A Four-Second Avatar Is Not a Benchmark. That Is the Point.

The best avatar output we have seen from our data came with a label: pilot, not benchmark.

Last week Dawnena Key, Deaf engineer and founder of SonZo AI, published a case study on a sign language AI system built on CLERC data. It includes a four-second generated avatar. The hands, the motion, the face and the mouth are the best result I have seen from our data so far.

The case study calls it a successful pilot candidate, not a completed benchmark result. That sentence is the reason I wanted to write this post.

What we shared

CLERC has two kinds of data. There is Épée, the open subset on Hugging Face, 1,200 clips from 6 native Deaf signers, which anyone can download and build on. And there is the licensed corpus: recorded by us, annotated by us, with signers who signed an agreement and are paid for their work.

SonZo AI received a slice of the licensed corpus: 10 hours of consented footage from 7 native Deaf signers, delivered as keypoints with segment-level annotations, every signer pseudonymized. It is the first time someone outside CLERC has put the licensed corpus through a formal evaluation and published what they found.

The system itself is not ours. The recognition model, the rendering pipeline, the evaluation protocol are SonZo's work. CLERC provided the data layer. That division is how we think the field should be organized.

Two questions that fail differently

The case study starts from a distinction that most demos skip: recognition and generation are two different problems, and they break in two different ways.

Recognition asks whether a model can read a signer it has never seen. The trap is the data split. Split clips at random and the same signer lands in both training and test, so the model learns that person's hands rather than the language. The score looks great and means very little. The protocol here is seven-fold leave-one-signer-out: each of the 7 signers becomes the untouched test signer once, and the result is reported on every fold, not only the best one. The primary metric is macro-F1, so a few frequent signs cannot carry the number. Confidence and abstention are evaluated too, because a model that cannot say "I am not sure" is not ready for a Deaf user.

Generation asks whether a rendered avatar preserved the sign. The trap is polish. A video can look smooth and still have changed the hand path, the timing, the orientation, or dropped the eyebrows that turn a statement into a question. So the evaluation measures motion fidelity against the control trajectories, measures non-manual features separately (mouth, eyebrows, gaze, head), and leaves the final question, is this still the same sign and is it intelligible, to Deaf reviewers. That part is not automated and should not be.

The four-second video is versioned: renderer, seed, prompt and file hashes are retained, so the result can be reproduced. That is what makes it a candidate rather than a nice clip.

Seven gates, none of them green

The 7 Production Gates are Dawnena's framework for deciding whether an AI system is ready for production. Every readiness claim has to point to evidence, every gap has to be named, and a good artifact is never allowed to become a production claim on its own. The case study records where each gate stands today.

01. Purpose and Scope. Research, recognition, generation and approval are four separate boundaries, so a result in one cannot be quietly claimed for another. In progress.

02. Data and Evidence. Consented, pseudonymized data with documented ownership boundaries. On the CLERC side, consent and ownership are settled: every signer in the delivery signed an agreement and is paid for the work. What remains is documenting those boundaries inside SonZo's own review. In progress.

03. Architecture and Tool Authority. SonZo's components, the external renderer, and who validates what are traceable. Partial.

04. Safety and Governance. Two failure modes need explicit handling: a recognition the model is unsure of, and an output whose meaning has drifted. In progress.

05. Evaluation and Validation. The leave-one-signer-out protocol is defined. Running all seven folds and smoke-testing generation are still ahead. In progress.

06. Deployment and Release. Release thresholds, rollback rules and final approvers are not frozen yet. Not ready.

07. Observability and Continuous Improvement. Inputs, prompts, hashes, diagnostics and outputs exist. A single run registry tying them together is still missing. Partial.

Four in progress, two partial, one not ready. Then the decision, in one line: "Evidence is advancing. Production approval is not implied." The recognition protocol exists but has not been fully executed. The avatar is a reproducible pilot candidate, not proof that the rest of the vocabulary will keep its meaning once rendered. One good four-second clip does not generalize to a language.

I have read a lot of sign language AI announcements. I do not remember one that ended with "not ready" printed in the open. This one does, on our data, and I am glad it is written that way. A working demo matters. The evidence behind it matters as much, and when the users are Deaf, it matters more.

What this gives CLERC

An evaluation like this is not a verdict on the data. It is a map of where the data has to go next.

If a fold fails on one signer, that tells us which signing style is underrepresented. If the avatar loses the eyebrows, that tells us non-manual features need denser annotation. If a sign is confused with another across several signers, that is the next recording session.

We already know this loop works, because we run it on the public demo. Users flag wrong recognitions, the corrections become a priority list, we record what is missing. That is how the demo went from 15 signs to 50, then to 75, on signers the model had never seen, without changing the model once. Only the data changed. The SonZo evaluation runs the same loop on the licensed corpus, with an external evaluator, under a written protocol. Same mechanism, higher standard of proof.

That is the feedback we asked for when we started working with SonZo, and the reason the collaboration goes beyond a single delivery. We are partners on three fronts: research, because the evaluation protocol is something we want to reuse and share; business, because a licensed dataset with an honest external evaluation is worth more to every client than a dataset with a marketing number; and the Deaf community, because both companies are Deaf-led, and I have watched sign language get flattened by systems that were never tested on people like us.

Democratizing the benchmark

The long-term goal is simple to state. Anyone building a sign language model should be able to evaluate it the way this case study does: signer-disjoint, per-fold, calibrated, with Deaf review on the output. That standard is rare today because the data that allows it is rare. Most public corpora cannot hold out a full signer, either because they have too few signers or because the signers are interpreters translating English rather than Deaf people signing. CLERC is building the corpus that makes the standard possible. SonZo is building the review that makes it enforceable.

What comes next

The review names the next evidence, and it is four things, not one.

• All seven signer-disjoint recognition folds, reported fold by fold: they will tell us which signers and which signs the corpus is thin on.
• A fixed generation smoke set, so every rendering candidate is judged on the same cases: it becomes a standing test for every future delivery.
• Measurable acceptance thresholds, decided before the results come in, not after: a number to build toward instead of a feeling.
• A reviewer decision trail, so that when a Deaf reviewer accepts or rejects an output, the reason is recorded: Deaf judgment turned into a dataset of its own.

We will publish what comes out, including the folds that fail.

Read the Client A case study on SonZo AI.

Thank you, Dawnena, for the work and for putting the word "pilot" on a video most people would have called a launch.

Follow CLERC to track the results.