# Épée 200 · input and output format Version 2026-09-24 (scorer 2.1: optional confidence and UNKNOWN answers, section 6). Every number below was measured on the 200 Qualifier clips, and the Final is built by the same code. ## 1. One clip `features/.npy`, shape `(n_frames, 128, 3)`, float32, no NaN. | Points | Block | Source | |---|---|---| | `0..20` | hand slot A | 21 MediaPipe hand landmarks: 0 wrist, 1-4 thumb, 5-8 index, 9-12 middle, 13-16 ring, 17-20 pinky | | `21..41` | hand slot B | same 21 landmarks, other hand | | `42..63` | body | MediaPipe pose landmarks 11 to 32, in order. 42, 43 shoulders · 44, 45 elbows · **46, 47 wrists (pose 15, 16)** · 48 to 53 pinky, index and thumb knuckles · 54, 55 hips · 56 to 63 knees to feet | | `64..91` | eyes and mouth | face mesh 33, 133, 159, 145, 153, 144 (eye on the image-left side) · 362, 263, 386, 374, 380, 373 (eye on the image-right side) · 61, 291, 39, 181, 0, 17, 269, 405 (outer lips) · 78, 308, 82, 87, 13, 14, 312, 317 (inner lips) | | `92..127` | head outline | face mesh FACE_OVAL, 36 points: 10, 338, 297, 332, 284, 251, 389, 356, 454, 323, 361, 288, 397, 365, 379, 378, 400, 377, 152, 148, 176, 149, 150, 136, 172, 58, 132, 93, 234, 127, 162, 21, 54, 103, 67, 109. Point 92 is the top of the forehead, the list runs down the image-right side, point 110 is the chin, then back up the image-left side | Not in the file: pose landmarks 0 to 10 (nose, eyes, ears and mouth of the pose model) and the other 404 face mesh points. There is no nose point. No video, no pixels. **The 128 points are an index selection of the 1,629-value MediaPipe Holistic layout (pose 33, face 468, hand 21, hand 21). Nothing is re-estimated.** Checked against the source files of the 200 Qualifier clips: hands, body, eyes and mouth are bit-identical on 200 of 200, the head outline on 198, and within 2e-5 on the other two. ## 2. Coordinates - `x`, `y`: MediaPipe image-normalized. `x` = pixel column / image width, `y` = pixel row / image height, origin at the top left, `y` pointing down. - All 400 clips of both rounds are 1280 x 720. One unit of `x` is therefore 16/9 of one unit of `y`: multiply `x` by 16/9 before measuring a distance or an angle. - `z`: MediaPipe relative depth, smaller is closer to the camera. Each block has its own origin (pose: middle of the hips, hand: its own wrist, face: centre of the head), so `z` is not comparable across blocks and is not metric. `baseline.py` ignores it. - **No normalization is applied**: no centring, no scaling, no smoothing, no interpolation. Body-relative normalization is your side of the job. 6.3 % of the non-zero coordinates fall outside `[0, 1]`, almost all of them hips and legs, which MediaPipe extrapolates below the frame on a seated signer (`y` up to 3.3). Ignore points 54 to 63. ## 3. Handedness - The video is not mirrored. Pose "left" and "right" are the signer's own: pose 15 (point 46) is the signer's left wrist and sits on the image-right side on 99.6 % of frames. - **The two hand slots carry no stable left or right meaning across clips.** In the Qualifier as published, slot A is the hand at pose wrist 15 in 46 clips, the hand at pose wrist 16 in 140, and 14 clips do not have enough two-hand frames to tell. The order is stable inside a clip. The public Épée v0.3 release carries the same defect, so training data and exam match, and the fix is the same: decide once per clip, at read time, by anchoring each hand on the pose wrists (the hand whose root, point 0 or 21, sits on point 46 goes first). `baseline.py` shows a cruder variant that orders the hands by image side. - MediaPipe sometimes returns the same physical hand in both slots for a few frames. ## 4. Missing points A block that was not detected on a frame is **exact zeros** on all its points. Nothing is interpolated. In the Qualifier: hand slot A is present on 49 % of frames, slot B on 62 %, both on 36 %, shoulders to wrists on 100 %, the head outline on 100 %. A hand goes missing when it rests below the frame, moves fast, or overlaps the other hand or the face. ## 5. Frames and sign boundaries - One row per video frame, the whole clip, rest position before and after signing included. - 30 frames per second on all 400 clips (29.97 to 30.01). Still, never assume a constant: `manifest.json` gives `duration_s` and `n_frames` per clip. - Boundaries are in seconds. `a = round(start / duration_s * n_frames)`, `b = round(end / duration_s * n_frames) + 1`, frames `[a, b)`. ```json { "version": "epee200_v2_qualifier", "n_videos": 200, "vocab": ["..."], "videos": [ { "video": "EQ26_0001", "tier": 3, "duration_s": 5.367, "n_frames": 161, "segments": [ { "index": 1, "start": 0.5, "end": 0.967 } ] } ] } ``` `tier` is the sentence length: 1 = 2 to 3 signs, 2 = 4 to 5, 3 = 6 to 7, 4 = 8 and more (the score card reports top-1 per tier). `vocab` is the list of the 71 scored signs. A clip holds 2 to 12 signs. The Qualifier gives boundaries for 1,098 signs, of which 606 are scored: a position counts for macro-F1 and top-1 only if its reference gloss is one of the 71. You are not told which positions those are. Gloss BLEU is computed on the whole sentence with an open vocabulary. ## 6. predictions.json One entry per clip, all 200, one item per sign boundary, in order: ```json [ { "video": "EQ26_0001", "glosses": ["THANK-YOU", "YOU"] }, { "video": "EQ26_0002", "glosses": [["YOU", "YOUR", "HE"], ["WANT", "NEED", "LIKE"]] } ] ``` - An item is one gloss, or up to three ranked candidates, best first, which adds a top-3 score. The first candidate is the one used for macro-F1, top-1, BLEU and exact match. - Glosses are compared upper-cased and trimmed, exactly as spelled in `vocab`. - The long form `"segments": [{ "index": 1, "gloss": "..." }]` is also accepted. - 1 MB maximum (a full run is about 65 KB). An incomplete run is checked, never scored. 5 scored runs per hour, 20 per day. ### Abstention (optional, never ranked) ```json [ { "video": "EQ26_0001", "glosses": ["THANK-YOU", "UNKNOWN"], "confidence": [0.93, 0] }, { "video": "EQ26_0002", "glosses": [["YOU", "YOUR", "HE"], ["WANT", "NEED", "LIKE"]], "confidence": [0.71, 0.38] } ] ``` - `confidence`: one number from 0 to 1 per sign, for the first candidate. Long form: `{ "index": 1, "gloss": "...", "confidence": 0.93 }`. - `UNKNOWN` as the first candidate means your model abstains. It is never a reference gloss: it counts as wrong for top-1 and macro-F1, emits no token for BLEU and GER, and is never a false alarm. - With a confidence on every answered sign, the card orders the scored signs from most to least confident (UNKNOWN last) and reports: accuracy on the 80 % most confident, the largest share answered at 90 % accuracy, AURC (area under the risk-coverage curve, lower is better), and the false alarms left at 80 % coverage. Equal confidences share their accuracy: the order of the file never matters. - Always reported, confidence or not: **false alarms**, the share of the signs outside the 71 (492 positions in the Qualifier) where your first answer is one of the 71. - Nothing here changes macro-F1, top-1 or the rank.