📦 WP-134 — feat(eval): keypoints wired into lucid-eval, canonical OKS ground truth¶
Engineering log · Phase 12 — Keypoint detection, v0.5.0
WP-134 — a checkpoint reported at one eightieth of its own score¶
WP-124 built the OKS scorer. WP-132 built the training path. Nothing ever joined them: evaluate_keypoints and keypoints_to_predictions had no caller anywhere outside their own module and its tests, and cli/eval.py branched on three tasks. WP-125's acceptance metric -- OKS keypoint mAP on COCO human pose -- was therefore not producible by any shipped command at all, which is the same shape of gap WP-132 opened with and the reason that row's own log ends where it does.
The 0.006288 was not a decode fault, and finding that out first is what kept this row small. A keypoints checkpoint run through lucid-eval fell through to detect_eval and reported map=0.006288 against a training loop that had logged val/mAP=0.48 at the same epoch. That reads as a broken decode, a broken un-letterbox, or a broken assignment, and any of the three would have made this a much larger row. It is none of them. detect_eval.run hardcodes annotations/instances_val2017.json regardless of what the checkpoint's task or class count is, and COCO's mAP averages average precision over every category the ground truth contains. A person-only checkpoint predicts contiguous class 0 and nothing else, so the other 79 categories each hold ground truth, receive no prediction, and score AP=0. The arithmetic closes: 0.006288 * 80 = 0.503, against the 0.48 the epoch metric logged. Confirmed independently on a synthetic three-category case through torchmetrics directly -- perfect predictions in one category of three give map=0.333 and map_per_class=[1.0, 0.0, 0.0] -- before any code here changed.
So the fix is the ground-truth file, not the mathematics. pose_eval.run reads person_keypoints_val2017.json, which names one category, and the dilution disappears as a side effect of scoring the protocol the checkpoint was trained for. detect_eval's behaviour is untouched: for a detect or segment checkpoint the GT category count matches the model's own, and its average is over exactly the right set.
Which row of the report compares. Training's val/mAP is logged from the o2o branch alone -- _val_decoder.decode_with_indices(head_out.o2o_cls, head_out.o2o_box, ...) against the datamodule's already single-class targets. The comparable figure in a new report is the e2e entry, and the nms entry is the dense branch's own answer, expected to sit above it by R1 sec. 4.4's deficit. The runner prints that in words, because a report that does not say which figure it quotes is how the oriented tier confused two of them for a whole release.
The keypoint mode enters DualPathEvaluator exactly where masks did, and differs where the pieces force it to. Structural detection off o2o_keypoints/o2m_keypoints rather than a flag, so a detect, segment or obb checkpoint provably takes the path it already took -- those fields are None on its head output. Per-branch and per-path, so the E2E path reads the one-to-one branch's points at the E2E decoder's own anchor indices and the dense path reads its own. Two differences from the mask case, both dictated rather than chosen: decode_keypoints composes the raw offsets with the whole (A, 2) anchor grid and has no per-detection indexing, so this path decodes the dense batch and then gathers, where the mask path gathers coefficients and then decodes; and COCO's keypoint protocol is one matching over one document with no incremental update, so the points are accumulated for the split and scored once where the boxes stream. The second is affordable only because the sizes are opposite -- a split of masks is hundreds of gigabytes and a split of points is a few megabytes.
The PAD_ANCHOR_INDEX sentinel is -1, which Tensor.gather accepts and answers with the last anchor's row. It is clamped to a valid position and the result zeroed, the same two steps _gather_coefficients takes, and those rows carry score 0 and are dropped regardless -- belt and braces on a failure whose symptom would be a plausible pose at a correct box.
Points invert the letterbox through Letterbox.inverse_map, which is already point-shaped: to_letterboxed_original is that same call plus the score and class columns of a detection tuple. Reaching for the box function would carry two columns onto coordinates, and writing a second affine would be a second definition of what a letterbox is, which is the rule WP-053a set.
The ground truth was the part that was quietly wrong, and it was wrong before this row. evaluate_keypoints reconstructed each instance's bbox and area from the extent of its visible keypoints and forced iscrowd=0 and num_keypoints to a recount. Every one of those four is a supplied field in COCO's protocol and each does real work: area is what OKS divides by and what draws the medium/large bucket boundary, and for a person it is the segmented area rather than the hull of the labeled joints; iscrowd and a declared num_keypoints == 0 are the two flags the backend marks an instance ignored on, verified by reading COCOeval_faster._prepare in the installed faster_coco_eval 1.7.2 rather than recalling pycocotools' behaviour. Reconstructing them scores a different protocol under the same name. They are now read from the annotation when the target carries them, annotations_to_target(with_keypoints=True) carries them, and the reconstruction stays as the fallback -- so WP-124's one-shot callers, its doctests and every synthetic fixture keep the exact numbers they had.
The regression test makes the two readings disagree on purpose: an instance whose visible points span 2500 px² and whose annotation declares 20000 belongs to the medium bucket under one reading and the large bucket under the other, and exactly one of the two can be populated. Reconstruction reports AP_medium=1.0, AP_large=-1.0; the annotation reports the reverse.
A K that is not 17 is refused rather than scored. This protocol's ground truth, its point ordering and its sigma table are all COCO's person schema. A checkpoint predicting some other K has no correspondence to it -- point i of the prediction is not point i of the annotation -- and R12's sigmas measure annotator variance on joints that are not the ones being predicted. That is A67's argument read from the other side: A67 already refuses to fabricate a per-point table for analytically-placed synthetic landmarks and scores them on its own uniform sigma in its own gate, so this row needs no new assumption and invents no table.
Validation is synthetic, and says so. There is no COCO checkout and no GPU on this machine, so nothing here ran against real data. Each claim is separated by construction instead: the two branches carry offsets of opposite sign and each anchor a different magnitude, so of the four combinations of branch and index the evaluator could take, only the right one scores OKS 1.0. Both mutations were checked to fail -- crossing the branches and dropping the padding clamp each turn the suite red. What that does not establish is the figure a real pose checkpoint will report; the first honest reading of this instrument is WP-125's, on real data, and the arithmetic above is a prediction of what it should show rather than a measurement of it.
One promotion rode along: _parse_keypoints in data/coco.py becomes parse_coco_keypoints, because the evaluation reader now parses the same field off the same file format and a second copy would be free to disagree about the triplet stride or the visibility dtype -- after which a training run and its own acceptance score would read different ground truth from one file.