📦 WP-137 — feat(ptl): keypoint OKS joins the epoch validation loop¶
Engineering log · Phase 12 — Keypoint detection, v0.5.0
🔗 Depends on WP-077, WP-087, WP-102, WP-122, WP-132, WP-134 · Needed by WP-138
WP-137 — a signal that only ever existed downstream of the run it should have warned about¶
Started as a request to prototype hotcoco (github.com/derekallman/hotcoco), a Rust-backed reimplementation of pycocotools's COCOeval, as a faster replacement for faster_coco_eval. It ended somewhere else, and the two findings that moved it there are both worth keeping.
hotcoco cannot be a torchmetrics backend, and the reason is structural rather than a version gap. torchmetrics.detection.MeanAveragePrecision's CocoBackend builds an empty COCO(), assigns a plain dict to its .dataset attribute, and calls .createIndex() -- a construction contract shared by pycocotools and faster_coco_eval alike. Installed and inspected directly (Tier-1 evidence, not the README's own drop-in claim): hotcoco's COCO.dataset is a read-only Rust-bound attribute (AttributeError: attribute 'dataset' of 'builtins.COCO' objects is not writable), and no createIndex/create_index method exists at all -- confirmed even after calling hotcoco's own init_as_pycocotools() monkeypatch, which does make from pycocotools.coco import COCO resolve to hotcoco's class (verified via sys.modules inspection) but returns an object that still fails both calls. A working hotcoco integration exists -- COCO(full_dict) -> .loadRes(results) -> COCOeval(gt, dt, iou_type=...) -> .evaluate()/.accumulate()/.summarize(), verified end to end with correct numbers -- but it is the same hand-driven shape evaluate_keypoints already uses for faster_coco_eval, not a backend= value, and it costs two things a subclass does not: hotcoco's dict schema requires height/width per image unconditionally (torchmetrics' own bbox-only dataset builder never produces them), and hotcoco offers no incremental .update() -- the whole dataset must be built in memory before scoring. For segm that reintroduces exactly what _StreamingScorer's own docstring exists to prevent: a full split's masks held at once, "hundreds of gigabytes" in this module's own words.
Benchmarked anyway, since a real cost needs a real number rather than a plausible one. A synthetic bbox-only benchmark at COCO val2017 scale (5000 images, 34,227 instances, matched to REPRODUCTION_REPORT.md's own cited count) measured faster_coco_eval (via evaluate_bbox) at 1.160 s and a hand-driven hotcoco path at 0.089 s -- roughly 13x, and the two engines agreed to three decimals (AP 0.888 both). But MeanAveragePrecision.update()/.compute() runs once per validation epoch (module.py's self._val_map), inside a training run whose own per-epoch wall-clock is measured in minutes; ~1.07 s saved per epoch over a 50-epoch run is roughly a minute total against hours, not a bottleneck a training loop notices. Separately, roboflow/rf-detr's own PR #1375 ("consolidate COCO mAP evaluation", 3.28x CPU speedup measured in its own PR body) turned out not to apply here either, for a reason worth stating precisely rather than waved past: that PR fixes a cost specific to class_metrics=True on torchmetrics <1.9, and this project is on torchmetrics>=1.9 with class_metrics never set -- already the fast path that PR exists to reach. Conclusion: hotcoco is rejected (A73) -- not for lacking a benefit, but because the benefit it has does not clear the cost of reaching it.
What the research surfaced instead was a real gap, one already named earlier this session and left open. A task="keypoints" module's validation_step fell into the if self.task != "obb": branch and called self._val_map.update(...) with plain box/label dicts -- the same call a plain detection module makes. Nothing scored keypoints during training validation at all; the only place OKS was ever computed was the standalone lucid-eval report, run by hand, after the fact. A user direction mid-turn reframed the whole question around this: not "should det/seg's eval get faster," but "should keypoints and det/seg have one eval experience" -- and roboflow/rf-detr's own PR #1183 (keypoint_oks.py's MetricKeypointOKS, read in full via gh pr diff) turned out to be a real precedent, but one that validates the shape this project already had for keypoints rather than motivating a change to it: a hand-rolled reset/update/compute facade over a faster_coco_eval-backed evaluator, built for the identical reason -- their own docstring names it, citing upstream torchmetrics PR #3348 as the eventual real fix rather than attempting one themselves.
So the unification landed as: give keypoints the same per-epoch validation monitoring det/seg/obb already had, not a new abstraction to hold it in. DetectionLitModule already carries this exact pattern for "obb"'s WP-063 rotated mAP -- self._val_rotated_preds/self._val_rotated_targets as plain lists, accumulated per batch and scored once at epoch end, because "mAP is not a per-batch quantity that can be averaged" (the module's own words) and there is no torchmetrics.Metric shape to hold an epoch-whole matching. Keypoints' OKS is the identical case, so self._val_keypoint_preds/self._val_keypoint_targets follow it exactly rather than reaching for a wrapper class: _update_val_keypoints decodes the one-to-one branch's raw point offsets (decode_keypoints) and gathers each kept detection's own set by the anchor index the same top-k selection already returned (gather_keypoints -- coco_eval.py's _gather_keypoints, made public, since DualPathEvaluator and this module now both need the identical padding-row-zeroing contract and a second hand-copied version of it is exactly the drift WP-109's "three strides, two of them private" lesson warns against); _log_oks_map scores the epoch once via the existing, unmodified evaluate_keypoints and logs val/oks_mAP.
Unlike "obb", this sits beside val/mAP, not instead of it (WP-102's own distinction, restated for the new case): a keypoints module's box branch is a real person detector, orthogonal to its pose quality, the same relationship val/segm_mAP already has to val/mAP. The sigma vector is picked by point count rather than stated once: COCO_KEYPOINT_OKS_SIGMAS only when num_keypoints == 17 matches COCO's own schema exactly, A67's uniform SYMBOL_KEYPOINT_OKS_SIGMA for every other count -- the same choice scripts/overfit_micro.py's synthetic gate already made for its own fixture, now read from one place (coco_eval.py) instead of two independently-hardcoded 0.072 literals that could have drifted apart.
Verification. tests/ptl/test_keypoint_training.py::TestKeypointValidation -- one validation step populates both buffers and on_validation_epoch_end logs a finite val/oks_mAP and clears them; val/mAP and val/oks_mAP are both present, confirming the "beside" relationship; a monkeypatched evaluate_keypoints confirms the sigma ternary picks COCO_KEYPOINT_OKS_SIGMAS only at num_keypoints == 17 and SYMBOL_KEYPOINT_OKS_SIGMA otherwise. Full regression sweep (test_keypoint_training.py, test_coco_eval.py, test_module.py, scripts/_tests/test_overfit_micro.py -- the last one exercising SYMBOL_KEYPOINT_OKS_SIGMA from its new location) -- 75 passed. make gate green.