# 📦 WP-137 — `feat(ptl): keypoint OKS joins the epoch validation loop`

> Engineering log · [Phase 12 — Keypoint detection, v0.5.0](https://borda.github.io/lucid-YOLO/engineering/phase-12-keypoints/)
>
> 🔗 Depends on WP-077, [WP-087](https://borda.github.io/lucid-YOLO/work_packages/wp-087/), [WP-102](https://borda.github.io/lucid-YOLO/work_packages/wp-102/), [WP-122](https://borda.github.io/lucid-YOLO/work_packages/wp-122/), [WP-132](https://borda.github.io/lucid-YOLO/work_packages/wp-132/), [WP-134](https://borda.github.io/lucid-YOLO/work_packages/wp-134/) · Needed by [WP-138](https://borda.github.io/lucid-YOLO/work_packages/wp-138/)
>
> ⬅️ [WP-136](https://borda.github.io/lucid-YOLO/work_packages/wp-136/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-138](https://borda.github.io/lucid-YOLO/work_packages/wp-138/) ➡️


## WP-137 — a signal that only ever existed downstream of the run it should have warned about

<a id="wp-137"></a>

Started as a request to prototype `hotcoco` (github.com/derekallman/hotcoco), a Rust-backed reimplementation of `pycocotools`'s `COCOeval`, as a faster replacement for `faster_coco_eval`. It ended somewhere else, and the two findings that moved it there are both worth keeping.

**`hotcoco` cannot be a torchmetrics backend, and the reason is structural rather than a version gap.** `torchmetrics.detection.MeanAveragePrecision`'s `CocoBackend` builds an empty `COCO()`, assigns a plain dict to its `.dataset` attribute, and calls `.createIndex()` -- a construction contract shared by `pycocotools` and `faster_coco_eval` alike. Installed and inspected directly (Tier-1 evidence, not the README's own drop-in claim): hotcoco's `COCO.dataset` is a read-only Rust-bound attribute (`AttributeError: attribute 'dataset' of 'builtins.COCO' objects is not writable`), and no `createIndex`/`create_index` method exists at all -- confirmed even after calling hotcoco's own `init_as_pycocotools()` monkeypatch, which does make `from pycocotools.coco import COCO` resolve to hotcoco's class (verified via `sys.modules` inspection) but returns an object that still fails both calls. A working hotcoco integration exists -- `COCO(full_dict)` -> `.loadRes(results)` -> `COCOeval(gt, dt, iou_type=...)` -> `.evaluate()/.accumulate()/.summarize()`, verified end to end with correct numbers -- but it is the same hand-driven shape `evaluate_keypoints` already uses for `faster_coco_eval`, not a `backend=` value, and it costs two things a subclass does not: hotcoco's dict schema requires `height`/`width` per image unconditionally (torchmetrics' own bbox-only dataset builder never produces them), and hotcoco offers no incremental `.update()` -- the whole dataset must be built in memory before scoring. For segm that reintroduces exactly what `_StreamingScorer`'s own docstring exists to prevent: a full split's masks held at once, "hundreds of gigabytes" in this module's own words.

**Benchmarked anyway, since a real cost needs a real number rather than a plausible one.** A synthetic bbox-only benchmark at COCO val2017 scale (5000 images, 34,227 instances, matched to `REPRODUCTION_REPORT.md`'s own cited count) measured `faster_coco_eval` (via `evaluate_bbox`) at 1.160 s and a hand-driven hotcoco path at 0.089 s -- roughly 13x, and the two engines agreed to three decimals (AP 0.888 both). But `MeanAveragePrecision.update()`/`.compute()` runs once per **validation epoch** (`module.py`'s `self._val_map`), inside a training run whose own per-epoch wall-clock is measured in minutes; ~1.07 s saved per epoch over a 50-epoch run is roughly a minute total against hours, not a bottleneck a training loop notices. Separately, `roboflow/rf-detr`'s own PR #1375 ("consolidate COCO mAP evaluation", 3.28x CPU speedup measured in its own PR body) turned out not to apply here either, for a reason worth stating precisely rather than waved past: that PR fixes a cost specific to `class_metrics=True` on torchmetrics `<1.9`, and this project is on `torchmetrics>=1.9` with `class_metrics` never set -- already the fast path that PR exists to reach. **Conclusion: `hotcoco` is rejected** (A73) -- not for lacking a benefit, but because the benefit it has does not clear the cost of reaching it.

**What the research surfaced instead was a real gap, one already named earlier this session and left open.** A `task="keypoints"` module's `validation_step` fell into the `if self.task != "obb":` branch and called `self._val_map.update(...)` with plain box/label dicts -- the same call a plain detection module makes. Nothing scored keypoints during training validation at all; the only place OKS was ever computed was the standalone `lucid-eval` report, run by hand, after the fact. A user direction mid-turn reframed the whole question around this: not "should det/seg's eval get faster," but "should keypoints and det/seg have one eval *experience*" -- and `roboflow/rf-detr`'s own `PR #1183` (`keypoint_oks.py`'s `MetricKeypointOKS`, read in full via `gh pr diff`) turned out to be a real precedent, but one that validates the shape this project already had for keypoints rather than motivating a change to it: a hand-rolled `reset/update/compute` facade over a `faster_coco_eval`-backed evaluator, built for the identical reason -- their own docstring names it, citing upstream torchmetrics `PR #3348` as the eventual real fix rather than attempting one themselves.

**So the unification landed as: give keypoints the same per-epoch validation monitoring det/seg/obb already had, not a new abstraction to hold it in.** `DetectionLitModule` already carries this exact pattern for `"obb"`'s [WP-063](https://borda.github.io/lucid-YOLO/work_packages/wp-063/) rotated mAP -- `self._val_rotated_preds`/`self._val_rotated_targets` as plain lists, accumulated per batch and scored once at epoch end, because "mAP is not a per-batch quantity that can be averaged" (the module's own words) and there is no `torchmetrics.Metric` shape to hold an epoch-whole matching. Keypoints' OKS is the identical case, so `self._val_keypoint_preds`/`self._val_keypoint_targets` follow it exactly rather than reaching for a wrapper class: `_update_val_keypoints` decodes the one-to-one branch's raw point offsets (`decode_keypoints`) and gathers each kept detection's own set by the anchor index the *same* top-k selection already returned (`gather_keypoints` -- `coco_eval.py`'s `_gather_keypoints`, made public, since `DualPathEvaluator` and this module now both need the identical padding-row-zeroing contract and a second hand-copied version of it is exactly the drift [WP-109](https://borda.github.io/lucid-YOLO/work_packages/wp-109/)'s "three strides, two of them private" lesson warns against); `_log_oks_map` scores the epoch once via the existing, unmodified `evaluate_keypoints` and logs `val/oks_mAP`.

Unlike `"obb"`, this sits **beside** `val/mAP`, not instead of it ([WP-102](https://borda.github.io/lucid-YOLO/work_packages/wp-102/)'s own distinction, restated for the new case): a keypoints module's box branch is a real person detector, orthogonal to its pose quality, the same relationship `val/segm_mAP` already has to `val/mAP`. The sigma vector is picked by point count rather than stated once: `COCO_KEYPOINT_OKS_SIGMAS` only when `num_keypoints == 17` matches COCO's own schema exactly, A67's uniform `SYMBOL_KEYPOINT_OKS_SIGMA` for every other count -- the same choice `scripts/overfit_micro.py`'s synthetic gate already made for its own fixture, now read from one place (`coco_eval.py`) instead of two independently-hardcoded `0.072` literals that could have drifted apart.

**Verification.** `tests/ptl/test_keypoint_training.py::TestKeypointValidation` -- one validation step populates both buffers and `on_validation_epoch_end` logs a finite `val/oks_mAP` and clears them; `val/mAP` and `val/oks_mAP` are both present, confirming the "beside" relationship; a monkeypatched `evaluate_keypoints` confirms the sigma ternary picks `COCO_KEYPOINT_OKS_SIGMAS` only at `num_keypoints == 17` and `SYMBOL_KEYPOINT_OKS_SIGMA` otherwise. Full regression sweep (`test_keypoint_training.py`, `test_coco_eval.py`, `test_module.py`, `scripts/_tests/test_overfit_micro.py` -- the last one exercising `SYMBOL_KEYPOINT_OKS_SIGMA` from its new location) -- 75 passed. `make gate` green.