📦 WP-166 — fix(eval): crowd and area reach the default scorer¶
Engineering log · Phase 15 — Audit remediation, v0.8.0
WP-166 — the scorer that shipped by default was reporting a different protocol¶
lucid-eval picks its engine with eval_backend="auto", and auto resolves to hotcoco wherever it is usable, which is everywhere this project runs. That engine's ground-truth records were built by _accumulate_boxes, which hardcoded "iscrowd": 0 and recomputed "area" as box width times height. annotations.py has parsed both fields off the COCO file since it was written and hands them over in the target dict, and the identical dicts reach both scorers — so the default engine implemented neither COCO's crowd-ignore rule nor its small/medium/large partition, while the faster_coco_eval path fed the same targets implemented both.
The size of it. On a crowd case — one normal ground truth, one iscrowd=1 ground truth, one detection sitting inside the crowd region — hotcoco reported map 0.5050 against faster_coco_eval's 1.0000. On an area case whose annotation area is 200 under a box measuring 40000, the instance landed in the large bucket instead of the small one. On a real split this is systematic rather than marginal: instances_val2017.json carries roughly 1.6% crowd annotations, whose detections became false positives and whose instances became false negatives, and its area field is the segmentation area, always at or below the box area, so objects shifted small→medium→large and the three size statistics described a different partition of the split than COCO's.
The defect was this project's, not the library's. Driving _hotcoco_stats with hand-built records that do carry the fields returns COCO's answer exactly. hotcoco honours both; _accumulate_boxes never gave them to it.
Why the gate could not see it. Every fixture TestHotcocoParity fed to both scorers omitted iscrowd and area entirely, so torchmetrics defaulted area to the box area and crowd to zero — exactly what the hotcoco path hardcoded — and the two engines agreed by construction on the only two fields where they could differ. The CHANGELOG's "verified bit-for-bit against the existing engine" was true of a fixture family that could not express the divergence. That is the shape worth remembering: a parity test proves two implementations agree, not that either is right, and it proves nothing at all about a field no fixture carries.
The mask half was measured, not assumed. The class docstring claims the two engines differ in how they swap box area for mask area, which would have made the same fix wrong on the segm side. A probe whose annotation area (200), mask area (2025) and box area (40000) name three different buckets scored small on both halves — so torchmetrics reads the supplied area there too, and the annotation's value is correct on both paths.
What travelled with it. Four consistency items in the same module, each the same shape as the main defect — a number that depends on which library computed it. _MAX_DETS now reaches both engines instead of hotcoco alone; the keypoint detection cap and area partition are pinned beside the sigmas that were pinned for exactly that reason; the sigma comment's claim to equal the library default "exactly" was false at 1.39e-17 and now states the measured figure; and the OKS category table is built from ground truth alone, as COCO fixes it, rather than from the union with prediction labels.
One narrowing worth its own line. _redirect_native_output blinded file descriptors 1 and 2 for the whole evaluator call, discarding native warnings about exactly the malformed records this scorer builds. Running each phase with both descriptors captured separately showed every phase but summarize writing zero bytes — including a run whose ground truth carried a negative-width box, a negative area and an out-of-range iscrowd. Blinding the rest bought no silence and cost the one channel a malformed record would be reported on.