Skip to content

📦 WP-138 — feat(eval): hotcoco as lucid-eval's default bbox/segm engine

Engineering log · Phase 12 — Keypoint detection, v0.5.0

🔗 Depends on WP-043, WP-069, WP-096, WP-134, WP-137 · Needed by none

⬅️ WP-137 · Roadmap · WP-139 ➡️

WP-138 -- a claim was wrong, and the fix it justified had already shipped

WP-137's own text claimed something false, and this row exists to say so plainly rather than quietly patch it. That entry rejected a hand-driven hotcoco path for segmentation on the grounds that it "would cost back the memory-bounded streaming _StreamingScorer exists for" -- an assumption never checked against the code it was reasoning about. Checked here, against torchmetrics' actual _get_safe_item_values source (the function _StreamingScorer's own docstring already cites): it RLE-encodes every mask immediately inside update() and discards the dense tensor the moment the batch is scored. Nothing about that is exclusive to torchmetrics' own machinery -- a hand-driven adapter can do the identical thing, encode-then-discard per batch, and only assemble the small already-encoded document at compute() time. The memory bound is about dense masks, not about "whole document at once," and conflating the two was the error. docs/ASSUMPTIONS.md A73 is revised in place rather than left standing, per this project's own rule that a wrong claim gets corrected where it was made, not silently worked around.

The correction changed the answer, not just the reasoning. With the memory objection gone, the remaining question was purely a feasibility one -- can hotcoco's actual API be driven correctly -- and the user asked for it reassessed with a harder constraint: even accepting a MeanAveragePrecision subclass if that is what it takes. Two things closed that question before any subclass was needed. First, roboflow/rf-detr merged PR #1402 in the interim (after WP-137's own research agent scanned develop's merged commits, which is why it was missed then) -- a real, working _HotCocoBackend(CocoBackend) subclass: constructed under the accepted "faster_coco_eval" string (passes CocoBackend's own validation), its coco/cocoeval/mask_utils properties overridden to return hotcoco's objects, and _get_coco_datasets (the assign-.dataset-then-createIndex() method hotcoco cannot support) skipped entirely on that path -- confirming a subclass is buildable, contradicting WP-137's blanket "cannot be a torchmetrics backend" framing. Their own measured verdict, independent of this project's: "not a training-throughput change... under 2% of an epoch," even with an array-fast-path optimization this project did not attempt. Second, this project has no DDP anywhere (grep across cli/train.py, module.py, docs/TRAINING.md -- nothing), which is Metric's other real selling point beyond what a hand-driven scorer already gets for free by copying the same encode-then-discard pattern. Between a training-loop win both this project's own benchmark and rf-detr's independently measure as negligible, and a subclass whose main other benefit does not apply here, the subclass (WP B) stays deferred rather than built.

What landed instead: hotcoco behind lucid-eval's own --eval_backend knob, where rf-detr's own PR says it actually matters -- "standalone evaluation of a checkpoint, large validation sets, sweeps that evaluate far more often than they train" is lucid-eval's exact shape, not module.py's per-epoch training loop, which stays on faster_coco_eval untouched. Default "auto" prefers hotcoco -- its wheels cover this project's real target platforms (cp39-abi3, macOS/Linux/Windows) -- and falls back to faster_coco_eval only if a runtime probe (hotcoco_available(): import, then construct an empty document, since a partially-broken compiled extension imports fine but cannot build anything) finds it unusable, recording the reason in the report's own info dict rather than only printing it. An explicit --eval_backend hotcoco is a stated requirement and raises rather than silently substituting -- the same "explicit override wins" shape --data.layout already has (A63). --eval_backend faster_coco_eval stays a fully-tested rollback, no reinstall, matching rf-detr's own choice to ship theirs the same way.

Five real hotcoco compatibility traps, each verified directly against this project's own installed package rather than taken from rf-detr's PR on faith (all five match their documented list exactly, which is itself worth recording -- independent verification, not a citation):

  1. params is copy-on-read. ev.params.recThrs = [...] is a silent no-op; the whole object has to be reassigned (p = ev.params; p.recThrs = ...; ev.params = p). Confirmed directly: a mutate-in-place maxDets change did not survive, the reassignment did.
  2. dataset is copy-on-read too, which is why _HotcocoStreamingScorer builds two independent ground-truth/result documents for a combined bbox+segm model (one box-area, one mask-area) rather than mutating one loaded document's area field between passes the way torchmetrics' own internal combined path does.
  3. RLE counts comes back bytes from faster_coco_eval's own mask.encode, and hotcoco's constructor decodes only the str form -- silently reading a bytes payload as an empty mask, no exception. Verified directly: a 9-pixel mask round-tripped through bytes scored 0 pixels; through str (.decode("utf-8")), 9. This is now centralized in one function (_encode_mask_rle) specifically so the fix cannot be half-applied at some call site that builds a record by hand.
  4. mask.encode accepts uint8 only, not the boolean arrays this project's mask tensors already are -- cast before encoding.
  5. hotcoco writes its own stdout (a summary table) and stderr (one warning per evaluator parameter differing from COCO's defaults -- this project overrides recThrs and maxDets, so every call would otherwise warn) directly from Rust. Verified directly, the same way rf-detr's PR states it: wrapping a hotcoco summarize() call in contextlib.redirect_stdout left the table on the real terminal and an empty capture buffer. Fixed with descriptor-level os.dup2 redirection (_redirect_native_output), scoped tightly around the three evaluator calls so a genuine failure -- hotcoco raises Python exceptions on error, never merely writes to descriptor 2 -- still surfaces normally.

The acceptance instrument is TestHotcocoParity, not a benchmark. Bit-for-bit (abs=1e-6, float32 precision noise) parity between _StreamingScorer and the new _HotcocoStreamingScorer, fed the identical accumulated predictions and targets: all 12 bbox statistics, all 24 bbox+segm statistics, the empty-preds all-zero case, and a full end-to-end DualPathEvaluator run through the real decode/letterbox pipeline on the detseg fixture (test_dual_path_report_matches_across_backends) -- the last one specifically to catch anything that only surfaces once real decoded detections, not a hand-built dict, reach the RLE/height-width/params machinery. hotcoco>=0.5,<0.6 joins pyproject.toml's core dependencies (a tight pre-1.0 ceiling, deliberate: three of the five traps above fail silently rather than raising, so each minor needs re-verifying against this same parity suite before it is trusted) -- faster-coco-eval stays required alongside it, since keypoints and module.py both still call it directly. make gate green, full suite plus the new parity class.