# 📦 WP-138 — `feat(eval): hotcoco as lucid-eval's default bbox/segm engine`

> Engineering log · [Phase 12 — Keypoint detection, v0.5.0](https://borda.github.io/lucid-YOLO/engineering/phase-12-keypoints/)
>
> 🔗 Depends on WP-043, WP-069, [WP-096](https://borda.github.io/lucid-YOLO/work_packages/wp-096/), [WP-134](https://borda.github.io/lucid-YOLO/work_packages/wp-134/), [WP-137](https://borda.github.io/lucid-YOLO/work_packages/wp-137/) · Needed by none
>
> ⬅️ [WP-137](https://borda.github.io/lucid-YOLO/work_packages/wp-137/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-139](https://borda.github.io/lucid-YOLO/work_packages/wp-139/) ➡️


## WP-138 -- a claim was wrong, and the fix it justified had already shipped

<a id="wp-138"></a>

**[WP-137](https://borda.github.io/lucid-YOLO/work_packages/wp-137/)'s own text claimed something false, and this row exists to say so plainly rather than quietly patch it.** That entry rejected a hand-driven `hotcoco` path for segmentation on the grounds that it "would cost back the memory-bounded streaming `_StreamingScorer` exists for" -- an assumption never checked against the code it was reasoning about. Checked here, against torchmetrics' actual `_get_safe_item_values` source (the function `_StreamingScorer`'s own docstring already cites): it RLE-encodes every mask **immediately inside `update()`** and discards the dense tensor the moment the batch is scored. Nothing about that is exclusive to torchmetrics' own machinery -- a hand-driven adapter can do the identical thing, encode-then-discard per batch, and only assemble the small already-encoded document at `compute()` time. The memory bound is about dense masks, not about "whole document at once," and conflating the two was the error. `docs/ASSUMPTIONS.md` A73 is revised in place rather than left standing, per this project's own rule that a wrong claim gets corrected where it was made, not silently worked around.

**The correction changed the answer, not just the reasoning.** With the memory objection gone, the remaining question was purely a feasibility one -- can hotcoco's actual API be driven correctly -- and the user asked for it reassessed with a harder constraint: even accepting a `MeanAveragePrecision` subclass if that is what it takes. Two things closed that question before any subclass was needed. First, `roboflow/rf-detr` merged PR #1402 in the interim (after [WP-137](https://borda.github.io/lucid-YOLO/work_packages/wp-137/)'s own research agent scanned `develop`'s merged commits, which is why it was missed then) -- a real, working `_HotCocoBackend(CocoBackend)` subclass: constructed under the accepted `"faster_coco_eval"` string (passes `CocoBackend`'s own validation), its `coco`/`cocoeval`/`mask_utils` properties overridden to return hotcoco's objects, and `_get_coco_datasets` (the assign-`.dataset`-then-`createIndex()` method hotcoco cannot support) skipped entirely on that path -- confirming a subclass *is* buildable, contradicting [WP-137](https://borda.github.io/lucid-YOLO/work_packages/wp-137/)'s blanket "cannot be a torchmetrics backend" framing. Their own measured verdict, independent of this project's: "not a training-throughput change... under 2% of an epoch," even with an array-fast-path optimization this project did not attempt. Second, this project has no DDP anywhere (`grep` across `cli/train.py`, `module.py`, `docs/TRAINING.md` -- nothing), which is `Metric`'s other real selling point beyond what a hand-driven scorer already gets for free by copying the same encode-then-discard pattern. Between a training-loop win both this project's own benchmark and rf-detr's independently measure as negligible, and a subclass whose main other benefit does not apply here, the subclass (WP B) stays deferred rather than built.

**What landed instead: hotcoco behind `lucid-eval`'s own `--eval_backend` knob, where rf-detr's own PR says it actually matters** -- "standalone evaluation of a checkpoint, large validation sets, sweeps that evaluate far more often than they train" is `lucid-eval`'s exact shape, not `module.py`'s per-epoch training loop, which stays on `faster_coco_eval` untouched. Default `"auto"` prefers hotcoco -- its wheels cover this project's real target platforms (`cp39-abi3`, macOS/Linux/Windows) -- and falls back to `faster_coco_eval` only if a runtime probe (`hotcoco_available()`: import, then construct an empty document, since a partially-broken compiled extension imports fine but cannot build anything) finds it unusable, recording the reason in the report's own `info` dict rather than only printing it. An explicit `--eval_backend hotcoco` is a stated requirement and raises rather than silently substituting -- the same "explicit override wins" shape `--data.layout` already has (A63). `--eval_backend faster_coco_eval` stays a fully-tested rollback, no reinstall, matching rf-detr's own choice to ship theirs the same way.

**Five real hotcoco compatibility traps, each verified directly against this project's own installed package rather than taken from rf-detr's PR on faith** (all five match their documented list exactly, which is itself worth recording -- independent verification, not a citation):

1. `params` is copy-on-read. `ev.params.recThrs = [...]` is a silent no-op; the whole object has to be reassigned (`p = ev.params; p.recThrs = ...; ev.params = p`). Confirmed directly: a mutate-in-place `maxDets` change did not survive, the reassignment did.
2. `dataset` is copy-on-read too, which is why `_HotcocoStreamingScorer` builds **two independent** ground-truth/result documents for a combined bbox+segm model (one box-area, one mask-area) rather than mutating one loaded document's `area` field between passes the way torchmetrics' own internal combined path does.
3. RLE `counts` comes back `bytes` from `faster_coco_eval`'s own `mask.encode`, and hotcoco's constructor decodes only the `str` form -- silently reading a bytes payload as an **empty mask**, no exception. Verified directly: a 9-pixel mask round-tripped through bytes scored 0 pixels; through `str` (`.decode("utf-8")`), 9. This is now centralized in one function (`_encode_mask_rle`) specifically so the fix cannot be half-applied at some call site that builds a record by hand.
4. `mask.encode` accepts `uint8` only, not the boolean arrays this project's mask tensors already are -- cast before encoding.
5. hotcoco writes its own stdout (a summary table) and stderr (one warning per evaluator parameter differing from COCO's defaults -- this project overrides `recThrs` and `maxDets`, so every call would otherwise warn) directly from Rust. Verified directly, the same way rf-detr's PR states it: wrapping a hotcoco `summarize()` call in `contextlib.redirect_stdout` left the table on the real terminal and an empty capture buffer. Fixed with descriptor-level `os.dup2` redirection (`_redirect_native_output`), scoped tightly around the three evaluator calls so a genuine failure -- hotcoco raises Python exceptions on error, never merely writes to descriptor 2 -- still surfaces normally.

**The acceptance instrument is `TestHotcocoParity`, not a benchmark.** Bit-for-bit (`abs=1e-6`, float32 precision noise) parity between `_StreamingScorer` and the new `_HotcocoStreamingScorer`, fed the *identical* accumulated predictions and targets: all 12 bbox statistics, all 24 bbox+segm statistics, the empty-preds all-zero case, and a full end-to-end `DualPathEvaluator` run through the real decode/letterbox pipeline on the detseg fixture (`test_dual_path_report_matches_across_backends`) -- the last one specifically to catch anything that only surfaces once real decoded detections, not a hand-built dict, reach the RLE/height-width/params machinery. `hotcoco>=0.5,<0.6` joins `pyproject.toml`'s core dependencies (a tight pre-1.0 ceiling, deliberate: three of the five traps above fail silently rather than raising, so each minor needs re-verifying against this same parity suite before it is trusted) -- `faster-coco-eval` stays required alongside it, since keypoints and `module.py` both still call it directly. `make gate` green, full suite plus the new parity class.