📦 WP-192 — perf(eval): complete oriented scoring uses the worker policy¶
Engineering log · Phase 17 — R1 pose-objective parity
WP-192 — the complete oriented evaluator uses the worker policy it already had¶
Timing evidence is host-specific and non-gated. The off-repository procedures were profiling the public scoring loop, comparing five-run medians with channels_last, traversing the loader, and calling public run; the warm-up and repetition procedure is recorded below where available. The benchmark harness and raw output are not committed.
The standalone path predates the image-size target and inherited-validation guard. The evaluator's explicit num_workers=0 arrived in 0fe08f4 on 12 August. The free-/dev/shm cap on automatic workers was already present from 9f1fa80 on 2 August; the image-size validation worker target (9d0fa88) and inherited-validation guard (d09ded9) arrived on 13 August, one day after the call site. 00af9e4 on 6 September renamed _val_worker_cap to _val_worker_target and clarified its documentation without changing the policy. Nothing revisited the call site, so automatic worker selection could only govern datamodules whose callers left worker selection open. Through the inherited-validation guard, the evaluator's explicit zero was raised to one validation worker wherever /dev/shm exists. All measured "existing route" baselines below, including 0.850 s, 9.757 s and 13.104 s, therefore used one validation worker; the current low-startup route runs zero workers and is unmeasured. A profile of the public scoring loop over 117 real DOTA tiles spent 3.60 of 5.10 s waiting for the loader. The rotated accumulator took 0.051 s, so even removing it entirely could save only 1.0% of that run. Converting the standalone module to channels_last, the other live candidate, moved the five-run median from 2.679 to 2.645 s and overlapped both ranges; it was rejected.
Completeness is not itself the crossover. Starting eight workers to score only eight tiles took a 1.784 s median against 0.850 s on the existing route. A positive --limit therefore keeps that route unconditionally — the measurement is the smallest size the diagnostic path runs at, not a sweep, so a large positive --limit keeps the same route by scope rather than by a remeasured crossover. With no limit, the evaluator asks the datamodule for its existing policy, then keeps it only when the loader has more batches than workers: otherwise no process can be reused after its first batch. That structural guard covers the measured one-batch regression and a complete but tiny custom split without introducing a dataset-size constant. No crossover measurement was taken on macOS / MPS, where workers spawn rather than fork, and the band of batch counts just above the worker count is also unmeasured. Nothing in the reader, letterbox, model, decoder, merge or metric changes. Tests walk run through both public branches and the tiny-complete fallback, while a separate real-pool case reads the same twelve tiles through one and two workers and compares images, boxes, labels, rotated boxes, difficult flags, order and the complete report.
The loader result was repeated on an isolated CPU. Five measured traversals after one warm-up on an n2-standard-8 Ice Lake VM (Python 3.12.3, torch 2.14.1) moved from a 9.757 s median (9.651–9.987 s) with one validation worker to 2.658 s (2.609–2.781 s) with the policy's eight, a 72.96% median reduction. Every traversal yielded the same 117 tiles, 2,691 instances and tensor checksum. That figure is the loader alone: CPU-device inference was not measured. The policy's worker count derives from os.cpu_count(), which is blind to container CPU quotas and process affinity, so a host that limits either may start more workers than it can run. A quota- or affinity-aware count would change the datamodule policy that training shares and belongs in its own work package, not in WP-192.
The complete pipeline keeps the gain. Five measured public run calls after one warm-up on an NVIDIA L4 (driver 580.178.04, torch 2.14.1+cu130, batch 8) included loading, model inference, rotated decoding and scoring, whole-image merge and scoring, and report construction. The existing route took a 13.104 s median (12.909–13.253 s); the policy route took 7.830 s (7.336–8.090 s), a 39.66% median paired reduction. Each pair used the same freshly initialised oriented nano model over the same official-DOTA-derived 117-tile slice; metrics and report payloads were equal in these runs after removing the recorded seconds and the intentionally different worker-count diagnostic. The in-repository, gated CPU evidence of report equivalence is tests/eval/test_eval_obb.py::test_run_through_real_workers_matches_run_through_low_startup_route, which compares a real worker pool with the zero-worker route after removing those same fields; it does not establish CUDA determinism.
The report now says which count ran. A complete run's worker count is no longer the fixed 0 the report used to imply; it is whatever the datamodule's shared-memory- and image-size-aware policy picked on the host that ran it. SplitScoring and the JSON report's info.workers field now carry that count, read off the same loader score_split already builds, so a report is self-describing about which route scored it without reaching into the datamodule's private state. A failure of the policy route's worker pool itself, such as a worker that dies mid-pass and surfaces as the loader's RuntimeError, is not one of the pairing ValueErrors run turns into a FAIL: line: it reaches the terminal as a traceback, lucid-eval exits with status 1, and no report is written.
That field is an additive schema change, and its consumers were checked. A repository-wide search over tests/, notebooks/, docs/ and src/ found no oriented-report golden and no consumer comparing the whole payload or the exact info key set. The tests and the other task reports read named keys, while demo_obb.py and the documentation only produce or name obb_report.json. The regression therefore pins the new field directly; downstream code that selects existing keys is unchanged, while a consumer enforcing an undocumented closed schema would need to accept the additive key.
The floor moved to its exact live count, not a one-row nudge. _WP_FLOOR only tracks additions correctly when it is raised to the roadmap's exact live count — the same ratchet rule WP-152b and WP-165b record, and the one _ASSUMPTION_FLOOR was caught violating at 47 rows of slack. This row found _WP_FLOOR at 177 against a live count already at 191 before it, 14 rows of slack that would have let most of Phase 17 be deleted without the ratchet firing; it is now 192, the exact count after this row, leaving zero slack for the next one to inherit.