📦 WP-167 — ci(gates): the accelerator half gets a schedule and the floors a gate¶
Engineering log · Phase 15 — Audit remediation, v0.8.0
WP-167 — the accelerator half gets a schedule and the floors a gate¶
The Makefile said it was the schedule. gate-gpu's comment claimed the target "is that schedule", and nothing invoked it: four workflows, none with schedule, none with workflow_dispatch, none mentioning the accelerator. ci-tests.yml runs -m "not gpu and not data" and calls check_goldens.py without --include-gpu, so the whole goldens/gpu/ subtree and every marked test were unrun remotely — which is how the frozen detection overfit golden drifted for six days under WP-078, and why RESEARCH_LOG.md still records shapes_regression_det.json as outstanding. gate-gpu.yml is nightly at 17 4 * * * plus workflow_dispatch; the off-the-hour minute keeps it out of the top-of-hour queue spike.
Arming it is one repository action, deliberately outside the file. The job carries if: vars.GPU_RUNNER_LABEL != '' and runs-on: ${{ vars.GPU_RUNNER_LABEL }}. Unset, both triggers resolve to a visible skipped job rather than a run queued forever against a self-hosted label no machine answers to — the failure mode where a green Actions tab and an unrun gate look identical. GitHub-hosted runners have no GPU, so there is no default worth hard-coding. cancel-in-progress: false, because a half-killed training run leaves no signal about the goldens at all.
The run record is the gate's own stdout, not a second opinion. Clause 1 admits "a run recorded under .reports/", and the workflow tees make gate-gpu into .reports/gate-gpu/gate.log under set -o pipefail. It deliberately does not re-run check_goldens.py --include-gpu to produce the record: every producer there retrains a model, so recording the result that way would cost a second full gate and could disagree with the run that decided the verdict.
Three of four floors were enforced by a human at a prompt. The comparison lived once, in main(). run_overfit returned its score unchecked, so the seg, obb and kp golden producers and every marked test accepted a below-floor run; only detection had a test asserting a floor. The comparison moved into run_overfit, raising FloorNotMet on the raw unrounded score. It subclasses ValueError and pointedly not RuntimeError, because run_overfit catches RuntimeError to retry training non-deterministically and a floor miss must never be swallowed as a missing deterministic kernel. The ordering matters as much as the placement: the raise precedes the freeze branch, so --freeze is now unreachable on a failing run and a bad re-freeze cannot happen quietly. Under make golden-gpu a below-floor task surfaces through check_golden's broad handler as a clean producer raised FloorNotMet: ... line rather than a traceback; check_goldens.py itself was not touched.
Two bands admitted values the published DoD rejects. write_golden stored spec.tolerance verbatim, so overfit_micro_kp.json at 0.335667 ± 0.05 had a lower edge of 0.285667 against a 0.30 floor, and overfit_micro_obb.json at 0.938966 ± 0.05 an edge of 0.888966 against 0.90. A golden could pass while the acceptance criterion it exists to protect had failed. clamped_tolerance(score, spec) returns max(0.0, min(spec.tolerance, round(score - spec.floor, 6))) and write_golden calls it, so the clamp lives in the writer and a re-freeze produces a correct band — no stored JSON was edited. The six-decimal rounding is not cosmetic: the natural assertion score - tolerance >= floor failed on obb at one ulp (0.8999999999999999), and pinning the arithmetic to the precision the goldens actually store fixed the representation rather than widening the assertion. shapes_regression.py was checked for the same defect class and has tolerances but no acceptance floors at all, so there is no band to clamp; reported rather than changed.
What this row cannot close, and why it stays open. Clauses 3 and 4 — goldens/gpu/ re-frozen with every band above its floor, and make gate-gpu green — need CUDA. This tree has MPS. WP-169 moved the first-step objective weighting, so the five accelerator goldens that passed on an L4 against f8225af will not pass now, and the re-freeze must run after 169 and 172 rather than before. The row therefore lands its code with its status still open; the regenerated goldens and the flip are a second commit. The freeze order is overfit for each of det, seg, obb, kp and then shapes for det — five files, not four, shapes_regression_det.json being the one RESEARCH_LOG.md records as outstanding.
Measured, and left alone. The L4 reference run sat near 10% GPU utilization and 5% of 24 GB, and Lightning named both causes itself: num_workers=0 against 12 vCPU, and default matmul precision on a Tensor Core card. Neither is taken here. Both change the numerics and therefore move every accelerator golden, which is a decision this row has no standing to make while it is busy fixing the gate that would catch it. The cheapest change that is not a numerics change is running the four overfit_micro tasks concurrently — separate processes, ~5% of the card each — and that is recorded as a recommendation rather than applied.
The re-freeze, measured before it was performed. Clauses 3 and 4 closed on an L4. make golden-gpu ran first, against the goldens frozen before WP-169, and returned 46/48: overfit_micro_kp expected 0.335667 and got 0.574728, overfit_micro_seg expected 0.814623 and got 0.86491. That failure is the evidence the re-freeze was necessary, rather than an assumption that it must have been. The other three passed while still moving — detection 0.994932 to 0.996622, oriented 0.938966 to 0.964928, shapes_regression_det's NMS mAP50-95 0.882337 to 0.870817 — so the stored bands absorbed a real 0.026 shift on obb without reporting it. A band wide enough to hide the movement it was sized for is the same defect class as a band admitting a sub-floor value; the first is recorded here, the second is what clamped_tolerance fixes.
The clamp does not bite today, and saying otherwise would be false. Every re-frozen score clears its floor by more than its own tolerance: keypoints by 0.2747 against a 0.05 band, oriented by 0.0649, detection by 0.0466 against 0.04, segmentation by 0.1649. min(spec.tolerance, score - spec.floor) therefore returns spec.tolerance unchanged in all four cases, and shapes_regression_det has no floors to clamp against at all. The two edges this row was opened to close — keypoints admitting 0.285667 against a 0.30 floor, oriented admitting 0.888966 against 0.90 — are moot rather than tightened: the scores moved out of reach of their own lower edges. The clamp stands as the guard that will bite on the next regression, not as something that changed a stored band here.
What moved the keypoint number. train_oks_ap rose 0.239061, from 0.335667 to 0.574728 — a 71% relative gain on a tier that shipped at 0.5.0. That is WP-169 landing, and it is the first place in the repository where the size of that defect is visible as a number: the one-to-many branch's entire localization objective had been annihilated, train/o2m_box pinned at 4.12e-09 in tests/ptl/prechange_detect_step.json. The offline gate could show that the objective was suppressed; only the accelerator floors could show what suppressing it cost.
Two runs, one set of values. The freeze ran twice, in two independent Colab sessions on two separate L4 VMs, each reporting NVIDIA L4 and default highest matmul precision. Agreement is claimed at the precision each run actually recorded, which is not the same number in both cases. Within the second session it is exact: golden-gpu's pre-freeze diagnostics report got 0.574728 and got 0.86491, matching the files the freeze then wrote at all six stored decimals, so the measure step and the write step agree digit for digit. Across the two sessions it is four decimals, because the first session's record is overfit_micro.py's own stdout, which rounds — 0.5747, 0.8649, 0.9966, 0.9649 — and its six-decimal values were not preserved. Four independent decimals on four tasks is the checkable claim; "identical to every stored decimal" would not be. The SHA-256 of all five files matches between the session that produced them and this tree. The second session's make gate-gpu did not reach golden-gpu: pytest collection failed at tests/models/test_onnx_export.py:71 on import onnx, because that session installed pip install -e . without the dev dependency group where onnx and onnxruntime are declared, and Colab's own image supplies pytest but not onnx. That is a provisioning gap in the throwaway notebook, not a defect in the tree — .github/workflows/gate-gpu.yml:52 installs --group dev, which is why the workflow this row adds would not hit it. The green gate-gpu of record is the first session's complete run on this same tree: 6 marked tests passed in 4464 s, then 48/48 goldens passed, exit 0.