# 📦 WP-167 — `ci(gates): the accelerator half gets a schedule and the floors a gate`

> Engineering log · [Phase 15 — Audit remediation, v0.8.0](https://borda.github.io/lucid-YOLO/engineering/phase-15-audit-remediation/)
>
> 🔗 Depends on [WP-170](https://borda.github.io/lucid-YOLO/work_packages/wp-170/), [WP-173](https://borda.github.io/lucid-YOLO/work_packages/wp-173/) · Needed by [WP-176](https://borda.github.io/lucid-YOLO/work_packages/wp-176/)
>
> ⬅️ [WP-166](https://borda.github.io/lucid-YOLO/work_packages/wp-166/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-168](https://borda.github.io/lucid-YOLO/work_packages/wp-168/) ➡️


## WP-167 — the accelerator half gets a schedule and the floors a gate

<a id="wp-167"></a>

**The Makefile said it was the schedule.** `gate-gpu`'s comment claimed the target "is that schedule", and nothing invoked it: four workflows, none with `schedule`, none with `workflow_dispatch`, none mentioning the accelerator. `ci-tests.yml` runs `-m "not gpu and not data"` and calls `check_goldens.py` without `--include-gpu`, so the whole `goldens/gpu/` subtree and every marked test were unrun remotely — which is how the frozen detection overfit golden drifted for six days under WP-078, and why `RESEARCH_LOG.md` still records `shapes_regression_det.json` as outstanding. `gate-gpu.yml` is nightly at `17 4 * * *` plus `workflow_dispatch`; the off-the-hour minute keeps it out of the top-of-hour queue spike.

**Arming it is one repository action, deliberately outside the file.** The job carries `if: vars.GPU_RUNNER_LABEL != ''` and `runs-on: ${{ vars.GPU_RUNNER_LABEL }}`. Unset, both triggers resolve to a *visible skipped job* rather than a run queued forever against a self-hosted label no machine answers to — the failure mode where a green Actions tab and an unrun gate look identical. GitHub-hosted runners have no GPU, so there is no default worth hard-coding. `cancel-in-progress: false`, because a half-killed training run leaves no signal about the goldens at all.

**The run record is the gate's own stdout, not a second opinion.** Clause 1 admits "a run recorded under `.reports/`", and the workflow `tee`s `make gate-gpu` into `.reports/gate-gpu/gate.log` under `set -o pipefail`. It deliberately does not re-run `check_goldens.py --include-gpu` to produce the record: every producer there retrains a model, so recording the result that way would cost a second full gate *and* could disagree with the run that decided the verdict.

**Three of four floors were enforced by a human at a prompt.** The comparison lived once, in `main()`. `run_overfit` returned its score unchecked, so the `seg`, `obb` and `kp` golden producers and every marked test accepted a below-floor run; only detection had a test asserting a floor. The comparison moved into `run_overfit`, raising `FloorNotMet` on the raw unrounded score. It subclasses `ValueError` and pointedly **not** `RuntimeError`, because `run_overfit` catches `RuntimeError` to retry training non-deterministically and a floor miss must never be swallowed as a missing deterministic kernel. The ordering matters as much as the placement: the raise precedes the freeze branch, so `--freeze` is now unreachable on a failing run and a bad re-freeze cannot happen quietly. Under `make golden-gpu` a below-floor task surfaces through `check_golden`'s broad handler as a clean `producer raised FloorNotMet: ...` line rather than a traceback; `check_goldens.py` itself was not touched.

**Two bands admitted values the published DoD rejects.** `write_golden` stored `spec.tolerance` verbatim, so `overfit_micro_kp.json` at `0.335667 ± 0.05` had a lower edge of `0.285667` against a `0.30` floor, and `overfit_micro_obb.json` at `0.938966 ± 0.05` an edge of `0.888966` against `0.90`. A golden could pass while the acceptance criterion it exists to protect had failed. `clamped_tolerance(score, spec)` returns `max(0.0, min(spec.tolerance, round(score - spec.floor, 6)))` and `write_golden` calls it, so the clamp lives in the writer and a re-freeze produces a correct band — no stored JSON was edited. The six-decimal rounding is not cosmetic: the natural assertion `score - tolerance >= floor` failed on `obb` at one ulp (`0.8999999999999999`), and pinning the arithmetic to the precision the goldens actually store fixed the representation rather than widening the assertion. `shapes_regression.py` was checked for the same defect class and has tolerances but no acceptance floors at all, so there is no band to clamp; reported rather than changed.

**What this row cannot close, and why it stays open.** Clauses 3 and 4 — `goldens/gpu/` re-frozen with every band above its floor, and `make gate-gpu` green — need CUDA. This tree has MPS. [WP-169](https://borda.github.io/lucid-YOLO/work_packages/wp-169/) moved the first-step objective weighting, so the five accelerator goldens that passed on an L4 against `f8225af` will not pass now, and the re-freeze must run *after* 169 and 172 rather than before. The row therefore lands its code with its status still open; the regenerated goldens and the flip are a second commit. The freeze order is `overfit` for each of `det`, `seg`, `obb`, `kp` and then `shapes` for `det` — five files, not four, `shapes_regression_det.json` being the one `RESEARCH_LOG.md` records as outstanding.

**Measured, and left alone.** The L4 reference run sat near 10% GPU utilization and 5% of 24 GB, and Lightning named both causes itself: `num_workers=0` against 12 vCPU, and default matmul precision on a Tensor Core card. Neither is taken here. Both change the numerics and therefore move every accelerator golden, which is a decision this row has no standing to make while it is busy fixing the gate that would catch it. The cheapest change that is *not* a numerics change is running the four `overfit_micro` tasks concurrently — separate processes, ~5% of the card each — and that is recorded as a recommendation rather than applied.

**The re-freeze, measured before it was performed.** Clauses 3 and 4 closed on an L4. `make golden-gpu` ran *first*, against the goldens frozen before [WP-169](https://borda.github.io/lucid-YOLO/work_packages/wp-169/), and returned **46/48**: `overfit_micro_kp` expected `0.335667` and got `0.574728`, `overfit_micro_seg` expected `0.814623` and got `0.86491`. That failure is the evidence the re-freeze was necessary, rather than an assumption that it must have been. The other three passed while still moving — detection `0.994932` to `0.996622`, oriented `0.938966` to `0.964928`, `shapes_regression_det`'s NMS mAP50-95 `0.882337` to `0.870817` — so the stored bands absorbed a real 0.026 shift on `obb` without reporting it. A band wide enough to hide the movement it was sized for is the same defect class as a band admitting a sub-floor value; the first is recorded here, the second is what `clamped_tolerance` fixes.

**The clamp does not bite today, and saying otherwise would be false.** Every re-frozen score clears its floor by more than its own tolerance: keypoints by `0.2747` against a `0.05` band, oriented by `0.0649`, detection by `0.0466` against `0.04`, segmentation by `0.1649`. `min(spec.tolerance, score - spec.floor)` therefore returns `spec.tolerance` unchanged in all four cases, and `shapes_regression_det` has no floors to clamp against at all. The two edges this row was opened to close — keypoints admitting `0.285667` against a `0.30` floor, oriented admitting `0.888966` against `0.90` — are moot rather than tightened: the scores moved out of reach of their own lower edges. The clamp stands as the guard that will bite on the next regression, not as something that changed a stored band here.

**What moved the keypoint number.** `train_oks_ap` rose `0.239061`, from `0.335667` to `0.574728` — a 71% relative gain on a tier that shipped at 0.5.0. That is [WP-169](https://borda.github.io/lucid-YOLO/work_packages/wp-169/) landing, and it is the first place in the repository where the size of that defect is visible as a number: the one-to-many branch's entire localization objective had been annihilated, `train/o2m_box` pinned at `4.12e-09` in `tests/ptl/prechange_detect_step.json`. The offline gate could show that the objective was suppressed; only the accelerator floors could show what suppressing it cost.

**Two runs, one set of values.** The freeze ran twice, in two independent Colab sessions on two separate L4 VMs, each reporting `NVIDIA L4` and default `highest` matmul precision. Agreement is claimed at the precision each run actually recorded, which is not the same number in both cases. Within the second session it is exact: `golden-gpu`'s pre-freeze diagnostics report `got 0.574728` and `got 0.86491`, matching the files the freeze then wrote at all six stored decimals, so the measure step and the write step agree digit for digit. Across the two sessions it is four decimals, because the first session's record is `overfit_micro.py`'s own stdout, which rounds — `0.5747`, `0.8649`, `0.9966`, `0.9649` — and its six-decimal values were not preserved. Four independent decimals on four tasks is the checkable claim; "identical to every stored decimal" would not be. The SHA-256 of all five files matches between the session that produced them and this tree. The second session's `make gate-gpu` did not reach `golden-gpu`: pytest collection failed at `tests/models/test_onnx_export.py:71` on `import onnx`, because that session installed `pip install -e .` without the `dev` dependency group where `onnx` and `onnxruntime` are declared, and Colab's own image supplies `pytest` but not `onnx`. That is a provisioning gap in the throwaway notebook, not a defect in the tree — `.github/workflows/gate-gpu.yml:52` installs `--group dev`, which is why the workflow this row adds would not hit it. The green `gate-gpu` of record is the first session's complete run on this same tree: 6 marked tests passed in 4464 s, then **48/48 goldens passed**, exit 0.