# WP-209 — the accelerator gate made green on an A100

> Engineering log · [Phase 19 — Codebase audit remediation](https://borda.github.io/lucid-YOLO/engineering/phase-19-codebase-audit/)
>
> 🔗 Depends on [WP-207](https://borda.github.io/lucid-YOLO/work_packages/wp-207/) · Needed by [WP-208](https://borda.github.io/lucid-YOLO/work_packages/wp-208/)
>
> ⬅️ [WP-208](https://borda.github.io/lucid-YOLO/work_packages/wp-208/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-210](https://borda.github.io/lucid-YOLO/work_packages/wp-210/) ➡️


<a id="wp-209"></a>

## WP-209 — the keypoint validation crash fixed, the pose replay comparisons bounded by what an A100 measures, the keypoint accelerator golden resealed, and the keypoint density and backbone attention kept finite under autocast

**Why this package exists.** The replay consolidation ran `make gate-gpu` on an A100-SXM4-40GB before its commit (colab_11, Python 3.11 / torch 2.9.1+cu129, never committed) and came back red. A control run (colab_12) repeated every red on `6109747`, the tree before it, with the same test ids and the same numbers, so none of them was the consolidation's. They were already on `main`, unseen because every earlier accelerator run had used an L4. The principal decided on 2026-10-24 to fix them first, in one package closed by one green `make gate-gpu`, with the replay consolidation waiting on it.

| Red | Where | Cause |
| -- | -- | -- |
| `test_runs_in_full[demo_keypoints]` | `_padding._host_split`: `torch.cat` "Expected size 17 but got size 0" in the validation sanity check | per-image keypoints concatenated without regard to shape |
| `test_runs_in_full[demo_keypoints]` at `16-mixed` | `RLELoss._per_point`, the flow's density | fp16 overflow under autocast of a finite float32 residual above 65504; surfaced in colab_14 only once part A let validation pass, under the same test id |
| `test_runs_in_full[demo_keypoints]` at `16-mixed`, validation | `SpatialAttention` logits in the P5 attention block | fp16 overflow under autocast of the attention logits before their scale, NaN through softmax; surfaced in colab_16 at the first validation after epoch 1, once part D let training pass |
| `test_pose_cuda_execution_keeps_objective_and_gradient_direction[2-True-True-32-true]`, `[…-16-mixed]` (Python 3.13 also `[…-bf16-mixed]`), `test_pose_cuda_execution_composes_with_network_replay` | `tests/ptl/test_pose_loss_execution.py` | bounds tighter than the arithmetic an A100 runs |
| `goldens/gpu/overfit_micro_kp.json` | `train_oks_ap` `0.582045` against `0.491935 ± 0.05` | trajectory spread, recorded in [ESCALATION](https://borda.github.io/lucid-YOLO/ESCALATION/) |

**A. An image without people in a keypoint batch.** `_host_split` ([WP-163](https://borda.github.io/lucid-YOLO/work_packages/wp-163/)) moves per-image tensors to the host with one `torch.cat` and slices the result. `Targets.empty()` stores keypoints as the canonical `(0, 0, 2)`, so beside an image with 17 points per person the concatenation fails, and a keypoint validation step crashed on the first image with no instances. Only tensors holding rows are concatenated now; a zero-row piece comes back as its own host copy with its own shape, so order and values are as before. The docstring states the exception and a doctest covers it. `TestHostSplit::test_zero_row_piece_keeps_own_trailing_shape` and `TestKeypointValidation::test_validation_step_accepts_image_without_instances` failed on `6109747` with the A100's error and pass now. The training step already handled the mixed batch; `TestTrainingStep::test_image_without_instances_beside_posed_image_trains` guards it.

**B. The pose replay comparisons.** Each test runs one fit natively and one with replay and compares loss and gradients. The gap was deterministic: a native fit repeated twice gives a gap of exactly zero, on every case, both runtimes. Three runs on the A100 (colab_13, colab_13b, colab_13c, never committed) took it apart.

- **TF32 explains the float32 single step.** cuDNN runs float32 convolutions in TF32 on an A100 by default, and an L4 run never did. With TF32 on, a native fit against itself with TF32 off already differs by 37,340 times the test's bound, and replay against native by 11 times. With TF32 off, replay sits at 0.14 of the bound. A `32-true` bound is a float32 claim, so the `true_fp32` fixture turns cuDNN TF32 off for both CUDA comparisons; it sets only the legacy `torch.backends.cudnn.allow_tf32` flag, because a partial set through torch's newer `fp32_precision` attributes makes the legacy getter Lightning reads raise.
- **Some gradients are zero in exact arithmetic.** With TF32 off, the composed test first failed on `backbone.attn_p5.blocks.0.attn.proj.1.bias`, a BatchNorm bias: 1.39e-6 against a floor of 1.0e-6. On CPU in float64 its gradient is 2.0e-15, against 6.3 for its weight, and 67 parameters of this model are below 1e-10. In float32 both arms hold rounding residue in multiples of 2^-23. The composed test's absolute floor goes from 1e-6 to the single-step test's 1e-5.
- **The rest is the rounding floor itself.** The noise floor was measured by scaling every head-output gradient by `1 + ε·U(-1, 1)`, a one-ulp change at the boundary loss replay works across, with five seeds per case, TF32 off:

| Case | Runtime | Per parameter (the test's ratio): replay / largest draw | Whole gradient, relative: replay / largest draw |
| -- | -- | -- | -- |
| float32 single step | 3.11 / 2.9.1 | 0.14 / 0.26 | 7.2e-7 / 9.8e-7 |
| float32 single step | 3.13 / 2.11.0 | 0.16 / 0.26 | 6.7e-7 / 1.1e-6 |
| 16-mixed single step | 3.11 / 2.9.1 | 464 / 798 | 2.6e-3 / 5.5e-3 |
| 16-mixed single step | 3.13 / 2.11.0 | 422 / 922 | 2.3e-3 / 6.5e-3 |
| bf16-mixed single step | 3.11 / 2.9.1 | 0.22 / 110 | 6.7e-6 / 3.2e-2 |
| bf16-mixed single step | 3.13 / 2.11.0 | 119 / 126 | 1.9e-2 / 4.9e-2 |
| composed, four steps | 3.11 / 2.9.1 | 42.3 / 43.0 | 2.3e-4 / 2.0e-4 |
| composed, four steps | 3.13 / 2.11.0 | 29.7 / 75.6 | 2.0e-4 / 4.0e-4 |

Replay sits at or below the largest draw on the test's own metric in every case, so its gap is the rounding a one-ulp change produces, not a different computation. One cell sits above its own runtime's largest draw: the composed run's whole-gradient distance on Python 3.11, 1.1 times it, and inside the other runtime's. The noise reaches backward only, so it leaves the loss untouched; replay's loss differences, at most 2.2e-5 relative, pass the tests' existing loss bounds. The noise also models loss replay better than network replay, whose forward kernels it does not perturb, so for the composed run the floor is an underestimate and replay sitting inside it is the stronger result.

Under autocast a per-parameter bound that admits the floor would test nothing: the zero-in-exact-arithmetic attention biases move by up to 922 times the bound under a one-ulp nudge, about 1.8 relative. The bounds therefore change shape where the floor demands it, each set at about 2.5 times the largest draw:

- **Single step, float32:** unchanged per parameter, plus the whole gradient within `ε`.
- **Single step, 16-mixed and bf16-mixed:** the whole gradient within `8ε` (largest draws 3.4ε and 3.2ε, replay 1.3ε and 1.2ε, with ε the test's `2 · finfo.eps`), the existing cosine bound `1 − ε` kept; no per-parameter bound.
- **Composed run:** step 0 keeps the per-parameter bound and the loss within `1e-5`. From step 1 on, both arms start from weights one optimizer update apart, and the whole gradient is held within `1e-3` (largest draw 4.0e-4) and the loss within `5e-5`. The noise moved a later step's loss by up to 1.7e-5, above the old `1e-5`, which replay passed only by its margin of 1.45.

What the whole-gradient bounds no longer catch is a defect confined to one small-norm parameter group under autocast or after step 0. The float32 single step and the composed run's step 0 carry that claim per parameter, with TF32 off.

**C. The keypoint accelerator golden.** `scripts/overfit_micro.py --task kp` scores `0.5820` on the A100 on `6109747` and twice on the replay consolidation tree, which sets no replay flag on that path, against the `0.491935 ± 0.05` sealed under torch 2.14.0+cu130. The other accelerator goldens pass on the same run. A seed sweep had measured this metric at `0.49`–`0.73` on unchanged per-step arithmetic. This is escalation trigger 4; the principal resealed it at `0.582045`, tolerance `0.05`, on 2026-10-24, and the [ESCALATION](https://borda.github.io/lucid-YOLO/ESCALATION/) entry of that date records it. `goldens/gpu/` is not under `goldens/frozen/MANIFEST.sha256`, so no manifest moves. The keypoint model card quotes the new figure.

**D. The keypoint density under autocast.** Once part A let validation pass, colab_14 reached a second failure under the same `test_runs_in_full[demo_keypoints]` id: pose training at `16-mixed` raised from the flow's standard-normal density. The colab_15 A100 probe measured it at training step 4, with batch 64, 640 px, learning rate 0.01 and COCO val2017 person keypoints. The float32 residual was finite, with absolute maximum 111554.7, and the flow parameters were finite; sigma's `1e-6` floor permits that residual, but autocast casts the flow's linear inputs to float16, whose maximum is 65504, so inversion becomes non-finite. The same residual inverted in float32 with autocast off was finite. Running `_per_point` in float32 with autocast off completed all 158 steps of the notebook's two epochs, with no non-finite value and a worst residual of `8.0e5`; validation was off (`limit_val_batches=0`), so this measures training only. Native `bf16-mixed` also completed 158 steps clean because its range is wider, though it computed the density in bf16.

The whole density now runs with autocast off: residual, Laplace term, flow log-density and log-sigma term are computed in at least float32, preserving float64 inputs. A CPU fp16-autocast regression covers finite loss and gradients for both native and paired scoring when the residual exceeds 65504. The CPU float32 path is unchanged because autocast is off there, so CPU goldens do not move; `goldens/gpu/overfit_micro_kp.json` does not move for part D either, because its script trains at the default `32-true`.

**E. The backbone attention under autocast.** With part D in place, colab_17 ran five arms on the A100, each a fresh process in the notebook's config (batch 64, 640 px, learning rate 0.01) with validation on. Native `16-mixed` reproduced at step 79, the first validation after epoch 1: `RLELoss._per_point` received NaN residuals, with 96% of `mu_hat` and `sigma_raw` NaN. Parameters and buffers were finite at validation start, with BatchNorm's minimum `running_var` at 0.0014. Forward hooks found the first non-finite module at `backbone.attn_p5.blocks.0.attn.proj.0`; `qkv` and `pe` stayed finite. `16-mixed` with validation under autocast off, `bf16-mixed` and `32-true` were each clean for 158 steps. The fifth arm, network replay, reported head outputs in float32 under autocast and did not discriminate; the discrimination comes from the finite state and the autocast-off arm.

A CPU probe with seed 0, `SpatialAttention(256).eval()` and `torch.randn(1, 256, 20, 20) * 100` puts the float32 logits' absolute maximum at 99756 and returns finite output. Under CPU fp16 autocast, 602 logits and the output become non-finite; at an input multiplier of 50, logits reach 25080 and the fp16 output is finite. The logits sum `key_dim` products and can exceed float16's 65504 before their scale is applied; softmax turns the resulting inf into NaN. Under mixed precision, a private autograd Function computes logits and softmax in at least float32 with autocast off in forward and backward, preserving float64 inputs. Backward recomputes the weights from detached saved queries and keys and uses `torch.autograd.grad`, rather than hand-written derivatives; weights and gradients return in their input dtypes, and attention is cast to the values' dtype for the second matmul. Without autocast, the float32 path is the original operation sequence, with no casts or autocast context: outputs and every parameter gradient stay bitwise identical, so no CPU or GPU golden moves (the GPU overfit goldens train at `32-true`). This block is in every task's backbone, so every task's mixed-precision forward changes.

On colab_18, F0 fail-fast stopped after 956 s with 8 failed and 53 passed. `demo_keypoints` and the pose loss tests passed; network capture's mixed-precision exactness tests failed. `torch.cuda.make_graphed_callables` records `torch.autograd.grad` without toggling autocast, and `network_capture.py` invokes it inside the caller's autocast state. The recorded backward therefore ran the float32 island's backward matmuls under autocast, while eager training calls `loss.backward()` outside autocast; replayed gradients differed by about 1e-3 relative, while float32 cases passed. A CPU proof with seed 0, `SpatialAttention(128).train()`, `torch.randn(2, 128, 8, 8)` and loss `out.float().square().mean()` found that backward inside versus outside fp16 autocast produced unequal parameter gradients, with maximum relative difference 0.23; the original operation sequence was bitwise equal. Disabling autocast explicitly in the Function's recomputed backward fixes this mismatch as well as retaining the finite forward.

Under `16-mixed`, GradScaler skipped 20 steps (2–18, 26, 28, 34), and its scale fell to 0.03125 by step 34, identically with and without the fix. This is recorded, not fixed, here.

**Verification.**

- **CPU.** `tests/ptl/test_module.py`, `test_keypoint_training.py` and `test_pose_loss_execution.py` pass with their CUDA ids skipped; `tests/losses/test_rle_loss.py::TestDensityAutocast::test_fp16_autocast_keeps_large_residual_loss_and_gradients_finite` covers native and paired scoring under CPU fp16 autocast. In `tests/models/test_blocks.py`, `test_spatial_attention_fp16_autocast_keeps_large_logits_finite` covers attention logits beyond float16's range; `test_spatial_attention_float32_matches_original_computation` guards bitwise outputs and every parameter gradient against the original float32 operation sequence; `test_spatial_attention_fp16_gradients_ignore_backward_autocast` guards bitwise parameter gradients with backward inside versus outside CPU fp16 autocast; and `test_full_precision_attention_weights_gradcheck` checks float64 derivatives numerically. The backward-autocast regression failed before the Function was added with `AssertionError: qkv.0.weight: backward autocast changed gradients`; `make gate` runs before the commit.
- **A100.** colab_19 (A100-SXM4-40GB, this commit's tree before this Verification text, never committed) ran a fail-fast arm first: the `demo_keypoints` full run plus `test_network_capture.py`, `test_segment_head_capture.py` and `test_pose_loss_execution.py`, 61 passed in 16 min. `make test-gpu` under Python 3.11.17 / torch 2.9.1+cu129 then passed 358 with 2 skipped in 4 h 04 min, every full notebook run included: `demo_obb` on DOTA-v1.0 (6395 s), `development_gates` (2876 s), `demo_keypoints` (612 s), `demo_segment` (560 s) and `demo_detect` (358 s). The two skips are `test_padded_musgd.py` (`grad_dtype` postdates torch 2.4) and `test_replay.py` (needs two CUDA devices). `make golden-gpu` passed 55 of 55, the resealed keypoint golden among them, and `tests/ptl -m "gpu or data"` under Colab's Python 3.13.16 / torch 2.11.0+cu130 passed 241.
- **The rounds before it.** colab_14 reached part D's red, colab_15 located it, colab_16 reached part E's red with `demo_obb` already green, colab_17 located it, and colab_18's fail-fast arm caught the first attention fix breaking capture exactness; parts D and E carry their measurements.

**Goldens.** `goldens/gpu/overfit_micro_kp.json` moves as part C says; no frozen golden is touched.

**Limits.**

- **One GPU model.** The bounds come from an A100 under two runtimes; another architecture may choose other kernels, and nothing here measures one.
- **Five noise draws.** The floor is the largest of five per case, not a distribution; the margin of about 2.5 is what absorbs that.
- **The Colab kernel configuration.** Colab's `/etc/ipython/ipython_config.py` sets `kernel_class google.colab._kernel.Kernel`, which leaks into a virtual environment's notebook kernel and fails the full notebook runs. The Colab runs move it aside and point `IPYTHONDIR` and `JUPYTER_CONFIG_DIR` at empty directories; that is the harness's problem, not the repository's, and nothing here changes for it.
- **The keypoint band.** The reseal calibrates one runtime. Whether the golden should hold a multi-seed band stays with the [WP-177](https://borda.github.io/lucid-YOLO/work_packages/wp-177/) entry.