# WP-207 — pose, oriented and segmentation network replay kept under gradient accumulation

> Engineering log · [Phase 19 — Codebase audit remediation](https://borda.github.io/lucid-YOLO/engineering/phase-19-codebase-audit/)
>
> 🔗 Depends on [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/) · Needed by [WP-208](https://borda.github.io/lucid-YOLO/work_packages/wp-208/), [WP-209](https://borda.github.io/lucid-YOLO/work_packages/wp-209/)
>
> ⬅️ [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-208](https://borda.github.io/lucid-YOLO/work_packages/wp-208/) ➡️


<a id="wp-207"></a>

## WP-207 — the accumulation retirement for pose, oriented and segmentation network replay removed, on the principal's answer to the 2026-10-20 escalation

**Why the retirement existed.** [WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/) kept pose replay native under gradient accumulation because the shared recorder's CUDA accumulation checks diverged on the unchanged baseline, and the oriented and segmentation replays inherited the same guard (`_ACCUMULATION_EAGER_LABELS` in `ptl/module.py`). The 2026-10-20 entry in [ESCALATION](https://borda.github.io/lucid-YOLO/ESCALATION/) asked whether detection had to stay native too. [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/) showed the divergence was a defect in the recorder: the live-gradient copy had moved into a pre-hook inside the replayed backward. [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/) moved it back to forward time.

**The evidence the decision rests on.** A probe on Colab (L4, never committed) emptied `_ACCUMULATION_EAGER_LABELS` and ran each task's own trainer parity scenario at `accumulate_grad_batches=2`. The probe required three things: replay records, nothing retires except for BF16 support, and the end state is bitwise equal to the native arm. That end state covers weights, optimizer and scaler, plus scheduler, EMA, losses and metrics for oriented and EMA for segmentation. There were 27 cases per arm: pose at three precisions and five layouts, oriented at three precisions and three layouts, and segmentation network replay at three precisions.

| Tree | Runtime | Result |
| -- | -- | -- |
| `bb3c0b8` ([WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/)) | Python 3.11 / torch 2.9.1+cu129 | 27 passed |
| `bb3c0b8` ([WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/)) | Python 3.13 / torch 2.11.0+cu130 | 27 passed |
| `3cf592d` (before [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/)) | Python 3.11 / torch 2.9.1+cu129 | 18 failed, 9 passed |

The 18 failures before the fix are real divergence, not probe errors:

- 5 pose cases ended in a bitwise state mismatch.
- 5 pose fits went non-finite: a keypoint `Laplace` density rejected its value.
- 8 oriented and segmentation cases ended in a state mismatch.

The principal answered all three decisions of the escalation on 2026-10-23, and the entry records them. This package carries out the first: replay under accumulation is allowed for every task.

**Change.** `_ACCUMULATION_EAGER_LABELS` and `DetectionLitModule._retire_accumulated_replay` are removed, together with their two calls in `_training_network` and the segmentation feature route. The `capture_network` docstring no longer says pose, oriented or segmentation training stays eager under accumulation. Detection never had the guard. **Segmentation head capture is unchanged:** `capture_segment_heads` is still ignored at setup unless `accumulate_grad_batches=1`. It still copies live gradients from a backward pre-hook, the pattern [WP-206](https://borda.github.io/lucid-YOLO/work_packages/wp-206/) found wrong for the network recorder, and nothing has tested it under accumulation.

**Tests.** The existing parity tests already ran accumulation 2, and asserted the retirement there. They now assert replay and parity instead:

- `test_pose_trainer_matches_native[*-2-*]`: expects 19 recordings and no retirement, and keeps the bitwise model, optimizer and scaler comparison.
- `test_oriented_trainer_preserves_native_state[2-*]`: expects one recording, no retirement and the per-batch graph launch counts that accumulation 1 checks. It keeps the full state comparison.
- `test_segment_fit_replays_across_empty_partial_validation_and_ema`:
    - Gains `network-fp32-accumulated`.
    - `heads-bf16-accumulated` now expects the head holder's setup warning and one feature recording, where the network replay used to retire.
    - The per-batch recording flags expect replay in both accumulation settings.
- `test_segment_accumulation_change_after_recording_retires_through_holder` becomes `..._keeps_replay_and_matches_native`. Accumulation rises from 1 to 2 after the recording is made, and the test asserts one recording, no retirement and bitwise-equal weights and optimizer state against a native fit with the same schedule.
- The accumulation cases of `test_pose_routing_retires_then_runs_native` and `test_oriented_routing_retires_then_runs_native` are removed, since accumulation no longer retires.
- The `accumulate` mode of `test_segment_host_routing_and_native_fallback` stays and now expects no retirement. On CPU, it asserts that the recorder is consulted under accumulation.

**Docs.** The CUDA performance page no longer says pose, oriented and segmentation replay stay native under accumulation. Its `capture_segment_heads` paragraph keeps `accumulate_grad_batches=1`. The three Unreleased CHANGELOG entries that said so are corrected in place, since none of them has shipped.

**Verification.**

- **CPU.** `tests/ptl` gives 986 passed and 243 skipped (the CUDA ids), and `make gate` runs before the commit.
- **CUDA, `colab_10`.** This ran on an L4 with the source and tests of this commit.
    - Python 3.11 / torch 2.9.1+cu129 ran every `gpu or data` test over `tests` and `scripts`, minus the four overfit runs and the full notebook run: 347 passed and 2 skipped in 12 min 10 s. The skips are a torch-version case in `test_padded_musgd.py` and a two-device case in `test_replay.py`.
    - Python 3.13 / torch 2.11.0+cu130 ran the `tests/ptl` accelerator selection: 241 passed.

**Goldens.** No frozen golden is touched. Replay is opt-in, and the overfit goldens run with `capture_network` off.

**Limits.**

- **Segmentation head capture** under accumulation stays guarded, as above. Moving its copy to forward time and lifting its guard belong with the replay consolidation.
- **Accuracy and throughput** under accumulation with replay are not measured. This package claims gradient and state equality with native training, on one GPU model, at accumulation 2.
- **[WP-193c](https://borda.github.io/lucid-YOLO/work_packages/wp-193c/) and [WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/) stay `🔄`.** Their DoD names runs that are still outstanding, and this package does not run them.