Skip to content

WP-207 — pose, oriented and segmentation network replay kept under gradient accumulation

Engineering log · Phase 19 — Codebase audit remediation

🔗 Depends on WP-206 · Needed by WP-208, WP-209

⬅️ WP-206 · Roadmap · WP-208 ➡️

WP-207 — the accumulation retirement for pose, oriented and segmentation network replay removed, on the principal's answer to the 2026-10-20 escalation

Why the retirement existed. WP-193d kept pose replay native under gradient accumulation because the shared recorder's CUDA accumulation checks diverged on the unchanged baseline, and the oriented and segmentation replays inherited the same guard (_ACCUMULATION_EAGER_LABELS in ptl/module.py). The 2026-10-20 entry in ESCALATION asked whether detection had to stay native too. WP-206 showed the divergence was a defect in the recorder: the live-gradient copy had moved into a pre-hook inside the replayed backward. WP-206 moved it back to forward time.

The evidence the decision rests on. A probe on Colab (L4, never committed) emptied _ACCUMULATION_EAGER_LABELS and ran each task's own trainer parity scenario at accumulate_grad_batches=2. The probe required three things: replay records, nothing retires except for BF16 support, and the end state is bitwise equal to the native arm. That end state covers weights, optimizer and scaler, plus scheduler, EMA, losses and metrics for oriented and EMA for segmentation. There were 27 cases per arm: pose at three precisions and five layouts, oriented at three precisions and three layouts, and segmentation network replay at three precisions.

Tree Runtime Result
bb3c0b8 (WP-206) Python 3.11 / torch 2.9.1+cu129 27 passed
bb3c0b8 (WP-206) Python 3.13 / torch 2.11.0+cu130 27 passed
3cf592d (before WP-206) Python 3.11 / torch 2.9.1+cu129 18 failed, 9 passed

The 18 failures before the fix are real divergence, not probe errors:

  • 5 pose cases ended in a bitwise state mismatch.
  • 5 pose fits went non-finite: a keypoint Laplace density rejected its value.
  • 8 oriented and segmentation cases ended in a state mismatch.

The principal answered all three decisions of the escalation on 2026-10-23, and the entry records them. This package carries out the first: replay under accumulation is allowed for every task.

Change. _ACCUMULATION_EAGER_LABELS and DetectionLitModule._retire_accumulated_replay are removed, together with their two calls in _training_network and the segmentation feature route. The capture_network docstring no longer says pose, oriented or segmentation training stays eager under accumulation. Detection never had the guard. Segmentation head capture is unchanged: capture_segment_heads is still ignored at setup unless accumulate_grad_batches=1. It still copies live gradients from a backward pre-hook, the pattern WP-206 found wrong for the network recorder, and nothing has tested it under accumulation.

Tests. The existing parity tests already ran accumulation 2, and asserted the retirement there. They now assert replay and parity instead:

  • test_pose_trainer_matches_native[*-2-*]: expects 19 recordings and no retirement, and keeps the bitwise model, optimizer and scaler comparison.
  • test_oriented_trainer_preserves_native_state[2-*]: expects one recording, no retirement and the per-batch graph launch counts that accumulation 1 checks. It keeps the full state comparison.
  • test_segment_fit_replays_across_empty_partial_validation_and_ema:
    • Gains network-fp32-accumulated.
    • heads-bf16-accumulated now expects the head holder's setup warning and one feature recording, where the network replay used to retire.
    • The per-batch recording flags expect replay in both accumulation settings.
  • test_segment_accumulation_change_after_recording_retires_through_holder becomes ..._keeps_replay_and_matches_native. Accumulation rises from 1 to 2 after the recording is made, and the test asserts one recording, no retirement and bitwise-equal weights and optimizer state against a native fit with the same schedule.
  • The accumulation cases of test_pose_routing_retires_then_runs_native and test_oriented_routing_retires_then_runs_native are removed, since accumulation no longer retires.
  • The accumulate mode of test_segment_host_routing_and_native_fallback stays and now expects no retirement. On CPU, it asserts that the recorder is consulted under accumulation.

Docs. The CUDA performance page no longer says pose, oriented and segmentation replay stay native under accumulation. Its capture_segment_heads paragraph keeps accumulate_grad_batches=1. The three Unreleased CHANGELOG entries that said so are corrected in place, since none of them has shipped.

Verification.

  • CPU. tests/ptl gives 986 passed and 243 skipped (the CUDA ids), and make gate runs before the commit.
  • CUDA, colab_10. This ran on an L4 with the source and tests of this commit.
    • Python 3.11 / torch 2.9.1+cu129 ran every gpu or data test over tests and scripts, minus the four overfit runs and the full notebook run: 347 passed and 2 skipped in 12 min 10 s. The skips are a torch-version case in test_padded_musgd.py and a two-device case in test_replay.py.
    • Python 3.13 / torch 2.11.0+cu130 ran the tests/ptl accelerator selection: 241 passed.

Goldens. No frozen golden is touched. Replay is opt-in, and the overfit goldens run with capture_network off.

Limits.

  • Segmentation head capture under accumulation stays guarded, as above. Moving its copy to forward time and lifting its guard belong with the replay consolidation.
  • Accuracy and throughput under accumulation with replay are not measured. This package claims gradient and state equality with native training, on one GPU model, at accumulation 2.
  • WP-193c and WP-193d stay 🔄. Their DoD names runs that are still outstanding, and this package does not run them.