# WP-199 — replay detection assignment and loss

> Engineering log · [Phase 18 — CUDA graph replay and pipeline throughput](https://borda.github.io/lucid-YOLO/engineering/phase-18-cuda-replay-throughput/)
>
> 🔗 Depends on [WP-198](https://borda.github.io/lucid-YOLO/work_packages/wp-198/) · Needed by [WP-199b](https://borda.github.io/lucid-YOLO/work_packages/wp-199b/), [WP-200](https://borda.github.io/lucid-YOLO/work_packages/wp-200/), [WP-200b](https://borda.github.io/lucid-YOLO/work_packages/wp-200b/)
>
> ⬅️ [WP-198](https://borda.github.io/lucid-YOLO/work_packages/wp-198/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-199b](https://borda.github.io/lucid-YOLO/work_packages/wp-199b/) ➡️


## WP-199 — the existing dynamic loss runs in a bounded CUDA workspace

<a id="wp-199"></a>

Our L4 CPU/CUDA profile identified repeated assignment and loss launch work after the already captured network. The new option replays that remaining detection objective, using this repository's existing expressions and R43's neutral runtime API.

`capture_loss`, off by default, replays the existing detection assignment and loss forward/backward through R43's graphed-callable API. It composes with network and orthogonalization replay and records no optimizer update. The private assignment entry point separates validation from unchanged arithmetic; the ordinary public assigner continues to validate exactly as before. Both branches share a label-space check when their class counts agree. Different class counts retain both checks, and invalid current labels still raise before input copying.

A ground-truth capacity grows to the next power of two of an observed count, replacing the previous graph rather than retaining a family of pools. Padding is zero boxes, zero labels and false masks. Added rows select no anchor and contribute zero to conflict counts and normalization maxima; original anchor rankings, ties and target order stay unchanged. Shrinking counts clear every stale row. An all-empty batch keeps the native empty-target route, including finite inputs whose intermediate area arithmetic overflows. This is a scheduling choice derived from actual sizes, with no method constant changed.

Masked padding needs finite arithmetic and an epsilon that remains positive in float32. Built-in assigners with subnormal/non-finite epsilon or invalid exponents keep the native route, including constructor-valid positive Python epsilons that underflow on the device. Current non-finite tensor inputs retain the native loss, including its finite empty-target classification result when only decoded boxes contain NaNs. The extra check runs outside recording.

Each result owns its scalar and assignment storage. An outstanding backward leaves another forward eager; the next automatic batch clears a hold left by a skipped backward. Fit completion, reuse, exceptions, deepcopy and pickle discard the recording. Global module hooks, partial batches, another CUDA stream, unsupported layouts, inference, anomaly detection and frozen predictions and unverified older BF16 runtimes retain native execution. Changed criterion implementations, scalar settings or precision retire recording. Capture failures clear the partial owner and raise. Recording collects unreachable graph cycles before opening capture, suspends automatic cycle collection until the first replay completes, then restores the caller's GC mode on success or failure.

Focused checks cover real CUDA scalar/assignment/gradient bits across target-count growth, shrinkage and empty batches; current-label errors; old-result ownership; partial batches; changed gains and criteria, including the native empty-target result; keyword-only custom losses; failed recording; the real CLI and a real automatic Trainer. Mutating away stale-row clearing or replacing replay with eager execution fails the dedicated regression suite. The full development gate and accelerator goldens are not part of this validation; this package remains open.

[MEASURED] Five alternating fresh-process pairs on an isolated g2-standard-8 (eight vCPUs, 32 GB), NVIDIA L4, driver 580.178.04, torch 2.9.1+cu129 and Lightning 2.6.6: complete fit time is **125.272 (124.536–126.679) s native** and **108.811 (108.582–109.466) s with replay**, a **13.140%** reduction (median and range). The workload is nano, batch 4, 640 px, BF16, five epochs on 1,024 COCO val2017 images used for training and 32 disjoint validation images, with four persistent training workers. Prepared caching, device HSV with its fused restore, epoch prefetch, network replay, orthogonalization replay and optimizer update replay are enabled in both arms. Timing includes construction, cache preparation, graph setup, training, schedules, EMA, validation and checkpoints. Dependency compiler and OS caches are warm; every fit rebuilds its image cache and CUDA recordings.

The model, full optimizer state, scheduler, EMA and complete loss tensor byte hashes match in each pair; validation metrics match too. The five native controls match each other. Actual replay plus native empty-batch counts cover every training batch. A separate batch-16 exploratory pair was 7.767% slower. The claim covers the measured small-batch workload. Long-run accuracy and full-COCO throughput remain unverified.

The final loss replay suite passes 59 cases on L4 torch 2.9.1 and 58 with one intentional BF16 skip on torch 2.4.0. Eight optimizer-flag CLI cases passed before the test-only composition addition, on the unchanged API. CUDA cases carry the GPU marker used by the project's test target. Real automatic Trainer coverage includes accumulation, clipping, FP16 overflow recovery, checkpoint plus assign=True restoration and simultaneous network/loss replay with both optimizer replay options. Seven mutation controls fail their dedicated regressions. The normal project environment passes the complete pre-commit set, including strict typing. Historical preliminary alpha compilation-counter and copied torch2.9 stub diagnostics reproduced on unchanged pre-merge main; these were not rerun on the merged source and neither is a green full-suite claim. Full development and accelerator-golden gates were not run; this package remains open.