# Opt-in CUDA performance flags

> **Opt-in, mostly CUDA only, mostly in progress.** Every flag on this page is off by default (or, for `fused_hsv`, on only where its toolchain exists). Most need a CUDA device and belong to a roadmap row that is still open ([ROADMAP](https://borda.github.io/lucid-YOLO/ROADMAP/) rows 192c to 200). Exceptions: the host-side image cache `cache_train_images` is built in the data module and needs no accelerator, and the rows for it ([WP-192b](https://borda.github.io/lucid-YOLO/work_packages/wp-192b/)) and `compile_step` (WP-184) are done. None of them is needed to train: the per-tier recipes in [Launching a training run](https://borda.github.io/lucid-YOLO/TRAINING/) work without any. Each gain below was measured on one workload and one device; check complete-fit throughput and memory on your own batch size before enabling a flag.

## The flags

The measured-gain column carries only figures stated in the sections below or in the linked work-package pages; a dash means no figure is stated there.

| Flag | Task | Default | Measured gain | WP |
| -- | -- | -- | -- | -- |
| `--data.device_hsv` | all (CUDA; `32-true`, `16-mixed`, `bf16-mixed`) | `false` | — (can help a CPU-limited run; image IPC and host-to-device bytes are four times the uint8 transport) | [WP-197](https://borda.github.io/lucid-YOLO/work_packages/wp-197/) |
| `--data.fused_hsv` | all (needs `device_hsv`; Linux NVIDIA compute capability 8.0 or newer, Triton and a host C compiler) | on where available | — | [WP-198](https://borda.github.io/lucid-YOLO/work_packages/wp-198/) |
| `--data.epoch_prefetch` (with `--data.persistent_workers true`) | all (single CUDA device, full epochs) | `false` | — (overlap measured on one L4 only) | [WP-198](https://borda.github.io/lucid-YOLO/work_packages/wp-198/) |
| `--data.cache_train_images` | all | `false` | — | [WP-192b](https://borda.github.io/lucid-YOLO/work_packages/wp-192b/) |
| `--data.cache_prepared_train_images` | all | `false` | — (3.30 MB per image measured on a 1,024-image slice, about 390 GB per process extrapolated for COCO train2017) | [WP-192c](https://borda.github.io/lucid-YOLO/work_packages/wp-192c/) |
| `--model.capture_segment_heads` | segment (with `--trainer.precision bf16-mixed`) | `false` | — (a small-batch result does not establish a large-batch speed-up) | [WP-193e](https://borda.github.io/lucid-YOLO/work_packages/wp-193e/) |
| `--model.capture_network` | detect | `false` | small-batch only (batch 4); batch-32 screens changed direction | [WP-193c](https://borda.github.io/lucid-YOLO/work_packages/wp-193c/) |
| `--model.capture_network` | keypoints | `false` | 12.97% saved, nano/640/BF16, batch 4, L4, five complete pairs, against shared-feature replay; reserved memory 1384 to 1730 MiB | [WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/), [WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/) |
| `--model.capture_network` | obb | `false` | 4.10% on a single batch-8 pair (Nano OBB, BF16, L4); peak reserved memory 3974 to 4300 MiB | [WP-200](https://borda.github.io/lucid-YOLO/work_packages/wp-200/) |
| `--model.capture_network` | segment | `false` | — | [WP-193g](https://borda.github.io/lucid-YOLO/work_packages/wp-193g/) |
| `--model.capture_gradient_norm` | all (single CUDA device, norm clipping, torch>=2.9) | `false` | — (flag-only share unmeasured) | [WP-200c](https://borda.github.io/lucid-YOLO/work_packages/wp-200c/) |
| `--model.capture_loss` | detect, keypoints | `false` | 13.14% saved, nano/640/BF16, batch 4, L4, five complete-fit pairs; a batch-16 exploratory pair was 7.767% slower | [WP-199](https://borda.github.io/lucid-YOLO/work_packages/wp-199/), [WP-199b](https://borda.github.io/lucid-YOLO/work_packages/wp-199b/) |
| `--model.capture_orthogonalization` | all (MuSGD) | `false` | — | [WP-197b](https://borda.github.io/lucid-YOLO/work_packages/wp-197b/) |
| `--model.capture_updates` | all (MuSGD, eligible dense float32 CUDA groups) | `false` | — | [WP-197c](https://borda.github.io/lucid-YOLO/work_packages/wp-197c/) |
| `--model.compile_step` | detect | `false` | x1.23 on the training step (74.7 ms eager, 60.9 ms compiled; RTX PRO 6000, variant `n`, COCO, batch 16, deterministic); 110–140 s compile per process | [WP-183](https://borda.github.io/lucid-YOLO/work_packages/wp-183/), D24 in [Decisions](https://borda.github.io/lucid-YOLO/DECISIONS/) |

Two combination rules apply across the table. `capture_network` and `capture_loss` both refuse `compile_step`, and `capture_network` raises at construction for a task outside detect, keypoints, obb and segment. `capture_segment_heads` and `capture_network` can be combined for segmentation, and `capture_loss` can combine with `capture_network`, `capture_orthogonalization` and `capture_updates`.

## What happens off CUDA, or off the supported case

Each row falls back with a `UserWarning` rather than failing, except where the last column says it raises. The first two rows are not flags: they apply to every run.

| Feature | CUDA | CPU | MPS | When the case is not met |
| -- | -- | -- | -- | -- |
| `channels_last` memory format (D23) | applied automatically to parameters and images, no flag | default layout | default layout | not applicable; nothing to configure |
| `deterministic: true` | strict `True` | strict `True` | `"warn_only"` (some backward kernels have no deterministic MPS implementation) | see [What `deterministic: true` and `seed` actually pin](https://borda.github.io/lucid-YOLO/TRAINING/#what-deterministic-true-and-seed-actually-pin) |
| `compile_step` | detect only | trains eager, nothing to gain | trains eager, Inductor crashes in `convolution_backward` | warns once at setup and trains eager; the alpha scalar lifted for the compiled region (`_alpha_tensor`) stays on the host on MPS, which has no `float64` |
| `device_hsv` | with `32-true`, `16-mixed` or `bf16-mixed` | worker-side HSV | worker-side HSV | warns once and keeps worker-side HSV, as do precision plugins that cast the images or gains before the data hook |
| `fused_hsv` | Linux, compute capability 8.0 or newer, Triton and a host C compiler | native restore | native restore | native CUDA restore; a failing first compile or launch warns once and keeps it for the process |
| `epoch_prefetch` | single device, full epochs, module's own shuffle generator, persistent workers | ordinary loader | ordinary loader | ordinary loader; warns once naming the first unmet condition |
| replay flags (`capture_network`, `capture_loss`, `capture_segment_heads`) | single device, automatic optimisation, supported task, BF16 needs a final PyTorch 2.9 or newer | warn and stay eager | warn and stay eager | other strategies warn and stay eager; an unsupported task or `compile_step` raises at construction; a recording failure raises |
| `capture_orthogonalization`, `capture_updates` | eligible dense float32 groups | original functions / native step | original functions / native step | native step; a recording failure raises (`capture_updates`) |

## CUDA colour augmentation

`--data.device_hsv true` moves training HSV and quantization to CUDA with `32-true`, `16-mixed` or `bf16-mixed` precision. The workers still sample the same gains and horizontal flips, and validation keeps its existing path. The default is `false`; other backends and precisions warn once and keep worker-side HSV. Any precision plugin whose input conversion casts the float32 images or float64 gains before the data hook (true-half and double, and the FSDP and DeepSpeed plugins under every accepted precision) keeps the original path as well, with the same one-time warning.

On Linux NVIDIA GPUs with compute capability 8.0 or newer, the float32 consumer restore can fuse the existing colour operations and quantization into one channels-last write. This is on by default wherever Triton and a host C compiler are present — torch's CUDA wheel already selects a Triton, so an existing `device_hsv` run there takes the fused path without installing anything (`uv pip install -e ".[cuda-hsv]"` adds Triton where it is missing). `--data.fused_hsv false` keeps `device_hsv` on the native CUDA restore, to step around a toolchain or bisect a difference between the two. The first call compiles the kernel and needs a host C compiler. The extra also installs the existing Setuptools build dependency needed by older Triton SDKs. A Triton `ImportError`, a missing compiler, other devices, noncontiguous buffers and other consumer precisions retain the native restore. So does a first compile or launch that fails: it warns once with its cause, and the native restore runs for the rest of the process. Gain sampling, rounding and the uint8 boundary are unchanged. Direct `unpack_batch` calls keep their existing path.

This can help a CPU-limited training run, including one with decoded images cached by `--data.cache_train_images true`. It transports float32 images until the colour transform finishes, so image IPC and host-to-device bytes are four times the default uint8 transport. The shared-memory worker cap budgets float32 images only when the module chooses the worker count (`--data.num_workers` unset) in one process; segmentation masks still need their own memory allowance. A named `--data.num_workers` is neither capped nor warned: batch 64 at 640 px with 8 workers and `prefetch_factor` 2 queues about 5.0 GB of `/dev/shm` in float32, against 1.26 GB in uint8. Each DDP rank budgets the free `/dev/shm` on its own, so ranks sharing one host can jointly overcommit it — a known limit. The cap cannot shrink a batch: with a small `/dev/shm`, enlarge it or use `--data.num_workers 0`. Measure the complete run on the intended host.

## Prepare the next epoch during validation

`--data.epoch_prefetch true` with `--data.persistent_workers true` lets the existing workers prepare up to `num_workers x prefetch_factor` batches of the next training epoch during the last step and validation. It is opt-in because the training queue can remain resident alongside validation inputs: `num_workers x prefetch_factor` batches of host memory, about 1.26 GB of uint8 images for the batch 64, 640 px, 8 worker, `prefetch_factor` 2 example above, or about 5.0 GB of float32 with `--data.device_hsv true`. With pinned host memory, which is on by default when CUDA is available, queued batches can sit in page-locked RAM that the `/dev/shm` worker cap does not budget. Use it when dataset and augmentation configuration remain stable between epochs. Every packaged config breaks that: its `CloseMosaicCallback` switches mosaic off for the final epochs through `set_mosaic_prob`, which persistent workers never see, because each keeps the pipeline copy it started with — those epochs would still train on mosaics. The module warns once when such a change arrives under `epoch_prefetch`; keep `--data.persistent_workers false` for these configs. It applies to full, single-CUDA epochs with the module's own shuffle generator; batch limits, a step bound, overfit, loader reloads and distributed training keep the ordinary loader, as do non-persistent workers, and the run warns once naming the first unmet condition. CUDA is the boundary because the overlap was measured only on one CUDA GPU (an L4); other accelerators are unmeasured, not known to fail. The configured final epoch does not prepare another one. Measure the complete fit: the benefit is greatest when first-batch preparation is a meaningful fraction of the epoch.

## Decoded training-image cache

`--data.cache_train_images true` decodes each distinct training file name during datamodule setup and keeps the RGB `uint8` tensor in host RAM. Mosaic, affine, colour and flip augmentation still run on fresh float storage for every draw; only the repeated JPEG/PNG decode disappears. COCO and YOLO training readers support the flag, and validation is deliberately not cached.

Budget the option from the uncompressed images, not their files: approximately `sum(3 * height * width)` bytes for 8-bit sources, before Python/tensor metadata — a 16-bit PNG decodes to `uint16` and costs two bytes per channel, so twice that — and once per device: a multi-GPU run spawns one process per rank, each building its own `DetectionDataModule` and therefore its own cache, so the single-process budget above multiplies by `trainer.devices`. The cache is keyed by file name (by path for the YOLO reader), so two records naming one file share an entry, while the same picture stored under two names is decoded and held twice. It defaults off.

Whether DataLoader workers share the cache depends on the multiprocessing start method, not on the operating system, and the data module leaves the start method at the platform default. Under `fork` the workers read the parent's cache through copy-on-write pages, so it costs one copy of RAM however many workers there are. Under `spawn` and `forkserver` — the default on macOS and Windows, and on Linux from Python 3.14 — the cache is pickled to every worker, torch moves each cached tensor into shared memory to do it (one segment or file descriptor per image), and with `--data.persistent_workers false`, the default, the hand-over repeats every epoch. The values are identical either way; a non-fork start method does not promise the one-copy cost or the speedup, and a large split can meet file-descriptor or `/dev/shm` limits at a size nobody has measured. The pipeline never writes to the cache — every draw converts into fresh float storage. Leave it off when the split does not fit comfortably beside the loader queue, model, optimizer and operating-system cache.

### Prepared training-source cache

`--data.cache_prepared_train_images true` also reuses float conversion and affine centring before random augmentation. It takes precedence over `cache_train_images`, keeps one float32 source per training record and snapshots YOLO labels with their images. Mosaic, geometry, colour, targets and RNG draws keep their existing results. The option defaults off.

Budget about `12 * sum(height * width)` bytes for the retained RGB pixels, plus target and reader metadata, per training process. Extrapolated from the 1,024-image slice measured in [WP-192c](https://borda.github.io/lucid-YOLO/work_packages/wp-192c/) (3.30 MB per image), the full COCO train2017 split would need about 390 GB per process; this is an extrapolation, not a measured run. Every DDP rank is a separate process and holds its own copy, so ranks on one host multiply that figure. Each float32 pixel channel uses four times the bytes of an 8-bit channel. The prepared cache retains separate images for duplicate COCO records, whereas the decoded cache deduplicates file names. COCO reports an estimate before decoding; YOLO has no annotation sizes and reports the retained pixel bytes afterwards. Under `spawn` or `forkserver` (see above), workers receive cached tensors through shared memory, so budget the cache alongside the queued batches. Forked workers share its read-only pages. Fit-stage setup builds the cache once, repeated fit setup reuses it, and other stages release the datamodule's references. Measure the complete fit on the intended host.

## Independent segmentation predictions

`--model.capture_segment_heads true --trainer.precision bf16-mixed` records the existing segmentation prediction routines, including their first-order backward work. The default is `false`. Box/class stems remain independent; coefficients, prototypes and semantic logits have separate backward nodes so an empty mask objective keeps absent gradients. With this option alone, common features, prototype fusion, assignment, losses, state updates and schedules stay native. Enable `capture_network` separately to replay the common features too.

Byte-exact agreement with native training is verified only on NVIDIA L4 (sm_89) with torch 2.9.1+cu129; other GPUs and torch builds are unverified. The route requires PyTorch 2.9 or newer, one CUDA device and automatic optimisation. Gradient accumulation replays too ([WP-208](https://borda.github.io/lucid-YOLO/work_packages/wp-208/)): a live gradient is copied out of a group's recorded gradient buffers at forward time, before the replay that would overwrite them, as network replay does. CPU, MPS, FP32, FP16 and older BF16 runtimes warn and use native calls; evaluation stays native too. Custom or changed prediction dispatch, hooks, frozen prediction parameters, aliased storage and retained feature gradients retire replay for the fit. Each prediction group records one input signature per fit (shape, stride, dtype, device, gradient flag and CUDA stream); a call with any other signature, a partial batch included, runs the native head without adding another graph, and the recorded graph pools stay reserved until the fit ends, so a run whose batch shapes keep changing pays that memory without replaying. Copying, checkpoints and fit teardown retain the original parameter names and no recording.

A recording failure raises after restoring running buffers and RNG state; disable the flag to keep that run native. The flag retains CUDA graph pools, so check reserved memory and throughput on the actual batch size. A replayed step's mask coefficients, prototypes and semantic logits live in CUDA graph memory that the next replay overwrites in place; box and class predictions are copies. Anything that keeps those three tensors, or a gradient adopted from their backward, beyond the training step that produced them has to clone it. Measurements and rejected alternatives belong to [WP-193e](https://borda.github.io/lucid-YOLO/work_packages/wp-193e/#wp-193e); a small-batch result does not establish a large-batch speed-up.

## Opt-in CUDA network replay

`--model.capture_network true` records the detection backbone, neck and dual head with PyTorch's `make_graphed_callables`, including their backward pass. Assignment, losses, optimizer, LR/alpha schedules and EMA keep the ordinary loop. It is separate from `compile_step` and cannot be combined with that flag. The default is `false`. The measured speed-up is limited to a small-batch workload (batch 4); batch-32 screens changed direction, so check throughput on your own batch size before enabling it.

For `task: keypoints`, the same flag records backbone and neck plus eighteen independent box/class/point stems through the private prediction runner. Each point stem has its own backward node: a batch with no assigned positives keeps its point-parameter gradients absent. The current RLE/OKS or Laplace objectives stay native. A fully frozen backbone and neck, changed head topology or dispatch, and head hooks/compilation keep pose native; a partial shared-feature freeze still replays. A fully frozen start or a later change to the shared-feature trainable set keeps the rest of the fit native after one warning. Changing a stem's trainable set or module mode retires that stem's recording, with a warning naming the stem; replacing a head module retires the parent before feature replay. The measured benefit is for nano/640/BF16, batch 4 on L4: five complete pairs save 12.97%, including setup, validation and checkpoints, against the existing shared-feature replay. Reserved memory rises from 1384 to 1730 MiB. Check time and memory on another batch or device. [WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/) records the measurements and unverified limits; [WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/) retains the earlier feature-only result.

For `task: obb`, the flag records backbone, neck and the complete dual head, including both angle outputs. The native classification and oriented losses stay outside capture. Their empty-target expressions retain zero angle gradients. Overridden training consumers, classification losses or oriented helpers keep the complete native call. The oriented speed-up is likewise limited to a small-batch workload (Nano OBB at batch 4, BF16, one L4): a single batch-8 pair changed fit time by only 4.10%, and replay raised peak reserved memory from 3974 to 4300 MiB, so check throughput and memory on your own batch size. The measurements and remaining limits are in [WP-200](https://borda.github.io/lucid-YOLO/work_packages/wp-200/).

This path requires single-device CUDA automatic optimisation and `task: detect`, `task: keypoints`, `task: obb` or `task: segment`. BF16 replay requires a final PyTorch 2.9 or newer release; 2.9 pre-release and NGC `a0` builds are excluded by design. 2.9 is the first verified runtime: older BF16 runs warn and stay eager because the 2.4 comparison did not meet the exact eager oracle. FP32 and FP16 replay of detection, oriented detection and pose shared features are checked on 2.4 as well; pose stem replay has run on 2.9.1 only ([WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/)). CPU, MPS and other strategies warn and stay eager; other tasks or `compile_step` raise at construction. One image signature is recorded per fit. A partial batch uses eager execution without growing a graph cache. Changes to the recorded stages’ parameter/buffer storage, module modes, hooks or precision retire replay for the rest of that fit. The supplied network must keep its ordinary stage implementations and static Python configuration. Root forward overrides, stage instance-forward overrides and registered hooks use the eager path. Anomaly detection (`--trainer.detect_anomaly true`) also stays eager so backward NaN checks run normally. A failure while recording raises instead of falling back to eager (RNG state and buffers are restored first), so disable `--model.capture_network` if recording fails on your model. Higher-order differentiation is not supported.

With detection and accumulation one, the module can omit snapshots that the native Lightning automatic-optimization closure clears immediately after forward. The version gate admits Lightning 2.4–2.6; 2.4.0 and 2.6.6 were executed and 2.5.0 was read from source only. Only the inspected native dispatch and pass-through profiler qualify; customized or unsupported contracts retain the snapshots. The module logs once per fit, at rank-zero info level, whether this snapshot-skip fast path is active and, if not, the first check that failed. This is an internal refinement of network replay; there is no additional flag. [WP-193h](https://borda.github.io/lucid-YOLO/work_packages/wp-193h/) records the measured workload and verification limits.

Recording warmup restores RNG and BatchNorm buffers before the first real batch. Accumulation or custom optimiser/zero-grad hooks preserve live gradients outside the graph-owned backward buffers before they can be overwritten. AMP uses an uncached autocast scope as the PyTorch API requires. Validation, prediction and public `forward` calls stay eager. Checkpoints retain the ordinary parameter names and the flag, with no graph state. Each fit records afresh and drops its recording on completion or an exception. CUDA graph pools can increase reserved memory even when peak allocated memory falls; measure both on the intended workload. The measured scope and remaining validation gaps are in [WP-193c](https://borda.github.io/lucid-YOLO/work_packages/wp-193c/).

For segmentation, the same `--model.capture_network true` option records only backbone and neck. Prediction groups, prototype fusion and both losses keep their existing arithmetic and gradient boundaries. It can combine with `--model.capture_segment_heads true`, which records the prediction groups separately. Overridden or compiled segmentation forwards stay native. The default remains `false`; a recording is transient and follows the same fit cleanup and input-signature rules above. See [WP-193g](https://borda.github.io/lucid-YOLO/work_packages/wp-193g/) for the measured workload and limits.

Network replay also shares one current membership traversal between its eligibility and storage checks. Certified scalar module metadata is seeded after recording warmup and reused only while module identities, class forwards, live training modes and ordered attribute bindings are unchanged. Live tensor storage, layout, dtype, gradient flags and hook checks remain independent. This host-bookkeeping change applies whenever `capture_network` replays, independently of `capture_gradient_norm`, without changing the recorded model arithmetic.

## Optional CUDA gradient-norm recording

`--model.capture_gradient_norm true` records PyTorch's global L2 gradient-norm reduction on single-device CUDA automatic optimisation with torch 2.9 or newer. The default is `false`. It is independent of network/loss/update replay and uses the trainer's existing norm clipping configuration, such as `--trainer.gradient_clip_val 10`. Gradient scaling stays in PyTorch's native call and reads the current limit on each step. Disabled clipping (a limit of `None` or at most zero) and value clipping retire recording once with the reason "clipping is disabled or not by norm" and keep the native call. Ordinary `16-mixed` GradScaler training can capture the reduction after unscaling; optimizers that unscale inside their own step (`_step_supports_amp_scaling`, such as fused optimizers) keep the native call, including Lightning's clipping rejection under mixed precision. One gradient shape/stride/dtype/device and stream signature is retained per fit. Offset views and other incompatible gradients or changed public clipping implementations retire recording for that fit. Precision plugins are matched through their type's MRO against stock `Precision`/`MixedPrecision` clipping methods, with each method checked by identity: subclasses inheriting those methods remain eligible; customized module or precision-plugin clipping retires capture with a reason naming the plugin type and keeps the native call. CPU, MPS and unsupported runtimes warn and stay native; the torch gate says the norm decomposition is "verified only on torch>=2.9". Fit completion, failure and copying release the transient recording, which adds no model tensor keys to checkpoints.

Setup and private CUDA buffers have a cost. The 9.55% complete-fit reduction in [WP-200c](https://borda.github.io/lucid-YOLO/work_packages/wp-200c/) covers the whole patch on a pre-integration revision (nano pose/640/BF16, batch 4, L4, five complete pairs), including the network-replay bookkeeping change above. The flag-only share is unmeasured, and the integrated tree was not re-measured. Long-run accuracy remains unverified.

## Optional CUDA detection and pose loss execution

`--model.capture_loss true` replays the existing detection assignment and loss forward/backward with fresh predictions, targets and the current alpha tensor. Default `false`: capture costs setup time and retains CUDA workspace, so short or compute-bound runs may not benefit. It accepts `task: detect` or `task: keypoints`, and requires single-device CUDA automatic optimisation and `compile_step: false`. It can combine with `capture_network`, `capture_orthogonalization` and `capture_updates`; `capture_loss` itself records no optimizer update, so the optimizer step is whatever those two optimizer options select (the existing eager path when both are off).

For pose, the unrotated detection objective uses that recorder while point gathers and OKS stay native. During training, the built-in RLE densities can score both branches in one batch, retaining separate visible-point means and branch weights. That batching is gated on the option being active for the fit, not on the recorder: it continues after the recorder retires and on steps whose detection loss takes its native call, such as a customized criterion or BF16 on torch older than 2.9. This reduces dispatch but can change matrix-kernel selection and parameter-gradient summation rounding: pose execution retains the mathematical objective, but training is not byte-identical. Hooked, overridden or compiled densities, altered pose scoring and validation use their original calls. Branches with different input dtypes/devices or tensor subclasses keep separate density calls. Laplace-only pose keeps native point scoring. See [WP-199b](https://borda.github.io/lucid-YOLO/work_packages/wp-199b/) for the measured scope and unverified accuracy checks.

One masked target workspace grows from observed instance counts and replaces the previous recording. Returned scalars and assignments own their storage. Current label bounds are checked before replay. Global module hooks, unsafe custom epsilon/exponent settings, entirely empty targets, partial batches, non-finite inputs, unsupported layouts or autograd contexts, another CUDA stream and older BF16 runtimes keep the native loss. FP32 and FP16 have focused CUDA coverage on torch 2.4.0 and 2.9.1; BF16 replay starts at the verified final 2.9 runtime. Changed criterion implementations, gains or precision retire the recording. A capture failure raises after clearing the partial owner. Higher-order differentiation and multiple backward passes through one recorded result are outside this automatic-training option.

On the measured L4 nano/640/BF16 workload, five batch-4 complete-fit pairs saved 13.14%. A separate batch-16 exploratory pair was 7.767% slower. Treat this as a small-batch launch-overhead option and measure the complete fit before enabling it for another workload.

[WP-199](https://borda.github.io/lucid-YOLO/work_packages/wp-199/) records the measured workload, discarded screens and remaining validation gaps. Check complete-fit throughput and memory on your own batch size before enabling it.

## CUDA graph replay of orthogonalization

`--model.capture_orthogonalization true` records MuSGD's existing Newton--Schulz matrix calls and replays them with each new gradient. The default is `false`: private graph pools retain extra device workspace, and setup may outweigh the gain on short or compute-bound runs. Momentum, parameter updates and the learning-rate schedule remain eager. CPU and MPS keep the original functions. [WP-197b](https://borda.github.io/lucid-YOLO/work_packages/wp-197b/) records the measured workload and compatibility envelope.

## CUDA update preparation replay

`--model.capture_updates true` additionally records MuSGD's update preparation, including momentum, for eligible dense float32 CUDA groups. Learning-rate changes and parameter writes remain outside capture. This option can be used with or without `capture_orthogonalization`; the two recordings do not nest. CPU, MPS and unsupported CUDA layouts or execution contexts retain the native step. A failure while recording raises instead of falling back to the native step (gradients and momentum are restored first), so disable `--model.capture_updates` if recording fails on your model. Loading optimizer state discards old recordings. The default is `false` because capture adds setup and retains workspace. [WP-197c](https://borda.github.io/lucid-YOLO/work_packages/wp-197c/) reports five complete public-data fits, numerical checks and measured memory cost; other workloads need their own measurement.

## torch.compile (CUDA, detect)

`--model.compile_step true` (or `compile_step: true` under `model:`; `det_nano_smoke.yaml` carries it at `false`) compiles the detection objective with Inductor. What is compiled is one region, `DetectionLitModule._detect_objective`: the forward, both branch decodes, the task-aligned assignment and the dual loss, traced as a single graph with no breaks — `tests/ptl/test_compile.py` pins that on CPU for a batch with ground truths and for one with none. Compiling the model half alone was measured first and was not worth a flag: x1.09–1.16 on the step, because the assignment and the loss are where the small kernels and the launch overhead live. With [WP-183](https://borda.github.io/lucid-YOLO/work_packages/wp-183/)'s dense box terms (no data-dependent gather) and the label check skipped under tracing, the whole region compiles: on an RTX PRO 6000, variant `n`, COCO, batch 16, deterministic, the eager training step is 74.7 ms and the compiled one 60.9 ms, x1.23. A bf16 run was recorded at x1.46 on the same bench and is **not** enabled by this flag — precision is a separate decision the fidelity gate has not been asked about.

What it costs: 110–140 s of compilation per process before the first step, and a handful of recompiles over a run: the training graph, the first `no_grad` evaluation step, and — because Dynamo specialises sizes 0 and 1 whatever is marked — one each for the first batch with no ground truths (`tal.py`'s own branch), the first with exactly one padded instance, and the first with one image, if the loader ever produces it. Measured on CPU with the `eager` backend over five batch shapes: four graphs. Batch size and padded instance count are marked dynamic on the images and the ground truths before every compiled call, so the last, smaller batch of an epoch and a batch with more instances than the one before share one graph; without the marks Dynamo would specialise until `cache_size_limit` (8) and then drop to eager without saying so, and with them the assigner's `F.one_hot(best_gt, num_gt)` — a concrete class count — had to become an `arange` comparison, since the enforcing marks refuse a pinned dim rather than silently recompiling. `TORCH_LOGS=recompiles` shows the count.

Where it refuses, it says so once at `setup` and trains eager. Off CUDA: the MPS Inductor backend crashes in `convolution_backward`, and CPU has nothing to gain. Under `task: segment`, `obb` or `keypoints`: their extra terms read tensors on the host inside `_task_extra_loss`, outside the region, so compiling the detection part alone would buy a fraction of the x1.23 for the same compile cost; they stay eager until their terms are made traceable. The compile wraps the bound method, not the module, so parameter names, the state dict, the EMA shadow, checkpoints and export are exactly what they are without the flag — a compiled *module* would prefix every key with `_orig_mod.`.

The default is off on cost, not on doubt. A78 — whether the compiled objective trains as eager does — was validated on an A100 (paired 4-epoch run: 1 recompile, step 180.6 → 144.2 ms, loss parity `3.6e-5` with TF32 off), but compilation added 210 s of wall to a 4-epoch run of 313 steps, a break-even near 19 epochs at that size and under one epoch on COCO train; turn it on per recipe when the run is long enough. When comparing the two arms, compare mAP, or set `torch.set_float32_matmul_precision("highest")` first: under TF32 the per-batch `train/loss` differs by up to `1e-1`, because the task-aligned assignment is discrete and TF32 rounding moves its top-k set. D24 carries the numbers.

## Lifetime of the replay flags

Each opt-in replay flag is kept or retired on its own record, under the principal's rule P5 ([D32](https://borda.github.io/lucid-YOLO/DECISIONS/)). A flag stays while its `make gate-gpu` parity is green and its documented speed-up holds. A flag is retired when its parity stays red past one phase, or when its gain on its documented workload is under 3%. No flag becomes the default in 0.x without a principal call. Nothing on this page is removed by the table below: a flag whose stated gain is already under 3% would be marked as a retirement candidate for the principal, and none is, since every documented-workload figure is well above it. The batch-16 and batch-32 screens quoted elsewhere on this page (2.01% and 2.90% for segmentation, 2.50% for pose features, 4.10% for oriented) are exploratory boundary cases, not the documented workload, and do not trigger the rule.

A known failing CUDA test is a red `main` until it is fixed, and a [GPU] work package does not close without a green `make gate-gpu` run (decision 3 of the 2026-10-20 entry in [ESCALATION](https://borda.github.io/lucid-YOLO/ESCALATION/)). A retirement trigger is therefore a call for the principal, not an automatic removal.

| Flag (task) | Work package | Documented workload and measured gain | Keep condition | Retire trigger |
| -- | -- | -- | -- | -- |
| `capture_network` (detect) | [WP-193c](https://borda.github.io/lucid-YOLO/work_packages/wp-193c/) | Nano detection on one L4, torch 2.9.1+cu129: complete-fit median 92.589 s to 77.330 s, 16.48%. | `make gate-gpu` parity green; the gain holds on this workload. | Parity red past one phase, or gain under 3% on this workload. |
| `capture_network` (keypoints) | [WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/), [WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/) | Feature replay ([WP-193d](https://borda.github.io/lucid-YOLO/work_packages/wp-193d/)), five pairs: complete-fit time falls 11.17%. Stem replay ([WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/)), nano/640/BF16, batch 4, L4, five pairs: complete runtime falls 12.97%. | Same. | Same. |
| `capture_network` (obb) | [WP-200](https://borda.github.io/lucid-YOLO/work_packages/wp-200/) | Nano oriented, 104 DOTA-v1.0 tiles, eight epochs, batch 4, 1024 px, BF16, one L4, five pairs: complete-fit median drops 12.011% (68.642 s to 60.397 s). | Same. | Same. |
| `capture_network` (segment) | [WP-193g](https://borda.github.io/lucid-YOLO/work_packages/wp-193g/) | Nano segmentation, 512 COCO val2017 images, five epochs, batch 4, 640 px, BF16, L4, five pairs, prediction replay already on: complete fit 107.412 s to 87.629 s, 18.42%. | Same. | Same. |
| `capture_loss` (detect) | [WP-199](https://borda.github.io/lucid-YOLO/work_packages/wp-199/) | Nano, batch 4, 640 px, BF16, five epochs on 1,024 COCO val2017 images, L4, five pairs: complete fit 125.272 s to 108.811 s, 13.140%. | Same. | Same. |
| `capture_loss` (keypoints) | [WP-199b](https://borda.github.io/lucid-YOLO/work_packages/wp-199b/) | Registered nano, batch 4, 640 px, BF16, five epochs over 384 COCO person images, L4, five pairs: 100.586 s to 87.058 s, 13.45% less complete-fit time. | Same. | Same. |
| `capture_segment_heads` (segment) | [WP-193e](https://borda.github.io/lucid-YOLO/work_packages/wp-193e/) | Nano segmentation, 384 training COCO val2017 images, 10 epochs, batch 4, 640 px, BF16, L4, five pairs: complete fit 202.794 s to 169.130 s, 16.60%. | Same. | Same. |
| `capture_orthogonalization` (MuSGD) | [WP-197b](https://borda.github.io/lucid-YOLO/work_packages/wp-197b/) | Nano detection, first 256 COCO val2017 images, six epochs, batch 4, 640 px, BF16, L4, five pairs: time saved 19.76% on the tree before compatible batching, 12.55% with compatible batching and device HSV applied (expected benefit on current main). | Same. | Same, read against the 12.55% row. |
| `capture_updates` (MuSGD) | [WP-197c](https://borda.github.io/lucid-YOLO/work_packages/wp-197c/) | Nano detection, first 256 COCO val2017 images, ten epochs, batch 4, 640 px, BF16, L4, five pairs: complete fit 120.850 s to 108.750 s, 10.01%. | Same. | Same. |
| `compile_step` (detect) | WP-184 ([roadmap row](https://borda.github.io/lucid-YOLO/ROADMAP/), D24 in [Decisions](https://borda.github.io/lucid-YOLO/DECISIONS/)) | Measured x1.23 on the detect step, RTX PRO 6000, variant `n`, COCO, batch 16; 110–140 s compile per process. WP-184 has no page of its own, so this is the roadmap row's figure, a step-time ratio rather than a complete-fit time. | Same; the compile cost is paid per process. | Same, with the step-time ratio converted to a complete-fit gain. |

Every row's "gain" is the figure its page states for the workload shown, on one device; none is a claim for another batch size, device or full-COCO run. The pose-feature row is not a complete-fit figure for the stem replay alone: [WP-193f](https://borda.github.io/lucid-YOLO/work_packages/wp-193f/) measures stem replay against feature replay already enabled.