WP-200c — perf(ptl): reduce CUDA training bookkeeping¶
Engineering log · Phase 18 — CUDA graph replay and pipeline throughput
WP-200c — current metadata and a recorded gradient reduction¶
The complete CUDA training loop still performs repeated module enumeration and scalar-attribute filtering before replay. On a real COCO keypoint training profile, network replay checks account for 0.211 of a 0.682 s four-step cProfile window (31.0%); signatures account for 0.115 s (16.9%). These interpreter-instrumented shares identify the mechanism and are not absolute throughput evidence. An independent initial segmentation Nsight timeline shows a large host idle budget, while its largest standalone kernel family contributes only 2.61% of that window, so a compute-only rewrite of that family has too little device-time budget for the intended complete-training saving there. Reducing submissions can also remove host gaps; that larger budget motivates recording the native reduction.
The membership walk now avoids unused names under native enumeration, shares the current parameter/buffer lists between checks and retains the recorded parameter list for backward protection. A custom enumeration route retains the original calls. Cached scalar metadata is used only while every ordered attribute binding and class forward remains identical. Tensor storage, shape, stride, dtype, device, trainability and hooks are still read independently, and changed contracts keep the existing retirement policy.
The optional capture_gradient_norm flag records the dependency's existing get_total_norm call on private CUDA gradient inputs. Current gradients are copied in before replay. The native clip_grads_with_norm_ call applies the current clipping limit to the live parameters; aliases and duplicate writes keep that call's semantics. The flag requires torch 2.9 and single-device CUDA automatic optimisation. Default off, value clipping, offset views, unsupported layouts and custom clipping keep their original route. One recording is retained per fit, and input/policy/stream drift retires it once. Successful fits, failed fits and copying release the recording. A callback retained by Trainer for a later unrelated model is a no-op, including the exception hook. Model, loss, assignment and optimizer expressions are unchanged.
This package derives from the current repository, our own measurements, R42 and R45. It introduces no method constant or numerical assumption. CUDA buffers and setup cost are explicit costs of the opt-in flag.
Five alternating fresh-process pairs against main 4dc5ddd on an isolated NVIDIA L4 reduce complete nano pose fit time by 9.55%, including setup and validation. Whole-process time decreases by 9.13%. These are the final corrected implementation, including warmup-failure ordering, offset-view fallback, unrelated-model lifecycle fallback, live public training-mode checks, post-warmup metadata certification with the original retirement stamp, and native clipping-conflict validation. Earlier production cohorts remain supporting evidence only.
| Median (range) | Main 4dc5ddd |
Patch |
|---|---|---|
| Complete fit, s | 98.342 (96.295–100.413) | 88.946 (88.575–88.961) |
| Whole process, s | 103.682 (101.545–105.764) | 94.218 (93.900–94.437) |
| Steady train step, ms | 108.671 (106.886–109.805) | 95.709 (94.944–96.188) |
The training-only step row fences the CUDA stream at epoch boundaries before validation or EMA weight swapping. It discards the first three epochs and divides the remaining seven training windows by their 448 steps. The complete-fit row retains every warmup/setup cost. Each fit uses 256 person-containing public COCO val2017 images for training and 32 disjoint images for validation, original keypoint annotations, nano, batch 4, 640 px, ten epochs and BF16 mixed precision. Four intraop threads and four persistent loader workers are used in both arms. Prepared-image caching, device HSV, epoch prefetch, applicable network/loss/orthogonalization/update replay and all-mosaic training are enabled in both. The timer starts before model/data/trainer construction and includes preparation, recording, training, optimizer, EMA, validation and checkpoint writing. Imports and output hashing are outside the complete-fit row and inside the whole-process row.
Model, optimizer, scheduler, EMA, all 640 step losses and validation metrics have matching value-byte hashes across all ten fits. Every loss is finite, every candidate batch observes the live norm recording, and every completed fit releases it. This is trajectory and execution validation on a public subset; it does not establish convergence or full-COCO accuracy.
The median training peak allocated memory is 587.10 to 587.10 MiB, and reserved memory is 1836.00 to 1846.00 MiB. These are separate allocator metrics, not a capacity claim. A separate single batch-16/three-epoch pair changes complete fit time by 3.60%; it is a boundary screen, not a repeated batch-16 claim. It changes both batch size and fit length, so it does not isolate batch scaling.
The measured source hashes are:
9ef4b97ff31077bdb7ff54591693972989bf75ab512bd12bb221d54276c92ad5 src/lucid_yolo/ptl/network_capture.py
0affdbc304d6be79723a92f7649ce4d7db1a3788babe9097d74d828edb67e986 src/lucid_yolo/ptl/module.py
300d5cf1314af0793effc758b7d5c7ad8864cbb44a1e044084a74c9b8a186b04 src/lucid_yolo/ptl/gradient_norm_capture.py
Python 3.11.17, torch 2.9.0+cu129, CUDA 12.9, Lightning 2.6.6, driver 580.178.04. The 57 focused cases include real public Trainer reuse across models, a later foreign optimizer update and preservation of its original training failure. The callback ignores a missing holder, following this repository's network-callback convention. Real stem retention is observed before three backward passes and the expected accumulated gradients; removing parameter retention fails that regression. Focused regressions also cover CUDA reduction/scaling for four dtypes, native aliases and duplicate writes, current limits, actual owner drift/retirement, copy/fit cleanup, controlled failed warmup/capture and a bounded linear FP16 Lightning fit with accumulation, one injected overflow and real optimizer updates. The FP16 fixture isolates the norm/precision hook; it does not certify the separate full-network FP16/accumulation/clipping interaction investigated in the controls. CPU fallback, older-runtime routing, custom precision behavior and the live metadata checks are also exercised. No new arithmetic or method constant is introduced. The original submission corrected the checkpoint-assignment CUDA test to compare the eager gradient pattern for a classification-only loss. After synchronization with main, that regression instead supervises all four detection outputs and requires every parameter gradient, retaining main's stronger oracle. At the measured revision, full make gate, aggregate offline coverage, accelerator goldens and long-run acceptance were not run.
Integration on 28 October 2026. This package is renumbered from WP-201 to WP-200c because main now owns WP-201. The merge retains main's pre-forward live-gradient copy and shared recording, tensor-signature and cleanup helpers. The parameter cache supplies the existing copy helper before replay; the regression follows that timing. The performance table and source hashes above describe the original measured revision, not this integrated tree. No performance measurement of the integrated tree was made.