📦 WP-192c — perf(data): reuse prepared source pixels during training¶
Engineering log · Phase 18 — CUDA graph replay and pipeline throughput
🔗 Depends on WP-192b · Needed by none
WP-192c — prepared source pixels and copy-paste masks¶
[THEORETICAL] Reuse immutable source work before random augmentation. cache_prepared_train_images is an opt-in float32 snapshot of the training reader's images and targets. Source conversion and subtraction of the existing affine fill happen once. Mosaic copies those centred values onto zero fill; the same resample restores the original fill afterwards. Preparation centres out of place, so the reader's tensor is never written and nothing depends on whether a reader hands out fresh or shared storage; the cost is one transient image per record during setup. The [MEASURED] timings below were taken at 672f0ba, which centred in place; the out-of-place subtraction adds only that setup-time allocation and was not re-timed. No second byte cache is retained. Fit setup reuses an existing snapshot; other stages release the datamodule's references, and failed preparation installs no partial cache. YOLO labels are snapshotted with the images.
[THEORETICAL] One source coordinate needs one copy. Copy-paste builds the union of its selected pixel masks and copies the immutable source once into the fresh destination clone. Overlapping writes already copied the same value. The instance targets remain separate and in their original order; the rasterisation, geometry and RNG draws are unchanged. Unlike the opt-in cache, the union changes the default (non-opt-in) copy-paste path; its output is byte-identical to the previous path, as TestCopyPasteUnion checks. Its inclusion in this WP awaits principal acceptance, because the screens do not carry it alone: cache-only reached 5.61% and union-only 4.32%. These changes follow from this repository's source, R21's existing fill interface and measured producer work.
[MEASURED] The complete short fit is the performance claim. Baseline d6f7ec8; candidate is d6f7ec8 plus the 672f0ba change. Five alternating pairs on an isolated NVIDIA L4, driver 580.178.04, torch 2.9.1+cu129 and Python 3.11.17 used detection n, batch 64, 640 px, BF16, four workers and decoded caching in both arms, with device_hsv=True. Each fresh-process fit covered five epochs and 80 steps over 1,024 distinct COCO val2017 training images and 32 held-out images. Cache construction, augmentation, forward/backward, optimiser, EMA, validation each epoch, checkpoints and final CUDA completion are inside the timer. Mosaic stayed enabled (close_mosaic=0), representing the augmentation-active part of training. Baseline median 116.184 s (114.011–116.872), candidate 97.579 s (96.728–98.340); paired reduction median 15.63% (13.93–16.75%).
[MEASURED] Correctness is byte-level. The five pairs and baseline-against-itself controls match model, optimiser, EMA, scheduler, loss and metric hashes. A separate 64-draw public-COCO check matches float-image bytes before quantisation, packed targets and RNG state with mosaic enabled and disabled, and preserves the source snapshots. Focused tests cover both readers, overlap/duplicate selections, non-contiguous images, source reuse, rollback, repeated setup and spawned workers. Direct affine checks cover exposed padding and identity parameters on fixed letterbox canvases, the training route, with byte parity. A standalone identity without letterbox can bypass the dependency resample and retain float32 centring round-off (observed maximum 5.96e-8 on RGB codes); a regression bounds its absolute error by float32 epsilon with zero relative tolerance. The declared torch 2.4 CPU floor passes the 148-test focused suite.
[MEASURED] The memory cost and scope stay explicit. This slice retains 3,375,379,188 float-image bytes, against 843,844,797 decoded bytes. Prepared images are per training record, including duplicates, plus target metadata; the option stays off by default. D27 governs the decoded uint8 cache only; the float32 shared-tensor cache is a different memory trade-off and awaits a principal decision. This is a small-slice, CPU-limited training result, not a claim for arbitrary worker counts, model sizes or full-size splits. Separate single-pair cache-only (eight epochs) and copy-paste-only (six epochs) fit screens reached 5.61% and 4.32%, respectively, so neither became a standalone claim above the investigation's eight-percent bar. A separate eight-worker, four-epoch single pair improved 4.48%, below that bar. Full make gate and accelerator goldens were not run; the row remains open.