📦 WP-188 — feat(docs): notebooks rendered from jupytext sources¶
Engineering log · Phase 17 — R1 pose-objective parity
WP-188 — notebooks as sources, the .ipynb as a build product¶
The format question was decided by what the gate can read. Recorded 2026-09-21. A notebook is worth publishing only while it still runs, and nothing about publishing one checks that; so the first decision was that every notebook is a .py the existing machinery already handles -- ruff lints it, license-headers scopes it, and tests/notebooks/test_notebooks_run.py runs it as a script under LUCID_NOTEBOOK_FAST=1 on every make gate. Jupytext's percent format is the one that costs nothing: a # %% line is a comment to Python and a cell to the converter. The .ipynb follows from the .py at build time in two places, docs/notebooks/ for the site and notebooks/<name>.ipynb on gh-pages for the Colab badge, and neither is ever committed to main; no-ipynb-tracked reads the index on every commit because .gitignore is advice and git add -f ignores it (D25). The script run ruled out one thing: the Colab bootstrap could not be a !git clone line, because jupytext writes a magic into the .py as a comment and a guarded block of comments alone is a SyntaxError when the file runs as a script -- so the bootstrap, and after it every console command the demo notebooks ran, went through subprocess.
The script run is withdrawn; the gate executes the notebook through a kernel. Revised 2026-09-21, once the four demo notebooks existed. Each of them is a sequence of console commands -- lucid-data download, lucid-yolo fit, lucid-eval, lucid-predict -- and as subprocess.run([...]) lists they read as argv arrays, not as the command lines docs/TRAINING.md spells. The user's ask was that they read as plain shell, and that decides the gate's mechanism rather than the other way round: each command is a %%bash cell -- # %% language="bash" in the .py with the script commented beneath, comments to a script run -- so the proof moves from "run the .py" to "convert the .py and execute the .ipynb with a kernel". The test now runs jupytext --from py:percent --to ipynb --execute --set-kernel python3 --run-path <repo root> per source, the same conversion make notebooks makes plus execution, and the subprocess reasoning above no longer applies because nothing runs the file as a script any more; the bootstrap is !git clone and %pip install -e lucid-YOLO under the same google.colab guard, with os.chdir and the sys.path insert keeping the block a block. Values cross into a shell cell through the environment -- the Python cell above does os.environ.update(DATA_ROOT=..., FIT_ARGS=...), the fast budget being one such string of flags -- and a switch the shell itself has to make reads $LUCID_NOTEBOOK_FAST, so the download the fast run skips and the lucid-predict loop only the fast run makes are if blocks in bash beside the command they guard. Three things were measured rather than assumed. A ! line does not fail the notebook -- !false followed by a print printed, under nbclient, with the cell green, IPython storing the status in _exit_code and continuing -- which is why the commands are %%bash cells and not ! lines: the script magic raises CalledProcessError on the cell's exit status and nbclient stops, as a probe with false confirmed; that status is the last command's, so a cell with more than one opens with set -e. --set-kernel -, jupytext's spelling for "the current interpreter", raises No kernel found that matches the current python executable: the kernelspec ipykernel's wheel installs into .venv/share/jupyter/kernels/python3 has argv[0] equal to the bare word python, and jupytext compares that to sys.executable with os.path.samefile. --set-kernel python3 names that spec directly, and jupyter_client.manager.KernelManager.format_kernel_cmd swaps a bare python for the launcher's sys.executable -- the venv's, since pytest runs jupytext as python -m jupytext -- which a probe confirmed from inside the kernel; so there is no ipykernel install step, and the test sets JUPYTER_PATH to the venv's share/jupyter so a user-level python3 spec, which the search order otherwise puts first, cannot take the name. And --from py:percent is not optional: jupytext's guess_format counts an indented # !git clone -- the Colab block's -- as an unescaped magic, because its escaped-magic test is not line.startswith("#") with no lstrip, and guesses hydrogen, a format that leaves every # ! line a comment. A jupytext header after the SPDX line and a [tool.jupytext] table in pyproject.toml were both tried and neither reaches the CLI reader, so the flag went into the test, the Makefile and docs.yml together; without it the published .ipynb would carry # !git clone as a comment and the badge would open a notebook that clones nothing. The dev group gains jupytext, nbconvert and ipykernel -- nbconvert and not nbclient, because jupytext --execute imports nbconvert.preprocessors.ExecutePreprocessor and declares neither as a hard requirement -- and the licence audit on the dev environment stayed clean at 152 distributions. Under LUCID_NOTEBOOK_FAST=1 the five sources take 18–32 s each through the kernel on an Apple-silicon CPU (the four-task wiring gate the 32 s), kernel start included, against the test's 60 s ceiling, which therefore did not move. The full runs' paths were settled at the same time: a dataset goes under the gitignored data/ of the checkout -- data/coco2017 for the three COCO demos, data/dota and data/dota_tiles for the oriented one -- and the run under results/demo_<task>, the Colab detect demo keeping its Drive guidance; the fast run keeps its temporary directory.
The demos moved off the shell cells; the kernel run stays. Revised again 2026-09-21, once the four demos were rewritten onto the library: every step is now a call on an object the kernel holds -- download_coco, DetectionLitModule and DetectionDataModule built from the tier's YAML values, Trainer.fit and Trainer.validate, lucid_yolo.predict and lucid_yolo._viz -- and the console command is quoted as prose, never run, so the %%bash cells, the os.environ hand-off and the $LUCID_NOTEBOOK_FAST test in bash are gone from them. The three measurements above still stand as measurements, they just no longer bear on the demos. What keeps the gate on jupytext --execute through a kernel rather than a script run is the bootstrap: !git clone and %pip install -e lucid-YOLO are magics, written into the .py as comments, and a script run would read them as comments and prove nothing about the two lines a Colab reader depends on; the --from py:percent reasoning is theirs as well. One consequence for a figure cell: its last expression is None if FAST else prediction_grid(640), because a bare name as the cell's last line is B018 to ruff and a call is not. Measured on an Apple-silicon CPU, the four demos take about 11 s each under the fast contract, against 18-32 s through the console scripts. The gate notebook went the other way for the same reason, one file later: the four overfit commands and the shapes regression are development tooling with no library entry point, so notebooks/development_gates.py runs them as the ! lines TRAINING.md spells, $PROBE appended under the fast contract -- --epochs 1 --batches 4, a probe override the overfit script gained on the shapes script's model, reporting PROBE with the score and enforcing no floor; 26 s for all five commands.
What the scratch-venv audit found. The docs group alone resolves to 43 distributions with jupytext added and 77 with mkdocs-jupyter, the 34 between them being nbconvert, nbclient, ipykernel and what they pull; pyzmq (BSD-3-Clause) is the one shipping a native binary the audit's wheel scan walks. Licence audit clean at both counts, nothing copyleft. docs.yml's build job is the only CI environment holding this tree, and its audit step already runs before the build, so the exposure is checked where it enters. The site is rendered with execute: false, so that job stays torch-free, which is what keeps the docs group small enough to audit by hand as well.
The --strict risk was checked with a throwaway notebook and did not fire. The plan flagged that mkdocs-jupyter might warn on a converted file missing kernel metadata, which --strict would turn into a failure. A percent-format source with one markdown cell and one code cell, converted by jupytext and listed in the nav, built strict with no warning and rendered its cells; jupytext writes the kernelspec. The source and its .ipynb were deleted after the build. Nothing in the shipped tree depends on this, since WP-188 lands zero notebooks -- it is recorded so WP-189 does not re-derive it.
Two checks moved from the site to the audit, and one rule changed shape. check_every_nav_entry_points_at_a_file_that_exists used to ask whether the nav target was a file under docs/; a notebooks/<name>.ipynb target now resolves against notebooks/<name>.py instead, whether or not the .ipynb has been written yet. The alternative -- accept the entry when the generated file is on disk -- would have answered differently on a warm checkout and a cold one, and a checker whose verdict depends on whether make notebooks was run is a checker that passes locally and fails in CI. check_every_notebook_source_has_a_nav_entry is the page rule (a register nobody can navigate to is one nobody reads) applied from the source side, and it tolerates a mkdocs.yml with no Notebooks section while no source exists, because MkDocs rejects a nav section with no children and the section therefore cannot arrive before the first notebook does.
The badge decided how the site is published. The first draft of this row added a publish-notebooks job writing the .ipynb files to an orphan notebooks branch, beside the existing Pages artifact deploy -- two publishing mechanisms for one site. The revision (2026-09-21) collapses them: the site itself now goes out as a branch deploy, ghp-import -n -p committing site/ onto gh-pages on top of that branch's history, and the converted notebooks are copied to site/notebooks/<name>.ipynb before the import so the badge's blob/gh-pages/notebooks/<name>.ipynb is a path on the branch Pages serves. mkdocs-jupyter's include_source places its own copy at site/notebooks/<name>/<name>.ipynb, beside the rendered page; the flat copy is the badge's, and both ride the same commit. ghp-import arrives with mkdocs, so the docs group gains nothing for it; deploy-pages, upload-pages-artifact, configure-pages and the pages/id-token permissions leave the workflow, and contents: write sits on the one job that remains -- a pull-request run holds it too, since a job's grant is fixed for the job, and never reaches the step that uses it. The Makefile and both workflow loops iterate a shell glob with an existence test rather than $(wildcard), since for f in ; is a syntax error under /bin/sh and an empty notebooks/ is the state this row ships in.