# 📦 WP-188 — `feat(docs): notebooks rendered from jupytext sources`

> Engineering log · [Phase 17 — R1 pose-objective parity](https://borda.github.io/lucid-YOLO/engineering/phase-17-pose-objective-parity/)
>
> 🔗 Depends on [WP-114](https://borda.github.io/lucid-YOLO/work_packages/wp-114/) · Needed by [WP-189](https://borda.github.io/lucid-YOLO/work_packages/wp-189/), WP-190
>
> ⬅️ [WP-183](https://borda.github.io/lucid-YOLO/work_packages/wp-183/) · [Roadmap](https://borda.github.io/lucid-YOLO/ROADMAP/) · [WP-189](https://borda.github.io/lucid-YOLO/work_packages/wp-189/) ➡️


## WP-188 — notebooks as sources, the `.ipynb` as a build product

<a id="wp-188"></a>

**The format question was decided by what the gate can read.** Recorded 2026-09-21. A notebook is worth publishing only while it still runs, and nothing about publishing one checks that; so the first decision was that every notebook is a `.py` the existing machinery already handles -- ruff lints it, `license-headers` scopes it, and `tests/notebooks/test_notebooks_run.py` runs it as a script under `LUCID_NOTEBOOK_FAST=1` on every `make gate`. Jupytext's percent format is the one that costs nothing: a `# %%` line is a comment to Python and a cell to the converter. The `.ipynb` follows from the `.py` at build time in two places, `docs/notebooks/` for the site and `notebooks/<name>.ipynb` on `gh-pages` for the Colab badge, and neither is ever committed to `main`; `no-ipynb-tracked` reads the index on every commit because `.gitignore` is advice and `git add -f` ignores it (D25). The script run ruled out one thing: the Colab bootstrap could not be a `!git clone` line, because jupytext writes a magic into the `.py` as a comment and a guarded block of comments alone is a `SyntaxError` when the file runs as a script -- so the bootstrap, and after it every console command the demo notebooks ran, went through `subprocess`.

**The script run is withdrawn; the gate executes the notebook through a kernel.** Revised 2026-09-21, once the four demo notebooks existed. Each of them is a sequence of console commands -- `lucid-data download`, `lucid-yolo fit`, `lucid-eval`, `lucid-predict` -- and as `subprocess.run([...])` lists they read as argv arrays, not as the command lines `docs/TRAINING.md` spells. The user's ask was that they read as plain shell, and that decides the gate's mechanism rather than the other way round: each command is a `%%bash` cell -- `# %% language="bash"` in the `.py` with the script commented beneath, comments to a script run -- so the proof moves from "run the `.py`" to "convert the `.py` and execute the `.ipynb` with a kernel". The test now runs `jupytext --from py:percent --to ipynb --execute --set-kernel python3 --run-path <repo root>` per source, the same conversion `make notebooks` makes plus execution, and the `subprocess` reasoning above no longer applies because nothing runs the file as a script any more; the bootstrap is `!git clone` and `%pip install -e lucid-YOLO` under the same `google.colab` guard, with `os.chdir` and the `sys.path` insert keeping the block a block. Values cross into a shell cell through the environment -- the Python cell above does `os.environ.update(DATA_ROOT=..., FIT_ARGS=...)`, the fast budget being one such string of flags -- and a switch the shell itself has to make reads `$LUCID_NOTEBOOK_FAST`, so the download the fast run skips and the `lucid-predict` loop only the fast run makes are `if` blocks in bash beside the command they guard. Three things were measured rather than assumed. A `!` line does not fail the notebook -- `!false` followed by a `print` printed, under `nbclient`, with the cell green, IPython storing the status in `_exit_code` and continuing -- which is why the commands are `%%bash` cells and not `!` lines: the script magic raises `CalledProcessError` on the cell's exit status and nbclient stops, as a probe with `false` confirmed; that status is the last command's, so a cell with more than one opens with `set -e`. `--set-kernel -`, jupytext's spelling for "the current interpreter", raises `No kernel found that matches the current python executable`: the kernelspec `ipykernel`'s wheel installs into `.venv/share/jupyter/kernels/python3` has `argv[0]` equal to the bare word `python`, and jupytext compares that to `sys.executable` with `os.path.samefile`. `--set-kernel python3` names that spec directly, and `jupyter_client.manager.KernelManager.format_kernel_cmd` swaps a bare `python` for the launcher's `sys.executable` -- the venv's, since pytest runs jupytext as `python -m jupytext` -- which a probe confirmed from inside the kernel; so there is no `ipykernel install` step, and the test sets `JUPYTER_PATH` to the venv's `share/jupyter` so a user-level `python3` spec, which the search order otherwise puts first, cannot take the name. And `--from py:percent` is not optional: jupytext's `guess_format` counts an *indented* `    # !git clone` -- the Colab block's -- as an unescaped magic, because its escaped-magic test is `not line.startswith("#")` with no `lstrip`, and guesses `hydrogen`, a format that leaves every `# !` line a comment. A jupytext header after the SPDX line and a `[tool.jupytext]` table in `pyproject.toml` were both tried and neither reaches the CLI reader, so the flag went into the test, the `Makefile` and `docs.yml` together; without it the published `.ipynb` would carry `# !git clone` as a comment and the badge would open a notebook that clones nothing. The `dev` group gains `jupytext`, `nbconvert` and `ipykernel` -- `nbconvert` and not `nbclient`, because `jupytext --execute` imports `nbconvert.preprocessors.ExecutePreprocessor` and declares neither as a hard requirement -- and the licence audit on the dev environment stayed clean at 152 distributions. Under `LUCID_NOTEBOOK_FAST=1` the five sources take 18–32 s each through the kernel on an Apple-silicon CPU (the four-task wiring gate the 32 s), kernel start included, against the test's 60 s ceiling, which therefore did not move. The full runs' paths were settled at the same time: a dataset goes under the gitignored `data/` of the checkout -- `data/coco2017` for the three COCO demos, `data/dota` and `data/dota_tiles` for the oriented one -- and the run under `results/demo_<task>`, the Colab detect demo keeping its Drive guidance; the fast run keeps its temporary directory.

**The demos moved off the shell cells; the kernel run stays.** Revised again 2026-09-21, once the four demos were rewritten onto the library: every step is now a call on an object the kernel holds -- `download_coco`, `DetectionLitModule` and `DetectionDataModule` built from the tier's YAML values, `Trainer.fit` and `Trainer.validate`, `lucid_yolo.predict` and `lucid_yolo._viz` -- and the console command is quoted as prose, never run, so the `%%bash` cells, the `os.environ` hand-off and the `$LUCID_NOTEBOOK_FAST` test in bash are gone from them. The three measurements above still stand as measurements, they just no longer bear on the demos. What keeps the gate on `jupytext --execute` through a kernel rather than a script run is the bootstrap: `!git clone` and `%pip install -e lucid-YOLO` are magics, written into the `.py` as comments, and a script run would read them as comments and prove nothing about the two lines a Colab reader depends on; the `--from py:percent` reasoning is theirs as well. One consequence for a figure cell: its last expression is `None if FAST else prediction_grid(640)`, because a bare name as the cell's last line is `B018` to ruff and a call is not. Measured on an Apple-silicon CPU, the four demos take about 11 s each under the fast contract, against 18-32 s through the console scripts. The gate notebook went the other way for the same reason, one file later: the four overfit commands and the shapes regression are development tooling with no library entry point, so `notebooks/development_gates.py` runs them as the `!` lines TRAINING.md spells, `$PROBE` appended under the fast contract -- `--epochs 1 --batches 4`, a probe override the overfit script gained on the shapes script's model, reporting PROBE with the score and enforcing no floor; 26 s for all five commands.

**What the scratch-venv audit found.** The docs group alone resolves to 43 distributions with `jupytext` added and 77 with `mkdocs-jupyter`, the 34 between them being `nbconvert`, `nbclient`, `ipykernel` and what they pull; `pyzmq` (BSD-3-Clause) is the one shipping a native binary the audit's wheel scan walks. Licence audit clean at both counts, nothing copyleft. `docs.yml`'s build job is the only CI environment holding this tree, and its audit step already runs before the build, so the exposure is checked where it enters. The site is rendered with `execute: false`, so that job stays torch-free, which is what keeps the docs group small enough to audit by hand as well.

**The `--strict` risk was checked with a throwaway notebook and did not fire.** The plan flagged that `mkdocs-jupyter` might warn on a converted file missing kernel metadata, which `--strict` would turn into a failure. A percent-format source with one markdown cell and one code cell, converted by jupytext and listed in the nav, built strict with no warning and rendered its cells; jupytext writes the kernelspec. The source and its `.ipynb` were deleted after the build. Nothing in the shipped tree depends on this, since WP-188 lands zero notebooks -- it is recorded so [WP-189](https://borda.github.io/lucid-YOLO/work_packages/wp-189/) does not re-derive it.

**Two checks moved from the site to the audit, and one rule changed shape.** `check_every_nav_entry_points_at_a_file_that_exists` used to ask whether the nav target was a file under `docs/`; a `notebooks/<name>.ipynb` target now resolves against `notebooks/<name>.py` instead, whether or not the `.ipynb` has been written yet. The alternative -- accept the entry when the generated file is on disk -- would have answered differently on a warm checkout and a cold one, and a checker whose verdict depends on whether `make notebooks` was run is a checker that passes locally and fails in CI. `check_every_notebook_source_has_a_nav_entry` is the page rule (a register nobody can navigate to is one nobody reads) applied from the source side, and it tolerates a `mkdocs.yml` with no `Notebooks` section while no source exists, because MkDocs rejects a nav section with no children and the section therefore cannot arrive before the first notebook does.

**The badge decided how the site is published.** The first draft of this row added a `publish-notebooks` job writing the `.ipynb` files to an orphan `notebooks` branch, beside the existing Pages artifact deploy -- two publishing mechanisms for one site. The revision (2026-09-21) collapses them: the site itself now goes out as a branch deploy, `ghp-import -n -p` committing `site/` onto `gh-pages` on top of that branch's history, and the converted notebooks are copied to `site/notebooks/<name>.ipynb` before the import so the badge's `blob/gh-pages/notebooks/<name>.ipynb` is a path on the branch Pages serves. `mkdocs-jupyter`'s `include_source` places its own copy at `site/notebooks/<name>/<name>.ipynb`, beside the rendered page; the flat copy is the badge's, and both ride the same commit. `ghp-import` arrives with mkdocs, so the docs group gains nothing for it; `deploy-pages`, `upload-pages-artifact`, `configure-pages` and the `pages`/`id-token` permissions leave the workflow, and `contents: write` sits on the one job that remains -- a pull-request run holds it too, since a job's grant is fixed for the job, and never reaches the step that uses it. The `Makefile` and both workflow loops iterate a shell glob with an existence test rather than `$(wildcard)`, since `for f in ;` is a syntax error under `/bin/sh` and an empty `notebooks/` is the state this row ships in.