Codex agentic explicit-population strata — 2026-09-08¶
results/codex-agentic-20260908T140949Z — one launch, two sequential child studies, 96 of 96 cells persisted, under manifest f82e99a0… and sweep authorization cb63fbb1…:
Terra—gpt-5.6-terra, started 2026-09-08T14:09:55Z, scope0cf127ad….Sol—gpt-5.6-sol, started 2026-09-08T15:44:33Z, scopef81b6a65….
Both ran at high reasoning effort on Codex CLI 0.153.4, one repetition per cell, 600 s per cell, against the same pytorch-lightning revision be98784a1 and frozen index 3c584089… every other table here uses. Neither is in the cross-provider table and neither replaces a row in it.
This is a fourth experiment revision, and the prompts are the third revision's, unchanged. experiment_revision reads codex-agentic-relocated-coordination-gate-2026-09-08. The comparison against the luna re-run is exact, because the two artifacts that determine prompt text are byte-identical across the two revisions: the task suite (semantic_suite_sha256 cd1a8bae…) and benchmarks/_bench_common/agentic_contracts.py (13ec30e5…), which carries both the rendered task text and the ORACLE_POPULATION_INSTRUCTION block. What changed is downstream of the prompt: the Codemap read/write coordination gate now lives outside the measured workspace at a disposable path beside each cell's home, addressed through the new CODEMAP_COORDINATION_DIR variable, and the frozen treatment is Codemap 0.35.0 rather than 0.34.0. That relocation is why this run exists — the previous attempt at these prompts, results/codex-agentic-20260908T094926Z/gpt-5.6-sol, aborted after 2 of 48 cells when a model wrote a scratch script into the gate beside the frozen index and the fail-closed validator refused to clean it up. Nothing here is poolable with any earlier agentic study, and no earlier row is rescored.
| Stratum | Arm | Pass | Component | Gross input | Fresh input | Output | Runtime | used / calls |
Help calls |
|---|---|---|---|---|---|---|---|---|---|
| Terra | A_plain |
16/16 | 1.000 | 2.13M | 348.1k | 71.4k | 36.2 min | 0/16 · 0/16 | 2 |
| Terra | B_auto |
16/16 | 1.000 | 2.85M | 368.1k | 63.0k | 28.6 min | 6/16 · 3/16 | 32 |
| Terra | C_strict |
13/16 | 0.965 | 2.29M | 364.6k | 44.1k | 28.5 min | 16/16 · 16/16 | 1 |
| Sol | A_plain |
16/16 | 1.000 | 2.42M | 340.2k | 70.6k | 48.4 min | 0/16 · 0/16 | 2 |
| Sol | B_auto |
15/16 | 1.000 | 3.41M | 462.5k | 64.5k | 46.2 min | 9/16 · 7/16 | 41 |
| Sol | C_strict |
15/16 | 0.991 | 1.90M | 336.7k | 38.0k | 30.0 min | 16/16 · 16/16 | 2 |
Paired under the revision's own both-pass policy, as per-task medians:
| Stratum | Pair | Both pass | Improved | Regressed | Tokens | Fresh | Output | Time |
|---|---|---|---|---|---|---|---|---|
| Terra | A_plain vs C_strict |
13 | 0 | 3 | +15.4% | +14.8% | −48.2% | −23.3% |
| Terra | A_plain vs B_auto |
16 | 0 | 0 | +21.0% | +9.6% | −23.5% | −29.6% |
| Sol | A_plain vs C_strict |
15 | 0 | 1 | −7.6% | −1.9% | −47.7% | −42.1% |
| Sol | A_plain vs B_auto |
15 | 0 | 1 | +43.2% | +32.6% | −5.9% | −1.8% |
The admission rule again removes nothing on its own terms: 96 of 96 cells scored, adherent, complete, uncontaminated, no envelope loss and no diagnostic bare-JSON recovery. Every exclusion above is a pass-gate exclusion, not a validity one.
The inversion generalizes: all three Codex strata now saturate the control. Both controls answer 16 of 16 at component mean 1.000, as luna's did. In the balanced-order sweep the same controls read 12 and 9 correct — terra failing BA-01, BA-12, BA-13, and BA-15, sol failing those four plus BA-02, BA-09, and BA-10. All eleven of those cells pass here. The sweep's flagship result, the strict arm converging on 14 of 16 while the controls read 13, 12, and 9, was measuring how much of the answer population each arm had to guess at rather than how much structural insight the tool supplied.
The strict arm is now the only arm that fails, in every stratum. Terra loses BA-12 on excluded_test_importer_count (56 against 57), BA-15 on a ranking with one entry deleted and the rest shifted up, and BA-16 on an invented high_centrality entry where the oracle expects none. Sol loses BA-08, again a ranking with one module dropped. The failure signature is luna's, unchanged: EREC and RREC read 1.000 in all 96 cells, no arm ever failed to surface an expected importer, and in both ranking failures the omitted module is present in that same cell's own correct production_importers list. Component scoring confines the damage to ranking (0.143 and 0.400 in the two failing cells) plus terra's single count and centrality fields; every enumeration field elsewhere reads 1.000. BA-08 is the one task the strict arm has now failed in three separate executions — sol's sweep cell, luna's re-run, and sol's cell here — but it drops a different module each time, which reads as an assembly slip rather than a disputed oracle.
All four strict-arm losses are hand assembly over correct tool output, and they trace to one missing query. Recomputing from the frozen index confirms the tool's data in every failing cell: the run's own rdeps output gives the terra count task 19 production and 57 test importers against the answered 56, and ranking the sol task's 70 importers by their own in-degree reproduces the oracle's top five exactly, including the module the arm dropped. Each task asks the same shape — rank or count within this candidate set — and Codemap 0.35.0 as measured could not express it: central ranked the whole repository and rdeps returned one module's importers as names with no count. So the arm either issued one rdeps per candidate and counted the lists by eye (20 and 9 calls in the two counting tasks) or filtered a repository-wide ranking against its candidate list by hand. Both failures of the second kind dropped or added exactly one row; the module the sol task lost sits at global rank 69 in that ranking, above the rank-103 and rank-116 rows the arm kept, so it is a scanning slip and not a truncation. The control never reaches that step: with no tool it derives each candidate's count as it enumerates, so its working set is already scoped. query central --among <modules> and the importer_count / excluded_test_importer_count fields were added after this run to close the gap — they are not in any measured artifact here, and testing whether they change the result needs a new revision and a fresh paid approval.
The optional arm's regression was prompt ambiguity, and these two strata close the question. B_auto recorded 10 semantic failures on terra and 11 on sol in the sweep, and zero in both here. Terra passes 16 of 16, the first Codex optional-arm execution in this record that does not regress against its control. The sweep inferred from component scores that this arm ranked by raw reverse-dependency counts including tests and examples while the task ranked within production importers; stating that population in the prompt removes the regression in all three strata. The behavioural difference outlives the accuracy one: heuristic help/discovery calls run 32 and 41 in B_auto against 2 in both controls and 1 and 2 in C_strict. Uptake moved the other way here — codemap_used reads 6 of 16 and 9 of 16 against 13 of 16 for both in the sweep, while luna rose from 14 of 16 to 16 of 16 at the same prompt — so that shift is per-stratum and no mechanism is attached to it.
Sol's one optional-arm loss is a measurement artifact, and the relocated gate both caused and contained it. BA-02 B_auto is scored execution_failure/cleanup_failed on an answer that is fully correct: quality.correct true, component 1.000, recall 1.000, envelope valid, treatment adherent. The frozen command list shows the model probing the sandbox for a writable path and then writing and running a scratch analysis script in the coordination gate, which is the single writable path in an otherwise read-only profile. Read the row at both layers — pass is 15 of 16 as measured, semantics are 16 of 16. Relocation did what it was for: the frozen index directory stayed clean and the run completed all 48 cells instead of aborting at 2. The cell-level cost is unchanged in this artifact. It is closed for future runs by the first of the two available routes: the harness now scrubs foreign entries from its own disposable gate at teardown and records their names on the row as coordination_scratch_entries, so a permitted write is an observation about how a cell worked rather than a failure that overwrites a correct answer. Teardown still fails closed on the two conditions that mean something is wrong outside the gate — a symlink in the path to it, or a lock a living process still holds. That change mints another revision, so no row above is rescored by it.
Only two of the four resource columns reproduce across strata. Output tokens read −48.2% and −47.7% against luna's −54.4%, and elapsed −23.3% and −42.1% against −42.3%; the strict arm is lower on both in 11 of 13 (terra) and 13 of 15 (sol) both-pass cells. Gross input does not hold a direction at all — −35.1% on luna, +15.4% on terra, −7.6% on sol — and fresh input agrees with it (−27.8%, +14.8%, −1.9%). Cached input is 84% and 82% of the strict arm's own gross, so every token column here remains a gross-input claim. DEFF has likewise lost its direction now that recall is pinned at 1.000: control 5.32 against strict 4.00 on terra, control 3.89 against strict 4.87 on sol.
Full per-task analysis, including the per-cell failure traces and the gate recommendation: .reports/benchmarks/codex-agentic-explicit-population-strata-analysis.md.