Codex agentic explicit-population re-run — 2026-09-08¶
results/codex-agentic-20260908T074844Z — gpt-5.6-luna, 48 of 48 cells persisted, started 2026-09-08T07:48:48Z, scope 57c54854… under manifest 38dffa99…, high reasoning effort on Codex CLI 0.153.4, one repetition per cell, against the same pytorch-lightning revision be98784a1 and frozen index 3c584089… every other table here uses. This run is not in the cross-provider table and does not replace any row in it.
This is a third experiment revision, and the change is in the prompt. experiment_revision reads codex-agentic-explicit-population-pass-metrics-2026-09-08. All 16 prompt_sha256 values differ from the sweep's even though only one task's own text changed, because a new ORACLE_POPULATION_INSTRUCTION block in benchmarks/_bench_common/agentic_contracts.py is now injected into every rendered prompt. It states the module-naming rule (a/b/__init__.py is a.b, never a.b.__init__), the test-module convention, that production importers exclude test modules and the target module itself, that every rdep count counts non-test direct importers excluding the counted module, deduplication, and the sort-then-tiebreak rule for rankings. The headline metric also changed: scoring/metrics drops SCORE for PASS and a secondary COMPONENT, reporting_version is agentic-pass-v1, and scoring/efficiency_policy now requires pairing control with treatment only where both cells pass. Arm order stays the sweep's cyclic-arm-order-v1 rotation. Nothing here is poolable with any earlier agentic study, and no earlier row is rescored.
| Arm | Pass | Component | Gross input | Fresh input | Output | Runtime | used / calls |
Help calls |
|---|---|---|---|---|---|---|---|---|
A_plain |
16/16 | 1.000 | 2.88M | 532.1k | 103.8k | 48.8 min | 0/16 · 0/16 | 3 |
B_auto |
15/16 | 0.996 | 3.78M | 596.8k | 63.5k | 33.3 min | 16/16 · 8/16 | 54 |
C_strict |
12/16 | 0.948 | 2.22M | 432.0k | 56.7k | 28.5 min | 16/16 · 16/16 | 2 |
Paired under the revision's own both-pass policy, as per-task medians:
| Pair | Both pass | Improved | Regressed | Tokens | Fresh | Output | Time |
|---|---|---|---|---|---|---|---|
A_plain vs C_strict |
12 | 0 | 4 | −35.1% | −27.8% | −54.4% | −42.3% |
A_plain vs B_auto |
15 | 0 | 1 | +25.3% | +9.0% | −35.4% | −22.4% |
The admission rule again removes nothing on its own terms: 48 of 48 cells scored, adherent, complete, uncontaminated, no envelope loss. Every exclusion above is a pass-gate exclusion, not a validity one.
The control saturates, and the prompt change is why. It answers all 16 tasks correctly at component mean 1.000, against 13 of 16 for the same stratum in the sweep. Run-to-run noise does not account for it: BA-01, BA-12, and BA-15 were failed by the control in every stratum of the sweep — luna, terra, and sol alike — and by every earlier execution's control; all three pass here. Three tasks that were stable failures across three independent strata do not flip together by chance. They flip because what they tested was undeclared population semantics, and the prompt now declares them. A large part of the Codex C_strict accuracy gain published above was the treatment arm compensating for prompt ambiguity the control had to guess at. With the ambiguity moved into the prompt for all three arms, that compensation has nothing left to buy, and this stratum's strict-arm accuracy gain inverts from +6.2 points to −25.0.
The strict arm's four misses are agent slips on correct tool output, not wrong tool data. Component scoring localizes them: every enumeration field — production_importers, buckets, overlap_importers, cross_namespace_importers, and all six *_count fields — reads 1.000 in all three arms, and only ranking (0.800) and rdep_counts (0.981) move, in C_strict alone. BA-10 settles the direction. The strict arm ran query --compact rdeps lightning.fabric.plugins --exclude-tests --limit 0; the frozen transcript's imported_by list holds exactly 29 modules, and recomputing the oracle set from the archived locked-index.json returns the same 29 with an empty difference both ways. The agent then reported 30. In each of the three ranking failures the omitted module is already in that arm's own, correct, importer list — lightning.pytorch.trainer.call on BA-03, lightning.pytorch.utilities.types on BA-08, lightning.pytorch.strategies.strategy on BA-11 — and BA-11's answer is the oracle's ranking with one entry deleted and the rest shifted up. On BA-03 the arm ranked the target module itself, which the stated population rule excludes. A truncated centrality window would explain a dropped candidate and is ruled out: BA-11 queried central --top 100 and the omitted module sits at global rank 48, inside it.
The optional arm's regression was prompt ambiguity, and this run turns that diagnosis into a result. B_auto ends at 15 of 16, its one miss an off-by-one in high_centrality on BA-16 — the same class of slip as BA-10. The sweep's analysis inferred from component scores that the optional arm ranked by raw reverse-dependency counts including tests and examples while the task ranked within production importers; spelling that population into the prompt moves this stratum's optional arm from −31.2 points to −6.2 and returns ranking to 1.000. The behavioural difference outlives the accuracy one: B_auto still spends 54 heuristic help/discovery calls against 2 and 3, and still reads more gross input than the control while the strict arm reads less.
The resource result survives the disambiguation intact, and is now the load-bearing claim for this stratum. The strict arm is lower on 11 of 12 both-pass cells for elapsed and 8 of 12 for fresh input, and the cohort totals agree in direction. Cached input is 81.5% of control gross, so the token column is a gross-input claim and the fresh figure is the one that carries a cost reading.
Recall no longer separates the arms here. EREC and RREC read 1.000 in all 48 cells, control included — the first execution in this record with no sub-recall control cell — so the evidence-recall column carries no signal for this stratum. DEFF inverts with it, 4.66 for the control against 2.88 for the strict arm, because the control issues fewer commands per cell and each touches proportionally more of the answer.
Only luna ran at this revision. Terra and sol were put on byte-identical prompts the same day and both follow it; see Codex agentic explicit-population strata. Full per-task analysis, including the per-cell failure traces: .reports/benchmarks/codex-agentic-explicit-population-rerun-analysis.md.