Codex results — gpt-5.6-sol — 2026-09-07¶
results/codex-combined-20260906T221515Z/structural/gpt-5.6-sol — gpt-5.6-sol at high reasoning effort, Codex CLI 0.153.4, 219 cells over 73 tasks in five stages, one repetition per cell, a 600 s wall-clock cap per cell, against pytorch-lightning 2.6.5 at be98784a1 with frozen index 3c5840893e9c939baa61a6c5ce95994ff69ffe4a67d225aeb412c73deb61e0c1. Every stage is marked completed in its run-metadata.json and all 219 cells persisted; the structural stage ran from 2026-09-06T22:20:45Z to 2026-09-07T01:26:08Z. execution.codex_cli records observed_version: codex-cli 0.153.4 against reviewed_version: codex-cli 0.146.1, so this executed on a CLI build the methodology had not reviewed, and auth_source_recorded is false. Full analysis: .reports/benchmarks/codex-sol-paid-analysis.md.
This run is half of the launch it came from, and remains a one-stratum result. Its sibling gpt-5.6-terra stratum ran no paid cell in that launch: the launcher refused with ERROR: paid Codex mode requires CODEX_PAID_APPROVAL=… before the first model call, leaving structural/gpt-5.6-terra with a run log and no telemetry. Terra was relaunched separately on 2026-09-07 and completed — its section follows — but it is a different study on a different launch, so nothing in this section may be restated as a two-stratum Codex result.
The binary paired rows on the shared 45-task headline cohort are in Structural results — every model on one cohort. On the run's own semantic-quality metric, over the stricter cohort the analysis uses — the 44 tasks where all three arms produced an admissible scored cell — mean quality is A_plain 0.9477, B_auto 0.9889, C_strict 0.9742, and the median is exactly 1.000 in every arm, with 36 of those 44 tasks scoring identically in all three arms. Paired per-task deltas on that cohort: C_strict against A_plain is +0.0265 mean and 0.0000 median at 6 wins / 36 ties / 2 losses, and B_auto against A_plain is +0.0412 mean and 0.0000 median at 7 / 36 / 1. Input tokens move where quality does not: C_strict is cheaper on 38 of those 44 tasks at a median paired delta of −51,666 gross tokens, while B_auto matches its control at +566. The honest reading is that the cost effect is real and large and the quality effect is indistinguishable from zero at this sample size.
Structural quality here is ceiling-bound: with medians of 1.000 in all three arms, this design can detect degradation and cannot detect improvement, so a 0.0 pp gain is a tie against a saturated control rather than evidence that Codemap does nothing.
Medians alone make the two treatment arms look equally good, and per-task spread is what separates them. Over the same 44 pairs, the per-task input-token ratio against A_plain runs 0.254–1.066 between the 10th and 90th percentiles for C_strict (full range 0.054–1.174, cheaper on 38 of 44) against 0.535–1.951 for B_auto (full range 0.205–2.740, cheaper on 19 of 44). The dispersion itself is nearly the same — the standard deviation of the log ratio is 0.593 for C_strict and 0.549 for B_auto — so the difference is where the distribution sits, not how wide it is: the required arm's interval lies below parity and the optional arm's straddles it with a heavier losing tail. Wall clock behaves the same way, 0.272–1.029 against 0.439–1.315. On quality the asymmetry reverses: B_auto's worst task loses 0.028 while C_strict's worst loses 0.500, so the one large quality regression in this run belongs to the required arm.
Optional access is not failing for lack of uptake. Twenty-six of those 44 B_auto cells issued at least one successful Codemap query, and splitting the token ratio on that boundary barely moves it — 0.974× median when Codemap was queried against 1.016× when it was not. The optional arm queries Codemap in addition to ordinary exploration rather than instead of it, which is why availability alone buys nothing.
The four executable and extraction stages are separate, nonpoolable strata, and their token direction is heterogeneous enough that averaging them would invert the finding:
| Model | Stage | Cells | A_plain correct | B_auto correct | C_strict correct | B tokens | C tokens | B time | C time |
|---|---|---|---|---|---|---|---|---|---|
| Sol | ReadCrop | 18 | 5/6 | 5/6 | 5/6 | +60% | −34% | +25% | −34% |
| Sol | Fix-Single | 12 | 4/4 | 4/4 | 4/4 | +16% | +15% | +3% | +16% |
| Sol | Fix-Multi | 9 | 3/3 | 3/3 | 3/3 | −15% | +34% | −55% | −41% |
| Sol | Patch | 15 | 5/5 | 5/5 | 4/5 | −36% | −67% | −11% | −23% |
Token and time columns are cohort totals restated as change against A_plain, the same estimator the Luna stage table above uses. Requiring Codemap costs more on localized and multi-file editing — +15% on Fix-Single and +34% on Fix-Multi — and saves on ReadCrop and Patch, so the four stages must be read one at a time. The Patch saving is carried almost entirely by two cells where A_plain spent 536,886 and 1,464,515 gross input tokens against C_strict's 112,210 and 153,314. ReadCrop's 5/6 is RC-03 failing primary_correct identically in all three arms, not an arm difference.
B_auto abandons the tool wherever code has to be modified. Voluntary uptake is 5 of 6 cells in ReadCrop and 35 of 55 in the structural stage, but 0 of 12 across Fix-Single, Fix-Multi, and Patch — not one editing cell chose to call Codemap. That is why the editing-stage B figures track A_plain, and why an optional rollout should not be expected to reproduce the C_strict result.
The run's only correctness failure in these four stages is PT-04 under C_strict, the Codemap-required arm, where A_plain and B_auto both passed — at 536,886 and 540,165 gross input tokens against C_strict's 112,210. The strict arm was 4.8× cheaper and wrong. Luna lost the same cell in the same arm; two failures of one task at one repetition each still support no efficiency or quality claim in either direction, but they do cut against reading the strict arm as strictly safer.
What the C_strict evidence does and does not establish: skill_delivery_observed is false in all 219 cells, including every C_strict cell whose codemap_delivery reads installed_skill. The _arm_compliance contract sets the delivery prefix by construction and treats skill_delivery_observed as the separate observational signal, and it never fired. There is therefore no observational proof that the Skill file was read in any cell; the claim the data supports is that required Codemap querying produced the effect, not that the Skill was read. Delivery is also confounded with arm — structural B_auto used direct_cli in all 35 of its Codemap cells and C_strict used installed_skill in all 55 — so no B-versus-C difference can be attributed to strictness alone.
Data-quality notes for this run: one repetition per cell throughout, so no cell has a variance estimate and no significance testing was performed; the agentic-family stages carry 3 to 6 tasks each, where a single task flip moves a pass rate by 17 to 33 points. A_plain CQ-05 timed out at 600.12 s with zero tokens and null quality — it is a diagnostic-cohort cell and cannot reach any headline figure. B_auto RV-02 is the run's one extraction_failed cell and is the reason the B comparison pairs 44 tasks where C pairs 45. canonical_telemetry_pooling_eligible is false on the structural stage, which is a whole-file admission flag raised by those two run-level failures; it forbids pooling the canonical file wholesale, not per-cell filtering. Token figures throughout this section are gross input including cached reads — see the fresh-token caveat under the aggregated table.
This run has no agentic blast-radius stage, so gpt-5.6-sol has no row in the agentic lane below.