Codex Quality Gates¶

Detailed codex-native workflows emit the same gate fields. Documented lightweight local paths run applicable project checks directly and report their evidence without creating this bundle.

Apply Acceptance and recovery: reviewer/report formatting and wording hints do not become failed acceptance checks. Preserve semantic structure and factual evidence; automatically handle equivalent presentation at the owning reader and retain raw evidence. Calibration benchmark wording misses are advisory entries in checks.txt; structural contract and behavioral failures retain their normal failed-check accounting. Configured project checks still run, with authorized automatic formatting/correction and relevant reruns; a tool's diagnostic alone does not establish its impact on delivery.

Review limitations and additional review¶

Deliver the full review supported by the evidence actually obtained. Missing independent coverage, unavailable reviewer tools, unproven context isolation, incomplete provenance, unexecuted checks and confidence limitations describe the review's limits; they are not source defects, severity findings, operational-blocker rows or automatic remediation tasks. Preserve actual failed checks and qualify readiness and conclusions honestly; never fabricate passing evidence or unconditional approval. A concrete source defect supported by evidence remains a finding even when coverage is limited.

If an additional review beyond the already authorized scope would resolve a material uncertainty, ask once for a decision before dispatch. State the exact question or gap, bounded scope, expected evidence/output and stopping condition, and offer to finish with the stated limitation. Existing explicit authorization for that same pass remains valid. A declined or dismissed optional review is skipped: do not re-ask, dispatch an alternative reviewer, or carry the request into final findings, blockers, recommendations, next steps or remediation selection. Retain only the factual limitation describing what was not verified. An unanswered optional question never authorizes dispatch; finish the available review with its limitation. An explicitly pending required decision keeps only its dependent work paused.

One accepted additional pass does not authorize another pass or a review/fix loop. An explicitly requested convergence workflow retains its bounded mechanics; its coverage gaps still remain limitations rather than synthetic source findings. Reviewers assess the assigned source and report real defects, not requests to review their own review. A report-intake consumer preserves this distinction, including for historical reports that incorrectly promoted a process limitation into a finding.

An explicit user decision may waive additional reviewers, review rounds or broader review coverage, including independence the user previously requested. Treat this as a change to the review scope, not a gate override. Record the user's decision, exact target/revision, waived coverage, retained requirements and resulting limitation in the review notes. Preserve frozen plans and rejected evidence; use the owning workflow's separate fallback plan for the revised scope. A declined optional repair alone does not revoke an earlier explicit independence requirement. Reuse an already explicit waiver for the same target and gap; never ask again or dispatch waived work.

Source identity/freshness, evidence integrity and provenance for claimed work, truthful check results, artifact validation, promotion and completion checks are never waivable. Keep required project checks and their actual failures; a coverage waiver never authorizes merge approval, fabricates independent evidence, turns an unexecuted check into a pass or admits an invalid artifact. A review can complete within its revised scope only after the normal validators and completion gates succeed; otherwise deliver the supported findings as a clearly incomplete report. State review completion separately from merge readiness.

Required checks¶

lint: ruff check (or project lint command)
format: formatter check or formatter run result
types: mypy (or project type check command)
tests: impacted tests at minimum, full suite for broad changes
review: self-review diff/risks; check request conformance, constraints, disclosed deviations

Required gate output¶

Each skill run must write:

status: pass, fail, or timeout
checks_run: list of check ids that were executed
checks_failed: list of check ids that failed
findings: severity counts {critical, high, medium, low}
confidence: numeric 0.0 to 1.0
artifact_path: absolute or repo-relative path to result artifact

Confidence needs objective evidence. Before user output, apply bands: <= 0.8 unacceptable; skill artifact returns status=fail or status=timeout with confidence-not-acceptable in checks_failed. 0.8 < confidence < 0.85 very questionable; return status=fail or status=timeout with confidence-very-questionable. 0.85 <= confidence < 0.9 cautious-low; proceed only with objective evidence, recovery actions, remaining limits. >= 0.9 fair but not automatic; name evidence-backed material limits. Result JSON records metadata.confidence_recovery: initial score, evidence, recovery actions, recomputed score, band status, remaining limits. Chat reports score and material limits; artifact holds detailed recovery/closure evidence. Every scored skill/agent output lists confidence gaps or degradation reasons. Close each with evidence or explicit unresolved/deferred record. Preserve result fields status, checks_failed, and confidence_gaps. Write metadata.confidence_gaps and metadata.confidence_gap_closures; when confidence < 1.0, include one concrete gap/residual limit and matching closed, unresolved, or deferred closure.

Each run_gates.py invocation writes gates.json with exactly five IDs. Entries contain id, status, exit_code, duration_seconds, command_path, stdout, stderr; missing-command, not-applicable, timeout also need reason. The three log fields record path relative to run's output directory in POSIX form (checks/<id>.<kind>.txt); absolute path is accepted only inside that directory. Readers accept any relative entry that resolves inside output directory, trying --out first, then its ancestors for runs written before this convention — never their own working directory, which is what once made same artifact valid in one directory, invalid in another. Every accepted log must resolve inside output directory. not-applicable passes only with explicit reason; missing-command/timeout fail. Result status/check lists reconcile with gates.json.

For --pytest-python source binding, keep the project's pytest selection and parallel options. The test gate records selected test paths and declared module origins from each executing pytest process, including xdist workers. A missing or crashed worker receipt, an observed origin outside the verified source, or an untracked origin without valid local snapshot membership cannot certify reviewed source; retain the failing or inconclusive gate evidence. A module absent in one worker remains neutral when another worker proves its tracked origin or admitted local snapshot membership. Clean PR mode still requires its expected head and tracked origins. A local review mirror may omit --expected-head only when --out already contains the valid collector local-source/review-worktree.json and local-source/source-snapshot.json for that exact mirror, with no competing PR receipt; the runner freezes a current runtime snapshot before gates. Local test/module origins must match regular-file snapshot members and current bytes, so a collector-admitted added file may keep tracked=false only with that proof. This local admission does not relax clean PR or release gate contracts, and old unbound logs cannot be promoted retroactively.

Optional but recommended:

recommendations: list of concrete next improvements
follow_up: list of prioritized next actions
metadata: required with confidence for machine-readable gaps/closures; otherwise skill-specific evidence not hidden in prose notes

Confidence Deduction Accounting¶

For every reported confidence gap or limitation, put its contribution first, before the description: (-0.NN) Description using an ASCII minus and two decimals. Unique deductions must sum to exactly 1.00 - score at the displayed precision; choose the displayed score and deductions consistently. Keep existing confidence bands and evidence requirements.

A limitation that did not reduce the score carries (-0.00); retain its evidence or unresolved/deferred rationale.
Name any cap, floor, or band that determined the score and show its contribution. Attribute overlapping causes without double-counting; when a limit repeats a gap, reference the same label and count its deduction once.
Put shortfall no named gap explains in one explicit residual entry with its deduction and reason; never hide it or silently distribute it across named gaps. A score below 1.00 requires at least one gap or residual entry.
This is transparent judgment accounting, not an empirically calibrated probability or measured causal decomposition. Deductions remain evidence-backed estimates; arithmetic precision does not establish statistical precision.

A disclaimer that a confidence score lacks empirical calibration is not itself a failed check or unfinished remediation task. Preserve it as an interpretation limit; do not invent a calibration requirement from it or from selecting all actionable items. If it arrives as a report obligation, retain its provenance and record an evidence-backed not-applicable disposition citing this contract. Explicitly requested calibration experiments, project-required calibration checks, actual coverage gaps, failed confidence thresholds and unsupported confidence claims still require their normal evidence and closure.

Example: confidence 0.96; (-0.02) unavailable target-host execution, (-0.02) unexercised live workflow, (-0.00) installation excluded from the requested scope. The two reducing limitations account for the 0.04 shortfall; the scope disclosure contributes zero.

Fail rules¶

An execution failure never makes an applicable check not-applicable. Missing tools, launcher panics, or sandbox restrictions require diagnosis and preserved failed evidence. Choose the required check entrypoint before execution. Do not rerun passing checks solely to wrap them in a different report or obtain a second approval: lightweight paths retain direct-check evidence, and detailed paths reuse an existing valid canonical receipt for unchanged source, selection, environment, and command. A direct check cannot silently replace a required source-bound canonical gate; if that proof was not collected, run the required gate or disclose it as incomplete. Same-directory reruns archive runner-owned receipts under gate-attempts/<NNN> before executing and reject failed-to-skipped reclassification. Incomplete prior state blocks overwrite; diagnose it before starting a separately scoped run.
Any critical finding => status=fail
Any failed check in checks_failed => status=fail
Missing command/tool for required gate => status=fail
Explicitly unsupported gate with reason => that gate's per-check gates.json entry status=not-applicable; non-failing, excluded from checks_failed (top-level run status stays pass/fail/timeout only — see "Required gate output")
All five gates.json entries not-applicable => top-level run status=pass (zero failures to report)
Missing artifact => status=fail
Gate timeout => status=timeout
For every skill/agent: confidence <= 0.8 cannot pass; 0.8 < confidence < 0.85 cannot be complete; 0.85 <= confidence < 0.9 needs recorded recovery/limits; confidence >= 0.9 still needs evidence.

Artifact path contract¶

Path: .reports/codex/<skill>/<YYYY-MM-DDTHH-MM-SSZ>/result.json
Optional: notes.md in same directory

Final Chat Contract¶

Every final chat is compact, outcome-coupled handoff in this order:

Outcome: name the topic or question being answered and state decision or completed result first, including gate status when it changes whether work is complete. The opening must stand alone; avoid unanchored “both,” “that,” or “yes” without repeating the conversation.
Results: show every decision unit needed to understand outcome. Use skill's required table when multiple findings, changes, checks, iterations, recommendations, or artifacts need distinct dispositions; do not collapse skipped, rejected, deferred, or unresolved rows.
Verification: name checks run and their exact results, including skips, failures, and unavailable checks.
Remaining: list every unresolved, deferred, skipped, externally owned, or environment-blocked obligation with its owner and next action. Write None only when evidence closes all obligations.
Recommendations / next steps: give prioritized owner/action entries that move unresolved work or accepted recommendation forward. Reference result rows instead of repeating them; write None when no follow-up is justified.
Confidence: report score, material limits, and any unresolved confidence gap under shared confidence-band policy and Confidence Deduction Accounting.
Artifact: link the canonical result artifact and any essential supporting report. An artifact is supplemental, never a substitute for the outcome, result rows, verification, remaining-work disclosure, or next steps.

Keep headings and tables proportional to result. A single decision may use one short result row or compact field list; branch-heavy results use separate bullets or tables instead of one long paragraph. Preserve skill-specific terminal-close or unavailable-output exceptions when their contracts prohibit normal tables or recommendations.

Skill contracts retain local outcomes, exact results, domain content, and terminal exceptions; other fields inherit.

Native contract¶

Skill files should follow native-skill-contract.md. Configured skills require:

Input Schema
Workflow
Fail-Fast Rules
Quality Gates
Calibration Hooks
Output Contract

Configured agents require:

Boundaries
Evidence Standard
Output Contract
clear TRIGGER, SKIP, and NOT for routing clauses

Execution helpers¶

Use PLUGIN_ROOT/bin/python PLUGIN_ROOT/shared/run_gates.py to execute five checks consistently.
Use executable PLUGIN_ROOT/shared/write-result.py to write canonical JSON result payloads.
Use PLUGIN_ROOT/shared/final_handoff.py and final-handoff-contract.md after gates to validate and render exact final-response structure. This checkpoint is not sixth gate; its failure blocks result promotion.
Use PLUGIN_ROOT/bin/python PLUGIN_ROOT/shared/collect_diff.py to collect scope-aware git diff artifacts consistently.
github_read.py is plugin-wide GitHub data boundary. Use it for all issue/release/repository reads and explicit GraphQL query for Discussions; use collect_pr.py --checkout for complete PR evidence. Never invoke gh directly. The broker prefers authenticated gh but never inspects credentials; it permits only audited built-in view groups (gist, issue, pr, project, release, repo, ruleset, run, workflow), REST GET, GraphQL query argv, persists no CLI failure output. Its public api.github.com fallback is unauthenticated, cannot replace private-only evidence, so complete PR collection fails closed if its review-thread query cannot run. gh pr checkout changes only local checkout. Codex Git marketplace add/upgrade remains separately authorized lifecycle operation.
Apply native-skill-contract.md Networked CLI Approval to each intentional shell-network path: keep persistent workspace networking disabled and approve complete owning command, not only nested executable.
PR checkout/update artifacts never record git/gh --force; force needs stop-and-ask confirmation with overwrite-risk rationale first.
Use PLUGIN_ROOT/shared/validate-artifacts.py to validate common report, ledger, gate-log, and result JSON artifacts.
Schema-v2 results must bind final-handoff.json, final.md, and final-handoff.validation.json; after validation and promotion, emit final.md verbatim. Historical schema-v1 results remain readable without this binding. Approved nested review uses Nested review delivery; all artifact and completion gates remain mandatory.
Use PLUGIN_ROOT/shared/severity-map.md to map findings to severity levels.

Behavior-Change Guardrails¶

Codex behavior changes must also update registrations, docs, calibration in same patch. Prefer existing agent/skill unless new role/workflow has distinct triggers, acceptance criteria, measurable gates.

A passing assessed audit result additionally requires an executed passing review gate. The generic runner still permits all not-applicable diagnostics; skill applicability checks decide whether those records can certify workflow completion.

New terminal handoffs use presentation version 8. Keep full evidence in machine records; display file paths outside every table as - Item: file-path lists. Presentation normalization is automatic; factual bindings, source coverage and authority checks retain their existing meaning.