Skip to content

Experiment retrospective

Read a completed managed numeric campaign. Do not execute measurements, modify its source, or change its original history. Historical or pass/fail runs without campaign.json have no managed reconstruction contract: explain that limitation rather than inventing state.

  1. Create a separate optimize result run. Copy the source run's campaign.json and byte-exact experiments.jsonl into it; retain the source path and both original hashes in hypothesis.md. Never replace a file in the source run. If source pending state exists, report unfinished work and keep the retrospective preliminary; do not promote it as complete.
  2. Run PLUGIN_ROOT/bin/python PLUGIN_ROOT/skills/optimize/campaign.py retro --run-dir <new-run> (native Windows: .cmd launcher). Save its stdout as retrospective.json. This validates configuration and the full ledger without requiring the original live checkout.
  3. Write comparison.md with baseline, after/best, delta, guard history, confidence and limits. Explain kept/reverted/inconclusive/failed counts, repeated hypotheses and rejected hypotheses worth revisiting. The helper's next-hypothesis list is a queue of prior non-kept approaches, not novel research or an instruction to repeat them unchanged. Add concrete next hypotheses only when evidence supports a changed mechanism.
  4. Flag large jumps relative to the campaign's observed scale for investigation, not as proof of a defect; record any absent per-iteration logs. Adaptive, selected iteration results do not establish statistical significance. Repeated independent trials and an appropriate experimental design are separate work, never fabricated from the ledger.
  5. Bind the copied campaign/log hashes in metadata.optimize_lifecycle with mode retrospective. Keep the existing final columns for every retained iteration or exactly one Iteration=campaign summary, using the exact numeric, guard and decision conventions in Managed campaigns. Explain in prose that guards and decisions describe historical attempts; they are not new measurements. Mark code lint/type/test execution inapplicable when no executable change or probe occurred; run review and artifact checks and complete the ordinary final handoff. Lifecycle admission validates retained history and reconciles the structured Results cells without requiring live source, but unresolved pending state cannot be promoted.