π¬ research β Claude Code Plugin¶
research turns a vague ML improvement idea into a reviewable path: gather literature, write a measurable experiment contract, check the methodology, run bounded iterations, and inspect what changed. Experiment state and reports stay in the project so the next maintainer can see what was tried.
Value at a glance: research connects literature, code, metrics, guards, commits, ablations, and retrospective evidence in one namespaced plugin while leaving datasets, compute, credentials, and scientific judgment with the project owner.
Current limits at a glance: the plugin does not provide data, GPUs, credentials, or companion plugins;
/research:run --codexrequires the installed and enabledbridge@borda-ai-rigplugin; unavailable explicit integrations stop the requested path rather than silently degrading; metric proxies still require human validation.
Contents
- [What research solves](#-what-research-solves) - [Install](#-install) - [Quick start](#-quick-start) - [Workflow index](#-workflow-index) - [`/research:topic`](#researchtopic) - [`/research:plan`](#researchplan) - [`/research:judge`](#researchjudge) - [`/research:run`](#researchrun) - [`/research:sweep`](#researchsweep) - [`/research:verify`](#researchverify) - [`/research:fortify`](#researchfortify) - [`/research:retro`](#researchretro) - [`/research:kaggle`](#researchkaggle) - [`/research:setup`](#researchsetup) - [Experiment contract](#-experiment-contract) - [Workflow overview](#-workflow-overview) - [Agents and optional integrations](#-agents-and-optional-integrations) - [Hooks, rules, artifacts, and bin tools](#-hooks-rules-artifacts-and-bin-tools) - [Current boundaries](#-current-boundaries) - [Troubleshooting](#-troubleshooting) - [Contributing and maintenance](#-contributing-and-maintenance) - [Acknowledgments and license](#-acknowledgments-and-license)π― What research solves¶
Without a contract, ML work often becomes intuition β experiment β unclear result β repeated effort. Baselines drift, proxy metrics go unquestioned, paper details are misimplemented, and GPU hours can be spent before a guard or split audit catches the design flaw.
With research, the evidence path is explicit:
/research:topicgathers literature and maps a recommendation to the current codebase./research:planrecords the goal, metric, guard, scope, strategy, and budget inprogram.md./research:judgechecks methodology, scientific rigor, and command executability before expensive work./research:runproposes one scoped change per iteration, commits before measurement, keeps guarded improvements, and reverts regressions./research:retroanalyzes significance, dead iterations, suspicious jumps, and next hypotheses./research:verifychecks whether an implementation matches a named paper across formulas, hyperparameters, evaluation, notation, and citations./research:fortifyisolates components in worktrees to test which changes mattered.
Every step is explicit and reviewable. The chain is not a guarantee of scientific validity, and a user still decides whether a result is worth adopting.
π¦ Install¶
Prerequisites are Claude Code with plugin support and Python 3.10+ for setup. /research:run also requires Git and starts only from a clean worktree. Datasets, project dependencies, compute, credentials, and optional companion plugins remain user-managed.
After installation, run /research:setup once to deliver this plugin's rules/quality-gates.md into the user rule directory. Run it again after an upgrade if the rule link is stale. The plugin is independently installable; most optional specialist paths degrade to general-purpose with a role prompt when Foundry is absent.
For companion development and release workflows:
claude plugin install foundry@borda-ai-rig
claude plugin install develop@borda-ai-rig
claude plugin install oss@borda-ai-rig
/research:kaggle requires foundry:sw-engineer from the foundry plugin and has no fallback. Other skills that request foundry:* agents use general-purpose with a role description when Foundry is unavailable, so review and implementation quality can be lower.
Upgrade and uninstall
Upgrade from the marketplace and refresh delivered rules: Uninstall the plugin with the Claude Code plugin manager: Uninstall does not remove created rule links. Delete only dangling `~/.claude/rules/research-*.md` links after confirming they target this plugin's former cache.β‘ Quick start¶
Start with a measurable optimization goal:
/research:plan "improve validation F1 from 0.82 to 0.87"
/research:judge program.md
/research:run program.md
/research:plan scans the project, proposes metric and guard commands, and writes program.md. /research:judge checks completeness, methodology, scientific rigor, and (unless skipped) runs the commands once. /research:run asks a specialist agent for one scoped change per iteration, measures the configured metric, runs the guard, and keeps or reverts the change.
The metric and guard commands are supplied by you. Research can check that a metric emits a number and a guard exits successfully, but it cannot prove that a proxy metric represents the real goal.
π§ Workflow index¶
| Need | Command | Result |
|---|---|---|
| Compare current AI/ML methods | /research:topic "<topic>" |
Literature report and codebase-mapped recommendation in .reports/research/ |
| Convert a goal into an experiment contract | /research:plan "<goal>" |
program.md or a requested output path |
| Check a contract before spending compute | /research:judge [<program.md>] |
APPROVED, NEEDS-REVISION, or BLOCKED report |
| Run a bounded improvement campaign | /research:run <program.md> |
State, iteration log, diary, and report under .experiments/ and .reports/research/ |
| Run plan β judge β campaign in one command | /research:sweep "<goal>" |
Contract, up to three judge/refinement passes, then a run |
| Check an implementation against a paper | /research:verify <paper> |
Formula, hyperparameter, evaluation, notation, and citation-chain audit |
| Test which changes mattered | /research:fortify <run-id> or /research:fortify program.md |
Isolated-worktree ablations and component-importance report |
| Analyze a completed campaign | /research:retro [<run-id>] |
Significance/descriptive statistics, dead-iteration analysis, and next hypotheses |
| Generate a Kaggle notebook script | /research:kaggle <competition-name> |
Jupytext # %% Python script under .experiments/kaggle/ |
| Install or refresh this plugin's rule links | /research:setup [--approve] |
Namespaced symlinks under ~/.claude/rules/ |
All ten commands are Claude Code skills under the research: namespace. Their SKILL.md files are the source of truth for validation rules, output schemas, flags, and edge cases.
/research:topic¶
Literature search, report gating, and plan follow-up
Searches AI/ML literature, compares methods, recommends an approach for the current codebase, and can turn the recommendation into a phased plan. A broad survey uses the workflow; a named-paper deep dive belongs to `research:scientist`./research:topic "efficient fine-tuning methods"
/research:topic "efficient fine-tuning methods" --team
/research:topic plan
/research:topic plan path/to/topic-report.md
/research:plan¶
Experiment wizard and profile-first mode
Builds a `program.md` contract from a measurable goal. A runnable Python file can be supplied instead to run `cProfile` before the wizard asks what to optimize. The wizard scans the codebase, proposes metric and guard commands, chooses a strategy, and asks before overwriting an existing output. It uses `foundry:solution-architect`, `foundry:perf-optimizer`, and `research:scientist` when available, with documented fallbacks. The output records `Goal`, `Metric`, `Guard`, `Config`, and optional `Notes`. `scope_files` constrain ideation, `max_iterations` is capped at 50, and `agent_strategy` accepts `auto`, `perf`, `code`, `ml`, or `arch`./research:judge¶
Methodology gate and verdict semantics
Reviews a contract before a campaign. It checks required fields, scope adequacy, metric/goal alignment, methodology, scientific rigor, and command execution./research:judge
/research:judge path/to/program.md
/research:judge path/to/program.md --skip-validation
Verdict: NEEDS-REVISION
Finding: target not set β the campaign will run to max_iterations
Finding: measurement validity β metric_cmd measures a proxy, not the stated metric
Required changes: (1) add `target:` under ## Metric (2) replace metric_cmd
/research:run¶
Bounded metric-improvement loop, flags, and state
Runs the core loop. Each iteration builds bounded context, proposes one scoped change, verifies that files changed, commits before measuring, runs the metric and guard, keeps a guarded improvement, or reverts the change. A campaign defaults to 20 iterations and is capped at 50. Five consecutive discards trigger strategy escalation and then stop rather than looping blindly./research:run program.md
/research:run program.md "focus on data augmentation"
/research:run program.md --resume
/research:run program.md --team --researcher --journal
Baseline: f1_score = 0.820
[β Iter 1/20 β best so far: 0.820 (Ξ0.0% vs baseline)]
[β Iter 1/20 β kept Β· metric=0.831 (Ξ1.3%) Β· agent=research:scientist]
[β Iter 2/20 β reverted Β· metric=0.818 (Ξ-0.2%) Β· agent=codex]
/research:sweep¶
Plan β judge/refine β run pipeline
Runs the non-interactive plan β judge/refine β run pipeline from a goal. It accepts the run's compute, team, Codex, researcher, architect, journal, and hypothesis options, plus `--skip-validation`, `--out/research:sweep "increase validation F1 to 0.87"
/research:sweep "reduce test runtime" --out .experiments/program.md
/research:sweep "improve recall" --researcher --journal
/research:verify¶
Paper-to-code fidelity audit
Audits whether code matches a named paper; it does not judge whether the paper's claims are valid. Input may be a PDF path, arXiv/PDF URL, or pasted paper text./research:verify paper.pdf
/research:verify paper.pdf --scope "src/model/**/*.py"
/research:verify paper.pdf --program program.md --strict
/research:verify paper.pdf --dim F,H --no-codemap
/research:fortify¶
Isolated ablations and optional reviewer Q&A
Runs one-component-at-a-time ablations after a completed run and an `APPROVED` judge report for the same program. It identifies candidates from the diff and diary, creates an isolated Git worktree for each variant, runs metric and guard commands locally, ranks importance, and can generate venue-specific reviewer Q&A./research:fortify
/research:fortify <run-id>
/research:fortify program.md --max-ablations 5 --skip-run
/research:fortify <run-id> --venue NeurIPS
/research:retro¶
Retrospective statistics and next-hypothesis queue
Reads a completed run's JSONL and diary without changing code or experiment state. It computes a one-sided one-sample Wilcoxon comparison of kept iterations against the baseline when at least six kept iterations and `scipy` are available; otherwise it reports descriptive statistics./research:retro
/research:retro <run-id>
/research:retro <run-id> --compare <run-id-2>
/research:retro <run-id> --threshold 0.005 --alpha 0.01
/research:kaggle¶
Grounded Jupytext notebook generation
Generates a Kaggle competition notebook as a Jupytext `# %%` Python script. It grounds schema and submission format through the authenticated Kaggle CLI, then produces an EDA β baseline β training β inference pipeline where the selected mode requires it. It requires `foundry:sw-engineer` and stops when that required agent is unavailable./research:kaggle competition-name
/research:kaggle competition-name "problem description"
/research:kaggle competition-name --type classification
/research:kaggle competition-name --eda-only
/research:kaggle competition-name --inference-only
/research:kaggle competition-name --resume .experiments/kaggle/existing.py
/research:setup¶
Rule delivery, ownership checks, and conflicts
Delivers this plugin's `rules/*.md` into Claude's flat user-rule namespace with a `research-` prefix, avoiding collisions with other plugins' `quality-gates.md` files. The default mode previews changes and asks before replacing a conflicting destination. `--approve` is the non-interactive mode used by synchronization. Only links whose targets resolve under this plugin's current cache or install-cache lineage are replaced or removed; a real file, another marketplace, a source checkout, or a dotfiles tree remains a conflict unless explicitly approved. Each rule becomes `~/.claude/rules/research-π§Ύ Experiment contract¶
program.md fields and validation rules
`program.md` is the boundary between planning and execution. Write it with `/research:plan` or by hand:
| Field | Values | Default | Contract |
| ----------------- | ------------------------------------ | -------- | ------------------------------------------------------------ |
| `max_iterations` | 1β50 | 20 | Hard ceiling; set deliberately before a campaign. |
| `agent_strategy` | `auto`, `perf`, `code`, `ml`, `arch` | `auto` | Auto infers from goal/metric keywords and warns on fallback. |
| `scope_files` | paths or globs | required | Bounds what ideation may inspect and modify. |
| `compute` | `local`, `colab`, `docker` | `local` | Routes metric and guard execution. |
| `colab_hw` | `H100`, `L4`, `T4`, `A100` | none | Hardware preference for Colab runs. |
| `sandbox_network` | `none`, `bridge` | `none` | Network isolation for Docker sandbox execution. |
The metric command must emit a numeric value, and the guard must exit successfully for a kept iteration. A target is recommended because running only to the iteration ceiling can waste compute. Scope files, direction, baseline, controls, and stopping criteria should be reviewable before `/research:run`.
π Workflow outputs¶
Reports, state, and handoff files
| Workflow | Primary report | Supporting state or output | | --------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------- | | `topic` | `.reports/research/topic-Colab MCP setup
`--colab` routes metric verification and GPU testing to a connected Colab runtime through `colab-mcp`. Before invoking it: 1. Enable `colab-mcp` in `settings.local.json`. 2. Ensure `colab-mcp` is defined in `.mcp.json` under `mcpServers`. 3. Open a Colab notebook with a connected runtime and execute its MCP connection cell. `--colab=H100` requests a specific hardware class; the run checks the observed GPU and reports a mismatch rather than silently treating another GPU as equivalent. `--colab` and `--compute=docker` are mutually exclusive.πΊοΈ Workflow overview¶
Common paths and run internals
Standard evidence path:1. /research:topic "<method>" understand current methods before coding
2. /research:plan "<goal>" write program.md
3. /research:judge validate methodology cheaply
4. /research:run program.md run bounded improvement loop
5. /research:retro analyze results and next hypotheses
6. /research:verify paper.pdf confirm paper-to-code fidelity
7. /research:fortify isolate component importance
π Agents and optional integrations¶
The plugin ships exactly two manually invocable agents:
| Agent | Purpose | Not for |
|---|---|---|
research:scientist |
Named-paper analysis, falsifiable hypotheses, experiment design, and result interpretation. | Broad SOTA survey, data acquisition, or general Python. |
research:data-steward |
Dataset acquisition, provenance, split integrity, leakage checks, and data-pipeline validation. | ML hypothesis design, throughput optimization, or undocumented web scraping. |
research:scientist expects a named paper, author, or arXiv anchor for paper implementation. It enforces one hypothesis per experiment, seed averaging, baselines, ablations, and mean Β± standard deviation rather than best-run reporting. research:data-steward audits pagination completeness, schema, boundaries, deduplication, split isolation, stateful transforms, augmentations, and DataLoader configuration; it delegates unknown URL discovery to foundry:web-explorer when Foundry is available.
Optional integrations are capability-gated:
foundry(requiresfoundryplugin) supplies software, performance, architecture, and web-research agents. Most skills fall back togeneral-purposerole prompts when it is absent; Kaggle stops.codemap-py(requirescodemap-pyplugin) supplies structural context torunandverifywhen enabled and indexed.--no-codemapopts out;--codemapmakes a usable index mandatory.bridge@borda-ai-rigenables/research:run --codex; it is never silently substituted when requested.- A connected
colab-mcpruntime enables--colabforrunandsweep. scipyenables Wilcoxon significance inretro; without it, the report uses descriptive statistics.- The authenticated Kaggle CLI is required for online competition grounding.
Agent operating boundaries
`research:scientist` is for a named paper, publication-backed method, falsifiable hypothesis, or experiment design. It separates paper claims from evidence, checks baselines and variance, identifies one central idea, audits attribution and contribution claims, plans one-variable-at-a-time experiments, estimates compute, and interprets results as confirmed, refuted, or partially supported. It should report mean Β± standard deviation over at least three seeds when stochastic results are being compared and should flag cherry-picked results, missing confidence intervals, test-set reuse, and leakage concerns. `research:data-steward` is for acquisition, provenance, DVC or version tracking, split integrity, leakage detection, schema validation, and DataLoader configuration. Its checklist includes pagination count/schema/boundary/dedup verification; mutually exclusive or group-aware splits; train-only fitting for stateful transforms; train-only augmentation and oversampling; temporal window direction; NaN/Inf, shape, dtype, and range checks; and `shuffle=False` for validation/test loaders. It delegates unknown URL discovery or scraping to `foundry:web-explorer` and validates the returned data itself. Useful data-steward search patterns include `fit_transform(` for pre-split normalization, `Random*` transforms for validation contamination, `train_test_split(` without group awareness, `patient_id` or `subject_id` split gaps, `random_split(` shared-transform risks, and augmentation calls before the split. These are investigation prompts, not automatic findings; the agent must confirm the surrounding data flow. Scientist handoffs should state the paper or method, core idea, actual contribution, mechanics, evidence, limitations, relevance, falsifiable prediction, variables, controls, success criterion, ablations, compute estimate, and expected outcome. A paper summary is not a benchmark claim unless the source and protocol are retained.π Hooks, rules, artifacts, and bin tools¶
Registered hooks and shared helper behavior
Hooks register from `hooks/hooks.json` when the plugin is enabled; no settings edit is needed for registration: | Hook | Event | Behavior | | ------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `agent-router.js` | `PreToolUse(Agent)` | Exact agent route, semantic fallback, then `general-purpose`. | | `sentinel-read-allow.js` | `PreToolUse(Bash)` | Allows only pre-canned read-only temporary-state idioms; other commands use normal permission checks. Whole-line `# β¦` comments inside a block are skipped rather than blocking it, and `..` is matched as a path component so ellipsis and version ranges are not mistaken for traversal. | | `blueprint-allow.js` | `PreToolUse(Bash)` | Exact-matches the normalized command against this plugin's committed `blueprint-manifest.json`; any deviation falls through to a normal prompt. | | `write-guard.js` | `PreToolUse(Edit, Write, NotebookEdit)` | Grants nothing; forces confirmation on writes to CI definitions, agent instructions, permission config, lockfiles and release metadata. Source and tests stay unprotected. | | `enforce-topic-header.js` | `PreToolUse(AskUserQuestion)` | Gates topic follow-up until its report exists; header-table checking remains additive and non-blocking. | `report-header-table.js` is a shared hook module rather than a registered hook. It checks whether a topic report header was rendered as the required table or documented fallback line. `rules/quality-gates.md` requires confidence blocks on analysis output and defines report-file and terminal-header formatting. `/research:setup` delivers it as a namespaced user rule instead of editing global configuration. Generated files remain at the project root:.experiments/state/<run-id>/ run state, JSONL log, diary, context, scripts
.experiments/<run-id>/ hypothesis queue, checkpoint, optional journal
.experiments/{judge,verify,fortify,retro}-* intermediate analysis artifacts
.experiments/kaggle/ generated Jupytext scripts and downloaded data
.reports/research/ topic, judge, run, verify, fortify, retro reports
.temp/state/ short-lived cross-phase skill contracts
π§ Current boundaries¶
These are current constraints, not promises about future releases:
/research:runoperates on the current Git worktree and can create commits or reverts. Reviewscope_files, metric commands, guard commands, and the resulting diff before accepting a campaign result.- Research does not provide datasets, GPU capacity, Kaggle credentials, Colab runtimes, Codex, or specialist companion plugins; those remain user-managed prerequisites.
fortifyexecutes ablations locally. Worktree isolation protects the main worktree, but it does not make arbitrary metric or guard commands safe.- Statistical conclusions are limited by run history, baseline design, data quality, and independence assumptions.
retroreports what it computed; it does not establish causal validity. - A missing or stale Codemap index reduces structural context, and an unavailable optional specialist falls back or stops according to the specific skill contract.
--colab,--compute=docker,--codex,--researcher, and--architectare explicit requirements when requested; the workflow reports missing capability instead of pretending it ran.- Rule links can outlive plugin uninstall because Claude Code has no uninstall cleanup hook; ownership must be checked before manual removal.
Potential future work includes richer native agent selection, broader compute backends for fortify, and more shared health-monitor orchestration. None is required for the current plugin contract.
π Troubleshooting¶
Common failure messages and recovery
**`No program.md found`**: run `/research:plan "π Contributing and maintenance¶
The canonical sources are the ten skills/*/SKILL.md files, the two agents/*.md files, rules/*.md, registered hooks, sidecar references, and bin/*. Keep this README synchronized when a public skill, flag, trigger, prerequisite, output path, hook, or boundary changes.
The plugin version is currently 0.19.0. This bridge integration is a designed capability change and therefore uses a minor version bump.
When editing a skill, update its README entry, flags, NOT-for boundaries, output paths, fallback behavior, and relevant troubleshooting guidance. Verify that references loaded from skills/_shared/, skills/*/modes/, agent sidecars, or bin/ remain installed-path safe and do not assume a source checkout.
The plugin's tests cover path safety, Codemap resolution, effect size, Docker sandboxing, hook contracts, run directories, rule resolution, shared-file resolution, retro analysis, and patient split validation. Run focused tests while editing and the full plugins/cc_research suite before release.
π Acknowledgments and license¶
Research automation design draws on fcakyon/phd-skills for hook-first guardrails and karpathy/autoresearch for metric-driven, commit-preserving iteration contracts. Those influences inform the design; this plugin's current behavior is defined by its shipped skills, agents, hooks, rules, and tests.
Research is licensed under Apache-2.0.