Skip to content

Sample Output

This page shows what you see. Each block below is what the tools in this repository return on fixtures: the decision and reason from the gate's JSON output, or a line the ledger helper or the scorer prints. A repository check (make eval-docs-samples) fails when a block stops matching the code, and no other code block is allowed on this page.

Not a recorded session

These are tool outputs on synthetic fixtures. The brief values (scope, credits, source) are illustrative, not a real price. No block here comes from a live Roboflow call or a real user session. A recorded end-to-end session is still pending; see Support & Evidence.

The spend gate (Claude Code)

The agent tries to start a training job. No action brief is recorded, so the gate denies the call and tells the agent what to record:

deny: Sentinel gate: trainings_create (training) moves data, spends credits, or changes provider state. Record a sourced action brief first: ledger_append.py --action action_brief_emitted --operation trainings_create --status success --scope "<what runs on what>" --cost-amount <upper estimate> --cost-unit <credits|USD> --cost-source "<where the price comes from>" --cost-as-of <YYYY-MM-DD> --data-movement "<what leaves this machine, or none>" --rollback "<how to undo, or not reversible>"; then retry.

A brief with no stated price source is refused by the ledger helper. The helper cannot check that a stated source is real:

ledger_append.py: error: a successful action_brief_emitted needs --cost-source

With a complete brief the gate returns ask with this text as its reason, and the host then asks you to approve or reject. How the host displays the text is not yet recorded:

ask: Sentinel action brief "manual:demo:brief:1" for trainings_create (training). Scope: "train a detection model on project pallets, version 3". Cost: up to 10 "credits" (source: "Roboflow plans-and-pricing skill, training table", as of 2026-11-10). Data movement: "none; the dataset is already in the workspace". Rollback: "cancel the training job; spent credits are not refunded".

In a permission mode that never shows a prompt, the same call is denied even with a brief:

deny: Sentinel gate: trainings_create needs an interactive human approval; permission mode "bypassPermissions" never shows a prompt. Rerun in default mode.

Single-image inference runs under one brief with a call budget, so scoring 40 images does not mean 40 briefs:

ask: Sentinel action brief "manual:demo:brief:2" for models_infer (inference). Scope: "score the candidate model on the 40 gold images, one image per call". Cost: up to 40 "credits" (source: "Roboflow plans-and-pricing skill, hosted inference table", as of 2026-11-10). Data movement: "40 gold images are sent to hosted inference". Rollback: "nothing to undo; inference changes no provider state". Call budget: 40 calls under this brief, 0 used so far (every attempt that reaches this gate uses one). After one call has run, later calls within the budget follow your normal permission rules; in auto mode each one is asked.

The agent that wants the call writes the brief. The gate checks that every field is present and well formed, not that the estimate is right. Trust and Safety lists what the gate does not cover.

The scorer verdict (any host, no account)

The first-baseline example scores 24 synthetic count labels against an 85% target. The point estimate is above the target, but the sample cannot rule out a true rate below it:

Per-image count agreement: 42 of 47 labeled objects (89%, 95% interval 81% to 96%); target >= 85%: INCONCLUSIVE. The 95% interval includes the target: label more images to settle it, or treat the result as not yet proven. This number does not penalize over-counting: the predictions hold 2 counts beyond the labels (see over_count_rate). Missing predictions: 0.

When one gold image has no prediction, the scorer refuses instead of counting the gap as a miss:

Per-image count agreement: 40 of 47 labeled objects (85.1%, 95% interval 70.3% to 95.9%); target >= 85.0%: NO VERDICT. No verdict because 1 of 24 gold images has no prediction. This number does not penalize over-counting: the predictions hold 2 counts beyond the labels (see over_count_rate). Missing predictions: 1.

PASS appears only when the whole 95% interval meets the target. Modes and the baseline ladder explains the four verdicts.

Still pending

  • A recorded real session: the conversation, the host permission prompt as the host renders it, and the resulting ledger.
  • A decision report from a real run.
  • A terminal recording.

These need a live run on public, licensed images. They will be added here when one is recorded. The maintainer's runbook for recording and packing such a run is examples/golden-run/README.md in the repository; the packer is scripts/package_golden_run.py.