feat(runner): zero-trust cost re-metering + crash-safe whole-repo output

.claude/skills/run-benchmark/SKILL.md

@@ -89,7 +89,13 @@ must keep `BUDGET_USD=20` and omit `MAX_RULES`.
 per rule with the budget split evenly; `whole-repo` runs ONE session over the whole library
 and lets the agent enumerate and triage the rules itself under a single budget. Both produce
 the same `out/submission.json` and are scored identically — set `AGENT_MODE=whole-repo` to try
-it. (`MAX_RULES` only applies to `per-rule`.)
+it. (`MAX_RULES` only applies to `per-rule`.) In `whole-repo`, the agent also writes each
+certificate to `TRAJECTORY_DIR/certs.txt` (default `/out`) as it finds it, and the trajectory
+is persisted every step — so an early-stop/crash still leaves the found bugs on disk.
+
+**Output is versioned.** `out/submission.json` is the stable "latest" pointer; every run ALSO
+writes a versioned archive `out/submission-<model>-<timestamp>.json` beside it, so runs don't
+overwrite each other.
 
 **Confirm the experiment parameters with the user — don't silently default them.** These
 shape the result and the spend, so state the resolved set and get an explicit OK before

.claude/skills/run-benchmark/references/env-and-troubleshoot.md

@@ -25,12 +25,14 @@ these as env vars (CLI flags would override, but the skill uses the env-file).
 | `PER_RULE_BUDGET` | 0.5 | per-rule cost cap |
 | `SAFETY_MARGIN` | 1.0 | USD held back so the budget-crossing call stays under cap |
 | `MAX_TOKENS` | 8192 | per-call output ceiling |
-| `MAX_RULES` | all | cap rules attempted — **smoke runs only**; omit for a ranked run |
+| `MAX_RULES` | all | cap rules attempted — **smoke runs only**; omit for a ranked run (per-rule only) |
+| `AGENT_MODE` | `per-rule` | `per-rule` (isolated session/rule, budget split evenly) or `whole-repo` (ONE session, the agent triages the rules itself) |
+| `TRAJECTORY_DIR` | `OUTPUT`'s dir (`/out`) | where **whole-repo** persists the trajectory + the durable incremental cert log (`certs.txt`); the agent writes each certificate here the moment it finds it, so an early-stop/crash still leaves the found bugs on disk |
 | `AGENT_CONFIG` / `AGENT_STRATEGY_FILE` | bundled | bring-your-own prompt; the files must be **mounted** into the container (`-v "$PWD/cfg:/cfg"`) and the path given as a container path |
 | `SUBMITTED_BY` | — | your handle, recorded in the envelope |
 | `EXPECTED_PRED_VERSION` / `EXPECTED_PRED_COMMIT` | baked | debugging only; `EXPECTED_PRED_VERSION=""` disables the version check |
 
-`REPO_DIR` and `OUTPUT` are container-internal and already baked — don't set them.
+`OUTPUT` (default `/out/submission.json`) is the stable "latest" pointer `prb submit` reads; every run **also** writes a versioned archive `submission-<model>-<timestamp>.json` beside it, so history isn't clobbered. `REPO_DIR` is container-internal and baked — don't set it.
 
 Non-standard endpoint example:
 ```ini

benchmark/config.yaml

@@ -83,14 +83,30 @@ environment:
     LESS: -R
 
 model:
+  # Head+tail elision (SWE-agent pattern): short output whole; long output keeps first + last
+  # 5000 chars + a notice, so a big command result can't flood the context and be re-sent every
+  # step. The runner MUST thread this into the model config (run_mini._build_model) or
+  # mini-swe-agent's non-truncating default template is used instead.
   observation_template: |
     {% if output.exception_info -%}
     <exception>{{output.exception_info}}</exception>
     {% endif -%}
     <returncode>{{output.returncode}}</returncode>
+    {% if output.output | length < 10000 -%}
     <output>
-    {{ output.output[:8000] -}}
+    {{ output.output -}}
     </output>
+    {%- else -%}
+    <warning>Output too long — showing head + tail only. Re-run narrowing it (head/tail/grep/sed) or write it to a file and read slices.</warning>
+    {%- set elided = output.output | length - 10000 -%}
+    <output_head>
+    {{ output.output[:5000] }}
+    </output_head>
+    <elided>{{ elided }} characters elided</elided>
+    <output_tail>
+    {{ output.output[-5000:] }}
+    </output_tail>
+    {%- endif -%}
   format_error_template: |
     Format error: {{error}}
     Provide exactly ONE bash command in triple backticks.

benchmark/config_repo.yaml

@@ -59,6 +59,16 @@ agent:
     }
     CERTIFICATE_END
 
+    THEN, immediately, append the SAME block to the durable log so the bug survives even if
+    the run is cut off (budget/crash) — one command, append only, never overwrite:
+      cat >> {{certs_file}} <<'CERT'
+      CERTIFICATE_START
+      { ...the identical certificate JSON... }
+      CERTIFICATE_END
+      CERT
+    Do this every time, the moment you find a bug, before moving to the next rule. The file
+    is your safety net: anything written there is recoverable even if you never reach submit.
+
     Only `rule`, `source`, and the target type (from `bundle`) are required; `target_config`
     is optional. The verifier re-derives everything from `source` with pred and never trusts
     your claim — a wrong or non-minimal certificate is simply rejected, so report only what
@@ -74,9 +84,10 @@ agent:
 
     Enumerate the rules with `pred list --rules --json`, triage by suspicion, and test the
     round-trip correctness of each with pred. Emit a CERTIFICATE_START...CERTIFICATE_END
-    block for every bug — many are expected; do not stop after the first. You have a limited
-    budget — be efficient and spend it on the most suspicious rules first. When done, run:
-    echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
+    block for every bug — many are expected; do not stop after the first — AND append the
+    same block to {{certs_file}} as you go, so no bug is lost if the run stops early. You
+    have a limited budget — be efficient and spend it on the most suspicious rules first.
+    When done, run: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
     </task>
 
   step_limit: 300
@@ -90,14 +101,31 @@ environment:
     LESS: -R
 
 model:
+  # Head+tail elision (SWE-agent pattern): keep short output whole; for long output keep the
+  # first and last 5000 chars + a notice, so a big result (e.g. `pred list --rules --json`,
+  # 290 rules) can't flood the context and get re-sent every step. The runner MUST thread this
+  # into the model config (see run_mini._build_model) or mini-swe-agent's non-truncating
+  # default template is used instead.
   observation_template: |
     {% if output.exception_info -%}
     <exception>{{output.exception_info}}</exception>
     {% endif -%}
     <returncode>{{output.returncode}}</returncode>
+    {% if output.output | length < 10000 -%}
     <output>
-    {{ output.output[:8000] -}}
+    {{ output.output -}}
     </output>
+    {%- else -%}
+    <warning>Output too long — showing head + tail only. Re-run narrowing it (head/tail/grep/sed) or write it to a file and read slices.</warning>
+    {%- set elided = output.output | length - 10000 -%}
+    <output_head>
+    {{ output.output[:5000] }}
+    </output_head>
+    <elided>{{ elided }} characters elided</elided>
+    <output_tail>
+    {{ output.output[-5000:] }}
+    </output_tail>
+    {%- endif -%}
   format_error_template: |
     Format error: {{error}}
     Provide exactly ONE bash command in triple backticks.

benchmark/cost.py

@@ -109,3 +109,43 @@ def extract_usage(messages: list) -> Usage:
         if resp is not None and _get(resp, "usage", None) is not None:
             total = total + usage_from_response(_get(resp, "usage", None))
     return total
+
+
+# ── (de)serialization: the 4-bucket token totals + the declared price snapshot ──────
+# These travel in the submission so the backend can RE-METER cost as tokens × price
+# (zero-trust, mirroring the bug re-verification) instead of trusting a self-reported
+# dollar total. Tokens are the reproducible primitive; the declared price (dated by the
+# submission's created_at) is a snapshot anyone can swap to recompute under other prices.
+
+def usage_as_dict(u: Usage) -> dict:
+    """Serialize a Usage to the 4-bucket dict shape used on rows and the envelope."""
+    return {"input": u.input_tokens, "output": u.output_tokens,
+            "cache_read": u.cache_read_tokens, "cache_write": u.cache_write_tokens}
+
+
+def usage_from_dict(d) -> Usage:
+    """Parse a 4-bucket dict (or None/missing → all zeros) back into a Usage."""
+    return Usage(
+        input_tokens=int(_get(d, "input")),
+        output_tokens=int(_get(d, "output")),
+        cache_read_tokens=int(_get(d, "cache_read")),
+        cache_write_tokens=int(_get(d, "cache_write")),
+    )
+
+
+def price_as_dict(p: Price) -> dict:
+    """Serialize a Price to a plain dict (USD per 1M tokens, per bucket)."""
+    return {"input": p.input, "output": p.output,
+            "cache_read": p.cache_read, "cache_write": p.cache_write}
+
+
+def price_from_dict(d) -> Price | None:
+    """Parse a price dict back into a Price, or None if absent (legacy submission)."""
+    if not d:
+        return None
+    return Price(
+        input=float(_get(d, "input")),
+        output=float(_get(d, "output")),
+        cache_read=float(_get(d, "cache_read")),
+        cache_write=float(_get(d, "cache_write")),
+    )

benchmark/results.schema.json

@@ -19,15 +19,39 @@
       "minimum": 0,
       "description": "Number of verified, novel bugs found"
     },
+    "test": {
+      "type": "boolean",
+      "description": "Carried through from the submission: scored + stored privately but excluded from the public leaderboard."
+    },
     "total_cost_usd": {
       "type": "number",
       "minimum": 0,
-      "description": "Total spend in USD"
+      "description": "Total spend in USD — re-metered by the backend as usage_totals × prices (falls back to the self-reported figure only for legacy submissions without those)."
     },
     "total_tokens_k": {
       "type": "number",
       "minimum": 0,
-      "description": "Total tokens used (in thousands)"
+      "description": "Total tokens used (in thousands); the sum of usage_totals / 1000."
+    },
+    "usage_totals": {
+      "type": ["object", "null"],
+      "description": "Aggregate token usage in the four billed buckets (input/output/cache_read/cache_write) the cost was metered from. The reproducible primitive — lets any reader recompute cost under other prices.",
+      "properties": {
+        "input": {"type": "integer", "minimum": 0},
+        "output": {"type": "integer", "minimum": 0},
+        "cache_read": {"type": "integer", "minimum": 0},
+        "cache_write": {"type": "integer", "minimum": 0}
+      }
+    },
+    "prices": {
+      "type": ["object", "null"],
+      "description": "Declared per-token price snapshot (USD per 1M tokens) the cost was metered from; dated by the submission's created_at. null for legacy/FAKE submissions.",
+      "properties": {
+        "input": {"type": "number", "minimum": 0},
+        "output": {"type": "number", "minimum": 0},
+        "cache_read": {"type": "number", "minimum": 0},
+        "cache_write": {"type": "number", "minimum": 0}
+      }
     },
     "efficiency_bugs_per_ktok": {
       "type": "number",