Add whole-repo agent mode alongside per-rule

.claude/skills/run-benchmark/SKILL.md

@@ -85,6 +85,18 @@ PRICE_OUT=15.0               # USD / 1M output tokens — REQUIRED
 For a cheap smoke run (don't spend the full $20) add `MAX_RULES=1`. A **ranked** submission
 must keep `BUDGET_USD=20` and omit `MAX_RULES`.
 
+**Agent mode** (`AGENT_MODE`, default `per-rule`): `per-rule` runs one isolated agent session
+per rule with the budget split evenly; `whole-repo` runs ONE session over the whole library
+and lets the agent enumerate and triage the rules itself under a single budget. Both produce
+the same `out/submission.json` and are scored identically — set `AGENT_MODE=whole-repo` to try
+it. (`MAX_RULES` only applies to `per-rule`.)
+
+**Confirm the experiment parameters with the user — don't silently default them.** These
+shape the result and the spend, so state the resolved set and get an explicit OK before
+running: **mode** (`AGENT_MODE`), **budget** (a full ranked run at `BUDGET_USD=20`, or a
+cheap smoke run via `MAX_RULES=1` / a smaller budget), and — only if they care —
+`PER_RULE_BUDGET` and `MAX_TOKENS`. Ranked runs require `BUDGET_USD=20` and no `MAX_RULES`.
+
 ## Step 4 — Preflight (one tiny real API call, ~a fraction of a cent)
 
 Always run this before the full run; it validates key/endpoint/price + pred/rules through the
@@ -99,6 +111,9 @@ name / pricing). Decode table in `references/env-and-troubleshoot.md`. Do not pr
 
 ## Step 5 — Full run → out/submission.json
 
+**Gate**: a full run spends real money and takes a while. Restate the resolved parameters
+(model, mode, budget, any smoke caps) and get an explicit OK before you launch it.
+
 - **docker**: `make run`
 - **podman/raw**: use the `RUN_FLAGS` from Step 1, e.g.
   ```bash

benchmark/config.yaml

@@ -67,7 +67,7 @@ agent:
     Library repo: {{repo_dir}} (commit {{commit_hash}})
     Rule source: {{rule_file}}
 
-    Use pred commands to test the round-trip correctness. Budget: ${{cost_limit}}.
+    Use pred commands to test the round-trip correctness. You have a limited budget — be efficient and don't over-explore.
     If you find a bug, emit the CERTIFICATE_START...CERTIFICATE_END block.
     If no bug found, run: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
     </task>

benchmark/config_repo.yaml

@@ -0,0 +1,103 @@
+agent:
+  system_template: |
+    You are an expert at finding bugs in problem reduction rules using the `pred` CLI. You
+    are given the WHOLE library and one budget; you decide which rules to probe and in what
+    order, and report every bug you find.
+
+    Each rule implements two functions:
+    - reduce_to(): transforms source problem A into target problem B
+    - extract_solution(): maps a target solution back to source space
+
+    A rule is BUGGY if the round-trip fails:
+    given A → reduce to B → solve B to get s → extract s → A', but A' is not a valid solution to A.
+
+    Your tool is `pred`. Key commands:
+      pred list --rules --json                     # enumerate every reduction rule
+      pred create <PROBLEM> [options] --json       # create a problem instance
+      pred reduce <file.json> --to <TARGET> --json # reduce source to target (produces a bundle)
+      pred solve <bundle.json> --json              # solve (finds optimal solution)
+      pred evaluate <file.json> --config <vals> --json   # check if a config is valid
+      pred extract <bundle.json> --config <vals> --json  # map target config back to source
+
+    On Windows, pred cannot read from stdin. Always write JSON to a file first, then pass the filename.
+
+    The single test for a bug (round-trip): solving the source directly must agree with
+    solving it THROUGH the reduction, compared by VALUE (optimization) or feasibility
+    (decision). `pred solve <bundle>` already does the whole round-trip, so you just compare
+    it against `pred solve <source>`:
+      pred solve source.json --json        # direct answer, e.g. {"evaluation": "Max(7)"}
+      pred reduce source.json --to <TARGET> --json -o bundle.json
+      pred solve bundle.json --json        # round-trip answer, e.g. {"evaluation": "Max(3)"}
+    Different evaluations (Max(7) vs Max(3), or Or(true) vs Or(false), or one infeasible) =
+    a genuine bug in the reduction.
+
+    {{strategy}}
+
+    How to work the whole repo:
+    1. `pred list --rules --json` to see every rule. Rule sources are under
+       {{repo_dir}}/src/rules/<rule>.rs — read the ones you probe.
+    2. Prioritize by SUSPICION — spend your budget where a bug is most likely first.
+       Good signals: rules touching weights/costs, offsets or constant shifts, min↔max or
+       feasibility flips, empty/degenerate inputs, or extraction code that drops information.
+       Don't sweep alphabetically; triage.
+    3. For each rule you pick, construct a small source instance with `pred create` (tiny —
+       minimal witnesses only) and compare `pred solve source` with `pred solve bundle`.
+    4. (Optional, stronger) If a SPECIFIC valid target solution extracts to an invalid or
+       worse source solution, note its config as `target_config`.
+
+    Report EVERY bug you find (you will likely find several). The moment a round-trip
+    disagrees for a rule, emit a certificate in this format and then MOVE ON to the next
+    rule — do NOT stop after the first:
+
+    CERTIFICATE_START
+    {
+      "rule": "<rule_name>",
+      "source": <paste the source JSON from pred create>,
+      "bundle": <paste the bundle JSON from pred reduce>,
+      "target_config": "<optional: comma-separated target solution that mis-extracts>",
+      "note": "<one sentence: source solves to X but the round-trip yields Y>"
+    }
+    CERTIFICATE_END
+
+    Only `rule`, `source`, and the target type (from `bundle`) are required; `target_config`
+    is optional. The verifier re-derives everything from `source` with pred and never trusts
+    your claim — a wrong or non-minimal certificate is simply rejected, so report only what
+    you have actually reproduced. Keep going until you run out of budget or have covered the
+    rules that look worth checking, then run: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
+
+  instance_template: |
+    <task>
+    Find as many bugs as you can across the reduction rules in the library.
+
+    Library repo: {{repo_dir}} (commit {{commit_hash}})
+    Rule sources: {{repo_dir}}/src/rules/
+
+    Enumerate the rules with `pred list --rules --json`, triage by suspicion, and test the
+    round-trip correctness of each with pred. Emit a CERTIFICATE_START...CERTIFICATE_END
+    block for every bug — many are expected; do not stop after the first. You have a limited
+    budget — be efficient and spend it on the most suspicious rules first. When done, run:
+    echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
+    </task>
+
+  step_limit: 300
+  cost_limit: 20.0
+
+environment:
+  timeout: 120
+  env:
+    PAGER: cat
+    MANPAGER: cat
+    LESS: -R
+
+model:
+  observation_template: |
+    {% if output.exception_info -%}
+    <exception>{{output.exception_info}}</exception>
+    {% endif -%}
+    <returncode>{{output.returncode}}</returncode>
+    <output>
+    {{ output.output[:8000] -}}
+    </output>
+  format_error_template: |
+    Format error: {{error}}
+    Provide exactly ONE bash command in triple backticks.

benchmark/run_mini.py

@@ -7,6 +7,7 @@
 import argparse
 import json
 import os
+import re
 from pathlib import Path
 
 import yaml
@@ -17,6 +18,7 @@
 from benchmark.verify import count_bugs, verify
 
 CONFIG_FILE = Path(__file__).parent / "config.yaml"
+REPO_CONFIG_FILE = Path(__file__).parent / "config_repo.yaml"
 SKIP_RULES = {"mod", "traits", "graph_helpers", "analysis", "cost", "registry", "graph"}
 
 # Rough average cost per 1K tokens (used as fallback when model doesn't report usage)
@@ -31,12 +33,13 @@ def list_rules(repo_dir: str) -> list[str]:
     return [f.stem for f in sorted(rules_dir.glob("*.rs")) if f.stem not in SKIP_RULES]
 
 
+_CERT_RE = re.compile(r"CERTIFICATE_START\s*\n(.*?)CERTIFICATE_END", re.DOTALL)
+
+
 def parse_certificate(messages: list) -> dict | None:
-    """Extract structured certificate JSON from agent message history."""
-    import re
+    """Extract the last structured certificate JSON from agent message history."""
     for msg in reversed(messages):
-        content = msg.get("content", "")
-        block = re.search(r"CERTIFICATE_START\s*\n(.*?)CERTIFICATE_END", content, re.DOTALL)
+        block = _CERT_RE.search(msg.get("content", "") or "")
         if block:
             try:
                 return json.loads(block.group(1).strip())
@@ -45,6 +48,24 @@ def parse_certificate(messages: list) -> dict | None:
     return None
 
 
+def parse_all_certificates(messages: list) -> list[dict]:
+    """Every certificate block in the trajectory (a whole-repo run emits many), de-duplicated
+    by (rule, source) in first-seen order. Unparseable blocks are skipped."""
+    seen, out = set(), []
+    for msg in messages:
+        for block in _CERT_RE.finditer(msg.get("content", "") or ""):
+            try:
+                cert = json.loads(block.group(1).strip())
+            except json.JSONDecodeError:
+                continue
+            key = (cert.get("rule"), json.dumps(cert.get("source"), sort_keys=True))
+            if key in seen:
+                continue
+            seen.add(key)
+            out.append(cert)
+    return out
+
+
 def extract_total_tokens(messages: list) -> int:
     total = 0
     for msg in messages:
@@ -62,6 +83,22 @@ def save_trajectory(messages: list, path: Path) -> None:
             f.write(json.dumps({"role": msg.get("role", ""), "content": msg.get("content", "")}) + "\n")
 
 
+def _session_cost(agent, price):
+    """Authoritative session spend from the trajectory. ``agent.cost`` is already our
+    token×price figure (see _build_model); cross-check against a fresh recompute and take
+    the max — never under-count for the budget guard. Returns (cost, tokens_k, usage)."""
+    usage = extract_usage(agent.messages)
+    cost = max(agent.cost, price.cost(usage) if price is not None else 0.0)
+    total_tokens = usage.total_tokens or extract_total_tokens(agent.messages)
+    tokens_k = round(total_tokens / 1000, 2) if total_tokens else round(cost / AVG_COST_PER_KTOK * 1000, 2)
+    return cost, tokens_k, usage
+
+
+def _trajectory(agent) -> list[dict]:
+    """The agent's own message history — provenance proof carried on cert-bearing rows."""
+    return [{"role": m.get("role", ""), "content": m.get("content", "")} for m in agent.messages]
+
+
 def _build_model(model_name: str, api_base: str | None, max_tokens: int, price: Price | None,
                  model_kwargs: dict | None = None, api_key: str | None = None):
     """A LitellmModel whose cost is OUR token×price figure, so mini-swe-agent's own
@@ -148,23 +185,11 @@ def run_one(
     cert = None
     try:
         agent.run(task=rule_name)
-        # agent.cost is already our token×price figure (see _build_model), which is what
-        # mini-swe-agent's per-step cost_limit enforced. Cross-check it against a fresh
-        # recompute over the whole trajectory and take the max — never under-count.
-        usage = extract_usage(agent.messages)
-        agent_cost = agent.cost
-        recomputed = price.cost(usage) if price is not None else 0.0
-        cost = max(agent_cost, recomputed)
-        total_tokens = usage.total_tokens or extract_total_tokens(agent.messages)
-        tokens_k = round(total_tokens / 1000, 2) if total_tokens else round(cost / AVG_COST_PER_KTOK * 1000, 2)
+        cost, tokens_k, usage = _session_cost(agent, price)
         usage_row = {"input": usage.input_tokens, "output": usage.output_tokens,
                      "cache_read": usage.cache_read_tokens, "cache_write": usage.cache_write_tokens,
-                     "accounted_cost_usd": round(agent_cost, 6)}
-        # Provenance record: the agent's own message history. The backend requires this on a
-        # bug_found row and checks the certificate was emitted here (not pasted from a public
-        # answer key), so we carry it on every cert-bearing row.
-        trajectory = [{"role": m.get("role", ""), "content": m.get("content", "")}
-                      for m in agent.messages]
+                     "accounted_cost_usd": round(agent.cost, 6)}
+        trajectory = _trajectory(agent)
         cert = parse_certificate(agent.messages)
         if trajectory_dir is not None:
             safe_model = model_name.replace("/", "_").replace(":", "_")
@@ -221,6 +246,88 @@ def run_one(
     }
 
 
+def run_repo_session(
+    model_name: str,
+    ctx: EnvContext,
+    cost_limit: float,
+    *,
+    api_base: str | None = None,
+    trajectory_dir: Path | None = None,
+    price: Price | None = None,
+    max_tokens: int = DEFAULT_MAX_TOKENS,
+    step_limit: int | None = None,
+    config_path: str | Path | None = None,
+    strategy: str | None = None,
+    model_kwargs: dict | None = None,
+    api_key: str | None = None,
+) -> dict:
+    """One WHOLE-REPO bug-hunting session: the agent gets the entire library + pred and the
+    full budget as its ``cost_limit``, chooses which rules to probe, and emits a certificate
+    per bug. Returns ``{"rows": [...], "cost": float, "tokens_k": float}`` — one result row
+    per distinct emitted certificate, each re-verified with pred and carrying the shared
+    session trajectory for provenance. Contrast with ``run_one`` (one isolated rule/session).
+    """
+    from minisweagent.agents.default import DefaultAgent
+    from minisweagent.environments.local import LocalEnvironment
+
+    cfg_file = Path(config_path) if config_path else REPO_CONFIG_FILE
+    config = yaml.safe_load(cfg_file.read_text(encoding="utf-8"))
+    if strategy is None:
+        strat_file = os.environ.get("AGENT_STRATEGY_FILE")
+        strategy = Path(strat_file).read_text(encoding="utf-8") if strat_file else ""
+    agent_cfg = config.get("agent", {})
+    agent_cfg["cost_limit"] = cost_limit
+    if step_limit is not None:
+        agent_cfg["step_limit"] = step_limit
+
+    agent = DefaultAgent(
+        _build_model(model_name, api_base, max_tokens, price,
+                     model_kwargs=model_kwargs, api_key=api_key),
+        LocalEnvironment(),
+        **agent_cfg,
+    )
+    agent.extra_template_vars = {
+        "repo_dir": str(ctx.repo_path),
+        "commit_hash": ctx.commit_hash[:7],
+        "cost_limit": cost_limit,
+        "strategy": strategy,
+    }
+
+    agent.run(task="find-bugs")
+    cost, tokens_k, _usage = _session_cost(agent, price)
+    trajectory = _trajectory(agent)
+    if trajectory_dir is not None:
+        safe_model = model_name.replace("/", "_").replace(":", "_")
+        save_trajectory(agent.messages, Path(trajectory_dir) / f"{safe_model}_whole-repo.jsonl")
+
+    rows = _rows_from_certificates(parse_all_certificates(agent.messages))
+    # The whole session is ONE trajectory shared by every bug — return it once for the
+    # envelope (build_submission) instead of copying it onto each row.
+    return {"rows": rows, "cost": cost, "tokens_k": tokens_k, "trajectory": trajectory}
+
+
+def _rows_from_certificates(certs: list[dict]) -> list[dict]:
+    """Verify each certificate with pred and build one result row per cert. bug_found when
+    pred confirms, else rejected. No per-row trajectory — a whole-repo run's provenance
+    trajectory lives once at the envelope level (submission["trajectory"])."""
+    rows = []
+    for cert in certs:
+        verdict = verify(cert)
+        row = {
+            "rule": cert.get("rule"),
+            "result": "bug_found" if verdict.accepted else "rejected",
+            "cost": 0.0,        # session cost/tokens live on the submission envelope, not per row
+            "tokens_k": 0.0,
+            "certificate": cert,
+        }
+        if verdict.accepted:
+            row["verify_details"] = verdict.details
+        else:
+            row["reject_reason"] = verdict.reason
+        rows.append(row)
+    return rows
+
+
 def main() -> None:
     parser = argparse.ArgumentParser(description="pred-based bug-finding benchmark")
     parser.add_argument("--model", default="anthropic/claude-sonnet-4-6", help="LiteLLM model name")