diff --git a/.claude/skills/run-benchmark/SKILL.md b/.claude/skills/run-benchmark/SKILL.md index 597f23d..b7b2944 100644 --- a/.claude/skills/run-benchmark/SKILL.md +++ b/.claude/skills/run-benchmark/SKILL.md @@ -85,6 +85,18 @@ PRICE_OUT=15.0 # USD / 1M output tokens — REQUIRED For a cheap smoke run (don't spend the full $20) add `MAX_RULES=1`. A **ranked** submission must keep `BUDGET_USD=20` and omit `MAX_RULES`. +**Agent mode** (`AGENT_MODE`, default `per-rule`): `per-rule` runs one isolated agent session +per rule with the budget split evenly; `whole-repo` runs ONE session over the whole library +and lets the agent enumerate and triage the rules itself under a single budget. Both produce +the same `out/submission.json` and are scored identically — set `AGENT_MODE=whole-repo` to try +it. (`MAX_RULES` only applies to `per-rule`.) + +**Confirm the experiment parameters with the user — don't silently default them.** These +shape the result and the spend, so state the resolved set and get an explicit OK before +running: **mode** (`AGENT_MODE`), **budget** (a full ranked run at `BUDGET_USD=20`, or a +cheap smoke run via `MAX_RULES=1` / a smaller budget), and — only if they care — +`PER_RULE_BUDGET` and `MAX_TOKENS`. Ranked runs require `BUDGET_USD=20` and no `MAX_RULES`. + ## Step 4 — Preflight (one tiny real API call, ~a fraction of a cent) Always run this before the full run; it validates key/endpoint/price + pred/rules through the @@ -99,6 +111,9 @@ name / pricing). Decode table in `references/env-and-troubleshoot.md`. Do not pr ## Step 5 — Full run → out/submission.json +**Gate**: a full run spends real money and takes a while. Restate the resolved parameters +(model, mode, budget, any smoke caps) and get an explicit OK before you launch it. + - **docker**: `make run` - **podman/raw**: use the `RUN_FLAGS` from Step 1, e.g. ```bash diff --git a/benchmark/config.yaml b/benchmark/config.yaml index 47de7ac..bbd441d 100644 --- a/benchmark/config.yaml +++ b/benchmark/config.yaml @@ -67,7 +67,7 @@ agent: Library repo: {{repo_dir}} (commit {{commit_hash}}) Rule source: {{rule_file}} - Use pred commands to test the round-trip correctness. Budget: ${{cost_limit}}. + Use pred commands to test the round-trip correctness. You have a limited budget — be efficient and don't over-explore. If you find a bug, emit the CERTIFICATE_START...CERTIFICATE_END block. If no bug found, run: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT diff --git a/benchmark/config_repo.yaml b/benchmark/config_repo.yaml new file mode 100644 index 0000000..03f5145 --- /dev/null +++ b/benchmark/config_repo.yaml @@ -0,0 +1,103 @@ +agent: + system_template: | + You are an expert at finding bugs in problem reduction rules using the `pred` CLI. You + are given the WHOLE library and one budget; you decide which rules to probe and in what + order, and report every bug you find. + + Each rule implements two functions: + - reduce_to(): transforms source problem A into target problem B + - extract_solution(): maps a target solution back to source space + + A rule is BUGGY if the round-trip fails: + given A → reduce to B → solve B to get s → extract s → A', but A' is not a valid solution to A. + + Your tool is `pred`. Key commands: + pred list --rules --json # enumerate every reduction rule + pred create [options] --json # create a problem instance + pred reduce --to --json # reduce source to target (produces a bundle) + pred solve --json # solve (finds optimal solution) + pred evaluate --config --json # check if a config is valid + pred extract --config --json # map target config back to source + + On Windows, pred cannot read from stdin. Always write JSON to a file first, then pass the filename. + + The single test for a bug (round-trip): solving the source directly must agree with + solving it THROUGH the reduction, compared by VALUE (optimization) or feasibility + (decision). `pred solve ` already does the whole round-trip, so you just compare + it against `pred solve `: + pred solve source.json --json # direct answer, e.g. {"evaluation": "Max(7)"} + pred reduce source.json --to --json -o bundle.json + pred solve bundle.json --json # round-trip answer, e.g. {"evaluation": "Max(3)"} + Different evaluations (Max(7) vs Max(3), or Or(true) vs Or(false), or one infeasible) = + a genuine bug in the reduction. + + {{strategy}} + + How to work the whole repo: + 1. `pred list --rules --json` to see every rule. Rule sources are under + {{repo_dir}}/src/rules/.rs — read the ones you probe. + 2. Prioritize by SUSPICION — spend your budget where a bug is most likely first. + Good signals: rules touching weights/costs, offsets or constant shifts, min↔max or + feasibility flips, empty/degenerate inputs, or extraction code that drops information. + Don't sweep alphabetically; triage. + 3. For each rule you pick, construct a small source instance with `pred create` (tiny — + minimal witnesses only) and compare `pred solve source` with `pred solve bundle`. + 4. (Optional, stronger) If a SPECIFIC valid target solution extracts to an invalid or + worse source solution, note its config as `target_config`. + + Report EVERY bug you find (you will likely find several). The moment a round-trip + disagrees for a rule, emit a certificate in this format and then MOVE ON to the next + rule — do NOT stop after the first: + + CERTIFICATE_START + { + "rule": "", + "source": , + "bundle": , + "target_config": "", + "note": "" + } + CERTIFICATE_END + + Only `rule`, `source`, and the target type (from `bundle`) are required; `target_config` + is optional. The verifier re-derives everything from `source` with pred and never trusts + your claim — a wrong or non-minimal certificate is simply rejected, so report only what + you have actually reproduced. Keep going until you run out of budget or have covered the + rules that look worth checking, then run: echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT + + instance_template: | + + Find as many bugs as you can across the reduction rules in the library. + + Library repo: {{repo_dir}} (commit {{commit_hash}}) + Rule sources: {{repo_dir}}/src/rules/ + + Enumerate the rules with `pred list --rules --json`, triage by suspicion, and test the + round-trip correctness of each with pred. Emit a CERTIFICATE_START...CERTIFICATE_END + block for every bug — many are expected; do not stop after the first. You have a limited + budget — be efficient and spend it on the most suspicious rules first. When done, run: + echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT + + + step_limit: 300 + cost_limit: 20.0 + +environment: + timeout: 120 + env: + PAGER: cat + MANPAGER: cat + LESS: -R + +model: + observation_template: | + {% if output.exception_info -%} + {{output.exception_info}} + {% endif -%} + {{output.returncode}} + + {{ output.output[:8000] -}} + + format_error_template: | + Format error: {{error}} + Provide exactly ONE bash command in triple backticks. diff --git a/benchmark/run_mini.py b/benchmark/run_mini.py index 851dfba..5f384e3 100644 --- a/benchmark/run_mini.py +++ b/benchmark/run_mini.py @@ -7,6 +7,7 @@ import argparse import json import os +import re from pathlib import Path import yaml @@ -17,6 +18,7 @@ from benchmark.verify import count_bugs, verify CONFIG_FILE = Path(__file__).parent / "config.yaml" +REPO_CONFIG_FILE = Path(__file__).parent / "config_repo.yaml" SKIP_RULES = {"mod", "traits", "graph_helpers", "analysis", "cost", "registry", "graph"} # Rough average cost per 1K tokens (used as fallback when model doesn't report usage) @@ -31,12 +33,13 @@ def list_rules(repo_dir: str) -> list[str]: return [f.stem for f in sorted(rules_dir.glob("*.rs")) if f.stem not in SKIP_RULES] +_CERT_RE = re.compile(r"CERTIFICATE_START\s*\n(.*?)CERTIFICATE_END", re.DOTALL) + + def parse_certificate(messages: list) -> dict | None: - """Extract structured certificate JSON from agent message history.""" - import re + """Extract the last structured certificate JSON from agent message history.""" for msg in reversed(messages): - content = msg.get("content", "") - block = re.search(r"CERTIFICATE_START\s*\n(.*?)CERTIFICATE_END", content, re.DOTALL) + block = _CERT_RE.search(msg.get("content", "") or "") if block: try: return json.loads(block.group(1).strip()) @@ -45,6 +48,24 @@ def parse_certificate(messages: list) -> dict | None: return None +def parse_all_certificates(messages: list) -> list[dict]: + """Every certificate block in the trajectory (a whole-repo run emits many), de-duplicated + by (rule, source) in first-seen order. Unparseable blocks are skipped.""" + seen, out = set(), [] + for msg in messages: + for block in _CERT_RE.finditer(msg.get("content", "") or ""): + try: + cert = json.loads(block.group(1).strip()) + except json.JSONDecodeError: + continue + key = (cert.get("rule"), json.dumps(cert.get("source"), sort_keys=True)) + if key in seen: + continue + seen.add(key) + out.append(cert) + return out + + def extract_total_tokens(messages: list) -> int: total = 0 for msg in messages: @@ -62,6 +83,22 @@ def save_trajectory(messages: list, path: Path) -> None: f.write(json.dumps({"role": msg.get("role", ""), "content": msg.get("content", "")}) + "\n") +def _session_cost(agent, price): + """Authoritative session spend from the trajectory. ``agent.cost`` is already our + token×price figure (see _build_model); cross-check against a fresh recompute and take + the max — never under-count for the budget guard. Returns (cost, tokens_k, usage).""" + usage = extract_usage(agent.messages) + cost = max(agent.cost, price.cost(usage) if price is not None else 0.0) + total_tokens = usage.total_tokens or extract_total_tokens(agent.messages) + tokens_k = round(total_tokens / 1000, 2) if total_tokens else round(cost / AVG_COST_PER_KTOK * 1000, 2) + return cost, tokens_k, usage + + +def _trajectory(agent) -> list[dict]: + """The agent's own message history — provenance proof carried on cert-bearing rows.""" + return [{"role": m.get("role", ""), "content": m.get("content", "")} for m in agent.messages] + + def _build_model(model_name: str, api_base: str | None, max_tokens: int, price: Price | None, model_kwargs: dict | None = None, api_key: str | None = None): """A LitellmModel whose cost is OUR token×price figure, so mini-swe-agent's own @@ -148,23 +185,11 @@ def run_one( cert = None try: agent.run(task=rule_name) - # agent.cost is already our token×price figure (see _build_model), which is what - # mini-swe-agent's per-step cost_limit enforced. Cross-check it against a fresh - # recompute over the whole trajectory and take the max — never under-count. - usage = extract_usage(agent.messages) - agent_cost = agent.cost - recomputed = price.cost(usage) if price is not None else 0.0 - cost = max(agent_cost, recomputed) - total_tokens = usage.total_tokens or extract_total_tokens(agent.messages) - tokens_k = round(total_tokens / 1000, 2) if total_tokens else round(cost / AVG_COST_PER_KTOK * 1000, 2) + cost, tokens_k, usage = _session_cost(agent, price) usage_row = {"input": usage.input_tokens, "output": usage.output_tokens, "cache_read": usage.cache_read_tokens, "cache_write": usage.cache_write_tokens, - "accounted_cost_usd": round(agent_cost, 6)} - # Provenance record: the agent's own message history. The backend requires this on a - # bug_found row and checks the certificate was emitted here (not pasted from a public - # answer key), so we carry it on every cert-bearing row. - trajectory = [{"role": m.get("role", ""), "content": m.get("content", "")} - for m in agent.messages] + "accounted_cost_usd": round(agent.cost, 6)} + trajectory = _trajectory(agent) cert = parse_certificate(agent.messages) if trajectory_dir is not None: safe_model = model_name.replace("/", "_").replace(":", "_") @@ -221,6 +246,88 @@ def run_one( } +def run_repo_session( + model_name: str, + ctx: EnvContext, + cost_limit: float, + *, + api_base: str | None = None, + trajectory_dir: Path | None = None, + price: Price | None = None, + max_tokens: int = DEFAULT_MAX_TOKENS, + step_limit: int | None = None, + config_path: str | Path | None = None, + strategy: str | None = None, + model_kwargs: dict | None = None, + api_key: str | None = None, +) -> dict: + """One WHOLE-REPO bug-hunting session: the agent gets the entire library + pred and the + full budget as its ``cost_limit``, chooses which rules to probe, and emits a certificate + per bug. Returns ``{"rows": [...], "cost": float, "tokens_k": float}`` — one result row + per distinct emitted certificate, each re-verified with pred and carrying the shared + session trajectory for provenance. Contrast with ``run_one`` (one isolated rule/session). + """ + from minisweagent.agents.default import DefaultAgent + from minisweagent.environments.local import LocalEnvironment + + cfg_file = Path(config_path) if config_path else REPO_CONFIG_FILE + config = yaml.safe_load(cfg_file.read_text(encoding="utf-8")) + if strategy is None: + strat_file = os.environ.get("AGENT_STRATEGY_FILE") + strategy = Path(strat_file).read_text(encoding="utf-8") if strat_file else "" + agent_cfg = config.get("agent", {}) + agent_cfg["cost_limit"] = cost_limit + if step_limit is not None: + agent_cfg["step_limit"] = step_limit + + agent = DefaultAgent( + _build_model(model_name, api_base, max_tokens, price, + model_kwargs=model_kwargs, api_key=api_key), + LocalEnvironment(), + **agent_cfg, + ) + agent.extra_template_vars = { + "repo_dir": str(ctx.repo_path), + "commit_hash": ctx.commit_hash[:7], + "cost_limit": cost_limit, + "strategy": strategy, + } + + agent.run(task="find-bugs") + cost, tokens_k, _usage = _session_cost(agent, price) + trajectory = _trajectory(agent) + if trajectory_dir is not None: + safe_model = model_name.replace("/", "_").replace(":", "_") + save_trajectory(agent.messages, Path(trajectory_dir) / f"{safe_model}_whole-repo.jsonl") + + rows = _rows_from_certificates(parse_all_certificates(agent.messages)) + # The whole session is ONE trajectory shared by every bug — return it once for the + # envelope (build_submission) instead of copying it onto each row. + return {"rows": rows, "cost": cost, "tokens_k": tokens_k, "trajectory": trajectory} + + +def _rows_from_certificates(certs: list[dict]) -> list[dict]: + """Verify each certificate with pred and build one result row per cert. bug_found when + pred confirms, else rejected. No per-row trajectory — a whole-repo run's provenance + trajectory lives once at the envelope level (submission["trajectory"]).""" + rows = [] + for cert in certs: + verdict = verify(cert) + row = { + "rule": cert.get("rule"), + "result": "bug_found" if verdict.accepted else "rejected", + "cost": 0.0, # session cost/tokens live on the submission envelope, not per row + "tokens_k": 0.0, + "certificate": cert, + } + if verdict.accepted: + row["verify_details"] = verdict.details + else: + row["reject_reason"] = verdict.reason + rows.append(row) + return rows + + def main() -> None: parser = argparse.ArgumentParser(description="pred-based bug-finding benchmark") parser.add_argument("--model", default="anthropic/claude-sonnet-4-6", help="LiteLLM model name") diff --git a/benchmark/run_submission.py b/benchmark/run_submission.py index eefb784..68dae70 100644 --- a/benchmark/run_submission.py +++ b/benchmark/run_submission.py @@ -50,20 +50,25 @@ def build_submission( created_at: str | None = None, submitted_by: str | None = None, total_cost_usd: float | None = None, + total_tokens_k: float | None = None, + trajectory: list[dict] | None = None, pred_version: str = "", ) -> dict: - """Assemble the submission envelope from the scheduler's per-rule result rows. - - ``rules_tested`` counts only rules actually attempted (skipped_budget rows don't - count as "reached"); ``bugs_found`` is distinct rules with a confirmed bug. - ``total_cost_usd`` defaults to the sum of row costs; pass the scheduler's tracked - spend to get the budget-faithful figure (the cap is enforced there, not per-row). + """Assemble the submission envelope from the runner's result rows. + + ``rules_tested`` is the number of DISTINCT rules with a result (skipped_budget rows + don't count as "reached"). For per-rule that is the rules attempted; for whole-repo it + is the distinct rules the agent emitted a certificate for — a floor, since rules the + agent probed but found clean aren't represented as rows. ``bugs_found`` is distinct + rules with a confirmed bug. ``total_cost_usd`` / ``total_tokens_k`` default to the sum + of row values; pass explicit session totals (per-rule: the scheduler's tracked spend; + whole-repo: the one session's cost/tokens, which don't live on individual rows). """ attempted = [r for r in rows if r.get("result") != "skipped_budget"] bugs = count_bugs(rows) cost = total_cost_usd if total_cost_usd is not None else sum(r.get("cost", 0.0) for r in rows) - tokens_k = sum(r.get("tokens_k", 0.0) for r in rows) - return { + tokens_k = total_tokens_k if total_tokens_k is not None else sum(r.get("tokens_k", 0.0) for r in rows) + envelope = { "schema_version": SCHEMA_VERSION, "model": model, "library_commit": library_commit, @@ -73,13 +78,17 @@ def build_submission( "total_tokens_k": round(tokens_k, 2), "efficiency_bugs_per_ktok": round(bugs / tokens_k, 4) if tokens_k else 0, "efficiency_bugs_per_dollar": round(bugs / cost, 4) if cost else 0, - "rules_tested": len(attempted), + "rules_tested": len({r.get("rule") for r in attempted}), "results": rows, "runner_version": runner_version, "pred_version": pred_version, "created_at": created_at, "submitted_by": submitted_by, } + # whole-repo: the one shared session log, stored once here (not copied onto each row). + if trajectory is not None: + envelope["trajectory"] = trajectory + return envelope def run( @@ -104,15 +113,20 @@ def run( strategy: str | None = None, model_kwargs: dict | None = None, api_key: str | None = None, + mode: str = "per-rule", ) -> dict: """Run the full budgeted session for one model and return the submission dict. + ``mode`` selects the runner: ``per-rule`` (default) schedules one isolated agent session + per rule under a shared budget; ``whole-repo`` runs ONE session over the whole library — + the agent enumerates and triages the rules itself and emits a certificate per bug. + ``price`` is the submitter's per-token rate (benchmark.cost.Price); with it, spend is recomputed from token usage so the budget is a hard cap. ``safety_margin`` is held back from the budget so the boundary-crossing call still lands under it. In ``fake`` mode no API key or pred binary is needed (FakeRunner) — used by tests - and for smoke-running the container wiring. + and for smoke-running the container wiring; ``fake`` always uses the per-rule path. """ repo = Path(repo_dir) commit = library_commit or pinned_commit() @@ -129,35 +143,52 @@ def run( pred_ver = verify_pred_version(pred_binary) # fail fast if pred != pinned version ctx = EnvContext(repo_path=repo, pred_binary=pred_binary, commit_hash=commit, pred_version=pred_ver) + # Only the per-rule path uses this, but the constructor just stores kwargs (no I/O), + # so building it unconditionally keeps `runner` assigned in exactly one place. runner = MiniSweRunner(api_base=api_base, price=price, max_tokens=max_tokens, config_path=config_path, strategy=strategy, model_kwargs=model_kwargs, api_key=api_key) - rules = list_rules(str(repo)) - if max_rules is not None: - rules = rules[:max_rules] - - with tempfile.TemporaryDirectory() as tmp: - scheduler = Scheduler( - runner=runner, - models=[model], - rules=rules, - total_budget=budget, - per_rule_budget=per_rule_budget, - results_dir=Path(tmp) / "results", - checkpoint_path=Path(tmp) / "checkpoint.json", - ctx=ctx, - resume=False, - parallelism=1, - safety_margin=safety_margin, + if mode == "whole-repo" and not fake: + from benchmark.run_mini import run_repo_session + session = run_repo_session( + model, ctx, cost_limit=max(budget - safety_margin, 0.0), + api_base=api_base, price=price, max_tokens=max_tokens, + config_path=config_path, strategy=strategy, + model_kwargs=model_kwargs, api_key=api_key, ) - completed = scheduler.run_all() - spent = scheduler._spent.get(model) + rows, total_cost, total_tokens = session["rows"], session["cost"], session["tokens_k"] + session_trajectory = session["trajectory"] + else: + rules = list_rules(str(repo)) + if max_rules is not None: + rules = rules[:max_rules] + + with tempfile.TemporaryDirectory() as tmp: + scheduler = Scheduler( + runner=runner, + models=[model], + rules=rules, + total_budget=budget, + per_rule_budget=per_rule_budget, + results_dir=Path(tmp) / "results", + checkpoint_path=Path(tmp) / "checkpoint.json", + ctx=ctx, + resume=False, + parallelism=1, + safety_margin=safety_margin, + ) + completed = scheduler.run_all() + spent = scheduler._spent.get(model) + # per-rule totals: cost is the scheduler's tracked spend; tokens sum from the rows. + # Per-rule rows carry their own trajectories, so there is no envelope-level one. + rows, total_cost, total_tokens = completed[model], spent, None + session_trajectory = None - rows = completed[model] sub = build_submission( model, rows, budget_cap=budget, library_commit=commit, - created_at=created_at, submitted_by=submitted_by, total_cost_usd=spent, + created_at=created_at, submitted_by=submitted_by, + total_cost_usd=total_cost, total_tokens_k=total_tokens, trajectory=session_trajectory, pred_version=getattr(ctx, "pred_version", ""), ) @@ -232,6 +263,11 @@ def main() -> None: "checks, then exit (run this before the full batch).") parser.add_argument("--fake", action="store_true", default=bool(_env("FAKE")), help="No API/pred — FakeRunner wiring run (mostly covered by tests)") + parser.add_argument("--mode", choices=("per-rule", "whole-repo"), + default=_env("AGENT_MODE", "per-rule"), + help="per-rule: one isolated agent session per rule (default). " + "whole-repo: ONE session over the whole library, the agent picks " + "which rules to probe (env AGENT_MODE).") args = parser.parse_args() if not args.model: @@ -283,8 +319,9 @@ def main() -> None: import datetime created_at = datetime.datetime.now(datetime.timezone.utc).isoformat() + detail = f"per-rule ${args.per_rule:.2f}" if args.mode == "per-rule" else "whole-repo" print(f"Running {args.model} at ${args.budget:.0f} budget " - f"(per-rule ${args.per_rule:.2f}){' [FAKE]' if args.fake else ''}...") + f"({detail}){' [FAKE]' if args.fake else ''}...") sub = run( args.model, args.repo_dir, @@ -303,6 +340,7 @@ def main() -> None: strategy=strategy, model_kwargs=model_kwargs, api_key=args.api_key, + mode=args.mode, ) print(f"\n{sub['bugs_found']} claimed bugs | ${sub['total_cost_usd']:.4f} | " f"{sub['rules_tested']} rules attempted") diff --git a/benchmark/submission.schema.json b/benchmark/submission.schema.json index 20ddce3..af9fa5d 100644 --- a/benchmark/submission.schema.json +++ b/benchmark/submission.schema.json @@ -50,7 +50,7 @@ "type": "object", "required": ["rule", "result", "cost", "tokens_k"], "if": {"properties": {"result": {"const": "bug_found"}}}, - "then": {"required": ["rule", "result", "cost", "tokens_k", "certificate", "trajectory"]}, + "then": {"required": ["rule", "result", "cost", "tokens_k", "certificate"]}, "properties": { "rule": {"type": "string"}, "result": { @@ -76,7 +76,7 @@ }, "trajectory": { "type": "array", - "description": "The agent's own message history for this rule — provenance proof. REQUIRED for result=bug_found: the backend checks the certificate was emitted in this trajectory (not pasted from a public answer key) before counting the bug.", + "description": "The agent's own message history for this rule — provenance proof (per-rule mode). For a bug_found row the backend checks the certificate was emitted in a trajectory (this row's, or the envelope-level one for whole-repo runs) before counting the bug.", "items": { "type": "object", "properties": { @@ -98,6 +98,17 @@ "created_at": {"type": "string", "description": "ISO-8601 timestamp the run finished"}, "notes": {"type": "string"}, "placeholder": {"type": "boolean", "description": "True for demo/placeholder rows, never ranked"}, - "test": {"type": "boolean", "description": "When true, the backend scores and stores the submission privately but EXCLUDES it from the public leaderboard (for end-to-end tests). Set via `prb submit --test`."} + "test": {"type": "boolean", "description": "When true, the backend scores and stores the submission privately but EXCLUDES it from the public leaderboard (for end-to-end tests). Set via `prb submit --test`."}, + "trajectory": { + "type": "array", + "description": "Envelope-level session trajectory for a whole-repo run (one session, many bugs) — stored once here instead of on every row. Provenance for its bug_found rows is checked against this. Per-rule runs put the trajectory on each row instead.", + "items": { + "type": "object", + "properties": { + "role": {"type": "string"}, + "content": {"type": "string"} + } + } + } } } diff --git a/benchmark/submit.py b/benchmark/submit.py index 2985e4d..ece1c33 100644 --- a/benchmark/submit.py +++ b/benchmark/submit.py @@ -43,9 +43,10 @@ def load_submission(path: Path) -> dict: def validate_submission(sub: dict) -> list[str]: """Client-side courtesy check. Returns a list of problems ([] == ok). - Mirrors submission.schema.json's hard requirement: a self-reported ``bug_found`` row - must carry both a ``certificate`` and a ``trajectory`` (provenance) — else the backend - would reject it, so catch it here before spending a submission. + Mirrors submission.schema.json's requirement: a self-reported ``bug_found`` row must + carry a ``certificate``, plus a provenance ``trajectory`` — on the row (per-rule) or once + at the envelope level (whole-repo). Else the backend rejects it, so catch it here before + spending a submission. """ problems: list[str] = [] if not isinstance(sub, dict): @@ -57,6 +58,7 @@ def validate_submission(sub: dict) -> list[str]: if not isinstance(results, list): problems.append("results must be a list") return problems + envelope_traj = bool(sub.get("trajectory")) for i, row in enumerate(results): if not isinstance(row, dict): problems.append(f"results[{i}] is not an object") @@ -65,9 +67,9 @@ def validate_submission(sub: dict) -> list[str]: rule = row.get("rule", "?") if not row.get("certificate"): problems.append(f"results[{i}] ({rule}): bug_found row has no certificate") - if not row.get("trajectory"): + if not row.get("trajectory") and not envelope_traj: problems.append(f"results[{i}] ({rule}): bug_found row has no trajectory " - "(required as provenance)") + "(required as provenance — on the row or the envelope)") return problems diff --git a/benchmark/tests/test_verify_submission.py b/benchmark/tests/test_verify_submission.py index a2f30ef..58ab2f9 100644 --- a/benchmark/tests/test_verify_submission.py +++ b/benchmark/tests/test_verify_submission.py @@ -90,6 +90,39 @@ def test_trajectory_source_mismatch_not_counted(self, monkeypatch): scored, _ = vs.score_submission(sub) assert scored["bugs_found"] == 0 + def test_multi_cert_trajectory_provenance(self, monkeypatch): + # A whole-repo session emits MANY certificates in one shared trajectory. Each bug row + # must pass provenance against ANY matching block — not just the last one. + monkeypatch.setattr(vs, "verify", lambda c, r=None: Verdict(True, "ok")) + c1 = {"rule": "r1", "violation": "solve_mismatch", "source": {"n": 1}, "bundle": {}} + c2 = {"rule": "r2", "violation": "solve_mismatch", "source": {"n": 2}, "bundle": {}} + shared = _traj(c1) + _traj(c2) # both certs in the one session trajectory + sub = _submission([_bug_row(c1, trajectory=shared), + _bug_row(c2, trajectory=shared)]) + scored, _ = vs.score_submission(sub) + assert scored["bugs_found"] == 2 # both counted, even though c1 is not the last block + assert all(r["result"] == "bug_found" for r in scored["results"]) + + def test_envelope_trajectory_provenance(self, monkeypatch): + # whole-repo: rows carry NO trajectory; the shared session log is on the envelope, + # parsed once. Each bug's cert is matched against it. + monkeypatch.setattr(vs, "verify", lambda c, r=None: Verdict(True, "ok")) + c1 = {"rule": "r1", "violation": "solve_mismatch", "source": {"n": 1}, "bundle": {}} + c2 = {"rule": "r2", "violation": "solve_mismatch", "source": {"n": 2}, "bundle": {}} + rows = [{"rule": "r1", "result": "bug_found", "cost": 0.0, "tokens_k": 0.0, "certificate": c1}, + {"rule": "r2", "result": "bug_found", "cost": 0.0, "tokens_k": 0.0, "certificate": c2}] + sub = _submission(rows, trajectory=_traj(c1) + _traj(c2)) + scored, _ = vs.score_submission(sub) + assert scored["bugs_found"] == 2 + + def test_no_trajectory_anywhere_not_counted(self, monkeypatch): + # A cert with neither a row trajectory nor an envelope trajectory fails provenance. + monkeypatch.setattr(vs, "verify", lambda c, r=None: Verdict(True, "ok")) + c1 = {"rule": "r1", "violation": "solve_mismatch", "source": {}, "bundle": {}} + rows = [{"rule": "r1", "result": "bug_found", "cost": 0.0, "tokens_k": 0.0, "certificate": c1}] + scored, _ = vs.score_submission(_submission(rows)) + assert scored["bugs_found"] == 0 + def test_distinct_rule_dedup(self, monkeypatch): monkeypatch.setattr(vs, "verify", lambda c, r=None: Verdict(True, "ok")) cert = lambda rule, v: {"rule": rule, "violation": v, "source": {"r": rule}, "bundle": {}} diff --git a/benchmark/tests/test_whole_repo.py b/benchmark/tests/test_whole_repo.py new file mode 100644 index 0000000..9c33daa --- /dev/null +++ b/benchmark/tests/test_whole_repo.py @@ -0,0 +1,92 @@ +""" +Tests for the whole-repo runner mode (one agent session over the whole library, emitting +a certificate per bug). Pure parsing/assembly — no pred, no API (verify monkeypatched). +""" +import json + +from benchmark import run_mini, run_submission +from benchmark.verify import Verdict + + +def _msg(cert: dict) -> dict: + return {"role": "assistant", + "content": "CERTIFICATE_START\n" + json.dumps(cert) + "\nCERTIFICATE_END"} + + +class TestParseAllCertificates: + def test_returns_every_block(self): + c1 = {"rule": "r1", "source": {"n": 1}} + c2 = {"rule": "r2", "source": {"n": 2}} + certs = run_mini.parse_all_certificates([_msg(c1), {"role": "user", "content": "x"}, _msg(c2)]) + assert [c["rule"] for c in certs] == ["r1", "r2"] + + def test_two_blocks_in_one_message(self): + c1 = {"rule": "r1", "source": {"n": 1}} + c2 = {"rule": "r2", "source": {"n": 2}} + merged = {"role": "assistant", "content": _msg(c1)["content"] + "\n" + _msg(c2)["content"]} + assert len(run_mini.parse_all_certificates([merged])) == 2 + + def test_dedups_same_rule_and_source(self): + c = {"rule": "r1", "source": {"n": 1}} + certs = run_mini.parse_all_certificates([_msg(c), _msg(c)]) + assert len(certs) == 1 + + def test_skips_unparseable(self): + bad = {"role": "assistant", "content": "CERTIFICATE_START\n{not json\nCERTIFICATE_END"} + assert run_mini.parse_all_certificates([bad]) == [] + + +class TestRowsFromCertificates: + def test_bug_and_rejected_rows_share_trajectory(self, monkeypatch): + # r1 confirmed, r2 rejected by pred; both rows carry the same session trajectory. + monkeypatch.setattr(run_mini, "verify", + lambda c, r=None: Verdict(c.get("rule") == "r1", "ok")) + certs = [{"rule": "r1", "source": {}}, {"rule": "r2", "source": {}}] + rows = run_mini._rows_from_certificates(certs) + assert [r["result"] for r in rows] == ["bug_found", "rejected"] + # No per-row trajectory — the whole-repo trajectory lives once on the envelope. + assert all("trajectory" not in r for r in rows) + assert rows[0]["certificate"]["rule"] == "r1" + + +class TestBuildSubmissionTotals: + def test_explicit_session_totals_override_row_sums(self): + # whole-repo rows carry 0 cost/tokens; the session totals come in as explicit args. + rows = [{"rule": "r1", "result": "bug_found", "cost": 0.0, "tokens_k": 0.0}] + sub = run_submission.build_submission( + "m", rows, budget_cap=20, library_commit="c", + total_cost_usd=3.5, total_tokens_k=42.0) + assert sub["total_cost_usd"] == 3.5 + assert sub["total_tokens_k"] == 42.0 + assert sub["efficiency_bugs_per_dollar"] == round(1 / 3.5, 4) + + +class TestRunWholeRepoWiring: + def test_run_dispatches_to_repo_session(self, monkeypatch): + # run(mode="whole-repo") must call run_repo_session and build an envelope from it, + # without touching the per-rule Scheduler. + captured = {} + + traj = [{"role": "assistant", "content": "run log"}] + + def fake_repo_session(model, ctx, cost_limit, **kw): + captured["cost_limit"] = cost_limit + return {"rows": [{"rule": "r1", "result": "bug_found", "cost": 0.0, "tokens_k": 0.0, + "certificate": {"rule": "r1", "source": {}}}], + "cost": 5.0, "tokens_k": 30.0, "trajectory": traj} + + monkeypatch.setattr(run_submission, "find_pred_binary", lambda: "pred") + monkeypatch.setattr(run_submission, "verify_pred_version", lambda p: "1.2.3") + monkeypatch.setattr(run_submission, "EnvContext", + lambda **kw: type("C", (), {"pred_version": "1.2.3", **kw})()) + monkeypatch.setattr(run_submission, "Scheduler", + lambda **kw: (_ for _ in ()).throw(AssertionError("scheduler used"))) + monkeypatch.setattr("benchmark.run_mini.run_repo_session", fake_repo_session) + + sub = run_submission.run("m", "/repo", budget=20, safety_margin=1.0, + library_commit="deadbeef", mode="whole-repo") + assert captured["cost_limit"] == 19.0 # budget - safety_margin + assert sub["total_cost_usd"] == 5.0 + assert sub["bugs_found"] == 1 + assert sub["trajectory"] is traj # stored once on the envelope + assert "trajectory" not in sub["results"][0] # not duplicated onto rows diff --git a/benchmark/verify_submission.py b/benchmark/verify_submission.py index f768f79..ba1698f 100644 --- a/benchmark/verify_submission.py +++ b/benchmark/verify_submission.py @@ -28,45 +28,43 @@ CERT_BLOCK = re.compile(r"CERTIFICATE_START\s*\n(.*?)CERTIFICATE_END", re.DOTALL) -def _cert_from_trajectory(trajectory) -> dict | None: - """Parse the last CERTIFICATE_START…END block emitted in an agent trajectory. +def _certs_from_trajectory(trajectory) -> list[dict]: + """Parse every CERTIFICATE_START…END block emitted in an agent trajectory. - ``trajectory`` is a list of {role, content} messages (as saved by the runner). - Returns the parsed certificate dict, or None if absent/unparseable. + ``trajectory`` is a list of {role, content} messages (as saved by the runner). A + whole-repo run emits many certificates in one trajectory, so return all of them; + unparseable blocks are skipped. """ - if not trajectory: - return None - for msg in reversed(trajectory): + certs: list[dict] = [] + for msg in trajectory or []: content = msg.get("content", "") or "" - m = CERT_BLOCK.search(content) - if m: + for m in CERT_BLOCK.finditer(content): try: - return json.loads(m.group(1).strip()) + certs.append(json.loads(m.group(1).strip())) except json.JSONDecodeError: - return None - return None + continue + return certs -def _provenance_ok(row: dict, cert: dict) -> tuple[bool, str]: +def _provenance_ok(row: dict, cert: dict, session_certs: list[dict] = ()) -> tuple[bool, str]: """Check the certificate was actually produced by this model's own run. - Guards against copied answer keys: a scored bug must appear in a CERTIFICATE block the - agent emitted in its trajectory, with the same rule and source instance. This can't - make copying impossible (the target library is public), but it lifts the bar from - "paste a rule name" to "produce a full run artifact whose source still round-trip-fails - under pred". + Guards against copied answer keys: a scored bug must appear as a CERTIFICATE block the + agent emitted in its trajectory, matching both rule and source instance. The trajectory + is either the row's own (per-rule: one session per row) or the shared session log at the + envelope level (whole-repo: one session, many bugs) — ``session_certs`` is that envelope + log pre-parsed once. Any emitted block that matches counts. This can't make copying + impossible (the library is public), but it lifts the bar from "paste a rule name" to + "produce a run artifact whose source still round-trip-fails". """ - traj = row.get("trajectory") - if not traj: + emitted = _certs_from_trajectory(row.get("trajectory")) + list(session_certs) + if not emitted: return False, "no trajectory attached (required for a scored bug)" - emitted = _cert_from_trajectory(traj) - if emitted is None: - return False, "no CERTIFICATE block found in trajectory" - if emitted.get("rule") not in (cert.get("rule"), row.get("rule")): - return False, "trajectory certificate targets a different rule" - if emitted.get("source") != cert.get("source"): - return False, "trajectory certificate source does not match the submitted source" - return True, "reproduced in the model's own trajectory" + want_rules = (cert.get("rule"), row.get("rule")) + for e in emitted: + if e.get("rule") in want_rules and e.get("source") == cert.get("source"): + return True, "reproduced in the model's own trajectory" + return False, "no trajectory certificate matches the submitted rule + source" def score_submission(submission: dict, repo_dir: str | None = None) -> tuple[dict, list[dict]]: @@ -79,6 +77,9 @@ def score_submission(submission: dict, repo_dir: str | None = None) -> tuple[dic """ rescored: list[dict] = [] report: list[dict] = [] + # Whole-repo runs store ONE session trajectory at the envelope level; parse it once here + # rather than re-parsing a copy on every bug row. + session_certs = _certs_from_trajectory(submission.get("trajectory")) for row in submission.get("results", []): cert = row.get("certificate") @@ -88,7 +89,7 @@ def score_submission(submission: dict, repo_dir: str | None = None) -> tuple[dic cert.setdefault("rule", row.get("rule")) verdict = verify(cert, repo_dir) - prov_ok, prov_reason = _provenance_ok(row, cert) if verdict.accepted else (False, "") + prov_ok, prov_reason = _provenance_ok(row, cert, session_certs) if verdict.accepted else (False, "") accepted = verdict.accepted and prov_ok new = dict(row) if accepted: diff --git a/submission.env.example b/submission.env.example index 0951fbc..158d92f 100644 --- a/submission.env.example +++ b/submission.env.example @@ -41,6 +41,11 @@ PRICE_OUT=15.0 # USD / 1M output tokens # MAX_TOKENS=8192 # per-call output ceiling # MAX_RULES=5 # cap rules attempted (smoke runs only; omit for a real run) +# ── AGENT MODE ──────────────────────────────────────────────────────────────── +# AGENT_MODE=per-rule # per-rule (default): one isolated session per rule, budget split +# # evenly. whole-repo: ONE session over the whole library — the agent +# # enumerates and triages the rules itself under a single budget. + # ── CUSTOM AGENT PROMPT (mount the files too: -v "$PWD/cfg:/cfg") ───────────── # AGENT_STRATEGY_FILE=/cfg/strategy.md # extra hints into the prompt's {{strategy}} slot # AGENT_CONFIG=/cfg/config.yaml # full prompt-config replacement