Add whole-repo agent mode alongside per-rule#38
Merged
Conversation
A second, opt-in run mode for comparison. per-rule (default, unchanged) runs one isolated agent session per rule with the budget split evenly. whole-repo runs ONE session over the whole library: the agent enumerates the rules with `pred list`, triages them by suspicion, and emits a certificate per bug under a single budget (its cost_limit). Both modes produce the same submission.json and are scored by the same zero-trust pred re-verification, so results are directly comparable. Select with AGENT_MODE=whole-repo (or --mode whole-repo). - benchmark/config_repo.yaml: whole-repo prompt (enumerate + triage by suspicion, emit many certificates, don't stop at the first). - run_mini.py: parse_all_certificates (multi-cert, de-duped) and run_repo_session (single session → one row per cert, shared trajectory). - run_submission.py: --mode/AGENT_MODE, whole-repo branch, and a total_tokens_k override on build_submission (session totals don't live on per-rule rows). - verify_submission.py: provenance now matches ANY certificate block in the trajectory, not just the last — a whole-repo run emits many in one trajectory. - config.yaml: per-rule prompt gives a qualitative budget hint instead of the literal dollar figure (the agent can't meter its own spend). - submission.env.example + run-benchmark skill: document AGENT_MODE; the skill now confirms experiment parameters (mode/budget) with the user before a run. - tests: test_whole_repo.py + a multi-cert provenance test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xw6L5yUrz33UrQpanbwnH
Code-quality cleanup from a /simplify pass: - run_mini.py: extract _session_cost() + _trajectory() helpers shared by run_one and run_repo_session — the budget-faithful cost/token accounting lived in two runners and could drift. Drop a redundant `strategy or ""`. - run_submission.py run(): assign `runner` in exactly one place (build the per-rule MiniSweRunner unconditionally in the real branch — its constructor is free), and call build_submission ONCE after the mode branch instead of in both. - run_submission.py build_submission: rules_tested counts DISTINCT rules with a result, not row count — for whole-repo that's the distinct rules with a cert (a floor), fixing a docstring that claimed "rules attempted" for both modes. Skipped (noted as follow-ups): the whole-repo session trajectory is duplicated onto every bug row, so the backend re-parses it once per bug (O(bugs) over the same log) and stores N copies — the real fix is to hold the trajectory once at the envelope level, a schema change beyond this cleanup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xw6L5yUrz33UrQpanbwnH
A whole-repo run is one session that emits many bugs, but each bug row carried a full copy of the shared trajectory — so submission.json duplicated the log N times and the backend re-parsed it once per bug (O(bugs) over identical data). Store it once: run_repo_session returns the trajectory; build_submission puts it at submission["trajectory"]; rows carry only their certificate. score_submission parses the envelope trajectory once and _provenance_ok matches each bug's cert against it (or the row's own trajectory, unchanged for per-rule). Schema gains an envelope-level `trajectory` and drops the row-level `trajectory` from the bug_found requirement (provenance is enforced by the backend / submit CLI, which now accept the trajectory on the row OR the envelope). Per-rule mode is unchanged: each row keeps its own trajectory, no envelope one. Tests: envelope-trajectory provenance + no-trajectory-anywhere rejection; updated whole-repo row/wiring tests. 153 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018xw6L5yUrz33UrQpanbwnH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second, opt-in run mode so the two can be compared on the same scoring pipeline.
pred list, triages by suspicion, and emits a certificate per bug under a single budget. Select withAGENT_MODE=whole-repo/--mode whole-repo.Both produce the same
submission.jsonand are scored by the same zero-trustpredre-verification, so results are directly comparable.Changes
benchmark/config_repo.yaml— whole-repo prompt (enumerate + triage, emit many certificates, don't stop at the first).run_mini.py—parse_all_certificates(multi-cert, de-duped) +run_repo_session(single session → one row per cert, shared trajectory).run_submission.py—--mode/AGENT_MODE, whole-repo branch,total_tokens_koverride onbuild_submission.verify_submission.py— provenance now matches any certificate block in the trajectory (a whole-repo run emits many), not just the last.config.yaml— per-rule prompt gives a qualitative budget hint instead of the literal dollar figure.submission.env.example+ run-benchmark skill — documentAGENT_MODE; skill now confirms experiment parameters (mode/budget) with the user before a run.test_whole_repo.py+ a multi-cert provenance test (151 unit tests pass).🤖 Generated with Claude Code