diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5c8995f..59018cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,25 +1,19 @@ # Submitting a model run The benchmark gives every model the **same $20 API budget** and asks: how many distinct -reduction-rule bugs (counterexamples) can it find? This document describes the end-to-end -submission pipeline. +reduction-rule bugs can it find? ``` make run ─▶ submission.json ─▶ python -m benchmark.submit ──▶ private store (R2) - (certificate + trajectory = answer key, never in git) │ maintainer's scorer re-verifies every certificate with pred - (zero trust) + a provenance check, OFF the public repo │ - only the AGGREGATE (counts / cost / tokens — no rules, no certs) - is published ─▶ PR to site/results.json ─▶ GitHub Pages (deploy) + only the aggregate is published ─▶ PR ─▶ GitHub Pages ``` -Your submission carries the certificate + trajectory — the answer key. On a fixed public -library commit a `pred`-confirmed certificate counts regardless of who produced it, so it -must stay private: the CLI uploads to a private store, and only the aggregate becomes -public. Self-reported counts are never trusted — the score is recomputed from `pred` -re-verification. +Your submission carries the certificate + trajectory, so it uploads to a private store; +only the aggregate is published. Self-reported counts are never trusted — the score is +recomputed by `pred`. ## 1. Produce a `submission.json` (dockerized runner) @@ -122,14 +116,12 @@ make preflight # docker run --env-file submission.env --preflight ``` It checks the `pred` binary + version, that the library rules are present, and makes one -minimal model call through the exact batch code path (validating credentials, endpoint, -`model_kwargs`, and that pricing computes). It exits non-zero on any failure. (The runner's -no-API wiring is covered by the pytest suite, not a separate command.) +minimal model call through the exact batch code path. It exits non-zero on any failure. ## 2. Submit it (CLI upload) -Submission is a **CLI upload** — there's no web form and the file never enters git (it -carries the answer key). Get the endpoint URL + a token from the maintainer, then: +Submission is a **CLI upload** — no web form, and the file never enters git. Get the +endpoint URL + a token from the maintainer, then: ```bash export PRB_SUBMIT_URL= # from the maintainer @@ -140,10 +132,9 @@ python -m benchmark.submit --predictions out/submission.json # --test scored + stored privately, but excluded from the public leaderboard ``` -The CLI validates the file against `submission.schema.json`, then uploads it over HTTPS to -a private store (Cloudflare R2). The maintainer's scorer re-verifies every certificate with -`pred` off-repo (see §3) and opens a PR that updates only the aggregate `site/results.json`; -merging deploys the site to **GitHub Pages**. Your self-reported counts are never trusted. +The CLI validates the file against `submission.schema.json`, then uploads it to a private +store (Cloudflare R2). The maintainer's scorer re-verifies it with `pred` (see §3) and opens +a PR that updates the aggregate `site/results.json`; merging deploys to **GitHub Pages**. See `intake/cloudflare-worker/README.md` for the intake setup. ## 3. Backend verification (automatic, zero-trust) diff --git a/README.md b/README.md index f54e221..d53a6b6 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ A benchmark that measures how efficiently AI models find bugs in reduction rules from the [problem-reductions](https://github.com/CodingThrust/problem-reductions) library (290+ rules). -The leaderboard is a static site (`site/`) published to **GitHub Pages**. Submitting uses a CLI (`python -m benchmark.submit`) that uploads your run to a **private store** — the certificate + trajectory are the answer key, so they never enter the public repo. The maintainer re-verifies every certificate with `pred` off-repo, and only the **aggregate** (counts, no rules/certs) becomes public. See [CONTRIBUTING.md](CONTRIBUTING.md) to run and submit. +The leaderboard is a static site (`site/`) published to **GitHub Pages**. Submitting uses a CLI (`python -m benchmark.submit`) that uploads your run to a private store; the maintainer re-verifies it with `pred` and publishes only the aggregate. See [CONTRIBUTING.md](CONTRIBUTING.md) to run and submit. ## What this measures @@ -46,7 +46,7 @@ class MyRunner(AgentRunner): Then pass it to `Scheduler` in `benchmark/scheduler.py`. See `MiniSweRunner` for a full example. -A run is packaged as a `submission.json` (envelope around the per-rule rows, see `benchmark/submission.schema.json`) and uploaded with `python -m benchmark.submit` to a private store; the maintainer re-verifies every certificate with `pred` off-repo and publishes only the aggregate. See [CONTRIBUTING.md](CONTRIBUTING.md). +A run is packaged as a `submission.json` (see `benchmark/submission.schema.json`) and uploaded with `python -m benchmark.submit`. See [CONTRIBUTING.md](CONTRIBUTING.md). ## How to run locally diff --git a/site/README.md b/site/README.md index 1eb64bd..f9cd16c 100644 --- a/site/README.md +++ b/site/README.md @@ -1,16 +1,13 @@ # Leaderboard site -Static leaderboard (one `index.html`, no app server, instant first paint), published to -**GitHub Pages** by `.github/workflows/publish-on-merge.yml`. Same **$20** budget for every -model — who finds the most bugs in the problem-reductions reduction rules? Every bug is -independently re-verified by `pred` (one rule = one bug). +Static leaderboard (`index.html`, no app server) published to **GitHub Pages** by +`.github/workflows/publish-on-merge.yml`. Every model gets the same **$20** budget; every +bug is re-verified by `pred`. `index.html` reads two data files served alongside it: -- `results.json` — the scored aggregate leaderboard (refreshed by `score-from-r2.yml`, - which re-verifies submissions off-repo and opens a PR; merging deploys the site) +- `results.json` — the aggregate leaderboard, refreshed by `score-from-r2.yml` - `tasks.json` — the rule set shown on the Tasks tab -Preview locally with `make serve` (the data files are loaded via `fetch`, so open it over -HTTP, not `file://`). `results.json` holds only aggregate counts — never certificates or -buggy-rule identities. +Preview locally with `make serve` (data files load via `fetch`, so serve over HTTP, not +`file://`). `results.json` holds only aggregate counts. diff --git a/site/index.html b/site/index.html index 6ad2473..4265ddd 100644 --- a/site/index.html +++ b/site/index.html @@ -297,7 +297,7 @@

Leaderboard

From your model to the leaderboard

Give a model the same $20 and see how many bugs it can find in the library's problem reductions. You run it on your own model and key; we independently re-check every bug before - it scores — so the ranking is earned, not claimed.

+ it scores.

You run it
@@ -305,7 +305,7 @@

From your model to the leaderboard

1

Get the runner

One Docker image: the solver (pred) plus the rule library, pinned to the benchmark version.

2

Plug in any model

OpenAI, Anthropic, DeepSeek, a self-hosted endpoint — you bring the API key and the price.

3

Hunt for bugs

Your model probes each rule for a counterexample until the $20 runs out; each find is checked by the solver on the spot.

a bug = solve(x) ≠ solve(reduce(x))
-
4

Submit from the command line

One hf upload of submission.json to the dataset, or open a pull request — no web form.

+
4

Submit from the command line

Upload submission.json with benchmark.submit. No web form.

@@ -319,11 +319,11 @@

From your model to the leaderboard

What counts as a bug

-

Each reduction turns problem A into an equivalent problem B. It's buggy on an input when solving that input directly gives a different answer than solving it through the reduction — a fact anyone can re-check, so a wrong claim simply fails to reproduce.

+

Each reduction turns problem A into an equivalent problem B. It's buggy on an input when solving that input directly gives a different answer than solving it through the reduction.

Why the $20 is fair

-

Every model gets the same budget, measured from real token usage at the price you declare — not a gateway's dollar guess. Ranking is by confirmed distinct-rule bugs, with bugs per 1K tokens as the efficiency tie-break.

+

Every model gets the same budget, measured from real token usage at the price you declare. Ranking is by confirmed distinct-rule bugs, with bugs per 1K tokens as the efficiency tie-break.

@@ -331,30 +331,29 @@

Why the $20 is fair

Submit a run

-

Run the dockerized runner at the fixed $20 budget against problem-reductions v0.6.0 — any provider, all config in one submission.env — then add the submission.json it produces to the submissions dataset. Every certificate is re-verified by pred on the backend before it counts.

+

Run the dockerized runner at the fixed $20 budget against problem-reductions v0.6.0, then upload the submission.json it produces. Every certificate is re-verified by pred before it counts.

1

Produce submission.json

-

Configure once (model · key · price), validate with one tiny real call, then run.

- 
# configure once (model · key · price — any provider), then:
-cp submission.env.example submission.env
+        
Configure once, validate with one tiny real call, then run.

+        
cp submission.env.example submission.env   # model · key · price
 make preflight   # validate config with one tiny real call
 make run         # → ./out/submission.json
2

Upload it

-

It lands as PENDING; the backend picks it up and re-verifies every certificate.

- 
hf upload isPANN/problem-reductions-submissions submission.json \
-  submissions/<your-handle>/<model>.json --repo-type dataset
+

The backend picks it up and re-verifies every certificate.

+ 
export PRB_SUBMIT_URL=… PRB_API_KEY=…
+python -m benchmark.submit --predictions out/submission.json
-

…or open a pull request adding it to the GitHub repo. Self-reported counts are advisory — only confirmed, distinct-rule bugs are ranked.

+

Self-reported counts are advisory; only confirmed distinct-rule bugs are ranked.

About the benchmark

-

An open measurement of how many bugs a model can find in the problem-reductions library — each reduction is meant to translate one computational problem into an equivalent one. Every model gets the same $20, and every claimed bug is independently re-verified server-side, so the ranking reflects confirmed findings, not self-reported counts.

+

An open measurement of how many bugs a model can find in the problem-reductions library. Every model gets the same $20, and every claimed bug is independently re-verified server-side.

How to cite

@misc{TODO_citation_key,
@@ -541,7 +540,7 @@ 
How to cite

     
Bugs / Ktok
${row.effNum.toFixed(4)}

     
Bugs / $
${row.spentEff.toFixed(3)}

   
-  
Which rules each model found bugs in is not published — a confirmed certificate on this fixed commit counts regardless of provenance, so the buggy-rule identities are kept private to protect the benchmark.
`;
+  
Buggy-rule identities are not published.
`;
   openDrawer();
 }
 
diff --git a/site/results/README.md b/site/results/README.md
index 5b8efab..85fef61 100644
--- a/site/results/README.md
+++ b/site/results/README.md
@@ -1,10 +1,8 @@
 # Per-submission leaderboard entries
 
 Each file here is **one submission's** public leaderboard entry:
-`----.json` (aggregate only — counts, cost, tokens,
-efficiency; never certificates or buggy-rule identities).
+`----.json`. Aggregate only; no certificates or rule identities.
 
-- Written by `score-from-r2.yml`, one **PR per submission**, so each is reviewed, merged,
-  or reverted independently.
+- Written by `score-from-r2.yml`, one **PR per submission**.
 - On merge, `publish-on-merge.yml` aggregates them (best run per model) into the deployed
-  `site/results.json` (generated, not committed) and publishes to GitHub Pages.
+  `site/results.json` and publishes to GitHub Pages.
diff --git a/submissions/README.md b/submissions/README.md
index a34a7dc..cc6c04a 100644
--- a/submissions/README.md
+++ b/submissions/README.md
@@ -1,17 +1,12 @@
 # Submissions
 
-**Submissions are never committed to this public repo.** A submission carries the
-certificate + trajectory — the answer key. On a fixed public library commit a
-`pred`-confirmed certificate counts regardless of who produced it, so publishing one would
-be a free answer key. Everything under `submissions/*.json` is `.gitignore`d.
-
-This directory is only a **local scratch space** for the self-run scoring path.
+**Submissions are never committed to this public repo** — they carry the answer key
+(certificate + trajectory) and are `.gitignore`d. This directory is only local scratch for
+the self-run scoring path.
 
 ## How to submit (external)
 
-Use the CLI intake — it uploads over HTTPS to a private store (Cloudflare R2); only the
-maintainer ever reads it back, and only the aggregate (counts, no rules/certs) becomes
-public.
+Use the CLI intake — it uploads to a private store; only the aggregate becomes public.
 
 ```bash
 export PRB_SUBMIT_URL=   # from the maintainer
@@ -26,8 +21,8 @@ the public leaderboard. See `intake/cloudflare-worker/README.md`.
 ## Self-run scoring (maintainer / local)
 
 Drop scored submission files into this directory and run `make publish-local`: it scores
-them with `pred`, rebuilds the aggregate, and writes `site/results.json` (guarded so no
-answer-key field leaks). The files here stay local — they never enter git.
+them with `pred`, rebuilds the aggregate, and writes `site/results.json`. The files here
+stay local; they never enter git.
 
 ## Notes