Skip to content

fix(prover): do not persist aborted proving jobs so they can be re-proven after a restart#24558

Draft
AztecBot wants to merge 2 commits into
nextfrom
cb/prover-broker-abort-reenqueue
Draft

fix(prover): do not persist aborted proving jobs so they can be re-proven after a restart#24558
AztecBot wants to merge 2 commits into
nextfrom
cb/prover-broker-abort-reenqueue

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

When the prover broker cancels/aborts a proving job (cancelProvingJob#reportProvingJobError(..., aborted=true)), it wrote the abort to the database as a settled { status: 'rejected', reason: 'Aborted' } result. On the next broker start, ProvingBroker.start() restores every persisted job and, if it has a result, replays that result into resultsCache instead of re-enqueuing the job. So a proving job's id (a deterministic hash of its inputs) that had been aborted came back as a permanent Aborted rejection: any later enqueue of the same job hit the jobsCache.has(id) branch and returned the cached aborted status — Cached proving job … Not enqueuing again — and the facade failed the job with Aborted.

Net effect: once a proof was aborted, it stayed aborted across restarts, so the epoch could never be proven. A deploy restarts both the prover node and the broker, and this is what turned a planned mid-epoch redeploy into a testnet epoch prune.

Fix

An abort is a cancellation, not a terminal result, so the broker no longer persists it: #reportProvingJobError skips the database.setProvingJobError write when aborted is true. The in-memory rejection is still recorded (so any current waiter is notified with Aborted), but the database keeps the job as pending. Because start() re-enqueues every persisted job that has no stored result, the aborted job is retried after the restart instead of coming back as a poisoned Aborted rejection.

This is deliberately the whole change — an in-memory "these were aborted" set would be wiped by the very restart we need to survive, so the durable fix lives entirely in what does (not) get written to the database. Genuine proof failures and timeouts are still persisted and terminal, unchanged.

Test

Adds a unit test in proving_broker.test.ts (run against both the in-memory and persisted DB variants): enqueue a job, cancel it, restart the broker, and assert the job is in-queue again and can be picked up and completed. It fails on the current code (the restarted broker restores the job as rejected) and passes with this change.

Note: the test was not run locally in this session — executing the prover-client suite requires a full noir/wasm bootstrap that wasn't available here — so it is relied on to run in CI.

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 6, 2026
@AztecBot AztecBot changed the title fix(prover): re-run aborted proving jobs instead of caching the abort forever fix(prover): do not persist aborted proving jobs so they can be re-proven after a restart Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants