Skip to content

fix(prover-node): do not abort in-flight proving jobs on a clean shutdown#24562

Draft
AztecBot wants to merge 1 commit into
cb/prover-broker-abort-reenqueuefrom
cb/prover-node-keep-jobs-on-shutdown
Draft

fix(prover-node): do not abort in-flight proving jobs on a clean shutdown#24562
AztecBot wants to merge 1 commit into
cb/prover-broker-abort-reenqueuefrom
cb/prover-node-keep-jobs-on-shutdown

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #24558 (base is that PR's branch, cb/prover-broker-abort-reenqueue) — review/merge that one first.

Problem

When the prover node shuts down cleanly (a deploy/restart), ProverNode.stop() calls EpochProvingJob.stop('stopped')interruptProcessing()prover.cancel(). That cancel tears down the orchestrator's scheduler and, depending on config, aborts every in-flight broker job. Aborting on a clean restart is wasteful: the in-flight proofs are still valid, the agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the exact same (deterministically-hashed) job ids. Cancelling them throws that work away and forces a re-prove from scratch.

The orchestrator already had the right primitive — resetSchedulerState(abortJobs), whose own docs say aborting is correct on reorg and wrong on shutdown, and the top-tree sub-orchestrator already threads an abortJobs flag — but the top-level ProvingOrchestrator.cancel() applied a single static cancelJobsOnStop config to every cancellation reason, with no way to say "this one is a clean shutdown, keep the jobs".

Fix

Make the cancel reason-aware, so a clean shutdown never aborts the broker jobs while reorg/timeout/failure keep the existing configured behaviour:

  • EpochProver.cancel(abortJobs?: boolean) — optional override; when omitted, the orchestrator's configured cancelJobsOnStop is used (unchanged default).
  • ProvingOrchestrator.cancel(abortJobs = this.cancelJobsOnStop) passes the flag through to resetSchedulerState, and its graceful-stop path (cancelInternal) now passes false, matching the top-tree sub-orchestrator which already does this.
  • EpochProvingJob.interruptProcessing() passes false when the terminal state is 'stopped' (clean shutdown), and otherwise defers to the configured behaviour.

This composes with #24558: even if a stale in-flight job were somehow aborted, that PR already stops the abort from being persisted; together, a clean redeploy mid-epoch neither cancels the in-flight jobs nor poisons them, so the epoch keeps proving across the restart.

Tests

  • orchestrator_lifecycle.test.ts: cancel(false) leaves the in-flight jobs' abort signals untouched even when cancelJobsOnStop is true, and cancel(true) aborts them even when it is false.
  • epoch-proving-job.test.ts: a clean external stop calls prover.cancel(false); a reorg-driven stop does not force abortJobs=false.

Note: not run locally in this session — executing these suites needs a full noir/wasm bootstrap that wasn't available here — so relied on CI.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 6, 2026
@AztecBot AztecBot force-pushed the cb/prover-node-keep-jobs-on-shutdown branch from dd12b69 to 98c348b Compare July 7, 2026 10:03
@AztecBot AztecBot force-pushed the cb/prover-node-keep-jobs-on-shutdown branch 2 times, most recently from 9f93efd to faf519e Compare July 7, 2026 10:28
@AztecBot AztecBot force-pushed the cb/prover-node-keep-jobs-on-shutdown branch from faf519e to 602a8ba Compare July 7, 2026 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant