Deadlock after optimistic sync: L2 sync state only exits `Syncing` on a new peer block, so a full sequencer restart stalls the chain permanently

[shu-unifra]

Summary

After an optimistic sync, the chain orchestrator's L2 sync state can only transition from Syncing to Synced by receiving a new block from a peer that extends the head. Since both the sequencer event loop and L1 notification processing are gated on the node being synced, a network where all sequencer nodes restart at the same time (with local data far enough behind the network head to trigger optimistic sync) ends up in a permanent deadlock: no node can produce the block that everyone is waiting for, and the chain stalls forever.

We hit this on a devnet running a fork of this repo, but the relevant logic is identical on current main.

Observed behavior

• Two sequencer nodes and one follower (RPC) node restarted at roughly the same time. Their local reth DBs were far behind the network head (~1.1M local vs ~5.35M network, well past optimistic_sync_trigger).
• On startup each node received a peer block far ahead of its head, triggered optimistic_sync, and reth's staged pipeline synced to the target block successfully (Finish stage reached target, currentBlock == highestBlock).
• After the pipeline finished, no further FCUs were ever issued. reth logs repeat:
  Beacon client online, but no consensus updates received for a while with a monotonically growing period.
• eth_syncing keeps returning a sync-progress object on every node (network sync state never set to Idle), sequencers never start sequencing, derivation stalls, and the chain head never advances again. Readiness probes based on eth_syncing == false fail indefinitely.

Root cause

All in crates/chain-orchestrator/src/lib.rs (line refs on current main):

1. On receiving a peer block beyond optimistic_sync_threshold, the orchestrator calls engine.optimistic_sync(...) and sets sync_state.l2_mut().set_syncing() (lib.rs:997-998).
2. The only place the L2 sync state transitions back to Synced is import_chain() — i.e. it requires receiving another new block from a peer that extends the chain (lib.rs:1206-1208). Nothing marks L2 synced when the optimistic pipeline sync itself completes and the node is already at the network tip.
3. consolidate_chain() — which is what eventually calls update_sync_state(RethSyncState::Idle) (lib.rs:1303) and makes eth_syncing return false — only runs once sync_state.is_synced().
4. Meanwhile the event loop gates:
   • the sequencer arm on self.sequencer.is_some() && self.sync_state.is_synced() (lib.rs:216), so no sequencer produces blocks while "syncing";
   • the L1 notification arm on self.sync_state.l2().is_synced() (lib.rs:228), so L1Notification::Synced (lib.rs:607-609) is never even consumed and L1 sync state can't progress either.

Resulting cycle:

L2 synced  ⇐ requires new block from a peer (import_chain)
new block  ⇐ requires some sequencer to be sequencing
sequencing ⇐ requires sync_state.is_synced()  ⇐ requires L2 synced

Any single surviving synced node (or an externally produced block) breaks the cycle, which is why this only manifests when every block producer restarts into optimistic sync simultaneously — e.g. a coordinated redeploy of all sequencers, or disaster recovery from a snapshot.

Suggested direction

When the optimistic sync target is reached (pipeline Finish at the FCS head) and no newer peer block is pending, the orchestrator should transition L2 to Synced and run consolidate_chain() on its own, rather than waiting indefinitely for the next NewBlockWithPeer. Alternatively (or additionally), the sequencer-enabled node could be allowed to start sequencing once its head matches the optimistic sync target, since it is by definition at the tip it was told to sync to.

Happy to provide more logs or test the fix on our devnet.