feat(workflow): StateSync recipe completes at release — mark-ready is terminal

[bdchatham]

Summary

The StateSync recipe ends at mark-ready. Complete means every mutation was performed and the node was released to re-bootstrap; catch-up is verified node-side (sdk WaitCaughtUp, seictl node watch --until=caught-up, alerts) by whoever triggered the workflow.

Why: catching_up is a proxy for health the workflow cannot own. The old final await-condition(catchingUp) held the node's single adoption slot for the full catch-up (hours at real chain height) with no deadline anywhere, and the only exit from a wedged wait was the force-delete ceremony. Completing at release frees the slot in minutes, deletes the unbounded-wait failure mode outright, and separates execution (the workflow's job) from verification (the node's health surfaces).

Two properties this buys, both test-pinned:

• Failed ⟺ held: mark-ready is the terminal step and it is fire-and-forget (success == submitted), so a Failed workflow always means the node is still held. The old shape could fail after release, which made "Failed holds the node" conditional.
• The adoption slot frees at release, so drift work and queued workflows resume promptly.

Accepted costs, on the record

• WorkflowInProgress clears at release, so height-lag alerts can fire during legitimate catch-up. Traced against the platform alert rules: at most 1-2 auto-resolving warning tickets on protocol-low, no pages.
• Drift suppression lifts at release, so drift work (an image roll, a config reconcile) can interrupt catch-up at any point in the window. Verified against sei-chain: the giga startup safety checks key on SS version markers written only at the end of a snapshot restore, so an interrupted restore re-enters state-sync and converges. One sub-second wedge window exists between the sequential cosmos/evm marker writes at the restore tail; recovery is re-running the recipe.
• A queued workflow can adopt and re-wipe a node mid-catch-up: wasteful, never corrupting (the recipe quiesces before it wipes).
• Config drift cannot undo the migration flags: managed nodes have no config-drift detector, and the only running-node config surface (the update progression's p2p patch) never touches app.toml state-store keys.

A lost mark-ready self-heals from two directions: the sidecar persists the Running row before returning and rehydrates on restart, and the controller repair loop (sidecarNeedsReapproval, wired in every mode planner) re-issues mark-ready when the post-release health probe sees a 503.

Changes

• internal/planner/workflow.go: BuildPlan ends at mark-ready; no await-condition step.
• Planner + envtest + controller tests pin the terminal-step shape (envtest stall point moves to configure-state-sync; mark-ready is fire-and-forget, so GetTask-based stalls cannot hold it).
• api/v1alpha1 kind doc + sdk/sei StateSyncWorkflow doc state the release semantics.
• No CRD schema change (make manifests generate produced no config/ diff).

Review

xreview ledger: bdchatham-designs/designs/seinode-task/xreview/workflow-complete-at-release.md — unanimous over two rounds (idiomatic-reviewer, systems-engineer as assigned dissenter, kubernetes-specialist), all findings resolved and verified.

Companion seictl PR (handoff print, 15m default timeout, README) follows.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Changes operational meaning of workflow Complete and frees the adoption slot during catch-up, which can affect alerts, drift reconciliation, and queued workflows on production nodes.

Overview
The StateSync workflow no longer ends with await-condition(catchingUp). The compiled plan stops at mark-ready, so Complete means mutations finished and the node was released to re-bootstrap—not that CometBFT finished catching up. Callers and integration tests assert catch-up with node-side checks (e.g. SDK WaitCaughtUp).

Docs in the CRD kind comment and sdk/sei describe this release semantics and that Failed implies the node is still held when failure happens before the terminal step.

Planner, controller, envtest, and integration tests were updated: plans are one step shorter, lifecycle tests assert the last task is mark-ready with no await-condition, and envtest stalls moved to configure-state-sync because mark-ready cannot be held via the fake sidecar.

^{Reviewed by Cursor Bugbot for commit 6e88d31. Bugbot is set up for automated code reviews on this repo. Configure here.}