fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24535
Draft
AztecBot wants to merge 1 commit into
Draft
fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24535AztecBot wants to merge 1 commit into
AztecBot wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly barretenberg "Debug Build" (
barretenberg-nightly-debug-build.yml→./ci3.sh barretenberg-debug, run on theaztec-claudemirror of thisci3/source) fails immediately with:Latest failing run: https://github.com/AztecProtocol/aztec-claude/actions/runs/28731746984 — the step dies in ~120ms, before
CI Log:is even printed and long before any EC2 builder is requested.Root cause
ci.sh barretenberg-debugrunsbootstrap_ec2, whose very first action logs a boot marker:redis_setexzwas the only redis helper that did not guard onCI_REDIS_AVAILABLE:When redis is unreachable (
CI_REDIS_AVAILABLE=0),redis_cliis deliberately a no-op that returns immediately without reading stdin. Sogzip's downstream reader vanishes,gzipdies withEPIPE/ "Broken pipe", and becauseci3/source_optionssetsset -euo pipefail, that failed pipeline element aborts the whole script on its first line.source_redisexplicitly treats redis as optional ("Log and test cache will be disabled"), so the script is meant to keep running when redis is down —redis_setexzviolated that contract while its siblingsredis_cli/redis_publishalready guard on the flag.Redis is unavailable on this path because the run has no
BUILD_INSTANCE_SSH_KEY(empty on the mirror), sosource_rediscan't open the bastion tunnel to the remote cache,nc localhost:6379fails, andCI=1disables the local-docker-redis fallback.Fix
Guard
redis_setexzonCI_REDIS_AVAILABLE, matching the other helpers; when redis is unavailable, drain stdin (cat >/dev/null) so the upstream producer never gets SIGPIPE. All five callers (bootstrap_ec2,cache_log,denoise,run_test_cmd, and the pipe itself) feed the value via stdin, so draining is correct in every case.Verification (red/green)
Reproduced the pipeline under
set -euo pipefailwithCI_REDIS_AVAILABLE=0:echo ... | redis_setexzaborts on a broken pipe (exit 141); the script never reaches the next line.bash -n ci3/source_redispasses.Why this keeps coming back — please merge, don't let it auto-close
This identical fix has already been made twice and both were auto-closed unmerged by the stale-draft reaper, not on technical grounds:
Because neither landed, the underlying bug is still in
nextand the nightly failed again. This is the third re-land off currentnext. Please mark ready and merge rather than letting it auto-close again.Scope note
This removes the immediate
ci3crash whenever redis is unavailable (a real robustness bug for any such run). For the nightly to fully complete inside theaztec-claudemirror it additionally needs AWS credentials /BUILD_INSTANCE_SSH_KEYconfigured there to spin up the EC2 builder — a repo-secrets/infra matter separate from this code fix.Created by claudebox · group:
slackbot