Skip to content

fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24535

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/fix-redis-setexz-broken-pipe-relanded
Draft

fix(ci3): make redis_setexz a no-op instead of crashing bootstrap_ec2 when redis is unavailable#24535
AztecBot wants to merge 1 commit into
nextfrom
cb/fix-redis-setexz-broken-pipe-relanded

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

The nightly barretenberg "Debug Build" (barretenberg-nightly-debug-build.yml./ci3.sh barretenberg-debug, run on the aztec-claude mirror of this ci3/ source) fails immediately with:

--- Run barretenberg-debug CI ---
gzip: stdout: Broken pipe
##[error]Process completed with exit code 1.

Latest failing run: https://github.com/AztecProtocol/aztec-claude/actions/runs/28731746984 — the step dies in ~120ms, before CI Log: is even printed and long before any EC2 builder is requested.

Root cause

ci.sh barretenberg-debug runs bootstrap_ec2, whose very first action logs a boot marker:

echo "CI booting..." | redis_setexz "$CI_LOG_ID" 300

redis_setexz was the only redis helper that did not guard on CI_REDIS_AVAILABLE:

function redis_setexz {
  gzip | redis_cli -x SETEX $1 $2 &>/dev/null
}

When redis is unreachable (CI_REDIS_AVAILABLE=0), redis_cli is deliberately a no-op that returns immediately without reading stdin. So gzip's downstream reader vanishes, gzip dies with EPIPE / "Broken pipe", and because ci3/source_options sets set -euo pipefail, that failed pipeline element aborts the whole script on its first line. source_redis explicitly treats redis as optional ("Log and test cache will be disabled"), so the script is meant to keep running when redis is down — redis_setexz violated that contract while its siblings redis_cli/redis_publish already guard on the flag.

Redis is unavailable on this path because the run has no BUILD_INSTANCE_SSH_KEY (empty on the mirror), so source_redis can't open the bastion tunnel to the remote cache, nc localhost:6379 fails, and CI=1 disables the local-docker-redis fallback.

Fix

Guard redis_setexz on CI_REDIS_AVAILABLE, matching the other helpers; when redis is unavailable, drain stdin (cat >/dev/null) so the upstream producer never gets SIGPIPE. All five callers (bootstrap_ec2, cache_log, denoise, run_test_cmd, and the pipe itself) feed the value via stdin, so draining is correct in every case.

Verification (red/green)

Reproduced the pipeline under set -euo pipefail with CI_REDIS_AVAILABLE=0:

  • before: echo ... | redis_setexz aborts on a broken pipe (exit 141); the script never reaches the next line.
  • after: stdin is drained, the pipeline returns 0, execution continues.

bash -n ci3/source_redis passes.

Why this keeps coming back — please merge, don't let it auto-close

This identical fix has already been made twice and both were auto-closed unmerged by the stale-draft reaper, not on technical grounds:

Because neither landed, the underlying bug is still in next and the nightly failed again. This is the third re-land off current next. Please mark ready and merge rather than letting it auto-close again.

Scope note

This removes the immediate ci3 crash whenever redis is unavailable (a real robustness bug for any such run). For the nightly to fully complete inside the aztec-claude mirror it additionally needs AWS credentials / BUILD_INSTANCE_SSH_KEY configured there to spin up the EC2 builder — a repo-secrets/infra matter separate from this code fix.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant