2026.7.19

fix: deferred copy tail double-counted in completed object plaintext size (#126)

The deferred copy tail was counted both in the client part that produced it and in the part that eventually stored it (next part or complete-time flush). This inflated total_plaintext_size by the tail size, so HEAD reported size+tail and rclone failed every >copy-cutoff copy with "corrupted on transfer: sizes differ".

• Stop adding the deferred tail to the producing part's plaintext total.
• Count the tail's plaintext on the part that flushes it at complete.
• Add a PART_PLAINTEXT_MISMATCH drift warning in _build_s3_parts.
• Regression tests for the 2-part defer and complete-time flush paths.

2026.7.18

Fix InvalidPart on 500+ part uploads (ClickHouse, Postgres barman, Scylla compressed streams) and noisy GetBucketVersioning 400s.

Problem: Sparse internal-part allocation maps client part N → internal (N-1)*20+1. Client part 501 needs internal 10001, exceeding S3's 10k part limit. Affected ClickHouse shadow tars, Postgres barman base backups (5MB parts), and Scylla compressed stream uploads.

Fix (#125):

• Dense single-internal mode — client part N → internal N while every client part uses exactly one internal part (typical 5MB uploads)
• Scylla SST path preserved — first multi-internal client part clears dense mode and reverts to sparse ranges (out-of-order safe)
• Legacy Redis state — infer dense_single_internal=false from existing sparse parts so in-flight uploads don't switch modes mid-stream
• GetBucketVersioning — answer locally with Suspended (Hetzner backend rejects the probe with 400)
• Diagnostic logs: allocation_mode, HIGH_CLIENT_PART_ALLOCATED, LEGACY_DENSE_MODE_INFERRED

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.18

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.18

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'allocation_mode=dense|HIGH_CLIENT_PART_ALLOCATED|INTERNAL_PART_ALLOCATION_REJECTED'

Expect allocation_mode=dense and internal_start=501 on part 501. Zero new INTERNAL_PART_ALLOCATION_REJECTED on fresh upload IDs.

Note: Uploads already past part 500 under sparse numbering cannot be salvaged — clients must abort and retry with a fresh upload ID.

2026.7.17

Fix EntityTooSmall on Scylla manifest dedup copies after hybrid passthrough (2026.7.16).

Problem: After 2026.7.16, part 1 hybrid passthrough worked (~572 segments) but the ~1.08 MB mid-frame tail was uploaded as internal part 573. Part 2 then added parts 574+, so S3 rejected CompleteMultipartUpload with EntityTooSmall (part 573 < 5 MB and not last). Dedup bytes counted, then the same bytes retried as Failed.

Fix (#124):

• Defer hybrid tail when part 1 is large (≥ 1 GiB) and more client parts follow — store sub-5 MB plaintext suffix in upload state instead of uploading it on part 1
• Part 2 prepends deferred tail into streaming re-encrypt (_PlaintextReader prefix)
• Complete flushes any leftover deferred tail
• Debug logs: HYBRID_TAIL_DEFER_DECISION, UPLOAD_PART_COPY_PASSTHROUGH_TAIL_DEFERRED, UPLOAD_PART_COPY_CONSUME_DEFERRED_TAIL, COMPLETE_MULTIPART_SUCCESS

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.17

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.17

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'HYBRID_TAIL_DEFER_DECISION defer=true|PASSTHROUGH_TAIL_DEFERRED|CONSUME_DEFERRED_TAIL|COMPLETE_MULTIPART_SUCCESS|ENTITY_TOO_SMALL'

Expect defer=true → TAIL_DEFERRED → CONSUME_DEFERRED_TAIL → COMPLETE_MULTIPART_SUCCESS on fresh .sm_ manifest upload IDs. Zero new ENTITY_TOO_SMALL on new uploads.

2026.7.16

Fix Scylla manifest UploadPartCopy timeouts, zombie copies, and two latent CI integration bugs.

Problem: Live-cluster analysis showed failing manifest copies are UploadPartCopy calls that send zero response bytes until the copy completes. rclone (scylla-manager-agent) hangs up after exactly 300s idle — a 4.7GB part took ~400s streaming, so every attempt was doomed. The proxy kept abandoned copies running as zombies (1,953 streaming copies started vs 232 completed in 9h, plus 18k MEMORY_BACKPRESSURE events from the retry storm).

Fix (#123):

• Keepalive streaming: UploadPartCopy now commits to 200 OK and trickles whitespace (default every 5s, S3PROXY_COPY_KEEPALIVE_INTERVAL) while the copy runs — the AWS S3 long-copy pattern — then emits CopyPartResult or an <Error> document as the body
• Client disconnect cancels work — no more zombie copies burning the memory budget after the client gives up
• Parallel passthrough segments (bounded, default 8, S3PROXY_PASSTHROUGH_SEGMENT_CONCURRENCY) instead of ~570 sequential backend calls
• MD5 pass overlapped with segment copies via TaskGroup instead of serially re-reading the whole 4.7GB source afterwards
• Hybrid passthrough + tail when Scylla manifest range ends mid internal frame
• CompleteMultipartUpload now forwards stored backend part etags (fixes InvalidPart on single-segment passthrough complete)
• Error handler drains small unread request bodies on keep-alive connections (fixes Abort after rejected UploadPart)
• Chart default memoryLimitMb raised to 128; backpressure timeout 120s; Redis WATCH retries hardened

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.16

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.16

Post-deploy check:

kubectl logs -n s3proxy-python -l app.kubernetes.io/name=s3proxy-python --since=30m \
  | rg 'UPLOAD_PART_COPY_KEEPALIVE|UPLOAD_PART_COPY_PASSTHROUGH_COMPLETE|UPLOAD_PART_COPY_CLIENT_GONE|FAILED_AFTER_200'

Expect PASSTHROUGH_COMPLETE with elapsed_sec well under 300s and CLIENT_GONE to disappear.

2026.7.15

Fix Scylla backup manifest UploadPartCopy still streaming after 2026.7.14.

Problem: PR #121 only normalized ranges matching sidecar total_plaintext_size. Scylla Manager sends bytes=0-(~4768MB-1) on .sm_* manifests while source big-Data.db metadata totals ~6GB. Copies still routed to UPLOAD_PART_COPY_STREAMING, queued behind the 2-slot pipeline, and timed out at ~300s → HAProxy 400.

Fix (#122):

• Range-aware passthrough for large copies from offset 0 when the range aligns to internal ciphertext segments
• Server-side upload_part_copy per segment (no decrypt/re-encrypt) instead of streaming
• UPLOAD_PART_COPY_ROUTE log with route, range sizes, and passthrough_blocked_reason for diagnosis

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.15

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.15

Post-deploy: Scylla manifest uploads (*.sm_*) should log UPLOAD_PART_COPY_ROUTE route=passthrough and UPLOAD_PART_COPY_PASSTHROUGH. Upgrade from 2026.7.14 if backups still show 400 @ ~300s on manifest copies.

2026.7.14

Fix Scylla backup manifest UploadPartCopy 400s at ~300s.

Problem: Scylla Manager sends UploadPartCopy with x-amz-copy-source-range: bytes=0-(size-1) on .sm_* manifest parts referencing already-uploaded encrypted SSTs (big-Data.db). s3proxy treated the full-object range as a partial copy, forcing UPLOAD_PART_COPY_STREAMING (decrypt/re-encrypt). Requests queued behind the 2-slot copy pipeline for many minutes while Scylla Manager timed out at ~300s → HAProxy 400.

Fix (#121):

• Normalize full-object CopySourceRange to whole-object copy → UPLOAD_PART_COPY_PASSTHROUGH (server-side ciphertext copy, no re-encrypt)
• Remove streaming pipeline semaphore from passthrough path so manifest copies are not blocked behind slow re-encrypt jobs
• Partial ranges still use streaming re-encrypt as before

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.14

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.14

Post-deploy: Scylla backup manifest uploads (*.sm_*) should log UPLOAD_PART_COPY_PASSTHROUGH and return 200 instead of 400 @ ~300s. Upgrade from 2026.7.13 if Scylla backups are failing on large manifest copies.

2026.7.13

Hotfix for Scylla Manager backup 403s after #116.

Problem: Scylla Manager uploads SSTs via presigned PUT URLs (X-Amz-Signature in query). #116 treated missing x-amz-content-sha256 as deferred header-auth, routing presigned requests through prepare_header_auth → 403 Forbidden.

Fix:

• Skip body preload and deferred signature when query contains X-Amz-Signature (or legacy Signature)
• Presigned large PUTs verify immediately via verifier.verify() and stream without double-buffering

Also in this release:

• E2E tests: generate_presigned_url SigV4 + legacy V2 through full proxy stack
• CI fix: isolated MinIO for heavy copy tests (disk exhaustion)

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.13

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.13

Post-deploy: Scylla backup PUTs should return 200. Upgrade from 2026.7.12 if backups are failing with 403.

2026.7.12

Deferred SigV4 for large Scylla SST PUTs — fixes s3proxy OOM during backups (#116).

Problem: Scylla Manager uploads ~179MB SSTables with SigV4 but without x-amz-content-sha256. s3proxy preloaded the full body for signature verification, then streamed it again for encryption — ~2× body size in memory per request. With concurrent backup nodes in a 1Gi pod, RSS hit ~965MB → OOMKilled (exit 137).

Fix:

• Deferred SigV4 for large PUTs (>8MB) missing x-amz-content-sha256: verify credentials up front, hash body once while streaming encrypt/upload, verify signature before complete
• Single-pass body iterator — no double-buffer from request.body() + request.stream()
• Stop polling is_disconnected() during body streaming (Starlette can drop bytes and corrupt SHA256)
• Dockerfile uses python -m s3proxy.main so TLS settings from Settings apply in prod

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.12

Helm: oci://ghcr.io/serversidehannes/charts/s3proxy-python version 2026.7.12

Post-deploy check: logs should show signature_deferred, not body_loaded body_size=187836908.

2026.7.11

UploadPartCopy passthrough — fixes s3proxy OOM during Scylla backup dedup (#115).

Problem: Dedup copies use UploadPartCopy on encrypted SSTables. The re-encrypt path downloaded, decrypted, and re-encrypted each ~1280MB object (~88MB peak per internal part × concurrent copies). With 13 agents and maxConnPerPod: 20, pods OOMKilled (exit 137) and backups showed Failed ≈ Deduplicated.

Fix:

• Passthrough for eligible copies: encrypted multipart sidecar source, full-object copy (no CopySourceRange), same credential — native server-side upload_part_copy of existing ciphertext with source DEK adoption (no decrypt/re-encrypt amplification)
• Hold governor reservation through upload_part completion (release after ciphertext is handed off)
• Default S3PROXY_MAX_PARALLEL_COPIES=2 pipeline semaphore per pod

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.11

Verify after rollout:

# During Scylla backup dedup: Failed should stop tracking Deduplicated
kubectl exec -n scylla-manager deploy/scylla-manager -- sctool progress <task-id> -c searchengine

# s3proxy OOMKills should drop; passthrough log lines appear
kubectl -n s3proxy-python logs deploy/s3proxy-python | grep UPLOAD_PART_COPY_PASSTHROUGH

Pre-merge gate:

make verify-passthrough

No config changes required. Encryption at rest unchanged — ciphertext is byte-copied, not decrypted to disk.

2026.7.10

Per-part copy memory reservation — fixes sustained 176MB governor hold during Scylla backup (#114).

Problem: #113 fixed O(1) peak sizing (~88MB per copy) but still reserved that amount for the entire copy duration (40 internal parts × ~24s for a 1280MB SSTable). Two concurrent copies held ~176MB continuously (active_mb=176.38), blocking a third and causing MEMORY_BACKPRESSURE even while INTERNAL_PART_UPLOADED events kept firing.

Fix: Streaming UploadPartCopy now acquires/releases governor memory per 32MB internal part (~88MB peak during encrypt only). Memory is free during S3 upload I/O between parts, so concurrent copies interleave within the 192MB budget.

Also: legacy encrypted objects without isec-kid now pass through byte-identically on COPY instead of failing re-encrypt.

Deploy: ghcr.io/serversidehannes/s3proxy-python:2026.7.10

Verify after rollout:

# Governor sawtooth during backup (not flat ~176MB)
kubectl -n s3proxy-python port-forward deploy/s3proxy-python 9090:9090 &
watch -n1 'curl -s localhost:9090/metrics | grep s3proxy_memory_reserved_bytes'

Or run pre-merge gate locally:

make verify-copy-memory

No config changes required. Keep S3PROXY_MEMORY_LIMIT_MB=192.