Disable backend payload signing; verify ciphertext MD5 against backend ETag#129
Open
ServerSideHannes wants to merge 1 commit into
Open
Disable backend payload signing; verify ciphertext MD5 against backend ETag#129ServerSideHannes wants to merge 1 commit into
ServerSideHannes wants to merge 1 commit into
Conversation
…backend ETag Backend SigV4 payload signing forced botocore to hold a second full copy of every uploaded internal part. Switch the backend hop to UNSIGNED-PAYLOAD and flexible checksums to when_required, and replace the lost integrity check with an incremental ciphertext-MD5 vs backend-ETag comparison on every upload_part and put_object (BackendIntegrityError -> 500 InternalError -> client retries). Non-MD5 ETags (SSE-KMS backends) skip verification with a one-time warning. Closes #127
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #127
What
UNSIGNED-PAYLOAD(payload_signing_enabled: False) and only computes flexible checksumswhen_required. SigV4 payload hashing forced botocore to hold/read a second full copy of every uploaded internal part — roughly half of the multi-frame upload peak documented instreaming_upload_peak().upload_part/put_objectcompares the backend's returned ETag against the MD5 of the ciphertext we sent (incremental for streaming bodies, direct for bytes). Mismatch raisesBackendIntegrityError→ 500InternalError→ the client (rclone/barman) retries the part. This is the no-regression condition from Backend payload signing doubles per-request upload memory — switch to UNSIGNED-PAYLOAD #127.BACKEND_ETAG_NOT_MD5warning; our RGW backend returns plain MD5 ETags.Side effect: local
~/.aws/configcan no longer flip payload signing behavior (the prior local-vs-CI divergence), since it's now pinned in the client config.Tests
tests/unit/test_backend_etag_verification.py: config pins (signing off, checksums when_required), match/mismatch/skip paths for both operations, streaming-body duck-typed verification, and the mapping to a retryable 500.Integration suites exercise the unsigned-payload path against real MinIO end-to-end — any ETag mismatch now fails loudly there.
Backup impact & rollout
None expected: same requests, same retry behavior, same data format. Deploy between backup runs; rollback is a plain image revert. Ship before #128 (frame-streaming bodies build on the unsigned-payload hop).