feat(storage): implement appendable upload connector by vsharonlynn · Pull Request #5956 · googleapis/google-cloud-rust

vsharonlynn · 2026-06-28T09:34:36Z

This PR comes after PR #5932.

gemini-code-assist

Code Review

This pull request introduces the appendable_object_writer module and supporting bidi_write infrastructure to enable bidirectional streaming writes for Google Cloud Storage. Key feedback includes avoiding .expect() on parameter conversions to prevent panics, simplifying complex string-folding logic, removing an unnecessary clone of the request object, and ensuring the background worker fails loudly with an explicit error if the stream closes unexpectedly while flushes are pending.

codecov · 2026-06-28T09:43:04Z

Codecov Report

❌ Patch coverage is 96.48562% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.75%. Comparing base (affbeb2) to head (c997071).

Files with missing lines	Patch %	Lines
src/storage/src/storage/bidi_write/connector.rs	97.24%	14 Missing ⚠️
src/storage/src/storage/bidi_write/mocks.rs	85.71%	4 Missing ⚠️
src/storage/src/storage/bidi_write/redirect.rs	95.50%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5956      +/-   ##
==========================================
+ Coverage   97.73%   97.75%   +0.01%     
==========================================
  Files         244      246       +2     
  Lines       61096    61703     +607     
==========================================
+ Hits        59710    60315     +605     
- Misses       1386     1388       +2

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joshuatants

We're gone through this extensively offline.

coryan

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

vsharonlynn · 2026-07-02T05:44:31Z

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

Ok. I am going to split this PR into smaller PRs. I'll update once done. Thank you and sorry about this!

vsharonlynn · 2026-07-08T06:23:45Z

This is still too large. Aim for at most 200 lines of new code, which probably means 400 to 600 lines per PR. 1600 lines is simply too large.

Ok. I am going to split this PR into smaller PRs. I'll update once done. Thank you and sorry about this!

I split this PR into three PRs:

This current PR is now for connector-related changes only, which includes some changes in redirect and mocks for testing.
feat(storage): implement appendable upload background worker #6001 for background worker. Still in draft, will wait for this PR to be merged before rebasing and sending it out.
feat(storage): implement appendable upload transport #6002 for transport. Still in draft, will wait for 6001 to be merged before rebasing and sending it out.

Thank you.

coryan · 2026-07-09T13:18:59Z


 #[allow(dead_code)]
-pub fn handle_redirect(spec: Arc<Mutex<AppendObjectSpec>>, status: Status) -> crate::Error {
+pub fn handle_redirect(state: Arc<Mutex<AppendObjectSpecState>>, status: Status) -> crate::Error {


Consider:

Suggested change

pub fn handle_redirect(state: Arc<Mutex<AppendObjectSpecState>>, status: Status) -> crate::Error {

pub fn handle_redirect(state: &mut AppendObjectSpecState, status: Status) -> crate::Error {

or make handle_redirect() a member function of AppendObjectSpecState:

impl AppendObjectSpecState { pub fn handle_redirect(&mut self, status: Status) -> crate::Error { }

Then you can use this (in tests) without an Arc<Mutex<>> and still use it in the production code as you want.

coryan · 2026-07-09T13:23:29Z

+            AppendObjectSpecState::Append(spec) => {
+                let mut new_spec = spec.clone();
+                new_spec.routing_token = redirect.routing_token;
+                new_spec.write_handle = redirect.write_handle;
+                if let Some(g) = redirect.generation {
+                    new_spec.generation = g;
+                }
+                AppendObjectSpecState::Append(new_spec)
+            }


I suspect if you receive a &mut self then this can be implemented using simpler assignments.

coryan · 2026-07-09T13:27:11Z

+
+        let new_state = match &*guard {
+            AppendObjectSpecState::Write(spec, _) => {
+                if let Some(generation) = redirect.generation {


The amount of copying and cloning here makes me sad... you should be able to do something like:

let new_spec = spec.resource.map(|r| /* move the data out of |r| */);

coryan · 2026-07-09T13:48:50Z

+#[derive(Clone, Debug)]
+#[allow(dead_code)]
+pub enum AppendObjectSpecState {
+    Write(Box<WriteObjectSpec>, Option<String>),


Why a Box<WriteObjectSpec> ?

coryan · 2026-07-09T13:49:41Z

+/// Represents a bidirectional streaming connection.
+/// Contains the transmission channel for requests and the receiving stream for responses.
+#[derive(Debug)]
+#[allow(dead_code)]


Instead of repeating this line in each element, put it at the place the module is imported.

coryan · 2026-07-09T13:53:13Z

+        let mut x_goog_request_params = format!("bucket={bucket_name}");
+        if let Some(token) = routing_token {
+            x_goog_request_params.push_str("&routing_token=");
+            x_goog_request_params.push_str(&token);


This needs to be percent-encoded, unless you happen to know that it never needs the percent encoding, in which can a comment justifying that would be useful.

coryan · 2026-07-09T13:54:21Z

+            x_goog_request_params.push_str(&token);
+        }
+
+        let (tx, rx) = tokio::sync::mpsc::channel::<BidiWriteObjectRequest>(100);


The magical 100 should be a configuration parameter or a constant. I apologize because I probably left a number of these in the code for open_object().

coryan · 2026-07-09T13:55:21Z

+                let new_state = match &*guard {
+                    AppendObjectSpecState::Write(s, _) => {
+                        AppendObjectSpecState::Append(AppendObjectSpec {
+                            bucket: s
+                                .resource
+                                .as_ref()
+                                .map(|r| r.bucket.clone())
+                                .unwrap_or_default(),
+                            object: s
+                                .resource
+                                .as_ref()
+                                .map(|r| r.name.clone())
+                                .unwrap_or_default(),
+                            generation: new_generation.unwrap_or(0),
+                            write_handle: m.write_handle.clone(),
+                            ..Default::default()
+                        })
+                    }
+                    AppendObjectSpecState::Append(s) => {
+                        AppendObjectSpecState::Append(AppendObjectSpec {
+                            generation: new_generation.unwrap_or(s.generation),
+                            write_handle: m.write_handle.clone().or_else(|| s.write_handle.clone()),
+                            ..s.clone()
+                        })
+                    }
+                };
+                *guard = new_state;


You should refactor this to a member function like I outline for handle_redirect()

coryan · 2026-07-09T13:57:02Z

+            let problem = SubstitutionFail::MismatchExpecting(
+                bucket_name.to_string(),
+                "projects/_/buckets/*",
+            );
+            let mismatch = SubstitutionMismatch {
+                field_name: "bucket",
+                problem,
+            };
+            let mismatch = PathMismatch {
+                subs: vec![mismatch],
+            };
+            let mismatch = BindingError {
+                paths: vec![mismatch],
+            };
+
+            return Err(crate::Error::binding(mismatch));


The connect_attempt() function is a bit long, consider refactoring things like this to helper functions.

product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Jun 28, 2026

gemini-code-assist Bot reviewed Jun 28, 2026

View reviewed changes

vsharonlynn mentioned this pull request Jun 28, 2026

feat(storage): implement API, request builders, integration tests #5957

Open

joshuatants reviewed Jun 29, 2026

View reviewed changes

Comment thread src/storage/src/storage/bidi_write/connector.rs Outdated

Comment thread src/storage/src/storage/bidi_write/redirect.rs Outdated

Comment thread src/storage/src/storage/bidi_write/transport.rs Outdated

vsharonlynn force-pushed the appendable_upload_3 branch 8 times, most recently from f3e8345 to 4c4b7be Compare June 29, 2026 11:57

vsharonlynn marked this pull request as ready for review June 29, 2026 12:02

vsharonlynn requested review from a team as code owners June 29, 2026 12:02

vsharonlynn force-pushed the appendable_upload_3 branch 2 times, most recently from 1713d73 to 9cca986 Compare June 29, 2026 17:13

joshuatants approved these changes Jun 30, 2026

View reviewed changes

coryan requested changes Jun 30, 2026

View reviewed changes

feat(storage): implement appendable upload connector

e86f327

vsharonlynn force-pushed the appendable_upload_3 branch from 9cca986 to e86f327 Compare July 8, 2026 02:58

Incorporate feedback

c997071

vsharonlynn changed the title ~~feat(storage): implement transport, connector, worker~~ feat(storage): implement appendable upload connector Jul 8, 2026

vsharonlynn mentioned this pull request Jul 8, 2026

feat(storage): implement appendable upload background worker #6001

Draft

vsharonlynn requested a review from coryan July 9, 2026 08:53

coryan requested changes Jul 9, 2026

View reviewed changes

	pub fn handle_redirect(state: Arc<Mutex<AppendObjectSpecState>>, status: Status) -> crate::Error {
	pub fn handle_redirect(state: &mut AppendObjectSpecState, status: Status) -> crate::Error {

Uh oh!

Conversation

vsharonlynn commented Jun 28, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joshuatants left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coryan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vsharonlynn commented Jul 2, 2026

Uh oh!

vsharonlynn commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Jun 28, 2026 •

edited

Loading