Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/styles/config/vocabularies/TraceMachina/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ API
APIs
Backpressure
Dedup
Deduplicate
XCode
CMake
Gradle
Expand Down
1 change: 1 addition & 0 deletions nativelink-config/src/backcompat.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ where
max_bytes_per_stream: old_config.max_bytes_per_stream,
persist_stream_on_disconnect_timeout_s: old_config
.persist_stream_on_disconnect_timeout_s,
experimental_write_dedup: false,
},
})
.collect();
Expand Down
21 changes: 21 additions & 0 deletions nativelink-config/src/cas_server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,27 @@ pub struct ByteStreamConfig {
alias = "persist_stream_on_disconnect_timeout"
)]
pub persist_stream_on_disconnect_timeout_s: usize,

/// Deduplicate concurrent uploads of the same digest ("join the flight").
/// When multiple clients upload a blob with the same digest at the same
/// time, only the first upload is streamed to the store; the others wait
/// for it to durably commit and then complete early without transferring
/// their payload, as permitted by the REAPI specification. Uploads of
/// blobs that already exist in the store also complete early after a
/// single existence check.
///
/// If the leading upload fails, waiting uploads receive a retryable
/// ABORTED error and one of the retrying clients becomes the new leader.
///
/// This saves upload bandwidth and store work when many actions produce
/// identical outputs, at the cost of one existence check per new upload.
/// The check uses the digest function in the upload resource name, and
/// proxy-backed `GrpcStore` instances participate in the same local
/// single-flight behavior.
///
/// Default: false (disabled)
#[serde(default, skip_serializing_if = "is_default")]
pub experimental_write_dedup: bool,
}

// Older bytestream config. All fields are as per the newer docs, but this requires
Expand Down
Loading
Loading