BlobContainer: add copyBlob method (8.x) #126558
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
It's possible that the client config, particularly its credentials, might change in the middle of a long-running operation such as a large multipart upload. Prior to this commit we would hold onto the same `AmazonS3` instance for the entire operation, but really there's no need to do so, we can obtain a potentially-fresher instance for each API call. (cherry picked from commit 09d4217)
If a container implements copyBlob, then the copy is performed by the store, without client-side IO. If the store does not provide a copy operation then the default implementation throws UnsupportedOperationException. This change provides implementations for the FS and S3 blob containers. More will follow. Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: David Turner <[email protected]> (cherry picked from commit c1a71ff)
bcully
added
>non-issue
v8.19.0
:Distributed Coordination/Distributed
elasticsearchmachine
added
the
Team:Distributed Coordination
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
|
If we do take it, I suppose we'll have to forward port the transport versions change to main first. |
- Translate a 404 during a multipart copy into a `FileNotFoundException` - Use multiple threads in `S3HttpHandler` to avoid `CopyObject`/`PutObject` deadlock Closes elastic#126576 (cherry picked from commit b10b35f)
|
I suppose we'd want to backport to 9.0 as well if we did this? I kind of forgot about the current state of backport branches. |
|
9.0 will live for much less time than 8.18 (and won't receive the SDKv2 upgrade) so I wouldn't expect to backport this there. However having played around with these changes I think it will work out ok to backport the SDKv2 upgrade without |
|
Doesn't seem worth backporting at this time. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a backport of #125737 to 8.x.
The copyBlob feature isn't expected to have any users in 8.x, but this may make it easier to backport the AWS SDKv2 changes that are incoming. Additionally it includes some coverage of copyBlob in the repository analyzer which may let it provide feedback on the suitability of the object store for 9.x before upgrade (though it may also trigger new warnings against an existing blob store that had previously passed, for users that upgrade to 8.19).
I'm not personally sure this backport is worthwhile given the amount of dead code it will bring along, but making that decision might be easier with something concrete to look at.