Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support part upload with x-amz-copy-source #98

Open
JustinKyleJames opened this issue Mar 1, 2024 · 0 comments
Open

Support part upload with x-amz-copy-source #98

JustinKyleJames opened this issue Mar 1, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@JustinKyleJames
Copy link
Contributor

JustinKyleJames commented Mar 1, 2024

When a large file is copied, the aws client does a CreateMultipartUpload followed by PUT's which have the following:

  • partNumber
  • x-amz-copy-source
  • x-amz-copy-source-range

Here is an example:

s3-api_1  | PUT
s3-api_1  | /alice-bucket/20Mfile2
s3-api_1  | partNumber=2&uploadId=19202868-e31f-4e1c-83cb-994f76a5b015
s3-api_1  | host:s3-api:8080 
s3-api_1  | x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
s3-api_1  | x-amz-copy-source:alice-bucket/20Mfile
s3-api_1  | x-amz-copy-source-range:bytes=8388608-16777215

This is currently not supported. The current code will simply call the copy operation in iRODS after detecting the x-amz-copy-source header.

We will need an enhancement to support this.

As a workaround, multipart should be disabled for any copy operation. This is preferred anyway as it is far more efficient to call the iRODS copy operation than to read the bytes for each part, save a part file, and then stream it all back to iRODS on CompleteMultipartUpload.

As a workaround for the test cases, we will disable multipart for the copy tests.

@JustinKyleJames JustinKyleJames self-assigned this Mar 1, 2024
@JustinKyleJames JustinKyleJames added the enhancement New feature or request label Mar 1, 2024
@alanking alanking added this to the 0.3.0 milestone Mar 1, 2024
@trel trel modified the milestones: 0.3.0, 0.4.0 Aug 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants