Cross Bucket Copy Support

[tustvold]

Draft as needs further design, and a motivating use-case

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently ObjectStore only supports copying files within the current bucket, this is despite most stores having APIs for performing efficient copies across buckets within the same cloud provider.

Describe the solution you'd like

I would like some mechanism to support copying between ObjectStore, that is able to make use of the more efficient APIs where they are possible. I don't have a good idea of how to achieve this, nor a motivating use-case for this functionality.

Describe alternatives you've considered

Additional context

[Xuanwo]

AWS S3 allows buckets in the same region to utilize the CopyObject feature, though including a bucket name directly in the API may seem odd.

[tustvold]

Agreed, I don't have a good idea for how to expose this currently. One option might be to tie into the URL parsing stuff we already have, so the user would provide the source and destination as URLs that would then get processed by a free function

[Xuanwo]

One option might be to tie into the URL parsing stuff we already have, so the user would provide the source and destination as URLs that would then get processed by a free function

Seems cool to have copy("s3://bucket_a/file", "s3://bucket_b/file").

The downside is that users might need to manually construct the URL. We might also need to provide let url = store.get_url("file")?

[tustvold]

Seems cool to have copy("s3://bucket_a/file", "s3://bucket_b/file").

Yeah something like that, not entirely sure how credentials would work, but that's broadly where my head was at.

We might also need to provide let url = store.get_url("file")?

I think perhaps the expectation here is that the user is already outside the "single-bucket" model of ObjectStore, and so probably already are constructing these URLs somehow. FWIW a get_url method has been discussed before on #259, it's non-trivial 😅

[Joe-Blount]

I also would find it useful to have some form cross bucket copy support. We're using Arrow and the DynObjectStore type. For backup and D/R purposes, we need to selectively copy some objects to another bucket. Doing it server side without buffering the object is necessary to allow high concurrency without high memory usage. I'm imagining a copy API that takes another DynObjectStore that's either the source or destination bucket (with Self being the other). But anything that lets me do server-side copies between buckets would be greatly appreciated.

[tustvold]

Have you considered using S3's built-in replication functionality? See here. It is designed precisely for that use-case, and can be driven off combinations of object tags, paths, etc...

[Joe-Blount]

Thanks for the idea, yes we've looked at that. In our case, the primary bucket is very high churn, and we need the second bucket to be low churn, only containing a few selected objects. Tagging the selected objects (and using that tag to drive replication) could work, but then we'd give up control on the ordering they're copied in. So we might as well just copy them ourselves and retain control of the copy order.

[tustvold]

Fair, just thought I'd suggest it as it wouldn't end up waiting on changes to object_store.

I'm imagining a copy API that takes another DynObjectStore that's either the source or destination bucket (with Self being the other)

It's unclear to me how such an API would work, ObjectStore does not expose Any nor would it be especially useful if it did given people tend to wrap ObjectStore in multiple layers.

I suspect #116 and adding the ability to provide a source bucket is the way to go here, but I'm afraid I don't really have the capacity to help design or implement such an API. #121 is also potentially related.

[crepererum]

ObjectStore does not expose Any nor would it be especially useful if it did given people tend to wrap ObjectStore in multiple layers.

That could be changed though:

• support Any: either via as_any or -- for new rust versions -- via dyn up/downcasting
• layers: add a new method to the base trait fn inner(&self) -> Option<&dyn ObjectStore>.

then add a method to the base trait async fn copy_from(&self, other: Arc<dyn ObjectStore>, path: ...) (or copy_to if you like that more) that uses the dynamic inspection capabilities.

[tustvold]

I would very strongly prefer we avoid adding dynamic inspection capabilities or building functionality reliant upon it, they break encapsulation and they tend to be extremely cumbersome to work with and maintain, tending to result in fragile coupling between components that isn't statically verified. Given #116 is inline with the current patterns of this crate, and is IMO a much cleaner design, I would strongly prefer such an approach.