Memory amplification

[ancapdev]

Hi,

This is more of question, and maybe issues isn't the best forum.

From your README:

For a GET request, Julia provides a buffer for the native library to write into. This requires Julia to know a suitable size before-hand and requires the native library to do an extra memory copy, but the upside is that Julia controls the lifetime of the memory.

When I first found this package I was excited that this may provide a route to not just higher throughput downloads, but also reduced memory amplification. The context is a use case running large jobs in AWS Batch, where I download a working set from S3 at the start of each job, 100s of GBs, and I need to scale my machines according to their peak memory usage. Individual objects are single digits GBs, and I download them concurrently to saturate the machine's bandwidth as much as I can. When I saw the API here providing a storage buffer with my S3 get, I thought finally, a zero copy API without data duplication, but reading the docs it would seem like the Rust code would write the entire object to an internal buffer and then copy it?

Appreciate any clarity and would be happy to contribute if possible.

[andrebsguedes]

Hey Christian, thank you for the interest in the library!

The copy we refer in the README is just from the 16KB TLS buffers to the Julia provided buffer so there is no extra allocation of the entire object size and as those buffers are small enough to live in the cache the cost of this copy is negligible (based on internal benchmarking)

Sorry for the long delay on the response, didn't catch the notification