Support for FUSE passthrough

[martingms]

Tell us more about this new feature.

Hi, first of all thanks for all the work on this project!

We're evaluating using mountpoint-s3 on our ML training nodes. We have a lot more data than we have room for on a single node, but we often read the same data multiple times on the same node, so the cache feature could be very useful.

We are, as expected from a FUSE filesystem, seeing quite bad performance on reads from cached files, however.

It seems like implementing support for FUSE passthrough could help alleviate this somewhat, is this feasible/something you've evaluated for mountpoint-s3?

Similarly, it would be very nice to be able to passthrough writes, and only write to S3 on close(2), for example.

[tadiwa-aizen]

Hello, thank you for taking the time to inquire about this new feature!

We have not yet explored using FUSE Passthrough in depth. One complication is that it would require downloading entire files to serve as backing files, whereas our current cache implementation works in blocks.

However, regarding passthrough writes, we’d be interested to hear how it would benefit your use case?

Thank you

[martingms]

Hi @tadiwa-aizen, thanks for the quick reply!

I understand that a block-based cache makes passthrough complicated. We only ever read complete files, so for us I guess the block approach doesn't do much but I'm sure it's very useful for other use cases.

W.r.t. to writes, we have some workloads that continuously write their output to disk throughout their runtime since the full results are too big to cache in memory and dump at the end. In those cases fast writes are nice to have. This is easily solved by writing to some local fs and moving the files into the S3-backed fs after completion of course, so it's not that big a deal, but in an ideal world we'd avoid this little extra step.

[tadiwa-aizen]

Hi @martingms

Mountpoint uses multipart uploads and doesn't keep all data in RAM since it uploads to S3 incrementally.

Have you found this an issue for your use case?

[martingms]

No, I meant that some of our workloads writes to files continuously during their normal operation, and because of that having as fast writes to disk as possible is advantageous, which I assume a theoretical passthrough / backing_id based solution could help with. But as I indicated above, that is easily addressed in other ways. As low overhead as possible on reading cached files is much more important to us.