Option to disable in-flight compaction

[lblasc]

Hello,

First of all, I must say I really enjoy using ZeroFS, thank you!

I would like to back up my local machine daily (~400 GB of files, ranging from a few kilobytes up to 10 GB) as quickly as possible to a 9p mount on Hetzner Object Storage. My internet connection is limited to 500/200 Mbps, and I’m using rclone.

On the initial run, I noticed the following log message:

WARN slatedb::db: unflushed memtable size exceeds max_unflushed_bytes. applying backpressure ...

This suggests that uploads cannot keep up, which is expected; I can see that the upload bandwidth is fully saturated.

Another potential slowdown is in-flight compaction. When it is triggered, it consumes upload bandwidth and slows down the overall process. I've lowered lsm->max_concurrent_compactions to 1, which helps a bit, but it would be more efficient if compaction could be paused (possibly over RPC?) or disabled via configuration.

My ideal backup workflow would be:

• pause compaction
• create a backup of my local machine using rclone
• resume compaction and let it run while the backup script is not active

[Barre]

Hi,

This would be quite a large design change because we do need to compact to be able to make room in l0 (the first stage where data is first written).

Perhaps the following workflow could work for you:

• Start ZeroFS with the local filesystem backend (file:///path/to/your/directory)
• Mount ZeroFS and write your data.
• rclone sync the directory to the object store
• Update the ZeroFS config to use the remote object store
• Remove the local directory