commitlog,durability: Support preallocation of disk space #3437
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a new commitlog segment is created, allocate disk space for it up
to the maximum segment size. Also do this when resuming writes to an
existing segment, such that segments created without preallocation will
allocate as well when the database is opened.
Preallocation is gated behind the feature "fallocate", because it is not
always desirable to preallocate, e.g. for local
standaloneusers.The feature can only be enabled on Linux targets, because allocation is
done using the Linux-specific
fallocate(2)system call.Unlike
ftruncate(2)or the portableposix_fallocate(3),fallocate(2)supports allocating disk space without zeroing. This is currently
required, because the commitlog format does not handle padding bytes.
If not enough space can be allocated, the commitlog refuses writes. For
commitlogs that were created without preallocation, this means that the
commitlog cannot even be opened in this situation.
The local durability impl will crash if it detects that the commitlog is
unable to allocate enough space.
This means that a database will eventually crash and be unable to start in
an out-of-space situation.
Allocated space is not included in the reported size of the commitlog.
Instead, allocated blocks are reported separately.
Expected complexity level and risk
3 - Disk size monitoring may need to be adjusted.
Testing
spacetimedb_durability::Local]when there is insufficient space. The test performs I/O against a loop device.
repo::Memoryimpl so that it can run out of space. No test currentlyutilizes this, but existing tests assuming infinite space still pass.