Quota enforcing sector allocator

[meroton-benjamin]

Moves disk quota enforcement to sector allocator, this allows the
file pool to have arbitrary large sparse files without running into
quota issues.

A consequence of this is that very large sparse files are now a first
class citizen of Buildbarn. We have therefore modified the sparse files
implementation to use a struct inspired by the unix inode pointer
struct.

[meroton-benjamin]

Latest version of an implementation of a quota enforcing sector allocator. There are some possible performance improvements that have not been pursued:

GetNextUnmappedSector when run in a dense region iterates through all sectors until it finds an unmapped one, this is not worse than what was done before but if run on e.g. a 16TB dense file from byte zero would require 2^32 iterations until it found an unmapped sector. With some bookkeeping this could probably be reduced.

GetLogicalSize keeps track of the last mapped sector of the pointer by manipulating an internal variable. This needs to be recalculated whenever truncated to smaller than the mapped sector. With a better implementation of GetNextUnmappedSector this might be possible to do faster by alternating between GetNextUnmappedSector and GetNextMappedSector until it reaches the last mapped sector less than what was truncated to.

[EdSchouten]

I think this code is a bit hard to read due to the excessive use of helper variables. What are your thoughts on getting rid of next and from? Can't we just write:

for {
    var list []uint32
    logical, list = sp.getNextDirectSectorList(logical)
    ...
}

[meroton-benjamin]

I don't really find this suggestion making things any clearer, I appreciate that the why of GetNextMappedSector is not very clear unless you know how the helper getNextDirectSectorList works. I have added some documentation to getNextDirectSectorList and reworked it somewhat to get around an overflow bug. Please have a look again if you still find it too unclear.

[meroton-benjamin]

With the modifications after the pull request GetLogicalSize could be removed. Tests that verify functionality around the boundary conditions the has been added (and the corresponding bug they discovered has been fixed).

[EdSchouten]

In #200 Jille and I added support for configuring the 'HoleSource' of a FilePool file. However, we didn't go as far that we changed BlockDeviceBackedFilePool to use an inode tree structure. It's still a list.

Benjamin, would you be interested in rebasing this change on top of what's in main right now? So:

• Change BlockDeviceBackedFilePool to use an inode tree structure.
• Change quota enforcement to account for sparseness.

[meroton-benjamin]

Sure I'll get on that

[aspect-workflows]

Test

3 test targets passed

Targets

//pkg/builder:builder_test [k8-fastbuild]            77ms
//pkg/filesystem/pool:pool_test [k8-fastbuild]       488ms
//pkg/filesystem/virtual:virtual_test [k8-fastbuild] 137ms

Total test execution time was 702ms. 16 tests (84.2%) were fully cached saving 3s.

[meroton-benjamin]

Rebased quota enforced sector allocator on top of what is in main and made the tests pass. As the rebase was pretty big it might be worth making another review pass at this.