Virtual contexts: Multiple file systems in an isolated mount tree

[jeff-hykin]

Thanks for all the work on ZenFS! I'm hoping to use it as a platform for a lot of projects.

To partially continue the conversation from #124 , my big question is: what variables/functions/classes are stopping ZenFS from being able to load two different backends and maintain them as separate file system trees.

I understand the challenges with Nodejs, and not having a giant ZenFS class. If I attempt a fork ZenFS, I'd attempt to use a binding technique rather than a giant class. And, well, it seems like V_Context is doing a lot of what I would've attempted. So I'm working on understanding what is missing from V_Context that limits it to just one file system tree.

To be clear on expected behavior; I expect backends like NodeFS to only ever have one file tree. In-memory trees that exist along side a global file system is the question for this issue.

Trying to answer that myself:

Other than the mounts variable in src/vfs/shared.ts, I don't see much that would prevent ZenFS having two file system trees.

That said, I imagine it is more than just mounts. After drowning in the complexity of the config.ts mounting/unmounting, I figured I should just ask how big the iceberg is.

[james-pre]

Hi Jeff, thanks for your interest in the project.

The single tree of mounts came from architectural requirements. Specifically, for Node.js compatibility, ZenFS has named exports for most FS functions (e.g. import { statSync } from '@zenfs/core'). This necessitates using functions (rather than methods) and prevented any kind of internal state shared between functions.

Originally, different contexts were not supported— the functions all shared some semi-global state (including PWD, descriptors, and credentials). This also lead to much of the VFS using a functional approach, since this is cleaner and easy to maintain (especially considering the lack of internal state).

Contexts are, in essence, a creative way to share internal state among separate functions by binding an object to the function's this value. This means import { exampleSync } ... works correctly, event when the imported function has some state associated with it. If you read through the VFS code, you'll notice a significant amount of .call(this, ...)— this is how functions are able to pass the state around without breaking compatibility with the node:fs API.

But how does this relate to multiple mount trees?

A little function called resolveMount was added before contexts. It goes through the current mounts and finds the one responsible for a given path. When contexts were added, resolveMount was extended to work with context roots. This function uses a Map called mounts to track what file system is mounted where. The mounts map is shared among contexts, meaning they all share the same set of mounts (though the root can change, meaning some mounts may not be visible to some contexts).

Now that you've brought this up, it would be theoretically possible to store a mounts map on contexts. Then, one could create a new context tree with its own mounts. This would introduce more complexity into configuration though, and could have a performance impact.

However, it is important to consider that this might not be needed— contexts already provide a way to isolate parts of the FS tree. They've been designed as a way for the "host" code to create a security boundary within the existing tree, preventing code given the bound context from escaping.

For example, you could leverage this with Compartments by exposing only the bound fs of a context to a compartment. (note the ses package polyfills this)

Please let me know if you'd like more information or if you have any questions.

[jeff-hykin]

Thanks @james-pre for such a detailed response! Your note about compartments is really interesting, I'll be looking into that for some of my other projects.

I think multiple trees still has valid usecases (ex: two libraries internally using ZenFS, without the application dev even knowing they use ZenFS).

So I'll try making a fork and putting the mounts inside the context. That should help me get an idea of the complexity and possible problems, then I can report back. (Might take a few weeks)

I'll probably:

1. add an argument to bindContext() that let's it use a separate mounts map.
2. Have bindContext output a (bound) configure function that allows it to get connected to a different backend

[james-pre]

I think multiple trees still has valid usecases (ex: two libraries internally using ZenFS, without the application dev even knowing they use ZenFS).

Most programs work fine using a single tree. Perhaps it'd be better for these libraries to follow the established way of doing things (e.g. using /etc, /bin, and /lib). This allows libraries to be deduplicated and stuff. After all, one of the goals of this project is to emulate Linux's file system behavior.

I'm still unsure of whether this is a feature that would be worth integrating, especially given the minor perf impact and more importantly the maintainability impact.

Anyway, here are some tips for your implementation:

I'll probably:
add an argument to bindContext() that let's it use a separate mounts map.

I'd recommend you extend ContextInit, since that allows for future changes without breaking the API.

Have bindContext output a (bound) configure function that allows it to get connected to a different backend

This could introduce a ton of complexity. I'd recommend just adding a mount and umount function directly to the bound context (but not it's fs property). This is a lot less complicated, and for such advanced use cases, configure may not be as useful anyway.

Hopefully this has been helpful!

[jeff-hykin]

This could introduce a ton of complexity.

Good news!

• less than 20 lines of src/ changed
• no changes to the existing arguments (only additional optional attributes or args)
• all tests (including new isolation tests) are passing!
• no measurable perf impact on tests

Conceptually it pretty much was as simple as ctx?.mounts || mounts, which was surprising to me. Also yeah, after understanding the code base better I agree returning a bound configure function would have been a terrible design.

I'll refine/check it more before making a PR, but I'm starting to think it won't conflict with your goals after seeing how nicely it fit into ctx and configure.

I'd also like to offer what help I can too. I noticed some of the tests are failing on the latest main branch, some awaits were missing in front of some assert.reject's and rmdir non-empty test seems flakey (at least on the 2.0 release commit). If you think it'd be useful I can try to get the tests passing and maybe expand the existing test cases.

Perhaps it'd be better for these libraries to follow the established way of doing things (e.g. using /etc, /bin, and /lib)

Oh, I think there's a big gap in our perspectives! None of the libraries I had in mind use any of those directories. Hopefully I can communicate the view I have, as I think its not nearly as advanced/niche as a use case like chroot or permissions or even just mixed backends on a single tree.

Most programs work fine using a single tree

If you said "most packages work fine with a single tree" I would agree immediately. But one program can have hundreds or thousands of packages.

What if the host doesn't even realize ZenFS is in their dependency tree? I'd argue most npm projects have more indirect dependencies than direct dependencies.

Imagine a tree like:

• ZenFS (leaf)
• package Z (using ZenFS)
• my software (using Z, could be nodejs or in-memory)
• package A using my software (only uses in-memory case)
• package B using package A (hides the FS setup of pkg A, there's no longer a way to configure or pass-in a fs)
• package C using package B (ex: a web component that renders the tree)
• host user (using only package C)

Or maybe a better example would be this:

1. The host doesn't want to deal with file trees, they just want to give the component a url and have number-of lines number of commits, etc visualized
2. The host couldn't give a central file tree to both tools (webgit and websvn) because the react components don't expose an interface for doing it
3. WebGit and SebSVN don't know about each other, it doesn't make sense for them to try and share a file system

Making standalone tools like unzip or isomorphic git, or image converters, or esbuild-wasm doing client side bundling on CodePen, etc. they just need a single isolated folder, not a /lib and /etc or a complex sub-tree system.

And in terms of manual isolation, the workarounds like duplicate ZenFS or Compartments have major downsides. Like platforms that don't have a file system, like CodePen or Val.town, where an isolated ZenFS would be great, don't work with Compartments since ses either needs a build step or has major runtime cost.

[james-pre]

Jeff,

I just wanted to say thanks again for your interest in the project, It's great to see so much community involvement.

Just FYI the current state of main is not golden (though it should be) due to #210. It's been a huge amount of work for such a seemingly minor issue. I'm still working on fixing some bugs that have arisen from the work done on that issue. These are the bugs currently causing tests to fail. My current priority is getting main back to a stable state, then I can take a look at merging any PRs.

You argue some compelling use cases. My reasoning against having separate mount trees has been based on comparing web programs to native ones. The web is very different, and I understand that use cases may want a different environment.

If you want to open a PR, that's fine. My current thoughts is that contexts would have a new mounts: Map<...> property, which is automatically inherited. ContextInit can have an optional mounts property that overrides the inherited one (and explicitly does not merge with it), giving the newly bound context a completely independent mount tree. You will also likely want to modify the existing mount and umount functions to have this: V_Context if they don't already.

By the way, I noticed your comment on assert.rejects and quickly made the change myself— I'm surprised I missed this when writing tests, nice catch! I'll hopefully be done fixing main soon; I just finished fixing a couple of very sneaky upstream bugs that I literally spent over 50 hours trying to find.

[jeff-hykin]

All good!

I'll just wait till main is stable again. I'm not in a rush.

I've fought with SharedArrayBuffer a couple times so I feel your pain with #210.

By the way, I noticed your comment on assert.rejects and quickly made the change myself

Great!

Speaking of changes that are easier without a PR. Could you try changing #!/usr/bin/bash to #!/usr/bin/env bash (at the top of tests/fetch/run.sh)? If that replacemnt works on your end, it would help me out cause the current hashbang doesn't work for me on MacOS Big Sur.

My current thoughts is that contexts would have a new mounts: Map<...> property, which is automatically inherited. ContextInit can have an optional mounts property

Haha, that exactly what's on my fork! Seems like we're on the same page. There are a couple details like addDevice, but I'll save that discussion for the PR.

[james-pre]

Hey Jeff, main is now stable and v2.1.0 is out. Also, I updated the hashbang. Please let me know if you have any questions.

[kylecarbs]

@james-pre could you provide an example of how to do this? I've tried to follow this thread, but an example would be great.

[james-pre]

Hey @kylecarbs, apologies for the delayed response.

I think the addition to the API would look like this:

const ctx = bindContext({
    mounts: new Map([
        ['/', InMemory.create({})],
    ]),
});

ctx.fs.mkdirSync('/ex');
const example = await resolveMountConfig(...);
ctx.mount('/ex', example);

Please note that this would be for advanced use cases that aren't satisfied with existing APIs (including contexts).

[jeff-hykin]

Hey Jeff, main is now stable and v2.1.0 is out. Also, I updated the hashbang. Please let me know if you have any questions.

Dang I missed this notification! I'm glad there were a few more comments here. Hopefully next week I'll have a good PR for this.

[james-pre]

Hey @jeff-hykin!

I was thinking about this issue some more, I think this API would keep things clean and separate:

• Keep mounts only on FSContexts
• The existing "global" mounts can be moved to the defaultContext
• Each set of mounts has a different context tree, and also different context IDs
  • This creates a clear security boundary, so programs can't gain information about what other programs might exist.
• bindContext and BoundContext.bind only work within an existing context tree
• A new function, maybe something like createContextTree is the way to create a new tree of contexts (with different mounts)

I came up with this in about 15 minutes, criticism is welcome. Please let me know what you think about this, especially if you think it is over-engineered.

[gwsbhqt]

Hey @kylecarbs, apologies for the delayed response.

I think the addition to the API would look like this:

const ctx = bindContext({
mounts: new Map([
['/', InMemory.create({})],
]),
});

ctx.fs.mkdirSync('/ex');
const example = await resolveMountConfig(...);
ctx.mount('/ex', example);
Please note that this would be for advanced use cases that aren't satisfied with existing APIs (including contexts).

We need this feature, keep going!

[jeff-hykin]

@james-pre I was working on this today, made a lot of progress, but when I went to test there was a problem. Even on the main branch I get a lot of failing tests. Is this normal?