Conversation
|
Review requested:
|
nodejs-github-bot
added
lib / src
mcollina
requested review from
Ethan-Arrowood,
RaisinTen,
avivkeller and
joyeecheung
avivkeller
added
fs
|
The
notable-change
Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
|
Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week. |
| */ | ||
| existsSync(path) { | ||
| // Prepend prefix to path for VFS lookup | ||
| const fullPath = this.#prefix + (StringPrototypeStartsWith(path, '/') ? path : '/' + path); |
| validateObject(files, 'options.files'); | ||
| } | ||
|
|
||
| const { VirtualFileSystem } = require('internal/vfs/virtual_fs'); |
There was a problem hiding this comment.
Shouldn't we import this at the top level / lazy load it at the top level?
| ArrayPrototypePush(this.#mocks, { | ||
| __proto__: null, | ||
| ctx, | ||
| restore: restoreFS, |
There was a problem hiding this comment.
| restore: restoreFS, | |
| restore: ctx.restore, |
nit
lib/internal/vfs/entries.js
Outdated
| * @param {object} [options] Optional configuration | ||
| */ | ||
| addFile(name, content, options) { | ||
| const path = this._directory.path + '/' + name; |
lib/internal/vfs/virtual_fs.js
Outdated
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const dirPath = '/' + segments.slice(0, i + 1).join('/'); |
lib/internal/vfs/virtual_fs.js
Outdated
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const parentPath = '/' + segments.slice(0, i + 1).join('/'); |
lib/internal/vfs/virtual_fs.js
Outdated
| } | ||
| } | ||
| callback(null, content); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
lib/internal/vfs/virtual_fs.js
Outdated
| const bytesToRead = Math.min(length, available); | ||
| content.copy(buffer, offset, readPos, readPos + bytesToRead); |
lib/internal/vfs/virtual_fs.js
Outdated
| } | ||
|
|
||
| callback(null, bytesToRead, buffer); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
| }).catch((err) => { | |
| }, (err) => { |
|
Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look |
|
It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #61478 +/- ##
==========================================
- Coverage 91.64% 89.65% -2.00%
==========================================
Files 337 691 +354
Lines 140455 213208 +72753
Branches 21779 40576 +18797
==========================================
+ Hits 128716 191144 +62428
- Misses 11517 14175 +2658
- Partials 222 7889 +7667
🚀 New features to boost your workflow:
|
|
And why not something like OPFS aka whatwg/fs? const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')OR const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.readFileSync('sandbox:/config.json')fs.createVirtual seems like something like a competing specification |
mcollina
force-pushed
the
vfs
branch
3 times, most recently
from
5e317de to
977cc3d
Compare
I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.) On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway. If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR. |
|
Small prior art: https://github.com/juliangruber/subfs |
mcollina
force-pushed
the
vfs
branch
2 times, most recently
from
8d711c1 to
73c18cd
Compare
|
I also worked on this a bit on the side recently: Qard@73b8fc6 That is very much in chaotic ideation stage with a bunch of LLM assistance to try some different ideas, but the broader concept I was aiming for was to have a module.exports = new VirtualFileSystem(new LocalProvider())I intended for it to be extensible for a bunch of different interesting scenarios, so there's also an S3 provider and a zip file provider there, mainly just to validate that the model can be applied to other varieties of storage systems effectively. Keep in mind, like I said, the current state is very much just ideation in a branch I pushed up just now to share, but I think there are concepts for extensibility in there that we could consider to enable a whole ecosystem of flexible storage providers. 🙂 Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system. |
just a bit off topic... but this reminds me of why i created this feature request: Would not lie, it would be cool if NodeJS also provided some type of static example that would only work in NodeJS (based on how it works internally) const size = 26
const blobPart = BlobFrom({
size,
stream (start, end) {
// can either be sync or async (that resolves to a ReadableStream)
// return new Response('abcdefghijklmnopqrstuvwxyz'.slice(start, end)).body
// return new Blob(['abcdefghijklmnopqrstuvwxyz'.slice(start, end)]).stream()
return fetch('https://httpbin.dev/range/' + size, {
headers: {
range: `bytes=${start}-${end - 1}`
}
}).then(r => r.body)
}
})
blobPart.text().then(text => {
console.log('a-z', text)
})
blobPart.slice(-3).text().then(text => {
console.log('x-z', text)
})
const a = blobPart.slice(0, 6)
a.text().then(text => {
console.log('a-f', text)
})
const b = a.slice(2, 4)
b.text().then(text => {
console.log('c-d', text)
})An actual working PoC(I would not rely on this unless it became officially supported by nodejs core - this is a hack) const blob = new Blob()
const symbols = Object.getOwnPropertySymbols(blob)
const blobSymbol = symbols.map(s => [s.description, s])
const symbolMap = Object.fromEntries(blobSymbol)
const {
kHandle,
kLength,
} = symbolMap
function BlobFrom ({ size, stream }) {
const blob = new Blob()
if (size === 0) return blob
blob[kLength] = size
blob[kHandle] = {
span: [0, size],
getReader () {
const [start, end] = this.span
if (start === end) {
return { pull: cb => cb(0) }
}
let reader
return {
async pull (cb) {
reader ??= (await stream(start, end)).getReader()
const {done, value} = await reader.read()
cb(done ^ 1, value)
}
}
},
slice (start, end) {
const [baseStart] = this.span
return {
span: [baseStart + start, baseStart + end],
getReader: this.getReader,
slice: this.slice,
}
}
}
return blob
}currently problematic to do: also need to handle properly clone, serialize & deserialize, if this where to be sent of to another worker - then i would transfer a MessageChannel where the worker thread asks main frame to hand back a transferable ReadableStream when it needs to read something. but there are probably better ways to handle this internally in core with piping data directly to and from different destinations without having to touch the js runtime? - if only getReader could return the reader directly instead of needing to read from the ReadableStream using js? |
Add loaderStat(), loaderReadFile(), and setLoaderFsOverrides() to helpers.js, and modify toRealPath() to support a VFS toggle. Replace direct internalFsBinding.internalModuleStat() and fs.readFileSync() calls in the CJS loader, ESM resolver, ESM loader, translators, and package_json_reader with these wrappers. The VFS module_hooks.js now calls setLoaderFsOverrides() first in installHooks(), making loader fs interception order-independent and eliminating conflicts with cached fs method references. Fix two pre-existing bugs in esm/resolve.js finalizeResolution(): - StringPrototypeEndsWith() was called with internalFsBinding as first arg instead of path - StringPrototypeSlice(path, -1) returned the last char instead of stripping the trailing slash (now correctly uses path, 0, -1) Existing fs patches for user-facing operations are kept unchanged.
|
@joyeecheung I added back the toggles for loading modules. PTAL. |
|
@avivkeller PTAL |
- Remove redundant #getBaseName/#getParentPath from MemoryProvider, use pathPosix.basename/dirname directly - Remove redundant getBaseName/getParentPath/splitPath from router.js, keep only functions with non-trivial VFS-specific logic - Convert RealFSProvider constant getters (readonly, supportsSymlinks) to readonly properties via ObjectDefineProperty - Fix joinVFSPath/normalizeVFSPath to detect Windows drive-letter paths by checking for ':' at position 1 instead of checking for leading '/', so bare '/' is always treated as a POSIX VFS path - Update test-vfs-internals.js to match router.js export changes
Use path.resolve() instead of pathPosix.normalize() for VFS path normalization so mount points resolve correctly on Windows (e.g. /virtual -> C:\virtual). Use path.sep and path.relative() in router for cross-platform mount point matching. Remove normalizeVFSPath and joinVFSPath wrappers in favor of direct path utility calls. Update tests to use path.resolve()/path.normalize() for platform-portable assertions. Co-Authored-By: Claude Opus 4.6 <[email protected]>
Remove blank lines between JSDoc description and @param tags (jsdoc/tag-lines) and remove unused `common` variable assignment (no-unused-vars).
|
I'm not sure if you're the right person to tag, but this may be of interest to the Electron team and worth some feedback on for potential Electron use cases. Off the top of my head for example, it may be a potential replacement for electron/asar or have some other relevant use cases? |
I think one thing that this cannot replace asar for is the ability to pack/unpack an archive based on what's in the VFS, which is useful especially in reproducibility. But the current design doesn't seem to be in conflict with implementing that later in a follow up. |
| @@ -27,7 +27,7 @@ const { kEmptyObject } = require('internal/util'); | |||
| const modulesBinding = internalBinding('modules'); | |||
There was a problem hiding this comment.
modulesBinding methods used below should still reroute to VFS, they are currently reading into the real FS via c++. Not patching them means syntax detection and error decoration that are re-loading the package.json are going to read into real FS for non-existent/different files and see mismatches - they can probably use some tests too.
Qard
left a comment
Qard
left a comment
There was a problem hiding this comment.
Mostly LGTM, but I worry the vfs-mount event on process may be insufficient as a security control. I think we'll probably want a proper permission system control for that which can't be tampered with.
doc/api/vfs.md
Outdated
| * Only specific targeted files are affected. | ||
| * Other operations appear to work normally. | ||
|
|
||
| ### Monitoring VFS mounts |
There was a problem hiding this comment.
Seems handy, but is there a reason for going for this rather than a stricter permission system block? Could also do both, but I think lots of users would want to be able to just hard-crash with a permission error if anything tries to mount unexpectedly. Process event handlers can be tampered with to remove the listener, so this alone is not so much of a security measure. 🤔
Wrap modulesBinding.readPackageJSON(), getNearestParentPackageJSON(), getPackageScopeConfig(), and getPackageType() with toggleable overrides so that VFS-mounted package.json files are read from virtual storage instead of the real filesystem. This fixes syntax detection and error decoration that re-read package.json bypassing VFS. The implementation follows the existing loaderStat/loaderReadFile toggle pattern in helpers.js.
Remove the vfs-mount/vfs-unmount events emitted on process, as they are insufficient as a security control and a proper permission system is needed instead.
Add tests covering require() of ESM modules from VFS with package.json type detection (.js with type:module, nested directory walk-up, ESM-to-ESM imports) and .mjs extension-based ESM loading without type:module.
| bool use_vfs; | ||
| if (field.value().get_bool().get(use_vfs)) { | ||
| FPrintF( | ||
| stderr, "\"useVfs\" field of %s is not a Boolean\n", config_path); |
There was a problem hiding this comment.
Can you add validation below to make sure that it errors when useVfs is enabled together with useCodeCache/useSnapshot/mainFormat: "module"?
|
|
||
| /** | ||
| * Install fs patches for user code transparency. | ||
| * These make fs.readFileSync('/vfs/path'), fs.statSync, etc. work for user code. |
There was a problem hiding this comment.
This function seems misplaced - they don't have much to do with module loading. It's also strange that for a VFS overlay implementation, lib/fs is not touched in this PR - I think what this tries to do belongs there instead. Since we have access to the true fs originals, it would be much more robust if in lib/fs we just wrap the fs methods and reroute to vfs when toggled - I don't imagine that'll be very expensive since when it's not enabled, it's just one extra boolean check, we can modify the exports in fs.js to add a layer of:
function readFileSync(...) {
if (vfsEnabled) return maybeVfs(...);
return originalFeadFileSync(...);
}And let the vfsEnabled be toggle-able in vfs.mount(). Otherwise this patching would not work if some other code cache access to fs first i.e. something like #62012, then after vfs.mount(), they would not see the effect.:
// 1. Some library code gets loaded and cache fs methods first
const { readFileSync } = require('fs');
function foo(path) {
readFileSync(path);
}
// 2. Some other code does the vfs mounting
vfs.mount();
// 3. User tries to use foo
foo('/vfs/path'); // This won't see the vfs overlayIf we wrap what gets exported in fs.js completely, however, even if any other code cache access to fs methods before the call to vfs.mount(), they will always see vfs interception once enabled. I think that will be the most important characteristics of a built-in vfs overlay implementation - rerouting fs methods regardless of loading order is something only Node.js core can safely do, if we don't do the wrapping and only do a hacky patch like user land solutions, then this isn't much better than a brittle user land solution.
| * @param {string} basePath The base path without extension | ||
| * @returns {string|null} The resolved path with extension, or null | ||
| */ | ||
| function tryExtensions(vfs, basePath) { |
There was a problem hiding this comment.
This and the code below does not look right - we are re-inventing a different version of module resolution with random quirks, and this will become a maintenance burden. Take tryExtensions here for example: the true tryExtensions in cjs/loader would also consider --preserve-symlinks-main and --preserve-symlinks, this version, however, doesn't , even if vfs support symlinks, so the resolution algorithm becomes different within the vfs than in a real fs. The list of extensions also get hard-coded here instead of being in sync with Object.keys(Module._extensions), which is what the true module resolution uses. We could've just redirect the stat call with a wrapper in the tryExtensions in cjs/loader to consult the vfs if necessary, similar to what other helpers do, and all the rest of the logic will remain in one place, instead of being duplicated here with footguns in the subtle differences.
The same applies to many other methods in this file e.g. the re-invented resolveConditions doesn't handle error recovery correctly, resolvePackageExports doesn't support patterns, resolveBareSpecifier has an odd choice of delegating to nextResolve for #imports, etc. Unless we want to document all these quirks, it's better to just update the canonical implementation to reroute to VFS if necessary, and reuse them.
Add SEA config validation to error when useVfs is enabled together with useSnapshot, useCodeCache, or mainFormat: "module", as these combinations are not supported.
|
@joyeecheung can we land this and iterate? This is starting to get immense. |
19 participants
A first-class virtual file system module (
node:vfs) with a provider-based architecture that integrates with Node.js's fs module and module loader.Key Features
Provider Architecture - Extensible design with pluggable providers:
MemoryProvider- In-memory file system with full read/write supportSEAProvider- Read-only access to Single Executable Application assetsVirtualProvider- Base class for creating custom providersStandard fs API - Uses familiar
writeFileSync,readFileSync,mkdirSyncinstead of custom methodsMount Mode - VFS mounts at a specific path prefix (e.g.,
/virtual), clear separation from real filesystemModule Loading -
require()andimportwork seamlessly from virtual filesSEA Integration - Assets automatically mounted at
/seawhen running as a Single Executable ApplicationFull fs Support - readFile, stat, readdir, exists, streams, promises, glob, symlinks
Example
SEA Usage
When running as a Single Executable Application, bundled assets are automatically available:
Public API
Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.
Fixes #60021