Internal format of *.data data-package files

[vadimkantorov]

Are there somehow concatenated LZMA-compressed files? Or are these simply ZIP files? (which can accommodate LZMA-compressed entries)

[kripken]

It just concatenates the files' binary data (implemented here). So there is no compression - that is left for the webserver, which can use gzip or brotli.

[kripken]

That option uses LZ4, yes. But the format of the files does not change otherwise. This just uses LZ4 compression on each file write, and decompression on each read, like a compressed linux filesystem.

[vadimkantorov]

Files are LZ4-compressed independently, right?

Am I right that it could just be a ZIP-file with LZ4-compressed entries? (and the final central directory) As in ZIP, the files can also be compressed independently

[kripken]

In theory the data files could use the zip format, yes.

But as webservers can gzip transparently, that hasn't been added.

The existing LZ4 support is not aimed to reduce the size of the downloaded data file, but on reducing the size at runtime (like a compressed linux filesystem, which is sometimes even faster).

[vadimkantorov]

I've implemented in C a read-only, in-memory FS (using syscall overriding and libarchive) allowing similar functionality. I'll send you a link when it's ready... The idea is that the virtual FS-kind of functionality is useful also in regular contexts - e.g. for packing a bunch of files in a statically-linked binary. But it also could be used to replace the virtual FS functionality of Emscripten - the benefit would be that the same code could be used for both in-browser context and regular x64 context... Ideally, some mini-project like this (2K LoC) could be supported part of emscripten-core org... Or some other established project...

[kripken]

Sounds interesting...