Add support for erofs and squashfs layer media types

[rchincha]

OCI artifacts support has landed in various OCI specs v1.1.0 which allows for arbitrary artifact types, small and large.

Large artifacts (even existing container images) pose a particular challenge that:

1. it takes too long to download
2. it takes too long to unpack

This PR begins to address 2. above.

squashfs (an older read-only) and erofs (newer) filesystems are already compressed full filesystems, and overlayfs compatible.

[sudo-bmitch]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

[rchincha]

#1190

[rchincha]

opencontainers/distribution-spec#537
to be combined with ^

[rchincha]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

[hsiangkao]

This needs a rebase on main to be considered. Also, I don't think we need to include non-OCI media types, especially if they've been registered with IANA.

This PR just represents possibilities. Needs a discussion if OCI wants to "adopt" this a recommended option. Almost all known solutions end up being a clever fuse or remotefs wrapper around a layer - so why not make the layer a filesystem itself! No further machinery needed.

JFYI, EROFS already has a IANA-registered media type "vnd.erofs" if some format is referred.

[rchincha]

Could also be application/vnd.oci.image.layer.erofs and application/vnd.oci.image.layer.squashfs
just so that we indicate this is OCI blessed.

Open question, whiteout handling

[syed]

The problems I see with large binary files (eg AI models) is that they don't tend well to compression. I would like to see if we can improve the distribution spec to allow parallel chunked upload/download. Right now we can only upload/download sequentially

[sudo-bmitch]

Right now we can only upload/download sequentially

I get the impression that most registries allow this already and we're just formalizing it.

[rchincha]

https://stackerbuild.io/
https://github.com/project-stacker/stacker/releases/tag/v1.1.0-rc1
v1.1.0-rc1 can now natively build OCI images with erofs layers+dmverity

[rchincha]

https://stackerbuild.io/ https://github.com/project-stacker/stacker/releases/tag/v1.1.0-rc1 v1.1.0-rc1 can now natively build OCI images with erofs layers+dmverity

In this context ^
containerd/containerd#11163

[sajayantony]

Honestly not in favor or non-oci types here.

[rchincha]

Honestly not in favor or non-oci types here.

Just to clarify further - the image is still OCI; this is about the layers. If they are complete filesystems (squashfs, erofs), then they can just be mounted directly. This is an alternative approach to what the recently added containerd erofs snapshotter does today - convert tgz into erofs and cache.

The only implication wrt this PR is that if squashfs, erofs are made "official" layer types (not a MUST btw, image still OCI), then the negotiation with containerd community becomes easier. Else maybe they will accept a tunable/config to allow such types.

If squashfs,erofs+verity are produced and signed at source, then integrity is maintained end-to-end and becomes a much simpler deployment model - no tgz mutation gimmickry.

[hsiangkao]

Honestly not in favor or non-oci types here.

Just to clarify further - the image is still OCI; this is about the layers. If they are complete filesystems (squashfs, erofs), then they can just be mounted directly. This is an alternative approach to what the recently added containerd erofs snapshotter does today - convert tgz into erofs and cache.

The only implication wrt this PR is that if squashfs, erofs are made "official" layer types (not a MUST btw, image still OCI), then the negotiation with containerd community becomes easier. Else maybe they will accept a tunable/config to allow such types.

If squashfs,erofs+verity are produced and signed at source, then integrity is maintained end-to-end and becomes a much simpler deployment model - no tgz mutation gimmickry.

@rchincha One thing I might need to mention here is prefix+compression format, currently
application/vnd.oci.image.layer.v1.tar+gzip
application/vnd.oci.image.layer.v1.tar+zstd
are reasonable because they are wraped with real gzip and zstd formats and can be decoded with gzip and zstd tools, also see https://www.rfc-editor.org/rfc/rfc6839#section-3.6
But I think it's incorrect to use if the case is "a filesystem contains lz4, lzma, deflate or zstd compressed data", it cannot be directly decoded with any tool above. IOWs, only uncompressed filesystem + gzip/zstd wrapper can be marked with application/vnd.xxx+gzip/zstd, otherwise it should be just application/vnd.xxx without any suffix.
Also EROFS supports multiple algorithms in one filesystem/image, so such suffix is meaningless to me.

[hsiangkao]

...

@rchincha One thing I might need to mention here is prefix+compression format, currently application/vnd.oci.image.layer.v1.tar+gzip application/vnd.oci.image.layer.v1.tar+zstd are reasonable because they are wraped with real gzip and zstd formats and can be decoded with gzip and zstd tools, also see https://www.rfc-editor.org/rfc/rfc6839#section-3.6 But I think it's incorrect to use if the case is "a filesystem contains lz4, lzma, deflate or zstd compressed data", it cannot be directly decoded with any tool above. IOWs, only uncompressed filesystem + gzip/zstd wrapper can be marked with application/vnd.xxx+gzip/zstd, otherwise it should be just application/vnd.xxx without any suffix. Also EROFS supports multiple algorithms in one filesystem/image, so such suffix is meaningless to me.

Also I try to enable EROFS metadata compression in the kernel 6.16 cycle: Compressed metadata is just placed in a special on-disk inode, so the metadata compress algorithm can be different from data algorithms, which could be useful to maximize runtime performance for metadata (lz4, zstd, or uncompressed) and leave high compression ratios for data (zstd, lzma, deflate, etc.)

[rchincha]

Pushed two test multi-layer OCI images with erofs layers (with and without dm-verity)

https://hub.docker.com/repository/docker/ramchi879/test-erofs-verity/tags
https://hub.docker.com/repository/docker/ramchi879/test-erofs-no-verity/tags

To download them,
skopeo copy --dest-oci-accept-uncompressed-layers docker://ramchi879/test-erofs-verity:c-erofs oci:erofs:test
skopeo copy --dest-oci-accept-uncompressed-layers docker://ramchi879/test-erofs-no-verity:c-erofs oci:erofs:test