Sideloading arbitrary OCI artifacts from predefined locations

[arikgrahl]

Describe the problem to be solved

Spegel relies on the container runtime, when serving OCI artifacts.
This means container image must be known to the underlying container runtime before they can be exchanged among different peers.
While this is without doubt the most important use-case, sideloading arbitrary OCI artifacts could streamline the process of ingesting OCI artifacts from upstream process or sources.
A simple lookup of artifacts from a local filesystem could therefore enable interesting new use-cases for Spegel.

Here is an example use-case on which I face on a regular basis.
To build my container images, I rely on Nix' pkgs.dockerTools.buildLayeredImage, which produces an gzipped OCI image somewhere on the filesystem (e.g. /nix/store/fq1rvmzk1bv8xwlrn3r4xdgk5i5hqzaq-example.tar.gz).
At the moment I need to load this image (e.g. ctr images import /nix/store/fq1rvmzk1bv8xwlrn3r4xdgk5i5hqzaq-example.tar.gz) in order to make the container runtime and therefore Spegel aware of this image.
Sideloading arbitrary OCI artifacts from predefined locations in combination with a simple lookup pattern would expose any built OCI images automatically without explicitly creating them in the container runtime's store.

I have implemented this Nix-specific feature in a standalone read-only OCI registry and would love to bring this to Spegel.
In a proof-of-concept, I integrated this into a fork of Spegel, which I could successfully test in a fork of K3s bundling it.
I believe it's worthwhile to further abstract this feature and support the sideloading of arbitrary OCI artifacts.

Proposed solution to the problem

We could introduce a configuration option specifying the location and pattern of OCI artifacts eligible for sideloading.
This configuration could include values such as:

• /nix/store/<ref>-<name>.tar.gz
• /opt/<name>/<ref>.tar

The first configuration example would sideload OCI artifacts for requests like HEAD /v2/example/manifests/fq1rvmzk1bv8xwlrn3r4xdgk5i5hqzaq, effectively serving the use-case outlined in the problem statement section.
The second configuration example would sideload tar balls in the /opt directory nested further by name and ref.
Even other OCI artifacts than (gzipped) tar balls could be served from configured locations in the filesystem.

I am willing to provide an initial implementation by next week.
Please let me know if further information is needed, and do not hesitate to reach out.

[phillebaba]

This would be a good replacement for the memory OCI implementation that is currently used in tests. I would want to add some restrictions to this feature to make sure we aren't moving away from the OCI spec. The behavior of the new file based OCI store backend should behave similar to the current solution. When you are doing this are you serving the whole tar och the image or the actual individual images? I would assume that we would need to un-tar the images into its separate layers for this to work?

Would it be possible to make this a generic implementation? We pass a directory or list of directories that contain OCI layers. Each layers name is the digest, which makes finding and listing the layers easy.

The only thing that would not be possible in this situation would be tagging of these images. Also we would miss out on any grabage collection that the kubelet normally does.

[brandond]

This seems like an odd thing to try to shoehorn into spegel. The whole point of Spegel is to share what's already available in the node's image store.

If you want to provide another source for content other than the image store and the upstream registry, then set up a standalone registry, and add that as a mirror after spegel, but before the upstream.