`podman manifest inspect` showing unexpected values for mixed-schema image structures

[codingben]

Issue Description

We have pushed OCI-based image index and manifests. Podman returns wrong results about media type of image index.

Steps to reproduce the issue

Build OCI image index that will contain OCI image manifests, and push it to any container registry.

╰─$ podman pull quay.io/boukhano/cirros:6.1            
Trying to pull quay.io/boukhano/cirros:6.1...
Error: copying system image from manifest list: parsing image configuration: invalid mixed OCI image with Docker v2s2 config ("application/vnd.docker.container.image.v1+json")

Describe the results you received

╰─$ podman manifest inspect quay.io/boukhano/cirros:6.1                   
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 418,
            "digest": "sha256:3a37dc3fd4e8460225ef5040308838ae344327c4901cffa653f3e157aae6c4f7",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 418,
            "digest": "sha256:cfefc66dad4addd779ff91774c1592f7a8cc87d3a905ea86bbcd9441045fe143",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

Describe the results you expected

╰─$ docker manifest inspect quay.io/boukhano/cirros:6.1                                         1 ↵
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 418,
         "digest": "sha256:3a37dc3fd4e8460225ef5040308838ae344327c4901cffa653f3e157aae6c4f7",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 418,
         "digest": "sha256:cfefc66dad4addd779ff91774c1592f7a8cc87d3a905ea86bbcd9441045fe143",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

N/A

Additional information

We have pushed OCI image index and manifests:

╰─$ curl -H "Accept: application/vnd.oci.image.index.v1+json" "https://quay.io/v2/boukhano/cirros/manifests/6.1"   

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","size":418,"digest":"sha256:3a37dc3fd4e8460225ef5040308838ae344327c4901cffa653f3e157aae6c4f7","platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","size":418,"digest":"sha256:cfefc66dad4addd779ff91774c1592f7a8cc87d3a905ea86bbcd9441045fe143","platform":{"architecture":"arm64","os":"linux"}}]}

[codingben]

@nalind Can you please take a look? Thanks!

[mheon]

@mtrmac PTAL

[codingben]

Thanks for looking on it. @nalind debugged and saw that layer/config still using Docker and not OCI, and that's why Podman doesn't work:

╰─$ skopeo inspect --raw docker://quay.io/boukhano/cirros@sha256:e96e9fe082ec1c9c056144f58b52de10314adb79353f35298caa2cd25c79d83c | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 329,
    "digest": "sha256:7760637f74788fa570057d4c2e7b5c21d996863744d6049807c218f01154fa74"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 20787697,
      "digest": "sha256:97565d0bbd9530b3349cabff269322e8686a4f3add243e72814ba51630e49be9"
    }
  ]
}

While image index and image manifests are based on OCI schemas.

[nalind]

Out of band, we have an image with an OCI media type, but which is using Docker media types to describe its config blob and layers:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 329,
    "digest": "sha256:7760637f74788fa570057d4c2e7b5c21d996863744d6049807c218f01154fa74"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 20787697,
      "digest": "sha256:97565d0bbd9530b3349cabff269322e8686a4f3add243e72814ba51630e49be9"
    }
  ]
}

being generated by https://github.com/kubevirt/containerdisks/blob/main/pkg/build/build.go with changes from https://github.com/kubevirt/containerdisks/compare/40fc44a45797c5e02bfdf72d1f161fc0bb155254..c39ddf6c0af1b76817541c0cdae8e744003b252d applied.

It looks like passing tarball.WithMediaType() as the second argument to the tarball.LayerFromOpener() call would change the layer's media type and running img through mutate.ConfigMediaType() would change its config blob's type.

But that leaves the question of whether or not we should be balking at such an image anyway. The spec seems to advise that images that are concerned with portability should be using OCI types for these things, and that seems easy enough to fix here, but the spec notes that implementations mustn't error out when encountering unknown types, and it looks to me like we're doing that here (for certain values of "unknown", anyway).

[nalind]

being generated by https://github.com/kubevirt/containerdisks/blob/main/pkg/build/build.go with changes from https://github.com/kubevirt/containerdisks/compare/40fc44a45797c5e02bfdf72d1f161fc0bb155254..c39ddf6c0af1b76817541c0cdae8e744003b252d applied.

... or reverted, as it may be.

[mtrmac]

But that leaves the question of whether or not we should be balking at such an image anyway. The spec seems to advise that images that are concerned with portability should be using OCI types for these things, and that seems easy enough to fix here, but the spec notes that implementations mustn't error out when encountering unknown types, and it looks to me like we're doing that here (for certain values of "unknown", anyway).

The spec says

Implementations storing or copying image manifests MUST NOT error on encountering a value that is unknown to the implementation.

That applies to skopeo copy, but not really to pulls which must consume the data.

It would be ~easy enough to handle some of the code paths; e.g. Inspect can ignore the manifest media type and choose an implementation based on just the config’s media type.

But I don’t think it’s longer-term reasonable, because the matrix of combinations blows up and capabilities can no longer be relied on.

E.g., (until recent Docker’s vendor-specific extension), “OCI images don’t support health checks” was true. If we support “OCI images with v2s2 configs”… what does that mean?

If we support “OCI images with v2s2 configs”, do we need to implement a “convert an OCI image to really-follows-spec-OCI-image” operation?

Hypothetically, if OCI config annotations are defined: What does it mean to have a v2s2 config with a an OCI-config-specific annotation?

[baude]

@codingben @nalind @mtrmac thanks for all the write ups ... i wanted to make sure we captured this for the future. it also seems to me that the title of the issue no longer reflects what is going on ... should it be updated for clarity?

[mtrmac]

All that said, this output

$ podman manifest inspect quay.io/boukhano/cirros:6.1                   
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

is either a bug, or maybe an intentionally-permissive implementation somehow trying to fall back to v2s2 based on the inconsistent per-platform instances.

[nalind]

I expect that was built using kubevirt/containerdisks#81, where the mediaType values being used are all hard-coded.

[mtrmac]

Was the image under that tag overwritten / replaced? At least right now, skopeo inspect --raw shows the image to use an OCI index media type.

[codingben]

Was the image under that tag overwritten / replaced? At least right now, skopeo inspect --raw shows the image to use an OCI index media type.

It should be OCI index media type and OCI image manifests, I just updated it again:

╰─$ docker manifest inspect quay.io/boukhano/cirros:6.1     
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 418,
         "digest": "sha256:a0e5b0df033c10034371d9711d90771816f349b57e246cf95b17b52b086a0d26",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 418,
         "digest": "sha256:7aa5762d40d1bbc545210629f9ff5a2036179d63551a5abbfe13f614f6cb4962",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

If you run podman manifest inspect quay.io/boukhano/cirros:6.1, it'll return Docker manifest list instead of OCI image index now.

[codingben]

@mtrmac Because of it, we decided that in our production registry quay.io/organization/containerdisks - We'll use Docker schemas and not OCI because of this issue [1].

So I'll leave quay.io/boukhano/cirros:6.1 to use OCI image index and OCI image manifests for testing purposes of this issue.

[1] https://github.com/kubevirt/containerdisks/compare/40fc44a45797c5e02bfdf72d1f161fc0bb155254..c39ddf6c0af1b76817541c0cdae8e744003b252d