Support buildkit binary when creating a container image

[diana-zvulun]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Since dockershim was deprecated in EKS, we no longer have the option to build container images in K8S with docker. It would be great if pulumi would support building container images with other tools such as buildkit (using buildctl) or kaniko.

Affected area/feature

Right now we build images using - awsx.ecr.Image & docker.Image but on the next k8s version this will not work so we need to resort to using Command that will build the image instead.

[blampe]

Thank you for the issue @diana-zvulun! If you could share a few more details it would be tremendously helpful:

• What version of awsx and pulumi/docker are you using?
• Are you able to test your existing code on the newer EKS version? What errors do you get when building your images?
• Can you share some anonymized examples of how you're configuring your awsx.ecr.Image and docker.Image resources?

It would be great if pulumi would support building container images with other tools such as buildkit (using buildctl) or kaniko.

Does this mean you'll have something like buildkitd available? I don't think anyone has tried building against that backend but it's an interesting idea.

[diana-zvulun]

• awsx version 1.0.6 and pulumi/docker version 3.6.1
• In order to be able to run docker inside our pulumi pod we mount the hostVolume /root/.docker inside the pod. Since we remove this volume (since EKS no longer supports it) this is the error we are getting when trying to use our docker.Image pulumi resource:

`
Error: ' docker build -f /workspace/stack/Dockerfile --build-arg APP=some-image-test --network host /workspace/stack -t c57d7350-container' failed with exit code 1

: Error: ' docker build -f /workspace/stack/Dockerfile --build-arg APP=some-image-test --network host /workspace/stack -t c57d7350-container' failed with exit code 1

at /snapshot/awsx/node_modules/@pulumi/docker/docker.js:406:19

at Generator.next (<anonymous>)

at fulfilled (/snapshot/awsx/node_modules/@pulumi/docker/docker.js:18:58)

at processTicksAndRejections (node:internal/process/task_queues:96:5)

error: ERROR: mkdir /root/.docker/buildx: read-only file system

' docker build -f /workspace/stack/Dockerfile --build-arg APP=some-image-test --network host /workspace/stack -t c57d7350-container' failed with exit code 1

error: Error: ' docker build -f /workspace/stack/Dockerfile --build-arg APP=some-image-test --network host /workspace/stack -t c57d7350-container' failed with exit code 1

`

This is how we create the docker.Image resource:

new awsx.ecr.Image(
      this.name,
      {
        args: {
          APP: this.name,
        },
        extraOptions: ['--network', 'host'],
        dockerfile: join(__dirname, `../Dockerfile`),
        path: join(__dirname, '../..'),
        repositoryUrl: repository.repository.repositoryUrl,
      }
    )

Yes, we currently have buildctl available in our pulumi pod that is able to connect to a buildkitd service in our cluster in order to build container images.

[blampe]

pulumi/docker version 3.6.1

@diana-zvulun are you able to use v4 of the provider, or at least run some tests with it? The newer version doesn't need a docker CLI and interacts with the build daemon directly. AFAIK the APIs involved are the same, so it should work equally well with Docker or buildkitd -- but I haven't confirmed this.

[blampe]

@diana-zvulun you might also be interested in the new Docker Build provider, which is compatible with any buildkit daemon.

The repository is here; our blog announcement is here; and finally API docs are here along with examples of how to migrate your existing Image resources to dockerbuild.Image.