Add garbage collection for docker images #2435

jsoriano · 2025-02-27T00:08:32Z

Related to #856.

If one is using elastic-package during enough time, Docker storage can end up full of docker images. Once the 90% threshold is reached, behavior can be unpredictable as described in #856.

This change introduces garbage collection for docker images that were downloaded by elastic-package. Before pulling new images, it checks first if there are old unused images or if the images are taking too much space, and remove images accordingly. It only removes images known to be downloaded by elastic-package.

Implementation is based on two methods:

Track is used to start tracking images, by recording their tags and last used times. It has to be called before pulling the images, it won't track images that are already pulled, to avoid GCing images not managed by elastic-package.
Run runs the garbage collection, based on two parameters: max_unused can be used to remove images older than some time and max_total_size to trigger removals when the total size of images (from docker system df) goes beyond some threshold. Less recently used images are removed first.

Tracking of known images is persisted under ~/.elastic-package/cache/docker-images-gc.json.

Garbage collection is disabled by default.

This can be configured from ~/.elastic-package/config.yml, for example:

stack:
  gc:
    enabled: true
    max_unused: 336h # 2 weeks
    max_total_size: 60GB

jsoriano · 2025-02-27T00:15:36Z

internal/common/bytesize.go

+)
+
+// ByteSize represents the size of a file.
+type ByteSize uint64


Type mostly copied from https://github.com/elastic/package-spec/blob/964c4a69e024cc464c4808720ba0db9f001a82a7/code/go/internal/spectypes/filesize.go, adding support for GB and floats, to support sizes reported by docker system df.

jsoriano · 2025-02-27T00:19:19Z

internal/stack/compose.go

@@ -50,6 +51,34 @@ func (eb *envBuilder) build() []string {
 	return eb.vars
 }

+func runDockerImagesGC(ctx context.Context, project *compose.Project, opts compose.CommandOptions, appConfig *install.ApplicationConfiguration) error {


Added by now only to stack operations, it could be added too to agent and service deployers. It may be specially interesting in the service deployer.

I've checked that running elastic-package test system -v in the mysql package (integrations repo), it creates 36 docker images to test the service:

$ elastic-package test system -v ... $ docker images |grep elastic-package-service |grep mysql | wc -l 36 $ docker images |grep elastic-package-service ... elastic-package-service-30135-mysql latest 040dcd09ffba 58 minutes ago 1.34GB elastic-package-service-13845-mysql_replica latest 9b7839234a41 About an hour ago 591MB ...

When trying with other packages using the new custom agents (e.g. oracle), elastic-package creates one elastic-agent docker image per each system test with different Dockerfile (hash of that file is part of the docker tag) link.

$ elastic-package test system -v --data-streams memory,performance ... $ docker images |grep elastic-package-test elastic-package-test-elastic-agent 8.17.3-1e83b6defffbb1d833d6235e1c502bad ecbc1bc5d908 22 minutes ago 1.95GB elastic-package-test-elastic-agent 8.17.3-2459e61e8ebaf237cffe70dc5ad649cb 159f23420959 31 minutes ago 1.95GB

As most of them are built with the same base image, there could be a lot of shared layers.

Removing all the previous docker images in my laptop (docker rmi), docker just untagged the docker images. Maybe does it need to clean the docker cache too ?

$ docker images |grep elastic-package-service |grep mysql | awk '{print $3}' | xargs -I {} docker rmi {} Untagged: elastic-package-service-40716-mysql:latest Deleted: sha256:1fc900526a35aeb6563b6fc4a393fe88e2282b41ca95e603684bd3dc72264245 Untagged: elastic-package-service-47488-mysql:latest Deleted: sha256:6625ce72c4b0b506cf0ad86e8b703b580d6d90707471aef59c39d2e62ab4fa82 Untagged: elastic-package-service-66823-mysql:latest Deleted: sha256:973241860b7932f231dead77a048d8a433bd468ed1e93bb90c8e0815707cc5ee Untagged: elastic-package-service-61615-mysql:latest Deleted: sha256:0385a33fc301b9b417ac9db269386af710767ef7e6342cfcfccb61717857051a ... $ docker rmi elastic-package-test-elastic-agent:8.17.3-1e83b6defffbb1d833d6235e1c502bad Untagged: elastic-package-test-elastic-agent:8.17.3-1e83b6defffbb1d833d6235e1c502bad Deleted: sha256:ecbc1bc5d908ca93d32d8af54ecc43dc067a8367abc400f2bcfc325aa37bb629 $ docker rmi elastic-package-test-elastic-agent:8.17.3-2459e61e8ebaf237cffe70dc5ad649cb Untagged: elastic-package-test-elastic-agent:8.17.3-2459e61e8ebaf237cffe70dc5ad649cb Deleted: sha256:159f234209595713fc882f7e6802c09a4b57381770610e2661defbf5dfe91618

jsoriano · 2025-02-27T00:20:15Z

internal/compose/compose.go

@@ -60,7 +61,23 @@ type Config struct {
 	Services map[string]service
 }

+// Images lists the images found in the configuration.


This only lists images found directly in the docker compose configuration, it won't find images found in Dockerfiles. Maybe we can think in other approaches to find the images to track. Ideas welcome.

internal/docker/imagesgc.go

internal/common/bytesize.go

efd6 · 2025-03-03T01:07:25Z

internal/common/bytesize.go

+	return s.unmarshalString(value.Value)
+}
+
+var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString))


Suggested change

var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString))

var bytesPattern = regexp.MustCompile(`^(\d+(\.\d+)?)(` + byteString + `|` + kiloByteString + `|` + megaByteString + `|` + gigaByteString + `|)$`)

(compile-time string construction instead of init-time)

but for a more efficient pattern

Suggested change

var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s|%s|%s|%s|)$`, byteString, kiloByteString, megaByteString, gigaByteString))

var bytesPattern = regexp.MustCompile(`^(\d+(\.\d+)?)(B|[KMG]B|)$`)

efd6 · 2025-03-03T01:28:24Z

internal/docker/imagesgc.go

+	path   string
+	images []gcEntry
+	clock  func() time.Time
+	client imagesGCClient


Field godoc?

efd6 · 2025-03-03T01:29:44Z

internal/docker/imagesgc.go

+// Persist saves the list of images to disk.
+func (gc *ImagesGC) Persist() error {
+	if gc.path == "" {
+		return errors.New("GC list was not created with a path")


Seems to me that path != "" is an invariant where non-adherence is a programmer error, so this should panic.

internal/docker/imagesgc.go

efd6 · 2025-03-03T01:42:28Z

internal/docker/imagesgc.go

+					continue
+				}
+				if err != nil {
+					gc.images = append(images, gc.images[i:]...)


Why do we do this? I can't see that this value is used after an error. What am I missing?

This is done in case gc is used after Run() fails, so gc.images is updated with images that may have been removed.
gc is not reused now, but the idea is that it could be.

Co-Authored-By: Dan Kortschak <[email protected]>

elasticmachine · 2025-03-04T16:37:28Z

💚 Build Succeeded

Buildkite Build
Commit: 9d5ec02

History

💔 Build #4931 failed 267b723
💔 Build #4930 failed 8b2f932
💔 Build #4899 failed 334f8a0

cc @jsoriano

Add GC for docker images

334f8a0

jsoriano self-assigned this Feb 27, 2025

jsoriano commented Feb 27, 2025

View reviewed changes

Linting

8b2f932

efd6 reviewed Mar 3, 2025

View reviewed changes

jsoriano and others added 2 commits March 4, 2025 16:52

Suggestions from review.

267b723

Co-Authored-By: Dan Kortschak <[email protected]>

Ensure cache directory exists

9d5ec02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add garbage collection for docker images #2435

Add garbage collection for docker images #2435

jsoriano commented Feb 27, 2025 •

edited

Loading

jsoriano Feb 27, 2025 •

edited

Loading

jsoriano Feb 27, 2025

mrodm Mar 14, 2025

jsoriano Feb 27, 2025

efd6 Mar 3, 2025

efd6 Mar 3, 2025

efd6 Mar 3, 2025

efd6 Mar 3, 2025

jsoriano Mar 4, 2025

efd6 Mar 17, 2025

elasticmachine commented Mar 4, 2025

	var bytesPattern = regexp.MustCompile(fmt.Sprintf(`^(\d+(\.\d+)?)(%s\|%s\|%s\|%s\|)$`, byteString, kiloByteString, megaByteString, gigaByteString))
	var bytesPattern = regexp.MustCompile(`^(\d+(\.\d+)?)(` + byteString + `\|` + kiloByteString + `\|` + megaByteString + `\|` + gigaByteString + `\|)$`)

Add garbage collection for docker images #2435

Are you sure you want to change the base?

Add garbage collection for docker images #2435

Conversation

jsoriano commented Feb 27, 2025 • edited Loading

jsoriano Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 4, 2025

💚 Build Succeeded

History

jsoriano commented Feb 27, 2025 •

edited

Loading

jsoriano Feb 27, 2025 •

edited

Loading