Skip to content

Commit 39f4cbb

Browse files
vitorsavianvitorsavian
authored
Add auto import images for containerd image store
* Add auto import images for containerd image store * Add auto import images Signed-off-by: Vitor Savian <[email protected]> * Fix EOF error log when importing tarball files Signed-off-by: Vitor Savian <[email protected]> * Delaying queue Signed-off-by: Vitor Savian <[email protected]> * Add parse for images Signed-off-by: Vitor Savian <[email protected]> --------- Signed-off-by: Vitor Savian <[email protected]>
1 parent 9d31106 commit 39f4cbb

File tree

7 files changed

+727
-23
lines changed

7 files changed

+727
-23
lines changed

‎.github/workflows/e2e.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ jobs:
4040
strategy:
4141
fail-fast: false
4242
matrix:
43-
etest: [startup, s3, btrfs, externalip, privateregistry, embeddedmirror, wasm]
43+
etest: [autoimport, startup, s3, btrfs, externalip, privateregistry, embeddedmirror, wasm]
4444
max-parallel: 3
4545
steps:
4646
- name: "Checkout"

‎docs/adrs/add-auto-import-containerd.md

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Easy way for auto adding images to k3s
2+
3+
Date: 2024-10-2
4+
5+
## Status
6+
7+
Proposed
8+
9+
## Context
10+
11+
Since the feature for embedded registry, the users appeared with a question about having to manually import images, specially in edge environments.
12+
13+
As a result, there is a need for a folder who can handle this action, where every image there will be watched by a controller for changes or new images, this new images or new changes will be added to the containerd image store.
14+
15+
The controller will watch the agent/images folder that is the default folder for the images, as the first iteration about the controller he will mainly work with the default image folder, but in the future we can set to watch more folders.
16+
17+
The main idea for the controller is to create a map for the file infos maintaining the state for the files, with that we can see if a file was modified and if the size changed.
18+
19+
### Map to handle the state from the files
20+
21+
This map will have the entire filepath of the file in the `key` value, since we can get the value from the key with only the `event.Name`
22+
23+
```go
24+
map[string]fs.FileInfo
25+
```
26+
27+
### Why use fsnotify
28+
29+
With this library we can easily use for any linux distros without the need to port for a specify distro and can also run in windows.
30+
31+
The main idea for the watch will be taking care of the last time that was modified the image file.
32+
33+
fsnotify has a great toolset for handling changes in files, since the code will have a channel to receive events such as CREATE, RENAME, REMOVE and WRITE.
34+
35+
### How the controller will work with the events
36+
37+
When the controller receive a event saying that a file was created, he will add to the map and import the images if the event that he has received is not a directory and then import the image.
38+
39+
When the controller receive a event saying that a file was writen, he will verify if the file has the size changed and if the file has the time modified based on the time and size from the state.
40+
41+
When the controller receive a event saying that a file was renamed, or removed, he will delete this file from the state. when a file is renamed, it is created a new file with the same infos but with a the new name, so the watcher will sent for the controller a event saying that a file was created.
42+
43+
## Decision
44+
45+
- Decided
46+
47+
## Consequences
48+
49+
Good:
50+
- Better use of embedded containerd image store.
51+
- Fsnotify it's a indirect dependency that upstream uses
52+
53+
Bad:
54+
- The need for another dependency

‎go.mod

+1-2
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ require (
9393
github.com/docker/docker v27.1.1+incompatible
9494
github.com/erikdubbelboer/gspt v0.0.0-20190125194910-e68493906b83
9595
github.com/flannel-io/flannel v0.25.7
96+
github.com/fsnotify/fsnotify v1.7.0
9697
github.com/go-bindata/go-bindata v3.1.2+incompatible
9798
github.com/go-logr/logr v1.4.2
9899
github.com/go-logr/stdr v1.2.3-0.20220714215716-96bad1d688c5
@@ -258,7 +259,6 @@ require (
258259
github.com/felixge/httpsnoop v1.0.4 // indirect
259260
github.com/flynn/noise v1.1.0 // indirect
260261
github.com/francoispqt/gojay v1.2.13 // indirect
261-
github.com/fsnotify/fsnotify v1.7.0 // indirect
262262
github.com/ghodss/yaml v1.0.0 // indirect
263263
github.com/go-errors/errors v1.4.2 // indirect
264264
github.com/go-jose/go-jose/v4 v4.0.2 // indirect
@@ -278,7 +278,6 @@ require (
278278
github.com/google/cel-go v0.22.0 // indirect
279279
github.com/google/gnostic-models v0.6.9 // indirect
280280
github.com/google/go-cmp v0.6.0 // indirect
281-
github.com/google/go-containerregistry v0.20.2 // indirect
282281
github.com/google/gofuzz v1.2.0 // indirect
283282
github.com/google/gopacket v1.1.19 // indirect
284283
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db // indirect

‎pkg/agent/containerd/containerd.go

+37-20
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import (
1818
"github.com/containerd/containerd/pkg/cri/constants"
1919
"github.com/containerd/containerd/pkg/cri/labels"
2020
"github.com/containerd/containerd/reference/docker"
21+
reference "github.com/google/go-containerregistry/pkg/name"
2122
"github.com/k3s-io/k3s/pkg/agent/cri"
2223
util2 "github.com/k3s-io/k3s/pkg/agent/util"
2324
"github.com/k3s-io/k3s/pkg/daemons/config"
@@ -115,24 +116,6 @@ func Run(ctx context.Context, cfg *config.Node) error {
115116
// any .txt files are processed as a list of images that should be pre-pulled from remote registries.
116117
// If configured, imported images are retagged as being pulled from additional registries.
117118
func PreloadImages(ctx context.Context, cfg *config.Node) error {
118-
fileInfo, err := os.Stat(cfg.Images)
119-
if os.IsNotExist(err) {
120-
return nil
121-
} else if err != nil {
122-
logrus.Errorf("Unable to find images in %s: %v", cfg.Images, err)
123-
return nil
124-
}
125-
126-
if !fileInfo.IsDir() {
127-
return nil
128-
}
129-
130-
fileInfos, err := os.ReadDir(cfg.Images)
131-
if err != nil {
132-
logrus.Errorf("Unable to read images in %s: %v", cfg.Images, err)
133-
return nil
134-
}
135-
136119
client, err := Client(cfg.Containerd.Address)
137120
if err != nil {
138121
return err
@@ -162,6 +145,28 @@ func PreloadImages(ctx context.Context, cfg *config.Node) error {
162145
return errors.Wrap(err, "failed to clear pinned labels")
163146
}
164147

148+
go watchImages(ctx, cfg)
149+
150+
// After setting the watcher, connections and everything, k3s will see if the images folder is already created
151+
// if the folder its already created, it will load the images
152+
fileInfo, err := os.Stat(cfg.Images)
153+
if os.IsNotExist(err) {
154+
return nil
155+
} else if err != nil {
156+
logrus.Errorf("Unable to find images in %s: %v", cfg.Images, err)
157+
return nil
158+
}
159+
160+
if !fileInfo.IsDir() {
161+
return nil
162+
}
163+
164+
fileInfos, err := os.ReadDir(cfg.Images)
165+
if err != nil {
166+
logrus.Errorf("Unable to read images in %s: %v", cfg.Images, err)
167+
return nil
168+
}
169+
165170
for _, fileInfo := range fileInfos {
166171
if fileInfo.IsDir() {
167172
continue
@@ -176,6 +181,7 @@ func PreloadImages(ctx context.Context, cfg *config.Node) error {
176181
}
177182
logrus.Infof("Imported images from %s in %s", filePath, time.Since(start))
178183
}
184+
179185
return nil
180186
}
181187

@@ -214,7 +220,7 @@ func preloadFile(ctx context.Context, cfg *config.Node, client *containerd.Clien
214220
}
215221
}
216222

217-
if err := labelImages(ctx, client, images); err != nil {
223+
if err := labelImages(ctx, client, images, filepath.Base(filePath)); err != nil {
218224
return errors.Wrap(err, "failed to add pinned label to images")
219225
}
220226
if err := retagImages(ctx, client, images, cfg.AgentConfig.AirgapExtraRegistry); err != nil {
@@ -265,7 +271,7 @@ func clearLabels(ctx context.Context, client *containerd.Client) error {
265271

266272
// labelImages adds labels to the listed images, indicating that they
267273
// are pinned by k3s and should not be pruned.
268-
func labelImages(ctx context.Context, client *containerd.Client, images []images.Image) error {
274+
func labelImages(ctx context.Context, client *containerd.Client, images []images.Image, fileName string) error {
269275
var errs []error
270276
imageService := client.ImageService()
271277
for i, image := range images {
@@ -277,6 +283,7 @@ func labelImages(ctx context.Context, client *containerd.Client, images []images
277283
if image.Labels == nil {
278284
image.Labels = map[string]string{}
279285
}
286+
280287
image.Labels[k3sPinnedImageLabelKey] = k3sPinnedImageLabelValue
281288
image.Labels[labels.PinnedImageLabelKey] = labels.PinnedImageLabelValue
282289
updatedImage, err := imageService.Update(ctx, image, "labels")
@@ -354,6 +361,16 @@ func prePullImages(ctx context.Context, client *containerd.Client, imageClient r
354361
for scanner.Scan() {
355362
name := strings.TrimSpace(scanner.Text())
356363

364+
if name == "" {
365+
continue
366+
}
367+
368+
// the options in the reference.ParseReference are for filtering only strings that cannot be seen as a possible image
369+
if _, err := reference.ParseReference(name, reference.WeakValidation, reference.Insecure); err != nil {
370+
logrus.Errorf("Failed to parse image reference %q: %v", name, err)
371+
continue
372+
}
373+
357374
if status, err := imageClient.ImageStatus(ctx, &runtimeapi.ImageStatusRequest{
358375
Image: &runtimeapi.ImageSpec{
359376
Image: name,

0 commit comments

Comments
 (0)