ipfs: add support to pull images by ref #1968

wswsmao · 2025-02-08T03:23:36Z

The solution for issue #1944 has been finalized, with the main design concepts outlined as follows:

IPNS
IPNS provides the ipfs name publish and ipfs name resolve commands, which allow for associating a file with an IPNS name, enabling publishing and retrieval.
For IPFS images, we can consider publishing the root descriptor file, allowing other nodes to resolve this file to pull the image. IPNS requires key management to control the publishing and resolution of IPNS names, with each key corresponding to a unique IPNS name.
This characteristic is well-suited for associating with image references, ultimately establishing the following relationship:
Image ref -> Key -> Root Descriptor -> Image File
Deterministic Keys
Different nodes will share the same deterministic key algorithm to ensure that each node receives the same key for the same image name. This allows every node to pull and push images. Specifically:
- Pull Phase: The corresponding image ref can be resolved using just the public key ID.
- Push Phase: The private key must first be imported into the local IPFS. After uploading the image, the updated root descriptor file will be published.

Additional Notes
This submission is compatible with the existing method. During the push / pull phase, the input will be checked to determine if it is a ref. If users still prefer to use CID, this method will remain valid.

Feature Demonstration
On Node A:

./ctr-remote i ipfs-push --estargz=false docker.io/abushwang/oc9-busybox:org
or
./ctr-remote i ipfs-push --estargz=true docker.io/abushwang/oc9-busybox:esgz

On other nodes:

./ctr-remote i rpull --snapshotter=overlayfs --ipfs docker.io/abushwang/oc9-busybox:org
or
./ctr-remote i rpull --snapshotter=stargz --ipfs docker.io/abushwang/oc9-busybox:esgz

Other nodes can also rebuild docker.io/abushwang/oc9-busybox:org, and then run:

./ctr-remote i ipfs-push --estargz=false docker.io/abushwang/oc9-busybox:org

At this point, Node A can pull the updated image:

./ctr-remote i rpull --snapshotter=overlayfs --ipfs docker.io/abushwang/oc9-busybox:org

AkihiroSuda · 2025-02-12T04:21:12Z

cmd/go.mod

@@ -49,6 +49,7 @@ require (
 	github.com/containernetworking/plugins v1.5.1 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect


Can we remove this dependency on "Decred" cryptocurrency to avoid potential confusion?

This might be a bit difficult. 'Decred' is introduced by the following two packages:

github.com/libp2p/go-libp2p/core/crypto github.com/libp2p/go-libp2p/core/peer

These two packages are used to compute the public key and peer ID in IPFS.
The Kubo project also introduces 'Decred':
https://github.com/ipfs/kubo/blob/master/go.mod#L119

Can we just exec the ipfs command for computing them?

Actually we got a compliant about a dependency on a bitcoin-originated library in the past, although it was not for mining nor trading bitcoin

go.mod: add a comment about indirect dependency on github.com/btcsuite/btcd/btcec (NOT bitcoin daemon itself) nerdctl#1573

Let me think. Do you mean that in ipnskey.go's (g *DetKeyGen) generateKey(name string), after we obtain the seed bytes for the deterministic keys through reader.Read(seedBytes), we execute the IPFS command using an OS command (for example, exec.Command) to calculate the public key and peer ID?

Oh, actually, there's no need to use the IPFS command; we can just use the API. We can import the private key into IPFS and then use the list API to obtain the public key ID and peer ID. I'll update the PR.

ipfs/converter.go

ktock · 2025-02-12T12:00:28Z

cmd/go.mod

@@ -49,6 +49,7 @@ require (
 	github.com/containernetworking/plugins v1.5.1 // indirect
 	github.com/cpuguy83/go-md2man/v2 v2.0.5 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect


This should be removed.

done
Refer to: #1968 (comment)

ktock · 2025-02-12T12:31:02Z

ipfs/ipnskey/ipnskey.go

+
+// KeyStore manages a collection of DetKeyGen instances
+type KeyStore struct {
+	store map[string]*DetKeyGen


Is this tolerant to restart?

KeyStore has been removed now

ipfs/client/client.go

ktock · 2025-02-12T12:34:10Z

ipfs/client/client.go

+	if err := json.NewDecoder(resp.Body).Decode(&rs); err != nil {
+		return "", err
+	}
+
+	parts := strings.Split(rs.Path, "/")
+	if len(parts) < 3 || parts[1] != "ipfs" {
+		return "", fmt.Errorf("invalid resolved path format: %s", rs.Path)
+	}
+
+	return parts[2], nil


done, add doc to docs/ipfs.md

can you add a comment of link to ipfs api docs?

ktock · 2025-02-12T12:35:40Z

ipfs/ipnskey/ipnskey.go

+package ipnskey
+
+import (
+	"bytes"
+	"crypto/ed25519"
+	"crypto/sha256"
+	"crypto/x509"
+	"encoding/pem"
+	"fmt"
+	"sync"
+
+	"github.com/libp2p/go-libp2p/core/crypto"
+	"github.com/libp2p/go-libp2p/core/peer"


This shouldn't be imported to stargz snapshotter but should be done in ipfs daemon.

done
Refer to: #1968 (comment)

wswsmao · 2025-02-13T11:58:34Z

@AkihiroSuda @ktock
The PR has been updated with the following changes:

Removed unnecessary dependencies: The generated key is now imported into IPFS, and the peer ID can be retrieved later using IPFS's list API, thus avoiding redundant dependencies.
Since the peer ID and related information are stored in IPFS, there is no longer a need for the KeyStore hash table to record the ref and peer ID; this can be accomplished directly through the list API.
Updated the docs/ipfs.md documentation.

ktock

Can you add integration tests?

ktock · 2025-02-13T12:36:19Z

cmd/go.mod

-	github.com/ipfs/go-cid v0.1.0 // indirect
+	github.com/ipfs/go-cid v0.4.1 // indirect


no need update to v0.4.1
done

ktock · 2025-02-13T12:36:56Z

ipfs/client/client.go

+	if err := json.NewDecoder(resp.Body).Decode(&rs); err != nil {
+		return "", err
+	}
+
+	parts := strings.Split(rs.Path, "/")
+	if len(parts) < 3 || parts[1] != "ipfs" {
+		return "", fmt.Errorf("invalid resolved path format: %s", rs.Path)
+	}
+
+	return parts[2], nil


can you add a comment of link to ipfs api docs?

ktock · 2025-02-13T12:37:27Z

ipfs/ipnskey/ipnskey.go

+   limitations under the License.
+*/
+
+package ipnskey


needs explanation comments and link to docs

wswsmao · 2025-02-14T08:57:13Z

Hi, @ktock
The PR has been updated with the following changes:

Add integration tests in script/integration/containerd/entrypoint.sh
Add comments of link to ipfs api docs for package ipnskey, Resolve, Publish and importKey

ktock · 2025-02-20T07:12:45Z

ipfs/client/client.go

+		client = http.DefaultClient
+	}
+
+	ipfsAPINamePublish := c.Address + "/api/v0/name/publish"


Can we have docs about lifetime and republishing of contents?

done, https://github.com/containerd/stargz-snapshotter/pull/1968/files#diff-0625292be4242bc6994d3dfc768c9a02f37e1c43a59848e38d8464e3228c128fR174

ktock · 2025-02-20T07:14:01Z

ipfs/ipnskey/ipnskey.go

+   limitations under the License.
+*/
+
+// Package ipnskey provides functionality for generating deterministic Ed25519 key pairs


Can't we use the gen API for this? https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-key-gen

@wswsmao can you check this ^

oh, I got a little trouble, check done

So this sounds like all users on the IPFS network can get the private key of an image ref and update and sign the image. How can a user verify the image is updated by a trusted one, just like the default IPNS settings?

When using IPFS, we have configured a IPFS private network by default, where the nodes within this private network are considered trusted. Additionally, nodes outside of the private network, even if they obtain the private key, will not be able to publish or resolve image files.

Signed-off-by: abushwang <[email protected]>

wswsmao · 2025-02-21T02:19:27Z

all review done @ktock

wswsmao force-pushed the ipfs branch 7 times, most recently from 20d3c42 to bfee105 Compare February 8, 2025 08:00

AkihiroSuda reviewed Feb 12, 2025

View reviewed changes

wswsmao force-pushed the ipfs branch 2 times, most recently from 6b511ab to 3545af1 Compare February 12, 2025 07:01

ktock requested changes Feb 12, 2025

View reviewed changes

wswsmao force-pushed the ipfs branch 3 times, most recently from b7a87c0 to 6c36913 Compare February 13, 2025 11:46

ktock reviewed Feb 13, 2025

View reviewed changes

wswsmao force-pushed the ipfs branch 4 times, most recently from 1eb20d7 to a015bb6 Compare February 14, 2025 08:29

wswsmao requested a review from ktock February 19, 2025 11:16

ktock reviewed Feb 20, 2025

View reviewed changes

ipfs: add support to pull images by ref

2b15ed2

Signed-off-by: abushwang <[email protected]>

wswsmao force-pushed the ipfs branch from a015bb6 to 2b15ed2 Compare February 20, 2025 07:41

wswsmao requested a review from ktock February 20, 2025 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ipfs: add support to pull images by ref #1968

ipfs: add support to pull images by ref #1968

wswsmao commented Feb 8, 2025 •

edited

Loading

AkihiroSuda Feb 12, 2025

wswsmao Feb 12, 2025

AkihiroSuda Feb 12, 2025

AkihiroSuda Feb 12, 2025

AkihiroSuda Feb 12, 2025

wswsmao Feb 12, 2025

wswsmao Feb 12, 2025

wswsmao Feb 13, 2025

ktock Feb 12, 2025

wswsmao Feb 13, 2025 •

edited

Loading

ktock Feb 12, 2025

wswsmao Feb 13, 2025

ktock Feb 12, 2025

wswsmao Feb 13, 2025

ktock Feb 13, 2025

wswsmao Feb 14, 2025

ktock Feb 12, 2025

wswsmao Feb 13, 2025 •

edited

Loading

wswsmao commented Feb 13, 2025 •

edited

Loading

ktock left a comment

ktock Feb 13, 2025

wswsmao Feb 14, 2025 •

edited

Loading

ktock Feb 13, 2025

ktock Feb 13, 2025

wswsmao Feb 14, 2025

wswsmao commented Feb 14, 2025 •

edited

Loading

ktock Feb 20, 2025

wswsmao Feb 20, 2025 •

edited

Loading

ktock Feb 20, 2025

ktock Feb 20, 2025 •

edited

Loading

wswsmao Feb 20, 2025

ktock Feb 21, 2025

wswsmao Feb 24, 2025

wswsmao commented Feb 21, 2025

		github.com/ipfs/go-cid v0.1.0 // indirect
		github.com/ipfs/go-cid v0.4.1 // indirect

ipfs: add support to pull images by ref #1968

Are you sure you want to change the base?

ipfs: add support to pull images by ref #1968

Conversation

wswsmao commented Feb 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wswsmao Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wswsmao Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

wswsmao commented Feb 13, 2025 • edited Loading

ktock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wswsmao Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wswsmao commented Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

wswsmao Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktock Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wswsmao commented Feb 21, 2025

wswsmao commented Feb 8, 2025 •

edited

Loading

wswsmao Feb 13, 2025 •

edited

Loading

wswsmao Feb 13, 2025 •

edited

Loading

wswsmao commented Feb 13, 2025 •

edited

Loading

wswsmao Feb 14, 2025 •

edited

Loading

wswsmao commented Feb 14, 2025 •

edited

Loading

wswsmao Feb 20, 2025 •

edited

Loading

ktock Feb 20, 2025 •

edited

Loading