refactor: schema 4 with extensible systems and APIs #4

lidel · 2025-07-28T23:35:55Z

This aims to incorporate feedback/discussions from #1 in a way that that does not require clients to hardcode any new mapping between HTTP endpoints and features.

This format is more open-ended, can serve as a discovery mechanism that allows IPFS Mainnet to deploy novel routing systems (e.g., new DHT variants, blockchain-based routing) immediately through well-known HTTP endpoints while implementations gradually add native support

cc @aschmahmann for feedback

BREAKING CHANGE: Schema v4 introduces system-centric configuration

Rationale:

Moves from static profile-based config to dynamic system registry
Enables gradual ecosystem evolution without breaking existing clients
Allows new routing systems to be introduced via HTTP adapters
Introduces more clear separation between native and delegated routing

Highlights:

SystemRegistry defines well known routing systems with capabilities
Delegated HTTP endpoints declare which systems and APIs they provide
Clients ignore unknown APIs (not entire systems) for extensibility
Profiles become optional hints rather than rigid configurations
API versioning moved to endpoint level for flexibility

BREAKING CHANGE: Schema v4 introduces system-centric configuration Rationale: - Moves from static profile-based config to dynamic system registry - Enables gradual ecosystem evolution without breaking existing clients - Allows new routing systems to be introduced via HTTP adapters - Introduces more clear separation between native and delegated routing Highlights: - SystemRegistry defines well known routing systems with capabilities - Delegated HTTP endpoints declare which systems and APIs they provide - Clients ignore unknown APIs (not entire systems) for extensibility - Profiles become optional hints rather than rigid configurations - API versioning moved to endpoint level for flexibility This format serves as a discovery mechanism that bridges the gap between innovation and compatibility in the IPFS ecosystem. It allows IPFS Mainnet to deploy novel routing systems (e.g., new DHT variants, blockchain-based routing) immediately through HTTP endpoints while implementations gradually add native support

aschmahmann · 2025-07-29T17:31:09Z

autoconfig.json

+      "PreferDelegated": ["AminoDHT", "IPNI", "IPNS", "DNS"]
+    },
+    "daemon": {
+      "PreferNative": ["AminoDHT", "IPNS", "DNS"],


DNS seems a little awkward here in how it plays interplays with the DNS resolvers

This way Kubo can ignore https://delegated-ipfs.dev/dns-query because it has own native DNS from OS, and still apply DNSResolvers for special TLDs.

We could remove it from autoconfig, but then Verified Fetch and inbrowser.link will have to hardcode https://delegated-ipfs.dev/dns-query or https://1.1.1.1/dns-query as the default resolver for ICANN TLDs.

For the sake of simplicity, I will remove DNS system.
Implementations will have to hardcode their own "defualt DNS resolver".

Removed: 6f83bc3

aschmahmann · 2025-07-29T17:32:25Z

autoconfig.json

+    },
+    "daemon": {
+      "PreferNative": ["AminoDHT", "IPNS", "DNS"],
+      "PreferDelegated": ["IPNI"]


I suppose it doesn't matter either way, but does it really prefer delegated, or is there just not really a native option today (yes, there's a custom API but not with a ton of added value) so we'd be forced to fallback to delegated anyway.

Good question.

IPNI's "native" is HTTP as well, so its mostly semantics?

The way I understand it, everyone is free to adopt IPNI's "native" APIs which (afak) are https://cid.contact/cid/ and https://cid.contact/multihash/. For example, lassie uses /multihash one 🤷

But for IPFS clients its more cost-effective to use /routing/v1 rather than use custom HTTP query code that is specific to IPNI without any benefit in doing so. Thus, recommend "delegated" /routing/v1.

aschmahmann · 2025-07-29T17:41:46Z

autoconfig.json

+        "Write": []
+      }
+    },
+    "IPNS": {


IPNS as a system name seems awkward. We don't have a "providers" system or a "peers" system these are just endpoints on the delegated routing endpoint.

The only thing that really comes to mind is defining IPNS as using AminoDHT and perhaps down the road other systems like IPNI, or IPNS over PubSub (and the associated systems that handle provider records) but this seems like overkill at the moment.

What do we gain from this system being here compared to something like one of:

Requiring the implementers to know that for IPNS they can leverage the systems they have natively, use delegation, or layer something like IPNS over PubSub on top of the provider record systems (or delegations) they have

Adding to the system registry the types of methods supported (even if there's no delegation option for them)

Ack. Right now its just DHT.

Simplified in d811a45: merged "IPNS" into "AminoDHT" for now. It should be acceptable, given how public good delegated-ipfs.dev/routing/v1/ipns works.

aschmahmann · 2025-07-29T17:49:19Z

autoconfig.json

-    "mainnet-for-ipns-publishers-with-http": [
-      "https://delegated-ipfs.dev/routing/v1/ipns"
-    ]
+  "Profiles": {


Who are the "profiles" here meant to be seen by / interacted with that it's abstracting complexity? For example:

End users where the implementers have given them flexibility to choose their profile

Implementers who want an easy way to figure out what to enable / disable for their implementation

e.g. does the browser profile really want to be here vs in some defaults file chosen by the implementers and that can be different whether the platform is nodejs vs browser?

FWIW I do like the concept of the profiles and suggestions for implementers my question is if this seems like it'll be valuable to implementers spec'd in the file vs as some prose next to it / on the website.

In my mind it was always closer to (2) – implementers. In places like Kubo, end user would never use these, but have something that implementers created.

To make this more clear, I've removed Profiles from JSON and instead put dome docs in index.html (9746ad1). We can refine examples in HTML later.

aschmahmann · 2025-07-29T17:51:53Z

autoconfig.json

+        "Write": []
+      }
+    },
+    "Example": {


It's a shame JSON doesn't have a good way to add comments, but for clarity here IIUC the idea is meant to force implementers to gracefully handle systems they can't understand / don't know what to do with since the delegated endpoints aren't understood. Right?

Yes. Clarified that in SystemRegistry.Example.Description (753001e)

#4 (comment)

moving to docs in html #4 (comment)

#4 (comment)

aschmahmann

@lidel thanks for the work here, this looks pretty good to me. Looking forward to seeing how this plays out in implementation and seeing if we missed anything (hopefully not 🤞).

aschmahmann · 2025-07-29T20:42:04Z

autoconfig.json

+  "AutoConfigSchema": 4,
+  "CacheTTL": 86400,
+  "SystemRegistry": {
+    "AminoDHT": {


As I think I've flagged elsewhere (but putting here so we can find it in the future since this PR is likely to merge into master rather than #1) requiring just names for the AminoDHT might make it harder for some kinds of custom DHT configuration. However, this use case seems sufficiently esoteric that we can probably wait for a user request before bumping the config version to handle it.

changes autoconfig.json format to one from ipshipyard/conf.ipfs-mainnet.org#4 at v4

lidel · 2025-07-30T16:37:11Z

Ok, I think everything is resolved, let's merge this and switch ipfs/kubo#10883 to use this version.

lidel mentioned this pull request Jul 28, 2025

feat: switch config to orient routing around the routing system names #1

Closed

lidel requested a review from aschmahmann July 28, 2025 23:52

aschmahmann reviewed Jul 29, 2025

View reviewed changes

lidel added 4 commits July 29, 2025 21:44

chore: remove DNS system

6f83bc3

#4 (comment)

chore: merge IPNS system into AminoDHT for now

d811a45

#4 (comment)

chore: remove Profiles from JSON

9746ad1

moving to docs in html #4 (comment)

docs: description of Example system

753001e

#4 (comment)

aschmahmann approved these changes Jul 29, 2025

View reviewed changes

lidel added a commit to ipfs/kubo that referenced this pull request Jul 30, 2025

refactor: schema v4

24d552d

changes autoconfig.json format to one from ipshipyard/conf.ipfs-mainnet.org#4 at v4

docs: cleanup system type examples

eec90be

lidel merged commit 7a451de into main Jul 30, 2025

lidel deleted the schema-4-extensible-systems-and-apis branch July 30, 2025 16:40

refactor: schema 4 with extensible systems and APIs #4

refactor: schema 4 with extensible systems and APIs #4

Uh oh!

Conversation

lidel commented Jul 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aschmahmann left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel commented Jul 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lidel Jul 29, 2025 •

edited

Loading

lidel Jul 29, 2025 •

edited

Loading

lidel Jul 29, 2025 •

edited

Loading