Blog post about blobs 0.90.0, setting expectations and referencing the iroh 0.90.0 blog post #335
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Floris Bruynooghe <[email protected]>
Co-authored-by: Floris Bruynooghe <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
matheus23
reviewed
Jul 11, 2025
|
|
||
| # API changes | ||
|
|
||
| The most notable change for an user of the crate is the API. In the old blobs, there were two levels of API. There was [very low level](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/store/trait.Store.html) layer that was possible to use only in-process and that had the lowest overhead, and a [more friendly API](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/rpc/client/blobs/index.html) that was feature gated with the `rpc` feature and used the [quic-rpc] crate. |
There was a problem hiding this comment.
Suggested change
| The most notable change for an user of the crate is the API. In the old blobs, there were two levels of API. There was [very low level](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/store/trait.Store.html) layer that was possible to use only in-process and that had the lowest overhead, and a [more friendly API](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/rpc/client/blobs/index.html) that was feature gated with the `rpc` feature and used the [quic-rpc] crate. | |
| The most notable change for a user of the crate is the API. In the old blobs, there were two levels of API. There was a [very low level](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/store/trait.Store.html) layer that was possible to use only in-process and that had the lowest overhead, and a [more friendly API](https://docs.rs/iroh-blobs/0.35.0/iroh_blobs/rpc/client/blobs/index.html) that was feature gated with the `rpc` feature and used the [quic-rpc] crate. |
| export const post = { | ||
| draft: false, | ||
| author: 'Rüdiger Klaehn', | ||
| date: '2025-07-01', |
There was a problem hiding this comment.
Remember to adjust the final publishing date
|
|
||
| In order to make the API easy to use, it is grouped into something similar to namespaces. There is a sub-API dealing with [tags](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/tags/index.html), with [remote](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/remote/index.html) operations, with complex [downloads](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/index.html) from multiple peers, with individual [blobs](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/index.html), and with the [blob store](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/struct.Store.html) as a whole. | ||
|
|
||
| These different namespaces are all basically newtype wrappers around an irpc client. They exist solely as a way to structure the API so we don't have to have a giant api with lots of fns with prefixes. |
There was a problem hiding this comment.
Suggested change
| These different namespaces are all basically newtype wrappers around an irpc client. They exist solely as a way to structure the API so we don't have to have a giant api with lots of fns with prefixes. | |
| These different namespaces are all basically newtype wrappers around an irpc client. They exist solely as a way to structure the API so we don't have to have a giant API with lots of fns with prefixes. |
Comment on lines
+98
to
+100
| To make complex requests easier to build, there are now builders for both `Get` and `GetMany` requests. There are also [extensions](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/index.html#reexport.ChunkRangesExt) to make working with [`ChunkRanges`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/type.ChunkRanges.html) more easy. | ||
|
|
||
| Various examples how to use the `Get` and `GetMany` request builders for complex requests are provided in the [`protocol module docs`](https://docs.rs/iroh-blobs/latest/iroh_blobs/protocol/index.html). |
There was a problem hiding this comment.
Suggested change
| To make complex requests easier to build, there are now builders for both `Get` and `GetMany` requests. There are also [extensions](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/index.html#reexport.ChunkRangesExt) to make working with [`ChunkRanges`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/type.ChunkRanges.html) more easy. | |
| Various examples how to use the `Get` and `GetMany` request builders for complex requests are provided in the [`protocol module docs`](https://docs.rs/iroh-blobs/latest/iroh_blobs/protocol/index.html). | |
| To make complex requests easier to build, there are now builders for both `Get` and `GetMany` requests. There are also [extensions](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/index.html#reexport.ChunkRangesExt) to make working with [`ChunkRanges`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/type.ChunkRanges.html) easier. | |
| Various examples of how to use the `Get` and `GetMany` request builders for complex requests are provided in the [`protocol module docs`](https://docs.rs/iroh-blobs/latest/iroh_blobs/protocol/index.html). |
|
|
||
| For the case where you want to get several blobs in a single request, but don't have a sequence of these hashes on the provider side, there is a new request type [`GetMany`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/protocol/struct.GetManyRequest.html). This allows you to just specify a set of hashes, where for each hash in the set you can specify ranges you want to download. | ||
|
|
||
| `GetMany` is useful when dealing with a large number of small blobs. If you want to just download a few large blobs, running multile Get requests in parallel is completely fine because QUIC has very cheap independent streams. |
There was a problem hiding this comment.
Suggested change
| `GetMany` is useful when dealing with a large number of small blobs. If you want to just download a few large blobs, running multile Get requests in parallel is completely fine because QUIC has very cheap independent streams. | |
| `GetMany` is useful when dealing with a large number of small blobs. If you want to just download a few large blobs, running multiple Get requests in parallel is completely fine because QUIC has very cheap independent streams. |
|
|
||
| #### ContentDiscovery | ||
|
|
||
| The ContentDiscovery trait has a single fn [`find_providers`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/trait.ContentDiscovery.html#tymethod.find_providers) to return a stream of providers. This can be either a finite stream, in which case the downloader will try each node in sequence and give up if the request can not be completed, or an infinite stream of possibly repeated node ids, in which case the downloader will try until success, or until the [`DownloadProgress`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/struct.DownloadProgress.html) object which acts as a handle for the request is dropped. |
There was a problem hiding this comment.
Suggested change
| The ContentDiscovery trait has a single fn [`find_providers`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/trait.ContentDiscovery.html#tymethod.find_providers) to return a stream of providers. This can be either a finite stream, in which case the downloader will try each node in sequence and give up if the request can not be completed, or an infinite stream of possibly repeated node ids, in which case the downloader will try until success, or until the [`DownloadProgress`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/struct.DownloadProgress.html) object which acts as a handle for the request is dropped. | |
| The ContentDiscovery trait has a single fn [`find_providers`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/trait.ContentDiscovery.html#tymethod.find_providers) to return a stream of providers. This can be either a finite stream, in which case the downloader will try each node in sequence and give up if the request cannot be completed, or an infinite stream of possibly repeated node ids, in which case the downloader will try until success, or until the [`DownloadProgress`](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/downloader/struct.DownloadProgress.html) object which acts as a handle for the request is dropped. |
|
|
||
| But none of the hooks that exist now will be removed. If anything, there will be more fine grained control before 1.0. | ||
|
|
||
| ## Batch add vs non-batch add. |
There was a problem hiding this comment.
Suggested change
| ## Batch add vs non-batch add. | |
| ## Batch add vs non-batch add |
Comment on lines
+220
to
+222
| When adding data without a batch, the default behaviour will be to create a *persistent* tag for every add operation. This means that your data is safe, but it can also lead to a large number of tags being created. | ||
|
|
||
| You can customize the behaviour by using different functions on AddProgress, such as [assigning a named tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.with_named_tag) or opting out of tag creation with [temp_tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.temp_tag). |
There was a problem hiding this comment.
Suggested change
| When adding data without a batch, the default behaviour will be to create a *persistent* tag for every add operation. This means that your data is safe, but it can also lead to a large number of tags being created. | |
| You can customize the behaviour by using different functions on AddProgress, such as [assigning a named tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.with_named_tag) or opting out of tag creation with [temp_tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.temp_tag). | |
| When adding data without a batch, the default behavior will be to create a *persistent* tag for every add operation. This means that your data is safe, but it can also lead to a large number of tags being created. | |
| You can customize the behavior by using different functions on AddProgress, such as [assigning a named tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.with_named_tag) or opting out of tag creation with [temp_tag](https://docs.rs/iroh-blobs/0.90.0/iroh_blobs/api/blobs/struct.AddProgress.html#method.temp_tag). |
|
|
||
| This makes it extremely flexible in terms of how its internals can look like. E.g. we are thinking about having an implementation of a file system based blob store using [io-uring](https://en.wikipedia.org/wiki/Io_uring) which would not use tokio at all for IO. | ||
|
|
||
| One downside is that it is harder to implement a fully featured store from scratch that behaves like the current store but e.g. stores data on S3. We will probably add a store implementation that leaves the behaviour of an individual entry/blob customizable via traits while implementing all the boilerplate for managing tags and garbage collection. |
There was a problem hiding this comment.
Suggested change
| One downside is that it is harder to implement a fully featured store from scratch that behaves like the current store but e.g. stores data on S3. We will probably add a store implementation that leaves the behaviour of an individual entry/blob customizable via traits while implementing all the boilerplate for managing tags and garbage collection. | |
| One downside is that it is harder to implement a fully featured store from scratch that behaves like the current store but e.g. stores data on S3. We will probably add a store implementation that leaves the behavior of an individual entry/blob customizable via traits while implementing all the boilerplate for managing tags and garbage collection. |
|
|
||
| There might be a *single* breaking change coming to the blobs protocol itself that would require changing the ALPN, before blobs 1.0. I have not yet decided if this is worthwhile. | ||
|
|
||
| The blob store format is compatible with the old iroh-blobs. You can open an 0.35 fs store without any migration. However, the new blobs will use one additional file per blob to keep track of the bitfield of available data. |
There was a problem hiding this comment.
Suggested change
| The blob store format is compatible with the old iroh-blobs. You can open an 0.35 fs store without any migration. However, the new blobs will use one additional file per blob to keep track of the bitfield of available data. | |
| The blob store format is compatible with the old iroh-blobs. You can open a 0.35 fs store without any migration. However, the new blobs will use one additional file per blob to keep track of the bitfield of available data. |
|
This has been published as 2 separate posts. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is quite a lot of text. Not sure if it is too much.