iroh-blobs v0.90 - The Upgrade Guide
by rklaehnThe new iroh-blobs
version 0.90
is not just a refactor. It is a complete rewrite of the file system blob store and in-memory blob store as well as a redesign of the API.
If you enjoy being on the bleeding edge and want to try out some of the new features, use 0.9x
. If you prefer stability, use the 0.35.x
series until we release 1.0
towards the end of the year.
For some context about our planned 1.0
release of iroh
, iroh-blobs
, and iroh-gossip
, see iroh v0.90 - The Canary Series đ„
API Changes
The most notable change in this new version of iroh-blobs
is the API. In the old blobs, there were two levels of API. The first was a very low level API that had lower overhead but was only possible to use in-process. The second was a more friendly API that was feature gated with the rpc
feature and used the quic-rpc crate.
quic-rpc
is quite fast when used in-process, but it does not have zero overhead. For every RPC request, no matter how small, there would be some additional allocations than would occur in a purely in-process design. For this reason, we still exposed the low-level API for extremely performance critical use cases. But this caused quite a bit of confusion, especially since the rpc
feature was not enabled by default for a long time.
In the new iroh-blobs
, we are using the irpc crate for RPC. irpc
is designed so that the in-process case has zero additional overhead over what you would normally do in rust if you need an async boundary (isolation via tokio
oneshot
and mpsc
channels). This is why we found it justified to have just one API - itâs fast if used in-memory and still able to cover cross-process or cross-machine RPC via quinn
connections.
There will be another blog post soon about the design of irpc
, but the TLDR is that it is just a thin wrapper around either tokio
(oneshot
or mpsc
) channels or quinn
connections. It does not attempt to fully abstract the connection type. At this time it only works for streams from the iroh-quinn crate, so when usingirpc
for cross-process or cross-machine RPC, the user will work with QUIC connections, created by either iroh
âs holepunching endpoints or iroh-quinn
non-holepunching endpoints. Dialing down the level of abstraction allowed us to optimize away the allocations that quic-rpc
had to do, while also getting rid of some quite unergonomic type parameters that infected code bases using quic-rpc
.
General API design
The iroh-blobs
API is quite complex due to the fact that you can interact not just with entire blobs, but also with ranges of them. You also need to express which blobs you want to keep permanently and which you are okay with getting garbage collected.
In order to make the API easy to use, there are sub-APIs that are grouped around specific concepts or namespaces. There is a sub-API dealing with tags, with remote operations, with complex downloads from multiple peers, with individual blobs, and with the blob store as a whole.
These different namespaces are all basically newtype wrappers around an irpc
client. They exist solely as a way to structure the API so we don't have to have a giant API with lots of methods with prefixes.
Progress
In many cases, the API dealing with blobs has the following problem: any operation on blobs will take significant time if you are dealing with large (GiB) blobs, but will be basically instantaneous when dealing with tiny (KiB) blobs. So we want the API to be pleasant to use if you just want to add a bit of data and see the hash, but also expressive enough to provide detailed progress for the operation in case you are adding a 100 GiB disk image or ML model.
To cover both use cases, every operation that isn't guaranteed to be constant time synchronously returns a ...Progress
struct (for example AddProgress
, DownloadProgress
, etc) which is a wrapper around a stream of progress events.
The progress struct implements IntoFuture for the case where you don't care about the progress events and just want to await
the final result (success or failure).
Each progress struct also provides a stream
method that allows you to convert it into the underlying stream and deal with the progress events one by one. Using this API, you can use the emitted event to, for example, feed a progress bar.
In addition, the ...Progress
struct will sometimes contains additional helper methods for common use cases.
Progress events will, in most cases, have two enum cases for successful and unsuccessful termination, in addition to events that contain information about the progress of the operation.
As an example of this pattern, AddProgress is returned from all operations that add data to the blob store, and has a stream method as well as an IntoFuture
implementation.
AddProgressItem
contains detailed information about the different stages of adding data to a blob store, which you can either use to provide very detailed progress information, or just ignore when using IntoFuture
.
Here is an example using the add_path
method illustrating how to do both:
// You don't care about progress events and just want to wait
// until the data is added to the blob store:
let tag_info = blobs.add_path("my/cool/data").await?;
// You want to track the progress:
let add_progress = blobs.add_path("my/cool/data");
let mut stream = add_progress.stream().await;
while Some(progress_item) = stream.next() {
match progress_item {
// process the item
}
}
Sendme contains a lot of examples on how to provide progress for adding, transferring, and exporting files from blobs.
Options
Many operations come with complex options. For example, operations for adding blobs often require a format
parameter to specify whether the blob being added is just a raw blob or a sequence of hashes. But in the vast majority of cases, users just want to add raw blobs. In other languages, you might solve this issue with either overloading or with default parameters. But rust has neither, for very good reasons.
So we have come up with the following pattern. For each operation there is a op_with_opts()
method which takes an ...Options
struct, such as AddPathOptions
or DownloadOptions
. This is always the method that most directly maps to the underlying RPC protocol (and in many cases the ...Options
struct is the RPC message!).
For convenience, there are methods to cover common use cases that delegate to the with_opts
methods. These overloads use rust tricks like impl Into<T>
to make them work with a wide variety of possible input types.
For example, when adding blobs, there is add_bytes_with_opts
method to add a Bytes
with an additional parameter to specify the format (BlobFormat::Raw
or BlobFormat::HashSeq
).
For convenience, there are also variants add_bytes
for adding anything that can be converted into a Bytes
and add_slice
to add anything that can be viewed as a slice.
The latter might have some overhead; if you add a Bytes
using add_slice
, a copy will be made. So if you have a giant Bytes
and want to add it to the store without a copy, use add_bytes
or add_bytes_with_opts
.
If you don't mind the verbosity and want to avoid overhead, the most direct way to use the API is always the ..._with_opts
methods.
Builders
Requests such as GetRequest
can be very simple (âjust give me the blobâ), but also very complex (âgive me this HashSeq
and the first and the last chunk of all its childrenâ).
To make complex requests easier to build, there are now builders for both Get
and GetMany
requests. There are also extensions to make working with ChunkRanges
easier.
Various examples on how to use the Get
and GetMany
request builders for complex requests are provided in the protocol module docs
.
If you look at the builder examples, you might wonder where RangeSpec and RangeSpecSeq went.
They both exist, but are no longer public, just an implementation detail of the wire protocol. What you interact with now is ChunkRanges and ChunkRangesSeq, which are hopefully less confusing.
Errors
Compared to the old blobs, we have vastly reduced the usage of anyhow
for errors. Instead we use the snafu
crate to provide concrete errors, with some additional features like backtraces and span traces.
Writing your own blob store
In iroh-blobs
0.35
stores were abstracted over two levels. At the low level, there was the store trait hierarchy. At the RPC level, there was a complex RPC protocol.
Two downsides of the trait hierarchy were that it was pretty confusing and that it baked in some assumptions about the exact implementation that might not always be true, e.g. IO futures being non-Send
.
So in the new blobs, the RPC protocol is the interface you have to implement to provide a new store implementation.
This makes it extremely flexible in terms of how its internals can look like. For instance, we are considering writing an implementation of a file-system based blob store using io-uring that would not use tokio
at all for IO.
One downside is that it is harder to implement a fully featured store from scratch that behaves like the current store but, for example, does something like store data on S3. To combat this, we will likely add a store implementation that allows the behaviour of an individual entry/blob to be customizable via traits, while still implementing all the boilerplate for managing tags and garbage collection.
Compatibility
The protocol for the Get
request is unchanged. You can do get requests from a node running the old (0.35
) iroh-blobs
to a node running the new blobs and vice versa.
There might be a single breaking change coming to the blobs protocol itself that would require changing the ALPN, before blobs 1.0
. We have not yet decided if this is worthwhile.
The blob store format is compatible with the old iroh-blobs
. You can open an 0.35
file store without any migration. However, the new blobs will use one additional file per blob that keeps track of the bitfield of available data. More on why this addition file tracking the bitfield of available data is worth it in the next blobs-focused blog post.
Performance
The old iroh-blobs
was already close to optimal when dealing with large files. Syncing a directory containing a few giant files is not going to get any faster using the new blobs (it might get faster due to optimisations we have done in iroh
connections, though).
Where there is a large improvement is when dealing with a large number of tiny blobs. So if you need to do something like sync a directory containing lots of small files, such as the linux kernel, you will see a noticeable performance improvement.
Here is the performance of old (iroh-blobs@0.35
based) sendme
syncing the linux kernel source code on localhost:
sendme receive 31.22s user 89.67s system 24% cpu 8:11.74 total
And here is the performance of new (iroh-blobs@0.90
based) sendme
doing the same thing:
cargo run --release receive 30.44s user 72.03s system 45% cpu 3:44.73 total
Note that you will only see a significant improvement if you are dealing with many small files and if you have a sufficiently fast connection that local-io speed matters!
Stability
This version of blobs has been thoroughly tested. Nevertheless, it is not yet fully production ready. Just like with iroh itself, iroh-blobs
0.90
is the start of the canary release series leading to blobs 1.0
.
Since the consequences of a bug in a blob store are more persistent than the consequences of a bug in a new iroh
connection, we would advise production deployments to wait for another 1 or 2 releases before switching to the new blobs. By that time, the API should also have stabilized further.
There will be several API changes as we work towards 1.0
. In particular the downloader API will grow and become more robust. The provider events API will also be refined.
Also, we will work on further refining the error types prior to freezing them for 1.0
.
To get started, take a look at our docs, dive directly into the code, or chat with us in our discord channel.