A new iroh-gossip release and the road to 1.0

iroh-gossip is a protocol to broadcast messages to a swarm of peers, without central coordination. It is one of the most used iroh protocols, and often forms the backbone for real-time communication between endpoints in apps.

Now that iroh 1.0 is out the door, we finally have more capacity to work on our protocols again. And today we released iroh-gossip version 0.102.0, which fixes many long-standing bugs and increases reliability and efficiency considerably.

This is also the last release on the current wire protocol. For the upcoming iroh-gossip 1.0 release, we have a couple of changes planned that break the wire protocol. The second half of this post describes our plans, to which we welcome your feedback!

So Many Fixes

While iroh-gossip has fared rather well in production already, we have had a number of long-standing bugs around connection lifecycle management, recovery, and dealing with slow or stalled peers in the swarm. We're happy to report that this release fixes many of those bugs and we're quite confident that this is the best iroh-gossip release yet :)

Slow Peers No Longer Block a Node

Before 0.102, a node had a queue of 64 messages per peer connection. When one of these queues became full because the peer wouldn't read fast enough, our main actor loop blocked on sending into the full queue. This means that a single slow or stalled peer stopped every topic and every new broadcast, potentially infinitely.

Now each peer has a size-limited queue, and pushing into this queue never waits. Instead, when a message does not fit, the node drops it for that peer and announces it again later. The peer can then fetch it while the message is still cached. Broadcasting waits only when the queues to all eager neighbors of the topic are half full, because then its own uplink is most likely the limit. (#185)

We measured this with patchbay, a tool for network simulations. We created a lab with 10 gossip nodes with some of them on slow 1 Mbit/s links. The publisher sends 1 KB messages for 60 s. Each cell in the table below is the median of three runs, as 0.101 → 0.102.

Scenario	Messages sent	Received per slow node	p99 latency of healthy nodes
1 slow node, 250 msg/s	5383 → 15001	3924 → 3547	20 s → 124 ms
8 slow nodes, 250 msg/s	4875 → 15001	4219 → 5684	36 s → 10 ms
1 slow node, 3 publishers on 2 topics	9496 → 15001	5757 → 4058	13 s → 235 ms
3 slow nodes, 100 msg/s	4621 → 6001	4092 → 3979	12 s → 20 ms

With 0.101, the slow nodes set the pace for everyone: the publisher could only send between 30 and 75% of its messages. With 0.102, the publisher keeps its full rate, and the healthy nodes keep their latency. The slow nodes receive about as many messages as before, because their link is the limit. But importantly they no longer block the publisher, and no longer block other fast nodes. That slow nodes sometimes receive a bit less than in 0.101 is related to the fact that their saturated link is doing a lot of duplicate grafts, which is not new, just slightly elevated by the higher publishing rate. This is something we want to improve on in the next release.

Correctness Fixes and Regression Tests

Both the Plumtree and HyParView papers report how their protocols behave in simulation, for example how fast a swarm recovers after half its nodes fail. We turned these claims into tests that run in our simulator. At first, 10 of these tests failed. We then went to fix a few issues, and now all of them pass on 50 random seeds. (#191)

• A node that lost all its neighbors did not look for new ones. It stayed alone until the app joined again. Now it joins again through the peers that the app joined with. It also tries one of its recent neighbors, in case those peers are gone. (#192)
• A node now looks for new neighbors before it has none left, also after a failed dial.
• The first shuffle fills the list of backup peers. It now runs one second after a join instead of after one minute.
• Late messages from a former neighbor no longer add it back to the tree.
• A link that a node adds to the tree after a slow message now stays, instead of being cut again right away. Before, the next message repeated the same churn. (#207)

Other Fixes

• Memory leaks. Several maps kept entries for peers that left, so the memory of long-running nodes grew without limit. (#161)
• Unused connections. A connection stayed open forever, even when no topic used the peer anymore. Now it closes ten seconds after neither side has a stream on it. (#178, #195, #206)
• Restarted peers. A peer that restarted and joined again quickly could get no reply to its join. (#161)
• Lost peers. When a connection to a peer closed before the peer became a neighbor, the node never dialed that peer again. (#178)
• Oversized messages. broadcast returned Ok for a message over the size limit. The node then dropped the message without notice. (#187)
• Shutdown. Shutting down the tokio runtime no longer makes the gossip actor panic. (#183)
• Fewer dials in a full swarm. A node with too few neighbors asked every full peer for a slot every five seconds, each time on a new connection. Now it asks at most three peers at a time, and waits longer after each round, up to one minute. (#196)
• Lost joins. A join that got lost on the way was not retried if the node had other neighbors. Now the node retries it once. (#217)

Swarms with Old and New Nodes

You can run 0.102 nodes and older nodes in the same swarm, and update one node at a time.

When testing things we first ran into a weird situation. Nodes up to 0.101 have a bug (#155): when the connection to a peer that is not their neighbor closes, they keep a dead handle for that peer. They lose all later messages to it, until the peer dials them again. Because 0.102 now properly closes unused connections, old nodes would hit this bug often.

We fixed it though. A 0.102 node now adds a flag to its stream headers. The flag says "I can handle closed connections", and old nodes ignore it. A new node closes an unused connection only after the peer sent this flag. Connections to old nodes stay open as before, so for them the leak stays until they update. (#189)

Thanks

Several of these fixes build on pull requests and issues from the community:

• @drlukeangel, @caiogondim, and @jgrund worked on the leaks (#145, #146, #147, #154).
• @adiibanez and @javruben reported and worked on slow peers (#47, #143).
• @TapGhoul found the dead handle bug (#155).
• @sigaloid fixed the chat example, so that it sends identical messages and shares nicknames with new peers (#200).

Thank you!

Next up: iroh-gossip 1.0

Our plan is to change the wire protocol in the next release, and to ship it as a canary for the upcoming iroh-gossip 1.0. We know that wire changes are hard for gossip users, because every node of a swarm has to update. But the most pressing features that users are asking for need wire-level breaking changes.

After 1.0, we want to stop changing the wire, and will design the new protocol in a way where we can add new features over time, something the current protocol hardly permits.

Here is what we have planned. These are all not set in stone yet, so if you have any feedback, concerns, different use cases or other ideas, please reply in the linked issues.

Authorized Topics (#179)

Today, anyone who knows the topic id and one online member can join a swarm. A secret topic id is the only access control. To remove someone, you have to move everyone to a new topic id.

We plan to let a node present a token when it opens a topic on a connection. The receiving node would pass the token to a verifier that the app provides, and the app would accept or reject the peer. The default verifier would accept everyone, so apps without this need wouldn't need to change at all.

iroh-gossip will treat the token as opaque bytes, so only the app decides what it means. It can be a shared secret or a signed capability, for example. Until the app accepts a peer, the node would send it only membership messages and no data. A node never forwards a token to other peers.

While structuring the tokens is up to the app, one rule will matter for all of them: you'll need to bind the token to the endpoint id of its holder. Otherwise, a peer that saw your token can use it as its own. We'll make sure to have some examples ready, both for shared secrets and for capability tokens.

One Broadcast Tree Per Sender (#180)

Today, all senders of a topic share one Plumtree tree. When a node gets a message twice, it makes the link of the duplicate lazy. This removes a cycle from the tree, and with one sender it works well and leads to optimal broadcast trees.

When several nodes send at the same time, their duplicates arrive on different links of the same cycle. The cycle then loses more than one link, and the tree splits. Most messages then arrive only after a lazy neighbor asks for them, and each request adds a timeout. In a simulation of 100 nodes, two concurrent senders reach half as many nodes through the tree as one sender. Five senders reach a tenth.

The Plumtree paper already recommends a tree per sender, and we want to implement this for 1.0. Every message would name the tree of its sender, and each node would keep its eager and lazy links per tree. In our prototype, a tree costs a node about 1.4 KB. Nodes would cap the number of trees, and drop the trees they have not used for a while. A dropped tree costs one flood of the next message of that sender. In a first test with four senders on one machine, the prototype moved about eight times more messages per second.

The implementation is rather straightforward, but this needs a wire breaking change to include the tree identifier in each message.

A Clear Model for Connections (#181)

Much of the work in 0.102 went into connections. The hardest case to reason about is a restarted peer, because the current protocol cannot tell it apart from a late connection of its previous run.

Our plan is a random session id that each node picks for each peer, and sends with every stream. A node would then see at once when a peer restarted, and drop the state of the old run. A node would send on one connection per peer. When any connection to the peer ends, both sides would end the session and clean up once, and the app would get one clear signal. The compatibility rules of 0.102 would go away, and every unused connection would close.

Migration and future extensibility

The upcoming releases will use a new protocol name (ALPN), so 0.102 and 1.0 nodes won't talk to each other. We don't plan to add backwards compatibility. This means that once you want to move to iroh-gossip 1.0, you will either have to update all your users at the same time, or ship both iroh-gossip 0.102 and 1.0 in your app for a while. While not ideal, this is the nature of upgrading gossip swarms. Be assured that this will be the last breaking change for a long time. We are designing the new wire protocol such that we can add extensions over time without breaking old clients.

We want your feedback on the design before we finish it. See the issues linked above, or this tracking issue.

Also - please try 0.102 and tell us how it works for you on GitHub or Discord!

The Changes in Detail

Breaking Changes

• The event and command enums in api and proto are now #[non_exhaustive]. Add a wildcard arm to your matches. (#198)
• metrics::Metrics is now #[non_exhaustive]. Create it with Metrics::default(), not with a struct literal. (#198)

Behavior Changes

These are not semver-breaking, but your app may notice them:

• A broadcast can wait, for at most 100 ms at a time, while the queues to all eager neighbors of its topic are half full. (#185)
• A full send queue drops data for that peer. A membership message that does not fit, or a write that stalls for 60 s, closes the connection. (#185)
• broadcast and broadcast_neighbors return ApiError::TooLarge for content over max_content_size. (#187)
• A connection closes ten seconds after neither side had a stream on it, including a connection that the app passed to Gossip::handle_connection. Connections to nodes up to 0.101 stay open. (#178, #189, #206)
• Gossip::handle_connection expects an accepted, server-side connection. (#185)
• A rejoin can dial a peer that the app did not name: one of the recent neighbors of the node. (#192)
• A join to a peer that is already a neighbor doesn't send a new message. An unanswered join is sent once more after ten seconds. (#217)

New Features

• #187 GossipSender::max_content_size and GossipTopic::max_content_size return the largest content size that a broadcast accepts. An RPC client asks its server once.
• #185 Builder::send_queue and QueueOptions set the size of the send queue to each peer. The new metrics msgs_dropped and peers_send_queue_full count the messages that a full queue dropped, and the connections that it closed.
• #192 A node with no neighbors and no backup peers also rejoins through one of its last eight neighbors.
• #191 proto::Ttl and proto::Round are now public. Four public config fields used these types before.

Fixes

• #185 A peer that read slowly, or not at all, stalled the gossip actor. Every topic and every other peer stopped with it.
• #186 A publisher with a command always ready starved the actor, and half a stream header blocked all streams of a connection.
• #161 State for departed peers leaked, and a topic could close a connection that another topic still used.
• #178 Unused connections never closed. A connection that closed before the peer became a neighbor left a dead handle, and later messages to that peer were lost.
• #189 A node closes unused connections only to peers that support it, so nodes up to 0.101 keep working.
• #195 A node also closes a connection on which neither side ever sent.
• #191 A node rejoins when it has no neighbors left, refills its neighbors before it runs out, and shuffles soon after a join.
• #187 broadcast of an oversized message returned Ok, and the node dropped the message later without notice.
• #183 Shutting down the tokio runtime made the actor panic.
• #201 The log shows a duplicate message with its short hash.
• #196 A node short of neighbors asked every full peer for a slot every five seconds. Now it asks at most three at a time, and backs off up to one minute.
• #206 A node could close a connection on which the peer still sent. A connection now closes only after neither side had a stream on it for ten seconds, and a message to an unused peer goes out on the existing connection.
• #206 A message for a topic that the node had left didn't get any answer. Now it answers with a Disconnect.
• #206 A Neighbor request could overtake the Disconnect sent before it. Now the streams of one topic are read in order.
• #207 A peer grafted after a slow message was often pruned again at once, and the next message repeated the graft. Now grafts are not pruned while pending.
• #215 The address lookup kept an unbounded number of addresses per peer. Now it keeps at most 32.
• #217 A lost join was not retried if the node had other neighbors. Now it is retried once. A join to a peer that is already a neighbor is no longer sent.

Performance

• #221 Removed some unecessary allocations.
• #222 Switched to more performant hash maps.

Docs

• #219 Improved the docs of the public items.