Use Case: AI/ML
Nous Research uses iroh to train foundation LLMs with compute distributed around the world, across AWS, GCP, Azure, and self-hosted infrastructure. Mesh LLM uses it to pool GPUs across machines into a single inference API. Neither needed a data center to do it.
Nous Research: Training
10x
Bandwidth reduction (10Gbps → 1Gbps)
50%
Cost reduction ($1M → $500K models)
30-50
Nodes in training runs
100%
GPU & network utilization
The amount of data that needs to be sent between every GPU during training is intense—basically the entire model. Traditional approaches require building massive data centers with specialized high-bandwidth interconnects. There's been no way to do this without concentrated infrastructure.
“Doubling the network speed halves our compute budget. That's the difference between a $1M model and a $500K model.”
But what if you could run distributed training over the internet? What if you could use the cheapest compute anywhere in the world and link them all together?
Psyche is Nous's distributed training framework. It brings the bandwidth requirements between each machine down from 10Gbps to just 1Gbps—making internet-based distributed training viable.
Data center operators can download a binary and use iroh to connect to every other node in a training run. They do training on their GPUs and communicate information through gossip, transferring large amounts of data via blobs.
The core question was simple: how do you get something to talk to something else? Iroh solves this. Gossip is especially useful because Psyche is building a swarm, not just a centralized service.
Built as a decentralized swarm, not a centralized service. Gossip enables coordination across all nodes.
Most training frameworks do a train step then synchronize. Psyche's asynchronous approach pegs GPUs at 100% and network connections at 100% simultaneously.
The use case: saturate all connections 100%. From iroh, they need reliable delivery of messages and file transfer as fast as it can possibly go.
High-level coordination through blockchain integration—useful when you need to pay someone and you don't know who they are.
Mesh LLM: Inference
Mesh LLM pools existing GPU resources across multiple machines and exposes them through a single OpenAI-compatible API, with no central server coordinating the mesh. Every node runs an iroh endpoint, so nodes connect directly to each other over authenticated QUIC instead of routing through a coordinator.
“Control over when models change, where your data goes, and what hardware runs your workloads.”
A request runs on the requesting machine's GPU, if it's big enough.
Otherwise, it routes directly to a peer node that already has the model loaded.
Models too large for any single machine get partitioned by layer ranges across multiple nodes, so modest machines can collectively run what none of them could load alone.
Nous runs 5 iroh relays to ensure reliable connectivity across their distributed training network. Mesh LLM runs two, in different regions, for the same reason. The key insight: when things deteriorate, they can't break.
Iroh automatically establishes direct connections when possible for maximum throughput. When direct connections aren't possible—due to NATs, firewalls, or network conditions—traffic flows through relays. This fallback mechanism means training and inference keep running even when network conditions change.
The n0.computer team hosts relays for Nous through the iroh services service. This provides reliable relay infrastructure without Nous having to manage it themselves.
We're also partnering with Nous to build better monitoring tools for distributed training—making it easy to understand what's happening at the network level during training runs.
“iroh does so much low-level networking for us. We don't have to learn about the low-level details of QUIC. When things go wrong, we want to look at the metrics and logs to understand what happened.”