Rust Distributed Mesh
This is the substrate layer a bigger event-streaming system gets built on — ordered, durable records over a self-organizing, NAT-traversing iroh fabric. It was never meant to be a proof of concept; it was meant to be a foundation solid enough to bet a product on. So I built the layer, then ran it hard enough to find every place it broke — and fixed each one until it stopped. What's left isn't a demo you admire. It's a layer you build on.
→ github.com/drlukeangel/rust-distributed-mesh
Why this exists
Every "production-ready" distributed system I've worked with assumed something the documentation didn't say out loud: that nodes share a datacenter network, that broker discovery is a config file, that the gossip cadence is some unstated default someone tuned in 2014. I wanted to feel that assumption from the inside — find out where the iroh-as-substrate story breaks at small scale, before betting a real product on it.
Spoiler from the journey below: it broke in places that had nothing to do with iroh. It broke in places where I'd written tokio::spawn(...) without thinking about who owned the resulting tasks.
What's in the box
| Piece | What it does |
|---|---|
crates/mesh-node-base | The substrate. Identity, gossip emit loop, peer registry, the LoadSampler for self-reported CPU/RAM, the staleness pruner. Built on iroh-gossip 0.98 (Plumtree + HyParView). |
crates/mesh-transport | Thin layer over iroh's Endpoint. Sets the ALPN, the mDNS toggle, the bind-addr surface, the idle timeout. |
crates/mesh-telemetry | OTLP/tracing init. Single source of truth for how every node's spans land in Jaeger. |
admin-ui/ | React + Vite topology view. The thing you stare at when you don't believe the numbers. Live node grid, hub-and-leaf layout, CPU/RAM bars per node, kill button per card. |
broker / gateway / compute / registry / bridge | Example node types. Each is just NodeRuntime::new("type").run().await and a .env.dev preset. From the substrate's perspective they're interchangeable. |
Stack: rust (substrate) · iroh 0.98 + iroh-gossip (QUIC + NAT traversal + Plumtree/HyParView) · tokio · opentelemetry → Jaeger · react + vite (UI).
The arc
| When | What |
|---|---|
| Early May 2026 | First 18-node bootstrap. Host pegged at 100% CPU. Hypothesis: "iroh-gossip is expensive at this scale." |
| Mid May | Killed mDNS (explicit seed nodes only), bumped gossip cadence 500ms → 2000ms, TRACE-demoted per-frame INFO spans. Host CPU: 100% → 35%. Better, still wrong. |
| Late May | Almost forked the project — spent two days seriously considering a centralized-Controller architecture to escape "gossip costs." |
| Late May | Took a flamegraph instead. Found the actual bug: a join_peers() call inside a 100ms tick that retriggered TLS handshakes for every known peer every 1/10th of a second. 3,240 QUIC handshakes per second across the cluster, accomplishing nothing. |
| End of May | Fixed the storm. Found four more leaks while in there: ghost QUIC connections on peer reconnect, span queue accumulation under #[instrument] on infinite loops, unbounded global state maps, busy-wait on a closed channel. Host CPU: 35% → 5%. Per-node settled at 0.05–0.10 cores. Competitive with an idle Bitcoin node. |
| Now | Hardened, soak-tested under a chaos battery, and open-sourced — a foundation to build on, not a prototype to admire. |
What I'd tell someone building one
- Comments lie. Flamegraphs don't. Every CPU bug I found had a comment next to it claiming the code was cheap. The flamegraph said otherwise on every one.
- Don't blame the protocol before you've measured. I was two days into "we need to abandon gossip" when a 30-second profile capture made the question moot. The protocol was fine. My usage of it was the bug.
- Per-node telemetry lies too. sysinfo's
cpu_usage()on Windows over-counts by 1.5–2× at high process counts. Task Manager was the ground truth; my own telemetry was inflating the cost picture. Always cross-check against the OS. - The substrate isn't the product. Half the bugs were because I treated gossip as a data plane — every node broadcasts its full state every 2 seconds, forever — when it's a control plane primitive. The day I stopped treating "everyone shouts everything constantly" as a normal pattern, the architecture made sense.
What's next
The full write-up lives in the notebook Building A Distributed Mesh in Rust — the work in order, including:
- Why I built a P2P mesh substrate in Rust — the architecture, the iroh choice, what the substrate buys you that direct TCP doesn't.
- When 18 nodes pegged my 80-core box at 100% — the bug cascade, with the actual diff next to each one.
- Flamegraphing your way out of "this can't possibly be right" — the engineering process, including the day I almost forked the architecture for the wrong reason.
The substrate is done — hardened, observable, and ready to build on. The event-streaming layer that rides on it is the next sprint; this is the foundation it stands on.