I work on connet [1] and from what I've seen iroh seem pretty cool. A few random thoughts I had while watching the presentations/reading the docs:
* the relays serve both for discovery and relay. In connet these are separate responsibilities, e.g. it has control server for discovery and a relay server for relaying connections.
* it seems that the connections to the relays in iroh are TCP (at least what was said in one of the videos), while connet uses QUIC in all cases. This probably makes iroh more resilient, but multiplexing on top of TCP might suffer from head of line blocking.
* it is pretty cool that iroh can seamlessly upgrade from relay to direct connection, connet doesn't do that on a connection level. It will however use direct in the subsequent virtual connections.
* using ALPNs for protocol selection is pretty cool, connet only offers "virtual connections" protocol, where one of the peers is "server" and the other is a "client".
* since there is a separate discovery server (with auth), in connet the endpoints are named separately with logical names, they don't represent peers necessarily. Because of this, you can have multiple peers with "server" role and "client" roles.
Anyhow, thanks for posting this, iroh looks great and I will draw some inspiration from it for sure.
There might be a small misunderstanding here. Our relays do two things. They relay user data, and relay a small number of special packets to help with hole punching. Other than that they are very simple. They never see unencrypted data, so they don't know anything more about the nodes they serve than what they need to function.
Connections are TCP https websocket connections, because this is most likely to get through even the most restrictive firewalls.
Discovery is handled outside the relays, via either a special DNS server or via the bittorrent mainline DHT. You can even implement your own discovery.
> * the relays serve both for discovery and relay. In connet these are separate responsibilities, e.g. it has control server for discovery and a relay server for relaying connections.
What are the relative advantages/disadvantages of these two strategies?
Some years ago, "iroh" was supposed to a replacement for ipfs. However since then, they (very smartly, in my opinion) dropped those ambitions and are just focused on being a high-quality library for anyone writing a P2P app (like ipfs).
I often see projects attempting to be a universe tool to solve every possible problem, and I think the iroh folks were smart to scale back and narrow their focus
I attended a workshop for iroh a while ago and really enjoyed it, and from what I can tell on the Discord server the folks developing it are gearing for a 1.0 release soon-ish.
There's also Dumb Pipe and SendMe which are demos (I believe) built on iroh to showcase some of its uses, and at the workshop we were shown a video of a startup using iroh for video game streaming (something similar to the old OnLive).
From what I understood (in spite of my lack of networking knowledge) and if I remember correctly clients have to be on the same relay (I think there's one for Europe and one for North America) and they use the Bittorent DHT Mainline (I had to google the iroh blog post about it because I forgot the exact name) for discovery. There was some stuff about BGP too, but it went over my head sadly.
I hope somebody more knowledgeable chimes in because iroh is really exciting, I feel like I could throw together a p2p application and it wouldn't be a daunting task due to it.
(disclosure: I work on iroh): you're selling yourself short! All of this is accurate, except for maybe the BGP stuff :)
Dumb Pipe & Sendme me are indeed demos, we do provide a set of default, public relays to use for free. The relay code is also open source, and if you want to pay us we can run a network for you.
We try to provide a few different options for discovery, the one we think has the most general utility is a custom DNS server, but both local mDNS and Bittorrent Mainline are also pluggable options.
Pardon me for jumping in the discussion, but I didn't know where else to ask this. Does Iroh support streaming, instead of moving blobs? I want to write a little p2p tool to forward one port from one machine to another.
Also, forwarding UDP packets doesn't require the congestion control of QUIC. Does Iroh allow disabling it for a certain "message" or stream?
Yes. Iroh itself provides direct QUIC connections. iroh-blobs is a protocol on top of iroh that provides content-addressed data transfer of BLAKE3 hashed data.
What you describe sounds like https://www.dumbpipe.dev/ , a tool/demo built on top of iroh to provide a bidirectional pipe across devices, somewhat like netcat.
Dumbpipe also has a mode where it listens on a port using TCP.
Iroh is intriguing. Dumbpipe is magical, and its implementation is easy to understand. I use dumbpipe daily to expose cross-stream (https://github.com/cablehead/xs) stores I run on different servers to my local laptop's `xs` client.
A bit off topic I guess, but what’s your usage for xs? I read the website, I think that I understand it and find it intriguing, but I’m not sure what one would use it for.
This thing is written in Rust. I wanted to use it on an embedded system in Rust (Embassy) using a CAN transport but unfortunately there's neither a no_std version nor a CAN plugin.
Otherwise it looks good.
Yeah, no_std is going to be very hard. We need a no_std implementation of QUIC that can be wielded by mere mortals first, which I don't think we'll be able to start on for at least a year.
Right now we can get down to an ESP32, which we think is a decent start.
Hm, how does this answer relate to the answer you gave to this comment: https://news.ycombinator.com/item?id=44381084 ? where, as far as I understand, you say it's possible to swap out quic for something else? asking sincerely, I'm confused here.
ah very sorry, I can see how this isn't all that clear. In the comment you've mentioned when I say "custom protocol" I mean a custom QUIC ALPNs: https://datatracker.ietf.org/doc/html/rfc7301
When we talk to mainline it's for discovery, which is separate from iroh connections, which always uses QUIC. Specifically: our fork of quinn, an implementation of QUIC in rust. Iroh is tightly coupled to quinn, and isn't swappable. Getting no_std support for us basically boils down to "can we get quinn to support no_std?". For that, see: https://github.com/quinn-rs/quinn/issues/579
Can this be a made to work as an adapter to play older, raw UDP multiplayer games with random strangers? E.g. telling someone in Twitch chat “bro 1v1 me in CS1.6, here’s my Iroh ticket:”, they put it into their “InstaFrag NetDriver” Windows Client and you both launch CS1.6 and just start playing in an ad-hoc p2p lobby.
With Tailscale this use case is very cumbersome as you’d need to add them to your tailnet and configure access controls to make it an ephemeral connection.
To expand on this: iroh is a rust library. A NodeId is just an Ed25519 public key, but of course it has a distinct type. If in the future we want a different public key standard, it would be a different rust type.
Encoding keys is mostly left to the user. The only exception are tickets. Tickets are postcard serialized and have a version field, so we can keep tickets compatible if we ever want to use a different public key standard or hash function.
I've been following Iroh's development for quite some time now and I have to say that I've been really impressed with what you've built so far.
At one point I'm going to use Iroh (or something heavily inspired by it) as the transport layer for a project I am working on. Can't wait.
I do have one question though while I have your attention: what was the reason you decided to use the Ed25519 public key as the NodeId directly? I mean, why not derive the NodeId from the public key instead (by hashing it for example)? Then the protocol itself would not be so tightly bound to Ed25519. A little indirection here would have been useful imho.
It's the one thing I have been wondering about Irohs design that I haven't really been able to answer by myself.
We decided to keep things simple. In general we try to provide one good way to do something instead of having a lot of options.
E.g. we only provide Ed25519 for keys and in iroh-blobs only BLAKE3 for hashing, instead of having a multihash scheme. Having the public key directly available is sometimes useful, e.g. for verifying signatures. It also allowed us to directly use the mainline extension BEP_0044 to store data for public keys.
That being said, I am very confident that we will be able to provide a relatively smooth transition if we ever have to switch from Ed25519 to another public key format.
For connection encryption we use a TLS extension called raw public keys in TLS, and here of course the keys are prefixed, and we could easily upgrade to another key format and then at some point stop supporting Ed25519 keys.
I've been wanting something like what Syncthing does for peer discovery for a while - something like this. Too bad it's written in such a low-level language.
lets say if i someone wants to keep using bittorrent dht for peer finding but swap out quic for something else maybe grpc, does the lib support that split clean? asking from a modular embed first tooling pov, where discovery logic needs to outlive or outswap transport depending on deployment
The two sides are peers when it comes to connection establishment, but once you have a connection they can and frequently will have different roles.
Many existing iroh protocols have clear client and server roles once the connection is established. E.g. gossip is a peer to peer protocol, blobs is a client server protocol in that one side provides data and the other requests it.
For a client you can use an ephemeral node id and not publish your info to discovery, since you will never be dialed yourself.
Each time I looked at libp2p I didn't even knew where to begin. With Iroh it was trivial to get connections.
Also, AFAIK, Iroh makes some architectural choices (using relays to help establish connections), that make it less "pure p2p", but much more likely to actually work reliably.
I work on connet [1] and from what I've seen iroh seem pretty cool. A few random thoughts I had while watching the presentations/reading the docs:
* the relays serve both for discovery and relay. In connet these are separate responsibilities, e.g. it has control server for discovery and a relay server for relaying connections.
* it seems that the connections to the relays in iroh are TCP (at least what was said in one of the videos), while connet uses QUIC in all cases. This probably makes iroh more resilient, but multiplexing on top of TCP might suffer from head of line blocking.
* it is pretty cool that iroh can seamlessly upgrade from relay to direct connection, connet doesn't do that on a connection level. It will however use direct in the subsequent virtual connections.
* using ALPNs for protocol selection is pretty cool, connet only offers "virtual connections" protocol, where one of the peers is "server" and the other is a "client".
* since there is a separate discovery server (with auth), in connet the endpoints are named separately with logical names, they don't represent peers necessarily. Because of this, you can have multiple peers with "server" role and "client" roles.
Anyhow, thanks for posting this, iroh looks great and I will draw some inspiration from it for sure.
[1] https://github.com/connet-dev/connet
There might be a small misunderstanding here. Our relays do two things. They relay user data, and relay a small number of special packets to help with hole punching. Other than that they are very simple. They never see unencrypted data, so they don't know anything more about the nodes they serve than what they need to function.
Connections are TCP https websocket connections, because this is most likely to get through even the most restrictive firewalls.
Discovery is handled outside the relays, via either a special DNS server or via the bittorrent mainline DHT. You can even implement your own discovery.
> * the relays serve both for discovery and relay. In connet these are separate responsibilities, e.g. it has control server for discovery and a relay server for relaying connections.
What are the relative advantages/disadvantages of these two strategies?
Some years ago, "iroh" was supposed to a replacement for ipfs. However since then, they (very smartly, in my opinion) dropped those ambitions and are just focused on being a high-quality library for anyone writing a P2P app (like ipfs).
I often see projects attempting to be a universe tool to solve every possible problem, and I think the iroh folks were smart to scale back and narrow their focus
appreciate the feedback, it was a hard decision to make, but has felt more right everyday since we made it
I attended a workshop for iroh a while ago and really enjoyed it, and from what I can tell on the Discord server the folks developing it are gearing for a 1.0 release soon-ish.
There's also Dumb Pipe and SendMe which are demos (I believe) built on iroh to showcase some of its uses, and at the workshop we were shown a video of a startup using iroh for video game streaming (something similar to the old OnLive).
From what I understood (in spite of my lack of networking knowledge) and if I remember correctly clients have to be on the same relay (I think there's one for Europe and one for North America) and they use the Bittorent DHT Mainline (I had to google the iroh blog post about it because I forgot the exact name) for discovery. There was some stuff about BGP too, but it went over my head sadly.
I hope somebody more knowledgeable chimes in because iroh is really exciting, I feel like I could throw together a p2p application and it wouldn't be a daunting task due to it.
(disclosure: I work on iroh): you're selling yourself short! All of this is accurate, except for maybe the BGP stuff :)
Dumb Pipe & Sendme me are indeed demos, we do provide a set of default, public relays to use for free. The relay code is also open source, and if you want to pay us we can run a network for you.
We try to provide a few different options for discovery, the one we think has the most general utility is a custom DNS server, but both local mDNS and Bittorrent Mainline are also pluggable options.
Pardon me for jumping in the discussion, but I didn't know where else to ask this. Does Iroh support streaming, instead of moving blobs? I want to write a little p2p tool to forward one port from one machine to another. Also, forwarding UDP packets doesn't require the congestion control of QUIC. Does Iroh allow disabling it for a certain "message" or stream?
Yes. Iroh itself provides direct QUIC connections. iroh-blobs is a protocol on top of iroh that provides content-addressed data transfer of BLAKE3 hashed data.
What you describe sounds like https://www.dumbpipe.dev/ , a tool/demo built on top of iroh to provide a bidirectional pipe across devices, somewhat like netcat.
Dumbpipe also has a mode where it listens on a port using TCP.
It sounds like you want to basically build dumbpipe for UDP. You can of course use a QUIC stream, but QUIC has an extension, which we support, to send datagrams: https://docs.rs/iroh/latest/iroh/endpoint/struct.Connection....
This basically allows you to opt out of QUIC streams, but you still do get TLS encryption.
It looks like they have examples with unreliable channels: https://github.com/n0-computer/iroh/tree/main/iroh/examples
You'll prob have to check the max packet size that you want to forward because quic adds a bit of overhead.
Was the workshop recorded?
Iroh is very cool and their YouTube explainers are pretty great: https://youtube.com/@n0computer
I just need good FFI now, which is on the roadmap!
Can't wait to be able to use it in Go or Python :)
Iroh is intriguing. Dumbpipe is magical, and its implementation is easy to understand. I use dumbpipe daily to expose cross-stream (https://github.com/cablehead/xs) stores I run on different servers to my local laptop's `xs` client.
A bit off topic I guess, but what’s your usage for xs? I read the website, I think that I understand it and find it intriguing, but I’m not sure what one would use it for.
I was never angry with you. I was sad, because I was afraid you'd lost your route.
Sharing tea with a fascinating stranger is one of life’s true delights.
Failure is only the opportunity to begin again. Only this time, more wisely.
Makes me cry everytime.
This may be, the worlds most perfect comment. Bravo.
I came here for this. Thanks!
this made my day
This thing is written in Rust. I wanted to use it on an embedded system in Rust (Embassy) using a CAN transport but unfortunately there's neither a no_std version nor a CAN plugin. Otherwise it looks good.
Yeah, no_std is going to be very hard. We need a no_std implementation of QUIC that can be wielded by mere mortals first, which I don't think we'll be able to start on for at least a year.
Right now we can get down to an ESP32, which we think is a decent start.
Hm, how does this answer relate to the answer you gave to this comment: https://news.ycombinator.com/item?id=44381084 ? where, as far as I understand, you say it's possible to swap out quic for something else? asking sincerely, I'm confused here.
ah very sorry, I can see how this isn't all that clear. In the comment you've mentioned when I say "custom protocol" I mean a custom QUIC ALPNs: https://datatracker.ietf.org/doc/html/rfc7301
When we talk to mainline it's for discovery, which is separate from iroh connections, which always uses QUIC. Specifically: our fork of quinn, an implementation of QUIC in rust. Iroh is tightly coupled to quinn, and isn't swappable. Getting no_std support for us basically boils down to "can we get quinn to support no_std?". For that, see: https://github.com/quinn-rs/quinn/issues/579
this would be great to have, but as we rely heavily on quic, we first need an implementation of quic in no_std which is the current biggest challenge
How does this compare to libP2P? https://libp2p.io/
less configuration. more reliable. less pure p2p (iroh uses relays)
hey I work on this! AMA!
Can this be a made to work as an adapter to play older, raw UDP multiplayer games with random strangers? E.g. telling someone in Twitch chat “bro 1v1 me in CS1.6, here’s my Iroh ticket:”, they put it into their “InstaFrag NetDriver” Windows Client and you both launch CS1.6 and just start playing in an ad-hoc p2p lobby.
With Tailscale this use case is very cumbersome as you’d need to add them to your tailnet and configure access controls to make it an ephemeral connection.
There's no reason this _can't_ be built. The thing that's missing is simulating a raw UDP socket.
We could adapt the pattern from dumbpipe wrapping a TCP listner: https://github.com/n0-computer/dumbpipe?tab=readme-ov-file#t... which is exactly the "here join this ticket" you're describing
This is very much like my usecase too, except that I need to be able to use it via FFI since my engine is not Rust.
we do have https://github.com/n0-computer/iroh-c-ffi which is lowlevel but directly usable from any c like language
do keys have a defined prefix or identifier? e.g., Veilid uses vld0? pkarr uses pk:?
keys are always ED25519, we use raw public key bytes, without prefixes.
Applications are more than welcome to use prefixes, but the use of ED25519 is not configurable
To expand on this: iroh is a rust library. A NodeId is just an Ed25519 public key, but of course it has a distinct type. If in the future we want a different public key standard, it would be a different rust type.
Encoding keys is mostly left to the user. The only exception are tickets. Tickets are postcard serialized and have a version field, so we can keep tickets compatible if we ever want to use a different public key standard or hash function.
(disclaimer, I also work on iroh)
I've been following Iroh's development for quite some time now and I have to say that I've been really impressed with what you've built so far.
At one point I'm going to use Iroh (or something heavily inspired by it) as the transport layer for a project I am working on. Can't wait.
I do have one question though while I have your attention: what was the reason you decided to use the Ed25519 public key as the NodeId directly? I mean, why not derive the NodeId from the public key instead (by hashing it for example)? Then the protocol itself would not be so tightly bound to Ed25519. A little indirection here would have been useful imho.
It's the one thing I have been wondering about Irohs design that I haven't really been able to answer by myself.
Anyways, great work! Keep it up!
We decided to keep things simple. In general we try to provide one good way to do something instead of having a lot of options.
E.g. we only provide Ed25519 for keys and in iroh-blobs only BLAKE3 for hashing, instead of having a multihash scheme. Having the public key directly available is sometimes useful, e.g. for verifying signatures. It also allowed us to directly use the mainline extension BEP_0044 to store data for public keys.
That being said, I am very confident that we will be able to provide a relatively smooth transition if we ever have to switch from Ed25519 to another public key format.
For connection encryption we use a TLS extension called raw public keys in TLS, and here of course the keys are prefixed, and we could easily upgrade to another key format and then at some point stop supporting Ed25519 keys.
raw public keys in TLS: https://datatracker.ietf.org/doc/html/rfc7250 storing arbitrary data in the DHT: https://www.bittorrent.org/beps/bep_0044.html
I've been wanting something like what Syncthing does for peer discovery for a while - something like this. Too bad it's written in such a low-level language.
these guys also have some really nice distributed systems explainer videos
lets say if i someone wants to keep using bittorrent dht for peer finding but swap out quic for something else maybe grpc, does the lib support that split clean? asking from a modular embed first tooling pov, where discovery logic needs to outlive or outswap transport depending on deployment
yep totally doable. You'd use iroh configured with mainline, Then write a custom protocol for the grpc bit: https://www.iroh.computer/docs/protocols/writing
We use this a bunch for writing rpc protocols on iroh: https://github.com/n0-computer/irpc , but there's no reason you couldn't write one with grpc.
Does this always have to be p2p or does it also allow for client server architecture
The two sides are peers when it comes to connection establishment, but once you have a connection they can and frequently will have different roles.
Many existing iroh protocols have clear client and server roles once the connection is established. E.g. gossip is a peer to peer protocol, blobs is a client server protocol in that one side provides data and the other requests it.
For a client you can use an ephemeral node id and not publish your info to discovery, since you will never be dialed yourself.
It just makes a connection between two sides. How you use it (e.g. client makes a request, server responds) is up to you. So yes.
This is what GIT should had from start ! Imagine...
The promise of this is super interesting. How would people compare it to libp2p? Is libp2p a lower level toolkit that leaves the assembly to you?
Each time I looked at libp2p I didn't even knew where to begin. With Iroh it was trivial to get connections.
Also, AFAIK, Iroh makes some architectural choices (using relays to help establish connections), that make it less "pure p2p", but much more likely to actually work reliably.