What is the differrence between OpenZiti and Nebula?

[romelBen]

Hey all, I have been experimenting on the usage of OpenZiti and Nebula on communicating to 6000+ raspberry pis and jetson devices securely. I can safely assume Wireguard/OpenVPN does the trick but looking at a simpler process such as OpenZiti/Nebula for managing high availability and scaling. (OpenZiti has this process but it is not simple to setup.)

The benchmark would be handling heavy throughput such as cameras.

[johnmaguire]

Hi @romelBen - apologies for the long delay in getting back to you.

In short, Nebula is an IP-layer overlay: you get a flat, encrypted network and your apps talk over it unmodified. It's generally exposed as a tun device on your host, which packets route over. OpenZiti is an application-layer zero-trust networking solution: applications address each other by name/identity rather than IP, usually via an SDK integrated into the application. They also offer a tunneler to bridge unmodified apps onto the fabric. Both use certificate-based identity and have policy layers, but they are solving networking at very different layers (the host layer vs. the application layer.)

Data path: Nebula tunnels are peer-to-peer, with Lighthouses performing discovery and coordination of hole punching. They can also run as relays (or you can use dedicated relays) as a fallback path when direct connections do not work. OpenZiti on the other hand routes through a router mesh by default. This means that you have additional chokepoints and bandwidth considerations, but the tradeoff is that OpenZiti offers more advanced smart-routing/failover.

Throughput: I'd recommend performing your own performance testing. One thing worth knowing before you benchmark: a single Nebula tunnel is handled by a single routine, since the kernel hashes UDP packets to sockets by 4-tuple. So per-tunnel throughput is bounded by one core, and the way to scale past that is more tunnels rather than more threads per tunnel. On Pi 4/5 and Jetson you'll have the ARMv8 crypto extensions, so that core is mostly spent on per-packet overhead rather than on AES - meaning packet sizes and batching will move your numbers more than cipher choice will.

Management: OpenZiti offers a management layer with a full enrollment story baked in. Nebula requires you to build this yourself: nebula-cert is how you manage your CA and mint new host identities. You either need to build or buy something to manage IPs, rotate certificates, and handle config management. (Defined Networking - the company many of us work at, myself included - offers a managed solution here.)

Nebula is deployed as a static binary and a YAML file per host, plus a couple of lighthouse hosts for discovery, hole punching, and optionally relaying. OpenZiti is a controller, routers, and a console.

Overall, if you're looking for a flat encrypted network with direct device-to-device communication, minimal moving parts, and are willing to build/buy your management plane, Nebula fits well - and I think the peer-to-peer communication is a genuine advantage at your scale. On the other hand, if you're looking for something more like a service-level fabric with identity-aware access that you can bake into your own application, OpenZiti is worth a look.