Tag Archives: Internet

Easily Accessing All Your Stuff with a Zero-Trust Mesh VPN

Probably everyone is familiar with a regular VPN. The traditional use case is to connect to a corporate or home network from a remote location, and access services as if you were there.

But these days, the notion of “corporate network” and “home network” are less based around physical location. For instance, a company may have no particular office at all, may have a number of offices plus a number of people working remotely, and so forth. A home network might have, say, a PVR and file server, while highly portable devices such as laptops, tablets, and phones may want to talk to each other regardless of location. For instance, a family member might be traveling with a laptop, another at a coffee shop, and those two devices might want to communicate, in addition to talking to the devices at home.

And, in both scenarios, there might be questions about giving limited access to friends. Perhaps you’d like to give a friend access to part of your file server, or as a company, you might have contractors working on a limited project.

Pretty soon you wind up with a mess of VPNs, forwarded ports, and tricks to make it all work. With the increasing prevalence of CGNAT, a lot of times you can’t even open a port to the public Internet. Each application or device probably has its own gateway just to make it visible on the Internet, some of which you pay for.

Then you add on the question of: should you really trust your LAN anyhow? With possibilities of guests using it, rogue access points, etc., the answer is probably “no”.

We can move the responsibility for dealing with NAT, fluctuating IPs, encryption, and authentication, from the application layer further down into the network stack. We then arrive at a much simpler picture for all.

So this page is fundamentally about making the network work, simply and effectively.

How do we make the Internet work in these scenarios?

We’re going to combine three concepts:

  1. A VPN, providing fully encrypted and authenticated communication and stable IPs
  2. Mesh Networking, in which devices automatically discover optimal paths to reach each other
  3. Zero-trust networking, in which we do not need to trust anything about the underlying LAN, because all our traffic uses the secure systems in points 1 and 2.

By combining these concepts, we arrive at some nice results:

  • You can ssh hostname, where hostname is one of your machines (server, laptop, whatever), and as long as hostname is up, you can reach it, wherever it is, wherever you are.
    • Combined with mosh, these sessions will be durable even across moving to other host networks.
    • You could just as well use telnet, because the underlying network should be secure.
  • You don’t have to mess with encryption keys, certs, etc., for every internal-only service. Since IPs are now trustworthy, that’s all you need. hosts.allow could make a comeback!
  • You have a way of transiting out of extremely restrictive networks. Every tool discussed here has a way of falling back on routing things via a broker (relay) on TCP port 443 if all else fails.

There might sometimes be tradeoffs. For instance:

  • On LANs faster than 1Gbps, performance may degrade due to encryption and encapsulation overhead. However, these tools should let hosts discover the locality of each other and not send traffic over the Internet if the devices are local.
  • With some of these tools, hosts local to each other (on the same LAN) may be unable to find each other if they can’t reach the control plane over the Internet (Internet is down or provider is down)

Some other features that some of the tools provide include:

  • Easy sharing of limited access with friends/guests
  • Taking care of everything you need, including SSL certs, for exposing a certain on-net service to the public Internet
  • Optional routing of your outbound Internet traffic via an exit node on your network. Useful, for instance, if your local network is blocking tons of stuff.

Let’s dive in.

Types of Mesh VPNs

I’ll go over several types of meshes in this article:

  1. Fully decentralized with automatic hop routing

    This model has no special central control plane. Nodes discover each other in various ways, and establish routes to each other. These routes can be direct connections over the Internet, or via other nodes. This approach offers the greatest resilience. Examples I’ll cover include Yggdrasil and tinc.

  2. Automatic peer-to-peer with centralized control

    In this model, nodes, by default, communicate by establishing direct links between them. A regular node never carries traffic on behalf of other nodes. Special-purpose relays are used to handle cases in which NAT traversal is impossible. This approach tends to offer simple setup. Examples I’ll cover include Tailscale, Zerotier, Nebula, and Netmaker.

  3. Roll your own and hybrid approaches

    This is a “grab bag” of other ideas; for instance, running Yggdrasil over Tailscale.

Terminology

For the sake of consistency, I’m going to use common language to discuss things that have different terms in different ecosystems:

  • Every tool discussed here has a way of dealing with NAT traversal. It may assist with establishing direct connections (eg, STUN), and if that fails, it may simply relay traffic between nodes. I’ll call such a relay a “broker”. This may or may not be the same system that is a control plane for a tool.
  • All of these systems operate over lower layers that are unencrypted. Those lower layers may be a LAN (wired or wireless, which may or may not have Internet access), or the public Internet (IPv4 and/or IPv6). I’m going to call the unencrypted lower layer, whatever it is, the “clearnet”.

Evaluation Criteria

Here are the things I want to see from a solution:

  • Secure, with all communications end-to-end encrypted and authenticated, and prevention of traffic from untrusted devices.
  • Flexible, adapting to changes in network topology quickly and automatically.
  • Resilient, without single points of failure, and with devices local to each other able to communicate even if cut off from the Internet or other parts of the network.
  • Private, minimizing leakage of information or metadata about me and my systems
  • Able to traverse CGNAT without having to use a broker whenever possible
  • A lesser requirement for me, but still a nice to have, is the ability to include others via something like Internet publishing or inviting guests.
  • Fully or nearly fully Open Source
  • Free or very cheap for personal use
  • Wide operating system support, including headless Linux on x86_64 and ARM.

Fully Decentralized VPNs with Automatic Hop Routing

Two systems fit this description: Yggdrasil and Tinc. Let’s dive in.

Yggdrasil

I’ll start with Yggdrasil because I’ve written so much about it already. It featured in prior posts such as:

Yggdrasil can be a private mesh VPN, or something more

Yggdrasil can be a private mesh VPN, just like the other tools covered here. It’s unique, however, in that a key goal of the project is to also make it useful as a planet-scale global mesh network. As such, Yggdrasil is a testbed of new ideas in distributed routing designed to scale up to massive sizes and all sorts of connection conditions. As of 2023-04-10, the main global Yggdrasil mesh has over 5000 nodes in it. You can choose whether or not to participate.

Every node in a Yggdrasil mesh has a public/private keypair. Each node then has an IPv6 address (in a private address space) derived from its public key. Using these IPv6 addresses, you can communicate right away.

Yggdrasil differs from most of the other tools here in that it does not necessarily seek to establish a direct link on the clearnet between, say, host A and host G for them to communicate. It will prefer such a direct link if it exists, but it is perfectly happy if it doesn’t.

The reason is that every Yggdrasil node is also a router in the Yggdrasil mesh. Let’s sit with that concept for a moment. Consider:

  • If you have a bunch of machines on your LAN, but only one of them can peer over the clearnet, that’s fine; all the other machines will discover this route to the world and use it when necessary.
  • All you need to run a broker is just a regular node with a public IP address. If you are participating in the global mesh, you can use one (or more) of the free public peers for this purpose.
  • It is not necessary for every node to know about the clearnet IP address of every other node (improving privacy). In fact, it’s not even necessary for every node to know about the existence of all the other nodes, so long as it can find a route to a given node when it’s asked to.
  • Yggdrasil can find one or more routes between nodes, and it can use this knowledge of multiple routes to aggressively optimize for varying network conditions, including combinations of, say, downloads and low-latency ssh sessions.

Behind the scenes, Yggdrasil calculates optimal routes between nodes as necessary, using a mesh-wide DHT for initial contact and then deriving more optimal paths. (You can also read more details about the routing algorithm.)

One final way that Yggdrasil is different from most of the other tools is that there is no separate control server. No node is “special”, in charge, the sole keeper of metadata, or anything like that. The entire system is completely distributed and auto-assembling.

Meeting neighbors

There are two ways that Yggdrasil knows about peers:

  • By broadcast discovery on the local LAN
  • By listening on a specific port (or being told to connect to a specific host/port)

Sometimes this might lead to multiple ways to connect to a node; Yggdrasil prefers the connection auto-discovered by broadcast first, then the lowest-latency of the defined path. In other words, when your laptops are in the same room as each other on your local LAN, your packets will flow directly between them without traversing the Internet.

Unique uses

Yggdrasil is uniquely suited to network-challenged situations. As an example, in a post-disaster situation, Internet access may be unavailable or flaky, yet there may be many local devices – perhaps ones that had never known of each other before – that could share information. Yggdrasil meets this situation perfectly. The combination of broadcast auto-detection, distributed routing, and so forth, basically means that if there is any physical path between two nodes, Yggdrasil will find and enable it.

Ad-hoc wifi is rarely used because it is a real pain. Yggdrasil actually makes it useful! Its broadcast discovery doesn’t require any IP address provisioned on the interface at all (it just uses the IPv6 link-local address), so you don’t need to figure out a DHCP server or some such. And, Yggdrasil will tend to perform routing along the contours of the RF path. So you could have a laptop in the middle of a long distance relaying communications from people farther out, because it could see both. Or even a chain of such things.

Yggdrasil: Security and Privacy

Yggdrasil’s mesh is aggressively greedy. It will peer with any node it can find (unless told otherwise) and will find a route to anywhere it can. There are two main ways to make sure you keep unauthorized traffic out: by restricting who can talk to your mesh, and by firewalling the Yggdrasil interface. Both can be used, and they can be used simultaneously.

I’ll discuss firewalling more at the end of this article. Basically, you’ll almost certainly want to do this if you participate in the public mesh, because doing so is akin to having a globally-routable public IP address direct to your device.

If you want to restrict who can talk to your mesh, you just disable the broadcast feature on all your nodes (empty MulticastInterfaces section in the config), and avoid telling any of your nodes to connect to a public peer. You can set a list of authorized public keys that can connect to your nodes’ listening interfaces, which you’ll probably want to do. You will probably want to either open up some inbound ports (if you can) or set up a node with a known clearnet IP on a place like a $5/mo VPS to help with NAT traversal (again, setting AllowedPublicKeys as appropriate). Yggdrasil doesn’t allow filtering multicast clients by public key, only by network interface, so that’s why we disable broadcast discovery. You can easily enough teach Yggdrasil about static internal LAN IPs of your nodes and have things work that way. (Or, set up an internal “gateway” node or two, that the clients just connect to when they’re local). But fundamentally, you need to put a bit more thought into this with Yggdrasil than with the other tools here, which are closed-only.

Compared to some of the other tools here, Yggdrasil is better about information leakage; nodes only know details, such as clearnet IPs, of directly-connected peers. You can obtain the list of directly-connected peers of any known node in the mesh – but that list is the public keys of the directly-connected peers, not the clearnet IPs.

Some of the other tools contain a limited integrated firewall of sorts (with limited ACLs and such). Yggdrasil does not, but is fully compatible with on-host firewalls. I recommend these anyway even with many other tools.

Yggdrasil: Connectivity and NAT traversal

Compared to the other tools, Yggdrasil is an interesting mix. It provides a fully functional mesh and facilitates connectivity in situations in which no other tool can. Yet its NAT traversal, while it exists and does work, results in using a broker under some of the more challenging CGNAT situations more often than some of the other tools, which can impede performance.

Yggdrasil’s underlying protocol is TCP-based. Before you run away screaming that it must be slow and unreliable like OpenVPN over TCP – it’s not, and it is even surprisingly good around bufferbloat. I’ve found its performance to be on par with the other tools here, and it works as well as I’d expect even on flaky 4G links.

Overall, the NAT traversal story is mixed. On the one hand, you can run a node that listens on port 443 – and Yggdrasil can even make it speak TLS (even though that’s unnecessary from a security standpoint), so you can likely get out of most restrictive firewalls you will ever encounter. If you join the public mesh, know that plenty of public peers do listen on port 443 (and other well-known ports like 53, plus random high-numbered ones).

If you connect your system to multiple public peers, there is a chance – though a very small one – that some public transit traffic might be routed via it. In practice, public peers hopefully are already peered with each other, preventing this from happening (you can verify this with yggdrasilctl debug_remotegetpeers key=ABC...). I have never experienced a problem with this. Also, since latency is a factor in routing for Yggdrasil, it is highly unlikely that random connections we use are going to be competitive with datacenter peers.

Yggdrasil: Sharing with friends

If you’re open to participating in the public mesh, this is one of the easiest things of all. Have your friend install Yggdrasil, point them to a public peer, give them your Yggdrasil IP, and that’s it. (Well, presumably you also open up your firewall – you did follow my advice to set one up, right?)

If your friend is visiting at your location, they can just hop on your wifi, install Yggdrasil, and it will automatically discover a route to you. Yggdrasil even has a zero-config mode for ephemeral nodes such as certain Docker containers.

Yggdrasil doesn’t directly support publishing to the clearnet, but it is certainly possible to proxy (or even NAT) to/from the clearnet, and people do.

Yggdrasil: DNS

There is no particular extra DNS in Yggdrasil. You can, of course, run a DNS server within Yggdrasil, just as you can anywhere else. Personally I just add relevant hosts to /etc/hosts and leave it at that, but it’s up to you.

Yggdrasil: Source code, pricing, and portability

Yggdrasil is fully open source (LGPLv3 plus additional permissions in an exception) and highly portable. It is written in Go, and has prebuilt binaries for all major platforms (including a Debian package which I made).

There is no charge for anything with Yggdrasil. Listed public peers are free and run by volunteers. You can run your own peers if you like; they can be public and unlisted, public and listed (just submit a PR to get it listed), or private (accepting connections only from certain nodes’ keys). A “peer” in this case is just a node with a known clearnet IP address.

Yggdrasil encourages use in other projects. For instance, NNCP integrates a Yggdrasil node for easy communication with other NNCP nodes.

Yggdrasil conclusions

Yggdrasil is tops in reliability (having no single point of failure) and flexibility. It will maintain opportunistic connections between peers even if the Internet is down. The unique added feature of being able to be part of a global mesh is a nice one. The tradeoffs include being more prone to need to use a broker in restrictive CGNAT environments. Some other tools have clients that override the OS DNS resolver to also provide resolution of hostnames of member nodes; Yggdrasil doesn’t, though you can certainly run your own DNS infrastructure over Yggdrasil (or, for that matter, let public DNS servers provide Yggdrasil answers if you wish).

There is also a need to pay more attention to firewalling or maintaining separation from the public mesh. However, as I explain below, many other options have potential impacts if the control plane, or your account for it, are compromised, meaning you ought to firewall those, too. Still, it may be a more immediate concern with Yggdrasil.

Although Yggdrasil is listed as experimental, I have been using it for over a year and have found it to be rock-solid. They did change how mesh IPs were calculated when moving from 0.3 to 0.4, causing a global renumbering, so just be aware that this is a possibility while it is experimental.

tinc

tinc is the oldest tool on this list; version 1.0 came out in 2003! You can think of tinc as something akin to “an older Yggdrasil without the public option.”

I will be discussing tinc 1.0.36, the latest stable version, which came out in 2019. The development branch, 1.1, has been going since 2011 and had its latest release in 2021. The last commit to the Github repo was in June 2022.

Tinc is the only tool here to support both tun and tap style interfaces. I go into the difference more in the Zerotier review below. Tinc actually provides a better tap implementation than Zerotier, with various sane options for broadcasts, but I still think the call for an Ethernet, as opposed to IP, VPN is small.

To configure tinc, you generate a per-host configuration and then distribute it to every tinc node. It contains a host’s public key. Therefore, adding a host to the mesh means distributing its key everywhere; de-authorizing it means removing its key everywhere. This makes it rather unwieldy.

tinc can do LAN broadcast discovery and mesh routing, but generally speaking you must manually teach it where to connect initially. Somewhat confusingly, the examples all mention listing a public address for a node. This doesn’t make sense for a laptop, and I suspect you’d just omit it. I think that address is used for something akin to a Yggdrasil peer with a clearnet IP.

Unlike all of the other tools described here, tinc has no tool to inspect the running state of the mesh.

Some of the properties of tinc made it clear I was unlikely to adopt it, so this review wasn’t as thorough as that of Yggdrasil.

tinc: Security and Privacy

As mentioned above, every host in the tinc mesh is authenticated based on its public key. However, to be more precise, this key is validated only at the point it connects to its next hop peer. (To be sure, this is also the same as how the list of allowed pubkeys works in Yggdrasil.) Since IPs in tinc are not derived from their key, and any host can assign itself whatever mesh IP it likes, this implies that a compromised host could impersonate another.

It is unclear whether packets are end-to-end encrypted when using a tinc node as a router. The fact that they can be routed at the kernel level by the tun interface implies that they may not be.

tinc: Connectivity and NAT traversal

I was unable to find much information about NAT traversal in tinc, other than that it does support it. tinc can run over UDP or TCP and auto-detects which to use, preferring UDP.

tinc: Sharing with friends

tinc has no special support for this, and the difficulty of configuration makes it unlikely you’d do this with tinc.

tinc: Source code, pricing, and portability

tinc is fully open source (GPLv2). It is written in C and generally portable. It supports some very old operating systems. Mobile support is iffy.

tinc does not seem to be very actively maintained.

tinc conclusions

I haven’t mentioned performance in my other reviews (see the section at the end of this post). But, it is so poor as to only run about 300Mbps on my 2.5Gbps network. That’s 1/3 the speed of Yggdrasil or Tailscale. Combine that with the unwieldiness of adding hosts and some uncertainties in security, and I’m not going to be using tinc.

Automatic Peer-to-Peer Mesh VPNs with centralized control

These tend to be the options that are frequently discussed. Let’s talk about the options.

Tailscale

Tailscale is a popular choice in this type of VPN. To use Tailscale, you first sign up on tailscale.com. Then, you install the tailscale client on each machine. On first run, it prints a URL for you to click on to authorize the client to your mesh (“tailnet”). Tailscale assigns a mesh IP to each system. The Tailscale client lets the Tailscale control plane gather IP information about each node, including all detectable public and private clearnet IPs.

When you attempt to contact a node via Tailscale, the client will fetch the known contact information from the control plane and attempt to establish a link. If it can contact over the local LAN, it will (it doesn’t have broadcast autodetection like Yggdrasil; the information must come from the control plane). Otherwise, it will try various NAT traversal options. If all else fails, it will use a broker to relay traffic; Tailscale calls a broker a DERP relay server. Unlike Yggdrasil, a Tailscale node never relays traffic for another; all connections are either direct P2P or via a broker.

Tailscale, like several others, is based around Wireguard; though wireguard-go rather than the in-kernel Wireguard.

Tailscale has a number of somewhat unique features in this space:

  • Funnel, which lets you expose ports on your system to the public Internet via the VPN.
  • Exit nodes, which automate the process of routing your public Internet traffic over some other node in the network. This is possible with every tool mentioned here, but Tailscale makes switching it on or off a couple of quick commands away.
  • Node sharing, which lets you share a subset of your network with guests
  • A fantastic set of documentation, easily the best of the bunch.

Funnel, in particular, is interesting. With a couple of “tailscale serve”-style commands, you can expose a directory tree (or a development webserver) to the world. Tailscale gives you a public hostname, obtains a cert for it, and proxies inbound traffic to you. This is subject to some unspecified bandwidth limits, and you can only choose from three public ports, so it’s not really a production solution – but as a quick and easy way to demonstrate something cool to a friend, it’s a neat feature.

Tailscale: Security and Privacy

With Tailscale, as with the other tools in this category, one of the main threats to consider is the control plane. What are the consequences of a compromise of Tailscale’s control plane, or of the credentials you use to access it?

Let’s begin with the credentials used to access it. Tailscale operates no identity system itself, instead relying on third parties. For individuals, this means Google, Github, or Microsoft accounts; Okta and other SAML and similar identity providers are also supported, but this runs into complexity and expense that most individuals aren’t wanting to take on. Unfortunately, all three of those types of accounts often have saved auth tokens in a browser. Personally I would rather have a separate, very secure, login.

If a person does compromise your account or the Tailscale servers themselves, they can’t directly eavesdrop on your traffic because it is end-to-end encrypted. However, assuming an attacker obtains access to your account, they could:

  • Tamper with your Tailscale ACLs, permitting new actions
  • Add new nodes to the network
  • Forcibly remove nodes from the network
  • Enable or disable optional features

Of note is that they cannot just commandeer an existing IP. I would say the riskiest possibility here is that could add new nodes to the mesh. Because they could also tamper with your ACLs, they could then proceed to attempt to access all your internal services. They could even turn on service collection and have Tailscale tell them what and where all the services are.

Therefore, as with other tools, I recommend a local firewall on each machine with Tailscale. More on that below.

Tailscale has a new alpha feature called tailnet lock which helps with this problem. It requires existing nodes in the mesh to sign a request for a new node to join. Although this doesn’t address ACL tampering and some of the other things, it does represent a significant help with the most significant concern. However, tailnet lock is in alpha, only available on the Enterprise plan, and has a waitlist, so I have been unable to test it.

Any Tailscale node can request the IP addresses belonging to any other Tailscale node. The Tailscale control plane captures, and exposes to you, this information about every node in your network: the OS hostname, IP addresses and port numbers, operating system, creation date, last seen timestamp, and NAT traversal parameters. You can optionally enable service data capture as well, which sends data about open ports on each node to the control plane.

Tailscale likes to highlight their key expiry and rotation feature. By default, all keys expire after 180 days, and traffic to and from the expired node will be interrupted until they are renewed (basically, you re-login with your provider and do a renew operation). Unfortunately, the only mention I can see of warning of impeding expiration is in the Windows client, and even there you need to edit a registry key to get the warning more than the default 24 hours in advance. In short, it seems likely to cut off communications when it’s most important. You can disable key expiry on a per-node basis in the admin console web interface, and I mostly do, due to not wanting to lose connectivity at an inopportune time.

Tailscale: Connectivity and NAT traversal

When thinking about reliability, the primary consideration here is being able to reach the Tailscale control plane. While it is possible in limited circumstances to reach nodes without the Tailscale control plane, it is “a fairly brittle setup” and notably will not survive a client restart. So if you use Tailscale to reach other nodes on your LAN, that won’t work unless your Internet is up and the control plane is reachable.

Assuming your Internet is up and Tailscale’s infrastructure is up, there is little to be concerned with. Your own comfort level with cloud providers and your Internet should guide you here.

Tailscale wrote a fantastic article about NAT traversal and they, predictably, do very well with it. Tailscale prefers UDP but falls back to TCP if needed. Broker (DERP) servers step in as a last resort, and Tailscale clients automatically select the best ones. I’m not aware of anything that is more successful with NAT traversal than Tailscale. This maximizes the situations in which a direct P2P connection can be used without a broker.

I have found Tailscale to be a bit slow to notice changes in network topography compared to Yggdrasil, and sometimes needs a kick in the form of restarting the client process to re-establish communications after a network change. However, it’s possible (maybe even probable) that if I’d waited a bit longer, it would have sorted this all out.

Tailscale: Sharing with friends

I touched on the funnel feature earlier. The sharing feature lets you give an invite to an outsider. By default, a person accepting a share can make only outgoing connections to the network they’re invited to, and cannot receive incoming connections from that network – this makes sense. When sharing an exit node, you get a checkbox that lets you share access to the exit node as well. Of course, the person accepting the share needs to install the Tailnet client. The combination of funnel and sharing make Tailscale the best for ad-hoc sharing.

Tailscale: DNS

Tailscale’s DNS is called MagicDNS. It runs as a layer atop your standard DNS – taking over /etc/resolv.conf on Linux – and provides resolution of mesh hostnames and some other features. This is a concept that is pretty slick.

It also is a bit flaky on Linux; dueling programs want to write to /etc/resolv.conf. I can’t really say this is entirely Tailscale’s fault; they document the problem and some workarounds.

I would love to be able to add custom records to this service; for instance, to override the public IP for a service to use the in-mesh IP. Unfortunately, that’s not yet possible. However, MagicDNS can query existing nameservers for certain domains in a split DNS setup.

Tailscale: Source code, pricing, and portability

Tailscale is almost fully open source and the client is highly portable. The client is open source (BSD 3-clause) on open source platforms, and closed source on closed source platforms. The DERP servers are open source. The coordination server is closed source, although there is an open source coordination server called Headscale (also BSD 3-clause) made available with Tailscale’s blessing and informal support. It supports most, but not all, features in the Tailscale coordination server.

Tailscale’s pricing (which does not apply when using Headscale) provides a free plan for 1 user with up to 20 devices. A Personal Pro plan expands that to 100 devices for $48 per year - not a bad deal at $4/mo. A “Community on Github” plan also exists, and then there are more business-oriented plans as well. See the pricing page for details.

As a small note, I appreciated Tailscale’s install script. It properly added Tailscale’s apt key in a way that it can only be used to authenticate the Tailscale repo, rather than as a systemwide authenticator. This is a nice touch and speaks well of their developers.

Tailscale conclusions

Tailscale is tops in sharing and has a broad feature set and excellent documentation. Like other solutions with a centralized control plane, device communications can stop working if the control plane is unreachable, and the threat model of the control plane should be carefully considered.

Zerotier

Zerotier is a close competitor to Tailscale, and is similar to it in a lot of ways. So rather than duplicate all of the Tailscale information here, I’m mainly going to describe how it differs from Tailscale.

The primary difference between the two is that Zerotier emulates an Ethernet network via a Linux tap interface, while Tailscale emulates a TCP/IP network via a Linux tun interface.

However, Zerotier has a number of things that make it be a somewhat imperfect Ethernet emulator. For one, it has a problem with broadcast amplification; the machine sending the broadcast sends it to all the other nodes that should receive it (up to a set maximum). I wouldn’t want to have a lot of programs broadcasting on a slow link. While in theory this could let you run Netware or DECNet across Zerotier, I’m not really convinced there’s much call for that these days, and Zerotier is clearly IP-focused as it allocates IP addresses and such anyhow. Zerotier provides special support for emulated ARP (IPv4) and NDP (IPv6). While you could theoretically run Zerotier as a bridge, this eliminates the zero trust principle, and Tailscale supports subnet routers, which provide much of the same feature set anyhow.

A somewhat obscure feature, but possibly useful, is Zerotier’s built-in support for multipath WAN for the public interface. This actually lets you do a somewhat basic kind of channel bonding for WAN.

Zerotier: Security and Privacy

The picture here is similar to Tailscale, with the difference that you can create a Zerotier-local account rather than relying on cloud authentication. I was unable to find as much detail about Zerotier as I could about Tailscale - notably I couldn’t find anything about how “sticky” an IP address is. However, the configuration screen lets me delete a node and assign additional arbitrary IPs within a subnet to other nodes, so I think the assumption here is that if your Zerotier account (or the Zerotier control plane) is compromised, an attacker could remove a legit device, add a malicious one, and assign the previous IP of the legit device to the malicious one. I’m not sure how to mitigate against that risk, as firewalling specific IPs is ineffective if an attacker can simply take them over. Zerotier also lacks anything akin to Tailnet Lock.

For this reason, I didn’t proceed much further in my Zerotier evaluation.

Zerotier: Connectivity and NAT traversal

Like Tailscale, Zerotier has NAT traversal with STUN. However, it looks like it’s more limited than Tailscale’s, and in particular is incompatible with double NAT that is often seen these days. Zerotier operates brokers (“root servers”) that can do relaying, including TCP relaying. So you should be able to connect even from hostile networks, but you are less likely to form a P2P connection than with Tailscale.

Zerotier: Sharing with friends

I was unable to find any special features relating to this in the Zerotier documentation. Therefore, it would be at the same level as Yggdrasil: possible, maybe even not too difficult, but without any specific help.

Zerotier: DNS

Unlike Tailscale, Zerotier does not support automatically adding DNS entries for your hosts. Therefore, your options are approximately the same as Yggdrasil, though with the added option of pushing configuration pointing to your own non-Zerotier DNS servers to the client.

Zerotier: Source code, pricing, and portability

The client ZeroTier One is available on Github under a custom “business source license” which prevents you from using it in certain settings. This license would preclude it being included in Debian. Their library, libzt, is available under the same license. The pricing page mentions a community edition for self hosting, but the documentation is sparse and it was difficult to understand what its feature set really is.

The free plan lets you have 1 user with up to 25 devices. Paid plans are also available.

Zerotier conclusions

Frankly I don’t see much reason to use Zerotier. The “virtual Ethernet” model seems to be a weird hybrid that doesn’t bring much value. I’m concerned about the implications of a compromise of a user account or the control plane, and it lacks a lot of Tailscale features (MagicDNS and sharing). The only thing it may offer in particular is multipath WAN, but that’s esoteric enough – and also solvable at other layers – that it doesn’t seem all that compelling to me. Add to that the strange license and, to me anyhow, I don’t see much reason to bother with it.

Netmaker

Netmaker is one of the projects that is making noise these days. Netmaker is the only one here that is a wrapper around in-kernel Wireguard, which can make a performance difference when talking to peers on a 1Gbps or faster link. Also, unlike other tools, it has an ingress gateway feature that lets people that don’t have the Netmaker client, but do have Wireguard, participate in the VPN. I believe I also saw a reference somewhere to nodes as routers as with Yggdrasil, but I’m failing to dig it up now.

The project is in a bit of an early state; you can sign up for an “upcoming closed beta” with a SaaS host, but really you are generally pointed to self-hosting using the code in the github repo. There are community and enterprise editions, but it’s not clear how to actually choose. The server has a bunch of components: binary, CoreDNS, database, and web server. It also requires elevated privileges on the host, in addition to a container engine. Contrast that to the single binary that some others provide.

It looks like releases are frequent, but sometimes break things, and have a somewhat more laborious upgrade processes than most.

I don’t want to spend a lot of time managing my mesh. So because of the heavy needs of the server, the upgrades being labor-intensive, it taking over iptables and such on the server, I didn’t proceed with a more in-depth evaluation of Netmaker. It has a lot of promise, but for me, it doesn’t seem to be in a state that will meet my needs yet.

Nebula

Nebula is an interesting mesh project that originated within Slack, seems to still be primarily sponsored by Slack, but is also being developed by Defined Networking (though their product looks early right now). Unlike the other tools in this section, Nebula doesn’t have a web interface at all. Defined Networking looks likely to provide something of a SaaS service, but for now, you will need to run a broker (“lighthouse”) yourself; perhaps on a $5/mo VPS.

Due to the poor firewall traversal properties, I didn’t do a full evaluation of Nebula, but it still has a very interesting design.

Nebula: Security and Privacy

Since Nebula lacks a traditional control plane, the root of trust in Nebula is a CA (certificate authority). The documentation gives this example of setting it up:

./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"
./nebula-cert sign -name "laptop" -ip "192.168.100.2/24" -groups "laptop,home,ssh"
./nebula-cert sign -name "server1" -ip "192.168.100.9/24" -groups "servers"
./nebula-cert sign -name "host3" -ip "192.168.100.10/24"

So the cert contains your IP, hostname, and group allocation. Each host in the mesh gets your CA certificate, and the per-host cert and key generated from each of these steps.

This leads to a really nice security model. Your CA is the gatekeeper to what is trusted in your mesh. You can even have it airgapped or something to make it exceptionally difficult to breach the perimeter.

Nebula contains an integrated firewall. Because the ability to keep out unwanted nodes is so strong, I would say this may be the one mesh VPN you might consider using without bothering with an additional on-host firewall.

You can define static mappings from a Nebula mesh IP to a clearnet IP. I haven’t found information on this, but theoretically if NAT traversal isn’t required, these static mappings may allow Nebula nodes to reach each other even if Internet is down. I don’t know if this is truly the case, however.

Nebula: Connectivity and NAT traversal

This is a weak point of Nebula. Nebula sends all traffic over a single UDP port; there is no provision for using TCP. This is an issue at certain hotel and other public networks which open only TCP egress ports 80 and 443.

I couldn’t find a lot of detail on what Nebula’s NAT traversal is capable of, but according to a certain Github issue, this has been a sore spot for years and isn’t as capable as Tailscale.

You can designate nodes in Nebula as brokers (relays). The concept is the same as Yggdrasil, but it’s less versatile. You have to manually designate what relay to use. It’s unclear to me what happens if different nodes designate different relays. Keep in mind that this always happens over a UDP port.

Nebula: Sharing with friends

There is no particular support here.

Nebula: DNS

Nebula has experimental DNS support. In contrast with Tailscale, which has an internal DNS server on every node, Nebula only runs a DNS server on a lighthouse. This means that it can’t forward requests to a DNS server that’s upstream for your laptop’s particular current location. Actually, Nebula’s DNS server doesn’t forward at all. It also doesn’t resolve its own name.

The Nebula documentation makes reference to using multiple lighthouses, which you may want to do for DNS redundancy or performance, but it’s unclear to me if this would make each lighthouse form a complete picture of the network.

Nebula: Source code, pricing, and portability

Nebula is fully open source (MIT). It consists of a single Go binary and configuration. It is fairly portable.

Nebula conclusions

I am attracted to Nebula’s unique security model. I would probably be more seriously considering it if not for the lack of support for TCP and poor general NAT traversal properties. Its datacenter connectivity heritage does show through.

Roll your own and hybrid

Here is a grab bag of ideas:

Running Yggdrasil over Tailscale

One possibility would be to use Tailscale for its superior NAT traversal, then allow Yggdrasil to run over it. (You will need a firewall to prevent Tailscale from trying to run over Yggdrasil at the same time!) This creates a closed network with all the benefits of Yggdrasil, yet getting the NAT traversal from Tailscale.

Drawbacks might be the overhead of the double encryption and double encapsulation. A good Yggdrasil peer may wind up being faster than this anyhow.

Public VPN provider for NAT traversal

A public VPN provider such as Mullvad will often offer incoming port forwarding and nodes in many cities. This could be an attractive way to solve a bunch of NAT traversal problems: just use one of those services to get you an incoming port, and run whatever you like over that.

Be aware that a number of public VPN clients have a “kill switch” to prevent any traffic from egressing without using the VPN; see, for instance, Mullvad’s. You’ll need to disable this if you are running a mesh atop it.

Other

Combining with local firewalls

For most of these tools, I recommend using a local firewal in conjunction with them. I have been using firehol and find it to be quite nice. This means you don’t have to trust the mesh, the control plane, or whatever. The catch is that you do need your mesh VPN to provide strong association between IP address and node. Most, but not all, do.

Performance

I tested some of these for performance using iperf3 on a 2.5Gbps LAN. Here are the results. All speeds are in Mbps.

Tool iperf3 (default) iperf3 -P 10 iperf3 -R
Direct (no VPN) 2406 2406 2764
Wireguard (kernel) 1515 1566 2027
Yggdrasil 892 1126 1105
Tailscale 950 1034 1085
Tinc 296 300 277

You can see that Wireguard was significantly faster than the other options. Tailscale and Yggdrasil were roughly comparable, and Tinc was terrible.

IP collisions

When you are communicating over a network such as these, you need to trust that the IP address you are communicating with belongs to the system you think it does. This protects against two malicious actor scenarios:

  1. Someone compromises one machine on your mesh and reconfigures it to impersonate a more important one
  2. Someone connects an unauthorized system to the mesh, taking over a trusted IP, and uses the privileges of the trusted IP to access resources

To summarize the state of play as highlighted in the reviews above:

  • Yggdrasil derives IPv6 addresses from a public key
  • tinc allows any node to set any IP
  • Tailscale IPs aren’t user-assignable, but the assignment algorithm is unknown
  • Zerotier allows any IP to be allocated to any node at the control plane
  • I don’t know what Netmaker does
  • Nebula IPs are baked into the cert and signed by the CA, but I haven’t verified the enforcement algorithm

So this discussion really only applies to Yggdrasil and Tailscale. tinc and Zerotier lack detailed IP security, while Nebula expects IP allocations to be handled outside of the tool and baked into the certs (therefore enforcing rigidity at that level).

So the question for Yggdrasil and Tailscale is: how easy is it to commandeer a trusted IP?

Yggdrasil has a brief discussion of this. In short, Yggdrasil offers you both a dedicated IP and a rarely-used /64 prefix which you can delegate to other machines on your LAN. Obviously by taking the dedicated IP, a lot more bits are available for the hash of the node’s public key, making “collisions technically impractical, if not outright impossible.” However, if you use the /64 prefix, a collision may be more possible. Yggdrasil’s hashing algorithm includes some optimizations to make this more difficult. Yggdrasil includes a genkeys tool that uses more CPU cycles to generate keys that are maximally difficult to collide with.

Tailscale doesn’t document their IP assignment algorithm, but I think it is safe to say that the larger subnet you use, the better. If you try to use a /24 for your mesh, it is certainly conceivable that an attacker could remove your trusted node, then just manually add the 240 or so machines it would take to get that IP reassigned. It might be a good idea to use a purely IPv6 mesh with Tailscale to minimize this problem as well.

So, I think the risk is low in the default configurations of both Yggdrasil and Tailscale (certainly lower than with tinc or Zerotier). You can drive the risk even lower with both.

Final thoughts

For my own purposes, I suspect I will remain with Yggdrasil in some fashion. Maybe I will just take the small performance hit that using a relay node implies. Or perhaps I will get clever and use an incoming VPN port forward or go over Tailscale.

Tailscale was the other option that seemed most interesting. However, living in a region with Internet that goes down more often than I’d like, I would like to just be able to send as much traffic over a mesh as possible, trusting that if the LAN is up, the mesh is up.

I have one thing that really benefits from performance in excess of Yggdrasil or Tailscale: NFS. That’s between two machines that never leave my LAN, so I will probably just set up a direct Wireguard link between them. Heck of a lot easier than trying to do Kerberos!

Finally, I wrote this intending to be useful. I dealt with a lot of complexity and under-documentation, so it’s possible I got something wrong somewhere. Please let me know if you find any errors.


This blog post is a copy of a page on my website. That page may be periodically updated.

Review of Secure, Privacy-Respecting Email Services

I’ve been hosting my own email for several decades now. Even before I had access to a dedicated Internet link, I had email via dialup UUCP (and, before that, a FidoNet gateway).

But self-hosting email is becoming increasingly difficult. The time required to maintain spam and virus filters, SPF/DKIM settings, etc. just grows. The importance of email also is increasing. Although my own email has been extremely reliable, it is still running on a single server somewhere and therefore I could stand to have a lot of trouble if it went down while I was unable to fix it

Email with Pretty Good Privacy & Security

(Yes, this heading is a pun.)

There’s a lot of important stuff linked to emails. Family photos. Password resets for banks, social media sites, chat sites, photo storage sites, etc. Shopping histories. In a lot of cases, if your email was compromised, it wouldn’t be all that hard to next compromise your bank account, buy stuff with your Amazon account, hijack your Netflix, etc. There are lots of good resources about why privacy matters; here’s one informative video even if you think you have “nothing to hide”.

There is often a tradeoff between security and usability. A very secure system would be airgapped; you’d always compose your messages and use your secret keys on a system that has no Internet access and never will. Such a system would be quite secure, but not particularly usable.

On the other end of the spectrum are services such as Gmail, which not only make your email available to you, but also to all sorts of other systems within the service that aim to learn about your habits so they can sell this information to advertisers.

This post is about the services in the middle – ones that are usable, can be easily used on mobile devices, and yet make a serious and credible effort to provide better security and privacy than the “big services” run by Google, Yahoo, and Microsoft. Some elements of trust are inherent here; for instance, that the description of the technical nuances of the provider’s services are accurate. (Elements of trust are present in any system; whether your firmware, binaries, etc. are trustworthy.) I used the list at Privacy Tools as a guide to what providers to investigate, supplemented by searches and NoMoreGoogle.

It so happens that most of these services integrate PGP in some way. PGP has long been one of the better ways to have secure communication via email, but it is not always easy for beginners to use. These services make it transparent to a certain degree. None of them are as good as a dedicated client on an airgapped machine, but then again, such a setup isn’t very practical for everyday use. These services give you something better — pretty good, even — but of course not perfect. All of these pay at least lip service to Open Source, some of them actually publishing source for some of their components, but none are fully open.

I pay particular attention to how they handle exchanges with people that do not have PGP, as this kind of communication constitutes the vast majority of my email.

A final comment – if what you really need is an easy and secure way to communicate with one or two people, email itself may not be the right option. Consider Signal.

Protonmail

Protonmail is, in many ways, the gold standard of privacy-respecting email. Every email is stored encrypted in a way that even they can’t see, being decrypted on the client side (using a Javascript PGP implementation or other clients). They definitely seem to be pushing the envelope for security and privacy; they keep no IP logs, don’t require any personal information to set up an account, and go into quite a lot of detail about how your keys are protected.

A side effect of this is that you can’t just access your email with any mail reader. Since the decryption is done on the client side, you pretty much have to use a Protonmail client. They provide clients for iOS and Android, the Web interface, and a “bridge” that exposes IMAP and SMTP ports to localhost and lets you connect a traditional mail client to the system. The bridge, in this case, handles the decryption for you. The bridge works really well and supports Windows, Mac, and Linux, though it is closed source. (The source for the Linux bridge has been “coming soon” for awhile now.) Protonmail provides very good support for bringing your own domain, and in my testing this worked flawlessly. It supports Sieve-based filters, which can also act on envelope recipients (yes!) The web interface is sleek, very well done, tightly integrated, and just generally exceptionally easy to use and just works.

Unfortunately, the mobile clients get the job done for only light use. My opinion: they’re bad. Really bad. For instance:

  • There’s no way to change the sort order on a mail folder
  • The Android client has an option to automatically download all message bodies. The iOS client lacks this option, but no matter; it doesn’t work on Android anyhow.
  • They’re almost completely unusable offline. You can compose a brand new message but that’s it.

There are some other drawbacks. For one, they don’t actually encrypt mail metadata, headers, or subject lines (though this is common to all of the solutions here, Protonmail’s marketing glosses over this). They also seem to have a lot of problems with overly-aggressive systems blocking people’s accounts: here’s a report from 2017, and I’ve seen more recent ones from people that had paid, but then had the account disabled. Apparently protonmail is used by scammers a fair bit and this is a side-effect of offering free, highly secure accounts – some of their deactivations have been legitimate. Nevertheless, it makes me nervous, especially given the high number of reports of this on reddit.

Unfortunately, Proton seems more focused on new products than on fixing these issues. They’ve been long-simmering in the community but what they talk about is more about their upcoming new products.

Protonmail’s terms of service include both a disclaimer that it’s as-is and an SLA, as well as an indemnification clause. Update 2019-03-04: Protonmail’s privacy policy states they use Matomo analytics, that they don’t record your login IP address by default (but IP logs might be kept if you enable it or if they suspect spamming, etc), collect mobile app analytics, IP addresses on incoming messages, etc. Data is retained “indefinitely” for active accounts and for 14 days after account deletion for closed accounts.

Support: email ticket only

Pricing: $4/mo if paid annually; includes 5GB storage, 5 aliases, and 1 custom domain

Location: Switzerland

MFA: TOTP only

Plus address extensions: yes

Transparency report: yes

Mailfence

Mailfence is often mentioned in the same sentence as ProtonMail. They also aim to be a privacy-respecting, secure email solution.

While it is quite possible they use something like LUKS to encrypt data at rest (safeguarding it from a stolen hard drive), unlike ProtonMail, Mailfence does have access to the full content of any plaintext messages sent or received by your account. Mailfence integrates PGP into the Web interface, claiming end-to-end encryption with a “zero-knowledge environment” using, of all things, the same openpgpjs library that is maintained by ProtonMail. While ProtonMail offers a detailed description of key management, I haven’t been able to find this with Mailfence – other than that the private key is stored encrypted on their servers and is protected by a separate passphrase from the login. If we assume the private key is decrypted on the client side, then for PGP-protected communications, the level of security is similar to ProtonMail. With Mailfence, decrypting these messages is a separate operation, while with ProtonMail it happens automatically once logged in. (Update 2019-03-01: Mailfence emailed me, pointing to their document on key storage – it is AES-256 encrypted by the client and stored on the server. They also passed along a link describing their PGP keystore. They also said they plan to work on a feature th encrypt plain text messages.)

While technical measures are part of the story, business policies are another, and Mailfence does seem to have some pretty good policies in place.

In experimenting with it, I found that Mailfence’s filters don’t support filtering based on the envelope recipient, which limits the utility of its aliases since BCC and the like won’t filter properly. A workaround might be possible via the IMAP connection filtering based on Received: headers, but that is somewhat ugly.

Mailfence also supports “secure” documents (word processor, spreadsheet, etc), WebDAV file storage, contacts, and calendars. There is no detail on what makes it “secure” – is it just that it uses TLS or is there something more? I note that the online document editor goes to a URL under writer.zoho.com, so this implies some sort of leakage to me and a possible violation of their “no third-party access to your data” claim. (Update 2019-03-01: Mailfence emailed me to point out that, while it’s not disclosed on the page I liked to, it is disclosed on their blog, and that since I evaluated it, they added a popup warning in the application before sending the documents to Zoho.)

Mailfence supports POP, IMAP, SMTP, and — interestingly — Exchange ActiveSync access to their services. I tested ActiveSync on my Android device, and it appeared to work exactly as planned. This gives a lot of client flexibility and very nice options for calendar and contact sync (*DAV is also supported).

Mailfence’s terms of use is fairly reasonable, though it also includes an indemnification clause. It makes no particular uptime promises. Update 2019-03-04: Per their privacy policy, Mailfence logs IP addresses and use Matomo analytics on the website but not within the application. Deleted messages and documents are retained for 45 days. The policy does not specify retention for logs.

Support: email ticket or business-hours phone support for paying customers

Pricing: EUR 2.50/mo paid annually, includes 5GB storage, 10 aliases, and 1 custom domain

Location: Belgium

MFA: TOTP only

Plus address extensions: Yes

Transparency report: Yes

Mailbox.org

Mailbox.org has been in the hosting business for a long time, and also has a privacy emphasis. Their security is conceptually similar to that of Mailfence. They offer two web-based ways of dealing with PGP: OX Guard and Mailvelope. Mailvelope is a browser extension that does all encryption and decryption on the client side, similar to Mailfence and ProtonMail. OX Guard is part of the Open-Xchange package which mailbox.org uses. It stores the encryption keys on the server, protected by a separate key passphrase, but all encryption and decryption is done server-side. Mailbox’s KB articles on this makes it quite clear and spell out the tradeoffs. The basic upshot is that messages you receive in plaintext will still be theoretically visible to the service itself.

Mailbox.org offers another interesting feature: automatic PGP-encryption of any incoming email that isn’t already encrypted. This encrypts everything inbound. If accessed using Mailvelope or some other external client, it provides equivalent security to ProtonMail. (OX Guard is a little different since the decryption happens server-side.)

They also offer you an @secure.mailbox.org email address that will reject any incoming mail that isn’t properly secured by TLS. You can also send from that address, which will fail to send unless the outgoing connection is properly secured as well. This is one of the more interesting approaches to dealing with the non-PGP-using public. Even if you don’t use that, if you compose in their web interface, you get immediate feedback about the TLS that will be used. It’s not end-to-end, but it’s better than nothing. Mailfence and Protonmail both offer an “secure email” that basically emails a link to a recipient, that links back to their server and requires the recipient to enter a password that was presumably exchanged out of band. Mailbox Guard will automatically go this route when you attempt to send email to someone for whom the PGP keys weren’t known, but goes a step further and invites them to reply there or set up their PGP keys.

Mailbox.org runs Open-Xchange, a semi-Open Source web-based office suite. As such, it also offers calendar, contacts, documents, task lists, IMAP/SMTP/POP, ActiveSync, and so forth. Their KB specifically spells out that things like the calendar are not encrypted with PGP. The filtering does the right thing with envelope recipients.

Mailbox.org has an amazingly comprehensive set of options, a massive knowledge base, even a user forum. Some of the settings I found to be interesting, besides the ones already mentioned, include:

  • Spam settings: greylisting on or off, RBL use, executable file attachment blocking, etc.
  • Restoring email from a backup
  • Disposable addresses (automatically deleted after 30 days)
  • A “catch-all” alias, that just counts as one of your regular aliases, and applies to all usernames under a domain not otherwise aliased.

I know Protonmail has frequent third-party security audits; I haven’t seen any mention of this on the mailbox.org site. However, it looks probable that less of their code was written in house, and it may have been audited without a mention.

Overall, I’ve been pretty impressed with them. They give details on EVERYTHING. It’s the geeky sort of comprehensive, professional solution I’d like. I wish it would have full end-to-end transparent encryption like ProtonMail, but honestly what they’re doing is more practical and useful to a lot of folks.

Mailbox has a reasonable T&C (though it does include an indemnity clause as many others do) and a thorough data protection and privacy policy. Some providers don’t log IP addresses at all; mailbox.org does, but destroys them after 4 days. (Update 2019-03-04: Discovered that all of the providers reviewed may do this at times; updated the other reviews and removed incorrect text; mailbox.org’s is actually one of the better policies) mailbox.org goes into a lot more detail than others, and also explicitly supports things such as Tor for greater anonymity.

Support: email ticket (phone for business-level customers)

Pricing: EUR 1/mo for 2GB storage and 3 aliases; EUR 2.50/mo for 5GB storage and 25 aliases. Expansions possible (for instance, 25GB storage costs a total of EUR 3.50/mo)

Location: Germany

MFA: Yubikey, OATH, TOTP, HOTP, MOTP (web interface only)

Plus address extensions: Yes

Transparency Report: Yes

Startmail

Startmail is a service from the people behind the privacy-respecting search engine Startpage. There is not a lot of information about the technical implementation of Startmail, with the exception of a technical white paper from 2016. It is unclear if this white paper remains accurate, but this review will assume it is. There are also some articles in the knowledge base.

I was unable to fully review Startmail, because the free trial is quite limited (doesn’t even support IMAP) and anything past that level requires an up-front payment of $60. While I paid a few dollars for a month’s real account elsewhere, this was rather too much for a few paragraphs’ review.

However, from the trial, it appears to have a feature set roughly akin to Mailfence. Its mail filters are actually more limited, and it’s mail only: no documents, calendars, etc.

Startmail a somewhat unique setup, in which a person’s mail, PGP keys, etc. are stored in a “vault” which turns out to be a LUKS-encrypted volume. This vault is opened when a person logs in and closed when they log out, and controlled by a derivative of their password. On the one hand, this provides an even stronger level of security than Protonmail (since headers are also encrypted). On the other hand, when the vault is “open” – when one must presume it is quite frequently for an account being polled by IMAP – it is no better than anything else.

They explicitly state that they have not had a third-party audit.

Support: ticket only

Pricing: $60/yr ($5/mo), must be paid as an entire year up-front

Location: Netherlands

MFA: TOTP only

Plus address extensions: unknown

Transparency report: no

Not Reviewed

Some other frequently-used providers I didn’t review carefully:

  • posteo.de: encrypts your mail using a dovecot extension that decrypts it using a derivation of your password when you connect. Something better than nothing but less than Protonmail. Didn’t evaluate because it didn’t support my own domain.
  • Tutanota: Seems to have a security posture similar to ProtonMail, but has no IMAP support at all. If I can’t use emacs to read my mail, I’m not going to bother.

Conclusions

The level of security represented by Protonmail was quite appealing to me. I wish that the service itself was more usable. It looks like an excellent special-needs service, but just isn’t quite there yet as a main mail account for people that have a lot of mail.

I am likely to pursue mailbox.org some more, as although it isn’t as strong as Protonmail when it comes to privacy, it is still pretty good and is amazing on usability and flexibility.

A Final Word on Trust

Trust is a big part of everything going on here. For instance, if you use ProtonMail, where does trust come into play? Well, you trust that they aren’t serving you malicious JavaScript that captures your password and sends it to them out of band. You trust that your browser provides a secure environment for JavaScript and doesn’t have leakage. Or if you use mailbox.org, you trust that the server is providing a secure environment and that when you supply your password for the PGP key, it’s used only for that. ProtonMail will tell you how great it is to have this code client-side. Startmail will tell you how bad Javascript in a browser is for doing things related to security. Both make good, valid points.

To be absolutely sure, it is not possible or practical for any person to verify every component in their stack on every use. Different approaches have different trust models. The very best is still standalone applications.

The providers reviewed here raise the average level of privacy and security on the Internet, and do it by making it easier for the average user. That alone is a good thing and worthy of support. None of them can solve every problem, but all of them are a step up from the standard, which is almost no security at all.

Review: Linux IM Software

I’ve been looking at instant messaging and chat software lately. Briefly stated, I connect to Jabber and IRC networks from at least three different computers. I don’t like having to sign in and out on different machines. One of the nice features about Jabber (XMPP) is that I can have clients signing in from all over the place and it will automatically route messages to the active one. If the clients are smart enough, that is.

Gajim

I have been using Gajim as my primary chat client for some time now. It has a good feature set, but has had a history of being a bit buggy for me. It used to have issues when starting up: sometimes it would try to fire up two copies of itself. It still has a bug when being fired up from a terminal: if you run gajim & exit, it will simply die. You have to wait a few seconds to close the terminal you launched it from. It has also had issues with failing to reconnect properly after a dropped network connection and generating spurious “resource already in use” errors. Upgrades sometimes fix bugs, and sometimes introduce them.

The latest one I’ve been dealing with is its auto-idle support. Sometimes it will fail to recognize that I am back at the machine. Even weirder, sometimes it will set one of my accounts to available status, but not the other.

So much for my complaints about Gajim; it also has some good sides. It has excellent multi-account support. You can have it present your multiple accounts as separate sections in the roster, or you can have them merged. Then, say, all your contacts in a group called Friends will be listed together, regardless of which account you use to contact them.

The Jabber protocol (XMPP) permits you to connect from multiple clients. Each client specifies a numeric priority for its connection. When someone sends you a message, it will be sent to the connection with the highest priority. The obvious feature, then, is to lower your priority when you are away (or auto-away due to being idle), so that you always get IMs at the device you are actively using. Gajim supports this via letting you specify timeouts that get you into different away states, and using the advanced configuration editor, you can also set the priority that each state goes to. So, if Gajim actually recognized your idleness correctly, this would be great.

I do also have AIM and MSN accounts which I use rarely. I run Jabber gateways to each of these on my server, so there is no need for me to use a multiprotocol client. That also is nice because then I can use a simple Jabber client on my phone, laptop, whatever and see all my contacts.

Gajim does not support voice or video calls.

Due to an apparent bug in Facebook, the latest Gajim release won’t connect to Facebook servers, but there is a patch that claims to fix it.

Psi

Psi is another single-protocol Jabber client, and like Gajim, it runs on Linux, Windows, and MacOS. Psi has a nicer GUI than Gajim, and is more stable. It is not quite as featureful, and one huge omission is that it doesn’t support dropping priority on auto-away (though it, weirdly, does support a dropped priority when you manually set yourself away).

Psi doesn’t support account merging, so it always shows my contacts from one account separately from those from another. I like having the option in Gajim.

There is a fork of Psi known variously as psi-dev or psi-plus or Psi+. It adds that missing priority feature and some others. Unfortunately, I’ve had it crash on me several times. Not only that, but the documentation, wiki, bug tracker, everything is available only in Russian. That is not very helpful to me, unfortunately. Psi+ still doesn’t support account merging.

Both branches of Psi support media calling.

Kopete

Kopete is a KDE multiprotocol instant messenger client. I gave it only about 10 minutes of time because it is far from meeting my needs. It doesn’t support adjustable priorities that I can tell. It also doesn’t support XMPP service discovery, which is used to do things like establish links to other chat networks using a Jabber gateway. It also has no way to access ejabberd’s “send message to all online users” feature (which can be accessed via service discovery), which I need in emergencies at work. It does offer multimedia calls, but that’s about it.

Update: A comment pointed out that Kopete can do service discovery, though it is in a very non-obvious place. However, it still can’t adjust priority when auto-away, so I still can’t use it.

Pidgin

Pidgin is a multiprotocol chat client. I have been avoiding it for years, with the legitimate fear that it was “jack of all trades, master of none.” Last I looked at it, it had the same limitations that Kopete does.

But these days, it is more capable. It supports all those XMPP features. It supports priority dropping by default, and with a plugin, you can even configure all the priority levels just like with Gajim. It also has decent, though not excellent, IRC protocol support.

Pidgin supports account merging — and in fact, it doesn’t support any other mode. You can, for instance, tell it that a given person on IRC is the same as a given Jabber ID. That works, but it’s annoying because you have to manually do it on every machine you’re running Pidgin on. Worse, they used to support a view without merged accounts, but don’t anymore, and they think that’s a feature.

Pidgin does still miss some nifty features that Gajim and Psi both have. Both of those clients will not only tell you that someone is away, but if you hover over their name, tell you how long someone has been away. (Gajim says “away since”, while Pidgin shows “last status at”. Same data either way.) Pidgin has the data to show this, but doesn’t. You can manually find it in the system log if you like, but unhelpfully, it’s not on the log for an individual person.

Also, the Jabber protocol supports notifications while in a chat: “The contact is typing”, paying attention to a conversation, or closed the chat window. Psi and Gajim have configurable support for these; you can send whatever notifications your privacy preferences say. Pidgin, alas, removed that option, and again they see this as a feature.

Pidgin, as a result, makes me rather nervous. They keep removing useful features. What will they remove next?

It is difficult to change colors in Pidgin. It follows the Gtk theme, and there is a special plugin that will override some, but not all, Gtk options.

Empathy

Empathy supports neither priority dropping when away nor service discovery, so it’s not usable for me. Its feature set appears sparse in general, although it has a unique desktop sharing option.

Update: this section added in response to a comment.

On IRC

I also use IRC, and have been using Xchat for that for quite some time now. I tried IRC in Pidgin. It has OK IRC support, but not great. It can automatically identify to nickserv, but it is under-documented and doesn’t support multiple IRC servers for a given network.

I’ve started using xchat with the bip IRC proxy, which makes connecting from multiple machines easier.

The Demise of PC Magazine

I just read the news that PC Magazine is being canceled. It’s not exactly a shock, given the state of technical magazines right now. I haven’t read one of those in years, since they turned to be more of a consumer than a technical publication.

But I hope I am not the only one out there that remembers PC Magazine from the mid to late 1980s. I had two favorite parts in each issue: the programming example, and the “Abort, Retry, Fail” page at the back of the magazine.

The programming example was usually some sort of DOS (or, on occasion, OS/2) utility. It was usually written in assembly, and would be accompanied by a BASIC program you could type in to get the resulting binary, as assemblers weren’t readily available. The BASIC program was line after line of decimal numbers that would decode them and write out the resulting binary — sort of a primitive uuencode for paper. Trying to type those in gave me some serious eyestrain on more than one occasion. By now, I forget what most of those utilities did, but I remember one: BatchMan. It was a collection of tools for use in DOS batch files, and could do things like display output in color or even — yes — play monophonic music. It came with an example that displayed some lyrics about batch programming on-screen, set to what I later realized was the Batman theme. Geek nirvana, right?

But Batchman was too big to publish the source code, or the BASIC decoder, in print. It might have been one of those things that eventually led me to a CompuServe account. PC Magazine had some deal with CompuServe that you could get their utilities for free, or reduced cost — I forget. CompuServe was probably where I sent my first email, from my account which was 71510,1421 — comma and all. In later years, you could pay a small fee to send email to the Internet, and I had the amazingly attractive email address of 71510.1421@cis.compuserve.com. Take that, gmail.

PC Magazine eventually stopped running utilities that taught people about assembly or batch programming and shifted more to the genre of Windows screensavers. They stopped their articles about how hard disks work and what SCSI is all about, and instead have cover stories like “Vista made easy!” I am, sadly, not making this up. Gone are the days of investigating alternative operating systems like OS/2.

It appears that “Abort, Retry, Fail” is gone, too. It was a one-page thing at the back of each magazine that featured braindead error messages and funny stories about people that did things like FAX an image of a floppy disk to a remote office — before such stories were cliche. Sort of like DailyWTF these days. The sad truth is that the people that would FAX an image of a floppy are probably the ones that are reading PC Magazine today.

I still have a bunch of PC Magazine issues — the good ones — in my parents’ basement. I also still have my floppies with the utilities on them somewhere. One day, when I get some time — I’m estimating this will be about when Jacob goes to college — I’ll go back and take another look at them.

Quick and Easy IPv6 for Debian


A lot of people have asked about IPv6 in Debian. There have been some instructions floating around, but all of them I’ve seen are overly complex. Here’s how to set up your own 6to4 tunnel in about 5 minutes (assuming your kernel is IPV6-ready), without the need of freenet6 or any other tunnel broker. You need only a real IP address (static is best) and a basic understanding of IPv6 to proceed. This article will configure your host or your router.




These instructions set you up with 6to4, which requires no outside tunnel broker. However, there are not many 6to4 routers out there. If you are connecting to other non-6to4 sites, chances are god that performance will not be good. This is not a flaw in IPv6 itself. I suggest setting up 6to4 first, since it is fairly easy; once you have it working, then move on to others if you like.



First, you need to obtain an IPV6-ready kernel. I strongly recommend 2.6.1 or above if possible. Check the IPv6 kernel system check page to make sure your kernel is IPV6-ready, and for info on compiling a new kernel if not. In addition to basic IPv6, I also recommend that you compile in IPv6 netfilter support.


Next, you need to add a tunnel to your /etc/network/interfaces file. First, you will need to know your public IP address in IPv4. It will look something like 10.20.30.40. Next, you need to get that in IPv6 notation. Here’s a quick shell script to do that:



#!/bin/sh
printf “2002:%x%02x:%x%02x::\n” `echo $1 | sed ‘s/\./ /g’`


Just run that with your IP address as an argument. In this example, for 10.20.30.40, the result is 2002:a14:1e28::. This is your prefix. All your IP addresses will begin with that. Please see the link above for more on IPv6 addressing if you don’t understand the “::” part of this.


Now, you have all the information to create your own IPv6 tunnel. Edit /etc/network/interfaces and add these lines:



iface sit1 inet6 v4tunnel
address 2002:a14:1e28::2
netmask 64
endpoint any
local 10.20.30.40
up ip -6 route add 2000::/3 via ::192.88.99.1 dev sit1
down ip -6 route flush dev sit1
up /etc/network/ipv6rules.sh
ttl 64


The address line contains the IPv6 address you calculated above, followed by a “2”. The local line contains your local IP address. Now bring up the link with ifup sit1. You should now be able to run ping6 www.ipv6.org and get results back. If you don’t have ping6 on your system, install the iputils-ping package. If this works, add sit1 to the auto line in /etc/network/interfaces.


The /etc/network/ipv6rules.sh is a little script that closes off some ports to your system. If you don’t want to use it, delete that “up” line. Here’s one version that I recommend:



#!/bin/bash
ip6tables -F
ip6tables -I INPUT -i sit+ -p tcp –syn -j DROP
ip6tables -I FORWARD -i sit+ -p tcp –syn -j DROP
ip6tables -I INPUT -i sit+ -p udp \! –dport 32768:60999 -j DROP
ip6tables -I FORWARD -i sit+ -p udp \! –dport 32768:60999 -j DROP
ip6tables -I INPUT -i sit+ -p tcp –dport 22 -j ACCEPT
ip6tables -I FORWARD -i sit+ -p tcp –dport 22 -j ACCEPT
# Uncomment the following lines if this is a router
#echo 0 > /proc/sys/net/ipv6/conf/all/autoconf
#echo 0 > /proc/sys/net/ipv6/conf/all/accept_ra
#echo 0 > /proc/sys/net/ipv6/conf/all/accept_redirects
#echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
#echo 0 > /proc/sys/net/ipv6/conf/all/router_solicitations


This script will close off incoming TCP connections, and UDP connections to low UDP ports, except for TCP to port 22 (ssh).


If you are setting up a host, you’re done. If this is a router, read on…


Router Configuration


If you’re setting up a router, there are a couple more quick steps. First, you need to configure your ethernet interface for ipv6. Insert a clause like this in /etc/network/interfaces:



iface eth0 inet6 static
address 2002:a14:1e28:1::1
netmask 64


Of course, replace the first first part of “address” with your real IPv6 address. (Note the added “:1::1” after the address.) Now run ifdown eth0; ifup eth0 to make the changes take effect.


Next, apt-get install radvd and edit /etc/radvd.conf. It should end up looking like this:



interface eth0
{
AdvLinkMTU 1480;
AdvSendAdvert on;
prefix 2002:a14:1e28:1::1/64
{
};
};


Mind the semicolons (and lack thereof); radvd is picky. Now /etc/init.d/radvd restart and use ps to make sure it’s running. radvd is similar to dhcp for IPv6, but a lot easier.


At this point, your IPv6 network is ready. All clients on your network that are IPv6 capable should automatically assign themselves an IPv6 address and be ready to go. For Debian clients, all you need is IPv6 support in your kernel; you do not need to do anything on them at all.


Revisions


  • Added a note about performance (1/19/2003 7PM). Suggested by Jeroen Massar.

  • Adjusted netmasks and router subnet (1/20). Suggested by Jeroen Massar.

  • Added ttl 64 (1/20). Adapted from a suggested from Thomas Habets.

  • Corrected sit0 to read sit1