A little while back, I spent a week in a remote area. It had no Internet and no cell phone coverage. Sometimes, I would drive in to town where there was a signal to get messages, upload photos, and so forth. I had to take several devices with me: my phone, my wife’s, maybe a laptop or a tablet too. It seemed there should have been a better way. And there is.
I’ll use this example to talk about a mesh network, but it could just as well apply to people wanting to communicate on a 12-hour flight that has no in-flight wifi, or spacecraft with an intermittent connection, or a person traveling.
Syncthing makes a wonderful solution for things like these. Here are some interesting things about Syncthing:
- You can think of Syncthing as a serverless, peer-to-peer, open source alternative to Dropbox. Machines sync directly with each other without a server, though you can add a server if you want.
- It can operate completely without Internet access or any central server, though if Internet access is available, it can readily be used.
- Syncthing devices connected to the same LAN or Wifi will detect each other’s presence and automatically communicate.
- Syncthing is capable of handling a constantly-changing topology. It can also, for instance, handle two disconnected clusters of nodes with one node that “travels” between them — perhaps just a phone.
- Syncthing scales from everything from a phone to thousands of nodes.
- Syncthing normally performs syncs in every direction, but can also do single-direction syncs
- An individual Syncthing node can register its interest or disinterest in certain files or directories based on filename patterns
Syncthing works by having you define devices and folders. You can choose which devices to share folders with. A shared folder has an ID that is unique across Sycnthing. You can share a folder from device A to device B, and then device B can share it with device C, even if A and C don’t know about each other or have no way to communicate. More commonly, though, all the devices would know about each other and will opportunistically communicate the best way they can.
Syncthing uses something akin to a Bittorrent protocol. Say you’re syncing videos from your phone, and they’re going to 3 machines. It doesn’t mean that Syncthing has to send it three times from the phone. Syncthing will send each block, most likely, just once; the other nodes in the swarm will register the block availability from the first other node to get it and will exchange blocks with themselves.
Syncthing will typically look for devices on the local LAN. Failing that, it will use an introduction server to see if it can reach them directly using P2P. Failing that, perhaps due to restrictive firewalls or NAT, communication can be relayed through volunteer-run Syncthing servers on the Internet. All Syncthing communications are cryptographically encrypted and verified. You can also configure Syncthing arbitrarily; for instance, to run over ssh or Tor tunnels.
So, let’s look at how Syncthing might help with the example I laid out up front.
All the devices at the remote location could communicate with each other. The Android app is quite capable of syncing photos and videos using Syncthing, for instance. Then one device could be taken to the Internet location and it would transmit data on behalf of all the others – perhaps back to a computer at your home, or to a server somewhere. Perhaps a script running on the remote server would then move files out of the syncthing synced folder into permanent storage elsewhere, triggering a deletion to be sent to the phone to free up storage. When the phone gets back to the other devices, the deletion can be propagated to them to free up storage there too.
Or maybe you have a computer out in a shed or somewhere without Internet access that you go to periodically, and need to get files to it. Again, your phone could be a carrier.
Taking it a step further
If you envision a file as a packet, you could, conceivably, do something like tunnel TCP/IP over Syncthing, assuming generous-enough timeouts. It can truly handle communication.
But you don’t need TCP/IP for this. Consider some other things you could do:
- Drop a script in a special directory that gets picked up by a remote server and run
- Drop emails in a special directory that get transmitted and then deleted by a remote system when they’re seen
- Drop files (eg, photos or videos) in a directory that a remote system will copy or move out of there
- Drop messages (perhaps gpg-encrypted) — which could be text files — for someone to see and process.
- Drop NNTP bundles for group communication
You can start to see how there are a lot of possibilities here that extend beyond just file synchronization, though they are built upon a file synchronization tool.
Enter NNCP
Let’s look at a tool that’s especially suited for this: NNCP, which I’ve been writing about a lot lately.
NNCP is designed to handle file exchange and remote execution with remote computers in an asynchronous, store-and-forward manner. NNCP packets are themselves encrypted and authenticated. NNCP traditionally is source-routed (that is, you configure it so that machine A reaches machine D by relaying through B and C), and the packets are onion-routed. NNCP packets can be exchanged by a TCP call, a tar-like stream, copying files to something like a USB stick and physically transporting it to the remote, etc.
This works really well and I’ve been using it myself. But it gets complicated if the network topology isn’t fixed; it is difficult to reroute packets due to the onion routing, for instance. There are various workarounds that could be used — but why not just use Syncthing as a transport in those cases?
nncp-xfer is the command that exchanges packets by writing them to, and reading them from, a directory. It is what you’d use to exchange packets on a USB stick. And what you’d use to exchange packets via Syncthing. It writes packets in a RECIPIENT/SENDER/PACKET directory structure, so it is perfectly fine to have multiple systems exchanging packets in a single Syncthing synced folder tree. This structure also allows leaf nodes to only carry the particular packets they’re interested in. The packets are all encrypted, so they can be freely synced wherever.
Since Syncthing opportunistically syncs a shared folder with any device the folder is shared with, a phone could very easily be the NNCP transport, even if it has no idea what NNCP is. It could carry NNCP packets back and forth between sites, or to the Internet, or whatever.
NNCP supports file transmission, file request, and remote execution, all subject to controls, of course. It is easy to integrate with Exim or Postfix to use as a mail transport, Git transport, and so forth. I use it for backups. It would be quite easy to have it send those backups (encrypted zfs send) via nncp-xfer to Syncthing instead of the usual method, and then if I’ve shared the Syncthing folder with my phone, all I need to do is bring the phone into Internet range and they get sent. nncp-xfer will normally remove the packets out of the xfer directory as it ingests them, so the space will only be consumed on the phone (and laptop) until we know the packets made it to their destination.
Pretty slick, eh?
A Simple, Delay-Tolerant, Offline-Capable Mesh Network with Syncthing (+ optional NNCP) | The Changelog
changelog.complete.org/archives/10219…
syncthingはこれか
syncthing.net
Have you looked at CouchDB? It serves some of the same use cases, but with a different approach. It’s very popular for offline-first applications. Two real-world examples I’ve heard:
1. forestry applications, where a mobile device is frequently in the middle of a national forest, with no network connection, but it can easily sync back to a centralized data store when back within range.
2. Fighting the Ebola outbreak in parts of Africa where network connecitivity was sporadic, at best. Medics in the field could track local data on tablets, to be easily synced with other sites when the network was made available, to allow global analytics.
It’s sometimes called “A replication protocol, with a database tacked on.”
I’ve been working on a project that allows bi-directional sync between CouchDB and a filesystem directory. Makes me think it could easily be coupled with something like syncthing…
It is definitely is on my list to look into. https://verbnetworks.com/research/store-forward-networks/ planted that idea in my head and it looks quite interesting.
A Simple, Delay-Tolerant, Offline-Capable Mesh Network with Syncthing (+ optional NNCP) | The Changelog changelog.complete.org/archives/10219…
Recently I have been taking another look at the services at rsync.net and it got me thinking: what would I do with a lot of storage? What might I want to run with it, if it were fairly cheap?
Backups are an obvious place to start. Borgbackup makes a pretty compelling option: very bandwidth-efficient thanks to block-level rolling hash dedup, encryption fully on the client side, etc. Borg can run over ssh, though does need a server-side program.
Nextcloud is another option. With Google Photos getting quite expensive now, if you could have a TB of storage that you control, what might you do with it? Nextcloud also includes IM, video chat, and online document editing similar to Google Docs.
I’ve written before about the really neat properties of Syncthing: distributed synchronization that needs no server component. It also supports untrusted nodes in the mesh, where all content is encrypted before it reaches them. Sometimes an intermediary node is useful; for instance, if nodes A and C are to sync but are rarely online at the same time, an untrusted node B that is always online can facilitate synchronization. A server with some space could help with this.
A relay for NNCP or UUCP.
More broadly, you could self-host your photo or video cllection.
Let’s start taking a look at what’s out there. I’m going to try to focus on things that are unique for some reason: pricing, features, etc. Incidentally, good reviews are hard to find due to the proliferation of affiliate links. I have no affiliate relationships with anyone mentioned here and there are no affiliate links in this post.
I’ll start with the highest-end community and commercial options (though both are quite competitive on price for what they are), and then move on to the cheaper options.
Community option: SDF
SDF is somewhat hard to define. “What is SDF?” could prompt answers like:
A community-run network offering free Unix shells to the public
A diverse community of people that connect with unique tools. A social network in the 80s sense, sort of.
A provider of… let me see… VPN, DSL, and even dialup access.
An organization that runs various Open Source social network services, including Mastodon, Pixelfed (image sharing), PeerTube (video sharing), WordPress, even Minecraft.
A provider of various services for a nominal charge: $3/mo gets you access to the MetaArray with 800GB of storage space which you have shell access to, and can store stuff on with Nextcloud, host public webpages, etc.
Thriving communities around amateur radio, musicians, Plan 9, and even – brace yourself – TOPS-20, a DEC operating system first released in 1976 and not updated since 1988.
There’s even a Wikipedia article about SDF.
There’s a lot there. SDF lets you use things for yourself, of course, but you can also join a community. It’s not a commercial service backed by SLAs — it’s best-effort — but it’s been around more than 30 years and has a great track record.
Top commercial option for backup storage: rsync.net
rsync.net offers storage broadly over SSH: sftp, rsync, scp, borg, rclone, restic, git-annex, git, and such. You do not get a shell, but you do get to run a few noninteractive commands via ssh. You can, for instance, run git clone on the rsync server.
The rsync special sauce is in ZFS. They run raidz3 on their arrays (and also offer dual location setups for an additional fee), offer both free and paid ZFS snapshots, etc. The service is designed to be extremely reliable, particularly for backups, and it seems to me to meet those goals.
Basic storage is $0.025 per GB/mo, but with certain account types such as borg, can be had for $0.015 per GB/mo. The minimum size is 400GB or $10/mo. There are no bandwidth charges. This makes it quite economical even compared to, say, S3. Additional discounts start at 10TB, so 10TB with rsync.net would cost $204.80/mo or $81.92 on the borg plan.
You won’t run Nextcloud on this thing, but for backups that must be reliable, or even a photo collection or something, it makes perfect sense.
When you look into other options, you’ll find that other providers are a lot more vague about their storage setup than rsync.net.
Various offerings from Hetzner
Hetzner is one of Europe’s large hosting companies, and they have several options of interest.
Their Storage Box competes directly with the rsync.net service. Their per-GB storage cost is lower than rsync.net, and although they do include a certain amount of free bandwidth with each account, bandwidth is not unlimited and could result in charges. Still, if you don’t drive 2x or more your storage usage in bandwidth each month, it would be cheaper than rsync. The Storage Box also uses ZFS with some kind of redundancy, though they don’t specifcy details.
What differentiates them from rsync.net is the protocol support. They support sftp, scp, Borg, ssh, rsync, etc. just as rsync.net does. But then they also throw in Samba/CIFS, FTPS, HTTPS, and WebDAV – all optionally enabled or disabled by you. Although things like sshfs exist, they aren’t particularly optimal for some use cases, and CIFS support may just be what you need in some situations.
10TB with Hetzner would cost EUR 39.90/mo, or about $48.84/mo. (This figure is higher for Europeans, who also have to pay VAT.)
Hetzner also offers a Storage Share, which is a private Nextcloud instance. 10TB of that is exactly the same cost as 10TB of the Storage Box. You can add your own users, groups, etc. to this as your are the Nextcloud admin of your instance. Hetzner throws in automatic updates (which is great, as updates have been a pain in my side for a long time). Nextcloud is ideal for things like photo sharing, even has email and chat built in, etc. For about the same price at 2TB of Google One, you can have 2TB of Nextcloud with all those services for yourself. Not bad. You can also mount a Nextcloud instance with WebDAV.
Interestingly, Nextcloud supports “external storages” as backend for the data. It supports another Nextcloud instance, OpenStack or S3 object storage, and SFTP, SMB/CIFS, and WebDAV. If you’re thinking you’d like both SFTP and Nextcloud access to a pool of storage, I imagine you could always get a large Storage Box from Hetzner (internal transfer is free), pair it with a small Nextcloud instance, and link the two with Nextcloud external storage.
Dedicated Servers
If you want a more DIY approach, you can find some interesting deals on actual dedicated server hardware – you get the entire machine to yourself. I’ve been using OVH’s SoYouStart for a number of years, with good experienaces, and they have a number of server configurations available. For instance, for $45.99, you can get a Xeon box with 4x2TB drives and 32GB RAM. With RAID5 or raidz1, that’s 6TB of available space – and cheaper than the 6TB from rsync.net (though less redundant) plus you get the whole box to yourself too. OVH directly has some more storage servers; for instance, you can get a box with 4x4TB + 1x500GB SSD for $86.75/mo, giving you 12TB available with RAID5/raidz1, plus a 16GB server to do what you want with.
Hetzner also has some larger options available, for instance 2x4TB at EUR39 or 2x8TB at EUR54, both with 64GB of RAM.
Bargain Corner
Yes, you can find 10TB for $25/mo. It’s hosted on ceph, by what appears to be mostly a single person (though with a lot of experience and a fair bit of transparency). You’re not going to have the round-the-clock support experience as with rsync.net, nor its raidz3 level of redundancy – but if you don’t need that, there are quite a few options.
Let’s start with Lima Labs. Yes, 10TB is $25/mo, and they support sftp, rsync, borg, and even NFS mounts on storage backed by Ceph. The owner, Sam, seems to be a nice guy but the service isn’t going to be on the scale of rsync.net or Hetzner. That may or may not be OK for your needs – I mean, you can even get 1TB for $5/mo, so there are some fantastic deals to be had here.
BorgBase does Borg hosting and borg hosting only. You can get 1TB for $6.67/mo or, for instance, 10TB for $53.46. They don’t say much about their infrastructure and it’s hard to get a read on the company, but for Borg backups, it could be a nice option.
Bargain Corner Part 2: Seedboxes
There’s a market out there of companies offering BitTorrent seeding and downloading services. Typically, these services offer you Unix ssh access to a shell, give you a bunch of space on completely non-redundant drives (theory being that the data on them is transient), lots of bandwidth, for a low price. Some people use them for BitTorrent, others for media serving and such.
If you are willing to take the lowest in drive redundancy, there are some deals to be had. Whatbox is a popular leader here, and has an extensive wiki with info. Or you can find some seedbox.io “shared storage” plans – for instance, 12TB for $32.49/mo. But it’s completely non-redundant drives.
Seedbox has a partner company, Walker Servers, with some interesting deals; for instance, 4x8TB for EUR 52.45. Not bad for 24TB usable with RAID5 – but Walker Servers is completely unknown to me and doesn’t publish a phone number. So, YMMV.
Conclusion
I’m sure I’ve left out many quality options here, but hopefully this is enough to lay out a general lay of the land. Leave other suggestions in the comments.
@elb @ajroach42 2/ I also looked into Dat and IPFS, but they are neither as capable nor as useful as Syncthing for personal synchronization.
@elb @ajroach42 3/ For, eg, downloading websites, archivebox.io could be very nice. Archive Team also has some wiki pages on how to do it with wget and httrack. There are also plugins like webrecorder that could help.
@elb @ajroach42 4/ What you are really after is more general asynchronous communication. I have a whole blog series about this, including #NNCP and other tools: https://changelog.complete.org/archives/tag/asynchronous will give you all the posts in the series. Many of them are somewhat focused on backups, but should give you some good ideas for other things also. NNCP can use things like USB sticks, serial links, regular Internet connections, Syncthing, etc. as transport.
NNCP
asynchronous – The Changelog
@elb @ajroach42 5/ The NNCP page on use cases may give you some ideas (whether or not you use NNCP) http://www.nncpgo.org/Use-cases.html Their integration page http://www.nncpgo.org/Integration.html also is useful. http://www.nncpgo.org/WARCs.html describes downloading webpages.
Use cases (NNCP)
@elb @ajroach42 6/ You talked about accessing web pages offline. I’ve tried that but mostly don’t really bother. It is fairly painful (you frequently want to click on a link you don’t have). In some cases, for things like larger articles, it can make good sense. But you might want to look into something more like rss2email . Email is already asynchronous and there are lots of ways to get asyncrhonous email across. NNCP is one and documents this workflow at http://www.nncpgo.org/Feeds.html
Feeds (NNCP)
@elb @ajroach42 7/ Offline email is two separate problems: sending and receiving. Sending can go across BSMTP (delivered “somehow” via NNCP, UUCP, Syncthing, etc). I talk about Exim with NNCP as part of my series here https://changelog.complete.org/archives/10165-asynchronous-email-exim-over-nncp-or-uucp and the NNCP docs go over the Postfix setup. For incoming, you can use OfflineIMAP or an offline-capable mail reader. Alternatively, forwarding to an account you can receive via NNCP/etc to a local mailstore.
Asynchronous Email: Exim over NNCP (or UUCP)
@elb @ajroach42 8/ So if you have a VPS or a machine “in town” or whatever, you can do some pretty nice things; take the photos you copied into the “to upload” Syncthing folder and upload them, then delete them out of there. Or a laptop can run those commands directly “in town”
@elb @ajroach42 9/ Finally the two best ways to improve your 4G signal are: 1) height, and 2) antenna. I got one of these https://smile.amazon.com/gp/product/B01NBSLNJ6 with a Nighthawk M1 awhile back. Tremendous difference. A booster can only boost what it can receive. A good antenna, mounted high, hardwired into the access point will almost certainly be better. That antenna has “gain”, meaning it’s directional, so figure out where your best towers are and point it at those.
@elb @ajroach42 10/ Also point-to-point wireless may help; if there’s a good place you can get Internet and you have line-of-sight from your house, you may be able to work something out, even something surprisingly fast. For more challenging conditions, LoRA or XBee could work… but at 100Kbps or less. Not suitable for browsing but could work for email.
@elb @ajroach42 11/ Finally, don’t understimate the utility of sshing to a VPS somewhere and reading email in text. My qualifications to anser: have lived in Internet-challenged areas for 20 years, frequently travel into no-Internet areas, have experience with modern communication over extremely low bandwidth links (1 to 100Kbps) including LoRA/XBee radio, AX.25 packet radio, and satellite.
@elb @ajroach42 Also, @joeyh lives off-grid and may have lots of ideas to contribute here.
@jgoerzen @djsundog mentioned rss2email, which seems like a good choice to me (although, frankly, RSS to html files in syncthing is probably fine too.) I think what I’m wanting in terms of capturing web pages is an interface where I provide, for example, a URL or a search term, and the next time a connected node has an internet connection, that web page and every page it links to directly, and every page each of those links to directly is captured for LAN transfer at a later date.
@jgoerzen @elb Thick canopy and mountains, so there’s only so much improving to do. We’ll stick an antenna up pretty high, and run that in to our little booster. It’ll get the job done (I’m posting now over that cellular connection. It works, when you’re in the right location.)
@jgoerzen @elb I did some experimenting with LoRA and tried to do some work with XBee for keeping nodes of a distributed BBS in sync over multiple KM, but ultimately we just didn’t have the mesh density, and I ended up building a solution that used a device I carried with me to rsync each location over wifi. Using syncthing and a cellphone (and NNCP) seems like a more viable longterm solution.
@ajroach42 @jgoerzen @elb I still say we should try the trebuchet of usb flash drives thing at some point
@djsundog @ajroach42 @jgoerzen @elb question: what would be the bandwidth of 90 kg of flash drives launched 300 m?How fast would that be travelling
@nev @djsundog @jgoerzen @elb https://what-if.xkcd.com/31/
FedEx Bandwidth
@ajroach42 @djsundog @jgoerzen @elb this doesn’t answer my question!!
@ajroach42 @djsundog @jgoerzen @elb I googled it myself. Top speed 70 m/s, apparently…someone do the math
@nev a USB drive on average weighs about 30 grams, according to the two sources I found that weighed USB drives. Microcenter cells 256 GB flash drives. You could fit 3000 of them in a 90 KG payload. So that’s 768 terabytes.A full sized trebuchet can launch a 90kg at roughly 70m/s or 156.586 MPH. So you’re sending 768 TB at 300 meters in 4.2 seconds for a transfer speed of 182.857143 terabytes TBps.
@ajroach42 @nev the bottleneck is going to be reading from the interface
@nev Or, in more conventional terms, 1462857144 Mbps, or 1462857144000 Kbps or 1.462857144e+15 bits per second.
@ajroach42 @djsundog You could pretty easily do that with httrack on some remote system. nncp-exec sending a URL to that, it returns a tarball to you, or something similar, I’d think.
@jgoerzen @ajroach42 with a possible pit stop on a shared lan server that untars it long enough to index into elasticsearch or whatever along with a pointer to the tarball on lan storage for caching etc. yeah, this is more than doable.
@ajroach42 thank you for this incredible contribution to modern science.
@djsundog @ajroach42 Interestingly, Sergey just (literally within the past few days) added “multicast” support to #NNCP http://www.nncpgo.org/Multicast.html . The Internet-connected machine could send the downloaded site to the LAN “area”. Upon arriving at the LAN NNCP gateway, it would be copied both to the PC and to the indexer. Wouldn’t require any extra scarce bandwidth, and also wouldn’t require the indexer to be up in order for the data to reach the PC.
NNCP
Multicast (NNCP)
@djsundog @ajroach42 NNCP 7.x is still a bit buggy for me (I make pretty heavy use of it, sending thousands of packets, some of them hundreds of GB in size, across it daily) but I’m sure it will shake out shortly. In the meantime, 6.6.0 is rock solid
@ajroach42 @nev this works as a value for peak throughput, but i think in order to give a full picture of the capabilities, this calculation should also consider loading time rather than just launching time
@jgoerzen @ajroach42 @elb Reg. offline: you know scuttlebutt, do you? That sneakernet-compatible social network?
I think it’s a pretty good idea that meshes direct connections to hubs/proxies and p2p connections to update offline data.
@ajroach42 @nev I thought 30g sounded like a /really/ heavy flashdrive, so I weighed the small jar of flashdrives on my desk. their weights: 10g, 9g, 4g, 8g, 16g. which would increase data throughput :) I can weigh some SD cards too I guess… nvm. microSD card doesn’t make it up to 1g on my scales. 0.5g would be best guess. regular sized SD card is about 2 or 2.5g.
@epoch @ajroach42 okay so by my calculation, assuming a micro SD card is 0.5g and holds 1 TB, a 90kg payload contains 180 PB…
@nev @epoch and it’ll travel 300 meters in 4.2 seconds. So 180PB/4.2 seconds is 342 Petabits per second.
@jtr @debian @kensanata So #NNCP at its core is different because it doesn’t require the source and destination to be online simultaneously. With TCP, the origin of a packet is responsible for ensuring it gets delivered, retransmitted if dropped, etc. With NNCP, you send data to the next hop and then that hop takes over responsibility. It may be days before that hop is able to deliver it. That hop may be a computer, or it may be a USB stick or a radio. 2/
NNCP
@jtr @debian @kensanata #NNCP can be used to send files and execute things remotely. Think of it like ssh/scp and authorized_keys. I can say: tar -cpf /usr | nncp-exec dest untar -C /backupswhere I defined untar on the dest node as something that runs “tar -xpf -” and then we add a -C telling it where to unpack.You could run a very similar command with ssh. Difference: dest doesn’t have to be online with NNCP. 3/
NNCP
@jtr @debian @kensanata In that nncp-exec example, the data piped to nncp-exec is saved, along with the command line to run, in a data packet that is encrypted using the public key of the destination. Now it can be transported, directly or indirectly (via other nodes), over the network, USB drives, CD-ROMs, tapes, radios, laptops, phones, a combination of these, whatever. It can traverse multiple NNCP hops along the way, and is onion-routed like tor 4/
@jtr @debian @kensanata So let’s make this practical. Say you have slow Internet at home but fast Internet at a coffee shop. You have 10GB to send out. On your desktop at home, you queue it up to go like this: desktop->laptop->remote_server. Your desktop encrypts the 10GB to remote_server, wrapping that in a packet encrypted to the laptop. The laptop decrypts the outer encryption when it gets the data, but still can’t see the actual data. Laptop sends it out when you get to coffee shop. 5/
@jtr @debian @kensanata I use #NNCP for backups. I take hourly snapshots with #ZFS. I have a backup system with no Internet. My backups are sent to a staging box, and then I can get those packets over to the backup box with the various methods I’ve mentioned. On my laptops, NNCP is configured to only automatically send packets to staging when they’re on the home LAN, because I may be tethered to 4G otherwise. But I can manually send them off when I’m on fast Internet away from home. 6/
NNCP
zfs
@jtr @debian @kensanata #NNCP also has an #asynchronous #multicast feature, which I use to sync my #orgmode #git repo as described here https://changelog.complete.org/archives/10274-distributed-asynchronous-git-syncing-with-nncp I also have more on NNCP here https://www.complete.org/nncp/ 7/
NNCP
asynchronous
git
multicast
orgmode
Distributed, Asynchronous Git Syncing with NNCP
@jtr @debian @kensanata So #NNCP can be used to transport #Usenet news and #email because fundamentally those things can be transported by piping data into the rnews and rmail commands, which fits perfectly with nncp-exec. In fact, a predecessor to NNCP, #UUCP, was the way email and news often flowed in the early days, and this support is mostly still there even in modern servers. It takes just a bit of tweaking to make it use NNCP instead. 8/
NNCP
email
usenet
uucp
@jtr @debian @kensanata So I hope this helps, feel free to ask followup questions 🙂 end/
@jtr @debian @kensanata So this is all just scratching the surface of what you can do with #NNCP. It’s sort of like explaining what you can do with #ssh or find. Hope that helps! /end
NNCP
SSH
Note: This post is also available on my webiste, where it will be periodically updated.
As I’ve been thinking and writing about privacy and decentralization lately, I had a conversation with a colleague this week, and he commented about how loss of privacy is related to loss of agency: that is, loss of our ability to make our own choices, pursue our own interests, and be master of our own attention.
In terms of telecommunications, we have never really been free, though in terms of Internet and its predecessors, there have been times where we had a lot more choice. Many are too young to remember this, and for others, that era is a distant memory.
The irony is that our present moment is one of enormous consolidation of power, and yet also one of a proliferation of technologies that let us wrest back some of that power. In this post, I hope to enlighten or remind us of some of the choices we have lost — and also talk about the ways in which we can choose to regain them, already, right now.
I will talk about the possibilities, the big dreams that are possible now, and then go into more detail about the solutions.
The Problems & Possibilities
The limitations of “online”
We make the assumption that we must be “online” to exchange data. This is reinforced by many “modern” protocols; Twitter clients, for instance, don’t tend to let you make posts by relaying them through disconnected devices.
What would it be like if you could fully participate in global communities without a constant Internet connection? If you could share photos with your friends, read the news, read your email, etc. even if you don’t have a connection at present? Even if the device you use to do that never has a connection, but can route messages via other devices that do?
Would it surprise you to learn that this was once the case? Back in the days of UUCP, much email and Usenet news — a global discussion forum that didn’t require an Internet connection — was relayed via occasional calls over phone lines. This technology remains with us, and has even improved.
Sadly, many modern protocols make no effort in this regard. Some email clients will let you compose messages offline to send when you get online later, but the assumption always is that you will be connected to an IP network again soon.
NNCP, on the other hand, lets you relay messages over TCP, a radio, a satellite, or a USB stick. Email and Usenet, since they were designed in an era where store-and-forward was valued, can actually still be used in an entirely “offline” fashion (without ever touching an IP-based network). All it takes is for someone to care to make it happen. You can even still do it over UUCP if you like.
The physical and data link layers
Many of us just accept that we communicate in a few ways: Wifi for short distances, and then cable modems or DSL for our local Internet connection, and then many people are fuzzy about what happens after that. Or, alternatively, we have 4G phones that are the local Internet connection, and the same “fuzzy” things happen after.
Think about this for a moment. Which of these do you control in any way? Sometimes just wifi, sometimes maybe you have choices of local Internet providers. After that, your traffic is handled by enormous infrastructure companies.
There is choice here.
People in ham radio have been communicating digitally over long distances without the support of the traditional Internet for decades, but the technology to do this is now more accessible to anyone. Long-distance radio has had tremendous innovation in the last decade; cheap radios can now communicate over several miles/km without any other infrastructure at all. We all carry around radios (Wifi and Bluetooth) in our pockets that don’t have to be used as mere access points to the Internet or as drivers of headphones, but can also form their own networks directly (Briar).
Meshtastic is an example; it’s an instant messenger that can form a mesh over many miles/km and requires no IP infrastructure at all. Briar is similar. XBee radios form a mesh in hardware, allowing peers to reach each other (also over many miles/km) with a serial or framed protocol.
Loss of peer-to-peer
Back in the late 90s, I worked at a university. I had a 386 on my desk for a workstation – not a powerful computer even then. But I put the boa webserver on it and could just serve pages on the Internet. I didn’t have to get permission. Didn’t have to pay a hosting provider. I could just DO it.
And of course that is because the university had no firewall and no NAT. Every PC at the university was a full participant on the Internet as much as the servers at Microsoft or DEC. All I needed was a DNS entry. I could run my own SMTP server if I wanted, run a web or Gopher server, and that was that.
There are many reasons why this changed. Nowadays most residential ISPs will block SMTP for their customers, and if they didn’t, others would; large email providers have decided not to federate with IPs in residential address spaces. Most people have difficulty even getting a static IP address in the first place. Many are behind firewalls, NATs, or both, meaning that incoming connections of any kind are problematic.
Do you see what that means? It has weakened the whole point of the Internet being a network of peers. While IP still acts that way, as a practical matter, there are clients that are prevented from being servers by administrative policy they have no control over.
Imagine if you, a person with an Internet connection to your laptop or phone, could just decide to host a website, or a forum on it. For moderate levels of load, they are certainly capable of this. The only thing in the way is the network management policies you can’t control.
Elaborate technologies exist to try to bridge this divide, and some, like Tor or cjdns, can work quite well. More on this below.
Expense of running something popular
Related to the loss of peer-to-peer infrastructure is the very high cost of hosting something popular. Do you want to share videos with lots of people? That almost certainly is going to require expensive equipment and bandwidth.
There is a reason that there are only a small handful of popular video streaming sites online. It requires a ton of money to host videos at scale.
What if it didn’t? What if you could achieve economies of scale so much that you, an individual, could compete with the likes of YouTube? You wouldn’t necessarily have to run ads to support the service. You wouldn’t have to have billions of dollars or billions of viewers just to make it work.
This technology exists right now. Of course many of you are aware of how Bittorrent leverages the swarm for files. But projects like IPFS, Dat, and Peertube have taken this many steps further to integrate it into a global ecosystem. And, at least in the case of Peertube, this is a thing that works right now in any browser already!
Application-level “walled gardens”
I was recently startled at how much excitement there was when Github introduced “dark mode”. Yes, Github now offers two colors on its interface. Already back in the 80s and 90s, many DOS programs had more options than that.
Git is a decentralized protocol, but Github has managed to make it centralized.
Email is a decentralized protocol — pick your own provider, and they all communicate — but Facebook and Twitter aren’t. You can’t just pick your provider for Facebook. It’s Facebook or nothing.
There is a profit motive in locking others out; these networks want to keep you using their platforms because their real customers are advertisers, and they want to keep showing you ads.
Is it possible to have a world where you get to pick your own app for sharing photos, and it works even if your parents use a different one? Yes, yes it is.
Mastodon and the Fediverse are fantastic examples for social media. Pixelfed is specifically designed for photos, Mastodon for short-form communication, there’s Pleroma for more long-form communication, and they all work together. You can use Mastodon to read Pleroma content or look at Pixelfed photos, and there are many (free) providers of each.
Freedom from manipulation
I recently wrote about the dangers of the attention economy, so I won’t go into a lot of detail here. Fundamentally, you are not the customer of Facebook or Google; advertisers are. They optimize their site to keep you on it as much as possible so that they can show you as many ads as possible which makes them as much money as possible. Ads, of course, are fundamentally seeking to manipulate your behavior (“buy this product”).
By lowering the cost of running services, we can give a huge boost to hobbyists and nonprofits that want to do so without an ultimate profit motive. For-profit companies benefit also, with a dramatically reduced cost structure that frees them to pursue their mission instead of so many ads.
Freedom from snooping (privacy and anonymity)
These days, it’s not just government snooping that people think about. It’s data stolen by malware, spies at corporations (whether human or algorithmic), and even things like basic privacy of one’s own security footage. Here the picture is improving; encryption in transit, at least at a basic level, has become much more common with TLS being a standard these days. Sadly, end-to-end encryption (E2EE) is not nearly as much, perhaps because corporations have a profit motive to have access to your plaintext and metadata.
Closely related to privacy is anonymity: that is, being able to do things in an anonymous fashion. The two are not necessarily equal: you could send an encrypted message but reveal who the correspondents are, as with email; or, you could send a plaintext message over a Tor exit node that hides who the correspondents are. It is sometimes difficult to achieve both.
Nevertheless, numerous answers exist here that tackle one or both problems, from the Signal messenger to Tor.
Solutions That Exist Today
Let’s dive in to some of the things that exist today.
One concept you’ll see in many of these is integrated encryption with public keys used for addressing. In other words, your public key is akin to an IP address (and in some cases, is literally your IP address.)
Data link and networking technologies (some including P2P)
Starting with the low-power and long-distance technologies, I’ve written quite a bit about LoRA, which are low-power long-distance radios. They can easily achieve several miles/km while still using much less than 1W of power. LoRA is a common building block of mesh off-the-grid messenger systems such as meshtastic, which forms an ad-hoc mesh of LoRA devices with days-long battery life and miles-long communication abilities. LoRA trades speed for bandwidth; in its longest-distance modes, it may operate at 300bps or less. That is not a typo. Some LoRAWAN devices have battery life measured in years (usually one-way sensors and such). Also, the Pine64 folks are working to integrate LoRA on nearly all their product line, which includes single-board computers, phones, and laptops.
Similar to LoRA is XBee SX from Digi. While not quite as long-distance as LoRA, it does still do quite a bit with low power and also goes many miles. XBee modules have automatic mesh routing in firmware, and can be used in either frame mode or “serial cable emulation” mode in which they act as if they’re a serial cable. Unlike plain LoRA, XBee radios do hardware retransmit. They also run faster, at up to about 150Kbps – though that is still a lot slower than wifi.
I’ve written about secure mesh messengers recently. One of them, Briar, particularly stands out in that it is able to form an ad-hoc mesh using phone’s Bluetooth radios. It can also route messages over the public Internet, which it does exclusively using Tor.
I’ve also written a lot about NNCP, the sort of modernized UUCP. NNCP is completely different than the others here in that it is a store-and-forward network – sort of a modern UUCP. NNCP has easy built-in support for routing packets using USB drives, clean serial interfaces, TCP, basically anything you can pipe to, even broadcast satellite and such. And you don’t even have to pick one; you can use all of the above: Internet when it’s available, USB sticks or portable hard drives when not, etc. It uses Tor-line onion routing with E2EE. You’re not going to run TCP over NNCP, but files (including videos), backups, email, even remote execution are all possible. It is the most “Unixy” of the modern delay-tolerant networks and makes an excellent choice for a number of use cases where store-and-forward and extreme flexibility in transportation make a lot of sense.
Moving now into the range of speeds and technologies we’re more used to, there is a lot of material out there on building mesh networks on Wifi or Wifi-adjacent technology. Amateur radio operators have been active in this area for years, and even if you aren’t a licensed ham and don’t necessarily flash amateur radio firmware onto your access points, a lot of the ideas and concepts they cover could be of interest. For instance, the Amateur Radio Emergency Data Network covers both permanent and ad-hoc meshs, and this AREDN video covers device selection for AREDN — which also happens to be devices that would be useful for quite a few other mesh or long-distance point-to-point setups.
Once you have a physical link of some sort, cjdns and the Hyperboria network have the goals of literally replacing the Internet – but are fully functional immediately. cjdns assigns each node an IPv6 address based on its public key. The network uses DHT for routing between nodes. It can run directly atop Ethernet (and Wifi) as its own native protocol, without an IP stack underneath. It can also run as a layer atop the current Internet. And it can optionally be configured to let nodes find an exit node to reach the current public Internet, which they can do opportunistically if given permission. All traffic is E2EE. One can run an isolated network, or join the global Hyperboria network. The idea is that local meshes could be formed, and then geographically distant meshes can be linked together by simply using the current public Internet as a dumb transport. This, actually, strongly resembles the early days of Internet buildout under NSFNet. The Torento Mesh is a prominent user of cjdns, and they publish quite a bit of information online. cjdns as a standalone identity is in decline, but forms the basis of the pkt network, which is designed to foster an explosion in WISPs.
Similar in concept to cjdns is Yggdrasil, which uses a different routing algorithm. It is now more active than cjdns and has active participants and developers.
Althea is a startup in this space, hoping to encourage communities to build meshes whose purpose is to provide various routes to access to the traditional Internet, including digital currency micropayments. This story documents how one rural community is using it.
Tor is a somewhat interesting case. While it doesn’t provide kernel-level routing, it does provide a SOCKS5 proxy. Traditionally, Tor is used to achieve anonymity while browsing the public Internet via an exit node. However, you can stay entirely in-network by using onion services (basically ports that are open to Tor). All Tor traffic is onion-routed so that the originating IP cannot be discovered. Data within Tor is E2EE, though if you are using an exit node to the public Internet, that of course can’t apply there.
GNUnet is a large suite of tools for P2P communication. It includes file downloading, Tor-like IP over the network, a DNS replacement, and facilitates quite a few of the goals discussed here. (Added in a 2021-02-22 update)
P2P Infrastructure
While some of the technologies above, such as cjdns, explicitly facitilitate peer-to-peer communication, there are some other application-level technologies to look at.
IPFS has been having a lot of buzz lately, since the Brave browser integrated support. IPFS headlines as “powers the distributed web”, but it is actually more than that; various other apps layer atop it. The core idea is that content you request gets reshared by your node for some period of time, somewhat akin to Bittorrent. IPFS runs atop the regular Internet and is typically accessed through an app.
The Dat Protocol is somewhat similar in concept to IPFS, though the approach is somewhat different; it emphasizes efficient distribution of updates at the expense of requiring a git-like history.
IPFS itself is based on libp2p, which is designed to be a generic infrastructure for adding P2P capabilities to your own code. It is probably fair to say libp2p is still quite complex compared to ordinary TCP, and the language support is in its infancy, but nevertheless it is quite an exciting development to watch.
Of course almost all of us are familiar with Bittorrent, the software that first popularized the idea of a distributed mesh sharing knowledge about which chunks of a dataset they have in order to maximize the efficiency of distributing the whole thing. Bittorrent is still in wide use (and, despite its reputation, that wide use includes legitimate users such as archive.org and Debian).
I recently wrote about building a delay-tolerant offline-capable mesh with Syncthing. Syncthing, on its surface, is something like an open source Dropbox. But look into a bit and you realize it’s fully P2P, serverless, can support various network topologies including intermittent connectivity between network parts, and such. My article dives into that in more detail. If your needs are mostly related to files, Syncthing can make a fine mesh infrastructure that is auto-healing and is equally at home on the public Internet, a local wifi access point with no Internet at all, a private mesh like cjdns, etc.
Also showing some promise is Secure Scuttlebutt (SSB). Its most well-known application is a social network, but in my opinion some of the other applications atop SSB are more interesting. SSB is designed to be offline-friendly, can do things like automatically exchange data with peers on the same Wifi (eg, a coffee shop), etc., though it is an append-only log that can be unwieldy on mobile sometimes.
Instant Messengers and Chat
I won’t go into a lot of detail here since I recently wrote a roundup of secure mesh messengers and also a followup article about Signal and some hidden drawbacks of P2P. Please refer to those articles for some interesting things that are happening in this space.
Matrix is a distributed IM platform similar in concept to Slack or IRC, but globally distributed in a mesh. It supports optional E2EE.
Social Media
I wrote recently about how to join the Fediverse, which covered joining Mastodon, a federeated, decentralized social network. Mastodon is the largest of these, with several million users, and is something of a much nicer version of Twitter.
Mastodon is also part of what is known as the “Fediverse”, which are applications that are loosely joined together by their support of the ActivityPub protocol. Other popular Fediverse applications include Pixelfed (similar to Instagram) and Peertube for sharing video. Peertube is particularly interesting in that it supports Webtorrent for efficiently distributing popular videos. Webtorrent is akin to Bittorrent running efficiently inside your browser.
Concluding Remarks
Part of my goal with this is encouraging people to dream big, to ask questions like:
What could you do if offline were easy?
What is possible if you have freedom in the physical and data link layers? Dream big.
We’re so used to thinking that it’s quite difficult for two devices on the Internet to talk to each other. What would be possible if this were actually quite easy?
The assumption that costs rise dramatically as popularity increases is also baked into our thought processes. What if that weren’t the case — could you take on Youtube from your garage? Would lowering barriers to entry lower the ad economy and let nonprofits have more equal footing with large corporations?
We have so many walled gardens, from Github to Facebook, that we almost forget it doesn’t have to be that way.
So having asked these questions, my secondary point is to suggest that these aren’t pie-in-the-sky notions. These possibilites are with us right now.
You’ll notice from this list that virtually every one of these technologies is ad-free at its heart (though some would be capable of serving ads). They give you back your attention. Many preserve privacy, anonymity, or both. Many dramatically improve your freedom of association and communication. Technologies like IPFS and Bittorrent ease the burden of running something popular.
Some are quite easy to use (Mastodon or Peertube) while others are much more complex (libp2p or the lower-level mesh network systems).
Clearly there is still room for improvement in many areas.
But my fundamental point is this: good technology is here, right now. Technical people can vote with their feet and wallets and start using it. Early adopters will help guide the way for the next set of improvements. Join us!
Note: this post is also available on my website, where it will be updated periodically.
When things are difficult – maybe there’s been a disaster, or an invasion (this page is being written in 2022 just after Russia invaded Ukraine), or maybe you’re just backpacking off the grid – there are tools that can help you keep in touch, or move your data around. This page aims to survey some of them, roughly in order from easiest to more complex.
Simple radios
Handheld radios shouldn’t be forgotten. They are cheap, small, and easy to operate. Their range isn’t huge – maybe a couple of miles in rural areas, much less in cities – but they can be a useful place to start. They tend to have no actual encryption features (the “privacy” features really aren’t.) In the USA, options are FRS/GMRS and CB.
Syncthing
With Syncthing, you can share files among your devices or with your friends. Syncthing essentially builds a private mesh for file sharing. Devices will auto-discover each other when on the same LAN or Wifi network, and opportunistically sync.
I wrote more about offline uses of Syncthing, and its use with NNCP, in my blog post A simple, delay-tolerant, offline-capable mesh network with Syncthing (+ optional NNCP). Yes, it is a form of a Mesh Network!
Homepage: https://syncthing.net/
Briar
Briar is an instant messaging service based around Android. It’s IM with a twist: it can use a mesh of Bluetooh devices. Or, if Internet is available, Tor. It has even been extended to support the use of SD cards and USB sticks to carry your messages.
Like some others here, it can relay messages for third parties as well.
Homepage: https://briarproject.org/
Manyverse and Scuttlebutt
Manyverse is a client for Scuttlebutt, which is a sort of asynchronous, offline-friendly social network. You can use it to keep in touch with your family and friends, and it supports syncing over Bluetooth and Wifi even in the absence of Internet.
Homepages: https://www.manyver.se/ and https://scuttlebutt.nz/
Yggdrasil
Yggdrasil is a self-healing, fully end-to-end Encrypted Mesh Network. It can work among local devices or on the global Internet. It has network services that can egress onto things like Tor, I2P, and the public Internet. Yggdrasil makes a perfect companion to ad-hoc wifi as it has auto peer discovery on the local network.
I talked about it in more detail in my blog post Make the Internet Yours Again With an Instant Mesh Network.
Homepage: https://yggdrasil-network.github.io/
Ad-Hoc Wifi
Few people know about the ad-hoc wifi mode. Ad-hoc wifi lets devices in range talk to each other without an access point. You just all set your devices to the same network name and password and there you go. However, there often isn’t DHCP, so IP configuration can be a bit of a challenge. Yggdrasil helps here.
NNCP
Moving now to more advanced tools, NNCP lets you assemble a network of peers that can use Asynchronous Communication over sneakernet, USB drives, radios, CD-Rs, Internet, tor, NNCP over Yggdrasil, Syncthing, Dropbox, S3, you name it . NNCP supports multi-hop file transfer and remote execution. It is fully end-to-end encrypted. Think of it as the offline version of ssh.
Homepage: https://nncp.mirrors.quux.org/
Meshtastic
Meshtastic uses long-range, low-power LoRa radios to build a long-distance, encrypted, instant messaging system that is a Mesh Network. It requires specialized hardware, about $30, but will tend to get much better range than simple radios, and with very little power.
Homepages: https://meshtastic.org/ and https://meshtastic.letstalkthis.com/
Portable Satellite Communicators
You can get portable satellite communicators that can send SMS from anywhere on earth with a clear view of the sky. The Garmin InReach mini and Zoleo are two credible options. Subscriptions range from about $10 to $40 per month depending on usage. They also have global SOS features.
Telephone Lines
If you have a phone line and a modem, UUCP can get through just about anything. It’s an older protocol that lacks modern security, but will deal with slow and noisy serial lines well. XBee SX radios also have a serial mode that can work well with UUCP.
Additional Suggestions
It is probably useful to have a Linux live USB stick with whatever software you want to use handy. Debian can be installed from the live environment, or you could use a security-focused distribution such as Tails or Qubes.
References
This page originated in my Mastodon thread and incorporates some suggestions I received there.
It also formed a post on my blog.
thank you very much!