All posts by John Goerzen

Joyful is the Dark

“Joyful is the dark
  coolness of the tomb,
waiting for the wonder
  of the morning.

Never was that midnight
  touched by dread and gloom;
darkness was the cradle
  of the dawning.”

Brian Wren

Most of us are not personally experiencing symptoms of a pandemic virus, but with all the changes around us, with all the worries within us, many of us have been touched by dread and gloom.

I can find many layers of meaning in this poem. Today I think this poem excerpt reminds us to find the joy where we are, in the moment we are. Are we just waiting for the morning? Or do we take advantage of this pause to gaze up at the beautiful colors of the night? Are we defined by this moment, or do we define it? We are OK right now, so let this time be the cradle of something beautiful.

If you haven’t noticed the moon in this photo, zoom in. I don’t see dread and gloom in this photo. I see color, and hope, and beauty.

Rural Kansas, 2017

O, Sunlight!

“O, Sunlight! The most precious gold to be found on Earth.”

– Roman Payne

There is much beauty in this world, much hope, much life. All we need to do is pause, breathe, and take a moment to see it. It might be as simple as the gift of sunlight. I hope you all have moments of sunlight and delight each day.

Marion County, KS, April 2013

How & Why To Use Airgapped Backups

A good backup strategy needs to consider various threats to the integrity of data. For instance:

  • Building catches fire
  • Accidental deletion
  • Equipment failure
  • Security incident / malware / compromise

It’s that last one that is of particular interest today. A lot of backup strategies are such that if a user (or administrator) has their local account or network compromised, their backups could very well be destroyed as well. For instance, do you ssh from the account being backed up to the system holding the backups? Or rsync using a keypair stored on it? Or access S3 buckets, etc? It is trivially easy in many of these schemes to totally ruin cloud-based backups, or even some other schemes. rsync can be run with –delete (and often is, to prune remotes), S3 buckets can be deleted, etc. And even if you try to lock down an over-network backup to be append-only, still there are vectors for attack (ssh credentials, OpenSSL bugs, etc). In this post, I try to explore how we can protect against them and still retain some modern conveniences.

A backup scheme also needs to make a balance between:

  • Cost
  • Security
  • Accessibility
  • Efficiency (of time, bandwidth, storage, etc)

My story so far…

About 20 years ago, I had an Exabyte tape drive, with the amazing capacity of 7GB per tape! Eventually as disk prices fell, I had external disks plugged in to a server, and would periodically rotate them offsite. I’ve also had various combinations of partial or complete offsite copies over the Internet as well. I have around 6TB of data to back up (after compression), a figure that is growing somewhat rapidly as I digitize some old family recordings and videos.

Since I last wrote about backups 5 years ago, my scheme has been largely unchanged; at present I use ZFS for local and to-disk backups and borg for the copies over the Internet.

Let’s take a look at some options that could make this better.

Tape

The original airgapped backup. You back up to a tape, then you take the (fairly cheap) tape out of the drive and put in another one. In cost per GB, tape is probably the cheapest medium out there. But of course it has its drawbacks.

Let’s start with cost. To get a drive that can handle capacities of what I’d be needing, at least LTO-6 (2.5TB per tape) would be needed, if not LTO-7 (6TB). New, these drives cost several thousand dollars, plus they need LVD SCSI or Fibre Channel cards. You’re not going to be hanging one off a Raspberry Pi; these things need a real server with enterprise-style connectivity. If you’re particularly lucky, you might find an LTO-6 drive for as low as $500 on eBay. Then there are tapes. A 10-pack of LTO-6 tapes runs more than $200, and provides a total capacity of 25TB – sufficient for these needs (note that, of course, you need to have at least double the actual space of the data, to account for multiple full backups in a set). A 5-pack of LTO-7 tapes is a little more expensive, while providing more storage.

So all-in, this is going to be — in the best possible scenario — nearly $1000, and possibly a lot more. For a large company with many TB of storage, the initial costs can be defrayed due to the cheaper media, but for a home user, not so much.

Consider that 8TB hard drives can be found for $150 – $200. A pair of them (for redundancy) would run $300-400, and then you have all the other benefits of disk (quicker access, etc.) Plus they can be driven by something as cheap as a Raspberry Pi.

Fancier tape setups involve auto-changers, but then you’re not really airgapped, are you? (If you leave all your tapes in the changer, they can generally be selected and overwritten, barring things like hardware WORM).

As useful as tape is, for this project, it would simply be way more expensive than disk-based options.

Fundamentals of disk-based airgapping

The fundamental thing we need to address with disk-based airgapping is that the machines being backed up have no real-time contact with the backup storage system. This rules out most solutions out there, that want to sync by comparing local state with remote state. If one is willing to throw storage efficiency out the window — maybe practical for very small data sets — one could just send a full backup daily. But in reality, what is more likely needed is a way to store a local proxy for the remote state. Then a “runner” device (a USB stick, disk, etc) could be plugged into the network, filled with queued data, then plugged into the backup system to have the data dequeued and processed.

Some may be tempted to short-circuit this and just plug external disks into a backup system. I’ve done that for a long time. This is, however, a risk, because it makes those disks vulnerable to whatever may be attacking the local system (anything from lightning to ransomware).

ZFS

ZFS is, it should be no surprise, particularly well suited for this. zfs send/receive can send an incremental stream that represents a delta between two checkpoints (snapshots or bookmarks) on a filesystem. It can do this very efficiently, much more so than walking an entire filesystem tree.

Additionally, with the recent addition of ZFS crypto to ZFS on Linux, the replication stream can optionally reflect the encrypted data. Yes, as long as you don’t need to mount them, you can mostly work with ZFS datasets on an encrypted basis, and can directly tell zfs send to just send the encrypted data instead of the decrypted data.

The downside of ZFS is the resource requirements at the destination, which in terms of RAM are higher than most of the older Raspberry Pi-style devices. Still, one could perhaps just save off zfs send streams and restore them later if need be, but that implies a periodic resend of a full stream, an inefficient operation. dedpulicating software such as borg could be used on those streams (though with less effectiveness if they’re encrypted).

Tar

Perhaps surprisingly, tar in listed incremental mode can solve this problem for non-ZFS users. It will keep a local cache of the state of the filesystem as of the time of the last run of tar, and can generate new tarballs that reflect the changes since the previous run (even deletions). This can achieve a similar result to the ZFS send/receive, though in a much less elegant way.

Bacula / Bareos

Bacula (and its fork Bareos) both have support for a FIFO destination. Theoretically this could be used to queue of data for transfer to the airgapped machine. This support is very poorly documented in both and is rumored to have bitrotted, however.

rdiff and xdelta

rdiff and xdelta can be used as sort of a non-real-time rsync, at least on a per-file basis. Theoretically, one could generate a full backup (with tar, ZFS send, or whatever), take an rdiff signature, and send over the file while keeping the signature. On the next run, another full backup is piped into rdiff, and on the basis of the signature file of the old and the new data, it produces a binary patch that can be queued for the backup target to update its stored copy of the file.

This leaves history preservation as an exercise to be undertaken on the backup target. It may not necessarily be easy and may not be efficient.

rsync batches

rsync can be used to compute a delta between two directory trees and express this as a single-file batch that can be processed by a remote rsync. Unfortunately this implies the sender must always keep an old tree around (barring a solution such as ZFS snapshots) in order to compute the delta, and of course it still implies the need for history processing on the remote.

Getting the Data There

OK, so you’ve got an airgapped system, some sort of “runner” device for your sneakernet (USB stick, hard drive, etc). Now what?

Obviously you could just copy data on the runner and move it back off at the backup target. But a tool like NNCP (sort of a modernized UUCP) offer a lot of help in automating the process, returning error reports, etc. NNCP can be used online over TCP, over reliable serial links, over ssh, with offline onion routing via intermediaries or directly, etc.

Imagine having an airgapped machine at a different location you go to frequently (workplace, friend, etc). Before leaving, you put a USB stick in your pocket. When you get there, you pop it in. It’s despooled and processed while you want, and return emails or whatever are queued up to be sent when you get back home. Not bad, eh?

Future installment…

I’m going to try some of these approaches and report back on my experiences in the next few weeks.

Every Storm Runs Out Of Rain

“Every storm runs out of rain.”

– Maya Angelou

There are a lot of rain clouds in life these days. May we all remember that days like this one are behind us — and also ahead of us.

Every storm runs out of rain.


That was the start of a series of photos from my collection & quotes I shared with friends during the initial lockdown in spring. I’ll be sharing some here.

And here we all are, still dealing with this — and it’s more severe in a lot of ways. One of my colleagues won’t be able to see his parents this Christmas for the first time in over 40 years.

But this storm will run out rain.

And look how the scene changed, in just a few minutes. This is coming!

Non-Creepy Technology Purchasing & Gifting Guides

This time of year, a lot of people are thinking of buying gadgets and phones as gifts. But there are a lot of tech companies that have unethical practices, from terrible working conditions in their factories to spying on their users. Here are some buying guides to help you find gadgets that are fun – and not creepy.

The Free Software Foundation’s Ethical Tech Giving Guide is a fantastic resource from what’s probably the pickiest organization out there when it comes to tech. Not only do they highlight good devices, they also explain why and why you should, for instance, avoid the iPhone (their history of silencing political activists and spying on users).

The FSF also has a Guide to DRM-Free Living talks about books, video, audio, and software that respects your freedom by letting you make your own backups, move it to other devices, and continue to use your purchases even if you have no Internet or the company you bought them from goes bankrupt. This is a fantastic and HUGE resource; there are hundreds of organizations out there that provide content in a way that respects your rights — and many of them do it for free, legally, as well.

PrivacyTools has a fantastic series of guides on everything from email providers to operating systems, as well as links to a number of other guides.

The DeGoogle wiki on Reddit (as well as the sidebar) has a lot of fantastic alternatives to things like Chromebooks, Chrome, Gmail, etc.

Related resources

Here are some resources for education (what the issues are) and information about what companies and products to avoid.

In addition to the FSF’s other fantastic resources above, they also have a list of proprietary malware. It lists things, practices, and companies to avoid, and talks about the reasons why. Their addictions page is particularly good and relevant to my recent post on the problems of the attention economy.

The Surveillance Self-Defense site from the Electronic Frontier Foundation is a fantastic introduction into how corporate surveillance works and how to defend against it.

Use with a grain of salt:

Mozilla, the people behind Firefox, have a site called Privacy Not Included that rates products by how “creepy” they are. They focus more narrowly on privacy than the more expansive set of freedoms the FSF considers (privacy is one of a number of things the FSF looks at), and in some cases I would say Mozilla is too generous (eg, with the Amazon Kindle, a number of their data points are just incorrect.)

How To Join the Fediverse and Cast Off the Attention Economy

In a recent post, I wrote about how the attention economy in use at big social networks hurts you.

In this post, I’m going to suggest what to do about it.

Mastodon and the Fediverse

When you use email, you can send a message from an account at Google to one at Yahoo, Microsoft, or any of millions of businesses and organizations running their own mail server. Unlike, say, Facebook, email isn’t a single service, but rather a whole bunch of independent systems that can communicate (or federate) with each other.

The Fediverse is similar, and the most advanced Fediverse client is Mastodon.

Mastodon:

  • Lets you easily migrate your account from one Mastodon instance to another, similar to how you can migrate a phone number from one carrier to another
  • Lets you communicate with users across the Fediverse (other Mastodon instances, and those running other software too)
  • Supports very high-quality conversations through sensible moderation controls

It’s easy to get started! Head over to joinmastodon.org and click “Get Started”. Pick a community — don’t worry, this isn’t a hugely consequential decision, as you can always move or change later. You can browse activity from across the Fediverse, or just on your local community, so if you find a community with similar interests, it can be a neat way to find others to follow.

If you’re looking for more details, mastodon.help has a nice guide.

Defeating the Attention Economy

So, why does Mastodon make a difference?

First of all, you get to pick your host (and even software). With Twitter, you pretty much are using Twitter (yes, I know of things like Hootsuite, but for the vast majority of people, it’s twitter.com only). With Mastodon, you have choice. Pick the host that runs the software and has the kind of moderation you like.

Secondly, Mastodon is not for profit. There is no money to be made in keeping you on the site. Almost all Mastodon instances are ad-free. And Mastodon’s completely open protocols make it easy to go elsewhere if you like.

It’s Not Just Mastodon!

There are plenty of other programs in the Fediverse. And, this is really key, they all interact with each other. You can share photos in Pixelfed (sort of like a federated Instagram) and see them and comment in Mastodon! Some things to point out:

And there are many others.

This blog, for instance, runs WordPress and uses an ActivityPub connector; comments from the Fediverse integrate here.

Find me in the Fediverse

You can look me up: just type in @jgoerzen@floss.social in the search box of any Mastodon instance and click Follow. You can also follow this blog at @jgoerzen@changelog.complete.org.

How the Attention Economy Hurts You via Social Media Sites like Facebook

There is a whole science to manipulating our attention. And because there is a lot of money to be made by doing this well, it means we all encounter attempts to manipulate what we pay attention to each day. What is this, and how is it harmful? This post will be the first on a series on the topic.

Why is attention so important?

When people use Facebook, they use it for free. Facebook generally doesn’t even try to sell them anything, yet has billions in revenues. What, then, is Facebook’s product?

Well, really, it’s you. Or, more specifically, your attention. Facebook sells your attention to advertisers. Everything they do is in service to that. They want you to spend more time on the site so they can show you more ads.

(I should say here that I’m using Facebook as an example, but this applies to other social media companies too.)

Seeking to maximize attention

So if your attention is so important to their profit, it follows naturally that they would seek ways to get people to spend more time on their site. And they do. They track all sorts of metrics, including “engagement” (if you click “like”, comment, share, or otherwise interact with content). They know which sorts of things are likely to capture your (and I mean you in specific!) attention and show you that. Your neighbor may have different interests and Facebook judges different things are likely to capture their attention.

Manipulating your attention

Attention turning into money isn’t unique for social media. In fact, in the article If It Bleeds, It Leads: Understanding Fear-Based Media, Psychology Today writes:

In previous decades, the journalistic mission was to report the news as it actually happened, with fairness, balance, and integrity. However, capitalistic motives associated with journalism have forced much of today’s television news to look to the spectacular, the stirring, and the controversial as news stories. It’s no longer a race to break the story first or get the facts right. Instead, it’s to acquire good ratings in order to get advertisers, so that profits soar.

News programming uses a hierarchy of if it bleeds, it leads. Fear-based news programming has two aims. The first is to grab the viewer’s attention. In the news media, this is called the teaser. The second aim is to persuade the viewer that the solution for reducing the identified fear will be in the news story. If a teaser asks, “What’s in your tap water that YOU need to know about?” a viewer will likely tune in to get the up-to-date information to ensure safety.

You’ve probably seen fear-based messages a lot on Facebook. They will highlight messages to liberals about being afraid of what Trump is doing, and to conservatives about being afraid of what Biden is doing. They may or may not even intentionally be doing this; it is their algorithm predicts that those would maximize time and engagement for certain people, so that’s what they see.

Fear leads to controversy

It’s not just fear, though. Social media also loves controversy. There’s nothing that makes people really want to stay on Facebook like anger. See something controversial and you’ll see hundreds or thousands of people are there arguing about it — and in the process, giving Facebook their attention. A quick Internet search will show you numerous articles on how marketing companes can leverage controvery to get attention and engagement with their campaigns.

Consequences of maximizing fear and controversy

What does it mean to society at large — and to you personally — that large companies make a lot of money by maximizing fear and controversy?

The most obvious way is it leads to less common ground. If the posts and reactions that show common ground are never seen because they don’t drive engagement, it poisons the well; left and right hate each other with ever more vigor — a profitable outcome to Facebook, but a poisonous one to all of us.

I have had several friendships lost because I — a liberal in agreement with these friends on political matters — still talk to Trump voters. On the other side, we’ve seen people storm the Michigan statehouse with weapons. How did that level of disagreement — and even fear behind it — get so firmly embedded in our society? Surely the fact that social media shows us things designed to stimulate fear and anger must play a role.

What does it do to our ability to have empathy for, and understand, others? The Facebook groups I’ve been in for like-minded people have largely been flooded with memes calling the President “rump” and other things clearly designed to make people angry or fearful. It’s a worthless experience, and not just that, but it’s a harmful experience.

When our major media — TV and social networks — all are optimizing for fear, anger, and controvesry, we have a society beholden to fear, anger, and controvesy.

In my next installment, I’m going to talk about what to do about this, including the decentralized social networks of the Fediverse that are specifically designed to put you back in charge of your attention.

Update 2020-12-16: There are two followup articles for this: how to join the Fediverse and non-creepy technology purchasing and gifting guides. The latter references the FSF’s page on software manipulation towards addiction, which is particularly relevant to this topic.

Thanksgiving in 2020

With COVID-19, Thanksgiving is a little different this year.

The kids enjoyed doing a little sightseeing by air – in our own plane (all socially-distanced of course!). We built a Prusa 3D printer from a kit (the boys and I, though Martha checked in periodically too). It arrived earlier than expected so that kept us busy for several days. And, of course, there was the Christmas decorating and Zoom church (where only our family is in the building, hosting the service for everyone).

What, so Thanksgiving doesn’t normally involve assembling printers, sightseeing from the sky, and printing tiny cups and dishes for miniature houses on a 3D printer?

I’ll be glad when COVID is over. Meantime, we have some memories to treasure too.

In Which COVID-19 Misinformation Leads To A Bunch of Graphs Made With Rust

A funny — and by funny, I mean sad — thing has happened. Recently the Kansas Department of Health and Environment (KDHE) has been analyzing data from the patchwork implementation of mask requirements in Kansas. They came to a conclusion that shouldn’t be surprising to anyone: masks help. They published a chart showing this. A right-wing propaganda publication got ahold of this, and claimed the numbers were “doctored” because there were two-different Y-axes.

I set about to analyze the data myself from public sources, and produced graphs of various kinds using a single Y-axis and supporting the idea that the graphs were not, in fact, doctored. Here’s one graph that’s showing that:

In order to do that, I had imported COVID-19 data from various public sources. Many states in the US are large enough to have significant variation in COVID-19 conditions, and many of the source people look at don’t show county-level data over time. I wanted to do that.

Eventually, I wrote covid19db, which ingests data from a number of public sources and generates a SQLite database file. Using Github Actions, this file is automatically updated every morning and available for download. Or, you can download the code and generate a database yourself locally.

Then, I wrote covid19ks, which generates various pretty graphs covering the data. These graphs, incidentally, turn out to highlight just how poorly the United States is doing compared to the rest of the industrialized world.

I hope that these resources, and especially covid19db, might be useful to others that would like to analyze the data. The code isn’t the prettiest since it was done in a hurry, but I think that functionally this is useful.

COVID-19 is serious for all ages. Treat it like WWII

Today I’d like to post a few updates about COVID-19 which I have gathered from credible sources, as well as some advice also gathered from credible sources.

Summary

  1. Coronavirus causes health impacts requiring hospitalization in a significant percentage of all adult age groups.
  2. Coronavirus also can cause no symptoms at all in many, especially children.
  3. Be serious about social distancing.

COVID-19 is serious for young adults too

According to this report based on a CDC analysis, between 14% and 20% of people aged 20 to 44 require hospitalization due to COVID-19. That’s enough to be taken seriously. See also this CNN story.

Act as if you are a carrier because you may be infected and not even know it, even children

Information on this is somewhat preliminary, but it is certainly known that a certain set of cases is asymptomatic. This article discusses manifestations in children, while this summary of a summary (note: not original research) suggests that 17.9% of people may not even know they are infected.

How serious is this? Serious.

This excellent article by Daniel W. Johnson, MD, is a very good read. Among the points it makes:

  • Anyone that says it’s no big deal is wrong.
  • If we treat this like WWI or WWII and everyone does the right things, we will be harmed but OK. If many but not all people do the right things, we’ll be like Italy. If we blow it off, our health care system and life as we know it will be crippled.
  • If we don’t seriously work to flatten the curve, many lives will be needlessly lost

Advice

I’m going to just copy Dr. Johnson’s advice here:

  1. You and your kids should stay home. This includes not going to church, not going to the gym, not going anywhere.
  2. Do not travel for enjoyment until this is done. Do not travel for work unless your work truly requires it.
  3. Avoid groups of people. Not just crowds, groups. Just be around your immediate family. I think kids should just play with siblings at this point – no play dates, etc.
  4. When you must leave your home (to get groceries, to go to work), maintain a distance of six feet from people. REALLY stay away from people with a cough or who look sick.
  5. When you do get groceries, etc., buy twice as much as you normally do so that you can go to the store half as often. Use hand sanitizer immediately after your transaction, and immediately after you unload the groceries.

I’m not saying people should not go to work. Just don’t leave the house for anything unnecessary, and if you can work from home, do it.

Everyone on this email, besides Mom and Dad, are at low risk for severe disease if/when they contract COVID-19. While this is great, that is not the main point. When young, well people fail to do social distancing and hygiene, they pick up the virus and transmit it to older people who are at higher risk for critical illness or death. So everyone needs to stay home. Even young people.

Tell every person over 60, and every person with significant medical conditions, to avoid being around people. Please do not have your kids visit their grandparents if you can avoid it. FaceTime them.

Our nation is the strongest one in the world. We have been through other extreme challenges and succeeded many times before. We WILL return to normal life. Please take these measures now to flatten the curve, so that we can avoid catastrophe.

I’d also add that many supermarkets offer delivery or pickup options that allow you to get your groceries without entering the store. Some are also offering to let older people shop an hour before the store opens to the general public. These could help you minimize your exposure.

Other helpful links

Here is a Reddit megathread with state-specific unemployment resources.

Scammers are already trying to prey on people. Here are some important tips to avoid being a victim.

Although there are varying opinions, some are recommending avoiding ibuprofen when treating COVID-19.

Bill Gates had some useful advice. Here’s a summary emphasizing the need for good testing.