All posts by John Goerzen

Remembering Tom Wallis, The System Administrator That Made The World Better

I never asked Tom why he hired me.

I was barely 17 at the time – already a Debian developer, full of more enthusiasm than experience, and Tom offered me a job. It was my first real sysadmin job, and to my delight, I got to work with Unix. For two years, I was the part-time assistant systems administrator for the Computer Science department at Wichita State University. And Tom was my boss, mentor, and old friend. Tom was walking proof that a system administrator can make the world a better place.

That amazing time was two decades ago now. And in the time since, every so often Tom and I would exchange emails. I enjoyed occasionally dropping by his office at work and surprising him.

So it was a shock to get an email this week that Tom had married for the first time at age 54, and passed away four days later due to a boating accident while on his honeymoon.

Tom was a man with a big laugh and an even bigger heart. When I started a Linux Users Group (LUG) on campus, there was Tom – helping to arrange a place to meet, Internet access when we needed it, and gave his evenings to simply be present and a supporter.

I had (and still have) a passion for Free/Open Source software. Linux was just new at the time, and was barely present in the department when I started. I was fortunate that CS was the “little dept. that could” back then, with wonderful people but not a lot of money, so a free operating system helped with a lot of problems. Tom supported me in my enthusiasm to introduce Debian all over the place. His trust meant much, and brought out the best in me.

I learned a lot from Tom, and more than just technology. A state university can be heavily bureaucratic place at times. Tom was friends with every “can-do” person on campus, it seemed, and they all managed to pull through and get things done – sometimes working around policies that were a challenge.

I have sometimes wondered if I am doing enough, making a big enough difference in the world. Does a programmer really make a difference in people’s lives?

Tom Wallis is proof that the answer is yes. From the stories I heard at his funeral today, I can only guess how many other lives he touched.

This week, Tom gave me one final gift: a powerful reminder that sysadmins and developers can make the world a better place, can touch people’s lives. I hope Tom knew how much I appreciated him. If I find a way to make a difference in someone’s life — maybe an intern I’ve hired, or someone I take flying — than I will have found a way to pass on Tom’s gift to another, and I hope I can.

tom-wallis-penguin

(This penguin was sitting out on the table of memorabilia from Tom today. I remember it from a shelf in his office.)

The downfall of… Trump or Democracy?

The future of the United States as a democracy is at risk. That’s plenty scary. More scary is that many Americans know this, but don’t care. And even more astonishing is that this same thing happened 45 years ago.

I remember it clearly. January 30, just a couple weeks ago. On that day, we had the news that FBI deputy director McCabe — a frequent target of apparently-baseless Trump criticism — had been pushed out. The Trump administration refused to enforce the bipartisan set of additional sanctions on Russia. And the House Intelligence Committee voted on party lines to release what we all knew then, and since have seen confirmed, was a memo filled with errors designed to smear people investigating the president, but which nonetheless contained enough classified material to cause an almighty kerfuffle in Washington.

I told my wife that evening, “I think today will be remembered as a turning point. Either to the downfall of Trump, or the downfall of our democracy, but I don’t know which.”

I have not written much about this scandal, because so many quality words have already been written. But it is time to add something.

I was interested in Watergate years ago. Back in middle school, I read All the President’s Men. I wondered what it must have been like to live through those events — corruption at the highest level of government, dirty tricks, not knowing how it would play out. I wished I could have experienced it.

A couple of decades later, I have got my wish and I am not amused. After all:

“If these allegations prove to be true, what they were seeking to steal was not the jewels, money or other property of American citizens, but something much more valuable — their most precious heritage, the right to vote in a free election…

If the allegations… are substantiated, there has been a very serious subversion of the integrity of the electoral process, and the committee will be obliged to consider the manner in which such a subversion affects the continued existence of this nation as a representative democracy, and how, if we are to survive, such subversions may be prevented in the future.”

Sen. Sam Ervin Jr, May 17, 1973

That statement from 45 years ago captures accurately my contemporary fears. If foreign interference in our elections is not only tolerated but embraced, where does that leave us? Are we really a republic anymore?

I have been diving back into Watergate. In One Man Against The World: The Tragedy of Richard Nixon, written by Tim Weiner in 2015, he dives into the Nixon story in unprecedented detail, thanks to the release of many more files from that time. In his very first page, he writes:

[Nixon] made war in pursuit of peace. He committed crimes in the name of the law. He tore the country apart while trying to unite it. He sabotaged his presidency by violating the Constitution. He destroyed himself and damaged the nation through deliberate acts of folly…

He practiced geopolitics without subtlety; he preferred subterfuge and brutality. He dropped bombs and napalm without remorse; he believed they delivered a political message beyond flood and fire. Hr charted the course of the war without a strategy; he delivered victory to his adversaries.

His gravest decisions undermined his allies abroad. His grandest delusions armed his enemies at home…

The truth was not in him; secrecy and deception were his touchstones.

That these words describe another American president, one that I’m sure Weiner had not foreseen, is jarring. The parallels between Nixon and Trump in the pages of Weiner’s book are so strong that one sometimes wonders if Weiner has a more accurate story of Trump than Wolff got – and also if the pages of his book let us see what’s in store for us this year.

Today I started listening to the excellent podcast Slow Burn. If you have time for nothing else, listen to episode 5: True Believers. It discusses the politicization of the Senate Watergate committee, and more ominously, the efforts of reports to understand the people that still supported Nixon — despite all the damning testimony already out there.

Gail Sheehy went to a bar where Nixon supporters gathered, wanting to get their reaction to the Watergate hearings. The supporters didn’t want to watch. They thought the hearings were just an attempt by liberals to take down Nixon. Sheehy found the president’s people to be “angry, demoralized, and disconcertingly comfortable with the idea of a police state run by Richard Nixon.”

These guys felt they were nobodies… except Richard Nixon gave them an identity. He was a tough guy who was “going to get rid of all those anti-war people, anarchists, terrorists… the people that were tearing down our country!”

Art Buchwald’s tongue-in-cheek handy excuses for Nixon backers seems to be copied almost verbatim by Fox News (substitute Hillary’s emails for Chappaquiddick).

And what happened to the scum of Richard Nixon’s era? Yes, some went to jail, but not all.

  • Steve King, one of Nixon’s henchmen that kidnapped Martha Mitchell (wife of Attorney General and Nixon henchman John Mitchell) for a week to keep her from spilling the beans on Watergate, beat her up, and had her drugged — well he was appointed by Trump to be ambassador to the Czech Republic and confirmed by the Senate.
  • The man that said that the Watergate burglars were “not criminal at heart” because “their only aim was to re-elect the president” later got elected president himself, and pardoned one of the burglars. (Ronald Reagan)
  • The man that said “just let the president do his job!” was also elected president (George H. W. Bush)
  • The man that finally carried out Nixon’s order to fire special prosecutor Archibald Cox was nominated to the Supreme Court, but his nomination was blocked in the Senate. (Robert Bork) He was, however, on the United States Court of Appeals for 6 years.
  • And in an odd conspiracy-laden introduction to a reprint of a youth’s history book on Watergate, none other than Roger Stone, wrapped up in Trump’s shenanigans, was trying to defend Nixon. Oh, and he was a business partner with Paul Manafort and lobbyist for Ferdinand Marcos.

One comfort from all of this is the knowledge that we had been there before. We had lived through an era of great progress in civil rights, and right after that elected a dictatorial crook president. We survived the president’s fervent supporters refusing to believe overwhelming evidence of his crookedness. We survived.

And yet, that is no guarantee. After all, as John Dean put it, Nixon “might have survived if there’d been a Fox News.”

The Big Green Weasel Song

One does not normally get into one’s car intending to sing about barfing green weasels to the tune of a Beethoven symphony for a quarter of an hour. And yet, somehow, that’s what I wound up doing today.

Little did I know that when Jacob started band last fall, it would inspire him to sing about weasels to the tune of Beethoven’s 9th. That may come as a surprise to his band teacher, too, who didn’t likely expect that having the kids learn the theme to the Ninth Symphony would inspire them to sing about large weasels.

But tonight, as we were driving, I mentioned that I knew the original German words. He asked me to sing. I did.

Then, of course, Jacob and Oliver tried to sing it back in German. This devolved into gibberish and a fit of laughter pretty quick, and ended with something sounding like “schneezel”. So of course, I had to be silly and added, “I have a big green weasel!”

From then on, they were singing about big green weasels. It wasn’t long before they decided they would sing “the chorus” and I was supposed to improvise verse after verse about these weasels. Improvising to the middle of the 9th Symphony isn’t the easiest thing, but I had verses about giving weasels a hug, about weasels smelling like a bug, about a green weasel on a chair, about a weasel barfing on the stair. And soon, Jacob wanted to record the weasel song to make a CD of it. So we did, before we even got to town. Here it is:

[Youtube link]

I love to hear children delight in making music. And I love to hear them enjoying Beethoven. Especially with weasels.

I just wonder what will happen when they learn about Mozart.

How are you handling building local Debian/Ubuntu packages?

I’m in the middle of some conversations about Debian/Ubuntu repositories, and I’m curious how others are handling this.

How are people maintaining repos for an organization? Are you integrating them with a git/CI (github/gitlab, jenkins/travis, etc) workflow? How do packages propagate into repos? How do you separate prod from testing? Is anyone running buildd locally, or integrating with more common CI tools?

I’m also interested in how people handle local modifications of packages — anything from newer versions of C libraries to newer interpreters. Do you just use the regular Debian toolchain, packaging them up for (potentially) the multiple distros/versions that you have in production? Pin them in apt?

Just curious what’s out there.

Some Googling has so far turned up just one relevant hit: Michael Prokop’s DebConf15 slides, “Continuous Delivery of Debian packages”. Looks very interesting, and discusses jenkins-debian-glue.

Some tangentially-related but interesting items:

Edit 2018-02-02: I should have also mentioned BuildStream

An old DOS BBS in a Docker container

Awhile back, I wrote about my Debian Docker base images. I decided to extend this concept a bit further: to running DOS applications in Docker.

But first, a screenshot:

It turns out this is possible, but difficult. I went through all three major DOS emulators available (dosbox, qemu, and dosemu). I got them all running inside the Docker container, but had a number of, er, fun issues to resolve.

The general thing one has to do here is present a fake modem to the DOS environment. This needs to be exposed outside the container as a TCP port. That much is possible in various ways — I wound up using tcpser. dosbox had a TCP modem interface, but it turned out to be too buggy for this purpose.

The challenge comes in where you want to be able to accept more than one incoming telnet (or TCP) connection at a time. DOS was not a multitasking operating system, so there were any number of hackish solutions back then. One might have had multiple physical computers, one for each incoming phone line. Or they might have run multiple pseudo-DOS instances under a multitasking layer like DESQview, OS/2, or even Windows 3.1.

(Side note: I just learned of DESQview/X, which integrated DESQview with X11R5 and replaced the Windows 3 drivers to allow running Windows as an X application).

For various reasons, I didn’t want to try running one of those systems inside Docker. That left me with emulating the original multiple physical node setup. In theory, pretty easy — spin up a bunch of DOS boxes, each using at most 1MB of emulated RAM, and go to town. But here came the challenge.

In a multiple-physical-node setup, you need some sort of file sharing, because your nodes have to access the shared message and file store. There were a myriad of clunky ways to do this in the old DOS days – Netware, LAN manager, even some PC NFS clients. I didn’t have access to Netware. I tried the Microsoft LM client in DOS, talking to a Samba server running inside the Docker container. This I got working, but the LM client used so much RAM that, even with various high memory tricks, BBS software wasn’t going to run. I couldn’t just mount an underlying filesystem in multiple dosbox instances either, because dosbox did caching that wasn’t going to be compatible.

This is why I wound up using dosemu. Besides being a more complete emulator than dosbox, it had a way of sharing the host’s filesystems that was going to work.

So, all of this wound up with this: jgoerzen/docker-bbs-renegade.

I also prepared building blocks for others that want to do something similar: docker-dos-bbs and the lower-level docker-dosemu.

As a side bonus, I also attempted running this under Joyent’s Triton (SmartOS, Solaris-based). I was pleasantly impressed that I got it all almost working there. So yes, a Renegade DOS BBS running under a Linux-based DOS emulator in a container on a Solaris machine.

The Yellow House Phone Company (Featuring Asterisk and an 11-year-old)

“Well Jacob, do you think we should set up our own pretend phone company in the house?”

“We can DO THAT?”

“Yes!”

“Then… yes. Yes! YES YES YESYESYESYES YES! Let’s do it, dad!”

Not long ago, my parents had dug up the old phone I used back in the day. We still have a landline, and Jacob was having fun discovering how an analog phone works. I told him about the special number he could call to get the time and temperature read out to him. He discovered what happens if you call your own number and hang up. He figured out how to play “Mary Had a Little Lamb” using touchtone keys (after a slightly concerned lecture from me setting out some rules to make sure his “musical dialing” wouldn’t result in any, well, dialing.)

He was hooked. So I thought that taking it to the next level would be a good thing for a rainy day. I have run Asterisk before, though I had unfortunately gotten rid of most of my equipment some time back. But I found a great deal on a Cisco 186 ATA (Analog Telephone Adapter). It has two FXS lines (FXS ports simulate the phone company, and provide dialtone and ring voltage to a connected phone), and of course hooks up to the LAN.

We plugged that in, and Jacob was amazed to see its web interface come up. I had to figure out how to configure it (unfortunately, it uses SCCP rather than SIP, and figuring out Asterisk’s chan_skinny took some doing, but we got there.)

I set up voicemail. He loved it. He promptly figured out how to record his own greetings. We set up a second phone on the other line, so he could call between them. The cordless phones in our house support SIP, so I configured one of them as a third line. He spent a long time leaving himself messages.

IMG_3465

Pretty soon we both started having ideas. I set up extension 777, where he could call for the time. Then he wanted a way to get the weather forecast. Well, weather-util generates a text-based report. With it, a little sed and grep tweaking, the espeak TTS engine, and a little help from sox, I had a shell script worked up that would read back a forecast whenever he called a certain extension. He was super excited! “That’s great, dad! Can it also read weather alerts too?” Sure! weather-util has a nice option just for that. Both boys cackled as the system tried to read out the NWS header (their timestamps like 201711031258 started with “two hundred one billion…”)

Then I found an online source for streaming NOAA Weather Radio feeds – Jacob enjoys listening to weather radio – and I set up another extension he could call to listen to that. More delight!

But it really took off when I asked him, “Would you like to record your own menu?” “You mean those things where it says press 1 or 2 for this or that?” “Yes.” “WE CAN DO THAT?” “Oh yes!” “YES, LET’S DO IT RIGHT NOW!”

So he recorded a menu, then came and hovered by me while I hacked up extensions.conf, then eagerly went back to the phone to try it. Oh the excitement of hearing hisown voice, and finding that it worked! Pretty soon he was designing sub-menus (“OK Dad, so we’ll set it up so people can press 2 for the weather, and then choose if they want weather radio or the weather report. I’m recording that now. Got it?”)

He has informed me that next Saturday we will build an intercom system “like we have at school.” I’m going to have to have some ideas on how to tie Squeezebox in with Asterisk to make that happen, I think. Maybe this will do.

Switching to xmonad + Gnome – and ditching a Mac

I have been using XFCE with xmonad for years now. I’m not sure exactly how many, but at least 6 years, if not closer to 10. Today I threw in the towel and switched to Gnome.

More recently, at a new job, I was given a Macbook Pro. I wasn’t entirely sure what to think of this, but I thought I’d give it a try. I found MacOS to be extremely frustrating and confining. It had no real support for a tiling window manager, and although projects like amethyst tried to approximate what xmonad can do on Linux, they were just too limited by the platform and were clunky. Moreover, the entire UI was surprisingly sluggish; maybe that was an induced effect from animations, but I don’t think that explains it. A Debisn stretch install, even on inferior hardware, was snappy in a way that MacOS never was. So I have requested to swap for a laptop that will run Debian. The strange use of Command instead of Control for things, combined with the overall lack of configurability of keybindings, meant that I was going to always be fighting muscle memory moving from one platform to another. Not only that, but being back in the world of a Free Software OS means a lot.

Now then, back to xmonad and XFCE situation. XFCE once worked very well with xmonad. Over the years, this got more challenging. Around the jessie (XFCE 4.10) time, I had to be very careful about when I would let it save my session, because it would easily break. With stretch, I had to write custom scripts because the panel wouldn’t show up properly, and even some application icons would be invisible, if things were started in a certain order. This took much trial and error and was still cumbersome.

Gnome 3, with its tightly-coupled Gnome Shell, has never been compatible with other window managers — at least not directly. A person could have always used MATE with xmonad — but a lot of people that run XFCE tend to have some Gnome 3 apps (for instance, evince) anyhow. Cinnamon also wouldn’t work with xmonad, because it is simply another tightly-coupled shell instead of Gnome Shell. And then today I discovered gnome-flashback. gnome-flashback is a Gnome 3 environment that uses the traditional X approach with a separate window manager (metacity of yore by default). Sweet.

It turns out that Debian’s xmonad has built-in support for it. If you know the secret: apt-get install gnome-session-flashback (OK, it’s not so secret; it’s even in xmonad’s README.Debian these days) Install that, plus gnome and gdm3 and things are nice. Configure xmonad with GNOME support and poof – goodness right out of the box, selectable from the gdm sessions list.

I still have some gripes about Gnome’s configurability (or lack thereof). But I’ve got to say: This environment is the first one I’ve ever used that got external display switching very nearly right without any configuration, and I include MacOS in that. Plug in an external display, and poof – it’s configured and set up. You can hit a toggle key (Windows+P by default) to change the configurations, or use the Display section in gnome-control-center. Unplug it, and it instantly reconfigures itself to put everything back on the laptop screen. Yessss! I used to have scripts to do this in the wheezy/jessie days. XFCE in stretch had numerous annoying failures in this area which rendered the internal display completely dark until the next reboot – very frustrating. With Gnome, it just works. And, even if you have “suspend on lid closed” turned on, if the system is powered up and hooked up to an external display, it will keep running even if the lid is closed, figuring you must be using it on the external screen. Another thing the Mac wouldn’t do well.

All in all, some pretty good stuff here. I continue to be impressed by stretch. It is darn impressive to put this OS on generic hardware and have it outshine the closed-ecosystem Mac!

The Joy of Exploring: Old Phone Systems, Pizza, and Discovery

This story involves boys pretending to be pizza deliverymen using a working automated Strowger telephone exchange demonstrator on display in a museum, which is very old and is, to my knowledge, the only such working exhibit in the world. (Yes, I have video.) But first, a thought on exploration.

There are those that would say that there is nothing left to explore anymore – that the whole earth is mapped, photographed by satellites, and, well, known.

I prefer to look at it a different way: the earth is full of places that billions of people will never see, and probably don’t even know about. Those places may be quiet country creeks, peaceful neighborhoods one block away from major tourist attractions, an MTA museum in Brooklyn, a state park in Arkansas, or a beautiful church in Germany.

Martha is not yet two months old, and last week she and I spent a surprisingly long amount of time just gazing at tree branches — she was mesmerized, and why not, because to her, everything is new.

As I was exploring in Portland two weeks ago, I happened to pick up a nearly-forgotten book by a nearly-forgotten person, Beryl Markham, a woman who was a pilot in Africa about 80 years ago. The passage that I happened to randomly flip to in the bookstore, which really grabbed my attention, was this:

The available aviation maps of Africa in use at that time all bore the cartographer’s scale mark, ‘1/2,000,000’ — one over two million. An inch on the map was about thitry-two miles in the air, as compared to the flying maps of Europe on which one inch represented no more than four air miles.

Moreover, it seemed that the printers of the African maps had a slightly malicious habit of including, in large letters, the names of towns, junctions, and villages which, while most of them did exist in fact, as a group of thatched huts may exist or a water hold, they were usually so inconsequential as completely to escape discovery from the cockpit.

Beyond this, it was even more disconcerting to examine your charts before a proposed flight only to find that in many cases the bulk of the terrain over which you had to fly was bluntly marked: ‘UNSURVEYED’.

It was as if the mapmakers had said, “We are aware that between this spot and that one, there are several hundred thousands of acres, but until you make a forced landing there, we won’t know whether it is mud, desert, or jungle — and the chances are we won’t know then!”

— Beryl Markham, West With the Night

My aviation maps today have no such markings. The continent is covered with radio beacons, the world with GPS, the maps with precise elevations of the ground and everything from skyscrapers to antenna towers.

And yet, despite all we know, the world is still a breathtaking adventure.

Yesterday, the boys and I were going to fly to Abilene, KS, to see a museum (Seelye Mansion). Circumstances were such that we neither flew, nor saw that museum. But we still went to Abilene, and wound up at the Museum of Independent Telephony, a wondrous place for anyone interested in the history of technology. As it is one of those off-the-beaten-path sorts of places, the boys got 2.5 hours to use the hands-on exhibits of real old phones, switchboards, and also the schoolhouse out back. They decided — why not? — to use this historic equipment to pretend to order pizzas.

Jacob and Oliver proceeded to invent all sorts of things to use the phones for: ordering pizza, calling the cops to chase the pizza delivery guys, etc. They were so interested that by 2PM we still hadn’t had lunch and they claimed “we’re not hungry” despite the fact that we were going to get pizza for lunch. And I certainly enjoyed the exhibits on the evolution of telephones, switching (from manual plugboards to automated switchboards), and such.

This place was known – it even has a website, I had been there before, and in fact so had the boys (my parents took them there a couple of years ago). But yesterday, we discovered the Strowger switch had been repaired since the last visit, and that it, in fact, is great for conversations about pizza.

Whether it’s seeing an eclipse, discovering a fascination with tree branches, or historic telephones, a spirit of curiosity and exploration lets a person find fun adventures almost anywhere.

The Eclipse

Highway US-81 in northern Kansas and southern Nebraska is normally a pleasant, sleepy sort of drive. It was upgraded to a 4-lane road not too long ago, but as far as 4-lane roads go, its traffic is typically light. For drives from Kansas to South Dakota, it makes a pleasant route.

Yesterday was eclipse day. I strongly suspect that highway 81 had more traffic that day than it ever has before, or ever will again. For nearly the entire 3-hour drive to Geneva, NE, it was packed — though mostly still moving at a good speed. And for our entire drive back, highway 81 and every other southbound road we used was so full it felt like rush hour in Dallas. (Well, not quite. Traffic was still moving.) I believe scenes like this were played out across the continent.

I’ve been taking a lot of photos, and writing about our new baby Martha lately. Now it’s time to write a bit about some more adventures with Jacob and Oliver – they’re now in third and fifth grades in school.

We had been planning to fly, and airports I called were either full, or were planning to park planes in the grass, or even shut down some runways to use for parking. The airport in the little town of Beatrice, NE (which I had visited twice before) was even going to have a temporary FAA control tower. At the last minute, due to some storm activity near home at departure time, we unloaded the plane and drove instead.

The atmosphere at the fairgrounds in Geneva was festive. One family had brought bubbles for their kids — and extras to share.

IMG_20170821_113229

I had bought the boys a book about the eclipse, which they were reading before and during the event. They were both great, safe users of their eclipse glasses.

IMG_20170821_124809

Jacob caught a toad, and played with it for awhile. He wanted to bring it home with us, but I convinced him to let me take a picture of him with his toad friend instead.

IMG_20170821_124553

While we were waiting for totality, a number of buses from the local school district arrived. So by the time the big moment arrived, we could hear the distant roar of delight and applause from the school children gathered at the far end of the field, plus all the excitement nearby. Both boys were absolutely ecstatic to be witnessing it (and so was I!) “Wow!” “Awesome!” And simple cackles of delight were heard. On the drive home, they both kept talking about how amazing it was, and it was “once in a lifetime.”

We enjoyed our “eclipse neighbors” – the woman from San Antonio next to us, the surprise discovery of another family from just a few miles from us parked two cars down, even running into relatives at a restaurant on the way home. The applause from all around when it started – and when it ended. And the feeling, which is hard to describe, of awe and amazement at the wonders of our world and our universe.

There are many problems with the world right now, but somehow there’s something right about people coming together from all over to enjoy it.

A new baby and deep smiles

IMG_2059

A month ago, we were waiting for our new baby; time seemed to stand still. Now she is here! Martha Goerzen was born recently, and she is doing well and growing! Laura and I have enjoyed moments of cuddling her, watching her stare at our faces, hearing her (hopefully) soft sounds as she falls asleep in our arms. It is also heart-warming to see Martha’s older brothers take such an interest in her. Here is the first time Jacob got to hold her:

IMG_1846

Oliver, who is a boy very much into sports, play involving police and firefighters, and such, has started adding “aww” and “she’s so cute!” to his common vocabulary. He can be very insistent about interrupting me to hold her, too.