First Experiences with Stretch

Uncategorized

I’ve done my first upgrades to Debian stretch at this point. The results have been overall good. On the laptop my kids use, I helped my 10-year-old do it, and it worked flawlessly. On my workstation, I got a kernel panic on boot. Hmm.

Unfortunately, my system has to use the nv drivers, which leaves me with an 80×25 text console. It took some finagling (break=init in grub, then manually insmoding the appropriate stuff based on modules.dep for nouveau), but finally I got a console so I could see what was breaking. It appeared that init was crashing because it couldn’t find liblz4. A little digging shows that liblz4 is in /usr, and /usr wasn’t mounted. I’ve filed the bug on systemd-sysv for this.

I run root on ZFS, and further digging revealed that I had datasets named like this:

  • tank/hostname-1/ROOT
  • tank/hostname-1/usr
  • tank/hostname-1/var

This used to be fine. The mountpoint property of the usr dataset put it at /usr without incident. But it turns out that this won’t work now, unless I set ZFS_INITRD_ADDITIONAL_DATASETS in /etc/default/zfs for some reason. So I renamed them so usr was under ROOT, and then the system booted.

Then I ran samba not liking something in my bind interfaces line (to be fair, it did still say eth0 instead of br0). rpcbind was failing in postinst, though a reboot seems to have helped that. More annoying was that I had trouble logging into my system because resolv.conf was left empty (despite dns-* entries in /etc/network/interfaces and the presence of resolvconf). I eventually repaired that, and found that it kept removing my “search” line. Eventually I removed resolvconf.

Then mariadb’s postinst was silently failing. I eventually discovered it was sending info to syslog (odd), and /etc/init.d/apparmor teardown let it complete properly. It seems like there may have been an outdated /etc/apparmor.d/cache/usr.sbin.mysql out there for some reason.

Then there was XFCE. I use it with xmonad, and the session startup was really wonky. I had to zap my sessions, my panel config, etc. and start anew. I am still not entirely sure I have it right, but I at do have a usable system now.

Fixing the Problems with Docker Images

Linux, Programming

I recently wrote about the challenges in securing Docker container contents, and in particular with keeping up-to-date with security patches from all over the Internet.

Today I want to fix that.

Besides security, there is a second problem: the common way of running things in Docker pretends to provide a traditional POSIX API and environment, but really doesn’t. This is a big deal.

Before diving into that, I want to explain something: I have often heard it said the Docker provides single-process containers. This is unambiguously false in almost every case. Any time you have a shell script inside Docker that calls cp or even ls, you are running a second process. Web servers from Apache to whatever else use processes or threads of various types to service multiple connections at once. Many Docker containers are single-application, but a process is a core part of the POSIX API, and very little software would work if it was limited to a single process. So this is my little plea for more precise language. OK, soapbox mode off.

Now then, in a traditional Linux environment, besides your application, there are other key components of the system. These are usually missing in Docker containers.

So today, I will fix this also.

In my docker-debian-base images, I have prepared a system that still has only 11MB RAM overhead, makes minimal changes on top of Debian, and yet provides a very complete environment and API. Here’s what you get:

  • A real init system, capable of running standard startup scripts without modification, and solving the nasty Docker zombie reaping problem.
  • Working syslog, which can either export all logs to Docker’s logging infrastructure, or keep them within the container, depending on your preferences.
  • Working real schedulers (cron, anacron, and at), plus at least the standard logrotate utility to help prevent log files inside the container from becoming huge.

The above goes into my “minimal” image. Additional images add layers on top of it, and here are some of the features they add:

  • A real SMTP agent (exim4-daemon-light) so that cron and friends can actually send you mail
  • SSH client and server (optionally exposed to the Internet)
  • Automatic security patching via unattended-upgrades and needsrestart

All of the above, including the optional features, has an 11MB overhead on start. Not bad for so much, right?

From here, you can layer on top all your usual Dockery things. You can still run one application per container. But you can now make sure your disk doesn’t fill up from logs, run your database vacuuming commands at will, have your blog download its RSS feeds every few minutes, etc — all from within the container, as it should be. Furthermore, you don’t have to reinvent the wheel, because Debian already ships with things to take care of a lot of this out of the box — and now those tools will just work.

There is some popular work done in this area already by phusion’s baseimage-docker. However, I made my own for these reasons:

  • I wanted something based on Debian rather than Ubuntu
  • By using sysvinit rather than runit, the OS default init scripts can be used unmodified, reducing the administrative burden on container builders
  • Phusion’s system is, for some reason, not auto-built on the Docker hub. Mine is, so it will be automatically revised whenever the underlying Debian system, or the Github repository, is.

Finally a word on the choice to use sysvinit. It would have been simpler to use systemd here, since it is the default in Debian these days. Unfortunately, systemd requires you to poke some holes in the Docker security model, as well as mount a cgroups filesystem from the host. I didn’t consider this acceptable, and sysvinit ran without these workarounds, so I went with it.

With all this, Docker becomes a viable replacement for KVM for various services on my internal networks. I’ll be writing about that later.

Family Spring: A Story in Photos

Family, Travel

This has been a spring with times to relax, times to be busy, times of anticipation of a new baby, and times of enjoying our family.

Rather than write a lot of words about it, I’m telling the story in photos.

To view, click here, then click Show Info in the upper right to see captions. You can pause it with the button in the lower left, and use arrow keys to advance.

Alternatively, there’s a captionless slideshow available here.

Here’s one photo to get you started:

Happy about the little sister on the way

Flying with my brothers

Aviation, Family, Travel

Picture one Sunday morning. Three guys are seemingly-randomly walking into a Mennonite church in rural Nebraska. One with long hair and well-maintained clothes from the 70s. Another dressed well enough to be preaching. And the third simply dressed to be comfortable, with short hair showing evidence of having worn a headset for a couple of hours that morning. This was the scene as we made a spur-of-the-moment visit to that church — which resulted in quite some surprise all around, since my brother knew a number of people there.

For instance:

Pastor: Peter! What are you doing here?

Peter: [jokingly] Is that how you greet visitors here?

And then, of course, Peter would say, “Well, we were flying home from South Dakota and figured we’d stop in at Beatrice for fuel. And drop in on you.” Followed by some surprise that we would stop at their little airport (which is quite a nice one).

This all happened because it was windy. This is the fun adventure of aviation. Sometimes you plan to go to Texas, but the weather there is terrible, so you discover a 100-year-old landmark in Indiana instead. Or sometimes, like a couple of weeks ago, we planned to fly straight home but spent a few hours exploring rural Nebraska.

The three of us flew to Sioux Falls, SD, in a little Cessna to visit my uncle and aunt up there. On our flight up, we stopped at the little airport in Seward, NE. It was complete with this unique elevated deck. In my imagination, this is used for people to drink beer while watching the planes land.

IMG_20170512_113323

In South Dakota, we had a weekend full of card and board games, horseshoes, and Crokinole with my uncle and aunt, who are always fun to visit. We had many memories of visits up there as children — and the pleasant enjoyment of the fact that we didn’t need an 8-hour drive to get there. We flew back with a huge bag of large rhubarb from their garden (that too is something of a tradition!)

It was a fun weekend to spend with my brothers — first time we’d been able to do this in a long while. And it marked the 11th state I’ve flown into, and over 17,000 miles of flying.

Is there any way to truly secure Docker container contents?

Uncategorized

There is much to like about Docker. Much has been written about it, and about how secure the containerization is.

This post isn’t about that. This is about keeping what’s inside each container secure. I believe we have a fundamental problem here.

Earlier this month, a study on security vulnerabilities on Docker Hub came out, and the picture isn’t pretty. One key finding:

Over 80% of the :latest versions of official images contained at least on high severity vulnerability!

And it’s not the only one raising questions.

Let’s dive in and see how we got here.

It’s hard to be secure, but Debian makes it easier

Let’s say you want to run a PHP application like WordPress under Apache. Here are the things you need to keep secure:

  • WordPress itself
  • All plugins, themes, customizations
  • All PHP libraries it uses (MySQL, image-processing, etc.)
  • MySQL
  • Apache
  • All libraries MySQL or Apache use: OpenSSL, libc, PHP itself, etc.
  • The kernel
  • All containerization tools

On Debian (and most of its best-known derivatives), we are extremely lucky to have a wonderful security support system. If you run a Debian system, the combination of unattended-updates, needrestart, debsecan, and debian-security-support will help one keep a Debian system secure and verify it is. When the latest OpenSSL bug comes out, generally speaking by the time I wake up, unattended-updates has already patched it, needrestart has already restarted any server that uses it, and I’m protected. Debian’s security team generally backports fixes rather than just say “here’s the new version”, making it very safe to automatically apply patches. As long as I use what’s in Debian stable, all layers mentioned above will be protected using this scheme.

This picture is much nicer than what we see in Docker.

Problems

We have a lot of problems in the Docker ecosystem:

  1. No built-in way to know when a base needs to be updated, or to automatically update it
  2. Diverse and complicated vendor security picture
  3. No way to detect when intermediate libraries need to be updated
  4. Complicated final application security picture

Let’s look at them individually.

Problem : No built-in way to know when a base needs to be updated, or to automatically update it

First of all, there is nothing in Docker like unattended-updates. Although a few people have suggested ways to run unattended-updates inside containers, there are many reasons that approach doesn’t work well. The standard advice is to update/rebuild containers.

So how do you know when to do that? It is not all that obvious. Theoretically, official OS base images will be updated when needed, and then other Docker hub images will detect the base update and be rebuilt. So, if a bug in a base image is found, and if the vendors work properly, and if you are somehow watching, then you could be protected. There is work in this area; tools such as watchtower help here.

But this can lead to a false sense of security, because:

Problem : Diverse and complicated vendor security picture

Different images can use different operating system bases. Consider just these official images, and the bases they use: (tracking latest tag on each)

  • nginx: debian:stretch-slim (stretch is pre-release at this date!)
  • mysql: debian:jessie
  • mongo: debian:wheezy-slim (previous release)
  • apache httpd: debian:jessie-backports
  • postgres: debian:jessie
  • node: buildpack-deps:jessie, eventually depends on debian:jessie
  • wordpress: php:5.6-apache, eventually depends on debian:jessie

And how about a few unofficial images?

  • oracle/openjdk: oraclelinux:latest
  • robotamer/citadel: debian:testing (dangerous, because testing is an alias for different distros at different times)
  • docker.elastic.co/kibana: ubuntu of some sort

The good news is that Debian jessie seems to be pretty popular here. The bad news is that you see everything from Oracle Linux, to Ubuntu, to Debian testing, to Debian oldstable in just this list. Go a little further, and you’ll see Alpine Linux, CentOS, and many more represented.

Here’s the question: what do you know about the security practices of each of these organizations? How well updated are their base images? Even if it’s Debian, how well updated is, for instance, the oldstable or the testing image?

The attack surface here is a lot larger than if you were just using a single OS. But wait, it gets worse:

Problem #3: No way to detect when intermediate libraries need to be updated

Let’s say your Docker image is using a base that is updated immediately when a security problem is found. Let’s further assume that your software package (WordPress, MySQL, whatever) is also being updated.

What about the intermediate dependencies? Let’s look at the build process for nginx. The Dockerfile for it begins with Debian:stretch-slim. But then it does a natural thing: it runs an apt-get install, pulling in packages from both Debian and an nginx repo.

I ran the docker build across this. Of course, the apt-get command brings in not just the specified packages, but also their dependencies. Here are the ones nginx brought in:

fontconfig-config fonts-dejavu-core gettext-base libbsd0 libexpat1 libfontconfig1 libfreetype6 libgd3 libgeoip1 libicu57 libjbig0 libjpeg62-turbo libpng16-16 libssl1.1 libtiff5 libwebp6 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxml2 libxpm4 libxslt1.1 nginx nginx-module-geoip nginx-module-image-filter nginx-module-njs nginx-module-xslt ucf

Now, what is going to trigger a rebuild if there’s a security fix to libssl1.1 or libicu57? (Both of these have a history of security holes.) The answer, for the vast majority of Docker images, seems to be: nothing automatic.

Problem #4: Complicated final application security picture

And that brings us to the last problem: Let’s say you want to run an application in Docker. exim, PostgreSQL, Drupal, or maybe something more obscure. Who is watching for security holes in it? If you’re using Debian packages, the Debian security team is. If you’re using a Docker image, well, maybe it’s the random person that contributed it, maybe it’s the vendor, maybe it’s Docker, maybe it’s nobody. You have to take this burden on yourself, to validate the security support picture for each image you use.

Conclusion

All this adds up to a lot of work, which is not taken care of for you by default in Docker. It is no surprise that many Docker images are insecure, given this picture. The unfortunate reality is that many Docker containers are running with known vulnerabilities that have known fixes, but just aren’t, and that’s sad.

I wonder if there are any practices people are using that can mitigate this better than what the current best-practice recommendations seem to be?

Parsing the GOP’s Health Insurance Statistics

Politics

There has been a lot of noise lately about the GOP health care plan (AHCA) and the differences to the current plan (ACA or Obamacare). A lot of statistics are being misinterpreted.

The New York Times has an excellent analysis of some of this. But to pick it apart, I want to highlight a few things:

Many Republicans are touting the CBO’s estimate that, some years out, premiums will be 10% lower under their plan than under the ACA. However, this carries with it a lot of misleading information.

First of all, many are spinning this as if costs would go down. That’s not the case. The premiums would still rise — they would just have risen less by the end of the period than under ACA. That also ignores the immediate spike and throwing millions out of the insurance marketplace altogether.

Now then, where does this 10% number come from? First of all, you have to understand the older people are substantially more expensive to the health system, and therefore more expensive to insure. ACA limited the price differential from the youngest to the oldest people, which meant that in effect some young people were subsidizing older ones on the individual market. The GOP plan removes that limit. Combined with other changes in subsidies and tax credits, this dramatically increases the cost to older people. For instance, the New York Times article cites a CBO estimate that “the price an average 64-year-old earning $26,500 would need to pay after using a subsidy would increase from $1,700 under Obamacare to $14,600 under the Republican plan.”

They further conclude that these exceptionally high rates would be so unaffordable to older people that the older people will simply stop buying insurance on the individual market. This means that the overall risk pool of people in that market is healthier, and therefore the average price is lower.

So, to sum up: the reason that insurance premiums under the GOP plan will rise at a slightly slower rate long-term is that the higher-risk people will be unable to afford insurance in the first place, leaving only the cheaper people to buy in.

Silent Data Corruption Is Real

Debian

Here’s something you never want to see:

ZFS has detected a checksum error:

   eid: 138
 class: checksum
  host: alexandria
  time: 2017-01-29 18:08:10-0600
 vtype: disk

This means there was a data error on the drive. But it’s worse than a typical data error — this is an error that was not detected by the hardware. Unlike most filesystems, ZFS and btrfs write a checksum with every block of data (both data and metadata) written to the drive, and the checksum is verified at read time. Most filesystems don’t do this, because theoretically the hardware should detect all errors. But in practice, it doesn’t always, which can lead to silent data corruption. That’s why I use ZFS wherever I possibly can.

As I looked into this issue, I saw that ZFS repaired about 400KB of data. I thought, “well, that was unlucky” and just ignored it.

Then a week later, it happened again. Pretty soon, I noticed it happened every Sunday, and always to the same drive in my pool. It so happens that the highest I/O load on the machine happens on Sundays, because I have a cron job that runs zpool scrub on Sundays. This operation forces ZFS to read and verify the checksums on every block of data on the drive, and is a nice way to guard against unreadable sectors in rarely-used data.

I finally swapped out the drive, but to my frustration, the new drive now exhibited the same issue. The SATA protocol does include a CRC32 checksum, so it seemed (to me, at least) that the problem was unlikely to be a cable or chassis issue. I suspected motherboard.

It so happened I had a 9211-8i SAS card. I had purchased it off eBay awhile back when I built the server, but could never get it to see the drives. I wound up not filling it up with as many drives as planned, so the on-board SATA did the trick. Until now.

As I poked at the 9211-8i, noticing that even its configuration utility didn’t see any devices, I finally started wondering if the SAS/SATA breakout cables were a problem. And sure enough – I realized I had a “reverse” cable and needed a “forward” one. $14 later, I had the correct cable and things are working properly now.

One other note: RAM errors can sometimes cause issues like this, but this system uses ECC DRAM and the errors would be unlikely to always manifest themselves on a particular drive.

So over the course of this, had I not been using ZFS, I would have had several megabytes of reads with undetected errors. Thanks to using ZFS, I know my data integrity is still good.

What is happening to America?

Politics

I still remember vividly my first visit to Europe, back in 2010. I had just barely gotten off a plane in Hamburg and on to a bus to Lubeck, and struck up a conversation with a friendly, well-educated German classical musician next to me. We soon started to discuss politics and religion. Over the course of the conversation, in response to his questions, I explained I had twice voted against George W. Bush, that I opposed the war in Iraq for many reasons, that I did thought there was an ethical imperative to work to defeat climate change, that I viewed health care as an important ethical and religious issue, that I thought evolution was well-established, and that I am a Christian.

Finally, without any hint of insult intended, and rather a lot of surprise written all over his face, he said:

“Wow. You’re an American, and a Christian, and you’re so…. normal!”

This, it seems to me, has a lot to do with Trump.

Ouch

It felt like a punch to the gut. The day after the election, having known that a man that appeared to stand for everything that honorable people are against won the election, like people all around the world, I was trying to make sense of “how could this happen?” As I’ve watched since, as he stacks government with wealthy cronies with records nearly as colorful as his own, it is easy to feel even more depressed.

Based on how Trump spoke and acted, it would be easy to conclude that the “deplorables” won the day – that he was elected by a contingent of sexists or racists ascendent in power.

But that would be too simple an explanation. This is, after all, the same country that elected Barack Obama twice. There are a many people that voted twice for a black man, and then for Trump. Why? Racism, while doubtless a factor, can’t explain it all.

How Trump could happen

Russ Allbery made some excellent points recently:

[Many Americans are] hurt, and they’re scared, and they feel like a lot of the United States just slammed the door in their faces.”

The status quo is not working for people.

Technocratic government by political elites is not working for people. Business as usual is not working for people. Minor tweaks to increasingly arcane systems is not working for people. People are feeling lost in bureaucracy, disaffected by elections that do not present a clear alternate vision, and depressed by a slow slide into increasingly dismal circumstances.

Government is not doing what we want it to do for us. And people are getting left behind. The left in the United States (of which I’m part) has for many years been very concerned about the way blacks and other racial minorities are systematically pushed to the margins of our economy, and how women are pushed out of leadership roles. Those problems are real. But the loss of jobs in the industrial heartland, the inability of a white, rural, working-class man to support his family the way his father supported him, the collapse of once-vibrant communities into poverty and despair: those problems are real too.

The status quo is not working for anyone except for a few lucky, highly-educated people on the coasts. People, honestly, like me, and like many of the other (primarily white and male) people who work in tech. We are one of the few beneficiaries of a system that is failing the vast majority of people in this country.

Russ is, of course, right. The Democrats have been either complicit in policies damaging to many, or ineffective in preventing them. They have often appeared unconcerned with the plight of people outside cities (even if that wasn’t really the case). And it goes deeper.

When’s the last time you visited Kansas?

I live in Kansas. The nearest paved road is about a 3-mile drive from my home. The nearest town, population 600, is a 6-mile drive. My governor — whom I did not vote for — cut taxes on the wealthy so much that our excellent local schools have been struggling for years. But my community is amazing, full of loving and caring people, the sort of people who you know you’ll be living with for 40 years, and so you make sure you get along well with.

I have visited tourist sites in Berlin, enjoyed an opera and a Broadway show in New York, taken a train across the country to Portland, explored San Francisco. I’ve enjoyed all of them. Many rural people do get out and experience the world.

I have been in so many conversations where I try to explain where I live to people that simply cannot fathom it. I have explained how the 18 acres I own is a very small amount where I am. How, yes, I do actually have electricity and Internet. How a bad traffic day is one where I have to wait for three cars to go past before turning onto the paved road. How I occasionally find a bull in my front yard, how I can walk a quarter mile and be at the creek on the edge of my property, how I can get to an airport faster than most New Yorkers and my kids can walk out the front door and play in a spot more peaceful than Central Park, and how all this is way cheaper than a studio apartment in a bad part of San Francisco.

It is rare indeed to see visitors actually traveling to Kansas as a destination. People have no concept of the fact that my mechanic would drop everything and help me get my broken-down car to the shop for no charge, that any number of neighbors or uncles would bring a tractor and come plow the snow off my 1/4-mile driveway out of sheer kindness, that people around here really care for each other in a way you don’t see in a city.

There are people that I know see politics way differently than me, but I know them to be good people. They would also do anything for a person in need, no matter who they are. I may find the people that they vote for to be repugnant, but I cannot say “I’ve looked this person in the eyes and they are nothing but deplorable.”

And so, people in rural areas feel misunderstood. And they are right.

Some perspectives on Trump

As I’ve said, I do find Trump to be deplorable, but not everyone that voted for him is. How, then, do people wind up voting for him?

The New Yorker had an excellent story about a man named Mark Frisbie, owner of a welding and fab shop. The recession had been hard on his business. His wife’s day-care center also closed. Health care was hard to find, and the long, slow decline had spanned politicians of every stripe. Mark and his wife supposedly did everything they were supposed to: they worked hard, were honest, were entrepreneurial, and yet — he had lost his business, his family house, his health coverage, everything. He doesn’t want a handout. He wants to be able to earn a living. Asked who he’d vote for, he said, “Is ‘none of the above’ an option?”

The Washington Post had another insightful article, about a professor from Madison, WI interviewing people in rural areas. She said people would often say: “All the decisions are made in Madison and Milwaukee and nobody’s listening to us. Nobody’s paying attention, nobody’s coming out here and asking us what we think. Decisions are made in the cities, and we have to abide by them.” She pushed back, hard, on the idea that Trump supporters are ignorant, and added that liberals that push that line of thinking are only making the problem worse.

I would agree; seeing all the talk about universities dis-inviting speakers that don’t hew to certain political views doesn’t help either.

A related article talks about the lack of empathy for Trump voters.

And then we have a more recent CNN article: Where Tump support and Obamacare use soar together, explaining in great detail how it can be logical for someone to be on Obamacare but not like it. We can all argue that the Republicans may have as much to do with that as anything, but the problem exists.

And finally, a US News article makes this point:

“His supporters realize he’s a joke. They do not care. They know he’s authoritarian, nationalist, almost un-American, and they love him anyway, because he disrupts a broken political process and beats establishment candidates who’ve long ignored their interests.

When you’re earning $32,000 a year and haven’t had a decent vacation in over a decade, it doesn’t matter who Trump appoints to the U.N., or if he poisons America’s standing in the world, you just want to win again, whoever the victim, whatever the price.

…

According to the Republican Party, the biggest threat to rural America was Islamic terrorism. According to the Democratic Party it was gun violence. In reality it was prescription drug abuse and neither party noticed until it was too late.”

Are we leaving people out?

All this reminded me of reading about Donald Knuth, the famous computer scientist and something of the father of modern computing, writing about his feelings of trepidation about sharing with his university colleagues that he was working on a project related to the Bible. I am concerned about the complaints about “the PC culture”, because I think it is good that people aren’t making racist or anti-semitic jokes in public anymore. But, as some of these articles point out, in many circles, making fun of Christians and conservatives is still one of the accepted targets. Does that really help anything? (And as a Christian that is liberal, have all of you that aren’t Christians so quickly forgotten how churches like the Episcopals blazed the way for marriage equality many years ago already?)

But they don’t get a free pass

I have found a few things, however, absolutely scary. One was an article from December showing that Trump voters actually changed their views on Russia after Trump became the nominee. Another one from just today was a study on how people reacted when showed inauguration crowd photos.

NPR ran a story today as well, on how Trump is treating journalists like China does. Chilling stuff indeed.

Conclusion

So where does this leave us? Heading into uncertain times, for sure, but perhaps — just maybe — with a greater understanding of our neighbors.

Perhaps we will all be able to see past the rhetoric and polarization, and understand that there is something, well, normal about each other. Doing that is going to be the only way we can really take our country back.

Singing with Kids

Family

For four years now, we’ve had a tradition: I go up to the attic one night, make a lot of noise, and pretend to be Santa. The boys don’t think Santa is real, but they get a huge kick out of this anyway.

The other day, this wound up with me singing a duet with my 7-year-old Oliver, and seeing a hugely delighted 10-year-old Jacob.

All last week, the boys had been lobbying for me to “be Santa”. They aren’t going to be able to be here on Christmas day this year, so I thought – why not let them have some fun. I chose one present to give them early too.

So, Saturday night, I said they could get ready for Santa. They found some cookies somewhere, got out some milk. And Oliver wrote this wonderful note to “Santa”:

IMG_20161217_204244_cropped

That is a note I’m going to keep for a long time. He helpfully drew arrows pointing to the milk, cookies, and even the pen. He even started Santa’s reply at the bottom!

So, Saturday night, I snuck up to the attic, pretended to be Santa, and ate some cookies, drank some milk, and wrote Oliver a note. And I left a present.

Jacob has been really getting into music lately, and Laura suggested I find something for the boys. I went looking for something that could record also, and came up with what has got to be a kid’s dream: a karaoke machine.

The particular one I found came with two microphones, a CD player, audio recording onto SD card (though it’s a little dodgy), and a screen for showing words on any music that’s karaoke-enhanced.

Cue gasps of awe and excitement from the boys when we came down in our PJs and sweats at 6:45 Sunday morning to check it out.

IMG_8895

Jacob excitedly began exploring all the knobs and options on it (they were particularly fond of the echo feature), while Oliver wanted to sing. So we found one of his favorite Christmas songs, and here he is singing it with me.

IMG_8908

When you have a system with a line in, line out, and several microphone jacks, you can get creative. With a few bits of adapters from my attic, the headset I use for amateur radio worked with it perfectly. Add on a little mic extension cord, and pretty soon Oliver was pretending to be an announcer for a football game!

IMG_8919

Then, Oliver decided he would act out a football game while Jacob and I were the announcers.

Something tells me there will be much fun had with this over the next while!

Just wait until I show them how to hook up a handheld radio to it in order to make a remotely-activated loudspeaker…

Giant Concrete Arrows, Old Maps, and Fascinated Kids

Aviation, Family

Let me set a scene for you. Two children, ages 7 and 10, are jostling for position. There’s a little pushing and shoving to get the best view.

This is pretty typical for siblings this age. But what, you may wonder, are they trying to see? A TV? Video game?

No. Jacob and Oliver were in a library, trying to see a 98-year-old map of the property owners in Township 23, range 1 East, Harvey County, Kansas. And they were super excited about it, somewhat to the astonishment of the research librarian, who I am sure is more used to children jostling for position over the DVDs in the youth section than poring over maps in the non-circulating historical archives!

All this started with giant concrete arrows in the middle of nowhere.

Nearly a century ago, the US government installed a series of arrows on the ground in Kansas. These were part of a primitive air navigation system that led to the first transcontinental airmail service.

Every so often, people stumble upon these abandoned arrows and there is a big discussion online. Even Snopes has had to verify their authenticity (verdict: true). Entire websites exist to tracking and locating the remnants of these arrows. And as one of the early air mail routes went through Kansas, every so often people find these arrows around here.

I got the idea that it would be fun to replicate a journey along the old routes. Maybe I’d spot a few old arrows and such. So I started collecting old maps: a Contract Airmail Route #34 (CAM 34) map from 1927, aviation sectionals from 1933 and 1946, etc.

I noticed an odd thing on these maps: the Newton, KS airport was on the other side of the city from its present location, sometimes even several miles outside the city. What was going on?

1927 Airway Map
(1927 Airway Map)

1946 Wichita Sectional
(1946 Wichita sectional)

So one foggy morning, I explained my puzzlement to the boys. I highlighted all the mysteries: were these maps correct? Were there really two Newton airports at one time? How many airports were there, and where were they? Why did they move? What was the story behind them?

And I offered them the chance to be history detectives with me. And oh my goodness, were they ever excited! We had some information from a very helpful person at the Harvey County Historical Museum (thanks Kris!) So we suspected one airport at least was established in 1927. We also had a description of its location, though given in terms of township maps.

So the boys and I made the short drive over to the museum. We reviewed their property maps, though they were all a little older than the time period we needed. We looked through books and at pictures. Oliver pored over a railroad map of Newton from a century ago, fascinated. Jacob was excited to discover on one map that there used to be a train track down the middle of Main Street! I was interested that the present Newton Airport was once known as Wirt Field, rather to my surprise. I somehow suspect most 2nd and 4th graders spend a lot less excited time on their research floor!

Then on to the Newton Public Library to see if they’d have anything more — and that’s when the map that produced all the excitement came out.

It, by itself, didn’t answer the question, but by piecing together a number of pieces of information — newspaper stories, information from the museum, and the maps — we were able to come up with a pretty good explanation, much to their excitement.

Apparently, a man named Tangeman owned a golf course (the “golf links” according to the paper), and around 1927 the city of Newton purchased it, because of all the planes that were landing there. They turned it into a real airport. Later, they bought land east of the city and moved the airport there. However, during World War II, the Navy took over that location, so they built a third airport a few miles west of the city — but moved back to the current east location after the Navy returned that field to them.

Of course, a project like this just opens up all sorts of extra questions: why isn’t it called Wirt Field anymore? What’s the story of Frank Wirt? What led the Navy to take over Newton’s airport? Why did planes start landing on the golf course? Where precisely was the west airport located? How long was it there? (I found an aerial photo from 1956 that looks like it may have a plane in that general area, but it seems later than I’d have expected)

So now I have the boys interested in going to the courthouse with me to research the property records out there. Jacob is continually astounded that we are discovering things that aren’t in Wikipedia, and also excited that he could be the one to add them. To be continued, apparently!