Administering Dozens of Debian Servers

Debian, ,

At work, we have quite a few Debian servers. We have a few physical machines, then a number of virtual machines running under Xen. These servers are split up mainly along task-oriented lines: DNS server, LDAP server, file server, print server, mail server, several web app servers, ERP system, and the like.

In the past, we had fewer virtual instances and combined more services into a single OS install. This led to some difficulties, especially with upgrades. If we wanted to upgrade the OS for, say, the file server, we’d have to upgrade the web apps and test them along with it at the same time. This was not a terribly sustainable approach, hence the heavier reliance on smaller virtual environments.

All these virtual environments have led to their own issues. One of them is getting security patches installed. At present, that’s a mainly manual task. In the past, I used cron-apt a bit, but it seemed to be rather fragile. I’m wondering what people are using to get security updates onto servers in an automated fashion these days.

The other issue is managing the configuration of these things. We have some bits of configuration that are pretty similar between servers — the mail system setup, for instance. Most of them are just simple SMTP clients that need to be able to send out cron reports and the like. We had tried using cfengine2 for this, but it didn’t work out well. I don’t know if it was our approach or not, but we found that hacking cfengine2 after making changes on systems was too time-consuming, and so that task slipped and eventually cfengine2 wasn’t doing what it should anymore. And that even with taking advantage of it being able to do things like put the local hostname in the right places.

I’ve thought a bit about perhaps providing some locally-built packages that establish these config files, or load them up with our defaults. That approach has worked out well for me before, though it also means that pushing out changes isn’t a simple hack of a config file somewhere anymore.

It seems like a lot of the cfengine2/bcfg tools are designed for environments where servers are more homogenous than ours. bcfg2, in particular, goes down that road; it makes it difficult to be able to log on to a web server, apt-get install a few PHP modules that we need for a random app, and just proceed.

Any suggestions?

Why Do Web Applications Stink So Badly?

Programming,

So today, I happen to be looking at wikis for two small to mid-sized public proojects (MoinMoin and DokuWiki look like frontrunners right now — any suggestions?) Recently, I’ve also looked at blog and CMS software, and a host of other web apps. It’s as if these people have learned nothing about good software practices over the last 20 years.

Warning: Rant Ahead

So how many of you have been here before? You download WebApp X. It tells you to cd to your DocumentRoot and unzip/untar it there. At this point, most of them will tell you to chmod -R 777 the install directory. Some of the better ones, such as WordPress, will tell you to chmod it 777, or if that makes you nervous, to instead chown it to the user that your webserver runs as.

It is at this point that you realize that the Java-based programs ship with their own webserver that takes 2 minutes to load and uses 2GB of RAM, while the PHP-based programs want you to give them 32MB RAM per process, and probably modify your global PHP settings in a way that breaks some other PHP web app you’re already using.

As if that isn’t enough to scare you off, generally speaking, config files — including passwords to databases — are stored in the same directory, along with .htaccess files. Many of these programs are also downloading and updating plugins over the Internet, usually without any kind of cryptographic authentication, and overwriting their own program files in the process.

Oh, and this is a class of app that is notorious for security problems to start with, and makes your server known to billions of people via search engines.

Absolutely no opportunity for trouble here, of course! That sentence was dripping with sarcasm, in case you didn’t get it.

It also makes it almost impossible for people such as Debian maintainers to package up some webapps (such as just about every single one that uses Ruby on Rails) because there is just no sane way to make it behave with respect to the Filesystem Hierarchy Standard.

I’d love to see web app developers do a few simple things:

  1. Separate code from data
  2. Separate code from configuration
  3. Separate all of the above from the DocumentRoot to the greatest extent possible

I realize that some of this is purportedly to make things easier to install when you have FTP access only. But to me it seems just really poor design. I’ve written webapps, and it’s not that hard to do this part right.

Plus, doing the above right means that I no longer have to do something like use git on my WordPress installations because it’s too much of a hassle to apply security and plugin updates on all three separate ones otherwise.

If Programming Languages Were Christmas Carols

Humor, Programming, , , , , ,

Last spring, I posted If Version Contol Systems Were Airlines, which I really enjoyed. Now, because I seem to have a desire to take a good joke way too far, it’s time for:

IF PROGRAMMING LANGUAGES WERE CHRISTMAS CAROLS

I apologize in advance. (Feel free to add your own verses/carols in the comments.)

Away in a Pointer (C)

(to Away in a Manger)

Away in a pointer, the bits in a row.
A little dereference to see where they go.
I look down upon thee, and what do I see?
A segfault and core dump, right there just for me.

I saw thy init there, a reaping away
My process, from its address space, so sorry to say.
I thought I had saved thee, from void pointers all,
But maybe I missed one, and doomed you to fall.

Be near me, debugger, I ask thee to stay
Close by my terminal, and help me, I pray;
To find all the bugs and the void pointers too,
And if my kernel oopses, help me reboot for you.

Joy to the Wall (Perl)

(to Joy to the World)

Joy to the Wall, the Perl is come!
Let awk receive her King;
Let every grep prepare him room,
And bash and sed shall sing,
And bash and sed shall sing,
And bash, and bash, and sed shall sing.

Joy to the keyboard, we’ll use it all!
Let men, shift keys, employ;
Implicit variables, and globals never fall.
Repeat the line noise now,
Repeat the line noise now,
Repeat, repeat, the line noise now.

Perl rules the world with truth and ASCII,
And makes the doctors prove
The glories of carpal tunnel hands,
And we do it more than one way,
And we do it more than one way,
And we do it, and we do it, more than one way.

Hark! The Herald Coders Sing (Haskell)

(to Hark! The Herald Angels Sing)

Hark! The herald coders sing,
“Map and fold, recursive King;
Recursion and patterns wild,
Pure and IO — they’re reconciled!”
Joyful, all ye functions rise,
Join the typeclasses of the types,
With recursion, do proclaim,
“Laziness is born in this domain.”

Refrain
Hark! The herald coders sing,
“Map and fold, recursive king!”

Monads, by highest Heav’n adored;
Monads, their depths still unexplored;
Late in time, behold they’re good,
Never once were understood.
Veiled in functions, the Monads stay,
Used for IO, and more, each day,
With excitement, Monads say,
“Arrows are stranger, so with us stay.”

(Refrain)

Hail the glorious compiler of Glasgow!
Hail the threaded run-time system!
Join the beautiful Cabal of Hackage,
Upload there thy perfect package.
We know best, what we will Handle,
You’re safe with us: no pointers, no vandals.
Born to make your exceptions throw,
Unless you unsafePerformIO.

(Refrain)

Lispy the Paren

(to Frosty the Snowman)

Lispy the paren was a jolly happy soul,
With a lot of cars and a little cons
And two ends made out of curves.
Lispy the paren is a fairy tale, they say,
He was just common, but the children know
how he came to life one day.
There must have been some magic in that
Old Symbolics they found.
For when they placed him on its disk,
It recursed around and ’round.

O, Lispy the paren,
Was recursive as can be.
And the coders say it would take a day
To put his parens away.
Clunkety clunk clunk,
Clunkety clunk clunk,
Look at Lispy go.
Clunkety clunk clunk,
Clunkety clunk clunk,
Consing on the car.

Lispy the snowman knew
The keyboard was hot the day,
So he said, “Let’s cons and we’ll have some fun
now before they Scheme away.”
Down to the function,
With a list there in his RAM,
Running here and there,
all around the LAN, saying
“cdr me if you can.”
He led them down the streets of disk
Right to the traffic bus.
And only paused a moment when
He heard them holler (quit).

Oh BASIC Night

(to O Holy Night)

Oh BASIC night, the LEDs are brightly glinting;
It is the night of the dear GOSUB’s birth!
Long lay the world in sin and error printing,
Till you appeared and the RAM felt its worth.
Shiver of fear, line numbers do inspire,
For yonder breaks a mostly harmless GOTO.
Fall on your bits, O hear the Visual voices!
O BASIC divine, O BASIC where GOTO was born!
O BASIC, O Holy BASIC, O BASIC, you’re mine!

Some want to say, “GOTO is harmful always,”
But what of them, in their post-modern world.
We PRINT the truth, in the line-numbered goodness,
But Dijkstra appeared, and the faith, it was lost.
A thrill of hope, when .NET BASIC announces,
But Visual BASIC, what kind of thing are you?
Fall on your GUI, O see the old line numbers!
Behold BASICA, O BASIC when DOS was born!
O numbers, O lines, spaghetti divine!

Guido We Have Heard on High (Python)

(to Angels We Have Heard on High)

Guido we have hard on high
Sweetly indenting o’re the code,
And the functions in reply
Their exceptions sweetly flowed.

Refrain

Indent….. in your whitespace careful!
Indent…… in your whitespace careful!

Spaces, why this jubilee?
Why semicolons have you so wronged?
What backslashes must we use
If we want our lines so long?

(Refrain)

Come to Guido here to see
“One Right Way” is good, of course.
There’s no need for Perl, you know,
We have to be more verbose.

(Refrain)

Now the PEP will show the way
To the future, we shall see.
Banish lambda and the rest
Of the things we liked the best.

(Refrain)

The Demise of PC Magazine

Software, ,

I just read the news that PC Magazine is being canceled. It’s not exactly a shock, given the state of technical magazines right now. I haven’t read one of those in years, since they turned to be more of a consumer than a technical publication.

But I hope I am not the only one out there that remembers PC Magazine from the mid to late 1980s. I had two favorite parts in each issue: the programming example, and the “Abort, Retry, Fail” page at the back of the magazine.

The programming example was usually some sort of DOS (or, on occasion, OS/2) utility. It was usually written in assembly, and would be accompanied by a BASIC program you could type in to get the resulting binary, as assemblers weren’t readily available. The BASIC program was line after line of decimal numbers that would decode them and write out the resulting binary — sort of a primitive uuencode for paper. Trying to type those in gave me some serious eyestrain on more than one occasion. By now, I forget what most of those utilities did, but I remember one: BatchMan. It was a collection of tools for use in DOS batch files, and could do things like display output in color or even — yes — play monophonic music. It came with an example that displayed some lyrics about batch programming on-screen, set to what I later realized was the Batman theme. Geek nirvana, right?

But Batchman was too big to publish the source code, or the BASIC decoder, in print. It might have been one of those things that eventually led me to a CompuServe account. PC Magazine had some deal with CompuServe that you could get their utilities for free, or reduced cost — I forget. CompuServe was probably where I sent my first email, from my account which was 71510,1421 — comma and all. In later years, you could pay a small fee to send email to the Internet, and I had the amazingly attractive email address of 71510.1421@cis.compuserve.com. Take that, gmail.

PC Magazine eventually stopped running utilities that taught people about assembly or batch programming and shifted more to the genre of Windows screensavers. They stopped their articles about how hard disks work and what SCSI is all about, and instead have cover stories like “Vista made easy!” I am, sadly, not making this up. Gone are the days of investigating alternative operating systems like OS/2.

It appears that “Abort, Retry, Fail” is gone, too. It was a one-page thing at the back of each magazine that featured braindead error messages and funny stories about people that did things like FAX an image of a floppy disk to a remote office — before such stories were cliche. Sort of like DailyWTF these days. The sad truth is that the people that would FAX an image of a floppy are probably the ones that are reading PC Magazine today.

I still have a bunch of PC Magazine issues — the good ones — in my parents’ basement. I also still have my floppies with the utilities on them somewhere. One day, when I get some time — I’m estimating this will be about when Jacob goes to college — I’ll go back and take another look at them.

Jacob Update

Family,

Let’s start with a photo:

img_5563r.jpg

That’s Jacob over at the pumpkin patch near us. He found something to inspect, and spent awhile doing it. As he does.

He’s taken a liking to our cat, Nash. Jacob calls him “cat Nash”. Never just “Nash”. When we get home from somewhere, if the cat is around, Jacob will say, “Hi cat Nash! Hi cat Nash!” Then he’ll bend over, touch his head to Nash’s back, and try to give him a hug. Nash, surprisingly, doesn’t mind this.

Jacob enjoys being a part of — well, everything. He will repeat back new words and phrases, trying to learn how to say them, even if he doesn’t understand what they mean yet. His favorite recent outdoor discovery is that grain silos are all over the place. He’ll point them out excitedly as we drive down the road. I had never noticed just how many there are.

One day, he pointed at a water tower and said “SILO!” I understood why he said that, but I told him it was a water tower. He remembered that, and learned to tell them apart in a day or two. Then one morning he surprised me with, “Water tower. Water inside.” How he figured that out, I don’t know.

img_5446r.jpg

There’s another photo of him at the pumpkin patch.

The other day, I accidentally triggered our smoke alarms while checking one for a battery. After that, Jacob loved to say “BEEP! BEEP!” Sometimes followed by “Smoke larm. Hurt ears.” We learned how to say BEEP BEEP loud and also quiet.

He’s certainly a lot of fun at this age.

Real World Haskell Update

Programming,

Times are exciting. Our book, Real World Haskell, is now available in a number of venues. But before I get to that, I’ve got to talk about what a thrill this project has been.

I created our internal Darcs repository in May, 2007. Since then, the three of us has made 1324 commits — and that doesn’t count work done by copyeditors and others at O’Reilly.

We made available early drafts of the book online for commenting, which served as our tech review process. By the time we finished writing the book, about 800 people had submitted over 7,500 comments. I’ve never seen anything like it, and really appreciate all those that commented about it.

As for availability, RWH is available:

  • For immediate purchase with electronic delivery, from O’Reilly’s page
  • For immediate viewing on Safari Books Online, at its book page
  • Paper editing timing is still tentative, but we’re estimating arrival in bookstores the week of December 8.

People are talking about it on blogs, twitter, etc. We’re excited!

Frozen Bicycling

Outdoors

Some of you might recall that I’ve been bicycling to work, about 10 miles each way.

Over the last two weeks, I haven’t been able to ride much because it’s been too muddy. Today I rode to work.

It was about 25F-30F out there, so this was my first below-freezing bicycle ride. It went OK, though I was somewhat on the cool side — I’ll add more layers next time.

Today, I wore wool socks, bicycling shorts, tights over that, my short sleeve shirt, a long-sleeve shirt over it, full gloves, and a balaclava. I should have worn probably one more layer everywhere, but I survived and I’m not frozen.

You may now commence speculation about whether or not I am crazy.

Web Design Companies That Understand Technology

Online Life, Software

There are a lot of companies out there that do web design work that looks fabulous.

Unfortunately, a lot of these sites look fabulous only when viewed in IE6 build xxxx, with a 75dpi monitor, fonts set to the expected size, running on Windows XP SP2, with JavaScript enabled. Try looking at the site through Safari, Firefox, with larger-than-expected fonts, and things break down: text boxes overlap each other, buttons that should work don’t, and it becomes a mess.

So, if your employer wanted a web design company that has a good grasp of Web standards and the appropriate use of them, where would you look? A company that can write good HTML, CSS, and JavaScript, and still make the site look appealing? A company that has heard of Apache and gets the appropriate nausea when someone mentions ColdFusion or Frontpage?

So far, I’ve seen these places mentioned by others:

WebDevStudios.com
Happy Cog
Crowd Favorite

Converted to WordPress

Online Life, Software, ,

I have been using Serendipity on my blog for some time now. Overall, I’ve been pleased with it, but the conversion was a pain.

Serendipity is a simple blog engine, and has a wonderful built-in plugin system. It can detect what plugins need upgrading, and install those upgrades, all from directly within the management interface. There’s no unzipping stuff in install directories as with WordPress.
Continue reading Converted to WordPress →