Comments on: Administering Dozens of Debian Servers

By: emag

emag — Wed, 04 Mar 2009 14:58:35 +0000

I was wondering if you’ve found a solution yet? I was talking with someone yesterday evening about config management, primarily for 40+ Debian boxes (primarily VMs), but with some RHEL 3-5, SLES, OpenSuSE, Solaris 7-10, and AIX boxes thrown in for flavoring, and they directed me here, saying “john is smart so i’d do whatever he does”.

I’ve been contemplating puppet, cfengine, and/or bcfg2, but haven’t moved past the navel-gazing aspects yet. I do know that the manual process and local “meta” packages for the Debian side is untenable going forward, as it’s already getting difficult to deal with updates and especially the etch -> lenny transition. A lot of the machines are web stuff, but like you, several have one-off packages needed for specific apps.

So have you come to any decisions yet, or is it still a matter of nothing quite meeting your needs?

By: Greg

Greg — Wed, 17 Dec 2008 11:01:40 +0000

For monitoring purpose you may also be interested by monit, munin and atsar.

By: jlouis

jlouis — Wed, 17 Dec 2008 10:28:35 +0000

Puppet is the way I would go as well. It is pretty simple to set up and get going. After that it is simply a matter of taking one service at a time and making it standard.

By: Nico

Nico — Wed, 17 Dec 2008 07:45:23 +0000

I’m running along with puppet with configuration management & capistrano for running commands on hosts. It may sound funny to use capistrano to do some system administration but it does its job. Grab some keyboardcast fun to, when a few machines are involved ;)

By: Jan Dittberner

Jan Dittberner — Tue, 16 Dec 2008 22:09:26 +0000

We use some Debian machines (virtual and physical) and use apticron for updates and a svn repository with shared configuration files and etckeeper for local changes. Some small manual work is needed from time to time but we are satisfied with the process.

By: Narayan Desai

Narayan Desai — Tue, 16 Dec 2008 20:58:44 +0000

Bcfg2 is actually designed for non-homogeneous environments, but this has caused some of the processes to be somewhat heavyweight. One of the things that has changed in bcfg2 since the discussion last year is that bcfg2 has gotten a lot more capable at pulling changes from clients. This allows you to perform manual administration on clients, and let bcfg2 detect the changes. Then you can associate those configuration changes with the appropriate client or sets of clients.

By: Josip Rodin

Josip Rodin — Tue, 16 Dec 2008 20:35:53 +0000

Maybe it’s obvious and redundant, but I have to say – cssh.

By: Markus Hochholdinger

Markus Hochholdinger — Tue, 16 Dec 2008 19:14:52 +0000

Here’s my solution:
* One Admin-Server, especially secured, with a ssh-key with no passphrase (seems most of you have this setup already).
* List of managed Servers on this Admin-Server.
* script: Every 5 minutes (cron), ssh (in parallel) to all servers in the list, execute a few commands (df, free, uptime, ..) and make a few remote checks (ping, http). If something changes, report this in ONE email for all servers.
* script: Once a day (cron), ssh to alle servers (sequential ) and perform a apt-get update && apt-get -d upgrade, parse output, and only if updates are waiting report this in ONE email for all servers. Also execute a few other checks like getting the debian version, the location of the server, if its on hardware or a virtual one and so on. The results are saved in the server list.

Script for manually updating all servers:
For each server in the server list, do a “ssh -t $SERVER $UPDATE”, where $UPDATE is apt-get update, aptitude update or aptitude safe-upgrade depending on the debian version. With “ssh -t $SERVER” you can perform interactive things on each server in a row.

To manage the configuration or installation of special things first i document them in a format i can make a script of. Something like:
aptitude install xyz
CONFIG=/etc/xyz.conf
test -f $CONFIG.original || cp -a $CONFIG $CONFIG.original
echo -e “1c1
OPTION=on” | patch $CONFIG

If i need this configuration frequently, i put this in a script which i can execute remotely in the form:
scp $PATHTO$SCRIPT $SERVER:/tmp/ && ssh -t $SERVER /tmp/$SCRIPT

With this method i have full control over heterogeneous installations while doing a lot automated but doing changes only while i’m sitting in front of.

By: Greg

Greg — Tue, 16 Dec 2008 18:41:44 +0000

Hi,

It depends on what you want to do. You can automate things with puppet (written in ruby) or bcfg2 (in python), or you can do some manual stuff with python pexpect (and pxssh), and cssh( cluster ssh).

You might also want to track the configuration files with a scm like git.

By: John Goerzen

John Goerzen — Tue, 16 Dec 2008 18:26:13 +0000

I should add that people on IRC have also suggested:

http://www.eyrie.org/~eagle/software/bundle/

cluster ssh

http://rsug.itd.umich.edu/software/radmind/