Category Archives: Software

Whose Distributed VCS Is The Most Distributed?

Software,

Lately I have been trying out a number of distributed version control systems (VCS or SCM).

One of my tests was a real problem: I wanted to track the Linux 2.6.16.x kernel tree, apply the Xen patches to it, and pull only specific patches (for the qla2xxx driver) from 2.6.17.x into this local branch. I wanted also to be able to upgrade to 2.6.17.x later (once Xen supports it) and have the version control system properly track which patches I already have.

But before going on, let’s establish what it means to be an ideal distributed VCS:

  • 1. The fundamental method of collaboration must be a branch. A checkout should mean creating a local branch on which a person can commit and work without having to involve the server. An update from some central server should take the form of a merge from that branch to the local branch.
  • 2. Branching should be cheap. It should be easy to create a local branch, the operation to do so should be fast, and it shouldn’t take an inordinate amount of space. It should also be as easy as possible to branch from a remote repository.
  • 3. Merging between branches is intelligent. It should be easy to merge another branch with your own. The VCS should know which changesets from the other branch are already on yours, and should not attempt to merge changesets that you have already merged previously.
  • 4. Inividial changesets should be mergeable without bringing across the whole history. You should be able to bring across the minimum number of changesets necessary to effect a specific change. This corresponds to my test case above. Future merges from the whole branch should, of course, recognize that these changesets are present already.
  • 5. Branching preserves full history. A branch should be a first-class copy of a repository, even if the repository is remote. It should contain the full history of the branch it was made from, including diffs for each individual changeset and full commit logs, unless otherwise requested by the user.
  • 6. Merging preserves full history. A merge from one branch to another should also preserve full history. Changesets merged to the local branch should retain the individual, distinct diffs and commit logs for each changeset.

There are also some things that we would generally want:

  • 7. It is possible to commit, branch, merge, and work with history offline.
  • 8. The program is fast enough for general-purpose use.

Evaluation

Let’s look at some common VCSs against these criteria. I’ll talk about Arch (tla, baz, etc), bzr (bazaar-ng), Darcs, Git, Mercurial (hg), and Subversion (svn) for reference.

1. The fundamental method of collaboration must be a branch

All of the tools pass this test except for svn.

2. Branching should be cheap

Everyone except svn generally does this reasonably well.

The tla interface for Arch had a pretty terrible interface for this, so it took awhile simply due to all the typing involved. That’s better these days.

Darcs supports hardlinking of history to other local repositories and will do this automatically by default. Git also supports that, but defaults to not doing it, or you can store a path to look in for changesets that aren’t in the current repo. I believe Mercurial also can use hardlinks, though I didn’t personally verify that. bzr appears to have some features in this area, but not hardlinks, and the features were too complex (or poorly documented) to learn about quickly.

svn does not support branching across repositories, so doesn’t really pass this test. Branches within a repository are not directly supported either, but are conventionally simulated by doing a low-cost copy into a specially-named area in the repository.

3. Merging between branches is intelligent

Arch was one of the early ones to work on this problem. It works reasonably well in most situations, but breaks in spectacular and unintelligble ways in some other situations.

When asked to merge one branch to another, Darcs will simply merge in any patches from the source branch onto the destination which the destination doesn’t already have. This goes farther than any of the other systems, which generally store a “head” pointer for each branch that shows how far you’ve gone. (Arch is closer to darcs here, though ironically bzr is more like the other systems)

Merging between branches in svn is really poor, and has no support for recognizing changesets that have been applied both places, resulting in conflicts in many development models.

4. Inividial changesets should be mergeable without bringing across the whole history

Darcs is really the only one that can do this right. I was really surprised that nobody else could, since it is such a useful and vital feature for me.

Both bzr and git have a cherry-pick mode that simulates this, but really these commands just get a diff from the specific changeset requested, then apply the diff as with patch. So you really get a different changeset committed, which can really complicate your history later — AND lead to potential conflicts in future merges. bzr works around some of the conflict problems because on a merge, it will silently ignore patches that attempt to perform an operation that has already occured. But that leads to even more confusing results, as the merge of the patch is recorded for a commit that didn’t actually merge it. (That could even be a commit that doesn’t modify the source.) Sounds like a nightmare for later.

Arch has some support for it, but in my experience, actually using this support tends to get it really confused when you do merges later.

Neither Mercurial nor svn have any support for this at all.

5. Branching preserves full history

git, darcs, and Mercurial get this right. Making a branch from one of these repos will give you full history, including individual diffs and commit logs for each changeset.

Arch and bzr preserve commit logs but not the individual changesets on a new branch. I was particularly surprised at this shortcoming with bzr, but sure enough, a standard bzr merge from a remote branch commited three original changesets into one and did not preserve the individual history on the one commit.

svn doesn’t support cross-repo branching at all.

6. Merging preserves full history

Again, darcs, git, and Mercurial get this right (I haven’t tested this in Mercurial, so I’m not 100% sure).

Arch and bzr have the same problem of preserving commit logs, but not individual changesets. A merge from one branch to another in Arch or bzr simply commits one big changeset on the target that represents all the changesets pulled in from the source. So you lose the distinctness of each individual changeset. This can result in the uncomfortable situation of being unable to re-create full history without access to dozens of repositories on the ‘net.

Subversion has no support for merging across repositories, and its support for merging across simulated local branches isn’t all that great, either.

7. It is possible to commit, branch, merge, and work with history offline

Everyone except Subversion does a good job of this.

8. The program is fast enough for general-purpose use

All tools here are probably fast enough for most people’s projects. Subversion can be annoying at times because many more svn commands hit the network than those from others.

In my experience, Arch was the slowest. Though it was still fine for most work, it really bogged down with the Linux kernel. bzr was next, somewhere between arch and darcs. bzr commands “felt” sluggish, but I haven’t used it enough to really see how it scales.

Darcs is the next. It used to be pretty slow, but has been improving rapidly since 1.0.0 was released. It now scales up to a kernel-sized project very well, and is quite usable and reasonably responsive for such a thing. The two main things that slow it down are very large files (10MB or above) and conflicts during a merge.

Mercurial and git appear to be fastest and pretty similar in performance.

All of these tools perform best with periodic manual (or scheduled cron jobs) intervention — once a month to once a year, depending on your project’s size. Arch users have typically created a new repo each year. Darcs users periodically tag things (if things are tagged as part of normal work, no extra work is needed here) and can create checkpoints to speed checkouts over the net. git and Mercurial also use a form of checkpoints. (not sure about bzr)

Subversion works so differently from the others that it’s hard to compare. (For one, a checkout doesn’t bring down any history.)

Conclusions

I was surprised by a few things.

First, that only one system actually got #4 (merging individual changesets) right. Second, that if you had to pick losers among VCSs, it seems to be Arch and bzr — the lack of history in branching and merging is a really big issue, and they don’t seem to have any compelling features that git, darcs, or Mercurial lack. #4 was a unique feature to Darcs a few years ago, but I figured it surely would have been cloned by all the other new VCS projects that have popped up since. It seems that people have realized it is important, and have added token workaround support for it, but not real working support.

On the other hand, it was interesting to see how VCS projects have copied from each other. Everyone (except tla) seems to use a command-line syntax similar to CVS. The influence of tla Arch is, of course, plainly visible in baz and bzr, but you can also see pieces of it in all the other projects. I was also interested to see the Darcs notion of patch dependencies was visible (albeit in a more limited fashion) in bzr, git, and Mercurial.

So, I will be staying with Darcs. It seems to really take the idea of distributed VCS and run with it. Nobody else seems to have quite gotten the merging thing right yet — and if you are going to make it difficult to do anything but merge everything up to point x from someone’s branch, I just don’t see how your tool is as useful as Darcs. But I am glad to see ideas from different projects percolating across and getting reused — this is certainly good for the community.

Updates / Corrections

I got an e-mail explaining how to get the individual patch diffs out of bzr. This will work only for “regular”, non-cherry-picked merges, and requires some manual effort.

You’ll need to run bzr log, and find the patch IDs (these are the long hex numbers on the “merged:” line) of the changeset you’re interested in, plus the changeset immediately before it on the same branch (which may not be on the same patch and may not be obvious at all on busy projects.) Then, run bzr diff -r revid:old-revid-string..new-revid-string.

I think this procedure really stinks, though, since it requires people to manually find previous commits from the same branch in the log.

First steps with git and I’m not all that pleased

Software,

I have been tracking 2.6.16.x here because Xen doesn’t have patches for 2.6.17 yet (why on earth that is, I don’t know.) But I need a few 2.6.17.x patches to the qla2xx driver. So I figure this is an opportunity to learn git.

I learned git, but then quickly learned that I can’t just pull random commits from one branch to another. I have to use git cherry-pick, which doesn’t actually pull the commit unmodified — it takes a diff, and commits a new patch based on that diff. So I expect problems later when I bump the local branch to 2.6.17.

This seems depressingly like arch/baz/bzr, and is a good reason for me to stay with darcs for now. This same operation with darcs would have been trivial, AND darcs would have automatically pulled in prerequisite patches rather than giving a merge failure and making me find them manually.

Thoughs on cfengine, bcfg2, and puppet

Software,

Yesterday I posted about my first steps with cfengine. By the end of the day today, I had things far along that I can:

  • cdebootstrap a directory
  • Run a special cfengine script to get the base files like /etc/fstab and /etc/hosts set up
  • Bring it up in Xen, apt-get install cfengine2, and use cfagent to bring up the rest of the system and install the necessary base packages (like xfsprogs)

Very nice.

I’ve had a few annoyances with the cfengine packages support, which doesn’t quite seem to work as documented al the time.

I also took a look at bcfg2 thanks to a comment yesterday. It looks very interesting, but I have a few gripes about it. I find cfengine files easier to read. I can look at a file, having never used cfengine before, and have a reasonable idea of what is trying to be done and how it will be accomplished. I can’t say the same for bcfg2, plus bcfg2 uses XML config files (ick) and a bunch of small otherfiles. While the architecture as the authors have described it certainly sounds appealing, I’m not sure that bcfg2 is a simple as cfengine. I am a strong believer in the KISS (Keep It Simple, Stupid) principle. But THANKS to the person that left the comment, and I hope that bcfg2 continues to evolve and provide an alternative to cfengine.

I also looked at Puppet. This thing looks very slick. Seems to be cfengine with a nicer syntax. On the other hand, it’s not really clear that anybody is using it. That makes me nervous — this is the kind of thing that can seriously harm machines if it does something unexpected.

First steps with cfengine

Software

This afternoon I started looking at cfengine. In very little time, I’ve already set up rules that can bring a system from pretty much cdebootstrap state up to a minimal production system in our environment. I’ve still got a little ways to go, but it’s already hacking on /etc/hosts, hosts.deny, sources.list, installing appropriate Debian packages for our systems, etc.

It’s come a long way since I last looked at it six years ago.

One thing I can’t figure out…

I have a /etc/bacula/bacula-fd.conf file that contains, among other things, this:

Director {
  Name = backup-dir
  Password = "foo"
}

Director {
  Name = backup-mon
  Password = "bar"
  Monitor = yes
}

I can’t figure out how to make cfengine delete just that second section. I tried this:

       BeginGroupIfLineMatching "  name = .+-mon"
         IncrementPointer "-1"
         DeleteToLineMatching "\}"
         DeleteNLines "1"
       EndGroup

But it seems that the pointer is never actually being decremented, even when examined under verbose mode. That is, it leaves the leading “Director {” line in the file.

RedHat Gripes

Linux

Lately we are looking at groupware options, and have been looking at Scalix and Zimbra. We may need the features in the proprietary versions of these products, unfortunately.

So I downloaded an evaluation copy of Scalix.

They say they support RedHat and SuSE. Fine, I think, I’ll just alien the RPMs to debs and be happy.

Not so fast. They have a whole proprietary install system. They check for /etc/redhat_release or /etc/SuSE_release (or something like that) and do different things depending on what is there. Ugh. Why can’t these proprietary vendors just target LSB? The differences seem mostly related to init anyway.

So I touch /etc/SuSE_release into existence, run the installer again. It complains that DISPLAY is not set. UGH. I log in with ssh forwarding, to root (sigh), and run it again.

Now it complains that the SuSE_release file doesn’t contain a valid release. I google a bit, but the file format doesn’t seem to be documented anywhere. I extract it from an RPM somewhere, but no luck.

So, I figure at this point, let’s try an actual RPM distro. I’m running this in a Xen domain anyway, so it should be no big deal, right?

I think CentOS will be a good choice. It’s RHEL with the non-free stuff stripped out. And they support RHEL and don’t need any non-free stuff. I google, and find instructions for installing via rpmstrap for Xen uses.

Let me say, rpmstrap is not nearly the nice tool that cdebootstrap is. rpmstrap totally hosed the networking on the Xen host machine, requiring me to reboot to get it back to proper state. The resulting install wouldn’t boot, either — I later found out that, even though I listed explicit devices in /etc/fstab like usual, it requires labels on all my partitions to boot. Ugh. There are a host of other problems with the rpmstrap-installed chroot, and it’s broken beyond my ability to repair due to problems with the rpm database.

So then I downloaded the “Server” CD for CentOS, which is supposed to have just the stuff a person would need for a server, and leave off all the graphical tools, multimedia, etc. I fired up VMware and did an install. Then I booted Debian From Scratch in VMware and used tar and netcat to copy the installed image over to Xen.

I got it booting fairly easily. But now I start to remember why I had this instinctive gag reflex last time I used RHEL.

First off, the network configuration, by default, is tied to the MAC address of your ethernet card. So if you replace your Ethernet card, your network is broken by default.

Then, there’s the way the network is brought up. It uses arping as part of its procedure to bring up a NIC. If it sees a reply anywhere on the network with the IP you’re trying to assign, it leaves the NIC half-up — it’s been ifconfig’d up, but without an IP. So that’s right, if somebody happens to have a rogue device plugged in at the moment your server boots, your server will come up without a network configured. This is *Enterprise* Linux and it’s pulling this sort of thing. Terrible design.

Next, there’s the way the network is *configured*. There are commands such as system-config-network-tui, -gui, -cmd, -druid, etc. I go for -tui. to start with. It’s a dialog-like interface, and asks the basics like IP address, etc. It doesn’t have any way to configure more than one Ethernet card that I can tell. And some of the settings — like nameserver — apparently require you to press F12 to visit. But the program doesn’t recognize F12 as sent by an xterm, so it doesn’t work.

All the other options require X. So, I reluctantly ssh -X into it as root and run system-config-network-gui. It doesn’t work — complains it can’t find DISPLAY. Strange, I think; DISPLAY is set properly to localhost:whatever. It turns out that /etc/hosts is empty by default, so the thing can’t resolve localhost! Argh. I add a line to /etc/hosts and it fires up.

This tool works decently. I save, uncheck the tie to a MAC address box, and exit. I then think it might be good to fire it up again and see what it did. I try running it again, and get the same error about DISPLAY. The stupid tool blew away /etc/hosts and replaced it with an empty file! This is NOT what I would expect from an Enterprise Linux. You don’t blow away a config file the administrator touched without asking, EVER.

Next, I figure, let’s try installing the XFS tools so I can switch the root filesystem to xfs. I start with “yum update”, which doesn’t quite do what I expect. (It is more like apt-get update && apt-get -u dist-upgrade) So I hit Ctrl-C, but — surprise — IT DOESN’T WORK. I press it a few more times, and it seems to just make the downloader cycle through mirrors because of a “download error”. So I hit Ctrl-Z and kill %1. I have my prompt, but it’s STILL DOWNLOADING STUFF and spewing all over my console. Ugh.

I finally use ps and kill -9 and eventually get it killed off. Stupid thing.

I don’t understand why anybody would want to use RedHat Enterprise Linux in an enterprise. It seems more suited to a hobbyist system at home. From reading some forums, it seems there are quite a few people out there using Debian for enterprise systems for similar reasons.

So now, maybe I’ll have the chance to actually try Scalix.

(BTW, our intern got Zimbra installed on Debian just fine, so that’s a plus for it.)

Multipath is working

Linux

Yesterday, we got multipath working with our HP MSA1500cs SAN. We have a fully redundant setup with redundant controllers, fibre channel switches, and two FC controllers per host.

We had been having a lot of trouble getting things to work right with active/passive controllers. We could get failover to work in some cases, but getting everything to communicate correctly in the event of a failure was difficult, since every machine would have to flip over to the passive controller simultaneously.

With a firmware upgrade, the MSA 1500cs can support active/active controllers. With the dual-active setup, both controllers are active simultaneously and both are valid paths.

Despite HP support’s indications to the contrary, HP does have information on using built-in multipathd in Linux instead of their proprietary multipath solution. It’s document c00635587, part AA-RW8RA-TE.

We’ve configured multipathd.conf like this:

      path_grouping_policy  multibus
      path_checker              tur
      failback                  immediate
      no_path_retry             60
      path_selector             "round-robin 0"

Just put that in your default block and it should work.

Announcing hpodder

Software, ,

Today I’m finally announcing hpodder.

I’ve been trying different podcatchers in Linux, and have been generally unhappy. ipodder looked nice at first, but turned out to be horribly buggy.

bashpodder/podracer looked like a nice idea. However, it didn’t have enough flexibility for me, its XML parser has some well-known failures (it’s not a real XML parser, after all), etc.

So I wrote hpodder. hpodder is a command-line podcast downloader for Linux. It features:

  • Extensive manual (installed as manpage, or you can view the PDF versoin). Documents all command-line options, the config file, a quick start, plus some basic information about the internal database
  • Database of seen URLs (in Sqlite3) — for use both for downloads and when processing feeds
  • Graceful handling of Ctrl-C, shutdowns, network troubles, etc — including ability to resume downloads later, plus the ability to detect servers that don’t handle download resuming properly (libsyn)
  • Automatic setting of ID3 tags based on the episode title and podcast title from the podcast’s feed (as iTunes does) — dramatically helps with viewing of all sorts of podcasts on the iPod and your PC
  • Support for download rate limits, progress bars, etc. via Curl
  • Seems to be stable for me
  • Command-line tools to: add new podcasts, remove podcasts, update podcast feed URLs, scan podcast feeds, list known podcasts & status, list known episodes & status, alter episode status (mark for downloading or not), “catch up” podcasts, etc.
  • Automatic retry of downloads that failed due to transient errors

You can download a source tarball, or apt-get install hpodder if you run Debian sid.

hpodder is written in Haskell, and calls the curl and id3v2 binaries. It uses the Sqlite3 library and my HDBC database interface for Haskell.

But you’d never need to know or care about that unless you’re a programmer.

In future hpodder versions, I intend to improve the download status display, add last-seen date tracking, and add multithreaded downloading.