Comments on: rdiff-backup, ZFS, and rsync scripts

By: thermionic

thermionic — Sat, 20 Aug 2011 22:44:30 +0000

if you’e looking at zfs snaps, how about rsync with link-desk instead of rdiff ?

I found this quite interesting
http://www.sanitarium.net/unix_stuff/backups/readme.txt

By: Angelo

Angelo — Tue, 15 Mar 2011 19:21:41 +0000

Hey there, I’ve been in the same boat as you–only in reverse. I started with ZFS on Solaris which worked beautifully. But because of the poor support and lack of active development on Solaris, I moved to Linux. I chose to use rdiff-backup as well, but I’ve noticed it slowing down quite a bit.

I’m running into the same problem as you: I want to use ZFS but it’s just not all there on Linux.

By: micah

micah — Thu, 10 Mar 2011 18:36:52 +0000

Thanks for your blog posts about this, I’m having a lot of trouble with our backup server, doing remote backups of the backup server itself, and you have given me some interesting ideas.

About backupninja, I’m surprised you have not heard of it before, its been around for ages. You can schedule different backup jobs with backupninja to fire at different times.

Other things that backupninja can do that you were looking for:

1. Provides “run before” job capability
2. Can be silent on success and loud on failures (although if your mail system fails, you may never notice that failures are happening!)
3. Continues to backup when there is an error
4. You can easily add the capability to do snapshots and removal of snapshots, there is a shell handler that can be run with any commands you wish.
5. Its easily extendable, and simply written in bash

By: John Goerzen

John Goerzen — Thu, 27 Jan 2011 16:55:45 +0000

Interesting program — especially this in the description: “Backupninja is a silent flower blossom death strike to lost data.”

I’m really liking BackupPC’s scheduler and management interface. One problem with backupninja is that it doesn’t have centralized scheduling. It will not be practical for backing up dozens of servers as a result.

By: John Goerzen

John Goerzen — Thu, 27 Jan 2011 16:51:00 +0000

Ah, interesting on the rdifffs — I’ll have to check it out. Glad you enjoyed RWH!

By: John Goerzen

John Goerzen — Thu, 27 Jan 2011 16:48:52 +0000

Well, I think I was right about it but maybe not sufficiently clear.

You are right that I can’t delete incrementals from the middle of history. What I meant is that I can say “delete all backups older than 4 months old” and this just happens without consequence to restoring newer files. That’s unlike duplicity, which uses forward history, so you have to periodically run full backups so that you can delete old backups.

By: Jon

Jon — Thu, 27 Jan 2011 09:41:27 +0000

Interesting to see you evaluate rdiff-backup. I wrote a FUSE filesystem for browsing rdiff-backup increments, rdifffs, fueled largely by your excellent RWH book.

I think you may be wrong about like #4: From what I have been able to figure out, binary diffs are calculated against the next-newest version of the file, so if you delete an increment from the middle of the deck you might prevent restoring some files in older increments. But you can probably safely delete the oldest increment at any given time.

By: anony

anony — Thu, 27 Jan 2011 08:10:27 +0000

WRT backup scripts, have you considered backupninja? It has already backends to backing up databases and rsync.

As to rotating log files, I’ve switched to date-based rotation instead of number-based rotation, that pretty much eliminates the undesired behavior.

By: jimcooncat

jimcooncat — Wed, 26 Jan 2011 14:11:20 +0000

I have a bit different scenario in my small office; small files (word docs, spreadsheets, MS Access databases), and high availability is really wanted. I ran into some difficulties as well with LVM snapshots, especially when I was working to make a DRBD setup.

But I think the fix for snapshots will be in reach for me soon by using NILFS. It still has a bit to go before it’s production-ready, but it seems to have what I’d need for good performance.

I envision using NILFS and DRBD for an asyncronous primary/secondary setup, where the secondary server mounts the NILFS snapshots and backs up to another off-site server using rdiff or some other diff tool.

Not suggesting this is any answer to your problems, but the developments with NILFS might be of interest to you.
Homepage: http://www.nilfs.org
Current discussion: http://www.mail-archive.com/linux-nilfs@vger.kernel.org/

By: John Goerzen

John Goerzen — Tue, 25 Jan 2011 22:56:38 +0000

Yes. At present, not being able to remove old backups is a major showstopper. Concerns about how reliable it is, not preserving symlinks or hardlinks, etc. also rank up there. But I think they are actively working on all those things and it may well turn out to be a very good option in the future.

By: Nicholas Lee

Nicholas Lee — Tue, 25 Jan 2011 22:40:07 +0000

Have you seen bup?

By: John Goerzen

John Goerzen — Tue, 25 Jan 2011 20:05:16 +0000

Git won’t be suitable for this. For one thing, you can’t delete the old backups very easily. For another, ever tried committing a 10GB file in Git? Yeah, it ain’t pretty. You’d need more RAM for that than I can afford.

By: John Goerzen

John Goerzen — Tue, 25 Jan 2011 20:03:41 +0000

I haven’t tried it; see my comment on http://changelog.complete.org/archives/5547-research-on-deduplicating-disk-based-and-cloud-backups

By: Anonymous

Anonymous — Tue, 25 Jan 2011 19:26:26 +0000

As far as deduplication sstems go, have you tested just dropping files into a git repository and packing periodically? Git has some pretty impressive deduplication bits.

By: ssam

ssam — Tue, 25 Jan 2011 17:48:35 +0000

how well does btrfs compare

By: John Goerzen

John Goerzen — Tue, 25 Jan 2011 17:09:18 +0000

That is an excellent point. Last time I looked at it, the requirement for the web server and the fact that they pretty much roll their own compression format and rsync server both bothered me. But it’s worth another look. I may miss having things directly accessible on the filesystem though.

By: martin f. krafft

martin f. krafft — Tue, 25 Jan 2011 16:50:26 +0000

> Things I don’t like about it:

[…]

4. It wreaks havoc backing up e.g. /var/log, if files are just being
rotated.

I’ve used rdiff-backup myself for years until I switched to
backuppc. I have never looked back. backuppc rocks, for a single
machine, but especially if you have a few. It uses content-addressed
storage (like Git), so the same file is only ever saved once, even
across systems.

The web interface requires a web server, which is a bit naff. But
the ability to browse backups and restore files, directories and
hosts interactively and intuitively quickly relatives that.