Comments on: How To Think About Compression, Part 2

By: Research on deduplicating disk-based and cloud backups | The Changelog

Research on deduplicating disk-based and cloud backups | The Changelog — Thu, 20 Jan 2011 17:55:07 +0000

[...] at the moment. I set up a 2GB ZFS pool for this test, and set dedupe=on and compress=gzip-2. When I evaluated compression in the past, I hadn’t looked at lzjb. I found a blog post comparing lzjb to the gzip options [...]

By: Ole Tange

Ole Tange — Sat, 01 Aug 2009 18:21:10 +0000

Did you have a chance to look at FreeArc.org?

I was impressed the time was comparable to Bzip2, but compressed 20-30% better.

By: Charles Darke

Charles Darke — Sat, 13 Jun 2009 21:38:34 +0000

You might be interested to google for pigz (parallel version of gzip). Personally, I use lzop (since it is free) or pigz as a good trade off. LZMA is good where time is not a factor (decompression speed is reasonable).

By: John Goerzen

John Goerzen — Mon, 02 Mar 2009 13:45:49 +0000

The scale is plainly visible. Sometimes the difference between numbers is such that in a size to be able to fit in a blog post, starting it at zero would be useless itself.

By: Marijn Schouten

Marijn Schouten — Mon, 02 Mar 2009 12:14:25 +0000

Graphs that don’t start at zero are misleading and thus worse than useless.

By: Palmax

Palmax — Wed, 18 Feb 2009 21:37:32 +0000

Did you use threads?

By: Bill

Bill — Wed, 18 Feb 2009 21:28:45 +0000

I’ve been wondering how to describe my results using debootstrap. I got into it by wanting to use the –save-tarball as well as (unrelated to this post, directly anyway, learn how to –second-stage scripting from Neil Williams emdebian stuff)
I debootstraped all etch variants ( I wanted to get a copy before I couldn’t easily anymore)
When the minbase vairant of etch is tarballed it is 31386007 bytes – I thought to myself Nice…
Also wanting to know what the others looked like and in case I needed them too.
buildd is 52514446 and fakechroot is 45102635
I’ve been intrigued by the notion of this thing called pristine tar since I first heard Joey Hess blog of it around the time he was moving from svn to git iirc..
“minbase seems to me at this point” to be a pristine debain install – although I have already removed all important priority packages as well using cat and diff to get the list from dpkg/info file.
After unpacking the tarball I got an urge to test tar on what I thought would be a decent test target.
So I
first made a copy of the minbase system
cp -a /mnt/md6 /srv/pristine.minbase
du -b /srv/pristine.minbase 92913288
than I made archives using tar, and tar switches for gzip, bzip2 and lzma
tar -xf pristine.tar /mnt/md6/*
tar -xzf pristine.tar.gz /mnt/md6/*
tar -xjf pristine.tar.bz2 /mnt/md6/*
tar -xf –lzma pristine.tar.lzma /mnt/md6/*
iirc
ls -l /srv
shows interestingly
drwxr-xr-x 20 root root 4096 2009-02-03 00:30 pristine.minbase
-rw-r–r– 1 root root 96634880 2009-02-03 11:56 pristine.tar
-rw-r–r– 1 root root 32455598 2009-02-03 11:58 pristine.tar.bz2
-rw-r–r– 1 root root 37310068 2009-02-03 11:57 pristine.tar.gz
-rw-r–r– 1 root root 22177523 2009-02-03 12:01 pristine.tar.lzma

Have I forgotten anything?

By: Markus Hochholdinger

Markus Hochholdinger — Wed, 18 Feb 2009 19:48:17 +0000

Great. The last time i made a gzip/bzip2 decission for my backups must be 2 years ago.
My reults are, the data is compressed with gzip because bzip2 was too slow fot the network and the tape. But i use bzip2 for my backup database to easily search for files in the backups because bzip2 saves me a lot of disk space compared to gzip.

By: Michael G

Michael G — Wed, 18 Feb 2009 17:19:55 +0000

Great analysis.

I’d like to see the numbers for decompression as well. There are a lot of cases where I’d be willing to pay a lot of computing time during compression as long as decompression isn’t too bad. If I’m putting something up for public download, for example, it’s going to be compressed just once, but decompressed by every single person who downloads it.

By: suscal

suscal — Wed, 18 Feb 2009 04:55:52 +0000

> For part 2, I will be compressing each individual file
> contained in that tarball individually. This is a good
> test if you back up to hard disk and want quick
> access to your files.

For this cases, I like squashfs!

By: How To Think About Compression | The Changelog

How To Think About Compression | The Changelog — Wed, 18 Feb 2009 03:20:00 +0000

[...] Part 2 of this story is now available, which considers more compression tools, and looks at performance compressing files individually rather than the large tar file. [...]