Category Archives: Software

Announcing hpodder

Software, ,

Today I’m finally announcing hpodder.

I’ve been trying different podcatchers in Linux, and have been generally unhappy. ipodder looked nice at first, but turned out to be horribly buggy.

bashpodder/podracer looked like a nice idea. However, it didn’t have enough flexibility for me, its XML parser has some well-known failures (it’s not a real XML parser, after all), etc.

So I wrote hpodder. hpodder is a command-line podcast downloader for Linux. It features:

  • Extensive manual (installed as manpage, or you can view the PDF versoin). Documents all command-line options, the config file, a quick start, plus some basic information about the internal database
  • Database of seen URLs (in Sqlite3) — for use both for downloads and when processing feeds
  • Graceful handling of Ctrl-C, shutdowns, network troubles, etc — including ability to resume downloads later, plus the ability to detect servers that don’t handle download resuming properly (libsyn)
  • Automatic setting of ID3 tags based on the episode title and podcast title from the podcast’s feed (as iTunes does) — dramatically helps with viewing of all sorts of podcasts on the iPod and your PC
  • Support for download rate limits, progress bars, etc. via Curl
  • Seems to be stable for me
  • Command-line tools to: add new podcasts, remove podcasts, update podcast feed URLs, scan podcast feeds, list known podcasts & status, list known episodes & status, alter episode status (mark for downloading or not), “catch up” podcasts, etc.
  • Automatic retry of downloads that failed due to transient errors

You can download a source tarball, or apt-get install hpodder if you run Debian sid.

hpodder is written in Haskell, and calls the curl and id3v2 binaries. It uses the Sqlite3 library and my HDBC database interface for Haskell.

But you’d never need to know or care about that unless you’re a programmer.

In future hpodder versions, I intend to improve the download status display, add last-seen date tracking, and add multithreaded downloading.

Why Is Expensive Software So Crappy?

Software

Today I was listening to Gary McGraw on Frontline Security talking about software security. One of his points was that a large part of security trouble is poor design.

That reminds me how I’ve been meaning to rant a bit about the really terrible security I’ve seen in proprietary software lately. Some of this is very expensive software that people pay lots of money for.

  • World-writable installations. In one case, the documentation for the software directed users to mark the entire program’s directory tree world-writable, including all files and directories within it. In a whole host of additional cases, consultants or support people tasked with installing the software make everything world-writable as a matter of routine. And some of these are programs specifically designed to be used on Unix shell hosts.
  • Overuse of telnet. There’s telnet use everywhere. One program actually telnets to a server to start their own server-side component and then send XML to it. So we have to have an account for the server-side component, put the password in plain text on the client side, and maintain *telnet* for the application. Have we never heard of, say, CGI, people???
  • Overuse of root. Again, I’ve seen this even in documentation — “run everything as root” or “if you have trouble, just run this as root.” I’ve seen installers actually check to see if they’re running as root, and fail if they’re not, even though they have no need for root privileges.

Sigh. Although I’ve seen some poor code out there in the Free Software community, I’ve never seen anything that even approaches this level of insanity.

Why is the most expensive software the least secure? And what can we do about it when the vendor doesn’t care?

HDBC 1.0.0

Software

This evening, I released HDBC 1.0.0.

HDBC is a database API for Haskell. I wrote it after being unsatisfied with HSQL. HDBC at a certain level feels similar to Perl’s DBI. But it is both simpler and more powerful, IMHO, thanks in large measure to Haskell language features.

The HDBC homepage is here.

How to solve “The following packages cannot be authenticated”

Debian, , ,

Users of Debian’s testing or unstable distributions may be noticing messages from apt saying things like:

WARNING: The following packages cannot be authenticated!
  foo bar baz
Install these packages without verification [y/N]?

I noticed today that google doesn’t turn up good hits for the fix. The fix is really simple:

apt-get install debian-archive-keyring
apt-get update

That’s it. You now have secure packages from Debian. Nice, eh?

An iPod under Linux

Hardware, Music, Software, ,

I finally purchased my first iPod: a black 60GB iPod video model. I had been holding off for years. The iPod sounded nifty, but I just didn’t quite go there.

The thing that finally won me over was the camera connector. It lets you plug your iPod directly in to a digital camera. The iPod can download photos from the camera to its internal disk without the need for a PC. Very slick.

So anyway, we got the iPod and the camera adapter at the Apple store in Cambridge — a quick subway ride from Usenix. They were out of stock on the FM tuner, so I ordered that online.

The next step was to get the iPod working with Linux. I currently have it working with both music and video. Here’s how I did it.

First Thougts on Xen

Software

At work, we’ve been using vserver for virtualization for some years now. Due to various reasons, we’re looking at Xen.

I’ve been trying to switch my workstation to use Xen. I’ve enountered a few issues so far. Probably these will go away as I learn the system.

Overall, my greatest gripe is the documentation. It is outdated and just plain wrong far too often. For instance, there’s a place where it says to run “make ARCH=xen xconfig”, but the Xen kernel patches don’t (any more, at least) provide a xen arch.

The next gripe is the very weird kernel build system. Xen doesn’t ship a diff against a kernel tree. They instead ship whole files to extract atop a particular kernel version. Annoying and unwieldy. There is a command to generate a diff, but you have to download the full kernel tree first.

A couple of other gripes: There is little documentation on memory management (can Xen adjust the RAM usage of running VMs?), on 64-bit systems (can you run a 32-bit kernel under a 64-bit hypervisor? how about a 64-bit kernel that supports 32-bit userspace?)

I’m also having trouble with my forcedeth card locking up under Xen.

However, I’ve heard of lots of people having good luck with it so I’m going to keep trying.

But one would think that basic docs could be actually worked on a bit more.

Announcing HSH, the Haskell Shell

Programming

Following the “release early, release often” motto, I am happy to announce version 0.1.0 of HSH, the Haskell shell.

You may obtain it with:

darcs get --tag 0.1.0 http://darcs.complete.org/hsh

Things are still very rough in many ways, but this version already lets you:

  • Run commands
  • Pipe things between commands
  • Pipe command input/output into and out of pure Haskell functions
  • Pure Haskell functions are as much a first-class citizen as is grep or cat

Here is an example session: (some lines wrapped for readability)

$ ghci -fglasgow-exts HSH

*HSH> run $ ("ls", ["."])
COPYING    HSH        HSH.hs    TODO    announcements  testsrc
COPYRIGHT  HSH.cabal  Makefile  _darcs  test.hs

*HSH> run $ ("ls", ["-l"]) -|- ("wc", ["-l"])
12

*HSH> :m +Text.Printf
*HSH Text.Printf> let countLines = (zipWith (\i line -> printf "%-5d %s" i line) 
       [(1::Int)..])::([String] -> [String])

*HSH Text.Printf> run $ ("ls", ["-l"]) -|- countLines -|- ("grep", ["hs$"])
6     -rw-r--r-- 1 jgoerzen jgoerzen  1285 Jun  6 09:43 HSH.hs
11    -rw-r--r-- 1 jgoerzen jgoerzen   565 Jun  6 09:43 test.hs

*HSH Text.Printf> :m +Data.List
*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|- 
         filter (isSuffixOf "hs")
6     -rw-r--r-- 1 jgoerzen jgoerzen  1285 Jun  6 09:43 HSH.hs
11    -rw-r--r-- 1 jgoerzen jgoerzen   565 Jun  6 09:43 test.hs

*HSH Text.Printf Data.List> run $ ("ls", ["-l"]) -|- countLines -|- filter (isSuffixOf "hs") 
       -|- ("tr", ["a-z", "A-Z"])
6     -RW-R--R-- 1 JGOERZEN JGOERZEN  1285 JUN  6 09:43 HSH.HS
11    -RW-R--R-- 1 JGOERZEN JGOERZEN   565 JUN  6 09:43 TEST.HS

*HSH Text.Printf Data.List> let generator = \(_::String) -> unlines . map show $ [1..20]
*HSH Text.Printf Data.List> generator ""
"1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n"
*HSH Text.Printf Data.List> run $ generator -|- ("grep", ["1"])
1
10
11
12
13
14
15
16
17
18
19

Future versions will likely simplify syntax to make it easier to write scripts and introduce a sh to hsh converter. I also plan to add pure Haskell tools for some common shell-ish things that one could do in Haskell.

Bacula

Software, ,

Lately we’ve been looking at backup solutions at work.

And I’ve got to say that Bacula is looking downright awesome. It’s GPL’d and it has just about every feature a person could ask for.

I am a complete Bacula newbie. Today, after using Bacula for a total of about 30-60 minutes, I added the first client machine to my Linux test box. The client machine was running the Windows bacula client. It took about 10 minutes to install and configure the client and the server. And both backup and restore worked perfectly the first time. Nice. Setting up a *nix client is even easier.

I’ve been using Amanda for many years at home and at various workplaces. Looks like we’re going to be switching.

We’ve also ordered an HP MSL4048, a 48-tape LTO3 library with barcode support. Each tape has a native storage capacity of 400GB. Should be nice when it arrives. With that library and Bacula, we should be able to back up all our servers using a single backup system. And both our Windows and Unix people can manage the system, including running restores to any machine, from any authorized console machine.

Debian From Scratch 0.99.0 Is Out

Debian

At long last, I’ve finally updated Debian From Scratch (DFS). For those of you not familiar with DFS, it’s a single, full rescue CD capable of working with all major filesystems, LVM, software RAID, and even compiling a new kernel. The DFS ISO images also contain a small Debian mirror subset that lets you use cdebootstrap, along with the other utilities on the CD, to perform a manual, “Gentoo-like” installation. It also serves as an excellent rescue CD, with a full compliment of filesystem tools, backup/restore software, and a development environment complete enough to build your own kernels.

DFS also refers to dfsbuild, the tool that generates DFS images. dfsbuild is available as a Debian package. dfsbuild is designed to make it trivial to build your own custom DFS images. You can have your own set of Debian packages on your images, your own kernels, etc. Unlike many other systems, you can go from the example dfs.cfg to a customized DFS build in just a few minutes, even if you’ve never used dfsbuild before.

Version 0.99.0 is a from-scratch rewrite and port to Haskell. You can read the full list of new features in the announcement, but the biggest is that it now supports standard Debian initramfs kernels in addition to ones that have enough drivers statically linked to be able to read the CD-ROM.

You can also download my DFS images or browse the docs online.

Dupes really fixed now

Software, , ,

Thanks to some assistance from Garvin (lead Serendipity developer), it looks like the bug that Planet dislikes so intensely is indeed the <slash:comments> tag. I don’t believe this is a bug in Serendipity bug rather in Planet.

(There is still, IMHO, a pubDate bug in Serendipity, but it appears to be unrelated)

So, the dupes you were seeing from me really are gone now. I hope.

The fix is to edit the file templates/default/feed_2.0.tpl and remove the line that provides the <slash:comments> tag.

BTW, seems that Planet Haskell also ran afoul of this.