Category Archives: Entertainment / Arts

Sound players: Adventures with Ampache, mpd, pulseaudio, Raspberry Pi, and Logitech Media Server

I finally decided it was about time to get my whole-house sound project off the ground. As an added bonus, I’d like to be able to stream music from my house to my Android phone.

Some Background

It was about 2.5 years ago that I last revisited the music-listening picture on Linux. I used Spotify for awhile, but the buggy nature of its support for local music eventually drove me up the wall. I have a large collection of music that will never be on Spotify (local choirs, for instance) and this was an important feature.

When Google Play Music added the feature of uploading your local collection, I used that; it let me stream music from my phone in my car (using the Bluetooh link to the car). I could also listen at home on a PC, or plug my phone into various devices to play. But that was a hassle, and didn’t let me have music throughout the house.

Google Play is reasonable for that, but it has a number of really glaring issues. One is that it often gets album artwork wrong; it doesn’t use the ID3 tags embedded in the files, but rather tries to “guess”. Another is that the “sync” is only “add”. Move files to another place in your collection, or re-rip them to FLAC and replace your old MP3s, and suddenly they’re in Google Play twice. It won’t ever see updated metadata, either — quite a hassle for someone that uses Musicbrainz and carefully curates metadata.

My Hardware

I already have an oldish PC set up as a entertainment box running MythTV. It is a diskless system (boots PAE over the network and has NFS root) and very quiet. It has video output to my TV, and audio output via S/PDIF to my receiver. It is one logical audio frontend.

My workstation in my office is another obvious place.

My kitchen has a radio with a line in jack, and I also have a small portable speaker with a line in jack to make the last two options. I also have a Raspberry Pi model B that I bought awhile ago and was looking for a purpose, so I thought – this should be cheap and easy, right? Well. Cheap, yes. Easy, not so much.

First attempt: Ampache

The Ampache project produces quite a nice piece of software. Ampache has matured significantly in the last 2.5 years, and the usability of its web-interface — with HTML5 and Flash players as options — is quite impressive. It is easily as usable as Google’s, though its learning curve is rather more steep. There are multiple Android apps for Ampache to stream remotely. And, while most are terribly buggy and broken, there is at least one that seems to work well.

Ampache can also output m3u/pls files for a standalone player. It does on-the-fly transcoding. There are some chinks in the armor, however. The set of codecs that are transcoded or passed through is a global setting, not a per-device setting. The bitrates are per-account, so you can’t easily have it transcode FLAC into 320Kbps mp3 for streaming on your LAN and 128Kbps MP3 for streaming to your phone. (There are some hacks involving IP address ranges and multiple accounts, but they are poorly documented and cumbersome.)

Ampache also has a feature called “localplay” in which it drives local players instead of remote ones. I tried to use this in combination with mpd to drive music to the whole house. Ampache’s mpd interface is a bit odd; it actually loads things up into mpd’s queue. Sadly it shares the same global configs as the rest. Even though mpd is perfectly capable of handling FLAC audio, the Ampache web player isn’t, so you have to either make it transcode mp3 for everything or forego the web player (or use a second account that has “transcode everything” set). Frustratingly, not one of the Android clients for Ampache is even remotely compatible with Localplay, and some will fail in surprising ways if you have been using Localplay on the web client.

So let’s see how this mpd thing worked out.

Ampache with mpd for whole-house audio

The primary method here is to use mpd’s pulseaudio driver. I configured it like so:


audio_output {
type "pulse"
name "MPD stream"
#server "remote_server" # optional
sink "rtp" # optional
mixer_type "software"
}

Then in /etc/pulse/system.pa:


load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
load-module module-rtp-send source=rtp.monitor

This tells Pulse to use multicast streaming to the LAN for the audio packets. Pulse is supposed to have latency synchronization to achieve perfect audio everywhere. In practice, this works somewhat poorly. Plus I have to install pulse everywhere, which inserts its tentacles way too deeply into the ALSA stack for my taste. (alsamixer suddenly turns useless by default, for instance.)

But I gave it a try. After much fiddling — Pulse is rather poorly documented and the interactions of configuration tools with it even more so — I found a working configuration. On my MythTV box, I added:


load-module module-rtp-recv
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;x.x.x.x

The real annoyance was getting it to set the output through the S/PDIF digital port. Finally I figured out the magic potion:


set-card-profile 0 output:iec958-stereo+input:analog-stereo

This worked reasonably well. The Raspberry Pi was a much bigger challenge, however.

I put Raspbian on it easily enough, and installed Pulse. But apparently it is well-known in the Pulse community that Pulse’s RTP does not work well with wifi. Multicast itself works poorly with wifi in general, and Pulse won’t do unicast RTP. However, Pulse assumes very low latency and just won’t work well with wifi at all.

Morever, pulseaudio on the rpi is something of a difficult beast to tame. It has crackling audio, etc. I eventually got it working decently with:

daemon.conf:


realtime-scheduling = yes
realtime-priority = 5
resample-method = src-sinc-fastest
default-sample-rate = 44100
default-fragments = 4
default-fragment-size-msec=20

and in system.pa, commented out the module-udev-detect and module-detect, and added:


load-module module-alsa-card device_id=0 tsched=0 fragments=10 fragment_size=640 tsched_buffer_size=4194384 tsched_buffer_watermark=262144

plus the rtp-recv and tcp protocol lines as before. This got it working with decent quality, but it was always out of sync by at least a few tenths of a second, if not a whole second, with the other rooms.

Brief diversion: mplayer

Many players besides pulseaudio can play the RTP streams that pulseaudio generates. mplayer, for instance, can. I found this worked on the Raspberry Pi:


mplayer -really-quiet -cache 64 -cache-min 95 -demuxer rawaudio -rawaudio format=0x20776172 rtp://224.0.0.56:46510

But this produced even worse sync. It became clear that Ampache was not going to be the right solution.

Logitech Media Server

I have looked at this a few times over the years, but I’ve somehow skipped it. But I looked at it again now. It is an open source (GPL) server, and was originally designed to work with Logitech hardware. There are now all sorts of software clients out there. There is a helpful wiki about it, although Logitech has rebranded the thing so many times, it’s not even consistent internally on what the heck it’s called (is it LMS? or Squeezebox Server? or Slimserver? The answer is: yes.)

LMS does no web-based streaming. At all. But I thought I’d give it a try. Installation is a little… weird (its .deb packages up binary perl modules for things that are already in Debian, for half a dozen architectures and many Perl versions.) I had some odd issues but eventually it worked. It scanned my media collection. I installed squeezelite on my workstation as my first player, and things worked reasonably well out of the box.

I proceeded to install squeezelite on the Raspberry Pi and MythTV server (where I had to compile libsoxr for wheezy, due to an obscure error I couldn’t find the cause of until I used strace). Log in to the LMS web interface, tell it to enable synchronization, and bam! Perfect synchronization! Incredible.

But I ran into two issues: one was that it used a lot of CPU even on my workstation (50% or more, even when idle). Strangely, it used far less CPU on the pi than did pulseaudio. Secondly, I’d get annoying clicks from it from time to time. Some debugging and investigation revealed that they were both somewhat related; it was getting out of sync with the pi and correcting. Initially I assumed this to be an issue with the pi, but tracked it down to something else. I told it to upsample to 48kHz and this made the problem go away. My command on PC hardware is:


squeezelite -a 200:10 -o dmix -u hLX -z

And on the Raspberry Pi:


squeezelite -a 800:10:16:0 -n kitchen -z

It is working perfectly now.

LMS even has an Android remote control app that can control all devices together, queue up playlists, etc. Very slick.

The Raspberry Pi

The Raspberry Pi, even the model B+, apparently has notoriously bad audio output. I noticed this first as very low volume on the output. I found a $11 USB DAC on Amazon that seems to do the trick, so we will see. I am also using the $10 Edimax USB Wifi and it seems to work well.

Other Options

The big-name other option here is Sonos. They have perfect sync audio already working. But they’re pricy, closed, and proprietary; you have to buy Sonos speakers, transmitters, etc. for the entire house. Their receivers start at $200, and it’s at least $350 to integrate an existing stereo into the system. I’d be shelling out over $1000 to use Sonos for this setup. As it is, I’ve bought less than $100 of equipment (a second Pi and its accessories) and getting output that, while no doubt not quite as pristine, it still quite nice and acceptable. Quite nice.

Update: LMS with Apache

It turns out that Logitech Media Server can automatically save created playlists to a spot where Ampache can find them. So an Ampache streaming player can be used to access the same collection LMS uses, with full features. Nice!

Glockenklang, Forever

Jacob (and Oliver too, somewhat) has taken quite the interest in Christmas music this year. Perhaps it’s singing in the choir at school, or perhaps it’s just him getting older, but in any case, Christmas music fascinates him.

And no song more than Jingle Bells. I have recordings of it by several artists in different styles, and he has his favorites and often wants to hear them – again and again.

The other night at supper, he said, “Dad, can you sing Jingle Bells in German?” Kind of a random question. I normally try to keep books and devices of all sorts away from the table, but my son had just asked me to sing. In German. I don’t believe that had ever happened before, so I wasn’t going to say no! I got my tablet, pulled up Google, and found some German lyrics. Not exactly a translation, but it fit the tune, so that’s what counts.

So I started singing, and when I got to the chorus, and sang Oh, Glockenklang, Glockenklang both boys bust up laughing. They thought Glockenklang was a hilarious word, and loved to hear it. Oliver requests I “sing Glockenklang” every so often now. He has this eager anticipation when he does it, as if he knows I’m going to be hilarious — so much so that he almost starts laughing before I even say a word.

Then yesterday at breakfast, Jacob requested more German songs. I finally got a couple of hymn books (one of which, Gesangbuch mit Noten, really is a German songbook). I sang some songs in German for the boys, while they enjoyed their blueberry crepes. Sadly Stille Nacht and O du fröhliche did not prove as hilarious as Glockenklang, but they got impatient as I looked through the idnex between each song, saying “Dad, just sing more German! Don’t look at those pages!” So I guess it was a hit.

I’m not sure where this sudden fascination with German music came from, but it appears to be leaving me hungry.

While driving around, Oliver requested I sing Glockenklang in the car. I said yes, despite not remembering even one German word to that song except for, well, Glockenklang. So I just filled in with some random German words I do know. Fortunately, they didn’t seem to have memorized the rest of it either.

The other day, we went to the annual Christmas concert at the local high school. It was perfect for the boys – various styles of music, an interesting old auditorium, nobody was going to care if they weren’t perfectly quiet, and the price was right, too.

The band played first, and both boys sat there, paying great attention, soaking it all in. They loved the cymbal crashes and Jacob seemed to dream of playing the chimes and bells. At one point, I whispered to Jacob that I could feel the timpani on the wood floor in the auditorium, and he tried it, and made his “Hmm, I just learned something interesting!” reaction.

As is tradition, the concert concluded with singing the Hallelujah Chorus – and anyone in the audience that wants to sing is invited on stage to join the choir.

I remember being in high school for that concert, and after practicing it in school, the great fun of being joined by many powerful voices from the community all around me on the risers. So this time, while my parents stayed with the boys, I was one of the many that went up to join the choir. It was no less thrilling all these years later. Ending with “Forever and ever, Hallelujah, Hallelujah. Hallelujah!”, and feeling it, right then, put a huge smile on my face.

Difficult Times & Hope

This past month has been the most difficult in my life – and that of my family as well. I’m not going to go into it for the whole Internet, but any well wishes, happy thoughts, prayers, whatever you want to send our way, whether we know about it or not, would certainly be welcome. And, of course, the reason I’m not very active online right now is that I’m focusing on family, work, and other pressing matters.

I have had some measure of comfort from hearing from others that have had pain in their lives. It is good to know I’m not alone, good to have people to talk and share with. And it is good to find some way to have hope in the midst of difficulty and uncertainty.

I sing with the Kansas Mennonite Men’s Chorus, and when listening to one of our recordings recently, was struck by these words in a whole new way. Let music never die in me is a powerful message.

I dreamed a dream, a silent dream,
of a land not far away.
Where no bird sang,
no steeples rang
and teardrops fell like rain.

I dreamed a dream.
No alleluia, not one hosanna,
No song of love, no lullaby.

And no choir sang to change the world.
No pipers played, no dancers twirled.
I dreamed a dream, a silent dream.
Silent.
Silent.

Awake! Awake!
Awake, my soul and sing!
The time for praise has come.
The silence of the night has passed;
a new day has begun.
Let music never die in me!
Forever let my spirit sing!
Wherever emptiness is found,
Let there be joy and glorious sound.

Let music never die in me!
Forever let my spirit sing!

Let all our voices join as one
to praise the giver of the song!

Awake! Awake!
Let music live!

The Awakening, Joseph M. Martin (excerpts)

See a performance here.

Even when hope is dim, or the music is playing only softly, it’s not dead. I hear it when an 85-year-old person in church, comes up to me with tears in her eyes and gives me a big, silent hug. I feel the music when when I can share about things with people, when I sing, when the beautiful Kansas sunset peeks out, when I share a smile or laugh with someone, and when I see the delight and happiness of children.

Let music live!

Decisions on Listening to Music

A couple of days ago, I commented about my thoughts on finding a better way to listen to music, and asked for suggestions. I’ve checked out the avenues suggested, and here are my thoughts so far.

Streaming service: Spotify

Of the streaming service offerings, Spotify seems the most compelling. It has both Linux and Android clients. The Linux client is unofficial and a bit buggy, but serviceable. The service is fairly nice, and the serendipity of listening to “radio” of tracks like any track or artist I may pick is an incredible feature. It will mean a different mindset; rather than carefully choosing what I buy, I have a vast catalog at my fingertips. I’d have to give up some control, but might be better off for it.

Spotify can play local files, but sadly lacks support for FLAC that most of my collection is in. A little work with the shell, and I had a quick multi-core FLAC to MP3 conversion done. It can also identify local files and match them up with its catalog, but it does a somewhat poor job of it, despite the fact that my files are tagged by Musicbrainz. There are also a fair number of albums in my collection (from Magnatune and from local artists) that aren’t in Spotify’s catalog and probably never will be. I may wind up using git-annex or something like it to sync them between PCs.

Surprisingly, it doesn’t have any kind of web-based player available. It has a lot of features, but many of them are under-documented.

Subsonic / Supersonic

The idea of being able to stream music anywhere certainly has appeal, and the idea of streaming that music from my own machine — my own collection — is quite the appealing one. Subsonic seems to be the standard that people recommend for this. It has a web-based player that works well, as well as mobile clients for various platforms. The Android mobile client is particularly well-regarded and is probably the best one here, aside from Spotify. The playlists work well, as expected, and generally it has the appearance of a well-rounded system. There is a lot of flexibility in bitrates; maximum bitrates can be set per-user and per-client. It, of course, transcodes on the fly.

My chief gripe about Subsonic is that it has no true album/artist/genre browser. Its main browser is mostly a filesystem browser. I have used clients that show album/artist/genre panes or sortable lists, based on tags, for so long that my filesystem organization is rather different – centered around bitrate of rip, origin, etc. It is not very easy to navigate my collection in that way. The search feature can search artist and album, and this can partially make up for it. But there is simply no way to see a list of all artists in the collection.

The server is written in Java, and takes up 362MB at startup. That’s, I guess, decent for a Java server, but doesn’t make me pleased as I’d be running it on lower-end hardware.

Supersonic is a git-maintained actively synced subsonic fork which removes the license cost (the source is GPL3 anyhow so that wasn’t a big deal).

Ampache

Often mentioned in the same breath as Subsonic, this is yet another frequent suggestion for streaming your own collection. Its primary player is web-based, like Subsonic. It has full artist/genre browsers, and adds tag cloud and various other options as well – by default, it loaded the tag cloud with my genre tags, which was a nice touch. The search feature is a little less robust, as it simply searches anywhere and returns a list of all matching tracks, rather than listing matching albums and artists before going into the list of all tracks. But that is somewhat of a minor nit.

Ampache’s web-based player, liks Subsonic’s, uses Flash, but the implementation is really quite awkward. You can’t simply click a play button and have playback start. Rather, you must add things to the “playlist” (queue), a frame on the web interface. Then, you must click play in THAT frame, which launches a new window for the player and transfers its contents to it. You thus have two separate running playlists. And if you think you can just add to the playlist of the player window, think again; you can only replace it. It is simply not possible to add a track to the end of that list without losing your existing place. This makes the usefulness of the web-based player rather, well, low.

Playlist functionality in Ampache is rather odd. You can have playlists, but if you want to add a track to a saved playlist, first you have to clear you current playlist, then add the track to it, then save it to the existing playlist.

Frontends for Ampache exist in banshee and in amarok. Banshee’s is broken, and simply silently fails to do anything. Unless you look in the console, in which case you’ll see a .NET backtrace. Banshee has had these mysterious errors for me for years. The Amarok plugin is so basic that you can browse by artist but not by album, and is almost unusable. There is also a rhythmbox plugin in Debian, but it is apparently for an old version of Rhythmbox and no longer works.

The best frontend is a dedicated Ampache frontend called Viridian. It has a nice interface, provides features such as taskbar integration, and generally works pretty well. The only thing it’s bad at is playlists, and here it’s even worse than the web interface. It doesn’t support modifying the server-side playlists at all. And accessing one is an annoying “load playlist” operation, that replaces your current play queue (also called the “playlist”) with its contents by default.

Several Android clients exist for Ampache, though it seems only one has seen any updates since 2010.

mpd

This is an interesting program, as it is designed more to provide multiple points of control for a single player than it is to stream a single collection to multiple devices. Nevertheless, I tested its HTTP and icecast stream modules. (It can also stream with PulseAudio, but PulseAudio causes devious breakage on my workstation, so I refuse to bother.) This could be made to work reasonably well on my LAN using the gmpc client. There are several Android clients as well, though none of them integrate as well as Subsonic or Spotify. Most seem focused on controlling a remote player. The ones that do stream do so with some hiccups, and don’t integrate controls with the lock screen.

gmpc is a reasonable control program and, combined with mplayer, makes a decent player. It can browse the database every way I’d want to. It doesn’t let me right-click on tracks to add them to playlists, but I can use copy/paste to do so, or add them to the play queue and from there to a playlist. It’s annoying, but not completely unworkable.

But here’s the real concern. mpd has no provision whatsoever for any sort of encryption such as SSL. If I want to be listening to my music collection from on the road, I don’t really wish to open up a streaming music service for the entire Internet. Nor do I really want to mess with SSH tunnels and the like, which are cumbersome on Android (and can be in general). This is pretty much a fatal flaw to me.

Google Music

An interesting service, but ultimately one I don’t see the point for in my situation – where I already have my music on a PC that is always on. It would probably accommodate my entire collection free, but how many weeks would it take me to get it there? And how would it stay in sync? It simply doesn’t offer me any compelling advantage over streaming direct from my PC, so I don’t see much need to bother with it.

Others

I briefly considered Streeme (poor mobile support), Audiogalaxy (no Linux support), Jinzora (appears dead), and Sockso (no mobile support). I also briefly considered other streaming services such as Pandora, Slacker, rdio, etc., but based on reviews didn’t think that they were worth bothering with.

Conclusions

I’m left mainly considering Spotify and Subsonic. For right now, I’m going to try Spotify and see where that leads me, but may wind up getting back to Subsonic if I can’t find tracks I want with Spotify.

How to listen to music?

This seems like a simple question: how do you listen to music?

But in a way, I think I’ve fallen behind the times.

I’ve enjoyed listening to digital music for years, since well before MP3 even existed (MIDI files could be comfortably downloaded over dialup; some of you may even remember MOD files).

Anyhow, a few years ago, I digitized my entire CD collection, ripping it to FLAC and encoding to high-bitrate MP3 for storage on my 160GB iPod. Since, I’ve purchased some music from DRM-free MP3 retailers, and even FLAC files from Magnatune, but at this point I basically say that I have enough music. I have only rarely bought anything new in the last couple of years. I have used Musicbrainz Picard to enforce consistent tagging across almost my entire collection.

I am usually in front of a Linux machine. I run mt-daapd on my fileserver, so I can stream music from it to any other machine in the house. However, this has a lot of drawbacks. It is generally impossible to create or modify persistent playlists with this protocol. I can’t modify metadata. I can’t assign ratings. Though perhaps I don’t need to.

There are a lot of online services I could think of: Pandora, Spotify, Rdio, Slacker. Plus, of course, the upload your music to the cloud services such as Google and Amazon. Have people tried these? I’m hesitant to use a service where I pay a monthly fee, unless I wind up owning the music in a DRM-free fashion. And I’m a little nervous about the privacy implications of uploading a vast amount of music to the cloud.

There are some items in my collection that I am certain are not available on any of those others, but for the most part I’m willing to take what is out there if it meets my tastes. My tastes, by the way, typically run to classical and opera but also can go towards middle eastern or certain new age music (though I’m very picky about that, since I don’t care for most new age stuff I hear.) But these are huge categories; sometimes I’d like to hear some load and noisy classical (say, Beethoven’s Ninth conclusion, some Wagner, Verdi arias, etc.) And other times, something more contemplative (a Mozert concerto) or even quiet (a nice sonata) meet my mood. I have playlists for these on my iPod Classic – a bit of a problem, since I don’t use that device much anymore except in the car, and it takes a LOT of time to categorize things into these playlists. (An opera or a symphony could have parts in various styles and various levels of energy, for instance.) I keep a small manually-managed subset on my Android phone, which works fine. A recommendation engine for things like this could be useful, if it works well.

My requirements, basically: I want something that will play the same music on my Linux machines. I want to be able to have playlists kept in sync across my machines. And I don’t want a monthly fee. Ideally, the software will work on Android as well, and provide similar features there. Anything that requires Windows is a non-starter. I’ve considered NFS mounts, but that’s a bit annoying when the laptop disconnects from the network or when it’s not at home.

What are your thoughts? Ideas?

Sing to Me, Muse

(a review of Homer’s Iliad)

Here, therefore, huge and mighty warrior though you be, here shall you die.

– Homer (The Iliad)

And with that formidable quote, I begin my review of The Iliad. I shall not exhaust you with a rehashing of the plot; that you can find on Wikipedia. Nor shall I be spending page upon page of analyzing the beautiful imagery, the implications of our understanding of fate and destiny, or all the other things that compel English majors to write page after page on the topic. Nor even shall I try to decipher whether it is a piece of history or a piece of legend.

Rather, I intend to talk about why I read it: it’s a really good story.

Sing to me, O goddess Muse, the wrath of Achilles, son of Peleus, which brought countless ills upon the Acheans.

As I’d be reading it, I’d think to myself: “Ah ha! Now here we finally have a section of the story that doesn’t speak to modern life.” Perhaps it was a section the fear of being enslaved or butchered by conquerors, or about pride leading both sides to fight a war they need not have, or a graphic description of a spear going clear through someone’s head or neck.

And then, I’d have to sit and think. Are we really so far removed from that? Here we are, in a supposedly civilized world. We use remotely-operated drones to drop bombs on people, and think it naught but regrettable “collateral damage” when the brains and intestines of dozens of innocent victims are scattered about, killing them, all for the chance of killing one enemy. But we aren’t the only ones to blame; we live in a world in which killing the innocent is often the goal. People fly airplanes into skyscrapers, drop atomic weapons to flatten entire cities, and kill noncombatants in terrible numbers. And for what? A little power, some prestige, some riches, some vengeance, some wounded pride?

Slavery is not dead, either. We that can happily afford Internet access most likely abhor the thought. What, though, can we make of the fact that people are starving in this world in record numbers? That our actions may literally wipe some nations off the map? Are those people as free as we are? Do our actions repress them?

I suspect you can’t read the Iliad without being at least a bit introspective.

Fear, O Achilles, the wrath of heaven; think on your own father and have compassion upon me, who am the more pitiable

One of the great things about reading the Iliad is that I can learn about the culture of the ancient civilizations. Now, of course, one can read about this in history books and Wikipedia. But reading some dry sentence is different from reading a powerful story written by and for them. I know that the Iliad isn’t exactly a work of history, but it sure shows what made people tick: their religion, their morals, their behavior, and what they valued. I feel that I finally have some sort of insight into their society, and I am glad of it.

The day that robs a child of his parents severs him from his own kind; his head is bowed, his cheeks are wet with tears, and he will go about destitute among the friends of his father, plucking one by the cloak and another by the shirt. Some one or other of these may so far pity him as to hold the cup for a moment towards him and let him moisten his lips, but he must not drink enough to wet the roof of his mouth; then one whose parents are alive will drive him from the table with blows and angry words.

There are gory battle scenes in the Iliad, but then there are also heart-wrenching tender moments: when Hector leaves his wife to go fight, for instance, and worries about the future of his child if he should die.

That’s not to say I found the entire story riveting. I would have liked it to be 1/3 shorter. The battle just dragged on and on. And yet, I will grant that there was some purpose served by that: it fatigued me as a reader, which perhaps gave me a small sense of the fatigue felt by the participants in the story.

Why, pray, must the Argives needs fight the Trojans? What made the son of Atreus gather the host and bring them?

A question that was never answered, for either side: why are we so foolish that we must go to war? One tragedy among many is that neither side got smart about it until way too late, if ever they did at all.

I read The Illiad in the Butler translation, which overall I liked. Some of these quotes, however, use the Lattimore one. This completes the first item on my 2010 reading list.

I leave you with this powerful hope for the future, written almost three thousand years ago. Despite my criticisms above, we have made a lot of progress, haven’t we? What a beautiful ideal it is, and what a long ways we have yet to walk.

I wish that strife would vanish away from among gods and mortals, and gall, which makes a man grow angry for all his great mind, that gall of anger that swarms like smoke inside of a man’s heart and becomes a thing sweeter to him by far than the dripping of honey.

Sing to me, O goddess, the anger of Achilles, that one day his story may speak less to our hearts, for then will we have outgrown it.

My Reading List for 2010

I can hear the question now: “What kind of guy puts The Iliad and War and Peace on a list of things to read for fun?” Well, me. I think that reading things by authors I’ve never read before, people that take positions I haven’t heard of before or don’t agree with, or works that are challenging, will teach me something. And learning is fun.

My entire list for 2010 is at Goodreads. I’ve highlighted a few below. I don’t expect to read all 34 books on the Goodreads list necessarily, but there is the chance.

The Iliad by Homer [done 1/11], 750BC, trans. by Alexander Pope, 704 pages. A recent NPR story kindled my interest in this work. I’m looking forward to it.

The Oxford History of the Classical World by Boardman, Griffin, and Murray, 1986, 882 pages. It covers ancient Greece and Rome up through the fall of the Roman empire.

The Fires of Heaven (Wheel of Time #5) by Robert Jordan, 1994, 912 pages [done 9/2010]. I’ve read books 1 through 4 already, and would like to continue on the series.

War and Peace by Lev “Leo” Nikolayevich Tolstoy, 1869, 1392 pages. Been on my list for way too long. Time to get to it. Haven’t read anything by Tolstoy before.

The Politics of Jesus by John Howard Yoder, 1972, 2nd ed., 270 pages. Aims to dispel the notion of Jesus as apolitical.

An Intimate History of Humanity by Theodore Zeldin, 1996, 496 pages. Picked this up at Powell’s in Portland on a whim, and it’s about time I get to it.

The Myth of a Christian Nation: How the Quest for Political Power Is Destroying the Church by Gregory A. Boyd, 2007, 224 pages. An argument that the American evangelical church allowed itself to be co-opted by the political right (and some on the left) and argues this is harmful to the church. Also challenges the notion that America ever was “a Christian nation.”

Daily Life in Ancient Rome: The People and the City at the Height of the Empire, by Jerome Carcopino, 2003, 368 pages. I’ve always been fascinated with how things were “on the ground” rather than at the perspective of generals and kings, and this promises to be interesting.

Slavery, Sabbath, War, and Women: Case Issues in Biblical Interpretation (Conrad Grebel Lectures) by Willard M. Swartley, 1983, 368 pages. Looking at how people have argued from different Biblical perspectives about various issues over the years.

To the Lighthouse by Virginia Woolf, 1927, 252 pages. I can’t believe I’ve never read Woolf before. Yet another one I’m really looking forward to.

Tales of the Jazz Age by F. Scott Fitzgerald, 1922, 319 pages. Per Goodreads: “This book of five confessional essays from the 1930s follows Fitzgerald and his wife Zelda from the height of their celebrity as the darlings of the 1920s to years of rapid decline leading to the self-proclaimed ‘Crack Up’ in 1936.”

Ulysses by James Joyce, 1922 (1961 unabridged version), 783 pages.

The Future of Faith by Harvey Cox, 2009, 256 pages. [done 3/2010] Per Goodreads, “Cox explains why Christian beliefs and dogma are giving way to new grassroots movements rooted in social justice and spiritual experience.” Heard about this one in an interview with Diane Rehm.

Being There by Jerzy Kosiński, 1970, 128 pages.

Jesus: Uncovering the Life, Teachings, and Relevance of a Religious Revolutionary by Marcus Borg, 2006, 352 pages. Whether or not you agree with Borg, this has got to be a thought-provoking title.

The Three Musketeers by Alexandre Dumas, 1844, 640 pages.

The Book of Tea by Kakuzo Okakura, 1906, 154 pages. Per Goodreads: “In 1906 in turn-of-the century Boston, a small, esoteric book about tea was written with the intention of being read aloud in the famous salon of Isabella Gardner. It was authored by Okakura Kakuzo, a Japanese philosopher, art expert, and curator. Little known at the time, Kakuzo would emerge as one of the great thinkers of the early 20th century, a genius who was insightful, witty—and greatly responsible for bridging Western and Eastern cultures. Nearly a century later, Kakuzo’s The Book of Tea is still beloved the world over. Interwoven with a rich history of tea and its place in Japanese society is poignant commentary on Eastern culture and our ongoing fascination with it, as well as illuminating essays on art, spirituality, poetry, and more.”

More of my list is at Goodreads.

MoinMoin as a Personal Wiki, Zen To Done, And A Bit of Ikiwiki

Since I last evaluated and complained about wikis last year, I’ve been using moinmoin for two sites: a public one and a personal one.

The personal site has notes on various projects, and my task lists. I’ve been starting out with the Zen To Done (ebook, PDF, paper) idea. It sounds great, by the way; a nice improvement on the better-known GTD.

My To Do Page

Anyhow, in MoinMoin, I have a ToDos page. At the top are links to pages with different tasks: personal, work, yard, etc. Below that, are the three “big rocks” (as ZTD calls them) for the day: three main goals for the day. I edit that section every day.

The Calendar

And below that, I use MoinMoin’s excellent MonthCalendar macro. I have three calendars in a row: this month, next month, and last month. Each day on the calendar is a link to a wiki page; for instance, ToDos/Calendar/2009-10-01. The day has a red background if the wiki page exists, and white otherwise. So when I need to do something on or by a specific day, I click on the link, click my TaskTemplate, and create a simple wiki page. When I complete all the tasks for that day, I delete that day’s wiki page (and can note what I did as the log message if I like). Very slick.

The Task Lists

My task pages are similar. They look like this:


= Personal =

<>

<>
<
>

So, my personal task page has a heading, then it has an input form with a text box and a button that says “Create new task.” Type something in and that becomes the name for a wiki page, and takes you do the editor to describe it. Below the button is a list of all the sub-pages under the Personal page, which represent the tasks. When a task is done, I delete the page and off the list it goes. I can move items from one list to another by renaming the page. It works very, very nicely.

Collecting

Part of both ZTD and GTD is that it must be very easy to get your thoughts down. The idea is that if you have to think, “I’ve got to remember this,” then you’ll be stressed and worried about the things you might be forgetting. I have a “Collecting” page, like the Personal or Work pages, that new items appear on when I’m not editing my wiki. They get there by email.

MoinMoin has a nice email system. I’ve set up a secret email address. Mail sent there goes directly into MoinMoin. It does some checks on it, then looks at a combination of the From and Subject lines to decide what to do with it. If I name an existing page, it will append my message the the end. If it’s a new page, it’ll create it. I have it set up so that it takes the subject line as a page name to create/append to under ToDos/Collecting/$subject (by putting that as the “name” on the To line).

So, on my computers, I have a “newtodo” script that invokes mail(1), asks for a subject, and optionally lets me supply a body. Quick and painless.

Also, I’ve added the address to my mobile phone’s address book. That way I don’t have to carry around pen and paper. Need to get down some thought ? No problem. Hit send email, pull the last address sent to, give it a subject and maybe a body. Very slick.

Wiki Software

As a way of updating my posts from last year: I’ve been very happy with MoinMoin overall. It has some oddities, and the biggest one that concerns me is its attachment support. It doesn’t let you specify a maximum upload size, and doesn’t very well let you restrict attachment work to only certain people. But the biggest problem is that it doesn’t track history on attachments. If a vandal deletes the attachment on a page, it’s GONE. They expect to have that fixed in 2.0, coming out in approximately November, 2010.

I also looked at Ikiwiki carefully over the past few days. Several things impressed me. First, everything can be in git. This makes for a very nice offline mode, better than Moin’s offline sync. The comment module is nicer than anything in Moin, and the tagging system is as well. Ikiwiki truly could make a nice blog, and Moin just couldn’t. It also puts backlinks at the bottom of each page automatically, a nice feature. And it’s by Joey Hess, who writes very solid software.

There are also some drawbacks. Chief on that list is that ikiwiki has no built-in history of a page feature. Click History and it expects to take you to gitweb or ViewVC or some such tool. That means that reverting a page requires either git access or cut and pasting. That’s fine for me, but throwing newbies to gitweb suddenly might not be the most easy. Since ikiwiki is a (very smart) wiki compiler, its permission system is a lot less powerful than Moin’s, and notably can’t control read access to pages at all. If you need to do that, you’d have to do it at the webserver level. It does have a calendar, but not one that works like Moin’s does, though I could probably write one easily enough based on what’s there.

A few other minor nits: the email receiving feature is not as versatile as Moin’s, you can’t subscribe to get email notifications on certain pages (RSS feeds only, which would have to be manually tweaked later), and you can’t easily modify the links at the top of each page or create personal bookmarks.

Ikiwiki looks like an excellent tool, but just not quite the right fit for my needs right at the moment. I’ve also started to look at DokuWiki a bit. I was initially scared off by all the plugins I’d have to use, but it does look like a nice software.

I also re-visited MediaWiki, and once again concluded that it is way too complicated for its own good. There are something like a dozen calendar plugins for it, some of which even are thought to work. The one that looked like the one I’d use had a 7-step (2-page) installation process that involved manually running SQL commands and cutting and pasting some obscure HTML code with macros in it. No thanks.

How To Record High-Definition MythTV Files to DVD or Blu-Ray

I’ve long had a problem. Back on January 20, I took the day off work to watch the inauguration of Barack Obama. I saved the HD video recordings MythTV made of the day (off the ATSC broadcast), intending to eventually save them somehow. I hadn’t quite figured out how until recently, so there they sat: 9 hours of video taking up about 60GB of space on my disk.

MythTV includes a program called mytharchive that will helpfully transcode your files and burn a DVD from them. But it helpfully will transcode the beautiful 1920x1080i down to DVD resolution of — on a good day — 720×480. I couldn’t have that.

Background

My playback devices might be a PC, the MythTV, or the PlayStation 3. Of these, I figured the PS3 was going to be hardest to accommodate.

ATSC (HD) broadcasts in the United States are an MPEG Transport Stream (TS). Things are a bit complicated, because there may be errors in the TS due to reception problems, the resolution and aspect ratio may change multiple times (for instance, down to SD for certain commercials). And, I learned that some ATSC broadcasts are actually 1920×1088 because the vertical resolution has to be a multiple of a certain number, and those bottom 8 pixels shouldn’t be displayed.

Adding to the complexity, one file was 7 hours of video and about 50GB itself. I was going to have to do quite some splitting to get it onto 4.7GB DVD+Rs. I also didn’t want to re-encode any video, both for quality and for time reasons.

Attempts

So, I set out to try and figure out how to do this. My first approach was the sledgehammer one: split(1). split takes a large file and splits it on byte or line boundaries. It has no knowledge of MPEG files, so it may split them in the middle of frames. I figured that, worst case, I could always use cat to reassemble the file later.

Surprisingly, both mplayer and xine could play back any of these files, but the PS3 would only play back the first part. I remembered this as an option if all else failed.

Next, I tried avidemux. Quite the capable program — and I thought I could use it to cut my file into DVD-sized bits. But I couldn’t get it to let me copy the valid MPEG TS into another MPEG TS — it kept complaining of incompatible formats, but wouldn’t tell me in what way they were incompatible. I could get it to transcode to MPEG4, and produce a result that worked on the PS3, but that wasn’t really what I was after.

Then, I tried mpgsplit. Didn’t recognize the MPEG TS as a valid file, and even when I used a different tool to convert to MPEG PS, acted all suspicious as if it bought the MPEG from a shady character on a grungy street corner.

dvbcut

I eventually wound up using dvbcut to split up the ATSC (DVB) recordings. It understood the files natively and did exactly what I wanted. Well, *almost* exactly what I wanted. It has no command-line interface and didn’t have a way to split by filesize, but I calculated that about 35 minutes of the NBC broadcast and 56 minutes of the PBS broadcast could fit on a single DVD+R.

It worked very, very nicely. The resulting files tested out well on both the PS3 and the Linux box.

So after that, I wrote up an index.txt file to add to each disc and a little shell scripting later and I had a directory for each disc. I started burning them with growisofs. Discs 1 and 2 burned well, but then I got an error like this:


File disc06/VIDEO/0930-inaug-ksnw-06.mpg is larger than 4GiB-1.
-allow-limited-size was not specified. There is no way do represent this file size. Aborting.

Eeeeepp. So apparently the ISO 9660 filesystem can’t represent files bigger than 4GB. My files on disc 1 had represented multiple different programs, and stayed under that limit; and disc 2’s file was surprisingly just a few KB short. But the rest of them weren’t. I didn’t want to have to go back and re-split the data to be under 4GB. I also didn’t want to waste 700MB per disc, or to have to make someone change video files every 15 minutes.

So I decided to investigate UDF, the filesystem behind Blu-Ray discs. mkisofs couldn’t make a pure UDF, only a hybrid 9660/UDF disc that risked compatibility issues with big files. There is a mkudffs, but it doesn’t take a directory of its own. So I wrote a script to do it. Note that this script may fail with dotfiles or files with spaces in them:

#!/bin/bash

set -e

if [ ! -d "$1" -o -e "$2" -o -z "$2" ]; then
   echo "Syntax: $0 srcdir destimage [volid]"
   echo "destimage must not exist"
   exit 5
fi

if [ "`id -u`" != "0" ]; then
   echo "This program must run as root."
fi

EXTRAARGS=""
if [ ! -z "$3" ]; then
   EXTRAARGS="--vid=$3"
fi

# Get capacities at http://en.wikipedia.org/wiki/DVD+R as of 9/27/2009

SECSIZE=2048

# I'm going to set it a few lower, than capacity 2295104 just in case.
# Must be at least one lower than the actual size for dd to do its thing.
# SECTORS=2295103
SECTORS=2295000

echo "Allocating image..."

dd if=/dev/zero "of=$2" bs=2048 "seek=$SECTORS" count=1

echo "Creating filesystem..."

mkudffs --blocksize=2048 $EXTRAARGS "$2"
mkdir "$2.mnt"

echo "Populating..."
mount -o rw,loop -t udf "$2" "$2.mnt"
cp -rvi "$1/"* "$2.mnt/"
echo "Unounting..."
umount "$2.mnt"
rmdir "$2.mnt"

echo "Done."

That was loosely based on a script I found on the Arch Linux site. But I didn’t like the original script, because it tried to do too much, wasted tons of time writing out 4.7GB of NULLs when it could have created a sparse file in an instant, and was interactive.

So there you have it. HD broadcast to playable on PS3, losslessly. Note that this info will also work equally well if you have a Bluray drive.

Resurrecting Old VHS Videos (and Panasonic DMR-EZ38VK Review)

I have a problem that I’m sure is pretty common. My parents used to rent a VHS camcorder from time to time. Not only that, but various school plays, musicals, etc. are on VHS tapes. As a result, they and I have a library of family memories on VHS. And it appears those tapes go as far back as 1987.

You might imagine there are several problems here. One is that VHS tapes degrade over time. Those that were recorded in EP mode (6 hours on a T-120 tape) are especially prone to this. I’ve been worried about how well those 22-year-old tapes will perform even now.

Another problem is that VHS tapes are getting hard to watch these days. We own a VCR, but it’s probably been 7 years since it was hooked up to anything on a regular basis.

So I have meant for some time to convert these old VHS recordings to DVD format. My initial plan was to use the PVR-250 hardware MPEG-2 encoder card that is used with MythTV to do that. But it’s in the basement, used with MythTV, and would generally be a hassle. As a result, I’ve been “meaning to do” this project for about 5 years, and haven’t.

Last night, I found that tape from 1987. It has a few priceless seconds of my grandpa Klassen on it — he passed away in 1990.

The Panasonic DMR-EZ38VK

I initially set out looking for a dedicated DVD recorder with an S-video input, but wound up buying one with an integrated VHS deck as well: the Panasonic DMR-EZ38VK.

I started with a DVD recorder review on CNet. I was primarily interested in video quality. Surprisingly, it seems there is significant difference in video quality among DVD recorders, which was what led me to the Panasonic line.

I was initially planning on a DMR-EA18K or DMR-EZ18K (the difference is whether or not they include a TV tuner). I was having trouble finding them in stock at the vendors I normally use, and wound up with the DMR-EZ38VK instead. B&H had a open-box demo unit at a special discount, so I snapped it up.

Video Quality

I’ve been recording most items to DVD in “SP” mode, which stores 2h per single-layer DVD. I’d concur with CNet: this produces spectacular results. I don’t think I’ve noticed any MPEG compression artifacts at all in this mode.

Some items, such as TV programs or home recordings with little motion, I’ve recorded in “LP” mode. This mode stores 4 hours on a single-layer DVD. It’s also surprisingly good, considering the amount of compression needed. I have noticed MPEG artifacts in that mode, though not to an extremely annoying degree.

The copying process

I start by popping an empty disc in the drive. Then I’ll put in the VHS tape and position it to the place where I want it to start copying. Then I hit Functions -> Copy -> VHS to DVD -> without finalizing, and away it goes. It automatically detects end-of-tape and helpfully won’t copy 6 hours of static.

When a tape is done copying, you can copy from more tapes to the disc, eject it and finalize it later, or work with it.

When I’m ready to finish a disc, I’ll go and change the “disc name”, which is what shows up at the top of the disc menu that the unit generates. If I feel ambitious, I might change the titles of individual titles as well. But all of this has to be done with an on-screen keyboard, and thus takes awhile, so I usually don’t. Finalizing commits the menu to disc and fixates it, and takes about a minute.

Track Detection

This feature is both a blessing and a curse.

The Panasonic recorder can often detect the break between a recording on a VHS. Newer VCRs would explicitly mark these, but it can detect it even with older camcorders with reasonable accuracy.

When it detects this, it creates a new title on the DVD. This takes a few seconds, so it also rewinds the VHS tape a few seconds, then starts copying again.

Unfortunately, if you’re just wanting to watch one long recording all the way through, this results in a few seconds being duplicated right before each scene transition, which is rather jarring. There is no way to disable this feature, either. The only workaround is to read from an external VCR. But if you do that, you lose the end-of-tape detection.

Generally I’ve decided to just live with it for now. It’s a cheap price to pay for an otherwise pretty good workflow.

Other annoyances

While copying, you can’t access the position indicators for either the VHS deck or the DVD recorder. So you don’t know how far along on the tape you are, or how much space is left on the DVD, until copying stops.

Also, it would be very nice to be able to tell it “copy 23 minutes and 15 seconds from VHS to DVD” when you know you don’t want to copy the whole tape.

The unit also has SD and USB ports for reading digital video. Frustratingly, a USB keyboard can’t be used to edit disc or track titles. That seems like an obvious and cheap feature to have.

Overall

Overall I am happy with the unit. It produces very good quality results, and is pretty easy to use overall. I don’t think I’d pick a different one if I had to do it again. But it could be made better for people that are copying large numbers of VHS tapes to DVD.

Generally, though, I can just start the copy and let it sit for a couple of hours, trusting it to do the reasonable thing with a tape. That’s convenient enough that I can get other things done while it’s copying, and takes little enough of my time that I’m actually working through stacks of tapes now.

Update 8/27 I have now tried some discs from this playing back on my PS3 connected to a 1080p HDTV. On that setup, compression artifacts are noticeable at the 2hr setting, and more are noticeable at the 4hr setting. I don’t think that they are any necessarily any more noticeable than any other home-produced DVD, though, especially on the SP setting. They had not been very visible on SD equipment.