How To Record High-Definition MythTV Files to DVD or Blu-Ray

MythTV,

I’ve long had a problem. Back on January 20, I took the day off work to watch the inauguration of Barack Obama. I saved the HD video recordings MythTV made of the day (off the ATSC broadcast), intending to eventually save them somehow. I hadn’t quite figured out how until recently, so there they sat: 9 hours of video taking up about 60GB of space on my disk.

MythTV includes a program called mytharchive that will helpfully transcode your files and burn a DVD from them. But it helpfully will transcode the beautiful 1920x1080i down to DVD resolution of — on a good day — 720×480. I couldn’t have that.

Background

My playback devices might be a PC, the MythTV, or the PlayStation 3. Of these, I figured the PS3 was going to be hardest to accommodate.

ATSC (HD) broadcasts in the United States are an MPEG Transport Stream (TS). Things are a bit complicated, because there may be errors in the TS due to reception problems, the resolution and aspect ratio may change multiple times (for instance, down to SD for certain commercials). And, I learned that some ATSC broadcasts are actually 1920×1088 because the vertical resolution has to be a multiple of a certain number, and those bottom 8 pixels shouldn’t be displayed.

Adding to the complexity, one file was 7 hours of video and about 50GB itself. I was going to have to do quite some splitting to get it onto 4.7GB DVD+Rs. I also didn’t want to re-encode any video, both for quality and for time reasons.

Attempts

So, I set out to try and figure out how to do this. My first approach was the sledgehammer one: split(1). split takes a large file and splits it on byte or line boundaries. It has no knowledge of MPEG files, so it may split them in the middle of frames. I figured that, worst case, I could always use cat to reassemble the file later.

Surprisingly, both mplayer and xine could play back any of these files, but the PS3 would only play back the first part. I remembered this as an option if all else failed.

Next, I tried avidemux. Quite the capable program — and I thought I could use it to cut my file into DVD-sized bits. But I couldn’t get it to let me copy the valid MPEG TS into another MPEG TS — it kept complaining of incompatible formats, but wouldn’t tell me in what way they were incompatible. I could get it to transcode to MPEG4, and produce a result that worked on the PS3, but that wasn’t really what I was after.

Then, I tried mpgsplit. Didn’t recognize the MPEG TS as a valid file, and even when I used a different tool to convert to MPEG PS, acted all suspicious as if it bought the MPEG from a shady character on a grungy street corner.

dvbcut

I eventually wound up using dvbcut to split up the ATSC (DVB) recordings. It understood the files natively and did exactly what I wanted. Well, *almost* exactly what I wanted. It has no command-line interface and didn’t have a way to split by filesize, but I calculated that about 35 minutes of the NBC broadcast and 56 minutes of the PBS broadcast could fit on a single DVD+R.

It worked very, very nicely. The resulting files tested out well on both the PS3 and the Linux box.

So after that, I wrote up an index.txt file to add to each disc and a little shell scripting later and I had a directory for each disc. I started burning them with growisofs. Discs 1 and 2 burned well, but then I got an error like this:


File disc06/VIDEO/0930-inaug-ksnw-06.mpg is larger than 4GiB-1.
-allow-limited-size was not specified. There is no way do represent this file size. Aborting.

Eeeeepp. So apparently the ISO 9660 filesystem can’t represent files bigger than 4GB. My files on disc 1 had represented multiple different programs, and stayed under that limit; and disc 2’s file was surprisingly just a few KB short. But the rest of them weren’t. I didn’t want to have to go back and re-split the data to be under 4GB. I also didn’t want to waste 700MB per disc, or to have to make someone change video files every 15 minutes.

So I decided to investigate UDF, the filesystem behind Blu-Ray discs. mkisofs couldn’t make a pure UDF, only a hybrid 9660/UDF disc that risked compatibility issues with big files. There is a mkudffs, but it doesn’t take a directory of its own. So I wrote a script to do it. Note that this script may fail with dotfiles or files with spaces in them:

#!/bin/bash

set -e

if [ ! -d "$1" -o -e "$2" -o -z "$2" ]; then
   echo "Syntax: $0 srcdir destimage [volid]"
   echo "destimage must not exist"
   exit 5
fi

if [ "`id -u`" != "0" ]; then
   echo "This program must run as root."
fi

EXTRAARGS=""
if [ ! -z "$3" ]; then
   EXTRAARGS="--vid=$3"
fi

# Get capacities at http://en.wikipedia.org/wiki/DVD+R as of 9/27/2009

SECSIZE=2048

# I'm going to set it a few lower, than capacity 2295104 just in case.
# Must be at least one lower than the actual size for dd to do its thing.
# SECTORS=2295103
SECTORS=2295000

echo "Allocating image..."

dd if=/dev/zero "of=$2" bs=2048 "seek=$SECTORS" count=1

echo "Creating filesystem..."

mkudffs --blocksize=2048 $EXTRAARGS "$2"
mkdir "$2.mnt"

echo "Populating..."
mount -o rw,loop -t udf "$2" "$2.mnt"
cp -rvi "$1/"* "$2.mnt/"
echo "Unounting..."
umount "$2.mnt"
rmdir "$2.mnt"

echo "Done."

That was loosely based on a script I found on the Arch Linux site. But I didn’t like the original script, because it tried to do too much, wasted tons of time writing out 4.7GB of NULLs when it could have created a sparse file in an instant, and was interactive.

So there you have it. HD broadcast to playable on PS3, losslessly. Note that this info will also work equally well if you have a Bluray drive.

Town Hall Questions

Uncategorized

Sen. Brownback (R-KS) will be at my employer Monday, and will have a short town hall session. I’m debating whether to go or not, and whether to say anything or not. I don’t agree with him on much, and highly doubt that I’d change his mind on anything. Should I go? If so, what should I say?

Here are some random facts I’m considering mentioning:

  • When Jacob was born 3 years ago, it cost us $250. When Oliver was born this summer, it cost us around $3000, and Oliver wasn’t a more complicated pregnancy.
  • Most of the past 8 years, my insurance premiums have gone up and my benefits have gone down.
  • If I lost my coverage, I couldn’t afford to buy it on my own, even if I still had my job. Insuring our family on the individual market would cost more each month than our mortgage.
  • Almost as much (80%) of my paycheck goes to health care as to federal taxes. I’d gladly take a tax increase if it slowed growth in health care costs.
  • Brownback says it’s too expensive to to comprehensive reform right now. The estimated cost of reform is $900 billion over 10 years, and it will be paid for.
  • Brownback voted for the Bush tax cuts, which cost $2.5 trillion over 10 years, yes to attack Iraq ($700 billion so far), yes on Medicare prescription drug coverage ($400 billion). He didn’t vote for a way to pay for any of these.

Suggestions?

One final note: I will not be doing anything disruptive or disrespectful.

Late Summer

Uncategorized

It’s that time of the year again. Everything is changing, and maybe for the better.

The days are getting shorter. When I left for work on my bicycle yesterday morning, it was still dark outside, and a little nippy. There’s nothing quite like riding a bicycle down a deserted country road at night, a cool breeze at your back, and having the sun come up as you ride.

And with the cooler weather, we can open our windows at night instead of running the air conditioner. It’s also nice to have a pleasant cool breeze flowing through the house, and hear the frogs, crickets, owls, coyotes, and other wildlife at night. Out here, we certainly don’t hear sounds of traffic, or loud car radios, though on a really clear night we might hear the rumble and whistle of a train from a few miles away.

This is also sunflower season in Kansas. The wild sunflowers, when their smaller-than-most-people-think flowers, grow everywhere. Some ditches turn into a sea of person-height yellow. Sunflowers are on the sides of bridges, around people’s mailboxes — just about anywhere that isn’t mowed or farmed. Then you also pass the sunflower fields, with their larger flowers, even more sea-like.

The beans are getting tall in the fields this time of year, and it won’t be long before the milo starts to turn its deep, dark reddish brown.

But, you know, we’re Kansans. We can’t really let ourselves enjoy it all that much. Just today, I heard a conversation — apparently the Farmer’s Almanac is predicting a brutally frigid winter this year. Gotta keep our sense of pessimism about weather alive now…

Google Groups Fail

Uncategorized

Last month, I wrote that I was looking for mailing list hosting. It looks like some of the lists I host will move to Alioth, and some to Google Groups.

Google Groups has a nice newbie-friendly interface with the ability to read a group as a forum or as a mailing list. Also, they don’t have criteria about the subject matter of a group, so the Linux user’s group list I host could be there but not at Alioth.

So I set up a Google Group for the LUG. I grabbed the subscriber list from Ecartis, and went to “Directly Add” the members. This was roughly August 12.

After doing so, I got a message saying that Google needs to review these requests to make sure they’re legit, and will get back to me in 1-2 days. OK, that’s reasonable.

Problem is, nobody appears to be reviewing them. This is three weeks later and no action.

So I decided I would ask someone at Google about it. The only way they give to do that is to post in the Google Groups Help Forum. So I did. Guess what? They ignore that, too.

Let me say: relying on this sort of service for something important really makes me think twice. It makes me nervous about using Google Voice (what if my Google Voice number goes down?) It certainly makes me think twice about ever relying on Gmail or the other Apps for anything important.

My own mail server may not have the features that theirs does, but if it breaks, I don’t have to worry about whether anybody even cares to fix it.

Looking for a Linux-compatible scanner

Uncategorized

I own two scanners: a Fujitsu ScanSnap S510 and an Epson Perfection 4180 Photo. The S510 is a sheetfed document scanner, and works great at that. It has perfect SANE support, duplex mode, excellent sheet feeder, and is a generally good document scanner. It’s passable for photos, but only that. The color isn’t great, and the precision of a sheetfed scanner just isn’t up to a flatbed.

The Epson has never been supported by SANE directly. There was an epkowa driver, but the 4180 epkowa driver hasn’t been updated in ages and isn’t compatible with any modern Linux distro.

So, I’m looking for a flatbed scanner for photos and documents (no need for negatives; that’s the THIRD scanner on my desk.) The most important requirement is that it work well with Linux. The next most important requirement is that it have as good scanning quality as possible for an under-$200 scanner.

If it’s an all-in-one (with a printer), I’m fine with that, as long as it meets the above requirements. Any suggestions?

Preschool Update

Uncategorized

Tuesday was open house at the preschool Jacob will start attending when he turns 3. It’s the same preschool I went to, and the only one in the small town closest to us. It’s been owned and operated by the same person for all of the 30 years it’s existed.

We got out of the car when we got there, and Jacob went it full explore mode. He walked a few feet, and happily said “Ooo! There a ditch here! It has water sometimes.” We went inside, and he looked around for about 5 seconds, then was off. It didn’t take him long to find the toy train, the bulldozer, and the school’s bird. He was so busy, in fact, that he didn’t even notice the snacks that were set out until I pointed them out to him.

It’s hard to describe the building. You walk in and you immediately understand that the owner knows a lot about kids, and providing them a creative, interesting, and educational environment. In one corner, there’s a record player that’s probably older than the preschool, and a CD player right above it. Then there’s a little reading nook with some books, headphones, and cassette player. I’m sure Jacob will enjoy learning about cassettes and records, which we don’t really have out at home anymore.

The owner’s husband built most of the furniture for the preschool himself 30 years ago. He figured it would last maybe 10 years, but is still holding up well and gets a new coat of paint every few years. So Jacob will be sitting on the same chairs I used to sit on.

Most importantly, there is no TV to be seen anywhere in the preschool. No computer in there, either. But there is a large outdoor playground — the fun type that I used to enjoy as a child, not the boring plastic type that I see so much these days.

It reminds me a lot of Mr. Rogers. There isn’t expensive technology or a fancy building, but good old-fashioned creative play. Just what children need. I’ll be very happy that Jacob will be there, and I think he will be too.

Resurrecting Old VHS Videos (and Panasonic DMR-EZ38VK Review)

Family, Reviews, TV,

I have a problem that I’m sure is pretty common. My parents used to rent a VHS camcorder from time to time. Not only that, but various school plays, musicals, etc. are on VHS tapes. As a result, they and I have a library of family memories on VHS. And it appears those tapes go as far back as 1987.

You might imagine there are several problems here. One is that VHS tapes degrade over time. Those that were recorded in EP mode (6 hours on a T-120 tape) are especially prone to this. I’ve been worried about how well those 22-year-old tapes will perform even now.

Another problem is that VHS tapes are getting hard to watch these days. We own a VCR, but it’s probably been 7 years since it was hooked up to anything on a regular basis.

So I have meant for some time to convert these old VHS recordings to DVD format. My initial plan was to use the PVR-250 hardware MPEG-2 encoder card that is used with MythTV to do that. But it’s in the basement, used with MythTV, and would generally be a hassle. As a result, I’ve been “meaning to do” this project for about 5 years, and haven’t.

Last night, I found that tape from 1987. It has a few priceless seconds of my grandpa Klassen on it — he passed away in 1990.

The Panasonic DMR-EZ38VK

I initially set out looking for a dedicated DVD recorder with an S-video input, but wound up buying one with an integrated VHS deck as well: the Panasonic DMR-EZ38VK.

I started with a DVD recorder review on CNet. I was primarily interested in video quality. Surprisingly, it seems there is significant difference in video quality among DVD recorders, which was what led me to the Panasonic line.

I was initially planning on a DMR-EA18K or DMR-EZ18K (the difference is whether or not they include a TV tuner). I was having trouble finding them in stock at the vendors I normally use, and wound up with the DMR-EZ38VK instead. B&H had a open-box demo unit at a special discount, so I snapped it up.

Video Quality

I’ve been recording most items to DVD in “SP” mode, which stores 2h per single-layer DVD. I’d concur with CNet: this produces spectacular results. I don’t think I’ve noticed any MPEG compression artifacts at all in this mode.

Some items, such as TV programs or home recordings with little motion, I’ve recorded in “LP” mode. This mode stores 4 hours on a single-layer DVD. It’s also surprisingly good, considering the amount of compression needed. I have noticed MPEG artifacts in that mode, though not to an extremely annoying degree.

The copying process

I start by popping an empty disc in the drive. Then I’ll put in the VHS tape and position it to the place where I want it to start copying. Then I hit Functions -> Copy -> VHS to DVD -> without finalizing, and away it goes. It automatically detects end-of-tape and helpfully won’t copy 6 hours of static.

When a tape is done copying, you can copy from more tapes to the disc, eject it and finalize it later, or work with it.

When I’m ready to finish a disc, I’ll go and change the “disc name”, which is what shows up at the top of the disc menu that the unit generates. If I feel ambitious, I might change the titles of individual titles as well. But all of this has to be done with an on-screen keyboard, and thus takes awhile, so I usually don’t. Finalizing commits the menu to disc and fixates it, and takes about a minute.

Track Detection

This feature is both a blessing and a curse.

The Panasonic recorder can often detect the break between a recording on a VHS. Newer VCRs would explicitly mark these, but it can detect it even with older camcorders with reasonable accuracy.

When it detects this, it creates a new title on the DVD. This takes a few seconds, so it also rewinds the VHS tape a few seconds, then starts copying again.

Unfortunately, if you’re just wanting to watch one long recording all the way through, this results in a few seconds being duplicated right before each scene transition, which is rather jarring. There is no way to disable this feature, either. The only workaround is to read from an external VCR. But if you do that, you lose the end-of-tape detection.

Generally I’ve decided to just live with it for now. It’s a cheap price to pay for an otherwise pretty good workflow.

Other annoyances

While copying, you can’t access the position indicators for either the VHS deck or the DVD recorder. So you don’t know how far along on the tape you are, or how much space is left on the DVD, until copying stops.

Also, it would be very nice to be able to tell it “copy 23 minutes and 15 seconds from VHS to DVD” when you know you don’t want to copy the whole tape.

The unit also has SD and USB ports for reading digital video. Frustratingly, a USB keyboard can’t be used to edit disc or track titles. That seems like an obvious and cheap feature to have.

Overall

Overall I am happy with the unit. It produces very good quality results, and is pretty easy to use overall. I don’t think I’d pick a different one if I had to do it again. But it could be made better for people that are copying large numbers of VHS tapes to DVD.

Generally, though, I can just start the copy and let it sit for a couple of hours, trusting it to do the reasonable thing with a tape. That’s convenient enough that I can get other things done while it’s copying, and takes little enough of my time that I’m actually working through stacks of tapes now.

Update 8/27 I have now tried some discs from this playing back on my PS3 connected to a 1080p HDTV. On that setup, compression artifacts are noticeable at the 2hr setting, and more are noticeable at the 4hr setting. I don’t think that they are any necessarily any more noticeable than any other home-produced DVD, though, especially on the SP setting. They had not been very visible on SD equipment.

Prebuilt PC suggestions?

Uncategorized

My MythTV box died today, and I think I would like to buy a new workstation for myself, and cascade my old workstation to the MythTV.

Trouble is, I don’t have time to spec out components and build it myself this time. I haven’t bought a prebuilt PC for myself since 6 years ago, and don’t really know what’s good.

My general requirements are: works with Linux, has a flexible BIOS (many of the low-end home PCs have a limited BIOS with few options), is a quiet and energy-efficient as possible, quad-core, 2GB or more RAM, SATA. Nice to have would be fanless nVidia video, but I could also take one with cheap onboard video and move an existing PCIe card to it.

Suggestions?

Mailing List Hosting

Online Life

I’ve hosted email lists of one sort or another probably all the way back to 1995, when I first bought complete.org as an email-only domain fed off a UUCP connection on a long-distance dialup link.

I’ve only used two mailing list hosting programs in that whole time: Majordomo and Ecartis (used to be known as Listar). Unfortunately, Ecartis has not seen upstream work in several years, and as a result was removed from Debian in May.

That got me to thinking: what am I going to do with the mailing lists I host? I’m not pleased with my current list archives, which are very similar to what you get from Mailman: no search engine, and every thread is broken at the end of each month.

I also want to preserve my archives.

I’m presently looking at whether to continue hosting the lists myself, or turn to something like Google Groups or Nabble. Hosting it myself, the main choice is Mailman, which really has more features than I need in most areas, and fewer than I need for archiving.

For other hosts, I’ve looked at Google Groups, Yahoo Groups, and Nabble. Google Groups looks like the best option, and even has a (somewhat hidden) way to subscribe via pure email without having a Google account. They can import and export subscriber lists, though not archives.

I’m thinking I’ll also make sure all the lists have full archives at Gmane. Then, based on message IDs, I can generate a bunch of RedirectPermanent lines for Apache to links to the archives don’t get broken.

My current thought for list hosting itself is Google Groups. It would be nice to be free of the hassle of administering a mailing list host, which is nothing special these days. Another benefit of Google Groups is that those people that like web forums (who ARE those people anyway?) get a forum-like interface to the list if they so choose.

Nabble has some interesting features, and can optionally import a full history of a list, but it concentrates far more on the forum than the email aspects. It doesn’t even appear to have basic moderation settings.

The Thrilling Conclusion of Goerzen vs. Dell: Sweet, Sweet Victory

General

When United Airlines recently broke some expensive guitars but refused to pay for their negligence, the owner of the guitars made a Youtube video. United corporate HQ noticed, and were so embarrassed that they fixed things.

I’ve had some trouble with Dell breaking the law, and their corporate HQ noticed, were embarrassed, but didn’t bother fixing things.

However, I have discovered something that Dell does care about: FEDERAL PROSECUTORS.

I Hate Junk Mail

Before continuing, I need to answer a FAQ: why I hate junk mail. It’s bad for the environment, takes time to process, and fills up my recycling bins. We only get our recycling picked up once a month (we’re lucky to get that where we live), and I hate filling them up with catalogs for things I’ll never use. Also junk mail has a way of multiplying like rabbits. Get on one list, and pretty soon you’re on dozens.

Normally when I get junk mail, I’ll find the website or call the company that sent it to me and ask to be removed. And then they will stop sending me junk mail.

That approach has worked with every single company that I’ve tried it on. With one exception: Dell. Even though ignoring my requests puts them in violation of their own privacy policy.

The Story So Far

It’s been a little while since I’ve written about this, so here’s the condensed version. Click the links for more details.

Back in early 2007 — yes, more than 2 years ago — I had a lapse of judgement and tried to get a Dell monitor serviced under warranty. After a frustrating evening of trying to explain to them that I have a Dell monitor but not a Dell PC, they finally agreed to fix it. And put me on their “flamingo pink Inspiron catalog” mailing list.

I went to their website trying to get off the list. They have many different list removal forms, and I tried them all. I called them. I even got a comment from Debbie at Dell HQ in Texas, offering to try to help. Despite repeated attempts, she didn’t (or couldn’t).

So, in December of 2007, I decided to let Jacob rip apart my junk mail (with associated cute photos).

By August 2008, I still wasn’t off their list. I tried everything, and Dell customer service replied to my request to be REMOVED from their snail mail list by saying they would ADD me to their email list. Lovely.

So I finally obtained a prohibitory order (see scanned copy on that link) in July 2008, which enforces federal law (39 USC 3008) prohibiting Dell from mailing me any more of those catalogs. From August 25, 2008 on, it was a federal offense for Dell to send me any more catalogs.

Guess how successful that was. By September 2008, they were back at their old tricks, sending me catalogs.

The New Bits

So — I sent in a couple these catalogs to the USPS as evidence of violation. By February, I received this letter, which made me Very Happy:

dell-court-order

(see also larger version)

Yes, that’s right. The United States Postal Service went to court to obtain a court order against Dell, prohibiting them from sending me more catalogs.

And — it was successful! It’s been several months since I’ve received any more catalogs from Dell.

It took two years (it wouldn’t have had to, but I didn’t push things along very fast from my end, giving them lots of time to comply each step of the way), but I am finally free of Dell mailings.

I suspect some federal attorneys in some remote office somewhere owe their jobs to Dell’s noncompliance of postal and privacy regulations.

Now if only I can get Rep. Tiahrt to stop sending me junk mail… He keeps sending me literature, and I don’t even live in his district.