February 11th, 2015
Apparently I touched a nerve with my recent post about the growing complexity of issues.
There were quite a few good comments, which I’ll mention here. It’s provided me some clarity on the problem, in fact. I’ll try to distill a few more thoughts here.
The value of simplicity and predictability
The best software, whether it’s operating systems or anything else, is predictable. You read the documentation, or explore the interface, and you can make a logical prediction that “when I do action X, the result will be Y.” grep and cat are perfect examples of this.
The more complex the rules in the software, the more hard it is for us to predict. It leads to bugs, and it leads to inadvertant security holes. Worse, it leads to people being unable to fix things themselves — one of the key freedoms that Free Software is supposed to provide. The more complex software is, the fewer people will be able to fix it by themselves.
Now, I want to clarify: I hear a lot of talk about “ease of use.” Gnome removes options in my print dialog box to make it “easier to use.” (This is why I do not use Gnome. It actually makes it harder to use, because now I have to go find some obscure way to just make the darn thing print.) A lot of people conflate ease of use with ease of learning, but in reality, I am talking about neither.
I am talking about ease of analysis. The Linux command line may not have pointy-clicky icons, but — at least at one time — once you understood ls -l and how groups, users, and permission bits interacted, you could fairly easily conclude who had access to what on a system. Now we have a situation where the answer to this is quite unclear in terms of desktop environments (apparently some distros ship network-manager so that all users on the system share the wifi passwords they enter. A surprise, eh?)
I don’t mind reading a manpage to learn about something, so long as the manpage was written to inform.
With this situation of dbus/cgmanager/polkit/etc, here’s what it feels like. This, to me, is the crux of the problem:
It feels like we are in a twisty maze, every passage looks alike, and our flashlight ran out of battieries in 2013. The manpages, to the extent they exist for things like cgmanager and polkit, describe the texture of the walls in our cavern, but don’t give us a map to the cave. Therefore, we are each left to piece it together little bits at a time, but there are traps that keep moving around, so it’s slow going.
And it’s a really big cave.
Other user perceptions
There are a lot of comments on the blog about this. It is clear that the problem is not specific to Debian. For instance:
- Christopher writes that on Fedora, “annoying, niggling problems that used to be straightforward to isolate, diagnose and resolve by virtue of the platform’s simple, logical architecture have morphed into a morass that’s worse than the Windows Registry.” Alessandro Perucchi adds that he’s been using Linux for decades, and now his wifi doesn’t work, suspend doesn’t work, etc. in Fedora and he is surprisingly unable to fix it himself.
- Nate bargman writes, in a really insightful comment, “I do feel like as though I’m becoming less a master of and more of a slave to the computing software I use. This is not a good thing.”
- Singh makes the valid point that this stuff is in such a state of flux that even if a person is one of the few dozen in the world that understand what goes into a session today, the knowledge will be outdated in 6 months. (Hal, anyone?)
This stuff is really important, folks. People being able to maintain their own software, work with it themselves, etc. is one of the core reasons that Free Software exists in the first place. It is a fundamental value of our community. For decades, we have been struggling for survival, for relevance. When I started using Linux, it was both a question and an accomplishment to have a useable web browser on many platforms. (Netscape Navigator was closed source back then.) Now we have succeeded. We have GPL-licensed and BSD-licensed software running on everything from our smartphones to cars.
But we are snatching defeat from the jaws of victory, because just as we are managing to remove the legal roadblocks that kept people from true mastery of their software, we are erecting technological ones that make the step into the Free Software world so much more difficult than it needs to be.
We no longer have to craft Modelines for X, or compile a kernel with just the right drivers. This is progress. Our hardware is mostly auto-detected and our USB serial dongles work properly more often on Linux than on Windows. This is progress. Even our printers and scanners work pretty darn well. This is progress, too.
But in the place of all these things, now we have userspace mucking it up. We have people with mysterious errors that can’t be easily assisted by the elders in the community, because the elders are just as mystified. We have bugs crop up that would once have been shallow, but are now non-obvious. We are going to leave a sour taste in people’s mouth, and stir repulsion instead of interest among those just checking it out.
The ways out
It’s a nasty predicament, isn’t it? What are your ways out of that cave without being eaten by a grue?
Obviously the best bet is to get rid of the traps and the grues. Somehow the people that are working on this need to understand that elegance is a feature — a darn important feature. Sadly I think this ship may have already sailed.
Software diagnosis tools like Enrico Zini’s seat-inspect idea can also help. If we have something like an “ls for polkit” that can reduce all the complexity to something more manageable, that’s great.
The next best thing is a good map — good manpages, detailed logs, good error messages. If software would be more verbose about the permission errors, people could get a good clue about where to look. If manpages for software didn’t just explain the cavern wall texture, but explain how this room relates to all the other nearby rooms, it would be tremendously helpful.
At present, I am unsure if our problem is one of very poor documentation, or is so bad that good documentation like this is impossible because the underlying design is so complex it defies being documented in something smaller than a book (in which case, our ship has not just sailed but is taking on water).
One theme that came up often in the comments is that this is necessary for progress. To a certain extent, I buy that. I get why udev is important. I get why we want the DE software to interact well. But here’s my thing: this already worked well in wheezy. Gnome, XFCE, and KDE software all could mount/unmount my drives. I am truly still unsure what problem all this solved.
Yes, cloud companies have demanding requirements about security. I work for one. Making security more difficult to audit doesn’t do me any favors, I can assure you.
The systemd angle
To my surprise, systemd came up quite often in the discussion, despite the fact that I mentioned I wasn’t running systemd-sysv. It seems like the new desktop environemt ecosystem is “the systemd ecosystem” in a lot of people’s minds. I’m not certain this is justified; systemd was not my first choice, but as I said in an earlier blog post, “jessie will still boot”.
A final note
I still run Debian on all my personal boxes and I’m not going to change. It does awesome things. For under $100, I built a music-playing system, with Raspberry Pis, fully synced throughout my house, using a little scripting and software. The same thing from Sonos would have cost thousands. I am passionate about this community and its values. Even when jessie releases with polkit and all the rest, I’m still going to use it, because it is still a good distro from good people.
February 9th, 2015
For years, I used to run Debian sid (unstable) on all my personal machines. Laptops, workstations, sometimes even my personal servers years ago ran sid. Sid was, as its name implies, unstable. Sometimes things broke. But it wasn’t a big deal, because I could always get in there and fix it fairly quickly, whatever it was. It was the price I paid for the latest and greatest.
For the last number of months, I’ve dealt with a small but annoying issue in jessie: None of Nautilus, Thunar, or digikam (yes, that represents Gnome, XFCE, and KDE) can mount USB drives I plug in anymore. I just get “Not authorized to perform operation.” I can, of course, still mount -o uid=1000 /dev/sdc1 /mnt, but I miss the convenience of doing it this way.
One jessie system I switched to systemd specifically to get around this problem. It worked, but I don’t know why. I haven’t had the time to switch my workstation, and frankly I am concerned about it.
Here’s the crux of the issue: I don’t even know where to start looking. I’ve googled this issue, and found all sorts of answers pointing to polkit, or dbus, or systemd-shim, or cgmanager, or lightdm, or XFCE, or… I found a bug report of this exact problem — Debian #760281, but it’s marked fixed, and nobody replied to my comment that I’m still seeing it.
Nowhere is it documented that a Digikam mounting issue should have me looking in polkit, let alone cgmanager. And even once I find those packages, their documentation suffers from Bad Unix Documentation Disease: talking about the nitty-gritty weeds view of something, without bothering to put it in context. Here is the mystifying heading for the cgmanager(8) manpage:
cgmanager is a daemon to manage cgroups. Programs and users can make D-Bus requests to administer cgroups over which they have privilege. To ensure that users may not exceed their privilege in manipulating cgroups, the cgroup manager accepts regular D-Bus requests only from tasks within its own process-id and user namespaces. For tasks in private namespaces (such as containers), SCM-enhanced D-Bus calls are available. Using these manually is not recommended. Rather, each container is advised to run a cgproxy, which will forward plain D-Bus requests as SCM-enhanced D-Bus requests to the host cgmanager.
That’s about as comprehensible as Vogon poetry to me. How is cgmanager started? What does “SCM-enhanced” mean? And I even know what a cgroup is.
This has been going on for months, which has me also wondering: is it only me? (Google certainly suggests it’s not, and there are plenty of hits for this exact problem with many distros, and some truly terrible advice out there to boot.) And if not, why is something so basic and obvious festering for so long? Have we built something that’s too complex to understand and debug?
This is, in my mind, orthogonal to the systemd question. I used to be able to say Linux was clean, logical, well put-together, and organized. I can’t really say this anymore. Users and groups are not really determinitive for permissions, now that we have things like polkit running around. (Yes, by the way, I am a member of plugdev.) Error messages are unhelpful (WHY was I not authorized?) and logs are nowhere to be found. Traditionally, one could twiddle who could mount devices via /etc/fstab lines and perhaps some sudo rules. Granted, you had to know where to look, but when you did, it was simple; only two pieces to fit together. I’ve even spent time figuring out where to look and STILL have no idea what to do.
systemd may help with some of this, and may hurt with some of it; but I see the problem more of an attitude of desktop environments to add features fast without really thinking of the implications. There is something to be said for slower progress if the result is higher quality.
Then as I was writing this, of course, my laptop started insisting that it needed the root password to suspend when I close the lid. And it’s running systemd. There’s another quagmire…
Update: Part 2 with some reactions to this and further thoughts is now available.
January 31st, 2015
In my part 1 post yesterday, I wrote about the start of the home automation project. I mentioned that I was using Insteon switches, and they mostly were working well (I forgot to mention an annoyance: you can dim, but not totally shut off, their LED status light.) Anyhow, the Insteon battery-operated sensors seem to not be as good as their Z-Wave competition.
Setting up Z-Wave
Z-Wave devices get joined to the Z-Wave network in much the same way Bluetooth devices get paired with each other. The first time you use a Z-Wave device, you join it to the controller. The controller assigns it an ID on your network, and both devices discover the best route to communicate with each other.
I discovered that the Z-Wave module on the ISY-994i has particularly poor reception. Combined with the generally short range of battery-operated Z-wave devices, this meant that my sensors didn’t work reliably. As with Insteon, AC-powered Z-wave devices tend to be repeaters, but I didn’t have any AC-powered Z-wave devices. I went looking for Z-wave repeaters, and found some. But then I discovered that a Z-wave relay-based appliance switch was actually $5 cheaper, acted as a repeater, and could be used as a switch down the road if needed. A couple of those solved my communications issues. You join them to the network as usual, but then either re-join the battery-powered sensors (so they see the new route) or do a “network heal” (every device on the network re-learns about its neighbors and routes to the other devices) so they see the new route.
Z-Wave Motion / Occupancy Sensors
If you have been in new buildings, chances are you have seen light switches with built-in occupancy sensors. My doctor’s office has these. They are typically the same size as a regular light switch, but with a passive infrared (PIR) motion sensor used to automatically turn the lights off after a timeout (or even on, when someone walks into the room.) These work fine for small rooms, but if you’ve ever been in a bathroom with a PIR lightswitch that goes off while you’re still in the room, you are aware of their faults for larger rooms!
Most of the time, when you read about “occupancy sensors”, it’s really a PIR sensor, and the term is used interchangeably with “motion detector.” Motion detectors in home automation systems are commonly used for a number of purposes: detecting occuption of rooms for automatic control of lighting or HVAC systems, triggering alerts, or even locking or unlocking doors.
My friend told me he had poor luck with the Insteon PIR sensors, so I tried two different Z-Wave models: the $48 Aeon/Aeotec multi-sensor and the $30 Ecolink PIRZWAVE2-ECO. The Aeon device is clearly the more capable; it can be used indoors or out, and has a lot of options that can be configured by Z-Wave configuration parameters. It also has a ball/socket mount, so it can be easily aimed in different directions. It can draw power from 4 AAA batteries, or a mini USB cable (cable, but not power supply, included). The Aeon multi-sensor also has a temperature, humidity, and illumination (lux) sensor – but, as you’ll see, they have some drawbacks.
The Ecolink device is more basic; it has a few settings that can be altered via jumpers, but none that can be altered via Z-Wave configuration commands. That makes it a bit of a hassle, since you have to open it up to change, and when you do, it triggers a “tamper alarm” that is both undocumented and never seems to go away. It is powered by a single CR123A 3V Lithium battery, which are about $2 each on Amazon.
Both units have a default PIR timeout of 4 minutes. That means that after sensing motion, they will transmit the “on” signal (meaning motion detected) and then transmit no further signals for at least 4 minutes. This is because operating the radio consumes far more battery power than simply monitoring the PIR sensor, and it cuts down on repeated on/off transmissions. In this configuration, both units should have battery life of 6 months to a year, I figure, with an edge for the Ecolink, perhaps.
Both can also be configured to transmit more frequently; the Ecolink has a jumper that can change its timeout to 10 seconds, whereas the Aeon can be configured over Z-Wave for any timeout between 10 seconds and 65535 seconds (values above 4 minutes are rounded to the nearest minute, for some reason.)
Both also have adjustable sensitivity; on the Aeon this is via an adjustment knob, and on the Ecolink it’s via jumpers. And both can report their battery level to enable software alerts when it’s getting low.
The Aeon ships with its temperature, humidity, battery, and lux sensors disabled. This appears to be the cause of much confusion online, as they send one transmission and then no more. Sadly, one has to resort to the hard-to-find but very helpful MultiSensor engineering spec document to figure out the way to enable those sensors and set their interval. (It must be said, however, that I doubt most consumers will understand how to set bitfields, and this is either not covered or covered incorrectly in their other manuals.) This can be a real power drain, so I just set it to report battery level every 6 hours (so I can alert myself if it gets low) and leave it at that.
The Ecolink manual claims a detection radius of 39 feet and a total angle of 90deg (45deg left or right from center) and a 3-year battery life (I’m skeptical). It also is rated for indoor use only. Aeotec claims a detection radius of 16 feet and a total angle of 120 deg. Be skeptical of all these figures.
Once set up with good reception, both devices have been working fine.
Programming the controller
The ISY-994i-Zw Pro controller I mentioned in Part 1 has its own sort of programming language. It’s a lot better than the sort of GUI clicky mess that is found in most of these things, but it still has a sort of annoying Java-based editor where you select keywords from a menu and such. Although you can backup and restore the device, and import and export the programs, the file formats are XML and not really suitable for hand-editing. Sigh.
The language is limited, but gets the job done. Here is a simple program that turns off the fan in a bathroom 30 minutes after the light was turned off:
Status '1st Floor / Bathroom + Laundry / Main Bath Fan' is On
And Status '1st Floor / Bathroom + Laundry / Main Bath Light' is Off
Wait 30 minutes
Set '1st Floor / Bathroom + Laundry / Main Bath Fan' Off
- No Actions - (To add one, press 'Action')
The if/then/else clauses should probably be more properly called when/do/finally. The “if” clauses are evaluated whenever a relevant event occurs. If the If evaluates to true, if executes the Then section. The program in “Then” is uninterruptible except for “wait” and “repeat” statements. So in this case, the program begins, and if the light stays off and the fan stays on, 30 minutes later it turns off the fan. But if the light comes back on, the program aborts (since there is nothing in “else”). Similarly, if the fan goes off, the program aborts.
A more complicated example: motion-activated lamp
There is a table lamp in our living room controlled by Insteon. By adding a motion sensor in that room, it can be automatically turned on by someone walking through the room. Now, to be useful, I don’t want it to turn on during the day. I also don’t want it to turn on or adjust itself if other lights are on in the room, or if I turned it on myself; that could cause it to go on and off while I’m watching TV, for instance. It’s just to be helpful at night. Because the ISY-994i is pretty limited, having almost no control flow operations, this — as with many tasks — requires several “programs”.
First, here’s my “main”:
Status '1st Floor / Living + Dining Room / LR Table Lamp' is Off
And $LR_Lamp_Lockout is 0
And Status '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is On
And From Sunset
To Sunrise (next day)
And $LR_Lamp_Motion_Working is not 0
And Status '1st Floor / Living + Dining Room / LR Floor Lamp' is Off
And Status '1st Floor / Living + Dining Room / Dining Light' is Off
And Status '1st Floor / Living + Dining Room / LR Light' is Off
Set '1st Floor / Living + Dining Room / LR Table Lamp' 22%
Set '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' 22%
- No Actions - (To add one, press 'Action')
So, let’s look at the conditions. This program triggers if the lamp is off, the sensor is on, the time is between sunset and sunrise, and three other lights are off. (I will explain the lockout and motion_working variables later). If this is the case, it sets the lamp to 22% (and also informs the wall switch for it that the lamp is at 22%). I pick this precise value because it is unlikely I would manually set it via the wall switch, and therefore “lamp is 22% bright” doubles as a “lamp was turned on by this program” flag.
So this is the turning the lamp on bit. Let’s look at the program that turns it back off later:
Status '1st Floor / Living + Dining Room / LR Table Lamp' is 22%
Status '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is Off
Or $LR_Lamp_Motion_Working is 0
Or Status '1st Floor / Living + Dining Room / LR Light' is not Off
Or Status '1st Floor / Living + Dining Room / LR Floor Lamp' is not Off
Or Status '1st Floor / Living + Dining Room / Dining Light' is not Off
Wait 15 seconds
Set '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' Off
Set '1st Floor / Living + Dining Room / LR Table Lamp' Off
- No Actions - (To add one, press 'Action')
This is using a sensor with a 4-minute timeout, so the lamp will always be on for at least 4 minutes and 15 seconds. This program runs if the lamp is still at the program-set level (so if, for instance, I turned the lamp full on, the program does nothing to override my setting.) Then, it looks for a condition to trigger turning the lamp off, which could be any of the sensor indicating no more motion, another program detecting it’s lost communication with the sensor, or somebody turning on one of the bigger lights in the room. Then it simply sets the light (and the wall switch controlling it) to off.
There are a couple more bits to this puzzle. What if the system turned the lamp on, but I really want it off? If I walked up to the wall switch and pushed “off”, the lamp would go off. And then, a couple seconds later, come back on, since the state of the system met the conditions for the lamp-on program. So we need a lockout that prevents this from happening. Here’s my “trigger lockout” program:
Control '1st Floor / Living + Dining Room / LR Table Lamp' is switched Off
Or Control '1st Floor / Living + Dining Room / LR Table Lamp' is switched Fast Off
Or Control '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' is switched Fast Off
Or Control '1st Floor / Living + Dining Room / LR S Remote (Table Lamp)' is switched Off
$LR_Lamp_Lockout += 1
Run Program 'LR Lamp Clear Lockout' (If)
- No Actions - (To add one, press 'Action')
So if someone pushes the “off” button at the lamp switch box at the outlet it’s plugged into (unlikely), or at the wall switch, it increments the lockout variable and runs another program. This other program is unique in that it is flagged “disabled”, meaning it is never run automatically by the system, only when called by another program. Here’s the clear lockout program:
$LR_Lamp_Lockout > 0
Wait 5 minutes
$LR_Lamp_Lockout = 0
- No Actions - (To add one, press 'Action')
Thus by pushing the “off” button on the switch, the motion-triggered program won’t turn the lamp back on for at least 5 minutes.
Before I had reliable Z-Wave communication to the device, I had some times where it would simply drop off the Z-Wave network until a reboot. This was particularly annoying if it occured after having detected motion, since the state of the sensor in the ISY controller is simply whatever state it last received. Therefore, I wrote this program to check if it believes the motion sensor is working:
Status '1st Floor / Living + Dining Room / ZW 005 Binary Sensor' is On
$LR_Lamp_Motion_Working = 1
Wait 1 hour
$LR_Lamp_Motion_Working = 0
$LR_Lamp_Motion_Working = 1
We don’t really care if the motion sensor is broken when the status is off; all that happens then is no lamp turns on. So this program activates when the status is set to on. It flips the working flag to true, then waits for an hour. If the sensor shows no motion within that hour, the program skips to the else (keeping the flag true). But if it is still on after an hour, it decides “it must not be working” and sets the working flag to false. You can see that flag used in the other programs’ logic. But because of the Else, which is run whenever the conditions that caused the Then clause to run become false, as soon as the system receives “no motion”, it will flag the sensor as working again.
The final piece to this puzzle is a program flagged to run at boot time of the controller:
- No Conditions - (To add one, press 'Schedule' or 'Condition')
$LR_Lamp_Motion_Working = 1
- No Actions - (To add one, press 'Action')
This simply initializes the motion working variable to a known state.
The Insteon KeypadLinc is a nice device. It can control a single load directly, but all 8 buttons are fully responsive in the Insteon system. They each also have individually-addressible LED backlights. They are commonly used to do things like “ALL OFF”, “TV” (to set lights for watching TV), “AWAY” (to set the system for everyone being out of the house for awhile), etc. They are the size of a regular Decora switch, and I have installed one already, but haven’t programmed much of it yet.
The ISY has an extensive REST API, which I’ve used to integrate it a bit with my Debian systems. More on that another time.
Mobile apps are a common thing people look for in these systems. You can’t use the Insteon app with the ISY, but they recommend Mobilinc Pro. It does the job. Mobilinc tries to sell a $10/mo connection service, which is totally unnecessary if you can figure out SSL, and has on-screen buttons to bypass, but judging by the Google Play reviews, a lot of people thought they had to pay for that and uninstalled it afterwards.
Many people put in electronic door locks. I don’t plan to do that. I do plan to have the house systems be aware of the general state of things (is the house empty? is everyone asleep?) and do appropriate things with lighting and HVAC. I don’t really expect the savings in power for lighting control to pay for the system anytime soon. However, if it can achieve some savings in heating and cooling, it may well be able to pay for itself in a few years. So my big next step is thermostats that can integrate with all this.
I have had a water mess in my basement before, and water leak sensors are a very common item people deploy in these setups. I certainly plan to add a few of them.
Door-open sensors are also useful; they trigger more instantly and reliably than motion detectors and can be used in some nice ways (is it after dark and the door is opening when the house is vacant? If so, turn on the light nearby in case their hands are full.)
Some issues I ran into so far are already discussed above. One other major one involves SSL on the ISY-994i Pro. The method for adding SSL keys is cumbersome, but the processor on the device — which appears to run some sort of Java — is just not up to working with SSL. Apparently they only recently got it fast enough to work with 2048-bit keys. This is rather undocumented, though, so I obtained a cert for a 4096-bit key, my usual. Attempting to connect to the box with SSL appeared to hang not just that but confuse a lot of other things on it as well. Turned out it wasn’t hung; it just too a minute and 45 seconds to complete the SSL handshake. Moral of the story: use 2048-bit keys, or stunnel4 or some such to re-wrap the SSL communications with a stronger key.
The KeypadLinc backlights can be completely shut off, and both their on and off levels can be customized. I have it set to shut off the backlight during the day and turn it on at night. The wall switches, however, can’t have their brightness status LED bar entirely shut off. They can be made dim, but don’t ever go away. That’s rather annoying.
Also annoying is that Insteon doesn’t make switches in the traditional toggle switch style in colors other than white. As our house had mostly black switches, I was forced into the Decora style.
This has been a great learning experience for me in a number of ways. I have only begun to tap what the system can do, and the real benefits will probably come once I get the heating and cooling into the mix. It’s quite a nice way for a geek to go, and the improvements in lighting have also been popular with everyone else in the house.
Categories: Home Automation
January 30th, 2015
I’ve been thinking about home automation — automating lights, switches, thermostats, etc. — for years. Literally decades, in fact. When I was a child, my parents had a RadioShack X10 control module and one or two target devices. I think I had fun giving people a “light show” turning on or off one switch and one outlet remotely.
But I was stuck — there are a daunting number of standards for home automation these days. Zigbee, UPB, Z-Wave, Insteon, and all sorts of Wifi-enabled things that aren’t really compatible with each other (hellooooo, Nest) or have their own “ecosystem” that isn’t all that open (helloooo, Apple). Frankly I don’t think that Wifi is a great home automation protocol; its power drain completely prohibits it being used in a lot of ways.
Earlier this month, my awesome employer had our annual meeting and as part of that our technical teams had some time for anyone to talk about anything geeky. I used my time to talk about flying quadcopters, but two of my colleagues talked about home automation. I had enough to have a place to start, and was hooked.
People use these systems to do all sorts of things: intelligently turn off lights when rooms aren’t occupied, provide electronic door locks (unlockable via keypad, remote, or software), remote control lighting and heating/cooling via smartphone apps, detect water leakage, control switches with awkward wiring environments, buttons to instantly set multiple switches to certain levels for TV watching, turning off lights left on, etc. I even heard examples of monitoring a swamp cooler to make sure it is being used correctly. The possibilities are endless, and my geeky side was intrigued.
Insteon and Z-Wave
Based on what I heard from my colleagues, I decided to adopt a hybrid network consisting of Insteon and Z-Wave.
Both are reliable protocols (with ACKs and retransmit), so they work far better than X10 did. Both have all sorts of controls and sensors available (browse around on smarthome.com for some ideas).
Insteon is a particularly interesting system — an integrated dual-mesh network. It has both powerline and RF signaling, and most hardwared Insteon devices act as repeaters for both the wired and RF network simultaneously. Insteon packets contain a maximum hop count that is decremented after each relay, and the packets repeat in such as way that they collide and strengthen one another. There is no need to maintain routing tables or anything like that; it simply scales nicely.
This system addresses all sorts of potential complexities. It addresses the split-phase problem of powerline-only systems, but using an RF bridge. It addresses long distances and outbuildings by using the powerline signaling. I found it to work quite well.
The downside to Insteon is that all the equipment comes from one vendor: Insteon. If you don’t like their thermostat or motion sensor, you don’t have any choice.
Insteon devices can be used entirely without a central controller. Light switches can talk to each other directly, and you can even set them up so that one switch controls dozens of others, if you have enough patience to go around your house pressing tiny “set” buttons.
Enter Z-Wave. Z-Wave is RF-only, and while it is also a mesh network, it is source-routed, meaning that if you move devices around, you have to “heal” your network as all your nodes have to re-learn the path to each other. It also doesn’t have the easy distance traversal of Insteon, of course. On the other hand, hundreds of vendors make Z-Wave products, and they mostly interoperate well. Z-Wave is said to scale practically to maybe two or three dozen devices, which would have been an issue for me, buut with Insteon doing the heavy lifting and Z-Wave filling in the gaps, it’s worked out well.
Controlling it all
While both Insteon (and, to a certain extent, Z-Wave) devices can control each other, to really spread your wings, you need more centralized control. This lets you have programs that do things like “if there’s motion in the room on a weekday and it’s dark outside, then turn on a light, and turn it back off 5 minutes later.”
Insteon has several options. One, you can buy their “power line modem” (PLM). This can be hooked up to a PC to run either Insteon’s proprietary software, or something open-source like MisterHouse, written in Perl. Or you can hook it up to a controller I’ll mention in a minute. Those looking for a fairly simpe controller might get the Insteon 2242-222 Hub, which has the obligatory smartphone app and basic schedules.
For more sophisticated control, my friend recommended the ISY-994i controller. Not only does it have a lot more capable programming language (though still frustrating), it supports both Insteon and Z-Wave in an integrated box, and has a comprehensive REST API for integrating with other things. I went this route.
First step: LED lighting
I began my project by replacing my light bulbs with LEDs. I found that I could get Cree 4-Flow 60W equivs for $10 at Home Depot. They are dimmable, a key advantage over CFL, and also maintain their brightness throughout their life far better. As I wanted to install dimmer switches, I got a combination of Cree 60W bulbs, Cree TW bulbs (which have a better color spectrum coverage for more true colors), and Cree 100W equiv bulbs for places I needed more coverage. I even put in a few LED flood lights to replace my can lights.
Overall I was happy with the LEDs. They are a significant improvement over the CFLs I had been using, and use even less power to boot. I have had issues with three Cree bulbs, though: one arrived broken, and two others have had issues such as being quite dim. They have a good warranty, but it seems their QA could be better. Also, they can have a tendency to flickr when dimmed, though this plagues virtually all LED bulbs.
I had done quite a bit of research. CNET has some helpful brightness guides, and Insteon has a color temperature chart. CNET also had a nifty in-depth test of LED bulbs.
Second step: switches
Once the LED bulbs were in place, I was then able to start installing smart switches. I picked up Insteon’s basic switch, the SwitchLinc 2477D at Menard’s. This is a dimmable switch and requires a neutral wire in the box, but acts as a dual-band repeater for the system as well.
The way Insteon switches work, they can be standalone, or controllers, responders, or both in a “scene”. A scene is where multiple devices act together. You can create virtual 3-way switches in a scene, or more complicated things where different lights are turned on at different levels.
Anyhow, these switches worked out quite well. I have a few boxes where there is no neutral wire, so I had to use the Insteon SwitchLinc 2474D in them. That switch is RF-only and is supposed to have a minimum load of 20W, though they seemed to work OK — albeit with limited range and the occasional glitch — with my LEDs. There is also the relay-based SwitchLinc 2477S for use with non-dimmable lights, fans, etc. You can also get plug-in modules for controlling lamps and such.
I found the Insteon devices mostly lived up to their billing. Occasionally I could provoke a glitch by changing from dimming to brightening in rapid succession on a remote switch controlling a load on a distant one. Twice I had to power cycle an Insteon switch that got confused (rather annoying when they’re hardwared). Other than that, though, it’s been solid. From what I gather, this stuff isn’t ever quite as reliable as a 1950s mechanical switch, but at least in this price range, this is about as good as it gets these days.
Well, this post got quite long, so I will have to follow up with part 2 in a little while. I intend to write about sensors and the Z-Wave network (which didn’t work quite as easily as Insteon), as well as programming the ISY and my lessons learned along the way.
Categories: Home Automation
January 1st, 2015
I finally decided it was about time to get my whole-house sound project off the ground. As an added bonus, I’d like to be able to stream music from my house to my Android phone.
It was about 2.5 years ago that I last revisited the music-listening picture on Linux. I used Spotify for awhile, but the buggy nature of its support for local music eventually drove me up the wall. I have a large collection of music that will never be on Spotify (local choirs, for instance) and this was an important feature.
When Google Play Music added the feature of uploading your local collection, I used that; it let me stream music from my phone in my car (using the Bluetooh link to the car). I could also listen at home on a PC, or plug my phone into various devices to play. But that was a hassle, and didn’t let me have music throughout the house.
Google Play is reasonable for that, but it has a number of really glaring issues. One is that it often gets album artwork wrong; it doesn’t use the ID3 tags embedded in the files, but rather tries to “guess”. Another is that the “sync” is only “add”. Move files to another place in your collection, or re-rip them to FLAC and replace your old MP3s, and suddenly they’re in Google Play twice. It won’t ever see updated metadata, either — quite a hassle for someone that uses Musicbrainz and carefully curates metadata.
I already have an oldish PC set up as a entertainment box running MythTV. It is a diskless system (boots PAE over the network and has NFS root) and very quiet. It has video output to my TV, and audio output via S/PDIF to my receiver. It is one logical audio frontend.
My workstation in my office is another obvious place.
My kitchen has a radio with a line in jack, and I also have a small portable speaker with a line in jack to make the last two options. I also have a Raspberry Pi model B that I bought awhile ago and was looking for a purpose, so I thought – this should be cheap and easy, right? Well. Cheap, yes. Easy, not so much.
First attempt: Ampache
The Ampache project produces quite a nice piece of software. Ampache has matured significantly in the last 2.5 years, and the usability of its web-interface — with HTML5 and Flash players as options — is quite impressive. It is easily as usable as Google’s, though its learning curve is rather more steep. There are multiple Android apps for Ampache to stream remotely. And, while most are terribly buggy and broken, there is at least one that seems to work well.
Ampache can also output m3u/pls files for a standalone player. It does on-the-fly transcoding. There are some chinks in the armor, however. The set of codecs that are transcoded or passed through is a global setting, not a per-device setting. The bitrates are per-account, so you can’t easily have it transcode FLAC into 320Kbps mp3 for streaming on your LAN and 128Kbps MP3 for streaming to your phone. (There are some hacks involving IP address ranges and multiple accounts, but they are poorly documented and cumbersome.)
Ampache also has a feature called “localplay” in which it drives local players instead of remote ones. I tried to use this in combination with mpd to drive music to the whole house. Ampache’s mpd interface is a bit odd; it actually loads things up into mpd’s queue. Sadly it shares the same global configs as the rest. Even though mpd is perfectly capable of handling FLAC audio, the Ampache web player isn’t, so you have to either make it transcode mp3 for everything or forego the web player (or use a second account that has “transcode everything” set). Frustratingly, not one of the Android clients for Ampache is even remotely compatible with Localplay, and some will fail in surprising ways if you have been using Localplay on the web client.
So let’s see how this mpd thing worked out.
Ampache with mpd for whole-house audio
The primary method here is to use mpd’s pulseaudio driver. I configured it like so:
name "MPD stream"
#server "remote_server" # optional
sink "rtp" # optional
Then in /etc/pulse/system.pa:
load-module module-null-sink sink_name=rtp format=s16be channels=2 rate=44100 sink_properties="device.description='RTP Multicast Sink'"
load-module module-rtp-send source=rtp.monitor
This tells Pulse to use multicast streaming to the LAN for the audio packets. Pulse is supposed to have latency synchronization to achieve perfect audio everywhere. In practice, this works somewhat poorly. Plus I have to install pulse everywhere, which inserts its tentacles way too deeply into the ALSA stack for my taste. (alsamixer suddenly turns useless by default, for instance.)
But I gave it a try. After much fiddling — Pulse is rather poorly documented and the interactions of configuration tools with it even more so — I found a working configuration. On my MythTV box, I added:
load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1;x.x.x.x
The real annoyance was getting it to set the output through the S/PDIF digital port. Finally I figured out the magic potion:
set-card-profile 0 output:iec958-stereo+input:analog-stereo
This worked reasonably well. The Raspberry Pi was a much bigger challenge, however.
I put Raspbian on it easily enough, and installed Pulse. But apparently it is well-known in the Pulse community that Pulse’s RTP does not work well with wifi. Multicast itself works poorly with wifi in general, and Pulse won’t do unicast RTP. However, Pulse assumes very low latency and just won’t work well with wifi at all.
Morever, pulseaudio on the rpi is something of a difficult beast to tame. It has crackling audio, etc. I eventually got it working decently with:
realtime-scheduling = yes
realtime-priority = 5
resample-method = src-sinc-fastest
default-sample-rate = 44100
default-fragments = 4
and in system.pa, commented out the module-udev-detect and module-detect, and added:
load-module module-alsa-card device_id=0 tsched=0 fragments=10 fragment_size=640 tsched_buffer_size=4194384 tsched_buffer_watermark=262144
plus the rtp-recv and tcp protocol lines as before. This got it working with decent quality, but it was always out of sync by at least a few tenths of a second, if not a whole second, with the other rooms.
Brief diversion: mplayer
Many players besides pulseaudio can play the RTP streams that pulseaudio generates. mplayer, for instance, can. I found this worked on the Raspberry Pi:
mplayer -really-quiet -cache 64 -cache-min 95 -demuxer rawaudio -rawaudio format=0x20776172 rtp://22.214.171.124:46510
But this produced even worse sync. It became clear that Ampache was not going to be the right solution.
Logitech Media Server
I have looked at this a few times over the years, but I’ve somehow skipped it. But I looked at it again now. It is an open source (GPL) server, and was originally designed to work with Logitech hardware. There are now all sorts of software clients out there. There is a helpful wiki about it, although Logitech has rebranded the thing so many times, it’s not even consistent internally on what the heck it’s called (is it LMS? or Squeezebox Server? or Slimserver? The answer is: yes.)
LMS does no web-based streaming. At all. But I thought I’d give it a try. Installation is a little… weird (its .deb packages up binary perl modules for things that are already in Debian, for half a dozen architectures and many Perl versions.) I had some odd issues but eventually it worked. It scanned my media collection. I installed squeezelite on my workstation as my first player, and things worked reasonably well out of the box.
I proceeded to install squeezelite on the Raspberry Pi and MythTV server (where I had to compile libsoxr for wheezy, due to an obscure error I couldn’t find the cause of until I used strace). Log in to the LMS web interface, tell it to enable synchronization, and bam! Perfect synchronization! Incredible.
But I ran into two issues: one was that it used a lot of CPU even on my workstation (50% or more, even when idle). Strangely, it used far less CPU on the pi than did pulseaudio. Secondly, I’d get annoying clicks from it from time to time. Some debugging and investigation revealed that they were both somewhat related; it was getting out of sync with the pi and correcting. Initially I assumed this to be an issue with the pi, but tracked it down to something else. I told it to upsample to 48kHz and this made the problem go away. My command on PC hardware is:
squeezelite -a 200:10 -o dmix -u hLX -z
And on the Raspberry Pi:
squeezelite -a 800:10:16:0 -n kitchen -z
It is working perfectly now.
LMS even has an Android remote control app that can control all devices together, queue up playlists, etc. Very slick.
The Raspberry Pi
The Raspberry Pi, even the model B+, apparently has notoriously bad audio output. I noticed this first as very low volume on the output. I found a $11 USB DAC on Amazon that seems to do the trick, so we will see. I am also using the $10 Edimax USB Wifi and it seems to work well.
The big-name other option here is Sonos. They have perfect sync audio already working. But they’re pricy, closed, and proprietary; you have to buy Sonos speakers, transmitters, etc. for the entire house. Their receivers start at $200, and it’s at least $350 to integrate an existing stereo into the system. I’d be shelling out over $1000 to use Sonos for this setup. As it is, I’ve bought less than $100 of equipment (a second Pi and its accessories) and getting output that, while no doubt not quite as pristine, it still quite nice and acceptable. Quite nice.
Update: LMS with Apache
It turns out that Logitech Media Server can automatically save created playlists to a spot where Ampache can find them. So an Ampache streaming player can be used to access the same collection LMS uses, with full features. Nice!
December 18th, 2014
This was my first attempt to send up the quadcopter in winter. It’s challenging to take good photos of a snowy landscape anyway. Add to that the fact that the camera is flying, and it’s cold, which is hard on batteries and motors. I was rather amazed at how well it did!
November 23rd, 2014
Yesterday, Jacob (age 8) asked to help me put together a 30-year-old computer from parts in my basement. Meanwhile, Oliver (age 5) asked Laura to help him learn cursive. Somehow, this doesn’t seem odd for a Saturday at our place.
Let me tell you how this came about.
I’ve had a project going on for a while now to load data from old floppies. It’s been fun, and had a surprise twist the other day: my parents gave me an old TRS-80 Color Computer II (aka “CoCo 2″). It was, in fact, my first computer, one they got for me when I was in Kindergarten. It is nearly 30 years old.
I have been musing lately about the great disservice Apple did the world by making computers easy to learn — namely the fact that few people ever bother to learn about them. Who bothers to learn about them when, on the iPhone for instance, the case is sealed shut, the lifespan is 1 or 2 years for many purchasers, and the platform is closed in lots of ways?
I had forgotten how finicky computers used to be. But after some days struggling with IDE incompatibilities, booting issues, etc., when I actually managed to get data off a machine that had last booted in 1999, I had quite the sense of accomplishment, which I rarely have lately. I did something that was hard to do in a world where most of the interfaces don’t work with equipment that old (even if nominally they are supposed to.)
The CoCo is one of those computers normally used with a floppy drive or cassette recorder to store programs. You type DIR, and you feel the clack of the drive heads through the desk. You type CLOAD and you hear the relay click closed to turn on the tape motor. You wiggle cables around until they make contact just right. You power-cycle for the times when the reset button doesn’t quite do the job. The details of how it works aren’t abstracted away by innumerable layers of controllers, interfaces, operating system modules, etc. It’s all right there, literally vibrating your desk.
So I thought this could be a great opportunity for Jacob to learn a few more computing concepts, such as the difference between mass storage and RAM, plus a great way to encourage him to practice critical thinking. So we trekked down to the basement and came up with handfulls of parts. We brought up the computer, some joysticks, all sorts of tangled cables. We needed adapters, an old TV. Jacob helped me hook everything up, and then the moment of truth: success! A green BASIC screen!
I added more parts, but struck out when I tried to connect the floppy drive. The thing just wouldn’t start up right whenever the floppy controller cartridge was installed. I cleaned the cartridge. I took it apart, scrubbed the contacts, even did a re-seat of the chips. No dice.
So I fired up my CoCo emulator (xroar), and virtually “saved” some programs to cassette (a .wav file). I then burned those .wav files to an audio CD, brought up an old CD player from the basement, connected the “cassette in” plug to the CD player’s headphone jack, and presto — instant programs. (Well, almost. It takes a couple of minutes to load a program from audio codes.)
The picture above is Oliver cackling at one of the very simplest BASIC programs there is: “number find.” The computer picks a random number between 1 and 2000, and asks the user to guess it, giving a “too low” or “too high” clue with each incorrect guess. Oliver delighted in giving invalid input (way too high numbers, or things that weren’t numbers at all) and cackled at the sarcastic error messages built into the program. During Jacob’s turn, he got very serious about it, and is probably going to be learning about how to calculate halfway points before too long.
But imagine my pride when this morning, Jacob found the new CD I had made last night (correcting a couple recordings), found my one-line instruction on just part of how to load a program, and correctly figured out by himself all the steps to do in order (type CLOAD on the CoCo, advance the CD to the proper track, press play on the player, wait for it to load on the CoCo, then type RUN).
I ordered a replacement floppy controller off eBay tonight, and paid $5 for a coax adapter that should fix some video quality issues. I rescued some 5.25″ floppies from my trash can from another project, so they should have plenty of tools for exploration.
It is so much easier for them to learn how a disk drive works, and even what the heck a track is, when you can look at a floppy drive with the cover off and see the heads move. There are other things we can do with more modern equipment — Jacob has shown a lot of interest in Arduino projects — but I have so far drawn a blank on ways to really let kids discover how a modern PC (let alone a modern phone or tablet) works.
Update Nov. 24: Every so often, the world surprises me by deciding to, well, read one of my random blog posts. For the benefit of those of you that don’t already know my boys, you might want to know that among their common play activites are turning trees into pretend trains, typing at a manual typewriter, reading, writing their own books, using a cassette recorder, building a PC and learning to use bash or xmonad, making long paper tapes with an adding machine, playing records on a record player, building electric gizmos, and even making mud balls.
I am often asked about the role of the computer in the lives, given that my hobby and profession involves computers. The answer: less than that of most of their peers. I look for opportunities for them to learn by doing, discovering, playing, or imagining. I make no presumption that they will develop the passion for computers that I did. What I want is for them to have the curiosity and drive to learn everything there is to know about whatever they do develop a passion for, so they will be great at it.
Categories: Children & Computing
November 16th, 2014
Sometimes I look out the window and can’t help but feel “this weather is deep.” Deep with meaning, with import. Almost as if the weather is confident of itself, and is challenging me to find some meaning within it.
This weekend brought the first blast of winter to the plains of Kansas. Saturday was chilly and windy, and overnight a little snow fell. Just enough to cover up the ground and let the tops of the blades of grass poke through. Just enough to make the landscape look totally different, without completely hiding what lies beneath. Laura and I stood silently at the window for a few minutes this morning, gazing out over the untouched snow, extending out as far as we can see.
Yesterday, I spent some time with my great uncle and aunt. My great uncle isn’t doing so well. He’s been battling cancer and other health issues for some time, and can’t get out of the house very well. We talked for an hour and a half – about news of the family, struggles in life now and in the past, and joys. There were times when all three of us had tears in our eyes, and times when all of us were laughing so loudly. My great uncle managed to stand up twice while I was there — this took quite some effort — once to give me a huge hug when I arrived, and another to give me an even bigger hug when I left. He has always been a person to give the most loving hugs.
He hadn’t been able to taste food for awhile, due to treatment for cancer. When I realized he could taste again, I asked, “When should I bring you some borscht?” He looked surprised, then got a huge grin, glanced at his watch, and said, “Can you be back by 3:00?”
His brother, my grandpa, was known for his beef borscht. I also found out my great uncle’s favorite kind of bread, and thought that maybe I would do some cooking for him sometime soon.
Today on my way home from church, I did some shopping. I picked up the ingredients for borscht and for bread. I came home, said hi to the cats that showed up to greet me, and went inside. I turned on the radio – Prairie Home Companion was on – and started cooking.
It takes a long time to prepare what I was working on – I spent a solid two hours in the kitchen. As I was chopping up a head of cabbage, I remembered coming to what is now my house as a child, when my grandpa lived here. I remembered his borscht, zwiebach, monster cookies; his dusty but warm wood stove; his closet with toys in it. I remembered two years ago, having nearly 20 Goerzens here for Christmas, hosted by the boys and me, and the 3 gallons of borscht I made for the occasion.
I poured in some tomato sauce, added some water. The radio was talking about being kind to people, remembering that others don’t always have the advantages we do. Garrison Keillor’s fictional boy in a small town, when asked what advantages he had, mentioned “belonging.” Yes, that is an advantage. We all deal with death, our own and that of loved ones, but I am so blessed by belonging – to a loving family, two loving churches, a wonderful community.
Out came three pounds of stew beef. Chop, chop, slice, plunk into the cast iron Dutch oven. It’s my borscht pot. It looks as if it would be more at home over a campfire than a stovetop, but it works anywhere.
Outside, the sun came up. The snow melts a little, and the cats start running around even though it’s still below freezing. They look like they’re having fun playing.
I’m chopping up parsley and an onion, then wrapping them up in a cheesecloth to make the spice ball for the borscht. I add the basil and dill, some salt, and plonk them in, too. My 6-quart pot is nearly overflowing as I carefully stir the hearty stew.
On the radio, a woman who plays piano in a hospital and had dreamed of being on that particular radio program for 13 years finally was. She played with passion and delight I could hear through the radio.
Then it’s time to make bread. I pour in some warm water, add some brown sugar, and my thoughts turn to Home On The Range. I am reminded of this verse:
How often at night when the heavens are bright
With the light from the glittering stars
Have I stood here amazed and asked as I gazed
If their glory exceeds that of ours.
There’s something about a beautiful landscape out the window to remind a person of all the blessings in life. This has been a quite busy weekend — actually, a busy month — but despite the fact I have a relative that is sick in the midst of it all, I am so blessed in so many ways.
I finish off the bread, adding some yeast, and I remember my great uncle thanking me so much for visiting him yesterday. He commented that “a lot of younger people have no use for visiting an old geezer like me.” I told him, “I’ve never been like that. I am so glad I could come and visit you today. The best gifts are those that give in both directions, and this surely is that.”
Then I clean up the kitchen. I wipe down the counters from all the bits of cabbage that went flying. I put away all the herbs and spices I used, and finally go to sit down and reflect. From the kitchen, the smells of borscht and bread start to seep out, sweeping up the rest of the house. It takes at least 4 hours for the borscht to cook, and several hours for the bread, so this will be an afternoon of waiting with delicious smells. Soon my family will be home from all their activities of the day, and I will be able to greet them with a warm house and the same smells I stepped into when I was a boy.
I remember this other verse from Home On the Range:
Where the air is so pure, the zephyrs so free,
The breezes so balmy and light,
That I would not exchange my home on the range
For all of the cities so bright.
Today’s breeze is an icy blast from the north – maybe not balmy in the conventional sense. But it is the breeze of home, the breeze of belonging. Even today, as I gaze out at the frozen landscape, I realize how balmy it really is, for I know I wouldn’t exchange my life on the range for anything.
November 12th, 2014
My desk today looks like this:
Yep, that’s a computer. Motherboard to the right, floppy drives and CD drive stacked on top of the power supply, hard drive to the left.
And it’s an OLD computer. (I had forgotten just how loud these old power supplies are; wow.)
The point of this exercise is to read data off the floppies that I have made starting nearly 30 years ago now (wow). Many were made with DOS, some were made on a TRS-80 Color Computer II (aka CoCo 2). There are 5.25″ disks, 3.25″ disks, and all sorts of formats. Most are DOS, but the TRS-80 ones use a different physical format. Some of the data was written by Central Point Backup (from PC Tools), which squeezed more data on the disk by adding an extra sector or something, if my vague memory is working.
Reading these disks requires low-level playing with controller timing, and sometimes the original software to extract the data. It doesn’t necessarily work under Linux, and certainly doesn’t work with USB floppies or under emulation. Hence this system.
It’s a bridge. Old enough to run DOS, new enough to use an IDE drive. I can then hook up the IDE drive to a IDE-to-USB converter and copy the data off it onto my Linux system.
But this was tricky. I started the project a few years ago, but life got in the way. Getting back to it now, with the same motherboard and drive, but I just couldn’t get it to boot. I eventually began to suspect some disk geometry settings, and with some detective work from fdisk in Linux plus some research into old BIOS disk size limitations, discovered the problem was a 2GB limit. Through some educated trial and error, I programmed the BOIS with a number of cylinders that worked, set it to LBA mode, and finally my 3-year-old DOS 6.2 installation booted.
I had also forgotten how finicky things were back then. Pop a floppy from a Debian install set into the drive, type dir b:, and the system hangs. I guess there was a reason the reset button was prominent on the front of the computer back then…
November 10th, 2014
I’m never sure whether to post such things here, but I hope that it’s of interest to people: I’m trying to hire a top-notch Linux person for a 100% telecommute position. I’m particularly interested in people with experience managing 500 or more OS instances. It’s a shop with a lot of Debian, by the way. You can apply at that URL and mention you saw it in my blog if you’re interested.