Well, I finally bit the bullet. My laptop, which runs jessie, got dist-upgraded for the first time in a few months. My brightness keys stopped working, and it no longer would suspend to RAM when the lid was closed, and upon chasing things down from XFCE to policykit, eventually it appears that suddenly major parts of the desktop breaks without systemd in jessie. Sigh.
So apt-get install systemd-sysv (and watch sysvinit-core get uninstalled) and reboot.
Only, my system doesn’t come back up. In fact, over several hours of trying to make it boot with systemd, it failed in numerous spectacular and hilarious (or, would be hilarious if my laptop would boot) ways. I had text obliterating the cryptsetup password prompt almost every time. Sometimes there were two processes trying to read a cryptsetup password at once. Sometimes a process was trying to read that while another one was trying to read an emergency shell password. Many times it tried to write to /var and /tmp before they were mounted, meaning they *wouldn’t* mount because there was stuff there.
I noticed it not doing much with ZFS, complaining of a dependency loop between zfs-mount and $local-fs. I fixed that, but it still wouldn’t boot. In fact, it simply hung after writing something about wall passwords.
I’ve dug into systemd, finding a “unit generator for fstab” (whatever the hack that is, it’s not at all made clear by systemd-fstab-generator(8)).
In some cases, there’s info in journalctl, but if I can’t even get to an emergency mode prompt, the practice of hiding all stdout and stderr output is not all that pleasant.
I remember thinking “what’s all the flaming about?” systemd wasn’t my first choice (I always thought “if it ain’t broke, don’t fix it” about sysvinit), but basically ignored the thousands of messages, thinking whatever happens, jessie will still boot.
Now I’m not so sure. Even if the thing boots out of the box, it seems like the boot process with systemd is colossally fragile.
For now, at least zfs rollback can undo upgrades to 800 packages in about 2 seconds. But I can’t stay at some early jessie checkpoint forever.
Have we made a vast mistake that can’t be undone? (If things like even *brightness keys* now require systemd…)
The brightness key thing is actually a major improvement; it means brightness changes work across all desktop environments, whether logged in or not, and remain persistent across reboots. It’s nice that desktop environments don’t all have to handle things like brightness, rfkill, and suspend/resume themselves, just because they’re a thing listening to X and thus receiving input events.
It sounds like you hit at least two bugs. The zfs thing sounds like a bug in the .service files for zfs: if systemd encounters a dependency loop, it breaks the loop forcibly in the hopes of booting so you can recover; a loop involving local-fs could break dependencies on local-fs, and that could potentially have caused the mount issues you saw with /var and /tmp.
Separately from that, you’ve hit some issue with cryptsetup. I think a bug was already filed about displaying on top of cryptsetup; you can work around that by installing plymouth, but I’m hoping that one gets fixed in a way that doesn’t require a splash screen. However, you should *never* have two processes prompting for cryptsetup passwords at the same time; those prompts are supposed to be serialized.
The brightness keys still seem to be handled by xfce’s power manager plugin, and the cause of the failure seems to be pkexec deciding it’s not authorized because it’s not in a systemd session or something.
The ZFS debs I’m using don’t actually provide .service files; systemd is using regular init scripts. Which I thought were supposed to still work.
init scripts still work in general, but possibly not for something as fundamental as mounting filesystems.
Without pouring oil on the flames (they are plenty) — I saw that coming. I left the Gnome ship once I saw that everything was becoming interdependent in ways I didn’t like (mixing policy and mechanism, co-development of interface producers and consumers instead of defining an interface, yadda, yadda).
After I saw XFCE going into the same direction (DBUS is usually a symptom of this), I finally switched to Xvwm2 as my “desktop environment”. Happy since.
Now there are people which will be happy with this “new stuff”. More power to you all. But me? I’ll keep with the loosely-coupled, where there’s room between the components.
It’s very unfortunate that you had such a bad first experience with systemd. Please file a bug report against the Debian systemd package and we will investigate what’s going wrong.
Hi Michael,
Thank you. I am a little unsure even which packages are in play here, but I will try.
Hey, I think someone figured out the issue with cgmanager that is causing similar problems to what you describe. Check out this message and the following workaround (adding cgroup_enable=memory to the kernel command line) to see if it fixes your problems: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=757348#125
I should clarify this is to fix your problems with sysvinit and brightness/suspend, not anything else.
Also, you may want to edit /etc/systemd/logind.conf to have the line
HandleLidSwitch=suspend
Re: password prompt and multiple processes reading keystrokes, making rescue shell useless; that’s something I mentioned in #759029 but have been unable to reproduce it on demand to generate a bug report.
Conversely I just tried out OpenRC; I’ve hit a bug which causes it to segfault at early boot. But it drops me to a regular login prompt. I was able to /sbin/fsck, remount rw, and manually run scripts from /etc/rcS.d until my system is fully booted.
@Steven C:
It’s unlikely to be the same issue, but the inittab on the debian wiki for openrc has errors in it, imo.
I don’t recall all the detail now, only that the wiki one merely imports a few lines, and other things are not well-integrated. I reported it to a Gentoo openrc developer to relay to the debian collaborators, since the wiki page was locked, but to the best of my knowledge nothing changed.
In any event you can see a gentoo inittab at
http://paste.debian.net/125929/
— obviously not everything will be the same, but it
should give you a fuller picture.
HTH,
steveL.
@steveL: no, unrelated to that; I’ve filed bug #765070 about it..
(Reply button on comments doesn’t work without JavaScript)
Anyway, my point was about serviceability; being able to use existing tools and skills to rescue the system when the worst happens.
Funny, I had pretty much the same experience since systemd was introduced into Debian. My system boots most of the time, but GDM doesn’t start up and several daemons fail to load. I switched to KDM which seems to work more realiably. That problem is probably related to bugs with VT switching.
One of the other problems in my case might have something to do with /tmp being a symlink to /var/tmp, which makes some daemons bail out with “failed at step NAMESPACE”. I don’t know why it should matter, but apparently it does now.
I have nothing against systemd, but I do really hope they fix the bugs both in the software itself and in Debian before implementing everything and the kitchen sink. The init system should not be the cause of a non-working system.
Time to move to Linux Arch.
Arch uses SystemD.
This page on Fedora’s wiki has some good grub commands for troubleshooting systemd, including debug mode and an emergency shell: https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
This seems more an indictment of your distro and it’s migration to systemd than any semblance of the state of systemd. I’ve been running systemd for some time with no issue on laptops, personal server and business servers. Good luck, but point the finger in the direction if should go. :-)
Frank, I don’t intend to encourage potshots on my blog.
Did you read the story? Do you use ZFS with whatever your distro is? No? Then you haven’t run into this.
Do you use cryptsetup? Does your distro use plymouth?
From what I can tell, these are not Debian-specific issues.
I haven’t personally had a lot of trouble with systemd, but when I was helping out on some CentOS boxes a couple of years ago I was baffled that I could no longer read plain text logs.
I guess at this point I’ve accepted that there just isn’t much I can do to avoid it, if RedHat, Debian, and Arch are all using it. I’m not going to pass judgement without reading more, and like I said, I’ve pretty much accepted it, but it doesn’t seem like the nice scriptable easy to understand linux I came to love years ago.
In fact I started on Slackware primarily because it had a more transparent init system.
I bet the first time somebody performed a brain transplant it didn’t work either.
Considering it has never seem done, yes, fair comparison…
Just use Windows, dude. Linux is crap…
Thankfully work on Debian/kFreeBSD is going well. So there should be future option with Debian, ZFS, encrypted disks and no systemd…
On a more serious note, it’s nice to see people are willing to try systemd — I guess I will hold off trying on my laptop for a while (no zfs, but I do run cryptsetup).
Pity things broke so badly.
I switched from Windows to Kubuntu (for a month) to Arch Linux in April 2013, mainly because Arch used systemd and I could build a system ground up.
I’ve never had any issues related to systemd or otherwise except sometimes the non-free nVidia driver would refuse to load after kernel updates and needed downgrading to work.
I would put it down to general “dist upgrade” pains. I’ve had Debian and Ubuntu break badly on dist-upgrade.
systemd is not technically flawed, it simply does things in a less UNIX like way, but this is 2014 and very few “desktop Linux” users even see the text console.
For the record, I updated my work Jessie laptop constantly(nearly everyday) and had very few issues with the systemd changeover, and the few I had took < 1 hour total to fix and just involved some package finagling.
My laptop had brightness settings and an encrypted hard drive, though I didn't use ZFS.
In any case, just another anecdote. Systemd changeover worked fine for me.
P.S. I did have an annoying bug with startup where it took about 5 times longer to startup due to some kind've freeze after mounting filesystems. However, that went away after a few weeks.