A Mail Delivery Mystery: Exim, systemd, setuid, and Docker, oh my!

On mail.quux, a node of NNCPNET (the NNCP-based peer-to-peer email network), I started noticing emails not being delivered. They were all in the queue, frozen, and Exim’s log had entries like:

unable to set gid=5001 or uid=5001 (euid=100): local delivery to [redacted] transport=nncp

Weird.

Stranger still, when I manually ran the queue with sendmail -qff -v, they all delivered fine.

Huh.

Well, I thought, it was a one-off weird thing. But then it happened again.

Upon investigating, I observed that this issue was happening only on messages submitted by SMTP. Which, on these systems, aren’t that many.

While trying different things, I tried submitting a message to myself using SMTP. Nothing to do with NNCP at all. But look at this:

 jgoerzen@[redacted] R=userforward defer (-1): require_files: error for /home/jgoerzen/.forward: Permission denied

Strraaannnge….

All the information I could find about this, even a FAQ entry, said that the problem is that Exim isn’t setuid root. But it is:

-rwsr-xr-x 1 root root 1533496 Mar 29  2025 /usr/sbin/exim4

This problem started when I upgraded to Debian Trixie. So what changed there?

There are a lot of possibilities; this is running in Docker using my docker-debian-base system, which runs a regular Debian in Docker, including systemd.

I eventually tracked it down to Exim migrating from init.d to systemd in trixie, and putting a bunch of lockdowns in its service file. After a bunch of trial and error, I determined that I needed to override this set of lockdowns to make it work. These overrides did the trick:

ProtectClock=false
PrivateDevices=false
RestrictRealtime=false
ProtectKernelModules=false
ProtectKernelTunables=false
ProtectKernelLogs=false
ProtectHostname=false

I don’t know for sure if the issue is related to setuid. But if it is, there’s nothing that immediately jumps out at me about any of these that would indicate a problem with setuid.

I also don’t know if running in Docker makes any difference.

Anyhow, problem fixed, but mystery not solved!

3 thoughts on “A Mail Delivery Mystery: Exim, systemd, setuid, and Docker, oh my!

  1. A similar issue was brought up on the exim-users mailing list, with the red-flag log words “Read-only file system”. The solution suggestion there was systemd’s ProtectSystem and/or ProtectHome configuration.

    Usually the Debian guys are better than this.

    1. Debian didn’t have ProtectHome enabled. It did have ProtectSystem, but disabling that wasn’t necessary to fix these issues.

  2. https://etbe.coker.com.au/2025/01/17/systemd-hardening-mail/

    After writing the above blog post about similar issues I received the following comment by email:

    Exim has the “queue_only = ” or “-odq” option to disable the “fork to background here and now” delivery. However it doesn’t ping a running Exim daemon that a message has been queued (unlike say Postfix), so the messages will only be sent during the next queue run depending on the daemon’s -q option, which may take a while unless you add some inotify watcher on top.

    (Overall, that’s one of the reasons I prefer Postfix for send-only systems.)

    When Arch was introducing systemd, one of my contributions was to add an exim-queue.path that would start a queue-runnner as soon as the queue directory became non-empty (which fit well together with a .timer to emulate the -q30m periodic flush).

Reposts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Find out more about Webmentions.)