Category Archives: Technology

Goodbye to a 15-year-old Debian server

Debian

It was October of 2003 that the server I’ve called “glockenspiel” was born. It was the early days of Linux-based VM hosting, using a VPS provider called memset, running under, of all things, User Mode Linux. Over the years, it has been migrated around, sometimes running on the metal and sometimes in a VM. The operating system has been upgraded in-place using standard Debian upgrades over the years, and is now happily current on stretch (albeit with a 32-bit userland). But it has never been reinstalled. When I’d migrate hosting providers, I’d use tar or rsync to stream glockenspiel across the Internet to its new home.

A lot of people reinstall an OS when a new version comes out. I’ve been doing Debian upgrades with apt for ages, and this one is a case in point. It lingers.

Root’s .profile was last modified in November 2004, and its .bashrc was last modified in December 2004. My own home directory still has a .pinerc, .gopherrc, and .arch-params file. I last edited my .vimrc in 2003 and my .emacs dates back to 2002 (having been copied over from a pre-glockenspiel FreeBSD server).

drwxr-xr-x  3 jgoerzen jgoerzen      4096 Dec  3  2003 irclogs
-rw-r--r--  1 jgoerzen jgoerzen       373 Dec  3  2003 .vimrc
-rw-r--r--  1 jgoerzen jgoerzen       651 Nov 27  2003 .reportbugrc
drwx------  3 jgoerzen jgoerzen      4096 Sep  2  2003 .arch-params
-rw-r--r--  1 jgoerzen jgoerzen      1115 Aug 23  2003 .gopherrc
drwxr-xr-x  3 jgoerzen jgoerzen      4096 Jul 18  2003 .subversion
-rw-r--r--  1 jgoerzen jgoerzen     15317 Jun 21  2003 .pinerc

Poking around /etc on glockenspiel is like a trip back in time. Various apache sites still have configuration files around, but have long since been disabled. Over the years, glockenspiel has hosted source code repositories using Subversion, arch, tla, darcs, mercurial and git. It’s hosted websites using Drupal, WordPress, Serendipity, and so forth. It’s hosted gopher sites, websites or mailing lists for various Free Software projects (such as Freeciv), and any number of local charitable organizations. Remnants of an FTP configuration still exist, when people used web design software to build websites for those organizations on their PCs and then upload them to glockenspiel.

-rw-r--r--   1 root  root                      268 Dec 25  2005 libnet.cfg
-rw-r-----   1 root  root                     1305 Nov 11  2004 mrtg.cfg
-rw-r--r--   1 root  root                      552 Jul 31  2004 pam.conf

All this has been replaced by a set of Docker containers running my docker-debian-base software. They’re all in git, I can rebuild one of the containers in a few seconds or a few minutes by typing “make”, and there is no cruft from 2002. There are a lot of benefits to this.

And yet, there is a part of me that feels it’s all so… cold. Servers having “personalities” was always a distinctly dubious thing, but these days as we work through more and more layers of virtualization and indirection and become more distant from the hardware, we lose an appreciation for what we have and the many shoulders of giants upon which we stand.

And, so with that, the final farewell to this server that’s been running since 2003:

glockenspiel:/etc# shutdown -P now
Shared connection to glockenspiel.complete.org closed.

A (Partial) Defense of Debian

Debian

I was sad to read on his blog that Michael Stapelberg is winding down his Debian involvement. In his post, he outlined some critiques of Debian. In his post, I want to acknowledge that he is on point with some of them, but also push back on others. Some of this is also a response to some of the comments on Hacker News.

I’d first like to discuss some of the assumptions I believe his post rests on: namely that “we’ve always done it this way” isn’t a good reason to keep doing something. I completely agree. However, I would also say that “this thing is newer, so it’s better and we should use it” is also poor reasoning. Newer is not always better. Sometimes it is, sometimes it’s not, but deeper thought is generally required.

Also, when thinking about why things are a certain way or why people prefer certain approaches, we must often ask “why does that make sense to them?” So let’s dive in.

Debian’s Perspective: Stability

Stability, of course, can mean software that tends not to crash. That’s important, but there’s another aspect of it that is also important: software continuing to act the same over time. For instance, if you wrote a C program in 1985, will that program still compile and run today? Granted, that’s a bit of an extreme example, but the point is: to what extent can you count on software you need continuing to operate without forced change?

People that have been sysadmins for a long period of time will instantly recognize the value of this kind of stability. Change is expensive and difficult, and often causes outages and incidents as bugs are discovered when software is adapted to a new environment. Being able to keep up-to-date with security patches while also expecting little or no breaking changes is a huge win. Maintaining backwards compatibility for old software is also important.

Even from a developer’s perspective, lack of this kind of stability is why I have handed over maintainership of most of my Haskell software to others. Some of my Haskell projects were basically “done”, and every so often I’d get bug reports that it no longer compiles due to some change in the base library. Occasionally I’d have patches with those bug reports, but they would inevitably break compatibility with older versions (even though the language has plenty good support for something akin to a better version of #ifdefs to easily deal with this.) The culture of stability was not there.

This is not to say that this kind of stability is always good or always bad. In the Haskell case, there is value to be had in fixing designs that are realized to be poor and removing cruft. Some would say that strcpy() should be removed from libc for this reason. People that want the latest versions of gimp or whatever are probably not going to be running Debian stable. People that want to install a machine and not really be burdened by it for a couple of years are.

Debian has, for pretty much its entire life, had a large proportion of veteran sysadmins and programmers as part of the organization. Many of us have learned the value of this kind of stability from the school of hard knocks – over and over again. We recognize the value of something that just works, that is so stable that things like unattended-upgrades are safe and reliable. With many other distros, something like this isn’t even possible; when your answer to a security bug is to “just upgrade to the latest version”, just trusting a cron job to do it isn’t going to work because of the higher risk.

Recognizing Personal Preference

Writing about Debian’s bug-tracking tool, Michael says “It is great to have a paper-trail and artifacts of the process in the form of a bug report, but the primary interface should be more convenient (e.g. a web form).” This is representative of a personal preference. A web form might be more convenient for Michael — I have no reason to doubt this — but is it more convenient for everyone? I’d say no.

In his linked post, Michael also writes: “Recently, I was wondering why I was pushing off accepting contributions in Debian for longer than in other projects. It occurred to me that the effort to accept a contribution in Debian is way higher than in other FOSS projects. My remaining FOSS projects are on GitHub, where I can just click the “Merge” button after deciding a contribution looks good. In Debian, merging is actually a lot of work: I need to clone the repository, configure it, merge the patch, update the changelog, build and upload. “

I think that’s fair for someone that wants a web-based workflow. Allow me to present the opposite: for me, I tend to push off contributions that only come through Github, and the reason is that, for me, they’re less convenient. It’s also harder for me to contribute to Github projects than Debian ones. Let’s look at this – say I want to send in a small patch for something. If it’s Github, it’s going to look like this:

  1. Go to the website for the thing, click fork
  2. Now clone that fork or add it to my .git/config, hack, and commit
  3. Push the commit, go back to the website, and submit a PR
  4. Github’s email integration is so poor that I basically have to go back to the website for most parts of the conversation. I can do little from the comfort of mu4e.
  5. Remember to clean up my fork after the patch is accepted or rejected.

Compare that to how I’d contribute with Debian:

  1. Hack (and commit if I feel like it)
  2. Type “reportbug foo”, attach my patch
  3. Followup conversation happens directly in email where it’s convenient to reply

How about as the developer? Github constantly forces me to their website. I can’t very well work on bug reports, etc. without a strong Internet connection. And it’s designed to push people into using their tools and their interface, which is inferior in a lot of ways to a local interface – but then the process to pull down someone else’s set of patches involves a lot of typing and clicking, much more that would be involved from a simple git format-patch. In short, I don’t have my shortcut keys, my environment, etc. for reviewing things – the roadblocks are there to make me use theirs.

If I get a contribution from someone in debbugs, it’s so much easier. It’s usually just git apply or patch -p1 and boom, I can see exactly what’s changed and review it. A review comment is just a reply to an email. I don’t have to ever fire up a web browser. So much more convenient.

I don’t write this to say Michael is wrong about what’s more convenient for him. I write it to say he’s wrong about what’s more convenient for me (or others). It may well be the case that debbugs is so inconvenient that it pushes him to leave while github is so inconvenient for others that it pushes them to avoid it.

I will note before leaving this conversation that there are some command-line tools available for Github and a web interface to debbugs, but it is still clear that debbugs is a lot easier to work with from within my own mail reader and tooling, and Github is a lot easier to work with from within a web browser.

The case for reportbug

I remember the days before we had reportbug. Over and over and over again, I would get bug reports from users that wouldn’t have the basic information needed to investigate. reportbug gathers information from the system: package versions, configurations, versions of dependencies, etc. A simple web form can’t do this because it doesn’t have a local agent. From a developer’s perspective, trying to educate users on how to do this over and over as an unending, frustrating, and counter-productive task. Even if it’s clearly documented, the battle will be fought over and over. From a user’s perspective, having your bug report ignored or told you’re doing it wrong is frustrating too.

So I think reportbug is much nicer than having some github-esque web-based submission form. Could it be better? Sure. I think a mode to submit the reportbug report via HTTPS instead of email would make sense, since a lot of machines no longer have local email configured.

Where Debian Should Improve

I agree that there are areas where Debian should improve.

Michael rightly identifies the “strong maintainer” concept as a source of trouble. I agree. Though we’ve been making slow progress over time with things like low-threshold NMU and maintainer teams, the core assumption that a maintainer has a lot of power over particular packages is one that needs to be thrown out.

Michael, and commentators on HN, both identify things that boil down to documentation problems. I have heard so many times that it’s terribly hard to package something up for Debian. That’s really not the case for most things; dh_make and similar tools will do the right thing for many packages, and all you have to do is add some package descriptions and such. I wrote a “concise guide” to packaging for my workplace that ran to only about 2 pages. But it is true that the documentation on debian.org doesn’t clearly offer this simple path, so people are put off and not aware of it. Then there were the comments about how hard it is to become a Debian developer, and how easy it is to submit a patch to NixOS or some such. The fact is, these are different things; one does not need to be a Debian Developer to contribute to Debian. A DD is effectively the same as a patch approver elsewhere; these are the people that can ultimately approve software for insertion into the OS, and you DO want an element of trust there. Debian could do more to offer concise guides for drive-by contributions and the building of packages that follow standard language community patterns, both of which can be done without much knowledge of packaging tools and inner workings of the project.

Finally, I have distanced myself from conversations in Debian for some time, due to lack of time to participate in what I would call excessive bikeshedding. This is hardly unique to Debian, but I am glad to see the project putting more effort into expecting good behavior from conversations of late.


Review of Secure, Privacy-Respecting Email Services

Freedom, Online Life, Technology,

I’ve been hosting my own email for several decades now. Even before I had access to a dedicated Internet link, I had email via dialup UUCP (and, before that, a FidoNet gateway).

But self-hosting email is becoming increasingly difficult. The time required to maintain spam and virus filters, SPF/DKIM settings, etc. just grows. The importance of email also is increasing. Although my own email has been extremely reliable, it is still running on a single server somewhere and therefore I could stand to have a lot of trouble if it went down while I was unable to fix it

Email with Pretty Good Privacy & Security

(Yes, this heading is a pun.)

There’s a lot of important stuff linked to emails. Family photos. Password resets for banks, social media sites, chat sites, photo storage sites, etc. Shopping histories. In a lot of cases, if your email was compromised, it wouldn’t be all that hard to next compromise your bank account, buy stuff with your Amazon account, hijack your Netflix, etc. There are lots of good resources about why privacy matters; here’s one informative video even if you think you have “nothing to hide”.

There is often a tradeoff between security and usability. A very secure system would be airgapped; you’d always compose your messages and use your secret keys on a system that has no Internet access and never will. Such a system would be quite secure, but not particularly usable.

On the other end of the spectrum are services such as Gmail, which not only make your email available to you, but also to all sorts of other systems within the service that aim to learn about your habits so they can sell this information to advertisers.

This post is about the services in the middle – ones that are usable, can be easily used on mobile devices, and yet make a serious and credible effort to provide better security and privacy than the “big services” run by Google, Yahoo, and Microsoft. Some elements of trust are inherent here; for instance, that the description of the technical nuances of the provider’s services are accurate. (Elements of trust are present in any system; whether your firmware, binaries, etc. are trustworthy.) I used the list at Privacy Tools as a guide to what providers to investigate, supplemented by searches and NoMoreGoogle.

It so happens that most of these services integrate PGP in some way. PGP has long been one of the better ways to have secure communication via email, but it is not always easy for beginners to use. These services make it transparent to a certain degree. None of them are as good as a dedicated client on an airgapped machine, but then again, such a setup isn’t very practical for everyday use. These services give you something better — pretty good, even — but of course not perfect. All of these pay at least lip service to Open Source, some of them actually publishing source for some of their components, but none are fully open.

I pay particular attention to how they handle exchanges with people that do not have PGP, as this kind of communication constitutes the vast majority of my email.

A final comment – if what you really need is an easy and secure way to communicate with one or two people, email itself may not be the right option. Consider Signal.

Protonmail

Protonmail is, in many ways, the gold standard of privacy-respecting email. Every email is stored encrypted in a way that even they can’t see, being decrypted on the client side (using a Javascript PGP implementation or other clients). They definitely seem to be pushing the envelope for security and privacy; they keep no IP logs, don’t require any personal information to set up an account, and go into quite a lot of detail about how your keys are protected.

A side effect of this is that you can’t just access your email with any mail reader. Since the decryption is done on the client side, you pretty much have to use a Protonmail client. They provide clients for iOS and Android, the Web interface, and a “bridge” that exposes IMAP and SMTP ports to localhost and lets you connect a traditional mail client to the system. The bridge, in this case, handles the decryption for you. The bridge works really well and supports Windows, Mac, and Linux, though it is closed source. (The source for the Linux bridge has been “coming soon” for awhile now.) Protonmail provides very good support for bringing your own domain, and in my testing this worked flawlessly. It supports Sieve-based filters, which can also act on envelope recipients (yes!) The web interface is sleek, very well done, tightly integrated, and just generally exceptionally easy to use and just works.

Unfortunately, the mobile clients get the job done for only light use. My opinion: they’re bad. Really bad. For instance:

  • There’s no way to change the sort order on a mail folder
  • The Android client has an option to automatically download all message bodies. The iOS client lacks this option, but no matter; it doesn’t work on Android anyhow.
  • They’re almost completely unusable offline. You can compose a brand new message but that’s it.

There are some other drawbacks. For one, they don’t actually encrypt mail metadata, headers, or subject lines (though this is common to all of the solutions here, Protonmail’s marketing glosses over this). They also seem to have a lot of problems with overly-aggressive systems blocking people’s accounts: here’s a report from 2017, and I’ve seen more recent ones from people that had paid, but then had the account disabled. Apparently protonmail is used by scammers a fair bit and this is a side-effect of offering free, highly secure accounts – some of their deactivations have been legitimate. Nevertheless, it makes me nervous, especially given the high number of reports of this on reddit.

Unfortunately, Proton seems more focused on new products than on fixing these issues. They’ve been long-simmering in the community but what they talk about is more about their upcoming new products.

Protonmail’s terms of service include both a disclaimer that it’s as-is and an SLA, as well as an indemnification clause. Update 2019-03-04: Protonmail’s privacy policy states they use Matomo analytics, that they don’t record your login IP address by default (but IP logs might be kept if you enable it or if they suspect spamming, etc), collect mobile app analytics, IP addresses on incoming messages, etc. Data is retained “indefinitely” for active accounts and for 14 days after account deletion for closed accounts.

Support: email ticket only

Pricing: $4/mo if paid annually; includes 5GB storage, 5 aliases, and 1 custom domain

Location: Switzerland

MFA: TOTP only

Plus address extensions: yes

Transparency report: yes

Mailfence

Mailfence is often mentioned in the same sentence as ProtonMail. They also aim to be a privacy-respecting, secure email solution.

While it is quite possible they use something like LUKS to encrypt data at rest (safeguarding it from a stolen hard drive), unlike ProtonMail, Mailfence does have access to the full content of any plaintext messages sent or received by your account. Mailfence integrates PGP into the Web interface, claiming end-to-end encryption with a “zero-knowledge environment” using, of all things, the same openpgpjs library that is maintained by ProtonMail. While ProtonMail offers a detailed description of key management, I haven’t been able to find this with Mailfence – other than that the private key is stored encrypted on their servers and is protected by a separate passphrase from the login. If we assume the private key is decrypted on the client side, then for PGP-protected communications, the level of security is similar to ProtonMail. With Mailfence, decrypting these messages is a separate operation, while with ProtonMail it happens automatically once logged in. (Update 2019-03-01: Mailfence emailed me, pointing to their document on key storage – it is AES-256 encrypted by the client and stored on the server. They also passed along a link describing their PGP keystore. They also said they plan to work on a feature th encrypt plain text messages.)

While technical measures are part of the story, business policies are another, and Mailfence does seem to have some pretty good policies in place.

In experimenting with it, I found that Mailfence’s filters don’t support filtering based on the envelope recipient, which limits the utility of its aliases since BCC and the like won’t filter properly. A workaround might be possible via the IMAP connection filtering based on Received: headers, but that is somewhat ugly.

Mailfence also supports “secure” documents (word processor, spreadsheet, etc), WebDAV file storage, contacts, and calendars. There is no detail on what makes it “secure” – is it just that it uses TLS or is there something more? I note that the online document editor goes to a URL under writer.zoho.com, so this implies some sort of leakage to me and a possible violation of their “no third-party access to your data” claim. (Update 2019-03-01: Mailfence emailed me to point out that, while it’s not disclosed on the page I liked to, it is disclosed on their blog, and that since I evaluated it, they added a popup warning in the application before sending the documents to Zoho.)

Mailfence supports POP, IMAP, SMTP, and — interestingly — Exchange ActiveSync access to their services. I tested ActiveSync on my Android device, and it appeared to work exactly as planned. This gives a lot of client flexibility and very nice options for calendar and contact sync (*DAV is also supported).

Mailfence’s terms of use is fairly reasonable, though it also includes an indemnification clause. It makes no particular uptime promises. Update 2019-03-04: Per their privacy policy, Mailfence logs IP addresses and use Matomo analytics on the website but not within the application. Deleted messages and documents are retained for 45 days. The policy does not specify retention for logs.

Support: email ticket or business-hours phone support for paying customers

Pricing: EUR 2.50/mo paid annually, includes 5GB storage, 10 aliases, and 1 custom domain

Location: Belgium

MFA: TOTP only

Plus address extensions: Yes

Transparency report: Yes

Mailbox.org

Mailbox.org has been in the hosting business for a long time, and also has a privacy emphasis. Their security is conceptually similar to that of Mailfence. They offer two web-based ways of dealing with PGP: OX Guard and Mailvelope. Mailvelope is a browser extension that does all encryption and decryption on the client side, similar to Mailfence and ProtonMail. OX Guard is part of the Open-Xchange package which mailbox.org uses. It stores the encryption keys on the server, protected by a separate key passphrase, but all encryption and decryption is done server-side. Mailbox’s KB articles on this makes it quite clear and spell out the tradeoffs. The basic upshot is that messages you receive in plaintext will still be theoretically visible to the service itself.

Mailbox.org offers another interesting feature: automatic PGP-encryption of any incoming email that isn’t already encrypted. This encrypts everything inbound. If accessed using Mailvelope or some other external client, it provides equivalent security to ProtonMail. (OX Guard is a little different since the decryption happens server-side.)

They also offer you an @secure.mailbox.org email address that will reject any incoming mail that isn’t properly secured by TLS. You can also send from that address, which will fail to send unless the outgoing connection is properly secured as well. This is one of the more interesting approaches to dealing with the non-PGP-using public. Even if you don’t use that, if you compose in their web interface, you get immediate feedback about the TLS that will be used. It’s not end-to-end, but it’s better than nothing. Mailfence and Protonmail both offer an “secure email” that basically emails a link to a recipient, that links back to their server and requires the recipient to enter a password that was presumably exchanged out of band. Mailbox Guard will automatically go this route when you attempt to send email to someone for whom the PGP keys weren’t known, but goes a step further and invites them to reply there or set up their PGP keys.

Mailbox.org runs Open-Xchange, a semi-Open Source web-based office suite. As such, it also offers calendar, contacts, documents, task lists, IMAP/SMTP/POP, ActiveSync, and so forth. Their KB specifically spells out that things like the calendar are not encrypted with PGP. The filtering does the right thing with envelope recipients.

Mailbox.org has an amazingly comprehensive set of options, a massive knowledge base, even a user forum. Some of the settings I found to be interesting, besides the ones already mentioned, include:

  • Spam settings: greylisting on or off, RBL use, executable file attachment blocking, etc.
  • Restoring email from a backup
  • Disposable addresses (automatically deleted after 30 days)
  • A “catch-all” alias, that just counts as one of your regular aliases, and applies to all usernames under a domain not otherwise aliased.

I know Protonmail has frequent third-party security audits; I haven’t seen any mention of this on the mailbox.org site. However, it looks probable that less of their code was written in house, and it may have been audited without a mention.

Overall, I’ve been pretty impressed with them. They give details on EVERYTHING. It’s the geeky sort of comprehensive, professional solution I’d like. I wish it would have full end-to-end transparent encryption like ProtonMail, but honestly what they’re doing is more practical and useful to a lot of folks.

Mailbox has a reasonable T&C (though it does include an indemnity clause as many others do) and a thorough data protection and privacy policy. Some providers don’t log IP addresses at all; mailbox.org does, but destroys them after 4 days. (Update 2019-03-04: Discovered that all of the providers reviewed may do this at times; updated the other reviews and removed incorrect text; mailbox.org’s is actually one of the better policies) mailbox.org goes into a lot more detail than others, and also explicitly supports things such as Tor for greater anonymity.

Support: email ticket (phone for business-level customers)

Pricing: EUR 1/mo for 2GB storage and 3 aliases; EUR 2.50/mo for 5GB storage and 25 aliases. Expansions possible (for instance, 25GB storage costs a total of EUR 3.50/mo)

Location: Germany

MFA: Yubikey, OATH, TOTP, HOTP, MOTP (web interface only)

Plus address extensions: Yes

Transparency Report: Yes

Startmail

Startmail is a service from the people behind the privacy-respecting search engine Startpage. There is not a lot of information about the technical implementation of Startmail, with the exception of a technical white paper from 2016. It is unclear if this white paper remains accurate, but this review will assume it is. There are also some articles in the knowledge base.

I was unable to fully review Startmail, because the free trial is quite limited (doesn’t even support IMAP) and anything past that level requires an up-front payment of $60. While I paid a few dollars for a month’s real account elsewhere, this was rather too much for a few paragraphs’ review.

However, from the trial, it appears to have a feature set roughly akin to Mailfence. Its mail filters are actually more limited, and it’s mail only: no documents, calendars, etc.

Startmail a somewhat unique setup, in which a person’s mail, PGP keys, etc. are stored in a “vault” which turns out to be a LUKS-encrypted volume. This vault is opened when a person logs in and closed when they log out, and controlled by a derivative of their password. On the one hand, this provides an even stronger level of security than Protonmail (since headers are also encrypted). On the other hand, when the vault is “open” – when one must presume it is quite frequently for an account being polled by IMAP – it is no better than anything else.

They explicitly state that they have not had a third-party audit.

Support: ticket only

Pricing: $60/yr ($5/mo), must be paid as an entire year up-front

Location: Netherlands

MFA: TOTP only

Plus address extensions: unknown

Transparency report: no

Not Reviewed

Some other frequently-used providers I didn’t review carefully:

  • posteo.de: encrypts your mail using a dovecot extension that decrypts it using a derivation of your password when you connect. Something better than nothing but less than Protonmail. Didn’t evaluate because it didn’t support my own domain.
  • Tutanota: Seems to have a security posture similar to ProtonMail, but has no IMAP support at all. If I can’t use emacs to read my mail, I’m not going to bother.

Conclusions

The level of security represented by Protonmail was quite appealing to me. I wish that the service itself was more usable. It looks like an excellent special-needs service, but just isn’t quite there yet as a main mail account for people that have a lot of mail.

I am likely to pursue mailbox.org some more, as although it isn’t as strong as Protonmail when it comes to privacy, it is still pretty good and is amazing on usability and flexibility.

A Final Word on Trust

Trust is a big part of everything going on here. For instance, if you use ProtonMail, where does trust come into play? Well, you trust that they aren’t serving you malicious JavaScript that captures your password and sends it to them out of band. You trust that your browser provides a secure environment for JavaScript and doesn’t have leakage. Or if you use mailbox.org, you trust that the server is providing a secure environment and that when you supply your password for the PGP key, it’s used only for that. ProtonMail will tell you how great it is to have this code client-side. Startmail will tell you how bad Javascript in a browser is for doing things related to security. Both make good, valid points.

To be absolutely sure, it is not possible or practical for any person to verify every component in their stack on every use. Different approaches have different trust models. The very best is still standalone applications.

The providers reviewed here raise the average level of privacy and security on the Internet, and do it by making it easier for the average user. That alone is a good thing and worthy of support. None of them can solve every problem, but all of them are a step up from the standard, which is almost no security at all.

The Python Unicode Mess

Programming

Unicode has solved a lot of problems. Anyone that remembers the mess of ISO-8859-* vs. CP437 (and of course it’s even worse for non-Western languages) can attest to that. And of course, these days they’re doing the useful work of…. codifying emojis.

Emojis aside, things aren’t all so easy. Today’s cause of pain: Python 3. So much pain.

Python decided to fully integrate Unicode into the language. Nice idea, right?

But here come the problems. And they are numerous.

gpodder, for instance, frequently exits with tracebacks due to Python errors converting podcast titles with smartquotes into ASCII. Then you have the case where the pexpect docs say to use logfile = sys.stdout to show the interaction with the virtual terminal. Only that causes an error these days.

But processing of filenames takes the cake. I was recently dealing with data from 20 years ago, before UTF-8 was a filename standard. These filenames are still valid on Unix. tar unpacks them, and they work fine. But you start getting encoding errors from Python trying to do things like store filenames in strings. For a Python program to properly support all valid Unix filenames, it must use “bytes” instead of strings, which has all sorts of annoying implications. What’s the chances that all Python programs do this correctly? Yeah. Not high, I bet.

I recently was processing data generated by mtree, which uses octal escapes for special characters in filenames. I thought this should be easy in Python, eh?

That second link had a mention of an undocumented function, codecs.escape_decode, which does it right. I finally had to do this:

    if line.startswith(b'#'):
        continue
    fields = line.split()
    filename = codecs.escape_decode(fields[0])[0]
    filetype = getfield(b"type", fields[1:])
    if filetype == b"file":

And, whatever you do, don’t accidentally write if filetype == "file" — that will silently always evaluate to False, because "file" tests different than b"file". Not that I, uhm, wrote that and didn’t notice it at first…

So if you want to actually handle Unix filenames properly in Python, you:

  • Must have a processing path that fully avoids Python strings.
  • Must use sys.{stdin,stdout}.buffer instead of just sys.stdin/stdout
  • Must supply filenames as bytes to various functions. See PEP 0471 for this comment: “Like the other functions in the os module, scandir() accepts either a bytes or str object for the path parameter, and returns the DirEntry.name and DirEntry.path attributes with the same type as path. However, it is strongly recommended to use the str type, as this ensures cross-platform support for Unicode filenames. (On Windows, bytes filenames have been deprecated since Python 3.3).” So if you want to be cross-platform, it’s even worse, because you can’t use str on Unix nor bytes on Windows.

Update: Would you like to receive filenames on the command line? I’ll hand you this fine mess. And the environment? it’s not even clear.

Please stop making the library situation worse with attempts to fix it

Linux

I recently had a simple-sounding desire. I would like to run the latest stable version of Digikam. My desktop, however, runs Debian stable, which has 5.3.0, not 5.9.0.

This is not such a simple proposition.


$ ldd /usr/bin/digikam | wc -l
396

And many of those were required at versions that weren’t in stable.

I had long thought that AppImage was a rather bad idea, but I decided to give it a shot. I realized it was worse than I had thought.

The problems with AppImage

About a year ago, I wrote about the problems Docker security. I go into much more detail there, but the summary for AppImage is quite similar. How can I trust all the components in the (for instance) Digikam AppImage image are being kept secure? Are they using the latest libssl and libpng, to avoid security issues? How will I get notified of a security update? (There seems to be no mechanism for this right now.) An AppImage user that wants to be secure has to manually answer every one of those questions for every application. Ugh.

Nevertheless, the call of better facial detection beckoned, and I downloaded the Digikam AppImage and gave it a whirl. The darn thing actually fired up. But when it would play videos, there was no sound. Hmmmm.

I found errors like this:

Cannot access file ././/share/alsa/alsa.conf

Nasty. I spent quite some time trying to make ALSA work, before a bunch of experimentation showed that if I ran alsoft-conf on the host, and selected only the PulseAudio backend, then it would work. I reported this bug to Digikam.

Then I thought it was working — until I tried to upload some photos. It turns out that SSL support in Qt in the AppImage was broken, since it was trying to dlopen an incompatible version of libssl or libcrypto on the host. More details are in the bug I reported about this also.

These are just two examples. In the rather extensive Googling I did about these problems, I came across issue after issue people had with running Digikam in an AppImage. These issues are not limited to the ALSA and SSL issues I describe here. And they are not occurring due to some lack of skill on the part of Digikam developers.

Rather, they’re occurring because AppImage packaging for a complex package like this is hard. It’s hard because it’s based on a fiction — the fiction that it’s possible to make an AppImage container for a complex desktop application act exactly the same, when the host environment is not exactly the same. Does the host run PulseAudio or ALSA? Where are its libraries stored? How do you talk to dbus?

And it’s not for lack of trying. The scripts to build the Digikam appimage support runs to over 1000 lines of code in the AppImage directory, plus another 1300 lines of code (at least) in CMake files that handle much of the work, and another 3000 lines or so of patches to 3rd-party packages. That’s over 5000 lines of code! By contrast, the Debian packaging for the same version of Digikam, including Debian patches but excluding the changelog and copyright files, amounts to 517 lines. Of course, it is reusing OS packages for the dependencies that were already built, but this amounts to a lot simpler build.

Frankly I don’t believe that AppImage really lives up to its hype. Requiring reinventing a build system and making some dangerous concessions on security for something that doesn’t really work in the end — not good in my book.

The library problem

But of course, AppImage exists for a reason. That reason is that it’s a real pain to deal with so many levels of dependencies in software. Even if we were to compile from source like the old days, and even if it was even compatible with the versions of the dependencies in my OS, that’s still a lot of work. And if I have to build dependencies from source, then I’ve given up automated updates that way too.

There’s a lot of good that ELF has brought us, but I can’t help but think that it wasn’t really designed for a world in which a program links 396 libraries (plus dlopens a few more). Further, this world isn’t the corporate Unix world of the 80s; Open Source developers aren’t big on maintaining backwards compatibility (heck, both the KDE and Qt libraries under digikam have both been entirely rewritten in incompatible ways more than once!) The farther you get from libc, the less people seem to care about backwards compatibility. And really, who can blame volunteers? You want to work on new stuff, not supporting binaries from 5 years ago, right?

I don’t really know what the solution is here. Build-from-source approaches like FreeBSD and Gentoo have plenty of drawbacks too. Is there some grand solution I’m missing? Some effort to improve this situation without throwing out all the security benefits that individually-packaged libraries give us in distros like Debian?

Emacs #5: Documents and Presentations with org-mode

Software,

The Emacs series

This is fifth in a series on Emacs and org-mode.

This blog post was generated from an org-mode source and is available as: a blog page, slides (PDF format), and a PDF document.

1 About org-mode exporting

1.1 Background

org-mode isn't just an agenda-making program. It can also export to lots of formats: LaTeX, PDF, Beamer, iCalendar (agendas), HTML, Markdown, ODT, plain text, man pages, and more complicated formats such as a set of web pages.

This isn't just some afterthought either; it's a core part of the system and integrates very well.

One file can be source code, automatically-generated output, task list, documentation, and presentation, all at once.

Some use org-mode as their preferred markup format, even for things like LaTeX documents. The org-mode manual has an extensive section on exporting.

1.2 Getting started

From any org-mode document, just hit C-c C-e. From there will come up a menu, letting you choose various export formats and options. These are generally single-key options so it's easy to set and execute. For instance, to export a document to a PDF, use C-c C-e l p or for HTML export, C-c C-e h h.

There are lots of settings available for all of these export options; see the manual. It is, in fact, quite possible to use LaTeX-format equations in both LaTeX and HTML modes, to insert arbitrary preambles and settings for different modes, etc.

1.3 Add-on packages

ELPA containts many addition exporters for org-mode as well. Check there for details.

2 Beamer slides with org-mode

2.1 About Beamer

Beamer is a LaTeX environment for making presentations. Its features include:

  • Automated generating of structural elements in the presentation (see, for example, the Marburg theme). This provides a visual reference for the audience of where they are in the presentation.
  • Strong help for structuring the presentation
  • Themes
  • Full LaTeX available

2.2 Benefits of Beamer in org-mode

org-mode has a lot of benefits for working with Beamer. Among them:

  • org-mode's very easy and strong support for visualizing and changing the structure makes it very quick to reorganize your material.
  • Combined with org-babel, live source code (with syntax highlighting) and results can be embedded.
  • The syntax is often easier to work with.

I have completely replaced my usage of LibreOffice/Powerpoint/GoogleDocs with org-mode and beamer. It is, in fact, rather frustrating when I have to use one of those tools, as they are nowhere near as strong as org-mode for visualizing a presentation structure.

2.3 Headline Levels

org-mode's Beamer export will convert sections of your document (defined by headings) into slides. The question, of course, is: which sections? This is governed by the H export setting (org-export-headline-levels).

There are many ways to go, which suit people. I like to have my presentation like this: 

#+OPTIONS: H:2
#+BEAMER_HEADER: \AtBeginSection{\frame{\sectionpage}}

This gives a standalone section slide for each major topic, to highlight major transitions, and then takes the level 2 (two asterisks) headings to set the slide. Many Beamer themes expect a third level of indirection, so you would set H:3 for them.

2.4 Themes and settings

You can configure many Beamer and LaTeX settings in your document by inserting lines at the top of your org file. This document, for instance, defines: 

#+TITLE:  Documents and presentations with org-mode
#+AUTHOR: John Goerzen
#+BEAMER_HEADER: \institute{The Changelog}
#+PROPERTY: comments yes
#+PROPERTY: header-args :exports both :eval never-export
#+OPTIONS: H:2
#+BEAMER_THEME: CambridgeUS
#+BEAMER_COLOR_THEME: default

2.5 Advanced settings

I like to change some colors, bullet formatting, and the like. I round out my document with: 

# We can't just +BEAMER_INNER_THEME: default because that picks the theme default.
# Override per https://tex.stackexchange.com/questions/11168/change-bullet-style-formatting-in-beamer
#+BEAMER_INNER_THEME: default
#+LaTeX_CLASS_OPTIONS: [aspectratio=169]
#+BEAMER_HEADER: \definecolor{links}{HTML}{0000A0}
#+BEAMER_HEADER: \hypersetup{colorlinks=,linkcolor=,urlcolor=links}
#+BEAMER_HEADER: \setbeamertemplate{itemize items}[default]
#+BEAMER_HEADER: \setbeamertemplate{enumerate items}[default]
#+BEAMER_HEADER: \setbeamertemplate{items}[default]
#+BEAMER_HEADER: \setbeamercolor*{local structure}{fg=darkred}
#+BEAMER_HEADER: \setbeamercolor{section in toc}{fg=darkred}
#+BEAMER_HEADER: \setlength{\parskip}{\smallskipamount}

Here, aspectratio=169 sets a 16:9 aspect ratio, and the remaining are standard LaTeX/Beamer configuration bits.

2.6 Shrink (to fit)

Sometimes you've got some really large code examples and you might prefer to just shrink the slide to fit.

Just type C-c C-x p, set the BEAMER_opt property to shrink=15.

(Or a larger value of shrink). The previous slide uses this here.

2.7 Result

Here's the end result:

screenshot1

3 Interactive Slides

3.1 Interactive Emacs Slideshows

With the org-tree-slide package, you can display your slideshow from right within Emacs. Just run M-x org-tree-slide-mode. Then, use C-> and C-< to move between slides.

You might find C-c C-x C-v (which is org-toggle-inline-images) helpful to cause the system to display embedded images.

3.2 HTML Slideshows

There are a lot of ways to export org-mode presentations to HTML, with various levels of JavaScript integration. See the non-beamer presentations section of the org-mode wiki for details.

4 Miscellaneous

4.1 Additional resources to accompany this post

4.2 Up next in my Emacs series…

mu4e for email!

Emacs #4: Automated emails to org-mode and org-mode syncing

Linux, Software, ,

This is fourth in a series on Emacs and org-mode.

Hopefully by now you’ve started to see how powerful and useful org-mode is. If you’re like me, you’re thinking:

“I’d really like to have this in sync across all my devices.”

and, perhaps:

“Can I forward emails into org-mode?”

This being Emacs, the answers, of course, are “Yes.”

Syncing

Since org-mode just uses text files, syncing is pretty easily accomplished using any number of tools. I use git with git-remote-gcrypt. Due to some limitations of git-remote-gcrypt, each machine tends to push to its own branch, and to master on command. Each machine merges from all the other branches and pushes the result to master after a merge. A cron job causes pushes to the machine’s branch to happen, and a bit of elisp coordinates it all — making sure to save buffers before a sync, refresh them from disk after, etc.

The code for this post is somewhat more extended, so I will be linking to it on github rather than posting inline.

I have a directory $HOME/org where all my org-stuff lives. In ~/org lives a Makefile that handles the syncing. It defines these targets:

  • push: adds, commits, and pushes to a branch named after the machine’s hostname
  • fetch: does a simple git fetch
  • sync: adds, commits, pulls remote changes, merges, and (assuming the merge was successful) pushes to the branch named after the machine’s hostname plus master

Now, in my user’s crontab, I have this:

*/15   *   *  *   *      make -C $HOME/org push fetch 2>&1 | logger --tag 'orgsync'

The accompanying elisp code defines a shortcut (C-c s) to cause a sync to occur. Thanks to the cronjob, as long as files were saved — even if I didn’t explicitly sync on the other boxen — they’ll be pulled in.

I have found this setup to work really well.

Emailing to org-mode

Before going down this path, one should ask the question: do you really need it? I use org-mode with mu4e, and the integration is excellent; any org task can link to an email by message-id, and this is ideal — it lets a person do things like make a reminder to reply to a message in a week.

However, org is not just about reminders. It’s also a knowledge base, authoring system, etc. And, not all of my mail clients use mu4e. (Note: things like MobileOrg exist for mobile devices). I don’t actually use this as much as I thought I would, but it has its uses and I thought I’d document it here too.

Now I didn’t want to just be able to accept plain text email. I wanted to be able to handle attachments, HTML mail, etc. This quickly starts to sound problematic — but with tools like ripmime and pandoc, it’s not too bad.

The first step is to set up some way to get mail into a specific folder. A plus-extension, special user, whatever. I then use a fetchmail configuration to pull it down and run my insorgmail script.

This script is where all the interesting bits happen. It starts with ripmime to process the message. HTML bits are converted from HTML to org format using pandoc. an org hierarchy is made to represent the structure of the email as best as possible. emails can get pretty complicated, with HTML and the rest, but I have found this does an acceptable job with my use cases.

Up next…

My last post on org-mode will talk about using it to write documents and prepare slides — a use for which I found myself surprisingly pleased with it, but which needed a bit of tweaking.

Emacs #3: More on org-mode

Software, ,

This is third in a series on Emacs and org-mode.

Todo tracking and keywords

When using org-mode to track your TODOs, it can have multiple states. You can press C-c C-t for a quick shift between states. I have set this:

(setq org-todo-keywords '(
  (sequence "TODO(t!)" "NEXT(n!)" "STARTED(a!)" "WAIT(w@/!)" "OTHERS(o!)" "|" "DONE(d)" "CANCELLED(c)")
))

Here, I set up 5 states that are for a task that is not yet done: TODO, NEXT, STARTED, WAIT, and OTHERS. Each has a single-character shortcut (t, n, a, etc). The states after the pipe symbol are ones that are considered “done”. I have two: DONE (for things that I have done) and CANCELED (for things that I haven’t done, but for whatever reason, won’t).

The exclamation mark means to log the time when an item was changed to a state. I don’t add this to the done states because those are already logged anyhow. The @ sign means to prompt for a reason; so when switching to WAIT, org-mode will ask me why and add this to the note.

Here’s an example of an entry that has had some state changes:

** DONE This is a test
   CLOSED: [2018-03-02 Fri 03:05]
  
   - State "DONE"       from "WAIT"       [2018-03-02 Fri 03:05]
   - State "WAIT"       from "TODO"       [2018-03-02 Fri 03:05] \\
     waiting for pigs to fly
   - State "TODO"       from "NEXT"       [2018-03-02 Fri 03:05]
   - State "NEXT"       from "TODO"       [2018-03-02 Fri 03:05]

Here, the most recent items are on top.

Agenda mode, schedules, and deadlines

When you’re in a todo item, C-c C-s or C-c C-d can set a schedule or a deadline for it, respectively. These show up in agenda mode. The difference is in intent and presentation. A schedule is something that you expect to work on at around a time, while a deadline is something that is due at a specific time. By default, the agenda view will start warning you about deadline items in advance.

And while we’re at it, the agenda view will show you the items that you have coming up, offers a nice way to search for items based on plain text or tags, and handles bulk manipulation of items even across multiple files. I covered setting the files for agenda mode in part 2 of this series.

Tags

Of course org-mode has tags. You can quickly set them with C-c C-q.

You can set shortcuts for tags you might like to use often. Perhaps something like this:

  (setq org-tag-persistent-alist 
        '(("@phone" . ?p) 
          ("@computer" . ?c) 
          ("@websurfing" . ?w)
          ("@errands" . ?e)
          ("@outdoors" . ?o)
          ("MIT" . ?m)
          ("BIGROCK" . ?b)
          ("CONTACTS" . ?C)
          ("INBOX" . ?i)
          ))

You can also add tags to this list on a per-file basis, and also set tags for something on a per-file basis. I use that for my inbox.org and email.org files to set an INBOX tag. I can then review all items tagged INBOX from the agenda view each day, and the simple act of refiling them into other files will cause them to lost the INBOX tag.

Refiling

“Refiling” is moving things around, either within a file or elsewhere. It has completion using your headlines. C-c C-w does this. I like these settings:

(setq org-outline-path-complete-in-steps nil)         ; Refile in a single go
(setq org-refile-use-outline-path 'file)

Archiving

After awhile, you’ll get your files all cluttered with things that are done. org-mode has an archive feature to move things out of your main .org files and into some other files for future reference. If you have your org files in git or something, you may wish to delete these other files since you’d have things in history anyhow, but I find them handy for grepping and searching.

I periodically want to go through and archive everything in my files. Based on a stackoverflow discussion, I have this code:

(defun org-archive-done-tasks ()
  (interactive)
  (org-map-entries
   (lambda ()
     (org-archive-subtree)
     (setq org-map-continue-from (outline-previous-heading)))
   "/DONE" 'file)
  (org-map-entries
   (lambda ()
     (org-archive-subtree)
     (setq org-map-continue-from (outline-previous-heading)))
   "/CANCELLED" 'file)
)

This is based on a particular answer — see the comments there for some additional hints. Now you can run M-x org-archive-done-tasks and everything in the current file marked DONE or CANCELED will be pulled out into a different file.

Up next

I’ll wrap up org-mode with a discussion of automatically receiving emails into org, and syncing org between machines.

Resources to accompany this article

Emacs #2: Introducing org-mode

Software, ,

In my first post in my series on Emacs, I described returning to Emacs after over a decade of vim, and org-mode being the reason why.

I really am astounded at the usefulness, and simplicity, of org-mode. It is really a killer app.

So what exactly is org-mode?

I wrote yesterday:

It’s an information organization platform. Its website says “Your life in plain text: Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.”

That’s true, but doesn’t quite capture it. org-mode is a toolkit for you to organize things. It has reasonable out-of-the-box defaults, but it’s designed throughout for you to customize.

To highlight a few things:

  • Maintaining TODO lists: items can be scattered across org-mode files, contain attachments, have tags, deadlines, schedules. There is a convenient “agenda” view to show you what needs to be done. Items can repeat.
  • Authoring documents: org-mode has special features for generating HTML, LaTeX, slides (with LaTeX beamer), and all sorts of other formats. It also supports direct evaluation of code in-buffer and literate programming in virtually any Emacs-supported language. If you want to bend your mind on this stuff, read this article on literate devops. The entire Worg website
    is made with org-mode.
  • Keeping notes: yep, it can do that too. With full-text search, cross-referencing by file (as a wiki), by UUID, and even into other systems (into mu4e by Message-ID, into ERC logs, etc, etc.)

Getting started

I highly recommend watching Carsten Dominik’s excellent Google Talk on org-mode. It is an excellent introduction.

org-mode is included with Emacs, but you’ll often want a more recent version. Debian users can apt-get install org-mode, or it comes with the Emacs packaging system; M-x package-install RET org-mode RET may do it for you.

Now, you’ll probably want to start with the org-mode compact guide’s introduction section, noting in particular to set the keybindings mentioned in the activation section.

A good tutorial…

I’ve linked to a number of excellent tutorials and introductory items; this post is not going to serve as a tutorial. There are two good videos linked at the end of this post, in particular.

Some of my configuration

I’ll document some of my configuration here, and go into a bit of what it does. This isn’t necessarily because you’ll want to copy all of this verbatim — but just to give you a bit of an idea of some of what can be configured, an idea of what to look up in the manual, and maybe a reference for “now how do I do that?”

First, I set up Emacs to work in UTF-8 by default.

(prefer-coding-system 'utf-8)
(set-language-environment "UTF-8")

org-mode can follow URLs. By default, it opens in Firefox, but I use Chromium.

(setq browse-url-browser-function 'browse-url-chromium)

I set the basic key bindings as documented in the Guide, plus configure the M-RET behavior.

(global-set-key "\C-cl" 'org-store-link)
(global-set-key "\C-ca" 'org-agenda)
(global-set-key "\C-cc" 'org-capture)
(global-set-key "\C-cb" 'org-iswitchb)

(setq org-M-RET-may-split-line nil)

Configuration: Capturing

I can press C-c c from anywhere in Emacs. It will capture something for me, and include a link back to whatever I was working on.

You can define capture templates to set how this will work. I am going to keep two journal files for general notes about meetings, phone calls, etc. One for personal, one for work items. If I press C-c c j, then it will capture a personal item. The %a in all of these includes the link to where I was (or a link I had stored with C-c l).

(setq org-default-notes-file "~/org/tasks.org")
(setq org-capture-templates
      '(
        ("t" "Todo" entry (file+headline "inbox.org" "Tasks")
         "* TODO %?\n  %i\n  %u\n  %a")
        ("n" "Note/Data" entry (file+headline "inbox.org" "Notes/Data")
         "* %?   \n  %i\n  %u\n  %a")
        ("j" "Journal" entry (file+datetree "~/org/journal.org")
         "* %?\nEntered on %U\n %i\n %a")
        ("J" "Work-Journal" entry (file+datetree "~/org/wjournal.org")
         "* %?\nEntered on %U\n %i\n %a")
        ))
(setq org-irc-link-to-logs t)

I like to link by UUIDs, which lets me move things between files without breaking locations. This helps generate UUIDs when I ask Org to store a link target for future insertion.


(require 'org-id)
(setq org-id-link-to-org-use-id 'create-if-interactive)

Configuration: agenda views

I like my week to start on a Sunday, and for org to note the time when I mark something as done.


(setq org-log-done 'time)
(setq org-agenda-start-on-weekday 0)

Configuration: files and refiling

Here I tell it what files to use in the agenda, and to add a few more to the plain text search. I like to keep a general inbox (from which I can move, or “refile”, content), and then separate tasks, journal, and knowledge base for personal and work items.

  (setq org-agenda-files (list "~/org/inbox.org"
                               "~/org/email.org"
                               "~/org/tasks.org"
                               "~/org/wtasks.org"
                               "~/org/journal.org"
                               "~/org/wjournal.org"
                               "~/org/kb.org"
                               "~/org/wkb.org"
  ))
  (setq org-agenda-text-search-extra-files
        (list "~/org/someday.org"
              "~/org/config.org"
  ))

  (setq org-refile-targets '((nil :maxlevel . 2)
                             (org-agenda-files :maxlevel . 2)
                             ("~/org/someday.org" :maxlevel . 2)
                             ("~/org/templates.org" :maxlevel . 2)
                             )
        )
(setq org-outline-path-complete-in-steps nil)         ; Refile in a single go
(setq org-refile-use-outline-path 'file)

Configuration: Appearance

I like a pretty screen. After you’ve gotten used to org a bit, you might try this.

(require 'org-bullets)
(add-hook 'org-mode-hook
          (lambda ()
            (org-bullets-mode t)))
(setq org-ellipsis "⤵")

Coming up next…

This hopefully showed a few things that org-mode can do. Coming up next, I’ll cover how to customize TODO keywords and tags, archiving old tasks, forwarding emails to org-mode, and using git to synchronize between machines.

You can also see a list of all articles in this series.

Resources to accompany this article

Emacs #1: Ditching a bunch of stuff and moving to Emacs and org-mode

Software, ,

I’ll admit it. After over a decade of vim, I’m hooked on Emacs.

I’ve long had this frustration over how to organize things. I’ve followed approaches like GTD and ZTD, but things like email or large files are really hard to organize.

I had been using Asana for tasks, Evernote for notes, Thunderbird for email, a combination of ikiwiki and some other items for a personal knowledge base, and various files in an archive directory on my PC. When my new job added Slack to the mix, that was finally the last straw.

A lot of todo-management tools integrate with email — poorly. When you want to do something like “remind me to reply to this in a week”, a lot of times that’s impossible because the tool doesn’t store the email in a fashion you can easily reply to. And that problem is even worse with Slack.

It was right around then that I stumbled onto Carsten Dominik’s Google Talk on org-mode. Carsten was the author of org-mode, and although the talk is 10 years old, it is still highly relevant.

I’d stumbled across org-mode before, but each time I didn’t really dig in because I had the reaction of “an outliner? But I need a todo list.” Turns out I was missing out. org-mode is all that.

Just what IS Emacs? And org-mode?

Emacs grew up as a text editor. It still is, and that heritage is definitely present throughout. But to say Emacs is an editor would be rather unfair.

Emacs is something more like a platform or a toolkit. Not only do you have source code to it, but the very configuration is a program, and there are hooks all over the place. It’s as if it was super easy to write a Firefox plugin. A couple lines, and boom, behavior changed.

org-mode is very similar. Yes, it’s an outliner, but that’s not really what it is. It’s an information organization platform. Its website says “Your life in plain text: Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.”

Capturing

If you’ve ever read productivity guides based on GTD, one of the things they stress is effortless capture of items. The idea is that when something pops into your head, get it down into a trusted system quickly so you can get on with what you were doing. org-mode has a capture system for just this. I can press C-c c from anywhere in Emacs, and up pops a spot to type my note. But, critically, automatically embedded in that note is a link back to what I was doing when I pressed C-c c. If I was editing a file, it’ll have a link back to that file and the line I was on. If I was viewing an email, it’ll link back to that email (by Message-Id, no less, so it finds it in any folder). Same for participating in a chat, or even viewing another org-mode entry.

So I can make a note that will remind me in a week to reply to a certain email, and when I click the link in that note, it’ll bring up the email in my mail reader — even if I subsequently archived it out of my inbox.

YES, this is what I was looking for!

The tool suite

Once you’re using org-mode, pretty soon you want to integrate everything with it. There are browser plugins for capturing things from the web. Multiple Emacs mail or news readers integrate with it. ERC (IRC client) does as well. So I found myself switching from Thunderbird and mairix+mutt (for the mail archives) to mu4e, and from xchat+slack to ERC.

And wouldn’t you know it, I liked each of those Emacs-based tools better than the standalone they replaced.

A small side tidbit: I’m using OfflineIMAP again! I even used it with GNUS way back when.

One Emacs process to rule them

I used to use Emacs extensively, way back. Back then, Emacs was a “large” program. (Now my battery status applet literally uses more RAM than Emacs). There was this problem of startup time back then, so there was a way to connect to a running Emacs process.

I like to spawn programs with Mod-p (an xmonad shortcut to a dzen menubar, but Alt-F2 in more traditional DEs would do the trick). It’s convenient to not run several emacsen with this setup, so you don’t run into issues with trying to capture to a file that’s open in another one. The solution is very simple: I created a script, named it em, and put it on my path. All it does is this:


#!/bin/bash
exec emacsclient -c -a "" "$@"

It creates a new emacs process if one doesn’t already exist; otherwise, it uses what you’ve got. A bonus here: parameters such as -nw work just fine, so it really acts just as if you’d typed emacs at the shell prompt. It’s a suitable setting for EDITOR.

Up next…

I’ll be talking about my use of, and showing off configurations for:

  • org-mode, including syncing between computers, capturing, agenda and todos, files, linking, keywords and tags, various exporting (slideshows), etc.
  • mu4e for email, including multiple accounts, bbdb integration
  • ERC for IRC and IM

You can also see a list of all articles in this series.