Monthly Archives: March 2018

Emacs #4: Automated emails to org-mode and org-mode syncing

This is fourth in a series on Emacs and org-mode.

Hopefully by now you’ve started to see how powerful and useful org-mode is. If you’re like me, you’re thinking:

“I’d really like to have this in sync across all my devices.”

and, perhaps:

“Can I forward emails into org-mode?”

This being Emacs, the answers, of course, are “Yes.”

Syncing

Since org-mode just uses text files, syncing is pretty easily accomplished using any number of tools. I use git with git-remote-gcrypt. Due to some limitations of git-remote-gcrypt, each machine tends to push to its own branch, and to master on command. Each machine merges from all the other branches and pushes the result to master after a merge. A cron job causes pushes to the machine’s branch to happen, and a bit of elisp coordinates it all — making sure to save buffers before a sync, refresh them from disk after, etc.

The code for this post is somewhat more extended, so I will be linking to it on github rather than posting inline.

I have a directory $HOME/org where all my org-stuff lives. In ~/org lives a Makefile that handles the syncing. It defines these targets:

  • push: adds, commits, and pushes to a branch named after the machine’s hostname
  • fetch: does a simple git fetch
  • sync: adds, commits, pulls remote changes, merges, and (assuming the merge was successful) pushes to the branch named after the machine’s hostname plus master

Now, in my user’s crontab, I have this:

*/15   *   *  *   *      make -C $HOME/org push fetch 2>&1 | logger --tag 'orgsync'

The accompanying elisp code defines a shortcut (C-c s) to cause a sync to occur. Thanks to the cronjob, as long as files were saved — even if I didn’t explicitly sync on the other boxen — they’ll be pulled in.

I have found this setup to work really well.

Emailing to org-mode

Before going down this path, one should ask the question: do you really need it? I use org-mode with mu4e, and the integration is excellent; any org task can link to an email by message-id, and this is ideal — it lets a person do things like make a reminder to reply to a message in a week.

However, org is not just about reminders. It’s also a knowledge base, authoring system, etc. And, not all of my mail clients use mu4e. (Note: things like MobileOrg exist for mobile devices). I don’t actually use this as much as I thought I would, but it has its uses and I thought I’d document it here too.

Now I didn’t want to just be able to accept plain text email. I wanted to be able to handle attachments, HTML mail, etc. This quickly starts to sound problematic — but with tools like ripmime and pandoc, it’s not too bad.

The first step is to set up some way to get mail into a specific folder. A plus-extension, special user, whatever. I then use a fetchmail configuration to pull it down and run my insorgmail script.

This script is where all the interesting bits happen. It starts with ripmime to process the message. HTML bits are converted from HTML to org format using pandoc. an org hierarchy is made to represent the structure of the email as best as possible. emails can get pretty complicated, with HTML and the rest, but I have found this does an acceptable job with my use cases.

Up next…

My last post on org-mode will talk about using it to write documents and prepare slides — a use for which I found myself surprisingly pleased with it, but which needed a bit of tweaking.

A Grand Adventure: Sailing the Springtime Sky

And forget not that the earth delights to feel your bare feet and the winds long to play with your hair.

– Khalil Gibran

To be a pilot of a small plane is to be a scientist, a mathematician, and a poet. We read charts, analyze weather reports and forecasts, calculating what the headwinds will do to our fuel situation.

But in the end, the wise ones let the earth speak to us through these charts, and go where it invites us — where the skies are smooth and blue.

And so it was last week that we did not go to California as planned, but instead to the mountains near Santa Fe, a canyon near Amarillo, and a remote museum in far southwest Kansas — all the while hearing the delighted exclamations of “wow!” from our children.

IMG_20180317_091955

As we sailed along up there in our flying machine, down below we saw the rugged, craggy mesas of New Mexico, here and there punctuated by a lake, a little town, or an isolated airport — each a friendly sight in its own way. Our boys read some books, and sometimes pressed their noses to the windows, while little Martha mostly slept and sometimes played or ate — she enjoys flying more than driving.

IMG_4629_bw

Mountains have a way of reminding us all that the earth is larger than we are. We drive around them, fly around them, and even on a pleasant day they make the air bumpy. But once down on the ground, Oliver got out of the plane, and looked at the mountains all around us. He couldn’t stop saying “Wow! Dad, wow! Amazing! Look at that!” Jacob was more excited that an American Airlines plane was taxiing by right where we had been a minute before.

IMG_4667_v1

The boys helped us plan our trip. They’re the ones that chose for us to head southwest, and the #1 thing on their Santa Fe agenda was riding the New Mexico Rail Runner Express. So, despite a strong and cold wind, ride it we did, all the way to Albuquerque for pizza, then back to Santa Fe. When they weren’t busy listening to the “meep meep” sound the doors make when they’re about to close, they were excitedly reading the timetable or taking in the world as it whizzed past their window.

IMG_4680_bw

Martha, too, took in the train — though she still enjoyed her chew toys. Those things out the window don’t fit into a mouth so easily.

IMG_4743

Up in the mountains, the Puye Cliff Dwellings brought home the history of the place. The stories of the peaceful people that lived there, told by their descendants, members of the Santa Clara pueblo. Our guide Elijah picked up a shard of pottery, many of which remain on the mesa. He explained why there were no intact pottery examples up there. When his ancestors were done with a pot, then would throw it on the ground, shattering it — to give it back with thanks to the earth from where it came. One gets the sense that these ancient peoples knew quite a few things that our modern societies have not yet learned.

IMG_4639

After a full day, a cool evening in our hotel was welcome. Our room had a wood-burning fireplace, burning the pinyon pine that gives Santa Fe such a distinct sweet smell in the winter. Jacob would gaze into the fireplace for quite awhile.

P1070297_v1

I have never seen a photo do justice to Palo Duro Canyon. As you drive along the desolate, high Texas prairie, complete with tumbleweeds, all of a sudden you go around a corner and the earth opens up. It’s the “painted canyon” for a reason, and even though we’d been there before, as we rounded that bend, I heard exclamations of “Dad, this is AMAZING” from the back seat once again.

The vastness of the place cannot be captured on a screen. How can one capture 60 miles of color, ridges and gorges stretching out into the horizon, in a few inches?

MVI_4844

The boys were excited, bubbly, and bouncy as we hiked along some trails on the canyon floor. They’d make up games to play, most of which involved teasing me in some way.

IMG_4841

Oliver particularly loved to tease me.

IMG_4812

Jacob insisted I take a picture with him and Martha.

IMG_4942

But even these excited, bubbly kids would stop to reflect sometimes. Sometimes Jacob would say, “Dad, I have GOT to take a picture of this!” And sometimes they would just stare, maybe even with a mouth agape. Children know beauty, too.

MVI_5075

The three siblings delight in each other, too. Oliver would play a version of peek-a-boo, saying “I’m alive! Horse pill!” (he’d say silly things, and whatever made Martha laugh he’d keep saying.)

map

All told, we traveled over a thousand miles by air, spending some 7 or 8 hours in the plane. Had we attempted to drive it, it would have been more than 30 hours. There’s something amazing about seeing so much of the world in such a short amount of time.

IMG_20180228_180503_bw

Perhaps this is why many pilots secretly give their plane a little pat after a long journey. After all the smiles, laughter, the memories, the feeling of being part of sky — if you pay attention, it truly is more poetry than physics.

And that is why, though it is always nice to return to our home, in my mind’s eye, the hangar door is always open.

Emacs #3: More on org-mode

This is third in a series on Emacs and org-mode.

Todo tracking and keywords

When using org-mode to track your TODOs, it can have multiple states. You can press C-c C-t for a quick shift between states. I have set this:

(setq org-todo-keywords '(
  (sequence "TODO(t!)" "NEXT(n!)" "STARTED(a!)" "WAIT(w@/!)" "OTHERS(o!)" "|" "DONE(d)" "CANCELLED(c)")
))

Here, I set up 5 states that are for a task that is not yet done: TODO, NEXT, STARTED, WAIT, and OTHERS. Each has a single-character shortcut (t, n, a, etc). The states after the pipe symbol are ones that are considered “done”. I have two: DONE (for things that I have done) and CANCELED (for things that I haven’t done, but for whatever reason, won’t).

The exclamation mark means to log the time when an item was changed to a state. I don’t add this to the done states because those are already logged anyhow. The @ sign means to prompt for a reason; so when switching to WAIT, org-mode will ask me why and add this to the note.

Here’s an example of an entry that has had some state changes:

** DONE This is a test
   CLOSED: [2018-03-02 Fri 03:05]
  
   - State "DONE"       from "WAIT"       [2018-03-02 Fri 03:05]
   - State "WAIT"       from "TODO"       [2018-03-02 Fri 03:05] \\
     waiting for pigs to fly
   - State "TODO"       from "NEXT"       [2018-03-02 Fri 03:05]
   - State "NEXT"       from "TODO"       [2018-03-02 Fri 03:05]

Here, the most recent items are on top.

Agenda mode, schedules, and deadlines

When you’re in a todo item, C-c C-s or C-c C-d can set a schedule or a deadline for it, respectively. These show up in agenda mode. The difference is in intent and presentation. A schedule is something that you expect to work on at around a time, while a deadline is something that is due at a specific time. By default, the agenda view will start warning you about deadline items in advance.

And while we’re at it, the agenda view will show you the items that you have coming up, offers a nice way to search for items based on plain text or tags, and handles bulk manipulation of items even across multiple files. I covered setting the files for agenda mode in part 2 of this series.

Tags

Of course org-mode has tags. You can quickly set them with C-c C-q.

You can set shortcuts for tags you might like to use often. Perhaps something like this:

  (setq org-tag-persistent-alist 
        '(("@phone" . ?p) 
          ("@computer" . ?c) 
          ("@websurfing" . ?w)
          ("@errands" . ?e)
          ("@outdoors" . ?o)
          ("MIT" . ?m)
          ("BIGROCK" . ?b)
          ("CONTACTS" . ?C)
          ("INBOX" . ?i)
          ))

You can also add tags to this list on a per-file basis, and also set tags for something on a per-file basis. I use that for my inbox.org and email.org files to set an INBOX tag. I can then review all items tagged INBOX from the agenda view each day, and the simple act of refiling them into other files will cause them to lost the INBOX tag.

Refiling

“Refiling” is moving things around, either within a file or elsewhere. It has completion using your headlines. C-c C-w does this. I like these settings:

(setq org-outline-path-complete-in-steps nil)         ; Refile in a single go
(setq org-refile-use-outline-path 'file)

Archiving

After awhile, you’ll get your files all cluttered with things that are done. org-mode has an archive feature to move things out of your main .org files and into some other files for future reference. If you have your org files in git or something, you may wish to delete these other files since you’d have things in history anyhow, but I find them handy for grepping and searching.

I periodically want to go through and archive everything in my files. Based on a stackoverflow discussion, I have this code:

(defun org-archive-done-tasks ()
  (interactive)
  (org-map-entries
   (lambda ()
     (org-archive-subtree)
     (setq org-map-continue-from (outline-previous-heading)))
   "/DONE" 'file)
  (org-map-entries
   (lambda ()
     (org-archive-subtree)
     (setq org-map-continue-from (outline-previous-heading)))
   "/CANCELLED" 'file)
)

This is based on a particular answer — see the comments there for some additional hints. Now you can run M-x org-archive-done-tasks and everything in the current file marked DONE or CANCELED will be pulled out into a different file.

Up next

I’ll wrap up org-mode with a discussion of automatically receiving emails into org, and syncing org between machines.

Resources to accompany this article