Managing email with Notmuch and Emacs
Web email interfaces have taken over the world a long time ago. Except for Outlook users, only a few people even consider using an actual application for it. This is one of the primary reasons for our low opinion of electronic mail today, since with limited tools, you get limited capabilities. While there are options for automatic filtering, tagging, or folding with most clients, those capabilities are barebones.
Luckily there are still compelling alternatives.
Managing email locally
Back in the day of POP, this was the standard: your computer downloaded new messages from the server, and you had local copies from this point. Then IMAP came, and after it, Exchange. Now you have a fast and convenient option to sync all changes with other clients and not bother with hard drive space. But this also puts us at the mercy of email providers. If a given service doesn’t support tags, folders, or saved searches, you cannot use it. I tried to reverse this direction and never looked back. I want all my email to be on my machine, available at all times, and accessible instantly.
Mbsync(1)1 is a popular application to populate local ~/Mail folder with remote messages fetched via IMAP. The config is quite simple. For Gmail23:
1MaildirStore gmail-local 2Path ~/Mail/gmail/ 3Inbox ~/Mail/gmail/INBOX 4Subfolders Verbatim 5 6IMAPStore gmail-remote 7Host imap.gmail.com 8SSLType IMAPS 9AuthMechs LOGIN 10User <username> 11PassCmd <password> 12 13Channel gmail 14Far :gmail-remote: 15Near :gmail-local: 16Create Both 17Expunge Both 18Patterns * !"[Gmail]/All Mail" !"[Gmail]/Important" !"[Gmail]/Starred" !"[Gmail]/Bin" 19SyncState *
mbsybc -a, wait for a few (hundred? depending on the mailbox size) minutes, and voila. Your
~/Mail/gmail/ is now populated with all your messages. Let’s break down the config.
Mbsync(1) assumes two stores exist: local (on your computer) and remote (the IMAP server). A store is a place where mail exists. We have then configured in lines 1-4 and 6-11.
The remote one is self-explanatory. One thing to remember: some providers will require you to use an app-specific password and reject auth attempts with the normal one. Our password in line 11 can be either a string with the password or an arbitrary command (think
cat ~/my-secret-password or a CLI password manager).
The local store is just a definition of local folders to use. It can be anywhere, but
On line 13, we start to define a channel, which is how mbsync(1) works. One store is
far (remote), while the other is
near (on your machine). The rest of the config defines behavior. Refer to the manual, but in my case:
- it will create non-matching mailboxes
- it will delete messages in a store if a message was deleted on the other
- it will sync all messages except a few matching the pattern.
- it will store the synchronization state file in the Mail dir.
One last thing to add is a simple cron rule. You can force mail fetching manually, but I opted for the automatic option. Therefore, my crontab(1) has:
1/5 * * * * killall mbsync &>/dev/null; <msync_dir>/mbsync -a -V 2>&1 > ~/.mbsync_log
We will now fetch new messages every 5 minutes.
Now we need to choose a local email client to use. There is a lot to choose from! Thunderbird4 seems to be the most popular option for GUI users. For terminal users, we have Mutt(1)5, its successor Neomutt(1)6, and many, many more. I used Neomutt(1)7 for a while, and it was a pleasure compared to web clients.
However, ever since I started to use Emacs more, I wanted to move my Email inside Emacs. And to little surprise, we also have a lot to choose from. By default, Emacs comes with Gnus8, a newsgroups reader with an email client capability. However, the two most popular packages are Mu4e9 and notmuch(1)10
The last two are based on fast email indexing and searching but assume different workflows. Mu4e is based on filtering, while notmuch around tagging.
A friend11 recommended Notmuch(1) to me. I have never tried mu4e as notmuch fully satisfies my needs.
Setting up notmuch(1)
Notmuch(1) is not a client but an indexer. It indexes and tags existing email and allows one to search messages. It should be present on most systems package management, so install it. They run
notmuch, answer a few questions, and you’ve got yourself a ready notmuch.
Whenever a new mail arrives, it won’t be known to notmuch before indexing. You can manually run
notmuch new or make a cron definition for it.
One killer feature of notmuch is its sheer speed. The name comes from the fact that it can work on gigantic mailboxes very swiftly - oh, you have one million messages? That’s not much!
Let’s try a simple search:
$ notmuch search 'from:*@github.com'
You can search based on sender, receiver, dates, attachments, contents, titles, etc. Refer to man pages for
notmuch-search-terms(7). However, to get the most out of notmuch, we need to learn about tags.
Let’s add a “gh” tag for messages from Git Hub.
$ notmuch tag "+gh" -- "from:@github.com"
Now, we can search for such messages
$ notmuch search "tag:gh"
We can also join multiple search criteria with “and”, “or” and other boolean operators. We now have a fully working local email reader - however we can not send email. I will not discuss sending email here as it’s a separate subject. Notmuch(1) is not for sending email.
Using CLI for reading email is far from pleasant. All those commands will come in handy, but first, let’s add a user interface.
Notmuch(1) in Emacs(1)
Notmuch(1) can be used with different UIs, like Mutt(1). However, it comes with an Emacs(1) package, so let’s enable it.
(use-package notmuch :commands notmuch-hello :bind (("C-c m" . notmuch-hello)))
The key binding “C-c m” is very popular, but you can use whatever you want.
notmuch-hello, we are not greeted with a list of messages but with a search interface. You’ve got access to saved searches, recent searches, and a list of tags. This tells us that we are dealing with a completely different beast than webmail, and the user needs to think of new workflows.
The way we are thought of thinking of email is a list of messages. Some clients can mark messages that are more important, favorite, tag them, move them into folders, etc. But everywhere I know, the primary interface is just a list - unread email on top, read on the bottom. The behavior I always expect is to open (or mark as read) all incoming messages and then ignore most of them. If you spent some time on configuration, maybe you have automatic rules - like moving all newsletters to a “newsletter” folder and removing them from your inbox. Hey12 is even based on separating all incoming messages into three tiers: important mail, newsletters, and paper trial.
But back to notmuch. Look at saved searches - we will use them later. Open “unread,” and we see a semi-normal list of messages. Use “n” and “p” to select email, enter to open it, and so on - standard Emacs stuff. One thing to remember is that by default, reading an email will not mark it as read. You need to manually remove the tag via
k u - either one by one, or on all messages in a selected region (C-spc, it’s still a buffer, after all). “Unread” here is just another tag. We can be much smarter with marking actionable items.
Automatic Github Pull Review workflow
What we’ve seen before is nothing more than a normal email client with extra steps. We read emails in Emacs(1), which is great, but we don’t get anything extra. It’s time for a real-world example.
I am a software engineer forced to work with GitHub. One thing I do is to review pull requests. The primary problem here is knowing that someone wants me to review something. The review itself is the easy part :-)
I rely solely on email for this information, ostensibly ignoring all nudges on Slack13.
First of all, we need to enable email notifications from GitHub. Remember to mark that you want to get emails about your own actions.
Now, let’s think about what we want to achieve. For me it is “I want to know about all the pull requests I should look at without opening browser14”. This means I want to see all the review requests I was assigned (personally or by being part of a team) that I have not yet reviewed.
Luckily, GitHub allows us to get that from email:
- When you are first assigned a review, you get a dedicated email
- When you approve or reject a PR, you get an email
- When someone asks you to re-review an email, you get the same email as it was the first request for this PR.
We now know that we can use Notmuch(1). There are two ways: we can use
notmuch-hooks(5) and place a shell script in `~/Mail/.notmuch/hooks/post-new, but it never worked reliably for me. Instead, I have a cron job that runs a script:
1#!/usr/bin/env sh 2 3notmuch tag +gh -unread -- '(from:email@example.com)' 4 5# Mark new review requests 6for thread in $(notmuch search --sort=oldest-first --output=threads -- "\"requested your review on\" and tag:gh and -tag:gh-pr-done"); do 7 for msg in $(notmuch search --sort=oldest-first --output=messages -- "$thread"); do 8 txt=$(notmuch show "$msg") 9 10 (echo "$txt" | grep "requested your review on") && notmuch tag +gh-pr-todo -- "$thread" 11 (echo "$txt" | grep "@michalsapka approved this pull request") && notmuch tag -gh-pr-todo -- "$thread" 12 (echo "$txt" | grep "@michalsapka requested changes on this pull request") && notmuch tag -gh-pr-todo -- "$thread" 13 (echo "$txt" | grep "Merged.*into") && notmuch tag -gh-pr-todo +gh-pr-done -- "$thread" 14 done 15done
Let’s break it down:
- First, we add a “gh” tag to all notification emails from GitHub and remove the “unread” tag. I don’t need to be notified about all such emails, but I can still look at the “gh” tag if needed. 2, Then we search for threads where an email informs me about a review request. I limit the search to emails from GitHub via the tag from #1 and those without “gh-pr-done” tag. More on the second one in a moment
- Then I search for all messages in such threads. I force order as oldest-first to make it possible to reason about. In normal PR, all actions happen with a significant delay between them, so this should be enough not to get lost in the timeframe. If I ask for a change in review, the re-request will not happen instantly. Note that I get the email body as a variable on line 8.
- Then comes the meat. I will tag the entire thread multiple times based on the body of the message. When a request comes, a “gh-pr-todo” is added. I need to look at it. When I approve or reject a PR, the tag is removed. If someone asks for a review, logic from line 10 will be triggered, and the tag will be added again. This means that I want to handle all email threads with the “gh-pr-todo” tag.
- Lastly, when a PR is merged, I ensure that the “gh-pr-todo” tag is removed, and I add the “gh-pr-done” tag so this thread will not be found in step 2 in the future.
There are other ways to tag, like afew(1)15, but keeping it to simple shell script working with notmuch(1) directly gives us the greatest amount of freedom and made it easier for me to tell this story.
Making it more visible
This alone would be a challenge to manage. An email with a tag would be easily missed. Notmuch has us covered yet again! My emacs config has a few dedicated lines:
This makes two changes.
- Firstly, all messages with “gh-pr-todo” will be shown in red in any email list. All red items are actionable since we remove this tag in the workflow.
- Secondly, amongst other saved searches, I have one dedicated to PRs.
With those two things, every time I enter
notmuch-hello screen, I get instant information about the work I need to do.
Making it extra visible
But we can go one step further. Prot’s16 excellent notmuch-indicator17 allows us to add saved searches to the mode line. After installing it, the configuration is straightforward:
This adds a “pr:” to the mode line. The count is the number of messages, not threads, but frankly, I want it to be 0. The counter will be refreshed every 60 seconds. And lastly, if the count is 0, the label will not be added to the mode line.
Notmuch comes with one significant downside: lack of multi-device support. It’s 2023, and most of us have more than one computer and those pesky mobile phones.
As for the mobile - I have no solution. The read statuses will sync via mbsync(1), but not much else. I try to purge myself from phone addiction, so maybe that’s a plus?
As for the other computers, we have muchsync(1)18. It’s an external application designed to sync entire mailboxes and tags between devices over ssh. I have not tried it yet19, but it looks promising.
With local email and tools like notmuch(1), we are not at the mercy of external tools for even sophisticated workflows. If you get transactional emails, you can extract actionable data. It can be JIRA tickets, Pager Duty alerts, heck - even Amazon deliveries. Here I have demonstrated how easy it is to leverage notmuch(1), simple shell script, and emacs(1) to have a fully automated notification setup. It does not try to hijack your attention (like mobile notifications do) and is not hidden on some webpage (like GitHub notification), but it still gives actionable results. And all that without leaving the comfort of Emacs.
One cool thing I plan to apply soon is integrating notmuch(1) with Org-mode with the ol-notmich[oln] package. But for now, I am in the process of moving as many external services to a similar workflow as possible.
I have my gripes with Gmail, but it is still the standard. While this article assumes Gmail, you can easily use mbsync(1) with any IMAP service. ↩︎
Back when Slack was first sold, it was proposed not only as a chat tool but also as a single place for all information. We see it now: we connect Slack to everything - GitHub, jira, Data Dog, or pager duty. The general idea is great, but Slack is a pretty mediocre application. The only way to manage what you receive is to leave a channel. But then you lose all other messages sent there, so the price may be high. ↩︎
There is the GitHub CLI which is amazing by itself - one of the best things that GitHub has done in the last few years. And it can achieve the same. However, let’s ignore it for now, as the same model of email-based dashboards can be expanded to many other things. ↩︎
my work computer gets all work messages, and my private one gets all private ones—complete separation. When I get a second personal machine, I will set it up, but for now, there is no use case for me. ↩︎