trollfilter

• Introduction
• Download
• Usage
• Options
• Workflow
• dannyboy

Introduction

trollfilter is a special mail filter designed to create the illusion that everyone obeys the rule Please don't feed the trolls. It uses a sqlite database to store the emal addresses of known trolls and the message IDs of mail threads that have been infected by trolling.

As new messages in a troll thread arrive, it can recognize them by looking at the message IDs the new message references and so future trolls in the same thread are marked accurately for appropriate disposition.

Of course, the use of trollfilter is not restricted to just trolling. If a mail thread degenerates into a name calling cat fight, you can add the message IDs of those threads to the trollfilter database as well. Threads that have turned into boring rehashes of the same circular arguments are good candidates as well. Any mail thread you'd rather stop seeing can be picked out and added the the database.

It would be nice to see trollfilter integrated with mail clients to simplify the adding of email addresses and message IDs to the database, but that is beyond the scope of the trollfilter program itself (see below to see how I used claws-mail actions to integrate it with claws-mail). There are options provided for manually running trollfilter to add addresses and message IDs, but that is obviously not the most convenient interface.

Download

You can download the C++ source for the tool here:

trollfilter.tar.bz2

The code depends on the sqlite libraries, which are normally available in most linux distro repos. As of the Jul 21, 2017 release all dependence on vmime is gone. It was too much trouble to keep up with the constantly changing library, so now I do all my own email header parsing and editing.

On Jun 8, 2013 I tried to build this with the new libvmime in the Fedora 19 beta release and found I needed to make a change for it to compile with that version of vmime. The tarball has now been updated to the new version of the source code.

On Dec 19, 2013 I tried to build this with the new libvmime in the Fedora 20 release and found I needed to change it back again since the library seems to have reverted the interface to the way it was before (Sheeesh!). The tarball has now been updated again to the new (old) version of the source code.

Usage

trollfilter [options...] < infile > outfile

The default mode of operation for trollfilter is to act as a filter, reading an email message on stdin and writing a modified message to stdout which has a X-TrollFilter header added with the value Yes or No (yes meaning this message is recognized as part of a troll thread and no meaning it is not a troll thread).

The options described below can be used to change this mode of operation or run various database maintenance operations.

Options

--help Print a summary of the options and exit.

--database=file Name the database file to use for any operations. The default file is $HOME/.trollfilter. The database is created if it does not already exist.

--troll=address Specify an email address to be added to the database. Any mail from this address will be treated as a troll message and all follow-ups to that mail will be treated as part of a troll thread. You can give several --troll options to specify several troll addresses.

--trolladdr This reads an email message on stdin and extracts the From address. That address is added to the database as a known troll address. The Message-ID is also extracted and added as a known troll thread.

--untroll=address Specify an email address to remove from the list of trolls in the database. You can give this option several times to remove multiple addresses.

--untrolladdr Read an email message on stdin, extract the From email address, and remove that address from the list of trolls.

--dropping=id Add a message ID to the database to treat as the head of a new troll thread (a troll dropping :-). This is the way to mark an individual thread as a troll without having to call a particular email address a troll. You can give this option multiple times to mark multiple threads.

--trollid Read an email message on stdin, extract the Message-ID and add it to the database as a troll thread.

--clean=days Remove any threads from the database that have not had any activity for the specified number of days. This is something you probably want to run in cron to clean up the database and prevent it from growing without bound.

--filter This is the default option (so it isn't required). It tells trollfilter to operate in the standard filter mode, reading an email message on stdin and writing it to stdout with the X-TrollFilter header added.

--test This reads a message on stdin and processes it much like --filter, but rather than writing a new message to stdout, it exits with a zero status if the message is not part of a troll thread and a three exit status (starts with t which stands for Troll :-) if the message is part of a troll thread.

--list This reads the database and lists the troll addresses as well as a summary of the threads being tracked as troll threads.

Workflow

This section describes how I am using trollfilter. I have no doubt there are other ways to integrate it with mail processing, but this is mine.

First, let me note, I'm not running an external mail server, all my mail shows up in various gmail, hotmail, etc. accounts, so to get it all onto my system, I use fetchmail, which I run in cron to check all my mail accounts every five minutes and send all of it to my local mail user (tom) on my desktop.

My ~/.fetchmailrc file looks a bit like this, but all mail addresses and passwords have been changed:

poll mail.comcast.net with proto POP3 timeout 30
user "notmycomcastusername"
pass "notreallyapassword"
ssl fetchall
is 'tom' here

poll pop.gmail.com with proto POP3 timeout 30
user "notmymailname"
pass "notmygmailpassword"
ssl fetchall
is 'tom' here

poll pop.gmail.com with proto POP3 timeout 30
user "recent:notmyothergmailname"
pass "stillnotmygmailpassword"
ssl fetchall
is 'tom' here

poll pop3.live.com with proto POP3 timeout 30
user "notmyhotmailaddress"
pass "notmyhotmailpassword"
ssl fetchall
is 'tom' here

That is how external mail from different accounts all winds up going to my local user tom mail account. As postfix is processing the mail, it will look at the ~tom/.forward file for mail destined for user tom (sendmail would do the exact same thing, but I happen to use postfix on my system).

The ~/.forward file is where another utility program, dannyboy, comes in:

dannyboy

Included in the source is a handy utility program dannyboy for building a chain of several filter programs to pipe together. It takes several --prog=program arguments to name the programs to pipe together. Each --prog argument may be followed by any number of --arg=argument args to specify arguments to pass to the previous --prog

I use it in my ~/.forward file to run trollfilter piped into bogofilter then into deliver. I achieve all this with the ~/.forward file that contains the line:

(Note that the quotes around the entire command are required.)

You could also use the mda option to fetchmail rather than a ~/.forward file.

By running these filters before deliver, the X-TrollFilter and X-Bogosity headers are already in the message and available for testing when the deliver program delivers the mail to the dovecot IMAP server I use on my desktop to allow me to access my mail via IMAP on any system in my local LAN (and even from work via ssh port forwarding, but that's getting off topic).

The sieve filter provided by the dovecot pigeonhole project now has the X-TrollFilter header available in the mail as it arrives, so a ~/.dovecot.sieve filter like this will sort the troll messages into a special inbox:

require "fileinto";
if header :contains "X-TrollFilter" "Yes" {
   fileinto "INBOX.TrollCage";
   stop;
}

I have lots more filter rules than that, but this is the one that moves all troll messages to a special inbox (when I am sure this all works flawlessly, I may go ahead and delete them).

(By the way, this is the main reason I run my own IMAP server - to get consistent mail client independent mail filtering done by the server before the client even sees it. If I want to change clients or I have to use a different client on my android phone I still get a consistent view of my mail. By consolidating all the mail from my different mail accounts on the same IMAP server, I also get consistent filtering rules regardless of where the mail came from.)

That is everything required to recognize and filter troll message from information in the ~/.trollfilter database. So now what I need is a convenient way to update the database when I'm reading mail and encounter an annoying message.

My favorite mail client is Claws Mail, and it supports an Actions feature which can, among other things, pass a mail message to an external program.

So, I can now add these actions to the ~/.claws-mail/actionsrc file:

Troll Message: /usr/local/bin/trollfilter --trollid < %f &
Troll Sender: /usr/local/bin/trollfilter --trolladdr < %f &

Now I have the Troll Message and Troll Sender menu items in the Tools/Actions menu which I can use to run the selected message through trollfilter to add message IDs and email addresses to the database directly from claws-mail.

But note that this only works when running the mail client on the same machine as the IMAP server. You'd need tricks with ssh to run the trollfilter command on the mail host machine if you wanted to add similar actions to mail clients being used remotely.