six demon bag

Wind, fire, all that kind of thing!

2010-05-30

Backscatter protection

What is backscatter?

When mail servers accept mail and later discover that for some reason they are unable to actually deliver it, RFC 821 specifies that a Non-Delivery Notification (NDN, also known as "bounce") must be sent to the originator of the mail.

However, the "From" address can be spoofed most easily, so there is no guarantee whatsoever that the mail actually originated from that address. In case of a spoofed address, the NDN will be sent to someone who hadn't sent the original mail to begin with. These bounces going back to someone else but the original sender are called "backscatter".

Why is that a problem?

Because spammers tend to send their bulk e-mails to anything that looks even remotely like an e-mail address, the "To" addresses usually include lots of invalid addresses. Therefore spam-runs can cause massive waves of backscatter flooding the mailboxes of those people whose addresses were spoofed in the "From" field. However, it's not sensible to simply block all incoming bounces, because there are legitimate bounces as well.


What to do about it?

Instead of indiscriminately rejecting all bounces you want to selectively accept only those bounces that were generated for mails sent from one of your hosts. Therefore you check incoming mails in the SMTP dialog, which is a better approach than "accept first, bounce later" anyway. Rejecting ensures that the notification goes back to the sending server instead of some innocent bystander.

To be able to do this you need to identify which bounces relate to mails that originated from your domain. This can be achieved by sending all your outbound mail through your own mail server, which calculates and stores a checksum for each outgoing mail. Incoming bounces are checked whether they match an existing checksum and are rejected in the SMTP dialog otherwise.

How to do it?

Since some mail servers mangle the original mail when sending an NDN, it's not feasible to calculate a digest of the entire mail. Instead the filter checks a select few header fields (namely "From", "To" and "Date"), which seem to appear in almost every bounce. It then cononicalizes the values of these fields and calculates an MD5 hash of the concatenated values. With this procedure, bounces that don't contain these header fields will be rejected even if they're legitimate, but frankly, without at least those headers included, bounces are useless anyway.

This implementation is a filter for the Postfix mail server. The current version uses either BerkeleyDB, SQLite or PostgreSQL as a backend for storing the hashes. The BerkeleyDB and SQLite backends are just for single host setups:

Message flow and hash database access for outbound e-mails in a single host setup.Message flow and hash database access for inbound e-mails in a single host setup.

The PostgreSQL backend allows for setups with frontend/backend servers as well:

Message flow and hash database access for outbound e-mails in a frontend/backend server setup.Message flow and hash database access for inbound e-mails in a frontend/backend server setup.

However, in each case you need two instances of smtpprox. One must be run as an after-queue filter to calculate and store the hashes of outgoing mails, the other must be run as a before-queue filter to check incoming backscatter.

Of course this is a very simplified view. For a more detailed explanation of how Postfix handles mails see the Postfix Architecture Overview as well as the two READMEs linked in the above paragraph.

Download

Version 1.3.2 (2010-05-30; updated README)

Version 1.3.1 (2009-04-06; for some reason I had forgotten to actually patch the copy of `MSDW::SMTP` in my trunk, tsk)

Version 1.3 (2009-03-22; updated, because PostgreSQL 8.3 returns the difference of two dates as an interval)

Version 1.2 (2009-02-23; fixed a bug in the cleanup routine of the `FilterDB::PostgreSQL` module)

Version 1.1 (2009-01-21; added support for SQLite backends, and an example StartupItem for Mac OS X)

Version 1.0 (2008-11-28; initial release, supports BerkeleyDB and PostgreSQL backends)

Required Perl modules

Install smtpprox

To install the backscatter filter, copy the Perl modules FilterDB and MSDW::SMTP to a directory where Perl can find them (e.g. /usr/local/lib/perl). The above archive contains an already patched version of MSDW::SMTP, or you can download the original module and Jonathan Hitchcock's patch from the smtpprox homepage.

Copy smtpprox (the actual filter) and sentdb.pl (a maintenance utility for the filter backend) to an appropriate location (e.g. /usr/local/sbin). Copy the respective configuration files (smtpprox.conf and sentdb.conf) to /etc and edit them.

Create a user and group "filter" with the home directory /var/spool/filter, so that smtpprox can be run under a user account of its own. Also create a directory /var/run/smtpprox (that's the default location for smtpprox to put its PID files) and make it writable to the group "filter".

In case you're using PostgreSQL as your backend, use the included filterdb_pg.sql file to create the database. Make sure to edit it and set a password for the database user. Change the group membership of the configuration files to "filter" and make them readable for owner and group only. The configuration files contain the password for the filter database, and you don't want that information to be world-readable.

In case you're using SQLite as your backend, use the included filterdb_sqlite.sql file to create the database.

To have smtpprox start automatically, copy the example init script to /etc/init.d (or wherever your operating system places its init scripts) and link it to the respective runlevels. On Max OS X copy the example StartupItem to /Library/StartupItems.

Configure Postfix

Single host setup

In a setup with a single host, both smtpprox instances must run on the same host, either by starting them with the appropriate commandline options:

smtpprox -b -l 127.0.0.1:10025 -t 127.0.0.1:10026 -p /var/run/bqf.pid
smtpprox -l 127.0.0.1:10027 -t 127.0.0.1:10028 -p /var/run/aqf.pid

or by using separate configuration files:

smtpprox -f /etc/smtpprox/before_queue.conf
smtpprox -f /etc/smtpprox/after_queue.conf

Either way you have to modify your init script to handle two instances of smtpprox.

The master.cf should contain the following lines:

# First stage SMTP server (running before filters). Receive mail from the
# network and pass it to the SMTP proxy filter on localhost port 10025.
#
smtp      inet  n       -       -       -       20      smtpd
    -o smtpd_proxy_filter=127.0.0.1:10025
    -o smtpd_client_connection_count_limit=10
#
# Second stage SMTP server. Receives mail from before-queue and after-queue
# filters.
#
127.0.0.1:10026 inet n  -       -       -       -       smtpd
    -o content_filter=
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
    -o smtpd_client_restrictions=
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8
    -o smtpd_use_tls=no
    -o smtpd_timeout=3600s
#
# Run submitted mail through after-queue filter for registration.
# (uncomment submission lines if you use submission)
#
#submission inet n       -       -       -       -       smtpd
#    -o content_filter=smtp:127.0.0.1:10027
#    -o smtpd_enforce_tls=yes
#    -o smtpd_sasl_auth_enable=yes
#    -o smtpd_client_restrictions=permit_sasl_authenticated,reject
pickup    fifo  n       -       -       60      1       pickup
    -o content_filter=smtp:127.0.0.1:10027

Since the second stage SMTP server is listening on localhost, I chose to disable TLS, to avoid authentication problems. Also I had to increase $smtpd_timeout, because some sending mail servers took longer than the default timeout (300 s) to finish their DATA command. When that happened, the connection between before-queue filter and second stage SMTP server timed out and the mail bounced when the before-queue filter tried to write to the already closed connection.

Frontend/backend server setup

In a setup with separate frontend and backend servers, you need to run the before-queue filter on the frontend server(s) and the after-queue filter on the backend server (note that you cannot use the BerkeleyDB and SQLite backends in this kind of setup). Unlike with the single host setup you don't have to modify the example init script here, except for adjusting the path to the smtpprox script maybe.

The frontend servers' master.cf should contain the following lines:

# First stage SMTP server (running before filters). Receive mail from the
# network and pass it to the SMTP proxy filter on localhost port 10025.
#
smtp      inet  n       -       -       -       20      smtpd
    -o smtpd_proxy_filter=127.0.0.1:10025
    -o smtpd_client_connection_count_limit=10
#
# Second stage SMTP server (running behind before-queue filter). Receive mail
# from the proxy filter on localhost port 10026.
#
127.0.0.1:10026 inet n  -       -       -       -       smtpd
    -o receive_override_options=no_unknown_recipient_checks,no_address_mappings
    -o content_filter=
    -o smtpd_client_restrictions=
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8
    -o smtpd_use_tls=no
    -o smtpd_timeout=3600s

The backend server's master.cf should contain the following lines:

# First stage SMTP server (running before filters). Receive mail from the
# network and pass it to the content filter on localhost port 10025.
#
smtp      inet  n       -       -       -       20      smtpd
    -o content_filter=smtp:127.0.0.1:10025
    -o smtpd_client_connection_count_limit=10
#
# Second stage SMTP server (running behind after-queue filter). Receive mail
# from the content filter on localhost port 10026.
#
127.0.0.1:10026 inet n  -       -       -       -       smtpd
    -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
    -o content_filter= 
    -o smtpd_client_restrictions=
    -o smtpd_helo_restrictions=
    -o smtpd_sender_restrictions=
    -o smtpd_recipient_restrictions=permit_mynetworks,reject
    -o smtpd_data_restrictions=
    -o mynetworks=127.0.0.0/8
    -o smtpd_authorized_xforward_hosts=127.0.0.0/8
    -o smtpd_use_tls=no
    -o smtpd_timeout=3600s
#
# Run submitted mail through after-queue filter for registration.
# (uncomment submission lines if you use submission)
#
#submission inet n       -       -       -       -       smtpd
#    -o content_filter=smtp:127.0.0.1:10027
#    -o smtpd_enforce_tls=yes
#    -o smtpd_sasl_auth_enable=yes
#    -o smtpd_client_restrictions=permit_sasl_authenticated,reject
pickup    fifo  n       -       -       60      1       pickup
    -o content_filter=smtp:127.0.0.1:10027

Since this is a distributed setup, and you don't want anyone to be able to tamper with requests to the filter database as they go over the network, I strongly recommend to secure the connections with some kind of encryption. This can be achieved by using either SSH tunnels, a virtual private network (e.g. OpenVPN) or PostgreSQL's built-in SSL support.

Should you decide to use PostgreSQL's SSL support: although PostgreSQL does accept self-signed certificates, I'd recommend against using them, because man-in-the-middle attacks would still be possible. Better create an SSL root certificate of your own (it doesn't need to be signed by an official certification authority) and sign the certificate for your server with it.

Note: You can have any number of frontend servers, but as of yet smtpprox-backscatterfilter does not support multiple backend servers. If you have redundant backend servers, you're on your own.

Maintenance

Maintenance of the filter database is done on the host running the filter database (in a setup with frontend/backend servers that's usually the backend server). The sentdb.pl commandline utility allows to list the contents of the database as well as remove entries which are older than a given number of days (default is 4 days). To remove old entries from the database run

sentdb.pl -c

via cron on a daily basis.

References

smtpprox-backscatterfilter is based on an idea of Volker Birk. It was implemented using/modifying smtpprox by Bennet Todd, and the patch and smtpprox-contentfilter script by Jonathan Hitchcock.

Copyright

The script is distributed according to the terms of the GNU General Public License Version 2.0.

Thanks to

Alexander Bernauer
Ulrich Dangel

Posted 22:41 [permalink]