Parliament Hill Computers Ltd

 

Implement Greylisting with Exim

A non technical overview

Greylisting is one defense against unsolicited or junk email (SPAM). The way that it works is that the computer remembers everywhere that has sent it email and when it first got something. When it is asked to accept email from somewhere new, it will tell the sending computer that it is unable to take the email at the moment and please try again in ten minutes.

Genuine email senders will come back and try to deliver the email again, machines that send SPAM will often not bother to try again.

It works because much SPAM is sent from Botnets, generally Microsoft machines that have been taken over and are under the control of a Bot Master. The SPAM sending software will often not attempt to resend deferred email.

Greylisting must be used in conjunction with other SPAM elimination techniques.

This has been in use since early 2010 and there are no known problems with the implementation. However: this does not come with a guarantee. It may throw all your email away or drink all of your best wine. It may even prevent you receiving offers of untold wealth from Nigeria.

What is the big picture ?

The idea is that many spam bots do not retry sending email if the at first attempt they are told to defer: SMTP code 451. A genuine MTA will retry. A record of the pair of sender domain and the IP address trying to deliver the mail is kept in a database along with a time; mail will be deferred until the pair record is at least 10 minutes old. The first time that email is received from the pair there is a small delay, subsequent attempts are not delayed.

In practice this seems to be quite successful.

This is a new implementation that uses a Mysql stored procedure, this makes the Exim configuration simpler than other implementations.

Mysql setup

Note that you need Mysql 5.0.10 or later; your Exim should be able to connect to a Mysql server.

You need to create:

• A database to hold the table, in this HOWTO I call that exim_db.
• A Mysql account (username & password) that exim can use, this needs to be able to access the exim_db database.
• A suitable table, I use greylist.
• The stored procedure, a function in fact greylist_defer().

You will find the Mysql commands to do that here: greylist.database.function.setup.mysql

You execute the SQL commands in that file, something like this will do it (you may need to give a password):

   mysql -u root -p < greylist.database.function.setup.mysql
 

NOTE that this file contains a username and password, you should change the password at least.

Exim setup

You will need to update your exim configuration file, this will probably be called something like /etc/exim/exim.conf. This is where this HOWTO assumes that it is.

Near the top of the configuration file, before the line that contains:

    begin acl

Put the following lines:

    # Get Mysql password for grey listing:
    .include /etc/exim/grey_pass

    # Macro to call the mysql function that returns 'yes' if the mail should be deferred:
    GREYLIST_DEFER = SELECT greylist_defer('${quote_mysql:$sender_address_domain}', '${quote_mysql:$sender_host_address}')

Your RCPT ACL (probably called acl_check_rcpt) can be constructed in various ways, but often it will end with a set of accept acls followed by an unconditional deny. Before the set of accept acls insert:

    # Defer if GREYLIST_DEFER is 'yes':
    defer   condition = ${lookup mysql{GREYLIST_DEFER}}
            message   = Now greylisted - please try again in ten minutes.

If your RCPT ACL ends with an accept insert the above before it.

The file /etc/exim/grey_pass should contain the following. The file should be readable by exim and only by exim.
Change the password to match that used to create the database account.

    # Password for mysql exim database that is used to implement greylisting
    # This file should only be readable by exim.
    hide mysql_servers = localhost/exim_db/exim_user/code419

The greylisting checks should be made after:

	accept  hosts         = +relay_from_hosts

and:

	accept  authenticated = *

since you will always accept them.

You should also put greylisting checks after other checks, eg: SPF, RBL, ... since you don't want to fill your database with junk.

Operation

You need to ensure that mysql is running on the machine, it should just work.

You can check the exim log file on the receiving server (probably /var/log/exim/main.log) and you will see lines like:

2010-01-27 21:31:50 H=testmint.phcomp.co.uk (mint.phcomp.co.uk) [10.239.239.254] F=<addw@phcomp.co.uk> temporarily rejected RCPT <addw@test1.phcomp.co.uk>: Now greylisted - please try again in ten minutes.

If you have access to the sending machine an exim log file would contain entries like:

2010-01-27 21:31:51 1NaFUI-0006Jb-9O == addw@test1.phcomp.co.uk <addw@test1.phcomp.co.uk> R=dnslookup T=remote_smtp defer (-44): SMTP error from remote mail server after RCPT TO:<addw@test1.phcomp.co.uk>: host test1.phcomp.co.uk [10.239.239.1]: 451 Now greylisted - please try again in ten minutes.

If you need to clear the mysql greylist table down, do not worry. All that will happen is that incoming mail will be delayed slightly as you build up the history again.

Clearing old entries

Machines that send spam will, hopefully, not send you spam again; or at least not from the same domain. You will accumulate old entries in the greylist table, you need to clear these out occasionally.

You should run the following script daily; cron is a good way of doing this. Entries where you have not received mail for 30 days will be removed, tweak 30 to suit your taste.

    # Clear old entries from the greylist table
    mysql -u exim_user -pcode419 exim_db <<END
        DELETE FROM greylist WHERE last_received < NOW() - INTERVAL 30 DAY;
    END

Beware: this script contains the password in clear, so it should be protected so that only trusted users can read it. However: the password will be visible to someone running ps when the script is running, for better ways of doing this see: http://dev.mysql.com/doc/refman/5.1/en/password-security-user.html

Discussion & config changes

• This test is made in the RCPT ACL which is before the body of the email is received, thus network bandwidth is not wasted by transferring the body of genuine email several times and also network bandwidth is not wasted by receiving spam that is not resent.
• There is a race condition between the greylist_defer() function and the script that clears old entries from the greylist table. If the function finds (SELECTs) an entry that is then deleted as old before the function UPDATEs it there will be an SQL error.

  It would be possible to guard against this with locking or transactions; however I do not think that it is worth the performance hit for a very unlikely event where the result is not really harmful. If this does happen the email will not be subject to greylist delay and so one potential spam immediately accepted. This mail could, however be ham, or if it really is spam it still has to run the gauntlet of other anti spam measures.

• The code/method described in this HOWTO was developed by Alain Williams <addw@phcomp.co.uk>. I looked at Chris Wilson's notes on Reducing Spam, dwmw2's notes on Simple Greylisting with Exim and The Internet Company's notes on Greylisting with MySQL and Exim.

  You may use this in your own exim configuration in what ever way you see fit. It is your responsibility to ensure that this will work as you wish, Alain Williams accepts no liability for any damage or loss however caused. Please do send me suggestions and bug fixes.

  Preservation of the Author line would be nice.

• The first_received column in the greylist is not really necessary, you could remove it. If you do then modify the INSERT in greylist_defer(). I put it in since it may be useful for reports.

  The SQL INSERT should be modified to read:

       INSERT INTO greylist (sender_host_ip, from_domain, last_received)
           VALUES (sender_ip, sender_domain, NOW());
        

• An ISP may want to add a column to_domain, as its hosted domains will probably see the same spamming machines. So select on the triplet. This needs a new GREYLIST_DEFER macro. See below for an implemntation.
• You may find it interesting to count how many times the pair of sender domain and the IP address have tried to send you email. To do so add to the table a column rcpt_count and increment it when mail comes in.

  To the table creation CREATE TABLE greylist add:

       rcpt_count INT NOT NULL,       -- number of attempted emails
        

  The SQL INSERT should be modified to read:

       INSERT INTO greylist (sender_host_ip, from_domain, first_received, last_received, rcpt_count)
           VALUES (sender_ip, sender_domain, NOW(), NOW(), 1);
        

  The SQL UPDATE should be modified to read:

        UPDATE greylist SET last_received = NOW(), rcpt_count = rcpt_count + 1
            WHERE from_domain = sender_domain AND sender_host_ip = sender_ip;
        

  These changes to the Mysql commands are added in here: greylist-collect-info.database.function.setup.mysql It also uses the $domain in column to_domain and counts the number of times a triplet is seen. This needs a new GREYLIST_DEFER macro:

        GREYLIST_DEFER = SELECT greylist_defer('${quote_mysql:$sender_address_domain}', '${quote_mysql:$domain}', '${quote_mysql:$sender_host_address}')
        

• If you want to dry run it, ie see what information is collected in the database you may do so by replacing the defer ACL with the following (with exim older than 4.64 use acl_m9):

        warn   set acl_m_grey   = ${lookup mysql{GREYLIST_DEFER}}
  

  You can then see what the table contains after some time by typing (your input in italic):

        $ mysql -u exim_user -p exim_db
        Enter password: code419
        mysql> select * from greylist_old;
        +-----------------+---------------------------------+---------------------+---------------------+
        | sender_host_ip  | from_domain                     | first_received      | last_received       |
        +-----------------+---------------------------------+---------------------+---------------------+
        | 80.68.91.63     | bytemark.phcomp.co.uk           | 2010-01-28 18:14:45 | 2010-01-28 22:33:35 | 
        | 217.146.97.5    | magmag.info                     | 2010-01-28 18:32:14 | 2010-01-28 18:32:14 | 
        | 211.147.215.206 | hotmail.co.uk                   | 2010-01-28 18:37:05 | 2010-01-28 18:37:05 | 
        ....
  

  It can be a good idea to dry run for a few days before before using it to test for DEFER. This will put many high volume correspondents into the database so that they are not greylisted when you first switch it on.
• Comments on Mysql:
  • If the stored function will not execute due to permissions you may need to:

    	  grant execute on exim_db.* to exim_user@localhost;
    	  

    You can see what grants are specified for a user:

    	  show grants for exim_user@localhost;
    	  

  • If the mysql_servers is for a different database you will need to specify exim_db in the GREYLIST_DEFER macro:

    	   GREYLIST_DEFER = SELECT exim_db.greylist_defer('${quote_mysql:$sender_address_domain}', '${quote_mysql:$domain}', '${quote_mysql:$sender_host_address}')
    	   

Other comments and implementations of greylisting

• I hear of greyd, written in C++ http://www.beastsoft.net/trac/greyd. This seems to work on pair of IP and Name.
• Keith Brazington <keith.brazington@gmail.com> uses this for greylisting:

       BATVKEY               = mysecretkey
       HASH                  = ${hmac{sha1}{BATVKEY}{${lc:$sender_address$sender_host_address$local_part@$domain}}}
  
       rcpt_acl:
         ...
         # Various tests which put content into acl_c_grey if the connection is perceived as poor. No Reverse DNS, invalid HELO/EHLO etc.
         defer   message        = Greylisted
                 condition      = ${if def:acl_c_grey}
                 !ratelimit     = 1 / 1w / strict / HASH
     

• You may wish to read notes on using Postgrey with Exim.
• Martin A. Brooks has a filesystem-based greylister for Exim.
• Frank Elsner has another filesystem-based greylister for Exim.
• Chris Wilson's post to the exim mail list points to a grip on greylisting & Exim new users.