!!! RETAINED FOR HISTORICAL PURPOSES ONLY !!!

OpenSMTPd, jails and amavisd-new for virus scanning and DKIM signing and spam classification using SpamAssassin and DSPAM. Nowadays, you're better of with rspamd. Not documented here.

Contents

Installation
Housekeeping
Sub-component notes
Example configurations
Testing
1. amavisd
2. dspam
Scripts

Installation

Required ports

WIP mail/spamd Throttling
mail/opensmtpd OpenSMTPd MTA
mail/opensmtpd-extras OpenSMTPd extra filters
security/amavisd-new "A Mail and Virus Scanner"
mail/spamassassin Apache SpamAssassin e-mail classifier
mail/dspam Bayesian e-mail classifier
mail/rspamd rspamd e-mail classifier
security/clamav Open source virus scanner
DSPAM !SpamAssassin plug-in
One of MariaDB/MySQL, PostreSQL, SQLite or Hash database

Jails

mail (opensmtpd, dovecot)
scan (amavis, spamassassin, dspam, clamav)

Housekeeping

Jail	What	When	Why
scan	sa-update	daily	Keep SpamAssassin rules up-to-date
scan	freshclam	hourly	Keep ClamAV signatures up-to-date
scan\|db	dspam-purge	daily	Remove old messages and signatures from dspam database
scan	train dspam	hourly	Process false-positives/-negatives

Sub-component notes

Base OS

Disable sendmail

sendmail_enable="NONE"

spamd

Listens on 25 and proxies the inbound connections using black-/grey-/white-listing, tarpitting, etc.

Generates and maintains a hash-database of black-/grey-/whitelisted senders.

OpenSMTPd

Intended flow of mail from authenticated users to be relayed

Receive on port 25/465/587 (smtp/smtps/submission) with authentication
Classified as local when user is authenticated
DKIM-sign message
Relay

Intended flow for mail for any of my domains

Receive on port 25 (smtp)
Domain of receiver is one of allowed domains
Forward to AMaViS service
1. Bayesian classification using DSPAM
  b. Rule-based classification using SpamAssassin c. Virus scan using ClamAV
Receive from AMaViS service (tag Scanned)
Deliver to Dovecot using aliases db

All other mail must be bounced.

DKIM signing

Generate a 4096-bit DKIM signing key using LibreSSL in the location where the DKIM-signer filter expects it

$ /usr/local/bin/openssl genrsa -out /etc/ssl/private/rsa.private 4096

Once added to your amavisd config you can generate relevant DNS TXT records using

$ amavisd showkeys
default._domainkey.example.org.    3600 TXT (
  "v=DKIM1; p="
  "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3bcAwPnDM89FdJqWo8Om"
  "C4qfsfPIWIjqCnqp8H0QT03jAYwwzTwrKNb2w2dWjf9fp8D/Jm21XAT4GDhUWEQA"
  "XBxA2Ds5cHIPiSHRCstj7+kww1S40mQcS9WcUJjjO0jaGzH+Lfm02ZXC/E0i7YJY"
  "m9GKwPBz8M/8gMNCIbTuf9exVCmFW1gjnHMfQHOX0x8oBVLiby67vFl9UBwrIBSe"
  "+6T73t10fIWyTtkuFkJKNiqDnat8p0KMzpYwviBU8Aec1wTUFo9rK9bUgFJYdXUF"
  "q3gRN3glLlj4ho7yjZdqlofYO1CbdZ5FjyYs559jRd+dOUZh84Tmazg5PNx8Jeus"
  "MwIDAQAB")

Testing via any gmail account shows you if your DKIM setup is correct, look for the following in the smtp headers of a message that passed by gmail's service

Authentication-Results: mx.google.com;
    spf=neutral (google.com: 139.254.214.162 is neither permitted nor denied by best guess record for domain of brnrd@example.org) smtp.mail=brnrd@example.org;
    dkim=pass header.i=@example.org

Lessons learnt

OpenSMTPd foot-shooting notes:

accept/reject rules work in order
accept/reject rules make sure you have a from and for check
Ip-addresses of the host itself are seen as 'local'

Dovecot

Using Dovecot's LDA in OpenSMTPd provides the following benefits:

Mailbox indexing during mail delivery, providing faster mailbox access later
Quota enforcing (plugin)
Sieve language support (plugin)

This setup adds mail/dovecot2-antispam-plugin to the configuration for easy (re-)classification of mails. This triggers copying of a message when copied/moved into or out of the Spam folder. The NotSpam and IsSpam have to be trained with DSPAM.

AMaViSd

Listens on 10024 to scan mails

Scanning

ClamAV
DSPAM
rspamd
SpamAssassin

Viruses can be rejected without spam classification/scanning.
First pass through DSPAM for bayesian classification (I've found it to be far superior to SpamAssassin's bayesian classifier) and then through SpamAssassin to allow SpamAssassin to take the X-DSPAM headers generated by dspam into account when scoring

All this results in the following headers:

X-DSPAM-Processed: Sun Jun  7 11:03:51 2015
X-DSPAM-Confidence: 0.9899
X-DSPAM-Probability: 0.0000
X-Virus-Scanned: amavisd-new at brnrd.eu
X-Spam-Flag: NO
X-Spam-Score: -2.618
X-Spam-Level:
X-Spam-Status: No, score=-2.618 tagged_above=-999 required=6.2
        tests=[DSPAM.Innocent=-1.000, DSPAM_HAM_99=-3.23,
        MSGID_FROM_MTA_HEADER=0.001, RDNS_DYNAMIC=0.363, TO_MALFORMED=1.247,
        UNPARSEABLE_RELAY=0.001] autolearn=disabled
X-DSPAM-Result: Innocent
X-DSPAM-Signature: 447408f7488584401825356

DSPAM

NOTE: Documentation seems to be gone from the sourceforge.net project page NOTE2: I'm no longer using DSPAM, I've moved to rspamd (with spamassassin rules added)

This setup creates a 'site-wide' configuration, all mail is classified and trained under the vscan user.

To use dspamc from amavisd you must make /usr/local/etc/dspam.conf readable for the vscan user so dspamc can read its ClientHost configuration.

Training

Get a corpus of spam and ham to train dspam (in separate folders) as the vscan user.
dspam_train vscan /path/to/spamdir /path/to/hamdir

False positives/negatives

The Dovecot antispam-plugin must be configured to put mis-classified messages in /path/to/NotHam and /path/to/NotSpam folders. You'll need to create a script to feed these to dspam to update the database tokens

rspamd

Yet to be documented

SpamAssassin

NOTE: Now integrated into my rspamd config

Download the dspam SpamAssassin module (dspam.pm) and ruleset (dspam.cf) and drop them in /usr/local/etc/mail/spamassassin
Add to your init.pre

loadplugin Mail::SpamAssassin::Plugin::dspam /usr/local/etc/mail/spamassassin/dspam.pm`

and to your local.cf

include dspam.cf

The Bayesian tests and auto-learning can be disabled as they are replaced with DSPAM.

Validate your configuration with spamassassin --lint, when there's problems, adding -D shows you debug output that helps find the root of the problem.

Example configurations

NOTE: These are not complete configurations but show the relevant parts that require modification!

etc/mail/smtpd.conf

   1 # generate db using makemap
   2 table aliases db:/usr/local/etc/mail/aliases.db
   3 table domains { example.org, example.net, example.com }
   4 table secrets db:/usr/local/etc/mail/secrets.db
   5 
   6 filter dkim-sign      dkim-signer "-Dexample.org"
   7 filter dnsbl-sorbs    dnsbl       "-h dnsbl.sorbs.net"
   8 filter dnsbl-spamcop  dnsbl       "-h bl.spamcop.net"
   9 filter dnsbl-spamhaus dnsbl       "-h zen.spamhaus.org"
  10 filter dnsbl-all      chain       dnsbl-sorbs dnsbl-spamcop dnsbl-spamhaus
  11 
  12 lan_addr = "192.0.2.2"
  13 scan_addr = "192.0.2.5"
  14 
  15 # Define keys and certs (PEM encoded)
  16 pki example.org certificate "/etc/ssl/certs/example.org.cer"
  17 pki example.org key         "/etc/ssl/priv/example.org.key"
  18 
  19 filter dkim-sign dkim-signer "-d example.org -p /usr/local/etc/mail/example.org.dkim.key"
  20 
  21 # Inbound mail smtp, smtps, deliver
  22 listen on $lan_addr port  25 filter dnsbl-all tls \
  23        pki example.org hostname example.org auth-optional
  24 listen on $lan_addr port 465 smtps \
  25        pki example.org hostname example.org auth
  26 listen on $lan_addr port 587 dkim-sign tls-require \
  27        pki example.org hostname example.org auth \
  28        filter dkim-sign
  29 # Receive scanned mails from amavisd-new
  30 listen on $lan_addr port 10025 tag Scanned
  31 
  32 # Deliver locally messages coming back in from scanner
  33 accept tagged Scanned from source $scan_addr \
  34        for domain <domains> alias <aliases> \
  35        deliver to lmtp "/var/run/dovecot/lmtp"
  36 reject tagged Scanned
  37 
  38 # Forward all mail received for local domains to amavis
  39 accept from any \
  40        for domain <domains> \
  41        relay via "smtp://192.0.2.5:10024"
  42 
  43 # Relay anything that came in from authenticated users
  44 accept from local \
  45        for any \
  46        relay
  47 # If you need to use a 'smarthost' use
  48 #      relay via tls+auth://user@mta.example.net:587 <secrets>

Line 33: Use dovecot's LDA so it updates the indexes for the Maildir as well (otherwise indexing takes place when the user 'opens' the mailbox)
Line 39: There's no macro expansion in quoted strings, so you must use the IP-address or a valid hostname.

etc/amavisd.conf

   1 $MYHOME = '/var/amavis';
   2 $pid_file = "/var/run/amavisd/amavisd.pid";
   3 
   4 @inet_acl = qw(192.0.2.1/24);
   5 
   6 $inet_socket_port = [10024];
   7 
   8 $notify_method  = 'smtp:[192.0.2.2]:10025';
   9 $forward_method   = 'smtp:[192.0.2.2]:10025';
  10 
  11 @spam_scanners = (
  12   ['DSPAM', 'Amavis::SpamControl::ExtProg', '/usr/local/bin/dspamc',
  13   [ qw(--mode=teft --deliver=stdout --user vscan) ],
  14     mail_body_size_limit => 256000, score_factor => 1,
  15   ],
  16   ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ],
  17 );
  18 
  19 @av_scanners = (
  20    ['ClamAV-clamd',
  21      \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
  22      qr/\bOK$/m, qr/\bFOUND$/m,
  23      qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
  24 );

dspam

   1 Preference "signatureLocation=headers"

And make sure you have all storage backend params in order

SpamAssassin

use_bayes 0

loadplugin Mail::SpamAssassin::Plugin::dspam dspam.pm

Disable bayes (you can remove all lines referring to bayes as well)

Testing

amavisd

telnet or netcat/nc work fine for me to test amavisd. Prepare a complete mail-transmission to paste

EHLO localhost
MAIL FROM: <someone@example.org>
RCPT TO: <noone@example.net>
DATA
<the original mail content including headers>
.
QUIT

Remember that amavisd will deliver the mail to OpenSMTPd on port 10025 tagged Scanned

dspam

You can test dspam using the dspamc client program:
dspamc --mode=notrain --deliver=stdout --user vscan < test-ham.mail
Simulation works best when you switch user to vscan before executing the command (tests if dspamc can read the client config)

Scripts

dspam-learn

   1 #!/bin/sh
   2 
   3 notHamPath=/path/to/notHam
   4 notSpamPath=/path/to/not/Spam
   5 spamCorpusPath=/path/to/SpamCorpus
   6 
   7 IFS="
   8 "
   9 
  10 extractHeader () {
  11 headers=`grep -Eim3 '^(to|from|subject):' $1`
  12 
  13 for header in $headers ; do
  14    case $header in
  15       [Ff][Rr][Oo][Mm]:* ) from="${header#[Ff][Rr][Oo][Mm]:}" ;;
  16       [Tt][Oo]:* ) to="${header#[Tt][Oo]:}" ;;
  17       [Ss][Uu][Bb][Jj]???:* ) subject="${header#[Ss][Uu][Bb][Jj][Ee][Cc][Tt]:}" ;;
  18    esac
  19 done
  20 echo -e "From:    ${from}\nTo:      ${to}\nSubject: ${subject}\n"
  21 }
  22 
  23 trainIsSpam () {
  24    cd "$notHamPath"
  25    [ "$(ls -A .)" ] || return
  26 
  27    # Mails that have been mis-classified
  28    for file in `grep -E -l "^X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
  29       extractHeader "${file}"
  30       /usr/local/bin/dspamc --user vscan --class=spam --source=error < "${file}"
  31       mv "${file}" "${spamCorpusPath}"
  32    done
  33    [ "$(ls -A .)" ] || return
  34 
  35    # Mails that have already been classified as Spam
  36    for file in `grep -E -l "X-DSPAM-Result: Spam" *` ; do
  37       rm "${file}"
  38    done
  39    [ "$(ls -A .)" ] || return
  40 
  41    # Mails that were not previously classified
  42    for file in * ; do
  43       extractHeader "${file}"
  44       /usr/local/bin/dspamc --user vscan --class=spam --source=inoculation < "${file}"
  45       mv "${file}" "${spamCorpusPath}"
  46    done
  47 }
  48 
  49 trainIsHam () {
  50    cd "$notSpamPath"
  51    [ "$(ls -A .)" ] || return
  52 
  53    # Mails that have been mis-classified
  54    for file in `grep -E -l "^X-DSPAM-Result: Spam" *` ; do
  55       extractHeader "${file}"
  56       /usr/local/bin/dspamc --user vscan --class=innocent --source=error < "${file}"
  57       rm "${file}"
  58    done
  59    [ "$(ls -A .)" ] || return
  60 
  61    # Mails that have already been classified as Ham
  62    for file in `grep -E -l "X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
  63       rm "${file}"
  64    done
  65    [ "$(ls -A .)" ] || return
  66 
  67    # Mails that were not classified
  68    for file in * ; do
  69       extractHeader "${file}"
  70       /usr/local/bin/dspamc --user vscan --class=innocent --source=inoculation < "${file}"
  71       rm "${file}"
  72    done
  73 }
  74 
  75 trainIsSpam
  76 trainIsHam

sa-update

   1 #!/bin/sh
   2 
   3 PATH=$PATH:/usr/local/bin ; export PATH
   4 
   5 result=`sa-update -v 2>&1`
   6 RC=$?
   7 
   8 # Output of sa-update is "Update finished, no fresh updates were available"
   9 # when no new update is available, additionally RC=1
  10 [ "$result" == "Update finished, no fresh updates were available" ] && exit 0
  11 
  12 # If we reach this, we have a new ruleset
  13 echo New ruleset: RC=$RC
  14 
  15 # Generate some some text for your periodic/cron output so it sends you a mail
  16 echo "$result" | grep "Update available for channel"
  17 
  18 result=`sa-compile 2>&1`
  19 
  20 # AMaViDS needs to be restarted to pick up the new rules
  21 service amavisd restart

dspam-purge

Run daily

#!/bin/sh
mysql -u <dspamuser> -p <dspampass> -h db.host <dspamdb> < /usr/local/share/examples/dspam/mysql/purge.sql

freshclam.sh

Run every hour, main purpose is to silence cron mails

#!/bin/sh

result=`/usr/local/bin/freshclam`
RC=$?

# All databases are up to date, exit
[ `echo "$result" | grep -c "is up to date"` -eq 3 ] && exit 0

# DB Updated, Clamd notified, no errors, exit
[ "$result" == "${result%Database updated (*) from *}" ] && DBupdated=yes
[ "$result" == "${result%Clamd successfully notified about the update.}" ] && clamdNotified=yes
[ -n "$DBupdated" -a -n "$clamdNotified" -a $RC -eq 0 ] && exit 0

# If we reach this, we have a new ruleset and something non-default happened
echo ClamAV updates: RC=$RC

# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result"

https://paste.xinu.at/ypY62/

CategoryHowTo