OpenSMTPd, jails and amavisd-new for virus scanning and DKIM signing and spam classification using SpamAssassin and DSPAM
Contents
Installation
Required ports
WIP mail/spamd Throttling
mail/opensmtpd OpenSMTPd MTA
mail/opensmtpd-extras OpenSMTPd extra filters
security/amavisd-new "A Mail and Virus Scanner"
mail/spamassassin Apache SpamAssassin e-mail classifier
mail/dspam Bayesian e-mail classifier
mail/rspamd rspamd e-mail classifier
security/clamav Open source virus scanner
DSPAM !SpamAssassin plug-in
One of MariaDB/MySQL, PostreSQL, SQLite or Hash database
Jails
- mail (opensmtpd, dovecot)
- scan (amavis, spamassassin, dspam, clamav)
Housekeeping
Jail |
What |
When |
Why |
scan |
daily |
Keep SpamAssassin rules up-to-date |
|
scan |
hourly |
Keep ClamAV signatures up-to-date |
|
scan|db |
daily |
Remove old messages and signatures from dspam database |
|
scan |
hourly |
Process false-positives/-negatives |
Sub-component notes
Base OS
Disable sendmail
sendmail_enable="NONE"
spamd
Listens on 25 and proxies the inbound connections using black-/grey-/white-listing, tarpitting, etc.
Generates and maintains a hash-database of black-/grey-/whitelisted senders.
OpenSMTPd
Intended flow of mail from authenticated users to be relayed
- Receive on port 25/465/587 (smtp/smtps/submission) with authentication
Classified as local when user is authenticated
- DKIM-sign message
- Relay
Intended flow for mail for any of my domains
- Receive on port 25 (smtp)
- Domain of receiver is one of allowed domains
- Forward to AMaViS service
- Bayesian classification using DSPAM
b. Rule-based classification using SpamAssassin c. Virus scan using ClamAV
- Bayesian classification using DSPAM
- Receive from AMaViS service (tag Scanned)
- Deliver to Dovecot using aliases db
All other mail must be bounced.
DKIM signing
Generate a 4096-bit DKIM signing key using LibreSSL in the location where the DKIM-signer filter expects it
$ /usr/local/bin/openssl genrsa -out /etc/ssl/private/rsa.private 4096
Once added to your amavisd config you can generate relevant DNS TXT records using
$ amavisd showkeys default._domainkey.example.org. 3600 TXT ( "v=DKIM1; p=" "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3bcAwPnDM89FdJqWo8Om" "C4qfsfPIWIjqCnqp8H0QT03jAYwwzTwrKNb2w2dWjf9fp8D/Jm21XAT4GDhUWEQA" "XBxA2Ds5cHIPiSHRCstj7+kww1S40mQcS9WcUJjjO0jaGzH+Lfm02ZXC/E0i7YJY" "m9GKwPBz8M/8gMNCIbTuf9exVCmFW1gjnHMfQHOX0x8oBVLiby67vFl9UBwrIBSe" "+6T73t10fIWyTtkuFkJKNiqDnat8p0KMzpYwviBU8Aec1wTUFo9rK9bUgFJYdXUF" "q3gRN3glLlj4ho7yjZdqlofYO1CbdZ5FjyYs559jRd+dOUZh84Tmazg5PNx8Jeus" "MwIDAQAB")
Testing via any gmail account shows you if your DKIM setup is correct, look for the following in the smtp headers of a message that passed by gmail's service
Authentication-Results: mx.google.com;
spf=neutral (google.com: 139.254.214.162 is neither permitted nor denied by best guess record for domain of brnrd@example.org) smtp.mail=brnrd@example.org;
dkim=pass header.i=@example.org
Lessons learnt
OpenSMTPd foot-shooting notes:
accept/reject rules work in order
accept/reject rules make sure you have a from and for check
- Ip-addresses of the host itself are seen as 'local'
Dovecot
Using Dovecot's LDA in OpenSMTPd provides the following benefits:
- Mailbox indexing during mail delivery, providing faster mailbox access later
- Quota enforcing (plugin)
- Sieve language support (plugin)
This setup adds mail/dovecot2-antispam-plugin to the configuration for easy (re-)classification of mails. This triggers copying of a message when copied/moved into or out of the Spam folder. The NotSpam and IsSpam have to be trained with DSPAM.
AMaViSd
Listens on 10024 to scan mails
Scanning
- ClamAV
DSPAM
- rspamd
SpamAssassin
Viruses can be rejected without spam classification/scanning.
First pass through DSPAM for bayesian classification (I've found it to be far superior to SpamAssassin's bayesian classifier) and then through SpamAssassin to allow SpamAssassin to take the X-DSPAM headers generated by dspam into account when scoring
All this results in the following headers:
X-DSPAM-Processed: Sun Jun 7 11:03:51 2015
X-DSPAM-Confidence: 0.9899
X-DSPAM-Probability: 0.0000
X-Virus-Scanned: amavisd-new at brnrd.eu
X-Spam-Flag: NO
X-Spam-Score: -2.618
X-Spam-Level:
X-Spam-Status: No, score=-2.618 tagged_above=-999 required=6.2
tests=[DSPAM.Innocent=-1.000, DSPAM_HAM_99=-3.23,
MSGID_FROM_MTA_HEADER=0.001, RDNS_DYNAMIC=0.363, TO_MALFORMED=1.247,
UNPARSEABLE_RELAY=0.001] autolearn=disabled
X-DSPAM-Result: Innocent
X-DSPAM-Signature: 447408f7488584401825356
DSPAM
NOTE: Documentation seems to be gone from the sourceforge.net project page NOTE2: I'm no longer using DSPAM, I've moved to rspamd (with spamassassin rules added)
This setup creates a 'site-wide' configuration, all mail is classified and trained under the vscan user.
To use dspamc from amavisd you must make /usr/local/etc/dspam.conf readable for the vscan user so dspamc can read its ClientHost configuration.
Training
Get a corpus of spam and ham to train dspam (in separate folders) as the vscan user.
dspam_train vscan /path/to/spamdir /path/to/hamdir
False positives/negatives
The Dovecot antispam-plugin must be configured to put mis-classified messages in /path/to/NotHam and /path/to/NotSpam folders. You'll need to create a script to feed these to dspam to update the database tokens
rspamd
Yet to be documented
SpamAssassin
NOTE: Now integrated into my rspamd config
Download the dspam SpamAssassin module (dspam.pm) and ruleset (dspam.cf) and drop them in /usr/local/etc/mail/spamassassin
Add to your init.pre
loadplugin Mail::SpamAssassin::Plugin::dspam /usr/local/etc/mail/spamassassin/dspam.pm`
and to your local.cf
include dspam.cf
The Bayesian tests and auto-learning can be disabled as they are replaced with DSPAM.
Validate your configuration with spamassassin --lint, when there's problems, adding -D shows you debug output that helps find the root of the problem.
Example configurations
NOTE: These are not complete configurations but show the relevant parts that require modification!
etc/mail/smtpd.conf
1 # generate db using makemap
2 table aliases db:/usr/local/etc/mail/aliases.db
3 table domains { example.org, example.net, example.com }
4 table secrets db:/usr/local/etc/mail/secrets.db
5
6 filter dkim-sign dkim-signer "-Dexample.org"
7 filter dnsbl-sorbs dnsbl "-h dnsbl.sorbs.net"
8 filter dnsbl-spamcop dnsbl "-h bl.spamcop.net"
9 filter dnsbl-spamhaus dnsbl "-h zen.spamhaus.org"
10 filter dnsbl-all chain dnsbl-sorbs dnsbl-spamcop dnsbl-spamhaus
11
12 lan_addr = "192.0.2.2"
13 scan_addr = "192.0.2.5"
14
15 # Define keys and certs (PEM encoded)
16 pki example.org certificate "/etc/ssl/certs/example.org.cer"
17 pki example.org key "/etc/ssl/priv/example.org.key"
18
19 filter dkim-sign dkim-signer "-d example.org -p /usr/local/etc/mail/example.org.dkim.key"
20
21 # Inbound mail smtp, smtps, deliver
22 listen on $lan_addr port 25 filter dnsbl-all tls \
23 pki example.org hostname example.org auth-optional
24 listen on $lan_addr port 465 smtps \
25 pki example.org hostname example.org auth
26 listen on $lan_addr port 587 dkim-sign tls-require \
27 pki example.org hostname example.org auth \
28 filter dkim-sign
29 # Receive scanned mails from amavisd-new
30 listen on $lan_addr port 10025 tag Scanned
31
32 # Deliver locally messages coming back in from scanner
33 accept tagged Scanned from source $scan_addr \
34 for domain <domains> alias <aliases> \
35 deliver to lmtp "/var/run/dovecot/lmtp"
36 reject tagged Scanned
37
38 # Forward all mail received for local domains to amavis
39 accept from any \
40 for domain <domains> \
41 relay via "smtp://192.0.2.5:10024"
42
43 # Relay anything that came in from authenticated users
44 accept from local \
45 for any \
46 relay
47 # If you need to use a 'smarthost' use
48 # relay via tls+auth://user@mta.example.net:587 <secrets>
Line 33: Use dovecot's LDA so it updates the indexes for the Maildir as well (otherwise indexing takes place when the user 'opens' the mailbox)
Line 39: There's no macro expansion in quoted strings, so you must use the IP-address or a valid hostname.
etc/amavisd.conf
1 $MYHOME = '/var/amavis';
2 $pid_file = "/var/run/amavisd/amavisd.pid";
3
4 @inet_acl = qw(192.0.2.1/24);
5
6 $inet_socket_port = [10024];
7
8 $notify_method = 'smtp:[192.0.2.2]:10025';
9 $forward_method = 'smtp:[192.0.2.2]:10025';
10
11 @spam_scanners = (
12 ['DSPAM', 'Amavis::SpamControl::ExtProg', '/usr/local/bin/dspamc',
13 [ qw(--mode=teft --deliver=stdout --user vscan) ],
14 mail_body_size_limit => 256000, score_factor => 1,
15 ],
16 ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ],
17 );
18
19 @av_scanners = (
20 ['ClamAV-clamd',
21 \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
22 qr/\bOK$/m, qr/\bFOUND$/m,
23 qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
24 );
dspam
1 Preference "signatureLocation=headers"
And make sure you have all storage backend params in order
SpamAssassin
use_bayes 0
loadplugin Mail::SpamAssassin::Plugin::dspam dspam.pm
Disable bayes (you can remove all lines referring to bayes as well)
Testing
amavisd
telnet or netcat/nc work fine for me to test amavisd. Prepare a complete mail-transmission to paste
EHLO localhost MAIL FROM: <someone@example.org> RCPT TO: <noone@example.net> DATA <the original mail content including headers> . QUIT
Remember that amavisd will deliver the mail to OpenSMTPd on port 10025 tagged Scanned
dspam
You can test dspam using the dspamc client program:
dspamc --mode=notrain --deliver=stdout --user vscan < test-ham.mail
Simulation works best when you switch user to vscan before executing the command (tests if dspamc can read the client config)
Scripts
dspam-learn
1 #!/bin/sh
2
3 notHamPath=/path/to/notHam
4 notSpamPath=/path/to/not/Spam
5 spamCorpusPath=/path/to/SpamCorpus
6
7 IFS="
8 "
9
10 extractHeader () {
11 headers=`grep -Eim3 '^(to|from|subject):' $1`
12
13 for header in $headers ; do
14 case $header in
15 [Ff][Rr][Oo][Mm]:* ) from="${header#[Ff][Rr][Oo][Mm]:}" ;;
16 [Tt][Oo]:* ) to="${header#[Tt][Oo]:}" ;;
17 [Ss][Uu][Bb][Jj]???:* ) subject="${header#[Ss][Uu][Bb][Jj][Ee][Cc][Tt]:}" ;;
18 esac
19 done
20 echo -e "From: ${from}\nTo: ${to}\nSubject: ${subject}\n"
21 }
22
23 trainIsSpam () {
24 cd "$notHamPath"
25 [ "$(ls -A .)" ] || return
26
27 # Mails that have been mis-classified
28 for file in `grep -E -l "^X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
29 extractHeader "${file}"
30 /usr/local/bin/dspamc --user vscan --class=spam --source=error < "${file}"
31 mv "${file}" "${spamCorpusPath}"
32 done
33 [ "$(ls -A .)" ] || return
34
35 # Mails that have already been classified as Spam
36 for file in `grep -E -l "X-DSPAM-Result: Spam" *` ; do
37 rm "${file}"
38 done
39 [ "$(ls -A .)" ] || return
40
41 # Mails that were not previously classified
42 for file in * ; do
43 extractHeader "${file}"
44 /usr/local/bin/dspamc --user vscan --class=spam --source=inoculation < "${file}"
45 mv "${file}" "${spamCorpusPath}"
46 done
47 }
48
49 trainIsHam () {
50 cd "$notSpamPath"
51 [ "$(ls -A .)" ] || return
52
53 # Mails that have been mis-classified
54 for file in `grep -E -l "^X-DSPAM-Result: Spam" *` ; do
55 extractHeader "${file}"
56 /usr/local/bin/dspamc --user vscan --class=innocent --source=error < "${file}"
57 rm "${file}"
58 done
59 [ "$(ls -A .)" ] || return
60
61 # Mails that have already been classified as Ham
62 for file in `grep -E -l "X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
63 rm "${file}"
64 done
65 [ "$(ls -A .)" ] || return
66
67 # Mails that were not classified
68 for file in * ; do
69 extractHeader "${file}"
70 /usr/local/bin/dspamc --user vscan --class=innocent --source=inoculation < "${file}"
71 rm "${file}"
72 done
73 }
74
75 trainIsSpam
76 trainIsHam
sa-update
1 #!/bin/sh
2
3 PATH=$PATH:/usr/local/bin ; export PATH
4
5 result=`sa-update -v 2>&1`
6 RC=$?
7
8 # Output of sa-update is "Update finished, no fresh updates were available"
9 # when no new update is available, additionally RC=1
10 [ "$result" == "Update finished, no fresh updates were available" ] && exit 0
11
12 # If we reach this, we have a new ruleset
13 echo New ruleset: RC=$RC
14
15 # Generate some some text for your periodic/cron output so it sends you a mail
16 echo "$result" | grep "Update available for channel"
17
18 result=`sa-compile 2>&1`
19
20 # AMaViDS needs to be restarted to pick up the new rules
21 service amavisd restart
dspam-purge
Run daily
#!/bin/sh
mysql -u <dspamuser> -p <dspampass> -h db.host <dspamdb> < /usr/local/share/examples/dspam/mysql/purge.sql
freshclam.sh
Run every hour, main purpose is to silence cron mails
#!/bin/sh
result=`/usr/local/bin/freshclam`
RC=$?
# All databases are up to date, exit
[ `echo "$result" | grep -c "is up to date"` -eq 3 ] && exit 0
# DB Updated, Clamd notified, no errors, exit
[ "$result" == "${result%Database updated (*) from *}" ] && DBupdated=yes
[ "$result" == "${result%Clamd successfully notified about the update.}" ] && clamdNotified=yes
[ -n "$DBupdated" -a -n "$clamdNotified" -a $RC -eq 0 ] && exit 0
# If we reach this, we have a new ruleset and something non-default happened
echo ClamAV updates: RC=$RC
# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result"