Talon
You shall not pass! Unless you can deal with TLS, SPF, DKIM, DMARC, pass several checks…..
Part 1 of “The Network” which is a series of posts where I live the principle that security through obscurity is not the way. Part 2 was released Sept 24th and is called Daemon.
Today is mail server day. My mail server, the mx server for the nmrc.org domain is talon.nmrc.org, aka Talon. I’m talking about why I have Talon, and more specifically how it is set up.
As an old school hacker I never trusted my ISP’s mail server, hell I broke into the thing regularly. This was not because I hated my ISP, it was because if I learned of some mail server security issue I’d want to see if my mail spool was in danger (sorry Fastlane, nothing personal, it was the 90s). Still, they were way more secure than those free services like Yahoo, Hotmail, USA.net and so on. But this was a big part of why I wanted to have my own mail server. I wanted control.
There were other reasons too. The Electronic Communications Privacy Act (ECPA) of 1986 came through Congress and President Reagan signed it into its nightmarish existence. Still used today, it was designed pre-cloud so it only protected data for a somewhat short amount of time, and requests can be made to mail services by The Man to get that data. Sound like bullshit paranoia? ECPA originally classified data sitting on a third party’s server for more than 180 days as “abandoned” (think email in your Gmail account, Google had to hand it over), and as long as it was considered relevant to a case they could get it.
Things did improve by 2010 (US v Warshak) but I was making this decision in the 90s. And while there is still a lot that The Man can do despite rulings stemming from US v Warshak (The Man now needs a warrant to get data from any provider, thanks 4th amendment), things like National Security Letters still add to the suck and I stand by my decision even now.
The Beginning
So in 1997 I registered the nmrc.org domain, and my registrar Gandi.net got me all fixed up. A hacker buddy at an ISP in France provided DNS, my own ISP allowed me to host the web server in a subdirectory on their main server, and nmrc.org email was kept on my ISP’s mail server in a single account. My own mail server Talon would connect up to Fastlane and issue an ETRN command via Sendmail, download the spool and sort it into the individual mail accounts.
That first incarnation of Talon from 1997 was an old 386 computer with 4MB RAM and a 40MB hard drive loaded up with Slackware. Things grew from there.
Let’s Talk Configuration
We’ll start with the basics. Talon is currently an HPE Proliant ML10 v2, with an Intel i3-4150 3.50GHz 2 core processor, 8GB RAM, and a terabyte of drive space. Somewhat older, quite modest, but more than capable of performing its primary function of mail services. It is currently running Ubuntu Server 20.10, and very shortly it will get the upgrade to 21.04.
The major services exposed to the Internet include mail on port 25 and a web server on port 80 and 443. Sendmail handles the port 25 traffic, and nginx handles the web server. While I mainly use nginx to handle a shitty DNS lookup problem, it is also used to get letsencrypt keys, which are used by the web server as well as Sendmail.
I have SPF, DKIM, and DMARC set up in DNS with TXT records (I’ll cover the DNS server in another blog post edit: covered here). I use SMTPS so that when it sends mail it tries to encrypt. The encryption used is TLS, with the key coming from letsencrypt like I mentioned earlier. I use greylisting, have several DNSBLs and of course various security settings tweaked.
Here are the relevant snippets from the sendmail.mc file. This has the main milters I use - greylisting via milter-greylist and DKIM via OpenDKIM:
dnl ### milter-greylist INPUT_MAIL_FILTER(`greylist',`S=local:/var/run/milter-greylist/milter-greylist.sock')dnl dnl ### opendkim INPUT_MAIL_FILTER(`opendkim',`S=inet:8891@localhost')dnl define(`confMILTER_MACROS_CONNECT', `j, ')dnl define(`confMILTER_MACROS_HELO', `, ')dnl define(`confMILTER_MACROS_ENVFROM', `i, ')dnl define(`confMILTER_MACROS_ENVRCPT', `')dnl
This is the anti-spam part of the sendmail.mc file. As you can see, I use no less than five DNSBLs, and only then do I check my access.db file:
dnl ### anti-spam area define(`DNSBL_MAP', `dns -R A -r2')dnl FEATURE(`dnsbl', `zen.spamhaus.org', `"550 Mail from " $`'& " refused - see zen list at www.spamhaus.org"')dnl FEATURE(`dnsbl', `dbl.spamhaus.org', `"550 Mail from " $`'& " refused - see dbl list at www.spamhaus.org"')dnl FEATURE(`dnsbl', `cbl.abuseat.org', `"550 Mail from " $`'& " refused - see cbl.abuseat.org"')dnl FEATURE(`dnsbl', `bl.spamcop.net', `"550 Mail from " $`'& " refused - see bl.spamcop.net"')dnl FEATURE(`dnsbl', `dnsbl.sorbs.net', `"550 Mail from " $`'& " refused - see dnsbl.sorbs.net"')dnl FEATURE(`access_db',`hash -T<TMPF> -o /etc/mail/access.db')dnl FEATURE(`blacklist_recipients')dnl FEATURE(`greet_pause', `1000')dnl 1 seconds FEATURE(lookupdotdomain)dnl
The access.db file is the one that is updated most, probably 2-3 times per week. It has evolved over the years. It started with a way to block and allow connectivity to Sendmail but has morphed into a large resource. The source file for it is well over 25K lines long. At one point, there were several of us hacker types running our own mail servers that exchanged our blocking lists with each other, and as things have moved forward I’ve continued to add to this beast. It allows for some fun, like custom error messages:
. . . ### # ppl and domains we hate ### marcus.wade@techinstallinfo.com 550 Seriously Chris, fuck off and die in a fire islababbidge17@gmail.com 550 Seriously Isla, fuck off and die in a fire alphasights.com 550 delete all nmrc.org addresses from your spam lists, otherwise die in a fire crimsoncoal.com 550 you assholes at crimsoncoal stop trying to spam us or whatever with the repeated AUTH commands broadwick.com 550 you assholes at broadwick need to stop spamming us, fuck you and rot in hell atinav.com 550 you assholes at atinav need to stop spamming us, fuck you and rot in hell sysrec.co.uk 550 you assholes at sysrec.co.uk stop spamming us, fuck you and rot in hell resultsmail.com 550 you assholes at resultsmail stop spamming us, fuck you and rot in hell healthhaven.com 550 you assholes at healthhaven stop spamming us, fuck you and rot in hell secondits.com 550 you assholes at secondits stop spamming us, fuck you and rot in hell rosieweber.com 550 you assholes at rosieweber stop trying your lame scam, fuck you and rot in hell unifiedlayer.com 550 you assholes at unified layer need to close all those holes and block spammers ############ # unalloc'd and fucking asshole class A's ############ 1 DISCARD 2 DISCARD 7 DISCARD 23 DISCARD 27 DISCARD 31 DISCARD . . .
While I can block entire domains, there are ways to add exceptions, which is in the file as a comment:
### # individual exceptions ### # Example of allowing a single user from a blocked domain (must include the # inbound and outbound): # From:thegnome@example.com OK # To:thegnome@example.com OK # example.com REJECT ###
So people can further explore some of the configuration options, I’m including links to these files at the end of the blog post. I’ve only edited them enough to remove the privacy-related items such as email addresses of those that are allowed through from domains I normally block, otherwise they are freely available for perusal.
One thing I wanted to make clear - while I can REJECT a domain or IP address, I usually DISCARD instead. If the sender has completely forged the email, the REJECT message might bounce and I’m stuck with it. However if I use DISCARD, assuming the sender gets through the DNSBLs and greylisting, I will allow their sender daemon to waste its time until the very end, and then I simply delete the email. Because the hell with them - if I can waste the sender’s time, even just a few seconds, maybe it will prevent someone else from getting some spam, a scam, or even an attack.
Reading Email
Mail is retrieved by those with an nmrc.org email account via one of two ways. The main way is via IMAPS. Dovecot is used and is currently configured to use the same letsencrypt keys, mainly out of convenience. The other way is via direct access on the box itself. Shell access is is gained via SSH, restricted slightly by IP address, and protected via Duo Security’s Duo for Unix. Users that access mail this way can use Alpine or Mutt.
Access to IMAPS (port 993) and SSH (port 22) is restricted, but I’ll cover that in a bit with the firewall rules. In the future as I configure proper VPN access, I’ll move SSH access to Duo with PAM support, as well as a more sophisticated Dovecot configuration. That entire setup is currently in progress, and will be another blog post when complete.
The Firewall Rules
Some of the rules that reveal the IP addresses of remote users so I’ve left those rules out, but here are the most relevant:
$ sudo ufw status verbose Status: active Logging: on (medium) Default: deny (incoming), allow (outgoing), disabled (routed) New profiles: skip To Action From -- ------ ---- Anywhere REJECT IN 103.141.136.42 Anywhere REJECT IN 104.129.4.186 Anywhere REJECT IN 117.86.104.143 Anywhere REJECT IN 124.6.159.130 Anywhere REJECT IN 140.255.59.105 Anywhere REJECT IN 141.98.80.0/24 Anywhere REJECT IN 180.121.129.33 Anywhere REJECT IN 185.211.245.170 Anywhere REJECT IN 185.234.218.210 Anywhere REJECT IN 185.50.149.0/24 Anywhere REJECT IN 187.190.155.126 Anywhere REJECT IN 188.165.219.27 Anywhere REJECT IN 193.35.48.18 Anywhere REJECT IN 195.154.113.107 Anywhere REJECT IN 203.177.13.54 Anywhere REJECT IN 206.162.240.66 Anywhere REJECT IN 218.95.110.163 Anywhere REJECT IN 31.44.185.44 Anywhere REJECT IN 37.49.230.63 Anywhere REJECT IN 38.87.45.142 Anywhere REJECT IN 50.20.76.203 Anywhere REJECT IN 59.50.24.203 Anywhere REJECT IN 59.61.220.196 Anywhere REJECT IN 59.63.188.0/24 Anywhere REJECT IN 60.167.119.96 Anywhere REJECT IN 67.225.224.62 Anywhere REJECT IN 72.250.171.30 Anywhere REJECT IN 82.103.135.10 Anywhere REJECT IN 93.95.98.60 25/tcp ALLOW IN Anywhere 80/tcp ALLOW IN Anywhere 443/tcp ALLOW IN Anywhere 22/tcp ALLOW IN 65.70.17.130 587/tcp ALLOW IN 65.70.17.130 993/tcp ALLOW IN 65.70.17.130
All of the REJECT rules are tied to repeated attacks from specific email servers that were perceived to be attacking Talon. This includes mail relay attempts, enumeration of user accounts, and phishing attacks. Some of those IP addresses were compromised mail servers being used for spear phishing attacks. I’ve left those rules in place, mainly because if they’re lax enough to be compromised and become an attack platform for someone else, it could happen again in the future. So leaving them in place seems to make the most sense.
The 65.70.17.130 IP address is the gateway system for the internal network, and allows access to port 22 for shell access, and ports 587 and 993 for mail sending and mail retrieval respectively. And of course port 25 for inbound mail, and 80 and 443 for web.
Sophisticated Attacks
A part of my background includes previously working for a US government defense contractor, and this - coupled with a few “activities” by NMRC members against nation states with poor human rights records - have led to attacks from nation state APT groups, typically in the form of spear phishing. While that has influenced a lot of the configuration decisions, I feel that by making sure I can more readily thwart such attacks, it has led to a safer system overall. In a future blog post I’ll detail out some of the attack attempts I’ve dealt with.
Conclusion
Yes, this is a lot of trouble but it keeps my mail spool out of the hands of a cold and uncaring corporation that would just as soon as harvest it for ad fodder as it would hand it over to a government request with or sometimes even without a subpoena. In my mind, worth it. More than once the knowledge gained of the underlying protocols has paid dividends to employers whose systems I’ve helped protect. And it certainly helps keep me on my toes as I get to see live attacks up close and personal.
For full disclosure, here are the relevant config files in (almost) their entire glory, only the private info of NMRC members and friendly email servers has been edited out. Enjoy!
/etc/mail/access file: talon-access.md
/etc/mail/sendmail.mc file: talon-sendmail-mc.md
/etc/nginx/sites-available/default file: talon-nginx-conf.md
Firewall configuration: talon-firewall-rules.md
