Spam: Protecting Your Users/System

It is utterly impossible to do this job completely. However, there are a many things you can do to nail most spam.

• Delete all email that is malformed. A lot of spam has one or more of these technical flaws (usually due to automated bad forgery), and if any legitimate mail has it, well, who could blame you for dropping malformed cruft???

  • The timestamps in the Received line contain a mismatch between the time zone initials and the time zone offset. The most common goofs are "-0600 (EST)" and "-0700 (EDT)", though I have seen a few variations on these themes.

  • The Received lines contain impossible IP addresses (too few/many pieces, or pieces out of range).

  • The domains in the From lines do not exist. There is no reason for this to happen, at least on purpose, on email. There may be occasional lusers using cruft like Nutscrape Kaboomicator, munging for web browsing and Usenet posting purposes, and forgetting to demunge for emailing purposes... but then, it's his own damn fault!

• Delete all mail that, for other reasons, you can be absolutely certain is spam. For instance, back when Cyberpromo was spewing spam with a header line that said: "Received: --CLOAKED!--", you could key on the presence of such a line.

• Sidetrack for manual inspection all email with things in them that are typical of spam. For instance, anything that:
  • mentions money, sex, web sites, or email addresses in the subject
  • is addressed to friend@public.com, you@yourdomain.com, or some such
  • is from nobody@nowhere.com or a username that begins with a digit
  • has "4u" anywhere in the from line
  • has any of the major free email services in the From or Reply-To line
  • came from or through spamhavens such as ScrewYouNet or Compuscum
  • was relayed through anywhere in Asia
  • used a forged domain in the earliest Received line
  • tons of others....

  However, there are some pretty serious downsides:

  • It can be highly labor intensive, especially when not combined with nuking definite spam, and especially for large sites. I can do this because first off, the local Fidonet/Internet gateway does a first pass of much more sophisticated filtering, and secondly, there are only a few hundred active users receiving email through my site anyway.

  • It presents a highly sensitive privacy issue, so it should only be used if you make it VERY clear to the user, BEFORE he signs up, that you do this. If you institute it as a new policy, again make sure that all users are aware before you start, and give all users the opportunity to cancel and get a full refund, even if they got special prices by signing up for a long-term deal.

  • You'll get a lot of false positives, which means you must pre-screen so as to "bless" those emails fitting the "probable spam" criteria, but which for other reasons you know are not spam. This is fairly common with legitimate marketing mailing lists.

To sum up, follow this algorithm:

1. If it's malformed, nuke it.
2. Else if it fits the criteria of definite spam, nuke it.
3. Else if it's something you know fits the criteria of probable spam, but isn't, process it as normal.
4. Else if it fits the criteria of probable spam, sidetrack it.
5. Else process it as normal.

(Of course, that's in addition to any relay rejection you may want to do, but that is beyond the scope of this document. Consider that part of "process as normal", which should include "if it's not to your site, or any site with which you have explicit forwarding agreements, bounce it".)

Then, inspect what's been sidetracked ASAP. When you find something that is spam but has something distinctive about it, add that something to the "definite spam" filters; when you find something that is not spam but has something distinctive about it, add that something to the "smells like spam but isn't" filter. Also ask your users to report to you any spam that slips through intact, so you can tune your "probable spam" filter too.

Now, what software do you use for all this? On a standard Unix setup, most of it can be done with sendmail. I have heard that it can also be done with qmail. I've even heard that these can reject definite malformed crap and other spam before it even gets into your system. However, I wouldn't know, I haven't done that kind of setup yet. My system is a Fidonet BBS, run under OS/2, receiving the email in Fidonet packet format, most of it bound for other local Fidonet BBSes. (My system is the email router for the Washington DC area Fidonet.) I currently use several large NetMgr scripts to do it, but firstly, NetMgr has Y2K problems, and secondly, and sadly, it is no longer supported. The previous local Fidonet/Internet gateway, running on a UUCP feed (as many such gateways do) cobbled together his own filtering software, that worked quite well, in Perl under OS/2.

Well, that's the technical side. Then there's the BOFHly side....