Dealing with spam

[bscott]

Having lurked here for a while it is time for me to emerge from the shadows and say hello, wish you all a Happy New Year and give my thanks to those who make this forum the great place that it is. (A few credits on the way to mouser).

Can I ask how people are dealing with Spam?

I have found Cloudmark (http://www.cloudmark.com/desktop/) a very effective solution on the desktop - see Tech Support Alert review  here http://www.techsupportalert.com/reviews/review-cloudmark.htm.

Cloudmark works fine for me on Thunderbird which I have just adopted in place of Outlook but I need to catch spam at my mail server for when I use the Roundcube client.

I have just set up Exim4+SpamAssassin on  Debian Etch  but a lot of spam is getting through on the default settings - I will need to spend some time finding out how to tweak it.

I would be interested in other peoples experience in dealing with Spam on the desktop and at the server.

Regards,

Bob Scott

[f0dder]

Humm, for thunderbird I personally just use the built-in junk marking thing, works fine enough for me.

Anyway, at the mail server side I would implement greylisting. It works by actually rejecting mails(!) whenever you get mail from a new domain, but instead of whitelisting (where you would then have to manually add the domain, quite tedious!) it relies on SMTP servers re-trying the send as per the SMTP specification.

I haven't seen any instances where valid mail doesn't get through, and it's pretty effective in blocking spam, since most spammers don't use fully-fledged SMTP servers that retry, they use shotgun tactics to try and reach as many as possible rather than reliably delivering to each recipient.

I haven't set up such a system yet myself, but I plan on moving to in-house mailhosting at the museum I do admin work for, using Dovecot and postfix. And of course with greylisting.

[yksyks]

I'm using for a quite long time SpamPal (freeware) on a dedicated server (rather old computer running Win2000) with the last freeware version of a mail server 602Lan Suite. SpamPal combines several filter techniques using plugins, which is inevitable nowadays, including RegEx, Bayesian, collaborative filtering and others. It also uses public blacklists, which are becoming more and more unreliable. Very useful is an automated whitelist, i.e., all persons you write to are automatically whitelisted. Manual black- and whitelists are of course available, too.

The program unfortunately lacks any further development due to serious health problems of its author, however, it still runs almost perfectly. It takes some time to tweak it properly according to your spam profile. There's also an inevitable period of time spent on teaching the Bayesian filter, but this all applies to all similar applications. I'm doing this management using RealVNC, but after some time there's almost no need to touch it.

Also promising might be Spamato, does anyone have any experience?

[Carol Haynes]

fOdder that's an interesting idea - but can you set it up to do it on a per email basis rather than domain name?

The reason I ask is that a lot of spam comes from common domain names (usually ISP domain names or common free webmail sites such as Yahoo and Hotmail). Mostly these are spoofed email address sources and so it is difficult to track down the actual source. Greylisting by domain name would only have partial success on the constant stream of crap I receive as a lot of the domains would have been automatically whitelisted by genuine emails being sent from the same domain name.

If you could do this you would effectively only have email coming from addresses from serious senders (or spammers that do respond to bounces).

FWIW I use POPfile as my filtering method of choice. It is very effective and a lot more flexible than pure spam filtering tools as you can use it to effectively set up intelligent rules to sort or label incoming email. It works with any POP client for windows and on any other operating system that supports PERL and is open source. See http://popfile.sourceforge.net

Edit/Update: Just looking at the latest release of POPfile it now supports IMAP (probably did before but I didn't notice it). I also see it support NNTP - which is a very interesting idea! POPfile has been at early release numbers (last version was 0.22.5) but in December came of age at version 1.0. Whooppee

[tinjaw]

I use Tuffmail as my email hosting provider. I can't recommend them enough. Small shop. One man, I believe. Customer service is excellent. Their feature set is top notch. I have very little problem with spam. One of the nicest thing is that it is so easy to do server side spam rejection. I'm talking checkbox easy! What gets through that then must run the gauntlet of Spam Assassin. By default you have three Inboxes: Inbox, Junkmail, and Discard. These are basically your green, yellow, and red. I don't keep track, but I can't get any more than one or two true spam emails in my inbox. Things that have passed the gauntlet but still look a bit fishy go in Junkmail. If somebody sent me a legitimate email and I didn't get it in my ebox, this is usually where it is. The stuff in Discard is five nines spam. I use the web front end (imp3/4, squirrelmail, and they're testing Roundcube (AJAX). Everything I do with them is secured, which matters when I connect through my workplace. They even have LDAP address book support. When at home I connect using Thunderbird via IMAP.

I don't get any kickbacks. I am just a fanboy customer. They have a 30 Day free trial plus a 30 day money back guarantee. I think this is where I am supposed to say, "You can't afford NOT to try it!!!" 

[attachimg=#]

I almost forgot to mention that Sieve filtering comes with every address as well.

[bscott]

I was previously using Outlook and found its junk mail filtering pretty useless so I got Cloudmark for that. Since I already had the license I used it for Thunderbird. I will try the Thunderbird junk email filtering and see how it compares to Cloudmark and maybe I can save myself the license renewal.

Not too sure how much I like the greylisting idea. Interesting, but if it becomes widely used and effective, the spammers have an easy work around.

[tinjaw]

Not too sure how much I like the greylisting idea. Interesting, but if it becomes widely used and effective, the spammers have an easy work around.

Greylisting with Tuffmail is specific to your account, and you choose what gets greylisted by the settings you choose. You can turn it up or down and they have a few presets for you to use.

[f0dder]

fOdder that's an interesting idea - but can you set it up to do it on a per email basis rather than domain name?

The reason I ask is that a lot of spam comes from common domain names (usually ISP domain names or common free webmail sites such as Yahoo and Hotmail). Mostly these are spoofed email address sources and so it is difficult to track down the actual source. Greylisting by domain name would only have partial success on the constant stream of crap I receive as a lot of the domains would have been automatically whitelisted by genuine emails being sent from the same domain name.

Well, afaik it doesn't look at the (possibly spoofed) "From: " header line, but rather the host that's connecting to it. And it's my experience that spam mails are sent from infected zombie PCs, not through valid mail accounts, so it should work just fine

I was previously using Outlook and found its junk mail filtering pretty useless so I got Cloudmark for that. Since I already had the license I used it for Thunderbird. I will try the Thunderbird junk email filtering and see how it compares to Cloudmark and maybe I can save myself the license renewal.

Thunderbird's junk filtering is only bayesian filtering, while Cloudmark also does filtering based on what all other Cloudmark users do? My primary email client right now is TheBat with the free version of AntiSpamSniper which is also just Bayesian, and once trained, so little spam gets through that I wouldn't pay for more comprehensive solutions.

Not too sure how much I like the greylisting idea. Interesting, but if it becomes widely used and effective, the spammers have an easy work around.

Considering that spammers mainly operate from open (and buggy) relays and zombie botnets (which don't have proper SMTP servers installed but "just enough to work"), I don't think they're going to defeat greylisting anytime soon.

Everything I do with them is secured, which matters when I connect through my workplace. They even have LDAP address book support. When at home I connect using Thunderbird via IMAP.

Sounds very nice, just keep in mind that most emails don't go through an entirely secure path when travelling through the internet, so even if you, as the final link use encrypted access, it's still possible for people to snoop on your mails through previous links.

[tinjaw]

Sounds very nice, just keep in mind that most emails don't go through an entirely secure path when travelling through the internet, so even if you, as the final link use encrypted access, it's still possible for people to snoop on your mails through previous links.

Yes, I understand. Let me clarify what I meant. I make a secured connection to IMAP. My password and my traffic to and from the email server is encrypted. If I wish to send the email out as encrypted, Tuffmail supports PGP fully. Even through the web client.

[Eóin]

My email provider offers Tagged Message Delivery Agent (TMDA) and personally I really like it. But do a google on the pros and cons, a lot of people aren't fond of it's challenge-response system.

[f0dder]

About Greylisting:
http://en.wikipedia.org/wiki/Greylisting
http://www.tuffmail.com/faq.php#greylisting

[Carol Haynes]

About Greylisting:
http://en.wikipedia.org/wiki/Greylisting
http://www.tuffmail.com/faq.php#greylisting

Whilst I was initially interested this quote from the wiki arttcle is pretty much what I suspected would be the major drawback:

Perhaps the most significant disadvantage of greylisting is the fact that, like all spam mitigation techniques, it destroys the near-instantaneous nature of email people have come to expect, and throws email back to the early days when it was slow and unreliable. A customer of a greylisting ISP can not always rely on getting every email in a small amount of time. Thus email loses its function as easy and effortless vehicle to transfer electronic information instantenously.

[f0dder]

Whilst I was initially interested this quote from the wiki arttcle is pretty much what I suspected would be the major drawback:

Perhaps the most significant disadvantage of greylisting is the fact that, like all spam mitigation techniques, it destroys the near-instantaneous nature of email people have come to expect, and throws email back to the early days when it was slow and unreliable. A customer of a greylisting ISP can not always rely on getting every email in a small amount of time. Thus email loses its function as easy and effortless vehicle to transfer electronic information instantenously.

Yes, that does sound nasty, but in practice it means that the first time you receive mail from a new domain, it can take a little time to get through; after that, it's near-instantaneous again. It is annoying when you visit a new e-commerce site or sign up to a new forum etc., but it's a small price to pay imho.

[Stoic Joker]

fOdder that's an interesting idea - but can you set it up to do it on a per email basis rather than domain name?

The reason I ask is that a lot of spam comes from common domain names (usually ISP domain names or common free webmail sites such as Yahoo and Hotmail). Mostly these are spoofed email address sources and so it is difficult to track down the actual source. Greylisting by domain name would only have partial success on the constant stream of crap I receive as a lot of the domains would have been automatically whitelisted by genuine emails being sent from the same domain name.

Well, afaik it doesn't look at the (possibly spoofed) "From: " header line, but rather the host that's connecting to it. And it's my experience that spam mails are sent from infected zombie PCs, not through valid mail accounts, so it should work just fine

Actually it's both. The zombies use the local valid (address book) addresses & the local users to pump out mail from valid addresses. There are actually 2 from lines in an Email header, buth can be spoofed, one's is just a bit harder than the other. Either or both can be checked by spam software. This is why there is a push in the industry to blacklist all of the ISP's dynamic IP address ranges.

Not too sure how much I like the greylisting idea. Interesting, but if it becomes widely used and effective, the spammers have an easy work around.

Considering that spammers mainly operate from open (and buggy) relays and zombie botnets (which don't have proper SMTP servers installed but "just enough to work"), I don't think they're going to defeat greylisting anytime soon.

True but Open Relays aren't necessarily buggy. Any comercial server that isn't secured properly can be exploited. The use of blank or simple passwords on a network can be used by leveraged by spamers to gain access to a server to pump tons of mail through a valid account. I once did a clean-up on an Exchange server that had been subjected to this kind of treatment. The box had shutdown when the 80Gig drive overflowed due to the volume of mail that had poured through it in a single weekend. All because some jackass thought it was simpler if they left their password blank!!!


Directory Harvesting is the biggest problem with Server Side spam control, because even if the mail doesn't make it through to a user's mailbox ... it still has to be processed by the server! A Spammer will send a dictionary full of  Emails to a single domain name using a huge list of common names in the hopes of gleening a list of valid account that can then be targeted further. This is why Exchange auto-deletes any mail that would have been destined for the (invalid address) BadMail folder.

NDR reports are used by the spamers to Identify which acounts are valid in a given address space. Disable them. NDR based DoS attacks on mail servers were common for a while, because it really wasn't that hard to send one into an infinite loop using a carefully crafted batch of bad mail. The server would simply and litterally NDR itself to death! The Email that contain large blocks of rembling nonsense garbage text are designed specifically to get less informed admins to feed them the their Baysien filters to in an effort to confuse and cripple its ability to filter mail properly. A common exploit that (sadly) works beautifully btw.

Our mail server received over 4 billion pieces Directory Harvest of spam last year.

[tranglos]

I was using PopFile for many years - one of the most active projects at SourceForge: http://popfile.sourceforge.net/
It was, I think, one of the first implementations of Bayesian filtering, and pretty good too, with relatively few false positives after a week of training or so.

Downsides are large memory footprint, rather slow startup, and slow operation when the database grew after several years of usage. Also, configuration is done via the browser, so when you see a miscategorized message, it takes some doing to teach Popfile a new trick.

A few months ago I replaced PopFile with AntiSpamSniper: http://www.antispamsniper.com/ - a Bayesian plugin for TheBat. I also supports Outlook and Outlook Express, but not Thunderbird, I'm afraid. There is a quite capable free version, but I went for the paid version ($19.95), which has a few nice additions, such as a dedicated toolbar and automatic whitelisting of addresses you send email to.

Screenshots: http://www.antispamsniper.com/thebat-plugin-screenshots.html

After about three months it produces even fewer false positives than PopFile (maybe one a month!), but a little more false negatives. Specifically, I've been recently getting lots of one-line spam, sometimes only a couple of words, and Antispamsniper has trouble recognizing those messages as spam, probably because there is very little content to hook on to, and each message is different.

I like it a lot - it does not slow down email download nearly as much as PopFile did, the memory footprint is negligible, and of course the training happens inside TheBat, so it's more convenient.

For a long time I used to use a procmail filter on my mail provider's shell account, but updating it was rather tedious, and every time I wanted to update the filter I had to refresh my memory of procmail syntax, which is somewhat involved. I finally switched to using procmail only for logging incoming mail, which is useful to investigate "missing" messages sometimes, but I no longer use it for blocking. Bayesian filtering is certainly much more effective than "dumb" keyword filters.

[Carol Haynes]

Whilst I was initially interested this quote from the wiki arttcle is pretty much what I suspected would be the major drawback:

Perhaps the most significant disadvantage of greylisting is the fact that, like all spam mitigation techniques, it destroys the near-instantaneous nature of email people have come to expect, and throws email back to the early days when it was slow and unreliable. A customer of a greylisting ISP can not always rely on getting every email in a small amount of time. Thus email loses its function as easy and effortless vehicle to transfer electronic information instantenously.

Yes, that does sound nasty, but in practice it means that the first time you receive mail from a new domain, it can take a little time to get through; after that, it's near-instantaneous again. It is annoying when you visit a new e-commerce site or sign up to a new forum etc., but it's a small price to pay imho.

Sorry - I think I am getting confused here - what do you mean by 'domain' in this context.

For me domain means the bit after the @ symbol so if I have some people who write to me from a common domain name such as hotmail.com which are perfectly legitimate emails so they correctly bounce back to me after being rejected then won't everything addressed to hotmail.com addresses also be validated without further checking?

I used hotmail as an example because I do know people who use hotmail (and similar domains) so how in these cases would greylisting help?

As for POPfile I have been using it for a number of years without problem but then I suppose there have been times when I started filter training again. I used to use Outpost (a plugin for Outlook which is a neat front end in Outlook for POPfile) but unfortunately that stopped development some time ago. More recently I use POPfile within Outlook by simply having a link in my mail folder list that opens the web interface page in Outlook - so it is very simple to go in and change false recognition.

[f0dder]

Sorry - I think I am getting confused here - what do you mean by 'domain' in this context.

For me domain means the bit after the @ symbol so if I have some people who write to me from a common domain name such as hotmail.com which are perfectly legitimate emails so they correctly bounce back to me after being rejected then won't everything addressed to hotmail.com addresses also be validated without further checking?

validated by the greylisting filter, yes. And only for mail originating from those servers, not spoofed to be from those servers. Spam filtering kicks in after greylisting to get rid of actual spam sent from a greylist-okay domain.

The idea with greylisting is that most spamming is done from very brutish zombie software that doesn't implement much of the SMTP protocol, namely re-transmits on temporary error.

[Carol Haynes]

So you are saying that the greylist keeps a list of approved servers (IP addresses?) not domain names? If an email arrives from an unapproved server it is rejected as a temporary error and the server is added if it is resent/received from the same server and the email accepted for processing further?

Sorry I thing I was getting muddled because the terminology was a bit muddled between servers and domains.

If this becomes a widespread tool wouldn't it be easy for spammers to simply check for returned mail and send it out again by beefing up their zombie servers - opening the floodgates to spam and also incidentally validating the email address (because it isn't simply bounced/ignored) so you get spammed even more ?

[f0dder]

So you are saying that the greylist keeps a list of approved servers (IP addresses?) not domain names? If an email arrives from an unapproved server it is rejected as a temporary error and the server is added if it is resent/received from the same server and the email accepted for processing further?

Sorry I thing I was getting muddled because the terminology was a bit muddled between servers and domains.

If this becomes a widespread tool wouldn't it be easy for spammers to simply check for returned mail and send it out again by beefing up their zombie servers - opening the floodgates to spam and also incidentally validating the email address (because it isn't simply bounced/ignored) so you get spammed even more ?

Well, I must admit I don't know 100% how it's implemented internally, and from the tuffmail FAQ it sounds like it's a bit different than what I thought - I was commenting from end-user experience using a greylisting mailserver 

The first time we see an IP address/sender/recipient tripple, and the sender/server meets one of the criteria for Greylisting, the message will be rejected with a temporary error code. A message from an SMTP server that attempts delivery 5 minutes or more after the first delivery attempt to the same IP address/sender/recipient tripple will be accepted.

Yes, I guess that if this was very widespread, spammers might implement more of the SMTP protocol and do proper re-send attempts. But it would probably stop the "dictionary" sends (ie, it tries a crapload of whatever@domain.com and not just web-harvested addresses). Doing things "proper" would make it a lot slower for them to send out their mail, so they'd have to be hard pressed to do it.

The obvious fix is of course to ban all SMTP servers that don't require authentication to send, to be very careful about relaying (if I contact the SMTP server at your.domain and say I have a mail for joe@other.domain , your.domain should NOT accept the mail¹). But it would require an internet-wide effort to do that, so it's not going to happen.

In general, imho "client machines" shouldn't make SMTP connections to whatever.domain to deliver mail, they should go through a mailprovider with a trusted.recognized.domain and relay mail through there - all other hosts could then be denied for incoming mail. But this would mean a lot of administrative mess in keeping up with who's big and trusted and recognized, and would also require a helluvalot of servers to be reconfigured. Not going to happen.

1: an exception is of course ISPs and other mail providers, but with authentication in place they won't work as open relays, which is what's dangerous.

[tinjaw]

If you are really interested in server side solutions, you should take a gander at SPF. It is the one that is gaining momentum over Domain Keys, etc. I have an SPF record in my DNS.

[Stoic Joker]

Yes, I guess that if this was very widespread, spammers might implement more of the SMTP protocol and do proper re-send attempts. But it would probably stop the "dictionary" sends (ie, it tries a crapload of whatever@domain.com and not just web-harvested addresses). Doing things "proper" would make it a lot slower for them to send out their mail, so they'd have to be hard pressed to do it.

You're only taking into account (the trojan infected client machine) half of the problem. Hackers don't send spam, marketing companies that want you to buy something do. Remember the TV ads "Make thousands with your home computer" ...really? Doing What? This is precisely how the theBat! (mass) Email client got blacklisted by many spam filters.

All Email servers must accept mail from any server that is destined for their domain to work (the fact that the destination address is internal is the authentication). The (Open Relay) problem stems from servers that will accept (and pass on) mail that is destined for a different domain, without requiring (user/pass) authentication. Those are the ones that are causing the other half of the problem.

The obvious fix is of course to ban all SMTP servers that don't require authentication to send, to be very careful about relaying (if I contact the SMTP server at your.domain and say I have a mail for joe@other.domain , your.domain should NOT accept the mail¹). But it would require an internet-wide effort to do that, so it's not going to happen.

The fact that most small companies can't afford a properly trained Admin are the driving force behind blacklisting IPS's dynamic address ranges, and the many different other DNS based blacklists used by server side spam filters. Every IT newsletter I've seen has been hammering on the Don't let your Email server be  (or get turned into) an open relay for the past few years.

In general, imho "client machines" shouldn't make SMTP connections to whatever.domain to deliver mail, they should go through a mailprovider with a trusted.recognized.domain and relay mail through there - all other hosts could then be denied for incoming mail. But this would mean a lot of administrative mess in keeping up with who's big and trusted and recognized, and would also require a helluvalot of servers to be reconfigured. Not going to happen.

Therein lying the problem ... It's not going to happen quickly. But it needs to happen. If just one guy at each one company would get up of their ass and check their one server it would help.

1: an exception is of course ISPs and other mail providers, but with authentication in place they won't work as open relays, which is what's dangerous.

Any client machine has to logon to the server to send mail, it's part of the basic client configuration. The exception is some of the companys that use (their) IP address ranges as a criteria for send authentication instead of u/p. Earthlink is one example, you can send mail through mail.earthlink.net without a u/p from anywhere inside their network. If you're outside their network (laptop at web café) the mail will be refused.

Even with everything in place and running full tilt, it still only takes one idiot using a blank, simple, or default password to send the whole thing down the drain.

[Carol Haynes]

How would you stop people setting up their own SMTP server (it is easy enough on Windows and Linux to do that)?

The first time we see an IP address/sender/recipient tripple, and the sender/server meets one of the criteria for Greylisting

What about dynamic IPs which almost all ISPs use. If I send emails out I can pretty much guarantee my "IP address/sender/recipient tripple" will change on at least a daily basis (if not every time I reboot my system or my ISP decides to refresh my IP).

[Stoic Joker]

How would you stop people setting up their own SMTP server (it is easy enough on Windows and Linux to do that)?

The first time we see an IP address/sender/recipient tripple, and the sender/server meets one of the criteria for Greylisting

What about dynamic IPs which almost all ISPs use. If I send emails out I can pretty much guarantee my "IP address/sender/recipient tripple" will change on at least a daily basis (if not every time I reboot my system or my ISP decides to refresh my IP).

I wouldn't stop people from setting up their own Email server. However I would set it up for them so it's done correctly...that's just part of what I do for a living.

If you're trying to run a mail server from a dynamic IP you're basically SOL because you'll be identified by most (if not all) spam software as part of the problem children. SPF, Reverse DNS, and MX record validation are just some of the many tests you will fail.

Options are: Get a static IP, it usually only $5 a month. and/or forward (relay...) your out bound mail through a SmartHost which is usually your ISP's mail server so it can validate you for you.

You'll also need to make sure your IP isn't blacklisted coming outa the gate. www.DNSStuff.com has been quite handy for me in the past for resolving mail flow issues.

It's not that it's impossible to setup a proper mail server for yourself, it's just that it's not as simple as click here and follow the prompts. There's a good bit of responsibility involved making sure you don't become part of the problem.

[f0dder]

Imho if you set up your own SMTP server (unless you mean serious business), it should only be for incoming mails, not for sending outgoing - use your ISP or "something big & well-known" for sending.

Problem is even with a static IP, unless you pay for what some ISPs in Denmark call a "global IP", you won't always have control of the IP's reverse-DNS... people & software get suspicious when my.serious.biz resolves to 1.3.3.7, but 1.3.3.7 reverse-dns is something like 0x12345678.slnxx7.adsl-dhcp.tele.dk instead of my.serious.biz.

[tinjaw]

my.serious.biz resolves to 1.3.3.7, but 1.3.3.7 reverse-dns is something like 0x12345678.slnxx7.adsl-dhcp.tele.dk instead of my.serious.biz.

At least with all of the ISP's that I have dealt with here in the US, if I have a static IP address, they will set the reverse lookup to what ever I ask them to.