The Hostile Email Landscape

[mouser]

There was an interesting post on slashdot today about how we've entered an age where only the big players in email domain hosting can reliably expect their emails to be received and not discarded as untrustworthy.

While it's not *yet* impossible to just run a mail server from your local domain, it is becoming increasingly likely that if you do, some recipients will have your mail automatically filed in their spam folders or outright rejected.

To me this is just another consequence of the big players slowly exerting their dominance over the rest of us.. there is just no incentive for them not to do this, and every incentive for them to make it harder and harder for small players to coexist with them.

As we consolidate on just a few major email services, it becomes more and more difficult to launch your own mail server. From the article: "Email perfectly embodies the spirit of the internet: independent mail hosts exchanging messages, no host more or less important than any other. Joining the network is as easy as installing Sendmail and slapping on an MX record. At least, that used to be the case. If you were to launch a new mail server right now, many networks would simply refuse to speak to you. The problem: reputation. ... Earlier this year I moved my personal email from Google Apps to a self-hosted server, with hopes of launching a paid mail service à la Fastmail on the same infrastructure. ... I had no issues sending to other servers running Postfix or Exim; SpamAssassin happily gave me a 0.0 score, but most big services and corporate mail servers were rejecting my mail, or flagging it as spam: Outlook.com accepted my email, but discarded it. GMail flagged me as spam. MimeCast put my mail into a perpetual greylist. Corporate networks using Microsoft's Online Exchange Protection bounced my mail."

http://tech.slashdot.org/story/15/10/19/1434236/the-hostile-email-landscape

[ayryq]

My main email address is hosted on a shared server at Bluehost. Not only is my  email occasionally marked as spam but the shared IP address is often blacklisted. It's very frustrating when you send a cold email and no reply is received... you don't know if the person just didn't respond or if your email just never got through.

[Stoic Joker]

MimeCast put my mail into a perpetual greylist.

Perpetual Greylist?? Greylisting just means to refuse everything on first contact to see if the try resending, because supposedly many(/most?) of the naughty mail servers don't do resends...to speed up Directory Harvesting I suppose..

Honestly it sounds like they got a semi blacklisted IP address from their ISP. I had that happen here when we switched ISP. Our mail servers previously pristine reputation (by IP) went out the window the instant I put it behind the new guilted by association because it was in an ISP customer address block IP address. MX-Toolbox said the address was "clean" but we were still getting rejections from (see above... ) other mail systems. After much - panicked 3am class digging - I found the system that had us listed as guilty by association and forwarded that info to the ISP who swore (front line support) that the address wasn't blacklisted. This resulted in me swearing, mild hostility (3am...), the finding of a supervisor...who dug up an engineer to "prove me wrong".

Thankfully the engineer was not nearly as retarded as the rest of the staff...and therefore was able to grasp the gravity of the situation, actually read the information I'd sent, and then contacted the listing company to have the address (actually a CIDR block of 16 addresses) removed from the naughty.


And the mail began to flow...

[skwire]

I've run my own mail server from home for about fifteen years now and came across this type of issue a couple years ago.  What I had to do was ask my ISP to create a reverse PTR record so that the reverse lookup of my IP address pointed at my mail domain instead of their pool name.  This solved the majority of the issues I was experiencing and I also thanked my lucky stars that my ISP was cool enough to allow that on a home connection.

[BGM]

I have hmailserver installed and it works VERY well for a full-fledged mail server.  But I discovered that my messages were being bounced by the other recipient domains because I did not have reverse DNS configured for my domain.  That was an issued to be solved with my ISP.  We ended up using a Google Apps domain anyway so the problem became moot.

Oh, I just noticed that Skwire said the same thing...  hahahhaa

[Renegade]

In order to remain civil, I'll keep my comments short and focused on a very tiny little bit...

Gmail filters emails from Microsoft as spam on a regular basis.

Do I need to say anything more?

Please run through every curse word you've ever heard, and then amplify that by several orders of magnitude, because my tidbit there is just the very tip of the top of the highest snowflake on the iceberg.

There is a Satan, he is very real, and no depth of evil you can possibly imagine can compare to him. Email is proof.

[Stoic Joker]

I've run my own mail server from home for about fifteen years now and came across this type of issue a couple years ago.  What I had to do was ask my ISP to create a reverse PTR record so that the reverse lookup of my IP address pointed at my mail domain instead of their pool name.  This solved the majority of the issues I was experiencing and I also thanked my lucky stars that my ISP was cool enough to allow that on a home connection.

I have hmailserver installed and it works VERY well for a full-fledged mail server.  But I discovered that my messages were being bounced by the other recipient domains because I did not have reverse DNS configured for my domain.  That was an issued to be solved with my ISP.  We ended up using a Google Apps domain anyway so the problem became moot.

Oh, I just noticed that Skwire said the same thing...  hahahhaa

Well understood guys, but I'd already had the RDNS record created days in advance of the switchover. I'd also found the blacklist companies info in an NDR after sending a flurry of test messages to various systems (with permission...) to try and gauge to scale of the issue. So after much battling with frontline TS, I finally got ahold of someone that could actually comprehend the information I was putting in front of them and we had it resolved in a matter of hours.


@BGM - +1 for hMailServer. After years of running a basic IIS POP/SMTP server in my home lab. Microsoft's decision to remove the POP server from Windows servers forced me to shop for other options. Since I didn't have the hardware, or a need for running a full blown Exchange server. I switched over to hMailServer last winter and have been quite happy with it.

[JavaJones]

To me this is just another consequence of the big players slowly exerting their dominance over the rest of us

How, exactly, is that the case? People don't like spam and detecting it automatically is non-trivial. Reputation-based blocking is not an unreasonable approach to the problem. It definitely has flaws, but it doesn't seem like a general anti-competitive conspiracy to me, more a consequence of a messy technological landscape in the email and security domain. It requires crude solutions because better ones (with equal efficacy) aren't readily available.

As others state in this thread, a lot of this comes down to just doing the right things (e.g. RDNS records, PTR), which the average home user just isn't aware of or equipped to setup. Again does that indicate some kind of big business conspiracy, or is it just the realities of the imperfect technological solutions that have been put in place to try to combat the very real issue of spam, that is out of all of our control?

Perhaps you disagree with the solutions we have, reputation-based, PTR, etc. Do you have better ideas that are easier to implement but not any easier to abuse?

- Oshyan

[Renegade]

How, exactly, is that the case? People don't like spam and detecting it automatically is non-trivial. Reputation-based blocking is not an unreasonable approach to the problem. It definitely has flaws, but it doesn't seem like a general anti-competitive conspiracy to me, more a consequence of a messy technological landscape in the email and security domain. It requires crude solutions because better ones (with equal efficacy) aren't readily available.

Am I the only one who always sees all email from Microsoft go directly to the spam folder of my Gmail interface?

[JavaJones]

How, exactly, is that the case? People don't like spam and detecting it automatically is non-trivial. Reputation-based blocking is not an unreasonable approach to the problem. It definitely has flaws, but it doesn't seem like a general anti-competitive conspiracy to me, more a consequence of a messy technological landscape in the email and security domain. It requires crude solutions because better ones (with equal efficacy) aren't readily available.

Am I the only one who always sees all email from Microsoft go directly to the spam folder of my Gmail interface?

Perhaps not the only one, but I certainly don't have that problem. Do you actively mark them as not spam, and if prompted, tell Gmail to not filter similar messages to spam in the future? Have you ever sent in any such emails to the spam team if/when prompted to help improve their filtering? (I get this message every once in a while when marking something as not spam in the spam folder)

- Oshyan

[Deozaan]

Am I the only one who always sees all email from Microsoft go directly to the spam folder of my Gmail interface?

Microsoft emails make it to my inbox in Gmail. And I have no special filters set up for them.

[Renegade]

I've flagged MS emails as not spam, but they still end up there. Not sure why. I figured it was just Google being malicious as I can't imagine them not being able to figure out that MS isn't a spammer.

[1NR1]

Hello,

OK so if you don't have the ability to run your own server, or weed through the problems of the BigCorporateFreeMailSystem, what's the suggestion for the rest of us to be as "safe" and "complete" as possible?  Thanks

[xtabber]

OK so if you don't have the ability to run your own server, or weed through the problems of the BigCorporateFreeMailSystem, what's the suggestion for the rest of us to be as "safe" and "complete" as possible?  Thanks

For starters, don't use Yahoo!