Two servers, same email sent from both, Gmail sees only one as spam

Question

I've been working on this problem for years with no success (I gave up a while back and just hoped Gmail would eventually "learn" that messages from one of my servers wasn't spam, but that apparently never happened).

I'm a game developer who runs forums and download servers for my customers. As part of my operation, I need to sent people emails, often at their request. For example, password-reset emails from my forums. This isn't a "bulk" mailing situation, nor is my server sending out lots of email.

I have two servers, both with the same hosting provider.

One is in a shared hosting environment, where I get a subdir and my domain name is resoloved using virtual hosting. Emails from that server have always been received just fine (though I didn't set up the email system, nor do I have much control over it).

The other is a VPS that I manage. I have my own IP address there, and have full control over everything. From the VPS, I've never been able to send emails that Gmail doesn't see as spam. I've set up spf, dkim, and all tests seem to pass, even according to Gmail when I "view original message". I'm not on any blacklists that I can see. And I've tested it with "personal" messages that don't even contain any links or images. Still spam.

HOWEVER, I just ran a controlled test (same message sent from webmail on each of these servers), and the headers on the non-spam server are substantially different. It looks like the mail is being sent through a mail relay. Maybe that helps somehow (if the mail relay is trusted, I guess?) Here's the raw message that Gmail sees as non-spam:

Received-SPF: pass (google.com: domain of jason@northcountrynotes.org designates 23.83.209.46 as permitted sender) client-ip=23.83.209.46;
Message-ID: <a6c63634f6f368814f401e02ef2df1fb.squirrel@northcountrynotes.org>
Date: Wed, 15 Feb 2017 01:46:30 -0500
Subject: Hello from an old friend
From: jason@northcountrynotes.org
To: jasonrohrer@gmail.com
User-Agent: SquirrelMail/1.5.2 [SVN]
MIME-Version: 1.0
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: 8bit

Hey Jason, my old friend.  Remember when we went to school together so
long ago at Cornell.  Boy!  Those were the days.  Hit me back if you want
to chat.

Justin J.

And here's the raw message from the VPS server, from which Gmail always puts messages in the spam folder:

Delivered-To: jasonrohrer@gmail.com
Received: by 10.182.233.195 with SMTP id ty3csp516102obc;
        Tue, 14 Feb 2017 22:33:25 -0800 (PST)
X-Received: by 10.36.198.133 with SMTP id j127mr7649805itg.72.1487140405772;
        Tue, 14 Feb 2017 22:33:25 -0800 (PST)
Return-Path: <jason@thecastledoctrine.net>
Received: from server.thecastledoctrine.net (server.thecastledoctrine.net. [108.165.22.58])
        by mx.google.com with ESMTPS id 127si5199936itx.63.2017.02.14.22.33.25
        for <jasonrohrer@gmail.com>
        (version=TLS1 cipher=AES128-SHA bits=128/128);
        Tue, 14 Feb 2017 22:33:25 -0800 (PST)
Received-SPF: pass (google.com: domain of jason@thecastledoctrine.net designates 108.165.22.58 as permitted sender) client-ip=108.165.22.58;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@thecastledoctrine.net;
       spf=pass (google.com: domain of jason@thecastledoctrine.net designates 108.165.22.58 as permitted sender) smtp.mailfrom=jason@thecastledoctrine.net
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thecastledoctrine.net; s=default; h=Content-Transfer-Encoding:Content-Type:MIME-Version:To:From:Subject:Date:Message-ID; bh=7hycuruqtuxJJjoFV/Gyd/n8jFs9UDvFVpJOWI96Abw=; b=yPnvk3ZVtOiXNFo9+zRP0mEdKWoahLgA4Au/GeUrmKC6uJiB9A6VJyEudRHdhidBiOalf30exm6gk+7+To0y1XEMwGzHpch8I32RY85qrJmzptX8mrqBBRC9Q51hfWCTcgmSXwS1YsiIgJYu96GSw+PZ2lgPAfOkZaR+i6lrIKs=;
Received: from localhost ([127.0.0.1]:54405 helo=thecastledoctrine.net) by server.thecastledoctrine.net with esmtpa (Exim 4.85) (envelope-from <jason@thecastledoctrine.net>) id 1cdt9X-0001bg-M5 for jasonrohrer@gmail.com; Wed, 15 Feb 2017 06:33:23 +0000
Received: from 50.1.19.178 ([50.1.19.178])
        (SquirrelMail authenticated user jason@thecastledoctrine.net)
        by thecastledoctrine.net with HTTP;
        Wed, 15 Feb 2017 06:33:23 -0000
Message-ID: <bb9ed865cadaedf1c99097d576ae7ac9.squirrel@thecastledoctrine.net>
Date: Wed, 15 Feb 2017 06:33:23 -0000
Subject: Hello from an old friend
From: jason@thecastledoctrine.net
To: jasonrohrer@gmail.com
User-Agent: SquirrelMail/1.5.2 [SVN]
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.thecastledoctrine.net
X-AntiAbuse: Original Domain - gmail.com
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - thecastledoctrine.net
X-Get-Message-Sender-Via: server.thecastledoctrine.net: authenticated_id: jason@thecastledoctrine.net

Hey Jason, my old friend.  Remember when we went to school together so
long ago at Cornell.  Boy!  Those were the days.  Hit me back if you want
to chat.

Justin J.

I'm guessing that I'm missing something simple here.

Update:

Here's another message sent from the VPS that is marked as spam. This is a message generated by one of my PHP scripts (a download server, sending someone the download key that they paid for). It contains links, but when an identical message is sent from the shared server, it is not spam.

Delivered-To: jasonrohrer@gmail.com
Received: by 10.182.233.195 with SMTP id ty3csp529399obc;
        Tue, 14 Feb 2017 23:55:38 -0800 (PST)
X-Received: by 10.36.161.70 with SMTP id n6mr7354632iti.76.1487145338304;
        Tue, 14 Feb 2017 23:55:38 -0800 (PST)
Return-Path: <jason@thecastledoctrine.net>
Received: from server.thecastledoctrine.net (server.thecastledoctrine.net. [108.165.22.58])
        by mx.google.com with ESMTPS id w204si3392935itc.67.2017.02.14.23.55.37
        for <jasonrohrer@gmail.com>
        (version=TLS1 cipher=AES128-SHA bits=128/128);
        Tue, 14 Feb 2017 23:55:38 -0800 (PST)
Received-SPF: pass (google.com: domain of jason@thecastledoctrine.net designates 108.165.22.58 as permitted sender) client-ip=108.165.22.58;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@thecastledoctrine.net;
       spf=pass (google.com: domain of jason@thecastledoctrine.net designates 108.165.22.58 as permitted sender) smtp.mailfrom=jason@thecastledoctrine.net
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=thecastledoctrine.net; s=default; h=Message-Id:Date:Subject:To:From; bh=GFDUYdq+5oW7xSkEaXekddRXEY2zhMAKlOU/+v0g27c=; b=s3kENICTVFbW8saQ0ya7bh1V+XIb9a4NCQlUghvQHa/52Rw40V3e6a33jRgJCJx2v9kc4NhLDjUM6Q/upbLh6kRGWwTCotG7bIrLlhlKGnLgNaraffq2kVuNkTYi+auts7QzofJBs3Gw1ctvYJf7LH3fxVgiEqUK06P2Q0Dnim4=;
Received: from [108.165.22.59] (port=39156 helo=localhost) by server.thecastledoctrine.net with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.85) (envelope-from <jason@thecastledoctrine.net>) id 1cduR6-0000KN-OG for jasonrohrer@gmail.com; Wed, 15 Feb 2017 07:55:36 +0000
From: Jason Rohrer <jason@thecastledoctrine.net>
To: jasonrohrer@gmail.com
Subject: Your [The Castle Doctrine] download is ready
Date: Wed, 15 Feb 2017 02:55:35 -0500
Message-Id: <58a40977f2d47@thecastledoctrine.net>
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.thecastledoctrine.net
X-AntiAbuse: Original Domain - gmail.com
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - thecastledoctrine.net
X-Get-Message-Sender-Via: server.thecastledoctrine.net: authenticated_id: jason@thecastledoctrine.net

Jason Rohrer Gmail:

Keep your ticket ID secure, because it is your sole way of accessing your account on the game server.

Your can now access your download at:

  http://thecastledoctrine.net/ts/server.php?action=show_downloads&ticket_id=8G5QX

You can also access your download manually by entering your ticket 8G5QX here:

  http://thecastledoctrine.net

And no, this is NOT the same question. Here I have two example non-bulk emails, one delivered, and one not. The usual answers are of no use, because I'm already doing all of that stuff.

Yes. You can see the spf=pass status in both headers. I don't think SPF is the issue here, but I might be missing something. — Jason Rohrer, Feb 15 '17 at 07:14
These messages are dramatically different. The one from your VPS has a bunch of cPanel-like headers which are unfortunately very commonly seen in rejected mail (as quite a bit of spam originates from compromised web apps on cPanel systems). The usual advice you'll get here is to get rid of cPanel. — Michael Hampton, Feb 15 '17 at 07:30
Well, unfortunately, I don't have an "I know it when I see it" sensibility about mail headers. Can you help me figure out which are cpanel-like? Also, BOTH servers are running cPanel. Both email accounts were created through cPanel, and the emails were sent through SquirrelMail, which was launched from inside cPanel. I'll post another example of a spam message that one of my PHP scripts sent outside of cPanel. I assume you're not saying that cPanel generating the messages is the problem, but that it somehow is adding extra mail headers just by being installed. — Jason Rohrer, Feb 15 '17 at 07:54
Possible duplicate of [How to send emails and avoid them being classified as spam?](http://serverfault.com/questions/48428/how-to-send-emails-and-avoid-them-being-classified-as-spam) — MadHatter, Feb 15 '17 at 08:06
I have no idea what ASO has done to cPanel. We don't support shared web hosting here either. — Michael Hampton, Feb 15 '17 at 08:11
Well, I'm not having a problem with my shared web hosting, as I mentioned. My VPS, which I have full control over (I can install any OS on it, currently running CentOS 5) is where the problem lies. I'm assuming that my hosting provider has carefully configured their shared hosting server and relay to avoid spam issues. I've been trying to do the same thing on my VPS for years with no luck. I've read every article I can find and jumped through all the various hoops. And I'm NOT sending something that "looks like a mailing list." Even personal emails (shown above) are blocked. — Jason Rohrer, Feb 15 '17 at 15:45
The biggest difference I see here is that the shared host is using a relay service, whereas my VPS is not. http://www.mailchannels.com claims to ensure emails don't end up in spam folders, etc. etc. I'm assuming that these companies forge relationships with Gmail---"trust us, we never send spam." SO... perhaps the answer is that you can no longer run your own email server in such a spam-ridden world. You're outside the circle of trust. Can someone confirm this? That I should just bite the bullet and pay for an SMTP service? — Jason Rohrer, Feb 15 '17 at 15:52

score 2 · Answer 1 · answered Feb 16 '17 at 05:12

Here's how I solved this:

I got myself whitelisted.

I purchased a $10/month account with SendGrid which allows me to send up to 40,000 emails per month, with no daily limits.

When I switched to their smtp server instead of my own, all the same emails started going to the GMail inbox instead of the spam folder.

mailchannels.com, which is used by my shared hosting provider, is a similar service to SendGrid, but meant for much larger organizations with multiple, untrusted users (like web hosting companies who want to give reliable email sending capabilities to their clients).

Anyway, these days, it seems like just NOT being on a blacklist, and having all the right email/DNS verfications (SPF, etc) is not enough.

Gmail and other email services are apparently depending on whitelists more than anything else. I'm guessing that SendGrid, MailGun, PostMark, etc., have special agreements, or maybe even paid contracts, with Google, to help them deliver all email. These services also heavily police any spam coming from their own clients---their reputation depends on it.

And this is an answer that I've never seen posted anywhere, as much as SPF and DKIM have been discussed in trying to help small domain owners. This is a perennial question after all (I've tried everything, but emails from my domain are still not inboxed).

SPF and DKIM didn't help to stop spam, so they're pretty much useless. Gmail doesn't even care about them anymore, from my experience. Well, except to prove what they do care about:

All that matters is WHERE your email is coming from, and whether that source is inside the circle of trust (and SPF is needed for that).

If you're some dude who put together his own VPS and has his own domain name, like me, you are outside the circle of trust. And why wouldn't you be outside? ANYONE can set up a domain on a cheap VPS and configure SPF and DKIM for that domain. Even a spammer! I mean, I could get a spamming VPS up and running tonight if I wanted to.

The only way to get IN the circle, assuming that you're not so big and well known that everyone will naturally trust you on their own, is to contract with someone who's already in the circle.

Like paying a certificate authority to bring you inside the encryption circle of trust.

Otherwise, you're kinda just vouching for yourself.

As the spam problem has continued to grow, vouching for yourself with SPF and DKIM isn't good enough anymore.

Which has sadly made a very lucrative business for these email relay services.

I tried to resist this conclusion, recalling the glory days of mom-and-pop mail servers running in broom closets... but it seems like those days are over, and I had to give up the dream of email independence.

Two servers, same email sent from both, Gmail sees only one as spam

1 Answers1