Emails directed to google-based mail server are disappearing into cyberspace

2

0

The set up

  • Using google suite to manage inbound/outbound emails. All MX records are configured properly for my domain.
  • Using google's web-based email client to send/receive emails
  • We get almost all emails, however, there are a few emails originating from a corporate domains that never arrive. There is no bounceback alerting the sender that email was never delivered and there is no evidence of the email ever arriving (I looked at the inbound mail logging using the admin > reports > audit > email log)
  • I have spf, dkim and dmarc set up and working properly.

I can rule out

  • Definitely not the normal culprits: spam folder, custom filters, etc. because I don't see emails ever arriving in the logs
  • MX Records setup because I am receiving the majority of other emails

It might be

  • DNS propagation issue? We recently switched over to a new DNS. I exported/imported all the DNS records from zoneedit to zeit.co DNS management around the same time that we started noticing this issue. I've since move back to my original zoneedit configuration, but the problem persists.
  • The individual outbound email set up of each of the domains that we cannot receive email from. There's no way to really verify this w/o getting access to their email logs. And trying to get cooperation is unlikely.

Things I've tested

  • client@their_corp_domain.com -> me@my_corp_domain.com - no email, no bounce
  • me@my_corp_domain.com -> client@their_corp_domain.com WORKS! and then when they hit reply and send - no email, no bounce
  • What's even weirder: me@my_corp_domain.com -> client@their_corp_domain.com cc: me@gmail.com, cc: me@hotmail.com WORKS! and when they group reply NOTHING! not even to my gmail or hotmail accounts.

Theory

  • Changing DNS messed stuff up in a way that when the sender's mail system trying to resolve my MX records, they see some inconsistent IP addresses and puke all over themselves. Or, there's some blacklisted IP addresses in the zeit dns managed servers that cause emails to disappear.

I'm basically looking for other ideas that I could try out and test out to help nail down this elusive bug. If you run your own outbound email server and are willing to help me w/ logs, I'd greatly appreciate it.

UPDATES

As per comments, I've done some additional testing:

Bounceback

  • them@their_corp_domain.com -> bogus email: BOUNCEBACK REC'D.

Emailing from other domains to me@my_corp_domain.com

  • me@gmail.com <--> me@my_corp_domain.com: WORKS BOTH WAYS.
  • me@hotmail.com <--> me@my_corp_domain.com: WORKS BOTH WAYS.
  • me@my_other_corp_domain (also google apps managed) <--> me@my_corp_domain.com: WORKS BOTH WAYS. Not certain about it, but I think google-based mail servers are able to see if it's an internal delivery and just route it directly. I've attached screenshot of the routing routing of email

Traceroute from their end

  • No issues here. I think the real interesting question would be what does the traceroute look like from their SMTP server. Each of these corporate domains that cannot send us email outsources their mail handling to 3rd party provider.

  • There are 3 different corp domains that I'm aware of that cannot email us.

Some investigation

I did some digging and found that REDACTED.gslb.pphosted.com and REDACTED.ecicloud.com show up in MX records for 2 of the 3 accounts. The other corp domain uses http://www.abacusgroupllc.com/ for their managed IT.

I tried to do a telnet directly to one of the servers above to spoof an email directly over smtp. It might a red herring, but for rcpt to: <someone@their_corp_domain.com> I got a Sender ok, but for any address outside their domain, i got Relaying denied. This wasn't an exhaustive test by any means.

MX Records TTL

  • Per suggestion in comments, I checked TTL. They were originally at 7200 sec for each MX record. I've now updated all of them to be 3600 sec.

MXToolbox Header Analysis

  • Per suggestion by harrymc, I ran the header analysis and came up with these results: results. Honestly, I'm not super clear as to what I'm looking at in this report, but I can see that SPF, DKIM and DMARC are not configured properly. I'm guessing that google mail server is just rejecting emails coming from this domain as a result.

awei

Posted 2019-09-16T16:34:32.770

Reputation: 21

Could you ask a client from one such a domain to run the usual network tools to verify connectivity to your server (nslookup, tracert etc)? – harrymc – 2019-09-19T08:04:34.370

@harrymc: I've checked the nslookup/tracert, nothing looks odd. I think the real acid test would be to check the nslookup/tracert from their managed IT providers' servers. – awei – 2019-09-20T14:32:04.357

@PimpJuiceIT. Thanks for your many suggestions. I've tried all of them and updated OP w/ results. – awei – 2019-09-20T14:32:48.333

A mail server will produce a "Relaying Denied" error when an unauthorized user attempts to send non-local email. The problem might be a matter of user permissions on their server or in google suite. Some investigation may be called for. – harrymc – 2019-09-20T15:05:16.667

@harrymc. Ok. I thought it was probably something like that when trying to spoof an email on their SMTP server. When you say "user permissions" in "google suite" what are you referring to specifically? Their domains don't use google suite email service. – awei – 2019-09-20T20:06:31.453

Your post included a reference to google suite, which I never used, so cannot say what permissions are involved. My thought is that the problem is bad setup on the client domains rather than a DNS error on your side, since your setup does work for most clients. The answer might be simple, but making the administrators cooperate might not be. – harrymc – 2019-09-20T20:19:09.113

1

Have you checked any of the outgoing SMTP servers of the failing domains for blacklists like at https://mxtoolbox.com/blacklists.aspx ? To do this properly you'd need some successfully received emails from those domains and figure out their SMTP serves from the Received: headers.

– Velo Traveler – 2019-09-21T19:03:00.660

1The normal way to trace outgoing mail problems like this where the recepient has no logs of the mail arriving is to check handoff logs from the source and then at each hop along the way. This means the folks at the failing domains would have to look at their logs to make sure the messages went to their provider, then open tickets with their providers and ask for logs of the lost messages. – Velo Traveler – 2019-09-21T19:08:39.300

@VeloTraveler I checked the 3 known failing domains. Only one of them showed up on a blacklist SORBS SPAM. The others did not show up on any blacklists. As for tracking down the handoff logs it is probably unlikely that I'll get their individual IT departments to cooperate. – awei – 2019-09-23T14:29:28.973

If you have contact with one such client, ask him to execute and report on the tests of MXToolbox Email Deliverability and click "View your full deliverability report" in the received email report, and also dnslytics email-test.

– harrymc – 2019-09-25T08:28:00.033

@harrymc. I did what you suggested. I have posted the image of the result in the update of my original question. I'm not totally sure what I'm looking at, but the report seems to indicate that their mail server is not configured for SPF/DKIM/DMARC and thus google might be rejecting the emails. – awei – 2019-09-26T14:20:40.470

Answers

0

With the information at hand, and since your setup does work for most clients, the conclusion is that the problem is bad setup on the client domains. The probability for a DNS error on your side is very low, since for most clients your servers do work.

It is hard to say what exactly is wrong, but Google servers have become very stringent as regarding emails, because of the amount of false emails being sent every day. Because of these measures, Gmail users in my experience receive much less spam.

The mail servers of the problematic domains have most likely not implemented DMARC (Domain-based Message Authentication, Reporting and Conformance).

The G Suite Admin Help About DMARC has this text:

DMARC actions for failed messages

When an incoming message doesn't pass the DMARC check, the DMARC policy defines what happens to the message. There are three possible actions:

  • Take no action on the message.
  • Mark the message as spam and deliver to recipient's Gmail spam folder.
  • Tell receiving servers to reject the message.

I have no knowledge of administering G Suite, so cannot say where you can define the action to take. Your default is probably to take no action on the message, so the email just evaporates. You could change it so that it leaves some trace for its existence.

The article Gmail settings, section Approved senders without authentication mentions a method for disabling the DMARC test on your servers:

When customizing the Spam setting for your domain in the Google Admin console, you have the option to not require sender authentication.

This might be better that trying to convince the administrators of these failing domains to upgrade their mail servers so they pass the DMARC test, although it would expose you to the usual attacks via email and force you to be extremely careful about clicking links in received emails.

harrymc

Posted 2019-09-16T16:34:32.770

Reputation: 306 093

I haven't accepted this as the correct answer yet because I just made some settings changes that may take a day to propagate. I'll update again when I have verified. In the meantime since the bounty is going to expire in 30 mins and because this is the solution that feels the most correct, I'm going to award the bounty. – awei – 2019-09-26T21:16:46.480

Asked: 2019-09-16T16:34:32.770

Viewed: 149 times

Active: 2019-09-26T16:06:39.887