Do mail servers follow links in emails as part of a security scan before inbox delivery?

Question

I've implemented a passwordless login using a magic link and email. The link can be used only once. One customer is complaining that once they click the link, the page reports that the link is already used. This is indeed what I implemented, but I'm convinced they actually click each link just once. I was in direct contact with the user.

We cannot reproduce this behavior (it works on my computer), so I checked the logs. From what I gathered from the logs, I suspect that some automated mechanism is clicking the login-link before delivering the email to the inbox. Once it reaches the inbox and the user clicks on the link, they're actually the second one clicking it. Some automated tool did it first, then the user does it second.

Here's what raised my suspicion in descending order of contribution:

1. there's an average of 2 seconds between submitting the email address and clicking the link in the email. Albeit not undoable, it takes some unlikely amount of effort to accomplish that by hand.
2. the email address was submitted from a completely different IP address from where the link was clicked. Geo-IP tells me that requests are all made in London, but the link is clicked in the US. Same user, next attempt 5 minutes later request from London, but this time clicked in Germany.
3. the email address was submitted from a different browser than the browser used to open the link (Chrome vs IE11).

Do mail servers actually follow links in emails as part of a security scan before inbox delivery? If so, what's a recommendable way to make one-time links work when used with such environments?

PS1: This customer works for a large multinational company. PS2: I'm aware of the pros and cons of passwordless logins.

Answer 1

This is not a new problem.

I haven't had to work around it in a few years, so I don't know if the state of the art has changed, but there are a few things you can try:

1) don't process your link unless it has a special query param in it. The query param is added by a client-side redirect (I used http://insider.zone/tools/client-side-url-redirect-generator/ to handle some of the cross-browser messiness) but the server-side code would return this with the query-paramless link to the browser. So:

user clicks 'https://magiclink.foo.org/ajskdfjwlakefj'

server does not see magic redirect, so

redirects to 'https://magiclink.foo.org/ajskdfjwlakefj?follow=true'

and server-side code, seeing "follow=true", would process and then invalidate that token.

2) I think you could do the same with a redirect from GET->POST with a form submit, but I haven't tried.

Good luck!

Answer 2

Easy enough to put a CAPTCHA on the one-time link to confirm the visitor is human before using the link. So, the mail filter will still visit the page but won't trigger the action expected by simply visiting the link (likely a cURL request instead of a browser with DOM/JS)

Answer 3

Depends on the functionality of the MTA, if the MTA have phishing detection capabilities (basically parse the message and extract URLs and check with a database) depends on how they make the detection for determine if is phishing for example. From my experience in general MTAs don't follow links or downloads the attachments that are on the messages, but probably some systems with advance capabilities could download and analyse attachments before the message is deliver to the inbox.

Answer 4

Obviously I don’t know your customer’s mail provider, but some service providers offer features specifically for this purpose — performing a security inspection of a message before delivery. Some, to another comment on this thread, go as far as inspecting attachments.

Office 365 Advanced Threat Protection (ATP), here’s a blog that describes it. Office 365 ATP