Can sendmail forward e-mail immediately instead of queueing?

Question

Some of the e-mail passing through my server is forwarded to external accounts.

Unfortunately, my upstream SMTP-server is very picky about spam -- and rejects some of the legitimate messages as such. When this happens to the forwarded mail, I get the bounces (as the postmaster) -- not the originators.

I understand, that this is because sendmail queues the messages locally, disconnects from the relay, and only then proceeds to forward them further. If the further forwarding breaks for any reason -- such as because the next relay misidentifies the message as spam -- my sendmail is left to hold the pieces.

Can things be configured so that the forwarding begins immediately instead (as soon as the forwarding destination is determined)? The status -- success or failure -- can then be communicated directly to the previous relay still on the line...

If sendmail can not do it, can any other MTAs? Thanks!

score 4 · Accepted Answer · edited Oct 07 '21 at 07:34

No, it's not possible as it's not implemented with any wide spread SMTP software; you would have to program your own SMTP server that supports this kind of behavior, which would be out of scope on Serverfault. In this answer I explain, why all MTAs have implemented the SMTP protocol very similarly, using queue, and how that is the best way to accomplish all requirements of the protocol.

A mail transport agent MTA always either denies a message or accepts and queues it, based on its own settings. Then, it's relayed or delivered from the queue.

That's because

there can be both permanent and temporary errors. If the MTA can't connect the nexthop immediate, it'll try again later and bounces only if the delay reaches the limit set. Neither can it wait for another MTA to respond before closing the connection, as it may have other messages to deliver first.
there can be several recipients. While a client can simply list all recipients at once with RCPT TO commands, the message can be finally delivered to several other servers, of which some can be available now and some later. Furthermore, the MTA can't open all these connections at once during the initial connection and wait for their responses. There's no practical reason to have totally different workflow for messages with a single recipient.
it should always be clear which MTA currently has the responsibility for delivering the message. (This has been explained by examples in MadHatter's answer.)

That's just how SMTP was designed. Rather than syntactical requirement for the connection commands this leads to very similar architectures; Sendmail, Postfix and even MS Exchange has separate components for sending and receiving mail.

The SMTP server component receives mail and adds it to the queue.
Then, separate SMTP client tries to send it further to other MTAs, or if a recipient is local, the message can be saved to a file or passed to a mail delivery agent MDA, e.g. Procmail.

The requirement still comes from the SMTP specification; RFC 5321 2.1 on SMTP model basic structure:

Fully-capable SMTP implementations, including the relays used by these less capable ones, and their destinations, are expected to support all of the queuing, retrying, and alternate address functions discussed in this specification. In many situations and configurations, the less-capable clients discussed above SHOULD be using the message submission protocol (RFC 4409) rather than SMTP.

And a bit further:

In other words, message transfer can occur in a single connection between the original SMTP-sender and the final SMTP-recipient, or can occur in a series of hops through intermediary systems. In either case, once the server has issued a success response at the end of the mail data, a formal handoff of responsibility for the message occurs: the protocol requires that a server MUST accept responsibility for either delivering the message or properly reporting the failure to do so (see Sections 6.1, 6.2, and 7.8).

Whatever the error with the next hop, the MTA could still echo it back to the previous hop. I struggle to understand, which part of the SMTP specification _mandates_ the "always queue first" behavior... — Mikhail T., Aug 08 '17 at 00:53
Sorry, it does not. My server can still pass the next server's verdict (be it 250 or some error) verbatim without breaking any part of the protocol -- acting more like a smart SMTP-_proxy_, perhaps, but still within the specification... — Mikhail T., Aug 09 '17 at 06:33

score 3 · Answer 2 · answered Aug 08 '17 at 08:28

I think Esa's answer is excellent, but I'm going to disagree with some of it. I think that what you want is possible, but it's a bad idea, and it won't help you. As he says, RFC5321 s2.1 notes that

once the server has issued a success response at the end of the mail data, a formal handoff of responsibility for the message occurs: the protocol requires that a server MUST accept responsibility for either delivering the message or properly reporting the failure to do so

In the case that server B receives a message from server A and delivers it on to server C, I don't see that this bars B from waiting to confirm receipt to A until it has had confirmation of receipt from C - which is what you're asking for. But the problem is that between two servers 250 OK is atomic (either the receiver has accepted it, and so is responsible for delivery, or it hasn't, and so the sender remains responsible), while between three it isn't.

Consider the case where client A is unintentionally disconnected after it has sent the mail but before it's had the 250 OK, while B is delivering it on to C. C then confirms receipt from B with its 250 OK, so B knows that C has it. But A doesn't, so A must still be responsible, and must continue to redeliver to B. If there is some systematic communications issue between A and B (eg, one of those lovely firewalls that think it's their job to mess with the contents of SMTP conversations), this could result in a very large number of copies of the same message being delivered.

Moreover, sendmail already does what you think it doesn't: in the event of failed delivery to C, it will try to pass the buck back to A. This generally only fails where A is malicious, and either lies in the envelope-From (to whom such notification should be made) or doesn't run a mail server at all. In such cases the mail must be delivered to B's postmaster, for B is responsible for delivery (having said 250 OK to A, but not had it from C), cannot deliver to C (it's tried and got a 5xx permanent failure) and cannot hand back to A (because A has made this impossible) so has no-one else to give it to. Here's an example I got this morning:

Date: Tue, 8 Aug 2017 02:53:55 +0100
From: Mail Delivery Subsystem <MAILER-DAEMON@lory.teaparty.net>
To: mailman-bounces@teaparty.net
Subject: Returned mail: see transcript for details
Parts/Attachments:
   1   Shown     17 lines  Text
   2   Shown    407 bytes  Message, "Delivery Status"
   3   Shown     15 KB     Message
   3.1           10 KB     Application
----------------------------------------

The original message was received at Tue, 8 Aug 2017 02:53:53 +0100
from localhost [IPv6:::1]

   ----- The following addresses had permanent fatal errors -----
<redacted@googlemail.com>
    (reason: 550-5.7.1 Unauthenticated email from aol.com is not accepted due to domain's)

   ----- Transcript of session follows -----
... while talking to gmail-smtp-in.l.google.com.:
>>> DATA
<<< 550-5.7.1 Unauthenticated email from aol.com is not accepted due to domain's
<<< 550-5.7.1 DMARC policy. Please contact the administrator of aol.com domain if
<<< 550-5.7.1 this was a legitimate mail. Please visit
<<< 550-5.7.1  https://support.google.com/mail/answer/2451690 to learn about the
<<< 550 5.7.1 DMARC initiative. 53si155585wrc.260 - gsmtp
554 5.0.0 Service unavailable

    [ Part 2: "Delivery Status" ]

Reporting-MTA: dns; lory.teaparty.net
Received-From-MTA: DNS; localhost
Arrival-Date: Tue, 8 Aug 2017 02:53:53 +0100

Final-Recipient: RFC822; redacted@googlemail.com
Action: failed
Status: 5.7.1
Remote-MTA: DNS; gmail-smtp-in.l.google.com
Diagnostic-Code: SMTP; 550-5.7.1 Unauthenticated email from aol.com is not accepted due to domain's
Last-Attempt-Date: Tue, 8 Aug 2017 02:53:55 +0100

    [ Part 3: "Included Message" ]

Date: Tue, 08 Aug 2017 01:53:46 -0000
From: rmusic80proof@aol.com
To: redacted@stphilipschurch.org.uk
[...]

Note how the original email purports to come from an aol.com address. How, then, am I not trying to deliver the failure report back to them? Because they lied in their original SMTP transaction:

Aug  8 02:53:51 lory sendmail[9457]: v781rmjA009457: from=<rmusic80proof@stphilipschurch.org.uk>, size=14095, class=0, nrcpts=1, msgid=<150215722683.22237.12283849532100059916@37.114.157.178>, proto=SMTP, daemon=MTA-v6, relay=[37.114.157.178]

It's my fault that I haven't yet set up SPF for that particular domain of mine (stphilipschurch.org.uk), but since I haven't, nothing stops me from accepting that lie - and then I'm stuck with an undeliverable message which cannot be returned to sender, for the sender is malicious and uninterested (being connected via an ISP in Azerbaijan).

tl; dr: sendmail already does what you're asking for, when it can. What you want to do won't help it when it can't, and will create problems. Don't do it.

That's an excellent clarification. I don't see how this disagrees with my answer, it just manages to explain some parts better and in more detail. — Esa Jokinen, Aug 08 '17 at 11:35
@EsaJokinen well, you say it's not possible; I say it is, but it's a dam' silly idea. But as long as you think this a useful addition to what you wrote, I'm delighted! — MadHatter, Aug 08 '17 at 11:41
I'm just a little more strict towards the situation "_where client A is unintentionally disconnected after it has sent the mail but before it's had the `250 OK`, while B is delivering it on to C_". I'd state that just because the mail might unintentionally get delivered twice, it would be against SMTP specification not to have unambiguous understanding on who is responsible for delivering it. That's why having a queue is indirectly mandatory architecture. — Esa Jokinen, Aug 08 '17 at 11:48
@EsaJokinen a very reasonable inference, but an inference nevertheless. The RFC *does* forbid a responsible server from dropping mail on the floor, but it doesn't to my knowledge forbid a server from delivering something twice. I stipulate that would be unhelpful and irresponsible, but I don't read it as forbidden. — MadHatter, Aug 08 '17 at 14:41
Good point. After all, a recipient may end up having the same message twice for other reasons, for example if two addresses are forwarded to same mailbox. Then it's eventually up to MDA to decide what to do with these duplicates, based on ID when `Received:` path is different. This discussion has been delightful. — Esa Jokinen, Aug 08 '17 at 15:05
What if there is no forwarding at all, I send the 250 to the sending server, but it does not receive it due to a network mishap? It would cause the same problem (duplication), which does not seem to bother anyone... — Mikhail T., Aug 09 '17 at 06:27
As Esa noted in his comment above, there are paths by which a duplicate can be received already; that doesn't excuse creating more. Let's be clear, no-one is stopping you writing an MTA that does what you describe. It's that just that no-one (including sendmail) is prepared to go out of their way to help you do it, because it's not a good idea. — MadHatter, Aug 09 '17 at 11:45
This is not "more" -- this is the exact same scenario: the sender does not get the receiver's confirmation due to a network disconnect causing a message-duplication. What it is the receiver is/was doing with the message during the transition (saving into a queue or forwarding upstream) seems irrelevant. I fail to understand the reasons for your irritation... _no-one is stopping you writing an MTA ..._ -- thanks, but @EsaJokinen's and your answers both state, that such implementation would violate SMTP. It would help, if you explicitly acknowledge, that's not true... Thank you! — Mikhail T., Aug 09 '17 at 12:49
We both explained several reasons why **no-one** has implemented this in wide spread SMTP servers. Be welcome to program your own. I have actually once programmed an SMTP relay that violates the specification even more directly, for a special internal use where it solved a temporary problem. ;) — Esa Jokinen, Aug 09 '17 at 12:54
No, Esa, your answer _begins with_: "It is not possible". Not merely "not done" (for a reason you deem good), but not possible (because SMTP). Since we seem to have established by now, that's not actually true -- that such an MTA could be implemented -- you should probably correct it... — Mikhail T., Aug 09 '17 at 12:56
@MikhailT where do I say that such a thing would violate the SMTP RFC? — MadHatter, Aug 10 '17 at 06:31
I've added an explanation why "it is not possible" is the correct answer in the scope of Serverfault; this is about configuring existing solutions, not programming new. However, it's not completely against RFC, like MadHatter explains. Now our answers complement each other. — Esa Jokinen, Aug 10 '17 at 09:51

Can sendmail forward e-mail immediately instead of queueing?

2 Answers2