The question as a whole touches on a few different aspects which all need to be taken into consideration to answer why RFC7505 adds something useful.
First of all, the pre-RFC7505 definition of how mail delivery should be made does not have a way to cleanly indicate that no mail delivery attempts should be made for a name which has address records.
From RFC7505 section 1:
SMTP clients have a prescribed sequence for identifying a server
that accepts email for a domain. Section 5 of [RFC5321] covers
this in detail; in essence, the SMTP client first looks up a DNS MX
RR, and, if that is not found, it falls back to looking up a DNS A
or AAAA RR. Hence, this overloads a DNS record (that has a
different primary mission) with an email service semantic.
If a domain has no MX records, senders will attempt to deliver mail
to the hosts at the addresses in the domain's A or AAAA records. If
there are no SMTP listeners at the A/AAAA addresses, message delivery
will be attempted repeatedly for a long period, typically a week,
before the sending Mail Transfer Agent (MTA) gives up. This will
delay notification to the sender in the case of misdirected mail and
will consume resources at the sender.
This document defines a null MX that will cause all mail delivery
attempts to a domain to fail immediately, without requiring domains
to create SMTP listeners dedicated to preventing delivery attempts.
Then there's the matter of how RFC7505 implements this (IN MX 0 .).
From RFC7505 section 3:
- MX Resource Records Specifying Null MX
To indicate that a domain does not accept email, it advertises a
single MX RR (see Section 3.3.9 of [RFC1035]) with an RDATA section
consisting of preference number 0 and a zero-length label, written in
master files as ".", as the exchange domain, to denote that there
exists no mail exchanger for a domain. Since "." is not a valid host
name, a null MX record cannot be confused with an ordinary MX record.
The use of "." as a pseudo-hostname meaning no service available is
modeled on the SRV RR [RFC2782] where it has a similar meaning.
A domain that advertises a null MX MUST NOT advertise any other MX
RR.
(emphasis added)
As is noted here, the implementation details for the "null MX" are based on an already established pattern from the SRV RR type. It makes sense to mimic this as the SRV RR type more or less is a generalized version of the MX RR type.
So, the decision was essentially taken already when defining the SRV RR type.
So, why not make use of `.invalid`?
From RFC2606 section2:
".invalid" is intended for use in online construction of domain
names that are sure to be invalid and which it is obvious at a
glance are invalid.
As can be seen here, this reserved TLD is for human consumption. There's no precedent of defining special handling of this in software.
Surely it could have been implemented in some different way but they chose to go with the minimal approach of using ., which is not a valid hostname and thus does not interfere with normal use anyway.
