I am not a cryptographer. Maybe that is why I don't see the issues with integrating PGP into SMTP.
In my head: Lea requests the server of Luke's domain jedi.com to tell her the public key of luke@jedi.com (The request includes an encryption method perhaps). She gets back the key PUBLIC. Then she encrypts the message and Luke can decrypt it easily.
It's not that hard, why isn't it standard for years?
It's not that hard, why isn't it standard for years?
Because that would not have solved the problem that PGP is trying to solve.
PGP is an end to end encryption, so if there is any way for the SMTP server to subvert the encryption, then the scheme fails.
In the case of the scheme you proposed, suppose Alice (alice@charlie.com) wants to send a private message to Bob (bob@dave.com). Using the scheme you proposed, Alice's mail client or Alice's SMTP server fetches Bob's public key by making TLS connection to dave.com. This is fine as long as dave.com are honest and actually return Bob's public key. But dave.com could have been configured by dave.com's operator to return a forged public key created by Eve, or Eve could hack into dave.com and set this up. Now Alice's mail client/mail server would happily accept Eve's certificate, thinking the public key is Bob's. In this model, the operator of dave.com can intercept any of Bob's emails.
Now, as long as dave.com is honest, this still protects against third party spoofing. Why don't we do this anyway if this protects at least against third party snooping? Mainly because SMTPS also provides the same level of protection, while also being much simpler. If MITM by the mail server operator is not your concern, you can already very well secure your emails by ensuring you both uses SMTPS.
Note that the difficulty of end to end encryption isn't about fetching public keys. Most email client that supports PGP also supports automatically fetching public keys from LDAP or HPKP. The difficulty of end to end encryption is verifying the public keys.
There is no known method of verifying public keys that is fully transparent to the users and fully secure. Web of Trust or Certificate Authority model comes closest, but Web of Trust comes with a lot of caveats and Certificate Authority model relies on a third party to do the verification.
integrating PGP into SMTP.
PGP is a container format for data (like mails but not restricted to mails), which adds encryption and/or signature to the data. SMTP is a transport protocol.
You don't integrate container formats into transport protocols. This would be the same as saying that you should integrate Office (container for text, images...) with SMTP (transport) to send an office document to somebody.
PGP is also used outside of SMTP, because it is just a container. And SMTP is also used to transport things different from PGP containers, because it is just a transport protocol.
If you instead ask about integrating end-to-end encryption like PGP or S/MIME into SMTP it will not work either, because SMTP is hop-by-hop delivery and not end-to-end. Apart from that SMTP does not even cover the last hop, i.e. the delivery from the last mail server to the client. This is done with protocols like POP or IMAP.
Lea requests the server of Luke's domain jedi.com to tell her the public key of luke@jedi.com ...
That's what you have key servers or other kinds of central directories for. But how does Lea know that this is actually Luke's key and not the key of someone claiming to be Luke? Thus you need to have some trust propagation, for example in the form of a web of trust (PGP) or a more centralized structure (S/MIME) or by trusting everything in a specific central directory.
Thus the task is not to integrate PGP with SMTP but to have better support for PGP in the mail clients, so that they automatically fetch the PGP keys of the recipients. But of course there must be first a verifiable PGP key for the recipient somewhere on a key server or other directory, so the other task is to make key creating, publishing and managing keys (renewing, revoking...) easier. These are all things outside of the mail delivery (SMTP) itself.
Encryption is already in place during mail transit (STARTTLS in SMTP), but not sophisticated enough to protect against MITM.
I believe PGP is more of an end-user experience between email clients, which is helpful if you don't have full trust of the servers involved.
(PGP is sometimes susceptible to MITM to the less-than-careful user, however, like in SSH, if you check for the correct key signature then that problem is solved)
However, in the case of cloud-based email services like Gmail, they would have to be available to the server anyway for a good user experience, so PGP would just get in the way.
Hopefully one day we'll get an MITM proof encryption in SMTP, but it is less of an issue there because mail servers are on controlled networks.
Writing a standard is easy. I thought about this very problem about ten years ago. It comes down to the human/cost factor. How do you convince a billion technologically illiterate people to update their software for no perceived benefit, and convince the thousands of developers across a smattering of platforms to implement this protocol, and millions of organizations to switch to a new protocol, even if you give it away. We've exhausted our IPv4 addresses in some parts of the world as early as 2011, and yet the Internet still does not globally use IPv6. We can't even convince people to switch to IPv6 despite the fact that we're literally stifling the growth of the Internet. Even if you could whip out a standard today, and submit it to IETF to be formalized and distributed, it'd be 2030 before it was widespread enough to shut down the countless "classic" mail servers that are running today.
Well, as you can read above you can use SMTPS and STARTTLS to harden security for SMTP servers while sending mails. MITM can be mitigated with DKIM.
DomainKeys Identified Mail (DKIM) lets an organization take responsibility for a message that is in transit. The organization is a handler of the message, either as its originator or as an intermediary. Their reputation is the basis for evaluating whether to trust the message for further handling, such as delivery. Technically DKIM provides a method for validating a domain name identity that is associated with a message through cryptographic authentication.
In short, DKIM lets you identify if mails come from the origin the mail header says it is coming from. Where the server can then decide in a reliable way if the message can be trusted (if DKIM authentication fails, you can drop the mail and not worry if a forged message gets the SMTP server on a spam house list).
And that kind of rounds the circle. With secure authentication STARTTLS you can reliably connect to an SMTP server and send your mail to a mail server. DKIM can identify if given SMTP server is authorized to send mails from a certain address. TLS for mail delivery from SMTP servery to mail server to encrypt the connection. STARTTLS between mail client and mail server (unless it is an webmail client, then a secure socket connection through POP or IMAP can suffice)
