Postfix ... Can I use SSH public keys instead of SASL password?
In short, no. Postfix uses SASL, and while SASL supports a variety of different authentication mechanisms, SSH pubkeys is not one of them.
That being said, SASL does support a number of sophisticated mechanisms that would serve your purpose of making login non-trivial for attackers to attempt. The biggest problem may be finding and configuring a complex mechanism that is supported by your legitimate clients!
The simplest way to incorporate public key authentication into your Postfix SMTP workflow is to use the Postfix SMTP TLS settings to require authentication, as described in this documentation.
Alternately, as this answer outlines, you can force your legitimate clients to connect via SSH and connect to the SMTP server over an SSH tunnel, thus meeting the same goal.
- Is it possible to completely change the login authentication such that
a public key must be used instead of password so that these automated
bots require more advanced methods to even hit the smtp server in the
first place?
Some of the SASL mechanisms listed on the page above are, effectively, public keys - for example, RFC 3163 (but note their caveat about it being less useful than simple TLS).
The lack of pubkey SASL mechanisms is likely due to the fact that everyone considering one looked at the alternative of doing the same thing at the TLS layer and said "Solved, no need to work on this!"
- For websites that are on a server also acting as a mailserver, is
there a method to only allow authentication from vhosts on this
webserver itself?
Certainly, if they're on the same machine, you can set up an SMTP listener on localhost (127.0.0.1) which your web server process can connect to but which is unavailable to outsiders. You could also trust relay without authentication on that server, since it's not accessible to others. Look at the Postfix master.cf
file for setting up different or additional listeners.