I understand that RSA fingerprints are used to verify that you are really connecting directly to who you want to connect to, and not someone else posing as that site.
Like when you do a git push, it shows you an RSA key, then you can go to a webpage and see if the key you got matches the key on their webpage.
But what's to stop someone else from finding out the public RSA key that the site is using, and then replying with that same key, in order to pretend to be that site?
Some good information in the other answers as well, but to answer your question directly:
But what's to stop someone else from finding out the public RSA key that the site is using, and then replying with that same key, in order to pretend to be that site?
An attacker could do this, and they could trick you into encrypting your data using the public key and sending it to them. However, without the corresponding private key, this information would be useless to them because they would be unable to decrypt it.
Public keys are exactly that - public, intended for distribution. Having a copy of a public key doesn't allow you, on its own, to masquerade as the key's owner. It gives you the ability to encrypt data that you send to the owner in a manner that only the owner can decrypt it, using the private key that corresponds with the public key. It also allows you to decrypt data that was encrypted using the private key, proving that the data came from the owner of the private key, which is the use here - your copy of the public key allows you to be reasonably certain that the server you are communicating with is trustworthy (to the extent that you can trust the owner of the private key to maintain the secrecy of that private key).
Provided the owner keeps the private key, private, the system works as intended.
The Wikipedia article on public-key cryptography gives a good introduction to how things like RSA, DSA etc work.
The fingerprint is computed from the associated public key and can be calculated by your computer. If the fingerprint matches what you expect, then the key you are encrypting to is the appropriate one for your target.
While one can report a false fingerprint, the fact that they can be independently calculated from the actual possessed key makes pretty much useless unless the hash used for fingerprinting is small enough that a collision is realistically possible. Without a collision, any exchange of data would still be encrypted to the appropriate public key (since you verified that the one you're using matches the expected hash) and could not be decrypted without the corresponding private key.
SSH host authentication occurs during the key exchange. The key exchange is generally done using Diffie-Hellman, which allows generating a shared secret that can be used as a symmetric session key.
At the start of the key exchange, a signing algorithm is negotiated (this can be RSA, DSA, etc). At the end of the key exchange, the host uses this signing algorithm with its private key to sign the previously exchanged data. The host sends its public key and the signature to the client (along with the Diffie-Hellman stuff), and the client uses the host's public key to verify the signature. Since a signature can only be created with the private key, the client knows that the host has access to the private key corresponding to the public key the host presented.
The client has a few ways of verifying that it's talking to the correct host. If it has previously connected to the host, it can match the host's public key against a local database to make sure the host presented the same public key that it did previously.
During the key exchange, the host can also opt to send a certificate signed by a CA that the client trusts instead of sending just the public key; the client can then verify that the host's public key was signed by the CA (although in practice certificates aren't used with ssh very often).
If the host key can't be automatically verified, a fingerprint (hash of the host's public key) is often presented to the user to allow them to manually verify it out-of-band. If accepted by the user, the public key will then generally be stored to allow automatic verification for future connections.
So to answer your question: Any host could copy any other host's public key, but it would be unable to produce a valid signature for the key exchange, and the client would abort with an error. As long as you verify the host key's fingerprint out-of-band (like you are by checking the website, over HTTPS I hope), MitM is prevented.
The MITM needs the private key of the server to pose as the server and decrypt private messages you send to to the server. The public key of the server is for authenticating messages signed by the server and sending private messages to the server, so you still need to verify its fingerprint. It's okay if the MITM has the public key too. All that means is that the MITM can also authenticate messages from the server and send private messages to the server.
In general, if all public keys are known to everyone, anyone can authenticate anyone's signatures and anyone can send a private message to anyone. The private key controls signing and decryption, so ideally no one can pose as another, and no one can read a private message sent to another.
NB: The following is simplified model for educational purposes only, intended to illustrate the point of the public key fingerprint. See this other question for more SSH details.
Review. For simplicity, instead of SSH over a computer network, consider a
public cork-board where agents can post text notes. At the start, the board is
blank. Agents A and S want to communicate, and wish to protect the privacy
and integrity of their messages from malicious agent M. Assume private keys
are known only to their respective owners.
A posts A_pub, and S posts S_pub.A encrypts a message with S_pub, signs with A_priv, and posts.S decrypts the message with S_priv, verifies signature using A_pub.S encrypts a response with A_pub, signs with S_priv, and posts.A does here.Agent M cannot decrypt these messages. M can post encrypted messages
to S or A. However, an attempt at forgery would fail. A message signed with
M_priv would not pass verification using A_pub or S_pub.
Now for the function of the public key fingerprint. Suppose M creates
M_pub and M_priv, and attempts to fool A by posting M_pub as S_pub:
Dear A,
My public key is
[contents of M_pub]
Love,
S
If
A believes this lie, A will encrypt messages to S with M_pub so that M
can decrypt them, and M can forge messages as S using M_priv. However, if
A has the fingerprint of S_pub, A can compare it with the fingerprint of
M_pub and discover the lie. Hence A will refuse to use M_pub for
encryption of messages to S and authentication of messages from S.
Therefore when SSH gives a warning about the pubic key fingerprint being unknown or changed, it means you could be the victim of an MITM attack. If you proceed, the MITM will be able to decrypt your messages and forge responses as the server. If the fingerprint check succeeds you have a very high confidence you are not a victim of MITM, assuming the fingerprint you have is the real fingerprint of the server, the private keys (both yours and the server's) are indeed private, the implementations of SSH are correct, and the public-key crypto math is not compromised.
It does not only show you the RSA key. It also encrypts the communication using a public / private key pair (including that RSA key). This way you are sure only server X can be the server ( since normally only server X has the private key).
You can read more about public / private keys here . There is quite some math involved.
If I am not mistaken, the function that is used to generate the fingerprint uses the private key to do so. So the fingerprint cannot* be recreated without the private key.
*There's the possibility of collisions since the range of values that valid fingerprints are less than the range of values of valid keys. It's sufficiently rare to still enable fingerprints to be useful.
