There is a mechanism to allow secure authentication over HTTP without SSL or TLS, but it's rarely ever used, and it's still not as good as HTTPS. Basically, it's a half-assed security measure of historical interest that never caught on, and you really ought to just use HTTPS anyway. But since you asked……
The HTTP protocol supports two authentication mechanisms: Basic and Digest Access Authentication, both described in RFC 2617. These are mechanisms that cause your browser itself to show an authentication dialog box, not embedded in the contents of the page. Basic authentication, which is sometimes used, is not much better than cleartext transmission.
The Digest mechanism, though, is a challenge-response protocol. The server issues a challenge containing a nonce (some random string). The client must reissue the request with a response that is a hash function of the nonce and the password (but not the password itself).
There are some significant caveats:
- The server usually stores the plaintext password (or a plaintext-equivalent version of it) in order to be able to verify the challenge. This is undesirable, since best practices dictate that only salted password hashes should be stored. (@user2829759 points out that the server could also store the MD5 hash of (username:realm:password).
- The Digest mechanism uses MD5, which is considered to be an insecure hash algorithm these days. Unlike SSL/TLS, there is no algorithm negotiation between the client and server.
- There is no verification of the server's identity. Spoofing is possible, as are man-in-the-middle attacks. The only thing that Digest Authentication is good at protecting is the password itself — which is not as useful as one might think.
In Apache, Digest Authentication support is provided by mod_auth_digest.
One lesson that can be drawn from this piece of trivia is that a JavaScript-based encryption hack is likely to suffer from the same weaknesses. If you need security, just use HTTPS!