114

CVE-2014-0160

http://heartbleed.com

This is supposed to be a canonical question on dealing with the Heartbeat exploit.

I run an Apache web server with OpenSSL, as well as a few other utilities relying on OpenSSL (as client). What should I do to mitigate the risks?

Obligatory XKCD

 I looked at some of the data dumps
 from vulnerable sites,
 and it was ... bad.
 I saw emails, passwords, password hints.
 SSL keys and session cookies.
 Important servers
 brimming with visitor IPs.
 Attack ships on fire off 
 the shoulder of Orion,
 c-beams glittering in the dark
 near the Tannhäuser Gate.
 I should probably patch OpenSSL.

Credit: XKCD.

Deer Hunter
  • 5,297
  • 5
  • 33
  • 50
  • 41
    It's worth pointing out that OpenSSH is not affected by the OpenSSL bug. While OpenSSH does use openssl for some key-generation functions, it does not use the TLS protocol (and in particular the TLS heartbeat extension that heartbleed attacks). So there is no need to worry about SSH being compromised, though it is still a good idea to update openssl to 1.0.1g or 1.0.2-beta2 (but you don't have to worry about replacing SSH keypairs). – dr jimbob Apr 08 '14 at 07:02
  • 2
    @Adnan : except there are a few other things to do with certs... If you think it serves no purpose, VTC. – Deer Hunter Apr 08 '14 at 07:38
  • 4
    @drjimbob unless your SSH keys are in the memory of a process that's using OpenSSL's TLS. Unlikely but possible. – OrangeDog Apr 08 '14 at 10:06
  • 11
    Judging by the active attempts being reported in the [DMZ](http://chat.stackexchange.com/rooms/151/the-dmz), the best thing now is **STOPPING THE FRIKKIN SERVER ASAP**. Sessions are being hijacked, passwords leaked, confidential business data revealed. – Deer Hunter Apr 08 '14 at 10:23
  • @OrangeDog - While "possible", that seems to require a completely separate exploit. There's no reason (at least on my systems) that any process having access to an SSH private key also uses TLS. SSH private keys are limited to ssh, sshd, and ssh-agent. So worry about changing your TLS certificates used for HTTPS, FTPS, email, etc. Worry about changing all passwords you've ever used over an HTTPS or TLS connection to a website that may have used OpenSSL in the past three years that you value. But really no legitimate reason to worry about SSH keys. – dr jimbob Apr 08 '14 at 15:01
  • 1
    Apparently, Gabriel Weinberg @ DuckDuckGo had his servers [patched](https://twitter.com/yegg/status/453590663695966208). Qualys incorporated Heartbleed check into their [assessment tool](https://www.ssllabs.com/ssltest/analyze.html?d=duckduckgo.com&s=50.18.192.251). – Deer Hunter Apr 09 '14 at 06:41
  • To link: [What should end users do about Heartbleed?](http://security.stackexchange.com/questions/55083), [How exactly does the OpenSSL TLS Heartbeat (Heartbleed) exploit work](http://security.stackexchange.com/questions/55116) – Deer Hunter Apr 09 '14 at 07:03
  • 3
    Node.JS update: [indutny commented April 08, 2014: No, it isn't we have HEARTBEATS disabled since node v0.10.2.](https://github.com/joyent/node/issues/7424#issuecomment-39820298) – Deer Hunter Apr 09 '14 at 07:25
  • [Another useful link](http://security.stackexchange.com/questions/7790/guidance-for-implementors-of-https-only-sites-server-side) – Deer Hunter Apr 09 '14 at 21:24
  • For ease of understanding: http://security.stackexchange.com/questions/55343/how-to-explain-heartbleed-without-technical-terms. For static binaries: http://security.stackexchange.com/questions/55107/heartbleed-how-to-find-out-applications-using-statically-compiled-version-of-ope – Deer Hunter Apr 11 '14 at 07:11
  • 1
    For even more 'ease of understanding': http://xkcd.com/1354/ – Frerich Raabe Apr 11 '14 at 07:31

5 Answers5

65

There is more to consider than just new certificates (or rather, new key pairs) for every affected server. It also means:

  • Patching affected systems to OpenSSL 1.0.1g
  • Revocation of the old keypairs that were just supersceded
  • Changing all passwords
  • Invalidating all session keys and cookies
  • Evaluating the actual content handled by the vulnerable servers that could have been leaked, and reacting accordingly.
  • Evaluating any other information that could have been revealed, like memory addresses and security measures

Neel Mehta (the Google Security engineer who first reported the bug) has tweeted:

Heap allocation patterns make private key exposure unlikely for #heartbleed #dontpanic.

Tomas Rzepka (probably from Swedish security firm Certezza) replied with what they had to do to recover keys:

We can extract the private key successfully on FreeBSD after restarting apache and making the first request with ssltest.py

Private key theft has been also demonstrated by CloudFlare Challenge.

And Twitter user makomk chimed in with:

I've recovered it from Apache on Gentoo as a bare prime factor in binary, but your demo's a lot clearer...It has a lowish success rate, more tries on the same connection don't help, reconnecting may, restarting probably won't...Someone with decent heap exploitation skills could probably improve the reliability. I'm not really trying that hard.

I summarized the bullet points above from heartbleed.com (emphasis mine):

What is leaked primary key material and how to recover?

These are the crown jewels, the encryption keys themselves. Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services and to impersonate the service at will. Any protection given by the encryption and the signatures in the X.509 certificates can be bypassed. Recovery from this leak requires patching the vulnerability, revocation of the compromised keys and reissuing and redistributing new keys. Even doing all this will still leave any traffic intercepted by the attacker in the past still vulnerable to decryption. All this has to be done by the owners of the services.

What is leaked secondary key material and how to recover?

These are for example the user credentials (user names and passwords) used in the vulnerable services. Recovery from this leaks requires owners of the service first to restore trust to the service according to steps described above. After this users can start changing their passwords and possible encryption keys according to the instructions from the owners of the services that have been compromised. All session keys and session cookies should be invalided and considered compromised.

What is leaked protected content and how to recover?

This is the actual content handled by the vulnerable services. It may be personal or financial details, private communication such as emails or instant messages, documents or anything seen worth protecting by encryption. Only owners of the services will be able to estimate the likelihood what has been leaked and they should notify their users accordingly. Most important thing is to restore trust to the primary and secondary key material as described above. Only this enables safe use of the compromised services in the future.

What is leaked collateral and how to recover?

Leaked collateral are other details that have been exposed to the attacker in the leaked memory content. These may contain technical details such as memory addresses and security measures such as canaries used to protect against overflow attacks. These have only contemporary value and will lose their value to the attacker when OpenSSL has been upgraded to a fixed version.

kravietz
  • 412
  • 2
  • 7
scuzzy-delta
  • 9,303
  • 3
  • 33
  • 54
  • "Leaked secret keys allows the attacker to decrypt any past and future traffic to the protected services" - weren't we supposed to have perfect forward secrecy to protect us from that? – user2357112 Apr 08 '14 at 10:58
  • 2
    Quoting from heartbleed.com: *"Use of Perfect Forward Secrecy (PFS), which is unfortunately rare but powerful, should protect past communications from retrospective decryption."* Caveat: [A leaked](https://twitter.com/ivanristic/status/453280081897467905) [ticket key](http://en.wikipedia.org/wiki/Transport_Layer_Security#Session_tickets) would compromise all sessions it signed. – scuzzy-delta Apr 08 '14 at 12:58
  • 3
    SSL keys are not likely to leak via Heartbleed because they are not likely to be present in these memory regions. On the other hand, user cookies, including session cookies, user passwords and AJAX data is **very likely** to be found there. – kravietz Apr 08 '14 at 14:46
  • @kravietz Good call (see edit/tweet from Neel Mehta) – scuzzy-delta Apr 09 '14 at 08:28
  • @scuzzy-delta Actually I'm doing some testing right now and in the HB dumps I find quite a lot of X.509 certificates, so I'm less confident about not finding private keys there. On the other hand, presence of the certs can be explained - SSL server sends a certificate path to each new client, so they can be present in operational memory. On the other other hand private keys are **also** used on each connection, just not sure if they are copied around in memory as frequently as certs. – kravietz Apr 12 '14 at 09:10
  • 2
    Oh, and now it's official - we do need to regenerate private keys and reissue certs... https://www.cloudflarechallenge.com/heartbleed – kravietz Apr 12 '14 at 11:18
21

Directly copied from OpenSSL site

OpenSSL Security Advisory [07 Apr 2014]

TLS heartbeat read overrun (CVE-2014-0160)

A missing bounds check in the handling of the TLS heartbeat extension can be used to reveal up to 64k of memory to a connected client or server.

Only 1.0.1 and 1.0.2-beta releases of OpenSSL are affected including 1.0.1f and 1.0.2-beta1.

Thanks for Neel Mehta of Google Security for discovering this bug and to Adam Langley and Bodo Moeller for preparing the fix.

Affected users should upgrade to OpenSSL 1.0.1g. Users unable to immediately upgrade can alternatively recompile OpenSSL with -DOPENSSL_NO_HEARTBEATS.

1.0.2 will be fixed in 1.0.2-beta2.

  • Check if you're using the mentioned versions of OpenSSL, if yes patch it to 1.0.1g (by compiling it from source and wrapping the package with e.g. FPM).

  • (Addition by atk) Afterwards, replace your exposed certificates and revoke them.

  • (Addition by dr.jimbob) It's worth pointing out that OpenSSH is not affected by the OpenSSL bug. While OpenSSH does use openssl for some key-generation functions, it does not use the TLS protocol (and in particular the TLS heartbeat extension that heartbleed attacks). So there is no need to worry about SSH being compromised, though it is still a good idea to update openssl to 1.0.1g or 1.0.2-beta2 (but you don't have to worry about replacing SSH keypairs).

    • (OrangeDog): @drjimbob unless your SSH keys are in the memory of a process that's using OpenSSL's TLS. Unlikely but possible.

  • (Addition by Deer Hunter): Judging by the active attempts being reported in the DMZ, the best thing now is STOPPING THE FRIKKIN SERVER ASAP. Sessions are being hijacked, passwords leaked, confidential business data revealed.

  • (An extra bit courtesy of EFF.org): "To reach a firmer conclusion about Heartbleed's history, it would be best for the networking community to try to replicate Koeman's findings. Any network operators who have extensive TLS-layer traffic logs can check for malicious heartbeats, which most commonly have a TCP payload of 18 03 02 00 03 01 40 00 or 18 03 01 00 03 01 40 00, although the 0x4000 at the end may be replaced with lower numbers, particularly in implementations that try to read multiple malloc chunk bins." In a nutshell, if you keep detailed TLS logs (not likely for the majority of operators out there), check for past exploitation attempts (and share what you've got).

Related Q&A over at ServerFault:

https://serverfault.com/questions/587338/does-heartbleed-affect-aws-elastic-load-balancer

https://serverfault.com/questions/587329/heartbleed-what-is-it-and-what-are-options-to-mitigate-it

https://serverfault.com/questions/587348/best-method-to-update-ubuntu-13-10-to-path-the-heartbleed-bug

https://serverfault.com/questions/587324/heartbleed-how-to-reliably-and-portably-check-the-openssl-version

A well-written summary over at AskUbuntu: https://askubuntu.com/a/444905

A comprehensive Q&A at Unix&Linux SE: https://unix.stackexchange.com/questions/123711/how-do-i-recover-from-the-heartbleed-bug-in-openssl

If by any chance you run a server on Mac OS X: https://apple.stackexchange.com/questions/126916/what-versions-of-os-x-are-affected-by-heartbleed

Retrieving Private SSL Key using heart bleed: http://blog.cloudflare.com/answering-the-critical-question-can-you-get-private-ssl-keys-using-heartbleed Yes, it's possible!

Community
  • 1
hoa
  • 441
  • 2
  • 8
9

[edited]

I made a tool to check the status of your SSL and see if heartbeat is enabled and vulnerable. Tool at: http://rehmann.co/projects/heartbeat/

There's another one at http://filippo.io/Heartbleed/

If you're vulnerable, please upgrade your OpenSSL packages & renew your certs!

Luke Rehmann
  • 111
  • 4
  • 2
    Man, have you got an offline version with source? If my site is vulnerable, I'd be crazy to leave it powered. – Deer Hunter Apr 08 '14 at 03:42
  • @Deer Hunter You can shutdown any service that uses TLS (apache, nginx, email?). SSH is **NOT** affected by this bug. – Augusto Apr 08 '14 at 07:44
  • 3
    Another checker: http://filippo.io/Heartbleed/ (Filippo put his source on GitHub!) – Deer Hunter Apr 08 '14 at 10:51
  • 4
    Why should we trust your tool to not exploit the vulnerability? – user Apr 08 '14 at 11:53
  • 1
    @MichaelKjorling : given the massive coverage of the bug, I'm sure there are scores of scanners all around. – Deer Hunter Apr 08 '14 at 12:37
  • 1
    @DeerHunter No doubt. But this is the Information Security Stack Exchange, and I'd rather not point those scanners squarely at my servers. :) – user Apr 08 '14 at 12:42
  • 2
    -1 because this is an online tool that is not auditable, so not trustable. – dolmen Apr 08 '14 at 14:00
  • @DeerHunter: An online tool, even with source code, is still an online tool. Nothing guatantees that the running code is the advertised code. The main issue is not that the site exploits directly the issue, but instead that it records the memory image captured and keep it for later. – dolmen Apr 08 '14 at 14:02
  • @dolmen: nothing *guarantees* that this tool won't scan your site even if you don't tell it your domain name (assuming your site can be connected to, ofc). There are people scanning every site they can find for the vuln. Some of those people are security researchers. – Steve Jessop Apr 08 '14 at 15:10
  • Unfortunately, checking the OpenSSL version using `openssl version -a` is not a very reliable way to see if you need to patch. Some Ubuntu and Debian versions keep showing a lower version number, even though all patches are applied. More details at http://serverfault.com/questions/587324/heartbleed-how-to-reliably-and-portably-check-the-openssl-version – Martijn Apr 08 '14 at 19:26
3

Jspenguin wrote an offline tool to check if a server has the flaw. Download it, audit it, and run it.

I also wrote ssl-heartbleed-check.pl (also an offline tool) to check if the OpenSSL used by your Perl stack (which on *n*x is often the openssl used by the whole system) is affected. This can help you to determine if you are affected on the client side.

Nmap 6.45 includes an ssl-heartbleed script. ssl-heartbleed.nse can also be used together with nmap ≥6.25.

Gilles 'SO- stop being evil'
  • 50,912
  • 13
  • 120
  • 179
dolmen
  • 131
  • 4
3

Note that if you're using a cloud based provider or content distribution network, and they are vulnerable, your website's leaking content will be mixed with content of all other websites using this provider. I've just just seen that with Incapsula, where a bank website's content was leaked along cryptocurrency website. They're fixed now fortunately.

kravietz
  • 412
  • 2
  • 7