203

This is a Canonical Question about understanding and remediating the Heartbleed security issue.

What exactly is CVE-2014-0160 AKA "Heartbleed"? What is the cause, what OSs and versions of OpenSSL are vulnerable, what are the symptoms, are there any methods to detect a successful exploit?

How can I check to see if my system is affected? How can this vulnerability be mitigated? Should I be concerned that my keys or other private data have been compromised? What other side effects should I be concerned about?

Jacob
  • 9,114
  • 4
  • 44
  • 56
  • 14
    Mitigation for Heartbleed [involves **more** than just new keys](http://security.stackexchange.com/a/55089/1811). (Link to my answer on Information Security StackExchange) – scuzzy-delta Apr 08 '14 at 06:59
  • I hear you, but I think EEAA pretty comprehensively covered that below. – MadHatter Apr 08 '14 at 07:16
  • I agree: it's a great answer, but [heartbleed.com](http://heartbleed.com/) goes to great effort to point out that there are considerations beyond just new key pairs - like forcing password changes and session invalidation. – scuzzy-delta Apr 08 '14 at 07:24
  • 1
    @scuzzy-delta - good point. I've made my answer CW now, so feel free to edit/improve it with that information. – EEAA Apr 08 '14 at 12:35
  • @scuzzy-delta Good catch, I'll do a writeup in a few hours when I get a minute. – Jacob Apr 08 '14 at 12:51
  • I wrote a CentOS Patch tutorial here: http://kevinflorida.com/heartbleed-patch – Kevin Apr 10 '14 at 04:01
  • "I have looked at the OpenSSL source code, I think it is a catastrophe waiting to happen." said phk https://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-April/018410.html –  Apr 10 '14 at 08:28
  • Thanks to everyone for upvoting the question, but please upvote the answers too! – Jacob Apr 11 '14 at 07:20
  • 3
    Best example on what it is - (unsurprisingly) XKCD: http://xkcd.com/1354/ – Wayne Werner Apr 12 '14 at 16:44
  • @EricDANNIELOU Wow, the signature in the message you linked is very much relevant here: "Never attribute to malice what can adequately be explained by incompetence." – molnarm Apr 13 '14 at 18:26

9 Answers9

116

First, before freaking out, be sure that you understand whether or not this vulnerability actually applies to you. If you have a server, but have never actually had any applications using TLS, then this is not a high-priority thing for you to fix. If, on the other hand, you have ever had TLS-enabled applications, well then you're in for a treat. Read on:

What exactly is CVE-2014-0160 aka "Heartbleed"?

It's a big fricking mess, that's what it is. In short, a remotely-exploitable vulnerability was discovered in OpenSSL versions 1.0.1 through 1.0.1f through which an attacker can read certain parts of system memory. Those parts being that which hold sensitive data such as private keys, preshared keys, passwords and high valued corporate data among other things.

The bug was independently discovered by Neel Mehta of Google Security (March 21, 2014) and Finnish IT security testing firm Codenomicon (April 2, 2014).

What is the cause?

Well, errant code in OpenSSL. Here is the commit that introduced the vulnerability, and here is the commit that fixed the vulnerability. The bug showed up in December of 2011 and was patched today, April 7th, 2014.

The bug can also be seen as a symptom of a larger problem. The two related problems are (1) what process are in place to ensure errant code is not introduced to a code base, and (2) why are the protocols and extensions so complex and hard to test. Item (1) is a governance and process issue with OpenSSL and many other projects. Many developers simply resist practices such as code reviews, analysis and scanning. Item (2) is being discussed on the IETF's TLS WG. See Heartbleed / protocol complexity.

Was the errant code maliciously inserted?

I won't speculate on whether this was truly a mistake or possibly a bit of code slipped in on behalf of a bad actor. However, the person who developed the code for OpenSSL states it was inadvertent. See Man who introduced serious 'Heartbleed' security flaw denies he inserted it deliberately.

What OSs and versions of OpenSSL are vulnerable?

As mentioned above, any operating system that is using, or application that is linked against OpenSSL 1.0.1 through 1.0.1f.

What are the symptoms, are any methods to detect a successful exploit?

This is the scary part. As far as we know, there is no known way to detect whether or not this vulnerability has been exploited. It is theoretically possible that IDS signatures will be released soon that can detect this exploit, but as of this writing, those are not available.

There is evidence that Heartbleed was being actively exploited in the wild as early as November, 2013. See the EFF's Wild at Heart: Were Intelligence Agencies Using Heartbleed in November 2013? And Bloomberg reports the NSA had weaponized the exploit shortly after the vulnerability was introduced. See NSA Said to Exploit Heartbleed Bug for Intelligence for Years. However, the US Intelligence Community denies Bloomberg's claims. See IC ON THE RECORD.

How can I check to see if my system is affected?

If you are maintaining OpenSSL on your system, then you can simply issue openssl version:

$ openssl version
OpenSSL 1.0.1g 7 Apr 2014

If the distribution is maintaining OpenSSL, then you probably can't determine the version of OpenSSL due to back patching using openssl command or the package information (for example, apt-get, dpkg, yum or rpm). The back patching process used by most (all?) distributions only uses the base version number (for example, "1.0.1e"); and does not include an effective security version (for example, "1.0.1g").

There's an open question on Super User to determine the effective security version for OpenSSL and other packages when packages are backpatched. Unfortunately, there are no useful answers (other than check the distro's website). See Determine Effective Security Version when faced with Backpatching?.

As a rule of thumb: if you have ever installed one of the affected versions, and have ever run programs or services that linked against OpenSSL for TLS support, then you are vulnerable.

Where can I find a program to test for the vulnerability?

Within hours of the Heartbleed announcement, several people on the internet had publicized publicly-accessible web applications that supposedly could be used to check a server for the presence of this vulnerability. As of this writing, I have not reviewed any, so I won't further publicize their applications. They can be found relatively easily with the help of your preferred search engine.

How is this vulnerability mitigated?

Upgrade to a non-vulnerable version and reset or re-secure vulnerable data. As noted on the Heartbleed site, appropriate response steps are broadly:

  1. Patch vulnerable systems.
  2. Regenerate new private keys.
  3. Submit new CSR to your CA.
  4. Obtain and install new signed certificate.
  5. Invalidate session keys and cookies
  6. Reset passwords and shared secrets
  7. Revoke old certificates.

For a more detailed analysis and answer, see What should a website operator do about the Heartbleed OpenSSL exploit? on the Security Stack Exchange.

Should I be concerned that my keys or other private data have been compromised? What other side effects should I be concerned about?

Absolutely. Systems Administrators need to assume that their servers which used vulnerable OpenSSL versions are indeed compromised and respond accordingly.

Shortly after the vulnerability was disclosed, Cloudfare offered a challenge to see if a server's private key could be recovered in practice. The challenge was independently won by Fedor Indutny and Ilkka Mattila. See The Heartbleed Challenge.

Where can I find more information?

Link dump, for those looking for more details:


A rather detailed timeline of the disclosure events can be found at Heartbleed disclosure timeline: who knew what and when.


If you are a programmer and are interested in various programming tricks like detecting a Heartbleed attack through OpenSSL's msg_cb callback, then see OpenSSL's Security Advisory 2014047.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • 42
    +1 for **SHUT. DOWN. YOUR. SERVERS.*** -- If you do ANYTHING where SSL is really important, shut it off until you fix the issue. Also don't forget to install new certificates (with ***new keys***) after you patch your servers - reusing your old keys (which may have been compromised) defeats the whole purpose of patching the vulnerability... – voretaq7 Apr 08 '14 at 01:30
  • 29
    **ALSO** - restart any services that link to OpenSSL libraries. Upgrading OpenSSL without restarting your daemons is as good as not upgrading at all. – EEAA Apr 08 '14 at 01:34
  • 14
    Indeed - after any kind of major patch (like OpenSSL) I consider it a good rule to just reboot the machine to make sure you don't miss anything. – voretaq7 Apr 08 '14 at 01:35
  • 5
    One of the testers has been open-sourced: https://github.com/FiloSottile/Heartbleed – Riking Apr 08 '14 at 06:57
  • @EEAA Good point on restarting the services. For those who cannot reboot immediately, the following command can find processes that need closer attention: `ps uwwp $(find /proc -maxdepth 2 -name maps -exec grep -HE '/libssl\.so.* \(deleted\)' {} \; | cut -d/ -f3 | sort -u)` – Lekensteyn Apr 08 '14 at 08:49
  • [snort/suricata signatures](https://lists.emergingthreats.net/pipermail/emerging-sigs/2014-April/024058.html) – that guy from over there Apr 08 '14 at 23:46
  • Why is the private key in plain in the memory ? – Eduard Florinescu Apr 09 '14 at 06:10
  • @EduardFlorinescu The web server needs to use the private key to decrypt SSL traffic, so it sits in RAM. It's pure luck to get anything useful (like the keys) with this exploit, though. – Nathan C Apr 09 '14 at 12:15
  • 1
    Don't **Shut down your servers.** Disconnect those cables. «Disconnect these cables. Overturn these tables. This place don't make sense to me no more. Can you tell me what we're waiting for, señor? » - Bob Dylan – Eduard Florinescu Apr 09 '14 at 12:22
  • @EduardFlorinescu - for physical servers in one's one data center, sure. For rented VPSs/dedicated servers/etc, not possible. – EEAA Apr 09 '14 at 12:36
  • @EEAA make me think that if you cannot touch the cable the data is not really yours. – Eduard Florinescu Apr 09 '14 at 12:43
  • @EduardFlorinescu How else would you use it? – Simon Lindgren Apr 09 '14 at 12:49
  • @EduardFlorinescu - you're just trolling now. There are *plenty* of situations, even with a company's own servers, where the sysadmin team doesn't have physical access to the datacenter. – EEAA Apr 09 '14 at 14:27
  • 3
    @EEAA, "shutdown your servers" does not mean you have to pull the power. It means shut down ( or reconfigure to disable ssl/tls ) apache, or whatever service is doing the serving. – psusi Apr 09 '14 at 22:29
  • @psusi If someone doesn't know whether or not they're vulnerable or whether or not they will be able to easily fix the vulnerability, the most foolproof thing to do is shut down. – EEAA Apr 09 '14 at 23:06
  • 3
    "I won't speculate on whether this was truly a mistake or possibly a bit of code slipped in on behalf of a bad actor." - the code was provided by Robin Seggelmann. Seggelmann was coauthor of [Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension](https://tools.ietf.org/rfc/rfc6520.txt) RFC. The actual commit was done by Dr. Henson; see the 4817504d0 commit. The IETF's TLS WG is debating the usefulness and added complexity of such extensions now at [TLS: Heartbleed / protocol complexity](https://www.ietf.org/mail-archive/web/tls/current/msg11891.html). –  Apr 10 '14 at 00:00
  • @noloader - it's a CW. Feel free to edit as you wish. Like I said, in my answer I wasn't going to comment on this - that wasn't the purpose of the answer. – EEAA Apr 10 '14 at 00:01
  • @EEAA, it is also the least desirable response and, as you pointed out, not always possible. – psusi Apr 10 '14 at 01:18
  • 1
    @psusi As I've pointed out to others. It's a community wiki. Edit away. – EEAA Apr 10 '14 at 01:23
  • @EEAA, I was responding to your comment that you can't always shutdown. – psusi Apr 10 '14 at 01:24
43

A simple explanation of the bug, by XKCD:

XKCD 1354

200_success
  • 4,701
  • 1
  • 24
  • 42
  • Bonus for "User Karen" for setting her password to "CoHoBaSt", a shorter version of [https://xkcd.com/936/](Correct Horse Battery Staple). – Gloweye Oct 21 '19 at 12:17
36

Ubuntu 12.04, 12.10, and 13.10

Ubuntu has issued USN-2165-1, which states that updated packages are now available in the archives. Run the following two commands to grab the fix.

sudo apt-get update
sudo apt-get upgrade

Ubuntu 14.04

I have uploaded a Debian package containing the new release (1.0.1g) to a PPA I have set up for this purpose. These three commands will add my PPA to your system, update the list of available packages, and upgrade everything:

sudo add-apt-repository ppa:george-edison55/openssl-heartbleed-fix
sudo apt-get update
sudo apt-get upgrade

Note: the PPA also provides packages for Ubuntu 12.04 and 13.10, just in case you prefer to actually run the new version (1.0.1g) instead of just using the patched versions in the archives.

Ubuntu 10.04

This is an LTS Version, the server version is still supported and receives security updates. But the heartbleed vulnerability did not affect the openssl package of a standard installation of ubuntu 10.04, because the version is below 1.0.1.

The desktop version has reached end of life and needs to be upgraded / reinstalled.

Ubuntu 13.04 and other outdated versions

Ubuntu 13.04 had a very short support cycle which you might not expect. It has reached end of life already and does not receive security updates any more. It should long have been upgraded. If still someone is using it, please upgrade now, either from scratch or it can be upgraded non-destructive to 13.10 following this easy procedure: http://www.tecmint.com/upgrade-ubuntu-13-04-raring-ringtail-to-ubuntu-13-10-saucy-salamander/ After the upgrade the system receives the heartbleed patch from 13.10.

For all other outdated ubuntu versions it means basically a fresh install is necessary.

Verify that the patch was applied

Essentially, run openssl version -a and make sure that the build date is April 7, 2014 or later, but see more here.

Reboot

The best way to make sure all services depending on OpenSSL are restarted is to reboot.

Nathan Osman
  • 2,705
  • 7
  • 31
  • 46
  • I can't speak for other versions, but there appears to be a patch available for precise (12.04). While I can't say for certain that this fixes the vulnerability, it was at least compiled after the relevant commit (`Mon Apr 7 20:31:55 UTC 2014`). – Calrion Apr 08 '14 at 04:52
  • @Calrion: A patch for OpenSSL or the Debian packaging for OpenSSL? OpenSSL has already been fixed and a new release issued. – Nathan Osman Apr 08 '14 at 04:54
  • what will happen to existing connections while openssl is being updated? will they be dropped? – pdeva Apr 08 '14 at 05:49
  • 2
    That depends on which web server you are using and how you update. That being said, I wouldn't worry about dropping existing connections since they are using the vulnerable version. – Nathan Osman Apr 08 '14 at 05:51
  • 3
    14.04 [has since received a patch](http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.0.1f-1ubuntu2/changelog). – Seth Apr 08 '14 at 19:51
  • @NathanOsman The `openssl` package available for precise has been updated. I applied the updated package to my server and [SSL Labs](http://www.ssllabs.com/ssltest/) reports that my server is **not** vulnerable to heartbleed. I simply ran `sudo apt-get update && sudo apt-get upgrade`. – Calrion Apr 09 '14 at 07:46
  • Doing this is not enough. See the Community Wiki answer by @EEAA. – Simon Lindgren Apr 09 '14 at 08:53
  • can you respond to this question http://askubuntu.com/questions/445821/upgrading-openssl-on-ubuntu-10-04-4-lts-heartbleed – Haris Hasan Apr 10 '14 at 07:03
14

RedHat 6.5 and CentOS 6.5

These are vulnerable. RedHat's erratum RHSA-2014-0376 says there are patched libraries available, and anyone affected should upgrade at the earliest opportunity.

At the time of writing, CentOS did not yet have a fixed version, but Karanbir Singh's posting to CentOS-announce says that they've produced an updated version of openssl (openssl-1.0.1e-16.el6_5.4.0.1, note the last four digits which are important) that has the exploitable TLS command disabled, and that can be safely applied as it will be overwritten by a fixed version when it is eventually released.

The temporarily-fixed version doesn't seem to have made it onto all the mirrors yet, but is in the main repository at http://mirror.centos.org/centos/6/updates/x86_64/Packages/ (and similarly for i686).

Edit: as Iain says, there does now appear to be a fully-patched version for C6.5, and it seems to have been pushed around the mirrors in a hurry. A straight yum update got it for my servers; it's openssl-1.0.1e-16.el6_5.7.

Versions of RH6 and C6 prior to 6.5

These are not vulnerable. According to this advisory from Red Hat,

This issue did not affect the versions of openssl as shipped with Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 6.4 and earlier.

Karanbir Singh's posting to CentOS-announce is equally clear about versioning:

Earlier in the day today, we were made aware of a serious issue in openssl as shipped in CentOS-6.5

MadHatter
  • 78,442
  • 20
  • 178
  • 229
  • Isn't http://lists.centos.org/pipermail/centos-announce/2014-April/020249.html the release of the fix ? – user9517 Apr 08 '14 at 10:07
13

Debian Wheezy

Debian has issed DSA-2896-1 and patched libraries are available here. A shell script is available here.

1. Patch

Apt-get repository was updated so now patched libraries are available via apt-get update && apt-get upgrade

apt-get upgrade libssl1.0.0 openssl

Alternatively (not recommended) the packages can be upgraded manually:

wget http://security.debian.org/pool/updates/main/o/openssl/libssl1.0.0-dbg_1.0.1e-2+deb7u5_amd64.deb
wget http://security.debian.org/pool/updates/main/o/openssl/openssl_1.0.1e-2+deb7u5_amd64.deb
wget http://security.debian.org/pool/updates/main/o/openssl/libssl1.0.0_1.0.1e-2+deb7u5_amd64.deb
wget http://security.debian.org/pool/updates/main/o/openssl/libssl-dev_1.0.1e-2+deb7u5_amd64.deb

dpkg -i openssl_1.0.1e-2+deb7u5_amd64.deb
dpkg -i libssl1.0.0_1.0.1e-2+deb7u5_amd64.deb
dpkg -i libssl1.0.0-dbg_1.0.1e-2+deb7u5_amd64.deb
dpkg -i libssl-dev_1.0.1e-2+deb7u5_amd64.deb

2. Restart server/services

For best protection restart the entire server or if server can't be offline then restart needed services.

3. Check OpenSSL Version

love@server:~$ openssl version
OpenSSL 1.0.1e 11 Feb 2013
love@server:~$ dpkg -l libssl1.0.0
||/ Name                    Version          Architecture     Description
+++-=======================-================-================-====================================================
ii  libssl1.0.0                 1.0.1e-2+deb7u6  amd64            SSL shared libraries
Alastair Irvine
  • 1,172
  • 10
  • 22
jacksoncage
  • 185
  • 1
  • 1
  • 11
  • 1
    If you're getting updates from `wheezy/security` then you'll be good with `apt-get update && apt-get upgrade`. Or, use an interactive package manager to only update the packages `openssl`, `libssl1.0.0`, `libssl1.0.0-dbg` and `libssl-dev` (as installed on your system). – user Apr 08 '14 at 12:40
  • using apt-get does not fix the problem for me - still showing OpenSSL 1.0.1e 11 Feb 2013 – user568829 Apr 08 '14 at 19:17
  • Thanks @michael-kjorling, it was not available when I did this, but it's most secure and correct way of upgrading. – jacksoncage Apr 09 '14 at 13:50
  • 2
    @user568829 after applying the patch openssl version will still show `OpenSSL 1.0.1e 11 Feb 2013` as the patch is called 1.0.1e-2. You can check with `dpkg -l openssl` and it should show version `1.0.1e-2+deb7u6` – jacksoncage Apr 09 '14 at 13:57
  • 2
    I'd suggest restarting the host after updating OpenSSL, not because it is strictly necessary but for peace of mind that at least everything that loads the OpenSSL libraries dynamically is using the new version. (Statically linked is another matter.) That said, I recognize that some servers can't easily be rebooted in all situations where a service restart might be acceptable. – user Apr 09 '14 at 14:12
9

FreeBSD 10.0 or openssl from ports

The FreeBSD security team has issued an advisory regarding CVE-2014-0160 (aka "Heartbleed") and : FreeBSD-SA-14:06.openssl

  1. Updating FreeBSD

    • Updating FreeBSD via a binary patch

      Systems running a RELEASE version of FreeBSD on the i386 or amd64 platforms can be updated via the freebsd-update(8) utility:

      # freebsd-update fetch
      # freebsd-update install
      
    • Updating FreeBSD from the sources

      1. Download the relevant patch from the location below, and verify the detached PGP signature using your PGP utility.

        # fetch http://security.FreeBSD.org/patches/SA-14:06/openssl-10.patch
        # fetch http://security.FreeBSD.org/patches/SA-14:06/openssl-10.patch.asc
        # gpg --verify openssl-10.patch.asc
        
      2. Execute the following commands as root:

        # cd /usr/src
        # patch < /path/to/patch
        
      3. Recompile the operating system

        using buildworld and installworld as described in the FreeBSD handbook.

  2. Update the openssl port with minimum version 1.0.1_10

  3. Restart all daemons using the library, or reboot the system

  4. Act as if your system has been compromised, re-issue all your ssl keys and/or certificates and potentially leaked information (see EEAA more general answer).

FreeBSD 9.x and FreeBSD 8.x

These system are not vulnerable to the Heartbleed issue by default, as relying on older 0.9.x version of the openssl library, unless you installed openssl from the ports (see upstairs).

If these systems are not vulnerable to the Heartbleed issue, it might be wise to upgrade your system rather sooner than later due to another local vulnerability (see FreeBSD-SA-14:06.openssl and the "FreeBSD 10.0" section upstairs):

A local attacker might be able to snoop a signing process and might recover the signing key from it. [CVE-2014-0076]

Note:

The original Heartbleed advisory lists FreeBSD 8.4 and 9.1 as being potentially vulnerable. This is not true due to the lack of Heartbeat Extension (default FreeBSD openssl library being version 0.9.x).

Ouki
  • 1,367
  • 1
  • 11
  • 16
9

I'd like to point out that private keys are not the only assets that should be considered compromised. The bug has the potential to leak any memory running in the same address space (i.e., the same process) as OpenSSL. Therefore, if you are running a server process where a vulnerable version of OpenSSL is statically or dynamically linked, any piece of information that that process has ever handled, including passwords, credit card numbers, and other personal data, should be considered potentially compromised.

200_success
  • 4,701
  • 1
  • 24
  • 42
3

I found it next to impossible to determine the versions of SSL in use on several of the appliances I work with. Although it's technically not mitigation being able to ID currently vulnerable hosts was at the top of my list.

I put together a small VM that will perform checks against arbitrary hosts and ports using FiloSottile's test module. On preliminary glance the code looks sound.

The release of the completed VM is here. It's in VMX format.

Words of Warning

This script and VM will only show the current status of your systems. It's entirely possible that at some point in the past that your systems were in a vulnerable state and could have been been abused.

Something showing up here is definitely a high priority to fix but it does not get you off the hook for applying updates and changing all your keys.

Tim Brigham
  • 15,465
  • 7
  • 72
  • 113
2

Amazon Linux (Linux distro used in Amazon EC2)

https://aws.amazon.com/amazon-linux-ami/security-bulletins/ALAS-2014-320/

Issue Overview: A missing bounds check was found in the way OpenSSL handled TLS heartbeat extension packets. This flaw could be used to reveal up to 64k of memory from a connected client or server.

Affected Versions: Any Amazon Linux AMI on which openssl 1.0.1 is installed, which is any Amazon Linux AMI 2013.03 or later, and any Amazon Linux AMI that has upgraded to 2013.03 or later. OpenSSL is installed by default on the Amazon Linux AMI.

Affected Packages: openssl

Issue Correction: Run yum update openssl to update your system. Once the new package is installed, it is required that you either manually restart all services that are using openssl, or that you reboot your instance. While the new package is still named openssl-1.0.1e, it does contain the fix for CVE-2014-0160.

New Packages: i686:

openssl-1.0.1e-37.66.amzn1.i686

openssl-static-1.0.1e-37.66.amzn1.i686

openssl-perl-1.0.1e-37.66.amzn1.i686

openssl-devel-1.0.1e-37.66.amzn1.i686

openssl-debuginfo-1.0.1e-37.66.amzn1.i686

x86_64:

openssl-devel-1.0.1e-37.66.amzn1.x86_64

openssl-1.0.1e-37.66.amzn1.x86_64

openssl-debuginfo-1.0.1e-37.66.amzn1.x86_64

openssl-perl-1.0.1e-37.66.amzn1.x86_64

openssl-static-1.0.1e-37.66.amzn1.x86_64
Garreth McDaid
  • 3,399
  • 26
  • 41