Server still vulnerable to HeartBleed after Openssl update

Question

On a Centos 6.5 Minimal install, I have compiled Apache, PHP, and rpm installed Percona.

After updating OpenSSL days ago, my site that uses SSL on this server is vulnerable to Heartbleed somehow.

My Apache binary doesn't show that it is using libssl.so* at all, yet when I check it through https://filippo.io/Heartbleed/ it says my site is in fact vulnerable.

Am I checking something incorrectly?

[root@centos user]# ldd /usr/local/apache2/bin/httpd |grep -i ssl
[root@centos user]#

I've updated OpenSSL through yum (yum update openssl openssl-devel), I've recompiled Apache (four times since the openssl update), I've recompiled PHP (since the update), and I've reinstalled Percona.

I still receive the "Website is vulnerable" message, in spite of the fact that everything is up to date.

[root@centos user]# openssl version -a
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Apr  8 02:39:29 UTC 2014
platform: linux-x86_64
options:  bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DKRB5_MIT -m64 -DL_ENDIAN -DTERMIO -Wall -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wa,--noexecstack -DPURIFY -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
OPENSSLDIR: "/etc/pki/tls"
engines:  dynamic

I compile apache with the following options if this is of any help:

./configure --prefix=/usr/local/apache2 \
            --with-apxs2=/usr/local/apache2/bin/apxs \
            --with-mysql=/opt \
            --with-mysqli=/usr/bin/mysql_config \
            --with-gd --with-ttf=/opt \
            --with-xpm-dir=/opt \
            --with-freetype-dir=/opt \
            --enable-gd-native-ttf \
            --with-zlib-dir=/opt \
            --with-curl=/opt/include/curl \
            --enable-mbstring \
            --with-xsl=/opt \
            --with-libexpat-dir=/opt \
            --with-jpeg-dir=/opt \
            --with-png-dir=/opt \
            --enable-soap \
            --with-openssl \
            --with-ssl=/usr/local/openssl \
            --with-included-apr \
            --with-pcre=/usr/local/pcre/ \
            --with-mpm=worker \
            --enable-rewrite \
            --enable-ssl;

I have compiled a version of OpenSSL 1.0.1g to /usr/local/openssl using:

export CFLAGS="-fPIC";
openssl-1.0.1g]# ./config shared no-ssl2 no-ssl3 --openssldir=/usr/local/openssl

I have tried updating OpenSSL, Recompiling Apache, Recompiling PHP, Reinstalling Percona, rebooting, compiling OpenSSL (admittedly incorrectly and with no luck), and am still not able to patch this vulnerability.

I have rebooted the server already to ensure no libraries are still loaded in Ram.

Here is output regarding the libraries:

[root@centos php-5.5.9]# grep 'libssl.*(deleted)' /proc/*/maps
/proc/4497/maps:7faf80018000-7faf80079000 r-xp 00000000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf80079000-7faf80278000 ---p 00061000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf80278000-7faf8027c000 r--p 00060000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4497/maps:7faf8027c000-7faf80283000 rw-p 00064000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d735d000-7f21d73be000 r-xp 00000000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d73be000-7f21d75bd000 ---p 00061000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d75bd000-7f21d75c1000 r--p 00060000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4506/maps:7f21d75c1000-7f21d75c8000 rw-p 00064000 fd:00 1591034                    /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b000000-311b061000 r-xp 00000000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b061000-311b260000 ---p 00061000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b260000-311b264000 r--p 00060000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)
/proc/4594/maps:311b264000-311b26b000 rw-p 00064000 fd:00 1591034                        /usr/lib64/libssl.so.6 (deleted)

To see what version I am using:

[root@centos php-5.5.9]# strings /usr/lib64/libssl.so.6|grep -i openssl
OpenSSLDie
OPENSSL_cleanse
OPENSSL_DIR_read
OPENSSL_DIR_end
OPENSSL_init_library
OPENSSL_1.0.1
OPENSSL_1.0.1_EC
SSLv2 part of OpenSSL 1.0.1e-fips 11 Feb 2013
SSLv3 part of OpenSSL 1.0.1e-fips 11 Feb 2013
TLSv1 part of OpenSSL 1.0.1e-fips 11 Feb 2013
DTLSv1 part of OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL 1.0.1e-fips 11 Feb 2013
OPENSSL_DIR_read(&ctx, '
OPENSSL_DEFAULT_ZLIB
OPENSSL_malloc Error

I am unaware if OpenSSL 1.0.1e-fips 11 Feb 2013 relates to the above output where openssl version -a reports being OpenSSL 1.0.1e-fips 11 Feb 2013 but patch on the 8th, for heartbleed, or if its vulnerable.

On the same server, I am running Tomcat and GlassFish, but even when these are off, the server flags as vulnerable. Any ideas? Thank you in advance for any advice.

The output for `openssl version -a` shows you as having the vulnerable 1.0.1e. 1.0.1g is the first non-vulnerable version, I believe. — ceejayoz, Apr 15 '14 at 16:43
OpenSSL 1.0.1e was patched by RedHat and updated in the Yum repository to guard against heartbleed. 99/100 servers of mine have been able to successfully update by running yum update openssl openssl-devel and restarting apache. — DevOops, Apr 15 '14 at 17:16
Did you look at the filippo FAQ? https://filippo.io/Heartbleed/faq.html Does SSL Labs say the same thing? https://www.ssllabs.com/ssltest/index.html as there were (perhaps older) reports of false positives with the filippo test. — cjc, Apr 15 '14 at 17:38
It is still flagged as vulnerable by Qualys and other tests, which lead me to believe that it is in fact somehow still vulnerable. — DevOops, Apr 15 '14 at 18:19

score 6 · Accepted Answer · edited Apr 13 '17 at 12:37

6

You likely still have a process that is using the old library.

You can test it like so:

grep 'libssl.*(deleted)' /proc/*/maps

You could alternatively bounce the entire system.

https://unix.stackexchange.com/questions/123711/how-do-i-recover-from-the-heartbleed-bug-in-openssl

edited Apr 13 '17 at 12:37

Community

1

answered Apr 15 '14 at 19:53

spuder

1,695
2
25
42

THANK YOU SO MUCH !!!!!!! I was struggling with this for 5 days, and your advice led me to remove the library I found, and recompile. Now it works, and hackers aren't stealing everything I own!!! – DevOops Apr 15 '14 at 20:25

score 2 · Answer 2 · answered Apr 15 '14 at 16:58

2

You seem to compile apache with local version of openssl (--with-ssl=/usr/local/openssl), that might differ from the one you have upgraded through yum.

answered Apr 15 '14 at 16:58

phoops

2,073
4
18
23

Even when I compiled it with --with-ssl which defaults to the yum based one I presume, because it was before I downloaded and compiled openssl, it still flagged as vulnerable. – DevOops Apr 15 '14 at 17:15
Is it possible that is was false positive? – phoops Apr 15 '14 at 17:29
I would think so but other tools out there also report the same such as: https://www.ssllabs.com/ssltest/index.html This makes me think that it must indeed still be vulnerable, even though all libraries, binaries, and compile options match my other servers which are no longer vulnerable. – DevOops Apr 15 '14 at 18:18
Did you replace the SSL certificate? – phoops Apr 15 '14 at 18:29
I sure did, and when I serve it from another one of my servers in the same pool that was upgraded the exact same way, it shows as not vulnerable. – DevOops Apr 15 '14 at 18:54

Server still vulnerable to HeartBleed after Openssl update

2 Answers2