SSL stopped working

1

My SSL suddenly stopped serving my site. It could have been because I installed nginx on my server, then uninstalled it using apt-get purge nginx*

For about 30 minutes it was serving pages instead of Apache. After I purged it, Apache went back to normal... almost. It still won't serve SSL content, but it will serve the other virtual hosts on the server.

I also ensured that my ports 80 and 443 are open.

I have 3 sites enabled right now. When I start Apache I don't get any error messages. Here are the contents of the 000-default.conf, default-ssl.conf, and elaan.conf

000-default.conf

<VirtualHost *:80>

ServerAdmin joseph.lee.esl@gmail.com
DocumentRoot /var/www/html/elaan/

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

default-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost _default_:443>
    ServerAdmin joseph.lee.esl@gmail.com
    ServerName elaan.com.tw:443

    DocumentRoot /var/www/html/elaan

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    SSLEngine on

    SSLCertificateFile /etc/apache2/ssl/ssl.crt
    SSLCertificateKeyFile /etc/apache2/ssl/private.key
        SSLCertificateChainFile /etc/apache2/ssl/sub.class1.server.ca.pem


    <FilesMatch "\.(cgi|shtml|phtml|php)$">
            SSLOptions +StdEnvVars
    </FilesMatch>
    <Directory /usr/lib/cgi-bin>
            SSLOptions +StdEnvVars
    </Directory>

    BrowserMatch "MSIE [2-6]" \
            nokeepalive ssl-unclean-shutdown \
            downgrade-1.0 force-response-1.0
    # MSIE 7 and newer should be able to use keepalive
    BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

</VirtualHost>
</IfModule>

elaan.conf

<VirtualHost *:80>

ServerAdmin elaan.english@gmail.com
ServerName elaan.com.tw
ServerAlias www.elaan.com.tw
DocumentRoot /var/www/html/elaan

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

all these sites are in the site-enabled folder when I run service apache2 reload, and there are no error messages in the error.log file.

Can anyone offer any ideas how to further test why Apache is not serving from https:// ?

# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1003/sshd       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1335/sendmail: MTA:
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1104/mysqld     
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      1335/sendmail: MTA:
tcp6       0      0 :::22                   :::*                    LISTEN      1003/sshd       
tcp6       0      0 :::443                  :::*                    LISTEN      1443/apache2    
tcp6       0      0 :::80                   :::*                    LISTEN      1443/apache2 

ports.conf

Listen 80

<IfModule ssl_module>
    Listen 443
</IfModule>

<IfModule mod_gnutls.c>
    Listen 443
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

ssl_engine.log

[Tue Jul 14 12:18:18.811639 2015] [ssl:debug] [pid 11551] ssl_engine_pphrase.c(181): AH02199: SSL not enabled on vhost 127.0.1.1:80, skipping SSL setup
[Tue Jul 14 12:18:18.811892 2015] [ssl:debug] [pid 11551] ssl_engine_pphrase.c(239): AH02202: Init: Read server certificate from '/etc/apache2/ssl/ssl.crt'
[Tue Jul 14 12:18:18.812048 2015] [ssl:info] [pid 11551] AH01887: Init: Initializing (virtual) servers for SSL
[Tue Jul 14 12:18:18.812589 2015] [ssl:info] [pid 11551] AH01876: mod_ssl/2.4.7 compiled against Server: Apache/2.4.7, Library: OpenSSL/1.0.1f
[Tue Jul 14 12:18:18.836129 2015] [ssl:debug] [pid 11552] ssl_engine_pphrase.c(181): AH02199: SSL not enabled on vhost 127.0.1.1:80, skipping SSL setup
[Tue Jul 14 12:18:18.840838 2015] [ssl:debug] [pid 11552] ssl_engine_pphrase.c(239): AH02202: Init: Read server certificate from '/etc/apache2/ssl/ssl.crt'
[Tue Jul 14 12:18:18.841171 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(389): AH00821: shmcb_init allocated 512000 bytes of shared memory
[Tue Jul 14 12:18:18.841180 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(405): AH00822: for 511912 bytes (512000 including header), recommending 32 subcaches, 88 indexes each
[Tue Jul 14 12:18:18.841185 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(438): AH00824: shmcb_init_memory choices follow
[Tue Jul 14 12:18:18.841189 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(440): AH00825: subcache_num = 32
[Tue Jul 14 12:18:18.841193 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(442): AH00826: subcache_size = 15992
[Tue Jul 14 12:18:18.841196 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(444): AH00827: subcache_data_offset = 2128
[Tue Jul 14 12:18:18.841210 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(446): AH00828: subcache_data_size = 13864
[Tue Jul 14 12:18:18.841214 2015] [socache_shmcb:debug] [pid 11552] mod_socache_shmcb.c(448): AH00829: index_num = 88
[Tue Jul 14 12:18:18.841286 2015] [socache_shmcb:info] [pid 11552] AH00830: Shared memory socache initialised
[Tue Jul 14 12:18:18.841290 2015] [ssl:info] [pid 11552] AH01887: Init: Initializing (virtual) servers for SSL
[Tue Jul 14 12:18:18.841851 2015] [ssl:info] [pid 11552] AH01876: mod_ssl/2.4.7 compiled against Server: Apache/2.4.7, Library: OpenSSL/1.0.1f
[Tue Jul 14 12:18:18.844054 2015] [mpm_prefork:notice] [pid 11552] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.9 OpenSSL/1.0.1f configured -- resuming normal operations
[Tue Jul 14 12:18:18.844069 2015] [mpm_prefork:info] [pid 11552] AH00164: Server built: Mar 10 2015 13:05:59
[Tue Jul 14 12:18:18.844085 2015] [core:notice] [pid 11552] AH00094: Command line: '/usr/sbin/apache2'
[Tue Jul 14 12:18:18.844090 2015] [mpm_prefork:debug] [pid 11552] prefork.c(995): AH00165: Accept mutex: fcntl (default: sysvsem)

Firewall status:

 To                         Action      From
 --                         ------      ----
[ 1] 22                         ALLOW IN    Anywhere
[ 2] 80                         ALLOW IN    Anywhere
[ 3] 443/tcp                    ALLOW IN    Anywhere
[ 4] 443/tcp                    ALLOW OUT   Anywhere (out)
[ 5] 587                        ALLOW OUT   Anywhere (out)
[ 6] 22 (v6)                    ALLOW IN    Anywhere (v6)
[ 7] 80 (v6)                    ALLOW IN    Anywhere (v6)
[ 8] 443/tcp (v6)               ALLOW IN    Anywhere (v6)
[ 9] 443/tcp (v6)               ALLOW OUT   Anywhere (v6) (out)
[10] 587 (v6)                   ALLOW OUT   Anywhere (v6) (out)

jonnyjandles

Posted 2015-07-13T16:12:35.017

Reputation: 123

You say that you've checked that ports 80 and 443 are open, but have you verified that the apache PID is actually listening to port 443 via netstat or something else? – ssnobody – 2015-07-13T22:39:49.113

I've posted the results of netstat in the question above. To me is seems that it's listening on 443, but I"m not sure because the local address and foreign address seem empty. Also, it seems that it's only listening to tcp6 on those ports. – jonnyjandles – 2015-07-14T16:01:24.620

Do connections to the server timeout, result in a different error, or something else entirely? – ssnobody – 2015-07-14T19:37:47.123

The info bar at the bottom of the browser says connecting. Then, eventually the connection times out. There are no logs in the access.log. Is a tcp6 connection also good enough to work with tcp v4? – jonnyjandles – 2015-07-14T22:14:31.910

The tcp6 connection should be enough to accept both IPv4 and IPv6 connections if mapping was enabled. See http://httpd.apache.org/docs/2.2/bind.html . If you are getting time-outs it indicates you were not able to reach the listening apache. This would likely be due to firewall or routing issues. Try to telnet from localhost to port 443 and see if the connection shows up in netstat, if it works, then try the same from remote.

– ssnobody – 2015-07-14T22:25:49.543

I was able to connect via telnet to locahost 443, and $ openssl s_client -connect localhost:443. I posted the firewall port configuration status above in the post. I was not able to connect from my machine. I was able to connect via port 80 from my local machine. – jonnyjandles – 2015-07-14T23:53:12.663

So it looks like you've narrowed your problem to a firewall or routing issue and not an apache issue at all. You might want to stop apache and troubleshoot with something like netcat to verify connectivity to the server. Your firewall config looks reasonable but I thought I'd note that the entries for 443 and 80 are not the same (e.g. no OUT entries for port 80), so if port 80 is working and 443 is not, you might want to have 443 use the same firewall settings at port 80. – ssnobody – 2015-07-14T23:59:49.547

I was also able to get the index page via GET / HTTP/1.1. I can try to shut down port 443 out, but everything was working fine before I installed nginx and then uninstalled it. :( – jonnyjandles – 2015-07-15T00:08:57.553

Interesting, so I just shut down the firewall all together, and it suddenly popped back up with https from my remote machine. – jonnyjandles – 2015-07-15T00:10:56.083

OK, you got it. I disabled the rule for allowing 443 out and the page was back up. :) Many thanks! – jonnyjandles – 2015-07-15T00:14:21.343

Glad you got it working again! Also glad to see you've posted the answer. Feel free to accept it after 48 hours. – ssnobody – 2015-07-15T00:16:19.373

Hello, just a quick note that you should be able to accept your own answer now and should likely do so. – ssnobody – 2015-07-17T17:49:41.360

Answers

1

Finally, we tracked the issue down to the firewall. I was able to connect via local host command line but not from remote machines. For some reason, disabling the ALLOW 443 OUT rule managed to fix the issue.

jonnyjandles

Posted 2015-07-13T16:12:35.017

Reputation: 123