Unable to SSH into ec2 after reboot

Question

I hoping that this is the right place to post this. Today I was locked out of my ec2 after a reboot. This is what it says when I try to ssh:

OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to xx.xx.xx.xx [xx.xx.xx.xx] port 22.
debug1: connect to address xx.xx.xx.xx port 22: Connection refused
ssh: connect to host xx.xx.xx.xx port 22: Connection refused

I've went through the following questions:

https://stackoverflow.com/questions/14026148/running-ec2-instance-suddenly-refuses-ssh-connection

Amazon EC2 - No SSH After Reboot, Connection Refused

https://www.youtube.com/watch?v=_P29ZHu_feU

I have already tried detaching my volume in attempt to fix sshd_config but that doesn't seem to rectify the problem. Thankfully I have an AMI before I did anything today, and I re-launched my instance and it worked fine. However, the moment I run yum update and rebooted my instance, I am locked out again.

These are the things that have been updated today:

Dependencies Resolved

================================================================================================================================================================
 Package                            Arch                   Version                                      Repository                                         Size
================================================================================================================================================================
Installing:
 kernel                             x86_64                 2.6.32-642.3.1.el6                           rhui-REGION-rhel-server-releases                   32 M
Updating:
 autofs                             x86_64                 1:5.0.5-123.el6_8                            rhui-REGION-rhel-server-releases                  721 k
 elasticsearch                      noarch                 2.3.5-1                                      elasticsearch-2.x                                  26 M
 httpd                              x86_64                 2.2.15-54.el6_8                              rhui-REGION-rhel-server-releases                  833 k
 httpd-tools                        x86_64                 2.2.15-54.el6_8                              rhui-REGION-rhel-server-releases                   78 k
 initscripts                        x86_64                 9.03.53-1.el6_8.1                            rhui-REGION-rhel-server-releases                  947 k
 java-1.7.0-openjdk                 x86_64                 1:1.7.0.111-2.6.7.2.el6_8                    rhui-REGION-rhel-server-releases                   26 M
 kernel-firmware                    noarch                 2.6.32-642.3.1.el6                           rhui-REGION-rhel-server-releases                   28 M
 kernel-headers                     x86_64                 2.6.32-642.3.1.el6                           rhui-REGION-rhel-server-releases                  4.4 M
 libcgroup                          x86_64                 0.40.rc1-18.el6_8                            rhui-REGION-rhel-server-releases                  130 k
 libipa_hbac                        x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  112 k
 libsss_idmap                       x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  118 k
 libtiff                            x86_64                 3.9.4-18.el6_8                               rhui-REGION-rhel-server-releases                  345 k
 libxml2                            x86_64                 2.7.6-21.el6_8.1                             rhui-REGION-rhel-server-releases                  805 k
 libxml2-python                     x86_64                 2.7.6-21.el6_8.1                             rhui-REGION-rhel-server-releases                  325 k
 mdadm                              x86_64                 3.3.4-1.el6_8.5                              rhui-REGION-rhel-server-releases                  348 k
 mod_ssl                            x86_64                 1:2.2.15-54.el6_8                            rhui-REGION-rhel-server-releases                   97 k
 nfs-utils                          x86_64                 1:1.2.3-70.el6_8.1                           rhui-REGION-rhel-server-releases                  335 k
 nss-softokn                        x86_64                 3.14.3-23.3.el6_8                            rhui-REGION-rhel-server-releases                  262 k
 nss-softokn-freebl                 x86_64                 3.14.3-23.3.el6_8                            rhui-REGION-rhel-server-releases                  168 k
 perf                               x86_64                 2.6.32-642.3.1.el6                           rhui-REGION-rhel-server-releases                  4.6 M
 php56w                             x86_64                 5.6.24-1.w6                                  webtatic                                          2.7 M
 php56w-cli                         x86_64                 5.6.24-1.w6                                  webtatic                                          2.6 M
 php56w-common                      x86_64                 5.6.24-1.w6                                  webtatic                                          1.2 M
 php56w-gd                          x86_64                 5.6.24-1.w6                                  webtatic                                          144 k
 php56w-mbstring                    x86_64                 5.6.24-1.w6                                  webtatic                                          944 k
 php56w-mcrypt                      x86_64                 5.6.24-1.w6                                  webtatic                                           25 k
 php56w-mysql                       x86_64                 5.6.24-1.w6                                  webtatic                                          111 k
 php56w-pdo                         x86_64                 5.6.24-1.w6                                  webtatic                                           94 k
 php56w-process                     x86_64                 5.6.24-1.w6                                  webtatic                                           35 k
 php56w-tidy                        x86_64                 5.6.24-1.w6                                  webtatic                                           29 k
 php56w-xml                         x86_64                 5.6.24-1.w6                                  webtatic                                          148 k
 python-libipa_hbac                 x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  107 k
 python-sssdconfig                  noarch                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  141 k
 samba4-libs                        x86_64                 4.2.10-7.el6_8                               rhui-REGION-rhel-server-releases                  4.4 M
 sssd                               x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  108 k
 sssd-ad                            x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  211 k
 sssd-client                        x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  164 k
 sssd-common                        x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  1.1 M
 sssd-common-pac                    x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  147 k
 sssd-ipa                           x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  277 k
 sssd-krb5                          x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  143 k
 sssd-krb5-common                   x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  163 k
 sssd-ldap                          x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  209 k
 sssd-proxy                         x86_64                 1.13.3-22.el6_8.4                            rhui-REGION-rhel-server-releases                  137 k
 tar                                x86_64                 2:1.23-15.el6_8                              rhui-REGION-rhel-server-releases                  810 k
 tzdata                             noarch                 2016f-1.el6                                  rhui-REGION-rhel-server-releases                  452 k
 tzdata-java                        noarch                 2016f-1.el6                                  rhui-REGION-rhel-server-releases                  180 k
 yum                                noarch                 3.2.29-75.el6_8                              rhui-REGION-rhel-server-releases                  1.0 M
Removing:
 kernel                             x86_64                 2.6.32-573.18.1.el6                          @rhui-REGION-rhel-server-releases                 126 M

Transaction Summary
================================================================================================================================================================
Install       1 Package(s)
Upgrade      48 Package(s)
Remove        1 Package(s)

Total download size: 144 M

So my question:

How can I make sure that ssh has started after my reboot? I am locked out of the server.
I highly suspect that one of the packages has errors, but I have no idea which. Any way I can pin-point?

Check logs files in detaching volume – Federico Sierra Aug 04 '16 at 18:25 — Federico Sierra, Aug 04 '16 at 18:25

score 1 · Answer 1 · answered Aug 06 '16 at 09:21

A connection refused error probably means the sshd service is not started and so is not listening on port 22 as it should.

Before rebooting, after updating your packages, check that OpenSSH is still running by typing telnet localhost 22. If the telnet package is not installed, install it using yum install telnet. If you see an OpenSSH banner, it means it is running, quit telnet by typing ^] (control-]). If it is not, it means the update process went bad. Try the following to see the state of the sshd service:

Non-systemd (pre-CentOS 7)

# chkconfig --list sshd

If you see it as off, enable it using the following command:

# chkconfig sshd on

And start it:

# service sshd start

Re-try telnet. If it is still not responding, checkout /var/log/messages and dig in the reason why it refused starting.

With systemd (CentOS 7 and up):

# systemctl status sshd -l

This command should tell you what occurred when last tried to start sshd. If it shows that sshd is not enabled, it means it will not start at boot time, enable it using the following:

# systemctl enable sshd

And start it if it doesn't respond:

# systemctl start sshd

Check again how did the starting process went using systemctl status sshd, errors happening during the process should be displayed in your console.

As for your question about debugging yum update process, you might want to rise debuglevel from 2 (default) to 10 (the higher) in /etc/yum.conf.

Unable to SSH into ec2 after reboot

1 Answers1

Non-systemd (pre-CentOS 7)

With systemd (CentOS 7 and up):