ubuntu upgrade went wrong: SSH connection closes after login - no access to terminal

Question

After an upgrade from Ubuntu 14.04 to 16.04 on my VPS done through SSH, SSH broke. do-release-upgrade always ended up with some package error, so I left it as it was. Client was left open, after ~12 hours later when I closed the connection, and wanted to reopen it, I realized that its broken: after a successful login the server terminates the connection with message Exit status 254

I did a ssh -vvv USERNAME@XXX.XXX.XXX.XXX, here are the results:

$ ssh -vvv USERNAME@XXX.XXX.XXX.XXX
OpenSSH_8.1p1, OpenSSL 1.1.1d  10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname XXX.XXX.XXX.XXX is address
debug2: ssh_connect_direct
debug1: Connecting to XXX.XXX.XXX.XXX [XXX.XXX.XXX.XXX] port 22.
debug1: Connection established.
debug1: identity file /c/Users/USER/.ssh/id_rsa type -1
debug1: identity file /c/Users/USER/.ssh/id_rsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_dsa type -1
debug1: identity file /c/Users/USER/.ssh/id_dsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_ecdsa type -1
debug1: identity file /c/Users/USER/.ssh/id_ecdsa-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_ed25519 type -1
debug1: identity file /c/Users/USER/.ssh/id_ed25519-cert type -1
debug1: identity file /c/Users/USER/.ssh/id_xmss type -1
debug1: identity file /c/Users/USER/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13 pat OpenSSH_6.6.1* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to XXX.XXX.XXX.XXX:22 as 'USERNAME'
debug3: hostkeys_foreach: reading file "/c/Users/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/USER/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from XXX.XXX.XXX.XXX
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,aes256-cbc,aes192-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:MjRBEXmM8Lpl5zUEXnixUr7GgOxV1ka9UXI5qhB8uhE
debug3: hostkeys_foreach: reading file "/c/Users/USER/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /c/Users/USER/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from XXX.XXX.XXX.XXX
debug1: Host 'XXX.XXX.XXX.XXX' is known and matches the ECDSA host key.
debug1: Found key in /c/Users/USER/.ssh/known_hosts:2
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /c/Users/USER/.ssh/id_rsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_dsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_ecdsa
debug1: Will attempt key: /c/Users/USER/.ssh/id_ed25519
debug1: Will attempt key: /c/Users/USER/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /c/Users/USER/.ssh/id_rsa
debug3: no such identity: /c/Users/USER/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_dsa
debug3: no such identity: /c/Users/USER/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_ecdsa
debug3: no such identity: /c/Users/USER/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_ed25519
debug3: no such identity: /c/Users/USER/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /c/Users/USER/.ssh/id_xmss
debug3: no such identity: /c/Users/USER/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
USERNAME@XXX.XXX.XXX.XXX's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to XXX.XXX.XXX.XXX ([XXX.XXX.XXX.XXX]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x48
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
Welcome to Ubuntu 16.04 LTS (GNU/Linux 3.13.0-042stab141.3 x86_64)

 * Documentation:  https://help.ubuntu.com/
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

No mail.
Last login: Wed Jun 16 21:58:00 2021 from MYHOST
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1)

debug3: send packet: type 1
debug3: fd 1 is not O_NONBLOCK
Connection to XXX.XXX.XXX.XXX closed.
Transferred: sent 2236, received 2512 bytes, in 0.3 seconds
Bytes per second: sent 6692.5, received 7518.6
debug1: Exit status 254

The problem is that SSH was my only option to access the server (self-managed VPS, no possibility to access a remote console or to get a snapshot), so support recommends destroying and rebuilding the server, which is not a friendly option.

I'm trying to save the day by somehow getting SSH access back, but can't get past the above error. I don't have any server-side logs. According to the stats available on the hosting side, Memory is available (RAM usage is ~16%),

I've tried to:

send ^C after logging in to intercept .bashrc or .bash_profile; it didnt work, and I'm not even sure that that's causing the problem
ssh USERNAME@XXX.XXX.XXX.XXX bash --noprofile --norc, same outcome
ssh USERNAME@XXX.XXX.XXX.XXX 'echo something >&2 I see debug1: Sending command: echo something >&2 in the log, but no result on the output
access the database: after rebooting the system using the Server Dashboard site MySQL server never got back online
ask for a remote terminal or a VM snapshot from the hosting company, didnt work out

All help is appreciated

They don't provide a remote console?! That's really rather ridiculous. — Michael Hampton, Jun 16 '21 at 22:53
@MichaelHampton That was my reaction as well... I should've checked this before starting to upgrade, now its the harder way to recover - if possible at all — Gardenee, Jun 16 '21 at 22:57
Oh, I just saw the godaddy tag. That explains everything. Do yourself a favor, get a backup of your VM and go anywhere else. — Michael Hampton, Jun 16 '21 at 22:59
@MichaelHampton cant even get a backup without SSH access :) — Gardenee, Jun 16 '21 at 23:07
What? That's utterly insane! They should be able to provide you a complete image of your VPS. — Michael Hampton, Jun 16 '21 at 23:26
Just to make sure, I contacted them once again with this exact question; they've told me that it cant be done. I made it clear that I'm willing to pay for such a service, still no — Gardenee, Jun 17 '21 at 08:29

ubuntu upgrade went wrong: SSH connection closes after login - no access to terminal

0 Answers0