Unable to SSH to LXC Ubuntu container

Question

I'm trying to connect from a CentOS host to my LXC Ubuntu container and it fails with this error:

packet_write_wait: Connection to 192.168.122.92 port 22: Broken pipe

The CentOS I'm using is:

CentOS Linux release 7.4.1708 (Core)

The Ubuntu version inside the container is:

Ubuntu 12.04.5 LTS

I have tried doing the following

On the host server:

Modify /etc/ssh/sshd_config

ClientAliveInterval 20
ClientAliveCountMax 5 

In the container (for root and ubuntu users):

Modify ~/.ssh/config

Host *
  ServerAliveInterval 30
  ServerAliveCountMax 5

This is the verbose output from SSH:

[root@ansible lxc]# ssh -C -vvv ubuntu@192.168.122.92
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /root/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "192.168.122.92" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 192.168.122.92 [192.168.122.92] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.10
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.10 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.122.92:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:5
debug3: load_hostkeys: loaded 1 keys from 192.168.122.92
debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[e-mail],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: [e-mail],[e-mail],[e-mail],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[e-mail],[e-mail],[e-mail],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: [e-mail],aes128-ctr,aes192-ctr,aes256-ctr,[e-mail],[e-mail],aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],[e-mail],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: [e-mail],zlib,none
debug2: compression stoc: [e-mail],zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]
debug2: MACs ctos: hmac-md5,hmac-sha1,[e-mail],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,[e-mail],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,[e-mail]
debug2: compression stoc: none,[e-mail]
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: [e-mail] compression: [e-mail]
debug1: kex: client->server cipher: aes128-ctr MAC: [e-mail] compression: [e-mail]
debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
debug3: send packet: type 30
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:0nGGO4e+fMOHUJPibVa+qPu8m1qm09IefeRJoCw+1kQ
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:5
debug3: load_hostkeys: loaded 1 keys from 192.168.122.92
debug1: Host '192.168.122.92' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:5
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: /root/.ssh/id_rsa (0x55d8bed534b0)
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug2: key: /root/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /root/.ssh/id_ed25519
debug3: no such identity: /root/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
ubuntu@192.168.122.92's password:
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (password).
Authenticated to 192.168.122.92 ([192.168.122.92]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting [e-mail]
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: send packet: type 1
packet_write_wait: Connection to 192.168.122.92 port 22: Broken pipe

Any idea?

Answer 1

[root@ansible lxc]# ssh -C -vvv ubuntu@192.168.122.92
Authenticated to 192.168.122.92 ([192.168.122.92]:22).
debug1: channel 0: new [client-session]
...
packet_write_wait: Connection to 192.168.122.92 port 22: Broken pipe

You are connecting to the remote system (your container) and authenticating. Then the connection terminates while setting up your interactive session on the remote system.

The simplest explanation is that the ssh server on the remote container is trying to invoke your shell for the remote system, and the shell invocation is failing. For example, the shell program may be missing, or some necessary shared library may be missing, or the shell process may be crashing as soon as it starts.

The client debug trace isn't going provide any more detail on what is happening. You would need to get into the container to troubleshoot this. Start by checking the ssh server log entries. I don't know about this container, but on a normal unix-like system, ssh log entries would normally be in one of the files in /var/logs.