I have 12 node of bare-metal CoreOS cluster (on SuperMicro Blade). Every nodes was installed with same image and cloud-config. (Stable 717.3.0)
The problem is that I fail to connect sshd frequently, means that sometimes I can connect to sshd.
So I tried it with ssh -v option and found some differences. What I noticed in fail case is that remote response of protocol is 'dropbear_2013.60', not 'OpenSSH_6.7'.
Another weird thing I found is that, when I did port scanning to the node, usually it report there's only one open port, 22/tcp for ssh like expected but if did it again and again, sometimes it report like below:
# nmap XXX.XXX.XXX.112
Starting Nmap 5.21 ( http://nmap.org ) at 2015-07-24 23:48 KST
Nmap scan report for xxx.xxx.xxx (XXX.XXX.XXX.112)
Host is up (0.0026s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
5900/tcp open vnc
MAC Address: 00:25:XX:XX:XX:XX (Super Micro Computer)
After quick duckduckgoing, I guess that ports are for SuperMicro's remote management service. I tried to disable it from BIOS configuration but couldn't find appropriate menu.
I opened it with a browser and it's not always working. Sometime I can get some login page, sometime it's simply not respondable. Does this SuperMicro things interfere sshd connecting? or any other reason? I attached ssh logs below.
This is a fail case log:
$ ssh -i ~/.ssh/se.pem -l core -v XXX.XXX.XXX.112
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/scari/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to XXX.XXX.XXX.112 [XXX.XXX.XXX.112] port 22.
debug1: Connection established.
debug1: identity file /Users/scari/.ssh/se.pem type -1
debug1: identity file /Users/scari/.ssh/se.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
ssh_exchange_identification: read: Connection reset by peer
and this is another fail case:
$ ssh -i ~/.ssh/se.pem -l core -v XXX.XXX.XXX.112
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/scari/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to XXX.XXX.XXX.112 [XXX.XXX.XXX.112] port 22.
debug1: Connection established.
debug1: identity file /Users/scari/.ssh/se.pem type -1
debug1: identity file /Users/scari/.ssh/se.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version dropbear_2013.60
debug1: no match: dropbear_2013.60
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: RSA 95:0d:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:f4:3e
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
95:0d:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:f4:3e.
Please contact your system administrator.
Add correct host key in /Users/scari/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/scari/.ssh/known_hosts:15
RSA host key for XXX.XXX.XXX.112 has changed and you have requested strict checking.
Host key verification failed.
and this is success case:
$ ssh -i ~/.ssh/se.pem -l core -v XXX.XXX.XXX.112
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/scari/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to XXX.XXX.XXX.112 [XXX.XXX.XXX.112] port 22.
debug1: Connection established.
debug1: identity file /Users/scari/.ssh/se.pem type -1
debug1: identity file /Users/scari/.ssh/se.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7
debug1: match: OpenSSH_6.7 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-sha1-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-sha1-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<2048<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 56:07:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:fc:8d
debug1: Host 'XXX.XXX.XXX.112' is known and matches the RSA host key.
debug1: Found key in /Users/scari/.ssh/known_hosts:15
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/scari/.ssh/se.pem
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to XXX.XXX.XXX.112 ([XXX.XXX.XXX.112]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Requesting authentication agent forwarding.
debug1: Sending environment.
debug1: Sending env LC_ALL = ko_KR.UTF-8
debug1: Sending env LANG = ko_KR.UTF-8
Last login: Fri Jul 24 13:56:14 2015 from XXX.XXX.XXX.153
CoreOS stable (717.3.0)
I also checked if sshd is running. Looks like it running.
core112 ~ # systemctl list-units sshd*
UNIT LOAD ACTIVE SUB DESCRIPTION
sshd-keygen.service loaded active exited Generate sshd host keys
sshd@38-XXX.XXX.XXX.112:22-XXX.XXX.XXX.153:57197.service loaded active running OpenSSH per-connection server daemon (XXX.XXX.XXX.153:57197)
sshd.socket loaded active listening OpenSSH Server Socket
