SSH permission denied in a closed system with twin machines

Question

I have a network with 3 machines: one server (192.168.200.1) connected to the outside world and two twin machines (192.168.200.2 and 192.168.200.3) connected to server through a switch (default setting, no filtering at all).

All machines have Debian, port 22 open and I want to access through ssh. I can access from server to twin2 using ssh but if I try to access twin1 it deny because wrong password (but it is the correct) All machines have the same root passwd without any firewall

If I connect from server to twin2 (using ssh) and then from twin2 to twin1 it works! twin1 cannot see server.

/etc/ssh/sshd_config and route configuration is exactly the same in twin machines

route -n
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.200.1   0.0.0.0         UG    0      0        0 eth0
192.168.200.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

The server see both machines:

root@server:# traceroute 192.168.200.2
traceroute to 192.168.200.2 (192.168.200.2), 30 hops max, 60 byte packets
 1  twin1 (192.168.200.2)  0.036 ms  0.012 ms  0.010 ms
root@server:# traceroute 192.168.200.3
traceroute to 192.168.200.3 (192.168.200.3), 30 hops max, 60 byte packets
 1  twin2 (192.168.200.3)  0.140 ms  0.122 ms  0.115 ms

twin1 see server:

root@twin1:~# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 60 byte packets
 1  server (192.168.200.1)  0.106 ms  0.087 ms  0.074 ms

twin2 don't see the server:

root@twin2:~# traceroute 192.168.200.1
traceroute to 192.168.200.1 (192.168.200.1), 30 hops max, 60 byte packets
 1  * * *
 ....
 30 * * *

SSH conection output

root@server:~# ssh -v twin2 (this work and even don't ask for passwd) 
OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to twin2 [192.168.200.3] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u1
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u1 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr [e-mail] none
debug1: kex: client->server aes128-ctr [e-mail] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA d4:0c:32:b6:84:2f:fa:89:cf:40:fd:b0:b8:47:d4:d6
debug1: Host 'twin2' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:3
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to twin2 ([192.168.200.3]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [e-mail]
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8

(this don't work)

root@server:~# ssh -v twin1
OpenSSH_6.7p1 Debian-5, OpenSSL 1.0.1k 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to twin1 [192.168.200.2] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5
debug1: Remote protocol version 2.0, remote software version         OpenSSH_6.7p1 Debian-5
debug1: match: OpenSSH_6.7p1 Debian-5 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr [e-mail] none
debug1: kex: client->server aes128-ctr [e-mail] none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 19:9c:99:3f:a1:aa:4b:86:bc:2d:6f:9c:66:29:d9:07
debug1: Host 'twin1' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug1: Trying private key: /root/.ssh/id_ed25519
debug1: Next authentication method: password
root@twin1's password: 
debug1: Authentications that can continue: publickey,password
Permission denied, please try again

If I check the /var/log/auth.log in twin1 when server try to connect through ssh nothing is registered! (as the ssh connection never reach it)

Any help will be welcome

Answer 1

It sounds to me like the problem is not with SSH, rather its with the underlying network.

I think you will find that the issue is with your switch - it almost looks looks like TWIN1 is configured behind a port using NAT - First thing I would do is replace the switch and see what happens.

Second thing I would do is break out TCPDump, and actually dump what is happening on port 22 on each of the machines when SSH'ing between them. This will shed light on whether packets are arriving and on their source and destination IP addresses.

I'd also take a careful look at the MAC addresses on each interface - I wonder if, in cloning the machines you have cloned the MAC addresses - this could explain some of your problems - of-course, each interface needs a unique MAC address - and it is possible to redefine a MAC address in software.