RedHat Linux - ssh shows "connection refused" using one user and also root, but another user can connect

From server Y, user A tries to ssh to server X and gets Connection Refused. From server Y, user B tries to ssh to server X and is allowed through. From server Y, root tries to ssh and gets Connection Refused. The issue is 100% repeatable and 100% consistent.

The first two would suggest an issue with the ACL, but I'm told the settings are the same for both users although I don't have permission to view them. They have also been deleted and readded without success. However, root isn't affected by the ACL and also cannot connect.

There are two servers with this problem, and many other servers with identical settings in ifconfig (except IP addresses etc) and routing tables that do not suffer this issue.

A tcpdump on server X looking for connections from server Y shows nothing when user A tries to connect, but produces reams of information when user B tries to connect. This shows that server X is receiving nothing when user A is trying to ssh.

All servers are running RedHat

I'm assuming the problem is on server Y, but what should I look for? Could a username based connectivity issue appear in firewalls in between the servers, or are such firewalls not based on username?

Edit - Both users A and B can ssh from server Y to many other servers in the group. It is just server X that has the problem. Ssh-ing does not use RSA, although logging in to server Y in the first place does.

mavdo

Posted 2014-07-10T17:41:29.043

Reputation: 11

I'm guessing that since user B is "allowed through" that you're using RSA authentication, is that correct? – Ben Plont – 2014-07-10T20:15:34.940

mavdo and @Mavdo – if you are the same person, in the future please use your registered account (Mavdo) exclusively. That way you will be able to edit your own posts without needing to go through the review process. Also, when you ask a question and then provide additional, clarifying information that is legitimately part of the question, just go straight for the [Edit]. That sort of thing belongs in the question, not in a comment. – Scott – 2014-07-10T22:28:48.087

Apologies - I only created my account after I had posted. I had thought that I could link my post back to my account. But I will absolutely follow those guidelines in future, now I have an account. – mavdo – 2014-07-11T08:09:10.127

Answers

Check user A's .ssh/config file. He may have a Host entry in that file for the hostname at issue, which is causing ssh to try a different port or IP address for that hostname. For example, if .ssh/config had these lines:

Host a.example.com
    Hostname b.example.com
    Port 42

Then running "ssh a.example.com" would try try to connect to b.example.com port 42.

Another (unlikely) possibility is that user A is running a different "ssh" program for some reason, e.g. because his command path is different, and the alternate program is not behaving as expected.

Edit: Another unlikely possibility is that you have two hosts on the network answering to the same IP address. The failed connection requests are going to and being answered by the wrong server. This would cause connections to fail randomly, rather than consistently per user, so it doesn't really match the behavior that you describe.

Kenster

Posted 2014-07-10T17:41:29.043

Reputation: 5 474

I would add, double check the RSA key pair assigned to the "connection-refused" UIDs are configured correctly. – Ben Plont – 2014-07-10T20:13:20.063

"Connection refused" means he's not even making a TCP connection. The server that the packets are addressed to is refusing to accept the connection. An SSH key problem wouldn't cause that. – Kenster – 2014-07-10T20:21:17.290

@Kenster: In fact, the server that the packets are (presumably) addressed to is not even *seeing* them. So, I agree; it isn’t a key problem. – Scott – 2014-07-10T22:31:25.333

1To be pedantic, "connection refused" means the client host sent a SYN packet (connection request) and got a RST packet (refused) back. I know the OP says the intended target host isn't seeing the packets, but some server responded to the client. So maybe the connection request didn't go to the server that they were intended to go to. – Kenster – 2014-07-10T22:46:01.810

I had high hopes for this answer, but sadly there is no .ssh/config file for either user. When server B is up, both users can ping it and get the ssh results shown. When server B is taken down, neither user can ping it, but the connection refused messages are still shown for user A, while user B gets a Destination Host Unreachable. I think this suggests we are connecting to the right server. – mavdo – 2014-07-11T10:42:46.477

This has been resolved.

There were TWO access lists - the first that was commonly known about (ACL) which lists the servers that a user can connect to, and the second previously unknown list of servers that are only allowed access to certain users, ie the reverse of an ACL. With the servers removed from this restrictions list, access was allowed for all users.

Thanks for all your comments and guidance.

Mavdo

Posted 2014-07-10T17:41:29.043

Reputation: 23

How do these ACLs relate to a "connection refused" error message (which normally indicates a TCP connection refusal)? Consider if someone comes across this in ten years; try to make your answer as helpful as possible.

– a CVn – 2014-07-11T13:50:17.493

If I knew, I'd post. I'm guessing it is acting like an internal firewall, while the first list will allow user A to connect to only servers X, Y and Z, the second list, which seems to trump it, is only allowing user B to access server X and no one else. This firewall refuses any other type of connection. I was posting in an effort to make the post useful at all. It is all too often I find my exact question on a forum left unanswered for the last 4 years or so. – Mavdo – 2014-07-11T14:10:08.500

"all too often I find my exact question on a forum left unanswered for the last 4 years or so" That's my point. :) Even just saying which access lists you are talking about and how to access them would probably add tremendous value to this answer. Especially since TCP connection refused happens ages before any user details are exchanged, this answer is non-obvious at best. – a CVn – 2014-07-11T19:02:46.843

A firewall (whatever is implementing the ACLs) on the source host (serverY), could hypothetically identify the user trying to open a connection and apply rules based on that. And I've heard of firewalls simulating connection refused instead of just dropping the blocked packets. – Kenster – 2014-07-18T16:38:55.687

Incidentally, Mavdo, you should "accept" your answer so this question shows as answered. There should be a checkmark beneath the score for your answer. – Kenster – 2014-07-18T16:40:33.830

@Kenster - Sadly not because I posted originally as a guest and then created an account off the back of that. So annoyingly the guest user, mavdo, is not the same as my user, Mavdo, and of course I can't log in as the guest user any more. I live and learn. So I'm not really sure how guests can set their questions to "answered" at all. – Mavdo – 2014-07-22T11:14:07.253