cannot use localhost but 127.0.0.1 works

Question

My usecase is that I manage remote computers using reverse SSH tunneling. This works great as our customer enable the tunnel and I can then log on using it.

However, on one machine the tunneling stopped working.

ssh -fN -R 9999:localhost:22 user@host

After a lot of troubleshooting it turned out that using 127.0.0.1 instead of localhost did still work.

ssh -fN -R 9999:127.0.0.1:22 user@host

So, when connected to the target machine I noticed that eve more things did not work with localhost. For example ping.

ping -v localhost
ping: socket: Permission denied, attempting raw socket...
ping: socket: Permission denied, attempting raw socket...
ping: localhost: System error

While using 127.0.0.1 does work

ping -v 127.0.0.1
ping: socket: Permission denied, attempting raw socket...
ping: socket: Permission denied, attempting raw socket...
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.022 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.017 ms

sshd gave me equally useless error messages

error: connect_to localhost: unknown host (Bad file descriptor)
error: connect_to localhost: unknown host (Bad file descriptor)
error: connect_to localhost: unknown host (System error)
error: connect_to localhost: unknown host (System error)

Naturally I looked to the /etc/hosts file. But it looks ok to me.

cat /etc/hosts
#
# /etc/hosts: static lookup table for host names
#

#<ip-address>   <hostname.domain.org>   <hostname>
127.0.0.1   localhost.localdomain   localhost
::1     localhost.localdomain   localhost

# End of file

Also, the lookup in the hosts file does work to some degree at least. Because trying for example ping localhostq gives me the usual "unknown host" reply.

Running arch linux on target machine. Up to date as of 2016-06-23 (also another similar computer with the exact same arch version does not have this problem).

EDIT (2016-06-23 15:30):

Iptables (very relaxed atm)

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Not using nftables at all.

Hosts from /etc/nsswitch.conf:

cat /etc/nsswitch.conf | grep hosts
hosts: = files mdns4_minimal [NOTFOUND=return] dns mdns4

[EDIT: 2016-06-23 15:45] Actually, on the machine that does work I have this:

cat /etc/nsswitch.conf | grep hosts
hosts: files dns myhostname

Very surprised to see the lines differ since this file isn't something I have changed. Nor does I know what it means :)

Check IPv6 settings on the problem machine and see if they differ. — Tim Brigham, Jun 23 '16 at 13:13
Not that I can see. Also, ping localhost -4 gives the same error — tkarls, Jun 23 '16 at 13:23
updated the question with ip/nftables rules and hosts line from /etc/nsswitch.conf — tkarls, Jun 23 '16 at 13:40
First add another alias with 127.0.0.1 and give the output with ping and do _chmod +s /bin/ping_ — Nullpointer, Jun 23 '16 at 13:55

score 0 · Accepted Answer · answered Jun 23 '16 at 14:08

Turned out it was the /etc/nsswitch file

See the problem?

cat /etc/nsswitch.conf | grep hosts
hosts: = files mdns4_minimal [NOTFOUND=return] dns mdns4

A '=' sign incorrectly in the script. This was done in a custom package of ours (not written by me, blame allocation FTW!). that simply wasn't installed on the one that worked but was installed on the one that didn't.

Now I can correct the packet (used for automatically configure avahi) and things should go back to normal.

Many many thanks to all that asked follow up questions. Especially @7171u that asked the right one :)

cannot use localhost but 127.0.0.1 works

1 Answers1