9

We currently have three webservers.

Servers one and two behave, but I'm having real problems with the third.

wget, curl and yum all fail to make their connections - that is, they all hang after resolving the host and attempting a connection.

Example (I have tried many different urls):

# wget http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
--2010-09-02 20:00:26--  http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
Resolving rpm.pbone.net... 85.14.85.4
Connecting to rpm.pbone.net|85.14.85.4|:80...

... hang

# curl -v http://rpm.pbone.net/index.php3/stat/4/idpl/13941547/dir/centos_5/com/httpd-2.2.3-43.el5.centos.i386.rpm.html
* About to connect() to rpm.pbone.net port 80
*   Trying 85.14.85.4...

... hang

#yum -d9 update
Loading "fastestmirror" plugin
Config time: 0.052
Running "init" handler for "fastestmirror" plugin
Yum Version: 3.2.22
COMMAND: yum -d9 update 
Installroot: /
Setting up Package Sacks
Running "postreposetup" handler for "fastestmirror" plugin
Loading mirror speeds from cached hostfile

... hang

but:

# ping rpm.pbone.net
PING gepard.pbone.net (85.14.85.4) 56(84) bytes of data.
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=1 ttl=49 time=449 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=2 ttl=49 time=448 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=3 ttl=49 time=444 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=4 ttl=49 time=445 ms
64 bytes from gepard.pbone.net (85.14.85.4): icmp_seq=5 ttl=49 time=457 ms

I'm far from a server expert, does anyone have any pointers on where to start solving this?

EDIT:

# netstat -lan | egrep LISTEN
tcp        0      0 0.0.0.0:941                 0.0.0.0:*                   LISTEN      
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      
tcp        0      0 :::80                       :::*                        LISTEN      
tcp        0      0 :::22                       :::*                        LISTEN      
unix  2      [ ACC ]     STREAM     LISTENING     7451   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     7678   @/tmp/fam-root-
unix  2      [ ACC ]     STREAM     LISTENING     5824   @/var/run/hald/dbus-3hUBzR5e9e
unix  2      [ ACC ]     STREAM     LISTENING     5087   /var/run/audispd_events
unix  2      [ ACC ]     STREAM     LISTENING     5825   @/var/run/hald/dbus-rDLe61j4bM
unix  2      [ ACC ]     STREAM     LISTENING     5545   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     5616   /var/run/sdp
unix  2      [ ACC ]     STREAM     LISTENING     5749   /var/run/pcscd.comm
unix  2      [ ACC ]     STREAM     LISTENING     5782   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     7075   /var/run/cups/cups.sock
unix  2      [ ACC ]     STREAM     LISTENING     7585   /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     7389   /dev/gpmctl

# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
RH-Firewall-1-INPUT  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            icmp any 
ACCEPT     esp  --  anywhere             anywhere            
ACCEPT     ah   --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns 
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:http 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited
Michael Robinson
  • 418
  • 2
  • 7
  • 19
  • Note that `iptables -L` doesn't give you as much info as `iptables-save` does; try it to get a better picture. – pjz Sep 02 '10 at 19:35

7 Answers7

3

You have firewall rules in place that are blocking port 80 outbound, or refusing the reciprocal inbound response. These can be software firewall rules which may be blocking port 80 specifically or all of TCP (PING is ICMP), check with:
iptables -L as ErikA pointed out above.
It could also be a hardware firewall issue - is the server behind a Cisco firewall? Consult your locate sysadmin. If you can curl from other machines, they have :80 open. It's also possible, but unlikely, that they're blocking you are their side, but if you can't curl anything (even google), it's your side.

James L
  • 5,915
  • 1
  • 19
  • 24
  • I can't curl anything, not even google. The servers are in China - but I've verified that `curl http://google.com` works on the other servers. – Michael Robinson Sep 02 '10 at 16:42
  • With it being China it adds another link in the "who could be blocking you" chain. Does curl http://google.cn work? Can you connect to other ports (eg. telnet to a 443?) – James L Sep 02 '10 at 17:08
  • Can't `curl http://google.cn/` on trouble server, can on the others. – Michael Robinson Sep 02 '10 at 17:11
  • telnet to port 443, 80 hang. – Michael Robinson Sep 02 '10 at 17:13
  • is that telnet to port 443 works, or they both hang? – James L Sep 02 '10 at 17:45
  • all attempts to telnet to the server hang – Michael Robinson Sep 02 '10 at 17:59
  • I can SSH to the server *from another server on that LAN* but not from any other machine. – Michael Robinson Sep 02 '10 at 18:01
  • Try to keep your responses below the comments they are responding to, swapping between makes it confusing. Try disabling IPTables to see if that fixes the issue (you have a Policy ACCEPT so you can easily do this). If not, and there's no hardware firewall, and no other server can connect to their server on their external IP, the problem is at their end. If you have SSH access to their server, you can tunnel: `SSH -L 1025:localhost:80 user@theirmachine` will allow you to connect to them on localhost:80 – James L Sep 02 '10 at 18:05
  • Sorry about the misplaced comments, and thank you for all your help so far. It's 6AM, brain fried. SSH -L 1025:localhost:80 user@HORRIBLE_SERVER_IP timesout when I attempt it from my local machine. – Michael Robinson Sep 02 '10 at 18:12
  • can you SSH normally? – James L Sep 02 '10 at 18:32
  • James - I give up. I've gone from being able to ssh from a site on it's LAN to not being able to connect at all - that part was my own stupid fault. I can't touch the server as it's 10,000KM away, I should have delegated this but I was in the *IT GETS DONE NOW* frame of mind. You helped the most, you get the tick. – Michael Robinson Sep 02 '10 at 22:04
2

Well, ping uses ICMP, whereas all of those HTTP clients use TCP port 80. Could that be blocked between your source and destination?

mfinni
  • 35,711
  • 3
  • 50
  • 86
  • Could be, but the other two servers don't hang in any of my examples – Michael Robinson Sep 02 '10 at 16:41
  • So that could be blocked for the server that's failing. – mfinni Sep 02 '10 at 17:06
  • It could be... is there anyway for me to determine whether this is the case? – Michael Robinson Sep 02 '10 at 17:10
  • If you can't even telnet to port 80, then it's blocked somewhere. It could be blocked on your server that you're starting the connection from. It could be blocked on the website that you're trying to connect to. It could be blocked at any network device in between. You have to work with the parties responsible for each/all of those to determine where the block is. – mfinni Sep 02 '10 at 17:30
  • curl to the offending server from within the lan works, *they* must have blocked the ip. – Michael Robinson Sep 02 '10 at 17:36
  • that's not true, if you've blocked TCP inbound or outbound in iptables then it will. If it's blocked for your IP (or only specifically allowed for the two other IPs) on a hardware firewall, then it will fial. In fact, if you have a NATted LAN then it's almost certainly at your side as they'd all share an external IP. Most likely scenario is you've configured something wrong, not them. – James L Sep 02 '10 at 17:49
  • I don't think we have a hardware firewall. I posted iptables in the question, do any of the rules there block in/out? The ips I'm using are all external, I can ssh to any of our machines with their ip. – Michael Robinson Sep 02 '10 at 17:58
1

How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from?

  • netstat -anp
  • iptables -vL
Rob Olmos
  • 2,220
  • 1
  • 15
  • 25
1

I don't fully understand how you're trying to test this but is this an EC2 instance? If so, make sure it's in the same "security group" as the other servers or that the policies within the group allow port 80 and such.

Rob Olmos
  • 2,220
  • 1
  • 15
  • 25
1

In my case, the problem was caused by stale DNS cache. The following helped:

# service nscd restart
jamix
  • 121
  • 4
0

Make sure that something is listening on port 80 and also that there are no firewall rules that would block traffic to port 80 from the IP address you're testing from.

To check if something is listening on port 80, run the following from the webserver itself:

$ netstat -lan | egrep LISTEN

In the output, if you see a line like this, then something is listening on port 80.

tcp        0      0 0.0.0.0:80            0.0.0.0:*               LISTEN

If something is indeed listening on port 80, then it's likely that a firewall rule is blocking the traffic. You can check your firewall rules by running $ iptables --list on the webserver.

EEAA
  • 108,414
  • 18
  • 172
  • 242
  • Thanks for your answer. How would I check that something is listening on port 80, and that there are no firewall rules that would block traffic to oprt 80 from the IP I'm testing from? I've verified on both our other webservers that the examples above work. – Michael Robinson Sep 02 '10 at 16:22
  • I've edited my answer to reflect your additional questions. – EEAA Sep 02 '10 at 16:31
  • Thanks, I have appended the output of the two commands to my question – Michael Robinson Sep 02 '10 at 16:36
  • Well, from your netstat output, nothing is listening on port 80. Are you sure your webserver process is running? – EEAA Sep 02 '10 at 17:12
  • Just installed httpd, see updated output. – Michael Robinson Sep 02 '10 at 17:15
  • Oh, I just re-read your question. Are you trying to run `curl` *from* this webserver, retrieving a file from another remote webserver? If so, the above recommendations aren't valid. – EEAA Sep 02 '10 at 18:32
0

Something between your misbehaving server and rpm.pbone.net is blocking TCP connections initiated by your misbehaving server with rpm.pbone.net port 80 as the destination. The offender could be your server itself, rpm.pbone.net itself, or any router/firewall in between. And what is blocked could be the connection establishment packet or its reply.

The first step in the investigation is to figure out who is blocking traffic. tcptraceroute rpm.pbone.net 80 should tell you how far the packets can go.

It may or may not help to run tcpdump and see if you get some kind of reply (ACK, ICMP, or something weirder) to your SYN packet when you start an outgoing connection to port 80.

Gilles 'SO- stop being evil'
  • 9,181
  • 1
  • 30
  • 48