HTTP Services inaccessible from some ISP's

Question

I got a really strange problem and I don't know what to do anymore. All my problems started a couple of weeks ago.

A basic schematic of my network is available in the fallowing image:

Basic Configuration:

M1 services:

• Apache on port 80
• RoR with WEBrick on port 4321
• SSH on port 22

M2 services:

• USBWebserver on port 8080

Router Port Forwards (Extern[E:] to Intern)

• E:81 to M1:80
• E:4321 to M1:4321
• E:8080 to Router:8080
• E:8081 to M2:8080
• E:22 to M1:22 (ssh)
• E:10000 to M1:10000 (https webmin)
• E:3389 to M2:3389 (rdc)

The problem:

On port 4321 I have a quite heavy incoming traffic. My problem started when I realized that my HTTPD services on the M1 was accessible only from some ISP's.

My Internet Service Provider is providing me with a static IP address (188.26.X.XXX), based on a PPPOE connection and is called RDS. Other ISP's in this discution are ROMTELECOM,ORANGE, UPC and INTERSAT.

After a few tests the problem was as fallows:

E:81,E:4321,E:8080,E:10000 was not loading from RDS(my own network), ROMTELECOM, INTERSAT. E:81,E:4321,E:8080,E:10000 was loading from ROMTELECOM (different geographical region), UPC and mobile networks like ORANGE.

E:8081,E:3389 and E:22 was accesible from anywhere.

All services available in LAN.

Troubleshooting Steps

1. Disabling any kind of firewall on the router and M1.
   • Result: Same problem.
2. Changing the LAN ip's and MAC addresses for Router(wan) and M1
   • Result: Same problem.
3. Changing the Router Asus WL-520GU running DD-WRT with a WRT54GL running default firmware and then Tomato.
   • Result: Same problem.
4. Removing the M1 from the network and installing 2 virtual machines, VM1 and VM2 on M2. VM1 has the same configuration as M1. VM2 was for testing purposes and was running WINDOWS XP with RoR for Windows and USBWebserver.
   • Result: VM1 same scenario as M1(ssh OK. RoR and apache not OK). VM2 inaccessible.
5. Removing the router from the network and connecting directly to the M2. Forwarding ports to VM1 and VM2.
   • Result: Same problem.
6. Freaking out.
   • Result: Same problem.
7. Calling ISP support and sending an email. A telephone conversation was made and the problem explained. No known issues from them. No traffic alteration from my ISP.
   • Result: Same problem.
8. Repeating steps from above in random order...

Other tests that I made

Using WFetch I realized that my request from the affected networks was recived by my server but the response was frizzing after a few bytes. Using my E:81 with _h5ai as a service for my screenshot's and other file sharing purposes, I removed all files and made a index.html with 1 character. Testing again the E:81 from affected networks, the service was accessible and the content of index.html was received. Adding characters to the file I discovered that a index.html with ~1499 characters was successively received. Adding one more character the effect will become unresponsive as described. No time out's from the browser, just waiting from a response. Unfortunately, this buffer was not the same on WEBrick. A bigger response from my server (~3k characters) was needed to reproduce the problem. The E:8081 was accessible all the time from all networks.

Also, I realized that if I request a non existing page :81/xyz or :4321/xyz, I receive the 404 error.From both HTTP servers, apache2 and WEBrick. This is have to do with that maximum amount of data allowed as response.

Reproducing the problem

My current configuration contains the VM1. M1 and VM2 was removed. Try to access 188.26.X.XXX:4321 and 188.26.X.XXX:81 by browser and if is not working, by WFetch.

Thank you and I hope that I made my problem clear.

Answer 1

I discovered that a index.html with ~1499 characters was successively received. Adding one more character the effect will become unresponsive as described.

This would suggest a MTU issue (Wikipedia link) where your router is sending large packets that another router is not capable of handling.

In laymans terms, all communication across the network/internet is broken down into little packets, but the size of the packet is not fixed. The MTU (maximum transmission unit) is the maximum size that a device (eg your router) is supposed to send. If you try to send a larger packet, your device should split the packet into two packets and send them separately.

But a problem can occur if your router sends a large packet to another router that is not able to cope with. In this situation, your router will assume the packet got lost and send it again and since that will also fail, it sends it again and again until it gives up).

Reducing the MTU on your router would mean your router would send smaller packets that the other router can accept and everything starts working.

So why doesn't everyone use a small MTU setting that everyone can cope with. Well every packet sent has an overhead , eg information about who the packet is from and who it is to be sent to and also it must get a reply back indicating that the packet was received correctly. Hence the smaller the packet size, the more packets you need to send and the greater the overhead and that slows down your connection. (For example, for performance reasons I understand that XBOX live requires a MTU of at least 1364)

Hence the normal recommendation is to use the default settings except when you are having trouble and in that case you keep reducing your MTU settings (in steps of 10) until you find a value that works reliably.

The default setting can depend on what type of network connection your are using. For example the default on an internal windows network is 1500, whereas the default for internet connections using PPPoE is 1492.

Answer 2

According to SANS, Port 4321 has been used by the BoBo trojan. As you didn't specify UDP or TCP, I wonder if it's possible that that port was shut down by some ISPs?!?! You might also want to look at this site for additional info on port 4321. Also, my understanding is many ISPs frown on people running their web servers unless this is a business with a business contract with the ISP.