4

The Problem:

How do you diagnose an ISP specific connectivity problem?

I host a few sites on local machine. I pay for a business class account with Centurylink and have a lot of 5 static ips. The sites are accessible from anywhere in the world except other Centurylink users. If my customers didn't live in the sticks like me, I wouldn't have ever known. Cox users, Verizon users, Cricket, At&t, Comcast, and a huge number of US and internationally based internet service providers can access my sites and ip addresses, EXCEPT other people using Centurylink.

What I've done:

While on location at another person's home with Centurylink internet, I attempted to access each ip address on ports 80 and 443. I attempted to ssh into the server at an arbitrary port that I've selected for that service. I ran tracert on both the domains and the ip addresses and got the following:

1   2ms 1ms 1ms modem.Home [192.168.0.1]
2   30ms    30ms    25ms    rbflxyza84.centurylink.net [x.x.123.45]
3   29ms    30ms    33ms    asdf-ghjk.inet.qwest.net [x.x.122.34]
4   29ms    30ms    33ms    x.x.x.18
5   200ms   79ms    59ms    x-x-45-67.orlf.qwest.net [x.x.45.67]
6   *   *   *   Request timed out.
7   *   *   *   Request timed out.
8   *   *   *   Request timed out.
9   *   *   *   Request timed out.
10  *   *   *   Request timed out.

Coincidentally x.x.45.67 is the ip address I get when I Google What's my ipv4 into Google. If this were working the static ip address would be the next result, so I would think that the request is being blocked at the router.

Using the originating public ip address I searched the access logs and the firewall logs on the router. There are no records of the remote ip address. I am able to find records when packets are dropped according to the rules I've set for OTHER ip addresses.

Just in case I checked the server logs and found no record of the remote address.

Where else could I check to diagnose this problem? I would have asked on SO Network Engineering, but they restrict their questions to only enterprise level issues and solutions.

I've checked blacklists, but my ip addresses don't come up. I've checked the router firewall, but when something there is blocked by one of my rules, it gets logged and I can find the record.

My worst case solution

Before someone suggests it, my worst case scenario is to reverse proxy the traffic through some remote servers I also administer.

Edit

I set up the reverse proxy last night in less time than the average hold time at Centurylink. I'm loathe to call them because they took a month just to provision my static IP addresses. It was like no one there understood why a business needs a static IP address, or even what that is.

Solution

Michael Hampton's comment about the netmask being incorrect is exactly right. All the clients who can't access my address share at least the the first 8 bits of the ip address. The netmask was 255.0.0.0 when it should have been 255.255.255.248. NetworkManager was overwriting my settings, so even after setting it correctly (probably for the second time), it was reverting to the incorrect netmask.

Altimus Prime
  • 334
  • 2
  • 7
  • 20
  • If it's NetworkManager you can just use `nmcli` to fix it, but you have to give the mask in CIDR format, e.g. `nmcli c mod enp4s0 ipv4.addresses 198.51.100.87/29` – Michael Hampton Jun 17 '20 at 01:25
  • Sorry to expose my ignorance, but do I understand correctly that 198.51.100.87 would be the first address in your example block? – Altimus Prime Jun 17 '20 at 02:26
  • It's an example. In reality it would be whatever your IP address is supposed to be. – Michael Hampton Jun 17 '20 at 02:28
  • many provider block ICMP expired in transit (how ping operates). Try using a tool called MTR, as it uses a number of methods to map networks. – The Unix Janitor Jul 03 '20 at 09:46

2 Answers2

8

This is an issue within CenturyLink network. You need to talk to them about the issue and get it resolved.

Tero Kilkanen
  • 34,499
  • 3
  • 38
  • 58
  • Indeed. Not every question needs to be on SE. Sometimes (most of the time??) you just need to _talk to the provider_. – Asteroids With Wings Jun 15 '20 at 12:45
  • I think you're probably right but could you teach me a little more about how you arrive at that conclusion? – Altimus Prime Jun 15 '20 at 14:07
  • A traceroute pinpoints where communications breaks down between two routers. Your traceroute shows that the trace dies sometime after a Qwest router. (CenturyLink and Qwest merged a few years ago.) So the problem is either with that router, or the router immediately after that. Since you are CenturyLink endpoint trying to reach a CenturyLink endpoint, then CenturyLink probably owns that next router, too. – longneck Jun 15 '20 at 15:38
2

It's most likely your netmask is incorrect. The netmask determines which IP addresses are considered to be on the same layer 2 network segment, and if it is incorrect, you will lose connectivity to addresses within that mask, as your device will think they are on the local layer 2 network segment, and will not attempt to route those packets anywhere.

Michael Hampton
  • 237,123
  • 42
  • 477
  • 940