15

I have a number of customers who have networking equipment managed by their ISP. This is usually in the form of an ISP-provided switch or router placed at the customer location(s).

For sites that have MPLS or multi-location connectivity, it would be extremely convenient to tie this equipment into the existing monitoring infrastructure (OpenNMS, Observium, etc.), especially since all other aspects of the environment are routinely checked.

Unfortunately, most providers restrict access to the equipment and force you to go through them for configuration changes. That's understandable, but how can I get more accurate information? I basically have a big black hole in my monitoring footprint.

A recent example was a client who was having VoIP troubles (dropped calls and quality issue) across an MPLS link between two facilities. I don't have any detail as to the level of QoS implemented (because we can't see inside the router). The ISP didn't have any suggestions other than to increase bandwidth from 4Mbps to 7Mbps (upsell - $$$). They said, "you're maxing out your connection at the remote site". So of course, the client agreed to this, without any engineering justification.

The best I could do was monitor the switch ports leading to the ISP's routers at both sites, and I saw no indication of bandwidth saturation... only big jumps in latency (measured switch-to-switch).

Primary site: enter image description here

Remote site: enter image description here

  • So, is this something that's negotiable with the ISP?
  • Have you ever convinced a provider to provide more in-depth monitoring data or to allow SNMP monitoring of their equipment?
  • What recourse do you have if you suspect that the problem lies with the ISP?
ewwhite
  • 194,921
  • 91
  • 434
  • 799

2 Answers2

11

Usually for a large contract you should be able to negotiate at least read only access to their edge router.

However even as building a new data center near one of our ISP connection (~40m cable) and a pretty big contract back then i was unable to get that. There was often some data leaking which you could get.

Depending on your local laws this might however be some sort of unauthorized access already and you should make sure you can get either a permission or take the risk no one is noticing :)

Getting some sort of regress is as usual negotiable , but it is most likely not worth it.

Example for a small contract: we get up to 2 month of what we pay them back if they really mess up and fail to provide the service we buy from them. We (back then) would lose a lot more in worst case so this was rather useless in terms of money.

We tried to document everything "external". What you cannot change yourself, write it down/monitor it so you immediately know if it changed.

We had documents about which routing which of our customers got, how long this usually takes, how big the bandwidth between the links is, every detail we could think of. This was done with the help of the ISP, so it did not take that long, i think it was a week or something like that.

For the actual fixing and/or blaming:

If we found a specific issue we notified them with monitoring data and with the parts of our documentation. There was an incident where they changed the routing and a part of our customer had a few ms delay more than before, which was crucial for our service .

They did not respond positively to that all the time however.

We got permission to "prove" that is their issue, spammed the link and could reproduce the additional delay when some threshold was reached. Soon after it was fixed, even with the different routing.

Basically there are two options:

Either you have really good connections to the technicians in charge (read CTO of ISP), and they are able to allow you read-only access, or you have to try and error , reproduce the issue (however hard that is) and then get someone to actually understand it.

Do not even try with normal customer support of the ISPs, even their "higher level support" is most of the time not allowed to acknowledge their faults.

All of this won't work with enough cash in the contract.

Dennis Nolte
  • 2,848
  • 4
  • 26
  • 36
3

Some ISPs might provide a looking glass server, which provides limited info for public access. These info generally include traceroute/ping from ISP router to specific location, as well as basic BGP info.

One can check out sites like lookinglass.org or bgp4.net wiki to see if concerned ISP is listed. However note that those info are very limited, and may not completely satisfy original question.

In general, answer from @dennis-nolte do apply: everything boils down to money, and the initial term of contract. It's quite easy to ask ISP to provide read-only access when ISP spots a big customer during discussion of initial contract, but they are reluctant to do anything as an afterthought.

Abel Cheung
  • 251
  • 1
  • 9