Light packet loss through router

Question

I just recently got a fiber line installed into my office, and with the exception of an odd bit of trouble that we're having, things are working good, and the network response is really amazing.

The problem that we're having is that every once in a while, my router will flake out and drop packets. It's not the line, and it's not the switch. It's the router itself, and I've switched out the hardware, and both pieces do it. The piece of equipment that I'm using is a Juniper Netscreen SSG5. Here are the symptoms:

I do a pingflood to the "internal" interface, with

 ping -f -c 10000 <internal-ip>

and I get 10,000 responses. Every time. Then, I'll do the same thing, except to the IP address of the external interface (but still on the same device). It drops somewhere between 10 and 15 packets out of 10,000. I've done the same test on every other gateway in the company, and nothing else shows this behavior. I'm perplexed.

I've talked with the support from the fiber company, and both of our interfaces are hard-coded to 100Mb with full duplex, if that could even cause the problem. Incidentally, when pinging the exterior interface from inside the router, I never lose a packet, which makes me think that it isn't the interface itself. And the local interface never loses a packet, so it isn't the switch.

I'm honestly not sure where the problem could lie, except with the design of the hardware itself. I've watched the graphs, and even during the pingflood, I'm nowhere near maximizing the CPU or memory on the router.

Any suggestions?

Edit

For Tom: The fiber is 13Mb/s, but when I ping the interface, it isn't crossing over to the fiber. The local LAN is running at 100Mb/s, and the internal interface responds to every packet. I'll have to see if I can borrow another piece of hardware, but I've got some older model Junipers (5GTs) at different sites that don't show the same symptoms.

Answer 1

Bear in mind two points:

1. The router will likely throttle ICMP traffic directed to it, although I'm not familiar with the SSG5 specifically.
2. A forwarding rate of 140MBit/sec assumes that the traffic is going through the router; traffic addressed to the router will cause an additional performance hit, as every packet will be passed to the router's own IP stack and will require a response packet to be generated.

A couple of tests to try:

1. Try pingflooding from your LAN through the router; perhaps the remote end of the WAN link? (I'm assuming that will be something with more processing power, if it's owned by your service provider.)
2. Run iperf between a node in your office and something outside on the internet, to get a good idea as to what you're being shaped to.

Answer 2

Just an idea.. But what's the speed of the fibre? Can the backplane of the router actually shift packets at that speed? I had a similar problem with filling the ethernet buffers on a Cisco 857 by maxing out the connections on the switchports.

Is the SSG5 running the latest version of ScreenOS? Latest firmware updates?
The spec claims that it can shift 140Mbit, or 30k Packets per second. So might not be that, but perhaps a beefier router could cope with the traffic?
Could you borrow a larger device from someone? Perhaps a Cisco 2811 or a Juniper J2320?

Answer 3

We had similar issues when we moved to fiber/metro ethernet (AT&T).

Do the interfaces on your router show any errors? We use Cisco and we would see CRC or input errors, depending on the interface.

We finally resolved it by swapping around different negotiation methods between auto, 10/half and full, and 100/half and full, for each of our locations, until either auto or 100/full “stuck.” You may also want to ask your provider to temporarily remove the 13Mbps cap, to see if it’s an issue with their bandwidth limiting.

AT&T blamed it on the switches they used (also Cisco), but wouldn’t swap them for alternative models. We stopped caring as long as we stopped getting errors and 100/full worked (either by hard coding or auto negotiation).

To this day, we still have some offices auto and some 100/full, just because it worked and we don’t want to break it.