15

Background

I have been struggling to get my SIP phones to register behind a brand new router and switch in our brand new office. Our PBX is hosted offsite. I have worked with our provider to attempt several different approaches. We have tried regular NAT to connect to their NAT-aware session border controller. We have tried using siproxd (the pfSense package) to intercept the SIP registration requests and register on the phones behalf. Finally, we have tried configuring the phones manually to register with the siproxd daemon on my local network.

Throughout testing we have seen the phones do all of the following successfully:

  • Contact the hosted FTP server by IP address
  • Download the configuration from said server
  • Perform DNS queries to resolve the IP address of the NTP server
  • Query the NTP server to set the time
  • Perform DNS queries to resolve the IP address of the SIP server(s)

Symptoms

After the phones have done all of the pre-registration tasks successfully, we never see the registration attempt hit the pfSense box, or the provider's PBX. I have enabled the highest level of debugging in siproxd on my end and have seen nary a TCP connection or UDP packet. However, a simple telnet to port 5060 from a workstation will generate expected log messages. Performing a packet capture on the pfSense box showed absolutely no SIP traffic attempts.

What the heck?

My final troubleshooting step that thoroughly stumped me and brought me to ask this question was as follows. I first mirrored the switch port that a phone was plugged into to my workstation switch port. I performed a packet capture of all traffic on the interface. To my surprise I saw SIP registration packets coming from the phone. Here is an example:

Phone Packet Capture

Clearly the phone is trying to register with the PBXs (those are the correct IP addresses as well).

My next step was to mirror the switch port that feeds into the LAN side of the pfSense router. I saw all of the FTP, NTP, and DNS traffic from the 172.200.22.102 phone coming out of the switch, but not a trace of the SIP packets. This is completely baffling to me! What is causing only the SIP traffic to vanish within the switch?

Environment

Switch Configuration

The phone with IP Address 172.22.200.102 is in port 4 of this switch, the router LAN link is in port 22.

VLAN Configuration

VLAN 200 participation

I can share any more settings that may be needed.

hobodave
  • 2,800
  • 2
  • 23
  • 33
  • I know it's common sense that the packets should be headed for the LAN interface of the router, but let's not take anything for granted. Could you get wireshark to show you the destination MAC addresses on those packets leaving the phone, and confirm that they are in fact the MAC address of the router? If for some reason they were not, that would explain why you weren't seeing them on the mirror port. – MadHatter Aug 22 '11 at 15:18
  • @Mad: confirmed proper destination MAC address of the router – hobodave Aug 22 '11 at 16:21
  • Damn. Well, it was worth checking. Sorry not to have any better ideas. – MadHatter Aug 22 '11 at 16:39

1 Answers1

16

I found the solution after spending roughly 40 hours on this problem.

There is a setting in the switch that enables "Auto DoS" protection. Apparently it considers TCP or UDP traffic that has matching source or destination ports to be a blat attack and drops the packet. This is ridiculously short-sighted since SIP traffic often (always?) relies on source and destination ports being 5060.

In case a textual description was insufficient:

enter image description here

hobodave
  • 2,800
  • 2
  • 23
  • 33
  • wow, that's brutal. Good job finding that. I bet it doesn't show up in the logs either, right? – gravyface Aug 22 '11 at 21:37
  • @gravyface: correct, nothing logged. All the logs show is basic link up/down and authentication attempts – hobodave Aug 22 '11 at 21:40
  • 2
    Whaaaaaaaaaaaaaaaat? Nice job HP! – voretaq7 Aug 22 '11 at 21:41
  • @hobodave: can you increase the verbosity? – gravyface Aug 22 '11 at 21:42
  • @gravy: It's on the second highest already (INFO). I could go to DEBUG I guess, but I don't feel like breaking my phones just to test that :) – hobodave Aug 22 '11 at 22:08
  • @hobodave: ok. Just wondered if that behaviour is detectable at any point in the switch logs. – gravyface Aug 22 '11 at 22:11
  • 1
    That sucks; it would screw (eg) NTP and server-server DNS as well. In the words of voretaq, "nice job. HP". Well diagnosed, though, hobodave; you deserve a beer! – MadHatter Aug 23 '11 at 06:55
  • 1
    +1 - I ran into this today w/ the same switch in a different application. The SonicWALL "Single Sign-On" protocol uses UDP datagrams with the same source/destination ports. I wish I'd looked here before tracking it down with an Ethernet tap. It's interesting to note that the "offending" frames are even removed when port-mirroring the ingress port. Complete fail, HP. I can confirm that the PK 1.15 firmware, released in January, still has this misfeature in it. – Evan Anderson Sep 30 '13 at 23:56