4

Recently ran into a very strange problem.

Several applications were having issues communication through our F5 Load-Balancer. When we looked into it we found that the router had an incorrect ARP and MAC-ADDRESS table entry on the Load-Balancer VLAN. Those entries were pointing towards a Windows Server 2008 R2 box instead of the Load-Balancers external interface.

Now here is the strange thing. The hardware address in the MAC/ARP table entries did not exist on the Windows 2008 Server but it was very close. The Windows Server was on router interface Gi1/37 (below). The Load-Balancer External Address was 192.168.111.61 and the Windows Server was 192.168.111.125. Two totally different IP addresses in the same /24 subnet.

IPConfig on Windows Server

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) 82574L Gigabit Network Connect
   Physical Address. . . . . . . . . : 00-E0-81-DF-15-FE
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::917f:6781:df6:f724%11(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.111.125(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : fe80::21e:f7ff:fe41:2a80%11
                                       fe80::21e:f7ff:fe41:3540%11
                                       192.168.111.1

MAC Info on Windows Box

C:\Users\Administrator>getmac

Physical Address    Transport Name
=================== =========================================================
00-E0-81-DF-15-FE   \Device\Tcpip_{5BB4FA88-7056-4303-8528-AA2293E4821B}
00-E0-81-DF-15-FD   Media disconnected

The ARP and MAC ADDRESS entry in the Router

Router#sh ip arp 192.168.111.61
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  192.168.111.61            1   00e0.81df.15fc  ARPA   Vlan50  

Router#sh mac-address-table addr 00e0.81df.15fc

Legend: * - primary entry
    age - seconds since last seen
    n/a - not available

  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Module 1[FE 1]:
*   50  00e0.81df.15fc   dynamic  Yes        275   Gi1/37

The last 4 bits on the hardware address although similar were not existing physical hardware addresses on the Windows 2008 Server. Logic dictates that the Windows Server had to have performed some sort of incorrect gratuitous ARP in order to poison the ARP and MAC table on the router. Or it was responding to an ARP request for an IP that it didn't own and a MAC ADDRESS that it didn't own.

The second we shut down the Windows 2008 interface and cleared the ARP/MAC tables the problem was solved.

For the life of me i am unable to understand how this happened (or why).

user770395
  • 101
  • 1
  • 5
  • What does an ipconfig /all say on the windows 2008 server? – artifex May 29 '13 at 23:24
  • 1
    What is the Windows server running? Is there a cluster or Network Load Balancing installed? – Noah Stahl May 30 '13 at 00:21
  • Added IPConfig Info Windows Server was running nothing but the OS. It was not load balanced nor dual homed. It had 1 single Ethernet connection. – user770395 May 30 '13 at 16:08
  • The Windows's servers IP-Adress and the line from the router do not match. `.61` will never show the same MAC as `125`. Those are different machines. – bjoster Jan 13 '20 at 14:19

2 Answers2

0

I suspect you are using a Dell server with iDrac, or a server with similar functionality (I.e IPMI).

The NIC is set to “shared” mode and the physical adapter has more than one MAC address. The MAC address iDrac uses is only one digit off from the address Windows sees. Windows will not see this extra MAC address, it is handled outside the operating system.

In other words, look in your iDrac settings and find out how it is misconfigured. It is either set to the same IP address as the load balancer or has some type of broadcast address or netmask problem that is responding to every ARP request on the network. Or perhaps it is just malfunctioning. Maybe perform a firmware update.

Appleoddity
  • 3,290
  • 2
  • 10
  • 27
0

If you have a virtual server and it has experienced a cluster failover a gratuitous ARP request is sent. Most likely your switch is not forwarding this and the ARP table on the other side of the switch is left with the incorrect MAC.

In a cluster fail over the server will/can switch MAC address.

The solution is to configure your switch(es) to forward the gratuitous ARP request.

artifex
  • 1,634
  • 1
  • 17
  • 22
  • This isn't related to clustering. The Router ARP cache literally had an entry pointing towards the Windows Server for an IP address that the Windows Server didn't own and a MAC address that the Windows Server didn't own (but was close). – user770395 May 30 '13 at 16:36
  • Which can happen in a cluster fail over scenario. http://support.microsoft.com/kb/244331 (Its the old cluster node's information that is seen in the switch) – artifex May 30 '13 at 19:37