How to troubleshoot a MAC flapping between switch ports. (Cisco)

Question

I'm working on a rather small Layer 2 network segment that contains a physical loop. (I have included a picture of the topology below). The core switches are stacked 3850s. The other switches are SG300 series small business switches.

The links between all switches are trunks that allow vlans 1,503,508,590

Spanning-tree seems to be functioning as expected and the interface on SW3 that connects to SW4 is put into a blocking state.

The problem I have is that the core switch periodically logs a MAC Flap event between the links to SW1 and SW2.

%SW_MATM-4-MACFLAP_NOTIF: Host 00eb.d5f2.0b9a in vlan 1 is flapping between port Po1 and port Po2

The offending MAC address seems to be a Cisco MAC (Possible one assigned to a port group?) however I am unable to physically locate the switch to which it belongs. When I remove the loop, The Core says the MAC is available through Po2 (The link to SW2), however SW2 says the MAC is available through Po1 (The uplink to the Core).

My questions are:

How can I determine which switch this MAC address actually belongs too?
What might be causing this periodic flap between ports?

Here is the current configuration for all switch-to-switch links as well as a the full mac address table for each switch.

Core Switch interface configurations:

    interface GigabitEthernet1/1/1
     description Po1 Member1
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
     channel-group 1 mode active
    end
    !
    interface GigabitEthernet1/1/2
     description Po2 Member1
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
     channel-group 2 mode active
    end
    !
    interface GigabitEthernet2/1/1
     description Po1 Member2
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
     channel-group 1 mode active
    end
    !
    interface GigabitEthernet2/1/2
     description Po2 Member2
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
     channel-group 2 mode active
    end
    !
    interface Port-channel1
     description SW1 Uplink
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
    end
    !
    interface Port-channel2
     description SW2 Uplink
     switchport trunk allowed vlan 1,503,508,590
     switchport mode trunk
     switchport nonegotiate
    end

SW1 interface configurations:

    interface gigabitethernet51
     channel-group 1 mode auto
    !
    interface gigabitethernet52
     channel-group 1 mode auto
    !
    interface Port-channel1
     description "Core Uplink"
     switchport trunk allowed vlan add 503,508,590
    !
    interface gigabitethernet50
     description "Uplink to SW3"
     switchport trunk allowed vlan add 503,508,590
    !

SW2 interface configurations:

    interface gigabitethernet51
     channel-group 1 mode auto
    !
    interface gigabitethernet52
     channel-group 1 mode auto
    !
    interface Port-channel1
     description "Core Uplink"
     switchport trunk allowed vlan add 503,508,590
    !
    interface gigabitethernet50
     description "Uplink to SW4"
     switchport trunk allowed vlan add 503,508,590
    !

SW3 interface configurations:

    interface gigabitethernet49
     description "UPLNK to SW4"
     switchport trunk allowed vlan add 503,508,590
    !
    interface gigabitethernet50
     description "UPLINK TO SW1"
     switchport trunk allowed vlan add 503,508,590
    !

SW4 interface configurations:

    interface gigabitethernet49
     description "UPLNK to SW3"
     switchport trunk allowed vlan add 503,508,590
    !
    interface gigabitethernet50
     description "UPLINK TO SW2"
     switchport trunk allowed vlan add 503,508,590
    !

Core Switch MAC Table for 00eb.d5f2.0b9a:

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   1    00eb.d5f2.0b9a    DYNAMIC     Po1

SW1 MAC Table for 00:eb:d5:f2:0b:9a:

    Vlan          Mac Address         Port       Type
------------ --------------------- ---------- ----------
     1         00:eb:d5:f2:0b:9a      gi50     dynamic

SW2 MAC Table for 00:eb:d5:f2:0b:9a:

    Vlan          Mac Address         Port       Type
------------ --------------------- ---------- ----------
     1         00:eb:d5:f2:0b:9a      Po1      dynamic
    503        00:eb:d5:f2:0b:9a      Po1      dynamic
    508        00:eb:d5:f2:0b:9a      Po1      dynamic
    590        00:eb:d5:f2:0b:9a      Po1      dynamic

SW3 MAC Table for 00:eb:d5:f2:0b:9a:

    Vlan          Mac Address         Port       Type
------------ --------------------- ---------- ----------

SW4 MAC Table for 00:eb:d5:f2:0b:9a:

    Vlan          Mac Address         Port       Type
------------ --------------------- ---------- ----------
     1         00:eb:d5:f2:0b:9a      gi50     dynamic
    503        00:eb:d5:f2:0b:9a      gi50     dynamic
    508        00:eb:d5:f2:0b:9a      gi50     dynamic
    590        00:eb:d5:f2:0b:9a      gi50     dynamic

show cdp nei for 'core switch':

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID

SW2
                 Gig 2/1/2         146              S I   SG300-52  gi52
SW2
                 Gig 1/1/2         146              S I   SG300-52  gi51
SW1
                 Gig 2/1/1         123              S I   SG300-52  gi52
SW1
                 Gig 1/1/1         123              S I   SG300-52  gi51

Just a shot in the dark: Are you sure you need to configure "switchport trunk..." for the individual interfaces on the core? Wouldn't the VLAN config for Port-channel1 suffice? (Just like you did on SW1) — Marki, Jul 20 '16 at 07:20
BTW in your text you're writing about VLAN 509, yet in your config I see VLAN 590 :) — Marki, Jul 20 '16 at 07:20
You could start by eliminating the layer-2 negotiations.You can use `channel-group 1 mode on` to unconditionally channel; this eliminates PAgP and LACP negotiation. Also, you unconditionally set the core ports to trunk, but the interfaces on SW1 are going to use DTP to negotiate a trunk. This is a bad practice. You should set both sides to `switchport mode trunk` and `switchport nonegotiate`, which will eliminate DTP negotiation. The configurations from all the switch-to-switch ports on all the switches would be helpful, as would the output from `show mac-address-table` on all the switches. — Ron Maupin, Jul 20 '16 at 15:33
Also, the `spanning-tree link-type point-to-point` is an incorrect configuration for your topology. — Ron Maupin, Jul 20 '16 at 17:44
@Marki Thanks for the response, however I believe it is necessary on the Catalyst switches. If I remove the 'switchport trunk' settings from the physical interfaces then I get the following error: `%EC-5-CANNOT_BUNDLE2: Gi1/1/2 is not compatible with Po2 and will be suspended (vlan mask is different)`. Additionally, when I put the trunk config on the port channel interface, then the switch automatically adds the same config to the individual interfaces. — blacklight, Jul 21 '16 at 00:12
@RonMaupin Thanks for the tips Ron. I have included pastebin links to the full switchport config as well as MAC tables in the original post. I do not believe the SG300 switches support DTP and set their trunk links to unconditional with no negotiation by default. (There is no option to set otherwise). I did however add `switchport nonegotiate` to the Core switch, I also removed the spanning tree link-type from the switch links. I have not seen the flapping MAC since making these changes, however the error was transient, so I will have to monitor it, to see if it's still occurring. Thanks. — blacklight, Jul 21 '16 at 00:20
The MAC address in question is connected to SW1 interface `Gi50` on VLAN 1, for which you have not included the configuration. It seems you are somehow bridging the VLANs together on the Core Switch. SW2, on the other side of the Core Switch, is getting that MAC address from the Core Switch on all the VLANs, and SW2 is passing it to SW4 on all the VLANs. You should carefully check your cabling and interface configurations. You may be on the verge of creating a broadcast storm. A full configuration of the Core Switch would help, as well as the output from `show cdp neighbor`. — Ron Maupin, Jul 21 '16 at 01:23
Apologies, I missed the config for the Gi50 config on SW1 & SW2. These are the uplinks to SW3 & SW4 respectively. I have added their config, as well as updating the picture to show the topology in greater detail. I also included the output of show cdp neighbor. I will make a sanitized config for the core and upload it shortly. I have double checked all cabling, and it is connected as detailed in the updated picture. — blacklight, Jul 21 '16 at 01:58
I thought I would just add that I still haven't seen the error logged again after making the initial changes you suggested. I did a `clear mac address-table dynamic` on all switches in the topology, and now the offending MAC address is only appearing in the MAC table for SW2 & SW4 (On all VLANs). I don't want to speak too soon, but I believe it may now be resolved. — blacklight, Jul 21 '16 at 02:13

score 3 · Accepted Answer · answered Jul 21 '16 at 22:10

With much assistance from Ron Maupin in the comments I was able to resolve this by following one of his initial suggestions; Completely disabling DTP.

The SG300 switches do not support DTP and all trunks are run with Switchport mode trunk by default. After adding the line switchport nonegotiate to the port groups from the Core down to SW1 & SW2 then the flapping stopped occurring.

I tested this by enabling DTP again (By removing the switchport nonegotiate) command, and the flapping returned.

I do not fully understand why DTP on the core switches was causing this. Perhaps DTP frames were being forwarded by the access switches rather than being dropped? If anyone can shed any light why this fix worked then that would be much appreciated.

How to troubleshoot a MAC flapping between switch ports. (Cisco)

1 Answers1