Using MCP (MisCabling Protocol) for ACI

MisCabling Protocol (or MCP) detects loops from external sources (i.e., misbehaving servers, external networking equipment running STP) and will err-disable the interface on which ACI receives its own packet. Enabling this feature is a best practice and it should be enabled globally and on all interfaces, regardless of the end device.

MCP limits the blast radius that can result when Layer-2 loops form. While a layer-2 loop will do nothing to harm the ACI Fabric itself (ACI will transmit the broadcast churn @ line-rate), the loop and resulting broadcast storm will cause issues with attached hosts and other networking equipment that reside in the broadcast domain.

How does MCP work?

When configured properly, MCP will send Layer-2 packets on every EPG with a custom MAC address. If the ACI Fabric receives an MCP packet on any interface, it will take action (actions could be just alerting with a fault, or err-disabling the interface on which the MCP packet was received).

Notables

  • The ACI Fabric does not participate in spanning tree protocol (STP) but instead acts as hub with respect to STP.
  • MCP can be enabled globally and per-interface.
  • By default, MCP is disabled globally but is enabled on each port. For MCP to work, it must also be enabled globally.
  • This feature (and the option to send MCP packets on a per-EPG basis), was first available starting with APIC 2.0(2) code.

How do I enable MCP?

For MCP to be enabled, you need to have it enabled globally and on a per-interface basis.

While MCP is enabled on all interfaces by default, it is not turned “on” until you also enable it globally. The global configuration knob for MCP can be enabled by configuring the global settings here: Fabric > Access Policies > Global Policies > MCP Instance Policy default

Screen Shot 2018-03-27 at 9.54.26 AM.png
Fabric > Access Policies > Global Policies > MCP Instance Policy default
  1. Enable the admin state of the feature
  2. Make sure and enable the “Enable MCP PDU per VLAN” option, which enables MCP to send packets on a per-EPG basis, otherwise, these packets will only be sent on untagged EPGs (which basically makes it useless from a loop-detection perspective).
  3. (Optional) – Alter the “loop detection multiplication factor” setting. This setting refers to the amount of MCP packets that will be received by the ACI Fabric BEFORE the Loop Protection Action takes place.

Helpful MCP related Commands

Helpful MCP-related CLI commands (to be executed on LEAF switches)

  • show mcp internal info global
  • show mcp internal event-history interface fsm-all
  • show mcp internal info interface eth 1/4
  • show mcp internal event-history errors
  • show mcp internal event-history trace detail

Verification

To verify that you have MCP enabled (via the CLI), login to a LEAF switch and run the show mcp internal info global command. This will show display the following info:

  • Is MCP enabled globally?
  • Is MCP transmitting Layer-2 packets on a per-epg basis < VERY IMPORTANT
  • What is the Loop Detection Action set to?
Leaf202# show mcp internal info global

----------- STP HIF Configurations -----------
 HIF Port BPDU Guard: enabled
 HIF Port BPDU Filter: disabled

------------ STP Flush Internal Info -----------
 TC Batch: enabled
 TC Batch Wait Time: 2000 MSECS
 TC Batch Hold Time: 5000 MSECS

----------- Loop-Detection Configurations ---------
 MCP Enabled: True
 MCP PDU PER VLAN Mode: True
 TX Freq: 2000 MSECS
Loop Detect Multiplier: 1
 Loop Detect Action: Port Err-Disable

----------- System Information ---------
 Node ID: 202
 Fabric ID: 1
 Lscope Info: N 0 0
Bootup Loop Action Delay: 180 SECS
Loop action effective from: Fri Mar 16 01:47:39 2018

 

There are two quick ways to determine if MCP has triggered on an interface.

  1. GUI – Look under faults (System > Faults)
  2. CLI on LEAF – show mcp internal event-history interface fsm-all. The CLI command will show the FSM states for all interfaces. In the case below, using the grep command in combination with the FSM CLI command would display all interfaces that had an MCP Loop event. show mcp internal event-history interface fsm-all | grep MCP_EV_LOOP_DETECTED
Leaf202# show mcp internal event-history interface fsm-all

... Output Suppressed for brevity

>>>>FSM: <Ethernet1/6> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/5> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/4> has 6 logged transitions<<<<<

1) FSM:<Ethernet1/4> Transition at 360024 usecs after Fri Mar 16 01:47:38 2018
 Previous state: [MCP_ST_INITIALIZE]
 Triggered event: [MCP_EV_ADMIN_ENABLE]
 Next state: [MCP_ST_OPER_UP]

2) FSM:<Ethernet1/4> Transition at 354986 usecs after Fri Mar 16 07:35:43 2018
 Previous state: [MCP_ST_OPER_UP]
 Triggered event: [MCP_EV_LOOP_DETECTED]
 Next state: [MCP_ST_OPER_LOOP_DETECTED]

3) FSM:<Ethernet1/4> Transition at 355459 usecs after Fri Mar 16 07:35:43 2018
 Previous state: [MCP_ST_OPER_LOOP_DETECTED]
 Triggered event: [MCP_EV_LOOP_DETECTED]
 Next state: [No transition found]

4) FSM:<Ethernet1/4> Transition at 355513 usecs after Fri Mar 16 07:35:43 2018
 Previous state: [MCP_ST_OPER_LOOP_DETECTED]
 Triggered event: [MCP_EV_LOOP_DETECTED]
 Next state: [No transition found]

5) FSM:<Ethernet1/4> Transition at 355561 usecs after Fri Mar 16 07:35:43 2018
 Previous state: [MCP_ST_OPER_LOOP_DETECTED]
 Triggered event: [MCP_EV_LOOP_DETECTED]
 Next state: [No transition found]

6) FSM:<Ethernet1/4> Transition at 358978 usecs after Fri Mar 16 07:35:43 2018
 Previous state: [MCP_ST_OPER_LOOP_DETECTED]
 Triggered event: [MCP_EV_ADMIN_DISABLE]
 Next state: [MCP_ST_OPER_DOWN]

Curr state: [MCP_ST_OPER_DOWN]

>>>>FSM: <Ethernet1/3> has 1 logged transitions<<<<<

1) FSM:<Ethernet1/3> Transition at 653928 usecs after Fri Mar 16 01:48:18 2018
 Previous state: [MCP_ST_INITIALIZE]
 Triggered event: [MCP_EV_ADMIN_ENABLE]
 Next state: [MCP_ST_OPER_UP]

Curr state: [MCP_ST_OPER_UP]

>>>>FSM: <Ethernet1/2> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/1> has NO logged transitions<<<<<

 

Once you have narrowed down the interface on which MCP took action, lets dig a bit deeper with the show mcp internal info interface eth 1/4 command. This, btw, is one of the most useful commands. In addition to the wealth of MCP-related information, this command is also very helpful if you are troubleshooting a classic-STP related issue as well with your legacy environment, as you can see the amount of TCNs (topology change notifications) as well as the last time you received a TCN on the interface in question.

This will show display the following info:

  • If MCP is enabled on the Interface
  • STP packet counts (for MST, RSTP, PVRSTP)
  • TCN count
  • Last time a TC was received on the interface
  • Number of MCP Packets sent
  • Number of MCP Packets received << This is the important one, because receiving an MCP packet means a loop has occurred.
  • The Vlan (EPGs) on which MCP is being sent
  • The Vlan (or EPG) on which the MCP packet was received
Leaf202# show mcp internal info interface eth 1/4
------------------------------------------
Interface: Ethernet1/4        
Native PI VLAN: 0     
Native Encap VLAN: 0            
BPDU Guard: disabled           
BPDU Filter: disabled            
Port State: down           
Layer3 Port: false       
Switching State: enabled           
Mac Address: 50:f:80:42:5:64 
Interface MCP enabled: true
------------------- STP STATS --------------------            
MSTP Count: 0            
RSTP Count: 10443         
MSTP TC Count: 0         
RSTP TC Count: 0       
PVRSTP TC Count: 14             
TCN Count: 0 PVID Error BPDU Count: 0    
Error Packet Count: 10447
BPDU Guard Event Count: 0
Last TC received at Fri Mar 16 01:57:12 2018

--------------- LOOP-DETECTION STATS  ---------------
MCP packets sent(Per-vlan): 10442  
MCP packets received: 4
MCP invalid packets received: 0
MCP packets received with invalid digest: 0
MCP packets received when switching state is disabled: 0
Number of active VLANs : 7
Number of VLANS in MCP packets are sent: 7     
MCP enabled vlans:     50,   99,  100,  101,  102,  103, 1500        
MCP loop detected at: Fri Mar 16 07:35:43 2018   
MCP loop detected in VLAN: 100
-------------- MCP Remote Peer Info --------------
No remote peers exist

One thought on “Using MCP (MisCabling Protocol) for ACI

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.