MisCabling Protocol (or MCP) detects loops from external sources (i.e., misbehaving servers, external networking equipment running STP) and will err-disable the interface on which ACI receives its own packet. Enabling this feature is a best practice and it should be enabled globally and on all interfaces, regardless of the end device.
MCP limits the blast radius that can result when Layer-2 loops form. While a layer-2 loop will do nothing to harm the ACI Fabric itself (ACI will transmit the broadcast churn @ line-rate), the loop and resulting broadcast storm will cause issues with attached hosts and other networking equipment that reside in the broadcast domain.
NOTE – Because MCP works to stop Layer-2 loops, it should be enabled right away on an ACI Fabric prior to connecting Layer-2 devices (i.e., other network switches) for migration purposes.
How does MCP work?
When configured properly, MCP will send Layer-2 packets on every EPG with a custom MAC address. If the ACI Fabric receives an MCP packet on any interface, it will take action (actions could be just alerting with a fault, or err-disabling the interface on which the MCP packet was received).
Notables
- The ACI Fabric does not participate in spanning tree protocol (STP) but instead acts as hub with respect to STP.
- MCP can be enabled globally and per-interface.
- By default, MCP is disabled globally but is enabled on each port. For MCP to work, it must also be enabled globally.
- This feature (and the option to send MCP packets on a per-EPG basis), was first available starting with APIC 2.0(2) code.
- Starting with 3.2, the MCP timer can be tuned to sub-second. Previous to 3.2, the quickest MCP would react was 3 seconds; with MCP Per VLAN Aggressive Timers you can achieve failover in as little as 350 milliseconds.
- MCP supports 256 VLANs per interface. However, when the aggressive timer is used (tx rate < 2 sec), MCP will support up to 2000 total VLANs per leaf across interfaces (The aggressive timer was added into ACI from APIC version 3.2 onward.)
How do I enable MCP?
For MCP to be enabled, you need to have it enabled globally and on a per-interface basis.
While MCP is enabled on all interfaces by default, it is not turned “on” until you also enable it globally. The global configuration knob for MCP can be enabled by configuring the global settings here: Fabric > Access Policies > Global Policies > MCP Instance Policy default
- Enable the admin state of the feature
- Make sure and enable the “Enable MCP PDU per VLAN” option, which enables MCP to send packets on a per-EPG basis, otherwise, these packets will only be sent on untagged EPGs (which basically makes it useless from a loop-detection perspective).
- (Optional) – Alter the “loop detection multiplication factor” setting. This setting refers to the amount of MCP packets that will be received by the ACI Fabric BEFORE the Loop Protection Action takes place.
Helpful MCP related Commands
Helpful MCP-related CLI commands (to be executed on LEAF switches)
- show mcp internal info global
- show mcp internal event-history interface fsm-all
- show mcp internal info interface eth 1/4
- show mcp internal info vlan 101
- show mcp internal event-history errors
- show mcp internal event-history trace detail
Verification
To verify that you have MCP enabled (via the CLI), login to a LEAF switch and run the show mcp internal info global command. This will show display the following info:
- Is MCP enabled globally?
- Is MCP transmitting Layer-2 packets on a per-epg basis < VERY IMPORTANT
- What is the Loop Detection Action set to?
Leaf202# show mcp internal info global
----------- STP HIF Configurations -----------
HIF Port BPDU Guard: enabled
HIF Port BPDU Filter: disabled
------------ STP Flush Internal Info -----------
TC Batch: enabled
TC Batch Wait Time: 2000 MSECS
TC Batch Hold Time: 5000 MSECS
----------- Loop-Detection Configurations ---------
MCP Enabled: True
MCP PDU PER VLAN Mode: True
TX Freq: 2000 MSECS
Loop Detect Multiplier: 1
Loop Detect Action: Port Err-Disable
----------- System Information ---------
Node ID: 202
Fabric ID: 1
Lscope Info: N 0 0
Bootup Loop Action Delay: 180 SECS
Loop action effective from: Fri Mar 16 01:47:39 2018
There are two quick ways to determine if MCP has triggered on an interface.
- GUI – Look under faults (System > Faults)
- CLI on LEAF – show mcp internal event-history interface fsm-all. The CLI command will show the FSM states for all interfaces. In the case below, using the grep command in combination with the FSM CLI command would display all interfaces that had an MCP Loop event. show mcp internal event-history interface fsm-all | grep MCP_EV_LOOP_DETECTED
Leaf202# show mcp internal event-history interface fsm-all
... Output Suppressed for brevity
>>>>FSM: <Ethernet1/6> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/5> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/4> has 6 logged transitions<<<<<
1) FSM:<Ethernet1/4> Transition at 360024 usecs after Fri Mar 16 01:47:38 2018
Previous state: [MCP_ST_INITIALIZE]
Triggered event: [MCP_EV_ADMIN_ENABLE]
Next state: [MCP_ST_OPER_UP]
2) FSM:<Ethernet1/4> Transition at 354986 usecs after Fri Mar 16 07:35:43 2018
Previous state: [MCP_ST_OPER_UP]
Triggered event: [MCP_EV_LOOP_DETECTED]
Next state: [MCP_ST_OPER_LOOP_DETECTED]
3) FSM:<Ethernet1/4> Transition at 355459 usecs after Fri Mar 16 07:35:43 2018
Previous state: [MCP_ST_OPER_LOOP_DETECTED]
Triggered event: [MCP_EV_LOOP_DETECTED]
Next state: [No transition found]
4) FSM:<Ethernet1/4> Transition at 355513 usecs after Fri Mar 16 07:35:43 2018
Previous state: [MCP_ST_OPER_LOOP_DETECTED]
Triggered event: [MCP_EV_LOOP_DETECTED]
Next state: [No transition found]
5) FSM:<Ethernet1/4> Transition at 355561 usecs after Fri Mar 16 07:35:43 2018
Previous state: [MCP_ST_OPER_LOOP_DETECTED]
Triggered event: [MCP_EV_LOOP_DETECTED]
Next state: [No transition found]
6) FSM:<Ethernet1/4> Transition at 358978 usecs after Fri Mar 16 07:35:43 2018
Previous state: [MCP_ST_OPER_LOOP_DETECTED]
Triggered event: [MCP_EV_ADMIN_DISABLE]
Next state: [MCP_ST_OPER_DOWN]
Curr state: [MCP_ST_OPER_DOWN]
>>>>FSM: <Ethernet1/3> has 1 logged transitions<<<<<
1) FSM:<Ethernet1/3> Transition at 653928 usecs after Fri Mar 16 01:48:18 2018
Previous state: [MCP_ST_INITIALIZE]
Triggered event: [MCP_EV_ADMIN_ENABLE]
Next state: [MCP_ST_OPER_UP]
Curr state: [MCP_ST_OPER_UP]
>>>>FSM: <Ethernet1/2> has NO logged transitions<<<<<
>>>>FSM: <Ethernet1/1> has NO logged transitions<<<<<
Once you have narrowed down the interface on which MCP took action, lets dig a bit deeper with the show mcp internal info interface eth 1/4 command. This, btw, is one of the most useful commands. In addition to the wealth of MCP-related information, this command is also very helpful if you are troubleshooting a classic-STP related issue as well with your legacy environment, as you can see the amount of TCNs (topology change notifications) as well as the last time you received a TCN on the interface in question.
This will show display the following info:
- If MCP is enabled on the Interface
- STP packet counts (for MST, RSTP, PVRSTP)
- TCN count
- Last time a TC was received on the interface
- Number of MCP Packets sent
- Number of MCP Packets received << This is the important one, because receiving an MCP packet means a loop has occurred.
- The Vlan (EPGs) on which MCP is being sent
- The Vlan (or EPG) on which the MCP packet was received
Leaf202# show mcp internal info interface eth 1/4
------------------------------------------
Interface: Ethernet1/4
Native PI VLAN: 0
Native Encap VLAN: 0
BPDU Guard: disabled
BPDU Filter: disabled
Port State: down
Layer3 Port: false
Switching State: enabled
Mac Address: 50:f:80:42:5:64
Interface MCP enabled: true
------------------- STP STATS --------------------
MSTP Count: 0
RSTP Count: 10443
MSTP TC Count: 0
RSTP TC Count: 0
PVRSTP TC Count: 14
TCN Count: 0 PVID Error BPDU Count: 0
Error Packet Count: 10447
BPDU Guard Event Count: 0
Last TC received at Fri Mar 16 01:57:12 2018
--------------- LOOP-DETECTION STATS ---------------
MCP packets sent(Per-vlan): 10442
MCP packets received: 4
MCP invalid packets received: 0
MCP packets received with invalid digest: 0
MCP packets received when switching state is disabled: 0
Number of active VLANs : 7
Number of VLANS in MCP packets are sent: 7
MCP enabled vlans: 50, 99, 100, 101, 102, 103, 1500
MCP loop detected at: Fri Mar 16 07:35:43 2018
MCP loop detected in VLAN: 100
-------------- MCP Remote Peer Info --------------
No remote peers exist
Finally, a great command that it is not related to MCP per se, but rather useful when it comes to tracking down Spanning-Tree TCNs, the show mcp internal info interface vlan XXX command (where XXX is your encap vlan).
Why do we care about tracking down Spanning-tree TCNs? ACI will react to a Spanning-tree TCN by flushing endpoints from the EPG on which it was received. This can result in intermittent traffic loss in the EPG, especially if this is occurring frequently, as the endpoints are flushed and then re-learned.
The show mcp internal info interface vlan XXX command will show display the following info:
- The number of STP Topology Change Notifications
- Timestamp of last BD Flush due to STP TCN
- The interface the last TCN was received on
- A physical interface indicates the TCN was locally received (i.e., directly connected
- An SVI interface indicates the TCN was received via the BD interface (i.e., the source of the TCN is not local)
Leaf201# show mcp internal info vlan 101
-------------------------------------------------
PI VLAN: 28 Up
Encap VLAN: 101 << Encap Vlan defined in EPG
PVRSTP TC Count: 34 << # of STP Topology change packets
RSTP TC Count: 0
Last TC flush at Thu Feb 28 16:54:53 2019 << Last TCN packet received
on Ethernet1/13 << Interface last TCN received
Leaf204# show mcp internal info vlan 101
-------------------------------------------------
PI VLAN: 14 Up
Encap VLAN: 101
PVRSTP TC Count: 11
RSTP TC Count: 0
Last TC flush at Thu Feb 28 16:54:53 2019
on Vlan13 << TCN received on BD and the source is not local
Now – the above command executed at the leaf level is nice, but if you really want to gain an appreciation for how awesome ACI is, let’s bring the APIC into the picture and execute this command for ALL LEAFS on the fabric from the APIC itself. This will allow us to quickly track down which interface on which Leaf switch the TCN is being received!
The fabric 201-206 show mcp internal info interface vlan XXX command will execute our MCP command across Leaf switch nodes 201 through 206, in succession.
apic1# fabric 201-206 show mcp internal info vlan 101
-------------------------------------------------
Node 201 (Leaf201)
-------------------------------------------------
PI VLAN: 28 Up
Encap VLAN: 101
PVRSTP TC Count: 34
RSTP TC Count: 0
Last TC flush at Thu Feb 28 16:54:53 2019
on Ethernet1/13
-------------------------------------------------
Node 202 (Leaf202)
-------------------------------------------------
PI VLAN: 26 Up
Encap VLAN: 101
PVRSTP TC Count: 11
RSTP TC Count: 0
Last TC flush at Thu Feb 28 16:54:53 2019
on port-channel1
-------------------------------------------------
Node 203 (Leaf203)
-------------------------------------------------
PI VLAN: 14 Up
Encap VLAN: 101
PVRSTP TC Count: 11
RSTP TC Count: 0
Last TC flush at Thu Feb 28 16:54:53 2019
on Vlan13
-------------------------------------------------
Node 204 (Leaf204)
-------------------------------------------------
-------------------------------------------------
Node 205 (Leaf205)
-------------------------------------------------
-------------------------------------------------
Node 206 (Leaf206)
-------------------------------------------------
One thought on “Using MCP (MisCabling Protocol) for ACI”