Achieving a Loop-Free Layer 2 Topology
Netvisor One Loop Detection operates in conjunction with Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP). RSTP and MSTP are is used to ensure loop free topology of the VLANs in the Layer 2 network as far as the networking equipment is concerned.
RSTP prevents loops in the network caused by mis-cabled networking equipment, but does not address mis-configured hosts. Netvisor One Loop Detection goes beyond STP to protect the network from mis-configured or mis-cabled hosts attached to the network.
Netvisor One Control Plane — The Netvisor One control plane includes information about every MAC address attached to the Layer 2 network in a vPort database. The vport database is distributed throughout the fabric so that each Netvisor One switch has a copy of the vPort database for the entire fabric.
A MAC address is stored in a vPort, which includes the following information:
- MAC address, VLAN ID, and VXLAN ID
- Owner-port and local-port
- Migration history including owner, time, and port
- vPort state as active, static, moving, or loop-probe
Access to the Netvisor One fabric goes through the Netvisor One software. Netvisor One determines if endpoints access the network based on control plane data structures including the vPort database.
Netvisor One Loop Detection is implemented as part of Netvisor One source MAC address miss handling. Netvisor One disables hardware learning of MAC addresses, so when a packet arrives with an unknown MAC address, the switch sends the packet to Netvisor One rather than switching the packet normally. Netvisor One examines the vPort table to determine if a packet with an unknown MAC indicates a loop.
Netvisor One uses two criteria to detect a loop on the network:
- A MAC address associated with an in-band NIC of a node in the fabric appears as the source MAC on a packet that ingresses on a host port. Netvisor One detects this situation by noting the PN-internal status of a vPort that would otherwise migrate to a host port. Netvisor One does not allow the migration to take place and starts loop mitigation.
For the purposes of Netvisor One Loop Detection, a host port is defined as a port not connected to another Pluribus switch, not an internal port, and does not participate in STP with Netvisor One which means that Netvisor One is not configured for STP or the device connected on the port is not configured for STP.
- Packets with the same source MAC address arrive on multiple host ports in the fabric at approximately the same time. In order to support VM and host migration, some rapid movement of MAC addresses through the fabric is tolerated. When the same MAC address moves rapidly back and forth between two ports, a loop is assumed and loop mitigation starts.
VRRP MAC addresses are not subject to Loop Detection and Mitigation, and can migrate freely.
Loops are detected on a port by port basis. A single loop typically involves two ports, either on the same switch or on two different switches. When multiple loops occur with more than two ports then Netvisor One responds to each port separately.
When Netvisor ONE detects a loop, a message appears in the system log indicating the host port and VLAN involved in the loop. In addition the host port involved in the loop has the "loop" status added and Netvisor ONE adds the VLAN to the host port loop-vlans VLAN map. Looping ports and VLANs are displayed in the port-show output.
At the start of loop mitigation, Netvisor ONE creates vPorts to send loop probe packets. The vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state of loop-probe. Netvisor ONE propagates Loop-probe vPorts throughout the fabric. Netvisor ONE creates a loop-probe vPort for each looping VLAN.
At the start of loop mitigation Netvisor ONE deletes all vPorts from the looping host port and VLAN. This prevents the hardware from sending unicast packets to the looping port, and causes every packet arriving on the looping port to appear in the software as a source MAC miss. During loop mitigation, Netvisor ONE drops all packets arriving on the looping port.
During loop mitigation, Netvisor ONE sends loop probe packets on the looping VLANs every 3 seconds. As long as the loop persists, Netvisor ONE receives the probe packets as source MAC miss notification on the looping ports, so Netvisor ONE can determine if the loop is still present. If 9 seconds elapse with no received probe packets, Netvisor ONE detects the loop is resolved and ends loop mitigation.
At the end of loop mitigation, log messages are added the system log, loop-probe vPorts are removed, and loop stats and loop VLANS are removed from the looping port.
To view affected ports, use the port-show command and add the parameter, status loop:
CLI (network-admin@switch-31) > port-show status loop
switch port hostname status config
---------- ---- -------- --------------------- ------
switch-31 9 up,stp-edge-port,loop fd,10g
switch-32 9 up,stp-edge-port,loop fd,10g
Note the new status, loop, in the status column.
During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:
CLI (network-admin@switch-31) > vport-show state loop-probe
owner mac vlan ports state hostname status
---------- ----------------- ---- ----- ---------- ---------- -----------
switch-32 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal
switch-31 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal
Note the loop-probe state as well as the PN-internal state. The loop probes use the port MAC address format, and use the internal port for the in-band NIC.
If you notice a disruption in the network, use the port-show command to find the looping ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring virtual switches, or as a stop-gap measure, using the port-config-modify command to change port properties for the looping host ports. Once the loop is resolved, Netvisor One no longer detects probes and leaves the loop mitigation state, while logging a message:
2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system
host_port_loop_resolved(11381) : level=note : port=9 :
Traffic has stopped looping on host-port=9
At this point the loop status is removed from the port-show output for port 9 and the loop-probe vPorts are removed.
Netvisor One Loop Detection exposes loops using system log messages, port-show output, and vport-show output. Netvisor One Loop Detection is enabled or disabled by using the sys-flow-setting-modify command:
When Netvisor One detects an internal port MAC address on a host port, Netvisor One prints a log message:
system 2016-01-19,15:36:40.570184-07:00 mac_move_denied
11379 note MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9
reason=internal MAC of local switch not allowed to change ports
Netvisor One starts Loop Mitigation by logging a message:
system 2016-01-19,15:36:40.570334-07:00 host_port_loop_detected
11380 warn Looping traffic detected on host-port=9
vlan=1. Traffic on this port/VLAN will be ignored until loop resolved
During Loop Mitigation, Netvisor One sends loop probes. When these probes, as well as any other packets, are received on a looping host port, Netvisor One logs a message:
system 2016-01-19,15:59:54.734277-07:00 mac_move_denied
11379 note MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31
deny-port=9 reason=port is looping
Netvisor One limits mac_move_denied messages are limited to one every 5 seconds for each vPort. This prevents the system log from filling up with mac_move_denied messages during loop mitigation.
During loop mitigation, you can use the port-show command to see which ports are involved in the loop:
CLI (network-admin@Leaf1) > port-show status loop
switch port hostname status loop-vlans config
------ ---- -------- --------------------- ---------- ------
leaf1 9 up,stp-edge-port,loop 1 fd,10g
leaf1 9 up,stp-edge-port,loop 1 fd,10g
Note the loop status in the status column and the loop-vlans column.
During loop mitigation the MAC addresses for loop probes are displayed the vPort table:
CLI (network-admin@Leaf1) > vport-show state loop-probe,
owner mac vlan ports state hostname status
------ ----------------- ---- ----- ---------- -------- ---------
leaf1 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal
leaf1 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal