Loop-Free Layer 2 Topology
Netvisor ONE Loop Detection operates in conjunction with Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP). Netvisor ONE uses RSTP and MSTP to ensure loop free topology of the VLANs in the Layer 2 network.
RSTP prevents loops in the network caused by miscabled networking equipment, but does not address misconfigured hosts. Netvisor ONE Loop Detection goes beyond STP to protect the network from misconfigured or miscabled hosts attached to the network
Netvisor ONE Control Plane — The Netvisor ONE control plane includes information about every MAC address attached to the Layer 2 network in a vPort database. Netvisor ONE distributes the vPort database throughout the fabric so that each Netvisor ONE switch contains a copy of the vPort database for the entire fabric.
A MAC address stored in a vPort, includes the following information:
MAC address, VLAN ID, and VXLAN ID
Owner-port and local-port
Migration history including owner, time, and port
vPort state as active, static, moving, or loop-probe
Access to the Netvisor ONE fabric goes through the Netvisor ONE software. Netvisor ONE determines if endpoints access the network based on control plane data structures including the vPort database.
Detecting Loops
Netvisor ONE implements Netvisor ONE Loop Detection as part of Netvisor ONE source MAC address mishandling. Netvisor ONE disables hardware learning of MAC addresses, when a packet arrives with an unknown MAC address, the switch sends the packet to Netvisor ONE rather than switching the packet normally. Netvisor ONE examines the vPort table to determine if a packet with an unknown MAC indicates a loop.
Netvisor ONE uses two criteria to detect a loop on the network:
A MAC address associated with an in-band NIC of a node in the fabric appears as the source MAC on a packet that ingresses on a host port. Netvisor ONE detects this situation by noting the PN-internal status of a vPort migrating to a host port. Netvisor ONE prevents the migration to take place and starts loop mitigation.
For the purposes of Netvisor ONE Loop Detection, Netvisor ONE defines a host port as a port not connected to another Pluribus switch, not an internal port, and disables participation in STP with Netvisor ONE. Netvisor ONE disables STP on the switch and the device connected on the port.
Packets with the same source MAC address arrive on multiple host ports in the fabric at approximately the same time. In order to support VM and host migration, Netvisor ONE tolerates some rapid movement of MAC addresses through the fabric. When the same MAC address moves rapidly back and forth between two ports, Netvisor ONE detects a loop and loop mitigation starts.
VRRP MAC addresses do not participate Loop Detection and Mitigation, and migrate freely.
Netvisor ONE detects loops on a port by port basis. A single loop typically involves two ports, either on the same switch or on two different switches. When multiple loops occur with more than two ports then Netvisor ONE responds to each port separately.
Loop Mitigation
When Netvisor ONE detects a loop, a message appears in the system log indicating the host port and VLAN involved in the loop. In addition the host port involved in the loop has the "loop" status added and Netvisor ONE adds the VLAN to the host port loop-vlans VLAN map. Looping ports and VLANs are displayed in the port-show output.
At the start of loop mitigation, Netvisor ONE creates vPorts to send loop probe packets. The vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state of loop-probe. Netvisor ONE propagates Loop-probe vPorts throughout the fabric. Netvisor ONE creates a loop-probe vPort for each looping VLAN.
At the start of loop mitigation Netvisor ONE deletes all vPorts from the looping host port and VLAN. This prevents the hardware from sending unicast packets to the looping port, and causes every packet arriving on the looping port to appear in the software as a source MAC miss. During loop mitigation, Netvisor ONE drops all packets arriving on the looping port.
During loop mitigation, Netvisor ONE sends loop probe packets on the looping VLANs every 3 seconds. As long as the loop persists, Netvisor ONE receives the probe packets as source MAC miss notification on the looping ports, so Netvisor ONE can determine if the loop is still present. If 9 seconds elapse with no received probe packets, Netvisor ONE detects the loop is resolved and ends loop mitigation.
At the end of loop mitigation, log messages are added the system log, loop-probe vPorts are removed, and loop stats and loop VLANS are removed from the looping port.
To view affected ports, use the port-show command and add the parameter, status loop:
network-admin@switch-31>port-show status loop
switch port hostname status config
---------- ---- -------- --------------------- ------
switch-31 9 up,stp-edge-port,loop fd,10g
switch-32 9 up,stp-edge-port,loop fd,10g
Note the new status, loop, in the status column.
During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:
<CLI (network-admin@switch-31) > vport-show state loop-probe
owner mac vlan ports state hostname status
---------- ----------------- ---- ----- ---------- ---------- -----------
switch-32 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal
switch-31 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal
 
Note the loop-probe state as well as the PN-internal state. The loop probes use the port MAC address format, and use the internal port for the in-band NIC.
If you notice a disruption in the network, use the port-show command to find the looping ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring virtual switches, or as a stop-gap measure, using the port-config-modify command to change port properties for the looping host ports. Once you resolve the loop, Netvisor ONE no longer detects probes and leaves the loop mitigation state, while logging a message:
2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system
host_port_loop_resolved(11381) : level=note : port=9 :
Traffic has stopped looping on host-port=9
At this point Netvisor ONE removes the loop status from the port-show output for port 9 and deletes the loop-probe vPorts.
Netvisor ONE Loop Detection exposes loops using system log messages, port-show output, and vport-show output. Enable or disable Netvisor ONE Loop Detection by using the system-settings-modify command:
network-admin@e68-leaf-01>system-settings-modify block-loops
network-admin@e68-leaf-01>system-settings-modify no-block-loops
 
When Netvisor ONE detects an internal port MAC address on a host port, Netvisor ONE prints a log message:
system 2016-01-19,15:36:40.570184-07:00 mac_move_denied
11379 note MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9
reason=internal MAC of local switch not allowed to change ports
 
Netvisor ONE starts Loop Mitigation by logging a message:
system 2016-01-19,15:36:40.570334-07:00 host_port_loop_detected
11380 warn Looping traffic detected on host-port=9
vlan=1. Traffic on this port/VLAN will be ignored until loop resolved
 
During Loop Mitigation, Netvisor ONE sends loop probes. When these probes, as well as any other packets received on a looping host port, Netvisor ONE logs a message:
 
system 2016-01-19,15:59:54.734277-07:00 mac_move_denied
11379 note MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0
from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31
deny-port=9 reason=port is looping
 
Netvisor ONE limits mac_move_denied messages to one every 5 seconds for each vPort. This prevents the system log from filling up with mac_move_denied messages during loop mitigation.
During loop mitigation, use the port-show command to see ports involved in the loop:
 
CLI network-admin@Leaf1 > port-show status loop
switch port hostname status                loop-vlans config
---------- ---- -------- --------------------- ---------- ------
e68-leaf-01 9 up,stp-edge-port,loop 1 fd,10g
e68-leaf-01 9 up,stp-edge-port,loop 1 fd,10g
 
Note the loop status in the status column and the loop-vlans column.
During loop mitigation the MAC addresses for loop probes Netvisor ONE displays the vPort table:
 
CLI network-admin@Leaf1 > vport-show state loop-probe,
owner  mac     vlan  ports      state hostname status
---------- ----------------- ---- ----- ---------- --------   --------- --------
e68-leaf-01 06:c0:00:16:f0:45    42     69         loop-probe leo-ext-32 PN-internal
e68-leaf-01 06:c0:00:19:c0:45    42     69         loop-probe leo-ext-31 PN-internal