Glossary		Print Page		Document Feedback		Support & Services		Pluribus Networks		Pluribus Blog		LinkedIn		Facebook		Twitter

Achieving a Loop-Free Layer 2 Topology

Note: This feature can be configured only in a full mesh topology.

Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP) ensure a loop-free topology in the Layer 2 as far as the networking equipment is concerned. Though RSTP prevents loops in the network caused by mis-cabled networking equipment, the protocol does not address mis-configured hosts. Netvisor ONE Loop Detection operates in conjunction with RSTP and MSTP to detect, log, and mitigate misbehaving and misconfigured hosts to prevent looping layer 2 traffic.

Netvisor ONE Control Plane — The Netvisor ONE control plane includes information about every MAC address in the Layer 2 network in a vPort database. This database is distributed throughout the fabric so that each Netvisor ONE switch has a copy of it for the entire fabric.

A MAC address is stored in a vPort, which includes the following information:

MAC address, VLAN ID, and VXLAN ID
Owner-port and local-port
Migration history including owner, time, and port
vPort state as active, static, moving, or loop-probe

Based on control plane data structures including the vPort database, Netvisor ONE decides if endpoints are to be allowed to access the network.

Detecting Loops

Netvisor ONE Loop Detection is implemented as part of Netvisor ONE source MAC address miss handling. Netvisor ONE disables hardware learning of MAC addresses, so when a packet arrives with an unknown source MAC address, the switch sends the packet to Netvisor One rather than switching the packet normally. Netvisor ONE examines the vPort table to determine if a packet with an unknown source MAC indicates a loop.

Netvisor ONE uses two criteria to detect a loop in the network:

A MAC address associated with an in-band NIC of a node in the fabric appears as the source MAC on a packet that ingresses on a host port. Netvisor ONE detects this situation by noting the PN-internal status of a vPort that would otherwise migrate to a host port. Netvisor does not allow the migration to take place and starts loop mitigation.

For the purposes of Netvisor ONE Loop Detection, a host port is defined as a port not connected to another Pluribus switch, not an internal port, and does not participate in STP with Netvisor ONE which means that Netvisor One is not configured for STP or the device connected on the port is not configured for STP.

Packets with the same source MAC address arrive on multiple host ports in the fabric at approximately the same time. In order to support VM and host migration, some rapid movement of MAC addresses through the fabric is tolerated. When the same MAC address moves rapidly back and forth between two ports, a loop is assumed and loop mitigation starts.

VRRP MAC addresses are not subject to loop detection and mitigation, and can migrate freely.

Loops are detected on a port by port basis. A single loop typically involves two ports, either on the same switch or on two different switches. When multiple loops occur with more than two ports then Netvisor ONE responds to each port separately.

Loop Mitigation

When Netvisor ONE detects a loop, a message appears in the system log indicating the host port and VLAN involved in the loop. In addition the host port involved in the loop has the "loop" status added and Netvisor ONE adds the VLAN to the host port loop-vlans VLAN map. Looping ports and VLANs are displayed in the port-show output.

At the start of loop mitigation, Netvisor ONE creates vPorts to send loop probe packets. The vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state of loop-probe. Netvisor ONE propagates Loop-probe vPorts throughout the fabric. Netvisor ONE creates a loop-probe vPort for each looping VLAN.

Netvisor ONE deletes all vPorts from the looping host port and VLAN at the start of loop mitigation. This prevents the hardware from sending unicast packets to the looping port, and causes every packet arriving on the looping port to appear in the software as a source MAC miss. During loop mitigation, Netvisor ONE drops all packets arriving on the looping port.

During loop mitigation, Netvisor ONE sends loop probe packets on the looping VLANs every 3 seconds. As long as the loop persists, Netvisor ONE receives the probe packets as source MAC miss notification on the looping ports, so Netvisor ONE can determine if the loop is still present. If 9 seconds elapse with no received probe packets, Netvisor ONE detects the loop is resolved and ends loop mitigation.

At the end of loop mitigation, log messages are added to the system log, loop-probe vPorts are removed, and loop stats and loop VLANS are removed from the looping port.

To view affected ports, use the port-show command and add the parameter, status loop:

CLI (network-admin@switch-31) > port-show status loop

switch port hostname status config

---------- ---- -------- --------------------- ------

switch-31 9 up,stp-edge-port,loop fd,10g

switch-32 9 up,stp-edge-port,loop fd,10g

Note: the new status, loop, in the status column. When the loops are removed from the port, the loop flag is removed from the port-show status command output and log message is added regarding the removal of loop.

During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:

CLI (network-admin@switch-31) > vport-show state loop-probe

owner mac vlan ports state hostname status

---------- ----------------- ---- ----- ---------- ---------- -----------

switch-32 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal

switch-31 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal

Note the loop-probe state as well as the PN-internal state. The loop probes use the port MAC address format, and use the internal port for the in-band NIC.

Note: The state and the status columns are different in the above vport-show stats loop-probe command output. The status column refers to the vPort peer owner state in the fabric (the PN-internal parameter indicates that the MAC belongs to the PN fabric). The state column displays the vPort state.

If you notice a disruption in the network, use the port-show command to find the looping ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring virtual switches, or as a stop-gap measure, using the port-config-modify command to change port properties for the looping host ports. Once the loop is resolved, Netvisor ONE no longer detects probes and leaves the loop mitigation state, while logging a message:

2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system

host_port_loop_resolved(11381) : level=note : port=9 :

Traffic has stopped looping on host-port=9

At this point the loop status is removed from the port-show output for port 9 and the loop-probe vPorts are removed.

Netvisor ONE Loop Detection exposes loops using system log messages, port-show output, and vport-show output.

When Netvisor ONE detects an internal port MAC address on a host port, Netvisor ONE prints a log message as below:

system 2016-01-19,15:36:40.570184-07:00 mac_move_denied

11379 note MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0

from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9

reason=internal MAC of local switch not allowed to change ports

Netvisor ONE starts Loop Mitigation by logging a message:

system 2016-01-19,15:36:40.570334-07:00 host_port_loop_detected

11380 warn Looping traffic detected on host-port=9

vlan=1. Traffic on this port/VLAN will be ignored until loop resolved

During Loop Mitigation, Netvisor ONE sends loop probes. When these probes, as well as any other packets, are received on a looping host port, Netvisor ONE logs a message:

system 2016-01-19,15:59:54.734277-07:00 mac_move_denied

11379 note MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0

from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31

deny-port=9 reason=port is looping

Netvisor ONE limits mac_move_denied messages are limited to one every 5 seconds for each vPort. This prevents the system log from filling up with mac_move_denied messages during loop mitigation.

During loop mitigation, you can use the port-show command to see which ports are involved in the loop:

CLI (network-admin@Leaf1) > port-show status loop

switch port hostname status loop-vlans config

------ ---- -------- --------------------- ---------- ------

leaf1 9 up,stp-edge-port,loop 1 fd,10g

Note the loop status in the status column and the loop-vlans column.

During loop mitigation the MAC addresses for loop probes are displayed in the vPort table:

CLI (network-admin@Leaf1) > vport-show state loop-probe

owner mac vlan ports state hostname status

------ ----------------- ---- ----- ---------- -------- ---------

leaf1 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal

leaf1 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal