Glossary		Print Page		Document Feedback		Support & Services		Pluribus Networks		Pluribus Blog		LinkedIn		Facebook		Twitter

Achieving a Loop-Free Layer 2 Topology

Note: This feature can be configured only in a full mesh topology.

Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP) ensure a loop-free topology in the Layer 2 as far as the networking equipment is concerned. Though RSTP prevents loops in the network caused by mis-cabled networking equipment, the protocol does not address mis-configured hosts. NetVisor OS Loop Detection operates in conjunction with RSTP and MSTP to detect, log, and mitigate misbehaving and misconfigured hosts to prevent looping layer 2 traffic.

NetVisor OS Control Plane — The NetVisor OS control plane includes information about every MAC address in the Layer 2 network in a vPort database. This database is distributed throughout the fabric so that each NetVisor OS switch has a copy of it for the entire fabric.

A MAC address is stored in a vPort, which includes the following information:

MAC address, VLAN ID, and VXLAN ID
Owner-port and local-port
Migration history including owner, time, and port
vPort state as active, static, moving, or loop-probe

Based on control plane data structures including the vPort database, NetVisor OS decides if endpoints are to be allowed to access the network.

Detecting Loops

NetVisor OS Loop Detection is implemented as part of NetVisor OS source MAC address miss handling. NetVisor OS disables hardware learning of MAC addresses, so when a packet arrives with an unknown source MAC address, the switch sends the packet to NetVisor OS rather than switching the packet normally. NetVisor OS examines the vPort table to determine if a packet with an unknown source MAC indicates a loop.

NetVisor OS uses two criteria to detect a loop in the network:

A MAC address associated with an in-band NIC of a node in the fabric appears as the source MAC on a packet that ingresses on a host port. NetVisor OS detects this situation by noting the PN-internal status of a vPort that would otherwise migrate to a host port. NetVisor does not allow the migration to take place and starts loop mitigation.

For the purposes of NetVisor OS Loop Detection, a host port is defined as a port not connected to another Arista switch, not an internal port, and does not participate in STP with NetVisor OS which means that NetVisor OS is not configured for STP or the device connected on the port is not configured for STP.

Packets with the same source MAC address arrive on multiple host ports in the fabric at approximately the same time. In order to support VM and host migration, some rapid movement of MAC addresses through the fabric is tolerated. When the same MAC address moves rapidly back and forth between two ports, a loop is assumed and loop mitigation starts.

VRRP MAC addresses are not subject to loop detection and mitigation, and can migrate freely.

Loops are detected on a port by port basis. A single loop typically involves two ports, either on the same switch or on two different switches. When multiple loops occur with more than two ports then NetVisor OS responds to each port separately.

Loop Mitigation

When NetVisor OS detects a loop, a message appears in the system log indicating the host port and VLAN involved in the loop. In addition the host port involved in the loop has the "loop" status added and NetVisor OS adds the VLAN to the host port loop-vlans VLAN map. Looping ports and VLANs are displayed in the port-show output.

At the start of loop mitigation, NetVisor OS creates vPorts to send loop probe packets. The vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state of loop-probe. NetVisor OS propagates Loop-probe vPorts throughout the fabric. NetVisor OS creates a loop-probe vPort for each looping VLAN.

NetVisor OS deletes all vPorts from the looping host port and VLAN at the start of loop mitigation. This prevents the hardware from sending unicast packets to the looping port, and causes every packet arriving on the looping port to appear in the software as a source MAC miss. During loop mitigation, NetVisor OS drops all packets arriving on the looping port.

During loop mitigation, NetVisor OS sends loop probe packets on the looping VLANs every 3 seconds. As long as the loop persists, NetVisor OS receives the probe packets as source MAC miss notification on the looping ports, so NetVisor OS can determine if the loop is still present. If 9 seconds elapse with no received probe packets, NetVisor OS detects the loop is resolved and ends loop mitigation.

At the end of loop mitigation, log messages are added to the system log, loop-probe vPorts are removed, and loop stats and loop VLANS are removed from the looping port.

To view affected ports, use the port-show command and add the parameter, status loop:

CLI (network-admin@switch-31) > port-show status loop

switch port hostname status config

---------- ---- -------- --------------------- ------

switch-31 9 up,stp-edge-port,loop fd,10g

switch-32 9 up,stp-edge-port,loop fd,10g

Note: the new status, loop, in the status column. When the loops are removed from the port, the loop flag is removed from the port-show status command output and log message is added regarding the removal of loop.

During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:

CLI (network-admin@switch-31) > vport-show state loop-probe

owner mac vlan ports state hostname status

---------- ----------------- ---- ----- ---------- ---------- -----------

switch-32 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal

switch-31 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal

Note the loop-probe state as well as the PN-internal state. The loop probes use the port MAC address format, and use the internal port for the in-band NIC.

Note: The state and the status columns are different in the above vport-show stats loop-probe command output. The status column refers to the vPort peer owner state in the fabric (the PN-internal parameter indicates that the MAC belongs to the PN fabric). The state column displays the vPort state.

If you notice a disruption in the network, use the port-show command to find the looping ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring virtual switches, or as a stop-gap measure, using the port-config-modify command to change port properties for the looping host ports. Once the loop is resolved, NetVisor OS no longer detects probes and leaves the loop mitigation state, while logging a message:

2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system

host_port_loop_resolved(11381) : level=note : port=9 :

Traffic has stopped looping on host-port=9

At this point the loop status is removed from the port-show output for port 9 and the loop-probe vPorts are removed.

NetVisor OS Loop Detection exposes loops using system log messages, port-show output, and vport-show output.

When NetVisor OS detects an internal port MAC address on a host port, NetVisor OS prints a log message as below:

system 2016-01-19,15:36:40.570184-07:00 mac_move_denied

11379 note MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0

from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9

reason=internal MAC of local switch not allowed to change ports

NetVisor OS starts Loop Mitigation by logging a message:

system 2016-01-19,15:36:40.570334-07:00 host_port_loop_detected

11380 warn Looping traffic detected on host-port=9

vlan=1. Traffic on this port/VLAN will be ignored until loop resolved

During Loop Mitigation, NetVisor OS sends loop probes. When these probes, as well as any other packets, are received on a looping host port, NetVisor OS logs a message:

system 2016-01-19,15:59:54.734277-07:00 mac_move_denied

11379 note MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0

from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31

deny-port=9 reason=port is looping

NetVisor OS limits mac_move_denied messages are limited to one every 5 seconds for each vPort. This prevents the system log from filling up with mac_move_denied messages during loop mitigation.

During loop mitigation, you can use the port-show command to see which ports are involved in the loop:

CLI (network-admin@Leaf1) > port-show status loop

switch port hostname status loop-vlans config

------ ---- -------- --------------------- ---------- ------

leaf1 9 up,stp-edge-port,loop 1 fd,10g

Note the loop status in the status column and the loop-vlans column.

During loop mitigation the MAC addresses for loop probes are displayed in the vPort table:

CLI (network-admin@Leaf1) > vport-show state loop-probe

owner mac vlan ports state hostname status

------ ----------------- ---- ----- ---------- -------- ---------

leaf1 06:c0:00:16:f0:45 42 69 loop-probe leo-ext-32 PN-internal

leaf1 06:c0:00:19:c0:45 42 69 loop-probe leo-ext-31 PN-internal