Auto-Recovery of a Disabled Port

Physical ports are automatically disabled by Netvisor OS due to certain violations. For example, if a port receives BPDU messages from an edge port, Netvisor OS disables the port because receiving BPDUs on a edge port is a security violation. However, there is no way to indicate that the port is shut down because of a violation and not because of physical link issues.

The port may be disabled due to the following errors:

This feature allows you to configure an automatic retry to enable the port after a configured timeout.

CLI network-admin@Leaf1 > err-disable-counters-clear

bpduguard|no-bpduguard

Specify if you want BPDU guard enabled.

macsecurity|no-macsecurity

Specify if you want MAC recovery enabled.

recovery-timer duration: #d#h#m#s

Specify the recovery time value. The default timer value is 5 minutes

CLI network-admin@Leaf1 > err-disable-modify

bpduguard|no-bpduguard

Specify if you want BPDU guard enabled.

macsecurity|no-macsecurity

Specify if you want MAC recovery enabled.

recovery-timer duration: #d#h#m#s

Specify the recovery time value. The default timer value is 5 minutes.

CLI network-admin@Leaf1 > err-disable-show

bpduguard|no-bpduguard

Display BPDU guard settings.

macsecurity|no-macsecurity

Display MAC recovery settings.

recovery-timer duration: #d#h#m#s

Display the recovery time value.

CLI network-admin@Leaf1 > err-disable-show

switch:         Leaf1

bpduguard:      off

macsecurity:    off

recovery-timer: 5m

Loop-Free Layer 2 Topology

Netvisor OS Loop Detection operates in conjunction with Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP). RSTP and MSTP are is used to ensure loop free topology of the VLANs in the Layer 2 network as far as the networking equipment is concerned.

RSTP prevents loops in the network caused by miscabled networking equipment, but does not address misconfigured hosts. Netvisor OS Loop Detection goes beyond STP to protect the network from misconfigured or miscabled hosts attached to the network

Netvisor OS Control Plane — The Netvisor OS control plane includes information about every MAC address attached to the Layer 2 network in a vPort database. The vport database is distributed throughout the fabric so that each Netvisor OS switch has a copy of the vPort database for the entire fabric.

A MAC address is stored in a vPort, which includes the following information:

Access to the Netvisor OS fabric goes through the Netvisor OS software. Netvisor OS determines if endpoints access the network based on control plane data structures including the vPort database.

Detecting Loops

Netvisor OS Loop Detection is implemented as part of Netvisor OS source MAC address miss handling. Netvisor OS disables hardware learning of MAC addresses, so when a packet arrives with an unknown MAC address, the switch sends the packet to Netvisor OS rather than switching the packet normally. Netvisor OS examines the vPort table to determine if a packet with an unknown MAC is indicative of a loop.

Netvisor OS uses two criteria to detect a loop on the network:

For the purposes of Netvisor OS Loop Detection, a host port is defined as a port not connected to another Pluribus switch, not an internal port, and does not participate in STP with Netvisor OS which means that Netvisor OS is not configured for STP or the device connected on the port is not configured for STP.

VRRP MAC addresses are not subject to Loop Detection and Mitigation, and can migrate freely.

Loops are detected on a port by port basis. A single loop typically involves two ports, either on the same switch or on two different switches. When multiple loops are present, more than two ports are involved and each port is handled separately.

Loop Mitigation

When a loop is detected a message is logged to the system log indicating the host port and VLAN involved in the loop. In addition the host port involved in the loop has the "loop" status added and Netvisor OS adds the VLAN to the host port loop-vlans VLAN map, so that looping ports and VLANs are displayed in the port-show output.

At the start of loop mitigation, Netvisor OS creates vPorts to send loop probe packets. The vPorts use the port MAC address for the in-band NIC port, status of PN-internal, and a state of loop-probe. Loop-probe vPorts are propagated throughout the fabric. Netvisor OS creates a loop-probe vPort for each looping VLAN.

At the start of loop mitigation Netvisor OS deletes all vPorts from the looping host port and VLAN. This prevents the hardware from sending unicast packets to the looping port, and causes every packet arriving on the looping port to appear in the software as a source MAC miss. During loop mitigation, all packets arriving on the looping port are dropped.

During loop mitigation Netvisor OS sends loop probe packets on the looping VLANs every 3 seconds. As long as the loop persists, Netvisor OS receives the probe packets as source mac miss notification on the looping ports, so Netvisor OS can determine if the loop is still present. If 9 seconds elapse with no received probe packets, Netvisor OS detects the loop is resolved and ends loop mitigation.

At the end of loop mitigation, log messages are added the system log, loop-probe vPorts are removed, and loop stats and loop VLANS are removed from the looping port.

To view affected ports, use the port-show command and add the parameter, status loop:

network-admin@switch-31>port-show status loop

switch     port hostname status                config

   ---------- ---- -------- --------------------- ------

   switch-31 9             up,stp-edge-port,loop fd,10g

   switch-32 9             up,stp-edge-port,loop fd,10g

Note the new status, loop, in the status column.

During loop mitigation, the MAC addresses for loop probes are displayed in the vPort table:

CLI (network-admin@switch-31) > vport-show state loop-probe

owner      mac               vlan ports state      hostname   status      

---------- ----------------- ---- ----- ---------- ---------- -----------

switch-32 06:c0:00:16:f0:45 42   69    loop-probe leo-ext-32 PN-internal

switch-31 06:c0:00:19:c0:45 42   69    loop-probe leo-ext-31 PN-internal

 

Note the loop-probe state as well as the PN-internal state. The loop probes use the port MAC address format, and use the internal port for the in-band NIC.

If you notice a disruption in the network, use the port-show command to find the looping ports, and fix the loop. Fixing the loop typically involves correcting cabling issues, configuring virtual switches, or as a stop-gap measure, using the port-config-modify command to change port properties for the looping host ports. Once the loop is resolved, Netvisor OS no longer detects probes and leaves the loop mitigation state, while logging a message:

2016-01-12,12:18:41.911799-07:00 leo-ext-31 nvOSd(25695) system

host_port_loop_resolved(11381) : level=note : port=9 :

Traffic has stopped looping on host-port=9

At this point the loop status is removed from the port-show output for port 9 and the loop-probe vPorts are removed.

Netvisor OS Loop Detection exposes loops using system log messages, port-show output, and vport-show output. Netvisor OS Loop Detection is enabled or disabled by using the sys-flow-setting-modify command:

network-admin@e68-leaf-01>system-settings-modify block-loops

network-admin@e68-leaf-01>system-settings-modify no-block-loops

 

The block-loops argument for sys-flow-setting-modify is not available on the F64.

When Netvisor OS detects an internal port MAC address on a host port, Netvisor OS prints a log message:

system   2016-01-19,15:36:40.570184-07:00 mac_move_denied

       11379 note  MOVE DENIED mac=64:0e:94:c0:03:b3 vlan=1 vxlan=0

       from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31 deny-port=9

       reason=internal MAC of local switch not allowed to change ports

 

Netvisor OS starts Loop Mitigation by logging a message:

system   2016-01-19,15:36:40.570334-07:00 host_port_loop_detected

       11380 warn  Looping traffic detected on host-port=9

       vlan=1. Traffic on this port/VLAN will be ignored until loop resolved

 

During Loop Mitigation, Netvisor OS sends loop probes. When these probes, as well as any other packets, are received on a looping host port, Netvisor OS logs a message:

 

       system   2016-01-19,15:59:54.734277-07:00 mac_move_denied

       11379 note  MOVE DENIED mac=06:c0:00:19:c0:45 vlan=1 vxlan=0

       from switch=leo-ext-31 port=69 to deny-switch=leo-ext-31

       deny-port=9 reason=port is looping

 

mac_move_denied messages are limited to one every 5 seconds for each vPort. This prevents the system log from filling up with mac_move_denied messages during loop mitigation.

During loop mitigation, you can use the port-show command to see which ports are involved in the loop:

 

CLI network-admin@Leaf1 > port-show status loop

switch     port hostname status                loop-vlans config

---------- ---- -------- --------------------- ---------- ------

e68-leaf-01 9             up,stp-edge-port,loop 1          fd,10g

e68-leaf-01 9             up,stp-edge-port,loop 1          fd,10g

 

Note the loop status in the status column and the loop-vlans column.

During loop mitigation the MAC addresses for loop probes are displayed the vPort table:

 

CLI network-admin@Leaf1 > vport-show state loop-probe,

owner       mac                   vlan  ports      state      hostname   status      

---------- ----------------- ---- ----- ---------- --------   --------- --------

e68-leaf-01 06:c0:00:16:f0:45    42     69         loop-probe leo-ext-32 PN-internal

e68-leaf-01 06:c0:00:19:c0:45    42     69         loop-probe leo-ext-31 PN-internal

 

Managing Control Plane Traffic Protection (CPTP)

This feature is supported on the following platforms:

F9272-X

AS5712-54X

F9232-C

AS6712-32X

FF9372-T

AS5812-54T

This feature is supported on the following platforms:

S4048-ON

Z9100-ON

Control Plane Traffic Protection (CPTP) applies to the internal control, data, and span ports which all connect to the CPU, so the CPU resources are protected from large quantities of traffic arriving from different sources such as control packets, cluster communication, fabric updates as well as the regular flood traffic, learning packets and copy-to-cpu packets.

The purpose of CPTP is to classify the traffic on the hardware to different Class of Service (CoS), and perform priority scheduling between them, and also apply a rate limit for each of the CoS, to protect the CPU resources and at the same time, provide a Service Level Agreement (SLA) for critical traffic.

CLI network-admin@Leaf1 > port-cos-rate-setting-show

switch    port  port-number cos0-rate cos1-rate cos2-rate cos3-rate cos4-rate cos5-rate cos6-rate

--------- ----- ----------- --------- --------- --------- --------- --------- --------- --------

Spine1    pci-e 0           100       100       1000000   1000000   1000000   1000000   1000000

Spine1    data 65           100       100       1000000   1000000   1000000   1000000   1000000

Spine1    span 66           100       100       1000000   1000000   1000000   1000000   1000000

 

On ONVL switches, the following output is displayed:

switch:         Leaf1

port:           control-port

ports:          0

cos0-rate(pps): 5000

cos1-rate(pps): 5000

cos2-rate(pps): 5000

cos3-rate(pps): 5000

cos4-rate(pps): 5000

cos5-rate(pps): 5000

cos6-rate(pps): 5000

cos7-rate(pps): 5000

You can modify the CoS rate settings using the port-cos-rate-setting-modify command. The rate limits are set in packets per second.

CLI network-admin@Leaf1 > port-cos-stats-show

switch:     Spine1

time:        11:59:15

port:        0

cos0-out:    58.8M

cos0-drops:  180M

cos1-out:    58.8M

cos1-drops:  185M

cos2-out:    0

cos2-drops:  0

cos3-out:    0

cos3-drops:  0

cos4-out:    0

cos4-drops:  0

cos5-out:    0

cos5-drops:  0

cos6-out:    65.5M

cos6-drops:  1.06G

cos7-out:    483K

cos7-drops:  493MTo clear the statistics for CoS on the ports, use the port-cos-stats-clear command.