Restoring Ports for Cluster Configurations
Note: This feature is applied only during the initial start-up of the network.
Cluster configuration supports sub-second traffic loss in case of fail-over events.
There are two types of ports providing redundant data paths:
- Layer 3 ports over ECMP redundant routed paths
- Virtual LAGs (vLAGs) providing redundant Layer 2 paths
During failover and recovery port events, it may take measurable time to change the hardware routing and MAC tables on larger networks. This delay incurs traffic loss on the network. To reduce delay, this feature allows you to incrementally restore the ports at start-up. By incrementally restoring the ports, the changes to the hardware are prevented from contending with each other. This reduces the delay between a port up and the hardware updates with the appropriate Layer 3 and Layer 2 information for the port. This process ensures sub-second fail over.
All non-Layer 3 and non-vLAG ports are restored first. This allows the cluster links to activate and the cluster configuration to synchronize information. Layer 3 and vLAG port restoration starts after the cluster synchronizes. This is predicated on the cluster becoming active, all Layer 2 and Layer 3 entries, such as status updates, exchanged, cluster STP status synchronized, and all router interfaces initialized.
Netvisor ONE enforces the following sequence for port bring up (not guaranteed for first upgrade to Netvisor ONE and is allowed only for subsequent reboots):
- Cluster ports, VXLAN-loopback-trunk ports, and Loopback ports
- Layer 3 ports
- vLAG ports
- All other ports
The sequence for port bring-down is:
- Orphan ports (non-vLAG, non-Layer3, and non-Cluster)
- vLAG ports
- Layer 3 ports
- All other ports
If a port is configured with defer-bringup parameter, then that port is brought up along with other ports. All ports except cluster ports can be configured for defer-bringup using the port-config-modify command. You can specify the global timer value for delaying the port bringup by using the cluster-bringup-modify command with port-defer-bringup-delay duration parameter.
The parameter, maximum-sync-delay, controls the maximum time to wait for synchronization in the case where the cluster cannot synchronize information. After synchronization is complete, Layer 3 ports are restored first, since Layer 3 traffic can traverse the cluster link to the peer VLAG port if needed. Currently the reverse is typically not true.
If vLAG ports are restored first, a Layer 3 adjacency between the two cluster nodes may be needed but may not exist in some network configurations. After Layer 3 ports are restored, Netvisor One waits a configurable Layer 3 port to vLAG delay to allow time for the routing protocols to converge and insert the routes. The delay time defaults to 15 seconds.
After the delay, the vLAG ports are restored incrementally and restoring ports incrementally allows enough time to move Layer 2 entries from the cluster link to the port. This incremental restoration of ports also allows the traffic loss to occur in small, 200-300ms per port, rather than one large time span. This is particularly important for server clusters where temporary small losses are not problematic, but fail or timeout for a large continuous traffic loss. If the node coming up is the cluster master, then no staggering and no Layer 3 to VLAG wait is applied. And if the node is the cluster master node, that means the peer is down or coming up, and not handling traffic. Therefore Netvisor ONE safely restores the ports as soon as possible to start traffic flowing between the nodes.
In addition, with version 5.1.1, Netvisor ONE supports the staggered bring up of vRouter vNICs. When configured, the vRouter vNICs, which are not configured on Layer3 ports, are brought up in a staggered manner during the nvOS boot-up on a cluster peer. You can specify the wait time (0-60000 ms) between NIC bring up.
By default, the vRouter interfaces for which the VLAN is up is brought up simultaneously. The bring up can be staggered by specifying a non-zero vrouter-if-staggered-interval. The staggered bring up process is helpful to reduce the traffic loss caused by the simultaneous bring up of all VNICs.
Note: The vRouter interfaces on Layer 3 ports are brought up first and is not staggered. Also, after bootup, the subsequent VLAN interfaces being down or up is not affected by the staggered configuration.
To configure or modify the port bring up process, use the command:
CLI (network-admin@Leaf1) > cluster-bringup-modify
|
cluster-bringup-modify |
Modifies the cluster bring up configuration. |
|
Specify one or more of the following options |
|
|
l3-port-bringup-mode staggered|simultaneous |
Specify the Layer 3 port bring up mode during start up. |
|
l3-port-staggered-iterval duration: #d#h#m#s |
Specify the interval between Layer 3 ports in Layer 3 staggered mode. This can be in days, hours, minutes, or seconds. |
|
vlag-port-bringup-mode staggered|simultaneous |
Specify the vLAG port bring up mode during start up. |
|
vlag-port-staggered-interval duration: #d#h#m#s |
Specify the interval between vLAG ports in vLAG staggered mode. This can be in days, hours, minutes, or seconds. |
|
cluster-trunk-port-staggered-interval duration: #d#h#m#s |
Specify the interval between cluster trunk ports in staggered mode. This can be in days, hours, minutes, or seconds. |
|
maximum-sync-delay duration: #d#h#m#s |
Specify the maximum delay to wait for cluster to synchronize before starting Layer 3 or vLAG port bring up. This can be in days, hours, minutes, or seconds. |
|
l3-to-vlag-delay duration: #d#h#m#s |
Specify the delay between the last Layer 3 port and the first vLAG port bring up. This can be in days, hours, minutes, or seconds. The default value is 15 seconds. |
|
l3-to-vlan-interface-delay duration: #d#h#m#s |
Specify the delay between the last Layer 3 port and the vRouter VLAN interface bring up. |
|
port-defer-bringup-delay duration: #d#h#m#s |
Specify the global timer value to be used for port delay-bring up. |
|
port-defer-bringup-mode staggered|simultaneous |
Specify the port defer bring up mode during start up. |
|
port-defer-bringup-staggered-interval duration: #d#h#m#s |
Specify the interval between ports in defer bring up mode. |
|
vrouter-if-bringup-mode staggered|simultaneous |
Specify the vRouter VLAN interface bring up mode. |
|
vrouter-if-staggered-interval 0..60000 |
Specify the interval in ms between vRouter VLAN interface bring up in staggered mode. The value ranges between 0-60000 milli-seconds. |
|
start-port-enable-delay 0..240 |
Specify the time delay (sec) to start enabling ports (including cluster ports) on the standby switch. |
To display the status of the cluster bring up process, use the cluster-bringup-show command:
CLI (network-admin@Leaf1) > cluster-bringup-show
switch: Leaf1
state: ports-enabled
l3-port-bringup-mode: staggered
l3-port-staggered-interval: 3s
vlag-port-bringup-mode: staggered
vlag-port-staggered-interval: 3s
maximum-sync-delay: 1m
l3-to-vlag-delay: 15s
l3-to-vlan-interface-delay: 0s
port-defer-bringup-delay: 30s
port-defer-bringup-mode: staggered
port-defer-bringup-staggered-interval: 0s
vrouter-if-bringup-mode: staggered
vrouter-if-staggered-interval(ms): 0
To display the status of ports with defer-bringup details on select ports, use the command:
CLI (network-admin@Leaf1) > port-config-show format port,cluster-port,defer-bringup,vlag, port 11,15,42,49
port cluster-port defer-bringup vlag
---- ------------ ------------- ---------
11 no no host-vlag
15 no yes
42 yes no
49 no no
To view the details of a specified port, use the command:
CLI (network-admin@Leaf1) > port-show port 49 format port,ip,mac,vlan,status,config
port ip mac vlan status config
---- ---------- ----------------- ---- ------------------------------------------------ ------
49 50.50.50.2 66:01:01:01:01:01 4092 up,PN-fabric,LLDP,l3-port,remote-l3-port,vlan-up fd,10g