About the Netvisor OS Fabric

A fabric is defined as a distributed architecture based on a collection of compute clustering techniques to present an open, standard-based Ethernet fabric as one logical switch. Every node shares the same view of the fabric including MAC and IP addresses, connections, and application flows.

When you add switches to the fabric, all switches are under a single management domain which is highly available through multiple link aggregation and load balancing between network resources.

The fabric performs a classic database 3-phase commit for configuration changes. All members of the fabric must accept the configuration changes before the change is made in the fabric. Figure 1  Fabric Architecture displays the fabric architecture of Netvisor OS.

Figure 1: Fabric Architecture

ONVL-Fabric.v11.jpg

Types of Fabric Communications

For UDP and TCP protocols, Netvisor uses the default ports, however, you can modify the ports using the command, fabric-comm-ports range 1024-65435.

To display the current communication ports, use the command, fabric-comm-ports-show:

CLI network-admin@switch > fabric-comm-ports-show format all

switch:                          Spine1

range-start:                     23300

fabric-port:                     23399

notify-port:                     23398

proxy-port:                      23397

fabric-keepalive-port:           23394

filesystem-replication-port:     23392

cluster-traffic-forwarding-port: 23391

vport-statistics-port:           23390

l2-encap-port:                   23389

igmp-encap-port:                 23388

icmpv6-encap-port:               23387

arp-encap-port:                  23386

cluster-analytics-port:          23385

 

TCP Ports

Communication is unicast unless otherwise noted.

Port

Communication

23399

Transactions, vLAG Synchronization

23398

Status Updates

23397

Proxy I/O, Cluster Synchronization, STP, and Configuration

23396

Client Access Port

23392

File System Replication

UCP Ports

Communication is unicast unless otherwise noted.

Port

Communication

23399

Global Discovery/Keep-alive (multicast), Internal Keep-alive (multicast through a socket connection)

23398

Fabric Notifications

23395

Internal keep-alives (multicast through the CPU)

23394

Fabric Keep-alives

23391

Forwarded Packets

23390

Host Vport Statistics (multicast)

23389

L2-encapped

23388

IGMP-encapped

23387

ICMPV6-encapped

23386

ARP-encapped

Communication Types by Multicast Address

Fabric Networks and Control Networks

A Netvisor fabric has two networks controlling communication: the fabric network and control network. You can configure the communication over a management or in-band interface. Configuring a management interface means the associated communications run over the management interface. Configuring a in-band interface means associated communication runs over the in-band interface or the cluster VNIC interface. If there is no physical rear-facing network interface, traffic through the in-band or cluster VNIC interfaces traverses the PCIE interface.

A fabric network consists of the following communications:

A control network consists of the following communications:

A fabric configured without a fabric network or a control network consists of the following communications:

Creating an Initial Fabric

After you complete the initial setup on the switch, you must create a new fabric for the switch or join an existing fabric. When switches form a fabric, the fabric becomes one logical switch, and shares state as well as communicates commands so that any scope of a fabric- command is executed on each switch in the fabric. A switch must be in a fabric in order to keep track of the fabric state. However, a switch can be a member of fabric, and consist of a single switch. A switch leaving one fabric and joining another loses the fabric state of the first fabric and learns the fabric state of the second fabric.

1. To create a new fabric , use the following command:

CLI network-admin@switch > fabric-create name name-string

2. Create a name for the new fabric.

If you want to assign a password to the fabric that allows other switches to securely join the fabric only if the administrator knows the password, use the password option. Press the return key after typing the password parameter:

CLI network-admin@switch > fabric-create name name-string <return>

password:*******

Re-enter password:*******

By default, the fabric is created on VLAN1. You can specify a different VLAN, but if you change the VLAN, you must recreate the fabric.

To join a remote fabric use the fabric-join command and the switch IP address. For example,

CLI network-admin@switch > fabric-join switch-ip 192.168.2.2 vlan 20

3. To show fabric details, use the fabric-show command:

 

 

 

CLI network-admin@switch > fabric-show

name:                         Spine1

id:                           b00070e:5a6b66aa

vlan:                         1

fabric-network:               mgmt

control-network:              mgmt

tid:                          30

fabric-advertisement-network: inband-mgmt

name:                         l1-fabric

id:                           c000235:5a7b7996

vlan:                         1

fabric-network:               mgmt

control-network:              mgmt

tid:                          0

fabric-advertisement-network: inband-mgmt

Adding Switches to an Existing Fabric

For this example, the switches are connected as in Figure 4  Directly Connected Switches in a Fabric:

Figure 4: Directly Connected Switches in a Fabric

fabric-connection.png

When you have more than one switch, you must add it to the fabric to take advantage of the features offered by Netvisor OS. To add the new switch, use the following command on one of the switches:

CLI network-admin@switch > fabric-join name corp-fabric

 

You can join the fabric using either the fabric name or the switch IP address. If you use the Tab key to display the available options, all fabrics configured on the network are displayed as options.

If you specify a password for the fabric, you must type it in twice. The password is used to encrypt communication between the nodes in the fabric. When you join the fabric from a node, you must type in the password to join it.

You can specify a specific VLAN for the fabric when you create a new one, or by default, the fabric uses VLAN1. However, you cannot change the fabric VLAN without recreating the fabric.

 

Informational Note: 

Avoid creating fabrics with the same name.

 When the fabric is created, the switch begins sending multicast messages out on Layer 2 looking for other switches. These messages are not propagated to other networks. This is how Switch B in Figure 4 learns about the fabric.

Once Switch B joins the fabric, the fabric configuration (commands with scope fabric) is downloaded on Switch B and the switch reboots.

If you want to connect to a switch over Layer 3, you must specify the IP address for the switch in the fabric using the following command:

CLI network-admin@switch > fabric-join switch-ip 192.168.11.1

fabric joined.

 

Fabric Over Management Interface

You can now configure fabric communication run over either the management interface or the in-band interface. Because fabric communication over the in-band interface can be disrupted due to STP, ports flapping, and other factors, fabric communication over management provides a more consistent configuration.

If you create a fabric with the management interface, any nodes joining the fabric inherit this setting. All nodes in a single fabric all run on the same network type. You cannot run a mixed configuration of management and in-band interfaces. Fabrics advertised on an incompatible network are not available for when you issue the fabric-join command. This keeps a switch from joining an incompatible fabric.

If the fabric is configured on the management interface, all fabric-communication is on the management network, except for the following:

Two new states are added to the state field of fabric-node-show:

CLI network-admin@switch > fabric-node-show ?

        [state offline|online|in-band-only-online|mgmt-only-online| fabric-joined|eula-required|setup-required|fabric-required| fresh-install]

Because there are now two networks for Netvisor OS to monitor for connectivity, online means both management and in-band are reachable; in-band-only-online means the switch is only reachable through the in-band network; mgmt-only-online means it is only reachable through the management network; and offline means the switch is not reachable on either network.

Monitoring and reporting are reported on both the management and in-band network connectivity.

Displaying Fabric Information

You can display the fabric using the fabric-show command:

CLI network-admin@switch > fabric-show

name             id               vlan fabric-network control-network tid  

---------------- ---------------- ---- -------------- --------------- ----

info-dev         a000030:5537b46c 3    in-band        in-band         365  

ursa-lyon        6000210:566621ee 0    mgmt           in-band         5055

 

You can display information about the fabric using the fabric-info command:

CLI network-admin@switch > fabric-info format all layout vertical

name:             pn-EBC4

switch-ip:        ::

id:               a0000c5:53ab701e

mcast-ip:         239.4.10.111

fid:              327

cid:              0

 

You can display information about fabric statistics using the fabric-stats-show command:

CLI network-admin@switch > fabric-stats-show

switch   id servers storage VM vlan vxlan tcp-syn tcp-est tcp-completed

-------- -- ------- ------- -- ---- ----- ------- ------- -------------

pubdev02 0  0       0       0  0    0     14.0K   5       40

pubdev03 0  0       0       0  0    0     3.85K   3       24

tcp-bytes udp-bytes arp

--------- --------- ---

125K      0         0

110M      0         0

 

 

Displaying Information about Nodes in the Fabric

You can also display information about the nodes in the fabric. It is important to take note of the fab-tid value. If the fab-tid values do not match for each node, you can use the commands transaction-rollback-to or transaction-rollforward-to to resynchronize the fabric.

id:                  167772619

name:                Leaf2

fab-name:            fab1

fab-id:              a0001c8:53e2601b

cluster-id:          0:0

fab-mcast-ip:        239.4.10.94

local-mac:           64:0e:94:28:06:f2

mgmt-nic:            

mgmt-ip:             192.168.1.14/24

...

in-band-ip:          192.168.254.14/24

...

fab-tid:             9

out-port:            0

version:             2.1.201015836,pn-ONVLnvOS-2.0.2-2000212196

state:               online

firmware_upgrade:    not-required

device_state:        ok

ports:               72

id:                  201326827

name:                Leaf1

fab-name:            fab1

fab-id:              a0001c8:53e2601b

cluster-id:          0:0

fab-mcast-ip:        239.4.10.94

local-mac:           64:0e:94:30:03:97

mgmt-nic:            

mgmt-ip:             192.168.1.11/24

...

in-band-ip:          192.168.254.11/24

...

fab-tid:             9

out-port:            129

version:             2.1.201015836,pn-ONVLnvOS-2.0.2-2000212196

state:               online

firmware_upgrade:    not-required

device_state:        ok

ports:               72

id:                  167772618

name:                Spine2

fab-name:            fab1

fab-id:              a0001c8:53e2601b

cluster-id:          0:0

fab-mcast-ip:        239.4.10.94

local-mac:           64:0e:94:28:06:ee

mgmt-nic:            

mgmt-ip:             192.168.1.13/24

An example of a fabric that is out of sync for two nodes in the fabric:

CLI network-admin@switch > fabric-node-show format all layout vertical

id:                  100663365

name:                CBF-switch

fab-name:            pn-CBF4

fab-id:              a0000c5:53ab701e

cluster-id:          0:0

fab-mcast-ip:        239.4.10.111

local-mac:           64:0e:94:18:01:03

mgmt-nic:            

mgmt-ip:             192.168.1.61/24

...

in-band-ip:          192.168.77.61/24

...

fab-tid:             328

out-port:            128

version:             2.1.201005800,pn-ONVL-2.0.2-2000212196

state:               online

firmware_upgrade:    not-required

device_state:        ok

ports:               68

id:                  201326771

name:                CBF-Leaf-1

fab-name:            corp-CBF4

fab-id:              a0000c5:53ab701e

cluster-id:          0:0

fab-mcast-ip:        239.4.10.111

local-mac:           64:0e:94:30:02:4d

mgmt-nic:            

mgmt-ip:             192.168.1.53/24

...

in-band-ip:          192.168.77.53/24

...

fab-tid:             329

out-port:            128

version:             2.1.201005800,pn-ONVLnvOS-2.0.2-2000212196

state:               online

firmware_upgrade:    not-required

device_state:        ok

ports:               72

id:                  167772357

name:                CBF-Spine1

fab-name:            pn-CBF4

fab-id:              a0000c5:53ab701e

cluster-id:          0:0

fab-mcast-ip:        239.4.10.111

local-mac:           64:0e:94:28:02:de

mgmt-nic:            

mgmt-ip:             192.168.1.51/24

...

in-band-ip:          192.168.77.51/24

f you apply a configuration to the fabric, and a node does not respond to it, you can evict the node from the fabric, and then troubleshoot the problem. To evict a node, use the following command:

CLI network-admin@switch > fabric-node-evict name pleiades25

or

CLI network-admin@switch > fabric-node-evict id b000021:52a1b620

Configuring Ports for Fabric Communication

Netvisor vFlows did match fabric traffic based on the packet protocol, TCP or UDP, plus the Layer 4 destination port. That means any existing TCP or UDP traffic using the same Layer 4 destination port matches the vFlows.

You can now configure a port range for TCP or UDP traffic in order to avoid traffic conflicts with existing TCP or UDP traffic. A new command allows this functionality:

fabric-comm-ports-modify

range-start

Specify a port range from 1024 to 65435.

fabric-comm-port-show

switch:                          techpub-accton-2

range-start:                     23300

fabric-port:                     23399

notify-port:                     23398

proxy-port:                      23397

fabric-keepalive-port:           23394

filesystem-replication-port:     23392

cluster-traffic-forwarding-port: 23391

vport-statistics-port:           23390

l2-encap-port:                   23389

igmp-encap-port:                 23388

icmpv6-encap-port:               23387

arp-encap-port:                  23386

cluster-analytics-port:          23385

 

When you modify the port range, you must do so on each node in the fabric which temporarily interrupts fabric communication until you configure each node with the same port range. There is no forwarded traffic loss if the interruption is brief. Because application of this command prevents communication with other nodes, you must log in to each node directly and separately to apply the command.

 

Configuring Link Aggregation Control Protocol (LACP)

Link Aggregation Control Protocol (LACP) is part of the IEEE specification 802.3ad that allows you to bundle several physical ports to form a single logical channel. When you change the number of active bundled ports on a port channel, traffic patterns reflect the rebalanced state of the port channel.

LACP supports the automatic creation of Gigabit Ethernet port trunks by exchanging LACP packets between ports. It learns the capabilities of port groups and informs the other ports. Once LACP identifies correctly matched Ethernet links, it facilitates grouping the links into Gigabit Ethernet port trunks.

LACP performs the following functions on the switch:

LACP packets are exchanged between ports in these modes:

Active and passive modes allow LACP to negotiate between ports to determine if they can form a port channel based on criteria such as port speed and trunking state.

To enable or disable LACP, or change the system priority, use the following command:

CLI network-admin@switch > lacp-modify enable system-priority 35000

The default system priority value is 32768 with a range from 0 to 65535.

Understanding LACP Priority

LACP port priority is configured on each port using LACP. You can use the default value, 32768, or configure a specific value from 0 to 65535. LACP uses the port priority with the port number to form the port identifier. The port priority determines which ports should be in standby mode when there is a hardware limitation that prevents all compatible ports from aggregating.

LACP system priority can be configured automatically or using the CLI. LACP uses the system priority with the device MAC address to form the system ID and also during negotiations with other systems. The range of LACP system priority is from 0 to 65535. The default value is 32768.

To create a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-create name trunk23 port 20-36 lacp-mode active

To modify a trunk with LACP, use the following command:

CLI network-admin@switch > trunk-modify name trunk23 lacp-mode passive

To modify a port configuration and add LACP priority to the port, use the following command:

CLI network-admin@switch > port-config-modify port 33 lacp-priority 34

Active-Standby Link Aggregation on Management Interfaces

You can configure dual management ports to form LAG with both ports as active. Active-standby, also called active-backup, is supported for LAG on management ports. The management interfaces form a LAG, and only one of the members is active at anytime. The other link is in standby mode, and joins the LAG only if the primary link is down. When the primary link comes up, it joins the LAG and the backup link unjoins.

Use the switch-setup-modify command option for the parameter, mgmt-lag, active-standby.

Configuring Trunking for Link Aggregation (LAG)


 

Informational Note:You must create unique names for each VLAG.

To configure a trunk for aggregating the links connected to ports 1, 2, 3, use the following steps:

1. Create a trunk called trunk-1 on ports 1, 2, 3, enter the following command:

CLI network-admin@switch > trunk-create name trunk-1 port 1,2,3

2. To verify the configuration, use the trunk-show command:

CLI network-admin@switch > trunk-show

name           port       speed   autoneg    jumbo

--------       -----      ------  -------    ------

trunk-1        1-3        10g     off        off

 

3. Modify the trunk configuration by removing port 2:

CLI network-admin@switch > trunk-modify name trunk-1 port 1,3

4. Verify the updated trunk configuration.

CLI network-admin@switch > trunk-show

name           port       speed   autoneg    jumbo

--------       -----      ------  -------    ------

trunk-1        1,3        10g     off        off

 

Notice that the ports have changed from 1-3 to 1,3 indicating that port 2 is no longer a member of the trunk configuration.

5. Delete the trunk configuration from the switch:

CLI network-admin@switch > trunk-delete name trunk-1

Verify that the trunk configuration is removed by using the trunk-show command.

High Availability

Netvisor OS switches automatically perform functions that ease your administrative burden. With high availability, switches in a fabric automatically detect other switches in the fabric. If multiple connections exist between two switches, they automatically create an 801.3ad Link Aggregation Group (LAG) between the two switches for resiliency and load balancing. Other features require configuration such as connecting one device to two switches, or if LAGs are desired between Netvisor OS switches and other manufacturers’ equipment.

Figure 6:

high-availability.png

High Availability

Configuring a Cluster

If you have two Netvisor OS switches, and want them to work together to provide networking services in the event one of the switches fails, the switches must be members of the same fabric, and you must configure them as a cluster.

You configure one node as a local node and the other node as a peer. This reference is symmetric, as either node can be local or peer, depending on the point of reference.

When you create a cluster configuration, you specify the nodes as cluster-node-1 and cluster-node-2. These assignments do not change unless you explicitly change them.

Cluster-node-1 is the primary node and cluster-node-2 is the secondary node. These roles are used to add asymmetry to some protocols. This reference is asymmetric.

A cluster-link contains the port or ports directly connecting the two cluster nodes together. If there are more than one port, this refers to the trunk (LAG) of those ports.

VLAN 4094 is a reserved VLAN used for cluster synchronization traffic. It is added to the in-band interface port and cluster-link automatically when you create the cluster configuration.

Netvisor OS detects cluster-links using an extra data set send in LLDP messages. When a cluster-link is detected, VLAN 4094 is automatically added to it.

Netvisor OS performs cluster synchronization over the control network of the fabric. For the in-band interface, synchronization uses the clust4094 vNIC on VLAN 4094 over the cluster-links. For management, this is performed on the management interface.

Cluster synchronization uses keep-alive messages to detect if the peer cluster node is online. Cluster synchronization messages contain the following information:

The state synchronization is online or offline state of the cluster. Additionally, version numbers are exchanged so messages are adjusted to ensure backward compatibility. Each cluster node sends synch messages to the other node every 2 seconds. If a node misses three synch messages in a row, the cluster goes offline. When a node comes online, it triggers the following behavior:

To set up a cluster of two switches, pleiades4 and pleiades6, you must verify that they are members of the existing fabric:

CLI network-admin@switch > fabric-node-show layout vertical

id:                     184549641

name:                   pleiades4

fab-name:               corp-fab

fab-id:                 b000109:5695af4f

cluster-id:             0:0

local-mac:              3a:7f:b1:43:8a:0f

fabric-network:         in-band

control-network:        in-band

mgmt-ip:                10.9.19.203/16

mgmt-mac:               ec:f4:bb:fe:06:20

mgmt-secondary-macs:    

in-band-ip:             192.168.168.203/24

in-band-mac:            3a:7f:b1:43:8a:0f

in-band-vlan:           0

in-band-secondary-macs:

fab-tid:                1

cluster-tid:            0

out-port:               0

version:                2.4.204009451,#47~14.04.1-Ubuntu

state:                  online

firmware-upgrade:       not-required

device-state:           ok

ports:                  104

 

To create a cluster configuration, use the following command:

CLI network-admin@switch > cluster-create name cluster1 cluster-node-1 onvlnvOS-switch1 cluster-node-2 onvlnvOS-switch2

To verify the status of the cluster, use the cluster-show command:

CLI network-admin@switch > cluster-show

name        state      cluster-node-1     cluster-node-2

---------   -------    ---------------    --------------

cluster1    online     corp-switch1   corp-switch2

 

To replace a failed cluster node, use the cluster-repeer command. However, you must evict the failed node from the fabric, and then run the cluster-repeer command on an active node after replacing the failed node.

To display information about the cluster, use the cluster-info command:

CLI network-admin@switch > cluster-info format all layout vertical

name:           cluster-leaf

state:          online

cluster-node-1: draco01

cluster-node-2: draco02

tid:            2

mode:           master

ports:          69-71,129

remote-ports:   69-71,128

If you want to connect the cluster nodes to an uplink switch, you must configure a VLAG between the ports on the cluster nodes and the uplink switch. For example, if corp-switch1 has port 53 connected to the uplink switch and corp-switch2 has port 19 connected to the uplink switch, create a VLAG by executing the VLAG-create command on either of the switches:

CLI network-admin@switch > VLAG-create name VLAG-uplink local-port 53 peer-switch switch1 peer-port 19

This example assumes that you’ve entered the command on switch2.

To verify the configuration, use the following command:

CLI network-admin@switch > vlag-show

name     cluster     mode          switch     port        peer-switch

-------- ----------- ------------- ---------- ----        ---------

switch-1       cluster-2   active-active spine-1     34         spine-2

 

peer-port      status    local-state lacp-mode

-------------- --------- ----------- ---------

129            normal    enabled     active

 

 

Informational Note:

Before you can create a VLAG, you must configure the two switches in a cluster.

Modifying LACP Mode on an Existing VLAG Configuration

You can modify the LACP mode on an existing VLAG configuration:

CLI network-admin@switch > vlag-modify

name name-string

Specify the VLAG name. 

failover-move-L2|
failover-ignore-L2

If you specify the parameter, failover-move-L2, Netvisor sends gratuitous ARPs.

lacp-mode off|passive|active

Specify the LACP mode as off, passive or active.

lacp-timeout slow|fast

Specify the LACP timeout as slow (30 seconds) or fast (4 seconds).

lacp-fallback bundle|individual

Specify the LACP fallback mode as individual or bundled.

lacp-fallback-timeout seconds

Specify the LACP fallback timeout in seconds. The default is 50

seconds

 

Modifying a Trunk or VLAG Configuration by Changing the LACP Mode

You can now modify a trunk or VLAG configuration by changing the LACP fallback mode and timeout.

This feature enables ports of a static LACP trunk to operate as individual ports in the absence of proper LACP negotiation with a network peer. Once any one port member hears a LACP PDU from the peer, then all port members of the trunk are bundled to operate as a trunk. This feature is useful for servers with multiple network interfaces using PXE boot.

With this configuration, Netvisor OS creates the trunk in the switch, but does not add any of the ports to the trunk. The ports continue to operate individually until LACP PDUs are heard on any of the ports part of the trunk configuration. Once LACP PDUs are sent from the peer for trunk, then all ports of the trunk cease to operate individually and are added to the trunk.

If no LACP PDUs are received for the number of seconds configured as the fallback timeout, Netvisor OS LACP checks if LACP negotiation has expired. If LACP negotiation has expired, then ports fall back to individual mode. If LACP negotiation has not expired, another fallback timer is scheduled at a value equal to the fallback timeout.

For example:

To change the LACP failback mode and timeout, use the new parameters:

CLI network-admin@switch > vlag-modify

lacp-mode off|passive|active

Modify the LACP mode.

lacp-timeout slow|fast

Modify the LACP timeout.

lacp-fallback bundle|individua

Modify the LACP fallback.

lacp-fallback-timeout seconds

Modify the LACP fallback timer between 30 and 60 seconds. The default value is 50 seconds.

lacp-port-priority integer 

Configure the LACP port priority as an integer between 1 and 65535.

Safely Restoring Ports for Cluster Configurations


 

Informational Note:  This feature is only applied during the initial start up of the cluster slave switch.

Safely Restoring Ports for Cluster Configurations (EA) [RFE14341-EA2.6.0]


 

Informational Note:  Netvisor only applies this feature during the initial start up of the cluster slave switch.

Netvisor requires sub-second traffic loss for fail over events for a cluster configuration. Netvisor provides two types of ports for redundant data paths: 1) Layer 3 ports over ECMP redundant routed paths, and 2) virtual LAGS (VLAGs) providing redundant Layer 2 paths. During failover and recovery port events, it can take measurable time to change the hardware routing and MAC tables on larger networks. This delay incurs traffic loss on the network. To reduce delay, this feature allows you to incrementally restore these ports at start up. By incrementally restoring the ports, Netvisor prevents the changes to the hardware from contending with each other and reduces the delay between a port up and the hardware updates with the appropriate Layer 3 and Layer 2 information for the port. This process ensures sub-second fail over.

Netvisor restores all non-Layer 3 and non-VLAG ports first. This allows the cluster links to activate and the cluster configuration to synchronize information. Layer 3 and VLAG port restoration starts after the cluster synchronizes. This is predicated on the cluster becoming active, all Layer 2 and Layer 3 entries, such as status updates, exchanged, cluster STP status synchronized, and all router interfaces initialized.

The parameter, maximum-sync-delay, controls the maximum time to wait for synchronization in the case where the cluster cannot synchronize information. After the synchronization completes, Netvisor restores Layer 3 ports first, since Layer 3 traffic can traverse the cluster link to the peer VLAG port if needed. Currently the reverse is typically not true.

If Netvisor restores VLAG ports first, a Layer 3 adjacency between the two cluster nodes may be needed but may not exist in some network configurations. After Netvisor restores Layer 3 ports, Netvisor waits a configurable Layer 3 port to VLAG delay to allow time for the routing protocols to converge and insert the routes. The delay time defaults to 15 seconds.

After the delay, Netvisor incrementally restores the VLAG ports. Incrementally restoring ports allows enough time to move Layer 2 entries from the cluster link to the port. Incrementally restoring ports also allows the traffic loss to occur in small, 200-300ms per port, rather than one large time span. This is particularly important for server clusters where temporary small losses are no issue, but fail or timeout for a large continuous traffic loss. If the cluster master node comes up, then Netvisor does not apply staggering and no Layer 3 to VLAG wait. And if the node is the cluster master node, that means the peer is down or coming up, and not handling traffic. Therefore Netvisor safely restores the ports as soon as possible to start traffic flowing between the nodes.

To configure a cluster for restoring Layer 3 ports, use the following commands:

cluster-bringup-modify

Modifies the cluster bring up configuration.

Specify one or more of the following options

l3-port-bringup-mode staggered|simultaneous

Specify the Layer 3 port bring up mode during start up.

l3-port-staggered-interval duration: #d#h#m#s

Specify the interval between Layer 3 ports in Layer 3 staggered mode. This can be in days, hours, minutes, or seconds.

vlag-port-bringup-mode staggered|simultaneous

Specify the VLAG port bring up mode during start up.

vlag-port-staggered-interval duration: #d#h#m#s   

 Specify the interval between VLAG ports in VLAG staggered mode.

This can be in days, hours, minutes, or seconds.

maximum-sync-delay duration: #d#h#m#s

Specify the maximum delay to wait for cluster to synchronize before starting Layer 3 or VLAG port bring up.

This can be in days, hours, minutes, or seconds.

l3-to-vlag-delay duration: #d#h#m#s

Specify the delay between the last Layer 3 port and the first VLAG port bring up.

This can be in days, hours, minutes, or seconds. The default value is 15 seconds.

 

To display the cluster port restoration configuration, use the cluster-bringup-show command:

cluster-bringup-show

Displays the cluster bring up configuration information.

Configuring Layer 2 Multipathing for Virtual Chassis Link Aggregation (VLAG)

You can aggregate links between two switches by configuring Layer 2 multipathing and virtual chassis Link Aggregation.

A virtual chassis Link Aggregation Group (VLAG) allows links that are physically connected to two different switches to appear as a single Ethernet trunk to a third device. The third device can be a server, switch, or any other networking device. A VLAG can create Layer 2 multipathing which allows you to create redundancy, enabling multiple parallel paths between nodes.

A VLAG requires that a least one cross connection between the two switches, also called peers, where the VLAG links terminate. The specific ports that connect the different switches, do not require explicit configuration before creating a VLAG.

VLAGs can provide the following benefits:

Netvisor OS performs VLAG synchronization to coordinate active-standby and active-active configurations using the following rules:

Netvisor OS reports the state as up or down and synchronizes the state. For active-standby VLAGs, port up timestamps are exchanged to resolve any contest if both ports are up.

Netvisor OS performs synchronization from the primary node to the secondary node. If the secondary node requires synchronization, the secondary node sends a request to the primary node to perform the synchronization.

Synchronization messages are sent on a per-VLAG basis, and compare the local VLAG port state with the peer VLAG port state. The port state then determines any port enable or disable actions for active-standby VLAGs or port egress rule changes for active-active VLAGs.

VLAG synchronization occurs when a trigger happens on the configuration:

For any port in an active-standby VLAG, Netvisor records the time up of the port, and sends it as part of the VLAG synchronization message. the time up values are compared on both nodes to determine the active port.

VLAG Topology Examples

Figure 7:L2 Design - Leaf and Spine with Active-Passive VLAG

active-passive-VLAG.png

Figure 8:L2 Design - Leaf and Spine with Active-Active VLAG

active-active-VLAG.png

Figure 9:L2 Design - Leaf and Third Party Spine without Multichassis LAG or VPC Mode

active-passive-VLAG.png

 

Figure 10:L2 Design - Leaf and Third Party Spine with Multichassis LAG, vPC and MLAG

 

To create a VLAG for aggregating links connected to ports 70 on the local switch and the peer called, eng-switch-b, you must first create a cluster configuration between the two switches. Netvisor OS switches must be members of a cluster configuration before you can add VLAGs to them.

Configuring Active-Active VLAG

Using the sample topology in Figure 11, “” on page 94, use the following steps to configure Active-Active VLAG:


 

Informational Note:  There must be a physical connection between PN-0 and PN-1 before you can configure VLAG.

Figure 11:Active-Active VLAG over a Trunk with a Server-Switch and Host

fabric00003.jpg

 

Three Netvisor OS switches in a common fabric with the Spine switch as the RSTP root. It’s important to note that ports 19-22 on PN-0 and PN-1 are ports connected to PN-2 (Spine). Port 26 connects PN-0 to PN-1 for the cluster configuration required for VLAG.

1. On PN-2, use the following command:

CLI network-admin@switch > stp-modify bridge-priority 4096

2. Create the fabric and add the switches:

On PN-2, use the fabric-create command:

CLI network-admin@switch > fabric-create name fab-VLAG

On PN-1, join the fabric:

CLI network-admin@switch > fabric-join name fab-VLAG

On PN-0, join the fabric:

CLI network-admin@switch > fabric-join name fab-VLAG

3. Create VLAN connectivity from the top switch to the bottom:

On PN-2, create the VLAN with scope fabric:

CLI network-admin@switch > vlan-create id 25 scope fabric

On PN-0, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-1, add the VLAN and untag the port connected to the host.

CLI network-admin@switch > vlan-port-add vlan-id 25 untagged ports 9

On PN-0, modify the host STP port to be an edge port.

CLI network-admin@switch > stp-port-modify port 9 edge

On PN-1, modify the host STP port to be an edge port.

CLI network-admin@switch > stp-port-modify port 9 edge

4. Create a cluster configuration between PN-1 and PN-0. This creates the cluster across port 26.

On PN-0, enter the cluster-create command:

CLI network-admin@switch > cluster-create name VLAG cluster-node-1 PN-0 cluster-node-2 PN-1

5. You must disable ports between PN-2 and PN-0, and then create a static trunk between them:

On PN-0, modify the ports facing PN-2:

CLI network-admin@switch > port-config-modify port 19,20 disable

6. Then create the trunk on PN-0:

CLI network-admin@switch > trunk-create name pn0-to-pn2 port 19,20 lacp-mode off

CLI network-admin@switch > trunk-show format all layout vertical

switch:                tac-1

trunk-id:              253

name:                  pn0-to-pn2

ports:                 none

speed:                 disable

egress-rate-limit:     unlimited

autoneg:               off

jumbo:                 on

enable:                off

lacp-mode:             off

lacp-priority:         0

lacp-timeout:          slow

lacp-fallback:         bundle

lacp-fallback-timeout: 50

lacp-individual:       none

stp-port-cost:         2000

stp-port-priority:     128

reflect:               off

edge-switch:           no

pause:                 no

description:           

loopback:              off

receive-only:          on

unknown-ucast-level:   %

unknown-mcast-level:   %

broadcast-level:       %

lport:                 0

rem-rswitch-port-mac:  00:00:00:00:00:00

rswitch-default-vlan:  0

status:                

config:                

trunk-hw-id:           0

send-port:             4294967295

routing:               yes

host-enable:           no

 

From the above output, you can find the name of the trunk configuration, pn0-to-pn2. You need this information to create the VLAG.

Then, on PN-1, repeat the same commands to create a trunk between PN-1 and PN-2.

7. You must disable ports between PN-2 and PN-1, and then create a static trunk between them:

On PN-1, modify the ports facing PN-2:

CLI network-admin@switch > port-config-modify port 21,22 disable

CLI network-admin@switch > trunk-create name pn1-to-pn2 port 21,22 lacp-mode off

CLI network-admin@switch > trunk-show format all layout vertical

switch:               PN-0

intf:                 129

name:                 pn1-to-pn2

port:                 21-22

speed:                10g

autoneg:              off

jumbo:                off

enable:               off

lacp-mode:            off

lacp-priority:        32768

lacp-timeout:         slow

reflect:              off

edge-switch:          no

pause:                no

description:          

loopback:             off

mirror-only:          off

lport:                0

rswitch-default-vlan: 0

port-mac-address:     06:60:00:02:10:80

status:               

config:               

send-port:            0

8. Now create the VLAG from the bottom switches going upward and static trunk from the top down. Keep one side of the VLAG disabled while you configure this step.

On PN-0, use the VLAG-create command:

CLI network-admin@switch > VLAG-create name to-spine port 128 peer-port 129 peer-switch PN-1 lacp-mode off mode active-active

On PN-2, create a trunk with the name trunk-pn:

CLI network-admin@switch > trunk-create name trunk-pn port 19,20,21,22 lacp-mode off

9. Now, you can enable ports on all switches:

On PN-2, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 19,20,21,22 enable

On PN-0, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 19,20 enable

On PN-1, enter the port-config-modify command:

CLI network-admin@switch > port-config-modify port 21,22 enable

10. Create the server-facing VLAG:

On PN-0, enter the VLAG-create command:

CLI network-admin@switch > vlag-create name to-spine port 9 peer-port 9 peer-switch PN-1 lacp-mode active mode active-active

Display the VLAG configuration information:

CLI network-admin@switch > vlag-show format all layout vertical

id:               a000024:0

name:             to-spine

cluster:          VLAG

mode:             active-active

switch:           pubdev02

port:             trunk2

peer-switch:      pubdev01

peer-port:        129

failover-move-L2: no

status:           normal

local-state:      enabled,up

lacp-mode:        off

lacp-timeout:     slow

lacp-key:         26460

lacp-system-id:   110013777969246

 

Routing over VLAGs

When you configure a routing protocol such as OSPF for peering over a VLAG, the cluster egress filtering rules prevent routed packets from egressing VLAGs, if the packet crosses the cluster links before egressing the VLAG.

There are some suboptimal network designs, for example when vrouter interfaces are created on only one side of the cluster due to the use of /30 network addresses, and the cluster active-active routing feature does not help with the implementation. These suboptimal network designs may cause Layer 3 traffic to route through the cluster even when the point of exit is a VLAG that is locally up on each cluster member. To help in those cases, a new parameter is provided to allow packets crossing the cluster to egress out of VLAGs, if the packet is routed after crossing the cluster. To enable this feature, use the following command:

system-settings-modify

 

routing-over-vlags|
no-routing-over-vlags

Specify if you want to enable or disable routing to VLAGs from cluster links

CLI (network-admin@vs-spine1) > system-settings-show

switch:             spine1

routing-over-vlags: off

switch:             vanquish2

switch:             vanquish4

routing-over-vlags: off

switch:             vanquish3

routing-over-vlags: off

 

CLI (network-admin@spine11) > system-settings-modify routing-over-vlags

 

CLI (network-admin@spine1) > switch-local system-settings-show

switch: spine1

routing-over-vlags: on

 

Support for Routing over VLAG

When a routing protocol such as OSPF is used for peering over a VLAG, the cluster egress filtering rules prevent routed packets from egressing VLAGs, if the packet crosses the cluster links before egressing the VLAG.

There are some suboptimal network designs, for example when vrouter-interfaces are created on only one side of the cluster due to the use of /30 network addresses, and the cluster active-active routing feature does not help with the implementation. These suboptimal network designs may cause Layer 3 traffic to route through the cluster even when the point of exit is a VLAG that is locally up on each cluster member. To help in those cases, a new parameter is provided to allow packets crossing the cluster to egress out of VLAGs, if the packet is routed after crossing the cluster. To enable this feature, use the following command:

system-settings-modify

Modify system settings

routing-over-vlags|no-routing-over-vlags

Specify if you want to enable or disable routing to VLAGs from cluster links

 

CLI (network-admin@vs-spine1) > system-settings-show

switch:             spine1

routing-over-vlags: off

switch:             vanquish2

switch:             vanquish4

routing-over-vlags: off

switch:             vanquish3

routing-over-vlags: off

 

CLI (network-admin@spine1) > system-settings-modify routing-over-vlags

 

CLI (network-admin@spine1) > switch-local system-settings-show

switch: spine1

routing-over-vlags: on

Topic Feedback

Was this topic useful to you? Please provide feedback to improve the content.