Configuring the VXLAN Loopback Trunk
In order to perform routing on the overlay and to replicate Layer 2 broadcast, unknown unicast and multicast (BUM) traffic over multiple VXLAN tunnels on Pluribus switches, you must add ports to a vxlan-loopback-trunk using the following command:
CLI (network-admin@switch) > trunk-modify name vxlan-loopback-trunk ports <list of ports> jumbo
Where <list of ports> may vary based on the hardware architecture of the switch (on most models it is likely to be a list of front panel ports set aside for this use.)
Check the data sheet of your switch model to verify the number of internal ports as well as front-panel ports available and their respective speeds. Depending on the expected peak amount of routed and BUM traffic, you can use either N x 10 GbE ports or two (or more) 40 GbE ports (or even 100 GbE).
The above command also prints out the following informational message:
!!!! Ports updated on vxlan-loopback-trunk. Ports added only support vxlan recirculation. Any features configured on these ports may not work. Please update the necessary config. Any ports removed need necessary adjustment on autoneg,jumbo,etc !!!
If you create a VLAN with associated VXLAN ID when vxlan-loopback-trunk is without any active ports, you will get this warning message:
CLI (network-admin@switch) > vlan-create id 100 scope local vxlan 1001000
!!!! Vlan 100 created, but there are no ports configured in vxlan-loopback-trunk. vxlan forwarding may not function correctly. !!!!
Single-pass VXLAN Forwarding
Starting from Netvisor ONE release 6.0.0, on NRU03 and NRU-S0301 platforms and on the Dell S5200 Series, more advanced hardware capabilities enable the switches to perform routing in and out of tunnels (RIOT) in a single pass (that is, without requiring the loopback trunk).
Note: On the Dell S4100 series single pass RIOT is supported for distributed VXLAN routing with Unicast Fabric VRFs, however single pass RIOT is not supported for centralized VXLAN routing on a DC Gateway border leaf.
On the supported platforms, this capability can be enabled with the command:
CLI (network-admin@switch) > system settings-modify no-single-pass-riot|single-pass-riot
The default setting is no-single-pass-riot. A reboot is required for the configuration change to take effect and an appropriate message is displayed to remind the user of this requirement.
Furthermore on all hardware platforms, starting from Netvisor ONE release 6.0.0, a new global command is introduced to enable single-pass replication (i.e., flooding) of Layer 2 broadcast, unknown unicast and multicast (BUM) traffic over multiple VXLAN tunnels: system-settings-modify no-single-pass-flood|single-pass-flood.
By default, single-pass-flood mode is disabled. A reboot is required for the configuration change to take effect and an appropriate message is displayed to remind the user of this requirement.
For example:
CLI (network-admin@switch2) > system-settings-modify single-pass-flood
!!!! Please reboot the system for the new single-pass-flood setting to take effect correctly !!!!
CLI (network-admin@switch2) > system-settings-show format single-pass-flood
single-pass-flood: on
CLI (network-admin@switch2) > switch-reboot
After the reboot, the tunnel-show flood-nexthop command can be used to display the path that is selected as next hop for flooded traffic out of the available ECMP paths, when single-pass-flood is enabled:
CLI (network-admin@switch2) > tunnel-show flood-nexthop
switch name flood-nh-ip flood-nh-vlan flood-nh-port flood-nh-mac flood-nh-egress-id
------- ----- ----------- ------------- ------------- ---------------- ------------------
switch1
switch2 tnl2 12.12.12.7 4088 70 66:0e:94:1a:e3:d3 100008
switch2 tnl1 10.10.10.7 4092 69 66:0e:94:1a:e3:d3 100007
In the above output switch1 has no info as that node is not in single-pass-flood mode.
Note that, due to a hardware limitation in single-pass mode, ECMP load balancing of flooded traffic is only performed on a per-tunnel basis. In other words, flooded traffic for a specific tunnel will always take the same active ECMP path, which is selected at random out of the available ECMP next hops.
The port-stats-show command can be used to display the traffic statistics on the loopback ports and on tunnel ports:
CLI (network-admin@switch2) > port-stats-show format time,port,ibits,iUpkts,iMpkts,iCongDrops,ierrs,obits,oUpkts,oBpkts,oMpkts,oCongDrops
time port ibits iUpkts iBpkts iMpkts iCongDrops ierrs obits oUpkts oBpkts oMpkts oCongDrops
-------- ---- ----- ------ ------ ------ ---------- ----- ----- ------ ------ ------ ----------
18:28:56 1 0 0 0 0 0 0 0 0 0 0 0
18:28:56 49 110K 0 85 58 0 0 174K 0 104 127 0 0 0
18:28:56 69 41.8K 0 0 10 0 0 119K 85 0 11 0 0 0
In this example, port 1 is the VXLAN loopback port (with a zero packet count, as the switch is in single-pass-flood mode), while port 49 is the input port where the broadcast packets are received and port 69 is the tunnel output port where the encapsulated flooded packets are sent to. The same packet count (along with the corresponding byte count) is displayed in the HER-pkts and HER-bytes fields of tunnel-stats-show:
CLI (network-admin@switch2) > tunnel-stats-show format switch,HER-pkts,HER-bytes
switch HER-pkts HER-bytes
------------- -------- ---------
switch2 85 98K
Starting from Netvisor ONE release 6.0.1, analogously to the BUM traffic replication case described above, support for single-pass Layer 2 forwarding of known multicast traffic is introduced.
IGMP snooping is used to determine the output interfaces corresponding to the multicast receivers as described in the section titled Configuring IGMP Snooping with VXLAN (refer to that section for configuration details). A new parameter is introduced to perform single-pass L2 forwarding for this multicast traffic with known receivers:
CLI (network-admin@switch) > system-settings-modify no-single-pass-l2-known-multicast | single-pass-l2-known-multicast
The no-single-pass-l2-known-multicast is the default setting. A reboot is required for a change in setting to take effect, as reminded by the message shown below:
CLI (network-admin@switch) > system-settings-modify single-pass-l2-known-multicast
!!!! Please reboot the system for the new single-pass-l2-known-multicast setting to take effect correctly !!!!
CLI (network-admin@switch) > system-settings-show format single-pass-l2-known-multicast
single-pass-l2-known-multicast: on
Note: that as in the BUM forwarding case, due to a hardware limitation in single-pass mode, ECMP load balancing of multicast traffic is only performed on a per-tunnel basis. In other words, forwarded multicast traffic for a specific tunnel will always take the same active ECMP path, which is selected at random out of the available ECMP next hops.
Note: On capable devices, single-pass features can be used to avoid consuming front panel ports for the recirculation function.