Configuring Uplink Groups
Certain features require that a switch monitors the total available aggregate uplink bandwidth compared to the amount of downlink traffic being generated by the attached devices. The requirement is to prevent oversubscription of the uplinks by traffic sourced from the downlinks (host-facing ports).
For that purpose, starting from Netvisor ONE release 7.0.0, the software supports the configuration of uplink groups, which can be used with features like vLE and bidirectional vPG for uplink oversubscription avoidance. (Refer to the Configuring Virtual Link Extension and Configuring Pluribus Network Packet Broker sections of the Configuration Guide for more details on those features.)
Uplink oversubscription is defined as: total downlink capacity > total uplink capacity (for a given switch).
Note: The operational status of switch ports is not considered to calculate the uplink and downlink aggregated bandwidth capacities. Only the configuration/administrative state is considered.
Note: Uplink groups support only bidirectional vPG-driven connections and not unidirectional vPG-driven connections.
The configuration of an uplink group allows users to track the available aggregate bandwidth of the physical ports added to a group. (Trunks cannot be added.) Then, a system configuration allows the users to select which action to apply in case of aggregate bandwidth mismatch between the uplink ports and the downlink ports:
- No warning or error is generated (default behavior)
- Allow the creation of a connection across the fabric, but:
- Display an appropriate message to warn about oversubscription
- Log the event in system.log
- Refuse the creation of a connection across the fabric if its creation leads to oversubscription.
Note: The software currently does not support separate uplink port groups per feature connection to monitor oversubscription more granularly. Hence this could lead to inaccuracies in reporting (i.e., to false positives).
The software tracks port additions/modifications/deletions as well as port speed changes to calculate the aggregate bandwidth. Whenever a feature-driven connection across the fabric is created/deleted/disabled, the downlink port capacity is also modified (when applicable) and checked.
Note: Uplink/downlink port speed changes when enabling/disabling auto-negotiation are not reflected in the respective group’s aggregated bandwidth. So, after auto-negotiation, if a port speed changes and leads to oversubscription, the event is not reported.
Release 7.0.0 introduces a new set of commands to configure an uplink group, as shown below:
CLI (network-admin@switch) > uplink-group-port-show
switch: switch
ports: none
CLI (network-admin@switch) > uplink-group-port-add ports 12-14
CLI (network-admin@switch) > uplink-group-port-show
switch: switch
ports: 12-14
Note: Only physical ports can be added to an uplink group. Configuring trunks will produce an error.
You can remove ports from an uplink group like so:
CLI (network-admin@switch) > uplink-group-port-show
switch: switch
ports: 12-14
CLI (network-admin@switch) > uplink-group-port-remove ports 12
CLI (network-admin@switch) > uplink-group-port-show
switch: switch
ports: 13-14
You can configure the action to take when uplink oversubscription is detected with the following command:
CLI (network-admin@switch) > system-settings-modify uplink-oversubscription-action default | warn | error
default means no warning or error is generated.
warn allows the creation of a connection across the fabric, but a warning message is produced if the creation causes uplink oversubscription.
If the switch is already in oversubscribed state, the system generates a specific warning message right when the option is configured:
CLI (network-admin@switch) > system-settings-modify uplink-oversubscription-action warn
Warning: The switch is already in uplink oversubscription state. Configure ports to uplink group
You can configure the error option so that the software refuses the creation of a connection across the fabric when causing uplink oversubscription:
CLI (network-admin@switch) > system-settings-modify uplink-oversubscription-action error
CLI (network-admin@switch) > system-settings-show format uplink-oversubscription-action
uplink-oversubscription: error
If the switch is already in oversubscribed state, the command will fail right away with an error when the option is configured:
CLI (network-admin@switch) > system-settings-modify uplink-oversubscription-action error
system-setting-modify: failed to change to error action as the switch is already in uplink oversubscription state. Add ports to uplink group or remove existing VLE/VPG(bidir) connections.
If you remove ports from an uplink group, oversubscription may occur due to the capacity change. The warn and error options trigger, respectively, the following messages (and action):
CLI (network-admin@switch) > uplink-group-port-remove port 25
Warning: removing ports from uplink group is causing oversubscription of uplink ports.
CLI (network-admin@switch) > uplink-group-port-remove port 25
uplink-group-port-remove: removing these ports from uplink group failed as it is leading to oversubscription of uplink ports.
If you change uplink/downlink port speeds, that may lead to oversubscription. The warn and error options trigger, respectively, the following messages (and action):
CLI (network-admin@switch) > port-config-modify port 45 speed 40g
Warning: Change of speed for the port(s) 45 has caused uplink oversubscription.
CLI (network-admin@switch) > port-config-modify port 45 speed 40g
port-config-modify: change port speed failed for given port(s) 45 as leading to oversubscription of uplink ports.
In case of the warn option, this is the warning message generated for a vLE configuration with uplink oversubscription:
CLI (network-admin@switch) > vle-create node-1 switch node-1-port 12 node-2 switch2 node-2-port 13 name vle1
Warning: The creation/enabling of vle vle1 is causing uplink oversubscription by 1000 mbps.
In this scenario, the calculated difference between the downlink and uplink capacity is 1000 Mbps.
In case of the error option, when oversubscription occurs locally to a vLE switch, this message is displayed:
CLI (network-admin@switch) > vle-create node-1 switch node-1-port 12 node-2 switch2 node-2-port 13 name vle1
vle-create: creation/enabling of VLE vle1 failed because it is causing uplink oversubscription by 1000 mbps. Remove associated transparent VLAN and VXLAN mapping for the VLE.
Similarly, when oversubscription happens on a remote switch due to the vle-create command, this message is generated:
CLI (network-admin@switch) > vle-create node-1 switch node-1-port 12 node-2 switch2 node-2-port 13 name vle1
vle-create: error from switch2: creation/enabling of VLE vle1 failed because it is causing uplink oversubscription by 1000 mbps. Remove associated transparent VLAN and VXLAN mapping for the VLE.
In case of the warn option, this is the warning message generated for a vPG configuration with uplink oversubscription:
CLI (network-admin@switch) > vflow-create name v1 scope fabric bidir-vpg-1 vpg1 bidir-vpg-2 vpg2
Warning: The creation/enabling of (bidir vpg)vflow v1 is causing uplink oversubscription by 1000 mbps.
In case of the error option, when oversubscription occurs locally to a vPG switch, this message is displayed:
CLI (network-admin@switch) > vflow-create name v1 scope fabric bidir-vpg-1 vpg1 bidir-vpg-2 vpg3
vflow-create: The creation/enabling of (bidir vpg)vflow v1 is causing uplink oversubscription by 1000 mbps.
Similarly, when oversubscription happens on a remote switch due to the vflow-create command, this message is generated:
CLI (network-admin@switch) > vflow-create name v1 scope fabric bidir-vpg-1 vpg1 bidir-vpg-2 vpg3
vflow-create: error from switch2: The creation/enabling of (bidir vpg)vflow v1 is causing uplink oversubscription by 1000 mbps.
With the warn option, when a vLE is created with endpoints that are local to the switch and have a speed mismatch, when oversubscription occurs this warning message is generated:
CLI (network-admin@switch) > port-show port 9,49 format port,config,
port config
---- ------
9 fd,10g
49 fd,40g
CLI (network-admin@switch) > vle-create name v1 node-1 switch node-1-port 9
node-2 switch node-2-port 49
Warning: creation/enabling of VLE is causing endpt 9 being oversubscribed.
In the same scenario, but with the error option, this action is generated in case of oversubscription:
CLI (network-admin@switch) > vle-create name v1 node-1 switch node-1-port 9 node-2 switch node-2-port 49
vle-create: creation/enabling of local VLE failed because endpoint 9 is oversubscribed.
Remove associated transparent VLAN and VXLAN mapping for the VLE.
With the warn option, when vPGs are created with endpoints that are local to the switch (port 17,45 in the example below) and have a speed mismatch, this warning message is generated:
CLI (network-admin@switch) > vpg-show
scope name type ports vni vlan
------ ---- ------------- ----- -------- ----
fabric vpg3 bidirectional 17 12500000 2750
fabric vpg4 bidirectional 22 12500001 2751
fabric vpg5 bidirectional 45 12500002 2752
CLI (network-admin@switch) > port-show port 17,45 format port,config,
port config
---- ------
17 fd,10g
45 fd,40g
CLI (network-admin@switch) > vflow-create name v3 scope local bidir-vpg-1 vpg3
bidir-vpg-2 vpg5
Warning: creation/enabling of local vflow is causing endpoint 17 to be
oversubscribed.
In the same scenario, but with the error option, this action is generated:
CLI (network-admin@switch) > vflow-create name v3 scope local bidir-vpg-1 vpg3
bidir-vpg-2 vpg5
vflow-create: creation/enabling of local vflow failed because endpoint 17 is
oversubscribed.
With the warn option, when adding ports to a vPG with an active vFlow connection, the following warning message is generated when oversubscription occurs:
CLI (network-admin@switch) > vflow-show format name,bidir-vpg-1,bidir-vpg-2,inport,
name bidir-vpg-1 bidir-vpg-2 in-port
---- ----------- ----------- -------
v6 vpg3 vpg2 12
If port 12 (10g) is removed and trunk port 273 (20g) is added to vpg3 of vFlow v6:
CLI (network-admin@switch) > vpg-port-remove vpg-name vpg3 ports 12
CLI (network-admin@switch) > vpg-port-add vpg-name vpg3 ports 273
warning: creation/enabling of Vflow v6 is causing uplink oversubscription by
10000 mbps
In the same scenario, but with the error option, this action is generated in case of oversubscription:
CLI (network-admin@switch) > vpg-port-remove vpg-name vpg3 ports 12
CLI (network-admin@switch) > vpg-port-add vpg-name vpg3 ports 273
vpg-port-add: creation/enabling of Vflow v6 failed because it is causing uplink
oversubscription by 1000 mbps. Consider adding ports to uplink group.
With the warn option, when trunk ports of a bidirectional vPG belonging to active vFlow connections are modified, the following warning message is generated when oversubscription occurs:
CLI (network-admin@switch) > vflow-show format name,bidir-vpg-1,bidir-vpg-2,inport,
name bidir-vpg-1 bidir-vpg-2 in-port
---- ----------- ----------- -------
v6 vpg3 vpg2 27
CLI (network-admin@switch) > trunk-show name t1 format name,trunk-id,ports
name trunk-id ports
---- -------- -----
t1 273 21
If the uplink bandwidth is 10 Gbps and the trunk t1 is expanded to double its capacity, say, with ports 21-22 (20g), then oversubscription would occur:
CLI (network-admin@switch) > trunk-modify ports 21-22 name t1
Warning: modifying trunk parameters in use by bidir vpg is leading to uplink oversubscription.
In the same scenario, but with the error option, this action is generated instead:
CLI (network-admin@switch) > trunk-modify ports 21-22 name t1
trunk-modify: modifying trunk parameters in use by bidir vpg failed because it is leading to uplink oversubscription.
You can show the uplink oversubscription state with the following command:
CLI (network-admin@switch) > uplink-group-port-oversubscription-state-show
uplink-bw: 20000
downlink-bw: 50000
uplink-oversubscription: yes