Configuring VXLAN-based Bridge Domains for 802.1Q and QinQ Internetworking
Bridge domains’ configuration process is based upon the deployment of a VXLAN transport as described in the Configuring VXLAN chapter of this guide.
Subsequently, switch ports can be selectively added to a BD to accept either 802.1Q frames or double-tagged 802.1ad frames based on the connected device(s) and/or network(s), as depicted in the Figure 9-6.
Figure 9-6: VXLAN-based BD Internetworking
Switch ports can be connected to either servers/hosts or switches/provider gateways as shown above.
In case of interconnection with an external cloud provider’s gateway devices, as shown on the right hand side of Figure 9-6, the connectivity is based on double-tagged 802.1ad frames, as it is required to preserve all 16M (4094 x 4094) possible combinations of user IDs and service IDs.
On the other hand, connectivity to servers can be based on a single 802.1Q VLAN tag. This case is useful in private cloud networks for direct server-to-server connectivity (for example a Web services node that needs to communicate with a database node).
Alternatively, server NICs can be configured with both an outer VLAN and an inner VLAN ID. In such case, the inner VLAN represents for example the service implemented on such server. Certain network designs employ this double-tagging scheme to be able to scale the number of supported services to 16M. In this case, switch ports have to be configured in 802.1ad mode to preserve the inner tag.
The 802.1Q ports can also be used to connect to customer switches and traffic can be double tagged by the fabric leaf nodes themselves.
Informational note: BD ports can be configured in 802.1Q mode or in 802.1ad/QinQ mode but not in both on the same switch for the same BD.
Before adding ports to any of the aforementioned scenarios, the first basic configuration step is the creation of a VXLAN-based bridge domain entity with the command:
CLI (network-admin@leaf-5) > bridge-domain-create name bd1000 scope fabric vxlan 10000 rsvd-vlan 4002
A bridge domain’s scope can be fabric, cluster, or local.
The bridge-domain-create command uses a manually assigned VNI to uniquely identify the bridge domain at the hardware level.
Moreover, with fabric and cluster scope, it must set aside a VLAN ID as reserved for supporting switch clusters:
CLI (network-admin@leaf-5) > bridge-domain-create name bd1000 scope fabric vxlan 10000 rsvd-vlan 4002
This command reserves VLAN 4002 on all cluster switches in the fabric for BD’s internal use, so it can no longer be configured as a regular VLAN number. (The same requirement applies to bridge-domain-create scope cluster commands).
If for a particular pair of cluster switches the reserved VLAN has to be modified, on any one of the two switches you can use the following command:
CLI (network-admin@leaf-5) > bridge-domain-modify name bd1000 local-rsvd-vlan 4006
Note that the above command requires both cluster switches to be rebooted to take effect. Before rebooting, both VLAN 4002 and 4006 are no longer available but only 4006 is reserved on the cluster switches after reboot.
Also note that in a cluster both switch members are required to use the same mode for a port, either QinQ or 802.1Q, for the same BD.
Bridge domains can also be created with a textual description to provide more context like so:
CLI (network-admin@leaf-5) > bridge-domain-create name bd1000 scope fabric vxlan 10000 description business rsvd-vlan 4002
CLI (network-admin@leaf-5) > bridge-domain-show
name scope vxlan auto-vxlan description ports cluster-name
------ ------ ----- ---------- ----------- ---- ------------
bd1000 fabric 10000 no business 13
The next configuration step is to set up the fabric overlay either with manual tunnel creation or with VTEP object configuration. The latter way is preferred.
In order to manually create a VXLAN tunnel, say, on cluster member leaf-5 in the Figure 9-6, the following commands can be used:
CLI (network-admin@leaf-5) > tunnel-create scope cluster name tunnel-5 vrouter-name vr5 peer-vrouter-name vr1 local-ip 20.20.25.1 remote-ip 20.20.27.1
CLI (network-admin@leaf-5) > trunk-modify name vxlan-loopback-trunk ports 17 jumbo
The same steps are required on the other leaf switches to deploy an overlay comprising a full mesh of VXLAN-based interconnections.
Alternatively, VTEP objects can be configured so that a full mesh of VXLAN interconnections is brought up automatically without requiring manual configuration.
This is the command required to create a VTEP object on a non-clustered switch:
CLI (network-admin@leaf-1) > vtep-create name vtep1 vrouter-name vr1 ip 20.20.25.1
which can be used to create each fabric termination point (vtep1, vtep2, etc. on each leaf switch).
In case of clustered switches using a common VIP, the following command pairs should be used:
CLI (network-admin@leaf-5) > vtep-create name vtep5 vrouter-name vr5 ip 103.103.103.250 virtual-ip 103.103.103.10
CLI (network-admin@leaf-6) > vtep-create name vtep6 vrouter-name vr6 ip 103.103.103.251 virtual-ip 103.103.103.10
The next configuration step is to associate the BD’s VXLAN ID (e.g., 10000) to the newly created tunnel(s) or VTEP objects. For example, this command implements the association with a tunnel:
CLI (network-admin@leaf-1) > tunnel-vxlan-add name tunnel-1 vxlan 10000
whereas this command implements the association with a VTEP object:
CLI (network-admin@leaf-1) > vtep-vxlan-add name vtep1 vxlan 10000
Once the VXLAN overlay deployment is complete and the BD’s VXLAN ID is associated to it, then it is possible to start adding switch ports.
Informational note: When a port is assigned to a BD (as in the examples below) it cannot be associated with a regular VLAN configured with the vlan-create command, and vice versa.
In order to add a port connected to the cloud provider network to the BD, you can use the following command that explicitly maps both inner and outer VLANs to the BD:
CLI (network-admin@leaf-7) > bridge-domain-port-add name bd1000 port 13 outer-vlan 13 inner-vlan 100
This configuration also means that a frame’s MAC lookup in the Layer 2 table is performed including both the outer VLAN and the inner VLAN.
The same configuration can be used to add a port connected to an 802.1ad-configured server to the BD:
CLI (network-admin@leaf-2) > bridge-domain-port-add name bd1000 port 11 outer-vlan 13 inner-vlan 100
In order to add a port connected to an 802.1Q network to the BD, you can use the following command that explicitly maps only the outer VLAN to the BD (and preserves the inner VLAN):
CLI (network-admin@leaf-2) > bridge-domain-port-add name bd1000 port 15 outer-vlan 19
This configuration also means that a frame’s MAC lookup in the Layer 2 table is performed including only the outer VLAN while the inner VLAN is ignored.
In order to add a port connected to an 802.1Q server to the BD, you can use the following command that explicitly associates one or more VLANs on the same switch port to the same BD:
CLI (network-admin@leaf-1) > bridge-domain-port-add name bd2100 port 10 vlans 210,310
Note that multiple VLAN IDs on the same port can be part of the same bridge domain. However, VLAN IDs added to a BD cannot be reused with another BD on the same port. Also note that MAC addresses associated with these two VLAN IDs must be unique within the BD.
Finally, you should also configure all cluster ports/trunks and ports in QinQ mode to work with the 802.1ad TPID (i.e., 0x88A8) with the following command:
CLI (network-admin@switch) > port-config-modify port <port_number> allowed-tpid q-in-q
Port removal from a BD and BD deletion follow the reverse order compared to the addition process described in the above steps.
Ports have to be removed first with the command:
CLI (network-admin@leaf-2) > bridge-domain-port-remove name bd1000 port 15
Then a BD’s VXLAN ID may be manually removed from the associated tunnel(s):
CLI (network-admin@leaf-2) > tunnel-vxlan-remove name tunnel-2 vxlan 10000
or from the corresponding VTEP objects like so:
CLI (network-admin@leaf-2) > vtep-vxlan-remove name vtep2 vxlan 10000
Finally, the BD can be deleted with the following command:
CLI (network-admin@leaf-2) > bridge-domain-delete name bd1000