About Split Brain and Detection Script
In Pluribus parlance, a cluster has two switches (nodes) that operate as a single logical switch. These switches periodically as well as on an event driven basis, exchange control messages over the control network to keep the status and tables (such as L2 tables, vLAG, STP, cluster states, etc) between the two switches synchronized. If one of the nodes in the cluster fails to communicate with the peer node for three consecutive cluster-sync messages, then the node sets the cluster to offline mode and attempts to function in an independent mode. If one of the cluster nodes is down, then operating as an independent node helps to maintain continuity.
However, if both nodes are up and the nodes are unable to sync-up (for example, due to the cluster network going down), then operating as an independent node is not desirable. This situation can lead to duplicate packets such as broadcast-unknown unicast or multicast (BUM) traffic or traffic loss. This condition is known as split brain.
In Netvisor ONE version 6.1.1 HF1, a script is provided as part of Netvisor package for the detection and recovery of split brain.
Note: The control network can be set to management or in-band using the fabric-local-modify control-network [in-band|mgmt] command.
Handling of Split Brain Using Detection Script
Caution: We recommend disabling the split brain script before performing any software upgrades on the switches in the cluster.
As a pre-requisite, you must install split brain detection script as a service on both nodes of the cluster pair. Netvisor ONE supports control network over Management or over in-band IP.
Note: Starting with Netvisor ONE version 6.1.1 HF4, the split brain install/uninstall scripts are available in /opt/nvOS/bin/pn-scripts directory. Having the scripts in /opt/nvOS/bin/pn-scripts directory enables you to invoke the scripts from both the Netvisor ONE CLI and REST API shell prompt.
When the script is installed on leaf switches:
- The script detects the split brain condition based on the following factors:
- When the control network is over management, losing the management network connection results in split brain.
- When the control network is over in-band, cluster links going down results in split brain
- Once the cause is detected, the script proceeds to quarantine the cluster slave (backup) switch. That is, all ports on the slave switch are brought down except the cluster ports. Disabling of ports is not persistent with switch-reboot, hence no manual intervention to enable ports is required.
To install and run the script as a system service, run the pn_split_brain_install.sh command by using Netvisor ONE CLI. The service will persist a switch reboot. It is not required to re-run the script on every reboot or power-cycle. To stop the script from running, use the pn_split_brain_uninstall.sh command.
The script can also detect when the cluster is back online.
Below is an example on how to use the script. To install and run the script, use the command:
CLI (network-admin@switch*) > pn-script-run name pn_split_brain_install.sh
Executing /opt/nvOS/bin/pn-scripts/pn_split_brain_install.sh:
Created symlink from /etc/systemd/system/multi-user.target.wants/svc-nvOS-split-brain.service to /etc/systemd/system/svc-nvOS-split-brain.service.
To know or detect the status of the script, use the command:
CLI (network-admin@switch*) > exit
root@switch*:~#
root@switch*:~# systemctl status svc-nvOS-split-brain.service
● svc-nvOS-split-brain.service - Service to check for split brain functionality in cluster-slave and disable ports
Loaded: loaded (/etc/systemd/system/svc-nvOS-split-brain.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-11-17 02:27:24 PST; 7s ago
Main PID: 25780 (perl)
Tasks: 2
Memory: 12.8M
CPU: 1.697s
CGroup: /system.slice/svc-nvOS-split-brain.service
├─25780 /usr/bin/perl /usr/bin/pn_split_brain.pl
Dec 06 19:39:18 switch* systemd[1]: Started Service to check for split brain functionality in cluster-slave and disable ports.
To stop the script from running, use the pn_split_brain_uninstall.sh command as below:
CLI (network-admin@switch*) > pn-script-run name pn_split_brain_uninstall.sh
Executing /opt/nvOS/bin/pn-scripts/pn_split_brain_uninstall.sh:
Removed symlink /etc/systemd/system/multi-user.target.wants/svc-nvOS-split-brain.service.
To know or detect the status of the script, use the command:
CLI (network-admin@switch*) > exit
root@switch*:~#
root@switch*:~# systemctl status svc-nvOS-split-brain.service
● svc-nvOS-split-brain.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
To install the script on all switches of the fabric, use the command:
CLI (network-admin@switch*) > switch * pn-script-run name pn_split_brain_install.sh
To uninstall the script on all switches of the fabric, use the command:
CLI (network-admin@switch*) > switch * pn-script-run name pn_split_brain_uninstall.sh
Note:
- The allow-offline-cluster-nodes command option in the transaction-settings-modify command is turned OFF by default and further fabric and cluster scoped transactions are not allowed until the cluster node is back online.
- Quarantine of cluster slave (backup) node mitigates the traffic loss due to split brain issue.
- In Fabric over Layer 3, to detect split brain, you should have 'allow-as' enabled (so that in-band-ip of cluster master is reachable via spine switch) or the management link should be present.
Recovery handling
To recover from split brain, the quarantined switch should be brought back to active service after the cluster comes back online. The script detects when the cluster is back online and proceeds to reboot the cluster slave and it comes back to active service with the ports enabled.
To check the status of ports, use the command:
CLI (network-admin@switch) > cluster-bringup-show
Where the status displays ports-enabled indicating that both devices of cluster are now active and up for service.