Upgrading the Netvisor ONE Software

---

Software upgrades are a routine maintenance procedure that must be completed every so often.  However, there are some guidelines to consider before you start the upgrade procedures.  

Before you start the upgrade process, obtain the required upgrade software.  You can download the software manually and copy it to a switch before beginning the upgrade procedures.

Note: If you are upgrading from Netvisor ONE version 3.0.4 or earlier to Netvisor ONE version 5.x.x or 6.x.x, then, you must first upgrade to Netvisor ONE version 3.1.1 before upgrading to a later release. The direct upgrade from version 3.0.4 or earlier to version 5.x.x/6.x.x is not supported.

Caution: We recommend disabling the split brain script before performing any software upgrades on the switches in the cluster if you had installed split brain detection script as a service on both nodes of the cluster pair. 

Software and fabric upgrade process comprises of two distinct phases: (i) the installation (upgrade) of the new software and (ii) a switch reboot to activate the new software.

Read this section completely before starting the software upgrade procedure.

To upgrade the software on a switch, use the software-upgrade command.

CLI (network-admin@switch) > software-upgrade

Starting with Netvisor ONE 6.1.0 release, additional options such as auto-reboot|no-auto-reboot, abort, and manual reboot are supported on the software-upgrade command. By using these options, you can control the automatic reboot of switch after software upgrade completes or you can abort an upgrade that is in progress. 

If you start software upgrade by specifying auto-reboot parameter for software-upgrade command, switch automatically reboots after software upgrade completes and switch boots up with new software version. 

However, if you start the software upgrade by specifying the no-auto-reboot parameter for software-upgrade command, switch completes the software upgrade and waits for administrator to manually reboot the switch to boot into upgraded software. After determining that upgrade is complete,  you must issue a software-upgrade-reboot command to reboot the switch to boot with upgraded software. 

While software upgrade is in progress or software upgrade is complete and switch is waiting for administrator to issue software-upgrade-reboot command, if you want to abort the upgrade and keep switch in current software version, you can use software-upgrade-abort command to abort the upgrade process. This scenario is explained further in the examples below.

Note: The software-upgrade-abort and software-upgrade-reboot commands are supported only if upgrade is started using software-upgrade command. Do not run these commands if you started fabric wide upgrade using fabric-upgrade command. An error message is displayed if you do so.

To start a software upgrade follow the steps below:

1. View the current version of Netvisor ONE on the switch by using command:

For example,

CLI network-admin@switch > software-show

version: 6.0.1-6000116966

2. Identify the software package. Depending on the version of Netvisor ONE running on your switch, you should use appropriate software upgrade package.  Select the appropriate upgrade bundle from the following upgrade matrix table based on the current version on your switch:

3. Copy the upgrade package to the switch, to do:

1. Enable Secure File Transfer Protocol (SFTP) on the switch:

CLI (network-admin@switch)> admin-sftp-modify enable

sftp password:

confirm sftp password:

CLI (network-admin@switch)>

2. Upload the Software package to the switch:

root@server-os-9:~/# sftp sftp@switch

The authenticity of host 'switch (10.0.0.02)' can't be established.

RSA key fingerprint is SHA256:SI8VQZgJCppbrF4sRcby36Fx7rz3Hh5EJllPPyScLZU.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'switch1, 10.0.0.02 (RSA) to the list of known hosts.

* Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *

* ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *

* By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *

* End User License Agreement (EULA). The EULA can be accessed via *

* http://www.pluribusnetworks.com/eula or by using the command "eula-show" *

Password:

Connected to switch

sftp> cd import

sftp> put nvOS- nvOS-6.1.0-6010018118-onvl.pkg

Uploading nvOS- nvOS-6.1.0-6010018118-onvl.pkg 

nvOS- nvOS-6.1.0-6010018118-onvl.pkg

nvOS- nvOS-6.1.0-6010018118-onvl.pkg 100% 332MB 7.5MB/s 04:00

4. Start the upgrade process by using the software-upgrade command with package parameter, which allows you to specify the name of the upgrade file.  The switch gets automatically rebooted after software upgrade is complete. 

CLI (network-admin@switch) > software-upgrade package nvOS-6.1.0-6010018118-onvl.pkg

Scheduled background update. Use software-upgrade-status-show to check. Switch will reboot itself. DO NOT reboot manually.

Caution: Do not reboot or power off the switch during the upgrade procedure.  When software upgrade is complete, switch reboots automatically.

5. Monitor the upgrade process using the software-upgrade-status-show command:

CLI (network-admin@switch)>software-upgrade-status-show show-interval5

[Apr11.21:30:41] Starting software upgrade ...

[Apr11.21:30:42] Cleaning old package bundles

[Apr11.21:30:42] Checking available disk space...

[Apr11.21:30:42] Avbl free space: 12.69G, Required: 0.62G

[Apr11.21:30:42] Unpacking local package bundle...

[Apr11.21:30:42] Extracting initial bundle.

.

.

log

-----------------------------------------------------------------

[Apr11.21:30:41] Starting software upgrade ...

[Apr11.21:30:42] Cleaning old package bundles

[Apr11.21:30:42] Checking available disk space...

[Apr11.21:30:42] Avbl free space: 12.69G, Required: 0.62G

[Apr11.21:30:42] Unpacking local package bundle...

[Apr11.21:30:42] Extracting initial bundle.

[Apr11.21:30:50] Decrypting signed bundle.

[Apr11.21:30:52] Extracting signed bundle.

[Apr11.21:31:00] Extracting packages.

[Apr11.21:31:09] Fetching repository metadata.

[Apr11.21:31:09] Skipping dpkg update in current boot image

[Apr11.21:31:11] Computing package update requirements.

[Apr11.21:31:12] Upgrade agent version: 6.0.1-6000116966

[Apr11.21:31:12] Upgrading software upgrade framework

[Apr11.21:31:16] Fetching repository metadata.

[Apr11.21:31:17] Skipping dpkg update in current boot image

[Apr11.21:31:17] Computing package update requirements.

[Apr11.21:31:17] Upgrade agent version: 6.1.0-6010018118

[Apr11.21:31:17] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010018118

[Apr11.21:32:28] Cleaning up old BEs.

[Apr11.21:32:30] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010018118

[Apr11.21:32:30] Software upgrade completed. Rebooting.

After the switch reboots and you see following message in serial console of the switch, you can SSH to switch as the network-admin: 

nvOS system info:

serial number: 1XXXXXXX00059

hostid: 090XXX9d

device id:

[ OK ] Started NetVisor Operating System.

Starting nvOSd Monitor...

[ OK ] Started nvOSd Monitor.

[ OK ] Reached target Multi-User System.

[ OK ] Reached target Graphical Interface.

[ OK ] Started Stop ureadahead data collection 45s after completed startup.

Starting Update UTMP about System Runlevel Changes...

[ OK ] Started Update UTMP about System Runlevel Changes.

* Welcome to Pluribus Networks Inc. Netvisor(R). This is a monitored system. *

* ACCESS RESTRICTED TO AUTHORIZED USERS ONLY *

* By using the Netvisor(R) CLI,you agree to the terms of the Pluribus Networks *

* End User License Agreement (EULA). The EULA can be accessed via *

* http://www.pluribusnetworks.com/eula or by using the command "eula-show" *

switch1 login: network-admin

Password:

Netvisor OS Command Line Interface 6.1

Connected to Switch switch; nvOS Identifier:0x90XXXX9; Ver: 6.1.0-6010018118

CLI (network-admin@switch) > software-show

version: 6.1.0-6010018118

To verify that a standalone (non-cluster) switch is fully operational after the software upgrade, you can use the log-event-show command. The 'System is up for service' log message indicates that the switch is ready for forwarding. For example:

CLI (network-admin@Leaf1)> log-event-show

category time                       name           code  level event-type message

-------- -------------------------- -------------- ----- ----- ---------- ------------------------

event    2021-03-10,23:37:40.572119 systemup_alert 11513 note  system     System is up for service

Note: You can also use the 'System is up for service' log message to verify that a switch is functional after a reboot or restart triggered by the following operations: fabric-upgrade, switch-reboot, nvos-restart, fabric-join, cluster repeer, and config import commands.

Apart from upgrading Netvisor ONE on individual switches, starting from Netvisor ONE version 6.1.0, you can perform software upgrade on all switches in a fabric using software-upgrade from one switch and reboot the switches to perform  fabric-wide software upgrade (software upgrade on all the nodes in a fabric).

You can choose one of the below options to perform software upgrade on all switches from one switch:

• Start software upgrade on entire fabric and reboot the switches sequentially

OR

• Start software upgrade on entire fabric and reboot all switches, but not sequentially

Start software upgrade on entire fabric and reboot the switches sequentially

Follow the steps to perform software upgrade on entire fabric and reboot the switches sequentially:

1. Enable SFTP on all switches and provide the credentials for each switch using the CLI command:

CLI (network-admin@switch)> switch * admin-sftp-modify enable

2. Enable shell access for network-admin role on all switches in the fabric:

CLI (network-admin@switch)> switch * role-modify name network-admin shell

3. Download the software upgrade bundle to each switch by entering the below command only on one switch: 

CLI (network-admin@switch)> switch * shell cd /sftp/import;wget http://sandy.pluribusnetworks.com//offline-pkgs/onvl/nvOS-6.1.0/nvOS-6.1.0-6010018118-onvl.pkg  

4. Start the software upgrade process on all the switches using the command:

CLI (network-admin@switch)> switch * software-upgrade package nvOS-6.1.0-6010018118-onvl.pkg no-auto-reboot

5. Verify that all nodes display the "Waiting for software-upgrade-abort/software-upgrade-reboot" message by using the command individually on each switch:

CLI (network-admin@switch)> software-upgrade-status-show

6. Issue the software-upgrade-reboot command on each switch in the fabric one by one based on the reboot sequence that best works for your deployment. 

Start software upgrade on entire fabric and reboot all switches non-sequentially 

Follow the steps to perform fabric-wide software upgrade without reboot sequence:

1. Enable SFTP on all switches and provide the credentials for each switch using the CLI command:

CLI (network-admin@switch)> switch * admin-sftp-modify enable

2. Enable shell access for network-admin role on all switches in the fabric:

CLI (network-admin@switch)> switch * role-modify name network-admin shell

3. Download the software upgrade bundle to each switch by entering the below command only on one switch: 

CLI (network-admin@switch)> switch * shell cd /sftp/import;wget http://sandy.pluribusnetworks.com//offline-pkgs/onvl/nvOS-6.1.0/nvOS-6.1.0-6010018118-onvl.pkg  

4. Start the software upgrade process on all the switches with no-auto-reboot command parameter:

CLI (network-admin@switch)> switch * software-upgrade package nvOS-6.1.0-6010018118-onvl.pkg no-auto-reboot

5. Verify that all nodes display the "Waiting for software-upgrade-abort/software-upgrade-reboot" message by using the command individually on each switch

CLI (network-admin@switch)> software-upgrade-status-show

6. Reboot all the switches except the node where you are running the commands. This step ensures that the software-upgrade-reboot command is executed by all the switches before the node where you originally started the upgrade process goes down. 

CLI (network-admin@switch)> switch <comma separated non controller switches> software-upgrade-reboot

7. Reboot the node where you started the upgrade and complete the upgrade process of the fabric: 

CLI (network-admin@switch)> switch <controller node> software-upgrade-reboot

Below are some examples of the software upgrade that can be used when you want the switch to not reboot automatically after upgrade is complete, but wants to reboot manually or if you want to abort the software upgrade that is in progress.

• Usage of the software-upgrade command with no-auto-reboot option and later issuing an abort command to abort the upgrade process:

CLI (network-admin@switch) > software-upgrade package nvOS- 6.1.0-6010017911-onvl.pkg no-auto-reboot

CLI (network-admin@ switch) > software-upgrade-status-show

log

----------------------------------------------------------------------

[Mar11.07:47:21] Starting software upgrade ...

[Mar11.07:47:22] Checking available disk space...

[Mar11.07:47:22] Avbl free space: 82.67G, Required: 1.28G

[Mar11.07:47:22] Unpacking local package bundle...

[Mar11.07:47:22] Extracting initial bundle.

[Mar11.07:47:38] Decrypting signed bundle.

[Mar11.07:47:39] Extracting signed bundle.

[Mar11.07:47:56] Extracting packages.

[Mar11.07:48:13] Fetching repository metadata.

[Mar11.07:48:13] Skipping dpkg update in current boot image

[Mar11.07:48:13] Computing package update requirements.

[Mar11.07:48:13] Upgrade agent version: 6.1.0-6010017911

[Mar11.07:48:13] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.07:49:01] Waiting for software-upgrade-abort/software-upgrade-reboot

CLI (network-admin@switch) > software-upgrade-abort

Upgrade running, scheduled abort

CLI (network-admin@switch) > software-upgrade-status-show

log

---------------------------------------------------------------------

[Mar11.07:47:21] Starting software upgrade ...

[Mar11.07:47:22] Checking available disk space...

[Mar11.07:47:22] Avbl free space: 82.67G, Required: 1.28G

[Mar11.07:47:22] Unpacking local package bundle...

[Mar11.07:47:22] Extracting initial bundle.

[Mar11.07:47:38] Decrypting signed bundle.

[Mar11.07:47:39] Extracting signed bundle.

[Mar11.07:47:56] Extracting packages.

[Mar11.07:48:13] Fetching repository metadata.

[Mar11.07:48:13] Skipping dpkg update in current boot image

[Mar11.07:48:13] Computing package update requirements.

[Mar11.07:48:13] Upgrade agent version: 6.1.0-6010017911

[Mar11.07:48:13] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.07:49:01] Waiting for software-upgrade-abort/software-upgrade-reboot

[Mar11.07:50:39] User indicated software-upgrade-abort

[Mar11.07:50:40] User aborted the upgrade.

[Mar11.07:50:42] Software upgrade aborted

[Mar11.07:50:42] ERR: Upgrade failed: User aborted the upgrade.

• Usage of the software-upgrade command with no-auto-reboot option and later issuing a software-upgrade-reboot command to complete the upgrade process:

CLI (network-admin@switch) > software-upgrade package nvOS-6.1.0-6010017911-onvl.pkg no-auto-reboot

CLI (network-admin@switch) > software-upgrade-status-show

log

--------------------------------------------------------------

[Mar11.07:52:59] Starting software upgrade ...

[Mar11.07:52:59] Checking available disk space...

[Mar11.07:52:59] Avbl free space: 82.67G, Required: 1.28G

[Mar11.07:52:59] Unpacking local package bundle...

[Mar11.07:52:59] Extracting initial bundle.

[Mar11.07:53:16] Decrypting signed bundle.

[Mar11.07:53:17] Extracting signed bundle.

[Mar11.07:53:33] Extracting packages.

[Mar11.07:53:50] Fetching repository metadata.

[Mar11.07:53:50] Skipping dpkg update in current boot image

[Mar11.07:53:50] Computing package update requirements.

[Mar11.07:53:51] Upgrade agent version: 6.1.0-6010017911

[Mar11.07:53:51] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.07:54:38] Waiting for software-upgrade-abort/software-upgrade-reboot

CLI (network-admin@switch) > software-upgrade-reboot

Upgrade running, scheduled reboot

CLI (network-admin@switch) > Shared connection to switch closed.

Check the status on the new BE:

       

CLI (network-admin@switch) > software-upgrade-status-show

log

-------------------------------------------------------------

[Mar11.07:52:59] Starting software upgrade ...

[Mar11.07:52:59] Checking available disk space...

[Mar11.07:52:59] Avbl free space: 82.67G, Required: 1.28G

[Mar11.07:52:59] Unpacking local package bundle...

[Mar11.07:52:59] Extracting initial bundle.

[Mar11.07:53:16] Decrypting signed bundle.

[Mar11.07:53:17] Extracting signed bundle.

[Mar11.07:53:33] Extracting packages.

[Mar11.07:53:50] Fetching repository metadata.

[Mar11.07:53:50] Skipping dpkg update in current boot image

[Mar11.07:53:50] Computing package update requirements.

[Mar11.07:53:51] Upgrade agent version: 6.1.0-6010017911

[Mar11.07:53:51] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.07:54:38] Waiting for software-upgrade-abort/software-upgrade-reboot

[Mar11.07:57:07] User indicated software-upgrade-reboot

[Mar11.07:57:08] User issued software-upgrade-reboot, rebooting the switch.

[Mar11.07:57:08] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.07:57:08] Software upgrade completed. Rebooting.

[Mar11.07:59:35] Upgrade agent is listening.

• Usage of the software-upgrade command in default mode:

CLI (network-admin@switch) > software-upgrade package nvOS-6.1.0-6010017911-onvl.pkg

log

--------------------------------------------------------------

[Mar11.09:32:10] Starting software upgrade ...

[Mar11.09:32:10] Checking available disk space...

[Mar11.09:32:10] Avbl free space: 81.89G, Required: 1.28G

[Mar11.09:32:10] Unpacking local package bundle...

[Mar11.09:32:10] Extracting initial bundle.

[Mar11.09:32:27] Decrypting signed bundle.

[Mar11.09:32:28] Extracting signed bundle.

[Mar11.09:32:44] Extracting packages.

[Mar11.09:33:01] Fetching repository metadata.

[Mar11.09:33:01] Skipping dpkg update in current boot image

[Mar11.09:33:02] Computing package update requirements.

[Mar11.09:33:02] Upgrade agent version: 6.1.0-6010017911

[Mar11.09:33:02] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.09:33:51] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

[Mar11.09:33:51] Software upgrade completed. Rebooting.

Shared connection to switch closed.

• Usage of the software-upgrade command in default mode and later issuing an abort command to abort the upgrade process. The software-upgrade-abort command is also supported in the default mode (same as auto-reboot mode):

CLI (network-admin@switch) > software-upgrade package nvOS-6.1.0-6010017911-onvl.pkg

CLI (network-admin@switch) > software-upgrade-abort

CLI (network-admin@switch) > software-upgrade-status-show

log

-------------------------------------------------------------

[Mar11.09:43:04] Starting software upgrade ...

[Mar11.09:43:04] Checking available disk space...

[Mar11.09:43:04] Avbl free space: 81.15G, Required: 1.28G

[Mar11.09:43:04] Unpacking local package bundle...

[Mar11.09:43:04] Extracting initial bundle.

[Mar11.09:43:13] User indicated software-upgrade-reboot <============ time at which user initiated the command

[Mar11.09:43:21] Decrypting signed bundle.

[Mar11.09:43:22] Extracting signed bundle.

[Mar11.09:43:38] Extracting packages.

[Mar11.09:43:55] Fetching repository metadata.

[Mar11.09:43:56] Skipping dpkg update in current boot image

[Mar11.09:43:56] Computing package update requirements.

[Mar11.09:43:56] Upgrade agent version: 6.1.0-6010017911

[Mar11.09:43:56] User aborted the upgrade.  <================ place at which actual abort is performed (consistent state)

[Mar11.09:43:56] Software upgrade aborted

[Mar11.09:43:56] ERR: Upgrade failed: User aborted the upgrade.

Another new command added in Netvisor ONE version 6.1.0 is the software-upgrade-instant-status-show command, which displays the most current status of the upgrade process of all the switches in the fabric. This command is different from the software-upgrade-status-show command, where all details of the upgrade process is displayed.

Use this command to view the most recent status on each switch in the fabric when you perform the upgrade process on all switches of the fabric (fabric-wide upgrade):

CLI (network-admin@switch) >  switch * software-upgrade-instant-status-show show-interval 1

switch       log

------- --------------------------------------------------------------------

Switch1 [Mar11.07:48:13] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

Switch2 [Mar11.07:48:13] Upgrading nvOS 6.0.1-6000116966 -> 6.1.0-6010017911

The switch * in the above command indicates all the switches in that fabric which is undergoing the software upgrade process.