Glossary		Print Page		Document Feedback		Support & Services		Pluribus Networks		Pluribus Blog		LinkedIn		Facebook		Twitter

Configuring Kubernetes Visibility

The Kubernetes client in NetVisor OS connects to the Kubernetes API server for real time notifications of events and changes. To create a connection between the Kubernetes client in NetVisor OS and the Kubernetes API server, use the command:

CLI (network-admin@switch1) k8s-connection-create

k8s-connection-create	Create a new connection to the Kubernetes API server.
cluster-name cluster-name-string	Specify a name for the Kubernetes cluster.
rest-user user-name	Specify the username for vREST access.
kube-config kube-config-string	Specify the absolute path to the kubeconfig file. The default location is /root/.kube/config.
password	Specify the vREST password.
enable\|disable	Enable or disable the connection to Kubernetes API server.
track-history\|no-track-history	Enable or disable Kubernetes cluster history tracking.
disk-space disk-space-number	Specify the disk-space allocated for history (including rotated log files).

For example:

CLI (network-admin@switch) > k8s-connection-create cluster-name k8s01 kube-config /root/.kube/config

rest-user network-admin enable

rest-user password :

confirm rest-user password:

While configuring Kubernetes, please remember the following guidelines:

NetVisor OS supports Kubernetes versions 1.19, 1.21, and 1.22.
The Kubernetes client uses REST API to interact with NetVisor OS. Thus, you must enable web services on the data interface or the management interface of the switch by using the admin-service-modify command.
You must enable LLDP on all the fabric nodes before configuring Kubernetes visibility. All Kubernetes servers must be visible in the lldp-show outputs of the fabric switches.
To establish a connection with the Kubernetes API server succesfully, you must first copy the Kubernetes configuration file called kubeconfig from the Kubernetes control plane node to the switch. You can then specify the path to the file in the k8s-connection-create command.
The Kubernetes API server must be reachable from linux global routing table and the API server's fully qualified domain name (FQDN) must be resolvable via nameserver.

Issue the k8s-connection-show command to display the Kubernetes connection information.

CLI (network-admin@switch1) k8s-connection-show

k8s-connection-show	Display the connection to the Kubernetes API server.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
kube-config kube-config-string	The absolute path to the kube config file.
rest-user user-name	The username for vRest access on switch.
enable\|disable	Status of the connection to Kubernetes API server.
track-history\|no-track-history	Enable or disable Kubernetes cluster history tracking.
disk-space disk-space-number	The disk-space allocated for history (including rotated log files).
create-vport\|no-create-vport	Enable or disable vPort creation for Pod traffic.
overlay-vxlan-analytics\|no-overlay-vxlan-analytics	Enable or disable Pod traffic analytics.
location location-number	Location where the connection is running.
connection-error connection-error-string	Kubernetes connection error.
k8s-vlan vlan-id	The VLAN ID used by Kubernetes for Pod vPort creation

CLI (network-admin@switch1) > k8s-connection-show format all layout vertical

cluster-name: k8s01

kube-config: /root/.kube/config

rest-user: network-admin

enable: yes

track-history: yes

disk-space: 50M

create-vport: yes

overlay-vxlan-analytics: no

location: 1262261009

Use the k8s-connection-delete and the k8s-connection-modify commands to delete or modify the configuration.

Displaying Kubernetes Cluster Information

A Kubernetes cluster consists of a control plane node and a set of worker nodes that run containerized applications. Every cluster runs at least one worker node. You can view the Kubernetes cluster information by using the k8s-cluster-show command.

CLI (network-admin@switch1) k8s-cluster-show

k8s-cluster-show	Display Kubernetes cluster information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
control-plane-node control-plane-node-string	The name of the Kubernetes control plane node.
control-plane-node-ip control-plane-node-ip-string	The IP address of the control plane node.
controller-status controller-status-string	The status of the Kubernetes controller.
scheduler-status scheduler-status-string	The status of the Kubernetes scheduler.
action none\|add\|remove\|update	The last action.

For example:

CLI (network-admin@switch1) > k8s-cluster-show

cluster-name control-plane-node control-plane-node-ip controller-stauts scheduler-status last-changed

------------ ------------------ --------------------- ----------------- ---------------- --------------

k8s01 server-test-93 192.168.10.10 healthy healthy 07-12,23:25:21

Displaying Kubernetes Node Information

The control plane machine and the worker machines that are part of a Kubernetes cluster are called nodes. Nodes host the Pods that are the components of the application workload. You can display information on Kubernetes nodes by using the command:

CLI (network-admin@switch1) k8s-node-show

k8s-node-show	Display information on Kubernetes master and worker nodes.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
node node-string	The name of the Kubernetes node.
role role-string	The role assigned to the node.
status status-string	The status of the node.
ip ip-string	The IP address of the node.
pod-CIDR pod-CIDR-string	The Pod CIDR for the node.
labels labels-string	The Pod labels.
action none\|add\|remove\|update	The last action performed.

CLI (network-admin@switch) k8s-node-show

cluster-name node role status ip pod-CIDR last-changed

------------ --------------- ------------- ------------ ------------- ------------- --------------

k8s01 server-test-103 worker KubeletReady 192.168.99.22 10.244.2.0/24 11-11,05:59:13

k8s01 server-test-93 control-plane KubeletReady 192.168.99.20 10.244.0.0/24 11-11,06:02:21

k8s01 server-test-94 worker KubeletReady 192.168.99.21 10.244.1.0/24 11-11,05:58:36

Displaying Information on Kubernetes Pods

Pods are the smallest units that you can deploy using Kubernetes. Pods consist of one or more containers that are always scheduled together. Each Kubernetes pod has a unique IP address. To display information on Pods, use the command:

CLI (network-admin@switch1) k8s-pod-show

k8s-pod-show	Display Kubernetes Pod information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
name name-string	The name of the Daemonset.
namespace namespace-string	The namespace.
ip ip-string	The IP address of the Pod.
state state-string	The state of the Pod.
container-state container-state-string	The state of the Pod's container.
node node-string	The name of the Pod.
containers containers-string	The container images running on the Pod.
labels labels-string	The Pod labels.
action none\|add\|remove\|update	The last action performed.
connections connections-string	The connections to fabric switches.

For example:

CLI (network-admin@switch1) > k8s-pod-show format name,namespace,ip,state,node,containers,labels

name namespace ip state node containers labels ----------------------------------------- -------------------- ------------- ------- --------------- ------------------ ---------

kube-apiserver-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-apiserver,tier...

kube-proxy-d28w9 kube-system 192.168.99.22 Running server-test-103 k8s.gcr.io/kube-... controller-revision-hash:84f9...

kube-flannel-ds-ltctg kube-system 192.168.99.22 Running server-test-103 quay.io/coreos/f... app:flannel,controller-revisi...

dashboard-metrics-scraper-79c5968bdc-hwpcg kubernetes-dashboard 10.244.1.35 Running server-test-94 kubernetesui/met... k8s-app:dashboard-metrics-scr...

kube-scheduler-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-scheduler,tier...

kube-controller-manager-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-controller-man...

gloo-7dc68dd65b-clz98 gloo-system 10.244.2.36 Running server-test-103 quay.io/solo-io/... pod-template-hash:7dc68dd65b,...

Displaying Kubernetes Deployment Information

Deployments represent a set of multiple, identical Pods with no unique identities. A deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In this way, deployments help ensure that one or more instances of your application are available to serve user requests.

To display Kubernetes deployment information, use the command:

CLI (network-admin@switch1) k8s-deployment-show

k8s-deployment-show	Display Kubernetes deployment information.
cluster-name cluster-name-string	The name of the Kubernetes node.
name name-string	The name of the deployment.
namespace namespace-string	The name of the namespace.
replicas replicas-number	The total replicas of Pod.
ready-replicas ready-replicas-number	The ready replicas of Pod.
selector selector-string	The Pod label in deployment.
action none\|add\|remove\|update	The last action performed.

For example:

CLI (network-admin@switch1) > k8s-deployment-show

cluster-name name namespace replicas ready-replicas selector last-changed

------------ ------------------------- -------------------- -------- -------------- --------------------------------- ------------

pluribus my-app default 1 1 app:my-app 01:07:38

pluribus pingtest default 2 2 app:pingtest 01:07:38

pluribus kubernetes-dashboard kubernetes-dashboard 1 1 k8s-app:kubernetes-dashboard 01:07:38

pluribus dashboard-metrics-scraper kubernetes-dashboard 1 1 k8s-app:dashboard-metrics-scraper 01:07:39

pluribus hello-world default 5 5 run:load-balancer-example 01:07:39

pluribus petstore default 1 1 app:petstore 01:07:39

pluribus discovery gloo-system 1 1 gloo:discovery 01:07:39

pluribus gloo gloo-system 1 1 gloo:gloo 01:07:39

pluribus ingress gloo-system 1 1 gloo:ingress 01:07:39

pluribus ingress-proxy gloo-system 1 1 gloo:ingress-proxy 01:07:39

pluribus coredns kube-system 2 2 k8s-app:kube-dns 01:07:39

Displaying Information on Daemonsets

A DaemonSet manages groups of replicated Pods and ensures that all (or some) nodes run a copy of a Pod. DaemonSets adds Pods to nodes that are added to a cluster and removes the Pods from nodes which are removed from the cluster. Deleting a DaemonSet also deletes the Pods that it created.To display information on DaemonSets, use the command:

CLI (network-admin@switch1) k8s-daemonset-show

k8s-daemonset-show	Display Kubernetes Daemonset information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
name name-string	The name of the Daemonset.
namespace namespace-string	The name of the namespace.
desired-cnt desired-cnt-number	The desired number of Pods.
ready-cnt ready-cnt-number	The number of Pods ready.
selector selector-string	The Pod label selector for Daemonset.
last-changed date/time: yyyy-mm-ddTHH:mm:ss	The time when Daemonset was last changed.
action none\|add\|remove\|update	The last action performed.

For example:

CLI (network-admin@switch) > k8s-daemonset-show

cluster-name name namespace desired-cnt ready-cnt last-changed

------------ --------------------- -------------- ----------- --------- ------------

k8s01 speaker metallb-system 6 6 23:24:49

k8s01 csi-cephfsplugin rook-ceph 6 6 23:24:50

k8s01 csi-rbdplugin rook-ceph 6 6 23:24:50

k8s01 calico-node kube-system 6 6 23:24:50

k8s01 fluentd-elasticsearch kube-system 6 6 23:24:50

k8s01 kube-proxy kube-system 6 6 23:24:50

k8s01 nodelocaldns kube-system 6 6 23:24:50

k8s02 fluentd-elasticsearch kube-system 6 6 23:25:22

k8s02 kube-flannel-ds kube-system 6 6 23:25:22

k8s02 kube-proxy kube-system 6 6 23:25:22

Displaying Kubernetes Services

Kubernetes services provide a stable virtual IP addresses for a group of Pods. Unlike Pods, which are non-permanent resources that can be created and destroyed dynamically, services remain consistent. Services act as loadbalancers and are based on Linux IPTable or IPVS table. Use the k8s-service-show command to view the Kubernetes services information.

CLI (network-admin@switch1) k8s-service-show

k8s-service-show	Display Kubernetes service information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
name name-string	The name of the Daemonset.
namespace namespace-string	The name of the namespace.
type type-string	The service type.
service-ip service-ip-string	The IP address of the service.
external-ip external-ip-string	The external IP for services of type load balancer.
selector selector-string	The label selector for the service.
ports ports-string	The ports for the service.
target-ports target-ports-string	The ports targeted on Pods.
action none\|add\|remove\|update	The last action performed.

For example:

CLI (network-admin@switch) > k8s-service-show layout vertical

cluster-name: k8s02

namespace: default

type: ClusterIP

service-ip: 10.96.0.1

ports: 443/TCP

last-changed: 01-30,23:25:21

cluster-name: k8s02

namespace: default

type: ClusterIP

service-ip: 10.98.20.142

ports: 71/TCP

last-changed: 01-30,23:25:21

cluster-name: k8s02

namespace: kube-system

type: ClusterIP

service-ip: 10.96.0.10

ports: 53/UDP,53/TCP,9153/TCP

last-changed: 01-30,23:25:23

...

Displaying Kubernetes Endpoints

To view information about Pods targeted by a service (Pod IP addresses) in a Kubernetes cluster, use the command:

CLI (network-admin@switch) k8s-endpoint-show

k8s-endpoint-show	Display Kubernetes endpoint information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
name name-string	The name of the endpoint.
namespace namespace-string	Specify the namespace.
selector selector-string	Specify the label selector.
target-ports target-ports-string	Ports targeted on ports.
endpoints endpoints-string	The list of Pod IP addresses.
action none\|add\|remove\|update	The last changed action.

CLI (network-admin@switch) > k8s-endpoint-show layout vertical

cluster-name: k8s02

namespace: default

target-ports: 6443

last-changed: 01-30,23:25:21

cluster-name: k8s02

namespace: default

target-ports: 9378

endpoints: pending

last-changed: 01-30,23:25:21

cluster-name: k8s02

namespace: kube-system

target-ports: 53,53,9153

endpoints: 172.22.0.39,172.22.1.8

last-changed: 01-30,23:25:23

...

Displaying Kubernetes Ingress Information

Kubernetes Ingress manages the external access to services within a cluster. Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can give Services externally-reachable URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting. To display Ingress information, use the command:

CLI (network-admin@switch) k8s-ingress-show

k8s-ingress-show	Display Kubernetes Ingress information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
name name-string	The name of the endpoint.
namespace namespace-string	The namespace.
host host-string	Specify the hostname of the ingress.
path path-string	Path for the ingress rule.
service-name service-name-string	The backend service name.
service-port service-port-string	The backend service port.
resource-api-group resource-api-group-string	The backend resource API group
resource-kind resource-kind-string	The backend resource kind.
resource-name resource-name-string	The backend resource name.
action none\|add\|remove\|update	The last performed action.

For example:

CLI (network-admin@switch) > k8s-ingress-show

cluster-name name namespace host path service-name service-port last-changed

------------ ------------------------- --------- ---------------- ---- ------------ ------------ --------------

pluribus petstore-ingress default gloo.example.com /.* petstore 8080 07-12,23:25:22

pluribus name-virtual-host-ingress default foo.bar.com / service1 80 07-12,23:25:22

pluribus name-virtual-host-ingress default bar.foo.com / service2 80 07-12,23:25:22

Displaying Kubernetes Connectivity information.

the Kubernetes visibility feature enables you to locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity or performance issues. Use the k8s-connectivity-show command to view the connectivity information for a deployment, Daemonset, services or a Pod. Without any filter arguments, k8s-connectivity-show gives connectivity information for each Kubernetes node to the fabric.

CLI (network-admin@switch) > k8s-connectivity-show

k8s-connectivity-show	Display Kubernetes endpoint information.
namespace namespace-string	The namespace.
deployment deployment-string	The Kubernetes deployment.
daemonset daemonset-string	The Daemonset name.
service service-string	Kubernetes service name.
pod pod-string	The name of the Pod.
node node-string	The cluster node name.

CLI (network-admin@switch) > k8s-connectivity-show deployment petstore

cluster-name namespace deployment pod connections

------------ --------- ---------- ------------------------ ---------------------------------------

pluribus default petstore petstore-9d499b76f-hhmg4 server-test-94:enp1s0f0 -- 43:switch-1

pluribus default petstore petstore-9d499b76f-psv8w server-test-103:enp1s0f0 -- 16:switch-1

server-test-94:enp1s0f1 -- 44:switch-1

pluribus default petstore petstore-9d499b76f-qj58s server-test-94:enp1s0f0 -- 43:switch-1

server-test-94:enp1s0f1 -- 44:switch-1

pluribus default petstore petstore-9d499b76f-649g6 server-test-103:enp1s0f0 -- 16:switch-1

The output displays the connectivity of a deployment named petstore which has 4 replicas. The output shows how each replica (Pod) is connected to Arista fabric. For example, the first replica (petstore-9d499b76f-hhmg4) running on server-test-94 is connected through enp1s0f0 to port 43 of Arista switch switch-1.

Tracing Paths between Kubernetes Pods or Nodes

You can inspect all the paths that Kubernetes Pods or Kubernetes nodes use to communicate with each other across the fabric. Use the command k8s-trace-path and specify the source and destination to view all available shortest paths between them.

k8s-trace-paths	Display the available paths between two Kubernetes Pods or nodes.
src-pod	The name of the Kubernetes source Pod.
dst-pod	The name of the Kubernetes destination Pod.
src-node	The name of the Kubernetes source node.
dst-node	The name of the Kubernetes destination node.

The example below shows all the shortest paths from the Pod petstore running on node server-test-94 to Pod ingress-proxy running on node server-test-103. You can locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity issues.

CLI (network-admin@switch) > k8s-trace-paths src-pod petstore dst-pod ingress-proxy

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine2:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf4:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp7 -- swp7:Leaf4:swp6 -- enp1s0f0:server-test-103

Analyzing the Impact on Applications during Host or Leaf Replacement

The Kubernetes visibility feature allows you to proactively determine the impact on containerized applications during switch outages or server replacements. You can view the impact on a Kubernetes deployment, Daemonset, service, or Pods in the occasion of a switch or host replacement by using the k8s-impact-show command. The impact can be none if there is no impact, partial if alternate paths are present, or full if the Kubernetes Pods lose all connectivity.

CLI (network-admin@switch) > k8s-impact-show

k8s-impact-show	Display the available paths between two Kubernetes Pods or nodes.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
host host-string	The host or node for which the impact needs to be checked.
switch fabric-node-name	The switch for which the impact needs to be checked.
namespace namespace-string	The name of the namespace.
deployment deployment-string	The Kubernetes deployment name.
daemonset daemonset-string	The Kubernetes Daemonset name.
service service-string	The Kubernetes service name.
pod pod-string	The Kubernetes Pod.
impact unknown\|none\|partial\|full	The impact on the Pod when node or switch is replaced.

For example, you can view the impact on a Kubernetes deployment named petstore upon replacing a server named server-test-94 by using the command below:

CLI (network-admin@switch) > k8s-impact-show host server-test-94 deployment petstore

cluster-name namespace deployment pod impact connections

------------ --------- ---------- ------------------------ ------ ---------------------------------------

pluribus default petstore petstore-9d499b76f-psv8w none server-test-103:enp1s0f0 -- 16:switch-1

pluribus default petstore petstore-9d499b76f-hhmg4 full server-test-94:enp1s0f0 -- 43:switch-1

server-test-94:enp1s0f1 -- 44:switch-1

pluribus default petstore petstore-9d499b76f-qj58s full server-test-94:enp1s0f0 -- 43:switch-1

server-test-94:enp1s0f0 -- 43:switch-1

pluribus default petstore petstore-9d499b76f-649g6 none server-test-103:enp1s0f0 -- 16:switch-1

Displaying Kubernetes Network Information

Use the k8s-network-show command to show the relation between Kubernetes networking components like ingress, services, and Pods.

k8s-network-show	Display Kubernetes Pod information.
cluster-name cluster-name-string	The name of the Kubernetes cluster.
Specify any of the following options:
namespace namespace-string	The name of the namespace.
ingress-name ingress-name-string	The IP address of the Pod.
host host-string	The state of the Pod.
type type-string	The name of the Pod.
external-ip external-ip-string	The external IP address for services of type load balancer.
node-port node-port-string	The service type.
service-name service-name-string	The name of the service.
selector selector-string	The service selector to identify Pods.
endpoints endpoints-string	The list of Pod IP addresses.

For example, this command displays the ingress to Pods to services mapping:

CLI (network-admin@switch) > k8s-network-show layout vertical format all

cluster-name: k8s01

namespace: default

ingress-name: demo

host: www.demo.io

type: LoadBalancer

external-ip: 10.13.22.154

node-port: 31437

service-name: demo

selector: app:demo

endpoints: 172.19.126.9,172.19.127.74

cluster-name: k8s01

namespace: default

ingress-name: example-ingress

host: example.com

type: ExternalName

external-ip: my.database.example.com

service-name: my-service

...

Viewing Historical Information on Kubernetes Cluster

NetVisor OS implements a time machine feature that eliminates the need to reproduce past events that impacts Kubernetes applications in order to troubleshoot availability issues. This feature enables you to track historical data in a Kubernetes cluster. You can view various historical attributes of a cluster since the creation of a Kubernetes connection and inspect the changes to the cluster in a specific time interval.

Use the k8s-connection-modify command to enable history tracking and to specify the disk space for logging of historical data. For example:

CLI (network-admin@switch) > k8s-connection-modify cluster-name pluribus track-history disk-space 60M

The Kubernetes commands that allow history tracking have a set of statistics collection parameters that allows you to specify the start time, end time, duration of the data collection and so on. Listed below are the commands that you can view historical data for:

k8s-cluster-show

k8s-node-show

k8s-deployment-show

k8s-service-show

k8s-ingress-show

k8s-pod-show

k8s-connectivity-show

k8s-impact-show

k8s-cni-show

The data collection parameters of the commands above which enable you to view the history in different formats are:

start-time date/time: yyyy-mm-ddTHH:mm:ss	The start time of statistics collection.
end-time date/time: yyyy-mm-ddTHH:mm:ss	The end time of statistics collection.
duration duration: #d#h#m#s	The duration of statistic collection.
interval duration: #d#h#m#s	The interval between statistic collection.
since-start	Statistics collected since start time.
older-than duration: #d#h#m#s	Display statistics older than the specified duration.
within-last duration: #d#h#m#s	Display statistics within the last specified duration.

For example, to display the changes to the deployment named petstore in the last 30 minutes, use the command:

CLI (network-admin@switch) > k8s-deployment-show name petstore within-last 30m

cluster-name name namespace replicas ready-replicas selector last-changed action

------------ -------- --------- -------- -------------- ------------ ------------ ------

pluribus petstore default 4 4 app:petstore 09:11:51 add

pluribus petstore default 6 4 app:petstore 09:24:35 update

pluribus petstore default 6 5 app:petstore 09:24:45 update

pluribus petstore default 6 6 app:petstore 09:24:45 update