Configuring Kubernetes Visibility


The Kubernetes client in NetVisor OS connects  to the Kubernetes API server for real time notifications of events and changes. To create a connection between the Kubernetes client in NetVisor OS and the Kubernetes API server, use the command:


CLI (network-admin@switch1) k8s-connection-create

k8s-connection-create

Create a new connection to the Kubernetes API server.

cluster-name cluster-name-string

Specify a name for the Kubernetes cluster.

rest-user user-name

Specify the username for vREST access.

kube-config kube-config-string

Specify the absolute path to the kubeconfig file. The default location is /root/.kube/config.

password

Specify the vREST password.

enable|disable

Enable or disable the connection to Kubernetes API server.

track-history|no-track-history

Enable or disable Kubernetes cluster history tracking.

disk-space disk-space-number

Specify the disk-space allocated for history (including rotated log files).

For example:

CLI (network-admin@switch) > k8s-connection-create cluster-name k8s01 kube-config /root/.kube/config

rest-user network-admin enable

rest-user password :

confirm rest-user password:


While configuring Kubernetes, please remember the following guidelines:


  • NetVisor OS supports Kubernetes versions 1.19, 1.21, and 1.22.
  • The Kubernetes client uses REST API to interact with NetVisor OS. Thus, you must enable web services on the data interface or the management interface of the switch by using the admin-service-modify command.
  • You must enable LLDP on all the fabric nodes before configuring Kubernetes visibility. All Kubernetes servers must be visible in the lldp-show outputs of the fabric switches.
  • To establish a connection with the Kubernetes API server succesfully, you must first copy the Kubernetes configuration file called kubeconfig from the Kubernetes control plane node to the switch. You can then specify the path to the file in the k8s-connection-create command.
  • The Kubernetes API server must be reachable from linux global routing table and the API server's fully qualified domain name (FQDN) must be resolvable via nameserver.


Issue the k8s-connection-show command to display the Kubernetes connection information.


CLI (network-admin@switch1) k8s-connection-show

k8s-connection-show

Display the connection to the Kubernetes API server.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

kube-config kube-config-string

The absolute path to the kube config file.

rest-user user-name

The username for vRest access on switch.

enable|disable

Status of the connection to Kubernetes API server.

track-history|no-track-history

Enable or disable Kubernetes cluster history tracking.

disk-space disk-space-number

The disk-space allocated for history (including rotated log files).

create-vport|no-create-vport

Enable or disable vPort creation for Pod traffic.

overlay-vxlan-analytics|no-overlay-vxlan-analytics

Enable or disable Pod traffic analytics.

location location-number

Location where the connection is running.

connection-error connection-error-string

Kubernetes connection error.

k8s-vlan vlan-id

The VLAN ID used by Kubernetes for Pod vPort creation


CLI (network-admin@switch1) > k8s-connection-show format all layout vertical

cluster-name:            k8s01

kube-config:            /root/.kube/config

rest-user:               network-admin

enable:                  yes

track-history:           yes

disk-space:              50M

create-vport:            yes

overlay-vxlan-analytics: no

location:                1262261009


Use the k8s-connection-delete and the k8s-connection-modify commands to delete or modify the configuration.

Displaying Kubernetes Cluster Information


A Kubernetes cluster consists of a control plane node and a set of worker nodes that run containerized applications. Every cluster runs at least one worker node. You can view the Kubernetes cluster information by using the k8s-cluster-show command.


CLI (network-admin@switch1) k8s-cluster-show

k8s-cluster-show

Display Kubernetes cluster information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

control-plane-node control-plane-node-string

The name of the Kubernetes control plane node.

control-plane-node-ip control-plane-node-ip-string

The IP address of the control plane node.

controller-status controller-status-string

The status of the Kubernetes controller.

scheduler-status scheduler-status-string

The status of the Kubernetes scheduler.

action none|add|remove|update

The last action.


For example:


CLI (network-admin@switch1) > k8s-cluster-show


cluster-name control-plane-node control-plane-node-ip controller-stauts scheduler-status last-changed

------------ ------------------ --------------------- ----------------- ---------------- --------------

k8s01        server-test-93     192.168.10.10         healthy           healthy          07-12,23:25:21


Displaying Kubernetes Node Information


The control plane machine and the worker machines that are part of a Kubernetes cluster are called nodes. Nodes host the Pods that are the components of the application workload. You can display information on Kubernetes nodes by using the command:


CLI (network-admin@switch1) k8s-node-show

k8s-node-show

Display information on Kubernetes master and worker nodes.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

node node-string

The name of the Kubernetes node.

role role-string

The role assigned to the node.

status status-string

The status of the node.

ip ip-string

The IP address of the node.

pod-CIDR pod-CIDR-string

The Pod CIDR for the node.

labels labels-string

The Pod labels.

action none|add|remove|update

The last action performed.


CLI (network-admin@switch) k8s-node-show

cluster-name node            role          status       ip            pod-CIDR      last-changed

------------ --------------- ------------- ------------ ------------- ------------- --------------

k8s01        server-test-103 worker        KubeletReady 192.168.99.22 10.244.2.0/24 11-11,05:59:13

k8s01        server-test-93  control-plane KubeletReady 192.168.99.20 10.244.0.0/24 11-11,06:02:21

k8s01        server-test-94  worker        KubeletReady 192.168.99.21 10.244.1.0/24 11-11,05:58:36

Displaying Information on Kubernetes Pods

Pods are the smallest units that you can deploy using Kubernetes. Pods consist of one or more containers that are always scheduled together. Each Kubernetes pod has a unique IP address. To display information on Pods, use the command:


CLI (network-admin@switch1) k8s-pod-show

k8s-pod-show

Display Kubernetes Pod information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

name name-string

The name of the Daemonset.

namespace namespace-string

The namespace.

ip ip-string

The IP address of the Pod.

state state-string

The state of the Pod.

container-state container-state-string

The state of the Pod's container.

node node-string

The name of the Pod.

containers containers-string

The container images running on the Pod.

labels labels-string

The Pod labels.

action none|add|remove|update

The last action performed.

connections connections-string

The connections to fabric switches.

For example:


CLI (network-admin@switch1) > k8s-pod-show format name,namespace,ip,state,node,containers,labels


name                                      namespace            ip            state   node            containers          labels      ----------------------------------------- -------------------- ------------- ------- --------------- ------------------ ---------

kube-apiserver-server-test-93              kube-system          192.168.99.20 Running server-test-93  k8s.gcr.io/kube-... component:kube-apiserver,tier...

kube-proxy-d28w9                           kube-system          192.168.99.22 Running server-test-103 k8s.gcr.io/kube-... controller-revision-hash:84f9...

kube-flannel-ds-ltctg                      kube-system          192.168.99.22 Running server-test-103 quay.io/coreos/f... app:flannel,controller-revisi...

dashboard-metrics-scraper-79c5968bdc-hwpcg kubernetes-dashboard 10.244.1.35   Running server-test-94  kubernetesui/met... k8s-app:dashboard-metrics-scr...

kube-scheduler-server-test-93              kube-system          192.168.99.20 Running server-test-93  k8s.gcr.io/kube-... component:kube-scheduler,tier...

kube-controller-manager-server-test-93     kube-system          192.168.99.20 Running server-test-93  k8s.gcr.io/kube-... component:kube-controller-man...

gloo-7dc68dd65b-clz98                      gloo-system          10.244.2.36   Running server-test-103 quay.io/solo-io/... pod-template-hash:7dc68dd65b,...


Displaying Kubernetes Deployment Information


Deployments represent a set of multiple, identical Pods with no unique identities. A deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In this way, deployments help ensure that one or more instances of your application are available to serve user requests. 


To display Kubernetes deployment information, use the command:


CLI (network-admin@switch1) k8s-deployment-show

k8s-deployment-show

Display Kubernetes deployment information.

cluster-name cluster-name-string

The name of the Kubernetes node.

name name-string

The name of the deployment.

namespace namespace-string

The name of the namespace.

replicas replicas-number

The total replicas of Pod.

ready-replicas ready-replicas-number

The ready replicas of Pod.

selector selector-string

The Pod label in deployment.

action none|add|remove|update

The last action performed.


For example:


CLI (network-admin@switch1) > k8s-deployment-show


cluster-name name                      namespace            replicas ready-replicas selector                          last-changed

------------ ------------------------- -------------------- -------- -------------- --------------------------------- ------------

pluribus     my-app                    default              1        1              app:my-app                        01:07:38

pluribus     pingtest                  default              2        2              app:pingtest                      01:07:38

pluribus     kubernetes-dashboard      kubernetes-dashboard 1        1              k8s-app:kubernetes-dashboard      01:07:38

pluribus     dashboard-metrics-scraper kubernetes-dashboard 1        1              k8s-app:dashboard-metrics-scraper 01:07:39

pluribus     hello-world               default              5        5              run:load-balancer-example         01:07:39

pluribus     petstore                  default              1        1              app:petstore                      01:07:39

pluribus     discovery                 gloo-system          1        1              gloo:discovery                    01:07:39

pluribus     gloo                      gloo-system          1        1              gloo:gloo                         01:07:39

pluribus     ingress                   gloo-system          1        1              gloo:ingress                      01:07:39

pluribus     ingress-proxy             gloo-system          1        1              gloo:ingress-proxy                01:07:39

pluribus     coredns                   kube-system          2        2              k8s-app:kube-dns                  01:07:39



Displaying Information on Daemonsets

A DaemonSet manages groups of replicated Pods and ensures that all (or some) nodes run a copy of a Pod. DaemonSets adds Pods to nodes that are added to a cluster and removes the Pods from nodes which are removed from the cluster. Deleting a DaemonSet also deletes the Pods that it created.To display information on DaemonSets, use the command:


CLI (network-admin@switch1) k8s-daemonset-show

k8s-daemonset-show

Display Kubernetes Daemonset information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

name name-string

The name of the Daemonset.

namespace namespace-string

The name of the namespace.

desired-cnt desired-cnt-number

The desired number of Pods.

ready-cnt ready-cnt-number

The number of Pods ready.

selector selector-string

The Pod label selector for Daemonset.

last-changed date/time: yyyy-mm-ddTHH:mm:ss

The time when Daemonset was last changed.

action none|add|remove|update

The last action performed.

For example:

CLI (network-admin@switch) > k8s-daemonset-show

cluster-name name                  namespace      desired-cnt ready-cnt last-changed

------------ --------------------- -------------- ----------- --------- ------------

k8s01        speaker               metallb-system 6           6         23:24:49

k8s01        csi-cephfsplugin      rook-ceph      6           6         23:24:50

k8s01        csi-rbdplugin         rook-ceph      6           6         23:24:50

k8s01        calico-node           kube-system    6           6         23:24:50

k8s01        fluentd-elasticsearch kube-system    6           6         23:24:50

k8s01        kube-proxy            kube-system    6           6         23:24:50

k8s01        nodelocaldns          kube-system    6           6         23:24:50

k8s02        fluentd-elasticsearch kube-system    6           6         23:25:22

k8s02        kube-flannel-ds       kube-system    6           6         23:25:22

k8s02        kube-proxy            kube-system    6           6         23:25:22


Displaying Kubernetes Services


Kubernetes services provide a stable virtual IP addresses for a group of Pods. Unlike Pods, which are non-permanent resources that can be created and destroyed dynamically, services remain consistent. Services act as loadbalancers and are based on Linux IPTable or IPVS table. Use the k8s-service-show command to view the Kubernetes services information.


CLI (network-admin@switch1) k8s-service-show

k8s-service-show

Display Kubernetes service information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

name name-string

The name of the Daemonset.

namespace namespace-string

The name of the namespace.

type type-string

The service type.

service-ip service-ip-string

The IP address of the service.

external-ip external-ip-string

The external IP for services of type load balancer.

selector selector-string

The label selector for the service.

ports ports-string

The ports for the service.

target-ports target-ports-string

The ports targeted on Pods.

action none|add|remove|update

The last action performed.


For example:


CLI (network-admin@switch) > k8s-service-show layout vertical

cluster-name: k8s02

name:         kubernetes

namespace:    default

type:         ClusterIP

service-ip:   10.96.0.1

ports:        443/TCP

last-changed: 01-30,23:25:21

cluster-name: k8s02

name:         my-service

namespace:    default

type:         ClusterIP

service-ip:   10.98.20.142

ports:        71/TCP

last-changed: 01-30,23:25:21

cluster-name: k8s02

name:         kube-dns

namespace:    kube-system

type:         ClusterIP

service-ip:   10.96.0.10

ports:        53/UDP,53/TCP,9153/TCP

last-changed: 01-30,23:25:23

...

Displaying Kubernetes Endpoints


To view information about Pods targeted by a service (Pod IP addresses) in a Kubernetes cluster, use the command:


CLI (network-admin@switch) k8s-endpoint-show

k8s-endpoint-show

Display Kubernetes endpoint information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

name name-string

The name of the endpoint.

namespace namespace-string

Specify the namespace.

selector selector-string

Specify the label selector.

target-ports target-ports-string

Ports targeted on ports.

endpoints endpoints-string

The list of Pod IP addresses.

action none|add|remove|update

The last changed action.


CLI (network-admin@switch) > k8s-endpoint-show layout vertical

cluster-name: k8s02

name:         kubernetes

namespace:    default

target-ports: 6443

last-changed: 01-30,23:25:21

cluster-name: k8s02

name:         my-service

namespace:    default

target-ports: 9378

endpoints:    pending

last-changed: 01-30,23:25:21

cluster-name: k8s02

name:         kube-dns

namespace:    kube-system

target-ports: 53,53,9153

endpoints:    172.22.0.39,172.22.1.8

last-changed: 01-30,23:25:23

...

Displaying Kubernetes Ingress Information


Kubernetes Ingress manages the external access to services within a cluster. Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can give Services externally-reachable URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting. To display Ingress information, use the command:


CLI (network-admin@switch) k8s-ingress-show

k8s-ingress-show

Display Kubernetes Ingress information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

name name-string

The name of the endpoint.

namespace namespace-string

The namespace.

host host-string

Specify the hostname of the ingress.

path path-string

Path for the ingress rule.

service-name service-name-string

The backend service name.

service-port service-port-string

The backend service port.

resource-api-group resource-api-group-string

The backend resource API group

resource-kind resource-kind-string

The backend resource kind.

resource-name resource-name-string

The backend resource name.

action none|add|remove|update

The last performed action.

       

For example:


CLI (network-admin@switch) > k8s-ingress-show

cluster-name name                      namespace host             path service-name service-port last-changed

------------ ------------------------- --------- ---------------- ---- ------------ ------------ --------------

pluribus     petstore-ingress          default   gloo.example.com /.*  petstore     8080         07-12,23:25:22

pluribus     name-virtual-host-ingress default   foo.bar.com      /    service1     80           07-12,23:25:22

pluribus     name-virtual-host-ingress default   bar.foo.com      /    service2     80           07-12,23:25:22


Displaying Kubernetes Connectivity information.


the Kubernetes visibility feature enables you to locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity or performance issues. Use the k8s-connectivity-show command to view the connectivity information for a deployment, Daemonset, services or a Pod. Without any filter arguments, k8s-connectivity-show gives connectivity information for each Kubernetes node to the fabric.


CLI (network-admin@switch) > k8s-connectivity-show

k8s-connectivity-show

Display Kubernetes endpoint information.

namespace namespace-string

The namespace.

deployment deployment-string

The Kubernetes deployment.

daemonset daemonset-string

The Daemonset name.

service service-string

Kubernetes service name.

pod pod-string

The name of the Pod.

node node-string

The cluster node name.


CLI (network-admin@switch) > k8s-connectivity-show deployment petstore

cluster-name namespace deployment pod                      connections

------------ --------- ---------- ------------------------ ---------------------------------------

pluribus     default   petstore   petstore-9d499b76f-hhmg4 server-test-94:enp1s0f0 -- 43:switch-1

pluribus     default   petstore   petstore-9d499b76f-psv8w server-test-103:enp1s0f0 -- 16:switch-1

                                                           server-test-94:enp1s0f1 -- 44:switch-1

pluribus     default   petstore   petstore-9d499b76f-qj58s server-test-94:enp1s0f0 -- 43:switch-1

                                                           server-test-94:enp1s0f1 -- 44:switch-1

pluribus     default   petstore   petstore-9d499b76f-649g6 server-test-103:enp1s0f0 -- 16:switch-1


The output displays the connectivity of a deployment named petstore which has 4 replicas. The output shows how each replica (Pod) is connected to Arista fabric. For example, the first replica (petstore-9d499b76f-hhmg4) running on server-test-94 is connected through enp1s0f0 to port 43 of Arista switch switch-1.

Tracing Paths between Kubernetes Pods or Nodes


You can inspect all the paths that Kubernetes Pods or Kubernetes nodes use to communicate with each other across the fabric. Use the command k8s-trace-path and specify the source and destination to view all available shortest paths between them.


k8s-trace-paths

Display the available paths between two Kubernetes Pods or nodes.

src-pod

The name of the Kubernetes source Pod.

dst-pod

The name of the Kubernetes destination Pod.

src-node

The name of the Kubernetes source node.

dst-node

The name of the Kubernetes destination node.


The example below shows all the shortest paths from the Pod petstore running on node server-test-94 to Pod ingress-proxy running on node server-test-103. You can locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity issues.


CLI (network-admin@switch) > k8s-trace-paths src-pod petstore dst-pod ingress-proxy

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine2:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf4:swp6 -- enp1s0f0:server-test-103

server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp7 -- swp7:Leaf4:swp6 -- enp1s0f0:server-test-103

Analyzing the Impact on Applications during Host or Leaf Replacement


The Kubernetes visibility feature allows you to proactively determine the impact on containerized applications during switch outages or server replacements. You can view the impact on a Kubernetes deployment, Daemonset, service, or Pods in the occasion of a switch or host replacement by using the k8s-impact-show command. The impact can be none if there is no impact, partial if alternate paths are present, or full if the Kubernetes Pods lose all connectivity.


CLI (network-admin@switch) > k8s-impact-show

k8s-impact-show

Display the available paths between two Kubernetes Pods or nodes.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

host host-string

The host or node for which the impact needs to be checked.

switch fabric-node-name

The switch for which the impact needs to be checked.

namespace namespace-string

The name of the namespace.

deployment deployment-string

The Kubernetes deployment name.

daemonset daemonset-string

The Kubernetes Daemonset name.

service service-string

The Kubernetes service name.

pod pod-string

The Kubernetes Pod.

impact unknown|none|partial|full

The impact on the Pod when node or switch is replaced.


For example, you can view the impact on a Kubernetes deployment named petstore upon replacing a server named server-test-94 by using the command below:


CLI (network-admin@switch) > k8s-impact-show host server-test-94 deployment petstore

cluster-name namespace deployment pod                      impact  connections

------------ --------- ---------- ------------------------ ------  ---------------------------------------

pluribus     default   petstore   petstore-9d499b76f-psv8w none    server-test-103:enp1s0f0 -- 16:switch-1

pluribus     default   petstore   petstore-9d499b76f-hhmg4 full    server-test-94:enp1s0f0 -- 43:switch-1

                                                                      server-test-94:enp1s0f1 -- 44:switch-1

pluribus     default   petstore   petstore-9d499b76f-qj58s full    server-test-94:enp1s0f0 -- 43:switch-1

                                                                      server-test-94:enp1s0f0 -- 43:switch-1

pluribus     default   petstore   petstore-9d499b76f-649g6 none    server-test-103:enp1s0f0 -- 16:switch-1


Displaying Kubernetes Network Information


Use the k8s-network-show command to show the relation between Kubernetes networking components like ingress, services, and Pods.


k8s-network-show

Display Kubernetes Pod information.

cluster-name cluster-name-string

The name of the Kubernetes cluster.

Specify any of the following options:


namespace namespace-string

The name of the namespace.

ingress-name ingress-name-string

The IP address of the Pod.

host host-string

The state of the Pod.

type type-string

The name of the Pod.

external-ip external-ip-string

The external IP address for services of type load balancer.

node-port node-port-string

The service type.

service-name service-name-string

The name of the service.

selector selector-string

The service selector to identify Pods.

endpoints endpoints-string

The list of Pod IP addresses.


For example, this command displays the ingress to Pods to services mapping:


CLI (network-admin@switch) > k8s-network-show layout vertical format all

cluster-name: k8s01

namespace:    default

ingress-name: demo

host:         www.demo.io

type:         LoadBalancer

external-ip:  10.13.22.154

node-port:    31437

service-name: demo

selector:     app:demo

endpoints:    172.19.126.9,172.19.127.74

cluster-name: k8s01

namespace:    default

ingress-name: example-ingress

host:         example.com

type:         ExternalName

external-ip:  my.database.example.com

service-name: my-service

...

Viewing Historical Information on Kubernetes Cluster


NetVisor OS implements a time machine feature that eliminates the need to reproduce past events that impacts Kubernetes applications in order to troubleshoot availability issues. This feature enables you to track historical data in a Kubernetes cluster. You can view various historical attributes of a cluster since the creation of a Kubernetes connection and inspect the changes to the cluster in a specific time interval. 


Use the k8s-connection-modify command to enable history tracking and to specify the disk space for logging of historical data. For example:


CLI (network-admin@switch) > k8s-connection-modify cluster-name pluribus track-history disk-space 60M


The Kubernetes commands that allow history tracking have a set of statistics collection parameters that allows you to specify the start time, end time, duration of the data collection and so on. Listed below are the commands that you can view historical data for:


k8s-cluster-show

k8s-node-show

k8s-deployment-show

k8s-service-show

k8s-ingress-show

k8s-pod-show

k8s-connectivity-show

k8s-impact-show

k8s-cni-show


The data collection parameters of the commands above which enable you to view the history in different formats are:


start-time date/time: yyyy-mm-ddTHH:mm:ss        

The start time of statistics collection.

end-time date/time: yyyy-mm-ddTHH:mm:ss

The end time of statistics collection.

duration duration: #d#h#m#s

The duration of statistic collection.

interval duration: #d#h#m#s

The interval between statistic collection.

since-start 

Statistics collected since start time.

older-than duration: #d#h#m#s

Display statistics older than the specified duration.

within-last duration: #d#h#m#s

Display statistics within the last specified duration.


For example, to display the changes to the deployment named petstore in the last 30 minutes, use the command:


CLI (network-admin@switch) > k8s-deployment-show name petstore within-last 30m

cluster-name name     namespace replicas ready-replicas selector     last-changed action

------------ -------- --------- -------- -------------- ------------ ------------ ------

pluribus     petstore default   4        4              app:petstore 09:11:51     add

pluribus     petstore default   6        4              app:petstore 09:24:35     update

pluribus     petstore default   6        5              app:petstore 09:24:45     update

pluribus     petstore default   6        6              app:petstore 09:24:45     update

north
    keyboard_arrow_up
    keyboard_arrow_down
    description
    print
    feedback
    support
    business
    rss_feed
    south