Configuring Kubernetes Visibility
The Kubernetes client in Netvisor ONE connects to the Kubernetes API server for real time notifications of events and changes. To create a connection between the Kubernetes client in Netvisor ONE and the Kubernetes API server, use the command:
CLI (network-admin@switch1) k8s-connection-create
|
k8s-connection-create |
Create a new connection to the Kubernetes API server. |
|
cluster-name cluster-name-string |
Specify a name for the Kubernetes cluster. |
|
rest-user user-name |
Specify the username for vREST access. |
|
kube-config kube-config-string |
Specify the absolute path to the kubeconfig file. The default location is /root/.kube/config. |
|
password |
Specify the vREST password. |
|
enable|disable |
Enable or disable the connection to Kubernetes API server. |
|
track-history|no-track-history |
Enable or disable Kubernetes cluster history tracking. |
|
disk-space disk-space-number |
Specify the disk-space allocated for history (including rotated log files). |
For example:
CLI (network-admin@switch) > k8s-connection-create cluster-name k8s01 kube-config /root/.kube/config
rest-user network-admin enable
rest-user password :
confirm rest-user password:
Note:
- Netvisor ONE supports Kubernetes versions 1.19, 1.21, and 1.22.
- The Kubernetes client uses REST API to interact with Netvisor ONE. Thus, you must enable web services on the data interface or the management interface of the switch by using the admin-service-modify command.
- You must enable LLDP on all the fabric nodes before configuring Kubernetes visibility. All Kubernetes servers must be visible in the lldp-show outputs of the fabric switches.
- To establish a connection with the Kubernetes API server succesfully, you must first copy the Kubernetes configuration file called kubeconfig from the Kubernetes control plane node to the switch. You can then specify the path to the file in the k8s-connection-create command.
- The Kubernetes API server must be reachable from linux global routing table and the API server's fully qualified domain name (FQDN) must be resolvable via nameserver.
Issue the k8s-connection-show command to display the Kubernetes connection information.
CLI (network-admin@switch1) k8s-connection-show
|
k8s-connection-show |
Display the connection to the Kubernetes API server. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
kube-config kube-config-string |
The absolute path to the kube config file. |
|
rest-user user-name |
The username for vRest access on switch. |
|
enable|disable |
Status of the connection to Kubernetes API server. |
|
track-history|no-track-history |
Enable or disable Kubernetes cluster history tracking. |
|
disk-space disk-space-number |
The disk-space allocated for history (including rotated log files). |
|
create-vport|no-create-vport |
Enable or disable vPort creation for Pod traffic. |
|
overlay-vxlan-analytics|no-overlay-vxlan-analytics |
Enable or disable Pod traffic analytics. |
|
location location-number |
Location where the connection is running. |
|
connection-error connection-error-string |
Kubernetes connection error. |
|
k8s-vlan vlan-id |
The VLAN ID used by Kubernetes for Pod vPort creation |
CLI (network-admin@switch1) > k8s-connection-show format all layout vertical
cluster-name: k8s01
kube-config: /root/.kube/config
rest-user: network-admin
enable: yes
track-history: yes
disk-space: 50M
create-vport: yes
overlay-vxlan-analytics: no
location: 1262261009
Use the k8s-connection-delete and the k8s-connection-modify commands to delete or modify the configuration.
Displaying Kubernetes Cluster Information
A Kubernetes cluster consists of a control plane node and a set of worker nodes that run containerized applications. Every cluster runs at least one worker node. You can view the Kubernetes cluster information by using the k8s-cluster-show command.
CLI (network-admin@switch1) k8s-cluster-show
|
k8s-cluster-show |
Display Kubernetes cluster information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
control-plane-node control-plane-node-string |
The name of the Kubernetes control plane node. |
|
control-plane-node-ip control-plane-node-ip-string |
The IP address of the control plane node. |
|
controller-status controller-status-string |
The status of the Kubernetes controller. |
|
scheduler-status scheduler-status-string |
The status of the Kubernetes scheduler. |
|
action none|add|remove|update |
The last action. |
For example:
CLI (network-admin@switch1) > k8s-cluster-show
cluster-name control-plane-node control-plane-node-ip controller-stauts scheduler-status last-changed
------------ ------------------ --------------------- ----------------- ---------------- --------------
k8s01 server-test-93 192.168.10.10 healthy healthy 07-12,23:25:21
Displaying Kubernetes Node Information
The control plane machine and the worker machines that are part of a Kubernetes cluster are called nodes. Nodes host the Pods that are the components of the application workload. You can display information on Kubernetes nodes by using the command:
CLI (network-admin@switch1) k8s-node-show
|
k8s-node-show |
Display information on Kubernetes master and worker nodes. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
node node-string |
The name of the Kubernetes node. |
|
role role-string |
The role assigned to the node. |
|
status status-string |
The status of the node. |
|
ip ip-string |
The IP address of the node. |
|
pod-CIDR pod-CIDR-string |
The Pod CIDR for the node. |
|
labels labels-string |
The Pod labels. |
|
action none|add|remove|update |
The last action performed. |
CLI (network-admin@switch) k8s-node-show
cluster-name node role status ip pod-CIDR last-changed
------------ --------------- ------------- ------------ ------------- ------------- --------------
k8s01 server-test-103 worker KubeletReady 192.168.99.22 10.244.2.0/24 11-11,05:59:13
k8s01 server-test-93 control-plane KubeletReady 192.168.99.20 10.244.0.0/24 11-11,06:02:21
k8s01 server-test-94 worker KubeletReady 192.168.99.21 10.244.1.0/24 11-11,05:58:36
Displaying Information on Kubernetes Pods
Pods are the smallest units that you can deploy using Kubernetes. Pods consist of one or more containers that are always scheduled together. Each Kubernetes pod has a unique IP address. To display information on Pods, use the command:
CLI (network-admin@switch1) k8s-pod-show
|
k8s-pod-show |
Display Kubernetes Pod information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
name name-string |
The name of the Daemonset. |
|
namespace namespace-string |
The namespace. |
|
ip ip-string |
The IP address of the Pod. |
|
state state-string |
The state of the Pod. |
|
container-state container-state-string |
The state of the Pod's container. |
|
node node-string |
The name of the Pod. |
|
containers containers-string |
The container images running on the Pod. |
|
labels labels-string |
The Pod labels. |
|
action none|add|remove|update |
The last action performed. |
|
connections connections-string |
The connections to fabric switches. |
For example:
CLI (network-admin@switch1) > k8s-pod-show format name,namespace,ip,state,node,containers,labels
name namespace ip state node containers labels
------------------------------------------ -------------------- ------------- ------- --------------- ------------------- --------------------------------
kube-apiserver-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-apiserver,tier...
kube-proxy-d28w9 kube-system 192.168.99.22 Running server-test-103 k8s.gcr.io/kube-... controller-revision-hash:84f9...
kube-flannel-ds-ltctg kube-system 192.168.99.22 Running server-test-103 quay.io/coreos/f... app:flannel,controller-revisi...
dashboard-metrics-scraper-79c5968bdc-hwpcg kubernetes-dashboard 10.244.1.35 Running server-test-94 kubernetesui/met... k8s-app:dashboard-metrics-scr...
kube-scheduler-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-scheduler,tier...
kube-controller-manager-server-test-93 kube-system 192.168.99.20 Running server-test-93 k8s.gcr.io/kube-... component:kube-controller-man...
gloo-7dc68dd65b-clz98 gloo-system 10.244.2.36 Running server-test-103 quay.io/solo-io/... pod-template-hash:7dc68dd65b,...
Displaying Kubernetes Deployment Information
Deployments represent a set of multiple, identical Pods with no unique identities. A deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In this way, deployments help ensure that one or more instances of your application are available to serve user requests. To display Kubernetes deployment information, use the command:
CLI (network-admin@switch1) k8s-deployment-show
|
k8s-deployment-show |
Display Kubernetes deployment information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes node. |
|
name name-string |
The name of the deployment. |
|
namespace namespace-string |
The name of the namespace. |
|
replicas replicas-number |
The total replicas of Pod. |
|
ready-replicas ready-replicas-number |
The ready replicas of Pod. |
|
selector selector-string |
The Pod label in deployment. |
|
action none|add|remove|update |
The last action performed. |
For example:
CLI (network-admin@switch1) > k8s-deployment-show
cluster-name name namespace replicas ready-replicas selector last-changed
------------ ------------------------- -------------------- -------- -------------- --------------------------------- ------------
pluribus my-app default 1 1 app:my-app 01:07:38
pluribus pingtest default 2 2 app:pingtest 01:07:38
pluribus kubernetes-dashboard kubernetes-dashboard 1 1 k8s-app:kubernetes-dashboard 01:07:38
pluribus dashboard-metrics-scraper kubernetes-dashboard 1 1 k8s-app:dashboard-metrics-scraper 01:07:39
pluribus hello-world default 5 5 run:load-balancer-example 01:07:39
pluribus petstore default 1 1 app:petstore 01:07:39
pluribus discovery gloo-system 1 1 gloo:discovery 01:07:39
pluribus gloo gloo-system 1 1 gloo:gloo 01:07:39
pluribus ingress gloo-system 1 1 gloo:ingress 01:07:39
pluribus ingress-proxy gloo-system 1 1 gloo:ingress-proxy 01:07:39
pluribus coredns kube-system 2 2 k8s-app:kube-dns 01:07:39
Displaying Information on Daemonsets
A DaemonSet manages groups of replicated Pods and ensures that all (or some) nodes run a copy of a Pod. DaemonSets adds Pods to nodes that are added to a cluster and removes the Pods from nodes which are removed from the cluster. Deleting a DaemonSet also deletes the Pods that it created.To display information on DaemonSets, use the command:
CLI (network-admin@switch1) k8s-daemonset-show
|
k8s-daemonset-show |
Display Kubernetes Daemonset information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
name name-string |
The name of the Daemonset. |
|
namespace namespace-string |
The name of the namespace. |
|
desired-cnt desired-cnt-number |
The desired number of Pods. |
|
ready-cnt ready-cnt-number |
The number of Pods ready. |
|
selector selector-string |
The Pod label selector for Daemonset. |
|
last-changed date/time: yyyy-mm-ddTHH:mm:ss |
The time when Daemonset was last changed. |
|
action none|add|remove|update |
The last action performed. |
For example:
CLI (network-admin@switch) > k8s-daemonset-show
cluster-name name namespace desired-cnt ready-cnt last-changed
------------ --------------------- -------------- ----------- --------- ------------
k8s01 speaker metallb-system 6 6 23:24:49
k8s01 csi-cephfsplugin rook-ceph 6 6 23:24:50
k8s01 csi-rbdplugin rook-ceph 6 6 23:24:50
k8s01 calico-node kube-system 6 6 23:24:50
k8s01 fluentd-elasticsearch kube-system 6 6 23:24:50
k8s01 kube-proxy kube-system 6 6 23:24:50
k8s01 nodelocaldns kube-system 6 6 23:24:50
k8s02 fluentd-elasticsearch kube-system 6 6 23:25:22
k8s02 kube-flannel-ds kube-system 6 6 23:25:22
k8s02 kube-proxy kube-system 6 6 23:25:22
Displaying Kubernetes Services
Kubernetes services provide a stable virtual IP addresses for a group of Pods. Unlike Pods, which are non-permanent resources that can be created and destroyed dynamically, services remain consistent. Services act as loadbalancers and are based on Linux IPTable or IPVS table. Use the k8s-service-show command to view the Kubernetes services information.
CLI (network-admin@switch1) k8s-service-show
|
k8s-service-show |
Display Kubernetes service information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
name name-string |
The name of the Daemonset. |
|
namespace namespace-string |
The name of the namespace. |
|
type type-string |
The service type. |
|
service-ip service-ip-string |
The IP address of the service. |
|
external-ip external-ip-string |
The external IP for services of type load balancer. |
|
selector selector-string |
The label selector for the service. |
|
ports ports-string |
The ports for the service. |
|
target-ports target-ports-string |
The ports targeted on Pods. |
|
action none|add|remove|update |
The last action performed. |
For example:
CLI (network-admin@switch) > k8s-service-show layout vertical
cluster-name: k8s02
name: kubernetes
namespace: default
type: ClusterIP
service-ip: 10.96.0.1
ports: 443/TCP
last-changed: 01-30,23:25:21
cluster-name: k8s02
name: my-service
namespace: default
type: ClusterIP
service-ip: 10.98.20.142
ports: 71/TCP
last-changed: 01-30,23:25:21
cluster-name: k8s02
name: kube-dns
namespace: kube-system
type: ClusterIP
service-ip: 10.96.0.10
ports: 53/UDP,53/TCP,9153/TCP
last-changed: 01-30,23:25:23
...
Displaying Kubernetes Endpoints
To view information about Pods targeted by a service (Pod IP addresses) in a Kubernetes cluster, use the command:
CLI (network-admin@switch) k8s-endpoint-show
|
k8s-endpoint-show |
Display Kubernetes endpoint information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
name name-string |
The name of the endpoint. |
|
namespace namespace-string |
Specify the namespace. |
|
selector selector-string |
Specify the label selector. |
|
target-ports target-ports-string |
Ports targeted on ports. |
|
endpoints endpoints-string |
The list of Pod IP addresses. |
|
action none|add|remove|update |
The last changed action. |
CLI (network-admin@switch) > k8s-endpoint-show layout vertical
cluster-name: k8s02
name: kubernetes
namespace: default
target-ports: 6443
last-changed: 01-30,23:25:21
cluster-name: k8s02
name: my-service
namespace: default
target-ports: 9378
endpoints: pending
last-changed: 01-30,23:25:21
cluster-name: k8s02
name: kube-dns
namespace: kube-system
target-ports: 53,53,9153
endpoints: 172.22.0.39,172.22.1.8
last-changed: 01-30,23:25:23
...
Displaying Kubernetes Ingress Information
Kubernetes Ingress manages the external access to services within a cluster. Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. An Ingress can give Services externally-reachable URLs, load balance traffic, terminate SSL/TLS, and offer name-based virtual hosting. To display Ingress information, use the command:
CLI (network-admin@switch) k8s-ingress-show
|
k8s-ingress-show |
Display Kubernetes Ingress information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
name name-string |
The name of the endpoint. |
|
namespace namespace-string |
The namespace. |
|
host host-string |
Specify the hostname of the ingress. |
|
path path-string |
Path for the ingress rule. |
|
service-name service-name-string |
The backend service name. |
|
service-port service-port-string |
The backend service port. |
|
resource-api-group resource-api-group-string |
The backend resource API group |
|
resource-kind resource-kind-string |
The backend resource kind. |
|
resource-name resource-name-string |
The backend resource name. |
|
action none|add|remove|update |
The last performed action. |
For example:
CLI (network-admin@switch) > k8s-ingress-show
cluster-name name namespace host path service-name service-port last-changed
------------ ------------------------- --------- ---------------- ---- ------------ ------------ --------------
pluribus petstore-ingress default gloo.example.com /.* petstore 8080 07-12,23:25:22
pluribus name-virtual-host-ingress default foo.bar.com / service1 80 07-12,23:25:22
pluribus name-virtual-host-ingress default bar.foo.com / service2 80 07-12,23:25:22
Displaying Kubernetes Connectivity information.
the Kubernetes visibility feature enables you to locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity or performance issues. Use the k8s-connectivity-show command to view the connectivity information for a deployment, Daemonset, services or a Pod. Without any filter arguments, k8s-connectivity-show gives connectivity information for each Kubernetes node to the fabric.
CLI (network-admin@switch) > k8s-connectivity-show
|
k8s-connectivity-show |
Display Kubernetes endpoint information. |
|
namespace namespace-string |
The namespace. |
|
deployment deployment-string |
The Kubernetes deployment. |
|
daemonset daemonset-string |
The Daemonset name. |
|
service service-string |
Kubernetes service name. |
|
pod pod-string |
The name of the Pod. |
|
node node-string |
The cluster node name. |
CLI (network-admin@switch) > k8s-connectivity-show deployment petstore
cluster-name namespace deployment pod connections
------------ --------- ---------- ------------------------ ---------------------------------------
pluribus default petstore petstore-9d499b76f-hhmg4 server-test-94:enp1s0f0 -- 43:switch-1
pluribus default petstore petstore-9d499b76f-psv8w server-test-103:enp1s0f0 -- 16:switch-1
server-test-94:enp1s0f1 -- 44:switch-1
pluribus default petstore petstore-9d499b76f-qj58s server-test-94:enp1s0f0 -- 43:switch-1
server-test-94:enp1s0f1 -- 44:switch-1
pluribus default petstore petstore-9d499b76f-649g6 server-test-103:enp1s0f0 -- 16:switch-1
The output displays the connectivity of a deployment named petstore which has 4 replicas. The output shows how each replica (Pod) is connected to Pluribus fabric. For example, the first replica (petstore-9d499b76f-hhmg4) running on server-test-94 is connected through enp1s0f0 to port 43 of Pluribus switch switch-1.
Tracing Paths between Kubernetes Pods or Nodes
You can inspect all the paths that Kubernetes Pods or Kubernetes nodes use to communicate with each other across the fabric. Use the command k8s-trace-path and specify the source and destination to view all available shortest paths between them.
|
k8s-trace-paths |
Display the available paths between two Kubernetes Pods or nodes. |
|
src-pod |
The name of the Kubernetes source Pod. |
|
dst-pod |
The name of the Kubernetes destination Pod. |
|
src-node |
The name of the Kubernetes source node. |
|
dst-node |
The name of the Kubernetes destination node. |
The example below shows all the shortest paths from the Pod petstore running on node server-test-94 to Pod ingress-proxy running on node server-test-103. You can locate where Kubernetes Pods for specific applications are connected to the fabric to resolve any connectivity issues.
CLI (network-admin@switch) > k8s-trace-paths src-pod petstore dst-pod ingress-proxy
server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103
server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine2:swp4 -- swp5:Leaf3:swp6 -- enp1s0f0:server-test-103
server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf4:swp6 -- enp1s0f0:server-test-103
server-test-94:enp1s0f0 -- swp1:Leaf1:swp2 -- swp3:spine1:swp4 -- swp5:Leaf3:swp7 -- swp7:Leaf4:swp6 -- enp1s0f0:server-test-103
Analyzing the Impact on Applications during Host or Leaf Replacement
The Kubernetes visibility feature allows you to proactively determine the impact on containerized applications during switch outages or server replacements. You can view the impact on a Kubernetes deployment, Daemonset, service, or Pods in the occasion of a switch or host replacement by using the k8s-impact-show command. The impact can be none if there is no impact, partial if alternate paths are present, or full if the Kubernetes Pods lose all connectivity.
CLI (network-admin@switch) > k8s-impact-show
|
k8s-impact-show |
Display the available paths between two Kubernetes Pods or nodes. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
host host-string |
The host or node for which the impact needs to be checked. |
|
switch fabric-node-name |
The switch for which the impact needs to be checked. |
|
namespace namespace-string |
The name of the namespace. |
|
deployment deployment-string |
The Kubernetes deployment name. |
|
daemonset daemonset-string |
The Kubernetes Daemonset name. |
|
service service-string |
The Kubernetes service name. |
|
pod pod-string |
The Kubernetes Pod. |
|
impact unknown|none|partial|full |
The impact on the Pod when node or switch is replaced. |
For example, you can view the impact on a Kubernetes deployment named petstore upon replacing a server named server-test-94 by using the command below:
CLI (network-admin@switch) > k8s-impact-show host server-test-94 deployment petstore
cluster-name namespace deployment pod impact connections
------------ --------- ---------- ------------------------ ------ ---------------------------------------
pluribus default petstore petstore-9d499b76f-psv8w none server-test-103:enp1s0f0 -- 16:switch-1
pluribus default petstore petstore-9d499b76f-hhmg4 full server-test-94:enp1s0f0 -- 43:switch-1
server-test-94:enp1s0f1 -- 44:switch-1
pluribus default petstore petstore-9d499b76f-qj58s full server-test-94:enp1s0f0 -- 43:switch-1
server-test-94:enp1s0f0 -- 43:switch-1
pluribus default petstore petstore-9d499b76f-649g6 none server-test-103:enp1s0f0 -- 16:switch-1
Displaying Kubernetes Network Information
Use the k8s-network-show command to show the relation between Kubernetes networking components like ingress, services, and Pods.
|
k8s-network-show |
Display Kubernetes Pod information. |
|
cluster-name cluster-name-string |
The name of the Kubernetes cluster. |
|
Specify any of the following options: |
|
|
namespace namespace-string |
The name of the namespace. |
|
ingress-name ingress-name-string |
The IP address of the Pod. |
|
host host-string |
The state of the Pod. |
|
type type-string |
The name of the Pod. |
|
external-ip external-ip-string |
The external IP address for services of type load balancer. |
|
node-port node-port-string |
The service type. |
|
service-name service-name-string |
The name of the service. |
|
selector selector-string |
The service selector to identify Pods. |
|
endpoints endpoints-string |
The list of Pod IP addresses. |
For example, this command displays the ingress to Pods to services mapping:
CLI (network-admin@switch) > k8s-network-show layout vertical format all
cluster-name: k8s01
namespace: default
ingress-name: demo
host: www.demo.io
type: LoadBalancer
external-ip: 10.13.22.154
node-port: 31437
service-name: demo
selector: app:demo
endpoints: 172.19.126.9,172.19.127.74
cluster-name: k8s01
namespace: default
ingress-name: example-ingress
host: example.com
type: ExternalName
external-ip: my.database.example.com
service-name: my-service
...
Viewing Historical Information on Kubernetes Cluster
Netvisor ONE implements a time machine feature that eliminates the need to reproduce past events that impacts Kubernetes applications in order to troubleshoot availability issues. This feature enables you to track historical data in a Kubernetes cluster. You can view various historical attributes of a cluster since the creation of a Kubernetes connection and inspect the changes to the cluster in a specific time interval.
Use the k8s-connection-modify command to enable history tracking and to specify the disk space for logging of historical data. For example:
CLI (network-admin@switch) > k8s-connection-modify cluster-name pluribus track-history disk-space 60M
The Kubernetes commands that allow history tracking have a set of statistics collection parameters that allows you to specify the start time, end time, duration of the data collection and so on. Listed below are the commands that you can view historical data for:
k8s-cluster-show
k8s-node-show
k8s-deployment-show
k8s-service-show
k8s-ingress-show
k8s-pod-show
k8s-connectivity-show
k8s-impact-show
k8s-cni-show
The data collection parameters of the commands above which enable you to view the history in different formats are:
|
start-time date/time: yyyy-mm-ddTHH:mm:ss |
The start time of statistics collection. |
|
end-time date/time: yyyy-mm-ddTHH:mm:ss |
The end time of statistics collection. |
|
duration duration: #d#h#m#s |
The duration of statistic collection. |
|
interval duration: #d#h#m#s |
The interval between statistic collection. |
|
since-start |
Statistics collected since start time. |
|
older-than duration: #d#h#m#s |
Display statistics older than the specified duration. |
|
within-last duration: #d#h#m#s |
Display statistics within the last specified duration. |
For example, to display the changes to the deployment named petstore in the last 30 minutes, use the command:
CLI (network-admin@switch) > k8s-deployment-show name petstore within-last 30m
cluster-name name namespace replicas ready-replicas selector last-changed action
------------ -------- --------- -------- -------------- ------------ ------------ ------
pluribus petstore default 4 4 app:petstore 09:11:51 add
pluribus petstore default 6 4 app:petstore 09:24:35 update
pluribus petstore default 6 5 app:petstore 09:24:45 update
pluribus petstore default 6 6 app:petstore 09:24:45 update