Kubernetes Resource relationship graphs for Application-level insights
In designing monitoring solutions for Kubernetes applications, DevOps teams are guided by following high-level questions:
- If a worker node fails, which applications or services will get affected?
- How much CPU/Memory/Storage is being consumed by the entire application?
- How can we aggregate logs or events at application level from container/pod level?
An application is architected in Kubernetes with the help of Kubernetes Resources (built-in Resources like Pod, Service, Deployment and Custom Resources coming from various Operators). Application workflows are realized in Kubernetes YAMLs by establishing connections between Kubernetes Resources. These connections are based on various relationships such as labels, annotations, ownership, SpecProperty etc. Following figure shows an example of a Kubernetes application workflow with various connections between the resources.
Discovery of resource relationships in a workflow is fundamental to designing any monitoring solutions at application level.
Discovering these resource relationship graphs specific to application workflows is not simple today as there is no easy way to query such relationships. Additionally, when working with Custom Resources introduced by Operators, there is no standard way to define and understand the relationships that can be established with a Custom Resource.
We have developed KubePlus tooling to address this gap. It offers a set of kubectl plugins to discover resource relationships between various Kubernetes resources.
Let us understand how resource relationships can be discovered with a very simple application workflow for Wordpress application as shown below. It uses 4 Kubernetes resources (including one Custom Resource) and they are connected using various relationships as shown to realize a workflow.
Here is how KubePlus kubectl plugin can help you find out the relationships in this workflow.
$ kubectl connections service wordpress namespace1::Final connections graph::
------ Branch 1 ------
Level:0 Service/wordpress
Level:1 Pod/wordpress-pod [related to Service/wordpress by:label]------ Branch 2 ------
Level:0 Service/wordpress
Level:1 Ingress/wordpress-ingress [related to Service/wordpress by:specproperty]
Level:2 ClusterIssuer/wordpress-stack [related to Ingress/wordpress-ingress by:annotation]
Currently we support querying relationship graphs based on Service, Pod, or Custom Resource. Here is a look at the same graph when queried based on a Custom Resource instance.
$ kubectl connections cr ClusterIssuer wordpress-stack namespace1::Final connections graph::
------ Branch 1 ------
Level:0 ClusterIssuer/wordpress-stack
Level:1 Ingress/wordpress-ingress [related to ClusterIssuer/wordpress-stack by:annotation] Level:2 Service/wordpress [related to Ingress/wordpress-ingress by:specproperty]
Level:3 Pod/wordpress-pod [related to Service/wordpress by:label]
Let’s look at a more complicated example now by using a MysqlCluster Custom Resource (coming from PressLabs MySQL Operator) for the database needs of the Wordpress Pod. MysqlCluster Custom Resource adds a number of relationships in the graph as shown below.
Here is the resource relationship graph for the above workflow from MysqlCluster resource.
$ kubectl connections cr MysqlCluster cluster1 namespace1------ Branch 1 ------
Level:0 MysqlCluster/cluster1
Level:1 StatefulSet/cluster1-mysql [related to MysqlCluster/cluster1 by:owner reference]
Level:2 Pod/cluster1-mysql-0 [related to StatefulSet/cluster1-mysql by:owner reference]
Level:3 Service/cluster1-mysql-nodes [related to Pod/cluster1-mysql-0 by:envvariable]
Level:3 Service/cluster1-mysql [related to Pod/cluster1-mysql-0 by:label]
Level:3 Service/cluster1-mysql-master [related to Pod/cluster1-mysql-0 by:label]
Level:3 Service/cluster1-mysql-nodes [related to Pod/cluster1-mysql-0 by:label]
Level:4 Pod/wordpress-pod [related to Service/cluster1-mysql-master by:envvariable]
Level:5 Service/wordpress [related to Pod/wordpress-pod by:label]
Level:6 Ingress/wordpress-ingress [related to Service/wordpress by:specproperty]------ Branch 2 ------
Level:0 MysqlCluster/cluster1
Level:1 Service/cluster1-mysql [related to MysqlCluster/cluster1 by:owner reference]------ Branch 3 ------
Level:0 MysqlCluster/cluster1
Level:1 Service/cluster1-mysql-master [related to MysqlCluster/cluster1 by:owner reference]------ Branch 4 ------
Level:0 MysqlCluster/cluster1
Level:1 Service/cluster1-mysql-nodes [related to MysqlCluster/cluster1 by:owner reference]------ Branch 5 ------
Level:0 MysqlCluster/cluster1
Level:1 ConfigMap/cluster1-mysql [related to MysqlCluster/cluster1 by:owner reference]
You can find this example here.
CRD Annotations for Custom Resource relationships
In order to enable discovering relationships of a Custom Resource, we use special annotations on Custom Resource Definitions (CRDs) to capture the relationships assumptions made by the Operator developer. In the example above we have annotated the ClusterIssuer CRD with the following annotation.
resource/annotation-relationship: on:Ingress, key:cert-manager.io/cluster-issuer, value:INSTANCE.metadata.nameAnd MysqlCluster CRD with the following annotation.
resource/composition: StatefulSet, Service, ConfigMap, Secret, PodDisruptionBudgetCheck the KubePlus github repo to understand more about these annotations and how to write them on a CRD.
Application-level Insights
We have additional kubectl plugins in KubePlus that leverage these resource relationship graphs to provide application level insights. Here is an example:
$ kubectl metrics service wordpress namespace1
----------------------------------------------------------
Kubernetes Resources consumed:
Number of Pods: 2
Number of Containers: 7
Number of Nodes: 1Underlying Physical Resources consumed:
Total CPU(cores): 25m
Total MEMORY(bytes): 307Mi
Total Storage(bytes): 21Gi
----------------------------------------------------------
As compared to the number of existing monitoring solutions, the key benefit of these kubectl plugins is that you don’t require any additional component to be running on your cluster and the support for relationships introduced by the Custom Resources.
Conclusion
Kubernetes resource relationship graphs can be used to help answer higher-level monitoring questions at application level. Try KubePlus and let us know how you would use the resources relationship graphs in your Kubernetes environments.
