Kubernetes Resource relationship graphs for Application-level insights

In designing monitoring solutions for Kubernetes applications, DevOps teams are guided by following high-level questions:

  • If a worker node fails, which applications or services will get affected?
  • How much CPU/Memory/Storage is being consumed by the entire application?
  • How can we aggregate logs or events at application level from container/pod level?

An application is architected in Kubernetes with the help of Kubernetes Resources (built-in Resources like Pod, Service, Deployment and Custom Resources coming from various Operators). Application workflows are realized in Kubernetes YAMLs by establishing connections between Kubernetes Resources. These connections are based on various relationships such as labels, annotations, ownership, SpecProperty etc. Following figure shows an example of a Kubernetes application workflow with various connections between the resources.

Discovery of resource relationships in a workflow is fundamental to designing any monitoring solutions at application level.

Discovering these resource relationship graphs specific to application workflows is not simple today as there is no easy way to query such relationships. Additionally, when working with Custom Resources introduced by Operators, there is no standard way to define and understand the relationships that can be established with a Custom Resource.

We have developed KubePlus tooling to address this gap. It offers a set of kubectl plugins to discover resource relationships between various Kubernetes resources.

Let us understand how resource relationships can be discovered with a very simple application workflow for Wordpress application as shown below. It uses 4 Kubernetes resources (including one Custom Resource) and they are connected using various relationships as shown to realize a workflow.

Here is how KubePlus kubectl plugin can help you find out the relationships in this workflow.

Currently we support querying relationship graphs based on Service, Pod, or Custom Resource. Here is a look at the same graph when queried based on a Custom Resource instance.

Let’s look at a more complicated example now by using a MysqlCluster Custom Resource (coming from PressLabs MySQL Operator) for the database needs of the Wordpress Pod. MysqlCluster Custom Resource adds a number of relationships in the graph as shown below.

Here is the resource relationship graph for the above workflow from MysqlCluster resource.

You can find this example here.

CRD Annotations for Custom Resource relationships

In order to enable discovering relationships of a Custom Resource, we use special annotations on Custom Resource Definitions (CRDs) to capture the relationships assumptions made by the Operator developer. In the example above we have annotated the ClusterIssuer CRD with the following annotation.

And MysqlCluster CRD with the following annotation.

Check the KubePlus github repo to understand more about these annotations and how to write them on a CRD.

Application-level Insights

We have additional kubectl plugins in KubePlus that leverage these resource relationship graphs to provide application level insights. Here is an example:

As compared to the number of existing monitoring solutions, the key benefit of these kubectl plugins is that you don’t require any additional component to be running on your cluster and the support for relationships introduced by the Custom Resources.

Conclusion

Kubernetes resource relationship graphs can be used to help answer higher-level monitoring questions at application level. Try KubePlus and let us know how you would use the resources relationship graphs in your Kubernetes environments.

www.cloudark.io