In Kubernetes, Pods consume CPU and memory resources on the worker nodes in a cluster. In the case of Kubernetes Custom Resources, the Pods that are created as part of Custom Resource instances are the ones that consume physical resources of the cluster. This post discusses our ongoing work around tracking resource metrics for Custom Resources.
Today’s enterprise Kubernetes clusters typically use more than one Operators/CRDs to simplify building their application workflows. We have seen requirements from our customers for simplifying monitoring of these Operators and Custom Resources. Specifically, following Custom Resource related questions seem to be important to DevOps teams:
- How many sub-resources and specifically pods are created by a Custom Resource instance?
- Who (user or service account) has created this particular Custom Resource instance? What other resources are created by the same account identity?
- How much CPU / Memory is consumed by an instance of a Custom Resource?
- How much CPU / Memory is consumed by an account identity?
To support this requirement we are working on adding Custom Resource metrics tracking features in our KubePlus API add-on. It builds on top of the metrics functionality available from Kubernetes Metrics API Server and offers aggregations at various levels such as Custom Resource instance and account identity. Primary interface for using this functionality is kubectl. Following are the key metrics commands we are offering:
- kubectl metrics cr <Type> <Name> : Provides various metrics for custom resource instance (number of sub-resources, number of pods, number of containers, number of nodes on which the pods run, total CPU and Memory).
- kubectl metrics account <AccountName> : Provides various metrics for an account identity — user / service account. (number of custom resources, number of Deployments/StatefulSets/ReplicaSets/DaemonSets/ReplicationControllers, number of Pods, total CPU and Memory)
Here is the sample output of these commands.
$ kubectl metrics cr MysqlCluster cluster1 namespace1
----------------------------------------------------------
Creator Account Identity: devdattakulkarni@gmail.com
----------------------------------------------------------
Number of Sub-resources: 7
Number of Pods: 2
Number of Containers: 16
Number of Nodes: 1
Total CPU(cores): 84m
Total MEMORY(bytes): 302Mi
----------------------------------------------------------
$ kubectl metrics account devdattakulkarni@gmail.com
----------------------------------------------------------
Creator Account Identity: devdattakulkarni@gmail.com
----------------------------------------------------------
Number of Custom Resources: 3
Number of Deployments: 1
Number of StatefulSets: 0
Number of ReplicaSets: 0
Number of DaemonSets: 0
Number of ReplicationControllers: 0
Number of Pods: 0
Total CPU(cores): 288m
Total MEMORY(bytes): 524MiThis functionality can also be easily integrated with your external monitoring framework such as Prometheus.
We are looking for your feedback on how you would like to use this functionality in your environment and what additional metrics you would want to see in the context of Custom Resources. Let us know in the blog comments or on KubePlus Github repository.
