Kubernetes CRDs and Operators are now widespread. Today most of the Kubernetes distributions come prepackaged with a number of Operators/CRDs. DevOps teams also write their own Operators to pre-package required automation for their workloads. An Operator adds Custom Resources to the cluster. Variety of teams using this purpose-built cluster can leverage available Custom Resources as they build their YAMLs for deploying the applications.
For these YAML developers Custom Resources present an opportunity as well as a challenge. Custom resources are an opportunity because a Custom Resource provides a declarative method for enabling some complex task. For example, ‘CassandraDatacenter’ Custom Resource from the DataStax Operator provides the ability to create a Cassandra cluster on Kubernetes. This is very powerful. Without such a Custom Resource and the Operator behind it, spinning up a Cassandra cluster on Kubernetes would require enormous amounts of effort and knowledge about operating Cassandra. While this is great, Custom Resources are also a challenge today because there is no good way for YAML developers to understand their usage or their semantics. For instance, when a Custom Resource is created, what all Kubernetes resources get created behind the scene? Or, what labels/annotations are important from the perspective of a Custom Resource and its Operator? Or, what is the CPU/Memory/Storage consumption of a Custom Resource instance?
In this post we present a generic technique that we have developed to address such questions about Kubernetes Custom Resources. You can use this technique on all required Operators on your cluster and significantly simplify usage of Custom Resources for your YAML developers.
CRD annotations
The basis for our technique is the observation that inter-resource relationships form one of the fundamental aspects for Kubernetes YAML automation. Some examples of inter-resource relationships between Kubernetes built-in resources are — a Pod is related to a PersistentVolumeClaim through Spec property; a Service is related to a Pod through labels; a Pod is related to a ReplicaSet through ownerReference.
It turns out that such relationships are also key aspects of Custom Resources coming from Operators. For instance, there are several Operators that look for a particular annotation or a label on other Kubernetes resources as a trigger for their operation. For Kubernetes YAML developers a big pain point today is not knowing what such relationships are for a given Custom Resource and how they need to be used as part of their YAML automation. Our technique addresses these questions.
The technique consists of a set of annotations that need to be given on the CRD (Custom Resource Definition) for making inter-resource relationships of a Custom Resource explicit. Here are those annotations:
resource/compositionresource/annotation-relationshipresource/label-relationshipresource/specproperty-relationshipresource/usage
These annotations are to be defined on an Operator’s Custom Resource Definition (CRD) YAML.
- The ‘composition’ annotation is used to list Kubernetes sub-resources that are created by the Operator when instantiating a Custom Resource of that CRD’s Kind.
- The three relationship annotations that follow define annotation-based, label-based, and spec property based relationships that are important for the Operator in regards to that specific Custom Resource.
- The ‘usage’ annotation is used to define how to use a Custom Resource — think of a ‘man page’ like information about a Custom Resource. The value of this annotation is a ConfigMap with the actual usage information.
Here is an example of defining the annotation-relationship on the ClusterIssuer CRD from the CertManager Operator.
resource/annotation-relationship:on:Ingress, key:cert-manager.io/cluster-issuer, value:INSTANCE.metadata.nameIt defines that CertManager looks for cert-manager.io/cluster-issuer annotation on Ingress resources. The value that it looks for is the name of a ClusterIssuer instance. When the CertManager Operator finds this annotation on an Ingress resource, it issues a new SSL certificate for that Ingress instance.
Here is another example of defining the ‘composition’ annotation on the MysqlCluster CRD from the PressLabs’ MySQL Operator.
resource/composition: StatefulSet, Service, ConfigMap, Secret, PodDisruptionBudgetThis identifies the set of resources that will be created by that Operator as part of instantiating the MysqlCluster Custom Resource instance.
Once these relationship dependencies are defined on the relevant CRDs on your cluster, it unlocks our KubePlus tooling. You can use variety of kubectl plugins from it to perform activities like:
- build resource topology graphs for all Kubernetes resources in a cluster.
- find out resource consumption of an individual Custom Resource instance or of an entire Kubernetes YAML workflow.
- create linkages between multiple Kubernetes YAMLs or Helm charts coming from different team members.
Below is a sample topology that can be constructed using KubePlus if ClusterIssuer CRD is annotated with the annotation mentioned above.
You can try this out with some of our sample examples available in KubePlus repository. This technique is generic and hence can be used with any set of Operators used on your Kubernetes cluster. We refer to this approach as Platform-as-Code and it focuses on simplifying adoption of Operators. Contact us if you are interested in using this approach and tooling with your purpose-built Operator stacks. We are also building a list of these annotations for commonly used Operators here and would like recommendations from the community on what you would like to see here.
