We are happy to share with you KubePlus Cluster Add-on. It consists of a Kubernetes Aggregated API Server that you can run in your Kubernetes cluster to enable discovery of various kinds of static and runtime information about Kubernetes Operators and Custom Resources.
Kubernetes Custom Resource Definitions (CRDs), popularly known as Operators, enable running third-party software directly on Kubernetes. Today, number of Operators are being built for various kinds of softwares such as databases, queues, loggers, ssl certificate management, volume backup/restore, etc. Operators typically define one or more Kubernetes Custom Resources abstracting the workflow actions to be performed on the software that it is managing.
We are seeing enterprises building their platform automation on Kubernetes using multiple Operators. Operators are selected based on the desired platform elements in their platform stacks (e.g. MySQL, Spark, Cassandra, Cert Manager, etc.). The corresponding ensemble of Kubernetes YAMLs, consisting of native and Custom Resources, can be referred to as Platform-as-Code, since the Custom Resources are nothing but the platform level abstractions that enable declarative description of the platform stack.
While helping customers build platforms using multiple Operators, we observe following common challenges:
- CLI overload: Some of the Operators introduce new CLIs. Usability becomes an issue when end users have to learn multiple CLIs to use more than one Operators in a cluster.
- Discoverability: When using multiple Operators, it becomes a challenge for developers to discover the capabilities of the various Custom Resources available in the cluster. Finding answers to question such as, how to use different Custom Resources, what is the meaning of various Custom Resource Spec attributes, what values should be assigned to them, etc. is not straightforward.
- Interoperability: Another challenge is how different Custom Resources work together. For instance, a Moodle Custom Resource (Moodle is a popular open source e-learning software) and a Backup Custom Resource both work with Volumes. How to ensure that both these Custom Resources are using the same Volume in their operations so that we can take backups of Moodle’s volume correctly?
KubePlus Cluster Add-on
Our motivation to build KubePlus Cluster Add-on was to solve above challenges when using multiple Operators together. Specifically, we wanted to build a Kubernetes-native solution that will provide an easy way to discover static and runtime information about Custom Resources directly through ‘kubectl’. Equipped with this information, application developers can build their platform stacks composing different Custom Resources together.
Static information about Custom Resources consists of: a) how-to-use guides, b) any code level assumptions made by an Operator, c) OpenAPI Spec definitions etc.. Runtime information consists of: a) identification of Kubernetes’s native resources that are created as part of instantiating a Custom Resource instance, b) history of declarative actions performed on Custom Resource instances etc.
We achieve these goals through annotations, ConfigMaps, and custom endpoints on an aggregated API server. Annotations and ConfigMaps are used by Operator developers to define required information about Custom Resources. Custom endpoints on the aggregated API server are used by application developers to discover information about Custom Resources.
Platform-as-Code annotations:
KubePlus currently defines following annotations:
- platform-as-code/usage
- platform-as-code/constants
- platform-as-code/openapispec
- platform-as-code/composition
The annotations need to be added on your CRD YAML. The values for ‘usage’, ‘constants’, ‘openapispec’ annotations are names of ConfigMaps that store the corresponding data. Defining the annotations and creating the ConfigMaps is the responsibility of Operator developer. The ‘composition’ annotation is used to define Kubernetes’s native resources that are created as part of instantiating a Custom Resource instance. KubePlus uses the values in this annotation along with OwnerReferences, to build dynamic composition tree of Kubernetes’s native resources that are created as part of instantiating a Custom Resource instance. As an example, annotations on Moodle Custom Resource Definition (CRD) are shown below:
The Moodle Helm Chart is available here.
Platform-as-Code endpoints:
KubePlus currently exposes following endpoints:
- ‘man’ — find out ‘man page’ like information about Custom Resources. It essentially exposes the information packaged in ‘usage’ and ‘constants’ annotations on a CRD.
- ‘explain’ — find out openapispec information about Custom Resources.
- ‘composition’ — find out run time composition for an instance of a Custom Resource. We use listing of native resources available in ‘composition’ annotation and Custom Resource OwnerReferences to build this tree.
These endpoints are implemented using Kubernetes’s aggregated API Server.
Here are examples of how these endpoints can be used by application developers to discover information about Custom Resources.
$ kubectl get — raw “/apis/platform-as-code/v1/man?kind=Moodle”
$ kubectl get — raw “/apis/platform-as-code/v1/composition?kind=Moodle&instance=moodle1&namespace=namespace1”
Examples of possible future endpoints are: ‘provenance’ (for discovering history of actions performed on Custom Resource instances), ‘functions’ (for discovering static information on lifecycle operations supported by Custom Resources), and ‘configurables’ (for discovering static information on configurables supported by Custom Resources). We look forward to inputs from the community on what additional information on Custom Resources you would like to get from such endpoints.
Example of using KubePlus Cluster Add-on
As an example of how KubePlus Cluster Add-on is useful, you can check out this example of a custom Moodle Platform built from three Operators — Moodle, MySQL, and Volume backup/restore. The various Custom Resources available through these Operators are — Moodle, MysqlCluster, Restic, Recovery. KubePlus helps application developers discover following aspects of these Custom Resources:
- Moodle Custom Resource YAML definition needs a specific value to bind to a MysqlCluster Custom Resource instance. The ‘man’ endpoint helps here.
- In order to take backup of Moodle volume, the Deployment object for that Moodle Custom Resource instance needs to be given some label and that label needs to be used in the Restic Custom Resource label selector. The ‘man’ and ‘composition’ endpoints help here.
- The Moodle volume backup also needs name of the volume that needs to be backed up. ‘man’ endpoint helps here to surface the volume name which is an implementation detail of the Moodle Operator.
Comparing KubePlus Cluster Add-on with other tools
When thinking about platforms that are built using Operators and Custom Resources, following personas are involved — Operator developers/curators, Cluster administrators/DevOps engineers, and Application developers.
There are existing tools in the community to cater to these personas. For instance, for Operator developers, there exist sample-controller, kubebuilder, Operator SDK. For cluster administrators, there exist Helm and Operator Lifecycle Manager (OLM). KubePlus Cluster Add-on complements these tools.
Conclusion
Traditionally enterprises have been presented with two options for building their platform stacks: a) pre-built PaaSes, b) build your own platform automation. Multi-Operator platform stacks are creating foundation for a new option of Platform-as-Code by bringing in composability of selecting desired Operators and reusability through declarative YAML definitions of Custom Resources. KubePlus Cluster Add-on enables and simplifies this Platform-as-Code experience.
- If you are a Operator developer/curator, consider adding Platform-as-Code annotations on your Operator’s CRDs. This will make your Custom Resources easily discoverable in a cluster.
- If you are a cluster administrator/DevOps engineer, consider installing KubePlus Cluster Add-on in your cluster. It will enable your application developers to discover different kinds of information about Custom Resources through ‘kubectl’.
- If you are an application developer, use the Platform-as-Code endpoints to discover Custom Resources in your cluster and build your application platforms as code.
