CRD for CRDs to design multi-tenant platform services from Helm charts
Platform engineering teams prepare Kubernetes clusters for sharing between multiple users and workloads. This involves building Helm charts for variety of operational workflows. The challenge is in delivering these Helm charts as platform services in self service manner so that they can be used repeatedly with tenant level controls and consumption tracking.
To address this challenge, we have built an open-source framework (KubePlus) to create multi-tenant platform services with the required isolation guarantees and per-tenant consumption metrics tracking. It involves taking a Helm chart of an operational workflow and building a CRD (Kubernetes API) to deliver the workflow as-a-service, along with attaching required policies and Prometheus monitoring to it.
KubePlus comes with a CRD that allows you to construct your own CRDs on the fly that embed a Helm workflow. The variables from values.yaml become the Spec Properties of this new CRD. Additionally, you can attach policies such as resource requests/resource limits to the objects that will be created using this new CRD. You can also get aggregated CPU/Memory/Storage metrics in the Prometheus format for the overall workflow.
This framework is very powerful as it offers platform teams a consistent way to enable, govern and monitor multi-tenant Kubernetes environments that are extended with Operators.
KubePlus offers a CRD named ResourceComposition to
- Compose new CRDs (Custom Resource Definitions) to publish platform services from Helm charts
- Define policies (e.g. Node selection, CPU/Memory limits, etc.) for managing resources of the platform services
- Get aggregated CPU/Memory/Storage Prometheus metrics for the platform services
Here is the high-level structure of ResourceComposition CRD:
To understand this further let us see how a platform team can build a MySQL service for their product team/s to consume. The base Kubernetes cluster has MySQL Operator installed on it.
The platform workflow requirements are:
- Create a PersistentVolume of required type for MySQL instance.
- Create Secret objects for MySQL instance and AWS backup.
- Create a MySQL instance with a backup target as AWS S3 bucket.
- Setup a policy in such a way that Pods created under this service will have specified Resource Request and Limits.
- Get aggregated CPU/Memory metrics for the overall workflow.
Here is a new platform service named MysqlService as Kubernetes API.
A new CRD named MysqlService has been created here using ResourceComposition. You provide a platform workflow Helm chart that creates required underlying resources. The Spec Properties of MysqlService come from values.yaml of the Helm chart. Additionally you define the policy and monitoring inputs as part of CRD creation step. Product teams can use this service to get a MySQL database for their application and KubePlus will ensure that the underlying Mysql Pods are provided with appropriate resource requests and limits. You can checkout this example here.
Conclusion:
Platform teams can use this mechanism of CRD for CRDs to deliver a variety of their services in self-service manner and apply required governance and monitoring abstracting away all the details from the end users. In turn, this mechanism can become a foundation for simplifying the collaboration between platform and product teams and managing multi-tenant environments on a Kubernetes cluster.
Learn more about this approach here. Contact us here for a free demo and discussion.
