Kubernetes and the future of as-Code Systems

Kubernetes completed five years recently. In this short timespan it has become the most talked about and adopted project in the enterprise world. In this blog post, we explore how Kubernetes’s extensible architecture is giving the flexibility and control back to enterprises in consuming public/private clouds and how in-turn it is contributing to the evolution of ‘as-Code’ systems.

Cloud Consumption Models

Before containers and Docker came into the picture, the typical journey of enterprises adopting public/private clouds used to be — start with infrastructure virtualization (VMWare, EC2, OpenStack), build automation using infrastructure provisioning systems (CloudFormation, Heat, Terraform, Ansible), set up application deployment workflows on this infrastructure (CI/CD, PaaS). In this world, tools like CloudFormation and Terraform got quickly adopted as they codified infrastructure creation and provisioning steps as declarative templates. While these tools enabled repeatability and shareability, the automation was still tied to a particular cloud provider as there was no standard across underlying cloud APIs. The automation within these tools primarily abstracted the underlying cloud APIs for simplifying their use.

Kubernetes’s extensibility

In this world Kubernetes came in as a breath of fresh air.

  • It is open; the project is not owned by any single company but is managed by Linux foundation.
  • It has been adopted across all public and private cloud providers.
  • And its architecture provides the right building blocks for extending its core APIs to run any software needed in an application stack natively on Kubernetes.

Especially, Kubernetes’s extensible architecture has made it possible to morph it into a customized platform layer as per an enterprise’s needs. Kubernetes provides extension points using which one can add new REST APIs (also known as Operators) into a Kubernetes cluster. These custom APIs work alongside Kubernetes built-in REST APIs (e.g. Pod, Service, Deployment) and are accessible using the same native tools (‘kubectl’, ‘helm’). There has been a surge of these open source API extensions (or Operators) built to run inside Kubernetes. They span from APIs that manage stateful services, like databases, on Kubernetes, to APIs that manage AI/ML workloads (Spark, TensorFlow), to APIs for taking backups/restores of Kubernetes Volumes. These days an enterprise can select from such existing APIs, or even build their own, and run them inside their Kubernetes cluster and turn their cluster into a customized platform for their specific requirements. This approach has following advantages -

  • First, the fact that entire application platform is running on Kubernetes makes it portable across public/private/hybrid clouds.
  • Second, the declarative nature of Kubernetes YAMLs enable leveraging all the benefits of ‘as-Code’ systems like repeatability and shareability in the Kubernetes world.

as-Code Systems and Kubernetes

The ability to add new APIs into a Kubernetes cluster enables a novel approach for creating platform workflows on Kubernetes. In this approach platform elements like databases, loadbalancers, ssl certificates, application workloads like Spark, etc., all are represented as Custom Resources of Kubernetes. One can define their platform workflows as declarative Kubernetes YAML files using these Custom Resources. Hence we call these platform YAMLs as ‘Platform-as-Code’ (PaC). Following figure demonstrates evolution of ‘as-Code’ systems.

Typically any ‘as-Code’ system is designed to offer a common language to provision a technology stack leveraging underlying APIs and resources of a cloud.

  • First generation systems like AWS CloudFormation focussed on delivering this for a single cloud (AWS).
  • Second generation systems like Terraform, focussed on abstracting underlying cloud APIs but did this across multiple clouds, offering a common language to provision technology stack in multi-cloud environments.
  • There are couple of key differences between first and second generation as-Code systems with the third generation of ‘as-Code’ systems. In the first and second generation as-Code systems, the set of APIs that the orchestration layer (as-Code layer) could use was fixed and known a priori. This is not the case with Kubernetes as one can extend Kubernetes API set by installing API extensions/Operators anytime. Second, with Kubernetes offering a base platform layer across private/public clouds, it is no more about abstracting the underlying cloud infrastructure APIs but it is about utilizing Kubernetes APIs to realize your Platform workflows. In this context, a Platform-as-Code system needs to augment Kubernetes to simplify creation of these platform YAMLs using built-in and Custom Resources enabling Platform-as-Code experience in hybrid multi-cloud environments. We have developed KubePlus API add-on to enable this Platform-as-Code experience.

KubePlus approach to Platform-as-Code

Two of the key characteristics of any ‘as-Code’ system are-

  1. ability to discover underlying supported resources, and
  2. ability to inter-connect / bind these resources to realize different workflows.

In closed source systems like CloudFormation, ‘discovery’ amounts to figuring out what all resources are supported by the specific version of CloudFormation that you are using. Terraform provides CLI commands like ‘terraform providers’ to find the information about supported providers. For ‘binding’, these systems provide in-built mechanisms like imports, reference parameters, or connection parameters. In Kubernetes, building a solution that offers generic ‘discovery’ and ‘binding’ functions for Custom Resources is tricky as compared to earlier ‘as-Code’ systems as the API extensions / Operators are developed independently. There is no shared understanding of what Spec properties to define for APIs, their semantics, interoperable defaults, etc.

At CloudARK, we are building KubePlus API discovery and binding add-on that focuses on solving these ‘discovery’ and ‘binding’ problems for Kubernetes APIs. It is currently in active development. The KubePlus API add-on offers new discovery endpoints (e.g. ‘man’, ‘composition’) that enable application developers to query static and dynamic information about various Custom Resources available in the cluster. E.g.

kubectl get — raw “/apis/platform-as-code/v1/man?kind=MysqlCluster”kubectl get — raw “/apis/platform-as-code/v1/composition?kind=Moodle&instance=moodle1&namespace=ns1”

The information that is output from these commands is something which can not be found through ‘kubectl explain’ command. The discovery endpoints provide information to build interactions between Kubernetes Custom & built-in Resources manually in Kubernetes YAMLs and ultimately define Platform-as-Code workflows. We are also developing additional automation in this add-on that helps with automatic binding resolution between Custom Resources within Platform-as-Code workflows. Here is the on-going work that uses mutating web-hook admission controller to perform run-time binding between Custom Resources required for Platform-as-Code workflows.

Give KubePlus API add-on a try and let us know your feedback. In a future post we will present its architecture in more details.


Kubernetes is ushering in new era of platformization in enterprise space. The control of the API layer is being transferred back to the enterprise platform engineering teams through CRDs/Operators (technical terms for custom API extensions). As more and more Kubernetes CRDs/Operators get developed, enterprises are going to get choice and flexibility in creating their custom platform layers on Kubernetes. Platform-as-Code approach simplifies composition of multi-Operator platform stacks as per enterprises’ specific requirements, instead of them having to adopt a pre-built platform offering.

Sign up for our Platform-as-Code eBook to understand in detail the advantages of using Platform-as-Code approach for building your Kubernetes native platform layer.