News: This article has now been accepted in official Kubernetes documentation:
Power of Kubernetes is in its extendability. One of the ways in which you can extend Kubernetes is by writing your own custom controllers. You can write custom controllers that handle in-built Kubernetes objects, such as Deployment, Service, in new ways, or you can add new custom resources within Kubernetes and write a custom controller to handle these new resources.
Recently we have been developing a Custom Resource Definition (CRD) for Postgres. While developing this CRD we spent some time studying Kubernetes’s client-go library. This library contains various mechanisms that you can use when developing your custom controllers. These mechanisms are defined in the tools/cache folder of the library.
We have put together a pictorial representation showing how the various components in the client-go library work and its interaction points with the custom controller code that you will write.
Here is that representation.
The picture is divided into two parts— client-go and Custom Controller.
Below we explain each component pointing to appropriate places in the client-go code and our Postgres custom controller that shows the main actions of each component.
client-go components
- Reflector: A reflector watches the Kubernetes API for the specified resource type (kind). This could be an in-built resource or it could be a custom resource. When it receives notification about existence of new resource instance through the watch API, it gets the newly created object using the corresponding listing API. It then puts the object in a Delta Fifo queue.
- Informer: An informer pops objects from the Delta Fifo queue. Its job is to save object for later retrieval, and invoke the controller code passing it the object.
- Indexer: An indexer provides indexing functionality over objects. A typical indexing use-case is to create an index based on object labels. Indexer can maintain indexes based on several indexing functions. Indexer uses a thread-safe data store to store objects and their keys. There is a default function that generates an object’s key as<namespace>/<name> combination for that object.
Custom Controller components
- Informer reference: This is the reference to the Informer instance that knows how to work with your custom resource objects. Your custom controller code needs to create the appropriate Informer.
- Indexer reference: This is the reference to the Indexer instance that knows how to work with your custom resource objects. Your custom controller code needs to create this. You will be using this reference for retrieving objects for later processing. client-go provides functions to create Informer and Indexer according to your needs. In your code you can either directly invoke these functions or use factory methods for creating an informer.
- Resource Event Handlers: These are the callback functions which will be called by the Informer when it wants to deliver an object to your controller. The typical pattern to write these functions is to obtain the dispatched object’s key and enqueue that key in a work queue for further processing.
- Work queue: This is the queue that you create in your controller code to decouple delivery of an object from its processing. Resource event handler functions are written to extract the delivered object’s key and add that to the work queue.
- Process Item: This is the function that you create in your code which processes items from the work queue. There can be one or more other functions that do the actual processing. These functions will typically use the Indexer reference, or a Listing wrapper to retrieve the object corresponding to the key.
You can try out our Postgres custom resource to see how these components fit together in real code. This custom resource has been developed following the sample-controller available in Kubernetes.
We hope that above explanation of various components and mechanisms involved in writing custom Kubernetes controllers will help you in writing your own custom controllers.
Note: If you like this post, you may also want to check-out:
