Adding SSL/TLS support to applications in Kubernetes-native way
In this post we discuss our experience of adding SSL/TLS support for a web application running on Kubernetes.
We have been working with one of our customers for building a Moodle (eLearning software) based platform stack on Kubernetes. Moodle is a PHP based application with three tier architecture consisting of a web server, application layer, and a database server.
We are using various Kubernetes Operators to deliver this end-to-end platform stack in Kubernetes-native manner. We use 4 different Operators for this — Moodle Operator, Community MySQL Operator, cert-manager Operator for SSL/TLS support and another community Operator for Volume backup/restore. In this post we discuss our journey towards adding SSL/TLS support to this Moodle stack.
What is SSL/TLS?
TLS stands for Transport Layer Security. It is basically a protocol that enables encrypted communication between browsers and web applications for which TLS is enabled. Any web application that is accessible over https has TLS enabled. One of the key requirements when setting up TLS for an endpoint, is to obtain a ‘SSL Certificate’ (SSL stands for Secure Socket Layer) that vouches for the identity of the Domain Name of your web application. This Certificate should be obtained from a trusted Certificate Authority such as Let’s Encrypt.
Enabling SSL/TLS in Kubernetes
In Kubernetes, you have the Ingress object which can be used to associate a Domain Name and a Certificate with a Kubernetes Service, as shown below:
apiVersion: extensions/v1beta1kind: Ingressmetadata:name: example1-ingressspec:tls: - hosts: - www.example1.net secretName: example1-cert rules: - host: www.example1.net http: paths: - path: / backend: serviceName: example1 servicePort: 30005
The Certificate is to be stored as a Secret object and its name is to be provided in the secretName attribute.
Towards configuring the Ingress object, following are the key configuration decisions that one has to make.
- What approach to use for obtaining SSL Certificates?
- Which method to use for exposing your web application Services?
1) Certificate Management
In our architecture we decided to use JetStack’s cert-manager for certificate management as it provides a Kubernetes-native approach for creating and managing SSL certificates from Let’s Encrypt. cert-manager supports two ways of obtaining certificates.
- One is using the ‘http’ provider.
- And another is using the ‘dns’ provider.
Between the two, ‘http’ provider is simpler to use. Based on certain annotations on the Ingress object, it obtains a Certificate and also saves it as a Secret object. For using it though, port 80 of the IP address to which the Domain Name is mapped, needs to be open. This fact makes the decision to pick ‘http’ vs. ‘dns’ closely tied with how you expose your Services.
2) Ways to expose Web application Services
There are 3 options in Kubernetes for exposing your web application services: NodePort, ClusterIP, LoadBalancer.
In our initial design, we were using NodePort type Service for exposing different Moodle instances, each on a different port. But that meant we could not use the ‘http’ provider for obtaining certificates. This was because port 80 on a general Kubernetes cluster is not open. So we went with the option of using ‘dns’ provider (reference). Soon though we reconsidered the choice of NodePort for exposing Moodle services. Even without TLS in picture, it was clear to us that using NodePort is not a good idea as there is no guarantee that the ports that our Moodle Operator was allocating to different Moodle instances would be available on a general Kubernetes cluster. The simplicity of using ‘http’ provider further helped sway our decision away from NodePort completely.
The next question was should we go with ClusterIP Service type or LoadBalancer Service type. With LoadBalancer Service type, every Moodle Instance would get its own LoadBalancer. And we would be able to use the ‘http’ provider as well, since port 80 on each of these LBs would be open. While this seemed attractive, we decided against this for two reasons. First, adding a LB instance to a Moodle stack would make it costly. Second, with ‘http’ provider for obtaining certificates, we could create a single LB across all Moodle instances and use unique Domain names along with ClusterIP type Services to segregate different Moodle instances from each other. The Domain name of each Moodle instance will map to the same LB and the Ingress object for each Moodle Instance will map that instance’s Domain name to corresponding Moodle instance’s Service. This architecture is shown below.
We have chosen this option. You can check it out here. The single LB that is used as entrypoint for all the Moodle instances is the LB corresponding to the Nginx Controller, which needs to be installed in the cluster for using cert-manager.
Experience and Lessons Learned
1. Overall the experience of using cert-manager was smooth. We felt that the documentation can be improved in some places and the Custom Resources can be modified as well. We have filed Github issues for these (you can track them here and here).
2. When setting up SSL there are potentially six configuration combinations corresponding to, ‘how’ to expose your Service (‘NodePort’, ‘LoadBalancer’, ‘ClusterIP’), and ‘which’ method for certificate issuance to use (‘http’, ‘dns’). Be open to experimenting with these options and pick the option that works best for your use-case.
3. Getting https configured for the endpoints is only half the story. There may be additional work that needs to be done on the application itself to get complete end-to-end SSL/TLS setup working.
Conclusion
SSL/TLS for applications running on Kubernetes is a key capability needed by every enterprise today. In our view the best approach towards this is to develop a Kubernetes-native solution. It will enable you to use standard Kubernetes interfaces such as Kubernetes YAMLs and tools such as helm and kubectl to configure SSL for your platform stack declaratively as code.
If you are interested in building similar Kubernetes-native platform stack, we offer customized training around Kubernetes extensibility and building platform stacks using Kubernetes-native components. Please get in touch to leverage it for your platformization needs.
