CSI Driver

Enabling mTLS of Pods using the cert-manager CSI Driver

A Container Storage Interface (CSI) driver has been created to facilitate mTLS of Pods running inside your cluster through use of cert-manager. Using this driver will ensure that the private key and corresponding signed certificate will be unique to each Pod and will be stored on disk to the node that the Pod is scheduled to. The life cycle of the certificate key pair matches that of the Pod meaning that they will be created at Pod creation, and destroyed during termination. This driver also handles renewal on live certificates on the fly.

A CSI driver is a storage plugin that is deployed into your Kubernetes cluster that can honor volume requests specified on Pods, just like those enabled by default such as the Secret, ConfigMap, or hostPath volume drivers. In the case of the cert-manager CSI driver, it makes use of the ephemeral volume type, made beta as of v1.16 and as such will only work from the Kubernetes version v1.16. An ephemeral volumes means that the volume is created and destroyed as the Pod is created and terminated, as well as specifying the volume attributes, without the need of a PersistentVolume. This gives the feature of not only having unique certificates and keys per Pod, where the private key never leaves the hosts node, but that the desired certificate for that Pod template can be defined in line with the deployment spec.

Warning: Use of the CSI driver is mostly intended for supporting a PKI of your cluster and facilitating mTLS, and as such, a private Certificate Authority issuer should be used - CA, Vault, and perhaps Venafi, or other external issuers. It is not recommended to use public Certificate Authorities, for example Let's Encrypt, which hold strict rate limits on the number of certificates that can be issued for a single domain. Like Pods, these certificate key pairs are designed to be non-immutable and can be created and destroyed at any time during normal operation.

How Does it Work?

The CSI specification is a protocol and standard for building storage drivers for container orchestration platforms with the intention that a single driver may be ported across multiple platforms and outlines a consistent specification to how drivers should behave from an infrastructure perspective. Since cert-manager is designed to only be run with a Kubernetes cluster, so too does the cert-manager CSI driver.

The driver should be deployed as a DaemonSet which means a single instance of the driver may be run on each node. The driver will not work when running multiple instances on a single node. The set of nodes that the driver runs on can be restricted using the nodeSelector in its Pod template.

When a Pod is scheduled to a node with a cert-manager CSI volume specified, the Kubelet running on that node will send a NodePublishVolume call to the driver on that node, containing that Pods information as well as the attributes detailed from the in-line volume attributes. From this, the driver will generate a private key as well as a certificate request based upon that key using information built from the volume attributes. The driver will create a CertificateRequest resource in the same namespace in the Pod that, if valid, cert-manager will return a signed certificate.

The resulting signed certificate and generated key will be written to that node's file system to be mounted to the Pods file system. Since the driver needs access to the nodes file system it must be made privileged. Once mounted, the Pod will begin execution with the unique private key and certificate available in its file system, as defined by its mount path.

By default, the driver will keep track of certificates created in order to monitor when they should be marked for renewal. When this happens, the driver will request for a new signed certificate, and when successful, will simply overwrite the existing certificate in path.

When the Pod is marked for termination, the NodeUnpublishVolume call is made to the node's driver which in turn destroys the certificate and key from the nodes file system.

The CSI driver is able to recover its full state in the event the its Pod being terminated.