Certificates in a distributed world

As I explained in the previous post, the objective of our project was to connect some of our customer’s Pods to their customer’s premises, providing connectivity to some of the systems they run in-house, like databases, Splunk or Solr.

This kind of connectivity is provided with a connection between two Envoys:

1. an Envoy running as a sidecar in the Pod, acting as a regular ****HTTP proxy for other containers in the Pod,

2. a peer Envoy running in an Azure VM that is located in the same VNET as a VPN gateway,

All traffic is sent multiplexed in a HTTP/2 connection between these two Envoys. So we must have a way to:

• get certificates for the HTTP/2 connection between these two Envoys,
• this certificate must be unique for this customer (so an Envoy in a Pod for customer X cannot connect to an Envoy in a VM for customer Y)
• and certificates must be rotated periodically without any human supervision

I solved all these requirements with the help of Vault and the Vault Agent, leveraging the use of different authentications mechanisms depending on the environment where the Vault Agent is running.

Certificates in the Azure VMs

The first part of the equation is how to get certificates for the Envoy in the Azure VM in an automatic way.

VMs get certificates with the Vault Agent. The Vault Agent is a lightweight application that tries to authenticate in a Vault server and get a certificate (in this case, a certificate for a specific customer). The Vault Agent can be configured for authenticating in different ways, and one of them is the Azure authentication: it can obtain a “token” with some information about the VM, like the Resource Group or Subscription where it is, all signed by the Azure.

Once the Vault Agent has this token, it can then present this token to the Vault Server and the Vault Server can go to Azure and check if this token is valid or not (the Vault Server must have been preconfigured for this).

If the token is valid, the Vault Server can be sure this VM is who it says it is. Then the Server can use the role (previously created with Terraform for this customer) that maps Resource Group XYZ to customer p12345, and generate a corresponding certificate for customer p12345.

Certificates in the Pods

Pods must also obtain a certificate for establishing the communication with their Azure VM peer.

The cert-manager will authenticate with the Vault Issuer with a Service Account, using the kubernetes authentication preconfigured in the Vault Server. The Vault Server has been previously configured with a policy that specifies that “Service Accounts XYZ can generate certificates for customer p12345”.

The Vault Server will use a auth:delegate Service Account (preconfigured in the Server) for checking in the Kubernetes API if the Service Account XYZ provided by cert-manager is valid. Once this has been validated, cert-manager can obtain a certificate by using the PKI backend for customer p12345.