In the new setup we heavily rely on Terraform for Infrastructure as Code collaboration and automation. One thing that we're adding to our Terraform repository is automated management of secrets and ENV vars for our app deployments.
App Deployment in Kubernetes
First of all, let's show you the thing that this is all about in the end: the application deployment. Our Ruby on Rails applications are deployed using a Kubernetes Deployment. Here is a snippet of one of our applications from our Terraform repository:
As you can see, we load the application's environment from a single Kubernetes Secret named dispatch-env.
We also put all Kubernetes objects for a given application in a specific Kubernetes Namespace. Following this practice makes sure that by default we don't let secret stuff from one application deployment leak over to a different application.
ENV vars via a single Kubernetes Secret
Simply loading the application's environment from a single Kubernetes secret makes it easy to manage the whole runtime environment via a single Terraform resource. Here is a redacted and shortened example of one of our apps:
Secret storage: Azure Key Vault
We used to not store secrets at all or have only certain manually generated secrets stored in our 1Password business account.
Because Terraform can talk to Azure Key Vault via it's Azure provider, we can now start managing secrets without ever touching them or making then visible to a human eye. This allows us to read a secret key from Azure Key Vault in our Terraform repository:
Setting up the application ENV variables with secrets from Azure Key Vault
So uptil this point we have three things prepared.
- Applications are deployed using Kubernetes Deployments in their own namespace.
- An application uses one single Kubernetes Secret to load the environment.
- Azure Key Vault is set up and we can load secrets via Terraform resources.
Now let's bring it all together.
In the following redacted sample you'll see how we're pulling in SMTP login credentials and a database password to load those into the application deployment's ENV:
Why this is great
Such a setup is great for the following reasons:
- Anyone in the team can take a look at our Infrastructure as Code git repository to see what ENV vars are configured for a given application deployment.
- Secrets are safely stored in Azure Key Vault, including versioning, timestamps, access logs, and access policies for team members and applications.
- We don't have to manually log in to Dokku servers or into Heroku anymore to define the ENV variables for our apps.
What can be improved
One thing that still bugs me is that the actual secret values are stored in a Kubernetes Secret. In practice, this is not worse than storing them in Heroku or Dokku. But it would even be greater to have a setup where the secret value is not visible to humans with access to the Kubernetes cluster.