Kubernetes offers a way to create and manage sensitive information for your applications using an Object called Secret, however the information stored in a Secret is simply Base64 encoded, which is arguably as good as keeping them in plaintext. Hence for every production workload you would find solutions such as Hashicorp’s Vault or Azure Key Vault managing the secrets (keys, certificates etc) deployed and integrated with Kubernetes. This is the first entry of the series of blog posts that discusses this topic in detail. In this entry, we explore one of the ways you could configure Azure Kubernetes Service with Azure Key Vault.

Using a FlexVolume

One of the simplest way you could integrate Azure Key Vault with AKS is to mount the key vault as a FlexVolume to the pods running on AKS. Hang on, but…

What is a FlexVolume ?

FlexVolumes can be thought of an extension mechanism for volumes provided by Kubernetes and enables users to write their own drivers and add support for their volumes in Kubernetes.

There is an excellent blog post by Lee Briggs on this very topic for the curious minds.

Coming back to Azure Key Vault, the FlexVolume Pods running on every node on your cluster are responsible for mounting the Volumes containing the secrets onto the Pods running on that node. The application pods can then read these secret values from the filesystem at the mounted path, just like reading any other file on the filesystem. These flex-volume pods contains the Azure Key-Vault Volume Driver code which uses a Service Principal to authenticate and access the secrets stored in the Key Vault. The Service principal is made available to these pods via another Kubernetes Secret. Something like below:

Hands-on Exercise

Enough of the theory, let’s start by creating some resources we would need for this to work end-to-end – We will be using the example from the famous AKS Workshop where we deploy an Order API which exposes a simple API and writes order info to a MongoDB database. We will store the MongoDB password in Azure Key Vault.

Let’s define some variables that we will be using in the commands below

$RG_NAME=aksakvtestRG
$KV_NAME=akvOrderAPI
$SP_NAME=http://aksakvintegrationtest
$MONGODB_USERNAME=orders-users
$MONGODB_PASSWORD=orders-password

• Deploy MongoDB to your Kubernetes cluster using Helm

helm install stable/mongodb --name orders-mongo --set mongodbUsername=$MONGODB_USERNAME,mongodbPassword=$MONGODB_PASSWORD,mongodbDatabase=ordersdb

• Create a Kubernetes Secret to hold MongoDB specific information such as hostname, username and password to connect. We will improve on this very step by delegating the password to KeyVault in a bit but for now, lets use the Kubernetes Secret

kubectl create secret generic mongodb --from-literal=mongoHost="orders-mongo-mongodb.default.svc.cluster.local" --from-literal=mongoUser=$MONGODB_USERNAME --from-literal=mongoPassword=$MONGODB_PASSWORD

In the above example note that if you installed MongoDB in some other namespace ( other than default ) while using Helm, then the service exposed by the helm chart would be different from what’s given above. It is normally printed on the console what resources are created after do a helm install. Otherwise you can always query using kubectl

• Create the Order API deployment to work with the MongoDB and Secret provisioned above. Use the following gist to create the API deployment

Create this deployment using kubectl

kubectl apply -f orderapi-deployment.yaml

• Optional: Expose a service to test the API. If you need details, refer to the awesome AKS Workshop for details.

Now that we have an API deployed on our Kubernetes cluster which talks to MongoDB using Kubernetes secrets, we ought to improve this and use Azure key Vault to hold the sensitive information. Onto the actual work…

• Provision the Azure Key Vault and save MongoDB’s password in it

az keyvault create -g $RG_NAME -n $KV_NAME
az keyvault secret set --vault-name $KV_NAME --name mongo-password --value $MONGODB_PASSWORD

• Create a Service Principal to be used for authenticating against the Key Vault

az ad sp create-for-rbac --name $SP_NAME --skip-assignment
$SP_App_ID=$(az ad sp list --display-name $SP_NAME --query "[].appId" -o tsv)
$SP_Password=<value of the password from above command for creating SP>

• Assign the Service Principal Reader role on the Key Vault and create a Key Vault policy to give the Service Principal permissions to read the secrets

KEYVAULT_ID=$(az keyvault show --name $KV_NAME --query id --output tsv)
az role assignment create --role Reader --assignee $SP_NAME --scope $KEYVAULT_ID
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn $SP_App_ID

• Create a Kubernetes Secret that stores the application id and password for the Service Principal created above

kubectl create secret generic kvcreds --from-literal clientid=$SP_App_ID --from-literal clientsecret=$SP_Password --type=azure/kv

• Now we deploy the flex-volume pods to your Kubernetes cluster – which contains the code for Azure KeyVault Driver that creates and mounts volumes onto your pods.

kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml

This will deploy the flex-volume pods in the namespace kv. You can confirm that there are N number of these pods ( N is the number of nodes in your cluster )

• Modify your deployment by reading secret from the FlexVolume instead of Kubernetes Secret. Update the deployment with the following changes
  • Define the FlexVolume which will authenticate against the KeyVault using the Secret we created above
  • Mount the volume to the Order API Pod
  • Replace the environment variables definition to read from FlexVolume instead of from Secrets

Here is how the updated deployment file looks like

Use kubectl to update your deployment like

kubectl apply -f updated-orderapi=deployment.yaml

And this is it. Great, if you have followed through these steps. Congratulations ! You have successfully configured your Azure Key Vault with your applications running on Kubernetes.

Some of the issues I think exists with this approach are:

1. The need to change application code to read the secrets from a certain mounted path.
2. Containers deployed by the keyvault-flexvolume DaemonSet run with root privileges. If your Kubernetes clusters has Pod Security Policies enabled that forbids the use of root containers then you would have to apply this PSP policy in order for these containers to work
3. We wanted to get away from using a Kubernetes secret but now we are back to using it for saving our Service Principal for authorizing to Key Vault. Back to square one, eh ?

Conclusion

This is just the tip of iceberg where we have explored how you could secure the management of sensitive information while deploying your workloads. Future blog entries of this series would cover:

1. Using AAD Pod Identity with Azure Key Vault
2. Using AKS with Hashicorp’s Vault

Let me know in comments if you would like to hear about anything else or have any comments on how it worked out for you.