How to keep application secrets in Kubernetes
Whether migrating existing solutions to public clouds or developing cloud solutions from scratch, many organizations run up against security challenges. Luckily, there are a lot of security advisories, from cloud providers and independent cybersecurity organizations, that exist to offer guidance. One of these is the CIS Benchmarks for Kubernetes, which amounts to 270 pages of recommendations on how to secure a Kubernetes cluster. For professionals who are well familiar with the security and Kubernetes domains, these recommendations are totally applicable. However, for those just beginning to familiarize themselves with cloud environments, while developing plans for migrating existing solutions to the cloud, additional practical examples are most likely needed.
In this article, we will discuss a very important aspect of securing solutions deployed in Kubernetes: dealing with secrets. You will learn how to create and use Kubernetes Secrets, and how to sync secrets between Kubernetes and Azure Key Vault. For the purpose of illustration, a pilot Microsoft Azure cloud platform will be used.
Solution overview
To begin, let’s take a look at an IT solution that belongs to an imaginary company called Contoso AB. Contoso AB wants to migrate their solution to Microsoft Azure cloud. The solution has a classic 3-tier architecture.It consists of a web server farm, a load balancer, a business logic server, servers with background tasks (report generators, integration with external services, etc.), a file server, and a database server:
After analyzing various cloud migration strategies, the solution architects decided to use Azure Kubernetes Service as a target platform. Once they familiarized themselves with the basics of Kubernetes for enterprise , they developed a cloud architecture based on the Azure Kubernetes Service:
According to the cloud journey plan, the first step of migrating Contoso AB's solution to the cloud is moving applications to Docker and adding Helm to them. To build the web and business logic tiers, Kubernetes Deployments are used to ensure that a specified number of pods are launched at every moment. Additionally, Service and Ingress Kubernetes objects are used in order to implement the web tier.
To implement the data tier, a plan is made to both migrate relational databases into Azure SQL Managed instance and migrate the file server into Azure Files share, which is mounted to the pod.
How to create Kubernetes Secrets
When developing the architecture of the Contoso AB solution, a question arose: How can we keep Kubernetes secrets, which are needed for interaction between different solution components?
- Contoso AB’s solution contains the following secrets:
- Credentials needed for interaction between the web and business logic services
- Tokens and credentials needed for integration with external services
- Database and Azure Files that share connecting strings
- SSL certificates for web endpoints
Secrets inside Kubernetes are described in the yaml config files as Kubernetes objects with kind Secret. For example, config file backend_secret.yaml, with credentials for a business logic tier looks like this:
apiVersion: v1
kind: Secret
metadata:
name: backendsecret
data:
username: '<Base 64 encoded username>'
password: '<Base 64 encoded password>'
To create this object, first of all, you need to encode the username and password in base64 format:
echo -n '<user name>' | base64 > usernamefile.txt
echo -n '<password>' | base64 > passwordfile.txt
Next, there are two ways you can create a Kubernetes Secret:
- Copy the base64 encoded values into the backendsecret.yaml file, and execute command:
kubectl -f apply backend_secret.yaml
kubectl create secret generic backendsecret \
--from-file=username=./usernamefile.txt \
--from-file=password=./passwordfile.txt
How to use Kubernetes Secrets
All application secrets can be divided into two large groups: secret strings (credentials, tokens, connecting strings, etc.) and secret files (certificates, key files, license files, etc.). Let's look at how to use secret strings first. As a rule, in real applications, such secrets can be taken from the environment variables. In the webservice_deployment.yaml config file we can see how to inject values for environment variables from secrets.
apiVersion: apps/v1
kind: Deployment
metadata:
name: webservice-deployment
labels:
app: webservice-deployment
spec:
replicas: 2
selector:
matchLabels:
app: webservice-deployment
template:
metadata:
labels:
app: webservice-deployment
spec:
containers:
- name: weservicecontainer
image: webserviceimage
ports:
- containerPort: 80
livenessProbe:
httpGet:
path: /
port: 80
readinessProbe:
httpGet:
path: /
port: 80
env:
- name: BACKEND_USERNAME
valueFrom:
secretKeyRef:
name: backendsecret
key: username
- name: BACKEND_PASSWORD
valueFrom:
secretKeyRef:
name: backendsecret
key: password
This config file contains a reference to Kubernetes Secret with the name backendsecret, which contains a key value pair for username and password. The username will be available as a BACKEND_USERNAME environment variable, and the password will be available as a BACKEND_PASSWORD.
In order to gain access to a secret as a file from the code inside the pod (for instance, if an application is needed from a certificate or key file), this file, which is located inside the volume, should be mounted to the pod.
For example, consider how to provide access to the key file stored, as the Kubernetes secret externalservicekeyfile from the background service:
apiVersion: apps/v1
kind: Deployment
metadata:
name: backgroundintegration-deployment
labels:
app: backgroundintegration-deployment
spec:
replicas: 3
selector:
matchLabels:
app: backgroundintegration-deployment
template:
metadata:
labels:
app: backgroundintegration-deployment
spec:
containers:
- name: backgroundintegrationcontainer
image: backgroundintegrationimage
livenessProbe:
...
ports:
- containerPort: 80
volumeMounts:
- name: keyfile
mountPath: ‘/etc/keyfile’
readOnly: true
volumes:
- name: keyfile
secret:
secretName: externalservicekeyfile
For the application code, inside this Deployment, a key file is available in path /etc/keyfile/externalservicekeyfile.
To route external web requests from the Load Balancer to the web tier, an Ingress Controller is used. It can also terminate SSL traffic. This means that web traffic is decrypted inside Ingress Controller, and then directed to the necessary service, deployed in Kubernetes.
Rules, describing how to route traffic from the load balancer to given services, are described as an Ingress Kubernetes object. These objects also contain references to secrets with certificates, needed for terminating SSL traffic.
Kubernetes config file ingress.yaml with the ingress rule, needed for the Contoso AB solution, looks like the following:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: webservice-ingress
spec:
tls:
- hosts:
- contosoab.se
secretName: contosoab-se-tls
rules:
- host: contosoab.se
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: webservice-service
port:
number: 80
In this rule, for the routing request coming to contosoab.se, the Kubernetes service webservice-service is used as a destination, and SSL traffic is terminated on the way into Ingress Controller. The certificate needed for that is contained within the contosoab-se-tls secret.
How to sync secrets between Kubernetes and Azure Key Vault
During the technical discussion about the proposed solution, the Security Architect at Contoso AB had the following questions:
- If secrets will be kept directly in the config files, how can we make sure that they are only accessible to those people with the appropriate privileges?
- How can we protect secrets against leakage?
- How can we automate the secret rotation and notify secret owners about their expiration?
- How can we capture events and logs from the secret storage for integrating with the SIEM and determine possible attacks, intrusions and leakages?
To meet these challenges, Contoso's solution architect proposed managing secrets with the Azure KeyVault service. It can keep, create, and rotate keys, secrets and certificates. TheAzure KeyVault can also be integrated with the other Azure services, which use connecting strings or secret keys, as Azure SQL Managed Instances and Azure Files.
To synchronize secrets in Azure KeyVault with the Kubernetes secrets, the Azure Key Vault Controller can be used, deployed on the AKS cluster.
One such controller is the Key Vault controller from Sparebanken. This controller can be deployed on an AKS cluster using Helm. To do this, after receiving the credentials in AKS, you need to create a namespace in it for the Azure Key Vault controller:
kubectl create ns azure-keyvault-controller-ns
Next, you should update the Helm repo locally:
helm repo add spv-charts http://charts.spvapi.no
helm repo update
Finally, you execute installation command locally:
helm upgrade --install azure-keyvault-controller spv-charts/akv2k8s \
--namespace azure-keyvault-controller-ns
To synchronize secrets between AKS and Azure Key Vault, instead of creating Kubernetes objects with kind “Secret”, you must create Kubernetes objects with kind “AzureKeyVaultSecret”. Here is an example, which looks like an updated file external-service-token.yaml, created for synchronizing a token for external services stored with the Azure Key Vault.
apiVersion: spv.no/v1
kind: AzureKeyVaultSecret
metadata:
name: external-service-token
spec:
vault:
name: contoso-production-keyvault
object:
name: contoso-external-service-token # This is a name of secret in Azure Key Vault
type: secret
output:
secret:
name: external-service-token
dataKey: token-value # key to store object value in AKS secret external-service-token
Changes and deletions of secrets in Azure Key Vault will automatically reflect on Kubernetes secrets.
Conclusion
In this article, we have described the general principles of working with secrets on the Kubernetes platform. With the Azure Kubernetes Service, you can store secrets in the Kubernetes cluster itself or in the Azure Key Vault and, in the future, synchronize them using the Key Vault Controller.
If you would like more detailed information, support, or technical advice on how to configure secrets in Kubernetes, please feel free to get in touch. We can also offer support on Kubernetes monitoring, logging, continuous integration, continuous delivery or any other DevOps consultancy needs you may have.
