How to use Kubernetes secrets?

Before we jump onto kubernetes secrets let’s first try to understand What is Secret? - Secret is sensitive information which can not be written in a plain text or shared with others. The best example of Secrets would be the password.

But Why do we need to use Kubernetes Secret? - Running a bare minimum kubernetes cluster does need a kubernetes secret but when you start deploying a docker container application then there is need of Kubernetes Secret for storing sensitive data.

One of the simplest examples of kubernetes secrets would be running mysql container images inside the kubernetes cluster. As you now mysql is a database and to access the database we need username, password and since we are running the mysql container inside the kubernetes cluster so we need to store those credentials(Username, Password) somewhere inside the kubernetes cluster and it should be safe enough, so for storing mysql username and password we are going to use Kubernetes secrets.

This blog is a comprehensive guide on Managing Kubernetes secrets and we are going to touch upon following topics -

1. How does Kubernetes Handle the Secrets?
2. Create Kubernetes secrets using kubectl command
3. Create base64 encoded kubernetes secrets
4. Create Kubernetes secrets from files
5. Use the kubernetes secrets inside your POD or deployment manifest
6. Conclusion

1. How does Kubernetes Handels the secrets?

When you work with Kubernetes you always run multiple docker containers but one thing Kubernetes is good at is running the container in an ephemeral mode so that each container is independent of the other and does not share any resources with each other.

So if there is a need of storing sensitive information by docker container then Kubernetes secrets are created. But Kubernetes secrets are independent objects and are not bound to any Kubernetes POD or docker container.

Since Docker containers are running in the ephemeral mode, we need to make the Kubernetes secrets available to the deployment manifest of your docker container.

As a general practice, we always create Kubernetes secrets independently, and the secrets can only be accessed by the PODS.

2. Create Kubernetes secrets using kubectl and --from-literal

One of the easiest ways to create the Kubernetes secret is by using the kubectl command and --from-literal flag. Let’s take an example to understand Kubernetes secret creation. In this example, we need three things -

1. secret-name - test-secret
2. username - test-user
3. password - testP@ssword

2.1 Here is the command for creating kubernetes secret -

1kubectl create secret generic test-secret --from-literal=username=test-user --from-literal=password=testP@ssword
2secret/test-secret created

2.2 Verify the secret using the following command -

1$ kubectl get secret test-secret
2NAME          TYPE     DATA   AGE
3test-secret   Opaque   2      11s

2.3 Check the secret using the kubectl describe

Now we know that the Kubernetes secret has been created. Let's check a few more details by running the kubectl describe command. This command can show the secrets attribute like username and password but it will not show the sensitive details.

 1$ kubectl describe secret test-secret
 2Name:         test-secret
 3Namespace:    default
 4Labels:       <none>
 5Annotations:  <none>
 6
 7Type:  Opaque
 8
 9Data
10====
11password:  22 bytes
12username:  9 bytes

2.4 Copy kubernetes secret between the namespaces

If there is need to copy or share kubernetes secrets between multiple workspace then I would recommend reading this guide on - Share kubernetes secrets between namespaces

3. Create base64 encoded kubernetes secrets

In the previous step, we have seen how to create kubernetes secrets in the plain text format which is okay but plain text secrets are not recommended practice. You should always store the secrets in such a way that they can not be viewed as plain text.

So to avoid storing Kubernetes secrets in the plain text format it is a recommended practice to use base64 encoding.

3.1 Encode username and password

Here is an example on how to do the base64 encoding of plain text -

1$ echo -n ‘test-user’ | base64
24oCYdGVzdC11c2Vy4oCZ

Similarly we can use the same base64 encoding command to encode the password-

1$ echo -n 'testP@ssword' | base64
2dGVzdFBAc3N3b3Jk

3.2 Create test-secret.yaml using base64 encoded username and password

After encoding the username and password we need to create a kubernetes secrets manifest with the name test-secret.yaml which will hold the encoded values of username as well as password.

Here is test-secret.yaml

1apiVersion: v1
2kind: Secret
3metadata:
4  name: mysql-secret
5type: kubernetes.io/basic-auth
6stringData:
7  username: 4oCYdGVzdC11c2Vy4oCZ
8  password: dGVzdFBAc3N3b3Jk

3.3 Apply secrets

You can apply the kubernetes secrets using the command - $ kubectl apply -f test-secrets.yaml

1$ kubectl apply -f test-secrets.yaml
2secret/mysql-secret created

3.4 Verify the secretes

After applying the kubernetes secrets you can verify it by using the following kubectl describe secrets command.

 1$ kubectl describe secrets mysql-secret
 2Name:         mysql-secret
 3Namespace:    default
 4Labels:       <none>
 5Annotations:  <none>
 6
 7Type:  kubernetes.io/basic-auth
 8
 9Data
10====
11password:  16 bytes
12username:  20 bytes

4. Create Kubernetes secrets from files

The other way to create Kubernetes secrets would be to store the secrets in a file and reference those files to create Kubernetes secrets.

In the previous steps, we have used an example in which we have created the Kubernetes secrets for username and password using the plain text and also using base64 encoding. But Kubernetes secrets provide one more feature which helps you to store the secrets in a simple text file.

4.1 Create files to store username and password

Let’s create a file named username.txt and password.txt. Copy the username and password into those files.

Here are the commands to creating the files -

1vi username.txt

Copy the username .i.e. test-username and save the file(if you are using the vi editor then simple press the esc key and then type :wq and hit enter)

Use the same approach for creating the password.txt and store the plain text password into it.

4.2 Create kubernetes secrets by referencing the username.txt and password.txt file

Now we have created the username.txt as well as password.txt file, let’s apply those -

1kubectl create secret generic test-secret --from-file=username.txt  --from-file=password.txt

(*Note - Always remember to put the relative file path)

Verify the secrets -

 1kubectl describe secret test-secret
 2Name:         test-secret
 3Namespace:    default
 4Labels:       <none>
 5Annotations:  <none>
 6
 7Type:  Opaque
 8
 9Data
10====
11password.txt:  13 bytes
12username.txt:  10 bytes

5. Use the kubernetes secrets inside your POD or deployment manifest

In previous steps from 1-4, we have seen different ways to create Kubernetes secrets. But now the next question is about using the Kubernetes secrets which we have created.

We can use Kubernetes secrets inside the Kubernetes POD as well as into the Kubernetes deployment manifest as well.

5.1 Use kubernetes secrets inside kubernetes POD

Let’s create a busybox pod yaml in which we are going to use kubernetes secrets .i.e. Test-secret -

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: busybox
 5spec:
 6  containers:
 7  - image: busybox
 8    name: busybox
 9    command: ["/bin/sh"]
10    args: ["-c", "sleep 600"]
11    env:
12    - name: myusername
13      valueFrom:
14        secretKeyRef:
15          name: test-secret
16          key: username

You can save the above POD configuration with the name - test-pod.yaml

Apply the POD configuration using the command -

1$ kubectl apply -f test-pod.yaml
2pod/busybox created

You can verify the pod by running the kubectl describe command

 1$ kubectl describe pod busybox
 2Name:         busybox
 3Namespace:    default
 4Priority:     0
 5Node:         node1/100.0.0.2
 6Start Time:   Sun, 07 Nov 2021 21:08:54 +0000
 7Labels:       <none>
 8Annotations:  cni.projectcalico.org/containerID: 758e309190cc12fbc5c66bd379c4e05808a8e6c5e4c4d601940d1dfa44689354
 9cni.projectcalico.org/podIP: 10.233.90.10/32
10cni.projectcalico.org/podIPs: 10.233.90.10/32
11Status:       Running
12IP:           10.233.90.10
13IPs:
14IP:  10.233.90.10
15Containers:
16busybox:
17Container ID:  docker://d9f72b422ab8fc2605787b7419956547e9726607e8a8a9a1c7e0d5124b779e3f
18Image:         busybox
19Image ID:      docker-pullable://busybox@sha256:15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f
20Port:          <none>
21Host Port:     <none>
22Command:
23/bin/sh
24Args:
25-c
26sleep 600
27State:          Running
28Started:      Sun, 07 Nov 2021 21:08:57 +0000
29Ready:          True
30Restart Count:  0
31Environment:
32myusername:  <set to the key 'username' in secret 'test-secret'>  Optional: false
33Mounts:
34/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kdpf8 (ro)

You should pay attention to the status logs of the busybox POD, you should see the following environment message stating your kubernetes secret reference.

5.2 Using the kubernetes secrets inside the deployment manifest

In the previous point 5.1 we have seen how to use kubernetes secrets inside the kubernetes POD. But apart from POD you can also use the kubernetes secrets inside the deployment manifest.

Here is an example of a mysql deployment manifest in which we are going to deploy a mysql docker container inside the kubernetes cluster. A typical mysql database setup always contains username and password. So first create a secret to store password -

1apiVersion: v1
2kind: Secret
3metadata:
4  name: mysql-test-secret
5type: kubernetes.io/basic-auth
6stringData:
7  password: test1234

Store the above file with the name test-mysql-secret.yaml

How to deploy a secret in Kubernetes?

To deploy the secret you simply need to run the kubectl apply command and supply the secret yaml.

1$ kubectl apply -f test-mysql-secret.yaml
2secret/mysql-test-secret created

1$ kubectl get secret mysql-test-secret
2NAME           TYPE                       DATA   AGE
3mysql-test-secret   kubernetes.io/basic-auth   1      17s

5.3 Create a deployment manifest for mysql using the mysql-test-secret?

Now we have created mysql-test-secret, let’s create a deployment manifest for mysql using the same secrets.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: mysql
 5spec:
 6  selector:
 7    matchLabels:
 8      app: mysql
 9  strategy:
10    type: Recreate
11  template:
12    metadata:
13      labels:
14        app: mysql
15    spec:
16      containers:
17        - image: mysql:5.6
18          name: mysql
19          env:
20            - name: MYSQL_ROOT_PASSWORD
21              valueFrom:
22                secretKeyRef:
23                  name: mysql-test-secret
24                  key: password
25          ports:
26            - containerPort: 3306
27              name: mysql
28          volumeMounts:
29            - name: mysql-persistent-storage
30              mountPath: /home/vagrant/storage
31      volumes:
32        - name: mysql-persistent-storage
33          persistentVolumeClaim:
34            claimName: test-pvc

Apply the above configuration by running the command -

1$ kubectl apply -f mysql-deployment.yaml
2deployment.apps/mysql created

Verify the the POD also -

1$ kubectl get pod
2NAME                     READY   STATUS    RESTARTS         AGE
3busybox                  1/1     Running   16 (6m55s ago)   11h
4mysql-686cd9b97c-k6hqm   1/1     Running   0                8s

 1$ kubectl describe pod mysql-686cd9b97c-k6hqm
 2Name:         mysql-686cd9b97c-k6hqm
 3Namespace:    default
 4Priority:     0
 5Node:         node1/100.0.0.2
 6Start Time:   Mon, 08 Nov 2021 08:51:12 +0000
 7Labels:       app=mysql
 8pod-template-hash=686cd9b97c
 9Annotations:  cni.projectcalico.org/containerID: 1ab3fe1dd57eddb3cf14a6e492d720e98263c36ce5e10bad620a886abafa244d
10cni.projectcalico.org/podIP: 10.233.90.11/32
11cni.projectcalico.org/podIPs: 10.233.90.11/32
12Status:       Running
13IP:           10.233.90.11
14IPs:
15IP:           10.233.90.11
16Controlled By:  ReplicaSet/mysql-686cd9b97c
17Containers:
18mysql:
19Container ID:   docker://7f317c925e1c25e96b02c7662a4467e320273d17b270cc372577808d0c502827
20Image:          mysql:5.6
21Image ID:       docker-pullable://mysql@sha256:cdb7b3a69c0f36ce61dda653cdbe1bf086b6a98c1bf6fa023f7a37bc8325dc98
22Port:           3306/TCP
23Host Port:      0/TCP
24State:          Running
25Started:      Mon, 08 Nov 2021 08:51:14 +0000
26Ready:          True
27Restart Count:  0
28Environment:
29MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'mysql-test-secret'>  Optional: false
30Mounts:
31/home/vagrant/storage from mysql-persistent-storage (rw)
32/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhs62 (ro)
33Conditions:
34Type              Status
35Initialized       True
36Ready             True
37ContainersReady   True
38PodScheduled      True
39Volumes:
40mysql-persistent-storage:
41Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
42ClaimName:  jhooq-pvc
43ReadOnly:   false
44kube-api-access-lhs62:
45Type:                    Projected (a volume that contains injected data from multiple sources)
46TokenExpirationSeconds:  3607
47ConfigMapName:           kube-root-ca.crt
48ConfigMapOptional:       <nil>
49DownwardAPI:             true
50QoS Class:                   BestEffort
51Node-Selectors:              <none>
52Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
53node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
54Events:
55Type    Reason     Age    From               Message
56  ----    ------     ----   ----               -------
57Normal  Scheduled  4m29s  default-scheduler  Successfully assigned default/mysql-686cd9b97c-k6hqm to node1
58Normal  Pulled     4m28s  kubelet            Container image "mysql:5.6" already present on machine
59Normal  Created    4m28s  kubelet            Created container mysql
60Normal  Started    4m27s  kubelet            Started container mysql

6. Conclusion

Kubernetes provides many type of secrets, it depends on the needs of an end user. But kubernetes secrets are loosely coupled objects which can be managed separately and does not interfere with POD and deployment operations. Although if you have not created the kubernetes secret and you are trying to use it under the POD or deployment manifest then it will throw an error CreateContainerConfigError.

The one important recommendation while working with the kubernetes secret is you always have to encrypt the secrets and do not store the secrets in the plain text.

Learn more On Kubernetes -

1. Setup kubernetes on Ubuntu
2. Setup Kubernetes on CentOs
3. Setup HA Kubernetes Cluster with Kubespray
4. Setup HA Kubernetes with Minikube
5. Setup Kubernetes Dashboard for local kubernetes cluster
6. Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
7. How to use Persistent Volume and Persistent Volume Claims in Kubernetes
8. Deploy Spring Boot Microservice on local Kubernetes cluster
9. Deploy Spring Boot Microservice on Cloud Platform(GCP)
10. Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
11. CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
12. kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
13. How to setup kubernetes jenkins pipeline on AWS?
14. Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
15. How to fix kubernetes pods getting recreated?
16. How to delete all kubernetes PODS?
17. How to use Kubernetes secrets?
18. Share kubernetes secrets between namespaces?
19. How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
20. Delete Kubernetes POD stuck in terminating state?