How to use Kubernetes secrets?
Before we jump onto kubernetes secrets let’s first try to understand What is Secret? - Secret is sensitive information which can not be written in a plain text or shared with others. The best example of Secrets would be the password.
But Why do we need to use Kubernetes Secret? - Running a bare minimum kubernetes cluster does need a kubernetes secret but when you start deploying a docker container application then there is need of Kubernetes Secret for storing sensitive data.
One of the simplest examples of kubernetes secrets would be running mysql container images inside the kubernetes cluster. As you now mysql is a database and to access the database we need username, password and since we are running the mysql container inside the kubernetes cluster so we need to store those credentials(Username, Password) somewhere inside the kubernetes cluster and it should be safe enough, so for storing mysql username and password we are going to use Kubernetes secrets.
This blog is a comprehensive guide on Managing Kubernetes secrets and we are going to touch upon following topics -
- How does Kubernetes Handle the Secrets?
- Create Kubernetes secrets using kubectl command
- Create base64 encoded kubernetes secrets
- Create Kubernetes secrets from files
- Use the kubernetes secrets inside your POD or deployment manifest
- Conclusion
1. How does Kubernetes Handels the secrets?
When you work with Kubernetes you always run multiple docker containers but one thing Kubernetes is good at is running the container in an ephemeral mode so that each container is independent of the other and does not share any resources with each other.
So if there is a need of storing sensitive information by docker container then Kubernetes secrets are created. But Kubernetes secrets are independent objects and are not bound to any Kubernetes POD or docker container.
Since Docker containers are running in the ephemeral mode, we need to make the Kubernetes secrets available to the deployment manifest of your docker container.
As a general practice, we always create Kubernetes secrets independently, and the secrets can only be accessed by the PODS.
2. Create Kubernetes secrets using kubectl and --from-literal
One of the easiest ways to create the Kubernetes secret is by using the kubectl command and --from-literal flag. Let’s take an example to understand Kubernetes secret creation. In this example, we need three things -
- secret-name - test-secret
- username - test-user
- password - testP@ssword
2.1 Here is the command for creating kubernetes secret -
1kubectl create secret generic test-secret --from-literal=username=test-user --from-literal=password=testP@ssword
2secret/test-secret created
2.2 Verify the secret using the following command -
1$ kubectl get secret test-secret
2NAME TYPE DATA AGE
3test-secret Opaque 2 11s
2.3 Check the secret using the kubectl describe
Now we know that the Kubernetes secret has been created. Let's check a few more details by running the kubectl describe command. This command can show the secrets attribute like username and password but it will not show the sensitive details.
1$ kubectl describe secret test-secret
2Name: test-secret
3Namespace: default
4Labels: <none>
5Annotations: <none>
6
7Type: Opaque
8
9Data
10====
11password: 22 bytes
12username: 9 bytes
2.4 Copy kubernetes secret between the namespaces
If there is need to copy or share kubernetes secrets between multiple workspace then I would recommend reading this guide on - Share kubernetes secrets between namespaces
3. Create base64 encoded kubernetes secrets
In the previous step, we have seen how to create kubernetes secrets in the plain text format which is okay but plain text secrets are not recommended practice. You should always store the secrets in such a way that they can not be viewed as plain text.
So to avoid storing Kubernetes secrets in the plain text format it is a recommended practice to use base64 encoding.
3.1 Encode username and password
Here is an example on how to do the base64 encoding of plain text -
1$ echo -n ‘test-user’ | base64
24oCYdGVzdC11c2Vy4oCZ
Similarly we can use the same base64 encoding command to encode the password-
1$ echo -n 'testP@ssword' | base64
2dGVzdFBAc3N3b3Jk
3.2 Create test-secret.yaml using base64 encoded username and password
After encoding the username and password we need to create a kubernetes secrets manifest with the name test-secret.yaml which will hold the encoded values of username as well as password.
Here is test-secret.yaml
1apiVersion: v1
2kind: Secret
3metadata:
4 name: mysql-secret
5type: kubernetes.io/basic-auth
6stringData:
7 username: 4oCYdGVzdC11c2Vy4oCZ
8 password: dGVzdFBAc3N3b3Jk
3.3 Apply secrets
You can apply the kubernetes secrets using the command - $ kubectl apply -f test-secrets.yaml
1$ kubectl apply -f test-secrets.yaml
2secret/mysql-secret created
3.4 Verify the secretes
After applying the kubernetes secrets you can verify it by using the following kubectl describe secrets command.
1$ kubectl describe secrets mysql-secret
2Name: mysql-secret
3Namespace: default
4Labels: <none>
5Annotations: <none>
6
7Type: kubernetes.io/basic-auth
8
9Data
10====
11password: 16 bytes
12username: 20 bytes
4. Create Kubernetes secrets from files
The other way to create Kubernetes secrets would be to store the secrets in a file and reference those files to create Kubernetes secrets.
In the previous steps, we have used an example in which we have created the Kubernetes secrets for username and password using the plain text and also using base64 encoding. But Kubernetes secrets provide one more feature which helps you to store the secrets in a simple text file.
4.1 Create files to store username and password
Let’s create a file named username.txt and password.txt. Copy the username and password into those files.
Here are the commands to creating the files -
1vi username.txt
Copy the username .i.e. test-username and save the file(if you are using the vi editor then simple press the esc key and then type :wq and hit enter)
Use the same approach for creating the password.txt and store the plain text password into it.
4.2 Create kubernetes secrets by referencing the username.txt and password.txt file
Now we have created the username.txt as well as password.txt file, let’s apply those -
1kubectl create secret generic test-secret --from-file=username.txt --from-file=password.txt
(*Note - Always remember to put the relative file path)
Verify the secrets -
1kubectl describe secret test-secret
2Name: test-secret
3Namespace: default
4Labels: <none>
5Annotations: <none>
6
7Type: Opaque
8
9Data
10====
11password.txt: 13 bytes
12username.txt: 10 bytes
5. Use the kubernetes secrets inside your POD or deployment manifest
In previous steps from 1-4, we have seen different ways to create Kubernetes secrets. But now the next question is about using the Kubernetes secrets which we have created.
We can use Kubernetes secrets inside the Kubernetes POD as well as into the Kubernetes deployment manifest as well.
5.1 Use kubernetes secrets inside kubernetes POD
Let’s create a busybox pod yaml in which we are going to use kubernetes secrets .i.e. Test-secret -
1apiVersion: v1
2kind: Pod
3metadata:
4 name: busybox
5spec:
6 containers:
7 - image: busybox
8 name: busybox
9 command: ["/bin/sh"]
10 args: ["-c", "sleep 600"]
11 env:
12 - name: myusername
13 valueFrom:
14 secretKeyRef:
15 name: test-secret
16 key: username
You can save the above POD configuration with the name - test-pod.yaml
Apply the POD configuration using the command -
1$ kubectl apply -f test-pod.yaml
2pod/busybox created
You can verify the pod by running the kubectl describe command
1$ kubectl describe pod busybox
2Name: busybox
3Namespace: default
4Priority: 0
5Node: node1/100.0.0.2
6Start Time: Sun, 07 Nov 2021 21:08:54 +0000
7Labels: <none>
8Annotations: cni.projectcalico.org/containerID: 758e309190cc12fbc5c66bd379c4e05808a8e6c5e4c4d601940d1dfa44689354
9cni.projectcalico.org/podIP: 10.233.90.10/32
10cni.projectcalico.org/podIPs: 10.233.90.10/32
11Status: Running
12IP: 10.233.90.10
13IPs:
14IP: 10.233.90.10
15Containers:
16busybox:
17Container ID: docker://d9f72b422ab8fc2605787b7419956547e9726607e8a8a9a1c7e0d5124b779e3f
18Image: busybox
19Image ID: docker-pullable://busybox@sha256:15e927f78df2cc772b70713543d6b651e3cd8370abf86b2ea4644a9fba21107f
20Port: <none>
21Host Port: <none>
22Command:
23/bin/sh
24Args:
25-c
26sleep 600
27State: Running
28Started: Sun, 07 Nov 2021 21:08:57 +0000
29Ready: True
30Restart Count: 0
31Environment:
32myusername: <set to the key 'username' in secret 'test-secret'> Optional: false
33Mounts:
34/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kdpf8 (ro)
You should pay attention to the status logs of the busybox POD, you should see the following environment message stating your kubernetes secret reference.
5.2 Using the kubernetes secrets inside the deployment manifest
In the previous point 5.1 we have seen how to use kubernetes secrets inside the kubernetes POD. But apart from POD you can also use the kubernetes secrets inside the deployment manifest.
Here is an example of a mysql deployment manifest in which we are going to deploy a mysql docker container inside the kubernetes cluster. A typical mysql database setup always contains username and password. So first create a secret to store password -
1apiVersion: v1
2kind: Secret
3metadata:
4 name: mysql-test-secret
5type: kubernetes.io/basic-auth
6stringData:
7 password: test1234
Store the above file with the name test-mysql-secret.yaml
How to deploy a secret in Kubernetes?
To deploy the secret you simply need to run the kubectl apply command and supply the secret yaml.
1$ kubectl apply -f test-mysql-secret.yaml
2secret/mysql-test-secret created
1$ kubectl get secret mysql-test-secret
2NAME TYPE DATA AGE
3mysql-test-secret kubernetes.io/basic-auth 1 17s
5.3 Create a deployment manifest for mysql using the mysql-test-secret?
Now we have created mysql-test-secret, let’s create a deployment manifest for mysql using the same secrets.
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: mysql
5spec:
6 selector:
7 matchLabels:
8 app: mysql
9 strategy:
10 type: Recreate
11 template:
12 metadata:
13 labels:
14 app: mysql
15 spec:
16 containers:
17 - image: mysql:5.6
18 name: mysql
19 env:
20 - name: MYSQL_ROOT_PASSWORD
21 valueFrom:
22 secretKeyRef:
23 name: mysql-test-secret
24 key: password
25 ports:
26 - containerPort: 3306
27 name: mysql
28 volumeMounts:
29 - name: mysql-persistent-storage
30 mountPath: /home/vagrant/storage
31 volumes:
32 - name: mysql-persistent-storage
33 persistentVolumeClaim:
34 claimName: test-pvc
Apply the above configuration by running the command -
1$ kubectl apply -f mysql-deployment.yaml
2deployment.apps/mysql created
Verify the the POD also -
1$ kubectl get pod
2NAME READY STATUS RESTARTS AGE
3busybox 1/1 Running 16 (6m55s ago) 11h
4mysql-686cd9b97c-k6hqm 1/1 Running 0 8s
1$ kubectl describe pod mysql-686cd9b97c-k6hqm
2Name: mysql-686cd9b97c-k6hqm
3Namespace: default
4Priority: 0
5Node: node1/100.0.0.2
6Start Time: Mon, 08 Nov 2021 08:51:12 +0000
7Labels: app=mysql
8pod-template-hash=686cd9b97c
9Annotations: cni.projectcalico.org/containerID: 1ab3fe1dd57eddb3cf14a6e492d720e98263c36ce5e10bad620a886abafa244d
10cni.projectcalico.org/podIP: 10.233.90.11/32
11cni.projectcalico.org/podIPs: 10.233.90.11/32
12Status: Running
13IP: 10.233.90.11
14IPs:
15IP: 10.233.90.11
16Controlled By: ReplicaSet/mysql-686cd9b97c
17Containers:
18mysql:
19Container ID: docker://7f317c925e1c25e96b02c7662a4467e320273d17b270cc372577808d0c502827
20Image: mysql:5.6
21Image ID: docker-pullable://mysql@sha256:cdb7b3a69c0f36ce61dda653cdbe1bf086b6a98c1bf6fa023f7a37bc8325dc98
22Port: 3306/TCP
23Host Port: 0/TCP
24State: Running
25Started: Mon, 08 Nov 2021 08:51:14 +0000
26Ready: True
27Restart Count: 0
28Environment:
29MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-test-secret'> Optional: false
30Mounts:
31/home/vagrant/storage from mysql-persistent-storage (rw)
32/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lhs62 (ro)
33Conditions:
34Type Status
35Initialized True
36Ready True
37ContainersReady True
38PodScheduled True
39Volumes:
40mysql-persistent-storage:
41Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
42ClaimName: jhooq-pvc
43ReadOnly: false
44kube-api-access-lhs62:
45Type: Projected (a volume that contains injected data from multiple sources)
46TokenExpirationSeconds: 3607
47ConfigMapName: kube-root-ca.crt
48ConfigMapOptional: <nil>
49DownwardAPI: true
50QoS Class: BestEffort
51Node-Selectors: <none>
52Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
53node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
54Events:
55Type Reason Age From Message
56 ---- ------ ---- ---- -------
57Normal Scheduled 4m29s default-scheduler Successfully assigned default/mysql-686cd9b97c-k6hqm to node1
58Normal Pulled 4m28s kubelet Container image "mysql:5.6" already present on machine
59Normal Created 4m28s kubelet Created container mysql
60Normal Started 4m27s kubelet Started container mysql
6. Conclusion
Kubernetes provides many type of secrets, it depends on the needs of an end user. But kubernetes secrets are loosely coupled objects which can be managed separately and does not interfere with POD and deployment operations. Although if you have not created the kubernetes secret and you are trying to use it under the POD or deployment manifest then it will throw an error CreateContainerConfigError.
The one important recommendation while working with the kubernetes secret is you always have to encrypt the secrets and do not store the secrets in the plain text.
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?
Posts in this Series
- Kubernetes Cheat Sheet for day to day DevOps operations?
- Delete Kubernetes POD stuck in terminating state?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Share kubernetes secrets between namespaces?
- How to use Kubernetes secrets?
- How to delete all kubernetes PODS?
- kubernetes pods getting recreated?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- kubectl export yaml OR How to generate YAML for deployed kubernetes resources
- Kubernetes Updates
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- Kubernetes cluster setup with Jenkins
- How to use Persistent Volume and Persistent Claims | Kubernetes
- How to fix ProvisioningFailed persistentvolume controller no volume plugin matched
- Fixing – Cannot bind to requested volume: storageClasseName does not match
- Fixing – pod has unbound immediate persistentvolumeclaims or cannot bind to requested volume incompatible accessmode
- How to fix kubernetes dashboard forbidden 403 error – message services https kubernetes-dashboard is forbidden User
- How to fix Kubernetes – error execution phase preflight [preflight]
- Deploy Spring Boot microservices on kubernetes?
- How to fix – ansible_memtotal_mb minimal_master_memory_mb
- How to use kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster
- How to setup kubernetes on CentOS 8 and CentOS 7
- How to fix – How to fix - ERROR Swap running with swap on is not supported. Please disable swap
- 14 Steps to Install kubernetes on Ubuntu 20.04(bento/ubuntu-20.04), 18.04(hashicorp/bionic64)
- Kubernetes Dashboard | Kubernetes Admin GUI | Kubernetes Desktop Client
- Install Kubernetes with Minikube