How to fix - nginx ingress controller crashloopbackoff error
I faced this issue while I was trying to install the nginx ingress controller along with HAproxy loadbalancer on my kuberentes cluster.
To be honest I had this issue because I missed to create config map during the setup.
But there could be many reasons behind this issue. We will take each possible scenario and look at how to fix this issue.
1. nginx ingress controller crashloopbackoff error exit code 255
This was the first error which I got when I was trying to setup the HAProxy loadbalancer along with Nginx ingress controller on my kubernetes cluster.
After spending a couple of days scratching my head I ended up creating an Issue on the GitHub for the help
But later it turn out to be my mistake because I missed to create config map while setting it up.
To be precise I missed the following command -
1$ kubectl apply -f deployment/nginx-ingress.yaml</code></pre>
But as i said there could be many reasons for this error.
So How would you find the exact error causing this issue ?
1. First and foremost you need to check all the pods running inside the namespace nginx-ingress
1$ kubectl get all -n nginx-ingress
It should return something similar
1NAME READY STATUS RESTARTS AGE 2pod/nginx-ingress-86r84 0/1 CrashLoopBackOff 9 25m 3pod/nginx-ingress-zb5xs 0/1 CrashLoopBackOff 9 25m 4 5NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE 6daemonset.apps/nginx-ingress 2 2 0 2 0 <none> 25m 7
Once you see “CrashLoopBackOff” in the status column then for sure you messed up something during your setup.
2. The next step I would advice you to check for the pod status using the following command (please replace the pod name in the command)
1$ kubectl describe -n nginx-ingress pod nginx-ingress-86r84
It should return you the following output which is -
1Name: nginx-ingress-86r84 2Namespace: nginx-ingress 3Priority: 0 4Node: node2/188.8.131.52 5Start Time: Thu, 06 Aug 2020 18:38:41 +0000 6Labels: app=nginx-ingress 7 controller-revision-hash=f76b6fd76 8 pod-template-generation=1 9Annotations: <none> 10Status: Running 11IP: 10.233.96.3 12IPs: 13 IP: 10.233.96.3 14Controlled By: DaemonSet/nginx-ingress 15Containers: 16 nginx-ingress: 17 Container ID: docker://dd6b6fa3000024a657f2488bce64072570eded26019c4cdeb919b3b926047f6b 18 Image: nginx/nginx-ingress:edge 19 Image ID: docker-pullable://nginx/nginx-ingress@sha256:078686035bf3a43f5ad85c7338738eaf582680cb017ede5c72327758e8107658 20 Ports: 80/TCP, 443/TCP, 8081/TCP 21 Host Ports: 80/TCP, 443/TCP, 0/TCP 22 Args: 23 -nginx-configmaps=$(POD_NAMESPACE)/nginx-config 24 -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret 25 State: Waiting 26 Reason: CrashLoopBackOff 27 Last State: Terminated 28 Reason: Error 29 Exit Code: 255 30 Started: Thu, 06 Aug 2020 18:55:05 +0000 31 Finished: Thu, 06 Aug 2020 18:55:05 +0000 32 Ready: False 33 Restart Count: 8 34 Readiness: http-get http://:readiness-port/nginx-ready delay=0s timeout=1s period=1s #success=1 #failure=3 35 Environment: 36 POD_NAMESPACE: nginx-ingress (v1:metadata.namespace) 37 POD_NAME: nginx-ingress-86r84 (v1:metadata.name) 38 Mounts: 39 /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-token-42dtg (ro) 40Conditions: 41 Type Status 42 Initialized True 43 Ready False 44 ContainersReady False 45 PodScheduled True 46Volumes: 47 nginx-ingress-token-42dtg: 48 Type: Secret (a volume populated by a Secret) 49 SecretName: nginx-ingress-token-42dtg 50 Optional: false 51QoS Class: BestEffort 52Node-Selectors: <none> 53Tolerations: node.kubernetes.io/disk-pressure:NoSchedule 54 node.kubernetes.io/memory-pressure:NoSchedule 55 node.kubernetes.io/not-ready:NoExecute 56 node.kubernetes.io/pid-pressure:NoSchedule 57 node.kubernetes.io/unreachable:NoExecute 58 node.kubernetes.io/unschedulable:NoSchedule 59Events: 60 Type Reason Age From Message 61 ---- ------ ---- ---- ------- 62 Normal Scheduled <unknown> default-scheduler Successfully assigned nginx-ingress/nginx-ingress-86r84 to node2 63 Normal Pulled 20m (x4 over 21m) kubelet, node2 Successfully pulled image "nginx/nginx-ingress:edge" 64 Normal Created 20m (x4 over 21m) kubelet, node2 Created container nginx-ingress 65 Normal Started 20m (x4 over 21m) kubelet, node2 Started container nginx-ingress 66 Normal Pulling 19m (x5 over 21m) kubelet, node2 Pulling image "nginx/nginx-ingress:edge" 67 Warning BackOff 57s (x94 over 20m) kubelet, node2 Back-off restarting failed container
3. Well do not get scared because you need to read the logs carefully here and look for the State: Waiting, Reason: CrashLoopBackOff, Reason: Error, Exit Code: 255
1 State: Waiting 2 Reason: CrashLoopBackOff 3 Last State: Terminated 4 Reason: Error 5 Exit Code: 255
4. Now we need to check the logs of ingress controller pod
1$ kubectl logs nginx-ingress-tt6kc -n nginx-ingress
1I0807 11:37:35.562704 1 main.go:226] Starting NGINX Ingress controller Version=edge GitCommit=00618a60 2F0807 11:37:35.583599 1 main.go:423] Error when getting nginx-ingress/nginx-config: configmaps "nginx-config" not found
As it says in the logs “Error when getting nginx-ingress/nginx-config: configmaps “nginx-config” not found”
Which means you did not setup the nginx-ingress/nginx-config and it can be fixed by -
1$ kubectl apply -f deployments/common/nginx-config.yaml
But running this command alone will not fix this issue, it had to be run with proper sequence. I would highly recommend following this guide - https://jhooq.com/ingress-controller-nginx/
I hope this would help you to fix the CrashLoopBackOff exit code 255 issue
Other articles on Devops -
- Part-1 : Setup kubernetes on Ubuntu
- Part-2 : Setup Kubernetes on CentOs
- Part-3 : Setup HA Kubernetes Cluster with Kubespray
- Part-4 : Setup HA Kubernetes with Minikube
- Part-5 : Setup Kubernetes Dashboard for local kubernetes cluster
- Part-6 : Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- Part-7 : How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Part-8 : Deploy Spring Boot Microservice on local Kubernetes cluster
- Part-9 : Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Part-10 : Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- Part-11 : CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes