How to fix - nginx ingress controller crashloopbackoff error
I faced this issue while I was trying to install the nginx ingress controller along with HAproxy loadbalancer on my kuberentes cluster.
To be honest I had this issue because I missed to create config map during the setup.
But there could be many reasons behind this issue. We will take each possible scenario and look at how to fix this issue.
1. nginx ingress controller crashloopbackoff error exit code 255
This was the first error which I got when I was trying to setup the HAProxy loadbalancer along with Nginx ingress controller on my kubernetes cluster.
After spending a couple of days scratching my head I ended up creating an Issue on the GitHub for the help
But later it turn out to be my mistake because I missed to create config map while setting it up.
To be precise I missed the following command -
1$ kubectl apply -f deployment/nginx-ingress.yaml</code></pre>
But as i said there could be many reasons for this error.
So How would you find the exact error causing this issue ?
1. First and foremost you need to check all the pods running inside the namespace nginx-ingress
1$ kubectl get all -n nginx-ingress
It should return something similar
1NAME READY STATUS RESTARTS AGE
2pod/nginx-ingress-86r84 0/1 CrashLoopBackOff 9 25m
3pod/nginx-ingress-zb5xs 0/1 CrashLoopBackOff 9 25m
4
5NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
6daemonset.apps/nginx-ingress 2 2 0 2 0 <none> 25m
Once you see "CrashLoopBackOff" in the status column then for sure you messed up something during your setup.
2. The next step I would advice you to check for the pod status using the following command (please replace the pod name in the command)
1$ kubectl describe -n nginx-ingress pod nginx-ingress-86r84
It should return you the following output which is -
1Name: nginx-ingress-86r84
2Namespace: nginx-ingress
3Priority: 0
4Node: node2/100.0.0.3
5Start Time: Thu, 06 Aug 2020 18:38:41 +0000
6Labels: app=nginx-ingress
7 controller-revision-hash=f76b6fd76
8 pod-template-generation=1
9Annotations: <none>
10Status: Running
11IP: 10.233.96.3
12IPs:
13 IP: 10.233.96.3
14Controlled By: DaemonSet/nginx-ingress
15Containers:
16 nginx-ingress:
17 Container ID: docker://dd6b6fa3000024a657f2488bce64072570eded26019c4cdeb919b3b926047f6b
18 Image: nginx/nginx-ingress:edge
19 Image ID: docker-pullable://nginx/nginx-ingress@sha256:078686035bf3a43f5ad85c7338738eaf582680cb017ede5c72327758e8107658
20 Ports: 80/TCP, 443/TCP, 8081/TCP
21 Host Ports: 80/TCP, 443/TCP, 0/TCP
22 Args:
23 -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
24 -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
25 State: Waiting
26 Reason: CrashLoopBackOff
27 Last State: Terminated
28 Reason: Error
29 Exit Code: 255
30 Started: Thu, 06 Aug 2020 18:55:05 +0000
31 Finished: Thu, 06 Aug 2020 18:55:05 +0000
32 Ready: False
33 Restart Count: 8
34 Readiness: http-get http://:readiness-port/nginx-ready delay=0s timeout=1s period=1s #success=1 #failure=3
35 Environment:
36 POD_NAMESPACE: nginx-ingress (v1:metadata.namespace)
37 POD_NAME: nginx-ingress-86r84 (v1:metadata.name)
38 Mounts:
39 /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-token-42dtg (ro)
40Conditions:
41 Type Status
42 Initialized True
43 Ready False
44 ContainersReady False
45 PodScheduled True
46Volumes:
47 nginx-ingress-token-42dtg:
48 Type: Secret (a volume populated by a Secret)
49 SecretName: nginx-ingress-token-42dtg
50 Optional: false
51QoS Class: BestEffort
52Node-Selectors: <none>
53Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
54 node.kubernetes.io/memory-pressure:NoSchedule
55 node.kubernetes.io/not-ready:NoExecute
56 node.kubernetes.io/pid-pressure:NoSchedule
57 node.kubernetes.io/unreachable:NoExecute
58 node.kubernetes.io/unschedulable:NoSchedule
59Events:
60 Type Reason Age From Message
61 ---- ------ ---- ---- -------
62 Normal Scheduled <unknown> default-scheduler Successfully assigned nginx-ingress/nginx-ingress-86r84
63 to node2
64 Normal Pulled 20m (x4 over 21m) kubelet, node2 Successfully pulled image "nginx/nginx-ingress:edge"
65 Normal Created 20m (x4 over 21m) kubelet, node2 Created container nginx-ingress
66 Normal Started 20m (x4 over 21m) kubelet, node2 Started container nginx-ingress
67 Normal Pulling 19m (x5 over 21m) kubelet, node2 Pulling image "nginx/nginx-ingress:edge"
68 Warning BackOff 57s (x94 over 20m) kubelet, node2 Back-off restarting failed container
3. Well do not get scared because you need to read the logs carefully here and look for the State: Waiting, Reason: CrashLoopBackOff, Reason: Error, Exit Code: 255
1 State: Waiting
2 Reason: CrashLoopBackOff
3 Last State: Terminated
4 Reason: Error
5 Exit Code: 255
4. Now we need to check the logs of ingress controller pod
1$ kubectl logs nginx-ingress-tt6kc -n nginx-ingress
1I0807 11:37:35.562704 1 main.go:226] Starting NGINX Ingress controller Version=edge GitCommit=00618a60
2F0807 11:37:35.583599 1 main.go:423] Error when getting nginx-ingress/nginx-config: configmaps "nginx-config" not found
As it says in the logs "Error when getting nginx-ingress/nginx-config: configmaps "nginx-config" not found"
Which means you did not setup the nginx-ingress/nginx-config and it can be fixed by -
1$ kubectl apply -f deployments/common/nginx-config.yaml
But running this command alone will not fix this issue, it had to be run with proper sequence. I would highly recommend following this guide - https://jhooq.com/ingress-controller-nginx/
I hope this would help you to fix the CrashLoopBackOff exit code 255 issue
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?