How to fix - nginx ingress controller crashloopbackoff error

I faced this issue while I was trying to install the nginx ingress controller along with HAproxy loadbalancer on my kuberentes cluster.

To be honest I had this issue because I missed to create config map during the setup.

But there could be many reasons behind this issue. We will take each possible scenario and look at how to fix this issue.



1. nginx ingress controller crashloopbackoff error exit code 255

This was the first error which I got when I was trying to setup the HAProxy loadbalancer along with Nginx ingress controller on my kubernetes cluster.

After spending a couple of days scratching my head I ended up creating an Issue on the GitHub for the help

But later it turn out to be my mistake because I missed to create config map while setting it up.

To be precise I missed the following command -

1$ kubectl apply -f deployment/nginx-ingress.yaml</code></pre>

But as i said there could be many reasons for this error.

So How would you find the exact error causing this issue ?

1. First and foremost you need to check all the pods running inside the namespace nginx-ingress

1$ kubectl get all -n nginx-ingress

It should return something similar

1NAME                      READY   STATUS             RESTARTS   AGE
2pod/nginx-ingress-86r84   0/1     CrashLoopBackOff   9          25m
3pod/nginx-ingress-zb5xs   0/1     CrashLoopBackOff   9          25m
4
5NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
6daemonset.apps/nginx-ingress   2         2         0       2            0           <none>          25m


Once you see "CrashLoopBackOff" in the status column then for sure you messed up something during your setup.

2. The next step I would advice you to check for the pod status using the following command (please replace the pod name in the command)

1$ kubectl describe -n nginx-ingress pod nginx-ingress-86r84

It should return you the following output which is -

 1Name:         nginx-ingress-86r84
 2Namespace:    nginx-ingress
 3Priority:     0
 4Node:         node2/100.0.0.3
 5Start Time:   Thu, 06 Aug 2020 18:38:41 +0000
 6Labels:       app=nginx-ingress
 7              controller-revision-hash=f76b6fd76
 8              pod-template-generation=1
 9Annotations:  <none>
10Status:       Running
11IP:           10.233.96.3
12IPs:
13  IP:           10.233.96.3
14Controlled By:  DaemonSet/nginx-ingress
15Containers:
16  nginx-ingress:
17    Container ID:  docker://dd6b6fa3000024a657f2488bce64072570eded26019c4cdeb919b3b926047f6b
18    Image:         nginx/nginx-ingress:edge
19    Image ID:      docker-pullable://nginx/nginx-ingress@sha256:078686035bf3a43f5ad85c7338738eaf582680cb017ede5c72327758e8107658
20    Ports:         80/TCP, 443/TCP, 8081/TCP
21    Host Ports:    80/TCP, 443/TCP, 0/TCP
22    Args:
23      -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
24      -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
25    State:          Waiting
26      Reason:       CrashLoopBackOff
27    Last State:     Terminated
28      Reason:       Error
29      Exit Code:    255
30      Started:      Thu, 06 Aug 2020 18:55:05 +0000
31      Finished:     Thu, 06 Aug 2020 18:55:05 +0000
32    Ready:          False
33    Restart Count:  8
34    Readiness:      http-get http://:readiness-port/nginx-ready delay=0s timeout=1s period=1s #success=1 #failure=3
35    Environment:
36      POD_NAMESPACE:  nginx-ingress (v1:metadata.namespace)
37      POD_NAME:       nginx-ingress-86r84 (v1:metadata.name)
38    Mounts:
39      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-token-42dtg (ro)
40Conditions:
41  Type              Status
42  Initialized       True 
43  Ready             False 
44  ContainersReady   False 
45  PodScheduled      True 
46Volumes:
47  nginx-ingress-token-42dtg:
48    Type:        Secret (a volume populated by a Secret)
49    SecretName:  nginx-ingress-token-42dtg
50    Optional:    false
51QoS Class:       BestEffort
52Node-Selectors:  <none>
53Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
54                 node.kubernetes.io/memory-pressure:NoSchedule
55                 node.kubernetes.io/not-ready:NoExecute
56                 node.kubernetes.io/pid-pressure:NoSchedule
57                 node.kubernetes.io/unreachable:NoExecute
58                 node.kubernetes.io/unschedulable:NoSchedule
59Events:
60  Type     Reason     Age                 From               Message
61  ----     ------     ----                ----               -------
62  Normal   Scheduled  <unknown>           default-scheduler  Successfully assigned nginx-ingress/nginx-ingress-86r84
63 to node2
64  Normal   Pulled     20m (x4 over 21m)   kubelet, node2     Successfully pulled image "nginx/nginx-ingress:edge"
65  Normal   Created    20m (x4 over 21m)   kubelet, node2     Created container nginx-ingress
66  Normal   Started    20m (x4 over 21m)   kubelet, node2     Started container nginx-ingress
67  Normal   Pulling    19m (x5 over 21m)   kubelet, node2     Pulling image "nginx/nginx-ingress:edge"
68  Warning  BackOff    57s (x94 over 20m)  kubelet, node2     Back-off restarting failed container


3. Well do not get scared because you need to read the logs carefully here and look for the State: Waiting, Reason: CrashLoopBackOff, Reason: Error, Exit Code: 255

1    State:          Waiting
2    Reason:       CrashLoopBackOff
3    Last State:     Terminated
4    Reason:       Error
5    Exit Code:    255

4. Now we need to check the logs of ingress controller pod

1$ kubectl logs nginx-ingress-tt6kc -n nginx-ingress
1I0807 11:37:35.562704       1 main.go:226] Starting NGINX Ingress controller Version=edge GitCommit=00618a60
2F0807 11:37:35.583599       1 main.go:423] Error when getting nginx-ingress/nginx-config: configmaps "nginx-config" not found

As it says in the logs "Error when getting nginx-ingress/nginx-config: configmaps "nginx-config" not found"

Which means you did not setup the nginx-ingress/nginx-config and it can be fixed by -

1$ kubectl apply -f deployments/common/nginx-config.yaml

But running this command alone will not fix this issue, it had to be run with proper sequence. I would highly recommend following this guide - https://jhooq.com/ingress-controller-nginx/

I hope this would help you to fix the CrashLoopBackOff exit code 255 issue




Learn more On Kubernetes -

  1. Setup kubernetes on Ubuntu
  2. Setup Kubernetes on CentOs
  3. Setup HA Kubernetes Cluster with Kubespray
  4. Setup HA Kubernetes with Minikube
  5. Setup Kubernetes Dashboard for local kubernetes cluster
  6. Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
  7. How to use Persistent Volume and Persistent Volume Claims in Kubernetes
  8. Deploy Spring Boot Microservice on local Kubernetes cluster
  9. Deploy Spring Boot Microservice on Cloud Platform(GCP)
  10. Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
  11. CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
  12. kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
  13. How to setup kubernetes jenkins pipeline on AWS?
  14. Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
  15. How to fix kubernetes pods getting recreated?
  16. How to delete all kubernetes PODS?
  17. How to use Kubernetes secrets?
  18. Share kubernetes secrets between namespaces?
  19. How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
  20. Delete Kubernetes POD stuck in terminating state?

Posts in this Series