How to fix Kubernetes – error execution phase preflight [preflight]

While working with kubernetes somewhere down the line you might have faced this error and there could be many reasons behind the issue error execution phase preflight

With my experience in working with kubernetes I found that this issue mostly occurs when you are trying to setup kubernetes cluster on your own and since you are setting it up by yourself than I can easily guess that you might miss a lot of technicalities and end up with this issue.

Tip - First and far most look your log file very carefully because all the errors starting with error execution phase preflight is not the same. Look for the keyword in the error log such as

• [ERROR DirAvailable]
• [ERROR FileAvailable]
• [ERROR Swap]
• [ERROR IsPrivilegedUser]
• [ERROR NumCPU]
• [ERROR FileContent]

All the above-mentioned errors are different although the error message always starts with error execution phase preflight.

For troubleshooting please follow the individual issue here -

1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
3. [ERROR Swap]: running with swap on is not supported. Please disable swap
4. [ERROR IsPrivilegedUser]: user is not running as root
5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
7. kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
8. kubeadm token
9. CNI(Container Network interface) .i.e. - calico, flannel
10. Firewall settings

1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty

1error execution phase preflight: preflight] Some fatal errors occurred:

1 ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty

In my experience this issue happens when you have deleted the node and trying to re-create it. If you have setup your local kubernetes cluster on your local development machine then it is quite often you will face this issue.

It mostly occurs when you either shutdown or restart and in some cases if you are trying to delete one the kubernetes node.

Solution - You should delete the directory /var/lib/etcd. Use the following command to delete the directory

1$ rm -rf /var/lib/etcd

Generally speaking kubeadm reset should delete this diretory (/var/lib/etcd). But there are many possible cases where this directory or file is locked or do not have permission to delete, so in that cases kubeadm reset will not be successful.

This issue is not so frequent on cloud plateform such GCP(Google Cloud Platform), AWS Cloud(Amazon Web Service) or any other cloud service provider.

2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists {#file-already-exist

This error tells that you need to run the kubeadm reset command before initializing the kubernetes cluster (kubeadm init). In other word you need to properly recover a K8s cluster after reboot

But first let's see the error when I ran the command $ kubeadm reset

1$ kubeadm reset

 1error execution phase preflight: preflight] Some fatal errors occurred:
 2ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: 
 3/etc/kubernetes/manifests/kube-apiserver.yaml already exists
 4ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: 
 5/etc/kubernetes/manifests/kube-controller-manager.yaml already exists
 6ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: 
 7/etc/kubernetes/manifests/kube-scheduler.yaml already exists
 8ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: 
 9/etc/kubernetes/manifests/etcd.yaml already exists
10ERROR FileContent--proc-sys-net-ipv4-ip_forward]:
11 /proc/sys/net/ipv4/ip_forward contents are not set to 1

How to fix and properly recover kubernetes cluster after reboot?

So all the above errors can be fixed with the following steps -

1. Stop kubelet.service - You must stop the Kubelet Service using the command $ sudo systemctl stop kubelet. service
2. Kubeadm reset - Run the kubeadm reset before initializing the kubernetes cluster. The most common cause of this issue is you have already run the kubeadm init command on your kubernetes cluster and now again you are trying to run the same command before doing the kubeadm reset.

4. [ERROR IsPrivilegedUser]: user is not running as root

As the error say "user is not running as root", which mean you are not running kubeadm init command as root user. To fix the issue you should run the kubeadm init as root.

Solution - Add sudo before the running the kubeadm init command.

1$ sudo kubeadm init --apiserver-advertise-address=100.0.0.1 --pod-network-cidr=10.244.0.0/16  

5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

As the error says you are trying to run Kubernetes cluster with 1 CPU which is not sufficient enough to install Kubernetes cluster.

Although [ERROR NumCPU] can be ignored by setting the --ignore-preflight-errors=NumCPU flag in the kube init command but it is not recommended because if you have very limited computing power then running the kubernetes cluster will always be troublesome.

How to increase CPU if you are using Vagrant for installing kubernetes cluster?

If you are using the Vagrant then increase the CPUs count from more than 1.

Here is sample vagrant file which you could refer (look for similar vagrant file which you have with you and update the CPU count in it)

Look for the configuration cpus inside the vagrant file

 1Vagrant.configure("2") do |config|
 2  config.vm.define "amaster" do |amaster|
 3    amaster.vm.box_download_insecure = true
 4    amaster.vm.box = "hashicorp/bionic64"
 5    amaster.vm.network "private_network", ip: "100.0.0.1"
 6    amaster.vm.hostname = "amaster"
 7    amaster.vm.provider "virtualbox" do |v|
 8      v.name = "amaster"
 9      v.memory = 2048
10      v.cpus = 2
11    end
12  end

How to increase CPU if you are using only Virtual Box for installing kubernetes cluster?

In case you are using the Virtual Box then you can go to Virtual Box settings and increase the CPUs, so that it is more than 1.

Please refer to following screenshot -

Increase the CPU if you are using AWS?

It always recommended to use Amazon Elastic Kubernetes Service (Amazon EKS) because Amazon EKS service will take care of CPU and its computing power so you do not have to worry about it. Here is a guide if you do not know how to setup AWS EKS using eksctl

6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

In order to fix this error you need to edit /etc/sysctl.conf and then set the value for /proc/sys/net/bridge/bridge-nf-call-iptables

Here are following command which you can use -

1$ sudo vi /etc/sysctl.conf

After opening the file sysctl.conf in edit mode, add the following line if its not there.

1net.bridge.bridge-nf-call-iptables = 1

Then you need to execute

1$ sudo sysctl -p

The above changed please do the kubeadm reset and then initialize your Kubernetes cluster with kubeadm init command

7. kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

1error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Well if you are facing this issue then I could easily say there is problem with the kublet version and kubelet-config-map.

The above error signifies that you are trying to join kubelet which is 1.15 and which has kublet-config-1.14, so that is not supported at this moment.

But as per the github it should be fixed in 1.16 for more details you can refer to git issue – #1582

8. kubeadm token

The first and far most guess would be to verify your kubeadm token and its validity. Because it quite obvious that you might have generated the token sometimes back and now it has expired.

So if you are still trying to connect to kubernetes master node with expired token than you will end up with the error.

I would recommend you to use following command to verify the kubeadm token.

Check the token validity

1$ kubeadm token list

The above command will return you with all the token which has been generated previously during the kubernetes cluster setup.

Carefully look on the EXPIRES column and see if there are any token which has expired.

How to fix the expired kubeadm token?

kubeadm(kubernetes admin) provides commands for generating/creating new token out of the box, so you can use following command to generate/create new token

1$ kubeadm token create --print-join-command

1kubeadm join 100.0.0.2:6443 --token xg7pkc.atu3g3nlanmajc8j     --discovery-token-ca-cert-hash sha256:d19178104c84f7876602747a3b6e23c1df76f374fe9ba232ab406fd56d0b768e

The above command will create a new fresh token as well as print kubeadm join command, which you can copy and run from any node.

9. CNI(Container Network Interface) - calico, flannel

Well if you think that token validity of your cluster is okay and you do not have any expired token than I would recommend checking the CNI(container network interface) setting within your kubernetes cluster.

But in my experience troubleshooting CNI is needed when you setup kubernetes cluster with VMs(virtual machine) using Vagrant

But anyways this is what happens when you use the CNI either from calico of flannel. There will be two ethernet port on your kubernetes master node.

Use the following command to check the ethernet port

1$ ifconfig

 1eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
 2        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
 3        inet6 fe80::a00:27ff:febb:1475  prefixlen 64  scopeid 0x20<link>
 4        ether 08:00:27:bb:14:75  txqueuelen 1000  (Ethernet)
 5        RX packets 704152  bytes 948370649 (948.3 MB)
 6        RX errors 0  dropped 0  overruns 0  frame 0
 7        TX packets 122139  bytes 7772979 (7.7 MB)
 8        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 9
10eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
11        inet 100.0.0.2  netmask 255.255.255.0  broadcast 100.0.0.255
12        inet6 fe80::a00:27ff:fe36:b458  prefixlen 64  scopeid 0x20<link>
13        ether 08:00:27:36:b4:58  txqueuelen 1000  (Ethernet)
14        RX packets 100821  bytes 64254917 (64.2 MB)
15        RX errors 0  dropped 0  overruns 0  frame 0
16        TX packets 65066  bytes 23079670 (23.0 MB)
17        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And if the command returns with 2 ethernet adapter name .i.e. eth0 and eth1. Then i can easily guess your CNI(container network interface) took the default ethernet adapter .i.e. eth0 and which is wrong.

Ideally it should take the eth1 as its preferred communication channel.

How to fix or setup correct ethernet adapter name for preferred CNI(Container Network Interface) ?

(Note - I tried following fix for the flannel but you can do similar steps for calico or any other CNI)

Look for the kube-flannel.yml inside your kubernetes node. You can search the kube-flannel.yml using following command

1find / -name 'kube-flannel.yml' 2>/dev/null

After that you need to edit the file and and search for the text - flanneld

1$ vi kube-flannel.yml

In the args section add : – –iface=eth1

1- --iface=eth1
2        args:
3        - --ip-masq
4        - --kube-subnet-mgr
5        - --iface=eth1

Apply the flannel configuration

1$ kubectl apply -f kube-flannel.yml

 1podsecuritypolicy.policy/psp.flannel.unprivileged created
 2clusterrole.rbac.authorization.k8s.io/flannel created
 3clusterrolebinding.rbac.authorization.k8s.io/flannel created
 4serviceaccount/flannel created
 5configmap/kube-flannel-cfg created
 6daemonset.apps/kube-flannel-ds-amd64 created
 7daemonset.apps/kube-flannel-ds-arm64 created
 8daemonset.apps/kube-flannel-ds-arm created
 9daemonset.apps/kube-flannel-ds-ppc64le created
10daemonset.apps/kube-flannel-ds-s390x created

10. Firewall settings

This could be one of the potential reason for this error because by default Firewall's are not disabled in any Linux distribution(Ubuntu, Centos, Fedora or RedHat).

The best way for you to check the firewall status before, use the following command for checking the firewall status.

Ubuntu

1$ sudo ufw status

Disable the firewall

1$ sudo ufw disable

But suppose you do not wanna disable the firewall due to security reason than it is highly recommended for you to check the firewall rule or in other words check iptables

Use the following command to verify the iptables

1$ iptables -L

Note - The above command will tell you all the ports which are open for communication. Look for the port 443 because it the most default port for communication inside kubernetes cluster

Now the next question comes in my mind, how to setup a firewall rule or entry in iptables for port 443 inside kubernetes cluster?

Well use the following command for setting up the iptable rules.(Keep in mind the following firewall rules is for the port 443, but if you have any other port then please update the port accordingly)

1$ iptables -A INPUT -p tcp --dport 443 -j ACCEPT

CentOS

The theory is pretty much same for the CentOS also, but the commands are little different when we talk about CentOS.

Use the following command to check the status of the CentOS where your kubernetes cluster is running.

1$ sudo systemctl status firewalld

The above command will tell you about the firewall status.

If the firewall is up and running then i would recommend you to disable the firewall using the following command

1$ sudo systemctl enable firewalld

But just in case you not able to disable the firewall due to security reasons then you need to check the ports which are open using following command

1$ firewall-cmd --list-ports

If you didn't find any entry in the list then please add port in the firewall (change the port number if you have some other ports)

1$ firewall-cmd --permanent --add-port 443/tcp

I hope this firewall troubleshooting will help you to fix the issue "error execution phase preflight" inside your kubernetes cluster.

Learn more On Kubernetes -

1. Setup kubernetes on Ubuntu
2. Setup Kubernetes on CentOs
3. Setup HA Kubernetes Cluster with Kubespray
4. Setup HA Kubernetes with Minikube
5. Setup Kubernetes Dashboard for local kubernetes cluster
6. Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
7. How to use Persistent Volume and Persistent Volume Claims in Kubernetes
8. Deploy Spring Boot Microservice on local Kubernetes cluster
9. Deploy Spring Boot Microservice on Cloud Platform(GCP)
10. Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
11. CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
12. kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
13. How to setup kubernetes jenkins pipeline on AWS?
14. Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
15. How to fix kubernetes pods getting recreated?
16. How to delete all kubernetes PODS?
17. How to use Kubernetes secrets?
18. Share kubernetes secrets between namespaces?
19. How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
20. Delete Kubernetes POD stuck in terminating state?