How to fix Kubernetes – error execution phase preflight [preflight]
While working with kubernetes somewhere down the line you might have faced this error and there could be many reasons behind the issue error execution phase preflight
With my experience in working with kubernetes I found that this issue mostly occurs when you are trying to setup kubernetes cluster on your own and since you are setting it up by yourself than I can easily guess that you might miss a lot of technicalities and end up with this issue.
Tip - First and far most look your log file very carefully because all the errors starting with error execution phase preflight is not the same. Look for the keyword in the error log such as
- [ERROR DirAvailable]
- [ERROR FileAvailable]
- [ERROR Swap]
- [ERROR IsPrivilegedUser]
- [ERROR NumCPU]
- [ERROR FileContent]
All the above-mentioned errors are different although the error message always starts with error execution phase preflight.
For troubleshooting please follow the individual issue here -
- [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
- [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
- [ERROR Swap]: running with swap on is not supported. Please disable swap
- [ERROR IsPrivilegedUser]: user is not running as root
- [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
- [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
- kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
- kubeadm token
- CNI(Container Network interface) .i.e. - calico, flannel
- Firewall settings
1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
1error execution phase preflight: preflight] Some fatal errors occurred:
1 ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
In my experience this issue happens when you have deleted the node and trying to re-create it. If you have setup your local kubernetes cluster on your local development machine then it is quite often you will face this issue.
It mostly occurs when you either shutdown or restart and in some cases if you are trying to delete one the kubernetes node.
Solution - You should delete the directory /var/lib/etcd. Use the following command to delete the directory
1$ rm -rf /var/lib/etcd
Generally speaking kubeadm reset should delete this diretory (/var/lib/etcd). But there are many possible cases where this directory or file is locked or do not have permission to delete, so in that cases kubeadm reset will not be successful.
This issue is not so frequent on cloud plateform such GCP(Google Cloud Platform), AWS Cloud(Amazon Web Service) or any other cloud service provider.
2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists {#file-already-exist
This error tells that you need to run the kubeadm reset command before initializing the kubernetes cluster (kubeadm init). In other word you need to properly recover a K8s cluster after reboot
But first let's see the error when I ran the command $ kubeadm reset
1$ kubeadm reset
1error execution phase preflight: preflight] Some fatal errors occurred:
2ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]:
3/etc/kubernetes/manifests/kube-apiserver.yaml already exists
4ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]:
5/etc/kubernetes/manifests/kube-controller-manager.yaml already exists
6ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]:
7/etc/kubernetes/manifests/kube-scheduler.yaml already exists
8ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]:
9/etc/kubernetes/manifests/etcd.yaml already exists
10ERROR FileContent--proc-sys-net-ipv4-ip_forward]:
11 /proc/sys/net/ipv4/ip_forward contents are not set to 1
How to fix and properly recover kubernetes cluster after reboot?
So all the above errors can be fixed with the following steps -
- Stop kubelet.service - You must stop the Kubelet Service using the command $ sudo systemctl stop kubelet. service
- Kubeadm reset - Run the kubeadm reset before initializing the kubernetes cluster. The most common cause of this issue is you have already run the kubeadm init command on your kubernetes cluster and now again you are trying to run the same command before doing the kubeadm reset.
4. [ERROR IsPrivilegedUser]: user is not running as root
As the error say "user is not running as root", which mean you are not running kubeadm init command as root user. To fix the issue you should run the kubeadm init as root.
Solution - Add sudo before the running the kubeadm init command.
1$ sudo kubeadm init --apiserver-advertise-address=100.0.0.1 --pod-network-cidr=10.244.0.0/16
5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
As the error says you are trying to run Kubernetes cluster with 1 CPU which is not sufficient enough to install Kubernetes cluster.
Although [ERROR NumCPU] can be ignored by setting the --ignore-preflight-errors=NumCPU flag in the kube init command but it is not recommended because if you have very limited computing power then running the kubernetes cluster will always be troublesome.
How to increase CPU if you are using Vagrant for installing kubernetes cluster?
If you are using the Vagrant then increase the CPUs count from more than 1.
Here is sample vagrant file which you could refer (look for similar vagrant file which you have with you and update the CPU count in it)
Look for the configuration cpus inside the vagrant file
1Vagrant.configure("2") do |config|
2 config.vm.define "amaster" do |amaster|
3 amaster.vm.box_download_insecure = true
4 amaster.vm.box = "hashicorp/bionic64"
5 amaster.vm.network "private_network", ip: "100.0.0.1"
6 amaster.vm.hostname = "amaster"
7 amaster.vm.provider "virtualbox" do |v|
8 v.name = "amaster"
9 v.memory = 2048
10 v.cpus = 2
11 end
12 end
How to increase CPU if you are using only Virtual Box for installing kubernetes cluster?
In case you are using the Virtual Box then you can go to Virtual Box settings and increase the CPUs, so that it is more than 1.
Please refer to following screenshot -
Increase the CPU if you are using AWS?
It always recommended to use Amazon Elastic Kubernetes Service (Amazon EKS) because Amazon EKS service will take care of CPU and its computing power so you do not have to worry about it. Here is a guide if you do not know how to setup AWS EKS using eksctl
6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
In order to fix this error you need to edit /etc/sysctl.conf and then set the value for /proc/sys/net/bridge/bridge-nf-call-iptables
Here are following command which you can use -
1$ sudo vi /etc/sysctl.conf
After opening the file sysctl.conf in edit mode, add the following line if its not there.
1net.bridge.bridge-nf-call-iptables = 1
Then you need to execute
1$ sudo sysctl -p
The above changed please do the kubeadm reset and then initialize your Kubernetes cluster with kubeadm init command
7. kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
1error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Well if you are facing this issue then I could easily say there is problem with the kublet version and kubelet-config-map.
The above error signifies that you are trying to join kubelet which is 1.15 and which has kublet-config-1.14, so that is not supported at this moment.
But as per the github it should be fixed in 1.16 for more details you can refer to git issue – #1582
8. kubeadm token
The first and far most guess would be to verify your kubeadm token and its validity. Because it quite obvious that you might have generated the token sometimes back and now it has expired.
So if you are still trying to connect to kubernetes master node with expired token than you will end up with the error.
I would recommend you to use following command to verify the kubeadm token.
Check the token validity
1$ kubeadm token list
The above command will return you with all the token which has been generated previously during the kubernetes cluster setup.
Carefully look on the EXPIRES column and see if there are any token which has expired.
How to fix the expired kubeadm token?
kubeadm(kubernetes admin) provides commands for generating/creating new token out of the box, so you can use following command to generate/create new token
1$ kubeadm token create --print-join-command
1kubeadm join 100.0.0.2:6443 --token xg7pkc.atu3g3nlanmajc8j --discovery-token-ca-cert-hash sha256:d19178104c84f7876602747a3b6e23c1df76f374fe9ba232ab406fd56d0b768e
The above command will create a new fresh token as well as print kubeadm join command, which you can copy and run from any node.
9. CNI(Container Network Interface) - calico, flannel
Well if you think that token validity of your cluster is okay and you do not have any expired token than I would recommend checking the CNI(container network interface) setting within your kubernetes cluster.
But in my experience troubleshooting CNI is needed when you setup kubernetes cluster with VMs(virtual machine) using Vagrant
But anyways this is what happens when you use the CNI either from calico of flannel. There will be two ethernet port on your kubernetes master node.
Use the following command to check the ethernet port
1$ ifconfig
1eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
2 inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
3 inet6 fe80::a00:27ff:febb:1475 prefixlen 64 scopeid 0x20<link>
4 ether 08:00:27:bb:14:75 txqueuelen 1000 (Ethernet)
5 RX packets 704152 bytes 948370649 (948.3 MB)
6 RX errors 0 dropped 0 overruns 0 frame 0
7 TX packets 122139 bytes 7772979 (7.7 MB)
8 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
9
10eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
11 inet 100.0.0.2 netmask 255.255.255.0 broadcast 100.0.0.255
12 inet6 fe80::a00:27ff:fe36:b458 prefixlen 64 scopeid 0x20<link>
13 ether 08:00:27:36:b4:58 txqueuelen 1000 (Ethernet)
14 RX packets 100821 bytes 64254917 (64.2 MB)
15 RX errors 0 dropped 0 overruns 0 frame 0
16 TX packets 65066 bytes 23079670 (23.0 MB)
17 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
And if the command returns with 2 ethernet adapter name .i.e. eth0 and eth1. Then i can easily guess your CNI(container network interface) took the default ethernet adapter .i.e. eth0 and which is wrong.
Ideally it should take the eth1 as its preferred communication channel.
How to fix or setup correct ethernet adapter name for preferred CNI(Container Network Interface) ?
(Note - I tried following fix for the flannel but you can do similar steps for calico or any other CNI)
Look for the kube-flannel.yml inside your kubernetes node. You can search the kube-flannel.yml using following command
1find / -name 'kube-flannel.yml' 2>/dev/null
After that you need to edit the file and and search for the text - flanneld
1$ vi kube-flannel.yml
In the args section add : – –iface=eth1
1- --iface=eth1
2 args:
3 - --ip-masq
4 - --kube-subnet-mgr
5 - --iface=eth1
Apply the flannel configuration
1$ kubectl apply -f kube-flannel.yml
1podsecuritypolicy.policy/psp.flannel.unprivileged created
2clusterrole.rbac.authorization.k8s.io/flannel created
3clusterrolebinding.rbac.authorization.k8s.io/flannel created
4serviceaccount/flannel created
5configmap/kube-flannel-cfg created
6daemonset.apps/kube-flannel-ds-amd64 created
7daemonset.apps/kube-flannel-ds-arm64 created
8daemonset.apps/kube-flannel-ds-arm created
9daemonset.apps/kube-flannel-ds-ppc64le created
10daemonset.apps/kube-flannel-ds-s390x created
10. Firewall settings
This could be one of the potential reason for this error because by default Firewall's are not disabled in any Linux distribution(Ubuntu, Centos, Fedora or RedHat).
The best way for you to check the firewall status before, use the following command for checking the firewall status.
1$ sudo ufw status
Disable the firewall
1$ sudo ufw disable
But suppose you do not wanna disable the firewall due to security reason than it is highly recommended for you to check the firewall rule or in other words check iptables
Use the following command to verify the iptables
1$ iptables -L
Note - The above command will tell you all the ports which are open for communication. Look for the port 443 because it the most default port for communication inside kubernetes cluster
Now the next question comes in my mind, how to setup a firewall rule or entry in iptables for port 443 inside kubernetes cluster?
Well use the following command for setting up the iptable rules.(Keep in mind the following firewall rules is for the port 443, but if you have any other port then please update the port accordingly)
1$ iptables -A INPUT -p tcp --dport 443 -j ACCEPT
The theory is pretty much same for the CentOS also, but the commands are little different when we talk about CentOS.
Use the following command to check the status of the CentOS where your kubernetes cluster is running.
1$ sudo systemctl status firewalld
The above command will tell you about the firewall status.
If the firewall is up and running then i would recommend you to disable the firewall using the following command
1$ sudo systemctl enable firewalld
But just in case you not able to disable the firewall due to security reasons then you need to check the ports which are open using following command
1$ firewall-cmd --list-ports
If you didn't find any entry in the list then please add port in the firewall (change the port number if you have some other ports)
1$ firewall-cmd --permanent --add-port 443/tcp
I hope this firewall troubleshooting will help you to fix the issue "error execution phase preflight" inside your kubernetes cluster.
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?
Posts in this Series
- Kubernetes Cheat Sheet for day to day DevOps operations?
- Delete Kubernetes POD stuck in terminating state?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Share kubernetes secrets between namespaces?
- How to use Kubernetes secrets?
- How to delete all kubernetes PODS?
- kubernetes pods getting recreated?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- kubectl export yaml OR How to generate YAML for deployed kubernetes resources
- Kubernetes Updates
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- Kubernetes cluster setup with Jenkins
- How to use Persistent Volume and Persistent Claims | Kubernetes
- How to fix ProvisioningFailed persistentvolume controller no volume plugin matched
- Fixing – Cannot bind to requested volume: storageClasseName does not match
- Fixing – pod has unbound immediate persistentvolumeclaims or cannot bind to requested volume incompatible accessmode
- How to fix kubernetes dashboard forbidden 403 error – message services https kubernetes-dashboard is forbidden User
- How to fix Kubernetes – error execution phase preflight [preflight]
- Deploy Spring Boot microservices on kubernetes?
- How to fix – ansible_memtotal_mb minimal_master_memory_mb
- How to use kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster
- How to setup kubernetes on CentOS 8 and CentOS 7
- How to fix – How to fix - ERROR Swap running with swap on is not supported. Please disable swap
- 14 Steps to Install kubernetes on Ubuntu 20.04(bento/ubuntu-20.04), 18.04(hashicorp/bionic64)
- Kubernetes Dashboard | Kubernetes Admin GUI | Kubernetes Desktop Client
- Install Kubernetes with Minikube