How to fix Kubernetes – error execution phase preflight [preflight]
While working with kubernetes somewhere down the line you might have faced this error and there could be many reasons behind the issue “error execution phase preflight”
With my experience in working with kubernetes I found that this issue mostly occurs when you are trying to setup kubernetes cluster on your own and since you are setting it up by yourself than I can easily guess that you might miss a lot of technicalities and end up with this issue.
Tip - First and far most look your log file very carefully because all the errors starting with error execution phase preflight is not the same. Look for the keyword in the error log such as
- [ERROR DirAvailable]
- [ERROR FileAvailable]
- [ERROR Swap]
- [ERROR IsPrivilegedUser]
- [ERROR NumCPU]
- [ERROR FileContent]
All the above-mentioned errors are different although the error message always starts with error execution phase preflight.
For troubleshooting please follow the individual issue here -
- [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
- [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
- [ERROR Swap]: running with swap on is not supported. Please disable swap
- [ERROR IsPrivilegedUser]: user is not running as root
- [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
- [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
- kubelet-start: configmaps “kubelet-config-1.15” is forbidden: User “system:bootstrap:g0toug” cannot get resource “configmaps” in API group "” in the namespace “kube-system”
- kubeadm token
- CNI(Container Network interface) .i.e. - calico, flannel
- Firewall settings
1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
1error execution phase preflight: preflight] Some fatal errors occurred:
1 ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
In my experience this issue happens when you have deleted the node and trying to re-create it. If you have setup your local kubernetes cluster on your local development machine then it is quite often you will face this issue.
It mostly occurs when you either shutdown or restart and in some cases if you are trying to delete one the kubernetes node.
Solution - You should delete the directory /var/lib/etcd. Use the following command to delete the directory
1$ rm -rf /var/lib/etcd
Generally speaking kubeadm reset should delete this diretory (/var/lib/etcd). But there are many possible cases where this directory or file is locked or do not have permission to delete, so in that cases kubeadm reset will not be successful.
This issue is not so frequent on cloud plateform such GCP(Google Cloud Platform), AWS Cloud(Amazon Web Service) or any other cloud service provider.
2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
This error tells that you need to run the kubeadm reset command before initializing the kubernetes cluster or running kubeadm init command.
1$ kubeadm reset
The command kubeadm reset can be used when you see following errors in the logs while initializing the kubernetes cluster
1error execution phase preflight: preflight] Some fatal errors occurred:
1ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
1ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
So all the above errors can be fixed if you run the the kubeadm reset before initializing the kubernetes cluster.
The most common cause of this issue is you have already run the kubeadm init command on your kubernetes cluster and now again you are trying to run the same command before doing the kubeadm reset.
4. [ERROR IsPrivilegedUser]: user is not running as root
As the error say “user is not running as root”, which mean you are not running kubeadm init command as root user. To fix the issue you should run the kubeadm init as root.
Solution - Add sudo before the running the kubeadm init command.
1$ sudo kubeadm init --apiserver-advertise-address=220.127.116.11 --pod-network-cidr=10.244.0.0/16
5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
As the error says you are trying to run Kubernetes cluster with 1 CPU either on Vagrant or Virtual Box.
If you are using the Vagrant then increase the CPUs count from more than 1.
Here is sample vagrant file which you could refer (look for similar vagrant file which you have with you and update the CPU count in it)
Look for the configuration cpus inside the vagrant file
1Vagrant.configure("2") do |config| 2 config.vm.define "amaster" do |amaster| 3 amaster.vm.box_download_insecure = true 4 amaster.vm.box = "hashicorp/bionic64" 5 amaster.vm.network "private_network", ip: "18.104.22.168" 6 amaster.vm.hostname = "amaster" 7 amaster.vm.provider "virtualbox" do |v| 8 v.name = "amaster" 9 v.memory = 2048 10 v.cpus = 2 11 end 12 end
Please refer to following screenshot -
6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
In order to fix this error you need to edit /etc/sysctl.conf and then set the value for /proc/sys/net/bridge/bridge-nf-call-iptables
Here are following command which you can use -
1$ sudo vi /etc/sysctl.conf
After opening the file sysctl.conf in edit mode, add the following line if its not there.
1net.bridge.bridge-nf-call-iptables = 1
Then you need to execute
1$ sudo sysctl -p
The above changed please do the kubeadm reset and then initialize your Kubernetes cluster with kubeadm init command
7. kubelet-start: configmaps “kubelet-config-1.15” is forbidden: User “system:bootstrap:g0toug” cannot get resource “configmaps” in API group "” in the namespace “kube-system”
1error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
Well if you are facing this issue then I could easily say there is problem with the kublet version and kubelet-config-map.
The above error signifies that you are trying to join kubelet which is 1.15 and which has kublet-config-1.14, so that is not supported at this moment.
But as per the github it should be fixed in 1.16 for more details you can refer to git issue – #1582
8. kubeadm token
The first and far most guess would be to verify your kubeadm token and its validity. Because it quite obvious that you might have generated the token sometimes back and now it has expired.
So if you are still trying to connect to kubernetes master node with expired token than you will end up with the error.
I would recommend you to use following command to verify the kubeadm token.
Check the token validity
1$ kubeadm token list
The above command will return you with all the token which has been generated previously during the kubernetes cluster setup.
Carefully look on the EXPIRES column and see if there are any token which has expired.
How to fix the expired kubeadm token?
kubeadm(kubernetes admin) provides commands for generating/creating new token out of the box, so you can use following command to generate/create new token
1$ kubeadm token create --print-join-command
1kubeadm join 22.214.171.124:6443 --token xg7pkc.atu3g3nlanmajc8j --discovery-token-ca-cert-hash sha256:d19178104c84f7876602747a3b6e23c1df76f374fe9ba232ab406fd56d0b768e
The above command will create a new fresh token as well as print kubeadm join command, which you can copy and run from any node.
Well if you think that token validity of your cluster is okay and you do not have any expired token than I would recommend checking the CNI(container network interface) setting within your kubernetes cluster.
But in my experience troubleshooting CNI is needed when you setup kubernetes cluster with VMs(virtual machine) using Vagrant
But anyways this is what happens when you use the CNI either from calico of flannel. There will be two ethernet port on your kubernetes master node.
Use the following command to check the ethernet port
1eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 2 inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255 3 inet6 fe80::a00:27ff:febb:1475 prefixlen 64 scopeid 0x20<link> 4 ether 08:00:27:bb:14:75 txqueuelen 1000 (Ethernet) 5 RX packets 704152 bytes 948370649 (948.3 MB) 6 RX errors 0 dropped 0 overruns 0 frame 0 7 TX packets 122139 bytes 7772979 (7.7 MB) 8 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 9 10eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 11 inet 126.96.36.199 netmask 255.255.255.0 broadcast 188.8.131.52 12 inet6 fe80::a00:27ff:fe36:b458 prefixlen 64 scopeid 0x20<link> 13 ether 08:00:27:36:b4:58 txqueuelen 1000 (Ethernet) 14 RX packets 100821 bytes 64254917 (64.2 MB) 15 RX errors 0 dropped 0 overruns 0 frame 0 16 TX packets 65066 bytes 23079670 (23.0 MB) 17 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
And if the command returns with 2 ethernet adapter name .i.e. eth0 and eth1. Then i can easily guess your CNI(container network interface) took the default ethernet adapter .i.e. eth0 and which is wrong.
Ideally it should take the eth1 as its preferred communication channel.
How to fix or setup correct ethernet adapter name for preferred CNI(Container Network Interface) ?
(Note - I tried following fix for the flannel but you can do similar steps for calico or any other CNI)
Look for the kube-flannel.yml inside your kubernetes node. You can search the kube-flannel.yml using following command
1find / -name 'kube-flannel.yml' 2>/dev/null
After that you need to edit the file and and search for the text - flanneld
1$ vi kube-flannel.yml
In the args section add : – –iface=eth1
1- --iface=eth1 2 args: 3 - --ip-masq 4 - --kube-subnet-mgr 5 - --iface=eth1
Apply the flannel configuration
1$ kubectl apply -f kube-flannel.yml
1podsecuritypolicy.policy/psp.flannel.unprivileged created 2clusterrole.rbac.authorization.k8s.io/flannel created 3clusterrolebinding.rbac.authorization.k8s.io/flannel created 4serviceaccount/flannel created 5configmap/kube-flannel-cfg created 6daemonset.apps/kube-flannel-ds-amd64 created 7daemonset.apps/kube-flannel-ds-arm64 created 8daemonset.apps/kube-flannel-ds-arm created 9daemonset.apps/kube-flannel-ds-ppc64le created 10daemonset.apps/kube-flannel-ds-s390x created
10. Firewall settings
This could be one of the potential reason for this error because by default Firewall’s are not disabled in any Linux distribution(Ubuntu, Centos, Fedora or RedHat).
The best way for you to check the firewall status before, use the following command for checking the firewall status.
1$ sudo ufw status
Disable the firewall
1$ sudo ufw disable
But suppose you do not wanna disable the firewall due to security reason than it is highly recommended for you to check the firewall rule or in other words check iptables
Use the following command to verify the iptables
1$ iptables -L
Note - The above command will tell you all the ports which are open for communication. Look for the port 443 because it the most default port for communication inside kubernetes cluster
Now the next question comes in my mind, how to setup a firewall rule or entry in iptables for port 443 inside kubernetes cluster?
Well use the following command for setting up the iptable rules.(Keep in mind the following firewall rules is for the port 443, but if you have any other port then please update the port accordingly)
1$ iptables -A INPUT -p tcp --dport 443 -j ACCEPT
Use the following command to check the status of the CentOS where your kubernetes cluster is running.
1$ sudo systemctl status firewalld
The above command will tell you about the firewall status.
If the firewall is up and running then i would recommend you to disable the firewall using the following command
1$ sudo systemctl enable firewalld
But just in case you not able to disable the firewall due to security reasons then you need to check the ports which are open using following command
1$ firewall-cmd --list-ports
If you didn’t find any entry in the list then please add port in the firewall (change the port number if you have some other ports)
1$ firewall-cmd --permanent --add-port 443/tcp
I hope this firewall troubleshooting will help you to fix the issue “error execution phase preflight” inside your kubernetes cluster.