How to fix Kubernetes – error execution phase preflight [preflight]

Share on:

While working with kubernetes somewhere down the line you might have faced this error and there could be many reasons behind the issue “error execution phase preflight”

With my experience in working with kubernetes I found that this issue mostly occurs when you are trying to setup kubernetes cluster on your own and since you are setting it up by yourself than I can easily guess that you might miss a lot of technicalities and end up with this issue.

Tip - First and far most look your log file very carefully because all the errors starting with error execution phase preflight is not the same. Look for the keyword in the error log such as

  • [ERROR DirAvailable]
  • [ERROR FileAvailable]
  • [ERROR Swap]
  • [ERROR IsPrivilegedUser]
  • [ERROR NumCPU]
  • [ERROR FileContent]

All the above-mentioned errors are different although the error message always starts with error execution phase preflight.

For troubleshooting please follow the individual issue here -

  1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty
  2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
  3. [ERROR Swap]: running with swap on is not supported. Please disable swap
  4. [ERROR IsPrivilegedUser]: user is not running as root
  5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
  6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
  7. kubelet-start: configmaps “kubelet-config-1.15” is forbidden: User “system:bootstrap:g0toug” cannot get resource “configmaps” in API group "” in the namespace “kube-system”
  8. kubeadm token
  9. CNI(Container Network interface) .i.e. - calico, flannel
  10. Firewall settings

1. [ERROR DirAvailable-var-lib-etcd]: /var/lib/etcd is not empty

1error execution phase preflight: preflight] Some fatal errors occurred:
1 ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty

In my experience this issue happens when you have deleted the node and trying to re-create it. If you have setup your local kubernetes cluster on your local development machine then it is quite often you will face this issue.

It mostly occurs when you either shutdown or restart and in some cases if you are trying to delete one the kubernetes node.

Solution - You should delete the directory /var/lib/etcd. Use the following command to delete the directory

1$ rm -rf /var/lib/etcd

Generally speaking kubeadm reset should delete this diretory (/var/lib/etcd). But there are many possible cases where this directory or file is locked or do not have permission to delete, so in that cases kubeadm reset will not be successful.

This issue is not so frequent on cloud plateform such GCP(Google Cloud Platform), AWS Cloud(Amazon Web Service) or any other cloud service provider.

2. [ERROR FileAvailable-etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists

This error tells that you need to run the kubeadm reset command before initializing the kubernetes cluster or running kubeadm init command.

1$ kubeadm reset

The command kubeadm reset can be used when you see following errors in the logs while initializing the kubernetes cluster

1error execution phase preflight: preflight] Some fatal errors occurred:
1ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
1ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
1ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

So all the above errors can be fixed if you run the the kubeadm reset before initializing the kubernetes cluster.

The most common cause of this issue is you have already run the kubeadm init command on your kubernetes cluster and now again you are trying to run the same command before doing the kubeadm reset.

4. [ERROR IsPrivilegedUser]: user is not running as root

As the error say “user is not running as root”, which mean you are not running kubeadm init command as root user. To fix the issue you should run the kubeadm init as root.

Solution - Add sudo before the running the kubeadm init command.

1$ sudo kubeadm init --apiserver-advertise-address=100.0.0.1 --pod-network-cidr=10.244.0.0/16  

5. [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

As the error says you are trying to run Kubernetes cluster with 1 CPU either on Vagrant or Virtual Box.

Vagrant

If you are using the Vagrant then increase the CPUs count from more than 1.

Here is sample vagrant file which you could refer (look for similar vagrant file which you have with you and update the CPU count in it)

Look for the configuration cpus inside the vagrant file

 1Vagrant.configure("2") do |config|
 2  config.vm.define "amaster" do |amaster|
 3    amaster.vm.box_download_insecure = true
 4    amaster.vm.box = "hashicorp/bionic64"
 5    amaster.vm.network "private_network", ip: "100.0.0.1"
 6    amaster.vm.hostname = "amaster"
 7    amaster.vm.provider "virtualbox" do |v|
 8      v.name = "amaster"
 9      v.memory = 2048
10      v.cpus = 2
11    end
12  end

Virtual Box

In case you are using the Virtual Box then you can go to Virtual Box settings and increase the CPUs, so that it is more than 1.

Please refer to following screenshot -

6. [ERROR FileContent-proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist or [ERROR FileContent-proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

In order to fix this error you need to edit /etc/sysctl.conf and then set the value for /proc/sys/net/bridge/bridge-nf-call-iptables

Here are following command which you can use -

1$ sudo vi /etc/sysctl.conf

After opening the file sysctl.conf in edit mode, add the following line if its not there.

1net.bridge.bridge-nf-call-iptables = 1

Then you need to execute

1$ sudo sysctl -p

The above changed please do the kubeadm reset and then initialize your Kubernetes cluster with kubeadm init command

7. kubelet-start: configmaps “kubelet-config-1.15” is forbidden: User “system:bootstrap:g0toug” cannot get resource “configmaps” in API group "” in the namespace “kube-system”

1error execution phase kubelet-start: configmaps "kubelet-config-1.15" is forbidden: User "system:bootstrap:g0toug" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Well if you are facing this issue then I could easily say there is problem with the kublet version and kubelet-config-map.

The above error signifies that you are trying to join kubelet which is 1.15 and which has kublet-config-1.14, so that is not supported at this moment.

But as per the github it should be fixed in 1.16 for more details you can refer to git issue – #1582

8. kubeadm token

The first and far most guess would be to verify your kubeadm token and its validity. Because it quite obvious that you might have generated the token sometimes back and now it has expired.

So if you are still trying to connect to kubernetes master node with expired token than you will end up with the error.

I would recommend you to use following command to verify the kubeadm token.

Check the token validity

1$ kubeadm token list

The above command will return you with all the token which has been generated previously during the kubernetes cluster setup.

Carefully look on the EXPIRES column and see if there are any token which has expired.

How to fix the expired kubeadm token?

kubeadm(kubernetes admin) provides commands for generating/creating new token out of the box, so you can use following command to generate/create new token

1$ kubeadm token create --print-join-command
1kubeadm join 100.0.0.2:6443 --token xg7pkc.atu3g3nlanmajc8j     --discovery-token-ca-cert-hash sha256:d19178104c84f7876602747a3b6e23c1df76f374fe9ba232ab406fd56d0b768e

kubeadm token create --print-join-command

kubeadm token create -print-join-command

The above command will create a new fresh token as well as print kubeadm join command, which you can copy and run from any node.

9. CNI(Container Network Interface) - calico, flannel

Well if you think that token validity of your cluster is okay and you do not have any expired token than I would recommend checking the CNI(container network interface) setting within your kubernetes cluster.

But in my experience troubleshooting CNI is needed when you setup kubernetes cluster with VMs(virtual machine) using Vagrant

But anyways this is what happens when you use the CNI either from calico of flannel. There will be two ethernet port on your kubernetes master node.

Use the following command to check the ethernet port

1$ ifconfig
 1eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
 2        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
 3        inet6 fe80::a00:27ff:febb:1475  prefixlen 64  scopeid 0x20<link>
 4        ether 08:00:27:bb:14:75  txqueuelen 1000  (Ethernet)
 5        RX packets 704152  bytes 948370649 (948.3 MB)
 6        RX errors 0  dropped 0  overruns 0  frame 0
 7        TX packets 122139  bytes 7772979 (7.7 MB)
 8        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
 9
10eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
11        inet 100.0.0.2  netmask 255.255.255.0  broadcast 100.0.0.255
12        inet6 fe80::a00:27ff:fe36:b458  prefixlen 64  scopeid 0x20<link>
13        ether 08:00:27:36:b4:58  txqueuelen 1000  (Ethernet)
14        RX packets 100821  bytes 64254917 (64.2 MB)
15        RX errors 0  dropped 0  overruns 0  frame 0
16        TX packets 65066  bytes 23079670 (23.0 MB)
17        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And if the command returns with 2 ethernet adapter name .i.e. eth0 and eth1. Then i can easily guess your CNI(container network interface) took the default ethernet adapter .i.e. eth0 and which is wrong.

Ideally it should take the eth1 as its preferred communication channel.

How to fix or setup correct ethernet adapter name for preferred CNI(Container Network Interface) ?

(Note - I tried following fix for the flannel but you can do similar steps for calico or any other CNI)

Look for the kube-flannel.yml inside your kubernetes node. You can search the kube-flannel.yml using following command

1find / -name 'kube-flannel.yml' 2>/dev/null

After that you need to edit the file and and search for the text - flanneld

1$ vi kube-flannel.yml

In the args section add : – –iface=eth1

1- --iface=eth1
2        args:
3        - --ip-masq
4        - --kube-subnet-mgr
5        - --iface=eth1

Apply the flannel configuration

1$ kubectl apply -f kube-flannel.yml
 1podsecuritypolicy.policy/psp.flannel.unprivileged created
 2clusterrole.rbac.authorization.k8s.io/flannel created
 3clusterrolebinding.rbac.authorization.k8s.io/flannel created
 4serviceaccount/flannel created
 5configmap/kube-flannel-cfg created
 6daemonset.apps/kube-flannel-ds-amd64 created
 7daemonset.apps/kube-flannel-ds-arm64 created
 8daemonset.apps/kube-flannel-ds-arm created
 9daemonset.apps/kube-flannel-ds-ppc64le created
10daemonset.apps/kube-flannel-ds-s390x created

10. Firewall settings

This could be one of the potential reason for this error because by default Firewall’s are not disabled in any Linux distribution(Ubuntu, Centos, Fedora or RedHat).

The best way for you to check the firewall status before, use the following command for checking the firewall status.

Ubuntu

1$ sudo ufw status

Disable the firewall

1$ sudo ufw disable

But suppose you do not wanna disable the firewall due to security reason than it is highly recommended for you to check the firewall rule or in other words check iptables

Use the following command to verify the iptables

1$ iptables -L

Note - The above command will tell you all the ports which are open for communication. Look for the port 443 because it the most default port for communication inside kubernetes cluster

Now the next question comes in my mind, how to setup a firewall rule or entry in iptables for port 443 inside kubernetes cluster?

Well use the following command for setting up the iptable rules.(Keep in mind the following firewall rules is for the port 443, but if you have any other port then please update the port accordingly)

1$ iptables -A INPUT -p tcp --dport 443 -j ACCEPT

CentOS

The theory is pretty much same for the CentOS also, but the commands are little different when we talk about CentOS.

Use the following command to check the status of the CentOS where your kubernetes cluster is running.

1$ sudo systemctl status firewalld

The above command will tell you about the firewall status.

If the firewall is up and running then i would recommend you to disable the firewall using the following command

1$ sudo systemctl enable firewalld

But just in case you not able to disable the firewall due to security reasons then you need to check the ports which are open using following command

1$ firewall-cmd --list-ports

If you didn’t find any entry in the list then please add port in the firewall (change the port number if you have some other ports)

1$ firewall-cmd --permanent --add-port 443/tcp

I hope this firewall troubleshooting will help you to fix the issue “error execution phase preflight” inside your kubernetes cluster.

Learn more about kubernetes -

15 Steps: Install Kubernetes on CentOS “bento/centos-7

14 Steps to Install kubernetes on Ubuntu 18.04 and 16.04 (“hashicorp/bionic64”)

kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster

Deploy Spring Boot microservices on kubernetes and Google Cloud Platform(GCP)