kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster

Share on:

Before we jump into the steps of installation, if you are familiar with Puppet, Chef and Ansible (https://github.com/kubernetes-incubator/kubespray) then kubespray is going to be the best choice to set up a Kubernetes cluster.

In this article, we will be going through 12 steps starting from setting up vagrant VMs till running the final ansible-playbook.

Disclaimer - If you are a beginner with kubernetes then i would highly recommend to go through the manual installation of kubernetes on Ubuntu or CentOS and for the same you can refer to -

  1. 14 Steps to Install kubernetes on Ubuntu 18.04 and 16.04
  2. 15 Steps: Install Kubernetes on CentOS “bento/centos-7”

Okay now lets try some kubespray and kubernetes -

Step 1: Provision the VMs using Vagrant

First we need to provision the VMs using vagrant.

We will be setting up total 3 VMs (Virtual Machine) with its unique IP -

  1. Ansible Node (amaster) - 100.0.0.1 - 2 CPU - 2 GB Memory
  2. Kubernetes Master Node (kmaster) - 100.0.0.2 - 2 CPU - 2 GB Memory
  3. Kubernetes Worker Node (kworker) - 100.0.0.3 - 2 CPU - 2 GB Memory

Here is the Vagratnfile

 1Vagrant.configure("2") do |config|
 2  config.vm.define "amaster" do |amaster|
 3    amaster.vm.box_download_insecure = true
 4    amaster.vm.box = "hashicorp/bionic64"
 5    amaster.vm.network "private_network", ip: "100.0.0.1"
 6    amaster.vm.hostname = "amaster"
 7    amaster.vm.provider "virtualbox" do |v|
 8      v.name = "amaster"
 9      v.memory = 2048
10      v.cpus = 2
11    end
12  end
13
14  config.vm.define "kmaster" do |kmaster|
15    kmaster.vm.box_download_insecure = true
16    kmaster.vm.box = "hashicorp/bionic64"
17    kmaster.vm.network "private_network", ip: "100.0.0.2"
18    kmaster.vm.hostname = "kmaster"
19    kmaster.vm.provider "virtualbox" do |v|
20      v.name = "kmaster"
21      v.memory = 2048
22      v.cpus = 2
23    end
24  end
25
26  config.vm.define "kworker" do |kworker|
27    kworker.vm.box_download_insecure = true
28    kworker.vm.box = "hashicorp/bionic64"
29    kworker.vm.network "private_network", ip: "100.0.0.3"
30    kworker.vm.hostname = "kworker"
31    kworker.vm.provider "virtualbox" do |v|
32      v.name = "kworker"
33      v.memory = 2048
34      v.cpus = 2
35    end
36  end
37
38end

Start the vagrant box -

1$ vagrant up

Step 2: Update /etc/hosts on all the nodes

After starting the vagrant box you need to update the /etc/hosts file on each node .i.e -amaster, kmaster, kworker

So run the following command on all the three nodes

1vagrant@amaster:~$ sudo vi /etc/hosts

Add the following entries in the hosts files of each node (amaster, kmaster, kworker)

1100.0.0.1 amaster.jhooq.com amaster
2100.0.0.2 kmaster.jhooq.com kmaster
3100.0.0.3 kworker.jhooq.com kworker

Your /etc/hosts file should look like this on all the three nodes .i.e. - amaster, kmaster, kworker

1vagrant@amaster:~$ cat /etc/hosts
 1127.0.0.1	localhost
 2127.0.1.1	amaster	amaster
 3
 4# The following lines are desirable for IPv6 capable hosts
 5::1     localhost ip6-localhost ip6-loopback
 6ff02::1 ip6-allnodes
 7ff02::2 ip6-allrouters
 8100.0.0.1 amaster.jhooq.com amaster
 9100.0.0.2 kmaster.jhooq.com kmaster
10100.0.0.3 kworker.jhooq.com kworker

Step 3: Generate SSH key for ansible (only need to run on ansible node .i.e. amaster)

To setup the kubespray smoothly we need to generate the SSH keys for the ansible master(amaster) nodes and copy the ssh keys to other nodes. So that you do not have to provide username and password everytime you login/ssh into the other nodes .i.e. - kmaster, kworker

Generate SSH key (during the ssh key generation it might will ask for passphrase so either you create a new passphrase or leave it empty)-

1ssh-keygen -t rsa
 1Generating public/private rsa key pair.
 2Enter file in which to save the key (/home/vagrant/.ssh/id_rsa): 
 3Enter passphrase (empty for no passphrase): 
 4Enter same passphrase again: 
 5Your identification has been saved in /home/vagrant/.ssh/id_rsa.
 6Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub.
 7The key fingerprint is:
 8SHA256:LWGasiSDAqf8eY3pz5swa/nUl2rWc1IFgiPuqFTYsKs vagrant@amaster
 9The key's randomart image is:
10+---[RSA 2048]----+
11|          .      |
12|   .   . o . .   |
13|. . = . + . . .  |
14|o+ o o = o     . |
15|+.o = = S .   .  |
16|. .*.++...  ..   |
17|  ooo*.o ..o.    |
18| E .oo* .oo+ .   |
19|    .oo*+.  +    |
20+----[SHA256]-----+

Step 4: Copy SSH key to other nodes .i.e. - kmaster, kworker

In the step-3 we have generated the SSH keys, now we need to copy the SSH keys to other nodes .i.e. kmaster, kworker

Copy to kmaster node (During the ssh-copy-id it will ask for the other node password, so in case if you have not set any password then you can supply default password .i.e. vagrant) -

1ssh-copy-id 100.0.0.2
1/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/vagrant/.ssh/id_rsa.pub"
2The authenticity of host '100.0.0.2 (100.0.0.2)' can't be established.
3ECDSA key fingerprint is SHA256:uY6GIjFdI9qTC4QYb980QRk+WblJF9cd5glr3SmmL+w.

Type “yes” when it asks for - Are you sure you want to continue connecting (yes/no)? yes

1Are you sure you want to continue connecting (yes/no)? yes
1/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
2/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
3vagrant@100.0.0.2's password: 
4
5Number of key(s) added: 1
6
7Now try logging into the machine, with:   "ssh '100.0.0.2'"
8and check to make sure that only the key(s) you wanted were added.

Copy to kworker node (During the ssh-copy-id it will ask for the other node password, so in case if you have not set any password then you can supply default password .i.e. vagrant) -

1ssh-copy-id 100.0.0.3
1/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/vagrant/.ssh/id_rsa.pub"
2The authenticity of host '100.0.0.3 (100.0.0.3)' can't be established.
3ECDSA key fingerprint is SHA256:uY6GIjFdI9qTC4QYb980QRk+WblJF9cd5glr3SmmL+w.

Type “yes” when it asks for - Are you sure you want to continue connecting (yes/no)? yes

1Are you sure you want to continue connecting (yes/no)? yes
1/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
2/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
3vagrant@100.0.0.3's password: 
4
5Number of key(s) added: 1
6
7Now try logging into the machine, with:   "ssh '100.0.0.3'"
8and check to make sure that only the key(s) you wanted were added.

Step 5: Install python3-pip (only need to run on ansible node .i.e. amaster)

Before installing the python3-pip, you need to download and update the package list from the repository.

Run the following command(on all the nodes)

1sudo apt-get update

Now you need to install the python3-pip, use the following installation command to install the python3-pip (only need to run on ansible node .i.e. amaster)

1sudo apt install python3-pip

To proceed with the installation press “y”

1Do you want to continue? [Y/n] y

After the installation verify the python and pip version

1python -V
2Python 2.7.15+
1pip3 -V
2pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

Step 6: Clone the kubespray git repo (only need to run on ansible node .i.e. amaster)

In the next step we are going to clone the kubespray. Use the following git command to clone kubespray

1git clone https://github.com/kubernetes-sigs/kubespray.git
1Cloning into 'kubespray'...
2remote: Enumerating objects: 3, done.
3remote: Counting objects: 100% (3/3), done.
4remote: Compressing objects: 100% (3/3), done.
5remote: Total 43626 (delta 0), reused 1 (delta 0), pack-reused 43623
6Receiving objects: 100% (43626/43626), 12.72 MiB | 5.18 MiB/s, done.
7Resolving deltas: 100% (24242/24242), done.

Step 7: Install kubespray package from “requirement.txt” (only need to run on ansible node .i.e. amaster)

Goto “kubespray” directory

1cd kubespray

Install the kubespray packages

1sudo pip3 install -r requirements.txt
 1The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
 2The directory '/home/vagrant/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
 3Collecting ansible==2.9.6 (from -r requirements.txt (line 1))
 4  Downloading https://files.pythonhosted.org/packages/ae/b7/c717363f767f7af33d90af9458d5f1e0960db9c2393a6c221c2ce97ad1aa/ansible-2.9.6.tar.gz (14.2MB)
 5    100% |████████████████████████████████| 14.2MB 123kB/s 
 6Collecting jinja2==2.11.1 (from -r requirements.txt (line 2))
 7  Downloading https://files.pythonhosted.org/packages/27/24/4f35961e5c669e96f6559760042a55b9bcfcdb82b9bdb3c8753dbe042e35/Jinja2-2.11.1-py2.py3-none-any.whl (126kB)
 8    100% |████████████████████████████████| 133kB 4.1MB/s 
 9Collecting netaddr==0.7.19 (from -r requirements.txt (line 3))
10  Downloading https://files.pythonhosted.org/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
11    100% |████████████████████████████████| 1.6MB 954kB/s 
12Collecting pbr==5.4.4 (from -r requirements.txt (line 4))
13  Downloading https://files.pythonhosted.org/packages/7a/db/a968fd7beb9fe06901c1841cb25c9ccb666ca1b9a19b114d1bbedf1126fc/pbr-5.4.4-py2.py3-none-any.whl (110kB)
14    100% |████████████████████████████████| 112kB 7.0MB/s 
15Collecting hvac==0.10.0 (from -r requirements.txt (line 5))
16  Downloading https://files.pythonhosted.org/packages/8d/d7/63e63936792a4c85bea3884003b6d502a040242da2d72db01b0ada4bdb28/hvac-0.10.0-py2.py3-none-any.whl (116kB)
17    100% |████████████████████████████████| 122kB 6.0MB/s 
18Collecting jmespath==0.9.5 (from -r requirements.txt (line 6))
19  Downloading https://files.pythonhosted.org/packages/a3/43/1e939e1fcd87b827fe192d0c9fc25b48c5b3368902bfb913de7754b0dc03/jmespath-0.9.5-py2.py3-none-any.whl
20Collecting ruamel.yaml==0.16.10 (from -r requirements.txt (line 7))
21  Downloading https://files.pythonhosted.org/packages/a6/92/59af3e38227b9cc14520bf1e59516d99ceca53e3b8448094248171e9432b/ruamel.yaml-0.16.10-py2.py3-none-any.whl (111kB)
22    100% |████████████████████████████████| 112kB 5.6MB/s 
23Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from ansible==2.9.6->-r requirements.txt (line 1))
24Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible==2.9.6->-r requirements.txt (line 1))
25Collecting MarkupSafe>=0.23 (from jinja2==2.11.1->-r requirements.txt (line 2))
26  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
27Requirement already satisfied: six>=1.5.0 in /usr/lib/python3/dist-packages (from hvac==0.10.0->-r requirements.txt (line 5))
28Collecting requests>=2.21.0 (from hvac==0.10.0->-r requirements.txt (line 5))
29  Downloading https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl (58kB)
30    100% |████████████████████████████████| 61kB 6.9MB/s 
31Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" (from ruamel.yaml==0.16.10->-r requirements.txt (line 7))
32  Downloading https://files.pythonhosted.org/packages/53/77/4bcd63f362bcb6c8f4f06253c11f9772f64189bf08cf3f40c5ccbda9e561/ruamel.yaml.clib-0.2.0-cp36-cp36m-manylinux1_x86_64.whl (548kB)
33    100% |████████████████████████████████| 552kB 2.5MB/s 
34Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.21.0->hvac==0.10.0->-r requirements.txt (line 5))
35Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.21.0->hvac==0.10.0->-r requirements.txt (line 5))
36Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3/dist-packages (from requests>=2.21.0->hvac==0.10.0->-r requirements.txt (line 5))
37Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.21.0->hvac==0.10.0->-r requirements.txt (line 5))
38Installing collected packages: MarkupSafe, jinja2, ansible, netaddr, pbr, requests, hvac, jmespath, ruamel.yaml.clib, ruamel.yaml
39  Running setup.py install for ansible ... done
40  Found existing installation: requests 2.18.4
41    Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
42Successfully installed MarkupSafe-1.1.1 ansible-2.9.6 hvac-0.10.0 jinja2-2.11.1 jmespath-0.9.5 netaddr-0.7.19 pbr-5.4.4 requests-2.23.0 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0
43

Step 8: Copy inventory file to current users (only need to run on ansible node .i.e. amaster)

Now we need to copy the inventory file to current user using the following command

1cp -rfp inventory/sample inventory/mycluster

Step 9: Prepare host.yml for kubespray (only need to run on ansible node .i.e. amaster)

This step is little trivial because we need to update host.yml with the nodes IP.

Now we are going to declare a variable “IPS” for storing the IP address of other nodes .i.e. kmaster(100.0.0.2), kworker(100.0.0.3)

1declare -a IPS=(100.0.0.2 100.0.0.3)
1CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
 1DEBUG: Adding group all
 2DEBUG: Adding group kube-master
 3DEBUG: Adding group kube-node
 4DEBUG: Adding group etcd
 5DEBUG: Adding group k8s-cluster
 6DEBUG: Adding group calico-rr
 7DEBUG: adding host node1 to group all
 8DEBUG: adding host node2 to group all
 9DEBUG: adding host node1 to group etcd
10DEBUG: adding host node1 to group kube-master
11DEBUG: adding host node2 to group kube-master
12DEBUG: adding host node1 to group kube-node
13DEBUG: adding host node2 to group kube-node

After running the above commands do verify the hosts.yml and it should be like -

1vi inventory/mycluster/hosts.yml
 1all:
 2  hosts:
 3    node1:
 4      ansible_host: 100.0.0.2
 5      ip: 100.0.0.2
 6      access_ip: 100.0.0.2
 7    node2:
 8      ansible_host: 100.0.0.3
 9      ip: 100.0.0.3
10      access_ip: 100.0.0.3
11  children:
12    kube-master:
13      hosts:
14        node1:
15        node2:
16    kube-node:
17      hosts:
18        node1:
19        node2:
20    etcd:
21      hosts:
22        node1:
23    k8s-cluster:
24      children:
25        kube-master:
26        kube-node:
27    calico-rr:
28      hosts: {}

Step 10: Run the ansible-playbook on ansible node .i.e. - amaster (only need to run on ansible node .i.e. amaster)

Now we have done all the prerequisite for running the ansible-playbook.

Use the following ansible-playbook command to begin the installation

1ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Running ansible playbook takes little time because it depends on the network bandwidth also.

During playbook run if you face an error “ansible_memtotal_mb >= minimal_master_memory_mb”, please refer to - How to fix – ansible_memtotal_mb >= minimal_master_memory_mb

Step 11: Install kubectl on kubernetes master .i.e. - kmaster (only need to run on kuebernets node .i.e. kmaster)

Now you need to log into the kubernetes master .i.e. kmaster and download the kubectl onto it.

1curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
1  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2                                 Dload  Upload   Total   Spent    Left  Speed
3100 41.9M  100 41.9M    0     0  5893k      0  0:00:07  0:00:07 --:--:-- 5962k

Now we need to copy the admin.conf file to .kube

1sudo cp /etc/kubernetes/admin.conf /home/vagrant/config
1mkdir .kube
1mv config .kube/
1sudo chown $(id -u):$(id -g ) $HOME/.kube/config

Check the kubectl version after installation

1kubectl version
1Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
2Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Step 12: Verify the kubernetes nodes

Now we have done all the required steps for the installing kubernetes using kubespray.

Lets check the nodes status in out final step

1kubectl get nodes
1NAME    STATUS   ROLES    AGE   VERSION
2node1   Ready    master   13m   v1.18.2
3node2   Ready    master   13m   v1.18.2