CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes

In this article we are going to focus on Setting up CI/CD Jenkins pipeline for kubernetes. We will be using GitHub, Docker, DockerHub, Jenkins and Kubernetes Cluster to achieve this.

For the complete lab session we are going to setup every component from scratch and for that we will be using Vagrant for provisioning the virtual environment. All the VMs will use Ubuntu as its base operating system.

Pre-Requisite

1. Vagrant
2. Virtual Box
3. IDE(Sublime, Atom, or Visual Studio)
4. Kubernetes Cluster (We need kubernetes cluster already setup - If you do not have kubernetes cluster then please do follow this lab session for setting it up - Kubernetes Cluster Setup)

1. Understanding the complete setup

Alright so before we start setting up our CI/CD pipeline I am assuming you have setup your kuernetes cluster already.

As you can see in the sketch we will have three servers(virtual machine) -

1. Jenkins Server
2. Kubernetes Master
3. Kubernetes Worker

On left you can see Jenkins server where we have only installed Jenkins and you should keep in mind that we are not running Jenkins inside kubernetes cluster. (It is not recommended to run Jenkins inside kubernetes cluster. Here is a very good article which you can read if your interested in why?)

On the right hand side you will see two servers one is k8smaster for kubernetes master node and k8sworker for kubernetes worker.

So in short our Jenkins server will connect to k8smaster and k8sworker for doing the continuous deployment.

Since we have provisioned our Servers using Vagrant so here the configuration -

For the reference here is the vagrantfile -

 1Vagrant.configure("2") do |config|
 2  config.vm.define "jenkinsserver" do |jenkinsserver|
 3    jenkinsserver.vm.box_download_insecure = true
 4    jenkinsserver.vm.box = "hashicorp/bionic64"
 5    jenkinsserver.vm.network "forwarded_port", guest: 8080, host: 8080
 6    jenkinsserver.vm.network "forwarded_port", guest: 8081, host: 8081
 7    jenkinsserver.vm.network "forwarded_port", guest: 9090, host: 9090
 8    jenkinsserver.vm.network "private_network", ip: "100.0.0.1"
 9    jenkinsserver.vm.hostname = "jenkinsserver"
10    jenkinsserver.vm.provider "virtualbox" do |v|
11      v.name = "jenkinsserver"
12      v.memory = 2048
13      v.cpus = 2
14    end
15  end
16
17  config.vm.define "k8smaster" do |k8smaster|
18    k8smaster.vm.box_download_insecure = true
19    k8smaster.vm.box = "hashicorp/bionic64"
20    k8smaster.vm.network "private_network", ip: "100.0.0.2"
21    k8smaster.vm.hostname = "k8smaster"
22    k8smaster.vm.provider "virtualbox" do |v|
23      v.name = "k8smaster"
24      v.memory = 2048
25      v.cpus = 2
26    end
27  end
28
29
30  config.vm.define "k8sworker" do |k8sworker|
31    k8sworker.vm.box_download_insecure = true
32    k8sworker.vm.box = "hashicorp/bionic64"
33    k8sworker.vm.network "private_network", ip: "100.0.0.3"
34    k8sworker.vm.hostname = "k8sworker"
35    k8sworker.vm.provider "virtualbox" do |v|
36      v.name = "k8sworker"
37      v.memory = 2048
38      v.cpus = 2
39    end
40  end
41end

2. Where does Github and Docker Hub fits in the CI/CD

We will use Git Hub as version control to push our application code. In this lab session we will be using Spring Boot Application.

Secondly we will use DockerHub for uploading/pushing the Docker image. Here is the overview of our GitHub and DockerHub flow -

Step 1 - Checkin/Push your code to GitHub

Step 2 - Pull your code from GitHub into your Jenkins server

Step 3 - Use Gradle/Maven build tool for building the artifacts

Step 4 - Create Docker image

Step 5 - Push your latest Docker image to DockerHub

Step 6 - Pull the latest image from DockerHub into jenkins.

Step 7 - Then use k8s-spring-boot-deployment.yml to deploy your application inside your kubernetes cluster.

3. Install Jenkins on your jenkinsserver

Alright lets start with Jenkins installation on our jenkinsserver

3.1 Start vagrant box

If your vagrant box is not running then start up your vagrant box -

1vagrant up 

3.2 Log into vagrant

After starting your vagrant box you can login into it -

1vagrant ssh jenkinsserver

3.3 Update ubuntu repositories

Update the repositories of your ubuntu -

1sudo apt update

3.4 Install Java

You also need Java as pre-requisite for installing jenkins, so lets first install java -

1sudo apt install openjdk-8-jdk

Verify your java installation by running the following command

1java -version

It should return with java version

1openjdk version "1.8.0_265"
2OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01)
3OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode) 

3.5 Install Jenkins

Now we can install the jenkins on our Ubuntu server, use the following command to install it -

1wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -

1sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
2    /etc/apt/sources.list.d/jenkins.list'

1sudo apt-get update

1sudo apt-get install jenkins

(Note : For more installation options for MacOS, Windows, CentOS or RedHat please refer to official documentation for jenkins installation)

3.6 Verify Jenkins installation

After installing Jenkins you can verify the jenkins installation by accessing the initial login page of the jenkins.

Since we have installed Jenkins on virtual machine with IP 100.0.0.1, so you can access using following URL

1http://100.0.0.1:8080/

And if you have installed the jenkins correctly then you should be able to see the following initial login page of Jenkins

4. Fetch Default jenkins password

As you can see you need to provide Default jenkins(initialAdminPassword) administrator password.

So the question is - How to find Default Jenkins password(initialAdminPassword)?

There are two ways find Default Jenkins password-

1. Using the Jenkins Logs file located at - /var/log/jenkins/jenkins.log
2. Using the Jenkins secret - /var/lib/jenkins/secrets/initialAdminPassword

For Option 1 - Run the following command -

1cat /var/log/jenkins/jenkins.log

And then look for following -

 1*************************************************************
 2*************************************************************
 3*************************************************************
 4
 5Jenkins initial setup is required. An admin user has been created and a password generated.
 6Please use the following password to proceed to installation:
 7
 8e0d325fb18a64797bd81dcb77e865237
 9
10This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
11
12*************************************************************
13*************************************************************
14*************************************************************

For Option 2 - Run the following command -

1cat /var/lib/jenkins/secrets/initialAdminPassword

It should return you back the password

1e0d325fb18a64797bd81dcb77e865237

Customize Jenkins and install suggested plugin

After you have entered your initialAdminPassword you should be able to see the following Customize jenkins page -

Select install suggested plugin and then it should install all the default plugins which is required for running Jenkins.

After successful login setup username and password

After you have installed all the suggested default plugins, it will prompt you for setting up username and password -

Set desired username and password (In this lab session I am going to keep Username - admin and password - admin)

After that it will ask for Jenkins URL and I would recommend you to keep the default one -

Now you jenkins is ready to use -

5. Jenkins - Install "SSH Pipeline Steps" plugin and "Gradle"

5.1 SSH Pipeline Steps

Moving ahead we need to install one more plugin SSH Pipeline Steps which we are going to use for SSH into k8smaster and k8sworker server.

For installing plugin please goto - Manage Jenkins -> Manage Plugin -> Available then in the search box type SSH Pipeline Steps.

Select the plugin and install it without restart.

5.2 Setup Gradle

For this lab session we are going to use Spring Boot Application, so for that we need Gradle as our build tool.

To setup Gradle Goto - Manage Jenkins -> Global Tool Configuration -> Gradle

Click on Add Grdle and then enter name default.

After that click on the checkbox - Install Automatically and from the drop down Install from Gradle.org select latest version.

Refer to the following screenshot

Now we have installed Jenkins and all the required plugins which is needed for this lab session.

6. Install Docker on jenkinsserver

Alright now we need to install Docker on jenkinsserver since we need to push our docker image to DockerHub.

(Note - We are not installing any plugins for jenkins, till the step no 5 we have completed the jenkins and jenkins plugin setup)

6.1 Logback into jenkinsserver

1vagrant ssh jenkinsserver

6.2 Install Docker

Use the following command to install the Docker -

1sudo apt install docker.io

After installing docker verify the installation with following docker command -

1Client:
2 Version:           19.03.6
3 API version:       1.40
4 Go version:        go1.12.17
5 Git commit:        369ce74a3c
6 Built:             Wed Oct 14 19:00:27 2020
7 OS/Arch:           linux/amd64
8 Experimental:      false 

6.3 Add Current User to Docker Group

The next step you need to do is to add the current user to Docker Group to avoid the error - Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock (For more detail refer to this post)

Use the following command for adding current user to Docker Group

1sudo usermod -aG docker $USER 

6.4 Add Jenkins User to Docker Group

Similar to previous step we also need to add Jenkins User to Docker Group, so that Jenkins can use Docker for building and pusing the docker images.

Run the following command to add Jenkins user to Docker Group

1sudo usermod -aG docker jenkins 

7. Take a look at Spring Boot Application

Note - In this step we are using Spring Boot Application but if you have different application like NodeJs, Angular etc than you can use that application. Rest of the steps are going to be the same.

Now its time for you to look at our application which we are going to deploy inside kubernetes cluster using Jenkins Pipeline.

To keep the lab session simple we are going to write Hello World rest webservice using Spring Boot.

Here is Java class of rest end point -

1@RestController
2public class JhooqDockerDemoController {
3
4    @GetMapping("/hello")
5    public String hello() {
6        return "Docker Demo - Hello Jhooq";
7    }
8}

You can clone the code repo from - Source Code

The source code also include the Dockerfile for building the dockerimage -

1FROM openjdk:11
2ARG JAR_FILE=build/libs/*.jar
3COPY ${JAR_FILE} app.jar
4ENTRYPOINT ["java","-jar","/app.jar"]

8. Write the Pipeline script

To make this lab session more simple I have divided CI/CD Jenkins Pipeline script into 11 steps

8.1 Create Jenkins Pipeline

The first step for you would be to create a pipeline.

1. Goto : Jenkins -> New Items
2. Enter an item name : Jhooq-SpringBoot
3. Select Pipeline
4. Click Ok

8.2 Clone the Git Repo

The first principle of the CI/CD pipeline is to clone/checkout the source code, using the same principle we are going to clone the GIT Repo inside Jenkins

1stage("Git Clone"){
2
3        git credentialsId: 'GIT_HUB_CREDENTIALS', url: 'https://github.com/rahulwagh/spring-boot-docker.git'
4    }

8.3 Jenkins store git credentials

As you know we cannot store plain text password inside jenkins scripts, so we need to store it somewhere securely.

Jenkins Manage Credential provides very elegant way to store GitHub Username and Password.

Goto : Jenkins -> Manage Jenkins -> Manage Credentials

After that Goto : Stores scoped to Jenkins -> global

Then select Username with Password

And then input you GitHub Username and Password. But always remember the ID

Note - Keep the ID somewhere store so that you remember - GIT_HUB_CREDENTIALS

8.4 Build the Spring Boot Application

Next step would be to build the Spring Boot Application Using Gradle

1stage('Gradle Build') {
2
3       sh './gradlew build'
4
5    } 

8.5 Build Docker image and tag it

After successful Gradle Build we are going to build Docker Image and after that I am going to tag it with the name jhooq-docker-demo

1stage("Docker build"){
2        sh 'docker version'
3        sh 'docker build -t jhooq-docker-demo .'
4        sh 'docker image list'
5        sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
6    } 

8.6 Jenkins store DockerHub credentials

For storing DockerHub Credentials you need to GOTO: Jenkins -> Manage Jenkins -> Manage Credentials -> Stored scoped to jenkins -> global -> Add Credentials

From the Kind dropdown please select Secret text

1. Secret - Type in the DockerHub Password
2. ID - DOCKER_HUB_PASSWORD
3. Description - Docker Hub password

8.7 Docker Login via CLI

Since I am working inside Jenkins so every step I perform I need to write pipeline script. Now after building and tagging the Docker Image we need to push it to the DockerHub. But before you push to DockerHub you need to authenticate yourself via CLI(command line interface) using docker login

So here is the pipeline step for Docker Login

1stage("Docker Login"){
2        withCredentials([string(credentialsId: 'DOCKER_HUB_PASSWORD', variable: 'PASSWORD')]) {
3            sh 'docker login -u rahulwagh17 -p $PASSWORD'
4        }
5    } 

$DOCKER_HUB_PASSWORD - Since I cann't disclose my DockerHub password, so I stored my DockerHub Password into Jenkins Manage Jenkins and assigned the ID $DOCKER_HUB_PASSWORD

8.8 Push Docker Image into DockerHub

After successful Docker login now we need to push the image to DockerHub

1stage("Push Image to Docker Hub"){
2        sh 'docker push  rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
3    }

8.9 SSH Into k8smaster server

If you remember we have installed SSH Pipeline Steps in step no - 5, now we are going to use that plugin to SSH into k8smaster server

1stage("SSH Into k8s Server") {
2        def remote = [:]
3        remote.name = 'K8S master'
4        remote.host = '100.0.0.2'
5        remote.user = 'vagrant'
6        remote.password = 'vagrant'
7        remote.allowAnyHosts = true
8} 

8.10 Copy k8s-spring-boot-deployment.yml to k8smaster server

After successful login copy k8s-spring-boot-deployment.yml into k8smaster server

1stage('Put k8s-spring-boot-deployment.yml onto k8smaster') {
2            sshPut remote: remote, from: 'k8s-spring-boot-deployment.yml', into: '.'
3        } 

8.11 Create kubernetes deployment and service

Apply the k8s-spring-boot-deployment.yml which will eventually -

1. Create deployment with name - jhooq-springboot
2. Expose service on NodePort

1stage('Deploy spring boot') {
2          sshCommand remote: remote, command: "kubectl apply -f k8s-spring-boot-deployment.yml"
3        }

So here is the final complete pipeline script for my CI/CD Jenkins kubernetes pipeline

 1node {
 2
 3    stage("Git Clone"){
 4
 5        git credentialsId: 'GIT_CREDENTIALS', url: 'https://github.com/rahulwagh/spring-boot-docker.git'
 6    }
 7
 8     stage('Gradle Build') {
 9
10       sh './gradlew build'
11
12    }
13
14    stage("Docker build"){
15        sh 'docker version'
16        sh 'docker build -t jhooq-docker-demo .'
17        sh 'docker image list'
18        sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
19    }
20
21    withCredentials([string(credentialsId: 'DOCKER_HUB_PASSWORD', variable: 'PASSWORD')]) {
22        sh 'docker login -u rahulwagh17 -p $PASSWORD'
23    }
24
25    stage("Push Image to Docker Hub"){
26        sh 'docker push  rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
27    }
28
29    stage("SSH Into k8s Server") {
30        def remote = [:]
31        remote.name = 'K8S master'
32        remote.host = '100.0.0.2'
33        remote.user = 'vagrant'
34        remote.password = 'vagrant'
35        remote.allowAnyHosts = true
36
37        stage('Put k8s-spring-boot-deployment.yml onto k8smaster') {
38            sshPut remote: remote, from: 'k8s-spring-boot-deployment.yml', into: '.'
39        }
40
41        stage('Deploy spring boot') {
42          sshCommand remote: remote, command: "kubectl apply -f k8s-spring-boot-deployment.yml"
43        }
44    }
45
46}
47 

Conclusion

1. First thing which I did is - setup the kubernetes cluster
2. Install Jenkins on another server
3. Install plugin 'SSH Pipeline Step' for jenkins
4. Install Docker along with Jenkins
5. Setup user group for CurrentUser and Jenkins
6. I created Jenkins Pipeline script for continuous Deployment

Hope you liked lab session, please follow me on YouTube

Learn more On Kubernetes -

1. Setup kubernetes on Ubuntu
2. Setup Kubernetes on CentOs
3. Setup HA Kubernetes Cluster with Kubespray
4. Setup HA Kubernetes with Minikube
5. Setup Kubernetes Dashboard for local kubernetes cluster
6. Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
7. How to use Persistent Volume and Persistent Volume Claims in Kubernetes
8. Deploy Spring Boot Microservice on local Kubernetes cluster
9. Deploy Spring Boot Microservice on Cloud Platform(GCP)
10. Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
11. CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
12. kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
13. How to setup kubernetes jenkins pipeline on AWS?
14. Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
15. How to fix kubernetes pods getting recreated?
16. How to delete all kubernetes PODS?
17. How to use Kubernetes secrets?
18. Share kubernetes secrets between namespaces?
19. How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
20. Delete Kubernetes POD stuck in terminating state?