CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
In this article we are going to focus on Setting up CI/CD Jenkins pipeline for kubernetes. We will be using GitHub, Docker, DockerHub, Jenkins and Kubernetes Cluster to achieve this.
For the complete lab session we are going to setup every component from scratch and for that we will be using Vagrant for provisioning the virtual environment. All the VMs will use Ubuntu as its base operating system.
Pre-Requisite
- Vagrant
- Virtual Box
- IDE(Sublime, Atom, or Visual Studio)
- Kubernetes Cluster (We need kubernetes cluster already setup - If you do not have kubernetes cluster then please do follow this lab session for setting it up - Kubernetes Cluster Setup)
1. Understanding the complete setup
Alright so before we start setting up our CI/CD pipeline I am assuming you have setup your kuernetes cluster already.
As you can see in the sketch we will have three servers(virtual machine) -
- Jenkins Server
- Kubernetes Master
- Kubernetes Worker
On left you can see Jenkins server where we have only installed Jenkins and you should keep in mind that we are not running Jenkins inside kubernetes cluster. (It is not recommended to run Jenkins inside kubernetes cluster. Here is a very good article which you can read if your interested in why?)
On the right hand side you will see two servers one is k8smaster for kubernetes master node and k8sworker for kubernetes worker.
So in short our Jenkins server will connect to k8smaster and k8sworker for doing the continuous deployment.
Since we have provisioned our Servers using Vagrant so here the configuration -
Server Name | Memory | CPU | IP Address |
---|---|---|---|
Jenkins server (jenkinsserver) | 2 Gigs | 2 | 100.0.0.1 |
Kubernetes Master (k8smaster) | 2 Gigs | 2 | 100.0.0.2 |
Kubernetes Worker (k8sworker) | 2 Gigs | 2 | 100.0.0.3 |
For the reference here is the vagrantfile -
1Vagrant.configure("2") do |config|
2 config.vm.define "jenkinsserver" do |jenkinsserver|
3 jenkinsserver.vm.box_download_insecure = true
4 jenkinsserver.vm.box = "hashicorp/bionic64"
5 jenkinsserver.vm.network "forwarded_port", guest: 8080, host: 8080
6 jenkinsserver.vm.network "forwarded_port", guest: 8081, host: 8081
7 jenkinsserver.vm.network "forwarded_port", guest: 9090, host: 9090
8 jenkinsserver.vm.network "private_network", ip: "100.0.0.1"
9 jenkinsserver.vm.hostname = "jenkinsserver"
10 jenkinsserver.vm.provider "virtualbox" do |v|
11 v.name = "jenkinsserver"
12 v.memory = 2048
13 v.cpus = 2
14 end
15 end
16
17 config.vm.define "k8smaster" do |k8smaster|
18 k8smaster.vm.box_download_insecure = true
19 k8smaster.vm.box = "hashicorp/bionic64"
20 k8smaster.vm.network "private_network", ip: "100.0.0.2"
21 k8smaster.vm.hostname = "k8smaster"
22 k8smaster.vm.provider "virtualbox" do |v|
23 v.name = "k8smaster"
24 v.memory = 2048
25 v.cpus = 2
26 end
27 end
28
29
30 config.vm.define "k8sworker" do |k8sworker|
31 k8sworker.vm.box_download_insecure = true
32 k8sworker.vm.box = "hashicorp/bionic64"
33 k8sworker.vm.network "private_network", ip: "100.0.0.3"
34 k8sworker.vm.hostname = "k8sworker"
35 k8sworker.vm.provider "virtualbox" do |v|
36 v.name = "k8sworker"
37 v.memory = 2048
38 v.cpus = 2
39 end
40 end
41end
2. Where does Github and Docker Hub fits in the CI/CD
We will use Git Hub as version control to push our application code. In this lab session we will be using Spring Boot Application.
Secondly we will use DockerHub for uploading/pushing the Docker image. Here is the overview of our GitHub and DockerHub flow -
Step 1 - Checkin/Push your code to GitHub
Step 2 - Pull your code from GitHub into your Jenkins server
Step 3 - Use Gradle/Maven build tool for building the artifacts
Step 4 - Create Docker image
Step 5 - Push your latest Docker image to DockerHub
Step 6 - Pull the latest image from DockerHub into jenkins.
Step 7 - Then use k8s-spring-boot-deployment.yml to deploy your application inside your kubernetes cluster.
3. Install Jenkins on your jenkinsserver
Alright lets start with Jenkins installation on our jenkinsserver
3.1 Start vagrant box
If your vagrant box is not running then start up your vagrant box -
1vagrant up
3.2 Log into vagrant
After starting your vagrant box you can login into it -
1vagrant ssh jenkinsserver
3.3 Update ubuntu repositories
Update the repositories of your ubuntu -
1sudo apt update
3.4 Install Java
You also need Java as pre-requisite for installing jenkins, so lets first install java -
1sudo apt install openjdk-8-jdk
Verify your java installation by running the following command
1java -version
It should return with java version
1openjdk version "1.8.0_265"
2OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01)
3OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)
3.5 Install Jenkins
Now we can install the jenkins on our Ubuntu server, use the following command to install it -
1wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
1sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > \
2 /etc/apt/sources.list.d/jenkins.list'
1sudo apt-get update
1sudo apt-get install jenkins
(Note : For more installation options for MacOS, Windows, CentOS or RedHat please refer to official documentation for jenkins installation)
3.6 Verify Jenkins installation
After installing Jenkins you can verify the jenkins installation by accessing the initial login page of the jenkins.
Since we have installed Jenkins on virtual machine with IP 100.0.0.1, so you can access using following URL
1http://100.0.0.1:8080/
And if you have installed the jenkins correctly then you should be able to see the following initial login page of Jenkins
4. Fetch Default jenkins password
As you can see you need to provide Default jenkins(initialAdminPassword) administrator password.
So the question is - How to find Default Jenkins password(initialAdminPassword)?
There are two ways find Default Jenkins password-
- Using the Jenkins Logs file located at -
/var/log/jenkins/jenkins.log
- Using the Jenkins secret -
/var/lib/jenkins/secrets/initialAdminPassword
For Option 1 - Run the following command -
1cat /var/log/jenkins/jenkins.log
And then look for following -
1*************************************************************
2*************************************************************
3*************************************************************
4
5Jenkins initial setup is required. An admin user has been created and a password generated.
6Please use the following password to proceed to installation:
7
8e0d325fb18a64797bd81dcb77e865237
9
10This may also be found at: /var/lib/jenkins/secrets/initialAdminPassword
11
12*************************************************************
13*************************************************************
14*************************************************************
For Option 2 - Run the following command -
1cat /var/lib/jenkins/secrets/initialAdminPassword
It should return you back the password
1e0d325fb18a64797bd81dcb77e865237
Customize Jenkins and install suggested plugin
After you have entered your initialAdminPassword you should be able to see the following Customize jenkins page -
Select install suggested plugin and then it should install all the default plugins which is required for running Jenkins.
After successful login setup username and password
After you have installed all the suggested default plugins, it will prompt you for setting up username and password -
Set desired username and password (In this lab session I am going to keep Username - admin and password - admin)
After that it will ask for Jenkins URL and I would recommend you to keep the default one -
Now you jenkins is ready to use -
5. Jenkins - Install "SSH Pipeline Steps" plugin and "Gradle"
5.1 SSH Pipeline Steps
Moving ahead we need to install one more plugin SSH Pipeline Steps which we are going to use for SSH into k8smaster and k8sworker server.
For installing plugin please goto - Manage Jenkins -> Manage Plugin -> Available
then in the search box type SSH Pipeline Steps.
Select the plugin and install it without restart.
5.2 Setup Gradle
For this lab session we are going to use Spring Boot Application, so for that we need Gradle as our build tool.
To setup Gradle Goto - Manage Jenkins -> Global Tool Configuration -> Gradle
Click on Add Grdle
and then enter name default
.
After that click on the checkbox - Install Automatically
and from the drop down Install from Gradle.org
select latest version.
Refer to the following screenshot
Now we have installed Jenkins and all the required plugins which is needed for this lab session.
6. Install Docker on jenkinsserver
Alright now we need to install Docker on jenkinsserver since we need to push our docker image to DockerHub.
(Note - We are not installing any plugins for jenkins, till the step no 5 we have completed the jenkins and jenkins plugin setup)
6.1 Logback into jenkinsserver
1vagrant ssh jenkinsserver
6.2 Install Docker
Use the following command to install the Docker -
1sudo apt install docker.io
After installing docker verify the installation with following docker command -
1Client:
2 Version: 19.03.6
3 API version: 1.40
4 Go version: go1.12.17
5 Git commit: 369ce74a3c
6 Built: Wed Oct 14 19:00:27 2020
7 OS/Arch: linux/amd64
8 Experimental: false
6.3 Add Current User to Docker Group
The next step you need to do is to add the current user to Docker Group to avoid the error - Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock (For more detail refer to this post)
Use the following command for adding current user to Docker Group
1sudo usermod -aG docker $USER
6.4 Add Jenkins User to Docker Group
Similar to previous step we also need to add Jenkins User to Docker Group, so that Jenkins can use Docker for building and pusing the docker images.
Run the following command to add Jenkins user to Docker Group
1sudo usermod -aG docker jenkins
7. Take a look at Spring Boot Application
Note - In this step we are using Spring Boot Application but if you have different application like NodeJs, Angular etc than you can use that application. Rest of the steps are going to be the same.
Now its time for you to look at our application which we are going to deploy inside kubernetes cluster using Jenkins Pipeline.
To keep the lab session simple we are going to write Hello World rest webservice using Spring Boot.
Here is Java class of rest end point -
1@RestController
2public class JhooqDockerDemoController {
3
4 @GetMapping("/hello")
5 public String hello() {
6 return "Docker Demo - Hello Jhooq";
7 }
8}
You can clone the code repo from - Source Code
The source code also include the Dockerfile for building the dockerimage -
1FROM openjdk:11
2ARG JAR_FILE=build/libs/*.jar
3COPY ${JAR_FILE} app.jar
4ENTRYPOINT ["java","-jar","/app.jar"]
8. Write the Pipeline script
To make this lab session more simple I have divided CI/CD Jenkins Pipeline script into 11 steps
8.1 Create Jenkins Pipeline
The first step for you would be to create a pipeline.
- Goto :
Jenkins -> New Items
- Enter an item name :
Jhooq-SpringBoot
- Select Pipeline
- Click Ok
8.2 Clone the Git Repo
The first principle of the CI/CD pipeline is to clone/checkout the source code, using the same principle we are going to clone the GIT Repo inside Jenkins
1stage("Git Clone"){
2
3 git credentialsId: 'GIT_HUB_CREDENTIALS', url: 'https://github.com/rahulwagh/spring-boot-docker.git'
4 }
8.3 Jenkins store git credentials
As you know we cannot store plain text password inside jenkins scripts, so we need to store it somewhere securely.
Jenkins Manage Credential provides very elegant way to store GitHub Username and Password.
Goto : Jenkins -> Manage Jenkins -> Manage Credentials
After that Goto : Stores scoped to Jenkins -> global
Then select Username with Password
And then input you GitHub Username and Password. But always remember the ID
Note - Keep the ID somewhere store so that you remember - GIT_HUB_CREDENTIALS
8.4 Build the Spring Boot Application
Next step would be to build the Spring Boot Application Using Gradle
1stage('Gradle Build') {
2
3 sh './gradlew build'
4
5 }
8.5 Build Docker image and tag it
After successful Gradle Build we are going to build Docker Image and after that I am going to tag it with the name jhooq-docker-demo
1stage("Docker build"){
2 sh 'docker version'
3 sh 'docker build -t jhooq-docker-demo .'
4 sh 'docker image list'
5 sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
6 }
8.6 Jenkins store DockerHub credentials
For storing DockerHub Credentials you need to GOTO: Jenkins -> Manage Jenkins -> Manage Credentials -> Stored scoped to jenkins -> global -> Add Credentials
From the Kind
dropdown please select Secret text
- Secret - Type in the DockerHub Password
- ID - DOCKER_HUB_PASSWORD
- Description - Docker Hub password
8.7 Docker Login via CLI
Since I am working inside Jenkins so every step I perform I need to write pipeline script.
Now after building and tagging the Docker Image we need to push it to the DockerHub.
But before you push to DockerHub you need to authenticate yourself via CLI(command line interface) using docker login
So here is the pipeline step for Docker Login
1stage("Docker Login"){
2 withCredentials([string(credentialsId: 'DOCKER_HUB_PASSWORD', variable: 'PASSWORD')]) {
3 sh 'docker login -u rahulwagh17 -p $PASSWORD'
4 }
5 }
$DOCKER_HUB_PASSWORD
- Since I cann't disclose my DockerHub password, so I stored my DockerHub Password into Jenkins Manage Jenkins
and assigned the ID $DOCKER_HUB_PASSWORD
8.8 Push Docker Image into DockerHub
After successful Docker login now we need to push the image to DockerHub
1stage("Push Image to Docker Hub"){
2 sh 'docker push rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
3 }
8.9 SSH Into k8smaster server
If you remember we have installed SSH Pipeline Steps in step no - 5, now we are going to use that plugin to SSH into k8smaster server
1stage("SSH Into k8s Server") {
2 def remote = [:]
3 remote.name = 'K8S master'
4 remote.host = '100.0.0.2'
5 remote.user = 'vagrant'
6 remote.password = 'vagrant'
7 remote.allowAnyHosts = true
8}
8.10 Copy k8s-spring-boot-deployment.yml to k8smaster server
After successful login copy k8s-spring-boot-deployment.yml into k8smaster server
1stage('Put k8s-spring-boot-deployment.yml onto k8smaster') {
2 sshPut remote: remote, from: 'k8s-spring-boot-deployment.yml', into: '.'
3 }
8.11 Create kubernetes deployment and service
Apply the k8s-spring-boot-deployment.yml which will eventually -
- Create deployment with name - jhooq-springboot
- Expose service on NodePort
1stage('Deploy spring boot') {
2 sshCommand remote: remote, command: "kubectl apply -f k8s-spring-boot-deployment.yml"
3 }
So here is the final complete pipeline script for my CI/CD Jenkins kubernetes pipeline
1node {
2
3 stage("Git Clone"){
4
5 git credentialsId: 'GIT_CREDENTIALS', url: 'https://github.com/rahulwagh/spring-boot-docker.git'
6 }
7
8 stage('Gradle Build') {
9
10 sh './gradlew build'
11
12 }
13
14 stage("Docker build"){
15 sh 'docker version'
16 sh 'docker build -t jhooq-docker-demo .'
17 sh 'docker image list'
18 sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
19 }
20
21 withCredentials([string(credentialsId: 'DOCKER_HUB_PASSWORD', variable: 'PASSWORD')]) {
22 sh 'docker login -u rahulwagh17 -p $PASSWORD'
23 }
24
25 stage("Push Image to Docker Hub"){
26 sh 'docker push rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
27 }
28
29 stage("SSH Into k8s Server") {
30 def remote = [:]
31 remote.name = 'K8S master'
32 remote.host = '100.0.0.2'
33 remote.user = 'vagrant'
34 remote.password = 'vagrant'
35 remote.allowAnyHosts = true
36
37 stage('Put k8s-spring-boot-deployment.yml onto k8smaster') {
38 sshPut remote: remote, from: 'k8s-spring-boot-deployment.yml', into: '.'
39 }
40
41 stage('Deploy spring boot') {
42 sshCommand remote: remote, command: "kubectl apply -f k8s-spring-boot-deployment.yml"
43 }
44 }
45
46}
47
Conclusion
- First thing which I did is - setup the kubernetes cluster
- Install Jenkins on another server
- Install plugin 'SSH Pipeline Step' for jenkins
- Install Docker along with Jenkins
- Setup user group for CurrentUser and Jenkins
- I created Jenkins Pipeline script for continuous Deployment
Hope you liked lab session, please follow me on YouTube
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?
Posts in this Series
- Kubernetes Cheat Sheet for day to day DevOps operations?
- Delete Kubernetes POD stuck in terminating state?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Share kubernetes secrets between namespaces?
- How to use Kubernetes secrets?
- How to delete all kubernetes PODS?
- kubernetes pods getting recreated?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- kubectl export yaml OR How to generate YAML for deployed kubernetes resources
- Kubernetes Updates
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- Kubernetes cluster setup with Jenkins
- How to use Persistent Volume and Persistent Claims | Kubernetes
- How to fix ProvisioningFailed persistentvolume controller no volume plugin matched
- Fixing – Cannot bind to requested volume: storageClasseName does not match
- Fixing – pod has unbound immediate persistentvolumeclaims or cannot bind to requested volume incompatible accessmode
- How to fix kubernetes dashboard forbidden 403 error – message services https kubernetes-dashboard is forbidden User
- How to fix Kubernetes – error execution phase preflight [preflight]
- Deploy Spring Boot microservices on kubernetes?
- How to fix – ansible_memtotal_mb minimal_master_memory_mb
- How to use kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster
- How to setup kubernetes on CentOS 8 and CentOS 7
- How to fix – How to fix - ERROR Swap running with swap on is not supported. Please disable swap
- 14 Steps to Install kubernetes on Ubuntu 20.04(bento/ubuntu-20.04), 18.04(hashicorp/bionic64)
- Kubernetes Dashboard | Kubernetes Admin GUI | Kubernetes Desktop Client
- Install Kubernetes with Minikube