Kubernetes cluster setup with Jenkins

Sep 24, 2020 · 9 min read · Kubernetes ·

In the DevOps world if you do not talk about Jenkins, Continuous Integration, and Continuous Delivery then you are missing a big chunk of it and with the inception of Kubernetes DevOps has grown its territory by 10 folds.

In this article, we are going to integrate -

Jenkins
Kuberneteshttps

Pre-Requisite

Vagrant
Virtual Box
IDE(Sublime, Atom, or Visual Studio)

1. Spin-up the vagrant box Vagranfile

The first and the far most we need a Kubernetes cluster and for that we need a virtual machine.

So we are going to spin up three virtual machine -

amaster - Ansible master for provisioning kubernetes cluster using kubespray
k8smaster - Kubernetes Master
k8sworker - Kubernetes Worker

(**Note - We will be using Ubuntu as our base operating system in the virtual machine)

Here is the Vagrantfile

 1Vagrant.configure("2") do |config|
 2  config.vm.define "amaster" do |amaster|
 3    amaster.vm.box_download_insecure = true
 4    amaster.vm.box = "hashicorp/bionic64"
 5    amaster.vm.network "forwarded_port", guest: 8080, host: 8080
 6    amaster.vm.network "forwarded_port", guest: 8081, host: 8081
 7    amaster.vm.network "private_network", ip: "100.0.0.1"
 8    amaster.vm.hostname = "amaster"
 9    amaster.vm.provider "virtualbox" do |v|
10      v.name = "amaster"
11      v.memory = 2048
12      v.cpus = 2
13    end
14  end
15
16  config.vm.define "k8smaster" do |k8smaster|
17    k8smaster.vm.box_download_insecure = true
18    k8smaster.vm.box = "hashicorp/bionic64"
19    k8smaster.vm.network "private_network", ip: "100.0.0.2"
20    k8smaster.vm.hostname = "k8smaster"
21    k8smaster.vm.provider "virtualbox" do |v|
22      v.name = "k8smaster"
23      v.memory = 2048
24      v.cpus = 2
25    end
26  end
27
28
29  config.vm.define "k8sworker" do |k8sworker|
30    k8sworker.vm.box_download_insecure = true
31    k8sworker.vm.box = "hashicorp/bionic64"
32    k8sworker.vm.network "private_network", ip: "100.0.0.3"
33    k8sworker.vm.hostname = "k8sworker"
34    k8sworker.vm.provider "virtualbox" do |v|
35      v.name = "k8sworker"
36      v.memory = 2048
37      v.cpus = 2
38    end
39  end
40`
41end

Next Question - You need a Kubernetes cluster and How to setup you Kubernetes cluster?

Follow this article Setup Kubernetes Cluster

Once you have done setting up the Kubernetes cluster then go ahead and login to the Kubernetes cluster.

1vagrant ssh k8smaster

2. Install Docker and docker-compose

The next step would be for you to install Docker and docker-compose.

Let's go ahead and install Docker first

First, update the repository

1sudo apt-get update

Then run the following Docker install command

1sudo apt install docker.io

You can verify the Docker installation by running the following

1docker version

It should return you with the current docker installed version

1Client: Docker Engine - Community
2 Version:           19.03.13
3 API version:       1.40
4 Go version:        go1.13.15
5 Git commit:        4484c46d9d
6 Built:             Wed Sep 16 17:02:36 2020
7 OS/Arch:           linux/amd64
8 Experimental:      false

Alright now we are done with installing Docker on our system, now its time to install docker-compose

Use the following command to download all the required packages needed for docker-compose

1sudo curl -L "https://github.com/docker/compose/releases/download/1.27.3/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Then change the permission of the directory /usr/local/bin/docker-compose and make it executable

1sudo chmod +x /usr/local/bin/docker-compose

Then let's create some hard links to the directory /usr/local/bin/docker-compose

1sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Now you need to add the current user to the docker group otherwise you will get a docker exception

1sudo usermod -aG docker $USER

After that restart the vagrant box and verify the docker-compose installation by running the following command

1sudo usermod -aG docker $USER

If you successfully installed the docker-compose than it should return you

1docker-compose version 1.27.3, build 4092ae5d

3. Setup jenkins cluster role - jenkins-cluster-role.yml

The first step towards setting up Jenkins is to create the cluster role.

Use the following jenkins-cluster-role.yml and apply it.

 1kind: ClusterRole
 2apiVersion: rbac.authorization.k8s.io/v1
 3metadata:
 4  namespace: default
 5  name: service-reader
 6rules:
 7  - apiGroups: [""] # "" indicates the core API group
 8    resources: ["services"]
 9    verbs: ["get", "watch", "list"]
10  - apiGroups: [""]
11    resources: ["pods"]
12    verbs: ["create","delete","get","list","patch","update","watch"]
13  - apiGroups: [""]
14    resources: ["pods/exec"]
15    verbs: ["create","delete","get","list","patch","update","watch"]
16  - apiGroups: [""]
17    resources: ["pods/log"]
18    verbs: ["get","list","watch"]
19  - apiGroups: [""]
20    resources: ["secrets"]
21    verbs: ["get"]

To apply the jenkins-cluster-role.yml use the following kubectl command

1kubectl apply -f jenkins-cluster-role.yml

Now we need to create cluster role binding named "service reader pod" .

Type	Name
clusterrole	service reader
name	default

Use the following command to achieve that -

1kubectl create clusterrolebinding service-reader-pod --clusterrole=service-reader  --serviceaccount=default:default

4. Create a persistent volume for storing Jenkins settings - jenkins-persistent-volume.yml

Now after you have created cluster role now its time to create persistent volume because we need to store the Jenkins configuration.

Create a yaml - jenkins-persistent-volume.yml

 1apiVersion: v1
 2kind: PersistentVolume
 3metadata:
 4  name: k8sdemo-jenkins-pv
 5  labels:
 6    app: k8sdemo-jenkins
 7spec:
 8  capacity:
 9    storage: 1Gi
10  volumeMode: Filesystem
11  accessModes:
12  - ReadWriteOnce
13  persistentVolumeReclaimPolicy: Retain
14  storageClassName: local-storage
15  local:
16    path: /home/vagrant/storage
17  nodeAffinity:
18    required:
19      nodeSelectorTerms:
20      - matchExpressions:
21        - key: kubernetes.io/hostname
22          operator: In
23          values:
24          - node1

Now apply the peristent-volume configuration

1kubectl apply -f jenkins-persistent-volume.yml

5. Create persistent volume claim for Jenkins - jenkins-pvc.yml

In this step, you need to define the persistent volume claim for your Jenkins.

Create a YAML - jenkins-pvc.yml

 1apiVersion: v1
 2kind: PersistentVolumeClaim
 3metadata:
 4  name: k8sdemo-jenkins-pvclaim
 5  labels:
 6    app: k8sdemo-jenkins
 7spec:
 8  accessModes:
 9    - ReadWriteOnce
10  storageClassName: local-storage
11  resources:
12    requests:
13      storage: 1Gi #1 GB

As always apply the persistent volume claim configuration

1kubectl apply -f jenkins-pvc.yml

6. Jenkins deployment (jenkins/jenkins:lts) - jenkins-deployment.yml

So till now, everything which we did is just pre-requisite for the jenkins.

Now we are going to install the Jenkins and for that, we are going to create jenkins-deployment.yml

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: k8sdemo-jenkins-deployment
 5  labels:
 6    app: k8sdemo-jenkins
 7spec:
 8  selector:
 9    matchLabels:
10      app: k8sdemo-jenkins
11  strategy:
12    type: Recreate
13  template:
14    metadata:
15      labels:
16        app: k8sdemo-jenkins
17    spec:
18      serviceAccountName: default # The name of the service account is default
19      containers:
20        - image: jenkins/jenkins:lts
21          name: k8sdemo-jenkins-container
22          #imagePullPolicy: Never
23          ports:
24            - containerPort: 8080
25            - containerPort: 50000
26          volumeMounts:
27            - name: k8sdemo-jenkins-persistentstorage
28              mountPath: /var/jenkins_home
29      volumes:
30        - name: k8sdemo-jenkins-persistentstorage
31          persistentVolumeClaim:
32            claimName: k8sdemo-jenkins-pvclaim

Key point - Here we are using jenkins long term support docker image .i.e. jenkins/jenkins:lts

Okay let's apply the configuration

1kubectl apply -f jenkins-deployment.yml

You can verify your jenkins deployment using the following command -

1kubectl get deployment

It should return you with something similar -

jenkins deployment status inside kubernetes

7. Expose Jenkins deployment as Service - jenkins-service.yml

After deploying Jenkins now we need to expose the service so that it can be accessed outside of the Kubernetes cluster.

And for exposing it to the outside of the cluster lets create - jenkins-service.yml

 1apiVersion: v1
 2kind: Service
 3metadata:
 4  name: k8sdemo-jenkins-service
 5  labels:
 6    app: k8sdemo-jenkins
 7spec:
 8  type: NodePort
 9  selector:
10    app: k8sdemo-jenkins
11  ports:
12    - port: 8080
13      name: http
14      protocol : TCP
15      nodePort: 30080
16      targetPort: 8080
17    - port: 50000
18      name: agent
19      protocol: TCP
20      targetPort: 50000

Keypoint - Here we are exposing Jenkins service using NodePort so that it can be accessed outside of the cluster.

Alright let's apply this configuration

1kubectl apply -f jenkins-service.yml

Once you have applied the configuration then you can verify it with the following command

1kubectl get service

Expose jenkins service using NodePort from inside kubernetes cluster

Congratulations you have installed Jenkins on the Kubernetes cluster and it can be accessed using the following URL -

1http://100.0.0.2:30080/

Here is the login screen of the Jenkins

Jenkins login screen after installing on kuberenetes cluster

8. How to find Jenkins initial Administrator password

Now you need to find Jenkins initial admin password and for that first, you need to get your Kubernetes pod name

1kubectl get pods

It should return you with something similar -

Get Kubernetes pod details for finding Jenkins initial admin password

Now we know jenkins pod name and in my case, it is - k8sdemo-jenkins-deployment-7b4ccf5755-4hj8s

(*The Jenkins pod name can be different in your case, so be careful while picking up the pod name)

Alright let run the log command on the Kubernetes pod to find the Jenkins initial admin password

1kubectl logs k8sdemo-jenkins-deployment-7b4ccf5755-4hj8s

It should return the pods logs along with Jenkins initial admin password

 1Running from: /usr/share/jenkins/jenkins.war
 2webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
 32020-10-04 12:52:15.343+0000 [id=1]    INFO   org.eclipse.jetty.util.log.Log#initialized: Logging initialized @731ms to org.eclipse.jetty.util.log.JavaUtilLog
 42020-10-04 12:52:15.512+0000 [id=1]    INFO   winstone.Logger#logInternal: Beginning extraction from war file
 52020-10-04 12:52:16.815+0000 [id=1]    WARNING    o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
 62020-10-04 12:52:16.899+0000 [id=1]    INFO   org.eclipse.jetty.server.Server#doStart: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 1.8.0_242-b08
 72020-10-04 12:52:17.253+0000 [id=1]    INFO   o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
 82020-10-04 12:52:17.306+0000 [id=1]    INFO   o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
 92020-10-04 12:52:17.306+0000 [id=1]    INFO   o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
102020-10-04 12:52:17.316+0000 [id=1]    INFO   o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
112020-10-04 12:52:17.733+0000 [id=1]    INFO   hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
122020-10-04 12:52:17.950+0000 [id=1]    INFO   o.e.j.s.handler.ContextHandler#doStart: Started w.@733c423e{Jenkins v2.249.1,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
132020-10-04 12:52:17.996+0000 [id=1]    INFO   o.e.j.server.AbstractConnector#doStart: Started ServerConnector@387a8303{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
142020-10-04 12:52:17.996+0000 [id=1]    INFO   org.eclipse.jetty.server.Server#doStart: Started @3385ms
152020-10-04 12:52:17.997+0000 [id=20]   INFO   winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
162020-10-04 12:52:19.317+0000 [id=25]   INFO   jenkins.InitReactorRunner$1#onAttained: Started initialization
172020-10-04 12:52:19.355+0000 [id=25]   INFO   jenkins.InitReactorRunner$1#onAttained: Listed all plugins
182020-10-04 12:52:20.987+0000 [id=25]   INFO   jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
192020-10-04 12:52:20.992+0000 [id=25]   INFO   jenkins.InitReactorRunner$1#onAttained: Started all plugins
202020-10-04 12:52:21.013+0000 [id=26]   INFO   jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
212020-10-04 12:52:21.960+0000 [id=26]   INFO   jenkins.InitReactorRunner$1#onAttained: System config loaded
222020-10-04 12:52:21.960+0000 [id=26]   INFO   jenkins.InitReactorRunner$1#onAttained: System config adapted
232020-10-04 12:52:21.960+0000 [id=26]   INFO   jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
242020-10-04 12:52:21.961+0000 [id=26]   INFO   jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
252020-10-04 12:52:22.536+0000 [id=39]   INFO   hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
262020-10-04 12:52:22.549+0000 [id=39]   INFO   hudson.util.Retrier#start: Attempt #1 to do the action check updates server
272020-10-04 12:52:23.382+0000 [id=26]   INFO   o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@6a723808: display name [Root WebApplicationContext]; startup date [Sun Oct 04 12:52:23 UTC 2020]; root of context hierarchy
282020-10-04 12:52:23.382+0000 [id=26]   INFO   o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@6a723808]: org.springframework.beans.factory.support.DefaultListableBeanFactory@5930e957
292020-10-04 12:52:23.393+0000 [id=26]   INFO   o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5930e957: defining beans [authenticationManager]; root of factory hierarchy
302020-10-04 12:52:23.614+0000 [id=26]   INFO   o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@237624c2: display name [Root WebApplicationContext]; startup date [Sun Oct 04 12:52:23 UTC 2020]; root of context hierarchy
312020-10-04 12:52:23.616+0000 [id=26]   INFO   o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@237624c2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@61d56f3d
322020-10-04 12:52:23.617+0000 [id=26]   INFO   o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@61d56f3d: defining beans [filter,legacy]; root of factory hierarchy
332020-10-04 12:52:23.971+0000 [id=26]   INFO   jenkins.install.SetupWizard#init: 
34
35*************************************************************
36*************************************************************
37*************************************************************
38
39Jenkins initial setup is required. An admin user has been created and a password generated.
40Please use the following password to proceed to installation:
41
423d0647262e92486caff0c3aecc04bd14
43
44This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
45
46*************************************************************
47*************************************************************
48*************************************************************

For the reference here is the screenshot of the same -

How to find the initial Jenkins admin password

9. Supply Jenkins Admin password and install Kubernetes plugins on Jenkins

So now we are into step no 9 and if you are following along then I think you can login into the Jenkins.

Step 1: - Install Jenkins plugins

Step 2: - Create a first admin user for Jenkins

Step 3 : - Setup jenkins URL

Now Jenkins is ready for use

10. Install Kubernetes plugin on Jenkins

The next step would be for us to install the Kubernetes plugin.

For that go to - Manage Jenkins -> Manage Plugin

Search for plugin = kubernetes

Now install it with option - Install without restart

After installing the plugin again go to - Manage Jenkins -> Configure System

Now after clicking on Configure System goto the bottom of the page and click on Cloud -> a separate configuration page.

After that you need to choose for Kubernetes from the drop-down of Configure Cloud

Configure cloud for kubernetes on jenkins

In the next step, we need to fill in the Kubernetes cloud details

Name	Value
Kubernetes URL	https://kubernetes.default

After filling the value you should click on Test Connection

Congratulations you have successfully installed Jenkins inside your Kubernetes cluster and you can connect to it.

11. Lab Session

Conclusion

We have successfully set up the Kubernetes cluster using Kubespray
Installed the Jenkins using docker long term support image jenkins/jenkins:lts
After Jenkins installation we installed the Kubernetes plugin on Jenkins
Finally we were able to connect to the Kubernetes cluster from Jenkins

Learn more On Kubernetes -

Setup kubernetes on Ubuntu

Setup Kubernetes on CentOs

Setup HA Kubernetes Cluster with Kubespray

Setup HA Kubernetes with Minikube

Setup Kubernetes Dashboard for local kubernetes cluster

Setup Kubernetes Dashboard On GCP(Google Cloud Platform)

How to use Persistent Volume and Persistent Volume Claims in Kubernetes

Deploy Spring Boot Microservice on local Kubernetes cluster

Deploy Spring Boot Microservice on Cloud Platform(GCP)

Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster

CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes

kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)

How to setup kubernetes jenkins pipeline on AWS?

Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?

How to fix kubernetes pods getting recreated?

How to delete all kubernetes PODS?

How to use Kubernetes secrets?

Share kubernetes secrets between namespaces?

How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?

Delete Kubernetes POD stuck in terminating state?