Kubernetes cluster setup with Jenkins
In the DevOps world if you do not talk about Jenkins, Continuous Integration, and Continuous Delivery then you are missing a big chunk of it and with the inception of Kubernetes DevOps has grown its territory by 10 folds.
In this article, we are going to integrate -
- Jenkins
- Kuberneteshttps
Pre-Requisite
- Vagrant
- Virtual Box
- IDE(Sublime, Atom, or Visual Studio)
1. Spin-up the vagrant box Vagranfile
The first and the far most we need a Kubernetes cluster and for that we need a virtual machine.
So we are going to spin up three virtual machine -
- amaster - Ansible master for provisioning kubernetes cluster using kubespray
- k8smaster - Kubernetes Master
- k8sworker - Kubernetes Worker
(**Note - We will be using Ubuntu as our base operating system in the virtual machine)
Here is the Vagrantfile
1Vagrant.configure("2") do |config|
2 config.vm.define "amaster" do |amaster|
3 amaster.vm.box_download_insecure = true
4 amaster.vm.box = "hashicorp/bionic64"
5 amaster.vm.network "forwarded_port", guest: 8080, host: 8080
6 amaster.vm.network "forwarded_port", guest: 8081, host: 8081
7 amaster.vm.network "private_network", ip: "100.0.0.1"
8 amaster.vm.hostname = "amaster"
9 amaster.vm.provider "virtualbox" do |v|
10 v.name = "amaster"
11 v.memory = 2048
12 v.cpus = 2
13 end
14 end
15
16 config.vm.define "k8smaster" do |k8smaster|
17 k8smaster.vm.box_download_insecure = true
18 k8smaster.vm.box = "hashicorp/bionic64"
19 k8smaster.vm.network "private_network", ip: "100.0.0.2"
20 k8smaster.vm.hostname = "k8smaster"
21 k8smaster.vm.provider "virtualbox" do |v|
22 v.name = "k8smaster"
23 v.memory = 2048
24 v.cpus = 2
25 end
26 end
27
28
29 config.vm.define "k8sworker" do |k8sworker|
30 k8sworker.vm.box_download_insecure = true
31 k8sworker.vm.box = "hashicorp/bionic64"
32 k8sworker.vm.network "private_network", ip: "100.0.0.3"
33 k8sworker.vm.hostname = "k8sworker"
34 k8sworker.vm.provider "virtualbox" do |v|
35 v.name = "k8sworker"
36 v.memory = 2048
37 v.cpus = 2
38 end
39 end
40`
41end
Next Question - You need a Kubernetes cluster and How to setup you Kubernetes cluster?
Follow this article Setup Kubernetes Cluster
Once you have done setting up the Kubernetes cluster then go ahead and login to the Kubernetes cluster.
1vagrant ssh k8smaster
2. Install Docker and docker-compose
The next step would be for you to install Docker and docker-compose.
Let's go ahead and install Docker first
First, update the repository
1sudo apt-get update
Then run the following Docker install command
1sudo apt install docker.io
You can verify the Docker installation by running the following
1docker version
It should return you with the current docker installed version
1Client: Docker Engine - Community
2 Version: 19.03.13
3 API version: 1.40
4 Go version: go1.13.15
5 Git commit: 4484c46d9d
6 Built: Wed Sep 16 17:02:36 2020
7 OS/Arch: linux/amd64
8 Experimental: false
Alright now we are done with installing Docker on our system, now its time to install docker-compose
Use the following command to download all the required packages needed for docker-compose
1sudo curl -L "https://github.com/docker/compose/releases/download/1.27.3/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
Then change the permission of the directory /usr/local/bin/docker-compose
and make it executable
1sudo chmod +x /usr/local/bin/docker-compose
Then let's create some hard links to the directory /usr/local/bin/docker-compose
1sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
Now you need to add the current user to the docker group otherwise you will get a docker exception
1sudo usermod -aG docker $USER
After that restart the vagrant box and verify the docker-compose installation by running the following command
1sudo usermod -aG docker $USER
If you successfully installed the docker-compose than it should return you
1docker-compose version 1.27.3, build 4092ae5d
3. Setup jenkins cluster role - jenkins-cluster-role.yml
The first step towards setting up Jenkins is to create the cluster role.
Use the following jenkins-cluster-role.yml
and apply it.
1kind: ClusterRole
2apiVersion: rbac.authorization.k8s.io/v1
3metadata:
4 namespace: default
5 name: service-reader
6rules:
7 - apiGroups: [""] # "" indicates the core API group
8 resources: ["services"]
9 verbs: ["get", "watch", "list"]
10 - apiGroups: [""]
11 resources: ["pods"]
12 verbs: ["create","delete","get","list","patch","update","watch"]
13 - apiGroups: [""]
14 resources: ["pods/exec"]
15 verbs: ["create","delete","get","list","patch","update","watch"]
16 - apiGroups: [""]
17 resources: ["pods/log"]
18 verbs: ["get","list","watch"]
19 - apiGroups: [""]
20 resources: ["secrets"]
21 verbs: ["get"]
To apply the jenkins-cluster-role.yml use the following kubectl
command
1kubectl apply -f jenkins-cluster-role.yml
Now we need to create cluster role binding named "service reader pod" .
Type | Name |
---|---|
clusterrole | service reader |
name | default |
Use the following command to achieve that -
1kubectl create clusterrolebinding service-reader-pod --clusterrole=service-reader --serviceaccount=default:default
4. Create a persistent volume for storing Jenkins settings - jenkins-persistent-volume.yml
Now after you have created cluster role now its time to create persistent volume because we need to store the Jenkins configuration.
Create a yaml - jenkins-persistent-volume.yml
1apiVersion: v1
2kind: PersistentVolume
3metadata:
4 name: k8sdemo-jenkins-pv
5 labels:
6 app: k8sdemo-jenkins
7spec:
8 capacity:
9 storage: 1Gi
10 volumeMode: Filesystem
11 accessModes:
12 - ReadWriteOnce
13 persistentVolumeReclaimPolicy: Retain
14 storageClassName: local-storage
15 local:
16 path: /home/vagrant/storage
17 nodeAffinity:
18 required:
19 nodeSelectorTerms:
20 - matchExpressions:
21 - key: kubernetes.io/hostname
22 operator: In
23 values:
24 - node1
Now apply the peristent-volume configuration
1kubectl apply -f jenkins-persistent-volume.yml
5. Create persistent volume claim for Jenkins - jenkins-pvc.yml
In this step, you need to define the persistent volume claim for your Jenkins.
Create a YAML - jenkins-pvc.yml
1apiVersion: v1
2kind: PersistentVolumeClaim
3metadata:
4 name: k8sdemo-jenkins-pvclaim
5 labels:
6 app: k8sdemo-jenkins
7spec:
8 accessModes:
9 - ReadWriteOnce
10 storageClassName: local-storage
11 resources:
12 requests:
13 storage: 1Gi #1 GB
As always apply the persistent volume claim configuration
1kubectl apply -f jenkins-pvc.yml
6. Jenkins deployment (jenkins/jenkins:lts) - jenkins-deployment.yml
So till now, everything which we did is just pre-requisite for the jenkins.
Now we are going to install the Jenkins and for that, we are going to create jenkins-deployment.yml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: k8sdemo-jenkins-deployment
5 labels:
6 app: k8sdemo-jenkins
7spec:
8 selector:
9 matchLabels:
10 app: k8sdemo-jenkins
11 strategy:
12 type: Recreate
13 template:
14 metadata:
15 labels:
16 app: k8sdemo-jenkins
17 spec:
18 serviceAccountName: default # The name of the service account is default
19 containers:
20 - image: jenkins/jenkins:lts
21 name: k8sdemo-jenkins-container
22 #imagePullPolicy: Never
23 ports:
24 - containerPort: 8080
25 - containerPort: 50000
26 volumeMounts:
27 - name: k8sdemo-jenkins-persistentstorage
28 mountPath: /var/jenkins_home
29 volumes:
30 - name: k8sdemo-jenkins-persistentstorage
31 persistentVolumeClaim:
32 claimName: k8sdemo-jenkins-pvclaim
Key point - Here we are using jenkins long term support docker image .i.e.
jenkins/jenkins:lts
Okay let's apply the configuration
1kubectl apply -f jenkins-deployment.yml
You can verify your jenkins deployment using the following command -
1kubectl get deployment
It should return you with something similar -
7. Expose Jenkins deployment as Service - jenkins-service.yml
After deploying Jenkins now we need to expose the service so that it can be accessed outside of the Kubernetes cluster.
And for exposing it to the outside of the cluster lets create - jenkins-service.yml
1apiVersion: v1
2kind: Service
3metadata:
4 name: k8sdemo-jenkins-service
5 labels:
6 app: k8sdemo-jenkins
7spec:
8 type: NodePort
9 selector:
10 app: k8sdemo-jenkins
11 ports:
12 - port: 8080
13 name: http
14 protocol : TCP
15 nodePort: 30080
16 targetPort: 8080
17 - port: 50000
18 name: agent
19 protocol: TCP
20 targetPort: 50000
Keypoint - Here we are exposing Jenkins service using NodePort so that it can be accessed outside of the cluster.
Alright let's apply this configuration
1kubectl apply -f jenkins-service.yml
Once you have applied the configuration then you can verify it with the following command
1kubectl get service
Congratulations you have installed Jenkins on the Kubernetes cluster and it can be accessed using the following URL -
1http://100.0.0.2:30080/
Here is the login screen of the Jenkins
8. How to find Jenkins initial Administrator password
Now you need to find Jenkins initial admin password and for that first, you need to get your Kubernetes pod name
1kubectl get pods
It should return you with something similar -
Now we know jenkins pod name and in my case, it is - k8sdemo-jenkins-deployment-7b4ccf5755-4hj8s
(*The Jenkins pod name can be different in your case, so be careful while picking up the pod name)
Alright let run the log command on the Kubernetes pod to find the Jenkins initial admin password
1kubectl logs k8sdemo-jenkins-deployment-7b4ccf5755-4hj8s
It should return the pods logs along with Jenkins initial admin password
1Running from: /usr/share/jenkins/jenkins.war
2webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
32020-10-04 12:52:15.343+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @731ms to org.eclipse.jetty.util.log.JavaUtilLog
42020-10-04 12:52:15.512+0000 [id=1] INFO winstone.Logger#logInternal: Beginning extraction from war file
52020-10-04 12:52:16.815+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
62020-10-04 12:52:16.899+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 1.8.0_242-b08
72020-10-04 12:52:17.253+0000 [id=1] INFO o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
82020-10-04 12:52:17.306+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: DefaultSessionIdManager workerName=node0
92020-10-04 12:52:17.306+0000 [id=1] INFO o.e.j.s.s.DefaultSessionIdManager#doStart: No SessionScavenger set, using defaults
102020-10-04 12:52:17.316+0000 [id=1] INFO o.e.j.server.session.HouseKeeper#startScavenging: node0 Scavenging every 600000ms
112020-10-04 12:52:17.733+0000 [id=1] INFO hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
122020-10-04 12:52:17.950+0000 [id=1] INFO o.e.j.s.handler.ContextHandler#doStart: Started w.@733c423e{Jenkins v2.249.1,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
132020-10-04 12:52:17.996+0000 [id=1] INFO o.e.j.server.AbstractConnector#doStart: Started ServerConnector@387a8303{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
142020-10-04 12:52:17.996+0000 [id=1] INFO org.eclipse.jetty.server.Server#doStart: Started @3385ms
152020-10-04 12:52:17.997+0000 [id=20] INFO winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
162020-10-04 12:52:19.317+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
172020-10-04 12:52:19.355+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
182020-10-04 12:52:20.987+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
192020-10-04 12:52:20.992+0000 [id=25] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
202020-10-04 12:52:21.013+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
212020-10-04 12:52:21.960+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: System config loaded
222020-10-04 12:52:21.960+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: System config adapted
232020-10-04 12:52:21.960+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
242020-10-04 12:52:21.961+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
252020-10-04 12:52:22.536+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started Download metadata
262020-10-04 12:52:22.549+0000 [id=39] INFO hudson.util.Retrier#start: Attempt #1 to do the action check updates server
272020-10-04 12:52:23.382+0000 [id=26] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@6a723808: display name [Root WebApplicationContext]; startup date [Sun Oct 04 12:52:23 UTC 2020]; root of context hierarchy
282020-10-04 12:52:23.382+0000 [id=26] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@6a723808]: org.springframework.beans.factory.support.DefaultListableBeanFactory@5930e957
292020-10-04 12:52:23.393+0000 [id=26] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5930e957: defining beans [authenticationManager]; root of factory hierarchy
302020-10-04 12:52:23.614+0000 [id=26] INFO o.s.c.s.AbstractApplicationContext#prepareRefresh: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@237624c2: display name [Root WebApplicationContext]; startup date [Sun Oct 04 12:52:23 UTC 2020]; root of context hierarchy
312020-10-04 12:52:23.616+0000 [id=26] INFO o.s.c.s.AbstractApplicationContext#obtainFreshBeanFactory: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@237624c2]: org.springframework.beans.factory.support.DefaultListableBeanFactory@61d56f3d
322020-10-04 12:52:23.617+0000 [id=26] INFO o.s.b.f.s.DefaultListableBeanFactory#preInstantiateSingletons: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@61d56f3d: defining beans [filter,legacy]; root of factory hierarchy
332020-10-04 12:52:23.971+0000 [id=26] INFO jenkins.install.SetupWizard#init:
34
35*************************************************************
36*************************************************************
37*************************************************************
38
39Jenkins initial setup is required. An admin user has been created and a password generated.
40Please use the following password to proceed to installation:
41
423d0647262e92486caff0c3aecc04bd14
43
44This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
45
46*************************************************************
47*************************************************************
48*************************************************************
For the reference here is the screenshot of the same -
9. Supply Jenkins Admin password and install Kubernetes plugins on Jenkins
So now we are into step no 9 and if you are following along then I think you can login into the Jenkins.
Step 1: - Install Jenkins plugins
Step 2: - Create a first admin user for Jenkins
Step 3 : - Setup jenkins URL
Now Jenkins is ready for use
10. Install Kubernetes plugin on Jenkins
The next step would be for us to install the Kubernetes plugin.
For that go to - Manage Jenkins -> Manage Plugin
Search for plugin = kubernetes
Now install it with option - Install without restart
After installing the plugin again go to - Manage Jenkins -> Configure System
Now after clicking on Configure System
goto the bottom of the page and click on Cloud -> a separate configuration page.
After that you need to choose for Kubernetes
from the drop-down of Configure Cloud
In the next step, we need to fill in the Kubernetes cloud details
Name | Value |
---|---|
Kubernetes URL | https://kubernetes.default |
After filling the value you should click on Test Connection
Congratulations you have successfully installed Jenkins inside your Kubernetes cluster and you can connect to it.
11. Lab Session
Conclusion
- We have successfully set up the Kubernetes cluster using Kubespray
- Installed the Jenkins using docker long term support image
jenkins/jenkins:lts
- After Jenkins installation we installed the Kubernetes plugin on Jenkins
- Finally we were able to connect to the Kubernetes cluster from Jenkins
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?
Posts in this Series
- Kubernetes Cheat Sheet for day to day DevOps operations?
- Delete Kubernetes POD stuck in terminating state?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Share kubernetes secrets between namespaces?
- How to use Kubernetes secrets?
- How to delete all kubernetes PODS?
- kubernetes pods getting recreated?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- kubectl export yaml OR How to generate YAML for deployed kubernetes resources
- Kubernetes Updates
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- Kubernetes cluster setup with Jenkins
- How to use Persistent Volume and Persistent Claims | Kubernetes
- How to fix ProvisioningFailed persistentvolume controller no volume plugin matched
- Fixing – Cannot bind to requested volume: storageClasseName does not match
- Fixing – pod has unbound immediate persistentvolumeclaims or cannot bind to requested volume incompatible accessmode
- How to fix kubernetes dashboard forbidden 403 error – message services https kubernetes-dashboard is forbidden User
- How to fix Kubernetes – error execution phase preflight [preflight]
- Deploy Spring Boot microservices on kubernetes?
- How to fix – ansible_memtotal_mb minimal_master_memory_mb
- How to use kubespray – 12 Steps for Installing a Production Ready Kubernetes Cluster
- How to setup kubernetes on CentOS 8 and CentOS 7
- How to fix – How to fix - ERROR Swap running with swap on is not supported. Please disable swap
- 14 Steps to Install kubernetes on Ubuntu 20.04(bento/ubuntu-20.04), 18.04(hashicorp/bionic64)
- Kubernetes Dashboard | Kubernetes Admin GUI | Kubernetes Desktop Client
- Install Kubernetes with Minikube