How to setup kubernetes jenkins pipeline on AWS?
Hello guys It is been quite some time I was planning to write a blog post on How to set up your Kubernetes CI CD Jenkins pipeline on AWS step by step? It took some time for me to figure out all the details for setting up the continuous pipeline but at last once I was able to deploy it successfully on AWS, it was a sigh of relief for me.
I have listed all the steps which you can find inside the table of content and also I will also record a session and will upload on youtube.
(If you are interested in setting up similar pipeline on premise or using virtual machine the checkout this post )
Table of Content
- Setup an AWS EC2 Instance
- Connect to EC2 Instance
- Install JDK on AWS EC2 Instance
- Install and Setup Jenkins
- Update visudo and assign administrative privileges to Jenkins user
- Install Docker
- Install and Setup AWS CLI
- Install and Setup Kubectl
- Install and Setup eksctl
- Create eks cluster using eksctl
- Add Docker and GitHub Credentials into Jenkins
- Add jenkins stages
- Build, deploy and test CI CD pipeline
1. Setup an AWS EC2 Instance
The first step would be for us to set up an EC2 instance and on this instance, we will be installing -
- JDK
- Jenkins
- eksctl
- kubectl
1.1 Launch EC2 instance
- But first let's head over to AWS and in the search box type in
ec2
.
- Click on the EC2 and after that, you need to look for the
Launch Insance
option -
- Now select the image type for the EC2 instance(For this article we are going to select
Ubuntu Server 20.04
)
- Choose an Instance Type.
For this tutorial, we are going to use t2.medium because we will be installing Jenkins and t2.micro will not be sufficient enough to setup Jenkins
- Configure Instance Details - you can simply verify the detail and proceed to add the storage part.
- Add storage - In general 8 Gib of memory is sufficient enough for setting up Jenkins
- Add tags - This part is optional but you can add some meaningful tag names to your EC2 instance.
- Configure Security group - This is an important step because here we need to add Custom TCP Port 8080, if you do not add this port then you will not be able to access Jenkins using the public IP address of the AWS EC2 instance.
- Finally click on review and launch
- But before you launch your EC2 instance you need to create and download the key pair(private key and public key)
- Type in the key pair name and then click on the
Download Key Pair
.
Your ec2 instance should be up and running.
2. Connect to EC2 Instance
Before you connect to your EC2 instance you must start your EC2 instance.
Goto your AWS EC2 dashboard and click on EC2 after that click on Instances(running).
Remove the running filter and you should see your EC2 instance which you set up in Step 1.
Now we need to start the EC2 instance and it can be done by first selecting the instance and then
Goto-> Instance Start -> Start Instance
Once the instance state is Running you can select the instance and click on Connect
We will connect using SSH Client -
We will use the jenkins-ec2.pem
file to connect, so carefully copy the ssh command. (Following command will be different for you because the IP address of EC2 instance will always be different for you and also you need to supply your server pem file)
1ssh -i "my-key-pair-1.pem" ubuntu@ec2-18-185-59-154.eu-central-1.compute.amazonaws.com
After successful login, you should see something similar on your terminal -
3. Install JDK on AWS EC2 Instance
The next requirement is we need to install JAVA(JDK) on the EC2 instance.
In the previous step we have seen how to connect and ssh
into the EC2 instance.
Now before we do the JDK installation lets first update the package manager of the virtual machine -
1sudo apt-get update
Check if you have java already installed onto your EC2 machine by running the following command -
1java -version
In case if you do not have java installed then you will see the following message -
1Command 'java' not found, but can be installed with:
2
3sudo apt install openjdk-11-jre-headless # version 11.0.11+9-0ubuntu2~20.04, or
4sudo apt install default-jre # version 2:1.11-72
5sudo apt install openjdk-13-jre-headless # version 13.0.7+5-0ubuntu1~20.04
6sudo apt install openjdk-16-jre-headless # version 16.0.1+9-1~20.04
7sudo apt install openjdk-8-jre-headless # version 8u292-b10-0ubuntu1~20.04
8sudo apt install openjdk-14-jre-headless # version 14.0.2+12-1~20.04
But you can install java by running the following command
1sudo apt install openjdk-11-jre-headless
If you see the following message then you have installed java successfully -
1openjdk version "11.0.11" 2021-04-20
2OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
3OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
4. Install and Setup Jenkins
The next step would be to install the Jenkins. You can follow the official Jenkins Installation guide also. But here I have listed down the steps for installing the Jenkins on the EC2 instance.
First, we need to add the Jenkins repository to the package manager -
1wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
1sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
After adding the repository link of Jenkins update the package manager
1sudo apt-get update
Then finally install Jenkins using the following command
1sudo apt-get install jenkins
On successful installation, you should see Active Status
1sudo service jenkins status
1● jenkins.service - LSB: Start Jenkins at boot time
2 Loaded: loaded (/etc/init.d/jenkins; generated)
3 Active: active (exited) since Tue 2021-06-22 20:31:18 UTC; 37s ago
4 Docs: man:systemd-sysv-generator(8)
5 Process: 16297 ExecStart=/etc/init.d/jenkins start (code=exited, status=0/SUCCESS)
4.1 Setup jenkins
After installing jenkins lets go back to AWS dashboard -> EC2 -> Instances(running)
Click on the instance ID as mentioned in the above image.
Now we need to find the public IP address of the EC2 machine so that we can access the Jenkins.
Once you click on the instance ID you should see the following page with lots of information about the EC2 instance.
We need to look for Public IPv4 address
Alright now we know the public IP address of the EC2 machine, so now we can access the Jenkins from the browser using the public IP address followed by the port 8080
If you are installing the Jenkins for the first time then you need to supply the initialAdminPassword and you can obtain it from -
1sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Copy the password and paste it into the initial page of the Jenkins. After that, Jenkins will prompt you for installing the plugins.
Opt for install suggested plugin -
After completing the installation of the suggested plugin you need to set the First Admin User for Jenkins
Also, check the instance configuration because it will be used for accessing the Jenkins
And now your Jenkins is ready for use
4.2 Setup Gradle
In this lab session, we will be using the Spring Boot Application, so we need to use a build tool for compilation and building the JAR file and for that, we will be using Gradle as the preferred tool.
For setting up the gradle Goto -> Manage Jenkins -> Global Tool Configuration -> Gradle
Here is the screenshot for your reference -
5. Update visudo and assign administration privileges to jenkins user
Now we have installed the Jenkins on the EC2 instance. To interact with the Kubernetes cluster Jenkins will be executing the shell script
with the Jenkins user, so the Jenkins user should have an administration(superuser) role assigned forehand.
Let's add jenkins user
as an administrator and also ass NOPASSWD
so that during the pipeline run it will not ask for root
password.
Open the file /etc/sudoers
in vi mode
1sudo vi /etc/sudoers
Add the following line at the end of the file
1jenkins ALL=(ALL) NOPASSWD: ALL
After adding the line save and quit the file.
Now we can use Jenkins as root user and for that run the following command -
1sudo su - jenkins
6. Install Docker
Now we need to install the docker after installing the Jenkins.
The docker installation will be done by the Jenkins user because now it has root user privileges.
Use the following command for installing the docker -
1sudo apt install docker.io
After installing the docker you can verify it by simply typing the docker --version
onto the terminal
It should return you with the latest version of the docker
1Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2
6.1 Add jenkins user to Docker group
Jenkins will be accessing the Docker for building the application Docker images, so we need to add the Jenkins user to the docker group.
1sudo usermod -aG docker jenkins
7. Install and Setup AWS CLI
Okay so now we have our EC2 machine and Jenkins installed. Now we need to set up the AWS CLI on the EC2 machine so that we can use eksctl in the later stages
Let us get the installation done for AWS CLI
1sudo apt install awscli
Verify your AWS CLI installation by running the following command -
1aws --version
It should return you with the version of CLI
1aws-cli/1.18.69 Python/3.8.5 Linux/5.4.0-1045-aws botocore/1.16.19
7.1 Configure AWS CLI
Okay now after installing the AWS CLI, let's configure the AWS CLI so that it can authenticate and communicate with the AWS environment.
To configure the AWS the first command we are going to run is -
1aws configure
Once you execute the above command it will ask for the following information -
- AWS Access Key ID [None]:
- AWS Secret Access Key [None]:
- Default region name [None]:
- Default output format [None]:
You can find this information by going into AWS -> My Security Credentials
Then navigate to Access Keys (access key ID and secret access key)
You can click on the Create New Access Key and it will let you generate - AWS Access Key ID
, AWS Secret Access Key
.
(Note: - Always remember you can only download your access id and secret once, if you misplace the secret and access then you need to recreate the keys again)
Default region name - You can find it from the menu
Alright now we have installed and set up AWS CLI.
8. Install and Setup Kubectl
Moving forward now we need to set up the kubectl also onto the EC2 instance where we set up the Jenkins in the previous steps.
Here is the command for installing kubectl
1curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
1chmod +x ./kubectl
1sudo mv ./kubectl /usr/local/bin
Verify the kubectl installation
Verify the kubectl installation by running the command kubectl version
and you should see the following output
1Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
2Error from server (Forbidden): <html><head><meta http-equiv='refresh' content='1;url=/login?from=%2Fversion%3Ftimeout%3D32s'/><script>window.location.replace('/login?from=%2Fversion%3Ftimeout%3D32s');</script></head><body style='background-color:white; color:white;'>
9. Install and Setup eksctl
The next thing which we are gonna do is to install the eksctl, which we will be using to create AWS EKS Clusters.
Okay, the first command which we are gonna run to install the eksctl
1 curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
1sudo mv /tmp/eksctl /usr/local/bin
Verify the installation by running the command -
1eksctl version
And it will return you with the version -
10.52.0
So at the time of installation, I had 0.52.0
Installing eksctl on other OS
For Mac OS
1brew tap weaveworks/tap
2brew install weaveworks/tap/eksctl
For Windows
1 chocolatey install eksctl
or Scoop
1 scoop install eksctl
10. Create eks cluster using eksctl
In all the previous 9 steps we were preparing our AWS environment. Now in this step, we are going to create EKS cluster using eksctl
You need the following in order to run the eksctl command
- Name of the cluster : --name jhooq-test-cluster
- Version of Kubernetes : --version 1.17
- Region : --name eu-central-1
- Nodegroup name/worker nodes : worker-nodes
- Node Type : t2.micro
- Number of nodes: -nodes 2
Here is the eksctl command -
1eksctl create cluster --name jhooq-test-cluster --version 1.17 --region eu-central-1 --nodegroup-name worker-nodes --node-type t2.micro --nodes 2
(*Note - Be patient with the above command because it may take 20-30 minutes to complete)
For me it almost took 20 minutes, here are the timestamps
10.1 Verify the EKS kubernetes cluster from AWS
You can go back to your AWS dashboard and look for Elastic Kubernetes Service -> Clusters
Click on the Cluster Name to verify the worker nodes -
11. Add Docker and GitHub Credentials into Jenkins
As we know Kubernetes is a container orchestration tool and container management we are using docker.
(In case if you haven't set up Docker Hub Account then please create a DockerHub Account because we are gonna need it.)
Alright so if you are reading this line then I am assuming you have a DockerHub Account and GitHub Account.
Here is the link of GitHub Repository for this project
11.1 Setup Docker Hub Secret Text in Jenkins
You can set the docker credentials by going into -
Goto -> Jenkins -> Manage Jenkins -> Manage Credentials -> Stored scoped to jenkins -> global -> Add Credentials
11.2 Setup GitHub Username and password into Jenkins
Now we add one more username and password for GitHub.
Goto -> Jenkins -> Manage Jenkins -> Manage Credentials -> Stored scoped to jenkins -> global -> Add Credentials
12. Add jenkins stages
Okay, now we can start writing out the Jenkins pipeline for deploying the Spring Boot Application into the Kubernetes Cluster.
12.1 Jenkins stage-1 : Checkout the GitHub Repository
Add the following Jenkins script for checking out the GitHub Repository -
1 stage("Git Clone"){
2
3 git credentialsId: 'GIT_HUB_CREDENTIALS', url: 'https://github.com/rahulwagh/k8s-jenkins-aws'
4 }
12.2 Jenkins stage-2 : Gradle compilation and build
Now after checking out the repository let compile and build the application using Gradle
1stage('Gradle Build') {
2 sh './gradlew build'
3}
12.3 Jenkins stage-3 : Create Docker Container and push to Docker Hub
After successful compilation and build let's create a Docker image and push to the docker hub
1stage("Docker build"){
2 sh 'docker version'
3 sh 'docker build -t jhooq-docker-demo .'
4 sh 'docker image list'
5 sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
6}
7
8stage("Push Image to Docker Hub"){
9 sh 'docker push rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
10}
12.4 Jenkins stage-4 : Kubernetes deployment
Finally, do the Kubernetes deployment
1stage("kubernetes deployment"){
2 sh 'kubectl apply -f k8s-spring-boot-deployment.yml'
3}
Here is the complete final script for Jenkins pipeline -
1node {
2
3 stage("Git Clone"){
4
5 git credentialsId: 'GIT_HUB_CREDENTIALS', url: 'https://github.com/rahulwagh/k8s-jenkins-aws'
6 }
7
8 stage('Gradle Build') {
9
10 sh './gradlew build'
11
12 }
13
14 stage("Docker build"){
15 sh 'docker version'
16 sh 'docker build -t jhooq-docker-demo .'
17 sh 'docker image list'
18 sh 'docker tag jhooq-docker-demo rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
19 }
20
21 withCredentials([string(credentialsId: 'DOCKER_HUB_PASSWORD', variable: 'PASSWORD')]) {
22 sh 'docker login -u rahulwagh17 -p $PASSWORD'
23 }
24
25 stage("Push Image to Docker Hub"){
26 sh 'docker push rahulwagh17/jhooq-docker-demo:jhooq-docker-demo'
27 }
28
29 stage("kubernetes deployment"){
30 sh 'kubectl apply -f k8s-spring-boot-deployment.yml'
31 }
32}
13. Build, deploy and test CI/CD pipeline
Create new Pipeline: Goto Jenkins Dashboard or Jenkins home page click on New Item
Pipeline Name: Now enter Jenkins pipeline name and select Pipeline
Add pipeline script: Goto -> Configure
and then pipeline section.
Copy the Jenkins script from Step 12 and paste it there.
Build and Run Pipeline: Now goto pipeline and click on build now
Verify the build status:
Verify using kubectl commands
You can also verify the Kubernetes deployment and service with kubectl
command .e.g kubectl get deployments
, kubectl get service
You can access the rest end point from browser using the EXTERNAL-IP address
I hope you enjoyed this article. For other devops topic like Terraform, Helm Chart you can check the category and my YouTube Channel
Learn more On Kubernetes -
- Setup kubernetes on Ubuntu
- Setup Kubernetes on CentOs
- Setup HA Kubernetes Cluster with Kubespray
- Setup HA Kubernetes with Minikube
- Setup Kubernetes Dashboard for local kubernetes cluster
- Setup Kubernetes Dashboard On GCP(Google Cloud Platform)
- How to use Persistent Volume and Persistent Volume Claims in Kubernetes
- Deploy Spring Boot Microservice on local Kubernetes cluster
- Deploy Spring Boot Microservice on Cloud Platform(GCP)
- Setting up Ingress controller NGINX along with HAproxy inside Kubernetes cluster
- CI/CD Kubernetes | Setting up CI/CD Jenkins pipeline for kubernetes
- kubectl export YAML | Get YAML for deployed kubernetes resources(service, deployment, PV, PVC....)
- How to setup kubernetes jenkins pipeline on AWS?
- Implementing Kubernetes liveness, Readiness and Startup probes with Spring Boot Microservice Application?
- How to fix kubernetes pods getting recreated?
- How to delete all kubernetes PODS?
- How to use Kubernetes secrets?
- Share kubernetes secrets between namespaces?
- How to Delete PV(Persistent Volume) and PVC(Persistent Volume Claim) stuck in terminating state?
- Delete Kubernetes POD stuck in terminating state?