How to setup Prometheus, Grafana dashboard for Kubernetes monitoring on AWS

Prometheus and Grafana are the two popular open-source tools for collecting and visualizing performance metrics. In this blog post, we are going to set up Kubernetes cluster on AWS, later we are gonna set up Prometheus server for collecting the performance metrics and Grafana for.

Prometheus and Grafana are always a great combination when it comes to measuring the site reliability index. SREs (Software Reliability Engineers) heavily rely on Prometheus and Grafana because apart from collecting performance metrics, you can also set up some alerts, live views, and custom dashboards.

To accomplish the object we need to install some command-line tools.i.e. - AWS CLI, eksctl, kubectl, and Helm Chart. So in the first 4 steps, we are going to prepare the environment by installing the above-mentioned tools and then we will install Prometheus and Grafana.

1. Install AWS CLI and eksctl
2. Setup AWS Credentials
3. Install kubectl
4. Install Helm chart
5. Create Kubernetes cluster in AWS using eksctl
6. Install Kubernetes Metrics Server
7. Install Prometheus
8. Install grafana
9. Import Grafana dashboard from Grafana Labs
10. Deploy a spring boot microservice and monitor it on Grafana
11. Conclusion

1. Install AWS CLI and eksctl

The first requirement we have is to set up Kubernetes cluster on AWS and for that, you will need to install the following command-line tool -

1. AWS CLI
2. eksctl

1. Install AWS CLI

To start with first we need to install the AWS CLI. Here is the [official webpage of AWS for installing AWS CLI] 11, but you can use the following instruction based on your operating system -

MacOS

1. Run the following installation script from the terminal

1curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
2sudo installer -pkg AWSCLIV2.pkg -target / 

2. Verify the installation, first by checking the installation directory

1which aws

Check the installation version -

1aws --version

Windows AWS CLI has also supported on the Windows 64-bit operating system, use the following installation script for installing AWS CLI -

1msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

Verify the installation -

1aws --version 

Linux x86(64-bit) If you are using Linux x86(64-bit) operating system -

1curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
2unzip awscliv2.zip
3sudo ./aws/install

(Note - You might need to install the zip before running the unzip command - $ sudo apt install zip)

For Linux ARM operating system use the following installation script -

1curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
2unzip awscliv2.zip
3sudo ./aws/install 

Verify the installation -

1aws --version 

2. Install eksctl

Just like AWS CLI you need to install the eksctl for managing [Amazon Elastic Kubernetes Service (Amazon EKS)] 12. Use the following installation instruction based on your operating system -

1. Download the latest release -

1curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
2sudo mv /tmp/eksctl /usr/local/bin

2. MacOS homebrew installation -

1brew tap weaveworks/tap
2brew install weaveworks/tap/eksctl

3. Windows user can use chocolatey -

1chocolatey install eksctl

2. Setup AWS Credentials

In the next step, I am going to set up the AWS Credentials, so that we can use eksctl and kubectl command-line tools for setting up and managing the Kubernetes cluster.

Here are the instructions for fetching the AWS credentials -

1. To set up the AWS credentials you first need to fetch them from your AWS account.
2. Goto your AWS account
3. Then Goto Security Credentials
4. Click on Access Keys
5. Then Create New Access Key

Setup the AWS Credentials -

1. Open terminal/command prompt
2. Run the command aws configure and then enter AWS Access key ID and AWS secret access key

1AWS Access Key ID [None]: <ENTER_YOU_ACCESS_KEY>
2AWS Secret Access Key [None]: <ENTER_ACCESS_KEY>
3Default region name [None]: <ENTER_THE_REGION>
4Default output format [None]: json

3. Install kubectl

The next tool which we are going to install is the kubectl for managing the Kubernetes cluster. Use the following installation scripts for setting up kubectl -

1curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
2
3chmod +x ./kubectl 
4
5sudo mv ./kubectl /usr/local/bin

Verify Kubectl installation

Run the following kubectl command to verify the installation -

1kubectl version 

It should return you back with the Kubernetes version -

1Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
2Error from server (Forbidden): <html><head><meta http-equiv='refresh' content='1;url=/login?from=%2Fversion%3Ftimeout%3D32s'/><script>window.location.replace('/login?from=%2Fversion%3Ftimeout%3D32s');</script></head><body style='background-color:white; color:white;'>  

4. Install Helm chart

The next tool we need is Helm Chart. If you are working on setting up Prometheus and Grafan for the Kubernetes cluster then it is always recommended using the Helm Chart Repository - https://prometheus-community.github. io/helm-charts so that you do not miss any configuration step.

Helm Chart works out of the box and it will take care of everything for you by installing prometheus-alertnamanger, prometheus-server, prometheus-operator.

1. Install Helm Chart - Use the following script to install the helm chart -

1curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
2chmod 700 get_helm.sh
3./get_helm.sh

2. Verify Helm Chart installation

1WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/vagrant/.kube/config
2version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

5. Create Kubernetes cluster in AWS using eksctl

In the previous steps we have completed the pre-requisites needed for setting Prometheus and grafana, now let's create a Kubernetes cluster using eksctl.

1. Creae Kubernetes Cluster - Goto your terminal and run the following create cluster command -

1eksctl create cluster --name test-cluster-1 --version 1.22 --region eu-central-1 --nodegroup-name worker-nodes 
2--node-type t2.large --nodes 2 --nodes-min 2 --nodes-max 3

The above command might be a little overwhelming for you but here is the breakdown of the command-

1. eksctl create cluster - This is will tell eksctl to create a Kubernetes cluster
2. --name test-cluster-1 - The --name flag is used for assigning the name of the cluster
3. --version 1.18 - The --version flag is used for choosing the Kubernetes version which you want to install on AWS
4. --region eu-central-1 - As we are working on AWS so we need to specify the --region flag along with correct region-name
5. --nodegroup-name worker-nodes- The --nodegroup-name is used for assigning the name for the worker node 6.--node-type t2.large - --node-type flag is used for specifying which ec2 instance we are going to use for setting up the Kubernetes cluster
6. --nodes 2 - The --nodes flag is used for defining how many numbers of nodes we need inside the Kubernetes cluster
7. --nodes-min 2 - The --nodes-min flag is used to tell the minimum number for nodes we need inside the Kubernetes cluster
8. --nodes-max 3 - The --nodes-min flag is used to tell the maximum number of nodes we need inside the kubernetes cluster

2.Verify Kubernetes Cluster - After successfully executing the eksctl command, let's head over to AWS management console to verify the Kubernetes cluster.

After login to AWS management console, in the search bar type EKS -

On the left navigation panel of the AWS management console, you will find an option Cluster click on it.

And you should be able to view the Kubernetes cluster which you have created using the eksctl command.

Note run following command to update your kubeconfig aws eks update-kubeconfig --name test-cluster-1 https://stackoverflow.com/questions/72126048/error-exec-plugin-invalid-apiversion-client-authentication-k8s-io-v1alpha1-c

6. Install Kubernetes Metrics Server

Alright the next step would be to install the Kubernetes Metrics server onto the Kubernetes cluster so that Prometheus can collect the performance metrics of Kubernetes.

Kubernetes Metrics server collects the resource metrics from kubelets and exposes it to the Kubernetes api server through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

The Kubernetes metrics offers -

1. Fast autoscaling, collecting metrics
2. Scalable support up to 5,000 node clusters.
3. Resource efficiency and it requires very less cpu(1 mili core) and less memory(2 mb)

Installation of Kubernetes metrics server

Use the following installation script to install kubernetes metrics server

1kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Note - In case you face the issue error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" while running the kubectl apply -f command then follow these troubleshooting steps

Verify the Kubernetes metric server installation- You can verify the Kubernetes metric server installation kubectl get deployment metrics-server -n kube-system or kubectl get pods -n kube-system

7. Install Prometheus

Now we have set up the Kubernetes cluster in the previous step, let's install the prometheus using the helm chart.

1. Add Prometheus helm chart repository

1helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 

2. Update the helm chart repository

1helm repo update 

3. Create prometheus namespace

1kubectl create namespace prometheus

4. Install the grafana

1helm install prometheus prometheus-community/prometheus \
2    --namespace prometheus \
3    --set alertmanager.persistentVolume.storageClass="gp2" \
4    --set server.persistentVolume.storageClass="gp2" 

5. View the Prometheus dashboard by forwarding the deployment ports

1kubectl port-forward deployment/prometheus-server 9090:9090 -n prometheus

8. Install grafana

After installing Prometheus let's install the Grafana using the Helm Chart.

1. Add the Grafana helm chart repository

1helm repo add grafana https://grafana.github.io/helm-charts 

2. Update the helm chart repository

1helm repo update 

3. Now we need to create a Prometheus data source so that Grafana can access the Kubernetes metrics. Create a yaml file prometheus-datasource.yaml and save the following data source configuration into it -

1datasources:
2  datasources.yaml:
3    apiVersion: 1
4    datasources:
5    - name: Prometheus
6      type: prometheus
7      url: http://prometheus-server.prometheus.svc.cluster.local
8      access: proxy
9      isDefault: true

4. Create a namespace grafana

1kubectl create namespace grafana

5. Install the Grafana

1helm install grafana grafana/grafana \
2    --namespace grafana \
3    --set persistence.storageClassName="gp2" \
4    --set persistence.enabled=true \
5    --set adminPassword='EKS!sAWSome' \
6    --values prometheus-datasource.yaml \
7    --set service.type=LoadBalancer 

6. Verify the Grafana installation by using the following kubectl command -

1kubectl get all -n grafana

7. Public AWS IP of Grafana Kubernetes Server- To Access the Grafana dashboard we need to find Public AWS IP address and for that use the following command -

1kubectl get service -n grafana 

8. Copy the Public AWS IP address and open it in the browser -

9. Import Grafana dashboard from Grafana Labs

Now we have set up everything in terms of Prometheus and Grafana. For the custom Grafana Dashboard, we are going to use the open source grafana dashboard.

On opensource grafana dashboard you can find many opensource dashboards which can be imported directly into the Grafana. For this session, I am going to import a dashboard 6417

How to import Grafana Dashboard

1. On left navigation click on + sign

   

2. Then goto import and enter the grafana dashboard number 6417

   

3. Select the data source and save it.

   

4. Now you can see your Grafana dashboard displaying all the Kubernetes metrics.

   

10. Deploy a spring boot microservice and monitor it on Grafana

To make use of Grafana dashboard let's make a kubernetes deployment of spring boot application.

You can download the kubernetes manifest for spring boot from GitHub

1. To deploy the spring boot on kubernetes cluster user the following kubectl command

1kubectl apply -f k8s-spring-boot-deployment.yml 

2. Verify the deployment by running the following kubectl command

1$ kubectl get deployment jhooq-springboot
2NAME               READY   UP-TO-DATE   AVAILABLE   AGE
3jhooq-springboot   2/2     2            2           8m5s

3. Refresh the grafana dashboard to verify the deployment

Note- Follow this blog post on How to deploy spring boot inside kubernetes cluster

Conclusion

I hope you this little exercise will help you to understand the concepts of using kubernetes metrics monitoring using prometheus and grafana dashboard. Along with the monitoring prometheus also supports alert management in case of reporting some critical failure in the system, probably we could look into another post.

Error Troubleshooting

How to fix - error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

During the setup you might ran into issue where you are trying install the kuberentes metrics server but may lear to following issue -

1error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" 

This is an existing bug with the aws cli and there is issue reported on the GitHUb "aws eks update-kubeconfig invalid apiVersion". This issue is mainly associated with the kubectl version 1.24.0

How to Fix?

1. You should downgrade the kubectl version from 1.24.0 to 1.23.6
2. You can update kube config using the following commands -

1aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} --region ${REGION}

Or

1aws eks update-kubeconfig --name test-cluster-1 

Learn more On Prometheus and Grafana -

1. How to setup Prometheus, Node Exporter and Grafana
2. How to setup Prometheus, Grafana dashboard for Kubernetes monitoring on AWS
3. Prometheus-Grafana Vs Zabbix