How to setup Prometheus, Node Exporter and Grafana

Performance monitoring and alerting are very crucial to measure the performance metrics of an application running in a production environment. In this blog post, we are going to talk about two popular open-source tools Prometheus and Grafana. This blog post is aimed at anyone who is trying to learn Prometheus and Grafana or has some experience with these tools.

This is going to be a very beginners guide where we will start from -

1. Installation of Prometheus
2. How to create and Install Node exporter
3. Installing the Grafana
4. Setting up the Grafana Dashboard
5. Setting up Prometheus and Grafana for Kubernetes on AWS (Advance Use-Case)
6. Advantages of Prometheus
7. Advantages of Grafana
8. Conclusion

What is Prometheus? - It is an open-source tool for scrapping the performance metrics of any given hardware resource (CPU, VM, Cloud Virtual Machine, Router, etc.). Prometheus scraps the data with the timestamp which can be stored on a server and accessed using the PromQL. Prometheus has superb support API which makes Prometheus integration with any resource present in the Datacenter. Prometheus architecture is really scalable and 3rd party libraries and makes it more powerful.

What is Grafana? - In Lehman's term Grafana is a User interface for viewing the metrics scraped by Prometheus from various resources. Grafana is an open-source analytics and visualization tool. Grafana does not store any data, but instead, it relies on Prometheus to send the data so that dashboard can be prepared. Also, Grafana is used for sending notifications and mail alerts based on various thresholds. One of the cool features of the Grafana is Grafana Labs where you can go and download the dashboard prepared by other developers so that you do not have to re-invent the wheel.

1. Installation of Prometheus

Prometheus has a various ways of installation for example - pre-compiled binaries, from source, Docker. But to keep the installation simple we are going to use the pre-compiled binaries for installing Prometheus onto my Ubuntu Linux machine.

1. Goto Download Page of Prometheus and select the prometheus-x.xx.x.linux-amd64.tar.gz file for download. (Note- Here x.xx.x is version number)

   

2. Extract the download binary file. (Note- Replace x.xx.x with the downloaded version of Prometheus)

1
2tar xvfz prometheus-x.xx.x.linux-amd64.tar.gz

3. Go into the extracted directory cd prometheus-x.xx.x.linux-amd64.tar.gz

4. Start the prometheus server with the command ./prometheus

   

5. The Prometheus server should start on port 9090

6. You can access the Prometheus graph UI by visint http://localhost:9090/graph

   

7. You can access the Prometheus metrics UI by visint http://localhost:9090/metrics

   

2. How to create and Install Node exporter

Node exporter is responsible for fetching the statistics from various hardware and virtual resources in the format which Prometheus can understand and with the help of the prometheus server those statistics can be exposed on port 9100.

There are many third-party Node exporters which can be used by SREs as well as DevOps based on their application needs. But primarily we look for the following metrics -

1. CPU usage
2. Memory usage
3. Disk usage
4. Network usage

2.1 How to Install Node exporter

After installing the Prometheus in the previous step the next package we are going to install is **Node Exporter **. Node exported is used for collecting various hardware and kernel-level metrics of your machine.

Here are the download and installation steps -

1. Download the binary of Node exporter based on the operating system.(In my case I am using Ubuntu linux machine)

   

2. Extract the download node exporter binary file. (Note- Replace x.xx.x with the downloaded version of node exporter)

1
2tar xvfz node_exporter-*.*-amd64.tar.gz

3. Go into the extracted directory cd xvfz node_exporter-*.*-amd64.tar.gz

4. Start the node exported with the command ./node_exporter

5. Access the Node exporter metrics on the browser with URL - http://localhost:9100

2.2 Adding node exporter scrap_configs to prometheus as a YAML Configuration

In the previous node export installation, we had Prometheus and node exporter running on the same server. But if you want to add node exporter for any remote server then you need to define scrap_configs with target hosts inside the YAML configuration.

Here is an example of scrap_configs -

(*Note - In the below configuration you have replaced IP address 100.0.0.3 with your host IP or Hostname)

1
2global:
3   scrape_interval: 15s
4
5scrape_configs:
6   - job_name: node
7     static_configs:
8        - targets: ['localhost:9100','100.0.0.3:9100'] 

1. As you can see I have added two hosts localhost, 100.0.0.3 inside the scrap_config.
2. Save the above configuration as YAML(ex. exporter.yaml)
3. Start the Prometheus server bypassing scrap_config with the help of --config.file flag

Example -

1
2./prometheus --config.file=exporter.yml

Verify the Host status-

After adding the remote host into the node exporter configuration verify it by accessing the Prometheus target URL localhost:9090/targets

2.3 Enable and disable the collector

If you have more than one collector configured then node_exporter allows you to enable and disable some of the collectors with run time arguments -

Here is an example command for starting the node_exporter with disabled flag -

1
2node_exporter --no-collector.ENTER-THE-NAME-OF-COLLECTOR

For enabling the collector use the following command -

1
2node_exporter --collector.ENTER-THE-NAME-OF-COLLECTOR

3. Installing the Grafana

After installing the Prometheus server and node exporter let's install the Grafana for setting up visualization using Grafana dashboard. Here is the official download link of Grafana, choose the correct Grafana binary based on your operating system. For this blog post, I am going to use ubuntu Linux binary.

Steps for Installing Grafana-

In this installation, we are going to install the latest enterprise edition Grafana.

1. Update the package info -

1
2sudo apt-get install -y apt-transport-https
3sudo apt-get install -y software-properties-common wget
4wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

2. Add stable repository of Grafana -

1
2echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

3. Update repository and Install Grafana

1
2sudo apt-get update
3sudo apt-get install grafana-enterprise

4. Start the Grafana Server -

1
2sudo systemctl daemon-reload
3sudo systemctl start grafana-server
4sudo systemctl status grafana-server

5. Configure Grafana to run at boot time-

1
2sudo systemctl enable grafana-server.service
3 

6. Access the Grafana Dashboard using http://localhost:3000/login

7. The default login Username: Password for accessing the Grafana dashboard is admin: admin

8. Change the default admin password after the login

4. Setting up the Grafana Dashboard

The next step after installing the Grafana Server would be to set up the Grafana Dashboard. Till now, we have installed the Prometheus Server, Node Exporter as well as Grafana Server but there is a connection between Prometheus and Grafana.

As I mentioned earlier Grafana is an analytics and visualization dashboard, Grafana does not store data, but instead it relies on Datasource connection towards the Prometheus server because Prometheus server is responsible for scraping and storing the data.

4.1 Let's create a Prometheus Datasource inside Grafana Dashboard

1. Goto your Grafana Dashboard and click on Gear(Settings Icon)->Data Sources

2. Click on Add Data Source

3. Select the Prometheus as preferred data source -

4. Enter the hostname or IP address of the prometheus server

5. Save and test data source

4.2 Import Grafana Dashboard from Grafana Labs

Now after settings the data source we can import pre-existing opensource dashboard from Grafana Labs using the Dashboard ID.

1. Goto Grafana Dashboard and search for Linux Memory

2. Click on the result and then copy the board ID .e.g 2747

3. Goto Import and enter the board ID .e.g. 2747

5. Click load and then select the data source as Prometheus

6. Finally Click on Import

7. You should be able to see the memory dashboard

5. Advantages of Prometheus

Now I have talked about how to install and use Prometheus along with Grafana, let's see what are the advantages of using Prometheus in the landscape of DevOps and SREs -

1. Prometheus is a very independent framework that does not impact your existing cloud or in-premise infrastructure.
2. It can be customized to suit your infrastructure needs
3. To use Prometheus and Node exporter a developer does not have to install all the node exporters available in the market. A Developer can only install the required node exporter and to avoid extra overhead on the hardware resources
4. Prometheus node exporter can be customized to suit your infrastructure need, if you do not find a suitable node exporter in the library then Prometheus allows you to build a custom node exporter.
5. Prometheus comes along with PromQL and TSDB which make it a more powerful tool for debugging troubleshooting.
6. Prometheus has a built in database and you do not have to install the database separately.
7. The pull mechanism of Prometheus is very flexible, and it allows you to schedule variable scrappers at different intervals.
8. Also, Prometheus is being open-source, it has a wide variety of community support available for developers.

6. Advantages of Grafana

1. If you compare Grafana with traditional tools Zabbix and LibreNMS then Grafana allows you to create multiple datasource whereas Zabbix and LibreNMS allow you only to create a single Datasource.
2. In the Grafana dashboard you can customize various panels based on your requirement, also you can have integration with multiple data sources in the same Dashboard which is by far the biggest merit of using Grafana.
3. In Grafana open-source community you can share your dashboard settings with others, and similarly, you can use other's dashboards by searching it on Grafana Community, so you do not have to re-invent the wheel again.
4. Grafana dashboard's user experience is very pleasing and also it allows a developer to put PNG images on the dashboard to make it more appealing.
5. Grafana developer community is building a lot of features and various datasource support which makes Grafana more adaptable for various infrastructure monitoring needs.

7. Conclusion

No doubt Grafana and Prometheus are a really popular frameworks for monitoring, alerting, and data visualization although there is a bit of learning because Prometheus has custom query language PromQL which requires little of practice before you call yourself an expert in the PromQL.

But when you look from DevOps and SREs end you would always like to have Prometheus and Grafana in your arsenal because of various third-party Node exporters, setting up various data sources, customizable dashboards, and panels. I hope this beginners guide will help you to get you started with Prometheus, Grafana, and Node exporter setup, and if you are interested in some advance setup of Prometheus and Grafana then I would recommend this guide on - How to setup Prometheus, Grafana dashboard for Kubernetes monitoring on AWS

Learn more On Prometheus and Grafana -

1. How to setup Prometheus, Node Exporter and Grafana
2. How to setup Prometheus, Grafana dashboard for Kubernetes monitoring on AWS
3. Prometheus-Grafana Vs Zabbix