How to setup Prometheus, Node Exporter and Grafana on Ubuntu (Step by Step, with systemd services and the Node Exporter Full dashboard)

Performance monitoring and alerting are crucial to measure how an application and the servers under it behave in production. In this blog post we are going to set up the most popular open-source monitoring stack there is - Prometheus to collect and store the metrics, Node Exporter to expose the CPU, memory, disk and network numbers of each Linux server, and Grafana to put them on a dashboard.

I first wrote this guide in 2022 for Prometheus 2.33. Since then Prometheus 3 has shipped with a new UI, Grafana has gone through several major versions and changed how you install it (the old apt-key method no longer works), and node_exporter is at 1.12. So this is the refreshed version - same beginner-friendly order, current commands, and this time we run everything as proper systemd services instead of leaving a ./prometheus process hanging in a terminal.

This is a beginners guide where we will start from -

Table of Content

1. How Prometheus, Node Exporter and Grafana fit together
2. Installation of Prometheus
3. How to install Node Exporter
4. Adding Node Exporter targets to prometheus.yml (scrape_configs)
5. Running Prometheus and Node Exporter as systemd services
6. Installing Grafana
7. Setting up the Grafana dashboard - data source and Node Exporter Full
8. Common errors and how to fix them
9. Setting up Prometheus and Grafana for Kubernetes on AWS (advanced use case)
10. Advantages of Prometheus
11. Advantages of Grafana
12. Conclusion

1. How Prometheus, Node Exporter and Grafana fit together

What is Prometheus? - It is an open-source monitoring system and time-series database. Prometheus works on a pull model - every few seconds it connects to each configured target, fetches a plain-text /metrics page and stores the numbers with a timestamp in its own local TSDB. You query that data with PromQL, and alerting rules evaluated on it are sent to Alertmanager. Prometheus is a CNCF graduated project and the de-facto standard for monitoring Kubernetes.

What is Node Exporter? - An exporter is a small program that translates the metrics of some system into the Prometheus text format. Node Exporter is the one for Linux hosts - it exposes CPU, memory, disk, filesystem, network and hundreds of other kernel metrics on port 9100. There are exporters for almost everything - MySQL, PostgreSQL, Redis, NGINX, HAProxy, blackbox probes - and your own application can expose /metrics directly with a client library.

What is Grafana? - In layman's terms Grafana is the user interface. It is an open-source analytics and visualization tool that does not store any metrics itself - you add Prometheus as a data source and build dashboards out of PromQL queries. The best feature for beginners is the Grafana dashboards library, where you import a dashboard somebody else built by typing one ID.

Everything in this post runs on one Ubuntu 24.04 machine for simplicity, and in section 4 we add a second server to show how remote targets work. The versions used - Prometheus 3.15.0 (LTS line 3.13), node_exporter 1.12.1, Grafana 13. Check the Prometheus download page for the current numbers and substitute them in the commands.

2. Installation of Prometheus

Prometheus has various ways of installation - pre-compiled binaries, from source, Docker, and on Kubernetes the kube-prometheus-stack Helm chart. To keep it simple we are going to use the pre-compiled binaries on an Ubuntu Linux machine.

1. Go to the download page of Prometheus and copy the link of the prometheus-x.xx.x.linux-amd64.tar.gz file. (Note - here x.xx.x is the version number; for an ARM server such as a Raspberry Pi or an AWS Graviton instance pick linux-arm64)

   

2. Download and extract it -

1wget https://github.com/prometheus/prometheus/releases/download/v3.15.0/prometheus-3.15.0.linux-amd64.tar.gz
2tar xvfz prometheus-3.15.0.linux-amd64.tar.gz
3cd prometheus-3.15.0.linux-amd64

3. Have a look at what is inside - the prometheus binary, the promtool helper and a ready prometheus.yml which scrapes Prometheus itself -

1ls
2# LICENSE  NOTICE  prometheus  prometheus.yml  promtool

4. Start the Prometheus server with the command ./prometheus --config.file=prometheus.yml

   

5. The Prometheus server starts on port 9090. The last log lines should say Server is ready to receive web requests.

6. Open the Prometheus UI at http://localhost:9090 (or http://<server-ip>:9090). In Prometheus 3 the expression browser is at /query - the old /graph URL still redirects there. Type up and press Execute - you should see one series, up{instance="localhost:9090", job="prometheus"} with value 1.

   

7. You can also see the raw metrics Prometheus exposes about itself at http://localhost:9090/metrics - this is exactly what every exporter looks like.

   

Leave this terminal running for now; in section 5 we turn it into a service.

3. How to install Node Exporter

Node Exporter is responsible for collecting hardware and kernel-level statistics of a Linux machine in the format Prometheus understands, and exposing them on port 9100. Out of the box it gives you the four things everybody looks at first -

1. CPU usage
2. Memory usage
3. Disk usage and I/O
4. Network usage

3.1 Download and run Node Exporter

1. Download the binary of Node Exporter for your operating system. (In my case the Ubuntu Linux machine, so linux-amd64)

   

2. Extract it -

1wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
2tar xvfz node_exporter-1.12.1.linux-amd64.tar.gz
3cd node_exporter-1.12.1.linux-amd64

3. Start it with the command ./node_exporter

4. Access the Node Exporter page in the browser at http://localhost:9100 and click Metrics, or open http://localhost:9100/metrics directly - you will see hundreds of lines like node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67.

Repeat the same three commands on every server you want to monitor. Node Exporter needs no configuration - the only thing that changes per server is the firewall rule allowing Prometheus to reach port 9100.

3.2 Enable and disable collectors

Node Exporter is made of collectors (cpu, meminfo, diskstats, filesystem, netdev, systemd ...). Most are on by default, and you can switch them at start-up -

Example - enable the systemd collector so you can alert when a service dies, and drop the (noisy) hwmon one -

1./node_exporter --collector.systemd --no-collector.hwmon

The full list with what each one exposes is in the node_exporter README.

4. Adding Node Exporter targets to prometheus.yml (scrape_configs)

Right now Prometheus only scrapes itself. To make it scrape Node Exporter - on this machine and on a remote server - we add a job to the scrape_configs section of prometheus.yml. The configuration reference is on the Prometheus configuration page.

Here is the complete file. (Replace 100.0.0.3 with the IP or hostname of your second server)

 1global:
 2  scrape_interval: 15s        # how often to scrape every target (default 1m)
 3  evaluation_interval: 15s    # how often to evaluate alerting rules
 4
 5scrape_configs:
 6  # Prometheus monitoring itself
 7  - job_name: "prometheus"
 8    static_configs:
 9      - targets: ["localhost:9090"]
10
11  # Linux servers running node_exporter
12  - job_name: "node"
13    static_configs:
14      - targets: ["localhost:9100", "100.0.0.3:9100"]
15        labels:
16          env: "lab"

1. The job_name becomes the job label on every metric, and each entry in targets becomes the instance label - that is how you tell servers apart in PromQL and Grafana.
2. Extra labels are attached to every metric from those targets - handy for env="prod" vs env="lab".
3. Validate the file before restarting - promtool catches indentation mistakes that would otherwise stop Prometheus from starting -

1./promtool check config prometheus.yml
2# Checking prometheus.yml
3#   SUCCESS: prometheus.yml is valid prometheus config file syntax

4. Restart Prometheus (Ctrl+C and ./prometheus --config.file=prometheus.yml again, or systemctl restart prometheus once it is a service - you can also send SIGHUP or POST /-/reload if started with --web.enable-lifecycle, and Prometheus reloads the config without a restart).

Verify the targets - open http://localhost:9090/targets. Both node endpoints should show state UP.

Now try a real query in the expression browser - the CPU usage in percent per instance -

1100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

For a handful of servers static_configs is fine. When servers come and go you switch to service discovery - ec2_sd_configs, gce_sd_configs, kubernetes_sd_configs, file_sd_configs - which is a config-only change in the same file.

5. Running Prometheus and Node Exporter as systemd services

Running the binaries from a terminal is fine for learning, but the moment you log out they die. On a real server you want both as systemd services that start at boot and run under their own unprivileged users. This is the layout the Prometheus community and the distribution packages use.

Prometheus -

1# user, directories, binaries
2sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
3sudo mkdir -p /etc/prometheus /var/lib/prometheus
4sudo cp prometheus promtool /usr/local/bin/
5sudo cp prometheus.yml /etc/prometheus/
6sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Create /etc/systemd/system/prometheus.service with this content -

 1[Unit]
 2Description=Prometheus
 3Wants=network-online.target
 4After=network-online.target
 5
 6[Service]
 7User=prometheus
 8Group=prometheus
 9Type=simple
10ExecStart=/usr/local/bin/prometheus \
11  --config.file=/etc/prometheus/prometheus.yml \
12  --storage.tsdb.path=/var/lib/prometheus \
13  --storage.tsdb.retention.time=30d \
14  --web.enable-lifecycle
15Restart=on-failure
16
17[Install]
18WantedBy=multi-user.target

1sudo systemctl daemon-reload
2sudo systemctl enable --now prometheus
3sudo systemctl status prometheus

Node Exporter - same pattern on every monitored server -

1sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter
2sudo cp node_exporter /usr/local/bin/

/etc/systemd/system/node_exporter.service -

 1[Unit]
 2Description=Prometheus Node Exporter
 3Wants=network-online.target
 4After=network-online.target
 5
 6[Service]
 7User=node_exporter
 8Group=node_exporter
 9Type=simple
10ExecStart=/usr/local/bin/node_exporter
11Restart=on-failure
12
13[Install]
14WantedBy=multi-user.target

1sudo systemctl daemon-reload
2sudo systemctl enable --now node_exporter

Check with curl -s localhost:9100/metrics | head and journalctl -u prometheus -f if anything refuses to start. (If you prefer Docker, the equivalents are the prom/prometheus and quay.io/prometheus/node-exporter images - the installation docs have the docker run commands; for Kubernetes you do not do any of this by hand, see section 9.)

6. Installing Grafana

After installing Prometheus and Node Exporter let's install Grafana for the dashboards. Grafana publishes an apt repository, and the official Debian/Ubuntu installation guide is what the steps below follow. Two things changed since my 2022 version - the repository moved to apt.grafana.com, and the GPG key goes into /etc/apt/keyrings/ because apt-key has been removed from current Ubuntu releases.

Steps for installing Grafana -

1. Install the prerequisites and import the Grafana GPG key -

1sudo apt-get install -y apt-transport-https software-properties-common wget
2sudo mkdir -p /etc/apt/keyrings/
3wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

2. Add the stable repository of Grafana -

1echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

3. Update the package index and install Grafana. Grafana recommends the Enterprise edition - it is free, includes everything in the OSS edition, and only unlocks the paid features if you add a licence. If you want the pure open-source build use grafana instead of grafana-enterprise -

1sudo apt-get update
2sudo apt-get install grafana-enterprise

4. Start the Grafana server -

1sudo systemctl daemon-reload
2sudo systemctl start grafana-server
3sudo systemctl status grafana-server

5. Configure Grafana to start at boot -

1sudo systemctl enable grafana-server.service

6. Access Grafana at http://localhost:3000/login

7. The default login for Grafana is Username: admin, Password: admin

8. Grafana forces you to change the admin password at the first login - do it, and if the server is reachable from the internet put it behind HTTPS (a reverse proxy, or Grafana's own [server] protocol = https setting in /etc/grafana/grafana.ini).

7. Setting up the Grafana dashboard - data source and Node Exporter Full

Till now we have installed the Prometheus server, Node Exporter and the Grafana server, but there is no connection between Prometheus and Grafana yet. As I mentioned, Grafana does not store data - it needs a data source that points at the Prometheus server, which is the one scraping and storing everything.

7.1 Let's create a Prometheus data source inside Grafana

The menu names below are from Grafana 11 onwards; the screenshots are from the older Grafana 8 UI, where the same thing was under the gear icon. The official data source guide has the current screenshots.

1. In the left menu go to Connections → Data sources and click Add new data source

2. Search for and select Prometheus

3. In Connection → Prometheus server URL enter the address of the Prometheus server - http://localhost:9090 when Grafana and Prometheus run on the same machine, otherwise http://<prometheus-ip>:9090

4. Leave everything else at the defaults and click Save & test. You should see "Successfully queried the Prometheus API".

If the test fails, the most common reason is that you typed localhost while Grafana runs in a different container or VM than Prometheus - use the IP that is reachable from Grafana.

7.2 Import the Node Exporter Full dashboard from the Grafana library

Now, instead of building panels one by one, we import a ready dashboard from the Grafana dashboards library using its ID. For Node Exporter the one everybody uses is Node Exporter Full, ID 1860 - CPU, memory, disk, network, filesystem, systemd, everything, with a server selector at the top. (My 2022 screenshots show dashboard 2747, Linux Memory, which still works but covers only memory.)

1. Go to the dashboards library, search for Node Exporter Full and copy the ID - 1860

2. In Grafana go to Dashboards → New → Import

3. Enter the ID 1860 and click Load

4. Select the data source Prometheus which we created in the previous step

5. Finally click Import

6. You should see the full dashboard, and the Host dropdown at the top lists every target from the node job - localhost:9100 and 100.0.0.3:9100

From here, building your own panel is just picking the Prometheus data source and typing a PromQL query such as node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100. Grafana's query builder can also write the PromQL for you if you are still learning it.

8. Common errors and how to fix them

1. Target shows DOWN with connection refused on /targets - Node Exporter is not running on that host, or a firewall blocks port 9100. Check with curl http://<ip>:9100/metrics from the Prometheus server; on Ubuntu sudo ufw allow from <prometheus-ip> to any port 9100, on AWS add an inbound rule to the security group.

2. context deadline exceeded - The target is reachable but too slow (the default scrape_timeout is 10s). Usually a firewall that drops packets instead of refusing, or a very busy host - check the network first.

3. server returned HTTP status 403 / 401 - Something in front of the exporter (a reverse proxy, a cloud load balancer) needs authentication. Scrape the exporter port directly, or configure basic_auth in the scrape job.

4. Prometheus will not start - parsing YAML file prometheus.yml: yaml: line 12: did not find expected key - Indentation. Always run promtool check config prometheus.yml first.

5. permission denied / opening storage failed - The prometheus user does not own /var/lib/prometheus. sudo chown -R prometheus:prometheus /var/lib/prometheus.

6. Grafana says Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused - Grafana cannot reach Prometheus at that URL. If Grafana runs in Docker, localhost is the container; use the host IP or http://host.docker.internal:9090.

7. Dashboard panels show No data - Either the data source for the imported dashboard is wrong (edit the dashboard settings → Variables), or the metric names changed - very old dashboards use node_cpu instead of node_cpu_seconds_total. Dashboard 1860 is maintained and matches current node_exporter.

8. E: The repository 'https://packages.grafana.com/oss/deb stable Release' does not have a Release file or apt-key warnings - That is the old repository. Remove the old /etc/apt/sources.list.d/grafana.list line and follow section 6 (apt.grafana.com with signed-by).

9. Disk filling up - Prometheus keeps 15 days by default. Set --storage.tsdb.retention.time (e.g. 30d) or --storage.tsdb.retention.size=50GB, and for long-term storage look at Thanos, Mimir or Cortex.

9. Setting up Prometheus and Grafana for Kubernetes on AWS (advanced use case)

On Kubernetes you do not install the binaries by hand. The kube-prometheus-stack Helm chart installs Prometheus Operator, Prometheus, Alertmanager, Grafana, node_exporter as a DaemonSet and kube-state-metrics in one helm install, with service discovery already configured. I have a dedicated guide on it - How to setup Prometheus and a Grafana dashboard for Kubernetes monitoring on AWS, and if you are new to Helm start with getting started with Helm.

10. Advantages of Prometheus

Now that I have talked about how to install and use Prometheus along with Grafana, let's see what the advantages are of using Prometheus in the landscape of DevOps and SREs -

1. Prometheus is an independent, self-contained system - one binary with its own database; you do not install a separate database or a message queue.
2. The pull model means you configure what to monitor in one place (Prometheus), not on every server, and up == 0 tells you immediately when a target disappears.
3. You only install the exporters you need, and if there is no exporter for something, writing one with a client library is a few dozen lines.
4. Service discovery for AWS, GCP, Azure, Kubernetes, Consul and DNS means new servers are picked up automatically.
5. PromQL is a genuinely powerful query language - rates, aggregations, predictions (predict_linear), joins across metrics.
6. Alerting rules live next to the data and Alertmanager handles grouping, silencing and routing to Slack, email, PagerDuty and webhooks.
7. The metrics format is an open standard (OpenMetrics) - almost every modern piece of software exposes /metrics natively.
8. Being open-source and a CNCF project, it has a huge community and it is the default in every Kubernetes distribution.

11. Advantages of Grafana

1. If you compare Grafana with traditional tools like Zabbix then Grafana lets you mix many data sources in one dashboard - Prometheus, Loki (logs), Tempo (traces), CloudWatch, MySQL, Elasticsearch - whereas Zabbix visualises only its own data. (The full comparison is in Prometheus + Grafana vs Zabbix.)
2. Panels, variables and templating let one dashboard serve every server or every environment through a dropdown.
3. The community dashboards library means you rarely start from scratch - Node Exporter Full, Kubernetes cluster monitoring, MySQL, NGINX, all one ID away.
4. Grafana has its own unified alerting now too, so you can alert from a dashboard query if you prefer that over Prometheus rules.
5. The developer community keeps adding visualisations and data sources, which makes Grafana adaptable to almost any infrastructure monitoring need.

12. Conclusion

No doubt Prometheus and Grafana are the most popular open-source combination for monitoring, alerting and visualisation, although there is a bit of a learning curve because Prometheus has its own query language, PromQL, which needs some practice before you call yourself an expert.

To summarise what we did - we installed Prometheus from the official binary and ran it on port 9090, installed Node Exporter on port 9100 and added it to scrape_configs in prometheus.yml, turned both into systemd services, installed Grafana from apt.grafana.com, connected Prometheus as a data source and imported the Node Exporter Full dashboard. From here the natural next steps are Alertmanager for notifications, more exporters for your databases and web servers, and if you run Kubernetes, the Helm chart route in the AWS Kubernetes monitoring guide. The official references are the Prometheus getting started guide, the node_exporter README and the Grafana installation docs.

Learn more On Prometheus and Grafana -

1. How to setup Prometheus, Node Exporter and Grafana
2. How to setup Prometheus, Grafana dashboard for Kubernetes monitoring on AWS
3. Prometheus-Grafana Vs Zabbix