Reading · cleberg.net · RSS Amplifier

2024-09-20 · Monitoring Linux servers with Prometheus and visualizing in Grafana Cloud.

This tutorial will guide you through the process of:

Configuring a free Grafana cloud account.
Installing Prometheus to store metrics.
Installing Node Exporter to export machine metrics for Prometheus.
Installing Nginx Exporter to export Nginx metrics for Prometheus.
Visualizing data in Grafana dashboards.
Configure alerts based on Grafana metrics.

1. Grafana Cloud

To get started, visit the Grafana website and create a free account.

1.1. Prometheus Data Source

By default, a Prometheus data source should exist in your data sources page ($yourOrg.grafana.net/connections/datasources). If not, add a new data source using the Prometheus type.

Once you have a valid Prometheus data source, open the data source and note the following items:

Data	Example
Prometheus Server URL	https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push
User	1234567
Password	configured

1.2. Cloud Access Policy Token

Now let's create an access token in Grafana. Navigate to the Administration > Users and Access > Cloud Access Policies page and create an access policy.

The metrics > write scope must be enabled within the access policy you choose.

Once you have an access policy with the correct scope, click the Add Token button and be sure to copy and save the token since it will disappear once the modal window is closed.

1.3. Dashboards

Finally, let's create a couple dashboards so that we can easily explore the data that we will be importing from the server.

I recommend importing the following dashboards:

Refer to the bottom of the post for dashboard screenshots!

2. Docker

On the machine that you want to observe, make sure Docker and Docker Compose are installed. This tutorial will be using Docker Compose to create a group of containers that will work together to send metrics to Grafana.

Let's start by creating a working directory.

mkdir ~/prometheus && \
cd ~/prometheus    && \
nano compose.yml

Within the compose.yml file, let's paste the following:

# compose.yml

networks:
  monitoring:
    driver: bridge
volumes:
  prometheus_data: {}
services:
  nginx-exporter:
    image: nginx/nginx-prometheus-exporter
    container_name: nginx-exporter
    restart: unless-stopped
    command:
      - '--nginx.scrape-uri=http://host.docker.internal:8080/stub_status'
    expose:
      - 9113
    networks:
      - monitoring
    extra_hosts:
      - host.docker.internal:host-gateway
  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    expose:
      - 9100
    networks:
      - monitoring
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    expose:
      - 9090
    networks:
      - monitoring

sudo docker compose up -d

I'm not sure if it made a difference but I also whitelisted port 8080 on my local firewall with sudo ufw allow 8080.

Next, let's create a prometheus.yml configuration file.

nano prometheus.yml

# prometheus.yml

global:
  scrape_interval: 1m
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']
  - job_name: 'nginx'
    scrape_interval: 5s
    static_configs:
      - targets: ['nginx-exporter:9113']
remote_write:
  - url: 'https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push'
    basic_auth:
      username: 'prometheus-grafana-username'
      password: 'access-policy-token'

2.1. Nginx

To enable to the Nginx statistics we need for the nginx-exporter container, we need to modify the Nginx configuration on the host.

More specifically, we need to create a path for the stub_status to be returned when we query port 8080 on our localhost.

sudo nano /etc/nginx/conf.d/default.conf

server {
        listen 8080;
        listen [::]:8080;
        location /stub_status {
                stub_status;
        }
}

sudo systemctl restart nginx.service

2.2. Debugging

At this point, everything should be running smoothly. If not, here are a few areas to check and see if any obvious errors exist.

Nginx: Curl the stub_status from the Nginx web server on the host machine to see if Nginx and stub_status are working properly.

curl http://127.0.0.1:8080/stub_status
# EXPECTED RESULTS:
Active connections: 101
server accepts handled requests
 7510 7510 9654
Reading: 0 Writing: 1 Waiting: 93

Nginx-Exporter: Curl the exported Nginx metrics.

# Figure out the IP address of the Docker container
sudo docker network inspect grafana_monitoring
...
"Name": "nginx-exporter",
"EndpointID": "ef999a53eb9e0753199a680f8d78db7c2a8d5f442626df0b1bb945f03b73dcdd",
"MacAddress": "02:42:c0:a8:40:02",
"IPv4Address": "192.168.64.2/20",
...
# Curl the exported Nginx metrics
curl 192.168.64.2:9113/metrics
# EXPECTED RESULTS:
...
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.9927e-05
go_gc_duration_seconds{quantile="0.25"} 4.24e-05
go_gc_duration_seconds{quantile="0.5"} 4.8531e-05
...

Node-Exporter: Curl the exporter node machine metrics.

# Curl the exported Node metrics
curl 192.168.64.3:9100/metrics
# EXPECTED RESULTS:
...
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 47
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
...

Grafana: Open the Explore panel and look to see if any metrics are coming through the Prometheus data source. If not, something on the machine is preventing data from flowing through.

3. Alerts & IRM

Now that we have our data connected and visualized, we can define alerting rules and determine what Grafana should do when an alert is triggered.

3.1. OnCall

View of the Grafana OnCall dashboard with metrics. — Figure 1: OnCall

Within the Alerts & IRM section of Grafana (/alerts-and-incidents), open the Users page.

The Users page allows you to configure user connections such as:

Mobile App
Slack
Telegram
MS Teams
iCal
Google Calendar

In addition to the connections of each user, you can specify how each user or team is alerted for Default Notifications and Important Notifications.

Finally, you can access the Schedules page within the OnCall module to schedule users and teams to be on call for specific date and time ranges. For my purposes, I put myself on-call 24/7 so that I receive all alerts.

View of the grafanafd88 user with notification preferences. — Figure 2: User Information

3.2. Alerting

A dashboard with the current grafana-managed alert rules. — Figure 3: Alerting Insights

Now that we have defined users and team associated with an on-call schedule and configured to receive the proper alerts, let's define a rule that will generate alerts.

Within the Alerting section of the Alerts & IRM module, you can create alert rules, contact points, and notification policies.

Let's start by opening the Alert Rules page and click the New Alert Rule button.

As shown in the image below, we will create an alert for high CPU temperature by querying the node_hwmon_temp_celsius metric from our Prometheus data source.

Next, we will set the threshold to be anything above 50 (degrees Celsius). Finally, we will tell Grafana to evaluate this every 1 minute via our Default evaluation group. This is connected to our Grafana email, but can be associated with any notification policy.