{"id":2456,"date":"2021-10-03T19:52:16","date_gmt":"2021-10-03T19:52:16","guid":{"rendered":"https:\/\/dft.wiki\/?p=2456"},"modified":"2026-06-08T22:25:59","modified_gmt":"2026-06-09T02:25:59","slug":"monitoring-servers-with-prometheus-and-grafana-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/dft.wiki\/?p=2456","title":{"rendered":"Monitoring Servers with Prometheus and Grafana on Ubuntu"},"content":{"rendered":"

Prometheus<\/strong> is a systems and service monitoring tool [Link<\/a>]. Out of the box, it monitors the host where it is running and uses collectors such as Prometheus Node Exporter [Link<\/a>] to scrape metrics from other endpoints.<\/p>\n

Grafana<\/strong> is a visualization tool that consumes metrics from data sources like Prometheus<\/strong> to build graphs and dashboards [Link<\/a>].<\/p>\n

\n

INSTALLING PROMETHEUS<\/strong><\/p>\n

Some will say you should not use the distribution source because it is not frequently updated, but it is the most reliable option and will upgrade and patch with the system:<\/p>\n

sudo apt-get install prometheus prometheus-node-exporter prometheus-pushgateway prometheus-alertmanager -y\r\nsudo systemctl stop prometheus<\/pre>\n
Enable the API to unlock additional features:<\/p>\n
sudo nano \/etc\/systemd\/system\/multi-user.target.wants\/prometheus.service<\/pre>\n
Edit the following line:<\/p>\n
ExecStart=\/usr\/bin\/prometheus --web.enable-lifecycle<\/strong> $ARGS<\/pre>\n
Note: the default data retention period is 15 days. To customize it, add the argument –storage.tsdb.retention.time=30d<\/strong> with the desired value.<\/p>\n
Apply the change and start the service:<\/p>\n
sudo systemctl daemon-reload\r\nsudo systemctl start prometheus\r\nsudo systemctl status prometheus<\/pre>\n
It can also be deployed as a Docker container (for reference):<\/p>\n
sudo docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom\/prometheus<\/pre>\n
INSTALLING NODE EXPORTER ON EACH MONITORED HOST<\/strong><\/p>\n
sudo apt install prometheus-node-exporter -y\r\nsudo systemctl start prometheus-node-exporter\r\nsudo systemctl enable prometheus-node-exporter\r\nsudo systemctl status prometheus-node-exporter\r\nsudo ufw allow from 192.168.1.162<\/strong> to any port 9100<\/pre>\n
Note: replace 192.168.1.162<\/strong> with the IP of the server running Prometheus.<\/p>\n
CONFIGURING PROMETHEUS TO REACH THE MONITORED HOSTS<\/strong><\/p>\n
sudo nano \/etc\/prometheus\/prometheus.yml<\/pre>\n
Example configuration:<\/p>\n
global:\r\n  scrape_interval: 1s\r\n  evaluation_interval: 1s\r\nalerting:\r\n  alertmanagers:\r\n  - static_configs:\r\n    - targets: ['192.168.1.162:9093']\r\nscrape_configs:\r\n  - job_name: 'prometheus'\r\n    scrape_interval: 1s\r\n    scrape_timeout: 1s\r\n    static_configs:\r\n      - targets: ['192.168.1.162:9090']\r\n  - job_name: 'nodes'\r\n    scrape_interval: 1s\r\n    scrape_timeout: 1s\r\n    static_configs:\r\n      - targets: ['192.168.1.162:9100', '192.168.1.163<\/strong>:9100', '192.168.1.164<\/strong>:9100']<\/pre>\n
Note: using hostnames instead of IPs is recommended. You can configure name resolution in \/etc\/hosts<\/strong>.<\/p>\n
Reload the configuration using the API:<\/p>\n
curl -X POST http:\/\/localhost:9090\/-\/reload<\/pre>\n
Access the Prometheus web UI at http:\/\/192.168.1.162:9090\/<\/strong> and navigate to Status > Targets<\/strong>.<\/p>\n
\"\"<\/p>\n
\n
INSTALLING GRAFANA<\/strong><\/p>\n
Grafana is a powerful tool for visualizing data over time. It is the most popular dashboard viewer for Prometheus, but it supports many other data sources [Link<\/a>].<\/p>\n
sudo apt-get install -y apt-transport-https\r\nsudo apt-get install -y software-properties-common wget\r\ncurl -fsSL \"https:\/\/keyserver.ubuntu.com\/pks\/lookup?op=get&search=0x963FA27710458545\" | gpg --dearmor | sudo tee \/usr\/share\/keyrings\/repository-keyring.gpg >\/dev\/null\r\necho \"deb [signed-by=\/usr\/share\/keyrings\/repository-keyring.gpg] https:\/\/apt.grafana.com stable main\" | sudo tee -a \/etc\/apt\/sources.list.d\/grafana.list\r\nsudo apt-get update\r\nsudo apt-get install grafana -y\r\nsudo systemctl start grafana-server\r\nsudo systemctl enable grafana-server\r\nsudo systemctl status grafana-server\r\nsudo ufw allow 3000<\/pre>\n
Access the web UI at http:\/\/192.168.1.162:3000\/<\/strong> and change the default password (admin:admin) immediately<\/strong><\/span>.<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
From the Home page, import a dashboard from the online repository [Link<\/a>].<\/p>\n
\"\"<\/p>\n
Import dashboard 1860 and select Prometheus as the data source.<\/p>\n
Repeat the import with dashboard 405.<\/p>\n
Open the dashboard and customize the imported views as needed:<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\n
REFLECTION<\/strong><\/p>\n
Keep an eye on storage usage at \/var\/lib\/prometheus\/metrics2\/<\/strong> while you dial in the retention time and scraping frequency that best fit your needs.<\/p>\n
For reference: my setup runs Prometheus on one server and scrapes data from two additional servers (4 sources total) every 1 second. This accumulates roughly 800 MB of data every 24 hours.<\/p>\n
Email alerts render much better with the image renderer plugin:<\/p>\n
sudo grafana-cli plugins install grafana-image-renderer<\/pre>\n
SMTP settings can be configured in:<\/p>\n
sudo nano \/etc\/grafana\/grafana.ini<\/pre>\n
Always restart Grafana after making configuration changes or installing plugins:<\/p>\n
sudo systemctl restart grafana-server<\/pre>\n
To use AWS CloudWatch<\/strong> as a Grafana data source, follow these steps and adjust as needed:<\/p>\n
IAM > Policies > Add service: CloudWatch > Allow for: ListMetrics, GetMetricData, GetMetricStatistics.\r\nIAM > Roles > AWS Service > EC2 > Select the Policy to the Role.\r\nIAM > Users > Add User > Attach Existent Policy > Select the Policy to the User > Get AccessKey<\/strong> and SecretKey<\/strong>.\r\nEC2 > Select the Instance > Actions > Instance Settings > Attach\/Replace IAM Role.<\/pre>\n
Then use the AccessKey<\/strong> and SecretKey<\/strong> to add CloudWatch as a data source in the Grafana web UI.<\/p>\n
\n
BONUS<\/strong><\/p>\n
For monitoring Databases<\/span><\/strong>, use Percona Monitoring and Management<\/strong> (PMM) [Link<\/a>].<\/p>\n
Server<\/strong><\/p>\n
curl -fsSL https:\/\/www.percona.com\/get\/pmm | \/bin\/bash<\/pre>\n
Client<\/strong><\/p>\n
wget https:\/\/repo.percona.com\/apt\/percona-release_latest.$(lsb_release -sc)_all.deb\r\nsudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb\r\nsudo apt-get update\r\nsudo apt-get install pmm2-client<\/pre>\n
Connect the client to the server:<\/p>\n
sudo pmm-admin config --server-insecure-tls --server-url=https:\/\/admin:strong_password<\/strong>@pmm-server.local<\/strong><\/pre>\n
Create a database account for metrics collection (example for MySQL 8):<\/p>\n
CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'strong_password<\/strong>' WITH MAX_USER_CONNECTIONS 10;\r\nGRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';<\/pre>\n
\n
BONUS OF THE BONUS<\/strong><\/p>\n
For monitoring the uptime of services such as HTTP(S), DNS, Ping, and TCP, check out Uptime Kuma<\/strong> [Link<\/a>].<\/p>\n
Docker Compose Deployment<\/p>\n
apt update && apt-get upgrade -y\r\napt install docker.io docker-compose -y\r\nmkdir uptime-kuma && cd uptime-kuma\r\ncurl -o compose.yaml https:\/\/raw.githubusercontent.com\/louislam\/uptime-kuma\/master\/compose.yaml\r\ndocker-compose up -d<\/pre>\n
Docker Deployment (the lazy way)<\/p>\n
sudo docker run -d --restart=always --network=host -v uptime-kuma:\/app\/data --name uptime-kuma louislam\/uptime-kuma:base3<\/pre>\n
Ubuntu Installation<\/p>\n
git clone https:\/\/github.com\/louislam\/uptime-kuma.git\r\ncd uptime-kuma\r\nsudo apt install npm -y\r\nsudo npm run setup\r\nsudo npm install pm2 -g && sudo pm2 install pm2-logrotate\r\nsudo pm2 start server\/server.js --name uptime-kuma\r\nsudo pm2 save && sudo pm2 startup<\/pre>\n
To run it behind a reverse proxy with TLS termination and automated Let’s Encrypt certificates:<\/p>\n
sudo apt install nginx certbot python3-certbot-nginx -y\r\nsudo nano \/etc\/nginx\/sites-enabled\/default<\/pre>\n
server {\r\n   listen 80 default_server;\r\n   listen [::]:80 default_server;\r\n   root \/var\/www\/html;\r\n   index index.html;\r\n   server_name _;\r\n   location \/ {\r\n      proxy_pass http:\/\/127.0.0.1:3001;\r\n      proxy_set_header Host $host;\r\n   }\r\n}<\/pre>\n
sudo nginx -t && sudo nginx -s reload\r\nsudo certbot --nginx --non-interactive --agree-tos --redirect --email devops@domain.com<\/strong> -d uptime.domain.com<\/strong><\/pre>\n
If you need to monitor an external or public service to detect outages affecting your monitoring, check out Status Shield<\/strong> [Link<\/a>].<\/p>\n
\n
MOREOVER<\/strong><\/p>\n
For monitoring cron jobs or background tasks that could silently fail, try healthchecks.io<\/strong> [Link<\/a>].<\/p>\n
The following docker-compose.yaml<\/code> handles the full deployment and configuration:<\/p>\n
version: \"3\"\r\nservices:\r\n  healthchecks:\r\n    image: healthchecks\/healthchecks:latest\r\n    container_name: healthchecks\r\n    environment:\r\n      - DB=sqlite\r\n      - DB_NAME=\/data\/hc.sqlite\r\n      - DEBUG=False\r\n      - DEFAULT_FROM_EMAIL=healthchecks@example.com\r\n      - EMAIL_HOST=smtp.example.com\r\n      - EMAIL_HOST_USER=healthchecks@example.com\r\n      - EMAIL_HOST_PASSWORD=strong_password\r\n\r\n###### For STARTTLS - Explicit TLS\r\n      - EMAIL_PORT=587\r\n      - EMAIL_USE_TLS=True\r\n\r\n###### For SMTPS - Implicit TLS (deprecated)\r\n#      - EMAIL_PORT=465\r\n#      - EMAIL_USE_TLS=False\r\n#      - EMAIL_USE_SSL=True\r\n\r\n      - SECRET_KEY=f7993db8-1004-414b-be03-ad4a5d5153a3\r\n      - ALLOWED_HOSTS=healthchecks.simnet.cloud\r\n      - SITE_ROOT=https:\/\/healthchecks.simnet.cloud\/\r\n      - REGISTRATION_OPEN=False\r\n    ports:\r\n      - 8000:8000\r\n    volumes:\r\n      - healthchecks-data:\/data\r\n    restart: unless-stopped\r\nvolumes:\r\n    healthchecks-data:<\/pre>\n
It is also recommended to run it behind a reverse proxy with TLS termination.<\/p>\n
\n
See also Grafana Alloy<\/strong> and Grafana Loki<\/strong> for combining logs with metrics and exploring them in Grafana dashboards.<\/p>\n