{"id":2456,"date":"2021-10-03T19:52:16","date_gmt":"2021-10-03T19:52:16","guid":{"rendered":"https:\/\/dft.wiki\/?p=2456"},"modified":"2026-06-02T20:59:52","modified_gmt":"2026-06-03T00:59:52","slug":"monitoring-servers-with-prometheus-and-grafana-on-ubuntu-20-04","status":"publish","type":"post","link":"https:\/\/dft.wiki\/?p=2456","title":{"rendered":"Monitoring Servers with Prometheus and Grafana on Ubuntu"},"content":{"rendered":"

Prometheus<\/strong> is a systems and service monitoring system [Link<\/a>].\u00a0Out of the box, it monitors the host where it is running and uses collectors such as Prometheus Node Explorer [Link<\/a>] to scrape metrics from other endpoints.<\/p>\n

On the other hand, Grafana<\/strong> is a visualizer that consumes metrics from data sources like Prometheus<\/strong>, to build graphs and dashboards [Link<\/a>].<\/p>\n

\n

INSTALLING PROMETHEUS<\/strong><\/p>\n

Some will say that you should not use the distribution source because it is not frequently updated but this is the most reliable source and will upgrade\/patch with the system:<\/p>\n

sudo apt-get install prometheus prometheus-node-exporter prometheus-pushgateway prometheus-alertmanager -y\r\nsudo systemctl stop prometheus<\/pre>\n
Enable the API (it will provide additional features):<\/p>\n
sudo nano \/etc\/systemd\/system\/multi-user.target.wants\/prometheus.service<\/pre>\n
Edit the following line:<\/p>\n
ExecStart=\/usr\/bin\/prometheus --web.enable-lifecycle<\/strong> $ARGS<\/pre>\n
Note: the default retention period of the collected data is 15 days. To customize this period add the argument –storage.tsdb.retention.time=30d<\/strong> with the desired period.<\/p>\n
Apply the change and start the service:<\/p>\n
sudo systemctl daemon-reload\r\nsudo systemctl start prometheus\r\nsudo systemctl status prometheus<\/pre>\n
It could also be deployed as a Docker Container (just for reference):<\/p>\n
sudo docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom\/prometheus<\/pre>\n
INSTALLING NODE_EXPLORER ON EACH OF THE MONITORED HOSTS<\/strong><\/p>\n
sudo apt install prometheus-node-exporter -y\r\nsudo systemctl start prometheus-node-exporter\r\nsudo systemctl enable prometheus-node-exporter\r\nsudo systemctl status prometheus-node-exporter\r\nsudo ufw allow from 192.168.1.162<\/strong> to any port 9100<\/pre>\n
Note: replace the IP 192.168.1.162<\/strong> with the IP of the server where Prometheus is running.<\/p>\n
CONFIGURING THE PROMETHEUS TO REACH THE\u00a0 MONITORED HOSTS<\/strong><\/p>\n
sudo nano \/etc\/prometheus\/prometheus.yml<\/pre>\n
Example of configuration:<\/p>\n
global:\r\n  scrape_interval: 1s\r\n  evaluation_interval: 1s\r\nalerting:\r\n  alertmanagers:\r\n  - static_configs:\r\n    - targets: ['192.168.1.162:9093']\r\nscrape_configs:\r\n  - job_name: 'prometheus'\r\n    scrape_interval: 1s\r\n    scrape_timeout: 1s\r\n    static_configs:\r\n      - targets: ['192.168.1.162:9090']\r\n  - job_name: 'nodes'\r\n    scrape_interval: 1s\r\n    scrape_timeout: 1s\r\n    static_configs:\r\n      - targets: ['192.168.1.162:9100', '192.168.1.163<\/strong>:9100', '192.168.1.164<\/strong>:9100']<\/pre>\n
Note: I recommend you use names instead of IPs. You can configure the translation in the file \/etc\/hosts<\/strong>.<\/p>\n
Reload the configuration using the API:<\/p>\n
curl -X POST http:\/\/localhost:9090\/-\/reload<\/pre>\n
Access Prometheus WebUI with a browser http:\/\/192.168.1.162:9090\/<\/strong> and navigate to Status > Targets<\/strong>.<\/p>\n
\"\"<\/p>\n
\n
INSTALLING GRAFANA<\/strong><\/p>\n
Grafana is a powerful tool to visualize data over time. It is the most popular dashboard viewer for Prometheus but not limited to it [Link<\/a>].<\/p>\n
sudo apt-get install -y apt-transport-https\r\nsudo apt-get install -y software-properties-common wget\r\ncurl -fsSL \"https:\/\/keyserver.ubuntu.com\/pks\/lookup?op=get&search=0x963FA27710458545\" | gpg --dearmor | sudo tee \/usr\/share\/keyrings\/repository-keyring.gpg >\/dev\/null\r\necho \"deb [signed-by=\/usr\/share\/keyrings\/repository-keyring.gpg] https:\/\/apt.grafana.com stable main\" | sudo tee -a \/etc\/apt\/sources.list.d\/grafana.list\r\nsudo apt-get update\r\nsudo apt-get install grafana -y\r\nsudo systemctl start grafana-server\r\nsudo systemctl enable grafana-server\r\nsudo systemctl status grafana-server\r\nsudo ufw allow 3000<\/pre>\n
Access the Web UI\u00a0 on http:\/\/192.168.1.162:3000\/<\/strong> and change the default password (admin:admin) immediately<\/strong><\/span>.<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
Back to Home, import a Dashboard View from the online repository [Link<\/a>].<\/p>\n
\"\"<\/p>\n
Import the dashboard 1860 and select the data source as Prometheus.<\/p>\n
Repeat the import procedure with the number 405.<\/p>\n
Go to the dashboard, navigate and customize your imported view:<\/p>\n
\"\"<\/p>\n
\"\"<\/p>\n
\n
REFLECTION<\/strong><\/p>\n
Keep monitoring the storage usage on \/var\/lib\/prometheus\/metrics2\/<\/strong>\u00a0until you define the retention time and scraping frequency that best suits your need.<\/p>\n
For reference, I have set up the server where the Prometheus runs on, and two other servers. It scrapes data from 4 sources every 1 second and every 24h it accumulates about 800MB of data.<\/p>\n
The email alerts would look much nicer with the image renderer:<\/p>\n
sudo grafana-cli plugins install grafana-image-renderer<\/pre>\n
The SMTP configuration can be found at:<\/p>\n
sudo nano \/etc\/grafana\/grafana.ini<\/pre>\n
It is always recommended to restart Grafana after making changes to its configurations or installing plugins:<\/p>\n
sudo systemctl restart grafana-server<\/pre>\n
To use AWS CloudWatch<\/strong> as a data source for Grafana follow the step below making the adjustments accordingly:<\/p>\n
IAM > Policies > Add service: CloudWatch > Allow for: ListMetrics, GetMetricData, GetMetricStatistics.\r\nIAM > Roles > AWS Service > EC2 > Select the Policy to the Role.\r\nIAM > Users > Add User > Attach Existent Policy > Select the Policy to the User > Get AccessKey<\/strong> and SecretKey<\/strong>.\r\nEC2 > Select the Instance > Actions > Instance Settings > Attach\/Replace IAM Role.<\/pre>\n
Then use the created AccessKey<\/strong> and SecretKey\u00a0<\/strong>to add the data source using the Grafana web application.<\/p>\n
\n
BONUS<\/strong><\/p>\n
For monitoring Databases<\/span><\/strong> use Percona Monitoring and Management<\/strong> (PMM) [Link<\/a>].<\/p>\n
Server<\/strong><\/p>\n
curl -fsSL https:\/\/www.percona.com\/get\/pmm | \/bin\/bash<\/pre>\n
Client<\/strong><\/p>\n
wget https:\/\/repo.percona.com\/apt\/percona-release_latest.$(lsb_release -sc)_all.deb\r\nsudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb\r\nsudo apt-get update\r\nsudo apt-get install pmm2-client<\/pre>\n
Connect the Client to the Server<\/p>\n
sudo pmm-admin config --server-insecure-tls --server-url=https:\/\/admin:strong_password<\/strong>@pmm-server.local<\/strong><\/pre>\n
Create an account in the DB for acquiring metrics from the DB engine (example for MySQL 8):<\/p>\n
CREATE USER 'pmm'@'localhost' IDENTIFIED BY 'strong_password<\/strong>' WITH MAX_USER_CONNECTIONS 10;\r\nGRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';<\/pre>\n
\n
BONUS OF THE BONUS<\/strong><\/p>\n
For monitoring the uptime of different service types (e.g., HTTP(S), DNS, Ping, and TCP), there is an amazing tool called Uptime Kuma<\/strong> [Link<\/a>].<\/p>\n
Docker Compose Deployment<\/p>\n
apt update &&m apt-get upgrade -y\r\napt install docker.io docker-compose -y\r\nmkdir uptime-kuma && cd uptime-kuma\r\ncurl -o compose.yaml https:\/\/raw.githubusercontent.com\/louislam\/uptime-kuma\/master\/compose.yaml\r\ndocker-compose up -d<\/pre>\n
Docker Deployment (the lazy way)<\/p>\n
sudo docker run -d --restart=always --network=host -v uptime-kuma:\/app\/data --name uptime-kuma louislam\/uptime-kuma:base3<\/pre>\n
Ubuntu Installation<\/p>\n
git clone https:\/\/github.com\/louislam\/uptime-kuma.git\r\ncd uptime-kuma\r\nsudo apt install npm -y\r\nsudo npm run setup\r\nsudo npm install pm2 -g && sudo pm2 install pm2-logrotate\r\nsudo pm2 start server\/server.js --name uptime-kuma\r\nsudo pm2 save && sudo pm2 startup<\/pre>\n
To run it behind a reverse proxy with TLS termination and automated Let’s Encrypt certificates.<\/p>\n
sudo apt install nginx certbot python3-certbot-nginx -y\r\nsudo nano \/etc\/nginx\/sites-enabled\/default<\/pre>\n
server {\r\n   listen 80 default_server;\r\n   listen [::]:80 default_server;\r\n   root \/var\/www\/html;\r\n   index index.html;\r\n   server_name _;\r\n   location \/ {\r\n      proxy_pass http:\/\/127.0.0.1:3001;\r\n      proxy_set_header Host $host;\r\n   }\r\n}<\/pre>\n
sudo nginx -t && sudo nginx -s reload\r\nsudo certbot --nginx --non-interactive --agree-tos --redirect --email devops@domain.com<\/strong> -d uptime.domain.com<\/strong><\/pre>\n
If you need to monitor an external or a public service to identify if they are experiencing an outage that affects your monitoring, check out Status Shield<\/strong> [Link<\/a>].<\/p>\n
\n
MOREOVER<\/strong><\/p>\n
For monitoring cron jobs or background tasks that could silently fail, try healthchecks.io<\/strong> [Link<\/a>].<\/p>\n
The following docker-compose.yaml<\/code> takes care of the whole deployment and configuration.<\/p>\n
version: \"3\"\r\nservices:\r\n  healthchecks:\r\n    image: healthchecks\/healthchecks:latest\r\n    container_name: healthchecks\r\n    environment:\r\n      - DB=sqlite\r\n      - DB_NAME=\/data\/hc.sqlite\r\n      - DEBUG=False\r\n      - DEFAULT_FROM_EMAIL=healthchecks@example.com\r\n      - EMAIL_HOST=smtp.example.com\r\n      - EMAIL_HOST_USER=healthchecks@example.com\r\n      - EMAIL_HOST_PASSWORD=strong_password\r\n\r\n###### For STARTTLS - Explicit TLS\r\n      - EMAIL_PORT=587\r\n      - EMAIL_USE_TLS=True\r\n\r\n###### For SMTPS - Implicit TLS (deprecated)\r\n#      - EMAIL_PORT=465\r\n#      - EMAIL_USE_TLS=False\r\n#      - EMAIL_USE_SSL=True\r\n\r\n      - SECRET_KEY=f7993db8-1004-414b-be03-ad4a5d5153a3\r\n      - ALLOWED_HOSTS=healthchecks.simnet.cloud\r\n      - SITE_ROOT=https:\/\/healthchecks.simnet.cloud\/\r\n      - REGISTRATION_OPEN=False\r\n    ports:\r\n      - 8000:8000\r\n    volumes:\r\n      - healthchecks-data:\/data\r\n    restart: unless-stopped\r\nvolumes:\r\n    healthchecks-data:<\/pre>\n
It is also recommended to keep behind a reverse proxy that terminates TLS.<\/p>\n
\n
See also Grafana Alloy<\/strong> and Grafana Loki<\/strong> for combining logs with metrics and visualizing and drilling down with Grafana Dashboards.<\/p>\n