Monitoring a Linux server using Prometheus and Grafana is a powerful combination for tracking server metrics, visualizing them, and setting up alerts. Here's a step-by-step guide on how to set up this monitoring stack:
1. Download & Install Prometheus:
Download Prometheus from the official website: https://prometheus.io/download/
[root@siddhesh ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
--2023-10-22 16:43:21-- https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
Resolving github.com (github.com)... 20.207.73.82
Connecting to github.com (github.com)|20.207.73.82|:443... connected.
...
Output Truncated
...
100%[==================================================================================================================>] 95,708,924 250MB/s in 0.4s
2023-10-22 16:43:23 (250 MB/s) - ‘prometheus-2.47.2.linux-amd64.tar.gz’ saved [95708924/95708924]
[root@siddhesh ~]#Now we will extract the Prometheus setup file using the following commands.
[root@siddhesh ~]# tar xvpf prometheus-2.47.2.linux-amd64.tar.gz
prometheus-2.47.2.linux-amd64/
prometheus-2.47.2.linux-amd64/LICENSE
prometheus-2.47.2.linux-amd64/NOTICE
prometheus-2.47.2.linux-amd64/prometheus.yml
prometheus-2.47.2.linux-amd64/consoles/
prometheus-2.47.2.linux-amd64/consoles/prometheus.html
prometheus-2.47.2.linux-amd64/consoles/prometheus-overview.html
prometheus-2.47.2.linux-amd64/consoles/node-cpu.html
prometheus-2.47.2.linux-amd64/consoles/index.html.example
prometheus-2.47.2.linux-amd64/consoles/node.html
prometheus-2.47.2.linux-amd64/consoles/node-disk.html
prometheus-2.47.2.linux-amd64/consoles/node-overview.html
prometheus-2.47.2.linux-amd64/promtool
prometheus-2.47.2.linux-amd64/console_libraries/
prometheus-2.47.2.linux-amd64/console_libraries/prom.lib
prometheus-2.47.2.linux-amd64/console_libraries/menu.lib
prometheus-2.47.2.linux-amd64/prometheus
[root@siddhesh ~]#Copy Prometheus Binary files.
[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/prometheus /usr/local/bin/
‘prometheus-2.47.2.linux-amd64/prometheus’ -> ‘/usr/local/bin/prometheus’
[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/promtool /usr/local/bin/
‘prometheus-2.47.2.linux-amd64/promtool’ -> ‘/usr/local/bin/promtool’
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/consoles /etc/prometheus
sending incremental file list
consoles/
consoles/index.html.example
consoles/node-cpu.html
consoles/node-disk.html
consoles/node-overview.html
consoles/node.html
consoles/prometheus-overview.html
consoles/prometheus.html
sent 19,993 bytes received 153 bytes 40,292.00 bytes/sec
total size is 19,486 speedup is 0.97
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/console_libraries /etc/prometheus
sending incremental file list
console_libraries/
console_libraries/menu.lib
console_libraries/prom.lib
sent 9,235 bytes received 58 bytes 18,586.00 bytes/sec
total size is 9,040 speedup is 0.97
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/prometheus.yml /etc/prometheus
sending incremental file list
prometheus.yml
sent 1,039 bytes received 35 bytes 2,148.00 bytes/sec
total size is 934 speedup is 0.87
[root@siddhesh ~]#It will be necessary for us to create a Prometheus directory and Prometheus user.
[root@siddhesh ~]# useradd --no-create-home --shell /bin/false prometheus
[root@siddhesh ~]# useradd --no-create-home --shell /bin/false node_exporter
[root@siddhesh ~]# mkdir /etc/prometheus
[root@siddhesh ~]# mkdir /var/lib/prometheusModify the ownership of Prometheus users on Binaries and Folders.
[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/prometheus
[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/promtool
[root@siddhesh ~]# chown prometheus:prometheus /etc/prometheus
[root@siddhesh ~]# chown prometheus:prometheus /var/lib/prometheus
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/consoles
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/prometheus.yml Verify version of Prometheus by running below commands.
[root@siddhesh ~]# prometheus --version
prometheus, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
build user: root@79f2ad339b75
build date: 20231012-16:07:10
go version: go1.21.3
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
[root@siddhesh ~]# promtool --version
promtool, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
build user: root@79f2ad339b75
build date: 20231012-16:07:10
go version: go1.21.3
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
[root@siddhesh ~]#2. Create Prometheus Systemd file.
[root@siddhesh ~]# cat /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
[root@siddhesh ~]#3. Reload systemctl daemon & Start Prometheus service
[root@siddhesh ~]# systemctl daemon-reload
[root@siddhesh ~]# systemctl start prometheus
[root@siddhesh ~]# systemctl enable prometheus4. Check service staus & access Prometheus using http://<serverip>:9090
Check the current state of the default target, http://127.0.0.1:9090/metrics.
5. Grafana Installation and Configuration
Enable grafana repo and install grafana package.
[root@siddhesh ~]# cat /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/enterprise/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=0
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
[root@siddhesh ~]#[root@siddhesh ~]# yum install grafana
Installing : grafana-10.1.5-1.x86_64 1/1
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server.service
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-server.service
POSTTRANS: Running script
Verifying : grafana-10.1.5-1.x86_64 1/1
Installed:
grafana.x86_64 0:10.1.5-1
Complete!
[root@siddhesh ~]#6. Start Grafana Service & Enable in system startup.
[root@siddhesh ~]# systemctl start grafana-server
[root@siddhesh ~]# systemctl enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
[root@siddhesh ~]#7. Check status of grafana & access grafana portal using http://<serverip>:3000
[root@siddhesh ~]# systemctl status grafana-server
8. Install Node Exporter on Linux to gathers system parameters such as memory use, CPU utilisation, RAM, disc space, and so on.
To install Node Exporter, go to the official Prometheus download page, scroll down to the node_exporter section, and then pick Linux OS for amd64.
[root@siddhesh ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gzExtract the content of node_exporter tar.
[root@siddhesh ~]# tar -xvpf node_exporter-1.6.1.linux-amd64.tar.gz
node_exporter-1.6.1.linux-amd64/
node_exporter-1.6.1.linux-amd64/NOTICE
node_exporter-1.6.1.linux-amd64/node_exporter
node_exporter-1.6.1.linux-amd64/LICENSE
[root@siddhesh ~]#9. Configure node_exporter & start node_exporter service.
Copy node_exporter binary under /usr/loca/bin/
[root@siddhesh ~]# cp -pav node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
‘node_exporter-1.6.1.linux-amd64/node_exporter’ -> ‘/usr/local/bin/node_exporter’
[root@siddhesh ~]#Assing relevant permission with the user of node_exporter which we had created earlier in this session.
[root@siddhesh ~]# chown node_exporter:node_exporter /usr/local/bin/node_exporterCreate node_exporter.service with following content and start the service.
[root@siddhesh ~]# cat /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
[root@siddhesh ~]#[root@siddhesh ~]# systemctl start node_exporter
[root@siddhesh ~]# systemctl enable node_exporter10. Set up the Node Exporter to be a target for Prometheus.
Add Node Exporter target i.e localhost:9100 in Prometheus target as shown below & restart Prometheus service.
[root@siddhesh ~]# grep targets /etc/prometheus/prometheus.yml
- targets:
- targets: [‘localhost:9090’, ‘localhost:9100’]
[root@siddhesh ~]# systemctl restart prometheus11. Login to the Grafana console & Setup Data Source using Prometheus target.
Go to http://<serverip>:300 and log in using the admin/admin Grafana user and password by default. This will forcefully redirect you to set a new password the first time; if you don't want to change, skip this step.
After logging into Grafana Now, go to Settings -> Configuration -> Data Sources first.
Let's select Prometheus by clicking on Add Data sources now.
12. Create a Grafana Dashboard for Monitoring a Linux Server
Let's now create a Grafana dashboard that can display the metrics of the Linux system.
So we'll utilise 7675 to import Grafana.com. Go to the Grafana Home page and look for the "+" symbol. Select "Import" by clicking on it. Now enter the Grafana.com Dashboard ID 7675 and press the Load button.
Now select a name, take the Prometheus Datasource, and click Import.
Click On Import and continue to see the Dashboard view.
