Linux: How to set up monitoring with alerts to Telegram

This article describes how to set up monitoring with alerts to Telegram using Grafana, Prometheus, Alertmanager, Node-exporter and Cadvisor.

First step: Cloning the repository

Log in to the server or go to local terminal and run the following commands:

git clone https://github.com/digitalstudium/grafana-docker-stack.git
cd grafana-docker-stack
git checkout alertmanager

Second step: setting the external address of the server

Open the docker-compose.yml file and on lines 22 and 38 change the address 127.0.0.1 to the address of the server where you want to install Prometheus.

Third step: creating a bot in Telegram

In the Telegram search, type Botfather and follow the first link:

Then press Start:

After that, create a bot using the Botfather instructions. As a result, you should receive a message with an API token:

Copy this token and paste it on line 14 of the configs/alertmanager.yml file as the value of the bot_token parameter.

Then create a telegram group and add the created bot to this group. This group will receive notifications (alerts).

Fourth step: Get the group id

In the group you added the bot to, write some command, for example: /my_id foobar

Then in the browser follow the link https://api.telegram.org/botINSERT_BOT_TOKEN_HERE/getUpdates replacing INSERT_BOT_TOKEN_HERE with the token created in step 3. You should get something like this page:

If there is nothing on your page, try sending the command /my_id foobar to the group again.

You need to copy the chat id value from this page and paste it on line 15 of the configs/alertmanager.yml file as the value of the chat_id parameter. Note that the value of the chat_id parameter must be without quotes and with a hyphen at the beginning.

Fifth step: Deploying the docker stack

While in the grafana-docker-stack folder, run the following commands:

docker swarm init
docker stack deploy -c docker-compose.yml monitoring

Wait 5-10 minutes then run the command

docker ps

The command output should contain 5 containers: prometheus, grafana, alertmanager, node-exporter, grafana.

CONTAINER ID   IMAGE                              COMMAND                  CREATED      STATUS                PORTS                       NAMES
46fba26e7234   gcr.io/cadvisor/cadvisor:v0.47.0   "/usr/bin/cadvisor -…"   5 days ago   Up 5 days (healthy)   8080/tcp                    monitoring_cadvisor.1.q02qcn798dh0rydo1dzslylse
f212e3c66786   prom/alertmanager:v0.25.0          "/bin/alertmanager -…"   6 days ago   Up 6 days             9093/tcp                    monitoring_alertmanager.1.oysziztrqnur7xr0hr82avunz
c16fb14929e2   prom/prometheus:v2.42.0            "/bin/prometheus --c…"   6 days ago   Up 6 days             9090/tcp                    monitoring_prometheus.1.yslspi4fgjp7ic4f5e18gm99a
9bf01ce6b7a1   grafana/grafana:9.3.6-ubuntu       "/run.sh"                6 days ago   Up 6 days             3000/tcp                    monitoring_grafana.1.mypn85x12xw37ucprr33knuwk
58efdb46f5c3   kindest/node:v1.25.3               "/usr/local/bin/entr…"   6 days ago   Up 6 days             127.0.0.1:46579->6443/tcp   kind-control-plane
ae15e453e517   prom/node-exporter:v1.5.0          "/bin/node_exporter …"   7 days ago   Up 7 days             9100/tcp                    monitoring_node-exporter.1.uecim10ow12h1qlpox5lg0c5r

If you see this output, then everything turned out successfully. If not, try repeating all previous steps.

Sixth step: Check if it works

Go to [server ip-address or domain name]:3000 You should get Grafana opened. Enter login admin and password admin. Grafana will ask you to change your password, change it to any other. Under Dashboard -> Browse -> General you will see 2 dashboards: Cadvisor exporter and Node Exporter Full. Open them and make sure everything works.

Summary

Now you can evaluate server performance through Grafana graphs. You will also receive alerts about problems with the server in the Telegram group. Alert rules can be configured in configs/prometheus/alert_rules.yml