Prometheus : Set Alert Notification (Email)2024/06/12

	This is the Alert Notification Settings on Prometheus. There are many way to receive Alerts like Slack, HipChat, WeChat and others, though, on this example, Configure Alerting with Email Receiver. For more details of Alerting, Refer to the official documents. ⇒ https://prometheus.io/docs/alerting/configuration/
[1]	For Email notification, it needs SMTP Server. On this example, it based on the environment that SMTP Server is running on localhost.
[2]	Install Alertmanager on Prometheus Server Host.

root@dlp:~#

apt -y install prometheus-alertmanager

[3]	Configure Prometheus Alert Settings with Email notification.

root@dlp:~#

mv /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.org

root@dlp:~#

vi /etc/prometheus/alertmanager.yml

# create new

global:
  # SMTP server to use
  smtp_smarthost: 'localhost:25'
  # require TLS or not
  smtp_require_tls: false
  # notification sender's Email address
  smtp_from: 'Alertmanager <root@dlp.srv.world>'
  # if set SMTP Auth on SMTP server, set below, too
  # smtp_auth_username: 'alertmanager'
  # smtp_auth_password: 'password'

route:
  # Receiver name for notification
  receiver: 'email-notice'
  # grouping definition
  group_by: ['alertname', 'Service', 'Stage', 'Role']
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 4h

receivers:
# any name of Receiver
- name: 'email-notice'
  email_configs:
  # destination Email address
  - to: "root@localhost"

# configure Alerting rules

root@dlp:~#

vi /etc/prometheus/alert_rules.yml

# create new

# for example, monitor node-exporter's [Up/Down]

groups:
- name: Instances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

root@dlp:~#

vi /etc/prometheus/prometheus.yml

# line 17 : confirm - (Alertmanager Host):(Port)
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# line 23 : add alert rules created above
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "alert_rules.yml"

root@dlp:~#

systemctl restart prometheus prometheus-alertmanager

root@dlp:~#

systemctl enable prometheus-alertmanager

[4]	If [node-exporter] is down, following Email is sent. (mail body is HTML)

root@dlp:~#

mail

? 1
Return-Path: <root@dlp.srv.world>
X-Original-To: root@localhost
Delivered-To: root@localhost
Received: from localhost (localhost [127.0.0.1])
        by dlp.srv.world (Postfix) with UTF8SMTP id CBA8C1A08C8
        for <root@localhost>; Wed, 12 Jun 2024 02:04:37 +0000 (UTC)
Subject: [FIRING:1] InstanceDown (node01.srv.world:9100 Hiroshima example critic
al)
To: root@localhost
From: Alertmanager <root@dlp.srv.world>
Message-Id: <1718157877834688641.16331542935340113689@dlp.srv.world>
Date: Wed, 12 Jun 2024 02:04:37 +0000
Content-Type: multipart/alternative;  boundary=c6ec120ee879e4fd5c4da94d3b07f8b42
dc84c9de9ff108a0a59443388cf
MIME-Version: 1.0

--c6ec120ee879e4fd5c4da94d3b07f8b42dc84c9de9ff108a0a59443388cf
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8



<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.=
.....
.....

Matched Content