Prometheus : Set Alert Notification (Email)2021/06/03

	This is the Alert Notification Settings on Prometheus. There are many way to receive Alerts like Slack, HipChat, WeChat and others, though, on this example, Configure Alerting with Email Receiver. For more details of Alerting, Refer to the official documents. ⇒ https://prometheus.io/docs/alerting/configuration/
[1]	For Email notification, it needs SMTP Server. On this example, it based on the environment that SMTP Server is running on localhost.
[2]	Install Alertmanager on Prometheus Server Host.

[root@dlp ~]#

dnf -y install alertmanager

[3]	Configure Prometheus Alert Settings with Email notification.

[root@dlp ~]#

mv /etc/prometheus/alertmanager.yml /etc/prometheus/alertmanager.yml.org

[root@dlp ~]#

vi /etc/prometheus/alertmanager.yml

# create new

global:
  # SMTP server to use
  smtp_smarthost: 'localhost:25'
  # require TLS or not
  smtp_require_tls: false
  # notification sender's Email address
  smtp_from: 'Alertmanager <root@dlp.srv.world>'
  # if set SMTP Auth on SMTP server, set below, too
  # smtp_auth_username: 'alertmanager'
  # smtp_auth_password: 'password'

route:
  # Receiver name for notification
  receiver: 'email-notice'
  # grouping definition
  group_by: ['alertname', 'Service', 'Stage', 'Role']
  group_wait:      30s
  group_interval:  5m
  repeat_interval: 4h

receivers:
# any name of Receiver
- name: 'email-notice'
  email_configs:
  # destination Email address
  - to: "root@localhost"

# configure Alerting rules

[root@dlp ~]#

vi /etc/prometheus/alert_rules.yml

# create new

# for example, monitor node-exporter's [Up/Down]

groups:
- name: Instances
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      description: '{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.'
      summary: 'Instance {{ $labels.instance }} down'

[root@dlp ~]#

vi /etc/prometheus/prometheus.yml

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # line 12 : change to (Alertmanager Host):(Port)
      - 'localhost:9093'

# line 18 : add alert rules created above
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "alert_rules.yml"

[root@dlp ~]#

systemctl restart prometheus alertmanager

[root@dlp ~]#

systemctl enable alertmanager

[4]	If [node-exporter] is down, following Email is sent. (mail body is HTML)

[root@dlp ~]#

mail

Message 19:
From root@dlp.srv.world  Wed Jun  2 20:18:38 2021
Return-Path: <root@dlp.srv.world>
X-Original-To: root@localhost
Delivered-To: root@localhost
From: Alertmanager <root@dlp.srv.world>
Subject: [FIRING:1] InstanceDown (node01.srv.world:9100 Hiroshima critical)
To: root@localhost
Date: Wed, 02 Jun 2021 20:18:37 -0500
Content-Type: multipart/alternative;  boundary=bde99c641c75ec2265b8f69f8a2c06868
58896abaa6872b1b4a9d480447a
Status: R

Part 1:
Content-Type: text/html; charset=UTF-8

.....
.....

Matched Content