Annotate

mod_pubsub_alertmanager/README.md @ 4623:622c6308d7af

mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
author Kim Alvefur <zash@zash.se>
date Fri, 16 Jul 2021 17:34:24 +0200
parent 4622:61ce3394fe8b
child 4624:eba7e68120d8
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
4622
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
1 ---
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
2 labels:
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
3 - 'Stage-Alpha'
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
4 summary: Alertmanager webhook receiver for pubsub
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
5 ---
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
6
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
7 # Introduction
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
9 This module lets
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
10 [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
11 publish alerts to [pubsub][doc:pubsub] via
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
12 [webhooks](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config).
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
13
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
14 # Setup
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
15
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
16 The relevant pubsub nodes must be created and configured somehow.
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
17 Because the request IP address is used to publish, the `publisher`
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
18 affiliation should be given to the IP address Alertmanager sends
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19 webhooks from.
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
20
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
21 # Configuration
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
22
4623
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
23 ## Prometheus
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
24
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
25 A Prometheus `rule_files` might contain something along these lines:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
26
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
27 ``` yaml
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
28 groups:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
29 - name: Stuff
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
30 rules:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
31 - alert: Down
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
32 expr: up == 0
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
33 for: 5m
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
34 annotations:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
35 title: 'Stuff is down!'
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
36 labels:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
37 severity: 'critical'
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
38 ```
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
39
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
40 ## Alertmanager
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
41 On the Alertmanager site the webhook configuration may look something
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
42 like this:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
43
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
44 ``` yaml
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
45 receivers:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
46 - name: pubsub
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
47 webhook_configs:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
48 - url: http://pubsub.localhost:5280/pubsub_alertmanager
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
49 ```
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
50
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
51 And then finally some Alertmanager routes would point at that receiver:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
52
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
53 ``` yaml
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
54 route:
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
55 receiver: pubsub
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
56 ```
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
57
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
58 ## Prosody
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
59
4622
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
60 On the Prosody side, apart from creating and configuring the node(s)
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
61 that will be used, configure your pubsub service like this:
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
62
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
63 ``` lua
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
64 Component "pubsub.example.com" "pubsub"
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
65 modules_enabled = {
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
66 "pubsub_alertmanager",
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
67 }
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
68
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
69 -- optional extra settings:
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
70 alertmanager_body_template = [[
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
71 *ALARM!* {annotations.title?Alert} is {status}
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
72 Since {startsAt}{endsAt& until {endsAt}}
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
73 Labels: {labels%
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
74 {idx}: {item}}
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
75 Annotations: {annotations%
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
76 {idx}: {item}}
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
77 ]]
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
78
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
79 alertmanager_node_template = "alerts/{alert.labels.severity}"
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
80 ```
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
81
4623
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
82 ## All Options
622c6308d7af mod_pubsub_alertmanager: Minimal example of Prometheus and Alertmanager config
Kim Alvefur <zash@zash.se>
parents: 4622
diff changeset
83
4622
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
84 Available configuration options:
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
85
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
86 `alertmanager_body_template`
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
87 : Template for the textual representation of alerts.
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
88
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
89 `alertmanager_node_template`
61ce3394fe8b mod_pubsub_alertmanager: Add a README with initial docs
Kim Alvefur <zash@zash.se>
parents:
diff changeset
90 : Template for the pubsub node name, defaults to `"{path?alerts}"`