[Savannah-cvs] [735] add info on programs for receiving prometheus alerts

[iank]

Revision: 735
          
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=735
Author:   iank
Date:     2024-01-03 05:21:35 -0500 (Wed, 03 Jan 2024)
Log Message:
-----------
add info on programs for receiving prometheus alerts

Modified Paths:
--------------
    trunk/sviki/fsf/tools/prometheus.mdwn

Modified: trunk/sviki/fsf/tools/prometheus.mdwn
===================================================================
--- trunk/sviki/fsf/tools/prometheus.mdwn       2024-01-03 10:21:30 UTC (rev 
734)
+++ trunk/sviki/fsf/tools/prometheus.mdwn       2024-01-03 10:21:35 UTC (rev 
735)
@@ -265,6 +265,77 @@
 ```
 
 
+## Background on notification methods
+
+Highly desirable alerting properties:
+
+* Ability to silence all notifications on a phone except for alerts.
+
+* Sent as directly as possible from prometheus server to phone &
+  computer so failures in other systems don't disable alerts.
+
+* Not intermixed with security sensitive communication stream.
+
+Evaluation of options:
+
+
+Currently used options:
+
+xmpp server running on prometheus server:
+https://github.com/trazfr/prometheus-xmpp-alerting there are 3 xmpp apps
+in fdroid conversations, cheogram, and monocles chat, latter 2 are forks
+of cheogram. Even though none of them can be used for general xmpp
+communication and silence all notifications except for alerts, you can
+dedicate one app to only receiving alerts.
+
+Nagstamon: this works fine but only for a computer.
+
+
+Potentially good options but would need work to implement:
+
+Imap from the prometheus server: Needs investigation about notification
+if you want to receive other mail. Possible we could build something
+like k9mail but renamed to get a separate app with its own
+notifications.
+
+Liberachat IRC Bot: Not totally direct. Useful for volunteer
+transparency. Revolution IRC allows silencing notifications except for
+custom rules.
+
+Bot send to self-hosted irc server on prometheus server: Would have to
+learn how to run an irc server.
+
+
+Not great options but they could work:
+
+gotify: The gotify android app can only connect to one gotify
+server. This would mean we would have to send multiple data centers
+alerts to one gotify server, making that a single point of
+failure. There is a github issue for this years old, the creator said he
+won't do it because it isn't useful to him personally. Before I knew
+this, I got it running, it doesn't have a good message template by
+default. Also, only has an android client.
+
+SMS alerts via email bridge: We started with this. These work ok, but
+they rely on having non-voip sms. And the most minimal builtin template
+for them is far too verbose.
+
+SMTP: Not sent directly. This requires sysadmins to have some outside
+mail server and account setup that will only receive alerts. I have
+this, but other sysadmins don't. I could provide it to them, but that
+also makes it somewhat prone to temporary failures and downtime.
+
+
+
+Bad rejected options:
+
+xmpp server on prometheus host using
+https://github.com/trazfr/prometheus-xmpp-alerting This has only like 75
+lines of code. Run the version in t11 with debug flag, its logs just say
+connecting, but tcpdump shows nothing. I looked long and hard to figure
+out what was going wrong, to no avail.
+
+
 ## Background on why we are avoiding Grafana for graphs