-- and make sure, if those notifications are emails, that they are not triggered every single time the error occurs, potentially flooding you with thousands of emails.
What do people use for this kind of thing? I'm aware of Bosun (https://bosun.org/) and Prometheus (https://prometheus.io/). Both can alert based on aggregated metrics, using rich rules such as values moving away from historical averages by a certain threshold.
A quick plug for Pushover, which has completely changed how I deal with alerts like that. I used to have them sent to me via email, but now the critical / urgent stuff actually buzzes my phone & their API is really easy to use:
