Update - Together with our upstream provider, we have identified a few queries that can cause higher loads on the system. We are testing the changes and we will roll these out later this week.
Oct 16, 14:42 CEST

Update - We are still investigating with our upstream supplier what could cause the high peak loads of processing volumes. Meanwhile we are pro-actively monitoring each of the clusters for false positives to see if we can identify early warning signals.
Oct 10, 10:22 CEST

Update - We have not yet identified the root cause of the issue. What we can see is there are sudden spikes in the data volume that is processed by our alerting service. During these spikes, our alerting service falls behind in processing the runtime heartbeat. Due to the nature of the heartbeat (we alert when we have not seen one for a few minutes), this causes false alerts. We are investigating with our upstream supplier what causes these peaks. A circuit breaker is scheduled to be added to our alerting stack, so that we can avoid high volumes of false alerts during the peak loads.
Oct 9, 11:02 CEST

Identified - The cause of the issue appeared to be a temporary slowness in putting metrics data into our time series database. We will open a ticket with our Upstream provider to discover the cause of this. We will also implement a circuit breaker to prevent false positives to be sent in cases like these.
Sep 28, 14:19 CEST

Investigating - We're investigating why incorrect alerts were sent out to subscribers of application alerts.
Sep 24, 15:31 CEST