Our upstream provider has identified the issue and rolled out fixes to the operating system of their databases. We will mark the issue as resolved, but we will also keep monitoring the system with heightened awareness for some time.
Posted Nov 09, 2018 - 15:11 CET
Update
We continue to monitor the situation. Since implementing our early warning system, we have not seen the spikes in data volume that earlier caused false positives, so we have not been able to verify our temporary solution. We are also continuing to work with our upstream provider to investigate the cause of the underlying peaks in metrics data volume.
Posted Nov 07, 2018 - 20:15 CET
Monitoring
We have implemented an early feedback system that will prevent most false positives from being sent. We are currently monitoring the system's operation.
Posted Oct 30, 2018 - 18:04 CET
Update
Whilst we are still pinpointing the exact cause of this issue with our supplier, we have now deployed measures which should alleviate the symptoms that customers are experiencing. We are monitoring the situation closely.
Posted Oct 26, 2018 - 14:50 CEST
Update
There was another round of false positives yesterday evening and this morning. So far, we are still pinpointing the exact cause of the high extraneous loads. We will roll out a temporary measure today or tomorrow to alleviate the situation.
Posted Oct 25, 2018 - 08:07 CEST
Update
We are continuing to work on a fix for this issue.
Posted Oct 24, 2018 - 08:06 CEST
Update
Working with our supplier a number of possible causes have been identified which are currently being investigated.
Posted Oct 23, 2018 - 10:22 CEST
Update
Unfortunately the root cause of the issue is taking longer than expected. We are pro-actively monitoring for false positives.
The issue causing the false positives occurred yesterday and as a result, we temporary disabled runtime heartbeat for the duration of the spikes. The heartbeat monitoring has since been reinstated. To further try and reduce the possibilities for false positives, the heartbeat monitoring has been set to 15 minutes.
Posted Oct 23, 2018 - 09:07 CEST
Update
Unfortunately the root cause of the issue is taking longer than expected. We are pro-actively monitoring for false positives.
Posted Oct 16, 2018 - 14:42 CEST
Update
Unfortunately the root cause of the issue is taking longer than expected. We are pro-actively monitoring for false positives.
Posted Oct 10, 2018 - 10:22 CEST
Update
Investigations to the root cause of this issue are ongoing. Spikes are being observed in the rate at which the monitoring service receives messages. During these spikes, the alerting service falls behind in processing the runtime heartbeat. Due to the nature of the heartbeat, this causes false alerts.
We are working with our upstream supplier to establish the cause of these spikes. A circuit breaker is scheduled to be added to our alerting stack, so that the high volume of false alerts during spikes can be reduced.
Posted Oct 09, 2018 - 11:02 CEST
Identified
The investigation into the exact cause of the issue is ongoing.
Posted Sep 28, 2018 - 14:19 CEST
Investigating
We're investigating why incorrect alerts were sent out to subscribers of application alerts.
Posted Sep 24, 2018 - 15:31 CEST
This incident affected: Mendix Cloud (Mendix Cloud US-East, Mendix Cloud EU (Frankfurt), Mendix Cloud Asia Pacific (Tokyo), Mendix Cloud Free Tier EU, Mendix Cloud UK, Mendix Cloud IE).