As announced on Thursday, December 23, all our platform services and components have been updated so the vulnerability isn’t there. It was never there in the Mendix Runtime to start with, as this doesn’t use log4j-core. Our Marketplace content is also free of potentially vulnerable components. The only potential vulnerabilities left are in customer apps which have not yet been patched, and our urgent advice remains to update log4j-core to the latest version (currently 2.17.0) in case log4j-core is part of your project.
We have continued monitoring our traffic on the Mendix Cloud. As with all internet services at this time, we see a large amount of scans taking place, both by legitimate parties (penetration testers, for instance) as well as from potentially malicious sources. You may also see evidence of that in your application logs. Entries in which requests are visible that look similar to the string that exposes the vulnerability indicate that someone is looking for it. This is in no way evidence that a hack was successful.
We did not see traffic or usage patterns that indicate that a hacking attempt was successful on our platform. Using CrowdStrike and the services of the Siemens Cyber Defense Center, we have seen some false positives. Upon further inspection, in this limited number of cases there was a legitimate explanation. For our own services and components, we can rule out that they were compromised. For customer apps, that is not something that we can ascertain, as Mendix does not have access to customer apps and data. However, CrowdStrike has the capability to assess customer containers as well and no malicious behaviour has been detected in- and outside customer containers.
This is the final update on this incident, and, unless there are additional developments, we will close this incident shortly.
Posted Dec 24, 2021 - 15:36 CET
Monitoring
We have updated the entire platform including its components to the latest Log4J version (currently 2.17) where applicable. Marketplace content has been updated by contributors and all content is secure. We advise customers to upgrade Marketplace content part of their project to the version currently available on Marketplace. Our AppDynamics and Datadog integrations have been secured against all three vulnerabilities. Also, new Studio Pro patches have been released for all supported major Mendix versions. These can be downloaded through the regular channels.
As we continue to monitor traffic on the Mendix Cloud and closely follow all developments around Log4Shell, we’re updating the status of this message to “Monitoring”. Customers continue to act swiftly in updating their apps to the latest Log4J version and we’ll continue to communicate with customers who haven’t done so yet. To be clear: customer apps that have not been updated yet on the Mendix Cloud do not pose a threat to the platform or other customer apps. In the unlikely event that a non-patched customer app is compromised, your patched app and its data are safe.
Our urgent advice remains to update log4j-core to the latest version (currently 2.17.0) in case log4j-core is part of your project.
Posted Dec 23, 2021 - 10:20 CET
Update
On Tuesday, December 21, we have released an updated buildpack containing a patched Datadog agent that is not vulnerable to CVE-2021-44228 and CVE-2021-45046, and is not impacted by CVE-2021-45105. We recommend customers to enable the Datadog integration again by setting the Datadog-related environment variables in the Developer Portal and redeploying the relevant Mendix application. The deployment process will automatically pick up the changes in the buildpack.
Also, we got confirmation from AppDynamics that the default configuration they ship (and Mendix redistributes) is not impacted by CVE-2021-45105 either. This means that we’ll change our advice to our customers again. The current AppDynamics agent we redistribute is not vulnerable to CVE-2021-44228 and CVE-2021-45046 and not impacted by CVE-2021-45105. Therefore, we recommend AppDynamics users to set the AppDynamics-related environment variables in the Developer Portal and redeploy the relevant Mendix application. We are aware that we’re going back and forth on this advice, but in the interest of our customers’ safety and security, we prefer to err on the side of caution while being fully transparent on the latest developments in this unfolding story.
Our urgent advice continues to be to update log4j-core to the latest version (currently 2.17.0) in case log4j-core is part of your project. We are in the process of updating all our components and services to the latest log4j-core version as well. Also, we’re proactively reaching out to customers whose apps have not yet been updated with this version. We’ll continue to provide updates on our ongoing efforts to keep our platform and our customer apps safe and secure.
Posted Dec 21, 2021 - 11:29 CET
Update
We have received an updated agent including the latest Log4J version from Datadog today. We’re currently incorporating it into our buildpack, which we will release after succesful QA. We expect this to be on Tuesday, December 21. Until then, our advice to switch off the Datadog integration stands.
Our guidance with respect to AppDynamics has changed. The change that we made to our AppDynamics agent based on the advice from AppDynamics mitigates CVE-2021-44228 and CVE-2021-45046. However, it’s not resilient against CVE-2021-45105, which was disclosed on Saturday, December 18. Although your data remains safe, the newly-found vulnerability would enable an attacker to make your app unresponsive. We’re currently investigating if this can be mitigated in the current agent version, while we’re ready to incorporate a new and more secure agent from AppDynamics. We have not received an ETA for this yet. Our advice is to switch off the AppDynamics integration by removing the AppDynamics-related environment variables from the environment in the Developer Portal and redeploying the relevant Mendix application.
In the meantime, we’re working on upgrading all our platform services and components to the latest Log4J version, published this weekend, which mitigates all three vulnerabilities posted so far.
We will continue to keep you updated on our efforts to investigate the impact of these vulnerabilities on our platform, and our efforts to mitigate them.
Posted Dec 20, 2021 - 17:59 CET
Update
On Saturday, December 18, a new log4j-core vulnerability, CVE-2021-45105, was disclosed. Unlike the other two vulnerabilities, this one only has implications in terms of availability, but not in terms of integrity and confidentiality. Data and apps from customers who have followed our guidance so far are safe. We have measures in place to prevent availability issues on the Mendix Cloud caused by this vulnerability.
Also, a new log4j-core version was made available addressing that vulnerability. Our urgent advice to update log4j-core to the latest version (currently 2.17.0) in case log4j-core is part of your project, remains as is. We will also perform the same update to our components shortly.
We will keep you updated on our efforts to investigate the impact of these vulnerabilities on our platform, and our efforts to mitigate them.
Posted Dec 18, 2021 - 10:34 CET
Update
We see that many customers have acted swiftly and promptly on our urgent advice to update log4j-core to the latest version (currently 2.16.0) in case log4j-core is part of their project. This version remediates the vulnerabilities published as CVE-2021-44228 on Friday, December 10, and CVE-2021-45046 on Tuesday, December 14.
The latest agent version from AppDynamics turned out not to be entirely stable, so we have chosen not to include it into the buildpack we released on Friday, December 17. Instead, we have patched the AppDynamics agent already in the buildpack, according to AppDynamics’ own guidance. As soon as we receive a proven stable version, we’ll include that into our buildpack. Customers using AppDynamics are safe in any case using today’s buildpack release.
Status regarding the Datadog agent remains unchanged while we’re working with Datadog to get an updated agent including the latest Log4J version as soon as possible. Our advice remains to switch off the Datadog integration by removing the Datadog-related environment variables from the environment in the Developer Portal and redeploying the relevant Mendix application.
As stated before, for deploying directly from Studio Pro to your own private Cloud Foundry instance, Studio Pro relies on a script called mx-cf-client. This script contains a vulnerable log4j-core version. We have released the following versions to mitigate this vulnerability, which would only appear if your own Cloud Foundry cluster is vulnerable or compromised:
- Mendix 7: Studio Pro 7.23.27 - Mendix 8: Studio Pro 8.18.14 - Mendix 9: Studio Pro 9.6.4 - Mendix 9: Studio Pro 9.9.0
These versions are generally available, so although they fix a specific mx-cf-client vulnerability, they’re recommended for general use. All new releases are available through the regular channels.
As we continue to scan and monitor our Mendix Cloud security perimeter, we have noticed an uptake in scanning activities coming from legitimate sources, indicating that many parties are probing their own apps on our cloud platform for this vulnerability. We support this good practice.
We will keep you updated on our efforts to investigate the impact of this vulnerability on our platform, and our efforts to mitigate them.
Posted Dec 17, 2021 - 16:11 CET
Update
Today, Thursday December 16, we released a new version of our buildpack, upgrading APM integrations to the latest and safest versions.
This upgrade also includes the Datadog agent. Although this upgrade mitigates many well-known attack vectors within the Datadog integration, Mendix is still waiting for the Datadog agent to be patched to the latest version of Log4J, upon which we will update our buildpack again.
Customers using Datadog are therefor advised to disable Datadog completely in order to mitigate the Log4J vulnerability. This risk is only applies to specific Mendix applications for which the customer has enabled the Datadog integration. Disabling can be done by removing the Datadog-related environment variables for their environment in the Developer Portal and redeploying the applicable Mendix application.
Posted Dec 16, 2021 - 17:47 CET
Identified
We continue to strongly urge customers to update log4j-core to the latest version (currently 2.16.0) in case log4j-core is part of your project. This version remediates the vulnerabilities published as CVE-2021-44228 on Friday, December 10, and CVE-2021-45046 on Tuesday, December 14.
One of the ways in which Log4J can still be part of your project is if it uses Marketplace content that is vulnerable. We have reached out to owners of vulnerable content requesting them to patch their components. Content that was vulnerable has been removed and content owners have been informed. We advise customers using Marketplace content to update their components to the latest Marketplace version if they haven’t done so.
Furthermore, since early this week, we have been actively scanning all deployment archives running on the Mendix Cloud for the vulnerable log4j-core library. Customers who have apps which are still vulnerable have been proactively informed and we will continue to do so.
Also we have measures in place on the Mendix Cloud to block incoming traffic that can be identified as an attempt to detect and exploit these vulnerabilities. It is important to stress that in the unlikely case such an attempt is successful, this doesn’t affect any other running app, due to the security architecture of the Mendix Platform. Additionally the Siemens Cyber Defense Center continuously monitors Mendix Cloud using CrowdStrike to identify malicious behavior.
We will keep you updated on our efforts to investigate the impact of this vulnerability on our platform, and our efforts to mitigate them.
Posted Dec 16, 2021 - 14:47 CET
Update
On Tuesday, December 14, a new CVE (CVE-2021-45046) was published, indicating that contrary to earlier reports, log4j-core version 2.15 is still vulnerable to the zero-day exploit, shared on Friday, December 10. The bug, tracked as CVE-2021-44228 and dubbed Log4Shell, might allow an attacker to run any arbitrary code through the Log4j Java library used for logging. For more in depth information on this vulnerability, see the Security page of the Apache Logging Services website.
As stated before, the Mendix runtime itself is not vulnerable to this exploit. We nevertheless continue to strongly urge customers to upgrade log4j-core to the latest version (currently 2.16.0) in case log4j-core is part of your project. This is regardless of the JRE/JDK version the app runs on.
The following only applies to customers who deploy to their own private Cloud Foundry cluster. This has no impact on the Public Cloud:
The Studio Pro distribution contains a utility called `mx-cf-client`. `mx-cf-client` gives the possibility to deploy a Mendix application to Cloud Foundry using Studio Pro. `mx-cf-client` contains a vulnerable `log4j-core` library, the only scenario this could be exploited is when the applicable Cloud Foundry server is untrusted or hacked.
`mx-cf-client` is only used for Cloud Foundry installations for private cloud and not used to deploy to the Mendix public cloud.
To remediate this vulnerability, we are working on the patch release of Mx7.23.27, Mx8.18.14 and Mx9.6.4, these versions will be released in the coming days. Upgrading your applications to one of these versions, or a later one, will remove this vulnerability. The patch for this vulnerability is included in our monthly release starting Mx9.9.0.
For any additional questions, please contact support.
We will continue to keep you updated on our efforts to investigate the impact of this vulnerability on our platform, and our efforts to mitigate them.
Posted Dec 15, 2021 - 10:12 CET
Update
On Friday, December 10, a new zero-day exploit for the Apache Log4j Java library was shared. The bug, tracked as CVE-2021-44228 and dubbed Log4Shell, might allow an attacker to run any arbitrary code through the Log4j Java library used for logging. For more in depth information on this vulnerability, see the Security page of the Apache Logging Services website.
Although the Mendix runtime itself is not vulnerable to this exploit, we nevertheless strongly urge customers to upgrade log4j-core to the latest version (currently 2.16.0) in case log4j-core is part of your project. This is regardless of the JRE/JDK version the app runs on. Please be aware that on version 2.15, a new CVE (CVE-2021-45046) was published on Tuesday, December 13, indicating that contrary to earlier reports, this version is also subject to the vulnerability.
We will keep you updated on our efforts to investigate the impact of this vulnerability on our platform, and our efforts to mitigate them.
Posted Dec 13, 2021 - 13:19 CET
Investigating
The Mendix Platform does not seem to be affected by CVE-2021-44228, a critical vulnerability in Log4j, a commonly used Java Library. We will post a more in-depth update at a later time.
Posted Dec 13, 2021 - 12:28 CET
This incident affected: Mendix Cloud (Mendix Cloud US-East, Mendix Cloud EU (Frankfurt), Mendix Cloud Asia Pacific (Tokyo), Mendix Cloud Free Tier EU, Mendix Cloud UK, Mendix Cloud IE, Mendix Cloud US-West, Mendix Cloud Asia Pacific (Sydney), Mendix Cloud Canada (Central), Mendix Cloud Asia Pacific (Singapore)) and Mendix Cloud Dedicated (Mendix Cloud Dedicated EU).