Insight from CrowdStrike Regarding Friday Incident Causing Windows Devices to Crash in Large Numbers
According to CrowdStrike, a cybersecurity company, an incident last Friday led to the crashing of millions of Windows devices due to an issue in its validation system that resulted in a widespread outage.
CrowdStrike stated in its Preliminary Post Incident Review (PIR) that on July 19, 2024, at 04:09 UTC, a content configuration update for the Windows sensor was released to gather telemetry on potential new threat techniques as part of routine operations.
The company explained that such updates are regularly deployed to enhance the protection mechanisms of the Falcon platform and that this particular update led to a system crash on Windows devices.
Devices running sensor version 7.11 and above that were online during the specified time period and received the update were affected, while Apple macOS and Linux systems remained unaffected.
It was mentioned that security content configuration updates are delivered through Sensor Content and Rapid Response Content to identify emerging threats using behavioral pattern-matching techniques.
The crash was the outcome of a Rapid Response Content update that contained an unidentified error, which is distributed as Template Instances linked to specific behaviors for enabling new telemetry and detection.
These Template Instances are generated using a Content Configuration System, deployed to the sensor through Channel Files, written to disk on Windows machines, and validated by a Content Validator mechanism before publication.
CrowdStrike explained that Rapid Response Content aids with visibility and detection on the sensor by identifying adversary behavior using behavioral heuristics to prevent attacks, independently from the on-sensor AI capabilities.
The Falcon sensor’s Content Interpreter processes these updates to enable detection or prevention of malicious activities.
Although new Template Types undergo rigorous testing for various parameters, such as performance impact, CrowdStrike attributed the problem’s root cause to the deployment of the Interprocess Communication (IPC) Template Type on February 28, 2024, aiming to detect attacks involving named pipes.
The chain of events unfolded as follows –
- February 28, 2024 – Introduction of sensor 7.11 with new IPC Template Type
- March 5, 2024 – Validation of IPC Template Type after stress testing
- March 5, 2024 – Release of IPC Template Instance to production via Channel File 291
- April 8 – 24, 2024 – Deployment of three more IPC Template Instances in production
- July 19, 2024 – Deployment of two additional IPC Template Instances, one passing validation despite data issues
CrowdStrike clarified that due to successful prior deployments and content validation during testing, these instances were deployed into production, causing the system crash upon encountering problematic content triggering a Windows operating system crash (BSoD).
In response to the widespread disruptions caused by the incident, CrowdStrike mentioned enhancing testing procedures, improving error handling in the Content Interpreter, and planning to implement a staged deployment approach for Rapid Response Content.
About Author
