Tech tsunami: What triggered the global CrowdStrike crash, what to do next
Blue screen of death (BSOD) crashes blamed on CrowdStrike 'Falcon sensor' update
Highlights
- 'Falcon Sensor' update blamed for the widespread system failure that triggered blue screen of death (BSOD) error worldwide.
- Falcon is a Crowdstrike "endpoint detection and response platform"
- CrowdStrike says it has identified the problematic version, shows workaround.
A major tech meltdown rippled across the globe overnight on Thursday and Friday (July 19, 2024) as Microsoft reported a global outage of its online services.
Thousands of flights were grounded, causing inordinate suffering among tens of thousands of flyers. Many businesses faced IT nightmares as Windows machines malfunctioned worldwide.
The dreaded and so-called "blue screen of death" (BSOD) greeted thousands, perhaps millions, of frustrated Microsoft users.
The BSOD errors on Windows systems form a critical error that abruptly shuts down (or restarts) the computer.
also see
- Microsoft global outage: What’s CrowdStrike, and how does it affect you?
- Global IT collapse puts cyber firm CrowdStrike in spotlight
- Microsoft-CrowdStrike outage: UAE's Etihad, Emirates flights may face delays today, say officials
- UAE airlines, Dubai airports, MoFA, telecoms authority responds to Microsoft outages
CrowdStrike, a cybersecurity company based in Texas, said in a support note that it identified a “content deployment" related to the issue and reverted the changes.
It's the Channel file "C-00000291*.sys" with timestamp of 0409 UTC that was the "problematic version".
From Australia and Germany to India and the US, a cascade of technical glitch triggered disruptions for businesses, from banks and airports to TV stations and hotels, according to a Wired report.
Mac and Linux hosts were not impacted.
Here's what we know about the downtime, reportedly arising from Crowdstrike bug:
Cause
Buggy update: News sources initially suggested a recent Crowdstrike update might be responsible. It turned out to be true. Crowdstrike did confirm this to be the culprit.
Impact: The outage likely affected businesses that rely on Crowdstrike's cybersecurity solutions, potentially hindering their antivirus protection, threat intelligence, or incident response capabilities.
Downtime duration: Information on the exact duration of the outage is unavailable at this time.
Official cause: Crowdstrike CEO George Kurtz confirmed on X that a “defect” was found in a "single content update" for Windows hosts.
Points to consider
- The Falcon agent on a Windows machine was pushed out, though not sufficiently tested, thus causing the global outage. A post-mortem of the code would reveal the flaw, but this may take some time.
- Due to its potential impact on businesses and the critical role Crowdstrike plays in cybersecurity, the outage received significant news coverage.
- IT experts from the UAE Cybersecurity Council urged caution among all digital device users in order to avoid falling victim to hackers who may exploit this technical glitch.
What's 'Falcon sensor'?
A Falcon sensor, or "agent", is an endpoint detection and response platform that monitors the computers that it is installed on to detect intrusions like hacks and respond to them, University of Melbourne tech expert Toby Murray told Australian media.
Falcon is one of Crowdstrike software products that organisations install on their computers to keep them safe from cyber attacks and malware.
What Crowdstrike said
CrowdStrike CEO George Kurtz stated on X that the company is actively working with customers impacted by what he termed as a “defect” found in a single content update for Windows hosts.
“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”
“We further recommend organisations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilised to ensure the security and stability of CrowdStrike customers.
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.
Companies/organisations affected:
- Airlines
- Airports
- Banks
- Media
- Utilities
- Trading platforms
- Fastfood chains
The following countries were reportedly affected:
- Australia
- Japan
- India
- Philippines
- Spain
- Germany
- UK
- US
- New Zealand
Details
(As per Crowdstrike update 9:22am ET, July 19, 2024:)
- Symptoms include hosts experiencing a bugcheck|blue screen error related to the Falcon Sensor.
- Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
- Windows hosts which are brought online after 0527 UTC will also not be impacted
- Hosts running Windows 7/2008 R2 are not impacted
- This issue is not impacting Mac- or Linux-based hosts
- Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
- Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.
Current action
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:
Workaround steps for individual hosts (what to do next):
- Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
- Note: Bitlocker-encrypted hosts may require a recovery key.
It's important to stay updated on official announcements from Crowdstrike and Microsoft regarding the cause, resolution, and any recommendations for their clients.