Please register to access this content.
To continue viewing the content you love, please sign in or create a new account
Dismiss
This content is for our paying subscribers only
Explainer

Tech tsunami: What triggered the global CrowdStrike crash, what to do next

Blue screen of death (BSOD) crashes blamed on CrowdStrike 'Falcon sensor' update



CrowdStrike outage: A blue screen for an inaccessible website on a laptop computer following reports of a major global outage, arranged in Palma de Mallorca, Spain, on Friday, July 19, 2024.
Image Credit: Bloomberg

Highlights

  • 'Falcon Sensor' update blamed for the widespread system failure that triggered blue screen of death (BSOD) error worldwide.
  • Falcon is a Crowdstrike "endpoint detection and response platform"
  • CrowdStrike says it has identified the problematic version, shows workaround.

A major tech meltdown rippled across the globe overnight on Thursday and Friday (July 19, 2024) as Microsoft reported a global outage of its online services.

Thousands of flights were grounded, causing inordinate suffering among tens of thousands of flyers. Many businesses faced IT nightmares as Windows machines malfunctioned worldwide.

The dreaded and so-called "blue screen of death" (BSOD) greeted thousands, perhaps millions, of frustrated Microsoft users.

The BSOD errors on Windows systems form a critical error that abruptly shuts down (or restarts) the computer.

Advertisement

also see

WHAT HAPPENED:
The blue screen of death (BSOD) crippled Windows users globally as Microsoft experienced a major outage due to a faulty update from cybersecurity provider CrowdStrike.

CrowdStrike, a cybersecurity company based in Texas, said in a support note that it identified a “content deployment" related to the issue and reverted the changes.

It's the Channel file "C-00000291*.sys" with timestamp of 0409 UTC that was the "problematic version".

From Australia and Germany to India and the US, a cascade of technical glitch triggered disruptions for businesses, from banks and airports to TV stations and hotels, according to a Wired report.

Mac and Linux hosts were not impacted.

Here's what we know about the downtime, reportedly arising from Crowdstrike bug:

Advertisement

Cause

Buggy update: News sources initially suggested a recent Crowdstrike update might be responsible. It turned out to be true. Crowdstrike did confirm this to be the culprit.

Impact: The outage likely affected businesses that rely on Crowdstrike's cybersecurity solutions, potentially hindering their antivirus protection, threat intelligence, or incident response capabilities.

Downtime duration: Information on the exact duration of the outage is unavailable at this time.

Official cause: Crowdstrike CEO George Kurtz confirmed on X that a “defect” was found in a "single content update" for Windows hosts.

Read more

Advertisement

Points to consider

  • The Falcon agent on a Windows machine was pushed out, though not sufficiently tested, thus causing the global outage. A post-mortem of the code would reveal the flaw, but this may take some time.
  • Due to its potential impact on businesses and the critical role Crowdstrike plays in cybersecurity, the outage received significant news coverage.
  • IT experts from the UAE Cybersecurity Council urged caution among all digital device users in order to avoid falling victim to hackers who may exploit this technical glitch.

What's 'Falcon sensor'?

A Falcon sensor, or "agent", is an endpoint detection and response platform that monitors the computers that it is installed on to detect intrusions like hacks and respond to them, University of Melbourne tech expert Toby Murray told Australian media.

Falcon is one of Crowdstrike software products that organisations install on their computers to keep them safe from cyber attacks and malware.

What Crowdstrike said

CrowdStrike CEO George Kurtz stated on X that the company is actively working with customers impacted by what he termed as a “defect” found in a single content update for Windows hosts.

“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”

Advertisement

“We further recommend organisations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilised to ensure the security and stability of CrowdStrike customers.

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.

Companies/organisations affected:

  • Airlines
  • Airports
  • Banks
  • Media
  • Utilities
  • Trading platforms
  • Fastfood chains

The following countries were reportedly affected:

Advertisement
  • Australia
  • Japan
  • India
  • Philippines
  • Spain
  • Germany
  • UK
  • US
  • New Zealand

Details

(As per Crowdstrike update 9:22am ET, July 19, 2024:)

  • Symptoms include hosts experiencing a bugcheck|blue screen error related to the Falcon Sensor.
  • Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
  • Windows hosts which are brought online after 0527 UTC will also not be impacted
  • Hosts running Windows 7/2008 R2 are not impacted
  • This issue is not impacting Mac- or Linux-based hosts
  • Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
  • Channel file "C-00000291*.sys" with timestamp of 0409 UTC is the problematic version.

Current action

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

Advertisement

If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround steps for individual hosts (what to do next):

  • Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
  • Boot Windows into Safe Mode or the Windows Recovery Environment
  • NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Boot the host normally.
  • Note: Bitlocker-encrypted hosts may require a recovery key.

It's important to stay updated on official announcements from Crowdstrike and Microsoft regarding the cause, resolution, and any recommendations for their clients.

Advertisement