TTCSIRT 430.19.07.24: Global Outage Triggered by Faulty CrowdStrike Cybersecurity Update

A large-scale outage has occurred due to a broken CrowdStrike cybersecurity update, rendering Windows computers unable to start and affecting much of the world’s infrastructure. The problem stems from an issue with CrowdStrike’s Falcon Sensors, which encountered problems following an early Friday morning update.

PLEASE BE ADVISED:
Only accept information from the CrowdStrike support portal, as phishing attempts have already been observed.

Symptoms: Hosts experience a blue screen error related to the Falcon Sensor.

Unaffected Systems:

• Windows hosts not impacted do not need action as the problematic file has been reverted.
• Windows hosts brought online after 0527 UTC are also not impacted.
• Windows 7/2008 R2 hosts are not impacted.
• Mac and Linux hosts are unaffected.

Problematic File: “C-00000291*.sys” with a timestamp of 0409 UTC.

Reverted File: “C-00000291*.sys” with a timestamp of 0527 UTC or later.

Current Action:

• CrowdStrike Engineering has identified the issue and reverted the changes.
• If hosts still crash and cannot stay online to receive the update, the following workaround steps can be used:

Workaround Steps for Individual Hosts:

1. Reboot the host to attempt downloading the reverted file. If the host crashes again:
2. Boot Windows into Safe Mode or Windows Recovery Environment.
   • Use a wired network and Safe Mode with Networking for better results.
3. Navigate to %WINDIR%\System32\drivers\CrowdStrike.
4. Locate and delete the file matching “C-00000291*.sys”.
5. Boot the host normally.
   • Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for Public Cloud or Virtual Environments:
Option 1:

1. Detach the operating system disk volume from the impacted virtual server.
2. Create a snapshot or backup of the disk volume.
3. Attach/mount the volume to a new virtual server.
4. Navigate to %WINDIR%\System32\drivers\CrowdStrike.
5. Locate and delete the file matching “C-00000291*.sys”.
6. Detach the volume from the new virtual server.
7. Reattach the fixed volume to the impacted virtual server.

Option 2:

• Roll back to a snapshot before 0409 UTC.

Reference: https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/

If you have any queries, comments or require assistance, please feel free to contact TT-CSIRT via contacts@ttcsirt.gov.tt

PLEASE BE ADVISED

Only accept information from the CrowdStrike support portal, as phishing attempts have already been observed.