Crowdstrike BSOD: How to identify and remediate impacted assets

Summary

At 2024-07-19, CrowdStrike published an update to Falcon sensor agents that caused Windows hosts — updated from 04:09 to 05:27 UTC — to crash.

CrowdStrike reverted the update at 05:27 UTC, but affected hosts — still crashing and unable to stay online to receive the 05:27 UTC update — require manual workaround to fix the issue.

CrowdStrike is actively addressing the issue and providing details in their CrowdStrike's Statement on Falcon Content Update for Windows Hosts.

Symptom

Affected hosts will experience blue screen crashes and cannot reboot to receive the latest update from CrowdStrike that fixes the issue.

Impacted Systems

For updates, visit the CrowdStrike article.

Steps to identify affected hosts with Axonius

Note: This will ensure that you get the latest relevant snapshot of your environment.

To do so, access the Axonius Platform as an admin and click Discover Now (located in the top-right hand corner).

Alternatively, make sure that your last discovery ran after 2024-07-19 05:27 UTC:

• In the Axonius Dashboard, click Activity Logs.
• Expand the first combo box (Users) and click Select All.
• In the second combo box (Actions), select Discovery Ended.
• Confirm that the latest discovery cycle is after 2024-07-19 05:27 UTC.

2.  Identify potentially impacted hosts.

Potentially affected devices

Potentially affected devices with hard drive encryption

Note: This query is a subset of the query above and identifies potentially affected hosts with encrypted hard drives. These devices may use Bitlocker encryption, which requires a recovery key for the appropriate fix.

From the results, you can export a CSV or leverage your existing automations in the Enforcement Center to mobilize your IT team.

Remediation steps

For updates, visit the CrowdStrike article.

For individual hosts:

• Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
  • Boot Windows into Safe Mode or the Windows Recovery Environment.
    NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Boot the host normally.
    NOTE: Bitlocker-encrypted hosts may require a recovery key.

• Detach the operating system disk volume from the impacted virtual server.
• Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes.
• Attach/mount the volume to a new virtual server.
• Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
• Locate the file matching “C-00000291*.sys”, and delete it.
• Detach the volume from the new virtual server.
• Reattach the fixed volume to the impacted virtual server.

Option 2:

• Roll back to a snapshot before 0409 UTC.

Related documentation

From CrowdStrike:

• Main Article: CrowdStrike's Statement on Falcon Content Update for Windows Hosts.
• AWS-specific documentation:
  • To attach an EBS volume to an instance
  • Detach an Amazon EBS volume from an instance
• Azure environments:
  • Please see this Microsoft article
• Bitlocker recovery-related KBs:
  • BitLocker recovery in Microsoft Azure
  • BitLocker recovery in Microsoft environments using SCCM
  • BitLocker recovery in Microsoft environments using Active Directory and GPOs
  • BitLocker recovery in Microsoft environments using Ivanti Endpoint Manager

• System Queries (Creating Queries Using Filters)
• Enforcement Center Overview
• Crowdstrike Falcon Adapter reference