Skip to main content

CrowdStrike update crashes Windows systems, causes outages worldwide


Jasper_The_Rasper
Moderator
Forum|alt.badge.img+54

July 19, 2024 By Ionut Ilascu

 

CrowdStrike update crashes Windows systems, causes outages worldwide

A faulty component in the latest CrowdStrike Falcon update is crashing Windows systems, impacting various organizations and services across the world, including airports, TV stations, and hospitals.

The glitch is affecting Windows workstations and servers, with users reporting massive outages that took offline entire companies and fleets of hundreds of thousands of computers.

According to some reports, emergency services in the U.S. and Canada have also been impacted.

Workaround for CrowdStrike glitched update

For the past few hours, users have been complaining about Windows hosts being stuck in a boot loop or showing the Blue Screen of Death (BSOD) after installing the latest update for CrowdStrike Falcon Sensor.

The security vendor acknowledged the issue and published a technical alert explaining that its engineers “identified a content deployment related to this issue and reverted those changes.”

“Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor,” CrowdStrike says in the tech alert.

The company revealed that the culprit is a Channel File, which contains data for the sensor (e.g. Instructions). Since it is just a component of the update for the sensor, this type of file can be addressed individually without removing the Falcon Sensor update.

For those already affected, CrowdStrike provides the following workaround steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it.
  4. Boot the host normally.

 

>>Full Article<<
 

26 replies

TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 19, 2024

RIP to fellow crowdstrike peeps. I don’t wish that on anyone, even a competitor 


TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 19, 2024

 


TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 19, 2024

 


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024

I hope that never happens to AWS!


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024

Updated July 19, 2024, 10:12 AM EDT

 

Live updates: Major global IT outage grounds flights, hits banks and businesses around the world

The problem was caused by a “defect found in a single content update for Windows” from the cybersecurity company CrowdStrike.
 

TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024

Statement on Falcon Content Update for Windows Hosts

bird

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack.

The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.

 

https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024
TylerM wrote:

 

I love number 5!

 

 


ProTruckDriver
Moderator

No worries here on my well tuned Mac!

“An Apple A Day Will Keep Microsoft Away”

 


TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 19, 2024

@ProTruckDriver yeah baby you might as well enjoy it and ride it into the weekend sunset today

 


Jasper_The_Rasper
Moderator
Forum|alt.badge.img+54
TylerM wrote:

RIP to fellow crowdstrike peeps. I don’t wish that on anyone, even a competitor 

I absolutely agree, it is a total PR disaster amongst many other things and it is going to take a long time to recover from and that is even after the world-wide systems are patched and recovered.
It just shows though how much damage can be done just by 1 bad update.


Jasper_The_Rasper
Moderator
Forum|alt.badge.img+54

UPDATE

CrowdStrike fixes start at “reboot up to 15 times” and get more complex from there

 

Admins can also restore backups or manually delete CrowdStrike's buggy driver.

ANDREW CUNNINGHAM - 7/19/2024

 

If rebooting doesn’t work

If rebooting multiple times isn't fixing your problem, Microsoft recommends restoring your systems using a backup from before 4:09 UTC on July 18 (just after midnight on Friday, Eastern time), when CrowdStrike began pushing out the buggy update. Crowdstrike says a reverted version of the file was deployed at 5:27 UTC.

If these simpler fixes don't work, you may need to boot your machines into Safe Mode so you can manually delete the file that's causing the BSOD errors. For virtual machines, Microsoft recommends attaching the virtual disk to a known-working repair VM so the file can be deleted, then reattaching the virtual disk to its original VM.

The file in question is a CrowdStrike driver located at Windows/System32/Drivers/CrowdStrike/C-00000291*.sys. Once it's gone, the machine should boot normally and grab a non-broken version of the driver.

Deleting that file on each and every one of your affected systems individually is time-consuming enough, but it's even more time-consuming for customers using Microsoft's BitLocker drive encryption to protect data at rest. Before you can delete the file on those systems, you'll need the recovery key that unlocks those encrypted disks and makes them readable (normally, this process is invisible, because the system can just read the key stored in a physical or virtual TPM module).

This can cause problems for admins who aren't using key management to store their recovery keys, since (by design!) you can't access a drive without its recovery key. If you don't have that key, Cryptography and infrastructure engineer Tony Arcieri on Mastodon compared this to a "self-inflicted ransomware attack," where an attacker encrypts the disks on your systems and withholds the key until they get paid.

 

>>Full Article<<


TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 19, 2024

🤐


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024
TylerM wrote:

🤐

 


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 19, 2024
Alert

Widespread IT Outage Due to CrowdStrike Update

Release Date
July 19, 2024
 

Note: CISA will update this Alert with more information as it becomes available.

As of 1130am EDT July 19, 2024: 

CISA is aware of the widespread outage affecting Microsoft Windows hosts due to an issue with a recent CrowdStrike update and is working closely with Crowdstrike and federal, state, local, tribal and territorial (SLTT) partners, as well as critical infrastructure and international partners to assess impacts and support remediation efforts. CrowdStrike has confirmed the outage:

  • Impacts Windows 10 and later systems.
  • Does not impact Mac and Linux hosts.
  • Is due to the CrowdStrike Falcon content update and not to malicious cyber activity.

According to CrowdStrike, the issue has been identified, isolated and a fix has been deployed. CrowdStrike customer organizations should reference CrowdStrike guidance and their customer portal to resolve the issue.

Of note, CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources. CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links.

 

https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 20, 2024

Note: CISA will update this Alert with more information as it becomes available.

Update 7:30 p.m., EDT, July 19, 2024: 

 

Threat actors continue to use the widespread IT outage for phishing and other malicious activity. CISA urges organizations to ensure they have robust cybersecurity measures to protect their users, assets, and data against this activity.

 

Of note, CISA has observed threat actors taking advantage of this incident for phishing and other malicious activity. CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources. CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links.

 

https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update


ProTruckDriver
Moderator

What a mess!

 


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 20, 2024
ProTruckDriver wrote:

What a mess!

 

Agreed because now Threat actors continue to use the widespread IT outage for phishing and other malicious activity.


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 20, 2024

 

 

In 2010 McAfee caused a global IT meltdown due to a faulty upgrade.
The CTO at the time was George Kurtz. He is currently the CEO of Crowdstrike.

 

https://www.independent.co.uk/tech/crowdstrike-george-kurtz-microsoft-outages-b2582962.html


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 20, 2024

Be alert for phishing and malware, warns MyCERT after IT outage

-20 Jul 2024, 05:16 PM

The Malaysia Computer Emergency Response Team says scammers are exploiting yesterday’s CrowdStrike incident.

 

 

KUALA LUMPUR:

The Malaysia Computer Emergency Response Team (MyCERT) has issued a critical alert following reports of increased phishing attacks leveraging yesterday’s CrowdStrike incident.

 

These attacks use fraudulent domains, command-and-control (C2) internet protocol (IP) addresses and malware binaries to compromise systems and steal sensitive information, said MyCERT on its website today.

It said the phishing domains mimic legitimate websites, deceiving users into unknowingly installing malware or disclosing personal credentials.

Meanwhile, attackers use C2 servers to maintain control over compromised devices and extract sensitive data.

Additionally, malicious software, delivered through websites or emails, exacerbate the threat landscape by executing harmful actions on infected devices, MyCERT noted.

The agency has provided 30 potential indicators of compromise (IoC), including their value, type and additional information, available on its website.

 

https://www.freemalaysiatoday.com/category/nation/2024/07/20/be-alert-for-phishing-and-malware-warns-mycert/


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 20, 2024

Note: CISA will update this Alert with more information as it becomes available.

Update 12:30 p.m., EDT, July 20, 2024: 

 

Cyber threat actors continue to leverage the outage to conduct malicious activity, including phishing attempts. CISA continues to work closely with CrowdStrike and other private sector and government partners to actively monitor any emerging malicious activity.

  • According to a new CrowdStrike blog, threat actors have been distributing a malicious ZIP archive file. This activity appears to be targeting Latin America-based CrowdStrike customers. The blog provides indicators of compromise and recommendations.

https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 21, 2024

Note: CISA will update this Alert with more information as it becomes available.

Update 9:45 a.m., EDT, July 21, 2024: 

  • Microsoft released a recovery tool that uses a USB drive to boot and repair affected systems. 
  • Microsoft also published a blog post that provides links to various remediation solutions and outlines their actions in response to the outage, which include working with CrowdStrike to expedite restoring services to disrupted systems.
  • In the blog post, Microsoft estimates the outage affected 8.5 million Windows devices. Microsoft notes that this number makes up less than one percent of all Windows machines.

https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update


russell.harris
Popular Voice
Forum|alt.badge.img+5

It’s not a surprise this happened though and certainly a warning of what chaos evil doers could cause!


Jasper_The_Rasper
Moderator
Forum|alt.badge.img+54

CrowdStrike: 'Content Validator' bug let faulty update pass checks

 

July 24, 2024 By Bill Toulas 

 

Crowdstike

CrowdStrike released a Preliminary Post Incident Review (PIR) on the faulty Falcon update explaining that a bug allowed bad data to pass its Content Validator and cause millions of Windows systems to crash on July 19, 2024.

The cybersecurity company explained that the issue was caused by a problematic content configuration update meant to gather telemetry on new threat techniques.

After passing the Content Validator, the update didn't go through additional verifications due to trust in previous successful deployments of the underlying Inter-Process Communication (IPC) Template Type. Therefore, it wasn't caught before it reached online hosts running Falcon version 7.11 and later.

The company realized the error and reverted the update within an hour.

However, by then, it was too late. Approximately 8.5 million Windows systems, if not more, suffered an out-of-bounds memory read and crashed when the Content Interpreter processed the new configuration update.

 

>>Full Article<<


TripleHelix
Moderator
Forum|alt.badge.img+63
  • Moderator
  • 9081 replies
  • July 25, 2024
Last Revised
July 24, 2024
 

Note: CISA will update this Alert with more information as it becomes available.

Update 12:00 p.m., EDT, July 24, 2024: 

  • CrowdStrike continues to provide updates to its guidance, including:

An update to their initial remediation that accelerates remediation of impacted systems; CrowdStrike encourages customers to “follow the Tech Alerts for latest updates as they happen.”

 

A “Preliminary Incident Review,” which provides answers to why and how the outage occurred and how they will prevent such outages going forward.

 

CrowdStrike also published a list of domains impersonating the CrowdStrike brand, which threat actors could use to deliver malicious content. 

 

https://www.cisa.gov/news-events/alerts/2024/07/19/widespread-it-outage-due-crowdstrike-update


TylerM
Administrator
Forum|alt.badge.img+25
  • Sr. Security Analyst & Community Manager
  • 1272 replies
  • July 25, 2024
Jasper_The_Rasper wrote:

 

However, by then, it was too late. Approximately 8.5 million Windows systems, if not more, suffered an out-of-bounds memory read and crashed when the Content Interpreter processed the new configuration update.

 

>>Full Article<<

Lol they forgot to add a couple more zeros to that number