CrowdStrike has promised to improve how it tests software after its faulty content update for Windows systems caused a mass global IT outage on Friday.
The cybersecurity company’s mistake resulted in problems for banks, hospitals and airlines as millions of PCs displayed “blue screens of death”.
In a detailed review of the incident published on Wednesday, CrowdStrike said the problem occurred due to a “bug” in the system which was meant to check software updates were working properly.
The glitch meant its system did not identify “problematic content data” in a file.
The company said it could prevent the incident from happening again with better software testing and checks, including more scrutiny from developers.
The faulty update crashed 8.5 million Microsoft Windows computers around the world and George Kurtz, Crowdstrike’s boss, has apologised for the impact of the outage.
But cybersecurity experts told BBC News that the review revealed “major mistakes” were made by the firm.
“What’s clear from the post mortem is they didn’t seem to have the right guardrails in place to prevent this type of incident or to reduce the risk of it occurring,” said cyber-security consultant Daniel Card.
His thoughts were echoed by cybersecurity researcher Kevin Beaumont, who said the key lesson from CrowdStrike’s review was that the firm doesn’t “test in waves”.
“They just deploy to all customers at once in a so called ‘rapid response update’ which was obviously a huge mistake,” he said.
But Sam Kirkman from cybersecurity firm NetSPI told the BBC the review showed CrowdStrike “took steps” to prevent the outages.
He said these steps “have likely been effective to prevent incidents on countless occasions prior to last week”.
Congress calls
According to insurance firm Parametrix, the top 500 US companies by revenue, excluding Microsoft, had faced some $5.4bn (£4.1bn) in financial losses from the outage.
It said that only $540m (£418m) to $1.08bn (£840m) of these losses were insured.
And the US government has opened an investigation into Delta Airlines‘ handling of the outage after it continued to cancel hundreds of flights.
Delta chief executive Ed Bastian said in a letter to customers on Wednesday that “the worst impacts of the CrowdStrike-caused outage are clearly behind us” and it expects the airline to make a full recovery on Thursday.
Meanwhile, Mr Kurtz has been called to testify in front of Congress about the outage.
“This incident must serve as a broader warning about the national security risks associated with network dependency,” wrote the House Committee on Homeland Security.
It has given the cybersecurity company until Wednesday evening to respond by scheduling a hearing.
Additional reporting by Joe Tidy