Home / Latest News And Updates / Protecting Yourself From the Next Cyber CrowdStrike-Like Outage
31 Jul 2024
Written by Jack Bedell-Pearce, Director at Cloud2Me
Much has been written in the past few days about the CrowdStrike outage, from the major organisations and services that were affected, to what caused it and who was to blame. Very little though seems to have been written about those companies that were affected but recovered quickly. What lessons can accountancy practices learn from those organisations?
By way of quick summary, the global outage on the 18th July 2024 was caused by a cyber security firm called CrowdStrike pushing an automatic software update that crashed computers and servers running the Windows Operating System. Normally crashing software isn’t a big deal, but in this instance it caused affected machines to reboot into a Blue Screen of Death (BSOD), making the retracting of the software patch impossible. This incident underscores the critical importance of robust cybersecurity measures to protect digital assets and ensure quick recovery from such disruptions.
CrowdStrike were quick to acknowledge the problem and published a guide on how to manually fix the problem but these fixes required engineers to log into machines in person, a daunting task when for organisations with tens of thousands of computers around the world, such as airport departure terminals and critical power stations.
If you’re an accountant or tax professional, whether in practice or industry, and you need CPD training, marketing support or growth advice, book a demo today.
The reason so many people’s computer was affected at the same time was because the security settings on those systems were set to automatically download and install all updates from CrowdStrike. In many instances, this is a sensible approach to protect and keep organisations secure from zero-day vulnerabilities, however a more sensible approach is to have many layers of cyber security infrastructure in place (such as two factor authentication and enterprise grade firewalls) and take what is called a ‘Canary Deployment’ approach to software and security updates. Cloud security is a crucial element in modern cybersecurity strategies, providing rapid threat detection and remediation for applications, data, and user safety in cloud environments. Network security solutions can help in identifying and blocking potential threats, adding an essential layer of protection.
A canary deployment is a software deployment strategy that involves rolling out a new version of an application incrementally to a small subset of users before making it available to the entire user base on their computer. This approach helps to minimise the risk of introducing new bugs or issues by first exposing the new release to a limited audience, often referred to as a “canary group.” If the canary deployment is successful and no significant problems are detected, the new version can then be gradually rolled out to the rest of the users computer.
Small to medium sized accountancy practices that took this approach will have quickly noticed a major issue with the update from CrowdStrike and avoided the mass rollout and subsequent outage suffered by organisations that had auto-updates turned on as standard. It should be noted that for organisations with thousands of regular PCs, this approach may not be practical to implement, but even in companies like this, business critical servers (i.e. airport lounges or payment processing systems) should have their updates manually applied to avoid this kind of failure. Endpoint security is crucial in protecting devices from vulnerabilities, ensuring that end-user devices such as desktops, laptops, and mobile devices are safeguarded against cyber threats, so ensure endpoint security is a priority. Incorporating next gen antivirus as part of endpoint security strategies can significantly enhance network threat prevention.
While it may not be practical for all organisations to run virtual machines (sometimes also referred to as Desktop as a Service or Hosted Desktops), those that were running them had a significantly easier time. Disaster recovery strategies can help organizations quickly recover from incidents. While a virtual machine that had downloaded the CrowdStrike update will have also experienced a BSOD, administrators of such systems will have had the ability to simply restore the entire machine back to a state when it was working fine – in other words, just before the buggy update was installed.
Software platforms play a crucial role in providing solutions and support during such disruptions, ensuring that organizations can maintain continuity and recover swiftly.
Depending on how the Virtual Desktop Infrastructure (VDI) had been set up, this ‘last restore point’ can typically range from anything as soon as every 15 minutes to every two hours (with the latter being a bit more annoying for anyone who had just spent the preceding couple of hours on a complicated spreadsheet). Cloud computing solutions can enhance the flexibility and efficiency of virtual desktop infrastructures, making it easier to manage and recover from such incidents.
The time it takes for a whole virtual machine to rebuild will also vary according to how many resources VDI has at its disposal, but well configured systems can typically completely recover all virtual machines within a matter of a couple of hours at most.
It must be noted that in order to access virtual machines, you do still need thin clients (physical PCs, laptops, tablets etc), which could themselves be vulnerable to the same update error affecting the virtual machine. It’s for this reason, thin clients should be set up as bare bones machines (i.e. nothing other the operating system) and if you want to protect that machine it with anti-virus software, use the default technology that comes with the operating system (i.e. Microsoft Defender, Apple XProtect, Android Play Protect and Linux SELinux). Oh yes, and adopt a canary deployment approach to any updates!
Software vendors are increasingly providing cloud-hosted software and their collaboration during technical disruptions is vital for maintaining the interconnected tech ecosystem.
Discover our Tech-Talk service today which is specifically for accountants to help improve their working practices and firm technology. This includes focus groups, webinars, app stack templates and much more.
Unfortunately, the CrowdStrike incident wasn’t even the first of this type of outage. Back in 2010, another antivirus firm, McAfee, deployed an update with similar disastrous consequences. Eagle-eyed Anshel Sag, the Principal Analyst at Moor Insights & Strategy noted on ‘X’ that the then CTO during that incident, George Kurtz, happens to be the current day CEO of CrowdStrike.
So while this type of incident is rare with businesses, it’s not unique. What is more common though (and equally devastating to affected companies) is ransomware attacks and data breaches. Data breaches can have severe consequences for organizations, including financial loss and a loss of trust. The increasing sophistication of malicious emails, designed to mimic legitimate sources, necessitates effective cybersecurity measures to filter them out before they reach users. While good cyber security infrastructure will go some way towards protecting companies from this kind of attack, the quickest way to recover from a ransomware attack is to restore an entire virtual machine from a pre-defined restore point. Malicious software like ransomware can disrupt business operations significantly. Attackers can gain access to systems through various tactics, such as phishing and social engineering. If your company doesn’t already run virtual desktops, you can learn more about them here.
Securing mobile operating systems from threats such as rooting and jailbreaking is crucial. Employing Mobile Device Management (MDM) solutions ensures that only compliant devices can access corporate resources.
It is also important to secure other organizations that play crucial roles in society. Securing these entities is key to maintaining societal functions and mitigating vulnerabilities that can arise from trust relationships, particularly in the context of supply chain attacks.
If you’re interested in becoming a 20:20 Innovation member or would like to find out more about our products and services, please fill in the form below.
29 Jul 2024
The general election took place on 4 July, and since the new Labour government were elected, Prime Minister Ke...
24 Jul 2024
20:20 Innovation have confirmed they will have stands at Accountex Summit 2024 giving accountants and tax prof...
24 Jul 2024
Partner Blog
This article offers practice management advice to accountants regarding your office server. When should you up...
