The Crowdstrike-Microsoft incidents highlight the intricacies and interdependencies of today’s digital world. From aviation and banking to healthcare and emergency services, these incidents resulted in downtime and disruptions for organizations across industries, causing many to re-think their digital resilience strategies.
“The incident has highlighted the need for improved risk management practices. Enhanced due diligence, rigorous testing of updates, and phased rollouts are now critical.”
CrowdStrike Incident Spurs CIOs To Reassess Cybersecurity
This digital evolution path is familiar. An incident occurs, analysis follows, and new best practices emerge. Perhaps uniquely, the Crowdstrike-Microsoft situation bridges what was already previously known and understood as best practices, albeit in different areas of technology application and software release management. That is, in traditional IT security, the order of priorities is confidentiality, integrity, and availability (CIA). In the traditional OT security realm, the order is control, availability, integrity, and confidentiality (CAIC). Notably, the more mission-critical the system, the higher the availability and control rank.
As such, Crowdstrike and Microsoft highlight the need to reassess previous priorities and assumptions. Organizations need to examine their digital strategy and assess their resilience.
Today’s connected devices need an update process that is not just “like the desktop you know.” Devices, systems, and processes are more integrated, automated, smart, and connected. With those advancements come complexities. Complexity requires more granular and controlled management. There are two critical challenges unique for managing connected devices:
The update process is fundamental to any system and the root of the Crowdstrike and Microsoft incidents. Technologically advanced software, hardware, and embedded systems require updates throughout their lifecycle. The update process serves as the spine of a connected device; from it, all other components function together. Without a robust update process, issues can happen, and parts can break down.
Therefore, the first element every digital organization needs to examine is its current update process and the level of robustness, control, and granularity within.
The quality control (QC)/quality assurance (QA) and update process are by nature intertwined. One must rely on the other. However, a critical component in a robust update process are the failsafe mechanisms built to safeguard against, mitigate, or minimize errors earlier in the pipeline. However, as in the Crowdstrike update process, if these functions are not used or bypassed – such as automatic updates or fleetwide deployment – there is little recall available if an error does occur in QC/QA.
A/B updates are an integral building block for a robust update mechanism. The core idea is to have two versions of the main operating system: the currently active one and an inactive one. These two versions are usually implemented as storage partitions within the system or device.
As the name suggests, the active version, version A, is the one currently in operation and running. It is in a known state – operating as needed and expected.
The inactive version, B, serves two purposes.
Thus, establishing an A/B update process ensures the device or system remains functional before, during, and after the update process. A/B updates prevent the device from being ‘bricked’ or malfunctioning.
An often overlooked but important detail is that detection and eventual rollback, as exemplified in the update-aware bootloader above, must be independent of the actual operating system. It must function whether the operating system does or not.
If the detection and rollback functionality is only able to handle problems once the main operating system has started, then it can only tackle application-level updates. Dependent detection and rollback functions are not suitable for system updates.
If the detection and rollback functionality is independent, it can address system update issues and operate as needed in an A/B update scenario.
While the two items above are technical measures that apply at the device level, phased rollouts are management or process elements. Phased rollouts – staged or staggered rollouts, or sometimes canary deployments – occur at the fleet level. These can also be manual or human-controlled in scenarios where the update infrastructure does not support phased rollout out of the box (i.e., deploying to only a select group).
The concept is very straightforward. Instead of sending an update to the whole device fleet, potentially thousands, all at once and immediately, the update is initially deployed to only a small fraction, like 5% of the devices. Only if no substantial problems are detected during a grace period is the update also deployed to the remainder of the device fleet.
While none of the above three elements are novel, their importance can vary significantly with the organization, device, usage or application, and fleet size. But in light of the Crowdstrike and Microsoft incidents, perhaps it is time to rethink the security and resiliency priorities.
As the world further embeds technology into every facet of our societal operations, it begs whether there should be two distinct disciplines: IT and OT, non-critical and critical infrastructure. Or does such a technological-dependent state exist where at least some technology is critical infrastructure, even if it is not traditionally classified as such?
Regardless, Crowdstrike and Microsoft exemplify the need for organizations to once again evolve their digital strategies beyond cybersecurity and privacy basics to embrace digital resilience. Whether IT or OT, hardware or software, updating the systems society relies on is the backbone of a digital strategy. It must be robust, granular, and fail-safe.
“...when it comes to mission-critical systems where failure is unacceptable, you need to exercise extreme caution with any upgrade...there are many ways to do a staged rollout. They include rolling updates, blue/green, canary, and A/B testing. Pick one. Make it work for your enterprise, just don’t put all your upgrades into one massive basket. Besides, robust rollback procedures are essential to revert to a stable version if problems arise quickly. Wouldn’t you have liked just to hit a button and roll back to working systems? Tens of thousands of IT staffers must wish for that now.”
7 Urgent Lessons From the CrowdStrike Disaster