One of the truths of the world is that the internet is a dangerous place. Even things that are supposed to be safe and easy can turn into unmitigated disasters. For example, when an update for a network management system is compromised by hackers, adding in a backdoor that allows those hackers to enter the systems that download those updates. That is exactly what happened several months ago when an update for network management software developed by SolarWinds was compromised and distributed to tens or possibly hundreds of thousands of networks.
The compromise was made by the innocuous-sounding hacker collective Cozy Bear, a Russian state-sponsored organization, and revealed by FireEye. The revelation is a huge problem, as users of SolarWinds range from the likes of Microsoft to the US Department of Defense. The government has recognized a "significant and ongoing hacking campaign", the scope of which is unknown.
There are a number of high-impact aspects of the hack. The first and most important is that it could take decades to unravel the details of the hack and what data might have been compromised. Currently, what is known is that the malware gives the hackers a broad reach into the infected systems. As the total scope within the government's systems is unknown, the DOD will need to operate under the premise that the Russian government knows anything and everything - creating a national security disaster.
On the second front, the depth of the SolarWinds software within networks could mean that the servers that have been infected could be unsalvageable. There's talk that any server that has been infected by the malware might need to be replaced, at a great cost to the government and the thousands of other clients of SolarWinds.
This hack brings back to the forefront a few tenents of IT security that have been lax or entirely ignored over the past few years. Unproven software, which SolarWinds product is, should never be installed on mission-critical systems. A company like SolarWinds needs years of successful track record before it can be trusted on major networks. The second is that updates should not be installed until IT has tested them as safe, both in terms of security and compatibility. Third, large central systems are never a good idea. Cloud systems have been showing us this vulnerability with AWS outages taking down everything from websites to Netflix, but even internal central systems create a bottleneck that can destroy an organization.