The Real Lesson from the AWS Outage
The embarrassing outage of Amazon Web Services this week should open our eyes to a growing problem. Complex systems are difficult to manage, but if they are connected in dependent ways, a fragile result emerges. Such structures are subject to unexpected malfunctions which can sprawl quickly. One of the most knowledgeable technology companies on the planet learned just such a lesson this week. Amazon’s star-child, their cloud services, had a major disruption. It was not a nation-state attack, sophisticated teams of cyber-hackers, or even malicious insiders bent on destruction. Nonetheless, the lessons are telling. The ramifications of which will be important to all of us.
It was one employee, typing a few wrong codes, that caused a significant outage to major portions of the Internet. Amazon worked furiously to contain and recover from the incident. It will have to rebuild trust with customers whom were sold on the resiliency of ‘cloud’ services to avoid such events. Amazon has already stated they will learn from the event and will apply some compartmentalization controls to lessen potential damage in the future. But there is a more significant realization to be made.
The greater lesson for us all is that when hugely sophisticated systems interconnect with each other, there is an exponential increase in complexity. Due to reliance, authority, and trust, these structures can fail in spectacular fashion. The AWS example show how such a situation allows a series of cascading unintended effects, that cannot easily have been predicted, to occur and cause widespread impacts. As bad as it may have appeared, it was not too severe. If it were an intentional attack from a capable, motivated, and sophisticated attacker, I believe the results would have been catastrophic.
With the AWS outage we can see the impact of an unintentional accident and the difficulty to recover when everyone is working together to resolve the issue. Now imagine what a malicious and focused cyber-threat could do while being stealthy, striving for maximum damage, and actively undermining countermeasures and recovery actions of response teams.
If this were a malicious insider or professional hack, the damage would be a thousand times worse. We would still be picking up the shattered pieces. There would be tears falling from the AWS cloud.
This week it was cloud storage services making websites unavailable. What happens when it is a fleet of autonomous vehicles which put lives at risk or the complex national power grid infrastructure?
We must take a fresh look at understanding threats, risks, countermeasures, and protection practices as individual pieces of the computing world are growing much more complex and being connected. Traditional methods are not sufficient in understanding how chain reactions can occur in the next generation of new technologies and services.
Interested in more? Follow me on Twitter (@Matt_Rosenquist), Steemit, and LinkedIn to hear insights and what is going on in cybersecurity.


Great article, but I felt you did not go far enough into alternatives or what "taking a fresh look" could be.
Could you expand?
Wow you read fast! I just posted this a few minutes ago!
Yes, I agree I did not go into the new aspect the industry should be considering and how risk assessments of highly complex systems need to adapt. It would have turned this blog into a full position paper. Not what I wanted as it relates to AWS event. But I am working on a more strategic position which I will eventually share.
Thanks for reading and commenting!
I just happened to be looking at the time! 🙂
Fair enough, I look forward to reading it. It certain is a weak point, and it's the opposite of the redundancy the Internet was suppose to tolerate in nodes!
...well, if you are interested in those topics, then you will really like what I am working on right now. ...at 10 year forecast for the major paradigm shifts for cybersecurity as it relates to technology. Been conducting industry interviews working on the analysis for a few months now. Ha!
..although I am still working the approval processes to release a redacted version to the public, I hope to include some aspects in upcoming speaking engagements (CISO Executive Summit in San Francisco, CSE @ISC West in Las Vegas, and the PSC TEC Conference in Denver)
YES! Very nice, can't wait 😆 👍
I work at a very large state University. Our new CIO is pushing cloud above all, and I just don't see how it's a money saver when we already have built out infrastructure for decades. Don't get me wrong, distributed environments are a safe bet, but forcing all new campus initiatives into the cloud is going to be a recipe for disaster. That's what happens when you get a CIO that treats IT as a commodity instead of a service.
Cloud services can make financial, oversight, stability, and scalability sense for enviromments that are not static. This has been the great lure for many organizations. But as always there are risks and situations where other solution architectures are superior.
I think it's great for dev environments and issues like the last AWS outage can be mitigated by spreading duplication across vendors. rsync works just fine between a vm at AWS and another at Rackspace or in the office for your production systems. I think broadly declaring All new stuff goes to the cloud is kind of bogus. For example, we have a box onsite that has 24 invida gpus in it that students can use for GPU intensive sims and analysis. Invidia just gives us the cards for free, every two years. AWS is not about to give us that kind of compute power on the level this onsite box does for less than 60k. Well Invidia just sent us all new cards($1800 a pop), and for us to use them, we need a new chassis. Chassis is about 10k and we were told NO. I wonder who gets the kickback, or if I can throw a card in my workstation...