On December 7, AWS us-east-1 went down and took a significant chunk of the internet with it. Disney+, Slack, parts of the AWS console itself. My phone was blowing up within minutes.
I’m not going to pretend this was a surprise. We’ve known us-east-1 is a single point of failure for years. AWS has had major us-east-1 incidents before. The 2017 S3 outage. The 2020 Kinesis incident. It keeps happening because us-east-1 is the default region for everything, and “we’ll do multi-region later” is the most common lie in cloud architecture.
What annoys me isn’t that AWS had an outage. Large distributed systems fail. That’s physics. What annoys me is that the same lessons keep getting ignored.
The Control Plane Took Everything Down
Here is what actually happened. The networking layer in us-east-1 experienced issues that cascaded into the AWS control plane – the internal APIs that manage EC2 instances, ECS tasks, Lambda functions, and basically everything else. Your running workloads were mostly fine. But you couldn’t scale them. You couldn’t deploy new ones. You couldn’t see what was happening because CloudWatch was degraded too.
This is the insidious part. Your application was running but you had no ability to manage it. For hours. If something else had gone wrong during that window – a pod crash, a traffic spike – you couldn’t respond. Auto-scaling was broken. Deployments were broken. Even the AWS status page was slow to update because it runs in… you guessed it… us-east-1.
The Default Region Problem
us-east-1 is the default region in the AWS console. It’s the default in most SDKs. It’s where most people create their first resources. It’s where third-party SaaS tools point by default. Half the AWS documentation examples use it.
This creates a concentration risk that nobody planned for. Your application runs in us-west-2. Great. But your CI/CD pipeline stores secrets in us-east-1. Your monitoring backend defaults to us-east-1. That vendor tool you integrated last quarter? us-east-1.
One region goes down and your entire operational surface breaks even though your “production” is somewhere else.
What You Should Have Done (And Still Can)
Stop putting AWS API calls on the request path. If your application calls the EC2 metadata service, Parameter Store, or Secrets Manager on every request, you’re coupling your availability to the control plane. Cache those values. Refresh them periodically. If the refresh fails, use the cached value and alert. Your service should survive hours without a working AWS control plane.
Run external monitoring. If your monitoring runs in the same region as your application, it fails at exactly the moment you need it most. Use an external synthetic monitoring service. Run health checks from outside AWS. Use a status page hosted on completely independent infrastructure.
I’m amazed how many organizations I consult for have their monitoring in the same blast radius as their applications. Your Grafana is on an EC2 instance in the same region as the thing it monitors. Come on.
Have a multi-region story, even a simple one. You don’t need active-active. Active-passive is fine. What you need is a documented plan for “if us-east-1 is gone for 6 hours, what do we do?” with actual tested runbooks.
Active-active multi-region gives you the best continuity but it’s expensive and complex. Most teams I work with don’t need it. What they need is:
- Data replication to a secondary region
- Deployment artifacts stored in at least two regions
- DNS-based failover with tested cutover procedures
- A quarterly drill where they actually fail over and confirm it works
If you’ve never tested your failover, you don’t have failover. You have a document.
Audit your defaults. Go through every SaaS integration, every CI/CD tool, every internal service and check what region it’s configured for. You’ll find us-east-1 in places you didn’t expect. Move what you can. Document what you can’t.
The Honest Assessment
Multi-region is hard. It’s expensive. It adds operational complexity. I understand why teams defer it. But “we accept the risk of us-east-1 going down” is a different decision than “we never thought about it.” The second one is what I see at most organizations.
After every major AWS outage, there’s a week of panic, a few architecture diagrams get drawn, and then everyone goes back to shipping features. I know because I’m the one who gets called during the outage, and I’m the one whose multi-region proposal gets deprioritized three weeks later when the pain fades.
This will happen again. us-east-1 will have another bad day. The question isn’t whether. It’s whether you’ll be scrambling again or whether you’ll have a tested playbook.
The December 7 outage wasn’t an AWS problem. It was a reminder that the cloud doesn’t remove operational responsibility. It changes the shape of it. Your availability is still your problem. Your regional strategy is still your problem. Your ability to operate when the control plane is down is still your problem.
AWS will publish a post-incident review. It will be informative and thorough. And the next outage will still catch most of you off guard.