If you haven’t read about SolarWinds yet, stop everything and go read the FireEye disclosure. I’ll wait. Because what happened last week is one of the most significant cyber operations I’ve seen since my time working NATO cyber defense. And I don’t say that lightly.
Someone – almost certainly a nation-state actor – compromised SolarWinds’ build infrastructure. Not their website. Not a phishing campaign against their customers. Their actual build pipeline. They inserted malicious code into the Orion platform update, SolarWinds signed it with their own keys, and then 18,000 organizations installed it voluntarily. Through normal patching. The thing your security team begs you to do.
Let that sink in. The update process itself was the attack vector.
The timeline so far
This is still unfolding as I write this on December 14th:
- Sometime earlier in 2020, attackers got into SolarWinds’ build environment and injected a backdoor into the Orion update.
- Compromised updates shipped to customers throughout the year. Months of dwell time. Months.
- December 8th: FireEye publicly discloses they were breached and traces it back to a supply-chain compromise.
- December 13th: SolarWinds confirms the Orion update was compromised.
The customer list for SolarWinds Orion reads like a who’s who. Fortune 500 companies. Multiple U.S. government agencies. Defense contractors. We don’t know the full scope yet. We won’t for weeks. Maybe months.
Why this breaks everything you thought about security
I spent years in NATO cyber defense. We built threat models. We planned for adversaries targeting our networks, our endpoints, our people. But the fundamental assumption in most of those models? That signed software from a trusted vendor is clean. That your update channel isn’t the enemy.
SolarWinds just shattered that assumption for every organization on the planet.
Here’s the thing that keeps me up tonight: code signing didn’t help. In fact, it made things worse. The malicious update was properly signed by SolarWinds. Your EDR saw it arrive. Your security team saw a legitimate vendor update. Green checkmarks everywhere. Meanwhile, the backdoor was already phoning home.
Traditional perimeter defense? Useless. You invited the attacker in through the front door and handed them a badge.
What this means for every software team
This isn’t just a SolarWinds problem. If you build software – any software – your build pipeline is now a tier-one attack surface. Full stop.
Think about your own CI/CD for a minute. Who has access to your build servers? Are they segmented from the rest of your network? Do you have integrity checks on your build artifacts beyond “it compiled successfully”? Can you reproduce a build from source and get the same binary?
Most teams I consult with can’t answer yes to any of these. That’s terrifying.
Your build system deserves the same paranoia you apply to production databases. Maybe more. A compromised database leaks data. A compromised build pipeline lets an attacker ship whatever they want to all your customers, signed with your name.
If you run SolarWinds Orion
Stop reading my opinions and go do these things:
- Find every Orion instance. Every single one. The ones you forgot about in the lab environment count too. Check service accounts. Check any system that touches those servers.
- Isolate first, investigate second. Don’t wipe anything yet. You need the forensic evidence. Pull the network cables (metaphorically – VLAN isolation works) and start preserving logs.
- Assume credential compromise. Any credential that Orion could touch? Reset it. Domain admin accounts that Orion’s service account had access to? Reset those too. Assume the worst.
- Watch for lateral movement. The initial backdoor is just the beginning. Sophisticated actors establish multiple persistence mechanisms. The Orion compromise was the door. They’ve had months to furnish the house.
- Get help. This isn’t the time for pride. Engage your IR retainer. Call CISA. Coordinate with your legal team because disclosure obligations are going to be a mess.
The bigger lesson: trust is a vulnerability
We built modern software deployment on trust chains. I trust my package manager. My package manager trusts the maintainer. The maintainer trusts their CI system. The CI system trusts the dependencies it pulls. Every link in that chain is a potential SolarWinds.
I’ve been saying this for years: your software supply chain is your weakest link. Nobody wanted to hear it because fixing it’s hard and expensive and slows things down. Well. Here we’re.
What actually helps:
- Treat build infrastructure like crown jewels. Separate networks. Restricted access. Full audit logging. The same controls you put on your production database.
- Reproducible builds. If you can rebuild from source and get a different binary, you can’t verify integrity. Period.
- Egress filtering. Your build servers have no business talking to arbitrary internet hosts. Lock it down.
- Software bill of materials. You need to know exactly what’s in your software. Not approximately. Exactly. Every dependency, every transitive dependency, every build tool.
- Behavioral detection over signatures. A signed binary doing DNS lookups to avsvmcloud[.]com is still suspicious. Stop relying only on known-bad indicators.
What I expect happens next
The investigation will expand. More victims will be identified. There will be congressional hearings. There will be a lot of finger-pointing at SolarWinds. Some of it deserved.
But the real question every CTO should be asking right now isn’t “are we a SolarWinds customer?” It’s “could this happen to us as a vendor?” Because the attackers didn’t pick SolarWinds because they were uniquely careless. They picked SolarWinds because they were ubiquitous. Any widely-deployed software with a build pipeline connected to the internet is a potential target.
We’re going to see more of these. This isn’t the last supply-chain attack. It’s the one that finally made people pay attention.
Secure your build pipelines. Verify your dependencies. And maybe reconsider that automatic update policy you’ve been running on faith.
I’ll write more as the situation develops. For now: assume breach, verify everything, trust nothing.