It was a Tuesday in February, around 11 PM. I was finishing a late sprint at our shared mobility startup when our monitoring lit up. Unusual outbound traffic from one of our API servers. Not a spike in user traffic. Not a deploy gone wrong. Something else entirely.
I pulled up the logs and saw authenticated requests hitting an internal endpoint we had deprecated months ago but never torn down. The requests were coming from a session token that belonged to an engineer who had left the company six weeks earlier. His credentials should have been revoked. They weren’t. Somewhere between the offboarding checklist and our actual IAM setup, we had a gap.
The next four hours taught me more about incident response than any exercise or drill ever could. Not because the technical problem was hard. It wasn’t. The deprecated endpoint was read-only, the data it exposed was limited, and the token hadn’t been used for anything beyond what looked like automated probing. But the organizational response was chaos. Nobody knew who was supposed to make decisions. Our CEO was calling me every ten minutes asking for updates I didn’t have. One engineer started revoking tokens across the board, which knocked real users offline. Another started restoring from backups before we understood what had actually happened.
We contained it within a few hours. The blast radius was small. But the recovery took three days because of the collateral damage from our own panic. The token revocation took down a payment processing integration. The premature backup restore overwrote six hours of legitimate transaction data. We spent more time cleaning up our response than we spent on the actual incident.
That night changed how I think about security incidents. The technical compromise was a footnote. The real damage came from not having a plan. From people acting fast without acting together. From urgency without structure.
I’ve since built incident response processes at three different companies and helped several more. The lesson is always the same: the plan doesn’t need to be sophisticated. It needs to exist, it needs to be practiced, and it needs to be clear about who does what. That’s the bar. Most startups don’t clear it.
Here is what I’ve learned.
The line between event and incident
Draw it before you need it. A security event is suspicious activity that might be nothing. A security incident is confirmed or strongly suspected unauthorized access that demands immediate action.
When you’re unsure, treat it as an incident and downgrade later. This is a lesson reinforced in NATO cyber defense exercises, and it applies just as much to a twelve-person startup. You can walk back an overreaction. You can’t walk back a delayed containment that let an attacker pivot to your production database.
The distinction matters because it determines who gets woken up and how fast. If everything is an incident, people stop responding. If nothing is, you miss the real ones.
Ownership before the alarm goes off
At our mobility startup, the problem wasn’t that nobody cared. Everyone cared. That was the problem. Five people all making independent decisions in a crisis is worse than one person making imperfect decisions with authority.
You need three roles defined ahead of time:
Incident commander. This person makes decisions and keeps the response coordinated. They don’t need to be the most technical person. They need to be calm, organized, and empowered to say “stop” when someone is about to make things worse.
Technical lead. This person drives the investigation and remediation. They decide what logs to pull, what systems to isolate, and what the containment strategy looks like.
Communications lead. This person keeps internal and external messaging accurate. At a five-person startup, this might be the CEO. At a fifty-person company, it might be someone in ops or legal.
At a small startup, one person wears multiple hats. Fine. But the ownership has to be explicit. Write it down. Put it in the wiki. And for the love of everything, make sure the contact information works at 11 PM on a Tuesday. Test it quarterly. Phone numbers change. People leave. On-call lists rot faster than you think.
Severity in plain language
Fancy severity matrices are a waste of time at a startup. You need four levels that everyone understands without looking anything up:
Critical. Active data exfiltration or production systems are compromised. Drop everything.
High. Confirmed unauthorized access, scope unclear. Assemble the response team now.
Medium. Suspected compromise, limited scope. Investigate within hours, not days.
Low. Suspicious activity that needs a closer look. Triage it during business hours.
The most important thing about your severity levels is that people actually use them. If your incident commander has to consult a decision tree to figure out whether an incident is a P1 or a P2, you have failed.
Containment is where most startups break
Containment is the step where panic does the most damage. The goal is to stop the bleeding without cutting off the patient’s blood supply.
Revoke compromised credentials immediately. Isolate affected systems from the network, but preserve evidence while you do it. Cut off active data exfiltration paths. If malware is involved, isolate the system and keep the sample for analysis.
Every containment action has a tradeoff. Revoking all tokens is safe but might take down production. Isolating a server preserves evidence but removes capacity. These are real decisions with real business impact. Make them intentionally, communicate the impact, and document why you chose what you chose.
The engineer at my startup who revoked every token in the system wasn’t wrong to act fast. He was wrong to act alone, without telling anyone, during an active incident. By the time I realized what had happened, our payment integration was down and we were fielding angry support tickets alongside the security investigation. Two fires instead of one.
Investigation: slow down before you speed up
Once the immediate bleeding stops, resist the urge to fix everything at once. Investigation is about building a timeline and understanding the blast radius.
Pull logs across authentication, application, and network layers. Preserve forensic images before you start making changes. Build a timeline of attacker activity. Collect indicators of compromise so you can sweep for related access you might have missed.
Don’t rush to restore service before you understand how the attacker got in. I’ve seen startups bring compromised systems back online with the same vulnerability still open because they were in a hurry to get back to normal. That isn’t recovery. That’s a second incident waiting to happen.
Eradication and recovery
Eradication means removing every foothold. Close the vulnerability that allowed the initial access. Remove any malware or backdoors. Reset credentials that could have been compromised. Verify that no unauthorized accounts were created.
Attackers who know what they are doing leave multiple persistence mechanisms. A revoked token isn’t enough if they also dropped an SSH key or created a service account. Validate your cleanup thoroughly.
Recovery means rebuilding from known-good sources when possible. Restore data from clean backups if needed. Bring systems back gradually and watch closely for signs of re-entry. The monitoring should be tighter after an incident, not looser.
Communication during a crisis
Internal communication should be regular, calm, and honest. Keep leadership aligned on severity and business risk. Inform affected teams. Separate confirmed facts from open questions in every update. Nothing erodes trust faster than a retraction.
External communication should be coordinated with legal counsel, especially if customer data is involved. Move quickly, but don’t speculate. Explain what happened, what data was affected, what you’re doing about it, and what customers should do. If you’re in fintech or health, map your regulatory notification requirements now. Not during the incident. Now.
At the shared mobility startup, we got this part right almost by accident. Our CEO was a former lawyer and insisted on reviewing every external statement. It slowed us down by an hour. It saved us from saying something incorrect that would have required a correction, which always looks worse than a slight delay.
Documentation as you go
Write it down in real time. Decisions, actions, evidence, timestamps. You will need this record for compliance, for legal protection, and for learning. It also prevents confusion when responders rotate or when you need to brief someone new.
Use a shared document. A Google Doc is fine. A dedicated incident management tool is better. The format matters less than the habit. If you aren’t documenting during the incident, you won’t remember accurately afterward. Memory under stress is unreliable.
Preparation that costs almost nothing
Know your critical assets. What systems hold sensitive data? What access paths could cause maximum damage? If you can’t answer these questions in under a minute, you aren’t ready.
Build detection that actually works. Centralized logs. Alerts on unusual patterns. At minimum, you should know when someone authenticates from a new location or when an API sees traffic outside normal bounds. This doesn’t require expensive tooling. It requires attention.
Write draft communication templates. Under pressure, writing coherent messages is harder than you think. Have templates for internal updates, customer notifications, and regulatory disclosures. Fill in the specifics during the incident, but don’t start from a blank page.
Run tabletop exercises. Pick a plausible scenario: stolen laptop, exposed database, compromised former employee credentials. Walk through the response as a team. These discussions take an hour and reveal gaps that would cost you days during a real incident.
The former employee scenario isn’t hypothetical. It’s what hit us. If we had run that exercise once, someone would have asked whether our offboarding process actually revoked all access. We would have checked. We would have found the gap. The incident would never have happened.
The real lesson
Incident response planning isn’t paranoia. It’s the cheapest insurance a startup can buy. A simple playbook, three named roles, and one tabletop exercise per quarter will put you ahead of ninety percent of companies your size.
You will still have incidents. The goal isn’t to prevent them all. The goal is to respond with discipline instead of panic, contain the damage instead of amplifying it, and come out the other side with a team that trusts each other more, not less.
Preparation is what separates an incident from a disaster. Every time.