Why I Moved Our AWS Infrastructure to Terraform

After a botched production recovery at a mobility startup exposed how little we actually knew about our own infrastructure, I moved everything to Terraform. It wasn’t fast and it wasn’t glamorous, but now our AWS environments live in version control and I sleep better.

The Night That Changed My Mind

I got paged at 1 AM because an EC2 instance hosting our vehicle-availability API had died. No big deal, I thought. Spin up a new one. Except nobody could tell me the exact instance type, the security group rules, or the IAM role attached to it. The wiki said one thing. The console said another. The CloudFormation stack that supposedly managed it had drifted so far from reality that applying it would have torn down half the VPC.

We recovered in about four hours. Four hours for one instance. That was the moment I decided our infrastructure couldn’t live in people’s heads anymore.

What We Were Running Before

The mobility startup’s AWS setup was a patchwork. Some resources lived in a CloudFormation stack written by a contractor who had since left. Other pieces were created by hand in the console during late-night debugging sessions and never documented. A handful of bash scripts in a repo called infra-scripts handled things like spinning up staging environments, but they were brittle and nobody trusted them enough to run without reading every line first.

CloudFormation itself wasn’t the problem exactly. The problem was that we treated it as a one-time setup tool instead of a living definition. We would write a template, deploy it, then immediately start making manual changes on top. Within weeks the template was fiction.

Why Terraform Won

I evaluated three options: doubling down on CloudFormation, adopting Ansible for infrastructure provisioning, or trying Terraform. Terraform was still young in mid-2016, somewhere around version 0.6, but it had two properties that mattered to me.

First, the declarative model. You describe what you want, not the steps to get there. After years of imperative scripts where ordering and idempotency were my problem, this was a relief. I could define a VPC, subnets, security groups, and instances in HCL and let Terraform figure out the dependency graph.

Second, the plan step. Before Terraform touches anything, it shows you exactly what it intends to do. Create this, modify that, destroy the other. At the mobility startup we were a small team and we couldn’t afford surprises. The plan became our pre-flight checklist.

resource "aws_instance" "vehicle_api" {
  ami           = "ami-0a2e1043"
  instance_type = "t2.small"
  subnet_id     = "${aws_subnet.private.id}"

  vpc_security_group_ids = [
    "${aws_security_group.api.id}",
  ]

  iam_instance_profile = "${aws_iam_instance_profile.vehicle_api.name}"

  tags {
    Name        = "vehicle-api-${var.environment}"
    Service     = "vehicle-availability"
    Environment = "${var.environment}"
    ManagedBy   = "terraform"
  }
}

That snippet is close to our actual vehicle-availability API definition. Nothing clever. Readable. Reviewable. If someone asks how production is configured, the answer is in Git.

The Migration

I didn’t attempt a big bang. That would have been reckless with a small engineering team and live users on the platform.

Week one: all new infrastructure goes into Terraform. No exceptions. If you need a new security group, you write HCL.

Week two through four: I started bringing existing resources under Terraform. Terraform 0.6 has no import command, so this is the tedious part: write the HCL for a resource, get it into state, run plan, and iterate until plan shows no changes. For our setup it meant adopting roughly forty resources across the VPC, subnets, route tables, security groups, instances, RDS, and an Elastic Load Balancer.

I did this environment by environment. Development first, where mistakes were cheap. Then staging. Production came last, after I had already made every dumb mistake in a place where it didn’t matter. By the time I brought the production VPC under Terraform, the process was mechanical.

Modules helped keep things sane. I built a small VPC module and a service module. Every environment became a composition of those two building blocks with different variables. The upfront effort paid back immediately in consistency. Staging actually looked like production for the first time.

State: The Sharp Edge

Terraform’s state file is where it tracks the mapping between your HCL and real AWS resources. It’s powerful and it’s dangerous. Early on I kept state in a local file and committed it to Git. That worked fine for a solo operator but fell apart as soon as a second engineer touched the repo.

We moved state to S3 remote state within the first month. Terraform has no state locking yet, so two people applying at once is a real risk: concurrent applies can corrupt state and leave you in a worse position than having no IaC at all. The state file also contains secrets in plain text, things like RDS passwords and API keys. Encryption at rest on the S3 bucket and tight IAM policies aren’t optional. We treat state like production credentials because that’s what it contains.

What Changed Operationally

The biggest shift was cultural. Before Terraform, infrastructure changes happened whenever someone had console access and a reason. After Terraform, every change went through a pull request. Run terraform plan, paste the output into the PR, get a review, merge, apply. Simple process. Hard to skip.

We caught real problems this way. A plan once showed that changing a subnet CIDR would destroy and recreate every instance in the subnet. That would have been a fifteen-minute outage discovered at 2 AM instead of during a calm code review on a Tuesday afternoon.

Recovery got faster too. When that same vehicle-availability API instance died again two months later, rebuilding it was a terraform apply away. Five minutes instead of four hours. That alone justified the migration.

Trade-offs

Terraform isn’t free. The learning curve is real, especially around state management and provider quirks. HCL is readable but limited. Complex logic feels awkward compared to a real programming language. And in mid-2016, the provider ecosystem still has gaps. We hit bugs. We read source code. We worked around things.

But the alternative was worse. The alternative was guessing at 1 AM.

Where We Stand

Every piece of the mobility startup’s AWS infrastructure is now defined in Terraform and stored in version control. Environments are reproducible. Changes are reviewed. Recovery is predictable. The migration took about six weeks of focused work alongside normal feature development.

If your infrastructure still lives in consoles and half-remembered scripts, make the move. Terraform isn’t perfect, but the cost of not knowing what you’re running is higher than you think. You will find out exactly how high at 1 AM on a Tuesday.