Two Terraform configs, one live resource: how a tag became an access-control bug

A cost-report alert fired for untagged resources in a production AWS account. I went looking for the drift and found something worse than a missing tag: two Terraform configurations describing the same live resources, disagreeing about how to tag them.

The migration to a module layout had landed. Deleting the old flat directory had not. Identifiers are changed; this was a client engagement.

What was actually there

The current, module-based config that had really been applied:

# terraform/envs/production/providers.tf
provider "aws" {
  default_tags {
    tags = {
      Project = "app"
      Env     = "production"
    }
  }
}

And the abandoned flat config, seven .tf files up a directory, still describing the same EC2 instance and the same RDS instance:

# terraform/providers.tf
provider "aws" {
  default_tags {
    tags = {
      Project     = "app-backend"    # different value
      Environment = "production"     # different KEY
      # no Env key at all
    }
  }
}

Different value for Project. A different key entirely — Environment where the live config uses Env. Neither directory knows the other exists.

Why that is not a cosmetic problem

Tags in that account were load-bearing in two separate systems:

1. Cost Explorer grouped on Project + Env. Any resource whose Project changes from app to app-backend silently drops out of every historical cost grouping. Not an error — a gap, in the reports you use to make decisions.

2. The SSO permission set gated on Env. This is the sharp end:

{
  "Effect": "Allow",
  "Action": ["ec2:StartInstances", "ec2:StopInstances", "rds:RebootDBInstance"],
  "Resource": "*",
  "Condition": {
    "StringEquals": { "aws:ResourceTag/Env": ["dev", "production"] }
  }
}

That is attribute-based access control, and it is a perfectly good pattern — until the attribute it reads is deleted.

terraform apply from the stale directory would have rewritten Project and stripped Env entirely on the production EC2 and RDS instances. The consequences, in order of how quickly you'd notice:

• Cost reporting loses those resources from their grouping. Nobody notices for a month.
• The Env-gated permission set stops matching. Engineers lose the ability to restart production infrastructure — discovered during an incident, when they reach for the runbook and get AccessDenied on a resource they own.
• The untagged-resource alert fires again, which is how it was found, but only after the damage.

An ABAC policy has a failure mode that a resource-ARN policy does not: the thing it evaluates lives on the resource and can be edited by anything with write access, including your own IaC. Change the tag, change the permission — with no change to any policy document, no review, and nothing in CloudTrail that looks like an authorization change.

Why nobody caught it

Two reasons, both structural:

No CI job ran Terraform. The trigger was a human typing cd terraform && terraform apply in the wrong directory — one cd up from where they meant. Low likelihood, high blast radius. That combination is the hardest to prioritise, because the likelihood argument always wins the meeting.

terraform plan in the stale directory looks correct. It does not say "another config already manages this." It compares its own state against the real resources and offers a tidy, plausible diff. The output that would have destroyed the access model reads as routine tag maintenance.

The fix, and the fix I did not do

The right long-term fix is deleting the seven stale files. The migration doc said so explicitly. That deletion is a real change with a real review cost, and it was not what I was in the code to do.

So the shipped change was one file, tags only:

   default_tags {
     tags = {
-      Project     = "app-backend"
+      Project     = "app"
       Environment = "production"
+      Env         = "production"
       ManagedBy   = "terraform"
     }
   }

Environment and ManagedBy left alone — they are not read by anything, and removing them is a separate argument. Now an accidental apply from the stale directory is a no-op on the tags that matter, rather than an access-control incident.

The PR says out loud that this is a defusing, not a resolution, and points at the doc that calls for the deletion. A mitigation that pretends to be a fix is how the real fix gets forgotten.

Four things I would check in any AWS account today

1. Grep for every tag key referenced in a policy condition. Anything under aws:ResourceTag/* in an IAM policy, SCP, or permission set is not a label — it is part of your authorization model, and it deserves the same change control as the policy itself.

   $terminal

   aws iam list-policies --scope Local --query 'Policies[].Arn' --output text \
     | tr '\t' '\n' | while read -r arn; do
         v=$(aws iam get-policy --policy-arn "$arn" --query 'Policy.DefaultVersionId' --output text)
         aws iam get-policy-version --policy-arn "$arn" --version-id "$v" \
           --query 'PolicyVersion.Document' --output json | grep -o 'aws:ResourceTag/[A-Za-z0-9_-]*'
       done | sort -u
   

2. Find resources managed by more than one config. If two state files claim the same ARN, whichever ran last wins, and neither will warn you.

3. Treat an abandoned IaC directory as live code. It is executable. It has no tests, no CI, and no owner — the worst combination in the repo. Delete it, or make it fail loudly: an empty backend block, a lifecycle precondition, even a README that the on-call will actually see.

4. Alert on untagged resources. That alert is what surfaced this. A cost tool flagging governance drift did the job a security review had not.

The takeaway

Tags stop being metadata the moment a policy condition reads one. From then on, any code path that can write a tag is a code path that can grant or revoke access — including the Terraform directory everyone has forgotten about.

Delete the old config. If you can't delete it today, at least make sure running it by accident is boring.