Dangerous changes

01The problem

Three of Tallybook's six pull requests could cause serious damage if applied as they are, and none of their titles says so:

• PR 102, "rename database", destroys the production database.
• PR 101, "web autoscaling", deletes all six web servers in the same step that creates their replacement.
• PR 106, "multi-AZ database", also upgrades the database to a new major version, immediately.

Each has a standard fix. Knowing them is a core skill for anyone who approves infrastructure changes.

02The concept

Renames: moved blocks

Tell Terraform the resource moved, and the rename becomes a no-op:

moved {
  from = aws_db_instance.prod
  to   = aws_db_instance.main
}

Protect what can't be recreated

resource "aws_db_instance" "main" {
  # ...
  deletion_protection = true

  lifecycle {
    prevent_destroy = true
  }
}

prevent_destroy makes Terraform refuse any plan that would destroy the resource. deletion_protection makes the cloud provider refuse it too, even from the console.

Sequence risky changes

• Create the new thing in one PR; switch traffic and delete the old thing in another, after checking.
• create_before_destroy in a lifecycle block makes replacements create first.

Check every changed attribute

An update can hide changes the title doesn't mention. Compare before and after for every attribute, not just the one you expected.

03Example

Find the changed attributes in every update, across all plans:

import json
from urllib.request import urlopen

base = "https://academy.cloudtechanalytics.com/datasets/terraform/"
PRS = ["101-web-autoscaling", "102-rename-database", "103-reporting-access", "104-cost-tags", "105-rightsize-api", "106-multi-az-database"]

def load(name):
    with urlopen(base + name) as f:
        return json.load(f)

plans = {pr: load(f"plan-pr-{pr}.json") for pr in PRS}

def changed_attributes(rc):
    before, after = rc["change"]["before"] or {}, rc["change"]["after"] or {}
    return sorted(k for k in set(before) | set(after) if before.get(k) != after.get(k))

for pr, plan in plans.items():
    updates = [rc for rc in plan["resource_changes"] if rc["change"]["actions"] == ["update"]]
    if updates:
        attrs = sorted({a for rc in updates for a in changed_attributes(rc)})
        print(f"PR {pr}: {len(updates)} updates, changing {attrs}")

PR 104-cost-tags: 19 updates, changing ['tags']
PR 105-rightsize-api: 4 updates, changing ['instance_type']
PR 106-multi-az-database: 1 updates, changing ['allow_major_version_upgrade', 'apply_immediately', 'engine_version', 'multi_az']

PR 104 changes only tags, and PR 105 only instance types. PR 106 changes four attributes: multi_az as intended, plus engine_version (15.4 to 16.3, a major upgrade), allow_major_version_upgrade, and apply_immediately, which means "do it now, during business hours, not in the maintenance window". Now the deletions of stateful resources:

STATEFUL = {"aws_db_instance", "aws_s3_bucket", "aws_ebs_volume", "aws_efs_file_system"}

for pr, plan in plans.items():
    for rc in plan["resource_changes"]:
        if "delete" in rc["change"]["actions"] and rc["type"] in STATEFUL:
            before = rc["change"]["before"]
            print(f"PR {pr}: deletes {rc['address']} ({before['identifier']}, {before['allocated_storage']} GB), deletion_protection={before['deletion_protection']}")

PR 102-rename-database: deletes aws_db_instance.prod (tallybook-prod, 500 GB), deletion_protection=False

The production database has deletion_protection switched off, so nothing outside Terraform would stop this either. Two fixes, both needed: a moved block in PR 102, and prevent_destroy plus deletion_protection on the database in a separate PR.

04Walkthrough

1. Run the cells. Write the moved block that makes PR 102 safe.
2. Split PR 101 into two PRs. What does each contain, and what do you check in between?
3. Split PR 106: which attributes go in the first PR, and when should the version upgrade happen?
4. Write your review comments (the task below).

05Practice

06Check your understanding