Module 3 · Cloud & DevOps Engineer Capstone: Ready for the Sale
Reviewing the readiness plan
Read the Terraform plan for the sale-readiness changes, write policy checks in Python, and catch the change that would have deleted the orders database, plus two security holes, before anyone applies it.
About 25 minutes
The problem
The platform team has opened PR 214: sale readiness. It raises the autoscaling limits, moves the orders database to a bigger instance, adds a connection pooler (PgBouncer) and a burn-rate alarm, and cleans up an old test machine. The plan output is long, and the team is keen to apply it today.
A plan review is where infrastructure mistakes are cheapest to catch. Read what Terraform will do, not what the pull request says it does.
The concept
Actions in a plan
| Actions | Meaning |
|---|---|
create | A new resource |
update | Changed in place |
delete | Destroyed |
delete, create | Replaced: destroyed, then created again. For a database, that means the data is gone unless restored |
Some attribute changes can't be made in place, so Terraform replaces the resource. Renaming a database's identifier is one of them.
Policy as code
Write the rules once, in code, and run them on every plan:
- No production database may be deleted or replaced, and every database needs
deletion_protection. - No security group may open a database port to the internet (
0.0.0.0/0). - No bucket may be made public.
- Every resource created or updated needs an
ownertag.
Example
What the plan will do:
import json
import urllib.request
base = "https://academy.cloudtechanalytics.com/datasets/platform/"
plan = json.load(urllib.request.urlopen(base + "plan-sale-readiness.json"))
print(plan["pull_request"], "\n")
for rc in plan["resource_changes"]:
print(f"{' + '.join(rc['change']['actions']):15} {rc['address']}")PR 214: sale readiness
update aws_autoscaling_group.checkout_api
delete + create aws_db_instance.orders
create aws_instance.pgbouncer
create aws_security_group_rule.db_ingress
create aws_s3_bucket_acl.sale_banners
create aws_cloudwatch_metric_alarm.checkout_burn_rate
update aws_elasticache_cluster.session_cache
delete aws_instance.temp_test_9One line says delete + create for the orders database. The policy checks:
def check(plan):
findings = []
for rc in plan["resource_changes"]:
actions, after = rc["change"]["actions"], rc["change"]["after"] or {}
if rc["type"] == "aws_db_instance":
if "delete" in actions:
findings.append((rc["address"], "database would be DESTROYED (" + rc.get("action_reason", "delete") + ")"))
if after and not after.get("deletion_protection"):
findings.append((rc["address"], "deletion_protection is off"))
if rc["type"] == "aws_security_group_rule" and "0.0.0.0/0" in after.get("cidr_blocks", []) and after.get("from_port") == 5432:
findings.append((rc["address"], "database port open to the internet"))
if rc["type"] == "aws_s3_bucket_acl" and after.get("acl", "").startswith("public"):
findings.append((rc["address"], "bucket made public"))
if actions != ["delete"] and "tags" in after and not after["tags"].get("owner"):
findings.append((rc["address"], "no owner tag"))
return findings
findings = check(plan)
for address, problem in findings:
print(f"{address:45} {problem}")
print(f"\n{len(findings)} findings")aws_db_instance.orders database would be DESTROYED (replace_because_cannot_update)
aws_db_instance.orders deletion_protection is off
aws_security_group_rule.db_ingress database port open to the internet
aws_s3_bucket_acl.sale_banners bucket made public
aws_elasticache_cluster.session_cache no owner tag
5 findingsThe headline: renaming the database from orders-db to kasuwa-orders-db forces Terraform to destroy and recreate it, eight weeks before the sale, with deletion protection off. Applying this plan would delete every order. The fix is to change only the instance class, which happens in place, turn on deletion protection, and add a lifecycle guard so Terraform refuses to destroy it:
Terraform (HCL)
resource "aws_db_instance" "orders" {
identifier = "orders-db"
instance_class = "db.r6g.2xlarge"
deletion_protection = true
lifecycle {
prevent_destroy = true
}
}The security group rule would expose the database to the whole internet; it should allow only the PgBouncer instance's security group. The public bucket for sale banners should be served through the CDN instead.
Walkthrough
- Run the cells.
- Change the database entry in a copy of the plan to an in-place
updatewithdeletion_protectiontrue, and rerun the checks. - Add a rule of your own: for example, that
max_sizeof an autoscaling group never more than doubles in one change without a note. - Decide where these checks run: in the pull request pipeline, blocking the merge.
- Write the plan review (the task below).
Practice
Practice
How many findings do the policy checks report on this plan?
Practice
Which resource address would be destroyed and recreated?
Task
8 minWrite the plan review comment for PR 214 (60 to 150 words): the blocking problems, why the database would be replaced, the fix for each, and what's fine to merge.
Your work is checked for
- Says it's blocked (request changes, don't apply)
- Explains the replacement (identifier, rename)
- A fix for the database (in place, deletion protection, prevent_destroy)
- Covers the open database port
- Covers the public bucket
- Says what's fine
- Between 60 and 150 words
Check your understanding
Answer every question to check.