Modules and the pipeline

01The problem

Tallybook's web, API and worker servers are three copies of nearly the same code. When the team added encrypted disks, they changed two of the three and forgot the third. And until now, anyone with credentials could run terraform apply from their own laptop, with whatever version of the code they happened to have.

Two practices fix this: modules, so shared patterns are written once, and a pipeline, so every change follows the same safe path.

02The concept

Modules

A module is a folder of Terraform code with inputs (variables) and outputs, used like a function:

module "api" {
  source = "./modules/server-group"

  name          = "api"
  instance_type = var.api_instance_type
  server_count  = var.api_count
  tags          = local.common_tags
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.13.0"
  # ...
}

Pin versions of external modules and providers, so the same code gives the same result next month.

The pipeline

Only the pipeline has permission to apply. People review; the pipeline acts.

03Example

How much repetition is in Tallybook's servers? Compare the attribute names and the settings that vary across the four server groups in state:

import json
from urllib.request import urlopen

import pandas as pd

with urlopen("https://academy.cloudtechanalytics.com/datasets/terraform/terraform.tfstate") as f:
    state = json.load(f)

rows = []
for r in state["resources"]:
    if r["type"] == "aws_instance":
        a = r["instances"][0]["attributes"]
        rows.append({"group": r["name"], "servers": len(r["instances"]), "instance_type": a["instance_type"],
                     "ami": a["ami"], "encrypted_disk": a["root_block_device"][0]["encrypted"], "attributes": len(a)})
pd.DataFrame(rows)

group  servers instance_type                    ami  encrypted_disk  attributes
0      web        6     m5.xlarge  ami-0a1b2c3d4e5f60718            True           7
1      api        4    m5.2xlarge  ami-0a1b2c3d4e5f60718            True           7
2   worker        3     m5.xlarge  ami-0a1b2c3d4e5f60718            True           7
3  staging        6     t3.medium  ami-0a1b2c3d4e5f60718            True           7

Same AMI, same attributes, same disk settings: four groups differing only in name, count, size and tags. That's exactly what a server-group module's inputs would be, and a fix such as disk encryption would then be made once.

04Walkthrough

1. Run the cell. List the module's inputs and outputs you'd need for these four groups.
2. Which stage of the pipeline would have caught each of the six PRs' problems?
3. Who at Tallybook should be able to approve infrastructure PRs, and who should be able to apply?
4. Write the pipeline definition in plain language (the task below).

05Practice

06Check your understanding