Rollbacks and recovery

01The problem

Even with good tests and canaries, some bad releases reach customers. What matters then is how fast service is restored. Under the old process, Tallybook's engineers usually tried to fix the problem and push a new release ("fix forward"), which meant working out the cause first, under pressure. Under the new one, the first move is always to go back to the last good image.

02The concept

Roll back or fix forward?

Restore service first; understand the cause afterwards.

Making rollbacks possible

• Keep previous images in the registry, tagged by version.
• Make database changes backwards-compatible (add a column first, start using it in a later release, remove the old one later still), so the previous app version still works.
• Use feature flags: ship new code switched off, turn it on separately, and switch it off instantly if it misbehaves, with no deployment at all.

03Example

Incidents and their restore times under each process:

import pandas as pd

deploys = pd.read_csv("https://academy.cloudtechanalytics.com/datasets/cicd/deployments.csv")
incidents = deploys[deploys["caused_incident"] == 1]
print(incidents.groupby(["pipeline", "result"])["minutes_to_restore"].agg(["count", "median", "max"]))
print()
print("Total customer-facing minutes of failed releases:")
print(incidents.groupby("pipeline")["minutes_to_restore"].sum())

count  median    max
pipeline result
new      rolled_back        2    21.5   25.0
old      fixed_forward      3   216.0  278.0
         rolled_back        5   194.0  306.0

Total customer-facing minutes of failed releases:
pipeline
new      43.0
old    1600.0
Name: minutes_to_restore, dtype: float64

Under the old process, fixing forward took much longer than rolling back, and even the rollbacks took about three hours at the median, because rolling back meant reverting the code and re-running the whole manual process. Under the new pipeline, the two releases that slipped past the canary were rolled back in minutes.

04Walkthrough

1. Run the cells. How many hours of failed-release time did the old process cause in three months, and the new one?
2. Design a backwards-compatible change for renaming a database column used by the app.
3. Which kinds of Tallybook features would you put behind a feature flag?
4. Write the recovery runbook (the task below).

05Practice

06Check your understanding