Burn-rate alerts

01The problem

A fixed alert like "error rate above 1%" is either too sensitive (paging for a 2-minute blip at 3am) or too slow (missing a steady 0.5% that quietly eats the month's budget). The SLO already says how much failure is acceptable. Alerts should fire when the budget is being spent too fast.

02The concept

Burn rate

Burn rate = observed error rate ÷ the error rate the SLO allows. With a 99.9% SLO, the allowed rate is 0.1%:

Multi-window alerts

Page when both a long and a short window are burning fast:

• the 1-hour burn rate is above 14.4 (a serious amount of budget is going), and
• the 5-minute burn rate is above 14.4 (it's still happening now, not a past blip).

The long window prevents paging on blips; the short window lets the alert clear quickly once the problem stops. Slower burns (for example 6× over 6 hours) open a ticket rather than paging.

03Example

Per-minute burn rate on 31 August, then both windows:

import pandas as pd

metrics = pd.read_csv("https://academy.cloudtechanalytics.com/datasets/observability/metrics.csv", parse_dates=["minute"])
web = metrics[metrics["service"] == "web"].set_index("minute")[["requests", "errors"]]

ALLOWED = 0.001   # 99.9% SLO

def burn(window_minutes):
    rolled = web.rolling(window_minutes).sum()
    return (rolled["errors"] / rolled["requests"]) / ALLOWED

web["burn_1h"] = burn(60)
web["burn_5m"] = burn(5)
web["page"] = (web["burn_1h"] > 14.4) & (web["burn_5m"] > 14.4)

fired = web.index[web["page"]]
print("Burn-rate page fires at", fired.min().strftime("%H:%M"), "and clears after", fired.max().strftime("%H:%M"))
web.loc["2026-08-31 09:38":"2026-08-31 09:46", ["burn_5m", "burn_1h", "page"]].round(1)

Burn-rate page fires at 09:42 and clears after 10:37
                     burn_5m  burn_1h   page
minute
2026-08-31 09:38:00      0.7      0.6  False
2026-08-31 09:39:00      0.7      0.6  False
2026-08-31 09:40:00     60.7      5.6  False
2026-08-31 09:41:00    113.2     10.0  False
2026-08-31 09:42:00    176.0     15.3   True
2026-08-31 09:43:00    228.6     19.8   True
2026-08-31 09:44:00    277.1     23.8   True
2026-08-31 09:45:00    272.5     28.3   True
2026-08-31 09:46:00    270.8     32.4   True

The alert fires within a couple of minutes of the bulk job starting, and stops soon after the fix. Compare a naive alert that pages whenever a single minute's error rate is above 1%, over the quiet early morning:

web["naive"] = (web["errors"] / web["requests"]) > 0.01
print("Naive alert minutes:", int(web["naive"].sum()), " burn-rate alert minutes:", int(web["page"].sum()))
print("Highest 1-hour burn rate before 09:00:", round(web.loc[:"2026-08-31 08:59", "burn_1h"].max(), 2))

Naive alert minutes: 54  burn-rate alert minutes: 56
Highest 1-hour burn rate before 09:00: 0.71

On this day, both agree on the incident, but look at the morning: before 09:00 the budget burned at well under 1×. A burn-rate rule stays quiet when nothing threatens the SLO, however a single minute looks, and the same rule works for any traffic level.

04Walkthrough

1. Run the cells. Add a ticket-level alert: 6-hour burn rate above 6 and 30-minute above 6. Does it fire on 31 August?
2. What error rate corresponds to a burn rate of 14.4 under a 99.5% SLO?
3. Change the short window to 15 minutes. How much later does the page clear?
4. Write the alert rules (the task below).

05Practice

06Check your understanding