Monitoring with control limits

01The problem

On 14 July 2026, the model provider updated the model behind the name Paystream uses. Overnight, the assistant began refusing ordinary questions ("I'm not able to help with account matters") about three times as often as before. Paystream's dashboard showed it from the first day. Nobody was looking at the right number with a rule for what "unusual" means, so it took four days and a pile of customer complaints for anyone to act.

Dashboards don't detect problems. Alerts with sensible limits do.

02The concept

Control limits

Every metric varies from day to day. A control limit marks the edge of normal variation:

upper limit = recent average + 3 × recent standard deviation

computed over a baseline window, such as the previous 28 days. A day outside the limits is unusual enough to investigate.

Why from the metric's own history

A fixed rule like "alert if refusals pass 10%" is either too loose (a jump from 3% to 8% stays under it) or too tight (noisy days trigger it). Limits learned from the data fit each metric.

Practical details

• Exclude known incident days from the baseline, so a problem doesn't raise its own limit.
• Weekly patterns (Sundays are quieter) can need separate baselines for each day of the week, or rates rather than counts.
• Use rates (refusals ÷ conversations) so busy days don't look alarming.
• Every alert needs an owner and a first step.

03Example

import pandas as pd

base = "https://academy.cloudtechanalytics.com/datasets/llmops/"
daily = pd.read_csv(base + "daily_metrics.csv", parse_dates=["date"])
daily["refusal_rate"] = daily["refusals"] / daily["conversations"]

baseline = daily["refusal_rate"].shift(1).rolling(28, min_periods=14)
daily["upper"] = baseline.mean() + 3 * baseline.std()
daily["alert"] = daily["refusal_rate"] > daily["upper"]

print("Days above the limit:", daily.loc[daily["alert"], "date"].dt.strftime("%d %b").tolist())
daily.set_index("date").loc["2026-07-10":"2026-07-21", ["refusal_rate", "upper", "alert"]].round(4)

Days above the limit: ['14 Jul', '15 Jul', '16 Jul']
            refusal_rate   upper  alert
date
2026-07-10        0.0271  0.0367  False
2026-07-11        0.0269  0.0366  False
2026-07-12        0.0283  0.0367  False
2026-07-13        0.0278  0.0366  False
2026-07-14        0.0816  0.0365   True
2026-07-15        0.0885  0.0616   True
2026-07-16        0.0876  0.0778   True
2026-07-17        0.0806  0.0893  False
2026-07-18        0.0852  0.0967  False
2026-07-19        0.0850  0.1044  False
2026-07-20        0.0283  0.1110  False
2026-07-21        0.0281  0.1111  False

The alert fires on the very first day of the change. Notice also that the limit itself creeps up after the first bad day, because those days enter the baseline. That's why known incident days should be excluded once they're found. The same method on latency:

daily["latency_upper"] = daily["p95_latency_ms"].shift(1).rolling(28, min_periods=14).mean() + 3 * daily["p95_latency_ms"].shift(1).rolling(28, min_periods=14).std()
print(daily.loc[daily["p95_latency_ms"] > daily["latency_upper"], ["date", "p95_latency_ms", "latency_upper"]].round(0).to_string(index=False))

date  p95_latency_ms  latency_upper
2026-06-20           11800         3780.0

The outage day stands out clearly, which is why that incident was found the same day: a latency alert already existed. The refusal alert would have found the July incident four days sooner.

04Walkthrough

1. Run the cells. Compute the hand-over rate and set limits on it. Does it flag the July incident too?
2. Rebuild the refusal baseline excluding 14 to 19 July. How do the limits for later days change?
3. Try 2 standard deviations instead of 3. How many false alarms appear in May and June?
4. Write the alert definitions (the task below).

05Practice

06Check your understanding