Experimentation and A/B Testing

Curriculum

1. 01Why ExperimentWhy comparing users who did something with users who didn't can't tell you whether the thing worked, and how random assignment fixes that.20 min
2. 02Designing a TestWrite a hypothesis, choose one primary metric and the guardrails, set the smallest effect worth detecting, and calculate how many users and days the test needs before it starts.25 min
3. 03Running a Test ProperlyCheck that randomisation worked with a sample ratio mismatch test, and see by simulation why peeking at results and stopping early multiplies false alarms.15 min
4. 04Analysing Conversion TestsTest the difference between two conversion rates with a two-proportion z-test, report it with a confidence interval and relative lift, and say what it means in users and money.25 min
5. 05Analysing Continuous MetricsCompare averages such as transactions or value per user with Welch's t-test, handle heavily skewed money metrics with a bootstrap, and decide whether a few big users are driving the result.15 min
6. 06Segments, Multiple Tests and NoveltyLook inside a result by segment without fooling yourself, correct for running many tests at once, and spot novelty effects that fade once users get used to a change.25 min
7. 07Guardrails and DecisionsWeigh a winning primary metric against guardrails that get worse, value both sides in money over a sensible horizon, and make a decision you can defend.25 min
8. 08When You Can't RandomiseEstimate an effect when a change was rolled out to some places and not others, with difference-in-differences, and check the parallel-trends assumption it depends on.25 min
9. 09Final ProjectPlan your final project, a full review of Paystream's four experiments and a design for the next one, and start by checking every test before trusting any result.20 min
10. Final assessmentPass it (60% or more) to earn the Experimentation and A/B Testing course completion badge.