Module 9 · Agile Business Analysis
Reviews, pilots and outcomes
Run sprint reviews that produce real feedback, design a pilot with success criteria agreed in advance, and decide what to do next from the evidence.
About 20 minutes
The problem
The kiosk app goes to 15 kiosks in Surulere for a six-week pilot. Afterwards there will be a meeting to decide whether to roll it out to all 39 kiosks, change it, or stop. Without preparation, that meeting goes one of two ways: the people who championed the app call it a success because "the kiosks liked it", or the people who doubted it call it a failure because "only 9 kiosks used it every week". Both will have a point, and neither will have evidence.
A pilot is an experiment. It needs a question, success criteria agreed before it starts, and a plan for measuring them. Otherwise the result is decided by whoever argues best afterwards.
The concept
Sprint reviews that produce feedback
A sprint review isn't a presentation; it's a working session to inspect the increment and adapt the backlog. To make it useful:
- Invite real users (a kiosk owner, a sales rep), not just managers.
- Demonstrate against the sprint goal, with real-looking data.
- Ask specific questions: "Would you use reorder instead of calling your rep? What would stop you?"
- Turn feedback into backlog items before people leave the room.
Designing a pilot
| Part | Kiosk app pilot |
|---|---|
| Question | Does the app make kiosks order more often? |
| Who and how long | 15 Surulere kiosks, six weeks |
| Comparison | the same kiosks' ordering in the six weeks before, and similar kiosks elsewhere over the same weeks |
| Success criteria (agreed in advance) | average days between orders falls from about 22 to 14 or fewer; at least 60% of pilot kiosks' orders placed in the app; no fall in average order value |
| Decision rule | all three met: roll out; adoption high but frequency unchanged: rethink; adoption low: find out why before spending more |
The comparison group matters. If kiosks everywhere order more in the pilot weeks (say, before a holiday), a rise in the pilot group proves nothing about the app.
Validated learning
Treat every release as a test of an assumption: "kiosk owners will reorder in the app if it takes under a minute". Measure, learn, and adjust the backlog. Sometimes the right outcome of a pilot is to stop, and that's a success if it saves the money a full rollout would have cost.
Example
The BA's plan for the sprint 7 review, where reorder is demonstrated:
- Goal reminder (2 minutes): "Kiosks can reorder last week's basket in two taps."
- Demonstration by a developer, using a real kiosk's last basket (5 minutes).
- Hands-on: the kiosk owner tries it on her own phone while everyone watches in silence (5 minutes).
- Questions (10 minutes): "What did you expect to happen when you tapped reorder? Would you change anything before confirming? When would you use this instead of waiting for the rep?"
- Backlog (5 minutes): new items agreed and added; priorities checked against the product goal.
Step 3 usually teaches the team more than the rest of the review put together.
Walkthrough
- Write the pilot's question, groups, duration and comparison.
- Agree the success criteria and the decision rule with the product owner and the commercial director before the pilot starts (the task below).
- Check you can measure every criterion: frequency from the sales data, adoption from the app, order value from both.
- Plan the next sprint review with a real kiosk owner and specific questions.
- Plan the pilot-end meeting: the numbers, what they mean, and the decision the rule points to.
Practice
Task
10 minWrite the pilot plan for the kiosk app as lines starting with these headings: Question:, Who:, Comparison:, Success criteria: (at least two criteria, each with a number) and Decision rule:.
Your work is checked for
- A Question line
- A Who line, with a number of kiosks or a duration
- A Comparison line
- Success criteria with at least two numbers
- A Decision rule line saying what happens if criteria are or aren't met
Task
5 minWrite five questions you'd ask a kiosk owner at the sprint review after they try the reorder feature. Each on its own line, ending with a question mark. Make them open, and include at least one about when they'd use it and one about what would stop them.
Your work is checked for
- Five questions, each ending with ?
- Mostly open questions (what, how, when, why, tell me, describe)
- Asks when they'd use it
- Asks what would stop them
- No leading questions (don't you think, wouldn't you, isn't it)
Check your understanding
Answer every question to check.