Did the pilot work?

01The problem

From 1 April 2026, Lagos tried four changes:

1. A document checklist app for agents and phone staff, so claims can't be sent without the right documents.
2. Photo assessment for windscreen claims, instead of a physical inspection.
3. Assessors approve windscreen and accident claims under ₦1m, instead of waiting for the manager's Friday sign-off.
4. SMS updates telling customers what's happening and what's missing.

The Lagos claims manager says claims are "much faster". The head of IT says April to June is always different and the pilot proves nothing. They could both be partly right. Your job is to measure it fairly.

02The concept

Before and after isn't enough

Comparing Lagos in April to June with Lagos before mixes the pilot's effect with anything else that changed at the same time. April brings the rainy season, more accidents and busier assessors everywhere.

A comparison group

The other four regions didn't get the changes but did get the season. Their change over the same period shows what would probably have happened in Lagos without the pilot.

Difference in differences

(Lagos after − Lagos before) − (others after − others before). The second bracket removes whatever affected everyone. It assumes Lagos and the other regions would otherwise have moved together, so check that they were similar before.

Check what it mustn't harm

A faster process that rejects good claims or approves bad ones isn't a success. Look at rejection rates and the controls.

03Example

Days to settle, before and during the pilot, for Lagos and the others:

SELECT CASE WHEN region = 'Lagos' THEN 'Lagos' ELSE 'Other regions' END AS grp,
       CASE WHEN submitted_at < '2026-04-01' THEN '1 Before' ELSE '2 Pilot period' END AS period,
       COUNT(*) AS paid_claims,
       ROUND(AVG(julianday(closed_at) - julianday(submitted_at)), 1) AS avg_days
FROM claims
WHERE outcome = 'Paid'
GROUP BY grp, period
ORDER BY grp, period;

grp            period          paid_claims  avg_days
Lagos          1 Before                676      24.9
Lagos          2 Pilot period          296      17.6
Other regions  1 Before               1228      25.4
Other regions  2 Pilot period          507      26.4

Before the pilot, Lagos and the other regions were almost identical, which makes the others a good comparison. During the pilot, the other regions got about a day slower (the season), while Lagos got more than seven days faster. The difference in differences:

WITH g AS (
  SELECT region = 'Lagos' AS lagos, submitted_at >= '2026-04-01' AS pilot,
         AVG(julianday(closed_at) - julianday(submitted_at)) AS days
  FROM claims WHERE outcome = 'Paid' GROUP BY lagos, pilot
)
SELECT ROUND((SELECT days FROM g WHERE lagos = 1 AND pilot = 1) - (SELECT days FROM g WHERE lagos = 1 AND pilot = 0)
           - ((SELECT days FROM g WHERE lagos = 0 AND pilot = 1) - (SELECT days FROM g WHERE lagos = 0 AND pilot = 0)), 1) AS did_days;

did_days
    -8.4

A naive before-and-after would say 7.4 days. Allowing for the season, the pilot saved about 8.4 days per paid claim. Now the other measures:

WITH r AS (SELECT claim_id, MAX(activity = 'Documents requested') AS requested FROM events GROUP BY claim_id),
x AS (
  SELECT c.*, r.requested, (SELECT COUNT(*) FROM complaints m WHERE m.claim_id = c.claim_id) AS complaints
  FROM claims c JOIN r USING (claim_id)
)
SELECT CASE WHEN region = 'Lagos' THEN 'Lagos' ELSE 'Other regions' END AS grp,
       CASE WHEN submitted_at < '2026-04-01' THEN '1 Before' ELSE '2 Pilot period' END AS period,
       COUNT(*) AS claims,
       ROUND(100.0 * AVG(requested), 1) AS pct_docs_requested,
       ROUND(100.0 * AVG(outcome = 'Paid' AND julianday(closed_at) - julianday(submitted_at) > 30), 1) AS pct_paid_over_30,
       ROUND(100.0 * AVG(outcome = 'Withdrawn'), 1) AS pct_withdrawn,
       ROUND(100.0 * AVG(outcome = 'Rejected'), 1) AS pct_rejected,
       ROUND(100.0 * SUM(complaints) / COUNT(*), 1) AS complaints_per_100
FROM x
GROUP BY grp, period
ORDER BY grp, period;

grp            period          claims  pct_docs_requested  pct_paid_over_30  pct_withdrawn  pct_rejected  complaints_per_100
Lagos          1 Before           793                47.3              18.7            7.4           7.3                19.4
Lagos          2 Pilot period     337                  16               3.9            2.1          10.1                 9.8
Other regions  1 Before          1446                44.7              19.9            7.2           7.9                19.5
Other regions  2 Pilot period     590                  41              21.9            6.4           7.6                22.4

The checklist did what it was meant to: claims needing documents chased fell from 47% to 16% in Lagos, while the other regions barely changed. Withdrawals and complaints fell too, and slow claims (over 30 days) almost disappeared.

One number needs watching: Lagos rejected 10.1% of claims during the pilot, against 7.3% before. That's 34 claims out of 337, a small sample, and a rise in rejections doesn't suggest assessors are approving too freely. But it's worth a sample audit before rollout, and the finance controller will ask.

04Walkthrough

1. Run the queries and reproduce the difference in differences.
2. Check that Lagos and the other regions moved together before the pilot. Plot monthly days to settle for both, from July 2025.
3. Split the Lagos pilot claims by type. How much faster were windscreen claims, which no longer had an inspection?
4. List what the pilot can't tell you: for example, whether it would work in Port Harcourt with only two assessors.
5. Write the pilot result (the task below).

05Practice

06Check your understanding