The brief and the design

01The problem

This is the capstone of the AI Engineer track. You'll take one LLM feature all the way from design to production monitoring, using the habits from the track: validated outputs, rules in code, retrieval, evaluation, red-teaming, release gates and alerts.

The company is Shieldline Insurance, the motor insurer from the Business Analyst Capstone. Its claims were slow partly because customers sent incomplete information and then heard nothing. Most customers already message Shieldline on WhatsApp, so the head of claims has asked for an assistant that:

1. reads a customer's first message about an incident and turns it into a structured claim (type, date, plate number, injuries, police report);
2. tells the customer exactly which documents are still needed;
3. answers policy questions from the policy wording;
4. hands anything serious to a person.

The team has already run three configurations over 600 real messages and recorded every output, so you can evaluate them without an API key.

02The concept

The model reads; code decides

An LLM is good at reading messy text, including Pidgin, and at writing a clear reply from a source. It is not reliable at enforcing rules. So split the work:

What it must never do

Approve, reject or promise payment for a claim; quote an amount Shieldline will pay; reveal anything about another customer; or answer from memory when the policy doesn't say.

The test data

messages.csv holds 600 messages with gold labels (what a claims officer extracted) and the recorded outputs of three configurations:

03Example

import pandas as pd

base = "https://academy.cloudtechanalytics.com/datasets/assistant/"
messages = pd.read_csv(base + "messages.csv", keep_default_na=False)
print(f"{len(messages)} messages")
print(messages["language"].value_counts().to_string(), "\n")
print(messages["gold_claim_type"].value_counts().to_string(), "\n")
print(f"Need a person (gold): {messages['gold_needs_human'].mean():.1%}")

600 messages
language
English    482
Pidgin     118

gold_claim_type
Accident damage    250
Windscreen         180
Third party         90
Theft               80

Need a person (gold): 36.0%

One message, and what each configuration made of it:

example = messages[messages["message_id"] == "MSG-0001"].iloc[0]
print(example["text"], "\n")
for config in ["small_v1", "large_v1", "large_v2"]:
    print(f"{config}: {example[config + '_output']}")

Good morning o, stone break my windscreen on 26/06/2026 for Wuse 2. Na Lexus RX 350, number na FKJ 471 KT. Mechanic talk say e go cost like ₦275k. Which document una need?

small_v1: {"claim_type":"Windscreen","incident_date":"2026-06-26","vehicle_reg":"FKJ 471 KT","injuries":false,"police_report":false,"needs_human":false}
large_v1: {"claim_type":"Windscreen","incident_date":"2026-06-26","vehicle_reg":"FKJ 471 KT","injuries":false,"police_report":false,"needs_human":false}
large_v2: {"claim_type":"Windscreen","incident_date":"2026-06-26","vehicle_reg":"FKJ471KT","injuries":false,"police_report":false,"needs_human":false}

All three got this one essentially right, but notice large_v2 wrote the plate without spaces. Small differences like that matter when the plate is matched against Shieldline's policy records. The rest of this course measures them properly.

04Walkthrough

1. Download the dataset below and open messages.csv in Colab.
2. Read twenty messages, including some in Pidgin. What makes them hard to extract?
3. Draw the design: message in, model calls, the checks in code, and the three ways out (reply with a checklist, answer a question, hand to a person).
4. List what the assistant must never do, and where in the design each rule is enforced.
5. Write the design note (the task below).

05Practice

06Check your understanding