"Final project: Paystream's support assistant"

01The problem

Paystream's head of support wants a decision in a month: should the company launch an AI support assistant, and if so, which design? Your final project is the prototype and, more importantly, the evidence: how accurate it is, where it fails, what it costs, and what keeps it safe.

Experienced AI engineers build the evaluation harness first, before the feature itself. If you can't measure it, you can't improve it, and you can't tell the head of support whether it's ready.

02The concept

The prototype has two parts

The evaluation harness

A single notebook section that, given a version of the system, produces the same table every time: accuracy, fraud recall, retrieval hit@3, correct-answer rate, refusal accuracy, failure rate and cost per 1,000 requests. Run it on every change and keep the history.

The decision

Launch, launch with limits (for example, triage only, with people answering), or don't launch yet, with the evidence for each.

03Example

The start of an evaluation harness: one function that summarises triage quality for any column of predicted categories.

import pandas as pd

tickets = pd.read_csv("https://academy.cloudtechanalytics.com/datasets/genai/tickets.csv")

def triage_report(df, predicted):
    fraud = df[df["true_category"] == "Fraud or scam"]
    return pd.Series({
        "accuracy": round((df[predicted] == df["true_category"]).mean(), 3),
        "fraud_recall": round((fraud[predicted] == "Fraud or scam").mean(), 3),
        "tickets": len(df),
    })

pd.DataFrame({col: triage_report(tickets, col) for col in ["small_model_category", "large_model_category"]}).T

accuracy  fraud_recall  tickets
small_model_category     0.798         0.723    900.0
large_model_category     0.893         0.892    900.0

Every later version of the triage system gets a row in this table. Add a column for the classic baseline from lesson 5, and one for your own prompt if you have an API key.

04Walkthrough

1. Build the harness: triage report, retrieval hit rates and answer-grade summaries, in one place.
2. Add the safety layers from lesson 9 and measure them: redaction coverage and injection flags.
3. If you have an API key, run your own prompt version on a sample and add it to the table.
4. Open the project brief on the course page and plan the write-up.

05Practice

06Check your understanding