Module 6 · Software Engineering with Python
Validating input
Check every record before using it, collect every problem with a clear reason instead of crashing on the first, and decide what happens to bad records so they're fixed rather than silently dropped.
About 25 minutes
The problem
Lesson 5 fixed one problem in the invoice export. There are several more: invoices exported twice, discounts above the 20% limit, dates in two formats, invoices for customers who don't exist, due dates before issue dates, and lines with zero or negative quantities. Any of them can produce a wrong total without any error at all, which is worse than a crash.
The concept
Validate at the boundary
Check data where it enters your system (an import, an API request, a form), before any calculation uses it.
Collect problems, don't stop at the first
A validator returns a list of problems for each record. An empty list means valid. Then you can report every problem at once, with counts.
Decide what happens to invalid records
| Option | When |
|---|---|
| Reject with a clear message | input from a person or another system that can fix it |
| Quarantine for review | bulk imports, so good records still go through |
| Fix automatically | only when the fix is certain (a formatted amount) |
Never silently drop bad records: totals become wrong and nobody knows.
Raise errors with useful messages
When a function can't continue, raise ValueError(...) with a message that says what was wrong and what was expected, as invoice_total does for discounts.
Example
A validator for invoice rows:
import re
from datetime import date
import pandas as pd
base = "https://academy.cloudtechanalytics.com/datasets/invoicing/"
raw = pd.read_csv(base + "invoices_raw.csv", dtype=str, keep_default_na=False)
customers = set(pd.read_csv(base + "customers.csv", dtype=str)["customer_id"])
lines = pd.read_csv(base + "invoice_lines.csv")
bad_quantity = set(lines.loc[lines["quantity"] <= 0, "invoice_id"])
ISO_DATE = re.compile(r"\d{4}-\d{2}-\d{2}")
def validate(row):
problems = []
if not row["customer_id"]:
problems.append("missing customer")
elif row["customer_id"] not in customers:
problems.append("unknown customer")
if not ISO_DATE.fullmatch(row["issue_date"]):
problems.append("issue date not YYYY-MM-DD")
elif date.fromisoformat(row["due_date"]) < date.fromisoformat(row["issue_date"]):
problems.append("due before issued")
if not 0 <= int(row["discount_pct"]) <= 20:
problems.append("discount above 20%")
if row["invoice_id"] in bad_quantity:
problems.append("line with zero or negative quantity")
return problems
raw["duplicate"] = raw.duplicated(keep="first")
raw["problems"] = raw.apply(validate, axis=1)
print("Rows:", len(raw), " valid:", int((raw["problems"].str.len() == 0).sum() - raw["duplicate"].sum()))
raw.explode("problems")["problems"].value_counts()Rows: 1205 valid: 1133
problems
issue date not YYYY-MM-DD 34
line with zero or negative quantity 12
missing customer 7
discount above 20% 7
unknown customer 5
due before issued 5
Name: count, dtype: int64Plus the duplicates, flagged separately:
print("Exact duplicate rows:", int(raw["duplicate"].sum()))
print(raw.loc[raw["invoice_id"].isin(raw.loc[raw["duplicate"], "invoice_id"]), ["invoice_id", "customer_id", "issue_date", "amount_paid"]].sort_values("invoice_id").head(4).to_string(index=False))Exact duplicate rows: 5
invoice_id customer_id issue_date amount_paid
INV-100357 C0145 2026-07-04 425045.78
INV-100357 C0145 2026-07-04 425045.78
INV-100416 C0211 2026-08-25 116151.86
INV-100416 C0211 2026-08-25 116151.86Each kind of problem has its own owner and fix. Formatted dates can be converted with certainty, so they can be fixed automatically, like formatted amounts. Duplicates should be dropped, keeping one copy. Unknown customers, discounts above the limit and bad quantities go to the finance team for review: the code can't know the right answer.
Walkthrough
- Run the cells. Write a function that converts
31/08/2026to2026-08-31, and a test for it. - How many invoices have more than one problem?
- Why does the validator check
due_dateonly whenissue_dateis valid? - Write the import rules (the task below).
Practice
Practice
How many rows have a discount above 20%?
Task
6 minWrite the import rules for the invoice export, one per line starting with the problem and a colon: at least five problems, each saying whether it's fixed automatically, quarantined for review (and by whom), or rejected, and why.
Your work is checked for
- At least five problem lines
- Something fixed automatically
- Something quarantined or reviewed by a named team
- Duplicates handled
- Discount limit handled
- No silent dropping
Check your understanding
Answer every question to check.