Module 11 · Machine Learning Fundamentals
"Final project: Ladder's credit model"
Plan your final project, a complete, responsible default model and lending recommendation for a microfinance bank, and start with the questions every credit model must answer.
About 25 minutes
The problem
Ladder Microfinance's board has approved a pilot: for six months, new applications will be scored by a model, and those above the threshold will be referred for review rather than approved automatically. The head of credit risk asks you to deliver the model and everything around it: the analysis behind it, the evidence it works, the threshold and what it's worth, the explanation for declined borrowers, the fairness checks, and the plan for monitoring it.
This is what a junior data scientist's first real project looks like. The modelling is perhaps a third of the work. The rest is making sure the model is honest, explainable, fair and useful, which is what this course has been about.
The concept
The project, step by step
| Step | Deliverable | Lesson |
|---|---|---|
| Frame | the prediction, the decision it changes, the measure of success | 1 |
| Prepare | features, leakage checks, engineered features such as amount ÷ revenue | 2 |
| Split and baseline | a stratified split and the "no default" baseline | 3, 7 |
| Model | logistic regression and at least one tree-based model, compared fairly | 4, 5, 7 |
| Tune | cross-validated settings, test set used once | 6 |
| Evaluate | AUC, precision and recall, and the profit-based threshold | 8 |
| Explain and check | permutation importance, reasons for declines, fairness checks | 9 |
| Deploy | a pipeline, a scored example, a model card, a monitoring plan | 10 |
What makes it responsible
- No leakage: every feature exists on the day of the application.
- No protected characteristics, and proxies tested and removed if they add nothing.
- A threshold chosen from business costs, with sensitivity shown.
- Reasons a borrower can understand.
- Monitoring with concrete triggers for review.
Example
Two warm-up questions that belong in your exploratory analysis. Group loans (where borrowers guarantee each other) and the loan's size relative to revenue are two of the strongest signals in the data:
import pandas as pd
loans = pd.read_csv("https://academy.cloudtechanalytics.com/datasets/loans/loans.csv")
loans["amount_to_revenue"] = loans["loan_amount_ngn"] / loans["monthly_revenue_ngn"]
print(loans.groupby("group_loan")["defaulted"].mean().round(3))
loans.groupby("defaulted")["amount_to_revenue"].median().round(2)group_loan
No 0.140
Yes 0.086
Name: defaulted, dtype: float64
defaulted
0 1.51
1 1.86
Name: amount_to_revenue, dtype: float64Group loans default less often, and borrowers who defaulted had typically borrowed a larger multiple of their monthly revenue. Both make business sense, which is reassuring: a model built on them will be easier to explain and to trust.
Walkthrough
- Frame the project in a short paragraph: what's predicted, the decision, and how success will be measured in the pilot.
- Build the features, check each one for leakage, and decide about region before you start modelling.
- Compare at least two models with cross-validation, choose one, and score the test set once.
- Build the profit table, choose the threshold, and show how it changes if a default costs 80% of the loan.
- Open the project brief on the course page and plan the rest: explanations, fairness, pipeline and model card.
Practice
Practice
What is the default rate of group loans? As a percentage, one decimal place.
Practice
What is the median amount-to-revenue ratio for loans that defaulted? Two decimal places.
Task
8 minWrite the project framing for Ladder's pilot in 60 to 150 words: what the model predicts, the decision it changes, who uses it, and how success will be measured after six months (with at least one number).
Your work is checked for
- Says what is predicted (probability of default)
- Says the decision it changes (approve, refer, decline, review)
- Names the users
- Success measure with a number
- Between 60 and 150 words
Check your understanding
Answer every question to check.