Evaluating classifiers and choosing thresholds

01The problem

Ladder's model gives every application a probability of default. The credit manager now has a decision to make: above what probability should an application be declined (or sent for extra checks)?

Set the line low and the bank declines many borrowers who would have repaid, losing their interest. Set it high and it lends to borrowers who default, losing the principal. Neither mistake is free, and they don't cost the same. The right threshold isn't a statistics question; it's a business one, and the BA or data scientist's job is to put naira on both kinds of mistake so the bank can choose.

02The concept

The confusion matrix

Measures from it

• Recall (sensitivity) = TP ÷ (TP + FN): of all the defaults, how many did we catch?
• Precision = TP ÷ (TP + FP): of the loans we flagged, how many really defaulted?
• Lowering the threshold raises recall and lowers precision. There's always a trade-off.

AUC: ranking quality, independent of threshold

The ROC AUC measures how well the model ranks risky loans above safe ones, across all thresholds: 0.5 is random guessing, 1.0 is perfect. Credit scoring models typically score 0.70 to 0.85. Use it to compare models; use the threshold analysis to make the decision.

Choosing the threshold from costs

For each possible threshold, simulate the decision on the test set:

• each approved loan that repays earns the bank its interest margin;
• each approved loan that defaults loses the bank part of the principal;
• each declined loan earns and loses nothing.

Add it up, and pick the threshold with the highest total. The assumptions (what a default really loses, what a good loan really earns) must come from the finance team, and you should show how the answer changes if they're different.

03Example

The setup from lesson 7, in one cell:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.metrics import confusion_matrix, precision_score, recall_score, roc_auc_score

loans = pd.read_csv("https://academy.cloudtechanalytics.com/datasets/loans/loans.csv")
loans["amount_to_revenue"] = loans["loan_amount_ngn"] / loans["monthly_revenue_ngn"]
features = ["region", "business_type", "borrower_age", "years_in_business", "monthly_revenue_ngn",
            "loan_amount_ngn", "amount_to_revenue", "term_months", "interest_rate_monthly_pct",
            "previous_loans", "previous_late_payments", "group_loan", "has_guarantor",
            "mobile_money_txns_per_month"]
X = pd.get_dummies(loans[features], drop_first=True, dtype=int)
y = loans["defaulted"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_train, y_train)
prob = model.predict_proba(X_test)[:, 1]
print("AUC:", round(roc_auc_score(y_test, prob), 3))

AUC: 0.76

Precision and recall at several thresholds:

for t in [0.10, 0.15, 0.20, 0.25, 0.30, 0.50]:
    flag = prob >= t
    print(f"threshold {t:.2f}: recall {recall_score(y_test, flag):.2f}  precision {precision_score(y_test, flag):.2f}  declined {flag.mean():.0%}")

threshold 0.10: recall 0.78  precision 0.22  declined 41%
threshold 0.15: recall 0.58  precision 0.27  declined 25%
threshold 0.20: recall 0.47  precision 0.34  declined 16%
threshold 0.25: recall 0.37  precision 0.41  declined 11%
threshold 0.30: recall 0.27  precision 0.43  declined 7%
threshold 0.50: recall 0.07  precision 0.65  declined 1%

Now put naira on it. Finance's assumptions: a loan that repays earns half its total interest as margin (after funding and operating costs); a default loses 60% of the amount lent.

test = loans.loc[X_test.index]
margin = test["loan_amount_ngn"] * test["interest_rate_monthly_pct"] / 100 * test["term_months"] * 0.5
loss = test["loan_amount_ngn"] * 0.6
outcome = np.where(y_test == 1, -loss, margin)   # what each loan earns or loses if approved

profit = {}
for t in [0.10, 0.15, 0.20, 0.25, 0.30, 0.40, 0.50, 1.01]:
    approve = prob < t
    profit[t] = outcome[approve].sum() / 1e6
pd.Series(profit).round(1)   # ₦ millions; 1.01 means approve everyone

0.10    52.7
0.15    60.1
0.20    66.1
0.25    68.9
0.30    64.0
0.40    59.2
0.50    53.6
1.01    46.7
dtype: float64

Approving everyone earns about ₦46.7m on these loans. Declining applications above a probability of 0.25 earns about ₦68.9m: roughly ₦22m more, from the same borrowers, by turning away about 1 in 10 applicants.

04Walkthrough

1. Run the cells. Print the confusion matrix at 0.25: confusion_matrix(y_test, prob >= 0.25).
2. Change the loss on default from 60% to 80% and rerun the profit table. Does the best threshold move?
3. Change the margin from 50% to 30% of interest. What happens now?
4. Write down the threshold you'd recommend, and how sensitive it is to finance's assumptions.

05Practice

06Check your understanding