Bài 6: Classification Evaluation: ROC/PR, Thresholding, Cost-Sensitive, Calibration
Series: Business Analytics Cheatsheet Series
Topics:
classificationroc-prthresholdingcalibrationcost-sensitiveSource: BA7 — Classification Evaluation. CRISP-DM position: Evaluation (technical + business), feeding Deployment.
Core claim: decisions depend on evaluation, not on accuracy. Asymmetric costs (fraud, churn, health) and class imbalance plus capacity constraints make metric choice a business decision, not a technical default.
1. Confusion Matrix & Core Rates
| Term | Type | Definition / Usage |
|---|---|---|
| Confusion matrix | Framework | 2×2 of Predicted vs Actual: TP, FP, TN, FN. Computed only after picking a decision threshold — change the threshold and all four counts move. Foundation for every derived metric. |
| TP / FP / TN / FN | Definition | TP: positive predicted positive. FP: negative predicted positive (false alarm). TN: negative predicted negative. FN: positive predicted negative (miss). |
| TPR / Recall / Sensitivity | Metric | TP/(TP+FN). Share of actual positives caught. The metric when misses are expensive. |
| FPR | Metric | FP/(FP+TN). Share of actual negatives falsely flagged. |
| TNR / Specificity | Metric | TN/(TN+FP) = 1 − FPR. Share of actual negatives correctly cleared. |
| Precision (PPV) | Metric | TP/(TP+FP). Share of flagged cases that are real. The metric when review capacity is limited. |
| NPV | Metric | TN/(TN+FN). Share of cleared cases that are truly negative. |
| Accuracy under imbalance | Anti-pattern | (TP+TN)/N. Misleading when positives are rare. Always predicting negative can look excellent and be useless. Prefer class-sensitive metrics and curves. |
2. Single-Number Metrics
| Term | Type | Definition / Usage |
|---|---|---|
| F-beta score | Metric (formula) | Harmonic mean of Precision and Recall, weighted by β. F₁ = 2·(Precision·Recall)/(Precision+Recall) when β = 1 (equal importance). β > 1 weights Recall higher — choose when misses are costly. β < 1 weights Precision. Useful for early model selection. |
| MCC — Matthews Correlation Coefficient | Metric (formula) | MCC = (TP·TN − FP·FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN)). Range [−1, 1]; 0 ≈ random, 1 = perfect. Robust single number even under imbalance. |
| MCC worked example | Example | 1,000 samples, 990 negative / 10 positive. A lazy model predicts all negative: TN = 990, FP = 0, FN = 10, TP = 0. Accuracy = 99% (looks great). F1 = 0 (TP = 0). MCC = 0 (or undefined) — correctly reports no predictive power. This is the argument for MCC in one table. |
| Balanced Accuracy | Metric (formula) | (Sensitivity + Specificity)/2. The lazy majority-class model scores (0+1)/2 = 0.5 — correctly identified as no better than random, against its 99% raw accuracy. |
| G-Mean | Metric (formula) | √(TPR · TNR). Geometric mean of Sensitivity and Specificity. Measures balance between class performances; penalizes majority-class dominance more sharply than the arithmetic mean. |
3. ROC & AUC
| Term | Type | Definition / Usage |
|---|---|---|
| Score distributions | Concept | Overlapping score histograms for positives and negatives. Sweeping the threshold across them traces every ROC/PR point. The overlap is the model’s limitation. |
| ROC curve | Chart | Receiver Operating Characteristic. Plot TPR vs FPR across every threshold from 0.0 to 1.0. Above the diagonal ⇒ better than random. |
| AUC-ROC | Metric | Area under ROC = P(score of a random positive > score of a random negative) — the model’s ranking ability. Computed pairwise: +1 if score(pos) > score(neg), +0.5 if equal, 0 otherwise, divided by (Total Positives × Total Negatives). |
| AUC properties | Property | Invariant to monotonic transforms of the score. Ignores prevalence and costs — this is both its strength (comparable across datasets) and its critical weakness for decisions. |
| Iso-TPR line | Concept | A horizontal line on the ROC plot — all operating points with the same recall. |
| Iso-cost line | Concept | All operating points achieving the same total mistake bill. Its slope is set by the cost ratio; the optimal threshold is where an iso-cost line is tangent to the ROC curve. |
| Cost asymmetry examples | Example | Cancer screening: FP = $50 (anxiety + extra tests); FN = $10,000 (missed detection). ⇒ push recall up. Spam filtering: FP = $1,000 (interview invitation lost to spam); FN = $1 (annoyance). ⇒ push precision up. Same model, opposite thresholds. |
| Youden’s J | Metric (formula) | J = Sensitivity + Specificity − 1 = TPR − FPR. Range 0 to 1: J = 1 perfect, J = 0 no discriminative ability. Geometrically, J is the vertical distance from the ROC point to the 45° diagonal. |
| ROC cut via Youden’s J | Technique | Optimal cut-off = the ROC point with the largest vertical distance from the diagonal. Valid only when costs are equal and prevalence ≈ 50%. |
When ROC misleads
| Scenario | Why ROC misleads | Use instead |
|---|---|---|
| Highly imbalanced data | FPR is insensitive — it hides the absolute FP burden, producing an overly optimistic AUC. | Precision-Recall curve |
| Unequal misclassification costs | ROC treats all errors equally. | Cost-benefit analysis or explicit error weighting |
| Need for accurate probabilities | AUC measures only ranking, not probability accuracy. | Calibration plots / reliability diagrams |
Rule of thumb: ROC is often the wrong tool for highly imbalanced classification. Two ROC-similar operating points can differ enormously in economics.
4. Precision-Recall & Average Precision
| Term | Type | Definition / Usage |
|---|---|---|
| PR curve | Chart | Plot Precision vs Recall across every threshold from 0 to 1. Designed for imbalanced binary classification. |
| PR baseline | Reference | Baseline precision equals the prevalence. A PR curve without its prevalence baseline drawn is uninterpretable. |
| AP — Average Precision | Metric (formula) | AP = Σ_k P(k) · ΔR(k) — weighted mean of precision at each recall increase. Emphasizes early retrieval quality, which is exactly what matters under limited review capacity. |
| AP vs PR-AUC | Clarification | Used interchangeably in practice; AP is a precise way to compute the area under the PR curve. |
| Worked PR table | Example | At τ = 0.8: TP 17, FP 0, FN 23, TN 160 → Precision 1.00, Recall 0.425. τ = 0.6: 34/3/6/157 → 0.919 / 0.850. τ = 0.4: 40/26/0/134 → 0.606 / 1.000. τ = 0.3: 40/55/0/105 → 0.421 / 1.000. Note precision collapsing as recall saturates. |
| Simulated example | Example | AP ≈ 0.810 against a prevalence baseline of 0.08. The early high-precision region is the valuable part when review capacity is limited. |
ROC vs PR — when to use
| Use ROC when | Use PR when |
|---|---|
| Ranking ability is the question | Positives are rare |
| You need prevalence-invariant comparison across datasets | FP costs matter operationally |
| Classes are roughly balanced | Review capacity constrains action |
Report both for completeness.
5. Threshold Selection
| Term | Type | Definition / Usage |
|---|---|---|
| Threshold (τ) | Concept | Converts a score into a decision. Example: score 0.75. With τ = 0.5 ⇒ “Spam”. With τ = 0.8 ⇒ “Not Spam”. Same model, opposite action. τ should reflect objectives: F1, profit, recall@k, SLA, regulation. |
| Threshold by F1/Fβ | Technique | Maximize F1 on a validation set for balanced precision/recall. Use β > 1 when misses are high-risk (“rather flag 10 healthy patients than miss one sick one”). Avoid leakage — never tune τ on the test set. |
| Threshold by ROC geometry — closest to (0,1) | Technique | Minimize Euclidean distance √((FPR−0)² + (TPR−1)²). Finds a point good at both metrics simultaneously. |
| Threshold by ROC geometry — Youden’s J | Technique | Maximize J = TPR − FPR. The balanced choice when costs are equal and classes balanced. |
| Threshold by iso-cost slope | Technique | Otherwise, find where the iso-cost line (slope from the cost ratio) is tangent to the ROC curve. This is the cost-aware generalization of the above. |
| Threshold by precision target | Technique | Constrain precision ≥ P₀, maximize recall. Example: expensive-product marketing — only contact leads you’re ≥90% sure about; accept missing some. |
| Threshold by recall target | Technique | Constrain recall ≥ R₀, maximize precision. Example: airport security or critical medical diagnosis — never miss a threat, accept manual review of false alarms. |
| Top-k / quota thresholding | Technique | “I don’t know what τ should be, but I know how many items I can action.” Rank by score, take the top k or top q%. Example: $10,000 mail budget at $2/mailer ⇒ k = 5,000. Evaluate with precision@k, recall@k, lift. This is the most common real operating mode and is often overlooked. |
6. Cost-Sensitive Evaluation
| Term | Type | Definition / Usage |
|---|---|---|
| Cost matrix | Framework | Assign a business value or cost to every cell. Typical: TP = gain (e.g. +$100 from a saved sale) · FN = missed opportunity (−$100 lost profit) · FP = wasted action (−$5 for a useless ad) · TN = $0 (correct “do nothing”). Building this table is a stakeholder workshop, not a modeling task. |
| Expected cost at a threshold | Method | For each candidate τ (0.01 … 0.99): compute the confusion matrix, then total expected cost from the cost matrix. Plot the result; the minimum of the curve is the optimal threshold. |
| Bayes-optimal threshold | Formula | Assuming zero cost for correct predictions: τ* = C_FP / (C_FP + C_FN). Decision rule: if p = P(Y=1 given X) > τ* predict Positive, else Negative. Note it depends only on the cost ratio. |
| Cost curve | Chart | Total expected cost (or profit) plotted across every τ. The shape shows how sensitive the decision is to threshold error — a flat basin means τ choice is low-risk; a sharp V means it is not. |
| Risk & regulatory constraints | Constraint | Risk determines C_FP and C_FN. Spam filter: C_FP high (angering a customer who missed an important email). Fraud detection: C_FN very high (losing the transaction amount). Add hard limits: max FPR, min precision, fairness bounds. Solve as a constrained optimization on validation — find the minimum-cost point satisfying all constraints. |
| Cost-sensitive learning vs post-hoc | Method choice | In-training: class weights, focal loss, custom utility. Post-hoc: tune τ using costs. Often combine both. |
| Imbalanced data handling | Practice | Report PR, precision@k, calibration. Use stratified CV and keep the test distribution unchanged. Apply resampling or class weights to training only — never to the evaluation set. |
7. Lift & Gain Charts
| Term | Type | Definition / Usage |
|---|---|---|
| Cumulative Gain chart | Chart | Y-axis: cumulative % of positives captured (0–100%). X-axis: fraction of population targeted, sorted by score descending. Baseline: the 45° diagonal. Answers “how many responders do we capture?” Reads as: “targeting 20% captures 70% of all responders.” |
| Lift chart | Chart | Y-axis: lift factor (1x, 2x, 5x). Baseline: horizontal line at Y = 1. Answers “how many times better is our model than random?” Reads as: “targeting 20% is 3.5× better than random.” |
| When to use | Usage | Marketing top-k selection and any capacity-constrained targeting. The natural language for non-technical stakeholders — easier to defend in a budget meeting than AUC. |
8. Probability Calibration
| Term | Type | Definition / Usage |
|---|---|---|
| Calibration | Concept | A model’s 70% confidence actually corresponds to a 70% real-world chance. Of 100 items scored 0.8, about 80 should be positive. |
| Ranking vs calibrated probability | Distinction | A ranking model tells you A is riskier than B — fine for prioritizing (“call the 1,000 riskiest”). A calibrated model tells you A is 80% and B is 30% — which changes the action: expensive retention offer for A, low-cost email for B. Uncalibrated models that report both as “80%” waste money on B. Random Forests and neural nets are commonly over-confident. |
| Calibration and cost thresholds | Consequence | Business rule “review any transaction with >10% fraud chance” requires τ = 0.1. If the model is uncalibrated, its 0.1 might mean 30% real risk ⇒ the business rule is silently not implemented. Miscalibration ⇒ wrong ROI, wrong capacity planning. |
When calibration matters
| Scenario | Need calibration? | Why |
|---|---|---|
| Ranking contest (Kaggle-style) | No | Only the order matters (AUC, Lift). |
| Selecting “Top 100” users | No | Pure ranking and selection task. |
| Calculating ROI / profit | Yes — critical | The probability value enters the financial formula. |
| Setting insurance premiums | Yes — critical | The premium is a direct function of the predicted probability. |
Diagnostics
| Term | Type | Definition / Usage |
|---|---|---|
| Reliability curve | Chart | Bin predicted probabilities (e.g. 10 bins), plot observed rate vs predicted rate per bin. Best case: on the diagonal y = x. An “S” shape indicates over-confidence. Show counts per bin for stability context. |
| Brier score | Metric | Proper scoring rule — MSE applied to probability predictions. Measures both discrimination and calibration. Closer to 0 is better. |
| Log loss | Metric | Cross-entropy. Proper scoring rule; also the standard training loss for logistic regression. Penalizes confident errors heavily. Closer to 0 is better. |
| ECE — Expected Calibration Error | Metric | Average deviation of the reliability curve from the diagonal, over the bins. Closer to 0 is better. |
| MCE — Maximum Calibration Error | Metric | Worst-case bin deviation. Use when a single badly-calibrated region is unacceptable (regulated decisions). |
| Cross-validation for calibration | Practice | Train the calibrator on held-out data or out-of-fold predictions — it must see the base model’s predictions on data the base model was not trained on. Then evaluate the whole pipeline on a third, untouched set (data A: model, data B: calibrator, data C: evaluation). |
Before/after calibration — typical impact
| Metric | Before | After | Δ | Meaning |
|---|---|---|---|---|
| Brier Score | 0.135 | 0.110 | −0.025 | Probability MSE down; probabilities more accurate. |
| ECE | 0.092 (9.2%) | 0.014 (1.4%) | −0.078 | The biggest gain — average deviation from the diagonal collapses. |
| Log Loss | 0.450 | 0.390 | −0.060 | Penalty for confident-but-wrong predictions reduced. |
| AUC | 0.850 | 0.850 | ~0 | Unchanged — calibration does not alter the ranking. |
Effect on thresholding
| Model | How to find τ* | Meaning of τ* |
|---|---|---|
| Calibrated | Theoretical: τ* = C_FP/C_FN-derived from costs | A real business risk level (e.g. 15%) |
| Uncalibrated | Empirical: search the cost curve | An arbitrary number (e.g. 0.28) that happens to minimize validation cost |
Fix calibration first if scores are unreliable — otherwise every downstream cost calculation is built on a mislabeled quantity.
9. End-to-End Workflow & Ops
Train → Evaluate (ROC/PR) → Calibrate → Choose τ → Back-test
↳ check fairness, capacity, stability
| Practice | Type | Definition / Usage |
|---|---|---|
| Fix SRM before evaluating | Ops | Sample Ratio Mismatch in an experiment invalidates the comparison. Check assignment balance first. |
| Stratified splits | Ops | Preserve class balance across folds; mandatory under imbalance. |
| Track drift, recalibrate periodically | Ops | Prevalence shifts move the optimal threshold even when the model is unchanged. |
Common pitfalls
| Pitfall | Why it fails |
|---|---|
| Reporting AUC only | Ignores prevalence, costs, and probability accuracy. |
| Choosing τ on the test set | Leakage; reported performance is optimistic. |
| Ignoring capacity constraints | An optimal threshold that flags more cases than the team can review is not optimal. |
| Not communicating trade-offs | Stakeholders cannot approve a threshold they don’t understand. |
Communicating to stakeholders
- Show ROC and PR plots with the chosen operating point marked.
- Present a cost table and capacity impact for 2–3 τ scenarios, not one recommendation.
- Include a what-if analysis for prevalence shifts.
10. Implementation hints (Python)
from sklearn.metrics import (
roc_curve, precision_recall_curve, average_precision_score,
brier_score_loss, matthews_corrcoef, balanced_accuracy_score,
)
from sklearn.calibration import CalibratedClassifierCV
fpr, tpr, thr_roc = roc_curve(y_val, p_val)
prec, rec, thr_pr = precision_recall_curve(y_val, p_val)
j_star = thr_roc[(tpr - fpr).argmax()]
tau_star = C_FP / (C_FP + C_FN)
cal = CalibratedClassifierCV(base_model, method='isotonic', cv=5).fit(X_cal, y_cal)
Evaluation checklist / Definition of Done
- Stratified splits; no leakage; τ chosen on validation, never on test.
- ROC and PR reported (AUC and AP), with the PR prevalence baseline drawn.
- Class-sensitive summary metric reported (MCC / Balanced Accuracy / G-Mean), not accuracy alone.
- Calibration diagnostics run (reliability curve, Brier, ECE) — mandatory if probabilities enter a financial formula.
- A validated cost/benefit model exists, built with the business owner.
- Threshold selected under explicit constraints and review capacity.
- 2–3 threshold scenarios presented with cost and capacity impact, plus a prevalence-shift what-if.
- Drift monitoring and a recalibration cadence scheduled.
References: Saito & Rehmsmeier (2015), PR vs ROC · Flach (2016), ROC analysis · Niculescu-Mizil & Caruana (2005), Calibration.
🔗 Cùng series
Bài 5: Linear Regression (Simple & Multiple)
Bài 7: Segmentation & Dimensionality Reduction
Nếu bài này hữu ích, hãy chia sẻ cho người đang học Data Science / Business Analytics. Mọi góp ý về lỗi kỹ thuật rất được hoan nghênh.