Handling Imbalanced & Messy Real-World Data

Four hundred and ninety-two frauds among 284,807 card payments: should a fraud team reweight them, duplicate them, synthesize more, or throw away most of the legitimate payments? Those are the counts in two days of European card traffic from September 2013 that the payment processor Worldline and the Machine Learning Group of the Université Libre de Bruxelles (ULB) released for research, a fraud rate of 0.172%, and Chapter 6 used them to show what to measure at such a rate: accuracy and ROC-AUC flatter a model, and precision falls with prevalence. Knowing what to measure is the start. The team still has to decide what a model's probability means after any rebalancing, how to pick the few hundred alerts a day that investigators can actually check, and whether labels that arrive days later, and only for the transactions someone looked at, can be trusted.