Stochastic Gradient Boosting Approach to Daily Attrition Scoring
Transcription
Stochastic Gradient Boosting Approach to Daily Attrition Scoring
Stochastic Gradient Boosting Approach to Daily Attrition Scoring Based on High-dimensional RFM Features Dr. Gerald Fahner Senior Director Analytic Science, FICO © 2015 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. Agenda • Ultra-dynamic Attrition Scoring • Case Study—Credit Card Attrition • Category Attrition © 2015 Fair Isaac Corporation. Confidential. 2 Ultra-Dynamic (Daily) Attrition Scoring Approach Customer uses card Daily attrition risk score • Prolonged inactivity signals higher risk—drives up attrition risk score Re-engage customer when attrition risk exceeds some threshold … Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We … © 2015 Fair Isaac Corporation. Confidential. 3 Transaction Dynamics Hold Key Information • Given information at time of scoring, who is more likely to attrite? ─ • Which measures are most informative? How to combine Recency and Frequency into predicting attrition risk? Recency Spence Observation Period Attrite? Frequency Time of Scoring Attila Observation Period © 2015 Fair Isaac Corporation. Confidential. 4 Attrite? Recency: Days since last card use Frequency: Fraction of days card used during obs. period How Machine Learning Complements Domain Expertise Domain Expertise Machine Learning Good at intuiting key predictors 1 Doesn’t scale to many variables Lacks intuition 2 Poor at combining multiple predictors Poor at quantifying uncertainty Need story behind the numbers 4 3 Excels at combining many features into accurate probabilistic predictions Diagnose and visualize models to gain insight into effects # Recommended path © 2015 Fair Isaac Corporation. Confidential. 5 Key Elements of Approach Based on Recencies, Frequencies, Monetary values • High-dimensional feature space of complex events Featurization of transaction events • Machine learning / classification tools • • Stochastic Gradient Boosting Partial dependence visualization Performance evaluation • • Lift related to portfolio profit gain Out-of-sample / Out-of-time evaluation © 2015 Fair Isaac Corporation. Confidential. 6 Stochastic Gradient Boosting[1] Combines predictions from 100’s or 1000’s of shallow CARTs Training Data Prediction Function CART 2 Weighted average Score Outcomes CART 1 Scored ? New case CART M Predictors Inexplicable model by direct inspection © 2015 Fair Isaac Corporation. Confidential. 7 Predictors Agenda • Ultra-dynamic Attrition Scoring • Case Study—Credit Card Attrition ─ • Machine Learning for Higher Profit Category Attrition © 2015 Fair Isaac Corporation. Confidential. 8 Credit Card Case Study Data and Project Design ~5 million accounts. More than 1 billion transactions over 3 years • Transaction information: Date, Merchant Code, Amount, Authorized Flag • 2 years Model development 6 months Performanc e period Observation period Time of Scoring Out-of-Time validation Observation period Performance period Attrition Performance Definition Scoring Exclusions Binary indicator of card activity during Performance period Inactive © 2015 Fair Isaac Corporation. Confidential. 9 Statistical Measures of Model Performance Lift and Precision Target top α % High Scores with retention offer Would-be attriters Non-attriters λ= = Low Scores © 2015 Fair Isaac Corporation. Confidential. Lift at α % operating point: 10 Fraction of Attriters Among Targeted Base Attrition Rate Precision Base Attrition Rate # Attriters Among Targeted # Targeted = # Attriters Total # Total ( ) Profit from a Retention Campaign Actual Behavior of Targeted Customer Profit Contribution per Customer Would-be attriter we persuade to stay (CLV Gain – Contact Cost – Incentive Cost) Precision * Persuasion Rate Unpersuadable attriter (No CLV Gain – Contact Cost) Precision * (1–Persuasion Rate) Non-attriter, erroneously targeted (No CLV Gain – Contact Cost – Incentive Cost) © 2015 Fair Isaac Corporation. Confidential. 11 Fraction of Targeted Customers with this Behavior 1–Precision Profit Gain From Attrition Model Improvement[2] Gain = (λB − λ A ) Nαβ 0 (γ CLV + δ (1 − γ )) is Portfolio Profit Gain from improving model B over model A, where : λA Lift from model A λB Lift from model B α β0 Targeting Fraction 5% Base Attrition Rate 8% N CLV δ γ Portfolio Size 5 million Customer Lifetime Value $1,000 Incentive Cost $100 Persuasion Rate 20% © 2015 Fair Isaac Corporation. Confidential. 12 Will benchmark alternative models Portfolio-specific assumptions Benchmarking Predictive Models of Increasing Complexity • How much can we gain by making models more complex? • Are complex models robust over time? Model 3: Interaction model in R and F of complex events Model 2: Interaction model in R and F of card use Model 1: Additive model in R and F of card use Complex Event Examples • • • R: Recency F: Frequency Recent restaurant visit and frequent hotels More than $1,000 spent on travel last week Recent car deal and frequently at the pump Dimensionality of Feature Space © 2015 Fair Isaac Corporation. Confidential. 13 Interaction Detection Experiment Should Capture (Recency X Frequency) Interactions • Predictors: Recency and Frequency of card use ─ Model 1: Additive, nonlinear in R and F ─ Model 2: Captures interaction between R and F Out-of-sample / λ = 6.03 1 Out-of-time ⇒ λ2 = 6.54 validation • Gain = $2.86 MM s.t. portfolio assumptions Interaction effect in agreement with research by Fader and Hardie[3] © 2015 Fair Isaac Corporation. Confidential. 14 Interaction Visualization Tells Story Two-dimensional Partial Dependence Function[4] Probability to use card during next 6 months = 1–Pr(Attrition) Attila Spence Attila is at higher risk of attrition because his card use has lapsed for an unusually long time interval ? Spence: R=20, F=0.05 ? Frequency Recency Fraction of days card used Days since last card use © 2015 Fair Isaac Corporation. Confidential. 15 Attila: R=20, F=0.55 Featurization Experiment Should Capture Complex Events in Your Models • Define R and F features for complex events • Model 3: Candidate predictors include: Card use events + Hundreds of merchant category events + Monetary events defined by spending bands + No-authorization events Out-of-sample / Out-of-time validation λ3 = 7.52 Recall : λ1 = 6.03 λ2 = 6.54 ⇒ Gain over Model 1 (simple, additive) = $8.34 MM s.t. portfolio assumptions © 2015 Fair Isaac Corporation. Confidential. 16 Learning Curves Experiment Should Exploit Larger Samples to Develop More Complex Models Lift (O-o-S / O-o-T) Model 3 (high-dim complex events) 7 6 Model 2 (card R and F only) 5 1,000 © 2015 Fair Isaac Corporation. Confidential. 17 10,000 100,000 #Training Samples Agenda • Ultra-dynamic Attrition Scoring • Case Study—Credit Card Attrition • Category Attrition ─ Detecting Subtle Forms of Attrition © 2015 Fair Isaac Corporation. Confidential. 18 Merchant Category (MC) Attrition • Hundreds of credit card MC’s • Performance definition for a specific MC: ─ • Stop buying from this MC–while continuing card use for other MC’s May signal competitive influence or early belt-tightening—before total attrition occurs. Quick detection informs rapid intervention Card-level model Overall customer status Grocery model Grocery status Travel status Travel model Gas station status Gas station model © 2015 Fair Isaac Corporation. Confidential. 19 Possible interventions: Offer incentives at service stations, or start customer dialogue Summary • Daily attrition scoring quickly detects emergent attrition—signaled by unusually long time lapse since last transaction • With large transaction volumes, more complex models are more profitable • Machine learning helps with insight, automation, scale © 2015 Fair Isaac Corporation. Confidential. 20 References [1] Greedy Function Approximation: A Gradient Boosting Machine, by Jerome Friedman, The Annals of Statistics, 29(5), 2001, 1189-1232. [2] Defection Detection: Measuring and Understanding the Predictive Accuracy of Customer Churn Models, by Scott Neslin et al., Journal of Marketing Research, 43(2), 2006, 204-211. [3] RFM and CLV: Using Iso-Value Curves for Customer Base Analysis, by Peter Fader, Bruce Hardie, and Ka Lok Lee, Journal of Marketing Research, 42(4), 2005, 415-430. [4] Predictive learning via rule ensembles, by Jerome Friedman et al., The Annals of Applied Statistics, 2(3), 2008, 916-954. © 2015 Fair Isaac Corporation. Confidential. 21 Thank You Dr. Gerald Fahner ++1 512 5323621 [email protected] © 2015 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent.