Class(ic) Scorecards

Size: px

Start display at page:

Download "Class(ic) Scorecards"

Phebe Morrison
6 years ago
Views:

Class(ic) Scorcards Slcting Charactristics and Attributs in Logistic Rgrssion Edinburgh Crdit Scoring Confrnc - 25 August 2011 Grard Scallan grard.

1 Class(ic) Scorcards Slcting Charactristics and Attributs in Logistic Rgrssion Edinburgh Crdit Scoring Confrnc - 25 August 2011 Grard Scallan grard.scallan@scorplus.com 1 Class(ic) Scorcards Using th Statistics! What s th Problm? Nstd Dummy Variabls Stpwis Mthod Slcting Charactristics Lssons Larnd 2

2 Exampl: Ag Charactristic Typical Analysis Layout CHARACTERISTIC: AGE SAMPLE COUNTS COLUMN % 0.5 WEIGHT OF INFORMATION GLOBAL Attribut Goods Bads Total Goods Bads Total EVIDENCE VALUE CHI² TOTAL % 100.0% 100.0% % 1.1% 0.5% % 1.9% 0.9% % 1.9% 0.9% % 2.8% 1.1% % 2.8% 1.2% % 3.0% 1.4% % 2.6% 1.3% % 2.8% 1.6% % 0.1% 0.4% WoE = LnOdds(attr) LnOdds(popn) IV = Avg G (WoE) Avg B (WoE) Information Valu: Chi² DF 47 p-lvl E-45 Goal of Classing Maximis prdictiv powr 3 WoE Graph: Show ovrall pictur W i g h t o f E v i d n c WoE - Estimation Sampl Lowr Bound Uppr Bound CUSTOMER AGE Outlir Problm: Tsting Wrong Hypothsis Var(WoE i ) = 1/G i + 1/B i 1/G total 1/B total Poor man s hypothsis tst! WoEi WoEi + 1 2StDv Rjct qual risk Sparat classs Equivalnt to 2 x 2 Chi² But ral hypothsis is not quality 4

3 Currnt Practic: Classing Currnt Practic Fin brakdowns on ach prdictiv charactristic Manual or Automatic Classing Basd on Information Valu or Chi² masur 1 dummy variabl pr class Slct modl variabls using stpwis Logistic Rgrssion 5 And what s wrong with it On charactristic at a tim Anomalis in on charactristic oftn xplaind by anothr Lots of prdictors Lots of tim 700 chars x 3 mins. = 35 hours Variabl slction in modl at attribut lvl gap toothd modls Ag 18-21, Ag in modl Ag not in modl Stpwis masurs crtainty Not distanc Good tchnical solutions but wrong problm Solution 1: Continuous Variabls Risk improvs continuously with Ag W i g h t o f E v i d n c WEIGHT OF EVIDENCE STRAIGHT LINE WoE Lowr Bound Uppr Bound CUSTOMER AGE Simplr Hypothsis 1 paramtr vs.15+ Data do not contradict th linar hypothsis In most cass But sampl slicd into many small catgoris Combin catgoris Mor rliabl tsts Slop changs ~ ag 30 Again ~ ag 50? Bttr Starting Point 6

4 Why Discrtis? W i g h t o f E v i d n c Non-Linaritis WEIGHT OF EVIDENCE Lowr Bound Uppr Bound Tradition 1960s Scors calculatd by hand No pockt calculators Multiplication lss rliabl than addition Cofficints 2 digit intgrs CUSTOMER AGE Slop changs ~ ag 30 Again ~ ag 50? Not quit discrt No longr justifid 7 Class(ic) Scorcards Using th Statistics! What s th Problm? Nstd Dummy Variabls Stpwis Mthod Slcting Charactristics Lssons Larnd 8

5 Partition Variabls a.k.a. Nstd Dummy Variabls Variabl Ag 18 Ag 19 Ag 20 Ag 21 Ag 22 Ag 23 P P P P P P Partition variabl for ach fin class P18 = intrcpt will not ntr modl Scor for 22 yar old = P18 + P19 + P20 + P21 Cofficint P22 = incrmntal chang for Ag 22 compard to Ag 21 Partition modl givs sam scor to ach individual as Attribut modl Partition and Attribut variabls = two bass for sam linar spac Monoton incrasing Partition Cofficints > 0 Diffrnt coding Sam modl 9 Varianc of Cofficints and Significanc Tsting MODEL 1 - TmBooks, DaysXS, Bounc, Autocrdit No. Charactristic Variabl Estimat Std. Error z-valu Pr(> z ) Significanc [95% Conf. Intrval] 0 (Intrcpt) *** TmBooks 2y6m < 2-16 *** TmBooks 7y1m E-08 *** TmBooks 14y1m *** DaysXS Any E-07 *** DaysXS * DaysXS DaysXS E-05 *** Bounc 1m ** Bounc 41m Bounc Nvr *** AutoCrdit Any ** AutoCrdit E-05 *** LogLiklihood Dvianc DF modl 13 AIC BIC DF rsidual 8081 Numbr of Fishr Scoring Itrations 3 Maximum Liklihood Estimats Std. Error from Covarianc Matrix of Estimats Z-valu = Estimat/Std. Error OR Wald Statistic = Z² 10

6 Z-tst and Wald Chi² Tst: Is this variabl ncssary? Z-tst Z-valu = Estimat/Std. Error If tru valu of Cofficint = 0 Null Hypothsis thn sampl valu of Z has Normal distribution Man = 0, Varianc = 1 (From thory of Max Liklihood) If Null Hypothsis is tru, thn unlikly to gt this big z OR If z is larg, data ar not consistnt with NH = 11 Wald Chi² Tst Z² = Estimat²/Varianc Undr Null Hypothsis Z² has Chi² Distribution w/ 1 DF Squar of N(0,1) Sam tst! Tst at 10%, 5%, 1%,.1% *** p < 0.1% ** p < 1% * p < 5%. p < 10% Larg sampl approximation asy to apply Hypothsis Tsts with Partition Variabls Attribut Dummy Variabls Rfrnc Attribut on vry charactristic Rcivs 0 scor Avoids linar indtrminacy Usually last attribut E.g. Ag 60+ Cofficint = 0 Risk sam as Rfrnc Attribut E.g. Risk on Ag = Risk on Ag 60+ Partition Dummy Variabls Cofficint = 0 Risk sam as nighbour to lft E.g. No diffrnc in risk btwn Ag and Ag What ar ky turning points in risk pattrn? Uslss hypothsis Ignor statistics Ky information 12

7 Automatd classing Provisional Solution Algorithm Partition Vars. for fin classs Must b ordrd snsibly Natural ordr or WoE Possibly variabls/ charactristic All charactristics in modl Candidats in stpwis Logistic Stpwis algorithm idntifis significant brakpoints Partition variabl ntrs iff significant diffrnc btwn nighboring attributs Advantags Lss work for analyst! Classing adapts to sampl siz Small sampl Coarsr Larg sampl Finr Accounts for intractions btwn charactristics Fwr classs/charactristic Multivariat approach Equivalnt to systmatic us of Marginal Chi² But approximations ar bttr! Avoids gap-toothd scorcards Gt minimal classing ndd for prdictiv structur 13 Continuous Variabls Picwis Linar Ida Analogous ida for continuous prdictors Family of splin variabls E.g. Ag (Ag 20) + = max(0,ag-20) (Ag 22) + = max(0,ag-22) (Ag 24) + = max(0,ag-24) tc. Candidats in stpwis Logistic Trms ntring corrspond to significant changs in slop a.k.a. MARS Multivariat Adaptiv Rgrssion Splins 14 Scor Exampl Scor = Ag (Yars).2 x Ag -.06 x (Ag 22) x (Ag 30) x (Ag 38) x (Ag 46) +

8 Class(ic) Scorcards Using th Statistics! What s th Problm? Nstd Dummy Variabls Stpwis Mthod Slcting Charactristics Lssons Larnd 15 Stpwis Approach Forward Slction Start with null modl Add variabls Until no furthr variabl adds significant prdictiv powr 3 variants Backward Elimination Start with all variabls Drop variabl which maks last contribution to liklihood Until no furthr variabl can b droppd without significant loss of prdictiv powr Bidirctional Start with null modl Add variabls At ach stp, chck to s if variabls can b droppd Thn chck to s if any variabl can b addd Until no variabl to b droppd AND No variabl to b addd Computation: Forward < Backward < Bidirctional 16

9 What s wrong with Stpwis? If this mthod had just bn proposd it would most likly b rjctd bcaus it violats vry principl of statistical stimation and hypothsis tsting Harrll 2001 Rgrssion Modling Stratgis, p. 56 Paramtrs stimats too larg Slcts ovrstimatd cofficints Ovrstimats prcision Bcaus undrstimats varianc Collinarity maks variabl slction arbitrary Lots of candidats Lots of nois in modl It allows us not to think about th problm 17 Stpwis Logistic on Random Numbrs Simulatd Exampl Variabls Entring Modl Gini Cofficint of Modl Sampl of "Bads" (1000 Goods) Similar to Flom & Cassll (2007) 1000 Goods Bads from 100 to candidat variabls All whit nois Random from Normal Distribution Ral prdictiv powr = rplications for ach sampl siz Entry/Exit critrion: p < % 50% 40% 30% 20% 10% 0% Sampl of "Bads" (1000 Goods) Max Mdian Min Rsults on stimation sampl Won t validat (w hop!) All modls hav Dvianc statistics w/ p-lvl < 0.1% 2/3 of variabls significant at 5% p- lvl Adds nois to modl

10 Class(ic) Scorcards Using th Statistics! What s th Problm? Nstd Dummy Variabls Stpwis Mthod Slcting Charactristics Lssons Larnd 19 Goal: Minimal Sufficint Modl Bring in nough variabls to xplain th variation in outcom across th sampl But no mor Tll a (snsibl) story End point: prdictiv powr of sampl is xhaustd 20

11 Marginal Information and Dlta Scors Dbit OBSERVED EXPECTED -scor Turnovr Goods Bads WoE Goods Bads WoE <= <= <= <= > Total Chi² = D.F. = 4 p-valu % Marginal Information Valu Wight of Evidnc (WoE) = log (Attribut Odds) log (Population Odds) On-dimnsional scor cofficints Dlta Scor = Obsrvd WoE Expctd WoE Approximation to scor coffts ndd to lin up xpctd with obsrvd Marginal Information Valu = Avg Good (Dlta Scor) - Avg Bad (Dlta Scor) Similar to Kullback-Liblr Information Valu Incrasd sprad btwn avrag scor of goods and bads if this charactristic brought into modl 21 Slcting Scorcard Charactristics DaysXsL6m ToB SincDish AutoCr CurDaysXs Charactristic IV Scor1 Scor2 Scor3 Scor4 Scor5 CurBal CurCTO CurDaysXs CurDTO CurValXs ToB MthsInact MthsNoCTO NtTO DaysDbL3m DaysXsL6m CurMxBal DishL1m DishL3m SincDish IntrCTO IntrDTO AutoCr ValDishL6m Rank charactristics by Marginal IV Charactristic with maximum MIV ntrs modl 22 i.. partition variabls bcom candidats for ntry to modl Continu until no SIGNIFICANT MIV lft

12 Marginal IV and Collinarity As ach variabl ntrs MIV on rmaining charactristics rducs Rduction masurs collinarity ovrlap in prdictiv powr Improprly calld corrlation Undrstand rlationships btwn charactristics through MIV dcay Frquntly idntify familis Or Factors If on mmbr ntrs modl, MIV drops svrly on othr mmbrs Choic of mmbr is arbitrary PARTITION INFO VALUE PARTITION MIV I N F O V A L U E CUSTOMER AGE Zro Marginal Information = Sufficint Statistic 23 Automatd Classing with Marginal IV Customr Ag Exampl INFO VALUE PARTITION INFO VALUE PARTITION MIV (30+) PARTITION MIV (44+) PARTITION MIV (27+) PARTITION MIV (48+) PARTITION MIV (48+) Variabl 1: 30+ Variabl 2: CUSTOMER AGE Comput Marginal Info Valu for ach partition Slct partition with max. MIV Chck Significanc Dvianc Tst 24 Rbuild modl w/ nw variabl R-stimat MIVs Continu until no significant MIV lft All charactristics procssd simultanously

13 Automatd Classing with Marginal IV Customr Ag Exampl - Compltion INFO VALUE PARTITION INFO VALUE PARTITION MIV (30+) PARTITION MIV (44+) PARTITION MIV (27+) PARTITION MIV (48+) PARTITION MIV (48+) Max. MIV Variabl CUSTOMER AGE Continu until all MIVs < variabls 6 classs -v MIVs Wrong dirction In ral lif, do all chars simultanously End of procss: Zro Marginal Information 25 Actual vs. Fittd WoE W i g h t o f E v i d n c Lowr Bound WoE - Estimation Sampl Uppr Bound FITTED WOE Bounds = 95% confidnc limits Nb Attributs Information Valu Chi² p-lvl Fin Classd E E-58 CUSTOMER AGE 26 Fw significant diffrncs btwn fittd and actual Diffrncs in nighbouring groups all significant at 95%

14 Tripl Tst Bottom Lin Marginal Information Valu = Importanc Distanc masur Rul of Thumb: < MIV > Ngativ valu indicats ovr-fitting R-xamin history of MIV to drop variabl from modl Marginal Chi² = Rliability Masur of crtainty Thousands of tsts - bwar of fals positivs Snsitiv to classing usd for analysis Mor robust to us Stpwis approach for classing Businss sns = Cohrnc Dos charactristic tll a blivabl story? Dos th modl mak sns Modl complt whn no furthr variabl satisfis ths 3 critria 27 Class(ic) Scorcards Using th Statistics! What s th Problm? Nstd Dummy Variabls Stpwis Mthod Slcting Charactristics Lssons Larnd 28

15 Conclusions Standard statistical tools can b usd bttr Corollary: W don t nd lots of spcial-purpos analysis softwar No statistical tool can tak ovr th burdn of sns-chcking modls 29 Marginal Analysis Confidnc intrvals on Dlta scors (asy) Marginal Information valus (hard) R-dsign charactristic analysis to focus on partition variabls Charactristic Analysis for Continuous Charactristics Splins Cf. Ross Gaylr Outstanding Issus Topics for Rsarch 30 Scorcard Estimation Stpwis typ algorithm using Marginal IV rathr than Dvianc masurs but also using significanc chcks Logistic Rgrssion with constraints Monotonicity Sign constraint Would liminat much ovrfitting through stpwis MORE POWER FROM STANDARD TOOLS USE THE STATISTICS!

16 Rfrncs Frank HARRELL (2001) Rgrssion Modling Stratgis (Springr, 2 nd dition) Ptr L. FLOM, David L. CASSELL (2007) Stopping stpwis: Why stpwis and similar slction mthods ar bad, and what you should us (NESUG 2007 North Eastrn SAS Usr Group) Alan AGRESTI (2012) Catgorical Data Analysis (Wily, 3 rd dition forthcoming). S Chaptr 15. Grard SCALLAN (2009) Marginal Chi² Analysis: Byond Goodnss of Fit for Logistic Rgrssion Modls ( Grard SCALLAN (2011) Building Bttr Scorcards (Scorplus, Cours Nots, 2011 dition Sctions 5, 7, 8; Sctions 8, 11 in oldr ditions) 31

Applied Statistics II - Categorical Data Analysis Data analysis using Genstat - Exercise 2 Logistic regression

Applied Statistics II - Categorical Data Analysis Data analysis using Genstat - Exercise 2 Logistic regression Applid Statistics II - Catgorical Data Analysis Data analysis using Gnstat - Exrcis 2 Logistic rgrssion Analysis 2. Logistic rgrssion for a 2 x k tabl. Th tabl blow shows th numbr of aphids aliv and dad