Outlier Detection in Logistic Regression: A Quest for Reliable Knowledge from Predictive Modeling and Classification

Transcription

1 Outler Detecton n Logstc egresson: A Quest for elable Knowledge from Predctve Modelng and Classfcaton Abdul Nurunnab, Geoff West Department of Spatal Scences, Curtn Unversty, Perth, Australa CC for Spatal Informaton CCSI abdul.nurunnab@postgrad.curtn.edu.au g.west@curtn.edu.au

2 Objectves Identfcaton of multple nfluental observatons n logstc regresson Classfcaton of outlers on a graphcal plot Investgatng mportance of outler treatment for relable knowledge dscovery

3 Outler what, when and how? An outler s an observaton that devates so much from the other observatons as to arouse suspcons that t was generated by a dfferent mechansm. Hawkns 1980 Causes of outlers: Outlers occur very frequently n real data, and often go unnotced because much data s processed by computers wthout careful nspecton and screenng. They may appear because of human error such as keypunch errors, mechancal faults such as transmsson or recodng errors, changes n system behavor, exceptonal events natural dsasters such as earthquakes and floods, nstrument error, or smply through natural devatons n populatons. Outlers effects: The presence of outlers n a dataset may cause the parameter estmaton to be erroneous, msclassfyng the outcomes and consequently creatng problems when makng nferences wth the wrong model. Draws unrelable conclusons and decsons.

4 Outlers and elablty Issues The typcal steps consttutng the KDD process elablty ssue and outler nteract wth 5 questons [Fayyad et al. 1996; Da et al. 2012] :. What are the major factors that can make the dscovery process unrelable?. How can we make sure that the dscovered knowledge are relable?. Under what condtons can a relable dscovery be assured? v. What technques are there that can mprove the relablty of dscovered knowledge? v. When can we trust that the dscovered knowledge s relable and reflects the real data?

5 Logstc egresson Logstc regresson s useful for stuaton n whch we want to predct the response presence or absence of a characterstc or outcome based on values of a set of predctor varables. It can be classfed nto three types based on categorcal response varables: bnary, ordnary and nomnal. A bnary response has two categores wth no natural order for example, success-falure or yes-no. An ordnal response has three or more categores wth a natural orderng e.g. none, mld, and severe. A nomnal response has three or more categores wth no natural orderng for example, blue, black, red, yellow; or sunny, rany, and cloudy. In ths presentaton, we wll cover only the bnary logstc regresson.

6 Logstc egresson and Outler The customary model for L s: g π 1 π X = ln = where π = e 0 + e E Y X = π X The log of [π./1-π.] can be defned as a lnear functon called logt log odds of X logodds = β0 + β1 X1 + + β p X = β + β X β + β X + + β X p p p + + β X p, 0 π 1 The L model can be re-wrtten as: Y = π + ε where Y s a vector of bnary 0, 1 response and ε s the error term: 1 π ε = π wth wth probablty probablty π; 1 π; f f p y = 1, Xβ y = 0. Outlers, lnear and logstc S-curve models.

7 Types of Outlers Typcally outlers n regresson can be categorzed nto three classes: outlers, hgh leverage ponts and nfluental observatons. Devaton/change n X explanatory space, called leverage ponts Devaton n Y response varable not n X, called vertcal outlers Devaton n both X-Y spaces. Influental observatons are defned as ponts, whch ether Indvdually or together wth several other observatons, have a demonstrably larger mpact on the calculated values of varous estmates coeffcents, standard errors, t-values etc.. Belsley et al In logstc regresson, outlers and nfluental observatons may occur as msclassfcaton between the bnary 0, 1 responses. It may occur by meanngful devaton we also see low leverage n explanatory varables. Nurunnab et al. 2010

8 Outler Detecton: Sngle-case Deleton Approach The th resdual can be defned n L as: εˆ = y πˆ The projecton leverage matrx s a dagonal matrx that gves the ftted values of the response varable as the projecton onto the covarate space. It s defned as: H = V 1 / 2 X X TVX 1 X TV The th dagonal element of H defned as: ˆ ˆ 1/ 2 T T 1 h = π 1 π x X VX The standardzed Pearson resdual for L s defned as: y πˆ rs = v 1 h DFFITS n L defned as: DFFITS x where V s a dagonal matrx wth dagonal elements v = πˆ 1 πˆ h > ck/n c = 2 or 3 values are generally dentfed as hgh leverage ponts. k=p+1. r s 3 are generally dentfed as outlers yˆ yˆ = DFFITS > 3 k/n are dentfed as v h nfluental cases

9 Modfcaton for Group Deleton Approach The sngle-case deleton measures are naturally affected by the maskng and swampng phenomena and fal to detect outlers n the presence of multple outlers and/or nfluental cases. Maskng occurs when an outlyng subset goes undetected because of the presence of another, usually adjacent, subset. Swampng occurs when good observatons are ncorrectly dentfed as outlers because of the presence of another, usually remote subset of observatons. The group deleton approach forms a clean subset of the data that s presumably free of outlers, and then test the outlyngness of the remanng ponts relatve to the clean subset.

10 Outler Detecton: Group Deleton Approach = = = D D D V V V Y Y Y X X X 0 0,, ˆ exp 1 ˆ exp ˆ T T β x β x π + = ˆ 1 ˆ, ˆ ˆ π π v π y ε = = T T x X V X x π π h 1 ˆ 1 ˆ = + =, 1 ˆ, 1 ˆ * D for h v π y for h v π y r s + =. 1, 1 * D for h h for h h h are generally dentfed as outlers r * s 3 are dentfed as hgh leverage ponts 3 * * * h MAD h medan h + > Generalzed Standardzed Pearson esdual GSP Generalzed Weght GW These methods fnd a suspect group D of d outlyng/unusual cases wth the help of graphcal methods, robust technques such as LMS, LS and/or approprate dagnostcs measures. The data n explanatory varables X, response varable Y and the varancecovarance matrx V can be separated deleton group D and the clean set as:

11 Identfcaton of Multple Influental Observatons Mahalanobs Dstance MD = Z Z T Σ 1 Z Z where Z s an m varate wth mean Z and covarance matrx Σ. The proposed Influence Dstance ID: ID T G G Σ G G = 1 where G = [ r* * s h ] s the generalzed resdual-leverage matrx and and Σ are the mean and covarance matrx based on the group excludng the observatons are dentfed as outlers by GSP. G

12 Proposed Method Algorthm 1. * Calculate and usng the group deleton approach Construct the matrx Calculate and based on the group after the v v v deleton of outlyng cases Calculate Fnd nfluental observatons for whch ID 2 > χ 2,0.975 = To sketch the classfcaton plot : a draw an scatter plot r* s versus r* s h* b draw cut-off lnes at ±3 and * * 3 r* * h + MAD h s medan h Σ G ID G = [ * r * s h ] T G G Σ G G = 1 through the and h axes respectvely c draw an nfluence ellpse based on the ID values and the Ch-square cut-off value. Classfcaton plot.

13 Modfed Brown Data L.N.I. A.P. L.N.I. A.P. L.N.I. A.P. L.N.I. A.P Experment 1. Modfed Brown Data r s 3.00 h Dagnostc esults for Modfed Brown Data DFFITS r s * 3.00 h * ID r s 3.00 h DFFITS r s * 3.00 h * ID

14 Experment 1. Modfed Brown Data Modfed Brown data a scatter plot; L.N.I. versus A.P. b ndex plot of standardzed Pearson resdual c ndex plot of leverage values d ndex plot of DFFITS e ndex plot of GSP f ndex plot of GW g ndex plot of ID h classfcaton plot

15 Experment 2. Modfed Fnney Data Modfed Fnney Data Y Vol. ate Y Vol. ate Y Vol. ate Dagnostc esults for Modfed Fnney Data r s h DFFITS r s * h * ID r s h DFFITS r s * h * ID

16 Experment 2. Modfed Fnney Data Modfed Fnney data a character plot; rate versus volume wth the response values 1, 0 b ndex plot of standardzed Pearson resdual c ndex plot of leverage values d ndex plot of DFFITS e ndex plot of GSP f ndex plot of GW g ndex plot of logid h classfcaton plot.

17 elablty Checkng: Models Parameters Estmaton and Test Modfed Brown Data L model ft and sgnfcance test esults for all observatons esults wthout outlers Parameter estmaton Parameter estmaton Predctor Coef. S. E. Z P Odds 95% Conf. Int. Odds 95% Conf. Int. Coef. S. E. Z P ato Lower Upper ato Lower Upper Constant A.P Test Test Test that all slopes are zero: G = 0.183, df = 1, P-Value = Test that all slopes are zero: G = 7.31, df = 1, P-Value = Goodness -of-ft test Goodness -of-ft test Χ 2 df p χ 2 Df p Pearson Pearson Devance Devance Hosmer-Lemeshow H-L Hosmer-Lemeshow H-L Model Summery Model Summery Log-Lkelhood LL -2 LL Cox & Snell 2 Nagelkerke 2 Log-Lkelhood LL -2 LL Cox & Snell 2 Nagelkerke Predcted probabltes versus A.P.

18 Performance Evaluaton: Classfcaton Classfcaton results wth outlers and wthout outlers Predcted status Absence 0 Presence 1 Total Correct classf. All observatons Actual Status Absence Presence Total % % 100% % 0% 0% % % 55 Wthout outlers Actual status Absence Presence Total % 18.18% 63.64% % 23.64% 36.36% % 41.82% % 69.09% Mosac plot a classfcaton wth outlers b classfcaton wthout outlers

19 Performance Evaluaton: Predctve Ablty OC Curves results Area S. E. Sg. Asymptotc 95% Conf. Int. p Lower Bound Upper Bound All observatons Wthout outlers OC curve a data wth outlers b data wthout outlers

20 Conclusons Ths paper proposes a dagnostc measure for dentfyng multple nfluental observatons n logstc regresson. It ntroduces a classfcaton graph to classfy outlers, hgh leverage ponts and nfluental observatons n the same plot at one tme. Dagnostc results show that the proposed measure effcently dentfes multple nfluental cases, and the graph s helpful for vsualzng outler categores. esults show that wthout careful outler nvestgaton, t may not be possble to get relable knowledge usng logstc regresson for predctve modelng and classfcaton.

21 Conclusons The outler nvestgaton n logstc regresson s hghly related to the ssues rased for relable knowledge dscovery. Outler detecton s one of the major factors that affect the relablty of the dscovery process. The condtons for relable knowledge dscovery can be mproved by parameter estmaton and testng the sgnfcance of the estmates. Proper outler dagnostcs and treatment deleton or correcton of the outlyng observatons can mprove the relablty of dscovered knowledge. v We can trust the dscovered knowledge s relable and reflects the real data f the test results meet the requred statstcal sgnfcance level. Therefore outler detecton and proper treatment s vtal for obtanng relable knowledge, and should be consdered as a data preprocessng step n knowledge dscovery n databases KDD. The proposed dagnostc method s ntroduced for the bnomal response varable n logstc regresson. Future research wll nvestgate the dagnostc method for multnomal response varables, and large and hgh dmensonal data as hgher dmensonal data presents extra problems that need to be addressed.

22 Queston?