Indices of Distances: Characteristics and Detection of Abnormal Points

Size: px

Start display at page:

Download "Indices of Distances: Characteristics and Detection of Abnormal Points"

Sabrina McCarthy
5 years ago
Views:

1 Iteratioal Joural of Mathematics ad Computer Sciece, 8(2013), o. 2, M CS Idices of Distaces: Characteristics ad Detectio of Abormal Poits Hicham Y. Abdallah Departmet of Applied Mathematics Faculty of Sciece-1 Lebaese Uiversity Hadath, Lebao habdalah@ul.edu.lb (Received September 5, 2012, Accepted November 1, 2013) Abstract Gettig a robust regressio requires the detectio of abormal poits. I this article we give a solutio to this problem based o Cook s distace, DFFITS, DFBETAS, amog others. We the compare the bouds for those distaces which are used to detect abormal poit. 1 Itroductio Statistics is the sciece whose object is to collect, process ad aalyze data from the observatio of radom pheomea, that is to say where the accidet occurs. Data aalysis is used to describe the pheomea, make predictios ad decisios about them. For this, several statistical methods are available for these studies, but the most used is the regressio method. Despite its effectiveess, the problem of ifluetial poits ad outliers makes it less robust ad affects its optimum results. Our objective is to solve this problem by showig the differece betwee the ifluetial poits ad outliers. We begi our study with a defiitio of ifluetial poits ad outliers ad the we discuss the differet methods of detectio of abormal values. I Key words ad phrases: Outliers, ifluetial poits, distaces. AMS (MOS) Subject Classificatios: 62J07,62J12. ISSN c 2013,

2 56 H. Y. Abdallah additio, we propose a theoretical compariso betwee the differet idices to fid the most effective idex that helps us detect abormal poits. 2 Ifluetial Poits ad Outliers The study of residues y i yi ca idetify outliers observatios or commets that play a importat role i determiig regressio, where y is the predictio of y. The two types of abormal poits are outliers ad ifluetial poits. The first correspods to observatios outside the orm, while the secod to poits that weigh (urealistically) o estimates: if removed, the results would be differet sigificatly. There are several methods for detectig these values accordig to their types. Oce these observatios have bee idetified, it may be better to remove them or use other more robust criteria. 1 - Ifluetial Poits: A ifluetial item weighs heavily i the regressio; that is to say, the results are quite differet depedig o whether or ot the poit is take ito accout i the regressio. The problem of ifluetial values arises i busiess especially where surveys that collect ecoomic variables have distributios that are highly osymmetric. Ifluetial values are problematic because they geerally lead to ustable estimators. I other words, icludig or excludig a ifluetial sample value usually has a sigificat impact o the volatility of a estimator. It is possible to miimize their impact through a appropriate samplig pla. However, it is geerally ot possible to completely elimiate the problem of ifluetial values at each step of the pla. As a result, it is importat to develop robust estimatio methods i the presece of ifluetial values. 2 - Outliers: Before presetig cocepts related to outliers, it is ecessary to defie them more precisely as may authors have attempted to describe them ad the defiitios have chaged over time. Amog these authors, Grubbs has defied these values as follows: A outlier is a observatio that appears to deviate markedly compared to all other members of the sample i which it appears. I 1994, Barett ad Lewis defied a outlier i a data set as a observatio (or set of observatios) which appears to be icosistet with other data. That is, outliers correspod to observatios outside the orm of the populatio studied. These poits ca

3 Idices of Distaces 57 distort the results of the regressio. 3 The methods for detectio of abormal values 1 - Detectio of outliers: The idices that help us detect outliers are: a-stadardized Residue: e i This residue compares y with y the residue i the presece of the ith observatio. A abormally large value of a stadardized residual is cosidered suspected. More precisely, the model is correct 95% of the time if oe has: e i t(0.025; p) or e e i i = s. 1 h ii The observatios that qualify as outliers belog to the same regio: e i t(0.025; p). b-studet residue: e i This residue compares y with y without the ith observatio. I this case, a value of abormally large Studet residue is cosidered suspected. For such observatios we have: e i >t(0.025; p 1). I practice, we use the followig formulas: where s: stadard deviatio e i = e ( i) s( i) 1 h ii, e i = y i y i h ii = x i (X X) 1 x i where x i is the ith row of X, e ( i) is the Residue without the ith observatio, ad s ( i) is the Stadard Deviatio without the ith observatio.

4 58 H. Y. Abdallah The previous two criteria cotribute to detect potetially ifluetial observatios by their distace to the size of the residues. This iformatio is sythesized i the criteria directly assessig the ifluece of a observatio o some parameters. All these idicators suggest comparig a estimated parameter without the ith observatio ad this same parameter is estimated with all the observatios. C-treatmet of outliers We have already metioed the mai methods of outlier detectio to fially get to address these issues ad have a specific regressio model. Several methods of treatmet are: Reject: robustly elimiate extreme values determied followig the test mismatch. Icorporate: chage the distributio model i order to icorporate outliers. Idetify: keep outliers, sice they may represet particularly importat features. Accommodate: adopt statistical methods that miimize the impact of outliers o the statistical aalysis. 2-Detectio of ifluetial poits: The idices that help us detect ifluetial poits are as follows: a-leverage poit: I regressio aalysis, we call a Leverage poit a observatio i that sigificatly affects the estimators because its values over other variables differ much of the rest of the data ad the idicated distace betwee observatio i ad the ceter of gravity of the cloud of poits. Leverage poit h ii observatio is read from the mai diagoal of the matrix Hat Matrix, ad it appears as a measure of the ifluece of the ith observatio o its proper predictio. I practice, a observatio i is cosidered as a poit Leverage poit if h ii > 2(p+1). Note also that a observatio h ii approachig 1 is a observatio with a very importat Leverage poit. b-cook s distace: Cook s distace from a observatio is a measure of the ifluece of this observatio o all the set of predictios of a model. Oe calculates a distace betwee the vector β of coefficiets of the regressio ad the vector β(i) obtaied by repeatig the regressio without observatio i. Whe Cook s distace is ormalized, a value greater tha 1 is likely. 4 However, a limit of is ofte better as the calibratio of 1 ca p 1 permit ifluetial values.

5 Idices of Distaces 59 I practice, we have the followig two formulas i this case D i = i=1 (ŷ i ŷ i ( i)) 2 σ 2 ɛ (p +1) D i = ( β β ( i) ) (X X) 1 ( β β ( i)) σ ɛ 2 (p +1) c-dfbetas: The purpose of this method is to measure the ifluece of a poit o the estimated coefficiet. It is ormalized so as to be comparable from oe variable to aother. A suspected observatio is such that DFBETAS> 2 (). Note: If there are may variables, we first cosider the globally ifluetial observatios (Cook) ad the for this observatio the variable(s) causig that ifluece (DFBETAS). I practice, (DFBETAS) j i = β j β j ( i), s( i) (X X) 1 j,j where (X X) 1 j,j is the jth positio of the diagoal of (X X) Other criteria: a-the DFFITS: The DFFITS of a observatio is a measure of the ifluece of this observatio o predictio of its eigevalue by the model. It gives the differece betwee the adjusted value for observatio i ad the predicted value of y for i i the estimated model without this observatio i. We cosider a observatio is ifluetial whe: p +1 (DFBETAS) i > 2 or (DFBETAS) i > ŷi ŷ i ( i) s( i) h ii = e i hii 1 h ii b-covratio: The COVRATIO measures disparities betwee the precisio of the estimators; that is to say, the geeralized variace of estimators give by

6 60 H. Y. Abdallah Var( β) =s 2 det(x X) 1. The presece of the observatio i improves the accuracy i the sese that it reduces the variace estimators if COVRATIO> 1 Istead, COVRATIO <1 idicates that the presece of the observatio i degrades the variace. But the most commo detectio rule is: 3(p +1) COV RATIO i 1 >. V-Theoretical Compariso of distaces Based o the critical regios of these differet distaces, we ca choose the oe that gives the best detectio result. Rule: The method with the smallest critical regio is the most accurate. So compare these regios: For Leverage poit ad COVRATIO: If the critical regio for the Leverage poit 2(p+1) ad that of 2 COV RATIO 3(p+1) +1, the 2(p+1) < 3(p+1) + 1 ad so the Leverage poit is more accurate tha COVRATIO. For DFBETAS ad DFFITS: 2 If the critical regio for DFBETAS is p+1 ad that for DFFITS is 2, the 2 p+1 < 2 ad therefore DFBETAS is more accurate tha DFFITS. 2 For Cook s distace ad Leverage poit: The critical regio for the Leverage poit is 2(p+1) ad that of Cook s is 4 p 1 ad so 4 p 1 < 2(p+1). For Cook s distace ad DFBETAS: The critical regio is DFBETAS is 2 ad that of Cook s is 4 4 < 2 p 1. p 1 ad so Comparig the critical regio of the criteria already metioed above: 4 < 1 < 2ad 2 p 1 < 1 < 2ad 2(p+1) < 2 for p +1 <ad 3(p+1) +1> 2 shows that DCOOK is the best distace i detectig suspected poits.

7 Idices of Distaces 61 4 Relatioships betwee the distaces The purpose of this sectio is to explai the relatioships amog the differet distaces. Leverage poitage ad Cook The Leverage poit arm measures horizotally the differece betwee the observed poit ad the mea X of the explaatory variable. They deped oly o the values of that variable. As for Cook distaces, they measure somehow the overall importace of horizotal ad vertical gaps. I geeral, a poit ca be characterized by a sigificat residue, without beig very ifluetial, if the Leverage poit arm is ot very high. Similarly, a poit may have a large Leverage poit arm without beig particularly ifluetial, if the residue which is characterized is low. A poit is ot therefore ifluetial i the sese of Cook s distace if both, its residue ad Leverage poit arm are importat. DFFITS ad Cook s distace Cook s distace ad DFFITS deped o Leverage poit ad CookD ca be represeted as a fuctio of Leverage poit ad Studet residue ad eve DFFITS. This shows that the observatios with high Leverage poit are the highest values of DCook ad DFFITS the have a great ifluece o the predictios of the model. COVRATIO, DFFITS ad DFBETAS Observig the rule of COVRATIO ad those of DFFITS ad DFBETAS, we ote that they do ot deped o sample size while COVRATIO depeds o. DCook ad DFBETAS If there are may variables, we first look at the globally ifluetial observatios (DCook) ad the for these observatios with variable(s) causig that ifluece (DFBETAS). 5 Table summary The followig table summarizes the correspodig case detectio of abormal poits.

8 62 H. Y. Abdallah Stadardized residuals Studet residuals Leverage poit Cook s distace Result e i > 2 whereas i residue is sigificatly 0 e i > 2 whereas i observatio requires a ivestigatio h ii > 2(p+1) CookD>1 idicates a abormal effect DFBETAS DFBETAS > 2 COVRATIO COV RATIO 1 > 3(p+1) p+1 DFFITS DEFITS > 2 Purpose Large residue detectio thus atypical observatio Detectio of large residue thus atypical observatio Measure the ifluece of observatio i o the estimators Measure the effect of the removal of the observatio i o the predictio of values Measure the ifluece of a poit o the estimated coefficiet Measure the effect of the ith observatio accuracy Measure the ifluece of observatio i o the predictio of its eigevalue 6 Practical Applicatio To illustrate our study, we propose a real example for a example of 100 studets at the Faculty of Sciece of the Lebaese Uiversity takig as variable the average score i the Masters, first, secod ad third years. I order to detect outliers for this example regressio is performed i the first step to explai the marks i Masters accordig to the two explaatory variables are the grades i the first year ad those i the secod ad third year together ad, as a secod step, we determie the critical areas of idices cited i the study:

9 Idices of Distaces 63 Studet Major Master Average of Average of Number s Average(AM) secod ad first year third years (A1) (A23) 1 biology biology Chemical biochemistry biology biochemistry biology biochemistry biology Fudametal Electroics biology biology biochemistry biology biochemistry mathematical chemistrymolecular Molecular Chemistry 20 biochemistry Fudametal Electroics biochemistry Fudametal biology Biology: Elective Chemical Computer biology biochemistry biology biochemistry biochemistry

10 64 H. Y. Abdallah Studet Number Major Master Average of Average of s Average(AM) secod ad first year third years (A1) (A23) chemistrymolecular 35 Electroics biochemistry biochemistry Computer Fudametal Chemical biochemistry chemistrymolecular biochemistry biochemistry biology biology chemistrymolecular Chemistry optio Evirometal Scieces 49 biology biology biology Electroics biochemistry biology biology Computer biology Electroics biochemistry Computer biochemistry biochemistry biology chemistrymolecular Chemical

11 Idices of Distaces 65 Studet Number Major Master s Average(AM) Average of secod ad third years (A23) Average of first year (A1) 66 biology biochemistry biochemistry biology chemistrymolecular Fudametal chemistrymolecular Computer Fudametal biochemistry Chemical biochemistry Computer Electroics biochemistry mathematics biochemistry Electroics mathematics Electroics biochemistry Electroics biochemistry biochemistry Fudametal Computer biology Electroics biology Computer biochemistry biology biochemistry Electroics Biology: Elec

12 66 H. Y. Abdallah 7 Iterpretatio The proposed study has two variables (p = 2) ad 100 observatios, The model is obtaied AM= A A1. For a threshold usig the SAS software, we had the followig results: Usig the Studet residual ad stadardized residual, each observatio more tha two is a abormal fidig. By examiig the differet values??of the residues shows that the observatio of which 12 are medium (A1 = 57.67, A23 = 71.88, AM = 81.57) with a studet residue = is the first atypical value, observatio 78 (A1 = 60.42, A23 = 71.80, AM = 64.37) each have 2035 ad as the value of STUDENT RESIDUAL RSTUDENT ad is the secod atypical value. Usig the Leverage poit, each greater tha 2(p+1) = 2(2+1 =0.06 observatio 100 is a uusual observatio. The the software uses gives us the values??of the matrix diagram Hat. We takes a few examples: 6 (A1=79.92, A23=79.26, AM=81.18) ad the Leverage poit is (h=0,0611) ; 9 (A1=69.50, A23=79.20, AM=80.40) (h=0,0801) ; 48(A1=82.67, A23=83.88, AM=81.05) (h=0,0966)... 4 From Cook s distace, each more tha = 4 =0.041 observatio p is a uusual observatio. While examiig the colum COOK foud that 56 observatios whose average are (A1 = 77.67, A23 = 77.71, AM = 60.62) ad Cook (D = 0.056) ad 63 (A1 = 82.08, A23 = 83.12, AM = 87.97) (D = 0.080) ad 84 (A1 = 67.75, A23 = 61.10, AM = 57.30) (D = 0.045);) are just outliers. The dfbetas idicates that each variable has a value greater tha 2 =0.2 correspods to a uique value, the the variable A23 has a uique value 5 (A23 = 64.55),7 (63.13),12 (71.88), 63 (83.12),66 (70.55),84 (61.10),87 (58.47),90 (71.22) ad the A1 variable 5 (A1 = 67.00), 12 (57.67),25 (50.00), 45 (78.25), 66 (50.50), 84 (67.75),90 (53.92). Usig COVRATIO, each observatio has covratio 1 > 3(p+1) = 0.07 is a uusual observatio. The follows that the observatios 6 (Average A1 = 79.92, A23 = 79.26, AM = 81.18), 8 (A1 = 76.50, A23 = 77.54, AM = 76.00), 9 (A1 = 69.50, A23 = 79.20, AM = 80.40), 12 (A1 = 57.67, A23 = 71.88, AM = 81.57), 19 (A1 = 84.67, A23 = 78.58, AM = 77.13), 20 (A1 = 56.33, A23 = 74.92, AM = 77.62) are outliers. Fially, we ote that the outliers will differ from oe remote to aother this is due to the existig differece betwee the distaces. I our project, comparig outliers obtaied usig differet distaces shows that DCook is

13 Idices of Distaces 67 best. The results are give usig DCOOK as follows: For the idividual 56, the average i the first three years (A1 = 77.67, A23 = 77.71) is higher tha the average Masters (AM = 60.62) -for the idividual 63, the average i the first three years (A1 = 82.08, A23 = 83.12) is lower tha the average Masters (AM = 87.97) -for the idividual 84, the average i the first three years (A1 = 67.75, A23 = 61.10) is higher tha the average Masters (AM = 57.30) -for the idividual 87, the average i the first three years (A1 = 50.75, A23 = 58.47) is lower tha the average Masters (AM = 68.42) Thus, these results are cosistet with our study, as the fact that these poits are atypical of the variatio is i the opposite directio betwee the explaatory variables ad to explai that. 8 Coclusio The observatios cotaied i the databases must absolutely be validated because the appearace of outliers is ievitable because of the quality of data processed ad the various sources of errors durig acquisitio. To esure high quality iformatio, a search for stragglers or outliers must be doe before the use of databases. So this article has helped us to differetiate a outlier from ifluetial. We also studied several methods for the detectio of outliers by showig that the test suitable for the detectio of abormal poits is Cook s distace. Simulatios o large files are the subject of curret research. Refereces [1] Ricco Rakotomolala, Pratique de la régressio liéaire multiple (diagostic et sélectio de variable), uiversité lumière Lyo 2, [2] Ricco Rakotomolala, Poits atypiques et poits ifluets, régressio liéaire multiple, uiversité LYON 2, [3] David A Belsey, Edwi Kuhroy, Regressio diagostic idetifyig ifluetial data ad sources of coliearity, [4] Pierre Adré Comilo, Eric Matzer Lober, Regressio: théorie et applicatios, Spriger, 2007.

14 68 H. Y. Abdallah [5] Lauret Carraro, Itroductio a la régressio, [6] Mathieu Resche-Rigo, Outills de régressio, résidus, mesure d ifluece idividuelle, [7] Stéphae Cau, Diagostic de la régressio, [8] Philippe Besse, Pratique de la modélisatio statistique, 2000.

Properties and Hypothesis Testing

Properties and Hypothesis Testing Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.