On the detection of influential outliers in linear regression analysis

Transcription

1 Amercan Journal of Theoretcal and Appled Statstcs 04; 3(4): Publshed onlne July 30, 04 ( do: 0.648/j.ajtas ISSN: (Prnt); ISSN: (Onlne) On the detecton of nfluental outlers n lnear regresson analyss Armyaw Zakara, Nathanel Kwamna Howard, Bsmark Kwao Nkansah * Department of Mathematcs and Statstcs, Unversty of Cape Coast, Cape Coast, Ghana Emal address: bnkansah@ucc.edu.gh (B. K. Nkansah), zzarmyaw@yahoo.com (A. Zakara), nathoward965@yahoo.co.uk (N. K. Howard) To cte ths artcle: Armyaw Zakara, Nathanel Kwamna Howard, Bsmark Kwao Nkansah. On the Detecton of Influental Outlers n Lnear egresson Analyss. Amercan Journal of Theoretcal and Appled Statstcs. Vol. 3, No. 4, 04, pp do: 0.648/j.ajtas Abstract: In ths paper, we propose a measure for detectng nfluental outlers n lnear regresson analyss. The performance of the proposed method, called the Coeffcent of Determnaton ato (CD), s then compared wth some standard measures of nfluence, namely: Cook s dstance, studentsed deleted resduals, leverage values, covarance rato, and dfference n fts standardzed. Two exstng datasets, one artfcal and one real, are employed for the comparson and to llustrate the effcency of the proposed measure. It s observed that the proposed measure appears more responsve to detectng nfluental outlers n both smple and multple lnear regresson analyses. The CD thus provdes a useful alternatve to exstng methods for detectng outlers n structured datasets. Keywords: Coeffcent of Determnaton ato, Cook s Dstance, DFFITS, CV, Studentsed Deleted esduals, Leverage Values. Introducton An outler n a set of data s defned to be an observaton (or subset of observatons) whch appears to be nconsstent wth the remander of that set of data []. Outlers may represent data that are contamnated n some way (e.g., a recordng error, an error n the expermental procedure), or they may represent an accurate observaton of a rare case []. It s well known that snce the effect of outlyng observatons on parameter estmates and on nferences about models and ther sutablty are to be expected, studes on outlers would help to reduce ther nfluence. Outler dentfcaton s done relatve to a specfed model. If the form of the model s modfed, the status of ndvdual observatons as outlers may change [3]. Consequentally, when outlers are present n a dataset, t leads to msleadng results. egresson analyss, as we know, s one of the most mportant statstcal technques for model fttng. If a regresson model s approprately selected, most observatons should be farly close to the regresson lne or hyperplane. The observatons whch are far away from the regresson lne or hyperplane may not be deal observatons for the selected model and could potentally be dentfed as the outlers for the model. The least squares method s undoubtedly the most popular parameter estmaton technque, manly due to ts computatonal smplcty and underlyng optmal propertes [4, 5]. It s well known that nferences based on least squares regresson can be strongly nfluenced by only a few observatons n the data, and the ftted model may reflect unusual features of those observatons rather than the overall relatonshp between the varables, [6]. Several technques have been developed for detectng problems wth dataset n regresson analyss. They dffer n the partcular regresson result on whch the effect of a deleton of an observaton s measured. For nstance, the Cook s dstance measures the effect of observatons on the estmated regresson coeffcents. Other measures of nfluence measure the effect of observatons on the ftted values and the varance-covarance of the parameter estmates. Ths paper s another attempt at dentfyng a more responsve measure for detectng even the more subtle suspect outlers. In ths secton, we provde a bref revew of the measures that are used for detectng nfluental observatons n structured data. Then n the next secton, we wll propose an alternatve measure for outler detecton. The thrd secton then compares the proposed measure wth the standard ones usng some datasets.

2 Amercan Journal of Theoretcal and Appled Statstcs 04; 3(4): evew of Measures of Influence In ths secton, we dscuss some standard measures of nfluence. These measures are the leverage value, studentsed deleted resduals, Cook s dstance, DFFITS, and the Covarance ato.... Leverage Values Leverage values are employed to dentfy outlers wth respect to ther x values. Ths value s a measure of the dstance between the observaton s x values and the centre of the data. If the leverage value of an observaton s large, the observaton s outlyng wth respect to ts x values. The dagonal elements of the hat matrx (called leverage values) are a useful ndcator of whether or not an observaton s outlyng wth respect to ts x values. The leverage value,, h for the th observaton n the data matrx X s gven by h x x,,,, n () If the th observaton s outlyng n terms of ts x values and therefore has a large value of h, t exercses substantal weght (leverage) n determnng the ftted value y ˆ. A leverage value h s usually consdered to be large, f t s more than twce as large as the mean leverage value. That s, leverage values greater than ( k + ) n are consdered by ths rule to ndcate outlyng observatons wth regard to ther x values.... Studentsed Deleted esduals The studentsed deleted resduals, t s gven by n k t ˆ ( ) ˆ SSE h ε ε () Whch s calculated from the resduals εˆ, the error sum of squares SSE, and the hat matrx values h, all for the ftted regresson based on the n observatons. We dentfy as outlyng those observatons whose studentsed deleted resduals are large n absolute value. In addton, we can conduct a formal test by means of Bonferron test of whether the observaton wth the largest absolute studentsed deleted resdual s an outler. If the regresson model s approprate, so that no observaton s outlyng because of a change n the model, then each studentsed deleted resdual wll follow the t dstrbuton wth ( n k ) degrees of freedom. The approprate Bonferron crtcal value therefore s t ; nk α. An observaton s consdered to be outler wth respect to ts y value f. t t α ; nk..3. Cook s Dstance Cook s dstance measures the squared dstance between the least squares estmate of β based on all n observatons and the estmate, β (), obtaned when the th observaton s removed. Cook s dstance measure s an aggregate nfluence measure, showng the effect of the th observaton on all n ftted values. It s gven by D ( ˆ ˆ β β) X X( βˆ βˆ ) ( ) ( k + ) s ( ) or by a more computatonally convenent form as r h D k + where r s the squared studentsed resdual, whch reflects how well the model fts the th observaton, y. For nterpretng Cook s dstance measure, a rule of thumb s that D 4 n ( k + ) whch ndcates that the observaton s nfluental...4. DFFITS A useful measure of nfluence that observaton has on the ftted value ŷ s gven by: ( DFFITS) ( ) (3) (4) yˆ yˆ( ) (5) s h The letters DF denote the dfference between the ftted value ŷ for the th observaton when all n observatons are used n fttng the regresson functon and the correspondng predcted value y ˆ( ) obtaned when the th observaton s omtted n fttng the regresson functon. The denomnator of Equaton (5) s the estmated standard devaton of yˆ, but t uses the standard error, s (), when the th observaton s omtted n fttng the regresson functon for σ estmatng the error varance. The denomnator provdes standardzaton so that the value ( DFFITS) for the th observaton represents the amount of ncrease or decrease n the estmated standard devatons of ŷ wth ncluson of the th observaton n fttng the regresson model. It can be shown that the DFFITS values can be computed by usng only the results from the entre dataset, as follows: ( n k h ) ˆ ( ) ˆ h DFFITS t SSE h ε h h ε (6) As a gude for dentfyng nfluental observatons, t s suggested to consder an observaton as nfluental f the absolute value of DFFITS exceeds for small to medum datasets and ( k + ) n for large datasets...5. Covarance ato One can assess the nfluence of the th observaton by comparng the estmated varance of βˆ and the estmated

3 0 Armyaw Zakara et al.: On the Detecton of Influental Outlers n Lnear egresson Analyss varance of βˆ (). Mathematcally, the Covarance ato (CV) s gven by ˆ σ( ) CV ˆ σ Ideally, when all observatons have equal nfluence on the covarance matrx, CV s approxmately equal to one. Devaton from unty ndcates that the th observaton s potentally nfluental. A rough calbraton pont for Equaton (7) s CV > 3k n.. The Coeffcent of Determnaton ato The general procedure for assessng the nfluence of an observaton n a regresson analyss s to determne the changes that occur when that observaton s omtted. Several measures of nfluence have been developed usng ths concept. We now propose a measure of nfluence that s based on the value of the coeffcent of determnaton ( ) of the lnear regresson model. To formulate the proposed measure, we frst ft a lnear regresson model to the full data and determne the value. Secondly, we compute the ( ), the coeffcent of k () (7) determnaton value when the th observaton s deleted from the dataset. We then compare the values of and () by takng ther rato. Ths measure s what we refer to n ths paper as the Coeffcent of Determnaton ato (CD). The CD for the th observaton s defned as ( ) ( ) CD,,,, n (8) ( ) It has been shown (see Appendx D, and [7] ) that a sutable expresson for () s ( ) ( ) ε y + and that y y y y. Substtutng these nto Equaton (8), some further algebrac steps gves ˆ In computng CD for each observaton n a gven dataset, there s no need to actually delete observatons one after the other and reft the lnear regresson model each tme. A lnear regresson analyss s carred out only once, and then regresson results are used to evaluate CD for each observaton. As a rule of thumb, f the CD for the th observaton devates from unty, then the th observaton s nfluental. Ths dea s somewhat general; hence we need to fnd a method whch wll determne the exact cutoff values for the CD. However, n ths paper, we examne all CD values graphcally. (The use of cutoff rule for the CD s under study). An ndex plot of CD may be a useful graphcal devce for vsualzng suspect outlers. When the CD values are all about the same, no suspect outlyng observatons are present. On the other hand, f there are observatons wth CD values that stand out from the rest, these observatons can be dentfed as outlers. 3. Implementaton of CD 3.. Usng CD to Detect Outlers n Smple Lnear egresson Analyss In ths secton, we llustrate the use of the proposed measure ( CD ) to detect outlers n smple lnear regresson analyss. The results obtaned by CD are compared wth those from some known nfluence measures revewed n Secton. The dataset used s an artfcal one created by [8] to llustrate the features of Mathematcal package for unmaskng regresson outlers. We examne for outlyng observatons by consderng the observatons that do not follow the man pattern of the bulk of the data. Even though ths procedure s an nformal way of detectng outlers, t s used as a prelmnary tool to dentfy susceptble observatons. A scatter plot for the data s shown n Fgure. CD y ˆ ε y ( y ) (9) (See Appendx C for proof). In Equaton (9), the quantty y y s the proporton of total varaton contrbuted byy ; ( ) s the amount of varaton n the dataset that excludes y ; and varaton due to y. y ˆ ε h s the amount of explaned Fgure. Scatter plot of Artfcal Data

4 Amercan Journal of Theoretcal and Appled Statstcs 04; 3(4): From Fgure, the majorty of the observatons follow a lnear pattern. Fve observatons {8, 9, 30, 3, 3} le separately from the bulk of the data. These observatons are suspected to be outlers. Observaton 8 s outlyng wth respect to ts x value. Ths observaton s not nfluental because, t les along the pattern of the bulk of the data. It can be seen from Fgure that observaton 9 s outlyng wth respect to ts y value, and therefore may be nfluental. Further, observatons {30, 3, 3} are outlers wth respect to ther x and y values. These observatons are also nfluental. A smple regresson analyss of the artfcal data yelds a regresson model summary whch s presented n Table. Table. egresson Model Summary for Artfcal Data Model sq Adj. sq Std Error of Estmate From Table, t s observed that as low as 0.3% of the varaton n the response Varable Y s accounted for by the predctor varable X. We now examne the performances of the measures of nfluence for ths dataset. The results are shown n Table. From Table, the CD measure detects observatons {8, 9, 30, 3, 3} as outlers (by means of an ndex plot, n Appendx A, of the values of CD ). These observatons have CD values that markedly devate from unty. Also, each of D and ( DFFITS) detected observatons {8, 9, D exceed 30, 3, 3} as outlers. The values of 4 4 the cut-off value f D Also, n ( k + ) 3 these observatons have absolute values of ( DFFITS) that exceed the calbraton pont of 0.5. The t measure suggests that observatons 8 and 9 are outlers snce ther absolute t values exceed the cutoff pont t 0.05, 9 ±.045. In addton, h dentfes observatons 8, 30, 3, and 3 to be outlyng but not 9. Ther values are greater than twce the average of all leverage values (0.5). Fnally, t can be seen that the CD, just lke t, classfes only observatons 8 and 9 as suspect outlers. The values of CD for observatons 8 and 9 do not fall wthn the cutoff nterval (0.83,.88). It s worth notng that the CD, besdes D and DFFITS ), s successful n detectng all the outlers n the ( data. However, h dentfes all but one outler. Each of t and CD detect only two of the fve outlers. Next, we consder an assessment of the nfluence of the outlyng sets of observatons on the value of. The results are presented n Table 3. The table gves the change n when the specfed observatons have been deleted from the data for each of the measures. CD Table. Influence Measures for Artfcal Data t h D ( DF) CV Table 3. Effect of Deleton of Outlyng Observatons on Value Outlyng Measure Observatons new change t 8, h 8, 30, 3, CV 8, D 8, 9, 30, 3, DF 8, 9, 30, 3, CD 8, 9, 30, 3, Table 3 ndcates that the omsson of the observatons {8, 9, 30, 3, 3} from the dataset results n an ncrease n the value of from 0.03 to 0.975, a substantal ncrease usng the CD. The result s the same for D and DFFITS), denoted DF n the table. ( 3.. Usng CD to Detect Outlers n Multple Lnear egresson Analyss For llustratve purposes, we use the data by Moore [9](as cted n [0]). Ths data has also been used by [9] to compare the performance of varous nfluence measures to detect nfluental observatons, hgh leverage ponts, and outlers n lnear regresson. The measured varables are: Y log (oxygen demand n dary waste), mg/mn; X bologcal oxygen demand, mg/ltre; X total Kjeldahl ntrogen, mg/ltre; X 3 total solds, mg/ltre; X 4 total volatve solds (a component of X 3), mg/ltre; X 5 chemcal oxygen demand, mg/ltre. Correlaton analyss of the data shows the presence of some lnear relatonshp between Y and each of the predctor varables, except X. Further, there s somewhat strong postve lnear relatonshp between any par of the varables X, X 3, X 4, and X 5. These lnear assocatons among the predctor varables are lkely to pose mult-collnearty problems. The summary of the ftted regresson of Y on the fve predctors s shown n Table 4.

5 04 Armyaw Zakara et al.: On the Detecton of Influental Outlers n Lnear egresson Analyss Table 4. egresson Model Summary for Moore s Data Model sq Adj. sq Std Error of Estmate Full CD Table 5. Influence Measures for Moore s Data t h D ( DF) CV From Table 4, we note that even though the value s hgh, t may not be a true representaton of the explanatory power of the ftted regresson model. In an deal stuaton, each observaton n the dataset contrbutes equally to the formaton of the value of. It s, therefore, legtmate to assess each observaton vs-à-vs ther nfluence on value n order to dentfy unusual ones. Table 5 shows the values of varous measures of nfluence for Moore s data when the th observaton s omtted from the dataset. Frst, we consder the coeffcent of determnaton rato (CD) n Table 5. It can be observed that the (CD) for all observatons are approxmately equal to one except observatons {, 7, 5, 0}. The removal of these observatons from the dataset s expected to substantally mprove the. Therefore, the observatons {, 7, 5, 0} are nfluental. To evaluate the studentsed deleted resdual t for an observaton, we compare ths quantty wth t α based on ( n k ) degrees of freedom. Specfcally, f the t s greater n absolute value than, then there s some t α, nk evdence that the observaton s an outler wth respect to ts y value. From Table 5, we see that the t for observatons and 0 (3.584 and.05, respectvely) are both greater than t.77. Therefore, we should be very concerned that 0.05,3 {, 0} are outlers wth respect to ther y values. For the Moore s data, there are n 0 observatons and snce the ftted lnear regresson model utlzes k 5 ndependent varables, twce the average leverage value s From Table 5, we see that the leverage value for observaton 7 s Snce ths value s greater than 0.60, t suggests that observaton 7 s an outler wth respect to ts x values. From Table 5, Cook s dstance for each of observatons {, 7, 0} s greater than the cut-off value Ths means that removng the group of observatons {, 7, 0} from the dataset would substantally change the least squares estmate of the regresson parameters. Hence, observatons, 7, and 0 are flagged as nfluental. It can be seen n Table 5 that the DFFITS for observatons, 7, and 0 exceed the cut-off value of.095, and therefore, they should be dentfed as outlers. For the Moore s data, the cutoff values for CV s ( 0.00,.900). The cut-off nterval s rather conservatve n that t declares too many observatons as outlers. In vew of ths, we use the ndex plot (see Appendx B) of CV to fnd outlers. Observatons and 7 are consdered as outlers. The CV value for observaton 7 s the largest ndcatng that ts presence would have the greatest mpact on ncreasng the precson of the parameter estmates. The CV for observaton s the lowest. Ths shows that the presence of observaton n the dataset greatly decrease the precson of the estmates. The nfluence of the dfferent sets of suspect outlyng observatons, from the varous measures of nfluence, on the value of s dsplayed n Table 6. Table 6. Effect of Deleton of Outlyng Observatons on Measure Outlyng Observatons Value new change t, h CV, D, 7, DF, 7, CD, 7, 5, It s observed from Table 6 that the value has ncreased as a result of deletng varous sets of suspect outlyng observatons. However, the magntude of the ncrement dffers across the sets of outlyng observatons. The greatest change n value (from 0.8 to 0.938) s assocated wth the omsson of the outlyng set {, 7, 5, 0}, whch s based on the CD measure of nfluence. However, the least change n value s lnked wth the omsson of {7}, whch s based on the h measure of nfluence. A comparson of values of n Table 6 shows that the set of observatons {, 7, 5, 0} detected by CD s the most nfluental than those emanatng from the other measures of nfluence. The deleton of ths set from the dataset would subsequently lead to a substantal change n the regresson estmates. The result shows that the CD s more responsve to dentfyng even the subtle outlers.

6 Amercan Journal of Theoretcal and Appled Statstcs 04; 3(4): Concluson and ecommendaton The man objectve of the paper was to assess the standard measures for detectng nfluental outlers n structured data. The result shows that the new measure, called the Coeffcent of Determnaton ato ( CD ) dentfes suspect outlers whch all the other measures detect. In addton, t has the property to detect other and even more subtle suspect outler observatons whch other measures do not detect. Ths means that by usng the CD, any observaton whch s potentally an outlers can be dentfed for assessment. The beneft of usng ths method s that the sgnfcance of the model obtaned eventually for summarsng the dataset s more relable. The results also show that the CD, D, and DF detect almost the same sets of nfluental outlers. However, the CD always perform dstnctly from the other nfluence measures such as the t, h, and CV. The mplementaton of the new method reled mostly on the scatter plot of the values of the CD. Lke the other measures, t would be more formal to dentfy suspect outlers usng exact cut-off values. Future studes n ths area should focus on obtanng a generalzed cut-off value for the new measure. Appendx A Index Plots for Artfcal Data Appendx B Index Plots for Moore s Data (a) (c) (b) (d) (e) (f) (a) (b) Appendx C Proof of Coeffcent of Determnaton ato The CD for the th observaton s defned as ( ) CD,,,, n () (c) (d) or ( ) CD () ( ) Now ˆ ˆ ( ) β ( ) X ( ) X( ) β( ). It can be shown ([7]) that ( ) X ( ) X( ) XX xx, where, () s sum of squares due to regresson wth the th observaton deleted, the correspondng sum of squares. We express and () (e) (f)

7 06 Armyaw Zakara et al.: On the Detecton of Influental Outlers n Lnear egresson Analyss and βˆ βˆ ( ) Substtutng y ˆ Xβ ˆ, we have βˆ ˆ ( ) X ( ) X( ) β( ) ( ) y ( ) y( ) ( ). Substtutng, we have ( ) ( ) ( ) x. (3) ( ) ( ) ( ) βˆ XXβ ˆ y y X X βˆ ( XX x x )ˆ β βˆ XXβ ˆ βˆ x x βˆ Further substtutons usng Eq. (3) gves ( ) βˆ βˆ Xβˆ Substtutng ( ) x βˆ x X X βˆ ˆ X X X x x x βˆ ( ) x Xβˆ X x ( ) ( ) ε x x βˆ x x ˆ xβ ŷ and h x x x x we have x βˆ X x X Xβˆ X x h h yˆ h yˆ h h h Expandng gves ( ) βˆ X Xβˆ h + x h βˆ X Xβˆ h Substtuton for βˆ (X x x ( ) yˆ βˆ x h y h x ˆ y h x βˆ + h h ˆ y h h X X βˆ h εˆ, further smplfcaton gves ˆ ˆ ˆ ( ) β X Xβ + ( y ) ε ( ) y + h ( y ) h ˆ ˆ ˆ h ( y ) h ε ε ε + ( ) (4) Now, ( ) y. (5) Substtutng Eq (4) and (5) nto Eq () yelds CD y y Some few further steps gves CD y eferences ε + ˆ ˆ ε y ( y ). [] Barnett, V., & Lews, T. (994). Outlers n statstcal data (3rd ed.). New York, NY: John Wley and Sons. [] Cohen, J., Cohen, P., West, S. G., & Aken, L. S. (003). Appled multple regresson/correlaton analyss for the behavoral scences (3rd ed.). London, England: Lawrence Erlbaum Assocates. [3] Wesberg, S. (005). Appled lnear regresson (3rd ed.). New York, NY: John Wley and Sons. [4] Nurunnab, A. A. M., Imon, A. H. M.., Al, A. B. M. S., & Nasser, M. (0). Outler detecton n lnear regresson. etreved June 9, 0 from [5] Chatterjee, S., & Had, A. S. (988). Senstvty analyss n lnear regresson. New York, NY: John Wley & Sons. [6] Cook,. D. & Wesberg, S. (98). esduals and Influence n egresson. New York, NY: Chapman and Hall. [7] encher, A. C. & Schaalje, G. B. (008). Lnear models n statstcs (nd ed.). New Jersey, NJ: John Wley & Sons. [8] Snksaran, E. & Satman, M. H. (0). PUO: A package for unmaskng regresson outlers. Gaz Unversty Journal of Scence, 4 (), [9] Moore, J. (975): Total bochemcal oxygen demand of dary manures. Ph. D. Thess, Unv. of Mnnesota, Dept. Agrcultural Engneerng. [0] Chatterjee, S. & Had, A. S. (986). Influental observatons, hgh leverage ponts, and outlers n lnear regresson. Statstcal Scence, (3),