ASSESSING GOODNESS OF FIT

Transcription

1 ASSESSING GOODNESS OF FIT 1. Iroducio Ofe imes we have some daa ad wa o es if a paricular model (or model class) is a good fi. For isace, i is commo o make ormaliy assumpios for simpliciy, bu ofe i is ecessary o check if hese assumpios are reasoable. I Goodess-of-Fi (GoF) ess we srive o check he compaibiliy of he daa wih a fixed sigle model (simple GoF) or wih a model i a give class (composie GoF). Le X 1,..., X be i.i.d. samples from a ukow disribuio. If we wish o ifer wheher his sample comes from a cerai hypoized disribuio F 0 his problem ca be cas as he followig hypohesis es: H 0 : F = F 0 vs. H 1 : F F 0. This is kow as he simple Goodess-of-Fi (GoF)-problem. The composie GoF problem arises whe we wa o es wheher he disribuio of he sample belogs o a cerai class F of disribuio fucios. I his case we cosider he esig problem H 0 : F F vs. H 1 : F F. This ull hypohesis is of composie ype, which ypically makes a formal aalysis much more difficul. A ypical applicaio of such a es arises whe we fi a liear model ad wa o check wheher he ormaliy assumpio o he residuals is reasoable. There are various approaches o GoF esig, ad here we will focus o wo of hem: (i) EDF ess; (ii) Chi-squared ype ess. The Chi-square ess are a obvious choice whe he hypohesized disribuios (ad daa) are discree, ad he empirical CDF mehods are very adequae for he coiuous case. 2. Simple GoF: EDF ess The basic idea follows from he properies of he ECDF see before. Uder H 0, for sufficiely large, ˆF is close o F 0. Recall ha, uder H 0, he Gliveko-Caelli heorem ells us ha sup ˆF () F 0 () a.s. 0, as. Hece ay discrepacy measure bewee ˆF ad F 0 serves as a es saisic. A good saisic much be somewha easy o compue ad characerize. Here are some impora examples. Dae: February 7,

2 2 ASSESSING GOODNESS OF FIT The Kolmogorov-Smirov (KS) es saisic D := sup ˆF () F 0 (). The Cramér-Vo Mises (CvM) saisic C := ( ˆF () F 0 ()) 2 df 0 (). The Aderso-Darlig (AD) saisic ( ˆF () F 0 ()) 2 A := F 0 ()(1 F 0 ()) df 0(). Alhough hese expressios migh look somewha complicaed hey ca be simplified sigificaly if F 0 is coiuous. Noe ha ˆF is piecewise cosa ad F 0 is a odecreasig fucio, herefore he maximum deviaio bewee ˆF () ad F 0 () mus occur i a eighborhood of he pois X (i), ad so D = max 1 i max{ ˆF (X (i) ) F 0 (X (i) ), ˆF (X (i) ) F 0(X (i) ) }, where X (i) := X (i) ɛ for a arbirarily small ɛ (here is a sligh abuse of oaio here). Now, akig io accou ha F 0 is coiuous F 0 (X (i) ) = F 0(X (i) ). Furhermore ˆF (X (i) ) = i/ ad ˆF (X (i)) = (i 1)/ ad herefore D = max 1 i max{ i/ F 0(X (i) ), (i 1)/ F 0 (X (i) ) }. Fially defie U i = F 0 (X i ), ad le U (i) deoe he correspodig order saisics (oe ha, sice F 0 is moooe U (i) = F 0 (X (i) )). We ca herefore wrie he above expressio as D = max max{ i/ U (i), (i 1)/ U (i) }. 1 i I a similar fashio we ge simplified expressios for he CvM ad AD saisics: (1) C = 1 ( 12 + U (i) 2i 1 ) 2, 2 ad A = 1 i=1 (2i 1) [ log U (i) + log(1 U ( i+1) ) ]. i=1 Now suppose ha ideed we are workig uder H 0, which meas he daa {X i } came from he coiuous disribuio F 0. I ha case U i are i.i.d. uiform radom variables i [0, 1] (see foooe 1 ) ad herefore we coclude ha, uder H 0, he disribuio of D, C ad A does o deped o he uderlyig disribuio F 0. I oher words hese saisics are disribuio free uder he ull hypohesis. So, o 1 This resul is kow as he Probabiliy Iegral Trasform: If X has coiuous disribuio F he Y = F (X) has a uiform disribuio suppored i (0, 1).

3 ASSESSING GOODNESS OF FIT 3 devise simple GoF ess i suffices o sudy he case whe he ull hypohesis is he uiform disribuio i (0, 1). To use he above es saisics oe eeds o kow he properies of heir disribuio. For small hese have bee abulaed, ad for large we ca use asympoics. The aalyical sudy requires some machiery of empirical processes heory, ad is ou of he scope of hese oes. As meioed before i suffices o sudy he case F 0 = Uif(0, 1). Le U i Uif(0, 1) ad defie Û() = 1 i=1 1{U i }. Noe ha i his case Û() Pr(U ) = sup Û(). D = D sup [0,1] A well-kow resul from empirical processes heory saes ha he process ( Û () ) coverges i disribuio o a process B 0, which is kow as a sadard Browia Bridge o [0, 1]. This is a Gaussia process defied for [0, 1] wih E[B 0 ()] ad Cov(B 0 (s), B 0 ()) = mi{s, } s. Now, wih a bi of hadwavig (a formal reame requires he use of ivariace priciples) we have as. Similarly C 1 D 0 D D sup B 0 (), B 2 0()d ad A D 1 0 B 2 0() (1 ) d. Foruaely he asympoic disribuios above ca be sudied aalyically ad we have lim P F 0 ( D λ) = 1 2 ( 1) j 1 e 2j2 λ 2, lim P F 0 (C > x) = 1 π j=1 4j 2 π 2 ( 1) j+1 j=1 (2j 1) 2 π 2 y si( y) e xy/2 y dy. Fially A where Y i i.i.d χ 2 1. D A, wih A D = j=1 Y j j(j + 1), 2.1. Cosisecy uder he aleraive. A very impora ad pleasa propery of he ess we ve see is ha hese are cosise uder ay aleraive. This meas ha if he rue disribuio is o F 0 he eveually, as, we will rejec he ull hypohesis o maer wha he rue disribuio is. Le s see his i he case of he KS saisic.

4 4 ASSESSING GOODNESS OF FIT Le G be he CDF of D uder F 0. Tha is G () = P ( D ). We rejec he ull hypohesis (wih sigificace α) if D 0 < α < 1. We will show he followig resul Lemma 2.1. If he daa {X i } i=1 comes from a disribuio F F 0 he as. P F ( D > G 1 (1 α)) 1, > G 1 (1 α), wih Proof. Sice F F 0 here is a leas oe poi a such ha F 0 (a) F (a). Now P F ( D > G 1 = P F ( sup (1 α)) = P F ( sup ˆF () F 0 () > G 1 (1 α)) ˆF () F () + F () F 0 () > G 1 (1 α)) P F ( ˆF (a) F (a) + F (a) F 0 (a) > G 1 (1 α)) P F ( F (a) F 0 (a) ˆF (a) F (a) > G 1 (1 α)), where he las sep follows from x + y x y. Now oe ha he CLT implies ˆF (a) F (a) = O P (1) (meaig δ > 0 c < : P F ( ˆF (a) F (a) c) 1 δ) ad ha F (a) F 0 (a). Therefore we coclude ha P F (D > G 1 (1 α)) coverges o oe as. A similar argume applies also o C ad A. 3. Composie GoF ess As alluded before, he composie GoF sceario is sigificaly more complicaed. However, i is also more releva from a pracical sadpoi, sice we are hardly ever i he siuaio ha we wa o es e.g. if a sample is from a expoeial disribuio wih parameer 2, or from a ormal disribuio wih parameers 1.05 ad Ofe, he composie GoF-problem comes dow o esig H 0 : F {F θ : θ Θ} }{{} F vs. F {F θ : θ Θ}. As a example F θ may be he expoeial disribuio wih mea θ. Perhaps he simples idea o come o mid i his case is o compare he bes disribuio i he class wih he empirical CDF. This ca be doe by esimaig he parameer θ from he daa (deoe his esimaor by ˆθ ) ad comparig ˆF wih, Fˆθ where F F. Therefore we ed up wih he followig es saisics. D = sup ˆF () () or C = Fˆθ ad similarly for he AD saisic. ( ˆF () Fˆθ ()) 2 dfˆθ (),

5 ASSESSING GOODNESS OF FIT 5 Remark 3.1. Pluggig i a parameer esimae affecs he disribuio of hese saisics, ad he disribuio uder he ull will be heavily iflueced by he ype of esimaor you use. Therefore oe ca o loger use he disribuios derived i he previous sessio. Praciioers ofe misakely plug i ˆθ ad subsequely use a ordiary KS-es or CvM-es. This will resul i iadequae esig procedures. Some ess are specifically desiged for GoF wih esimaed parameers. Example 3.2. The Lilliefors es (1967) is a adapaio of he KS-es for which he ull hypohesis equals H 0 : X 1,..., X is a sample from a ormal disribuio wih ukow parameers. The ukow (populaio) mea ad variace are esimaed by he sample mea ad sample variace. The disribuio of his saisic uder he ull has bee abulaed (by Moe-Carlo simulaio). Example 3.3. The Jarque-Bera es (1980) is a es for ormaliy ha is especially popular i he ecoomerics lieraure. This es is based o he sample kurosis ad skewess. 1 1 (X i X ) 3 skewess b 1 = kurosis b 2 = 1, s 3 1 (X i X ) 4. s 4 Uder ormaliy b 1 D N(0, 6) ad (b 2 3) D N(0, 24). The Jarque-Bera saisic is defied by JB = (b 2 1/6 + (b 2 3) 2 /24). Is limiig disribuio is Chi-squared wih 2 degrees of freedom. Example 3.4. The Shapiro-Wilk es is aoher powerful es for ormaliy. The es saisic is ( i=1 W = a ) 2 ix (i) i=1 (X i X ( (0, 1]), ) 2 where he weighs a 1,..., a are specified by a adequae formula. Uder H 0, he umeraor is a esimaor for ( 1)σ 2, whereas he deomiaor is also a esimaor for ( 1)σ 2. Hece, uder H 0, W 1. Uder H 1, he umeraor is eds o be smaller. Therefore, we rejec he ull hypohesis for small values of W. Example 3.5. A simulaio sudy o assess he performace of ess for ormaliy. We compue he fracio of imes ha he ull hypohesis of ormaliy is rejeced for a umber of disribuios (i oal we simulaed 1000 imes). Resuls for = 20 orm cauchy exp Shapiro KS AD CvM JB

6 6 ASSESSING GOODNESS OF FIT Resuls for = 50 orm cauchy exp Shapiro KS AD CvM JB Resuls for = 200 orm cauchy exp Shapiro KS AD NA NA NA CvM JB Resuls for = 5000 orm cauchy exp Shapiro KS AD NA NA NA NA CvM JB The R-code for obaiig his resuls is i he file compare gofess ormaliy.r. The AD es implemeaio appears o have some problems for large sample sizes. Alhough mos exbooks oly rea he KS-es/Lilliefors es, from his simulaio sudy i appears ha his is a raher poor esig procedure i pracice. The JB ad Shapiro-Wilk seem o work sigificaly beer whe esig ormaliy. [D Agosio ad Sephes (1986 war... for esig for ormaliy, he Kolmogorov-Smirov es is oly a hisorical curiosiy. I should ever be used. I has poor power i compariso o specialized ess such as Shapiro-Wilk, D Agosio-Pearso, Bowma-Sheo, ad Aderso-Darlig ess. As ca be see from his quoe, here are may more specialized GoF-ess. 4. Chi-Square-ype GoF ess This is a simple approach o GoF for boh discree ad coiuous radom variables. I has several advaages, amely Suiable for boh coiuous ad discree seigs Easy o use, eve i high dimesios (so far we have bee discussig oly he oe-dimesioal seig). However, here is a drawback: for coiuous radom variables he procedure requires some arbirary choices (ha mus be doe before seeig ay daa). As a cosequece

7 ASSESSING GOODNESS OF FIT 7 some iformaio is los, ad hese ess o loger have he propery of beig cosise agais ay aleraive. Firs cosider he simple GoF-problem. Suppose S = supp(f 0 ). Fix k ad le S = k A i, i=1 be a pariio of S (meaig he ses A i are disjoi). For discree radom variables here is a aural choice for such a pariio. For coiuous radom variables a pariio ca be obaied by formig appropriae cells usig some kowledge abou F 0. Usually oe chooses he umber of cells o saisfy 5 k 15. I is ofe hard o fully jusify a ceraily chose pariio or value for k. Defie N i o be he observed frequecy i cell A i N i = #{j : X j A i }. Uder H 0, e i := E 0 N i = P 0 (X A i ) (he subscrip emphasizes ha he expecaio ad probabiliy have o be compued uder he ull hypohesis). Uder H 0 we expec e i ad N i o be close. Ay discrepacy measure bewee hese wo quaiies ca serve as a basis for a es saisic. I paricular we defied he chi-square saisic as Q = k i=1 (N i e i ) 2 e i. I is o hard o show ha Q coverges o a χ 2 -disribuio wih k 1 degrees of freedom. As a rule of humb, his approximaio is reasoable if all he expeced cell frequecies e i are a leas Composie χ 2 -GoF ess.. I he composie GoF case higs ge more complicaed as you oe migh expec. I paricular he expeced cell frequecies have o be esimaed from he daa. By pluggig-i hese esimaors he disribuio of Q uder he ull is goig o chage. The properies of ha disribuio deped o he properies of he esimaor. If he esimaors are chose usig a maximum likelihood priciple he, uder some mild assumpios, he resulig limiig disribuio will be chi-square wih k 1 s degrees of freedom, where s is he umber of idepede parameers ha mus be esimaed for calculaig he esimaed expeced cell frequecies. So i case we es for ormaliy, s = 2 (mea ad variace). 5. Probabiliy Ploig ad Quaile-Quaile Plos Probabiliy plos provide a visual way for GoF ess. These are o formal ess, bu provide a quick ool o check if a cerai disribuioal assumpio is somewha reasoable. Le F be a locaio-scale disribuio family, ha is F = {F a,b : a R, b > 0},

8 8 ASSESSING GOODNESS OF FIT for some disribuio F. I he above F a,b is he CDF of a + bx whe X F, ha is ( ) x a F a,b (x) = F. b As a example, a N (µ, σ 2 ) radom variable is obaied from a sadard ormal radom variable Z by he liear rasformaio Z µ + σz. Le X 1,..., X be daa from some disribuio. Recall ha ˆF (X (i) ) = i/. Therefore ( ) F 1 a,b ( ˆF i (X (i) )) = F 1 a,b. Now, if he daa comes from a disribuio i F he ˆF (X (i) ) F a,b (X (i) ), ad so ( ) ( ) i i X (i) F 1 a,b = a + bf 1 ( Said differely, if he daa comes from a disribuio i F we expec he pois X(i), F ( 1 i +1)) o lie approximaely i a sraigh lie. Noe ha we replaced i/ by i/( + 1): his is o esure ha we say away from evaluaig F 1 (1), which ca be ifiie. The plo of he above pois is commoly called he quaile-quaile plo. The use of probabiliy plos requires some raiig, bu hese are very commoly used ad helpful. If we wa o es for a disribuio F θ ha is o i a locaio-scale family, he he precedig reasoig implies ha he pois ( F 1 θ. ( i +1), x(i) ) should be o a sraigh lie if θ is kow. If θ is ukow, a esimaor for i ca be plugged i. Mos sofware packages ca geerae such plos auomaically. As poied ou i [Veables ad Ripley (1997)] (page 165), a QQ-plo for e.g. a 9 -disribuio ca be geeraed by execuig he followig code i r: plo(q(ppois(x),9), sor(x)) (We assume he daa are sored i a vecor x). Addig he commad qqlie(x), produces a sraigh lie hrough he lower ad upper quariles. This helps assess wheher he pois are (approximaely) o a sraigh lie. You may also wa o cosider he fucio qq.plo i he library car, which gives a direc implemeaio of he QQ-plo. Example 5.1. Comparig QQ-ormaliy plos ad AD-es for ormaliy. We simulae samples of size 10, 50, 100, 1000, 5000, from a sadard ormal ad 15 disribuio. Figures 5.1 ad 2 show QQ-ormaliy plos for boh cases respecively. Readig hese figure from upper-lef o lower-righ, he correspodig AD p-values are:

9 ASSESSING GOODNESS OF FIT 9 Figure 1. QQ-plos for samples of sizes 10, 50, 100, 1000, 5000, from a sadard ormal disribuio. The upper-lef figure is for sample size 10, he lower-righ is for sample ormal e-08 For almos every purpose i pracice, he differece bewee a 15 ad a Normal disribuio is of o imporace. However, as we obai sufficiely may daa, hypohesis ess will always deec ay fixed deviaio from he ull hypohesis, as ca be see very clearly from he compued p-values. The R-code for producig hese figures is i he file compare qq esig.r. For large daases here are difficulies wih he ierpreaio of QQ-plos, as idicaed by he followig heorem. Theorem 5.2. Pu z i = Φ 1 ( i +1). Le r = i=1 z ix i i=1 z2 i i=1 (X i X) 2, i.e. r is he correlaio coefficie of he pairs ( X (i), z i ). Le F be he rue CDF wih variace σ 2,

10 10 ASSESSING GOODNESS OF FIT Figure 2. QQ-plos for samples of sizes 10, 50, 100, 1000, 5000, from a 15 disribuio. The upper-lef figure is for sample size 10, he lower-righ is for sample F ρ F ormal 1 uiform 0.98 double exp χ expoeial 0.90 logisic 0.97 Table 1. Limiig values for r he lim r = 1 σ 1 0 F 1 (x)φ 1 (x)dx =: ρ F See heorem i [DasGupa (2008)]. Table 1 provides values for he limiig value for various disribuio. We coclude ha asympoically we ge a perfec sraigh lie i case of ormaliy (as i should be). However, for may oher disribuios we obai a correlaio coefficie ha is very close o oe. I is hard o disiguish a se of pois wih correlaio coefficie 0.97 from a se wih correlaio a.s.

11 ASSESSING GOODNESS OF FIT 11 coefficie equal o 1 wih he huma eye. The differece is maily i he ails. For small sample sizes, probabiliy plos help o assess wheher ormaliy holds approximaely, which is ofe all we eed (for example i assessig approximae ormaliy of residuals of liear models). 6. Useful R commads Goodess of fi Tes R-fucio wihi package Chi-square GOF chisq.es sas KS GOF ks.es sas KS-disribuio ksdis PASWR QQ-plo qq.plo car Specialized es for ormaliy. Tes R-fucio wihi package Shapiro-Wilk shapiro.es sas Aderso Darlig ad.es ores Cramer-Vo Mises (for composie ormaliy) cvm.es ores Jarque-Bera (for composie ormaliy) jarque.bera.es series Refereces [D Agosio ad Sephes (1986)] D Agosio, R.B. ad Sephes, M.A. (1986) Goodess-of- Fi Techiques, New York: Marcel Dekker. [Sue e al. (1993)] Sue, W. Gozáles-Maeiga, W. ad Quidimil, M.P. (1993) Boosrap Based Goodess-Of-Fi-Tess Merika 40, [DasGupa (2008)] DasGupa, A. (2008) Asympoic Theory of Saisics ad Probabiliy, Spriger. chapers 26 ad 27 [Veables ad Ripley (1997)] Veables, W.N. ad Ripley, B.D. (1997) Moder Applied Saisics wih S-PLUS, secod ediio, Spriger.