Unifying the Derivations for. the Akaike and Corrected Akaike. Information Criteria. from Statistics & Probability Letters,

Transcription

1 Uifyig the Derivatis fr the Akaike ad Crrected Akaike Ifrmati Criteria frm Statistics & Prbability Letters, Vlume 33, 1997, pages 201{208. by Jseph E. Cavaaugh Departmet f Statistics, Uiversity f Missuri, Clumbia 1 Abstract The Akaike 1973, 1974 ifrmati criteri, AIC, ad the crrected Akaike ifrmati criteri Hurvich ad Tsai, 1989, AICc, were bth desiged as estimatrs f the expected Kullback-Leibler discrepacy betwee the mdel geeratig the data ad a tted cadidate mdel. AIC is justied i a very geeral framewrk, ad as a result, ers a crude estimatr f the expected discrepacy: e which exhibits a ptetially high degree f egative bias i small-sample applicatis Hurvich ad Tsai, AICc crrects fr this bias, but is less bradly applicable tha AIC sice its justicati depeds up the frm f the cadidate mdel Hurvich ad Tsai, 1989, 1993; Hurvich, Shumway, ad Tsai, 1990; Bedrick ad Tsai, Althugh AIC ad AICc share the same bjective, the derivatis f the criteria prceed alg very dieret lies, makig it dicult t reccile hw AICc imprves up the apprximatis leadig t AIC. T address this issue, we preset a derivati which uies the justicatis f AIC ad AICc i the liear regressi framewrk. Keywrds: AIC, AICc, ifrmati thery, Kullback-Leibler ifrmati, mdel selecti. 1 Direct crrespdece t: Jseph E. Cavaaugh, Departmet f Statistics, 222 Math Scieces Bldg., Uiversity f Missuri, Clumbia, MO

2 1. Itrducti The Akaike ifrmati criteri, AIC, was develped by Akaike 1973, 1974 t estimate the expected Kullback-Leibler discrepacy betwee the mdel geeratig the data ad a tted cadidate mdel. I istaces where the sample size is large ad the dimesi f the cadidate mdel is relatively small, AIC serves as a apprximately ubiased estimatr. I ther settigs, AIC may be characterized by a large egative bias Hurvich ad Tsai, 1989 which limits its eectiveess as a mdel selecti criteri. Fr such istaces, Hurvich ad Tsai 1989 prpsed the crrected Akaike ifrmati criteri, AICc. Origially suggested fr liear regressi by Sugiura 1978, Hurvich ad Tsai 1989 justi- ed AICc fr liear ad liear regressi ad autregressive mdelig, ad ivestigated the small-sample superirity f AICc ver AIC i these settigs. Sice the, AICc has bee exteded t a umber f additial framewrks, icludig autregressive mvig-average mdelig Hurvich, Shumway, ad Tsai, 1990, vectr autregressive mdelig Hurvich ad Tsai, 1993, ad multivariate regressi mdelig Bedrick ad Tsai, The advatage f AICc ver AIC is that i small-sample applicatis, AICc estimates the expected discrepacy with less bias tha AIC. The advatage f AIC ver AICc is that AIC is mre uiversally applicable, sice the derivati f AIC is quite geeral whereas the derivati f AICc relies up the frm f the cadidate mdel. Althugh AIC ad AICc share the same fudametal bjective, the justicatis f the criteria prvided by Akaike 1973 ad Hurvich ad Tsai 1989 prceed alg dieret directis, makig it dicult t reccile hw AICc rees the apprximatis used t establish AIC. A derivati which liks the justicatis wuld be istructive. With this as ur gal, we frmulate a derivati which cects the mtivatis fr the criteria i the settig f the liear regressi mdel. We csider this framewrk i rder t keep ur develpmet straightfrward: aalgus derivatis culd be easily cstructed fr ther settigs i which AICc has bee justied. Our wrk leads t sme useful guidelies characterizig liear regressi settigs i which AIC may prvide a iadequate estimatr f the expected discrepacy. 1

3 2. A Uied Derivati f AIC ad AICc A well-kw measure f separati betwee tw mdels is give by the -rmalized Kullback- Leibler ifrmati Kullback, 1968, als kw as the crss etrpy r discrepacy. Let represet the set f parameters fr the \true" r geeratig mdel, ad let represet the set f parameters fr a apprximatig r cadidate mdel. Let represet the parameter space fr. The discrepacy betwee the geeratig mdel ad the cadidate mdel is deed as d ; = E f?2 l L jy g; where E detes the expectati uder the geeratig mdel, ad L jy represets the likelihd crrespdig t the cadidate mdel. Nw fr a give set f maximum likelihd estimates ^, d ^ ; = E f?2 l L jy gj =^ 2.1 wuld prvide a useful measure f the separati betwee the geeratig mdel ad the tted cadidate mdel. Yet evaluatig 2.1 is t pssible, sice dig s requires kwledge f. Akaike 1973, hwever, ted that?2 l L^ jy serves as a biased estimatr f 2.1, ad that the bias adjustmet E E f?2 l L jy gj =^? E f?2 l L^ jy g 2.2 ca fte be asympttically estimated by twice the dimesi f ^. Thus, if we let k represet the dimesi f ^, the uder apprpriate cditis, the expected value f shuld be asympttically ear the expected value f 2.1, say AIC =?2 l L^ jy + 2k 2.3 k; = E fd ^ ; g: Specically, e ca establish that E faicg + 1 = k; : 2.4 The demstrati f this fact requires the strg assumpti that is a elemet f ; i.e., that the cadidate family icludes the geeratig mdel. See Lihart ad Zucchii, 1986, p

4 AIC prvides us with a apprximately ubiased estimatr f k; i settigs where is large ad k is cmparatively small. Yet i ther settigs, 2k may be much smaller tha the bias adjustmet 2.2, makig AIC substatially egatively biased as a estimatr f k;. T crrect fr this egative bias, Hurvich ad Tsai 1989 prpsed AICc fr liear ad liear regressi ad autregressive mdelig. Fr simplicity, we itrduce ad csider AICc i the ctext f the liear regressi mdel, althugh ur develpmet here culd easily be exteded t ther settigs i which AICc has bee justied. Suppse that the geeratig mdel fr the data is give by y = X + e; e N 0; 2 I; 2.5 ad that the cadidate mdel pstulated fr the data is f the frm y = X + e; e N 0; 2 I: 2.6 Here, y is a 1 bservati vectr, e is a 1 errr vectr, vectrs, ad X is a p desig matrix f full-clum rak. ad are p 1 parameter Nw assume is such that fr sme 0 < p p, the last p? p cmpets f are zer. Thus, mdel 2.5 is ested withi mdel 2.6. Let ad respectively dete the k = p + 1 dimesial vectrs 0 ; 2 0 ad 0 ; 2 0. Nte that the estig esures that is a elemet f, which is required t prve the aalgue f 2.4 fr AICc. See Hurvich ad Tsai, 1989, p Let ^ dete the least-squares estimatr f, ad let ^ 2 = y? X ^ 0 y? X ^ =. Hurvich ad Tsai 1989, p. 300 dee AICc as l ^ 2 + Fr cveiece, we will use the peratially equivalet deiti AICc = l ^ 2 + l p? p? 2 : p? p? 2 = l ^ 2 2p l 2 +? p? 2 ; 2.8 which diers frm 2.7 by a additive cstat that has impact the selecti behavir f the criteri. Oe ca prve that i the liear regressi settig, E faiccg = k; ; 2.9 thus establishig that AICc is exactly ubiased fr k;. The precedig hlds up t 1 fr ther mdelig framewrks i which AICc has bee justied ad develped. 3

5 We will derive AIC ad verify 2.4 i a geeral settig, ad derive AICc ad verify 2.9 i the liear regressi settig. Our bjective will be t d s i a maer which will clearly illustrate the way i which AICc imprves up the apprximatis leadig t AIC. T begi, csider writig k; as fllws: k; = E E f?2 l L jy gj =^ = E f?2 l L^ jy g h i E f?2 l L jy g? E f?2 l L^ jy g 2.11 i + he E f?2 l L jy gj =^? E f?2 l L jy g : 2.12 The derivati f AIC ad vericati f 2.4 are accmplished by establishig the fllwig lemma, which asserts that 2.11 ad 2.12 are bth withi 1 f k. The lemma ca be justied as a special case f Prpsitis 1 ad 2 f Lihart ad Zucchii 1986, pp. 240{242. A brief prf is sketched here fr cmpleteess. We assume the ecessary regularity cditis required t esure the csistecy ad asympttic rmality f the maximum likelihd vectr ^. Lemma 1 E f?2 l L jy g? E f?2 l L^ jy g = k + 1; 2.13 E E f?2 l L jy gj =^? E f?2 l L jy g = k + 1: 2.14 Prf: Dee I ; = E l L 0 # ad I ; Y = l L 0 # : First, csider takig a secd-rder expasi f?2 l L jy abut ^ ad evaluatig the expectati uder f the result. We btai E f?2 l L jy g = E f?2 l L^ jy g +E ^? 0 fi ^ ; Y g^? + 1:

6 Next, csider takig a secd-rder expasi f E f?2 l L jy gj =^ abut, agai evaluatig the expectati uder f the result. We btai E E f?2 l L jy gj =^ = E f?2 l L jy g +E ^? 0 fi ; g^? + 1: 2.16 Nw the quadratic frms ^? 0 fi ^ ; Y g^? ad ^? 0 fi ; g^? bth cverge t cetrally distributed chi-square radm variables with k degrees f freedm. Recall agai that we are assumig 2. Thus, the expectatis uder f bth quadratic frms are withi 1 f k. This fact alg with 2.15 ad 2.16 establishes 2.13 ad The asympttic apprximati f the sum f g by 2k may be smewhat crude fr small r mderate sample-size applicatis. Hwever, fr a mre precise assessmet f this quatity, e wuld eed t further specify the uderlyig mdelig framewrk. Fr example, i the liear regressi settig f iterest, the fllwig lemma will shw that f g ca be exactly evaluated as a fucti f p = k? 1 ad. This evaluati will lead t the derivati f AICc ad vericati f 2.9. Lemma 2 Fr the geeratig mdel 2.5 ad the cadidate mdel 2.6, E f?2 l L jy g? E f?2 l L^ jy g =?E l ^2 E E f?2 l L jy gj =^? E f?2 l L jy g = E l ^2 2 Prf: The lg-likelihd fr the cadidate mdel 2.6 is give by l L jy =? 2 l 2? 2 l 2? y? X0 y? X: Uder the geeratig mdel 2.5, e ca easily establish the fllwig relatis: ; p + 1? p? 2 : 2.18 E f?2 l L jy g = l l 2; 2.19 E f?2 l L^ jy g = E f l ^ 2 g l 2; 2.20 E f?2 l L jy gj =^ = l ^ ^ ^ ^ 2? 0 X 0 X ^? + l 2:

7 Nw t evaluate the expected value f 2.21 uder 2.5, te that ^ 2 =2 has a chi-square distributi with?p degrees f freedm, the quadratic frm f ^? 0 f1= 2 X 0 Xg ^? g has a chi-square distributi with p degrees f freedm, ad ^ 2 ad ^ are idepedet. Usig the fact that the expectati f the reciprcal f a chi-square radm variable with df degrees f freedm is give by 1=df? 2, e ca argue that the expectati f 2.21 uder 2.5 reduces t E E f?2 l L jy gj =^ = E f l ^ g 2 + See Hurvich ad Tsai, 1989, p ? p? 2 + By 2.19 ad 2.20, we have By 2.19 ad 2.22, we have p + l 2: 2.22? p? 2 AICc is derived ad 2.9 veried by usig Lemma 2 i cjucti with the represetati f k; as the sum f 2.10, 2.11, ad Nte that i the preset settig, we ca use 2.17 ad 2.18 t write k; as fllws: k; = E E f?2 l L jy gj =^ = E f?2 l L^ jy g?e l ^2 2 + E = E f?2 l L^ jy g + l ^ p + 1? p? 2 : 2p + 1? p? 2 The precedig justies that is exactly ubiased fr k;. Yet sice?2 l L^ jy + 2p + 1? p? ?2 l L^ jy = l ^ l 2; 2.23 is precisely the same as the deiti f AICc prvided by 2.8. I summary, we have derived AIC ad veried 2.4 thrugh utilizig Lemma 1 alg with the represetati f k; as the sum f 2.10, 2.11, ad Fr the liear regressi settig, we have derived AICc ad veried 2.9 thrugh utilizig Lemma 2 alg with the same represetati f k;. The third ad al lemma prvides a predictable albeit meaigful cecti betwee AIC ad AICc. 6

8 Lemma 3 Fr the geeratig mdel 2.5 ad the cadidate mdel 2.6, Prf: E l ^2 2?E + l ^2 2 2p + 1? p? 2 = p ; 2.24 = p : 2.25 We begi by csiderig a asympttic apprximati fr?e f l^ 2 =2 g: Let X df represet a chi-square radm variable with df degrees f freedm. By takig a secdrder expasi f l X df relati abut df ad evaluatig the expectati f the result, e ca justify the Efl X df g = l df? 1 df + O 1 See Bickel ad Dksum, 1977, p. 31. Usig this relati alg with the fact that ^ 2 =2 has a chi-square distributi with? p degrees f freedm, we ca write?e l ^2 2 = l? E = l? l ^2 2 df 2 l? p? 1? p + O : 1? p 2 Next, csider takig a rst-rder expasi f l? p abut t btai l? p = l? p + O p2 2! : 2.26 : 2.27 Substitutig 2.27 it 2.26 yields?e l ^2 2 = p p? p + O p! 2 + O? p 2 : 2.28 Asympttically, each f the al three terms the right f 2.28 is 1 as! 1 whe p is held cstat. Hwever, each f these terms is als 1 whe p is allwed t grw at a rate less tha p as! 1. Thus, assumig! 1 ad p is O 1=2?, > 0, 2.28 allws us t write?e l ^2 2 = p : This establishes The justicati f 2.25 fllws sice we ca write E l ^ p + 1? p? 2 = f?p g + = p : 2p p + 1p + 2? p? 2 7

9 3. Discussi The develpmet i Secti 2 alg with the results f Hurvich ad Tsai 1989 suggest that fr liear regressi, AICc is preferable t AIC as a estimatr f k; uless p is \small" ad is \large". The pealty term f AICc is exactly equal t the sum f g, whereas the pealty term f AIC ly prvides a apprximati t this sum. Hw crude is the apprximati f 2.11 ad 2.12 by k = p + 1? T address this questi i the liear regressi settig, we evaluate 2.11 ad 2.12 fr varius values f p ad ad preset the results i Table 1. The exact cmputati f 2.11 ad 2.12 is pssible usig the fact that where E l ^2 2 = l 2 +? p ; is the digamma r psi fucti Ktz, Jhs, ad Read, 1982, p See Hurvich ad Tsai, 1989, p Berard 1976 presets a simple algrithm fr cmputig values f. The results help t illustrate tw iterestig priciples. First, the apprximati f 2.12 by p+1 is uifrmly wrse tha the apprximati f 2.11 by p+1, due t the fact that p+1 is always less tha 2.11 ad 2.11 is always less tha Thus, the bias crrecti prvided by AICc t AIC plays a mre imprtat rle i reig the apprximati f 2.12 tha i reig the apprximati f 2.11, particularly where is small ad p is large. Secd, i referece t 2.24 ad 2.25, it ca be easily shw that if > p + 22p + 3, the 2p + 1 is less tha e uit smaller tha 2p + 1=? p? 2. If exceeds p + 22p + 3, the p + 1 appears t prvide a gd apprximati t 2.12, ad hece t Hwever, if is substatially less tha p + 22p + 3, the p + 1 may be much smaller tha 2.12 r 2.11 eve whe is \large". I such settigs, AIC may greatly uderestimate k;, ad as a result, may perfrm prly as a mdel selecti criteri. We clse by tig that the develpmet i Secti 2 suggests that?2 l L^ jy is a justiable chice fr the gdess-f-t term i AICc. The derivatis f AICc preseted by Hurvich ad Tsai 1989, 1993 ad Bedrick ad Tsai 1994 arrive at gdess-f-t terms f the frm l ^ 2, where ^ 2 represets the estimate f the errr variace. Of curse i liear regressi, the terms l ^2 ad?2 l L^ jy dier ly by the additive cstat 1 + l 2; thus, it des t matter which quatity is used fr the gdess-f-t term. Yet i may applicatis, such as with autregressive 2 r autregressive mvig-average mdels, there may be a substatial dierece betwee l ^ 2 8

10 ad?2 l L^ jy. Hurvich, Shumway, ad Tsai 1990 preset a cvicig case fr preferrig the latter t the frmer as a gdess-f-t term i deig mdel selecti criteria f the same basic frm f AIC r AICc: e.g., the criteria f Schwarz 1978, Haa ad Qui 1979, etc. The develpmet i Secti 2 shws that the chice f?2 l L^ jy i the deiti f AICc is theretically well mtivated. 9

11 Table 1. Evaluati f 2.11 ad 2.12 fr Varius Values f ad p 2:11 = E f?2 l L jy g? E f?2 l L^ jy g 2:12 = E E f?2 l L jy gj =^? E f?2 l L jy g p p

12 Ackwledgemets The authr wishes t express his gratitude t a aymus referee fr carefully readig ad cstructively critiquig the rigial versi f this mauscript. Refereces Akaike, H. 1973, Ifrmati thery ad a extesi f the maximum likelihd priciple, i: B. N. Petrv ad F. Csaki, eds., 2d Iteratial Sympsium Ifrmati Thery Akademia Kiad, Budapest, pp. 267{281. Akaike, H. 1974, A ew lk at the statistical mdel ideticati, IEEE Trasactis Autmatic Ctrl AC-19, 716{723. Bedrick, E. J. ad Tsai, C. L. 1994, Mdel selecti fr multivariate regressi i small samples, Bimetrics 50, 226{231. Berard, J. M. 1976, Psi digamma fucti, Applied Statistics 25, 315{317. Bickel, P. J. ad Dksum, K. A. 1977, Mathematical Statistics: Basic Ideas ad Selected Tpics Pretice Hall, New Jersey. Haa, E. J. ad Qui, B. G. 1979, The determiati f the rder f a autregressi, Jural f the Ryal Statistical Sciety B 41, 190{195. Hurvich, C. M., Shumway, R. H. ad Tsai, C. L. 1990, Imprved estimatrs f Kullback-Leibler ifrmati fr autregressive mdel selecti i small samples, Bimetrika 77, 709{719. Hurvich, C. M. ad Tsai, C. L. 1989, Regressi ad time series mdel selecti i small samples, Bimetrika 76, 297{307. Hurvich, C. M. ad Tsai, C. L. 1993, A crrected Akaike ifrmati criteri fr vectr autregressive mdel selecti, Jural f Time Series Aalysis 14, 271{279. Ktz, S., Jhs, N. L. ad Read, C. B., eds. 1982, Ecyclpedia f Statistical Scieces, Vlume 2 Wiley, New Yrk. Kullback, S. 1968, Ifrmati Thery ad Statistics Dver, New Yrk. Lihart, H. ad Zucchii, W. 1986, Mdel Selecti Wiley, New Yrk. Schwarz, G. 1978, Estimatig the dimesi f a mdel, The Aals f Statistics 6, 461{464. Sugiura, N. 1978, Further aalysis f the data by Akaike's ifrmati criteri ad the ite crrectis, Cmmuicatis i Statistics A7, 13{26. 11