Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Transcription

1 Stt Comput (27) 27: DOI.7/s Prctcl Byesn model evluton usng leve-one-out cross-vldton nd WAIC Ak Vehtr Andrew Gelmn 2 Jonh Gry 2 Receved: 23 Jnury 26 / Accepted: 22 August 26 / Pulshed onlne: 3 August 26 Sprnger Scence+Busness Med New York 26 Astrct Leve-one-out cross-vldton (LOO) nd the wdely pplcle nformton crteron (WAIC) re methods for estmtng pontwse out-of-smple predcton ccurcy from ftted Byesn model usng the log-lkelhood evluted t the posteror smultons of the prmeter vlues. LOO nd WAIC hve vrous dvntges over smpler estmtes of predctve error such s AIC nd DIC ut re less used n prctce ecuse they nvolve ddtonl computtonl steps. Here we ly out fst nd stle computtons for LOO nd WAIC tht cn e performed usng exstng smulton drws. We ntroduce n effcent computton of LOO usng Preto-smoothed mportnce smplng (PSIS), new procedure for regulrzng mportnce weghts. Although WAIC s symptotclly equl to LOO, we demonstrte tht PSIS-LOO s more roust n the fnte cse wth wek prors or nfluentl oservtons. As yproduct of our clcultons, we lso otn pproxmte stndrd errors for estmted predctve errors nd for comprson of predctve errors etween two models. We mplement the computtons n n R pckge clled loo nd demonstrte usng models ft wth the Byesn nference pckge Stn. B Ak Vehtr k.vehtr@lto.f Andrew Gelmn gelmn@stt.colum.edu Deprtment of Computer Scence, Helsnk Insttute for Informton Technology HIIT, Alto Unversty, Espoo, Fnlnd 2 Deprtment of Sttstcs, Colum Unversty, New York, USA Keywords Byesn computton Leve-one-out crossvldton (LOO) K -fold cross-vldton Wdely pplcle nformton crteron (WAIC) Stn Preto smoothed mportnce smplng (PSIS) Introducton After fttng Byesn model we often wnt to mesure ts predctve ccurcy, for ts own ske or for purposes of model comprson, selecton, or vergng (Gesser nd Eddy 979; Hoetng et l. 999; Vehtr nd Lmpnen 22; Ando nd Tsy 2; Vehtr nd Ojnen 22). Cross-vldton nd nformton crter re two pproches to estmtng out-ofsmple predctve ccurcy usng wthn-smple fts (Akke 973; Stone 977). In ths rtcle we consder computtons usng the log-lkelhood evluted t the usul posteror smultons of the prmeters. Computton tme for the predctve ccurcy mesures should e neglgle compred to the cost of fttng the model nd otnng posteror drws n the frst plce. Exct cross-vldton requres re-fttng the model wth dfferent trnng sets. Approxmte leve-one-out crossvldton (LOO) cn e computed esly usng mportnce smplng (IS; Gelfnd et l. 992; Gelfnd 996) ut the resultng estmte s nosy, s the vrnce of the mportnce weghts cn e lrge or even nfnte (Perugg 997; Epfn et l. 28). Here we propose to use Preto smoothed mportnce smplng (PSIS), new pproch tht provdes more ccurte nd relle estmte y fttng Preto dstruton to the upper tl of the dstruton of the mportnce weghts. PSIS llows us to compute LOO usng mportnce weghts tht would otherwse e unstle. The wdely pplcle or Wtne-Akke nformton crteron (WAIC; Wtne 2) cn e vewed s n

2 44 Stt Comput (27) 27: mprovement on the devnce nformton crteron (DIC) for Byesn models. DIC hs gned populrty n recent yers, n prt through ts mplementton n the grphcl modelng pckge BUGS (Spegelhlter et l. 22; Spegelhlter et l. 994, 23), ut t s known to hve some prolems, whch rse n prt from not eng fully Byesn n tht t s sed on pont estmte (vn der Lnde 25; Plummer 28). For exmple, DIC cn produce negtve estmtes of the effectve numer of prmeters n model nd t s not defned for sngulr models. WAIC s fully Byesn n tht t uses the entre posteror dstruton, nd t s symptotclly equl to Byesn cross-vldton. Unlke DIC, WAIC s nvrnt to prmetrzton nd lso works for sngulr models. Although WAIC s symptotclly equl to LOO, we demonstrte tht PSIS-LOO s more roust n the fnte cse wth wek prors or nfluentl oservtons. We provde dgnostcs for oth PSIS-LOO nd WAIC whch tell when these pproxmtons re lkely to hve lrge errors nd computtonlly more ntensve methods such s K -fold cross-vldton should e used. Fst nd stle computton nd dgnostcs for PSIS-LOO llows sfe use of ths new method n routne sttstcl prctce. As yproduct of our clcultons, we lso otn pproxmte stndrd errors for estmted predctve errors nd for the comprson of predctve errors etween two models. We mplement the computtons n pckge for R (R Core Tem 26) clled loo (Vehtr et l. 26, ) nd demonstrte usng models ft wth the Byesn nference pckge Stn (Stn Development Tem 26, ). All the computtons re fst compred to the typcl tme requred to ft the model n the frst plce. Although the exmples provded n ths pper ll use Stn, the loo pckge s ndependent of Stn nd cn e used wth models estmted y other softwre pckges or custom user-wrtten lgorthms. 2 Estmtng out-of-smple pontwse predctve ccurcy usng posteror smultons Consder dt y,...,y n, modeled s ndependent gven prmeters θ; thus p(y θ) = n = p(y θ). Ths formulton lso encompsses ltent vrle models wth p(y f,θ), where f re ltent vrles. Also suppose we hve pror dstruton p(θ), thus yeldng posteror dstruton p(θ y) nd posteror predctve dstruton p(ỹ y) = p(ỹ θ)p(θ y)dθ. To mntn comprlty wth the gven dtset nd to get eser nterpretton of the dfferences n scle of effectve numer of prmeters, we defne The loo R pckge s vlle from CRAN nd stn-dev/loo. The correspondng code for Mtl, Octve, nd Python s vlle t mesure of predctve ccurcy for the n dt ponts tken one t tme: elpd = expected log pontwse predctve densty for new dtset = p t (ỹ ) log p(ỹ y)d ỹ, () = where p t (ỹ ) s the dstruton representng the true dtgenertng process for ỹ.thep t (ỹ ) s re unknown, nd we wll use cross-vldton or WAIC to pproxmte (). In regresson, these dstrutons re lso mplctly condtoned on ny predctors n the model. See Vehtr nd Ojnen (22) for other pproches to pproxmtng p t (ỹ ) nd dscusson of lterntve predcton tsks. Insted of the log predctve densty log p(ỹ y), other utlty (or cost) functons u(p(ỹ y), ỹ) could e used, such s clssfcton error. Here we tke the log score s the defult for evlutng the predctve densty (Gesser nd Eddy 979; Bernrdo nd Smth 994; Gnetng nd Rftery 27). A helpful quntty n the nlyss s lpd = log pontwse predctve densty = log p(y y) = log p(y θ)p(θ y)dθ. (2) = = The lpd of oserved dt y s n overestmte of the elpd for future dt (). To compute the lpd n prctce, we cn evlute the expectton usng drws from p post (θ), the usul posteror smultons, whch we lel θ s, s =,...,S: lpd = computed log pontwse predctve densty ( ) S = log p(y θ s ). (3) S = s= 2. Leve-one-out cross-vldton The Byesn LOO estmte of out-of-smple predctve ft s elpd loo = log p(y y ), (4) = where p(y y ) = p(y θ)p(θ y )dθ (5) s the leve-one-out predctve densty gven the dt wthout the th dt pont.

3 Stt Comput (27) 27: Rw mportnce smplng As noted y Gelfnd et l. (992), f the n ponts re condtonlly ndependent n the dt model we cn then evlute (5) wth drws θ s from the full posteror p(θ y) usng mportnce rtos r s = p(y θ s ) p(θ s y ) p(θ s y) to get the mportnce smplng leve-one-out (IS-LOO) predctve dstruton, Ss= r s p(ỹ y ) p(ỹ θ s ) Ss= r s. (7) Evlutng ths LOO log predctve densty t the held-out dt pont y, we get p(y y ) S Ss= p(y θ s ) (6). (8) However, the posteror p(θ y) s lkely to hve smller vrnce nd thnner tls thn the leve-one-out dstrutons p(θ y ), nd thus drect use of (8) nduces nstlty ecuse the mportnce rtos cn hve hgh or nfnte vrnce. For smple models the vrnce of the mportnce weghts my e computed nlytclly. The necessry nd suffcent condtons for the vrnce of the cse-deleton mportnce smplng weghts to e fnte for Byesn lner model re gven y Perugg (997). Epfn et l. (28) extend the nlytcl results to generlzed lner models nd nonlner Mchels-Menten models. However, these condtons cn not e computed nlytclly n generl. Koopmn et l. (29) propose to use the mxmum lkelhood ft of the generlzed Preto dstruton to the upper tl of the dstruton of the mportnce rtos nd use the ftted prmeters to form test for whether the vrnce of the mportnce rtos s fnte. If the hypothess test suggests the vrnce s nfnte then they ndon mportnce smplng Truncted mportnce smplng Iondes (28) proposes modfcton of mportnce smplng where the rw mportnce rtos r s re replced y truncted weghts w s = mn(r s, S r), (9) where r = S Ss= r s. Iondes (28) proves tht the vrnce of the truncted mportnce smplng weghts s gurnteed to e fnte, nd provdes theoretcl nd expermentl results showng tht truncton usng the threshold S r gves n mportnce smplng estmte wth men squre error close to n estmte wth cse specfc optml truncton level. The downsde of the truncton s tht t ntroduces s, whch cn e lrge s we demonstrte n our experments Preto smoothed mportnce smplng We cn mprove the LOO estmte usng Preto smoothed mportnce smplng (PSIS; Vehtr nd Gelmn 25), whch pples smoothng procedure to the mportnce weghts. We refly revew the motvton nd steps of PSIS here, efore movng on to focus on the gols of usng nd evlutng predctve nformton crter. As noted ove, the dstruton of the mportnce weghts used n LOO my hve long rght tl. We use the emprcl Byes estmte of Zhng nd Stephens (29) to ft generlzed Preto dstruton to the tl (2 % lrgest mportnce rtos). By exmnng the shpe prmeter k of the ftted Preto dstruton, we re le to otn smple sed estmtes of the exstence of the moments (Koopmn et l. 29). Ths extends the dgnostc pproch of Perugg (997) nd Epfn et l. (28) to e used routnely wth IS-LOO for ny model wth fctorzng lkelhood. Epfn et l. (28) show tht when estmtng the leveone-out predctve densty, the centrl lmt theorem holds f the dstruton of the weghts hs fnte vrnce. These results cn e extended v the generlzed centrl lmt theorem for stle dstrutons. Thus, even f the vrnce of the mportnce weght dstruton s nfnte, f the men exsts then the ccurcy of the estmte mproves s ddtonl posteror drws re otned. When the tl of the weght dstruton s long, drect use of mportnce smplng s senstve to one or few lrgest vlues. By fttng generlzed Preto dstruton to the upper tl of the mportnce weghts, we smooth these vlues. The procedure goes s follows:. Ft the generlzed Preto dstruton to the 2% lrgest mportnce rtos r s s computed n (6). The computton s done seprtely for ech held-out dt pont. In smulton experments wth thousnds nd tens of thousnds of drws, we hve found tht the ft s not senstve to the specfc cutoff vlue (for consstent estmton, the proporton of the smples ove the cutoff should get smller when the numer of drws ncreses). 2. Stlze the mportnce rtos y replcng the M lrgest rtos y the expected vlues of the order sttstcs of the ftted generlzed Preto dstruton ( ) z /2 F, z =,...,M, M

4 46 Stt Comput (27) 27: where M s the numer of smulton drws used to ft the Preto (n ths cse, M =.2 S) nd F s the nverse- CDF of the generlzed Preto dstruton. Lel these new weghts s w s where, gn, s ndexes the smulton drws nd ndexes the dt ponts; thus, for ech there s dstnct vector of S weghts. 3. To gurntee fnte vrnce of the estmte, truncte ech vector of weghts t S 3/4 w, where w s the verge of the S smoothed weghts correspondng to the dstruton holdng out dt pont. Fnlly, lel these truncted weghts s w s. The ove steps must e performed for ech dt pont. The result s vector of weghts w s, s =,...,S, for ech, whch n generl should e etter ehved thn the rw mportnce rtos r s from whch they re constructed. The results cn then e comned to compute the desred LOO estmtes. The PSIS estmte of the LOO expected log pontwse predctve densty s êlpd pss loo = ( Ss= w s log p(y ) θ s ) Ss= w s. () = The estmted shpe prmeter ˆk of the generlzed Preto dstruton cn e used to ssess the rellty of the estmte: If k < 2, the vrnce of the rw mportnce rtos s fnte, the centrl lmt theorem holds, nd the estmte converges quckly. If k s etween 2 nd, the vrnce of the rw mportnce rtos s nfnte ut the men exsts, the generlzed centrl lmt theorem for stle dstrutons holds, nd the convergence of the estmte s slower. The vrnce of the PSIS estmte s fnte ut my e lrge. If k >, the vrnce nd the men of the rw rtos dstruton do not exst. The vrnce of the PSIS estmte s fnte ut my e lrge. If the estmted tl shpe prmeter ˆk exceeds.5, the user should e wrned, lthough n prctce we hve oserved good performnce for vlues of ˆk up to.7. Even f the PSIS estmte hs fnte vrnce, when ˆk exceeds.7 the user should consder smplng drectly from p(θ s y ) for the prolemtc,use K -fold cross-vldton (see Sect. 2.3), or use more roust model. The ddtonl computtonl cost of smplng drectly from ech p(θ s y ) s pproxmtely the sme s smplng from the full posteror, ut t s recommended f the numer of prolemtc dt ponts s not too hgh. A more roust model my lso help ecuse mportnce smplng s less lkely to work well f the mrgnl posteror p(θ s y) nd LOO posteror p(θ s y ) re very dfferent. Ths s more lkely to hppen wth non-roust model nd hghly nfluentl oservtons. A roust model my reduce the senstvty to one or severl hghly nfluentl oservtons, s we show n the exmples n Sect WAIC WAIC (Wtne 2) s n lterntve pproch to estmtng the expected log pontwse predctve densty nd s defned s êlpd wc = lpd p wc, () where p wc s the estmted effectve numer of prmeters nd s computed sed on the defnton 2 p wc = vr post (log p(y θ)), (2) = whch we cn clculte usng the posteror vrnce of the log predctve densty for ech dt pont y, tht s, Vs= S log p(y θ s ), where Vs= S represents the smple vrnce, Vs= S s = S Ss= ( s ā) 2. Summng over ll the dt ponts y gves smulton-estmted effectve numer of prmeters, p wc = = Vs= S ( log p(y θ s ) ). (3) For DIC, there s smlr vrnce-sed computton of the numer of prmeters tht s notorously unrelle, ut the WAIC verson s more stle ecuse t computes the vrnce seprtely for ech dt pont nd then tkes the sum; the summng yelds stlty. The effectve numer of prmeters p wc cn e used s mesure of complexty of the model, ut t should not e overnterpreted, s the orgnl gol s to estmte the dfference etween lpd nd elpd. As shown y Gelmn et l. (24) nd demonstrted lso n Sect. 4, n the cse of wek pror, p wc cn severely underestmte the dfference etween lpd nd elpd. For p wc there s no smlr theory s for the moments of the mportnce smplng weght dstruton, ut sed on our smulton experments t seems tht p wc s unrelle f ny of the terms Vs= S log p(y θ s ) exceeds.4. The dfferent ehvor of LOO nd WAIC seen n the experments cn e understood y comprng Tylor seres pproxmtons. By defnng genertng functon of functonl cumulnts, 2 In Gelmn et l. (23), the vrnce-sed p wc defned here s clled p wc 2. There s lso men-sed formul, p wc, whch we do not use here.

5 Stt Comput (27) 27: F(α) = log E post (p(y θ) α ), (4) = nd pplyng Tylor expnson of F(α) round wth α = we otn n expnson of lpd loo elpd loo = F () 2 F () + 6 F(3) () ( ) F () (). (5)! =4 From the defnton of F(α) we get F() = F() = log E post (p(y θ)) F () = F () = = E post (log p(y θ)) = vr post (log p(y θ)). (6) = Furthermore lpd = F() = F () + 2 F () + 6 F(3) () + nd the expnson for WAIC s then WAIC = F() F () = F () 2 F () + 6 F(3) () + =4 =4 F () (),! (7) F () ().! (8) The frst three terms of the expnson of WAIC mtch the expnson of LOO, nd the rest of the terms mtch the expnson of lpd. Wtne (2) rgues tht, symptotclly, the ltter terms hve neglgle contruton nd thus symptotc equvlence wth LOO s otned. However, the error cn e sgnfcnt n the cse of fnte n nd wek pror nformton s shown y Gelmn et l. (24), nd demonstrted lso n Sect. 4. If the hgher order terms re not neglgle, then WAIC s sed towrds lpd. To reduce ths s t s possle to compute ddtonl seres terms, ut computng hgher moments usng fnte posteror smple ncreses the vrnce of the estmte nd, sed on our experments, t s more dffcult to control the s-vrnce trdeoff thn n PSIS-LOO. WAIC s lrger s compred to LOO s lso demonstrted y Vehtr et l. (26, ) n the cse of Gussn processes wth dstrutonl posteror pproxmtons. In the experments we lso demonstrte tht we cn use truncted IS-LOO wth hevy truncton to otn smlr s towrds lpd nd smlr estmte vrnce s n WAIC. 2.3 K-fold cross-vldton In ths pper we focus on leve-one-out cross-vldton nd WAIC, ut, for sttstcl nd computtonl resons, t cn mke sense to cross-vldte usng K n hold-out sets. In some wys, K -fold cross-vldton s smpler thn leveone-out cross-vldton ut n other wys t s not. K -fold cross-vldton requres refttng the model K tmes whch cn e computtonlly expensve wheres pproxmtve LOO methods, such s PSIS-LOO, requre only one evluton of the model. If n PSIS-LOO ˆk >.7 forfew we recommend smplng drectly from ech correspondng p(θ s y ),ut f there re more thn K prolemtc, then we recommend checkng the results usng K -fold cross-vldton. Vehtr nd Lmpnen (22) demonstrte cses where IS-LOO fls (ccordng to effectve smple sze estmtes nsted of the ˆk dgnostc proposed here) for lrge numer of nd K- fold-cv produces more relle results. In Byesn K -fold cross-vldton, the dt re prttoned nto K susets y k,fork =,...,K, nd then the model s ft seprtely to ech trnng set y ( k), thus yeldng posteror dstruton p post( k) (θ) = p(θ y ( k) ).Ifthe numer of prttons s smll ( typcl vlue n the lterture s K = ), t s not so costly to smply re-ft the model seprtely to ech trnng set. To mntn consstency wth LOO nd WAIC, we defne predctve ccurcy for ech dt pont, so tht the log predctve densty for y, f t s n suset k,s log p(y y ( k) ) = log p(y θ)p(θ y ( k) )dθ, k. (9) Assumng the posteror dstruton p(θ y ( k) ) s summrzed y S smulton drws θ k,s, we clculte ts log predctve densty s ( ) S êlpd = log p(y θ k,s ) (2) S s= usng the smultons correspondng to the suset k tht contns dt pont. We then sum to get the estmte êlpd xvl = êlpd. (2) = There remns s s the model s lernng from frcton K less of the dt. Methods for correctng ths s exst ut re rrely used s they cn ncrese the vrnce, nd f K the sze of the s s typclly smll compred to

6 48 Stt Comput (27) 27: the vrnce of the estmte (Vehtr nd Lmpnen 22). In our experments, exct LOO s the sme s K -fold-cv wth K = N nd we lso nlyze the effect of ths s nd s correcton n Sect For K -fold cross-vldton, f the sujects re exchngele, tht s, the order does not contn nformton, then there s no need for rndom selecton. If the order does contn nformton, e.g. n survvl studes the lter ptents hve shorter follow-ups, then rndomzton s often useful. In most cses we recommend prttonng the dt nto susets y rndomly permutng the oservtons nd then systemclly dvdng them nto K sugroups. If the sujects re exchngele, tht s, the order does not contn nformton, then there s no need for rndom selecton, ut f the order does contn nformton, e.g. n survvl studes the lter ptents hve shorter follow-ups, then rndomzton s useful. In some cses t my e useful to strtfy to otn etter lnce mong groups. See Vehtr nd Lmpnen (22), Arolot nd Celsse (2), nd Vehtr nd Ojnen (22) for further dscusson of these ponts. As the dt cn e dvded n mny wys nto K groups t ntroduces ddtonl vrnce n the estmtes, whch s lso evdent from our experments. Ths vrnce cn e reduced y repetng K -fold-cv severl tmes wth dfferent permuttons n the dt dvson, ut ths wll further ncrese the computtonl cost. 2.4 Dt dvson The purpose of usng LOO or WAIC s to estmte the ccurcy of the predctve dstruton p(ỹ y). Computton of PSIS-LOO nd WAIC (nd AIC nd DIC) s sed on computng terms log p(y y) = log p(y θ)p(θ y) ssumng some greed-upon dvson of the dt y nto ndvdul dt ponts y. Although often y wll denote sngle sclr oservton, n the cse of herrchcl dt, t my denote group of oservtons. For exmple, n cogntve or medcl studes we my e nterested n predcton for new suject (or ptent), nd thus t s nturl n cross-vldton to consder n pproch where y would denote ll oservtons for sngle suject nd y would denote the oservtons for ll the other sujects. In theory, we cn use PSIS-LOO nd WAIC n ths cse, too, ut s the numer of oservtons per suject ncreses t s more lkely tht they wll not work s well. The fct tht mportnce smplng s dffcult n hgher dmensons s well known nd s demonstrted for IS-LOO y Vehtr nd Lmpnen (22) nd for PSIS y Vehtr nd Gelmn (25). The sme prolem cn lso e shown to hold for WAIC. If dgnostcs wrn out the rellty of PSIS-LOO (or WAIC), then K -fold cross-vldton cn e used y tkng nto ccount the herrchcl structure n the dt when dong the dt dvson s demonstrted, for exmple, y Vehtr nd Lmpnen (22). 3 Implementton n Stn We hve set up code to mplement LOO, WAIC, nd K -fold cross-vldton n R nd Stn so tht users wll hve quck nd convenent wy to ssess nd compre model fts. Implementton s not utomtc, though, ecuse of the need to compute the seprte fctors p(y θ) n the lkelhood. Stn works wth the jont densty nd n ts usul computtons does not know whch prts come from the pror nd whch from the lkelhood. Nor does Stn n generl mke use of ny fctorzton of the lkelhood nto peces correspondng to ech dt pont. Thus, to compute these mesures of predctve ft n Stn, the user needs to explctly code the fctors of the lkelhood (ctully, the terms of the log-lkelhood) s vector. We cn then pull prt the seprte terms nd compute cross-vldton nd WAIC t the end, fter ll smultons hve een collected. Smple code for crryng out ths procedure usng Stn nd the loo R pckge s provded n Appendx. Ths code cn e dpted to pply our procedure n other computng lnguges. Although the mplementton s not utomtc when wrtng custom Stn progrms, we cn crete mplementtons tht re utomtc for users of our new rstnrm R pckge (Gry nd Goodrch 26). rstnrm provdes hghlevel nterfce to Stn tht enles the user to specfy mny of the most common ppled Byesn regresson models usng stndrd R modelng syntx (e.g. lke tht of glm). The models re then estmted usng Stn s lgorthms nd the results re returned to the user n form smlr to the ftted model ojects to whch R users re ccustomed. For the models mplemented n rstnrm, we hve preprogrmmed mny tsks, ncludng computng nd svng the pontwse predctve mesures nd mportnce rtos whch we use to compute WAIC nd PSIS-LOO. The loo method for rstnrm models requres no ddtonl progrmmng from the user fter fttng model, s we cn compute ll of the needed qunttes nternlly from the contents of the ftted model oject nd then pss them to the functons n the loo pckge. Exmples of usng loo wth rstnrm cn e found n the rstnrm vgnettes, nd we lso provde n exmple n Appendx 3 of ths pper. 4 Exmples We llustrte wth sx smple exmples: two exmples from our erler reserch n computng the effectve numer of prmeters n herrchcl model, three exmples tht were used y Epfn et l. (28) to llustrte the estmton of the vrnce of the weght dstruton, nd one exmple of multlevel regresson from our erler ppled reserch. For ech exmple we used the Stn defult of 4 chns run for wrmup nd post-wrmup tertons, yeldng

7 Stt Comput (27) 27: totl of 4 sved smulton drws. Wth Gs smplng or rndom-wlk Metropols, 4 s not lrge numer of smulton drws. The lgorthm used y Stn s Hmltonn Monte Crlo wth No-U-Turn-Smplng (Hoffmn nd Gelmn 24), whch s much more effcent, nd s lredy more thn suffcent n mny rel-world settngs. In these exmples we followed stndrd prctce nd montored convergence nd effectve smple szes s recommended y Gelmn et l. (23). We performed ndependent replctons of ll experments to otn estmtes of vrton. For the exct LOO results nd convergence plots we run longer chns to otn totl of, drws (except for the rdon exmple whch s much slower to run). 4. Exmple: Scled 8 schools For our frst exmple we tke n nlyss of n educton experment used y Gelmn et l. (24) to demonstrte the use of nformton crter for herrchcl Byesn models. The gol of the study ws to mesure the effects of test preprton progrm conducted n eght dfferent hgh schools n New Jersey. A seprte rndomzed experment ws conducted n ech school, nd the dmnstrtors of ech school mplemented the progrm n ther own wy. Run (98) performed Byesn met-nlyss, prtlly poolng the eght estmtes towrd common men. The model hs the form, y N(θ,σ 2) nd θ N(μ, τ 2 ),for =,...,n = 8, wth unform pror dstruton on (μ, τ). The mesurements y nd uncertntes σ re the estmtes nd stndrd errors from seprte regressons performed for ech school, s shown n Tle. The test scores for the ndvdul students re no longer vlle. Ths model hs eght prmeters ut they re constrned through ther herrchcl dstruton nd re not estmted Tle In controlled study, ndependent rndomzed experments were conducted n 8 dfferent hgh schools to estmte the effect of specl preprton for college dmsson tests School Estmted effect, y j Stndrd error of estmte, σ j A 28 5 B 8 C 3 6 D 7 E 9 F G 8 H 2 8 Ech row of the tle gves n estmte nd stndrd error from one of the schools. A herrchcl Byesn model ws ft to perform metnlyss nd use prtl poolng to get more ccurte estmtes of the 8 effects. From Run (98) ndependently; thus we would ntcpte the effectve numer of prmeters should e some numer etween nd 8. To etter llustrte the ehvor of LOO nd WAIC, we repet the nlyss, resclng the dt ponts y y fctor rngng from. to 4 whle keepng the stndrd errors σ unchnged. Wth smll dt sclng fctor the herrchcl model ners complete poolng nd wth lrge dt sclng fctor the model pproches seprte fts to the dt for ech school. Fgure shows êlpd for the vrous LOO pproxmton methods s functon of the sclng fctor, sed on 4 smulton drws t ech grd pont. When the dt sclng fctor s smll (here, less thn.5), ll mesures lrgely gree. As the dt sclng fctor ncreses nd the model pproches no poolng, the populton pror for θ gets flt nd p wc p 2. Ths s correct ehvor, s dscussed y Gelmn et l. (24). In the cse of exct LOO, lpd êlpd loo cn e lrger thn p. As the pror for θ pproches fltness, the log predctve densty p post( ) (y ). At the sme tme, the full posteror ecomes n ndequte pproxmton to p post( ) nd ll pproxmtons ecome poor pproxmtons to the ctul out-of-smple predcton error under the model. WAIC strts to fl when one of the posteror vrnces of the log predctve denstes exceeds.4. LOO pproxmtons work well even f the tl shpe k of the generlzed Preto dstruton s etween 2 nd, nd the vrnce of the rw mportnce rtos s nfnte. The error of LOO pproxmtons ncreses wth k, wth clerer dfference etween the methods when k > Exmple: Smulted 8 schools In the prevous exmple, we used exct LOO s the gold stndrd. In ths secton, we generte smulted dt from the sme sttstcl model nd compre predctve performnce on ndependent test dt. Even when the numer of oservtons n s fxed, s the scle of the populton dstruton ncreses we oserve the effect of wek pror nformton n herrchcl models dscussed n the prevous secton nd y Gelmn et l. (24). Comprng the error, s nd vrnce of the vrous pproxmtons, we fnd tht PSIS-LOO offers the est lnce. For =,...,n = 8, we smulte θ, N(μ,τ 2 ) nd y N(θ,,σ 2, ), where we set σ, =,μ =, nd τ {, 2,...,3}. The smulted dt s smlr to the rel 8 schools dt, for whch the emprcl estmte s ˆτ. For ech vlue of τ we generte trnng sets of sze 8 nd one test dt set of sze. Posteror nference s sed on 4 drws for ech constructed model. Fgure 2 shows the root men squre error (RMSE) for the vrous LOO pproxmton methods s functon of τ, the scle of the populton dstruton. When τ s lrge ll of the pproxmtons eventully hve ever ncresng

8 42 Stt Comput (27) 27: elppd WAIC TIS-LOO 4 IS-LOO 42 PSIS-LOO LOO Dt sclng fctor lppd elppd Dt sclng fctor c.2 d.6 Tl shpe k V s= S log p(y θ s ) Dt sclng fctor Dt sclng fctor Fg. 8 Schools exmple: WAIC, Truncted Importnce Smplng LOO, Importnce Smplng LOO, Preto Smoothed Importnce Smplng LOO, nd exct LOO (whch n ths cse corresponds to eghtfold-cv); estmted effectve numer of prmeters for ech of these mesures; c tl shpe ˆk for the mportnce weghts; nd d the posteror vrnces of the log predctve denstes, for scled versons of the 8 schools dt (the orgnl oservtons y hve een multpled y common fctor). We consder sclng fctors rngng from. (correspondng to ner-zero vrton of the underlyng prmeters mong the schools) to 4 (mplyng tht the true effects n the schools vry y much more thn ther stndrd errors of mesurement). As the sclng ncreses, eventully LOO pproxmtons nd WAIC fl to pproxmte exct LOO s the leve-one-out posterors re not close to the full posteror. When the estmted tl shpe ˆk exceeds, the mportnceweghted LOO pproxmtons strt to fl. When posteror vrnces of the log predctve denstes exceeds.4, WAIC strts to fl. PSIS-LOO performs the est mong the pproxmtons consdered here RMSE, whle exct LOO hs n upper lmt. For medum scles the pproxmtons hve smller RMSE thn exct LOO. As dscussed lter, ths s explned y the dfference n the vrnce of the estmtes. For smll scles WAIC hs slghtly smller RMSE thn the other methods (ncludng exct LOO). Wtne (2) shows tht WAIC gves n symptotclly unsed estmte of the out-of-smple predcton error ths does not hold for herrchcl models wth wek pror nformton s shown y Gelmn et l. (24) ut exct LOO s slghtly sed s the LOO posterors use only n oservtons. WAIC s dfferent ehvor cn e understood through the truncted Tylor seres correcton to the lpd, tht s, not usng the entre seres wll s t towrds lpd (see Sect. 2.2). The s n LOO s neglgle when n s lrge, ut wth smll n t cn e e lrger. Fgure 2 shows RMSE for the s corrected LOO pproxmtons usng the frst order correcton of Burmn (989). For smll scles the error of s corrected LOOs s smller thn WAIC. When the scle ncreses the RMSEs re close to the non-corrected versons. Although the s correcton s esy to compute, the dfference n ccurcy s neglgle for most pplctons. We shll dscuss Fg. 2c n moment, ut frst consder Fg. 3, whch shows the RMSE of the pproxmton methods nd the lpd of oserved dt decomposed nto s nd stndrd devton. All methods (except the lpd of oserved dt) hve smll ses nd vrnces wth smll populton dstruton scles. Bs corrected exct LOO hs prctclly zero s for ll scle vlues ut the hghest vrnce. When the scle ncreses the LOO pproxmtons eventully fl nd s ncreses. As the pproxmtons strt to fl, there s certn regon where mplct shrnkge towrds the lpd of oserved dt decelertes the ncrese n RMSE s the vrnce s reduced, even f the s contnues to grow. If the gol were to mnmze the RMSE for smller nd medum scles, we could lso shrnk exct LOO nd ncrese

9 Stt Comput (27) 27: WAIC WAIC c TIS-LOO BC-PSIS-LOO IS-LOO BC-LOO 4 PSIS-LOO 4 LOO RMSE 2 RMSE 2 RMSE WAIC TIS(/4)-LOO PSIS-LOO BC-LOO Shrunk LOO Populton dstruton scle τ Populton dstruton scle τ Populton dstruton scle τ Fg. 2 Smulted 8 schools exmple: Root men squre error of WAIC, Truncted Importnce Smplng LOO, Importnce Smplng LOO, Preto Smoothed Importnce Smplng LOO, nd exct LOO wth the true predctve performnce computed usng ndependently smulted test dt; the error for ll the methods ncreses, ut the RMSE of exct LOO hs n upper lmt. Eventully the LOO pproxmtons nd WAIC fl to return exct LOO, s the leve-one-out posterors re not close to the full posteror. When the estmted tl shpe k exceeds, the mportnce-weghted LOO pproxmtons strt to fl. Among the pproxmtons IS-LOO hs the smllest RMSE s t hs the smllest s, nd s the tl shpe k s mostly elow, t does not fl dly. Root men squre error of WAIC, s corrected Preto Smoothed Importnce Smplng LOO, nd s corrected exct LOO wth the true predctve performnce computed usng ndependently smulted test dt. The s correcton lso reduces RMSE, hvng the clerest mpct wth smller populton dstruton scles, ut overll the reducton n RMSE s neglgle. c Root men squre error of WAIC, Truncted Importnce Smplng LOO wth hevy truncton ( 4 S r), Preto Smoothed Importnce Smplng LOO, s corrected exct LOO, nd shrunk exct LOO wth the true predctve performnce computed usng ndependently smulted test dt. Truncted Importnce Smplng LOO wth hevy truncton mtches WAIC ccurtely. Shrnkng exct LOO towrds the lpd of oserved dt reduces the RMSE for some scle vlues wth smll ncrese n error for lrger scle vlues shrnkge n pproxmtons. Fgure 2c shows the RMSE of the LOO pproxmtons wth two new choces. Truncted Importnce Smplng LOO wth very hevy truncton (to 4 S r) closely mtches the performnce of WAIC. In the experments not ncluded here, we lso oserved tht ddng more correct Tylor seres terms to WAIC wll mke t ehve smlr to Truncted Importnce Smplng wth less truncton (see dscusson of Tylor seres expnson n Sect. 2.2). Shrunk exct LOO (α elpd loo + ( α) lpd, wth α =.85 chosen y hnd for llustrtve purposes only) hs smller RMSE for smll nd medum scle vlues s the vrnce s reduced, ut the prce s ncresed s t lrger scle vlues. If the gol s roust estmton of predctve performnce, then exct LOO s the est generl choce ecuse the error s lmted even n the cse of wek prors. Of the pproxmtons, PSIS-LOO offers the est lnce s well s dgnostcs for dentfyng when t s lkely flng. 4.3 Exmple: Lner regresson for stck loss dt To check the performnce of the proposed dgnostc for our second exmple we nlyze the stck loss dt used y Perugg (997) whch s known to hve nlytclly proven nfnte vrnce of one of the mportnce weght dstrutons. The dt consst of n = 2 dly oservtons on one outcome nd three predctors pertnng to plnt for the oxdton of mmon to ntrc cd. The outcome y s n nverse mesure of the effcency of the plnt nd the three predctors x, x 2, nd x 3 mesure rte of operton, temperture of coolng wter, nd ( trnsformton of the) concentrton of crcultng cd. Perugg (997) shows tht the mportnce weghts for leve-one-out cross-vldton for the dt pont y 2 hve nfnte vrnce. Fgure 4 shows the dstruton of the estmted tl shpes ˆk nd estmton errors compred to LOO n ndependent Stn runs. 3 The estmtes of the tl shpe ˆk for = 2 suggest tht the vrnce of the rw mportnce rtos s nfnte, however the generlzed centrl lmt theorem for stle dstrutons holds nd we cn stll otn n ccurte estmte of the component of LOO for ths dt pont usng PSIS. Fgure 5 shows tht f we contnue smplng, the estmtes for oth the tl shpe ˆk nd êlpd do converge (lthough slowly s ˆk s close to ). As the convergence s slow t would e more effcent to smple drectly from p(θ s y ) for the prolemtc. Hgh estmtes of the tl shpe prmeter ˆk ndcte tht the full posteror s not good mportnce smplng pproxmton to the desred leve-one-out posteror, nd thus the oservton s surprsng ccordng to the model. It s nturl to consder n lterntve model. We tred replcng the norml oservton model wth Student-t to mke the model 3 Smoothed densty estmtes were mde usng logstc Gussn process (Vehtr nd Rhmäk 24).

10 422 Stt Comput (27) 27: WAIC PSIS-LOO BC-LOO LPD WAIC PSIS-LOO BC-LOO LPD Bs 2 Std Populton dstruton scle τ Populton dstruton scle τ Fg. 3 Smulted 8 schools exmple: Asolute s of WAIC, Preto Smoothed Importnce Smplng LOO, s corrected exct LOO, nd the lpd (log predctve densty) of oserved dt wth the true predctve performnce computed usng ndependently smulted test dt; stndrd devton for ech of these mesures; All methods except the lpd of oserved dt hve smll ses nd vrnces wth smll populton dstruton scles. When the scle ncreses the s of WAIC ncreses fster thn the s of the other methods (except the lpd of oserved dt). Bs corrected exct LOO hs prctclly zero s for ll scle vlues. WAIC nd Preto Smoothed Importnce Smplng LOO hve lower vrnce thn exct LOO, s they re shrunk towrds the lpd of oserved dt, whch hs the smllest vrnce wth ll scles ˆk LOO PSIS-LOO Fg. 4 Stck loss exmple wth norml errors: Dstrutons of tl shpe estmtes nd PSIS-LOO estmton errors compred to LOO, from ndependent Stn runs. The pontwse clculton of the terms n PSIS-LOO revels tht much of the uncertnty comes from sngle dt pont, nd t could mke sense to smply re-ft the model to the suset nd compute LOO drectly for tht pont Shpe prmeter k for = Numer of posteror drws 5 elppd for = WAIC TIS-LOO IS-LOO PSIS-LOO LOO Numer of posteror drws 5 Fg. 5 Stck loss exmple wth norml errors: Tl shpe estmte nd LOO pproxmtons for the dffcult pont, = 2. When more drws re otned, the estmtes converge (slowly) followng the generlzed centrl lmt theorem more roust for the possle outler. Fgure 6 shows the dstruton of the estmted tl shpes ˆk nd estmton errors for PSIS-LOO compred to LOO n ndependent Stn runs for the Student-t lner regresson model. The estmted tl shpes nd the errors n computng ths component of LOO re smller thn wth Gussn model.

11 Stt Comput (27) 27: ˆk LOO PSIS-LOO Fg. 6 Stck loss exmple wth Student-t errors: Dstrutons of tl shpe estmtes nd PSIS-LOO estmton errors compred to LOO, from ndependent Stn runs. The computtons re more stle thn wth norml errors (compre to Fg. 4).4 Shpe prmeter k LOO PSIS-LOO Fg. 7 Puromycn exmple: Dstrutons of tl shpe estmtes nd PSIS-LOO estmton errors compred to LOO, from ndependent Stn runs. In n ppled exmple we would only perform these clcultons once, ut here we replcte tmes to gve sense of the Monte Crlo error of our procedure 4.4 Exmple: Nonlner regresson for Puromycn recton dt As nonlner regresson exmple, we use the Puromycn ochemcl recton dt lso nlyzed y Epfn et l. (28). For group of cells not treted wth the drug Puromycn, there re n = mesurements of the ntl velocty of recton, V, otned when the concentrton of the sustrte ws set t gven postve vlue, c. Velocty on concentrton s gven y the Mchels-Menten relton, V N(mc /(κ + c ), σ 2 ). Epfn et l. (28) show tht the rw mportnce rtos for oservton = hve nfnte vrnce. Fgure 7 shows the dstruton of the estmted tl shpes k nd estmton errors compred to LOO n ndependent Stn runs. The estmtes of the tl shpe k for = suggest tht the vrnce of the rw mportnce rtos s nfnte. However, the generlzed centrl lmt theorem for stle dstrutons stll holds nd we cn get n ccurte estmte of the correspondng term n LOO. We could otn more drws to reduce the Monte Crlo error, or gn consder more roust model. 4.5 Exmple: Logstc regresson for leukem survvl Our next exmple uses logstc regresson model to predct survvl of leukem ptents pst 5 weeks from dgnoss. These dt were lso nlyzed y Epfn et l. (28). Explntory vrles re whte lood cell count t dgnoss nd whether Auer rods nd/or sgnfcnt grnulture of the leukemc cells n the one mrrow t dgnoss were present. Epfn et l. (28) show tht the rw mportnce rtos for dt pont = 5 hve nfnte vrnce. Fgure 8 shows the dstruton of the estmted tl shpes k nd estmton errors compred to LOO n ndependent Stn runs. The estmtes of the tl shpe k for = 5 suggest tht the men nd vrnce of the rw mportnce rtos do not exst. Thus the generlzed centrl lmt theorem does not hold. Fgure 9 shows tht f we contnue smplng, the tl shpe estmte stys ove nd êlpd wll not converge. Lrge estmtes for the tl shpe prmeter ndcte tht the full posteror s not good mportnce smplng pproxmton for the desred leve-one-out posteror, nd thus the oservton s surprsng. The orgnl model used the whte lood cell count drectly s predctor, nd t would e nturl

12 424 Stt Comput (27) 27: Shpe prmeter k LOO PSIS-LOO Fg. 8 Leukem exmple: Dstrutons of tl shpe estmtes nd PSIS-LOO estmton errors compred to LOO, from ndependent Stn runs. The pontwse clculton of the terms n PSIS-LOO revels tht much of the uncertnty comes from sngle dt pont, nd t could mke sense to smply re-ft the model to the suset nd compute LOO drectly for tht pont Shpe prmeter k for = Numer of posteror drws 5 elppd for = WAIC TIS-LOO IS-LOO PSIS-LOO LOO Numer of posteror drws 5 Fg. 9 Leukem exmple: Dstrutons of tl shpe estmte nd LOO pproxmtons for = 5. If we contnue smplng, the tl shpe estmte stys ove nd êlpd wll not converge Shpe prmeter k LOO PSIS-LOO Fg. Leukem exmple wth log-trnsformed predctor: Dstrutons of tl shpe estmtes for ech dt pont nd PSIS-LOO estmton errors compred to LOO, from ndependent Stn runs. Computtons re more stle compred to the model ft on the orgnl scle nd dsplyed n Fg. 8 to use ts logrthm nsted. Fgure shows the dstruton of the estmted tl shpes k nd estmton errors compred to LOO n ndependent Stn runs for ths modfed model. Both the tl shpe vlues nd errors re now smller. 4.6 Exmple: Multlevel regresson for rdon contmnton Gelmn nd Hll (27) descre study conducted y the Unted Sttes Envronmentl Protecton Agency desgned to

13 Stt Comput (27) 27: Shpe prmeter k LOO PSIS-LOO Fg. Rdon exmple: Tl shpe estmtes for ech pont s contruton to LOO, nd error n PSIS-LOO ccurcy for ech dt pont, ll sed on sngle ft of the model n Stn mesure levels of the crcnogen rdon n houses throughout the Unted Sttes. In hgh concentrtons rdon s known to cuse lung cncer nd s estmted to e responsle for severl thousnds of deths every yer n the Unted Sttes. Here we focus on the smple of 99 houses n the stte of Mnnesot, whch re dstruted (unevenly) throughout 85 countes. We ft the followng multlevel lner model to the rdon dt ( α j β j ( y N ) N α j[] + β j[] x,σ 2), =,...,99 (( γ α + γ αu ) ( j σ 2 )), α ρσ α σ β, γ β + γ β u j j =,...,85, ρσ α σ β where y s the logrthm of the rdon mesurement n the th house, x = for mesurement mde n the sement nd x = f on the frst floor (t s known tht rdon enters more esly when house s ult nto the ground), nd the county-level predctor u j s the logrthm of the sol urnum level n the county. The resdul stndrd devton σ nd ll hyperprmeters re gven wekly nformtve prors. Code for fttng ths model s provded n Appendx 3. The smple sze n ths exmple (n = 99) s not huge ut s lrge enough tht t s mportnt to hve computtonl method for LOO tht s fst for ech dt pont. Although the MCMC for the full posteror nference (usng four prllel chns) fnshed n only 93 s, the computtons for exct rute force LOO requre fttng the model 99 tmes nd took more thn 2 h to complete (Mcook Pro, 2.6 GHz Intel Core 7). Wth the sme hrdwre the PSIS-LOO computtons took less thn 5 s. Fgure shows the results for the rdon exmple nd ndeed the estmted shpe prmeters k re smll nd ll of the tested methods re ccurte. For two oservtons the σ 2 β estmte of k s slghtly hgher thn the preferred threshold of.7, ut we cn esly compute the elpd contrutons for these ponts drectly nd then comne wth the PSIS-LOO estmtes for the remnng oservtons. 4 Ths s the procedure we refer to s PSIS-LOO+ n Sect. 4.7 elow. 4.7 Summry of exmples Tle 2 compres the performnce of Preto smoothed mportnce smplng, rw mportnce smplng, truncted mportnce smplng, nd WAIC for estmtng expected out-of-smple predcton ccurcy for ech of the exmples n Sects Models were ft n Stn to otn 4 smulton drws. In ech cse, the dstrutons come from ndependent smultons of the entre fttng process, nd the root men squred error s evluted y comprng to exct LOO, whch ws computed y seprtely fttng the model to ech leve-one-out dtset for ech exmple. The lst three lnes of Tle 2 show ddtonlly the performnce of PSIS-LOO comned wth drect smplng for the prolemtc wth ˆk >.7 (PSIS-LOO+), -fold-cv, nd tmes repeted -fold-cv. 5 For the Stcks-N, Puromycn, nd Leukem exmples, there ws one wth ˆk >.7, nd thus the mprovement hs the sme computtonl cost s the full posteror nference. -fold-cv hs hgher RMSE thn LOO pproxmtons except n the Leukem cse. The hgher RMSE of -fold-cv s due to ddtonl vrnce from the dt dvson. The repeted -fold-cv hs smller RMSE thn sc -fold-cv, ut now the cost of computton s lredy tmes the orgnl full posteror nference. 4 As expected, the two slghtly hgh estmtes for k correspond to prtculrly nfluentl oservtons, n ths cse houses wth extremely low rdon mesurements. 5 -fold-cv results were not computed for dt sets wth n, nd tmes repeted -fold-cv ws not fesle for the rdon exmple due to the computton tme requred.

14 426 Stt Comput (27) 27: Tle 2 Root men squre error for dfferent computtons of LOO s determned from smulton study, n ech cse sed on runnng Stn to otn 4 posteror drws nd repetng tmes Method 8 schools Stcks-N Stcks-t Puromycn Leukem Leukem-log Rdon PSIS-LOO IS-LOO TIS-LOO WAIC PSIS-LOO fold-cv fold-cv Methods compred re Preto smoothed mportnce smplng (PSIS), PSIS wth drect smplng f ˆk >.7 (PSIS-LOO+), rw mportnce smplng (IS), truncted mportnce smplng (TIS), WAIC, -fold-cv, nd tmes repeted -fold-cv for the dfferent exmples consdered n Sects : the herrchcl model for the 8 schools, the stck loss regresson (wth norml nd t models), nonlner regresson for Puromycn, logstc regresson for leukem (n orgnl nd log scle), nd herrchcl lner regresson for rdon. See text for explntons. PSIS-LOO nd PSIS-LOO+ gve the smllest error n ll exmples except the 8 schools, where t gves the second smllest error. In ech cse, we compred the estmtes to the correct vlue of LOO y the rute-force procedure of fttng the model seprtely to ech of the n possle trnng sets for ech exmple Tle 3 Prtl replcton of Tle 2 usng 6, posteror drws n ech cse Method 8 schools Stcks-N Stcks-t Puromycn Leukem Leukem-log Rdon PSIS-LOO IS-LOO TIS-LOO WAIC Monte Crlo errors re slghtly lower. The errors for WAIC do not smply scle wth / S ecuse most of ts errors come from s not vrnce These results show tht K -fold-cv s needed only f LOO pproxmtons fl dly (see lso the results n Vehtr nd Lmpnen 22). As mesured y root men squred error, PSIS consstently performs well. In generl, when IS-LOO hs prolems t s ecuse of the hgh vrnce of the rw mportnce weghts, whle TIS-LOO nd WAIC hve prolems ecuse of s. Tle 3 shows replcton usng 6, Stn drws for ech exmple. The results re smlr results nd PSIS- LOO s le to mprove the most gven ddtonl drws. 5 Stndrd errors nd model comprson We next consder some pproches for ssessng the uncertnty of cross-vldton nd WAIC estmtes of predcton error. We present these methods n seprte secton rther thn n our mn development ecuse, s dscussed elow, the dgnostcs cn e dffcult to nterpret when the smple sze s smll. 5. Stndrd errors The computed estmtes êlpd loo nd êlpd wc re ech defned s the sum of n ndependent components so t s trvl to compute ther stndrd errors y computng the stndrd devton of the n components nd multplyng y n.for exmple, defne êlpd loo, = log p(y y ), (22) so tht êlpd loo n (4)sthesumofthesen ndependent terms. Then se (êlpd loo ) = nv =êlpd n loo,, (23) nd smlrly for WAIC nd K -fold cross-vldton. The effectve numers of prmeters, p loo nd p wc, re lso sums of ndependent terms so we cn compute ther stndrd errors n the sme wy. These stndrd errors come from consderng the n dt ponts s smple from lrger populton or, equvlently, s ndependent relztons of n error model. One cn lso compute Monte Crlo stndrd errors rsng from the fnte numer of smulton drws usng the formul from Gelmn et l. (23) whch uses oth etween nd wthn-chn nformton nd s mplemented n Stn. In prctce we expect Monte Crlo stndrd errors to not e so nterestng ecuse we would hope to hve enough smultons tht the computtons re stle, ut t could mke sense to look t them just

15 Stt Comput (27) 27: to check tht they re low enough to e neglgle compred to smplng error (whch scles lke /n rther thn /S). The stndrd error (23) nd the correspondng formul for se (êlpd wc ) hve two dffcultes when the smple sze s low. Frst, the n terms re not strctly ndependent ecuse they re ll computed from the sme set of posteror smultons θ s. Ths s generc ssue when evlutng the stndrd error of ny cross-vldted estmte. Second, the terms n ny of these expressons cn come from hghly skewed dstrutons, so the second moment mght not gve good summry of uncertnty. Both of these prolems should susde s n ecomes lrge. For smll n, one could nsted compute nonprmetrc error estmtes usng Byesn ootstrp on the computed log-lkelhood vlues correspondng to the n dt ponts (Vehtr nd Lmpnen 22). 5.2 Model comprson When comprng two ftted models, we cn estmte the dfference n ther expected predctve ccurcy y the dfference n êlpd loo or êlpd wc (multpled y 2, f desred, to e on the devnce scle). To compute the stndrd error of ths dfference we cn use pred estmte to tke dvntge of the fct tht the sme set of n dt ponts s eng used to ft oth models. For exmple, suppose we re comprng models A nd B, wth correspondng ft mesures êlpd A loo = n = êlpd A loo, nd êlpd B loo = n = êlpd B loo,. The stndrd error of ther dfference s smply, se (êlpd A loo êlpdb loo ) = nv n = (êlpd A loo, êlpdb loo, ), (24) nd smlrly for WAIC nd K -fold cross-vldton. Alterntvely the non-prmetrc Byesn ootstrp pproch cn e used (Vehtr nd Lmpnen 22). As efore, these clcultons should e most useful when n s lrge, ecuse then non-normlty of the dstruton s not such n ssue when estmtng the uncertnty of these sums. In ny cse, we suspect tht these stndrd error formuls, for ll ther flws, should gve etter sense of uncertnty thn wht s otned usng the current stndrd pproch for comprng dfferences of devnces to χ 2 dstruton, prctce tht s derved for Gussn lner models or symptotclly nd, n ny cse, only pples to nested models. Further reserch needs to e done to evlute the performnce n model comprson of (24) nd the correspondng stndrd error formul for LOO. Cross-vldton nd WAIC should not e used to select sngle model mong lrge numer of models due to selecton nduced s s demonstrted, for exmple, y Pronen nd Vehtr (26). We demonstrte the prctcl use of LOO n model comprson usng the rdon exmple from Sect Model A s the multlevel lner model dscussed n Sect. 4.6 nd Model B s the sme model ut wthout the county-level urnum predctor. Tht s, t the county-level Model B hs ( ) α j N β j (( μα μ β ), ( σ 2 α ρσ α σ β ρσ α σ β σ 2 β )), j =,...,85. Comprng the models on PSIS-LOO revels n estmted dfference n elpd of.2 (wth stndrd error of 5.) n fvor of Model A. 5.3 Model comprson usng pontwse predcton errors We cn lso compre models n ther leve-one-out errors, pont y pont. We llustrte wth n nlyss of survey of resdents from smll re n Bngldesh tht ws ffected y rsenc n drnkng wter. Respondents wth elevted rsenc levels n ther wells were sked f they were nterested n gettng wter from neghor s well, nd seres of models were ft to predct ths nry response gven vrous nformton out the households (Gelmn nd Hll 27). Here we strt wth logstc regresson for the wellswtchng response gven two predctors: the rsenc level of the wter n the resdent s home, nd the dstnce of the house from the nerest sfe well. We compre ths to n lterntve logstc regresson wth the rsenc predctor on the logrthmc scle. The two models hve the sme numer of prmeters ut gve dfferent predctons. Fgure 2 shows the pontwse results for the rsenc exmple. The scttered lue dots on the left sde of Fg. 2 nd on the lower rght of Fg. 2 correspond to dt ponts whch Model A fts prtculrly poorly tht s, lrge negtve contrutons to the expected log predctve densty. We cn lso sum these n terms to yeld n estmted dfference n elpd loo of 6.4 wth stndrd error of 4.4. Ths stndrd error derves from the fnte smple sze nd s scled y the vrton n the dfferences dsplyed n Fg. 2; t s not Monte Crlo error nd does not declne to s the numer of Stn smulton drws ncreses. 6 Dscusson Ths pper hs focused on the prctcltes of mplementng LOO, WAIC, nd K -fold cross-vldton wthn Byesn smulton envronment, n prtculr the codng of the loglkelhood n the model, the computtons of the nformton