Asymptotics of Predictive Distributions

Transcription

1 Asymptotics of redictive Distributios atrizia Berti, Luca ratelli ad ietro Rigo Abstract Let (X ) be a sequece of radom variables, adapted to a filtratio (G ), ad let μ = (1/) i=1 δ X i ad a ( ) = (X +1 G ) be the empirical ad the predictive measures. We focus o μ a =sup B D μ (B) a (B), where D is a class of measurable sets. Coditios for μ a 0, almost surely or i probability, are give. Also, to determie the rate of covergece, the asymptotic behavior of r μ a is ivestigated for suitable costats r. Special attetio is paid to r =. The sequece (X ) is exchageable or, more geerally, coditioally idetically distributed. 1 Itroductio 1.1 The roblem Throughout, S is a olish space ad X = (X : 1) a sequece of S-valued radom variables o the probability space (Ω, A, ). Further, B is the Borel σ-field o S ad G = (G : 0) a filtratio o (Ω, A, ). We fix a subclass D B ad we let deote the sup-orm over D, amely, α β =sup B D α(b) β(b) wheever α ad β are probabilities o B. Let μ = (1/) δ Xi ad a ( ) = (X +1 G ). i=1. Berti Uiversita di Modea e Reggio-Emilia, Modea, Italy patrizia.berti@uimore.it L. ratelli Accademia Navale di Livoro, Livoro, Italy pratel@mail.dm.uipi.it. Rigo (B) Uiversita di avia, avia, Italy pietro.rigo@uipv.it Spriger Iteratioal ublishig Switzerlad 2017 M.B. Ferraro et al. (eds.), Soft Methods for Data Sciece, Advaces i Itelliget Systems ad Computig 456, DOI 11007/ _7 53

2 54. Berti et al. Both μ ad a are regarded as radom probability measures o B; μ is the empirical measure ad (if X is G-adapted) a is the predictive measure. Uder some coditios, μ (B) a (B) a.s. 0 for fixed B B. I that case, a (atural) questio is whether D is such that μ a a.s. Such questio is addressed i this paper. Coditios for μ a 0, almost surely or i probability, are give. Also, to determie the rate of covergece, the asymptotic behavior of r μ a is ivestigated for suitable costats r. Special attetio is paid to r =. The sequece X is assumed to be exchageable or, more geerally, coditioally idetically distributed (see Sect. 2). Our mai cocer is to coect ad uify a few results from [1 4]. Thus, this paper is essetially a survey. However, i additio to report kow facts, some ew results ad examples are give. This is actually the case of Theorem 1(d), Corollary 1 ad Examples Heuristics There are various (o-idepedet) reasos for ivestigatig μ a. We ow list a few of them uder the assumptio that G = G X, where G X 0 ={,Ω} ad G X = σ(x 1,...,X ). Most remarks, however, apply to ay filtratio G which makes X adapted. Empirical processes for o-ergodic data. Slightly abusig termiology, say that X is ergodic if is 0 1 valued o the sub-σ-field σ ( lim sup μ (B) : B B ). I real problems, X is ofte o-ergodic. Most statioary sequeces, for istace, fail to be ergodic. Or else, a exchageable sequece is ergodic if ad oly if is i.i.d. Now, if X is i.i.d., the empirical process is defied as G = (μ μ 0 ) where μ 0 is the probability distributio of X 1. But this defiitio has various drawbacks whe X is ot ergodic; see [5]. I fact, uless X is i.i.d., the probability distributio of X is ot determied by that of X 1. More importatly, if G coverges i distributio i l (D) (the metric space l (D) is recalled before Corollary 1) the μ μ 0 = 1/2 G But μ μ 0 typically fails to coverge to 0 i probability whe X is ot ergodic. Thus, empirical processes for o-ergodic data should be defied i some differet way. I this framework, a meaigful optio is to replace μ 0 with a, amely, to let G = (μ a ). Bayesia predictive iferece. I a umber of problems, the mai goal is to evaluate a but the latter ca ot be obtaied i closed form. Thus, a is to be estimated by the available data. Uder some assumptios, a reasoable estimate of a is just μ. I these situatios, the asymptotic behavior of the error μ a plays a role. For istace, μ is a cosistet estimate of a provided μ a 0 i some sese.

3 Asymptotics of redictive Distributios 55 redictive distributios of exchageable sequeces. Let X be exchageable. Just very little is kow o the geeral form of a for give, ad a represetatio theorem for a would be actually a major breakthrough. Failig the latter, to fix the asymptotic behavior of μ a cotributes to fill the gap. de Fietti. Historically, oe reaso for itroducig exchageability (possibly, the mai reaso) was to justify observed frequecies as predictors of future evets. See [8 10]. I this sese, to focus o μ a is i lie with de Fietti s ideas. Roughly speakig, μ should be a good substitute of a i the exchageable case. 2 Coditioally Idetically Distributed Sequeces The sequece X is coditioally idetically distributed (c.i.d.) with respect to G if it is G-adapted ad ( ) ( ) X k G = X+1 G a.s. for all k > Roughly speakig, at each time 0, the future observatios (X k : k > ) are idetically distributed give the past G. Whe G = G X, the filtratio G is ot metioed at all ad X is just called c.i.d. The, X is c.i.d. if ad oly if ( ) ( ) X 1,...,X, X +2 X1,...,X, X +1 for all Exchageable sequeces are c.i.d. while the coverse is ot true. Ideed, X is exchageable if ad oly if it is statioary ad c.i.d. We refer to [3] for more o c.i.d. sequeces. Here, it suffices to metio a last fact. If X is c.i.d., there is a radom probability measure μ o B such that μ (B) a.s. μ(b) for every B B. As a cosequece, if X is c.i.d. with respect to G, for each 0 ad B B oe obtais E { μ(b) G } = limm E { μ m (B) G } = limm 1 m m k=+1 = ( X +1 B G ) = a (B) a.s. ( X k B G ) I particular, a (B) = E { } a.s. μ(b) G μ(b) ad μ (B) a (B) a.s. From ow o, X is c.i.d. with respect to G. I particular, X is idetically distributed ad μ 0 deotes the probability distributio of X 1.Wealsolet W = (μ μ), where μ is the radom probability measure o B itroduced above. Note that, if X is i.i.d., the μ = μ 0 a.s. ad W reduces to the usual empirical process.

4 56. Berti et al. 3 Results Let D B. To avoid measurability problems, D is assumed to be coutably determied. This meas that there is a coutable subclass D 0 D such that α β =sup B D0 α(b) β(b) for all probabilities α, β o B. For istace, D = B is coutably determied (for B is coutably geerated). Or else, if S = R k, the D ={(, t] :t R k }, D ={closed balls} ad D ={closed covex sets} are coutably determied. 3.1 A Geeral Criterio Sice a (B) = E { } μ(b) G a.s. for each B B ad D is coutably determied, oe obtais μ a = sup E { } { } μ (B) μ(b) G E μ μ G a.s. B D 0 This simple iequality has some ice cosequeces. Recall that D is a uiversal Gliveko-Catelli class if μ μ 0 a.s. 0 wheever X is i.i.d. Theorem 1 Suppose D is coutably determied ad X is c.i.d. with respect to G. The, (a) μ a a.s. 0 if μ μ a.s. 0 ad μ a 0 if μ μ (b) μ a a.s. 0 provided X is exchageable, G = G X ad D is a uiversal Gliveko-Catelli class. (c) r μ a 0 wheever the costats r satisfy r / 0 ad sup E { W b} < for some b 1. (d) u μ a a.s. 0 wheever u < 1/2 ad sup E { W b} < for each b 1. roof Sice μ μ 1, poit (a) follows from the martigale covergece theorem i the versio of [7]. (If μ μ 0, it suffices to apply a obvious argumet based o subsequeces). Next, suppose X, G ad D areasi(b). By de Fietti s theorem, coditioally o μ, the sequece X is i.i.d. with commo distributio μ. Sice D is a uiversal Gliveko-Catelli class, it follows that ( μ μ 0 ) = { μ μ 0 μ } d = 1d = 1. Hece, (b) is a cosequece of (a). As to (c), just ote that E{ (r μ a ) b } r b E{ μ μ b} = (r / ) b E { W b}.

5 Asymptotics of redictive Distributios 57 Fially, as to (d), fix u < 1/2 ad take b such that b(1/2 u) >1. The, ( u μ a > ɛ ) = E { W b} ɛ b (1/2 u)b E { μ a b} ɛ b ub E { μ μ b} ɛ b ub cost < for each ɛ > (1/2 u)b Therefore, u μ a a.s. 0 because of the Borel-Catelli lemma. Some remarks are i order. Theorem 1 is essetially kow. Apart from (d), it is implicit i [2, 4]. If X is exchageable, the secod part of (a) is redudat. I fact, μ μ 0 coverges a.s. (ot ecessarily to 0) wheever X is i.i.d. Applyig de Fietti s theorem as i the proof of Theorem 1(b), it follows that μ μ coverges a.s. eve if X is exchageable. Thus, μ μ 0 implies μ μ a.s. Sometimes, the coditio i (a) is ecessary as well, amely, μ a a.s. 0if ad oly if μ μ a.s. For istace, this happes whe G = G X ad μ λ a.s., where λ is a (o-radom) σ-fiite measure o B. I this case, i fact, a μ a.s. 0 by [6, Theorem 1]. Several examples of uiversal Gliveko-Catelli classes are available; see [11] ad refereces therei. Similarly, for may choices of D ad b 1 there is a uiversal costat c(b) such that sup E { W b} c(b) provided X is i.i.d.; see e.g. [11, Sects ad ]. I these cases, de Fietti s theorem yields sup E { W b} c(b) eve if X is exchageable. Thus, poits (b) (d) are especially useful whe X is exchageable. I (c), covergece i probability ca ot be replaced by a.s. covergece. As a trivial example, take D = B, G = G X, r = log log, ad X a i.i.d. sequece of idicators. Lettig p = (X 1 = 1), oe obtais E { W 2} = E {( μ {1} p ) 2} = p (1 p) for all. However, the LIL yields lim sup r μ a =lim sup We fially give a couple of examples. i=1 (X i p) log log = 2 p (1 p) a.s. Example 1 Let D = B. IfX is i.i.d., the μ μ 0 a.s. 0 if ad oly if μ 0 is discrete. By de Fietti s theorem, it follows that μ μ a.s. 0 wheever X is exchageable ad μ is a.s. discrete. Thus, uder such assumptios ad G = G X, Theorem 1(a) implies μ a a.s. This result has possible practical iterest. I fact, i Bayesia oparametrics, most priors are such that μ is a.s. discrete. Example 2 Let S = R k ad D ={closed covex sets}. Give ay probability α o B, deote by α (c) = α x α{x}δ x the cotiuous part of α.ifx is i.i.d. ad μ (c) 0 m,

6 58. Berti et al. where m is Lebesgue measure, the μ μ 0 a.s. Applyig Theorem 1(a) agai, oe obtais μ a a.s. 0 provided X is exchageable, G = G X ad μ (c) m a.s. While morally true, this argumet does ot work for D ={Borel covex sets} sice the latter choice of D is ot coutably determied. 3.2 The Domiated Case I this Subsectio, G = G X, A = σ ( ) G X, Q is a probability o (Ω, A) ad b ( ) = Q(X +1 G ) is the predictive measure uder Q. Also, we say that Q is a Ferguso-Dirichlet law if b ( ) = cq(x 1 ) + μ ( ), Q-a.s. for some costat c > c + If Q, the asymptotic behavior of μ a uder should be affected by that of μ b uder Q. This (rough) idea is realized by the ext result. Theorem 2 (Theorems 1 ad 2 of [4]) Suppose D is coutably determied, X is c.i.d., ad Q. The, μ a 0 provided μ b Q 0 ad the sequece (W ) is uiformly itegrable uder both ad Q. I additio, μ { a coverges a.s. to a fiite limit wheever Q is a Ferguso-Dirichlet law, sup E Q W 2} <, ad sup { { E Q (d/dq) 2 } { E Q E Q (d/dq G ) 2}} <. To make Theorem 2 effective, the coditio Q should be give a simple characterizatio. This happes i at least oe case. Let S be fiite, say S ={x 1,...,x k, x k+1 }, X exchageable ad μ 0 {x} > 0 for all x S. The Q, with Q a Ferguso-Dirichlet law, if ad oly if the distributio of ( μ{x 1 },...,μ{x k } ) is absolutely cotiuous (with respect to Lebesgue measure). This fact is behid the ext result. Theorem 3 (Corollaries 4 ad 5 of [4]) Suppose S ={0, 1} ad X is exchageable. The, ( μ {1} a {1} ) 0 wheever the distributio of μ{1} is absolutely cotiuous. Moreover, ( μ {1} a {1} ) coverges a.s. (to a fiite limit) provided the distributio of μ{1} is absolutely cotiuous with a almost Lipschitz desity. I Theorem 3, a real fuctio f o (0, 1) is said to be almost Lipschitz i case x f (x)x u (1 x) v is Lipschitz o (0, 1) for some reals u, v<1. A cosequece of Theorem 3 is to be stressed. For each B B, defie T (B) = { a (B) { X +1 B G B } }

7 Asymptotics of redictive Distributios 59 where G B = σ( I B (X 1 ),..., I B (X ) ). Also, let l (D) be the set of real bouded fuctios o D, equipped with uiform distace. I the ext result, W is regarded as a radom elemet of l (D) ad covergece i distributio is meat i Hoffma- Jørgese s sese; see [11]. Corollary 1 Let D be coutably determied ad X exchageable. Suppose (i) μ(b) has a absolutely cotiuous distributio for each B D such that 0 < (X 1 B) <1; (ii) the sequece ( W ) is uiformly itegrable; (iii) W coverges i distributio to a tight limit i l (D). The, μ a 0 if ad oly if T (B) 0 for each B D. roof Let U (B) = { μ (B) { } } X +1 B G B. The, U (B) 0for each B D. I fact, U (B) = 0a.s.if(X 1 B) {0, 1}. Otherwise, U (B) 0 follows from Theorem 3, sice (I B (X )) is a exchageable sequece of idicators ad μ(b) has a absolutely cotiuous distributio. Next, suppose T (B) 0 for each B D. Lettig C = (μ a ),wehavetoprovethat C Equivaletly, regardig C as a radom elemet of l (D), wehavetoprovethat C (B) 0 for fixed B D ad the sequece (C ) is asymptotically tight; see e.g. [11, Sect. 1.5]. Give B D, sice both U (B) ad T (B) coverge to 0 i probability, the C (B) = U (B) T (B) Moreover, sice C (B) = E { } W (B) G a.s., the asymptotic tightess of (C ) follows from (ii) ad (iii); see [3, Remark 4.4]. Hece, C Coversely, if C 0, oe trivially obtais T (B) = U (B) C (B) U (B) + C 0 for each B D. If X is exchageable, it frequetly happes that sup E { W 2} <, which i tur implies coditio (ii). Similarly, (iii) is ot uusual. As a example, coditios (ii) ad (iii) hold if S = R, D ={(, t] :t R} ad μ 0 is discrete or (X 1 = X 2 ) = 0; see [3, Theorem 4.5]. Ufortuately, as show by the ext example, T (B) may fail to coverge to 0 eve if μ(b) has a absolutely cotiuous distributio. This suggests the followig geeral questio. I the exchageable case, i additio to μ (B), which further iformatio is required to evaluate a (B)? Or at least, are there reasoable coditios for T (B) 0? Eve if itriguig, to our kowledge, such a questio does ot have a satisfactory aswer. Example 3 Let S = R ad X = Y Z 1, where Y ad Z are idepedet real radom variables, Y N(0, 1) for all, ad Z has a absolutely cotiuous distributio supported by [1, ). Coditioally o Z, the sequece X = (X 1, X 2,...) is i.i.d. with commo distributio N(0, Z 2 ). Thus, X is exchageable ad μ(b) = (X 1 B Z) = f B (Z) a.s., where

8 60. Berti et al. f B (z) = (2 π) 1/2 z B exp ( (xz) 2 /2 ) dx for B B ad z 1. Fix B B, with B [1, ) ad (X 1 B) >0, ad defie C ={ x : x B}. Sice f B = f C, the μ(b) = μ(c) a.s. Further, μ(b) has a absolutely cotiuous distributio, for f B is differetiable ad f B = Nevertheless, oe betwee T (B) ad T (C) does ot coverge to 0 i probability. Defie i fact g = I B I C ad R = 1/2 i=1 g(x i). Sice μ(g) = μ(b) μ(c) = 0 a.s., the R coverges stably to the kerel N(0, 2μ(B)); see[3, Theorem 3.1]. O the other had, sice E { } { } g(x +1 ) G = E μ(g) G = 0 a.s., oe obtais R = { μ (B) μ (C) } = T (C) T (B) + + { μ (B) { X +1 B G B } } { μ (C) { X +1 C G C Hece, if T (B) 0 ad T (C) 0, Corollary 1 (applied with D ={B, C}) implies the cotradictio R } }. Refereces 1. Berti, Rigo (1997) A Gliveko-Catelli theorem for exchageable radom variables. Stat robab Lett 32: Berti, Mattei A, Rigo (2002) Uiform covergece of empirical ad predictive measures. Atti Sem Mat Fis Uiv Modea 50: Berti, ratelli L, Rigo (2004) Limit theorems for a class of idetically distributed radom variables. A robab 32: Berti, Crimaldi I, ratelli L, Rigo (2009) Rate of covergece of predictive distributios for depedet data. Beroulli 15: Berti, ratelli L, Rigo (2012) Limit theorems for empirical processes based o depedet data. Electro J robab 17: Berti, ratelli L, Rigo (2013) Exchageable sequeces drive by a absolutely cotiuous radom measure. A robab 41: Blackwell D, Dubis LE (1962) Mergig of opiios with icreasig iformatio. A Math Stat 33: Cifarelli DM, Regazzii E (1996) De Fietti s cotributio to probability ad statistics. Stat Sci 11: Cifarelli DM, Dolera E, Regazzii E (2016) Frequetistic approximatios to Bayesia previsio of exchageable radom elemets. arxiv: v1 1 Fortii S, Ladelli L, Regazzii E (2000) Exchageability, predictive distributios ad parametric models. Sakhya A 62: va der Vaart A, Weller JA (1996) Weak covergece ad empirical processes. Spriger