Cross Validation, Monte Carlo Cross Validation, and Accumulated Prediction Errors: Asymptotic Properties

Transcription

1 Cross Validation, Monte Carlo Cross Validation, and Accumulated Prediction Errors: Asymptotic Properties Ching-Kang Ing Institute of Statistics, National Tsing Hua University, Hsinchu, Taiwan

2 Outline 1 Cross Validation (CV) 2 Monte Carlo Cross Validation (MCCV) 3 Accumulated Prediction Errors (APE) 4 References

3 Cross Validation (CV) Cross Validation (CV) Consider p y i = x i β + e i = x ij β j + e i, j=1 where β = (β 1,..., β p) and e i are i.i.d. r.v.s with E(e i ) = 0 and E(e 2 i ) = σ2 > 0. Suppose β i = 0 for some i P = {1,..., p}. We are interested in choosing the smallest true model y i = x i,α β α + e i, where α = {i : β i 0}, β α = (β i, i α ) and x i,α = (x ij, j α ), among the candidate models E(y) C(X α), where y = (y 1,..., y n), X α = (x i,α,..., x n,α) with x i,α = (x ij, j α), and α 2 P. 1

4 Cross Validation (CV) Delete-n v -out CV (CV(n v )) Let (y i, x i ), i = 1,..., n, be observations. CV(n v) splits the data into two parts: {(y i, x i ), i s} and {(y i, x i ), i s c }, where s and s c are subsets of N = {1,..., n} with s s c = and s s c = N. The first part is referred to as the validation (testing) sample, whereas the second part is referred to as the training sample. Denote (s) and (s c ) by n v and n c, respectively, noting that n v + n c = n. The CV evaluates the performance of α using 1 ˆΓ α,n(n v) = ( n n ) (y i x ˆβ i,α α,s c ) 2 v n v All (s,s c ) combinations i s 1 = ( n n ) y s ŷ α,s c 2, (ŷ α,s c = X α,s ˆβα,s c ) v n v All (s,s c ) combinations where y A = (y i, i A), A N and ˆβ α,s c = (X α,s c X α,s c ) 1 X α,s c y s c with X α,a = (x i,α, i A). 2

5 Cross Validation (CV) Some statistical properties of CV Define P α = X α(x α Xα) 1 X α and Q α,a = X α,a (X α Xα) 1 X α,a. Fact 1 n 1 v ys ŷ α,s c 2 = n 1 v (Inv Qα,s) 1 (y s X α,s ˆβα) 2, where I nv denotes the n v-dimensional identity matrix and ˆβ α = (X αx α) 1 X αy, which is the least squares estimate of β α based on the full data. 3

6 Cross Validation (CV) Proof Since (X α,s c X α,s c ) 1 = (X α Xα X α,s Xα,s) 1, by making use of (A B B) 1 = A 1 + A 1 B (I BA 1 B ) 1 BA 1, (1) we obtain ŷ α,s c = X α,s ˆβα,s c (I: identity matrix of suitable dimension) = X α,s ˆβα + Q α,s(i Q α,s) 1 X α,s ˆβα Q α,sy s Q α,s(i Q α,s) 1 Q α,sy s, and hence y s X α,s ˆβα,s c = (I Q α,s) 1 (y s X α,s ˆβα). This completes the proof. Remark By Fact 1, we have ˆΓ n α,n(1) = n 1 {(1 w iα ) 1 (y i x ˆβ i,α α)} 2, Delete-1-out CV (conventional CV) where w iα = (P α) ii, the ith diagonal element of P α. 4

7 Cross Validation (CV) Theorem 1 Assume X X n where X = (x 1,..., x n), and Then (1) R, a positive definite matrix, (2) n max w iα 0 for any α 2 P. (3) 1 i n n ˆΓ α,n(1) = σ n β X (I P α)xβ + o p(1), (I : n-dimensional identity matrix) (2) if α is an incorrect model (namely α α ), ˆΓ α,n(1) = n 1 e e + 2dασ2 n 1 n e P αe + o p(n 1 ), if α is a correct model (α α = ), where d α = (α) and e = (e 1,..., e n). 5

8 Cross Validation (CV) Proof Since (1 w iα ) 2 = 1 + 2w iα + O(w 2 iα ), where O( ) denotes a bound uniform over 1 i n, it holds that ˆΓ α,n(1) = 1 γiα 2 n + 1 (2w iα + O(wiα 2 n ))γ2 i,α (I) + (II), where γ i,α = y i x i,α ˆβ α. If α is incorrect, then (I) = n 1 y (I P α)y = n 1 (e + β X )(I P α)(xβ + e) = n 1 e (I P α)e + 2n 1 e (I P α)xβ + n 1 β X (I P α)xβ why? = σ 2 + n 1 β X (I P α)xβ + o p(1), and (II) = o p(1). Then (1) follows. 6

9 Cross Validation (CV) If α is correct, then and This completes the proof. Corollary 1 Let (I) = n 1 e (I P α)e, n (II) = n 1 (2w iα + O(wiα 2 ))(e i x i,α ( ˆβ α β α)) 2 why? = = why? = 2 n w iα e 2 i + op(n 1 ) 2 w iα σ w iα (e 2 i n n σ2 ) + o p(n 1 ) 2d ασ 2 + o p(1). n ˆα = argmin α 2 P ˆΓ α,n(1). Then lim n P (α ˆα = ) = 1, but lim n P (ˆα = α ) is in general less than 1. 7

10 Cross Validation (CV) Notation b = ( n n v ) B nv = {s : s N and (s) = n v} P α,s = X α,s(x α,sx α,s) 1 X α,s ˆR α,s = 1 n v t s xt,αx t,α ˆR α,s c = 1 n c t s c xt,αx t,α γ α,s = y s X α,s ˆβα 8

11 Cross Validation (CV) Then, Fact 1 yields ˆΓ α,n(n v) = 1 n vb where (I) = (III) = s Bnv (IV) = 2 nv n c 1 n vb (V) = γ α,s (Inv Qα,s) 1 (I nv Q α,s) 1 γ α,s = (I) + (II) + + (VI), (4) 1 γ α,s n γα,s, vb s Bnv 1 1 γ n 2 α,s c n vb s Bnv γ α,s Pα,sγα,s, 2 n c 1 n vb (VI) = 2nv n 2 c s Bnv γ α,s s Bnv 1 γ α,s n vb s Bnv (II) = n2 v n 2 c 1 1 Xα,s( ˆR α,sc ˆR α,s )X α,s 1 γ α,s n Pα,sγα,s, vb s Bnv 1 1 Xα,s( ˆR α,sc ˆR α,s )X α,s γα,s, 1 1 Pα,sXα,s( ˆR α,sc ˆR α,s )X α,s γα,s. 1 1 Xα,s( ˆR α,sc ˆR α,s )X α,s γα,s, (Pα,sXα,s = Xα,s) Hint Using (1), one gets (I nv Q α,s) 1 = I nv + nv n c P α,s + 1 n c X α,s( 1 1 ˆR α,sc ˆR α,s )X α,s. 9

12 Cross Validation (CV) Theorem 2 Assume the same assumptions as in Theorem 1. Assume also that where ˆR A = 1 (A) t A xtx t with A N. Suppose Then (i) For α α, n v n lim max ˆR s ˆR s c = 0, (5) n s B nv 1 and nc = n nv. (6) ˆΓ α,n(n v) = n 1 e e + n 1 β X (I P α)xβ + o p(1) + R n, (7) where R n is a non-negative random variable. (ii) For α α =, ˆΓ α,n(n v) = n 1 e e + dασ2 n c (iii) lim n P (ˆα nv = α ) = 1, where ˆα nv = argmin α 2 P ˆΓ α,n(n v). + o p(n 1 c ). (8) 10

13 Cross Validation (CV) Proof We begin by considering the case of α α =. We will first show that ˆΓ α,n(n v) = (I) + (II)(1 + o(1)), (9) via proving (K) = (II)o(1), (10) where K = III, IV, V, and VI. 11

14 Cross Validation (CV) To show (10) holds with K = VI, note that (VI) 2{(A) + (B)}, (11) where (A) = nv n 2 c (B) = nv n 2 c 1 n γ α,s vb s Bnv 1 n vb s Bnv α,s X α,s γα,s, α,s X α,s γα,s. 1 1 Xα,s( ˆR α,sc ˆR α,s )( ˆR α,s ˆR 1 α,s c ) ˆR γ 1 α,sxα,s ˆR α,s ( ˆR α,s ˆR 1 α,s c ) ˆR In addition, and (A) why? (B) why? n2 v 1 n 2 c n vb n2 v 1 n 2 c n vb max s Bnv max ˆR α,s ˆR α,s c max 1 ˆR s Bnv s Bnv ˆR α,s ˆR α,s c 2 max s Bnv α,s s Bnv 1 ˆR α,s max 1 ˆR α,s s Bnv c γ α,spα,sγα,s, (12) s Bnv γ α,spα,sγα,s. (13) 12

15 Cross Validation (CV) By (2), we have 1 max ˆR α,s = O(1), s B nv which, together with (5), and (11) (13), yields Similarly, it can be shown that Since nv, it is easy to see that n c (VI) = (II)o(1). (14) (V) = (II)o(1). (15) (IV) = (IV) = (II)o(1). ((IV) is non-negative) (16) 13

16 Cross Validation (CV) Now, for (III), we have (III) nv 1 n 2 c n vb nv 1 n 2 c n vb + nv 1 n 2 c n vb s B nv γ α,sx α,s ˆR 1 γ 1 α,sxα,s ˆR s B nv s B nv γ α,sx α,s ˆR 1 α,s c ( ˆR α,s ˆR 1 α,s c ) ˆR α,s c ( ˆR α,s ˆR α,s c ) α,s ( ˆR α,s ˆR α,s c ) α,s( ˆR α,s ˆR 1 α,s c ) ˆR α,s c 1 ˆR α,s c ( ˆR α,s ˆR 1 α,s c ) ˆR ( ˆR α,s ˆR 1 α,s c ) ˆR α,s c ( ˆR α,s ˆR 1 α,s c ) ˆR α,s X α,s γα,s. This and an argument similar to that used to prove (14) give 1 ˆR α,sx α,sγ α,s α,s X α,s γα,s (III) = (II)o(1). (17) Hence, (10) follows from (14) (17). Now, by an argument similar to that use to prove (14) again, we have (II) = n2 v 1 n 2 c n vb s B nv γ α,s Qα,sγα,s + (II)o(1). (18) 14

17 Cross Validation (CV) Moreover, n 2 v 1 n 2 c n vb s B nv γ α,s Qα,sγα,s = n2 v 1 n 2 P α,ij γ i,α γ j,α ([P α,ij ] i,j s = P α) c n vb s B nv (i,j) s s = n2 v 1 n 2 P α,ii γi,α 2 c n + n2 v 1 vb n s B nv i s 2 P α,ij γ i,α γ j,α (P α,ii = w iα ) c n vb s B nv (i,j) s s,i j why? = n2 v 1 n 2 c n = n2 v 1 n 2 c n w iα γi,α 2 n2 v n v 1 1 n 2 w iα γi,α 2 c n 1 n ( ) n c w iα σ O w iα (γi,α 2 n 1 n σ2 ) c why? = dασ2 + o p(n 1 c ). n c (19) 15

18 Cross Validation (CV) By (18) and (19), we have Moreover, (II) = dασ2 n c + o p(n 1 c ). (20) (I) = 1 n y (I P α)y = 1 n e (I P α)e = n 1 e e + O p(n 1 ). (21) In view of (20), (21) and (9), the desired conclusion (8) follows. 16

19 Cross Validation (CV) To show (7), note that (4) implies (why?) Since α α, ˆΓ α,n(n v) 1 n vb s B nv γ α,sγ α,s = 1 n y (I P α)y. (22) 1 n y (I P α)y = n 1 e e + n 1 β X (I P α)xβ + 2n 1 β X (I P α)e and hence (7) is ensured by (22) and (23). = n 1 e e + n 1 β X (I P α)xβ + o p(1), (23) Finally, (iii) is an immediate consequence of (i) and (ii). 17

20 Monte Carlo Cross Validation (MCCV) Monte Carlo Cross Validation (MCCV) i.i.d. Let s i U(B nv ), where B nv is defined in the note for CV. Define MCCV as follows: ˆΓ MCCV α,n = 1 n vb y si ŷ α,s c i 2, where b is the number of Monte Carlo simulations used to calculate CV. 1

21 Monte Carlo Cross Validation (MCCV) Theorem 3 Assume the same assumptions as in Theorem 1. Suppose max 1 1 j b x i x i n 1 x i x i v n = op(1), (24) i s c j i / s j and Then, (i) for α α, Ee 4 1 <, n 2 bn 2 0, c ˆΓ MCCV α,n = 1 n vb n v n 1, and nc. (25) e s i e si + 1 n β X (I P α)xβ + o p(1) + R n, where e s i = (e j, j s i ) and R n is some positive r.v., (ii) for α α =, ˆΓ MCCV α,n = 1 n vb e s i e si + n 1 c d ασ 2 + o p(n 1 c ), (iii) lim n P (ˆα MCCV = α ) = 1, where ˆα MCCV = argmin α 2 P ˆΓ MCCV α,n. 2

22 Monte Carlo Cross Validation (MCCV) Proof By an argument similar to that used to prove (9) and (18) in the note for CV, one has for α α =, and ˆΓ MCCV α,n = 1 n vb n 2 v 1 n 2 c n vb γ α,s i γ α,si + n2 v 1 n 2 c n vb γ α,s i P α,si γ α,si (1 + o p(1)), (26) γ α,s i P α,si γ α,si (1 + o p(1)) = n2 v 1 n 2 c n vb γ α,s i Q α,si γ α,si. (27) 3

23 Monte Carlo Cross Validation (MCCV) In addition, by (19) in the note for CV, [ ] n 2 v 1 E n 2 γ α,s c n vb i Q α,si γ α,si (y 1, x 1 ),..., (y n, x n) implying where = why? = = n 2 v 1 n 2 E[γ α,s c n 1 Q α,s1 γ α,s1 (y 1, x 1 ),..., (y n, x n)] v n 2 v 1 1 ( n 2 c n n ) v n v d ασ 2 n c s B nv γ α,s Qα,sγα,s + o p(n 1 c ), E[n cv n F n] pr. d ασ 2, (28) V n = n2 v 1 n 2 c n vb γ α,s i Q α,si γ α,si and F n is the σ-field generated by (y 1, x 1 ),..., (y n, x n). 4

24 Monte Carlo Cross Validation (MCCV) We also have V ar(n cv n F n) = n2 v 1 n 2 c b 2 V ar(γ α,s i Q α,si γ α,si F n) n2 v 1 n 2 c b E((γ α,s 1 Q α,s1 γ α,s1 ) 2 F n) = n2 v 1 1 ( n 2 c b n ) (γ α,s Qα,sγα,s)2. n v s B nv Since Ee 4 1 <, we have E((γ α,sq α,sγ α,s) 2 ) C < for all s B nv, (C : some positive constant) which, together with n2 v 1 n 2 0, yields c b V ar[n cv n F n] pr. 0. (29) 5

25 Monte Carlo Cross Validation (MCCV) It is shown in the Appendix that (28) + (29) implies n cv n pr. d ασ 2. (30) In view of (26), (27) and (30), (ii) is proved once 1 n vb To show (31), note first that yielding γ α,s i γ α,si = 1 n vb γ α,si = e si X α,si ( ˆβ α β α), e s i e si + o p(n 1 c ). (31) 1 n vb γ α,s i γ α,si = 1 n vb e s i e si 2 n vb ( ˆβ α β α) X α,s i e si ( ) +( ˆβ α β α) 1 X α,s n vb i X α,si ( ˆβ α β α). (32) 6

26 Monte Carlo Cross Validation (MCCV) Since and one has 1 ( ˆβ α β α) X α,s n vb i e si ˆβ 1 α β α X α,s n vb i e si, ( ) 1 E X 1 α,s n vb i e si E( X α,s n vb i e si ) 1 = E[E( X α,s n 1 e s1 F n)] v 1 = E ( 1 n n ) X α,se s v n v s B nv 2 n vb why? = O(n 1 2 v ), ˆβ α β α = O p(n 1 2 ), ( ˆβ α β α) X α,s why? i e si = O p(n 1 ). (33) 7

27 Monte Carlo Cross Validation (MCCV) Similarly, it can be shown that ( ( ˆβ α β α) 1 n vb ) X α,s i X α,si ( ˆβ α β α) = O p(n 1 ). (34) Combining (32) (34), we obtain the desired conclusion (31). Thus, (ii) is proved. The proof of (i) is left as an exercise. Finally, we note that (iii) is an immediate consequence of (i) & (ii). 8

28 Monte Carlo Cross Validation (MCCV) Appendix: the proof of (30) To prove (30), we need the so-called conditional Chebyshev s inequality, which is stated as follows. Conditional Chebyshev s inequality Let X be a positive r.v., ɛ > 0, and F be a σ-field. Then Proof P (X > ɛ F) E(X F) ɛ Let D F. Then, by the definition of conditional expectation, P (X > ɛ F) dp = E(I X>ɛ F) dp = I X>ɛ dp D D D Hence (35) follows. a.s. (almost surely). (35) = D D D X ɛ I dp X>ɛ X ɛ dp E(X F) ɛ dp. 9

29 Monte Carlo Cross Validation (MCCV) Now, (30) is ensured by the following fact. Fact Let {X n} and {F n} be sequences of r.v.s and σ-field, respectively. If V ar(x n F n) pr. 0, (36) and then E(X n F n) pr. C (a constant), (37) X n pr. C. (38) Proof By the conditional Chebyshev s inequality and (36), it holds that for any ɛ > 0 P ( X n E(X n F n) > ɛ F n) V ar(xn Fn) ɛ 2 which, together with the dominated convergence theorem, yields This and (37) give (38). X n E(X n F n) pr. 0. (why?) pr. 0, 10

30 Accumulated Prediction Errors (APE) Accumulated Prediction Errors (APE) Consider the following stochastic regression model y t = p β i x ti + ɛ t, (39) where ɛ t i.i.d. (0, σ 2 ) and x t = (x t1,..., x tp) is F t 1 -measurable, meaning that x t can be completely decided by the information collected at time t 1. Example 1: Autoregressive (AR) models y t = a 1 y t a py t p + ɛ t, where A(z) = 1 a 1 z a 2 z 2 a pz p 0, z 1. Example 2: Autoregressive exogenous models y t = a 1 y t a py t p + η 1 Z t1 + + η qz tq + ɛ t, where A(z) 0, z 1, and (ɛ t, Z t1,..., Z tq) are i.i.d.. 1

31 Accumulated Prediction Errors (APE) Let α and α be defined as in the note for CV. We are interested in choosing α based on APE, which assigns α to a positive value: APE α = (y t x ˆβ t,α t 1,α ) 2, i=m+1 where t ˆβ t,α = x j,α x 1 t j,α x j,α y j j=1 j=1 and M is some integer to be specified later. 2

32 Accumulated Prediction Errors (APE) Theorem 4 Assume and 1 n x tx t t=1 pr. R (p.d.), E ɛ 1 q 1 < for some large q 1, (40) max E x ti q 2 < for some large q 2, E t( ˆβ t β) q 3 < k <, (41) 1 t n,1 i p for all t M and some large q 3. (i) For α α =, (ii) For α α, APE α = APE α i=m+1 ɛ 2 t + σ2 d α log n + o p(log n), i=m+1 ɛ 2 t + n(α) + Op(1), where 1 pr. n(α) γ > 0. n (iii) lim n P (ˆα APE = α ) = 1, where ˆα APE = argmin α 2P APE α. 3

33 Accumulated Prediction Errors (APE) Proof We first prove (i). Define ( t 1 t t V t,α = x i,α x i,α), Q t,α = x j,α ɛ j V t,α x j,α ɛ j, j=1 j=1 and d t,α = x t,α Vt,αxt,α. By making use of and one gets = V t,α = V t 1,α V t 1,αx t,αx t,α V t 1,α 1 + x t,α V t 1,αx t,α x t,α V t 1,αx t,α = 1 d t,α, t=m+1 2 t 1 x t,αv t 1,α x j,α ɛ j (1 d t,α) + Q n,α Q M,α j=1 d t,αɛ 2 t + 2 n x t,α ( ˆβ t 1,α β α)ɛ t(1 d t,α). (42) t=m+1 t=m+1 4

34 Accumulated Prediction Errors (APE) Moreover, by (40) and (41), it can be shown that t=m+1 2 t 1 x t,α V t 1,α x j,α ɛ j d t,α = O p(1), (43) j=1 Q n = O p(1), Q M = O p(1), (44) x t,α( ˆβ t 1,α β α)ɛ t(1 d t,α) = o p(log n). (45) t=m+1 Also, we have t=m+1 d t,αɛ 2 t = t=m+1 d t,ασ 2 + t=m+1 d t,α(ɛ 2 t σ 2 ) = σ 2 d α log n + o p(log n) + O p(1). (46) 5

35 Accumulated Prediction Errors (APE) Consequently, (i) follows from (42) (46), and the fact that APE α = = = (ɛ t x t,α ( ˆβ t 1,α β α)) 2 t=m+1 ɛ 2 t 2 ( ˆβ t 1,α β α)x t,αɛ t + [x t,α( ˆβ t 1,α β α)] 2 t=m+1 t=m+1 t=m+1 ɛ 2 t + [x t,α( ˆβ t 1,α β α)] 2 + o p(log n). t=m+1 t=m+1 To show (ii), note that by Theorem 2.1 of Wei (1992), we have (y t x ˆβ t,α t 1,α ) 2 (y t x ˆβ M t,α n,α) 2 (y t x ˆβ t,α M,α ) 2 t=m+1 t=1 t=1 = y (I P α)y + O p(1), which implies (ii) (why?). Now, (iii) follows directly from (i) and (ii). 6

36 References References 1 Shao, J. (1993). Linear model selection by cross-validation. Journal of the American Statistical Association, 88, Wei, C. Z. (1992). On predictive least squares principles. The Annals of Statistics, 20, 1 42.