SUPPLEMENTARY MATERIAL COBRA: A Combined Regression Strategy by G. Biau, A. Fischer, B. Guedj and J. D. Malley

Transcription

1 SUPPLEMENTARY MATERIAL COBRA: A Combined Regression Strategy by G. Biau, A. Fischer, B. Guedj and J. D. Malley A. Proofs A.1. Proof of Proposition.1 e have E T n (r k (X)) r? (X) = E T n (r k (X)) T(r k (X)) As for the double product, notice that But Consequently, and + E T(r k (X)) r? (X) E[(T n (r k (X)) T(r k (X)))(T(r k (X)) r? (X))]. E[(T n (r k (X)) T(r k (X)))(T(r k (X)) r? (X))] = E E (T n (r k (X)) T(r k (X)))(T(r k (X)) r? (X)) r k (X),D n = E (T n (r k (X)) T(r k (X)))E T(r k (X)) r? (X) r k (X),D n. E[r? (X) r k (X),D n ] = E[r? (X) r k (X)] (by independence of X and D n ) = E[E[Y X] r k (X)] = E[Y r k (X)] (since æ(r k (X)) Ω æ(x)) = T(r k (X)). E[(T n (r k (X)) T(r k (X)))(T(r k (X)) r? (X))] = 0 E T n (r k (X)) r? (X) = E T n (r k (X)) T(r k (X)) + E T(r k (X)) r? (X). Thus, by definition of the conditional epectation, and using the fact that T(r k (X)) = E[r? (X) r k (X)], E T n (r k (X)) r? (X) E T n (r k (X)) T(r k (X)) + infe f (r k (X)) r? (X), f 1

2 where the infimum is taken over all square integrable functions of r k (X). In particular, as desired. E T n (r k (X)) r? (X) min E r k,m(x) r? (X) + E T n (r k (X)) T(r k (X)),,...,M A.. Proof of Proposition. Note that the second statement is an immediate consequence of the first statement and Proposition.1, therefore we only have to prove that E T n (r k (X)) T (r k (X))! 0 as `!1. e start with a technical lemma, whose proof can be found in the monograph by Györfi et al. (00). Lemma A.1. Let B(n, p) be a binomial random variable with parameters n 1 and p > 0. Then 1 1 E 1 + B(n, p) p(n + 1) and E 1{B(n,p)>0} B(n, p) p(n + 1). For all distributions of (X,Y ), using the elementary inequality (a + b + c) 3(a + b + c ), note that E T n (r k (X)) T(r k (X)) = E n,i (X)(Y i T(r k (X i )) + T(r k (X i )) T(r k (X)) + T(r k (X))) T(r k (X)) 3E n,i (X)(T(r k (X i )) T(r k (X))) (A.1) + 3E n,i (X)(Y i T(r k (X i ))) (A.)! + 3E n,i (X) 1 T(r k (X)). (A.3)

3 Consequently, to prove the proposition, it suffices to establish that (A.1), (A.) and (A.3) tend to 0 as ` tends to infinity. This is done, respectively, in Proposition A.1, Proposition A. and Proposition A.3 below. Proposition A.1. Under the assumptions of Proposition., lim n,i (X)(T(r k (X i )) T(r k (X))) = 0. `!1 E Proof of Proposition A.1. By the Cauchy-Schwarz inequality, E n,i (X)(T(r k (X i )) T(r k (X))) q q = E n,i (X) n,i (X)(T(r k (X i )) T(r k (X))) E n, j (X) n,i (X) T(r k (X i )) T(r k (X)) j=1 = E n,i (X) T(r k (X i )) T(r k (X)) := A n. The function T is such that E[T (r k (X))] <1. Therefore, it can be approimated in an L sense by a continuous function with compact support, say T (see, e.g., Theorem A.1 in Györfi et al., 00). More precisely, for any > 0, there eists a function T such that Consequently, we obtain " A n = E 3E + 3E + 3E E T(rk (X)) T(r k (X)) <. n,i (X) T(r k (X i )) T(r k (X)) " n,i (X) T(r k (X i )) T(r k (X i )) " n,i (X) T(r k (X i )) T(r k (X)) " n,i (X) T(r k (X)) T(r k (X)) := 3A n1 + 3A n + 3A n3. 3

4 Computation of A n3. Thanks to the approimation of T by T, A n3 = E n,i (X) T(r k (X)) T(r k (X)) E T(rk (X)) T(r k (X)) <. Computation of A n1. Denote by µ the distribution of X. Then, Letting A n1 = E n,i (X) T(r k (X i )) T(r k (X i )) = `E 4 1T M { r k,m (X) r k,m (X 1 ) "`} j=1 1T M { r k,m (X) r k,m (X j ) "`} Ω = `E T(r k (u)) T(r k (u)) E A 0 n1 = E 1TM { r k,m() r k,m (u) "`} 1 T M { r k,m () r k,m (u) "`} + æ D k µ(du). 3 T(r k (X 1 )) T(r k (X 1 )) 5. j= 1T M { r k,m () r k,m (X j ) "`} 1TM { r k,m() r k,m (u) "`} 1 T M { r k,m () r k,m (u) "`} + D k, j= 1T M { r k,m () r k,m (X j ) "`} µ(d) µ(d) let us prove that A 0 n1. To this aim, observe that M` A 0 n1 = E 4 = E4 M X p=1 E4 1 { T M k,m ([r k,m(u) "`,r k,m (u)+"`])} 1 + j= 1 {X j T M k,m ([r k,m() "`,r k,m ()+"`])} 1 { S(a 1,...,a M ){1,} M k,1 (Ia 1 n,1 (u))\ \r 1 k,m (Ia M n,m (u))} 1 + j= 1 {X j T M k,m ([r k,m() "`,r k,m ()+"`])} 1 p {R n(u)} 1 + j= 1 {X j T M k,m ([r k,m() "`,r k,m ()+"`])} 3 µ(d) D 5 k 3 µ(d) D 5 k 3 µ(d) D k 5. 4

5 Here, I 1 n,m (u) = [r k,m(u) "`, r k,m (u)], I n,m (u) = [r k,m(u), r k,m (u) + "`], and Rn(u) p is the p-th set of the form k,1 (Ia 1 n,1 (u)) \ \r 1 k,m (Ia M (u)) assuming n,m that they have been ordered using the leicographic order of (a 1,...,a M ). Net, note that R p n(u) ) R p n(u) Ω M\ k,m ([r k,m() "`, r k,m () + "`]). To see this, just observe that, for all m = 1,...,M, if r k,m (z) [r k,m (u) "`, r k,m (u)], i.e., r k,m (u) "` r k,m (z) r k,m (u), then, as r k,m (u) "` r k,m () r k,m (u), one has r k,m () "` r k,m (z) r k,m () + "`. Similarly, if r k,m (u) r k,m (z) r k,m (u) + "`, then r k,m (u) r k,m () r k,m (u) + "` implies r k,m () "` r k,m (z) r k,m () + "`. Consequently, " A 0 M n1 X E = p=1 M X p=1 M X p=1 1 p {R n(u)} 1 + " µ{rn(u)} p E 1 + p µ{rn (u)} E `µ{rn(u)} p D k j= 1 {X j R p n(u)} j= 1 {X j R p n(u)} D k µ(d) D k M` (by the first statement of Lemma A.1). Thus, returning to A n1, we obtain A n1 M E T(r k (X) T(r k (X))) < M. Computation of A n. For any ± > 0, write A n = E n,i (X) T(r k (X i )) T(r k (X)) = E n,i (X) T(r k (X i )) T(r k (X)) 1 S M { r k,m (X) r k,m (X i ) >±} + E from which we get that n,i (X) T(r k (X i )) T(r k (X)) 1 T M { r k,m (X) r k,m (X i ) ±} A n 4 sup T(r k (u)) E u + " sup u,v, T M { r k,m (u) r k,m (v) ±} n,i (X)1 S M { r k,m (X) r k,m (X i ) >±} (A.4) T(r k (v)) T(r k (u)). (A.5) 5

6 ith respect to the term (A.4), if ± > "`, then n,i (X)1 S M { r k,m (X) r k,m (X i ) >±} = = 0. 1 T M { r k,m (X) r k,m (X i ) "`} 1S M { r k,m (X) r k,m (X i ) >±} j=1 1T M { r k,m (X) r k,m (X j ) "`} It follows that, for all ± > 0, this term converges to 0 as ` tends to infinity. On the other hand, letting ±! 0, we see that the term (A.5) tends to 0 as well, by uniform continuity of T. Hence, A n tends to 0 as ` tends to infinity. Letting finally go to 0, we conclude that A n vanishes as ` tends to infinity. Proposition A.. Under the assumptions of Proposition., lim `!1 E Proof of Proposition A.. n,i (X)(Y i T(r k (X i ))) E n,i (X)(Y i T(r k (X i ))) = = E = E j=1 " = 0. E[ n,i (X) n, j (X)(Y i T(r k (X i )))(Y j T(r k (X j )))] n,i (X) Y i T(r k (X i )) " n,i (X)æ (r k (X i )), where æ (r k ()) = E[ Y T(r k (X)) r k ()]. For any > 0, æ can be approimated in an L 1 sense by a continuous function with compact support æ, i.e., E æ (r k (X)) æ (r k (X)) <. 6

7 Thus E n,i (X)æ (r k (X i )) E n,i (X) æ (r k (X i )) " + E n,i (X) æ (r k (X i )) æ (r k (X i )) n,i (X) sup æ (r k (u)) E u " + E n,i (X) æ (r k (X i )) æ (r k (X i )). ith the same argument as for A n1, we obtain E n,i (X) æ (r k (X i )) æ (r k (X i )) M. h i Therefore, it remains to prove that E n,i (X)! 0 as `!1. To this aim, fi ± > 0, and note that n,i (X) = 1T M { r k,m (X) r k,m (X i ) "`} j=1 1T M { r k,m (X) r k,m (X j ) "`} 8 < min ± + : ±, 1 1T M { r k,m (X) r k,m (X i ) "`} 1 Ω æ 1T M { r k,m (X) r k,m (X i ) "`} >0 1T M { r k,m (X) r k,m (X i ) "`} To complete the proof, we have to establish that the epectation of the righthand term tends to 0. Denoting by I a bounded interval on the real line, we. 9 = ; 7

8 have E < : E 6 4 1Ω X i T M k,m ([r k,m (X) "`,r k,m (X)+"`]) 9 = æ >0 ; 1n X i T o M k,m ([r k,m(x) "`,r k,m (X)+"`]) 1 8 < : 1Ω X i T M k,m ([r k,m (X) "`,r k,m (X)+"`]) = 1 n æ X T o M k,m >0 (I) ; 1n X i T o M k,m ([r k,m(x) "`,r k,m (X)+"`]) = E 6 E M[ + µ 1 8 < : k,m (I c ) 1Ω X i T M k,m ([r k,m (X) "`,r k,m (X)+"`]) ii M[ D k,x + µ 9 = 1 n æ X T o M k,m >0 (I) ; 1n X i T o M k,m ([r k,m(x) "`,r k,m (X)+"`]) k,m (I c ) 1 n (` + 1) E X T o 3 M 4 k,m (I) µ( T 5 M k,m ([r k,m(x) "`, r k,m (X) + "`])) M[ + µ k,m (I c ). The last inequality arises from the second statement of Lemma A.1. By an appropriate choice of I, according to the technical statement (.), the second term on the right-hand side can be made as small as desired. Regarding the first term, there eists a finite number N` of points z 1,...,z N` such that M\ k,m (I) Ω [ ( j 1,..., j M ){1,...,N`} M k,1 (I n,1(z j1 )) \ \ k,m (I n,m(z jm )), where I n,m (z j ) = [z j "`/,z j +"`/]. Suppose, without loss of generality, that the sets k,1 (I n,1(z j1 )) \ \ k,m (I n,m(z jm )) are ordered, and denote by Rn p the p-th among the N M` = (d I /"`e) M sets. Here I denotes the length of the interval I and de denotes the smallest 8

9 integer greater than. For all p, R p n ) R p n Ω M\ k,m ([r k,m() "`, r k,m () + "`]). Indeed, if v R p n, then, for all m = 1,...,M, there eists j {1,...,N`} such that r k,m (v) [z j "`/,z j + "`/], that is z j "`/ r k,m (v) z j + "`/. Since we also have z j "`/ r k,m (X) z j + "`/, we obtain r k,m (X) "` r k,m (v) r k,m (X) + "`. In conclusion, p=1 1 n X T M k,m (I) o E4 µ( T 5 M k,m ([r k,m(x) "`, r k,m (X) + "`])) N M " 1 { XRn} p E p=1 µ( T M k,m ([r k,m(x) "`, r k,m (X) + "`])) N M 1{ XRn} p E µ(rn) p = N M` ª º I M =. "` The result follows from the assumption lim`!1 `" =1. M` Proposition A.3. Under the assumptions of Proposition., lim `!1 E! n,i (X) 1 T(r k (X)) 3 = 0. Proof of Proposition A.3. Since n,i(x) 1 1, one has! n,i (X) 1 T(r k (X)) T (r k (X)). Consequently, by Lebesgue s dominated convergence theorem, to prove the 9

10 proposition, it suffices to show that n,i (X) tends to 1 almost surely. Now,! P n,i (X) 6= 1 = P = P = =! 1 T M { r k,m (X) r k,m (X i )) "`} = 0 1 n X i T M P k,m([r k,m (X) "`,r k,m (X)+"`]) o = 0 µ 8i = 1,...,`,1 n X i T M k,m([r k,m () "`,r k,m ()+"`]) o = 0 µ(d) h 1 µ(\ M r 1 k,m [rk,m () "`, r k,m () + "`] )i` µ(d). Denote by I a bounded interval. Then,! P n,i (X) 6= 1 ep `µ(\ M r 1 k,m [rk,m () "`, r k,m () + "`] ) M[ 1 T { M (I)}µ(d) + µ k,m (I c ) k,m ma u ue u M[ + µ k,m (I c ). `µ(\ M r 1 k,m 1 T { M k,m (I)}! [rk,m () "`, r k,m () + "`] ) µ(d) Using the same arguments as in the proof of Proposition A., the probability l I P n,i(x) 6= 1 m M. is bounded by e 1 ` This bound vanishes as n tends "` to infinity since, by assumption, lim`!1 `" =1. M` A.3. Proof of Theorem.1 Choose. An easy calculation yields that E[ T n (r k ()) T(r k ()) rk (X 1 ),...,r k (X`),D k ] = E Tn (r k ()) E[T n (r k ()) rk (X 1 ),...,r k (X`),D k ] (A.6) r k (X 1 ),...,r k (X`),D k + E[Tn (r k ()) rk (X 1 ),...,r k (X`),D k ] T(r k ()) := E 1 + E. (A.7) 10

11 On the one hand, we have h Tn E 1 = E (r k ()) E[T n (r k ()) r k (X 1 ),...,r k (X`),D k ] i r k (X 1 ),...,r k (X`),D k = E" n,i ()(Y i E[Y i r k (X i )]) r k (X 1 ),...,r k (X`),D k. Developing the square and noticing that E Y j Y i,r k (X 1 ),...,r k (X`),D k = E[Yj r k (X j )], since Y j is independent of Y i and of the X j s with j 6= i, we have 1T M E 1 = E { r k,m () r k,m (X i ) "`} Y i E[Y i r k (X i )] 1T M { r k,m () r k,m (X i ) "`} r k(x 1 ),...,r k (X`),D k (A.8) Thus, = V(Y i r k (X i )) 1 T M { r k,m () r k,m (X i ) "`} 1T M { r k,m () r k,m (X i ) "`} E 1 4R 1Ω 1T M { r k,m () r k,m (X i ) "`} >0 æ 1T M { r k,m () r k,m (X i ) "`}., (A.9) where V() denotes the variance of a random variable. On the other hand, recalling the notation ß introduced in Section 3, we obtain for the second 11

12 term E : E = E[Tn (r k ()) r k (X 1 ),...,r k (X`),D k ] T(r k ()) = n,i ()E[Y i r k (X i )] T(r k ()) 1 {ß>0} + T (r k ())1 {ß=0} 1T M { r k,m () r k,m (X i ) "`} E[Y i r k (X i )] T(r k ()) + T (r k ())1 {ß=0} j=1 1T M { r k,m () r k,m (X j ) "`} 1 {ß>0} (A.10) (by Jensen s inequality) 1T M = { r k,m () r k,m (X i ) "`} T(r k(x i )) T(r k ()) 1 {ß>0} (A.11) j=1 1T M { r k,m () r k,m (X j ) "`} + T (r k ())1 {ß=0} L " ` + T (r k ())1 {ß=0}. (A.1) Now, E T n (r k (X)) T(r k (X)) E (T n (r k ()) T(r k ()) µ(d). Then, using the decomposition (A.7) and the upper bounds (A.9) and (A.1), E T n (r k (X)) T(r k (X)) 4R 1 {ß>0} E µ(d) + L " ` B + E T (r k ())1 {ß=0} µ(d) R Ω d 4R 1 {ß>0} E E D k æµ(d) + L " ` B + E E T (r k ())1 {ß=0} D k µ(d). Thus, thanks to Lemma A.1, E T n (r k (X)) T(r k (X)) 8R 1 (` + 1) µ( T M { r k,m () r k,m (X) "`}) µ(d) + L " `!` M\ + T (r k ()) 1 µ( { r k,m () r k,m (X) "`}) µ(d). 1

13 Consequently, E T n (r k (X)) T(r k (X)) 8R 1 (` + 1) µ( T M { r k,m () r k,m (X) "`}) µ(d) + L " `! M\ + T (r k ())ep `µ( { r k,m () r k,m (X) "`}) µ(d) 8R 1 (` + 1) µ( T M { r k,m () r k,m (X) "`}) µ(d) + L " ` µ + sup T (r k ())ma ue u ur + 1 `µ( T M { r k,m () r k,m (X) "`}) µ(d). Introducing a bounded interval I as in the proof of Proposition., we observe that the boundedness of the r k yields that! M[ µ k,m (I c ) = 0, as soon as I is sufficiently large, independently of k. Then, proceeding as in the proof of Proposition., we obtain E T n (r k (X)) T(r k (X)) ª º I M ª º 8R 1 I M "` ` L " ` + 1 R ma ue u ur + "` ` R C 1 + L " `, `" M` for some positive constant C 1, independent of k. Hence, for the choice "` / ` 1 M+, we obtain E T n (r k (X)) T(r k (X)) C` M+, for some positive constant C depending on L, R and independent of k, as desired. B. Numerical results 13

14 Table 4 (SM): Quadratic errors of the implemented machines and COBRA in high-dimensional situations. Means and standard deviations over 00 independent replications. Model 9 Model 10 Model 11 lars ridge fnn tree rf COBRA m sd m sd m sd Table 5 (SM): Quadratic errors of eponentially weighted aggregate (EA) and COBRA. 00 independent replications. Model 9 Model 10 Model 11 Model 1 EA COBRA m sd m sd m sd m sd

15 Figure (SM): Eamples of calibration of parameters "` and Æ. The bold point is the minimum. (a) Model 5, uncorrelated design. (b) Model 5, correlated design. Risk machine machines 3 machines 4 machines 5 machines 6 machines Risk machine machines 3 machines 4 machines 5 machines 6 machines Epsilon Epsilon (c) Model 9. (d) Model 1. Quadratic Risk machine machines 3 machines 4 machines 5 machines Quadratic Risk machine machines 3 machines 4 machines 5 machines Epsilon Epsilon 15

16 Figure 3 (SM): Boplots of quadratic errors, uncorrelated design. From left to right: lars, ridge, fnn, tree, randomforest, COBRA. (a) Model 1. (b) Model. (c) Model 3. (d) Model 4. Boplots of errors for the testing set Boplots of errors for the testing set LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF CR (e) Model 5. (f) Model 6. (g) Model 7. (h) Model LR Lasso Ridge KNN CART RF COBRA LR Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA Figure 4 (SM): Boplots of quadratic errors, correlated design. From left to right: lars, ridge, fnn, tree, randomforest, COBRA. (a) Model 1. (b) Model. (c) Model 3. (d) Model 4. Boplots of errors for the testing set Boplots of errors for the testing set LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF CR LR Lasso Ridge KNN CART RF COBRA (e) Model 5. (f) Model 6. (g) Model 7. (h) Model LR Lasso Ridge KNN CART RF COBRA LR Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA 16

17 Figure 5 (SM): Prediction over the testing set, uncorrelated design. The more points on the first bissectri, the better the prediction. (a) Model 1. (b) Model. (c) Model 3. (d) Model 4. Testing set. Prediction versus true response Testing set. Prediction versus true response (e) Model 5. (f) Model 6. (g) Model 7. (h) Model Figure 6 (SM): Prediction over the testing set, correlated design. The more points on the first bissectri, the better the prediction. (a) Model 1. (b) Model. (c) Model 3. (d) Model 4. Testing set. Prediction versus true response Testing set. Prediction versus true response (e) Model 5. (f) Model 6. (g) Model 7. (h) Model

18 Figure 7 (SM): Eamples of reconstruction of the functional dependencies, for covariates 1 to 4. (a) Model 1, uncorrelated design. (b) Model 1, correlated design. Reconstruction of the functional dependence for covariates 1 to 4 Reconstruction of the functional dependence for covariates 1 to 4 f1() f() f3() f4() f1() f3() f4() f() (c) Model 3, uncorrelated design. (d) Model 3, correlated design. f1() f() f3() f4() f1() f3() f4() f()

19 Figure 8 (SM): Boplot of errors, high-dimensional models. (a) Model 9 Lasso Ridge FNN CART RF COBRA (b) Model 10 Lasso Ridge FNN CART RF COBRA (c) Model 11 Lasso Ridge FNN CART RF COBRA Figure 9 (SM): How stable is COBRA? (a) Boplot of errors: Initial sample is randomly cut (1000 replications of Model 1). Lasso Ridge FNN CART RF COBRA (b) Empirical risk with respect to the size of subsample D k, in Model Split Risk Figure 10 (SM): Boplot of errors: EA vs COBRA (a) Model 9. EA COBRA (b) Model 10. EA COBRA (c) Model 11. EA COBRA (d) Model 1. EA COBRA

20 Figure 11 (SM): Prediction over the testing set, real-life data sets. (a) Concrete Slump Test. (b) Concrete Compressive Strength (c) ine Quality, red wine. (d) ine Quality, white wine

21 Figure 1 (SM): Boplot of quadratic errors, real-life data sets. (a) Concrete Slump(b) Concrete Com-(cpressive Strength. wine. white ine Quality, red(d) ine Quality, Test. wine Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA Lasso Ridge KNN CART RF COBRA 1