S1 Notation and Assumptions

Transcription

1 Statistica Siica: Supplemet Robust-BD Estimatio ad Iferece for Varyig-Dimesioal Geeral Liear Models Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag Uiversity of Wiscosi-Madiso Supplemetary Material S Notatio ad Assumptios For a matrix M, its eigevalues, miimum eigevalue, maximum eigevalue ad trace are labeledbyλ j (M),λ mi (M),λ max (M)adtr(M)respectively. Let M = sup x = Mx = {λ max (M T M)} /2 be the matrix L 2 orm; let M F = {tr(m T M)} /2 be the Frobeius orm. See Golub ad Va Loa (996) for details. Throughout the proof, C is used as a geeric fiite costat. We first impose some regularity coditios, which are ot the weakest possible but facilitate the techical derivatios. Coditio A: A. sup β <. A. X = max j p X j is bouded almost surely. A2. E( X XT ) exists ad is osigular. A4. There isalargeeoughopesubset ofr p + which cotaisthe trueparameter poit β, such that F ( X T β) is bouded almost surely for all β i the subset. A5. w( ) is a bouded fuctio. Assume that ψ(r) is a bouded, odd fuctio, ad twice differetiable, such that ψ (r), ψ (r)r, ψ (r), ψ (r)r ad ψ (r)r 2 arebouded; V( ) >, V (2) ( ) is cotiuous. The matrix H is positive defiite, with eigevalues uiformly bouded away from. A6. q (4) ( ) is cotiuous, ad q (2) ( ) <. G (3) ( ) is cotiuous. A7. F( ) is mootoe ad a bijectio, F (3) ( ) is cotiuous, ad F () ( ). Coditio B:

2 S2 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag B5. The matrices Ω ad H are positive defiite, with eigevalues uiformly bouded away from. Also, H Ω is bouded away from. Coditio C: C4. Thereis a largeeoughope subset ofr p + whichcotaisthe trueparameter poit β, such that A β = g, ad F ( X T β) is bouded almost surely for all β i the subset. Coditio D: D5. The eigevalues of H are uiformly bouded away from. Also, H /2 Ω /2 is bouded away from. S2 Proofs of Mai Results Proof of Theorem We follow the idea of the proof i Fa ad Peg (24). Let r = p / ad ũ = (u,u,...,u p ) T R p +. It suffices to show that for ay give ǫ >, there exists a sufficietly large costat C ǫ such that, for large we have { } P if l ( β +r ũ ) > l ( β ) ǫ. (S2.) ũ =C ǫ This implies that with probability at least ǫ, there exists a local miimizer β of l ( β) i the ball { β + r ũ : ũ C ǫ } such that β β = O P (r ). To show (S2.), cosider l ( β +r ũ ) l ( β ) = where ũ = C ǫ. {ρ q (Y i,f ( X T i ( β +r ũ )))w(x i ) ρ q (Y i,f ( X T i β ))w(x i )} I, (S2.2) By Taylor s expasio, I = I, +I,2 +I,3, where I, = r / p (Y i ; X T i β )w(x i ) X T iũ, I,2 = r 2 /(2) p 2 (Y i ; X T i β ; )w(x i )( X T iũ) 2, (S2.3)

3 Robust-BD Estimatio ad Iferece S3 I,3 = r/(6) 3 p 3 (Y i ; X T i β )w(x i )( X T iũ) 3 for β located betwee β ; ad β ; +r ũ. Hece I, r p (Y i ; X T i β ; )w(x i ) X i ũ = O P (r p /) ũ. (S2.4) For I,2 i (S2.3), I,2 = r2 2 E{p 2 (Y i ; X T i β ; )w(x i )( X T iũ) 2 } + r2 [p 2 2 (Y i ; X T i β ; )w(x i )( X T iũ) 2 E{p 2 (Y i ; X T i β ; )w(x i )( X T iũ) 2 }] I,2, +I,2,2, where I,2, = 2 r 2 ũt H ũ. Meawhile, we have I,2,2 r 2 Thus, = r 2 O P (p / ) ũ 2. For I,3 i (S2.3), we observe that I,3 r 3 [p 2 (Y i ; X T i β ; )w(x i ) X i XT i E{p 2 (Y i; X T i β ; )w(x i ) X i XT i } ] F ũ 2 I,2 = r2 2 ũt H ũ +O P (r 2 p / ) ũ 2. (S2.5) p 3 (Y i ; X T i β ) w(x i) X T iũ 3 = O P (r 3 p3/2 ) ũ 3, which follows from Coditios A, A, A4 ad A5. By (S2.4) ad p 4 /, we ca choose some large C ǫ such that I, ad I,3 are all domiated by the first term of I,2 i (S2.5), which is positive by the eigevalue assumptio o H. This implies (S2.). Proof of Theorem 2 Notice the estimatig equatios l( β) β β= β =, sice β is a local miimizer of l ( β). Taylor s expasio applied to the left side of the estimatio equatios yields { } = p (Y i ; X T i β ; )w(x i ) X i

4 S4 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag { { p 2 (Y i ; X T i β ; )w(x i ) X i XT i } ( β β; ) p 3 (Y i ; X T i β )w(x i){ X T i ( β β; )} 2 Xi p (Y i ; X T i β ; )w(x i ) X i }+K 2 ( β β; )+K 3, (S2.6) where β lies betwee β ; ad β. Below, we will show K 2 H = O P (p / ), (S2.7) K 3 = O P (p 5/2 /). (S2.8) First, to show (S2.7), ote that K 2 H = K 2 E(K 2 ) L. Similar argumets for the proof of I,2,2 i Theorem give L = O P (p / ). Secod, a similar proof used for I,3 i (S2.3) completes (S2.8). Third, by (S2.6) (S2.8) ad β β; = O P ( p /), we see that H ( β β; ) = p (Y i ; X T i β ; )w(x i ) X i +u, (S2.9) where u = O P (p 5/2 /). Note that by Coditio B5, Thus A Ω /2 u A F λ max (Ω /2 ) u = {tr(a A T )}/2 /λ /2 mi (Ω ) u = O P (p 5/2 / ) = o P (). A Ω /2 {H ( β β; )} = A Ω /2 p (Y i ; X T i β ; )w(x i ) X i +o P (). To complete provig Theorem 2, we apply the Lideberg-Feller cetral limit theorem(va dervaart,998)to Z i,wherez i = /2 A Ω /2 p (Y i ; X T i β ; )w(x i ) X i. Itsufficestocheck cov(z i) G; (II) E( Z i 2+δ ) = o()forsomeδ >. Coditio follows from the fact that var{p (Y; X T β ; )w(x ) X } = Ω. To verify coditio (II), otice that usig Coditios B5 ad A5, { [ E( Z i 2+δ ) (2+δ)/2 E A 2+δ F Ω /2 X {ψ(r(y,m(x ))) G (m(x ))} {q (m(x )) V(m(X ))} ] 2+δ } F w(x ) (m(x )) C (2+δ)/2 E[{λ /2 mi (Ω ) X } 2+δ {ψ(r(y,m(x ))) G (m(x ))}

5 Robust-BD Estimatio ad Iferece S5 {q (m(x )) V(m(X ))}/F (m(x )) 2+δ ] (2+δ)/2 E[ {ψ(r(y,m(x ))) G (m(x ))} Cp (2+δ)/2 {q (m(x )) V(m(X ))}/F (m(x )) 2+δ ] O((p /) (2+δ)/2 ). Thus, we get E( Z i 2+δ ) O((p /) (2+δ)/2 ) = O(p (2+δ)/2 / δ/2 ), which is o(). This verifies Coditio (II). Propositio (covariace matrix estimatio) Assume A, A, A2, A4, A5, B5, A6, ad A7 i the Appedix. Let V = H Ω H ad V = Ĥ Ω Ĥ. If p4/ as, the for ay /p -cosistet estimator β of β;, we have A ( V V )A T P for ay k (p +) matrix A satisfyig A A T G, where G is a k k matrix ad k is ay fixed iteger. Proof: Note A ( V V )A T V V A 2 F. Sice A 2 F to prove that V V = o P (). tr(g), it suffices First, we prove Ĥ H = o P (). Note that Ĥ H = {p 2 (Y i ; X T i { + I +I 2. β) p 2 (Y i ; X T i β ; )}w(x i ) X i XT i p 2 (Y i ; X T i β ; )w(x i ) X i XT i H From the proof of (S2.7) i Theorem 2, we kow that I 2 = O P (p / ) = o P (). We oly eed to cosider the term I. Let m i = m(x i ), m i = m(x i ), r i = r(y i, m i ) ad r i = r(y i,m i ). The I = [A (Y i, m i )+{ψ( r i ) G ( m i )}A ( m i ) A (Y i,m i ) {ψ(r i ) G (m i )}A (m i )]w(x i ) X i XT i = {G ( m i )A ( m i ) G (m i )A (m i )}w(x i ) X i XT i + {A (Y i, m i )+ψ( r i )A ( m i ) A (Y i,m i ) ψ(r i )A (m i )}w(x i ) X i XT i I, +I,2. Let g( ) = G ( )A ( ). By the assumptios, g( ) is differetiable. Thus g( m i ) g(m i ) = (g F ) ( X T i β )X T i( β β; ) = O P ()O P ( p )O P ( p /) = O P (p / ), }

6 S6 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag where β is betwee β ad β;. Thus g( m(x i )) g(m(x i )) w(x i ) X i XT i = O P (p / )O P (p ) = O P (p 2 / ). F Similar argumets give I, = O P (p 2 / ) ad I,2 = O P (p 2 / ). Thus I = O P (p 2 / ) = o P (). Secod, we show Ω Ω = o P (). It is easy to see that Ω Ω = {p 2 (Y i; X T i { + =, +,2, β) p 2 (Y i; X T i β ; )}w 2 (X i ) X i XT i p 2 (Y i; X T i β ; )w 2 (X i ) X i XT i Ω where, = O P (p 2 / ) ad,2 = O P (p / ). We observe that Ω Ω = O P (p 2 / ) = o P (). Third, we show V V = o P (). Note V V = L + L 2 + L 3, where L = Ĥ ( Ω Ω )Ĥ, L 2 = Ĥ (H Ĥ)H Ω Ĥ ad L 3 = H Ω Ĥ (H Ĥ )H. By Assumptio B5, it is straightforward to verify that H O(), Ĥ O P() ad H Ω O(). Sice L Ĥ Ω Ω Ĥ, we coclude L = o P (), ad similarly L 2 = o P () ad L 3 = o P (). Hece V V = o P (). } Proof of Theorem 3 For the matrix A i (4.3), there exists a (p + k) (p +) matrix B satisfyig B B T = I p + k ad A B T =. Therefore, A β = g is equivalet to β = B Tγ + c, where γ is a (p + k) vector ad c = A T G g. Thus uder H i (4.3), we have β ; = B Tγ ; + c. The miimizig l ( β ) subject to A β = g is equivalet to miimizig l (Bγ T + c ) with respect to γ, ad we deote by γ the miimizer. Note that uder (4.4), β is the uique miimizer of l ( β ). Hece Λ = 2{l (B γ T +c ) l ( β)}. Before showig Theorem 3, we eed Lemma. Lemma Assume coditios of Theorem 3. The uder H i (4.3), we have that B T( γ γ ; ) = B T(B H B T ) B p (Y i; X T i β ; )w(x i ) X i +o P ( /2 ), ad 2{l (B T γ +c ) l ( β)} = (B T γ +c β) T H (B T γ +c β)+op ().

7 Robust-BD Estimatio ad Iferece S7 Proof: To obtai the first part, followig the proof of (S2.9) i Theorem 2, we have a similar expressio for γ, B H B( γ T γ ; ) = B p (Y i ; X T i β ; )w(x i ) X i +w, with w = o P ( /2 ). As a result, B T ( γ γ ; ) = BT (B H B T ) B p (Y i ; X T i β ; )w(x i ) X i +B T (B H B T ) w. We otice that B(B T H B) T w (B H B) T w w /λ mi (H ) = o P ( /2 ), i which the fact λ mi (B H B T) λ mi(h ) is used. The proof of the secod part proceeds i three steps. I Step, we use the followig Taylor expasio for l (B γ T +c ) l ( β), l (B γ T +c ) l ( β) = p 2 2 (Y i ; X T i β)w(x i ){ X T i(b γ T +c β)} 2 + p 6 3 (Y i ; X T i β )w(x i ){ X T i(b γ T +c β)} 3 I +I 2, where β lies betwee β ad B T γ +c. I Step 2, we aalyze the stochastic order of B γ T + c β. For a matrix X whose colum vectors are liearly idepedet, set P X = X(X T X) X T. Defie H = I p + P /2 H = P B T /2 H. The H A T BT (B H B T) B = H /2 H H /2. By (S2.9) ad the first part of Lemma, we see immediately that B γ T +c β = B T ( γ γ ; ) ( β β; ) { = H /2 H H /2 p,i w(x i ) X i }+o P ( /2 ), (S2.) wherep,i = p (Y i ; X T i β ; ). Notethat H /2 H H /2 O P (/ ). This gives B T γ +c β = OP (/ ). { p,iw(x i ) X i } = (S2.) I Step 3, we coclude from (S2.) that I 2 = O P {(p /) 3/2 } = o P (/). The 2{l (B γ T +c ) l ( β)} = 2I +o P (). Similar to the proof of Propositio, it is straightforward to see that 2I = (B γ T +c β) { T } p 2 (Y i ; X T i β)w(x i ) X i XT i (B γ T +c β)

8 S8 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag = (B γ T +c β) { T } p 2 (Y i ; X T i β ; )w(x i ) X i XT i (B γ T +c β)+op () = (B γ T +c β) T E{p 2 (Y ; X T β ; )w(x i ) X XT }(BT γ +c β)+op () = (B γ T +c β) T H (B γ T +c β)+op (). The the secod part of Lemma is proved. to We ow show Theorem 3. For part (i), a direct use of Lemma ad (S2.) leads 2{l (B γ T +c ) l ( β)} { } T { = p,i w(x i ) X i H /2 H H /2 p,i w(x i ) X i }+o P (). Sice H is idempotet of rak k, it ca be writte as H = C T C, where C is a k (p +) matrix satisfyig C C T = I k. The 2{l (B γ T +c ) l ( β)} { } T { = C H /2 p,i w(x i ) X i C H /2 p,i w(x i ) X i }+o P (). Now cosider part (ii). If ψ(r) = r ad the q-fuctio satisfies (4.5), the p (y;θ) = q (y;θ), p 2 (y;θ) = q 2 (y;θ) ad H = Ω /C, where q j (y;θ) = j θ Q j q (y,f (θ)). I this case, similar argumets for Theorem 2 yield C H /2 which completes the proof. q (Y i ; X T i β ; )w(x i ) X i L N(,CIk ), Proof of Theorem 4 Before showig Theorem 4, Lemma 2 is eeded. Lemma 2 Assume coditios of Theorem 4. The β β ; = H p (Y i ; X T i β ; )w(x i ) X i +o P ( /2 ), (A Ĥ Ω Ĥ AT ) /2 L A ( β β; ) N(,I k ). Proof:Followig(S2.9)itheproofofTheorem2,weobservethat u = O P (p 5/2 /) = o P ( /2 ). Coditio B5 completes the proof for the first part.

9 Robust-BD Estimatio ad Iferece S9 Toshowthesecodpart,deoteU = A H Ω H AT adû = A Ĥ Ω Ĥ Notice that the eigevalues of H Ω H are uiformly bouded away from. So are the eigevalues of U. From the first part, we see that It follows that A ( β β; ) = A H p (Y i ; X T i β ; )w(x i ) X i +o P ( /2 ). U /2 A ( β β; ) = Z i +o P (), AT. where Z i = /2 U /2 A H p (Y i ; X T i β ; )w(x i ) X i. To show Z L i N(,I k ), similar to the proof for Theorem 2, we check (III) cov(z i) I k ; (IV) E( Z i 2+δ ) = o() for some δ >. Coditio (III) is straightforward sice cov(z i) = U /2 U U /2 = I k. To check coditio (IV), similar argumets used i the proof of Theorem 2 give that E( Z i 2+δ ) = O((p /) (2+δ)/2 ). This ad the boudedess of ψ yield E( Z i 2+δ ) O(p (2+δ)/2 / δ/2 ) = o(). Hece U /2 L A ( β β; ) N(,I k ). (S2.2) From the proof of Propositio, it ca be cocluded that Û U = o P () ad that the eigevalues of Û are uiformly bouded away from ad with probability tedig to oe. Cosequetly, Û /2 U /2 I k = o P (). (S2.3) Combiig(S2.2),(S2.3)adSlutsky stheoremcompletestheproofthat Û /2 A ( β L β ; ) N(,I k ). We ow show Theorem 4, which follows directly from H i (4.3) ad the secod part of Lemma 2. This completes the proof. Proof of Theorem 5 Note that W ca be decomposed ito three additive terms, I = {A ( β β; )} T (A V A T ) {A ( β β; )}, I 2 = 2(A β; g ) T (A V A T ) {A ( β β; )}, I 3 = (A β; g ) T (A V A T ) (A β; g ), where V = Ĥ Ω Ĥ. We observe that I χ 2 k followig the secod part of Lemma 2; I 3 = (A β; g ) T M (A β; g ){+o P ()} by Propositio ; I 2 = O P ( ) by Cauchy-Schwartz iequality. Thus I 3 λ mi (M ) A β; g 2 {+o P ()} = λ max(m) A β; g 2 +o P (). These complete the proof for W. L

10 S Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag Proof of Theorem 6 Followigthe secodpartoflemma 2, we observethat (A V A T ) /2 L (A β g ) N(M /2 c,i k ), which completes the proof. Proof of Theorem 7 We first eed to show Lemma 3. Lemma 3 Suppose that (X o,y o ) follows the distributio of (X,Y) ad is idepedet of the traiig set T. If Q is a BD, the E{Q(Y o, m(x o ))} = E{Q(Y o,m(x o ))}+E{Q(m(X o ), m(x o ))}. Proof: Let q be the geeratig fuctio of Q. The Q(Y o, m(x o )) = [q(m(xo )) E{q(Y o ) T,X o }]+[E{q(Y o ) T,X o } q(y o )] q(m(x o ))+q( m(x o ))+{Y o m(x o )}q ( m(x o )). (S2.4) Sice (X o,y o ) is idepedet of T, we deduce from Chow ad Teicher (989, Corollary 3, p. 223) that E{q(Y o ) T,X o } = E{q(Y o ) X o }. (S2.5) Similarly, E{Y o q ( m(x o )) T,X o } = E(Y o X o )q ( m(x o )) = m(xo )q ( m(x o )). (S2.6) Applyig (S2.5) ad (S2.6) to (S2.4) results i E{Q(Y o, m(x o )) T,X o } = E{Q(Y o,m(x o )) X o }+Q(m(X o ), m(x o )) ad thus the coclusio. Now show Theorem 7. Settig Q i Lemma 3 to be the misclassificatio loss gives /2[E{R( φ )} R(φ,B )] E[ m(x o ).5 I{m(X o ).5, m(x o ) >.5}] +E[ m(x o ).5 I{m(Xo ) >.5, m(xo ).5}] = I +I 2. For ay ǫ >, it follows that I = E[ m(x o ).5 I{m(Xo ) <.5 ǫ, m(xo ) >.5}] +E[ m(x o ).5 I{.5 ǫ m(x o ).5, m(x o ) >.5}] P{ m(x o ) m(xo ) > ǫ}+ǫ ad similarly, I 2 ǫ+p{ m(x o ) m(xo ) ǫ}. Recall that m(x o ) m(xo ) = F ( X o T β) F ( X o T β; ) (F ) ( X o T β ) Xo β β;,

11 Robust-BD Estimatio ad Iferece S forsome β betwee β ; ad β, where Xo = (,X o T ) T. ByCoditioA4, wecoclude that (F ) ( X o T β ) = O P (). This alog with β β; = O P () ad X o = O P ( p ) implies that m(x o ) m(xo ) = O P(r p ) = o P (). Therefore I ad I 2, which completes the proof. S3 Figures 7 i Sectio 6.2 cotamiated data, classical dev. loss, w= cotamiated data, robust dev. loss, with w j cotamiated data, classical exp. loss, w= 2 j cotamiated data, robust exp. loss, with w j 2 j Figure 7: (Simulated Beroulli respose data with cotamiatio) Boxplots of β j β j;, j =,,...,p (from left to right i each pael). Left paels: the o-robust estimates; right paels: the robust estimates.

12 S2 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag cotamiated data, robust dev. loss, with w, H cotamiated data, robust dev. loss, with w, composite H (II) cotamiated data, robust exp. loss, with w, H cotamiated data, robust exp. loss, with w, composite H (II) Figure 8: (Simulated Beroulli respose data with cotamiatio) Empirical quatiles (othey-axis)ofteststatisticsw versusquatiles(othex-axis)oftheχ 2 k distributio. Solid lie: the 45 degree referece lie. Left paels: for testig H ; right paels: for testig H (II).

13 Robust-BD Estimatio ad Iferece S3 deviace loss, H deviace loss, composite H (II) d expoetial loss, H d d expoetial loss, composite H (II) d Figure 9: Level of tests for the Beroulli respose data. The dashed lie correspods to the o-robust Wald-type test; the solid lie correspods to the robust Wald-type test; the dotted lie idicates the 5% omial level. Left paels: for testig H ; right paels: for testig H (II).

14 S4 Chumig Zhag Xiao Guo Che Cheg Zhegju Zhag o cotamiated case, deviace loss, H cotamiated case, deviace loss, H o cotamiated case, expoetial loss, H cotamiated case, expoetial loss, H Figure : Observed power fuctios of tests for the Beroulli respose data. The dashed lie correspods to the o-robust Wald-type test; the solid lie correspods to the robust Wald-type test; the dotted lie idicates the 5% omial level. Left paels: o-cotamiated case; right paels: cotamiated case.