Semiparametric Mixtures of Nonparametric Regressions

Transcription

1 Semiparametric Mixtures of Noparametric Regressios Sijia Xiag, Weixi Yao Proofs I tis sectio, te coditios required by Teorems, 2, 3 ad 4 are listed. Tey are ot te weakest sufficiet coditios, but could easily facilitate te proofs. Te proofs of Teorems, 2, 3 ad 4 are also preseted i tis sectio. Tecical Coditios: (C) 4 0 ad 2 log(/) as ad 0. (C2) as ad 0. (C3) Te sample {(X i, Y i ), i =,..., } are idepedetly ad idetically distributed from f(x, y) wit fiite sixt momets. Te port for x, deoted by X R, is bouded ad closed. (C4) f(x, y) > 0 i its port ad as cotiuous first derivative. (C5) 3 l(, x, y)/ i j k M ijk (x, y), were E(M ijk (x, y)) is bouded for all i, j, k ad all X, Y. (C6) Te ukow fuctios m j (x), j =,..., k, ave cotiuous secod derivative. (C7) σ 2 j > 0 ad π j > 0 for j =,..., k ad k π j =. (C8) E(X 2r ) < for some ɛ < r, 2ɛ. (C9) I (x) ad I m (x) are positive defiite. (C0) Te kerel fuctio K( ) is symmetric, cotiuous wit compact port. (C) Te margial desity f(x) of X is Lipscitz cotiuous ad bouded away from 0. X as a bouded port X. (C2) t 3 K(t) ad t 3 K (t) are bouded ad t 4 K(t)dt <. (C3) E q 4 <, E q m 4 <, were q ad q m are defied i te proof of Teorem 2.5. Correspodig autor, Scool of Matematics ad Statistics, Zejiag Uiversity of Fiace & Ecoomics, Hagzou, Zejiag 3008, P. R. Cia. address: sjxiag@zufe.edu.c. Departmet of Statistics, Uiversity of Califoria, Riverside, CA 92887, U.S.A.

2 Te ext lemma is from Fa ad Huag (2005), ad will be used trougout te rest of te proofs. Lemma. Let {(X i, Y i ), i =,..., } be i.i.d radom vectors from (X, Y ), were X is a radom vector ad Y is a scalar radom variable. Let f be te joit desity of (X, Y ), ad furter assume tat E Y r < ad x y r f(x, y)dy <. Let K( ) be a bouded positive fuctio wit bouded port, satisfyig a Lipscitz coditio. Te, [K (X i x)y i E{K (X i x)y i }] = O p(γ log /2 (/)), give 2ɛ, for some ɛ < /r, were γ = () /2. I order to prove te asymptotic properties of {ˆπ, ˆm, ˆσ 2 }, we first eed to study te asymptotic property of { π, m, σ 2 }, wic is te maximum local log-likeliood estimator of (5). Defie π j = { π j π j }, m j = { m j m j }, σ 2 j = { σ 2 j σ 2 j }. Let π = ( π,..., π k )T, m = ( m,..., m k )T, ad σ 2 = ( σ 2,..., σ k 2 )T. Furtermore, defie = (( m ) T, ( π ) T, ( σ 2 ) T ) T, β = (( π) T, ( σ 2 ) T ) T. Lemma 2. Suppose tat coditios (C2)-(C0) are satisfied, te, f (x)i (x)s = Op ( 2 + γ log /2 (/)), were S is defied i (3). Proof of Lemma 2. Sice { π, m, σ 2 } maximizes l (π, m, σ 2 ) defied i (5), it is easy to see tat maximizes l ( ) = {l((x) + γ, Y i ) l((x), Y i )}K (X i x), = S + 2 T W + o p ( 2 ), () were S = l((x), Y i ) K (X i x), W = 2 l((x), Y i ) T K (X i x), (2) ad te secod equality olds by Taylor expasio. It is easy to see tat W = f(x)i (x)+ o p (), ad terefore, l ( ) = S 2 f(x) T I (x) + o p ( 2 ). (3) By Lemma ad assumptio (C9), it ca be sow tat for all x X, W coverges to f(x)i (x) uiformly. From (3) ad assumptio (C7) ad (C9), we kow tat l ( ) is 2

3 covex fuctio defied o a covex ope set, we is large eoug. Terefore, by te covexity lemma (Pollard, 99), (S + 2 T W ) [S 2 f(x) T I (x) ] P 0 olds uiformly for all x X ad i ay compact set. We kow tat f (x)i (x)s is a uique maximizer of (3), ad by defiitio, is a maximizer of (), te, by Lemma A. of Carroll et al. (997), f (x)i (x)s P 0, wic also implies tat Sice maximizes (), 0 =γ =γ l((x) + γ, Yi ) K (X i x) l((x), Y i ) K (X i x) + γ 2 tat is, W + Op (γ 2 ) = S. Terefore, = f (x)i (x)s + o p (). (4) 2 l((x), Y i ) T K (X i x) + O p (γ 2 ), {W E(W )} + O p (γ 2 ) = S E(W ) = S + f(x)i (x). (5) From (4) ad (9), it is easy to sow tat = O p (). By Lemma, W E(W ) = O p { 2 + γ log /2 (/)}, tus {W E(W )} + O p (γ 2 ) = O p { 2 + γ log /2 (/)}. Combied wit (5), we ave S + f(x)i (x) = Op { 2 + γ log /2 (/)}. Sice f(x) ad I (x) are bouded ad cotiuous fuctios i a closed set of X ad I (x) is positive defiite, f (x)i S = Op { 2 + γ log /2 (/)}. Proof of Teorem. Defie ˆβ = (ˆβ β), were ˆβ maximizes l 2 (β) i (6). Let l( m(x i ), β, Y i ) = log{ k π j φ(y i m j (X i ), σj 2 }, l( m(x i ), β + β /, Y i ) = log{ k (π j + π j / )φ(y i m j (X i ), σ 2 j + σ 2 j / }. 3

4 Sice ˆβ maximizes l 2, it is easy to see tat ˆβ maximizes l (β ) = {l( m(x i ), β + β /, Y i ) l( m(x i ), β, Y i } = A β + 2 β T B β + o p ( β 2 ), were A = l( m(x i ),β,y i ) ad B β = 2 l( m(x i ),β,y i ). It ca be easily see β β T tat B = B + o p () wit B = E{I β (X)}, terefore, by quadratic approximatio lemma, Defie R = R + O p ( k etries of I ˆβ = B A + o p (). (6) 2 l(m(x i ),β,y i ) ( m(x β m T i ) m(x i )), te A = l(m(x i ),β,y i ) + β m m 2 ). Let ϕ(x t, Y t ) be a k vector wose elemets are te first (X t ) l((xt),yt). From assumptio (C), we kow tat O p { /2 [γ 2 + (X i ) γ 2 log /2 (/)]} = o p (). By Lemma 3, (X i ) (X i ) = f (X i )I X i ) + O p {γ 2 + γ 2 log /2 (/)}. Sice m(x i ) m(x t ) = O(X i X t ), R = 3/2 = R 2 + O p ( /2 2 ). 2 l(m(x i ), β, Y i ) β m T f (X i )ϕ(x t, Y t )K (X i X t ) + O p ( /2 2 ) l((x i ),Y t) K (X t It ca be sow tat E[ 2 l(m(x i ),β,y i ) f β m (X T i )K (X i X t )] = I βm (X t ). Let ϖ(x t, Y t ) = I βm (X t )ϕ(x t, Y t ), ad R 3 = /2 ϖ(x t, Y t ), te R 2 R P 3 0, ad terefore A = { l(m(x i), β, Y i ) β ϖ(x i, Y i )} + o p (), give 4 0. Let Σ = V ar{ l((x),y β ) see tat E(A ) = 0, terefore by (6), ϖ(x, Y )}, te Var(A ) = Σ. It ca be easily (ˆβ β) D N(0, B ΣB ). Proof of Teorem 2. Defie ˆm = ( ˆm(x) m(x)), were ˆm(x) maximizes (7). It ca be sow tat ˆm (x) = f(x) I m (x) Ŝ + o p (), (7) were Ŝ = l(m(x), ˆβ, Y i ) K (X i x). (8) m 4

5 Notice tat Ŝ = S + D + o p (). l(m(x), β, Y i ) K (X i x) + m (ˆβ β) 2 l(m(x), β, Y i ) m β T K (X i x) + o p () 2 l(m(x),β,y i ) were S is defied i (2). Sice (ˆβ β) = O p () ad K m β T (X i x) = f(x)iβm T (x) + o p(), te D = (ˆβ β) 2 l(m(x),β,y i ) K m β T (X i x) = f(x)iβm T (x) + o p(). Tus, from (7), ˆm (x) = f(x) I m (x) S + o p (). Let Λ(u x) = E[ l(m(x),β,y m ) X = u], it ca be sow tat E(S ) = [ 2 f(x)λ (x x) + f (x)λ (x x)]κ 2 2, V ar(s ) = f(x)i m (x)ν 0. (9) To complete te proof, let (x) = I m (x)[ 2 Λ (x x) + f (x)f (x)λ (x x)]κ 2 2, ad m (x) be a k vector wose elemets are te first k etries of (x), te ( ˆm(x) m(x) m (x) + o p ( 2 )) D N(0, f (x)i m (x)ν 0 ). Proof of Teorem 3. (i) Assume te latet variables {Z i, i =,..., } be a radom sample from populatio Z, te P (Z i = j Y, ) = π j φ(y m j, σ 2 j )/ k π jφ(y m j, σ 2 j ), ad terefore, log{ k π j φ(y i m j, σj 2 )} = log{π j φ(y i m j, σj 2 )} log{p (Z i = j Y, )}. (0) Give (l) (X i ) = (m (l) (X i ), π (l) (X i ), σ 2(l) (X i )), for ay i =,...,, P (Z i = j Y i, (l) (X i )) = p (l+) ij ad k p(l+) ij =. Terefore, by (0) l () = k { log{π j φ(y i m j, σj 2 )}p (l+) ij }K (X i x) k { log{p (Z i = j Y, )}p (l+) ij }K (X i x). () Based o te M-step of (8), (9) ad (0), we ave k { k { log{π (l+) j log{π (l) j (x)φ(y i m (l+) j (x)φ(y i m (l) j (x), σ2(l) j (x), σ 2(l+) j (x))}p (l+) ij }K (X i x) (x))}p (l+) }K (X i x). ij 5

6 To complete te proof, based o (), we oly eed to sow lim i probability. Defie L = U = k { log{ P (Z i = j Y i, (l+) (x)) P (Z i = j Y i, (l) (x)) }p(l+) ij }K (X i x) 0 k { log{ P (Z i = j Y i, (l+) (x)) P (Z i = j Y i, (l) (x)) }p(l+) ij }K (X i x), k log{ { P (Z i = j Y i, (l+) (x)) P (Z i = j Y i, (l) (x)) }p(l+) ij }K (X i x), te, by Jese s iequality, L U. We complete te proof by sowig tat U P 0. Witout loss of geerality, assume tat P (Z i = j Y, (l) (x)) δ > 0 for some small value δ. Sice E(U) = E{log[ k P (Z i =j Y i, (l+) (x)) P (Z i =j Y i, (l) (x)) P (Z i = j Y i, (l) (X i ))]K (X i x)}, by similar argumet as i te proof of Teorem 2 ad Teorem 3, it ca be sow tat E(U) 0, ad Var(U) = O p (() ). Terefore, by Cebysv s iequality, U = o p (), ad tus completes te proof. (ii) Notice tat P (Z i = j Y, m, ˆβ) = ˆπ j φ(y m j, ˆσ j 2 )/ k ˆπ jφ(y m j, ˆσ j 2 ), P (Z i = j Y i, m (l) (X i ), ˆβ) = p (l+) ij ad k p(l+) ij =, were p (l+) ij is defied i (). Te rest of te proof is i lie wit part (i), ad tus is omitted ere. (iii) Notice tat by fixig m( ) = m (l) ( ), l (π, m (l) ( ), σ 2 ) = l 2 (π, σ 2 ). Terefore, by te ascet property of te ordiary EM algoritm, l (π (l+), m (l) ( ), σ 2(l+) ) = l 2 (π (l+), σ 2(l+) ) l 2 (π (l), σ 2(l) ) = l (π (l), m (l) ( ), σ 2(l) ). Tus, to complete te proof, we oly eed to sow lim if [l (π (l+), m (l+) ( ), σ 2(l+) ) l (π (l+), m (l) ( ), σ 2(l+) )] 0. If we fix ˆπ = π (l+) ad ˆσ 2 = σ 2(l+), te by part (ii), lim if [l 3 (m (l+) (x)) l 3 (m (l) (x))] 0 i probability for ay x {X t, t =,..., }. Terefore, lim if lim if 2 f(x t ) [l 3 (m (l+) (X t )) l 3 (m (l) (X t ))] lim if f(x t ) [l 3 (m (l+) (X t )) l 3 (m (l) (X t ))] 0. Sice K( ) is symmetric about 0, 2 f(x t) l 3 (m (l) (X t )) = Γ(l) i, were Γ (l) i = k f(x t ) log[ ˆπ j φ(y i m (l) j (X t), ˆσ 2 j )]K (X t X i ). 6

7 It ca be sow tat E(Γ (l) i X i, Y i ) = log[ k ˆπ jφ(y i m (l) j (X i), ˆσ j 2 )](+o p ()), ad Var(Γ (l) i X i, Y i ) = O p (() ). Te fact tat E(Γ(l) i X i, Y i ) = l (π (l+), m (l) ( ), σ 2(l+) ), ad E(Γ(l+) i X i, Y i ) = l (π (l+), m (l+) ( ), σ 2(l+) ) completes te proof. Proof of Teorem 4. Sice ˆβ as faster covergece rate ta ˆm( ), ˆm( ) as te same asymptotic properties as if β were kow. Terefore, i te followig proof, we study te property of ˆm( ) assumig β to be kow. Defie l((x i),y i ) = q i, 2 l((x i ),Y i ) = q T i ad similarly, defie q mi, q mmi ad so o. Let be te estimator uder H (Huag et al., 203), ad ˆm be te estimator uder H 0 (model (). From previous proof, we ave i,j,l (X i ) (X i ) = f (X i )I (X i ) ˆm(X i ) m(x i ) = f (X i )I m (X i ) q t K (X t X i )( + o p ()), (2) q mt K (X t X i )( + o p ()). (3) By (2) ad (3), we ca obtai tat l( (X i ), Y i ) l((x i ), Y i ) = { q if T (X l )I (X l )q l K (X i X l ) i,l + q 2 if T 2 (X 2 l )I (X l )q l I (X l )q j K (X i X l )K (X j X l )}( + o p ()), l( ˆm(X i ), Y i ) i,j,l l(m(x i ), Y i ) = { q T mif (X l )Im (X l )q ml K (X i X l ) + q 2 mif T 2 (X 2 l )Im (X l )q mml Im (X l )q mj K (X i X l )K (X j X l )}( + o p ()), ad so, T = i,l [q T ii (X l )q l qmii T m (X l )q ml ]f (X l )K (X i X l ) i,l [qii T i,j,l I (X l )q j qmii T m (X l )q mml Im (X l )q mj ]f 2 (X l )K (X i X l )K (X j X l ) Λ + 2 Γ. (X l )q l By similar argumet as Fa et al. (200), it ca be sow tat uder coditios (C9)-(C2), as 0, 3/2, Λ = 2k K(0)Ef(X) + [q T ii (X l )q l qmii T m (X l )q ml ]f (X l )K (X i X l ) + o p ( /2 ), Γ = (2k ) Ef(X) l i K 2 (t)dt 2 K K (X i X j ) + o p ( /2 ). i<j [q T ii 7 (X i )q j qmii T m (X i )q mj ]f (X i )

8 Terefore, T = µ + W /2 + o p ( /2 ), were µ = (2k ) X [K(0) 0.5 K 2 (t)dt], W = i j {q T ii (X j )[2K (X i X j ) K K (X i X j )]f (X j )q j q T mii m (X j )[2K (X i X j ) K K (X i X j )]f (X j )q mj }. It ca be sow tat Var(W ) ζ, were ζ = 2(2k )Ef (X) [2K(t) K K(t)] 2 dt. Apply Propositio 3.2 i de Jog (987), we obtai tat W D N(0, ζ), ad completes te proof. Refereces Carroll, R. J., Fa, J., Gijbels, I., ad Wad, M.P. (997). Geeralized partially Liear Sigle-idex Models. Joural of America Statistical Associatio, 92, de Jog, P. (987). A cetral limit teorem for geeralized quadratic forms. Probablity Teory ad Related Fields, 75, Fa, J., Zag, C., ad Zag, J. (200). Geeralized likeliood ratio statistics ad Wilks peomeo. Te Auals of Statistics, 29, Fa, J. ad Huag, T. (2005). Profile Likeliood Iferece o Semiparametric Varyig- Coefficiet Partially Liear Models. Beroulli,, Huag, M., Li, R., ad Wag, S. (203). Noparametric mixture of regressio models. Joural of America Statistical Associatio, 08,