Chapter 10. U-statistics Statistical Functionals and V-Statistics

Transcription

1 , respectively. Chpter 0 U-sttistics Whe oe is willig to ssume the existece of simple rdom smple X,...,X, U- sttistics geerlize commo otios of ubised estimtio such s the smple me d the ubised smple vrice (i fct, the U i U-sttistics stds for ubised ). Eve though U-sttistics my be cosidered bit of specil topic, their study i lrge-smple theory course hs side beefits tht mke them vluble pedgogiclly. The theory of U- sttistics icely demostrtes the pplictio of some of the lrge-smple topics preseted thus fr. Furthermore, the study of U-sttistics ebles theoreticl discussio of sttisticl fuctiols, which gives isight ito the commo moder prctice of bootstrppig. 0. Sttisticl Fuctiols d V-Sttistics Let S be set of cumultive distributio fuctios d let T deote mppig from S ito the rel umbers R. The T is clled sttisticl fuctiol. If, sy, we re give simple rdom smple from distributio with ukow distributio fuctio F, we my wt to ler the vlue of = T (F ) for (kow) fuctiol T. I this wy, we my thik of the vlue of sttisticl fuctiol s prmeter we wish to estimte. Some prticulr istces of sttisticl fuctiols re s follows: If T (F )=F (c) for some costt c, the T is sttisticl fuctiol mppig ech F to P F (Y pple c). If T (F )=F (p) for some costt p, where F (p) is defied i Equtio (3.8), the T mps F to its pth qutile. If T (F )=E F (Y )ort (F )=Vr F (Y ), the T mps F to its me µ or its vrice 85 Suppose X,...,X is idepedet d ideticlly distributed sequece i other words, simple rdom smple with distributio fuctio F (x). We defie the empiricl distributio fuctio ˆF to be the distributio fuctio for discrete uiform distributio o {X,...,X }. I other words, ˆF (x) = #{i : X i pple x} = I{X i pple x}. Sice ˆF (x) is legitimte distributio fuctio, resoble estimtor of T (F ) is the soclled plug-i estimtor T ( ˆF ). For exmple, if T (F )=E F (Y ), the the plug-i estimtor give simple rdom smple X,X,... from F is T ( ˆF ) = E ˆF (Y ) = X i = X. (0.) I Equtio (0.), Y is rdom vrible whose distributio is the sme s the empiricl distributio of the dt, which mes tht the true popultio me of Y equls the smple me of X,...,X. This equtio illustrtes how we distiguish betwee the ottiol use of X d Y i this chpter: We use X d X i wheever we must refer specificlly to the dt X,...,X ; but we use Y d Y i wheever we refer geerlly to fuctiol d o specific referece to the dt is mde. As we will see lter, plug-i estimtor, such s X bove, is lso kow s V-sttistic or V-estimtor whe the fuctiol T (F ) is of prticulr type clled expecttio fuctiol. Suppose tht for some rel-vlued fuctio (y), we defie T (F )=E F (Y ). Note i this cse tht T { F + ( )F } = E F (Y ) + ( )E F (Y ) = T (F ) + ( )T (F ). For this reso, such fuctiol is sometimes clled lier fuctiol (see Defiitio 0.). To geerlize this ide, we cosider rel-vlued fuctio tkig more th oe rel rgumet, sy (y,...,y ) for some >, d defie T (F )=E F (Y,...,Y ), (0.) which we tke to me the expecttio of (Y,...,Y ) where Y,...,Y is simple rdom smple from the distributio fuctio F. Lettig deote some permuttio mppig {,...,} oto itself, the fct tht the Y i re idepedet d ideticlly distributed 86

2 mes tht the joit distributio of (Y,...,Y ) is the sme s the joit distributio of (Y (),...,Y () ). Therefore, E F (Y,...,Y )=E F (Y (),...,Y () ). Sice there re! such permuttios, cosider the fuctio (y,...,y ) def = X (y (),...,y () ).! ll Sice E F (Y,...,Y )=E F (Y,...,Y ) d is symmetric i its rgumets, we see tht i Equtio (0.) we my ssume without loss of geerlity tht is symmetric i its rgumets. I other words, (y,...,y )= (y (),...,y () ) for y permuttio of the itegers through. A fuctio defied s i Equtio (0.) is clled expecttio fuctiol, s summrized i the followig defiitio: Defiitio 0. For some iteger, let : R! R be fuctio symmetric i its rgumets. The expecttio of (Y,...,Y ) uder the ssumptio tht Y,...,Y re idepedet d ideticlly distributed from some distributio F will be deoted by E F (Y,...,Y ). The the fuctiol T (F ) = E F (Y,...,Y ) is clled expecttio fuctiol. If =, the T is lso clled lier fuctiol. Expecttio fuctiols re importt i this chpter becuse they re precisely the fuctiols tht give rise to V-sttistics d U-sttistics. The fuctio (y,...,y ) i Defiitio 0. is used so frequetly tht we give it specil me: Defiitio 0. Let T (F )=E F (Y,...,Y ) be expecttio fuctiol, where : R! R is fuctio tht is symmetric i its rgumets. The is clled the kerel fuctio ssocited with T (F ). Suppose T (F ) is expecttio fuctiol defied ccordig to Equtio (0.). If we hve simple rdom smple of size from F, the s oted erlier, turl wy to estimte T (F ) is by the use of the plug-i estimtor T ( ˆF ). This estimtor is clled V-estimtor or V-sttistic. It is possible to write dow V-sttistic explicitly: Sice ˆF ssigs probbility to ech X i, we hve V = T ( ˆF ) = E ˆF (Y,...,Y ) = (X i,...,x i ). (0.3) I the cse =, Equtio (0.3) becomes V = (X i ). (0.4) 87 i= i= It is cler i Equtio (0.4) tht E V = T (F ), which we deote by. Furthermore, if =Vr F (Y ) <, the the cetrl limit theorem implies tht p (V ) d! N(0, For >, however, the sum i Equtio (0.3) cotis some terms i which i,...,i re ot ll distict. The expecttio of such terms is ot ecessrily equl to = T (F ) becuse i Defiitio 0., requires idepedet rdom vribles from F. Thus, V is ot ecessrily ubised for >. Exmple 0.3 Let = d (y,y )= y y. It my be show (Problem 0.) tht the fuctiol T (F )=E F Y Y is ot lier i F. Furthermore, sice Y i Y i is ideticlly zero wheever i = i, it my lso be show tht the V-estimtor of T (F ) is bised: E V = XX i6=j ). E F Y i Y j = T (F ) becuse there re exctly ( ) pirs (i, j) for which i 6= j. Sice the bis i V is due to the duplictio mog the subscripts i,...,i, oe wy to correct this bis is to restrict the summtio i Equtio (0.3) to sets of subscripts i,...,i tht coti o duplictio. For exmple, we might sum isted over ll possible subscripts stisfyig i < <i. The result is the U-sttistic, which is the topic of Sectio 0.. Exercises for Sectio 0. Exercise 0. Let X,...,X be simple rdom smple from F. For fixed t for which 0 <F(t) <, fid the symptotic distributio of ˆF (t). Exercise 0. Let T (F )=E F Y Y. Show tht T (F ) is ot lier fuctiol by exhibitig distributios F d F d costt (0, ) such tht T { F + ( )F } 6= T (F ) + ( )T (F ). Exercise 0.3 Let X,...,X be rdom smple from distributio F with fiite third bsolute momet. () For =, fid (y,y ) such tht E F (Y,Y )=Vr F Y. Your fuctio should be symmetric i its rgumets. Hit: The fct tht =EY E Y Y leds immeditely to o-symmetric fuctio. Symmetrize it. 88

3 (b) For = 3, fid (y,y,y 3 ) such tht E F (Y,Y,Y 3 )=E F (Y E F Y ) 3. As i prt (), should be symmetric i its rgumets. 0. Asymptotic Normlity Recll tht X,...,X re idepedet d ideticlly distributed rdom vribles. Becuse the V-sttistic V = (X i,...,x i ) i= i= is i geerl bised estimtor of the expecttio fuctiol T (F )=E F (Y,...,Y ) due to the presece of summds i which there re duplicted idices o the X ik, oe wy to produce ubised estimtor is to sum oly over those (i,...,i ) i which o duplictes occur. Becuse is ssumed to be symmetric i its rgumets, we my without loss of geerlity restrict ttetio to the cses i which pple i < <i pple. Doig this, we obti the U-sttistic U : Defiitio 0.4 Let be positive iteger d let (y,...,y ) be the kerel fuctio ssocited with expecttio fuctiol T (F ) (see Defiitios 0. d 0.). The the U-sttistic correspodig to this fuctiol equls U = X X pplei< <ipple where X,...,X is simple rdom smple of size. (X i,...,x i ), (0.5) The U i U-sttistic stds for ubised (the V i V-sttistic stds for vo Mises, who ws oe of the origitors of this theory i the lte 940 s). The ubisedess of U follows sice it is the verge of terms, ech with expecttio T (F )=E F (Y,...,Y ). Exmple 0.5 Cosider rdom smple X,...,X from F, d let R = ji{w j > 0} j= be the Wilcoxo siged rk sttistic, where W,...,W re simply X,...,X reordered i icresig bsolute vlue. We showed i Exmple 8. tht R = j= ix I{X i + X j > 0}. 89 Lettig (, b) =I{ + b>0}, we see tht is symmetric i its rgumets d thus it is legitimte kerel fuctio for expecttio fuctiol. We fid tht R = U + I{X i > 0} = U + O P ( ), where U is the U-sttistic correspodig to the expecttio fuctiol T (F )= P F (Y + Y > 0). Therefore, some symptotic properties of the siged rk test tht we hve lredy derived elsewhere c lso be obtied usig the theory of U-sttistics. I the specil cse =, the V-sttistic d the U-sttistic coicide. I this cse, we hve lredy see tht both U d V re symptoticlly orml by the cetrl limit theorem. However, for >, the two sttistics do ot coicide i geerl. Furthermore, we my o loger use the cetrl limit theorem to obti symptotic ormlity becuse the summds re ot idepedet (ech X i ppers i more th oe summd). To prove the symptotic ormlity of U-sttistics, we shll use method sometimes kow s the H-projectio method fter its ivetor, Wssily Hoe dig. If (y,...,y ) is the kerel fuctio of expecttio fuctiol T (F )=E F (Y,...,Y ), suppose X,...,X is simple rdom smple from the distributio F. Let = T (F ) d let U be the U-sttistic defied i Equtio (0.5). For pple k pple, suppose tht the vlues of Y,...,Y k re held costt, sy, Y = y,...,y k = y k. This my be viewed s projectig the rdom vector (Y,...,Y ) oto the ( k)-dimesiol subspce i R give by {(y,...,y k,c k+,...,c ): (c k+,...,c ) R k }. If we tke the coditiol expecttio, the result will be fuctio of y,...,y k, which we will deote by k. To summrize, for k =,..., we shll defie k(y,...,y k ) = E F (y,...,y k,y k+,...,y ). (0.6) Equivletly, we my use coditiol expecttio ottio to write k(y,...,y k )=E F { (Y,...,Y ) Y,...,Y k }. (0.7) From Equtio (0.7), we see tht E F k (Y,...,Y k )=E F (Y,...,Y )= for ll k. The vrices of the ew ottio, lettig k fuctios will be useful i wht follows. Therefore, we itroduce k =Vr F k (Y,...,Y k ). (0.8) The importce of the k vlues, prticulrly, is see i the followig theorem, which gives closed-form expressio for the vrice of U-sttistic: 90

4 Theorem 0.6 The vrice of U-sttistic is X Vr F U = k If,..., re ll fiite, the k= Vr F U = + O k. Theorem 0.6 is proved i Exercise 0.4. This theorem shows tht the vrice of p U teds to, d ideed we my well woder whether it is true tht p (U ) is symptoticlly orml with this limitig vrice. It is the gol of Hoe dig s H-projectio method to prove exctly tht fct. We shll derive the symptotic ormlity of U i sequece of steps. The bsic ide will be to show tht U hs the sme limitig distributio s the sum Ũ = k. E F (U X j ) (0.9) j= of projectios. The symptotic distributio of Ũ follows from the cetrl limit theorem becuse Ũ is the sum of idepedet d ideticlly distributed rdom vribles. Lemm 0.7 For ll pple j pple, Proof: E F (U X j )= { (X j ) }. Expdig U usig the defiitio (0.5) gives E F (U X j )= X X E F { (X i,...,x i ) X j }, pplei< <ipple where from equtio (0.7) we see tht E F { (X i,...,x i ) X j } = (X j ) if j {i,...,i } 0 otherwise. The umber of wys to choose {i,...,i } so tht j is mog them is E F (U X j ) = { (X j ) } = { (X j ) }. 9, so we obti Lemm 0.8 If < d Ũ is defied s i Equtio (0.9), the p Ũ d! N(0, ). Proof: Lemm 0.8 follows immeditely from Lemm 0.7 d the cetrl limit theorem sice (X j ) hs me d vrice. Now tht we kow the symptotic distributio of Ũ, it remis to show tht U Ũ hve the sme symptotic behvior. Lemm 0.9 o E F Ũ (U ) =E F Ũ. Proof: By Equtio (0.9) d Lemm 0.7, E F Ũ = /. Furthermore, o E F Ũ (U ) = E F {( (X j ) )(U )} = = j= E F E F {( (X j ) )(U ) X j } j= E F { (X j ) } =, where the third equlity bove follows from Lemm 0.7. j= Lemm 0.0 If k < for k =,...,, the p U Ũ P! 0. d Proof: Sice covergece i qudrtic me implies covergece i probbility (Theorem.7), it su ces to show tht E F p(u Ũ )o! 0. By Lemm 0.9, E F U Ũ = Vr F U E F Ũ. But Vr F U = + O(/) by Theorem 0.6, d E F Ũ =, provig the result. Filly, sice p (U )= p Ũ + p (U Ũ ), Lemms 0.8 d 0.0 log with Slutsky s theorem result i the theorem we origilly set out to prove: 9

5 Theorem 0. If k < for k =,...,, the Therefore (sice ddig the costt does ot chge the vrice), p (U ) d! N(0, ). (0.0) Exmple 0. Cosider the expecttio fuctiol defied by the kerel fuctio (y,y )=(y y ) /. We obti T (F )=E F (Y,Y )=E F (Y + Y Y Y )/ =E F Y (E F Y ) =Vr F Y. Give simple rdom smple X,...,X from F, let us derive the symptotic distributio of the ssocited U-sttistic. First, we obti U = XX pplei<jpple (X i,x j )= ( ) j= (X i,x j ), where we hve used the fct tht (X i,x j ) = 0 i this exmple wheever i = j i order to llow both i d j to rge from to. Cotiuig, we obti U = = ( ) ( ) j= (Xi + Xj X i X j ) / Xi + ( ) j= X j ( ) j= X i X j. By observig tht X = P P i j X ix j /, we my ow coclude tht " # U = Xi = (X i X ), which is the usul ubised smple vrice. We hve lredy derived the symptotic distributio of the smple vrice twice! i Exmples 4. d 5.9, though we used bised versio of the smple vrice i ech of those exmples. Now, we my obti the sme result third time usig the theory of U-sttistics we just developed. Sice = here, we kow tht p (U ) d! N(0, 4 ). It remis to fid. To this ed, we must defie (y). Lettig µ =E F Y d =Vr F Y, we obti (y) =E F (y, Y )=E F (y yy + Y )/ =(y µy + + µ )/. 93 =Vr F (Y )= 4 Vr F (Y µy + µ )= 4 Vr F (Y µ). We coclude tht p (U ) d! N[0, Vr F (Y µ) ]. This cofirms the results obtied i Exmples 4. d 5.9. Exercise 0.4 Exercises for Sectio 0. Prove Theorem 0.6, s follows: () Prove tht for pple k pple, E F (Y,...,Y ) (Y,...,Y k,y +,...,Y +( k) )= k + d thus Cov F { (Y,...,Y ), (Y,...,Y k,y +,...,Y +( k) )} = k. Hit: Use coditioig! I this cse, it mkes sese to coditio o Y,...,Y k becuse coditiol o those rdom vribles, the expressio bove is the product of idepedet reliztios of k. (b) Show tht Vr F U = X k k= Cov F { (X,...,X ), (X,...,X k,x +,...,X +( k) )} k d the use prt () to prove the first equtio of theorem 0.6. (c) Verify the secod equtio of theorem 0.6. Exercise 0.5 Suppose kerel fuctio (y,...,y ) stifies E F (Y i,...,y i ) < for y (ot ecessrily distict) i,...,i. Prove tht if U d V re the correspodig U- d V-sttistics for simple rdom smple X,...,X, the p (V U ) P! 0 so tht V hs the sme symptotic distributio s U. Hit: Verify d use the equtio 3 X V U = 4V X (X i,...,x i ) 5 ll ij distict " # X + X! (X i,...,x i ). 94 ll ij distict

6 Exercise 0.6 For the kerel fuctio of Exmple 0.3, (, b) = b, the correspodig U-sttistic is clled Gii s me di erece d it is deoted G.For rdom smple from uiform(0, ), fid the symptotic distributio of G. Exercise 0.7 Let (y,y,y 3 ) hve the property ( + by,+ by,+ by 3 ) = (y,y,y 3 )sg(b) for ll, b. (0.) Let =E (Y,Y,Y 3 ). The fuctio sg(b) is defied s the sig of b, which my be expressed s I{b >0} I{b <0}. () We defie the distributio F to be symmetric if for Y F, there exists some µ (the ceter of symmetry) such tht Y µ d µ Y hve the sme distributio. Prove tht if F is symmetric the = 0. (b) Let y d ỹ deote the me d medi of y,y,y 3. Let (y,y,y 3 )= sg(y ỹ). Show tht this fuctio stisfies criterio (0.), the fid the symptotic distributio for the correspodig U-sttistic if F is the stdrd uiform distributio. Exercise 0.8 If the rgumets of the kerel fuctio (y,...,y ) of U-sttistic re vectors isted of sclrs, ote tht Theorem 0. still pplies with o modifictio. With this i mid, cosider for y, z R the kerel (y, z) = I{(y z )(y z ) > 0}. () Give simple rdom smple X,...,X, if U deotes the U-sttistic correspodig to the kerel bove, the sttistic U is clled Kedll s tu sttistic. Suppose the mrgil distributios of X i d X i re both cotiuous, with X i d X i idepedet. Fid the symptotic distributio of p (U ) for pproprite vlue of. (b) To test the ull hypothesis tht smple W,...,W is idepedet d ideticlly distributed gist the ltertive hypothesis tht the W i re stochsticlly icresig i i, suppose we reject the ull hypothesis if the umber of pirs (W i,w j ) with W i <W j d i<jis greter th c. This test is clled M s test gist tred. Bsed o your swer to prt (), fid c so tht the test hs symptotic level.05. (c) Estimte the true level of the test i prt (b) for simple rdom smple of size from stdrd orml distributio for ech {5, 5, 75}. Use 5000 smples i ech cse. Exercise 0.9 Suppose tht X,...,X is simple rdom smple from uiform(0, ) distributio. For some fixed, let U be the U-sttistic ssocited with the kerel 95 fuctio (y,...,y ) = mx{y,...,y }. Fid the symptotic distributio of U. 0.3 U-sttistics i the o-iid cse I this sectio, we geerlize the ide of U-sttistics i two di eret directios. First, we cosider sigle U-sttistics for situtios i which there is more th oe smple. Next, we cosider the joit symptotic distributio of two (sigle-smple) U-sttistics. We begi by geerlizig the ide of U-sttistics to the cse i which we hve more th oe rdom smple. Suppose tht X i,...,x ii is simple rdom smple from F i for ll pple i pple s. I other words, we hve s rdom smples, ech potetilly from di eret distributio, d i is the size of the ith smple. We my defie sttisticl fuctiol =E (Y,...,Y ; Y,...,Y ; ; Y s,...,y ss ). (0.) Notice tht the kerel i Equtio (0.) hs s rgumets; furthermore, we ssume tht the first of them my be permuted without chgig the vlue of, the ext of them my be permuted without chgig the vlue of, etc. I other words, there re s distict blocks of rgumets of, d is symmetric i its rgumets withi ech of these blocks. Filly, otice tht i Equtio (0.), we hve dropped the subscripted F o the expecttio opertor used i the previous sectio, whe we wrote E F this is becuse there re ow s di eret distributios, F through F s, d writig E F,...,Fs would mke bd ottiol situtio eve worse! Lettig N = + + s deote the totl smple size, the U-sttistic correspodig to the expecttio fuctiol (0.) is X U N = X X i,...,x i ; ; X sr,...,x srs. (0.3) s s pplei< <i pple ppler< <rs pples As we did i the cse of sigle-smple U-sttistics, defie for 0 pple k pple i,...,0 pple k s pple s d k ks(y,...,y k ; ; Y s,...,y sks ) = E { (Y,...,Y ; ; Y s,...,y ss ) Y,...,Y k,,y s,...,y sks } (0.4) k ks =Vr k ks(y,...,y k ; ; Y s,...,y sks ). (0.5) 96

7 By rgumet similr to the oe used i the proof of Theorem 0.6, but much more tedious ottiolly, we c show tht k ks = Cov { (Y,...,Y ; ; Y s,...,y ss ), (Y,...,Y k,y,+,...; ; Y s,...,x sks,y s,s+,...)}. (0.6) Notice tht some of the k i my equl 0. This ws ot true i the sigle-smple cse, sice 0 would hve merely bee the costt, so 0 would hve bee 0. I the specil cse whe s =, Equtios (0.4), (0.5) d (0.6) become d jk(y,...,y j ; Z,...,Z k ) = E { (Y,...,Y ; Z,...,Z ) Y,...,Y j,z,...,z k }, jk = Vr jk(y,...,y j ; Z...,Z k ), jk = Cov { (Y,...,Y ; Z,...,Z ), (Y,...,Y j,y +,...,Y +( j); Z,...,Z k,z +,...,Z +( k)), respectively, for 0 pple j pple d 0 pple k pple. Although we will ot derive it here s we did for the sigle-smple cse, there is lgous symptotic ormlity result for multismple U-sttistics, s follows. Theorem 0.3 Suppose tht for i =,...,s, X i,...,x ii is rdom smple from the distributio F i d tht these s smples re idepedet of ech other. Suppose further tht there exist costts,..., s i the itervl (0, ) such tht i /N! i for ll i d tht <. The where p N(UN = s ) d! N(0, ), s s Although the ottio required for the multismple U-sttistic theory is ightmrish, life becomes cosiderbly simpler i the cse s = d = =, i which cse we obti U N = X X j= 97 (X i ; X j ). Equivletly, we my ssume tht X,...,X m re simple rdom smple from F d Y,...,Y re simple rdom smple from G, which gives U N = mx (X i ; Y j ). (0.7) m j= I the cse of the U-sttistic of Equtio (0.7), Theorem 0.3 sttes tht p N(UN )! d 0 N 0, + 0, where = lim m/n, 0 =Cov{ (X ; Y ), (X ; Y )}, d 0 =Cov{ (X ; Y ), (X ; Y )}. Exmple 0.4 For idepedet rdom smples X,...X m from F d Y,...,Y from G, cosider the Wilcoxo rk-sum sttistic W, defied to be the sum of the rks of the Y i mog the combied smple. We my show tht W = m ( +)+ X I{X i <Y j }. j= Therefore, if we let (; b) = I{ <b}, the the correspodig two-smple U- sttistic U N is relted to W by W = ( + ) + mu N. Therefore, we my use Theorem 0.3 to obti the symptotic ormlity of U N, d therefore of W. However, we mke o ssumptio here tht F d G re merely shifted versios of oe other. Thus, we my ow obti i priciple the symptotic distributio of the rk-sum sttistic for y two distributios F d G tht we wish, so log s they hve fiite secod momets. The other directio i which we will geerlize the developmet of U-sttistics is cosidertio of the joit distributio of two sigle-smple U-sttistics. Suppose tht there re two kerel fuctios, (y,...,y ) d '(y,...,y b ), d we defie the two correspodig U-sttistics U () = X X (X i,...,x i ) d U () = b pplei< <ipple X X '(X j,...,x jb ) pplej< <jbpple for sigle rdom smple X,...,X from F. Defie = E U () d = E U (). Furthermore, defie jk to be the covrice betwee j (Y,...,Y j ) d ' k (Y,...,Y k ), where j d ' k re defied s i Equtio (0.7). Lettig ` = mi{j, k}, it my be proved tht jk =Cov (Y,...,Y ), '(Y,...,Y`,Y +,...,Y +(b `) ). (0.8) 98

8 Note i prticulr tht jk depeds oly o the vlue of mi{j, k}. The followig theorem, stted without proof, gives the joit symptotic distributio of U () d U (). Theorem 0.5 Suppose X,...,X is rdom smple from F d tht : R! R d ' : R b! R re two kerel fuctios stisfyig Vr (Y,...,Y ) < d Vr '(Y,...,Y b ) <. Defie =Vr (Y ) d =Vr' (Y ), d let jk be defied s i Equtio (0.8). The ( U () p U () )! d 0 N, b 0 b b Exercises for Sectio 0.3 Exercise 0.0 Suppose X,...,X m d Y,...,Y re idepedet rdom smples from distributios Uif(0, ) d Uif(µ, µ + ), respectively. Assume m/n! s m,!d 0 <µ<. () Fid the symptotic distributio of the U-sttistic of Equtio (0.7), where (x; y) =I{x <y}. I so doig, fid fuctio g(x) such tht E (U N )=g(µ). (b) Fid the symptotic distributio of g(y (c) Fid the rge of vlues of µ for which the Wilcoxo estimte of g(µ) is symptoticlly more e ciet th g(y X). (The symptotic reltive e ciecy i this cse is the rtio of symptotic vrices.) Exercise 0. Solve ech prt of Problem 0.0, but this time uder the ssumptios tht the idepedet rdom smples X,...,X m d Y,...,Y stisfy P (X pple t) =P (Y pple t) =t for t [0, ] d 0 < <. As i Problem 0.0, ssume m/n! (0, ). Exercise 0. Suppose X,...,X m d Y,...,Y re idepedet rdom smples from distributios N(0, ) d N(µ, ), respectively. Assume m/(m + )! / s m,!. Let U N be the U-sttistic of Equtio (0.7), where (x; y) = I{x <y}. Suppose tht (µ) d (µ) re such tht p N[UN (µ)] d! N[0, Clculte (µ) d (µ) for µ {.,.5,,.5, }. 99 X). (µ)].. Hit: This problem requires bit of umericl itegrtio. There re couple of wys you might do this. A symbolic mthemtics progrm like Mthemtic or Mple will do it. There is fuctio clled itegrte i R d Splus d oe clled qud i MATLAB for itegrtig fuctio. If you cot get y of these to work for you, let me kow. Exercise 0.3 Suppose X,X,...re idepedet d ideticlly distributed with fiite vrice. Defie S = (x i x) d let G be Gii s me di erece, the U-sttistic defied i Problem 0.6. Note tht S is lso U-sttistic, correspodig to the kerel fuctio (x,x )= (x x ) /. () If X i re distributed s Uif(0, ), give the joit symptotic distributio of G d S by first fidig the joit symptotic distributio of the U-sttistics G d S. Note tht the covrice mtrix eed ot be positive defiite; i this problem, the covrice mtrix is sigulr. (b) The sigulr symptotic covrice mtrix i this problem implies tht s!, the joit distributio of G d S becomes cocetrted o lie. Does this pper to be the cse? For 000 smples of size from Uiform(0, ), plot sctterplots of G gist S.Tke {5, 5, 00}. 0.4 Itroductio to the Bootstrp This sectio does ot use very much lrge-smple theory side from the wek lw of lrge umbers, d it is ot directly relted to the study of U-sttistics. However, we iclude it here becuse of its turl reltioship with the cocepts of sttisticl fuctiols d plug-i estimtors see i Sectio 0., d lso becuse it is icresigly populr d ofte misuderstood method i sttisticl estimtio. Cosider sttisticl fuctiol T (F ) tht depeds o. For istce, T (F ) my be some property, such s bis or vrice, of estimtor ˆ of = (F ) bsed o rdom smple of size from some distributio F. As exmple, let (F )=F be the medi of F.Tkeˆ to be the mth order sttistic from rdom smple of size =m from F. Cosider the bis T B (F )=E F ˆ (F ) d the vrice T V (F )=E F ˆ (E F ˆ ). 00

9 Theoreticl properties of T B d T V re very di cult to obti. Eve symptotics re t very helpful, sice p (ˆ )! d N{0, /(4f ( ))} tells us oly tht the bis goes to zero d the limitig vrice my be very hrd to estimte becuse it ivolves the ukow qutity f( ), which is hrd to estimte. Cosider the plug-i estimtors T B ( ˆF ) d T V ( ˆF ). (Recll tht ˆF deotes the empiricl distributio fuctio, which puts mss of o ech of the smple poits.) I our medi exmple, d T B ( ˆF ) = E ˆF ˆ ˆ T V ( ˆF ) = E ˆF (ˆ ) (E ˆF ˆ ), where ˆ is the smple medi from rdom smple X,...,X from ˆF. To see how di cult it is to clculte T B ( ˆF ) d T V ( ˆF ), cosider the simplest otrivil cse, = 3: Coditiol o the order sttistics (X (),X (),X (3) ), there re 7 eqully likely possibilities for the vlue of (X,X,X 3), the smple of size 3 from ˆF, mely (X (),X (),X () ), (X (),X (),X () ),...,(X (3),X (3),X (3) ). Of these 7 possibilities, exctly + 6 = 7 hve the vlue X () occurrig or 3 times. Therefore, we obti This implies tht P (ˆ = X () )= 7 7,P(ˆ = X () )= 3 7, d P (ˆ = X (3) )= 7 7. E ˆF ˆ = 7 (7X () +3X () +7X (3) ) d E ˆF (ˆ ) = 7 (7X () +3X () +7X (3)). Therefore, sice ˆ = X (), we obti d T B ( ˆF )= 7 (7X () 4X () +7X (3) ) T V ( ˆF )= 4 79 (0X () +3X () +0X (3) 3X () X () 3X () X (3) 7X () X (3) ). To obti the smplig distributio of these estimtors, of course, we would hve to cosider the joit distributio of (X (),X (),X (3) ). Nturlly, the clcultios become eve more di cult s icreses. 0 Altertively, we could use resmplig i order to pproximte T B ( ˆF ) d T V ( ˆF ). This is the bootstrppig ide, d it works like this: For some lrge umber B, simulte B rdom smples from ˆF, mely X,...,X,. X B,...,X B, d pproximte qutity like E ˆF ˆ by the smple me B BX where ˆ i is the smple medi of the ith bootstrp smple X i,...,x i. Notice tht the wek lw of lrge umbers sserts tht B BX ˆ i P ˆ i,! E ˆF ˆ. To recp, the, we wish to estimte some prmeter T (F ) for ukow distributio F bsed o rdom smple from F. We estimte T (F )byt ( ˆF ), but it is ot esy to evlute T ( ˆF ) so we pproximte T ( ˆF ) by resmplig B times from ˆF d obti bootstrp estimtor TB,. Thus, there re two relevt issues:. How good is the pproximtio of T ( ˆF )byt B,? (Note tht T ( ˆF ) is NOT ukow prmeter; it is kow but hrd to evlute.). How precise is the estimtio of T (F )byt ( ˆF )? Questio is usully ddressed usig symptotic rgumet usig the wek lw or the cetrl limit theorem d lettig B!. For exmple, if we hve expecttio fuctiol T (F )=E F h(x,...,x ), the s B!. T B, = B BX h(xi,...,x i)! P T ( ˆF ) Questio, o the other hd, is ofte tricky; symptotic results ivolve lettig!d re hdled cse-by-cse. We will ot discuss these symptotics here. O relted ote, 0

10 however, there is rgumet i Lehm s book (o pges ) bout why plug-i estimtor my be better th symptotic estimtor. Tht is, if it is possible to show T (F )! T s!, the s estimtor of T (F ), T ( ˆF ) my be preferble to T. We coclude this sectio by cosiderig the so-clled prmetric bootstrp. If we ssume tht the ukow distributio fuctio F comes from fmily of distributio fuctios idexed by prmeter µ, the T (F ) is relly T (F µ ). The, isted of the plug-i estimtor T ( ˆF ), we might cosider the estimtor T (Fˆµ ), where ˆµ is estimtor of µ. Everythig proceeds s i the oprmetric versio of bootstrppig. Sice it my ot be esy to evulte T (Fˆµ ) explicitly, we first fid ˆµ d the tke B rdom smples of size, X,...,X through X B,...,X B, from Fˆµ. These smples re used to pproximte T (Fˆµ ). Exmple 0.6 Suppose X,...,X is rdom smple from Poisso(µ). Tke ˆµ = X. Suppose T (F µ ) = Vr Fµ ˆµ. I this cse, we hppe to kow tht T (F µ )= µ/, but let s igore this kowledge d pply prmetric bootstrp. For some lrge B, sy 500, geerte B smples from Poisso(ˆµ) d use the smple vrice of ˆµ s pproximtio to T (Fˆµ ). I R, with µ = d = 0 we obti x <- rpois(0,) # Geerte the smple from F muht <- me(x) muht [] 0.85 muhtstr <- rep(0,500) # Allocte the vector for muhtstr for(i i :500) muhtstr[i] <- me(rpois(0,muht)) vr(muhtstr) [] Note tht the estimte 0.04 is close to the kow true vlue This exmple is simplistic becuse we lredy kow tht T (F )=µ/, which mkes ˆµ/ more turl estimtor. However, it is ot lwys so simple to obti closed-form expressio for T (F ). Icidetlly, we could lso use oprmetric bootstrp pproch i this exmple: for (i i :500) muhtstr[i] <- me(smple(x,replce=t)) vr(muhtstr) [] Of course, 0.04 is pproximtio to T ( ˆF ) rther th T (Fˆµ ). Furthermore, we c obti result rbitrrily close to T ( ˆF ) by icresig the vlue of B: muhtstr_rep(0,00000) for (i i :00000) muhtstr[i] <- me(smple(x,replce=t)) vr(muhtstr) [] I fct, it is i priciple possible to obti pproximte vrice for our estimtes of T ( ˆF ) d T (Fˆµ ), d, usig the cetrl limit theorem, costruct pproximte cofidece itervls for these qutities. This would llow us to specify the qutities to y desired level of ccurcy. Exercises for Sectio 0.4 Exercise 0.4 () Devise oprmetric bootstrp scheme for settig cofidece itervls for i the lier regressio model Y i = + x i + i. There is more th oe possible swer. (b) Usig B = 000, implemet your scheme o the followig dtset to obti 95% cofidece itervl. Compre your swer with the stdrd 95% cofidece itervl. Y x (I the dtset, Y is the umber of mtee deths due to collisios with powerbots i Florid d x is the umber of powerbot registrtios i thousds for eve yers from ) Exercise 0.5 Cosider the followig dtset tht lists the ltitude d me August temperture i degrees Fhreheit for 7 US cities. The residuls re listed for use i prt (b). City Ltitude Temperture Residul Mimi Phoeix Memphis Bltimore Pittsburgh Bosto Portld, OR Miitb gives the followig output for simple lier regressio: 03 04

11 Predictor Coef SE Coef T P Costt ltitude S = R-Sq = 6.5% R-Sq(dj) = 53.8% Note tht this gives symptotic estimte of the vrice of the slope prmeter s.3443 =.85. I () through (c) below, use the described method to simulte B = 500 bootstrp smples (x b,y b ),...,(x b7,y b7 ) for pple b pple B. For ech b, refit the model to obti ˆ b. Report the smple vrice of ˆ,..., ˆ B d compre with the symptotic estimte of.85. () Prmetric bootstrp. Tke x bi = x i for ll b d i. Let y bi = ˆ0+ ˆx i + i, where i N(0, ˆ). Obti ˆ0, ˆ, d ˆ from the bove output. (b) Noprmetric bootstrp I. Tke x bi = x i for ll b d i. Let ybi = ˆ0 + ˆx i + rbi, where r b,...,r b7 is iid smple from the empiricl distributio of the residuls from the origil model (you my wt to refit the origil model to fid these residuls). (c) Noprmetric bootstrp II. Let (x b,y b ),...,(x b7,y b7 ) be iid smple from the empiricl distributio of (x,y ),...,(x 7,y 7 ). Note: I R or Splus, you c obti the slope coe ciet of the lier regressio of the vector y o the vector x usig lm(y~x)$coef[]. p (c) Defie h(t) = t. Prove tht if we tke V i = Ui for ech i, the V i is rdom vrible with desity h(t). Prove tht with we hve ˆ P!. ˆ = g(v i ) h(v i ), (d) For = 000, simulte ˆ d ˆ. Give estimtes of the vrice for ech estimtor by reportig ˆ/ for ech, where ˆ is the smple vrice of the g(u i ) or the g(v i )/h(v i ) s the cse my be. (e) Plot, o the sme set of xes, g(t), h(t), d the stdrd uiform desity for t [0, ]. From this plot, expli why the vrice of ˆ is smller th the vrice of ˆ. [Icidetlly, the techique of drwig rdom vribles from desity h whose shpe is close to the fuctio g of iterest is vrice-reductio techique kow s importce smplig.] Note: This ws sort of silly exmple, sice umericl methods yield exct vlue for. However, with certi high-dimesiol itegrls, the curse of dimesiolity mkes exct umericl methods extremely time-cosumig computtiolly; thus, Mote Crlo itegrtio does hve prcticl use i such cses. Exercise 0.6 The sme resmplig ide tht is exploited i the bootstrp c be used to pproximte the vlue of di cult itegrls by techique sometimes clled Mote Crlo itegrtio. Suppose we wish to compute = Z 0 e x cos 3 (x) dx. () Use umericl itegrtio (e.g., the itegrte fuctio i R d Splus) to verify tht = (b) Let Defie g(t) =e t cos 3 (t). Let U,...,U be iid uiform(0,) smple. ˆ = g(u i ). Prove tht ˆ P!