Theorem. Let H be a class of functions from a measurable space T to R. Assume that for every ɛ > 0 there exists a finite set of brackets

Transcription

1 STATISTICAL THEORY SUMMARY ANDREW TULLOCH Theorem Hoeffig s Iequaliy. Suose EX, a a X b. The E e X e b a 8... Uiform Laws of Large Numbers. Bic Coces Defiiio Covergece almos surely. A sequece X, N of raom variables coverges almos surely o a raom variable X if We say ha X X. X X µω Ω X ω Xω. Defiiio Covergece i robabiliy. X X i robabiliy if for all ɛ >, X X > ɛ.. For raom vecors, we efie aalogously wih akig he orm i R. Defiiio Covergece i isribuio. X X or X coverges o X i isribuio if X X.3 wheever X is coiuous. roosiio. Le X, N, X akig values i X R. i X X X X X X.4 ii If X X i ay moe, a if g : X R is coiuous, he gx gxi he same moe. iii Slusky s lemma If X X a Y c a cosa. The i Y c ii X + Y X + c iii X Y cx where Y R. iv X Y c X where Y R, c. iv If A, N are raom marices wih A ij Aij for all i, j a X X, he AX AX, a if A is iverible, A X A X, where A A ij. Theorem Law of Large Numbers. le X,..., X be IID coies of X such ha E X i <, he X i EX.5 i Theorem Ceral limi heorem. Le X,..., X be IID coies of X o R wih VX σ <. The X i EX N, σ.6 i I he mulivariae ce, where X o R wih he covariace of X Σ, he X i EX N, Σ.7 i Theorem Gaussia Tail Iequaliy. If X N,, he X > ɛ e ɛ..8 ɛ Theorem Chebyshev s Iequaliy. Le µ EX, σ VX. The X µ σ.9 Theorem Markov s Iequaliy. Le X be a o-egaive raom variable a suose EX exiss. The for ay >, X > EX. Theorem. Le H be a cls of fucios from a meurable sace T o R. Assume ha for every ɛ > here exiss a fiie se of brackes [l j, u j ], j,..., Nɛ, such ha E l j X <, E u j X <, a E u j X l j X < ɛ for every j. Suose moreover ha for every h H here exiss j wih h [l j, u j ] h {f : T R lx fx ux, x T }. The we have a uiform law of large umbers, su hx h H i EhX. i roof. Le ɛ >, a choose brackes [l j, u j ] such ha E u j l j X < ɛ by hyohesis. The for every ω T ousie a ull se A, here exiss ω, ɛ such ha max u j Eu j < ɛ j,...,n ɛ i. by he srog law of large umbers a akig a uio over all j. The his gives for h H, ousie a se of zero meure, hx i EhX u j Eh.3 i i u j Eu j + Eu j Eh.4 i which is boue above by ɛ + ɛ ɛ, require. 3. Cosisecy of M-esimaors Theorem. Le Θ R be comac. Le Q : Θ R be a coiuous, o-raom fucio ha h a uique miimizer θ Θ. Le Q : Θ R be ay sequece of raom fucios such ha su Q θ Qθ 3.. If θ is ay sequece of miimizers of Q, he ˆθ θ. roof. The key is o cosier he se S ɛ {θ Θ : θ θ ɛ}. This is comac, wih Q coiuous o his se, a so a ifimum Qθ ɛ > Qθ is aaie o his se. Choose δ > so Qθ ɛ δ > Qθ + δ. The cosier he se A ɛ {su Q θ Qθ < δ}. O his se, we have if Q θ if Qθ δ Qθ ɛ δ > Qθ + δ Q θ. 3. S ɛ S ɛ So if ˆθ lay i S ɛ, he Q δ woul be sricly smaller ha Q ˆθ, coraicig ha ˆθ is a miimizer. Coclue ha A ɛ ˆθ θ < ɛ, bu A ɛ, we have ˆθ θ < ɛ. Theorem. Le Θ be comac i R, a le X R a cosier observig X,..., X ii from X o X. Le q : X Θ R ha is coiuous i θ for all x a meurable i x for all θ Θ. Assume E su qx, θ < 3.3 The su qx i, θ EqX, θ 3.4 roof. We seek o fi suiable brackeig fucios a rocee via he uiform law of large umbers. Firs, efie he brackes ux, θ, η su θ B,η, so E ux, θ, η < by sumio. By coiuiy of q, x, he suremum is achieve a ois θ u θ θ < η, a so lim η ux, θ, η qθ, x a every x a every θ, a usig he omiae covergece heorem gives lim η E ux, θ, η qθ, X. The for ɛ > a θ Θ we ca choose ηɛ, θ small so ha EuX, θ, η lx, θ, η < ɛ. 3.5

2 STATISTICAL THEORY SUMMARY The oe balls {Bθ, ηɛ, θ} are a oe cover of he comac se Θ, so here exiss a fiie subcover by Heie-Borel. This fiie subcover u j, θ j, ηɛ, θ j cosiues a brackeig se of he q, a so we aly he uiform law of large umbers. Theorem Cosisecy of he Maximum Likelihoo Esimaor. Cosier he moel fθ, y, θ Θ R, y Y R. Assume fθ, y > for all y Y a all θ Θ, a ha Y fθ, yy for every θ Θ. Assume furher ha Θ is comac a ha he ma θ fθ, y is coiuous o Θ for every y Y. Le Y,..., Y be ii wih commo esiy fθ, where θ Θ. Suose fially ha he ieificaio coiio 3 a he omiaio coiio su log fθ, y fθ, yy < 3.6 Y θ Θ hol. If ˆθ is he MLE i he moel {fθ, θ Θ} be o he samle Y,..., Y, he ˆθ is cosise, i ha ˆθ θ θ. roof. Seig his follows from he revious resuls. qθ, y log fθ, y, 3.7 Qθ E θ qθ, Y, 3.8 Q θ qθ, Y i, 3.9 i Defiiio Uiform Cosisecy. A esimaor T is uiformly cosise i θ Θ, if for every δ >,. su θ T θ > δ 3. Theorem. A esimaor is uiformly cosise if, for every ɛ >, a ha su if if Qθ Qθ > 3. : θ θ ɛ θ su Q θ; Y,..., Y Qθ > δ Asymoic Disribuio Theory Theorem. Cosier he moel fθ, y, θ Θ R, y Y R. Assume fθ, y > for all y Y a all θ Θ, a ha Y fθ, yy for every θ Θ. Le Y,..., Y be ii from esiy fθ, y for some θ Θ. Assume moreover i θ is a ierior oi o Θ. ii There exiss a oe se U saisfyig θ U Θ such ha fθ, y is, for every y Y, wice coiuously iffereiable wih resec o θ o U, iii E log fθ,y θ θθ T is osigular, a E θ log fθ, Y < 4. θ iv There exiss a comac ball K U wih oemy ierior ceere a θ such ha E θ su θ K su Y θ K su Y θ K log fθ, Y θθ T < 4. fθ, y y < 4.3 θ fθ, y y < 4.4 θθt Le ˆθ be he MLE i he moel {fθ, ; θ Θ} be o he samle Y,..., Y, a sume ˆθ θ θ. Defie he Fisher iformaio log fθ, Y log fθ, Y T iθ E θ θ θ 4.5 The iθ E log fθ,y θ θθ T, a ˆθ θ N, i i θ 4.6. roof. Firs, oe ha fθ, yy for all θ J, he θ fθ, yy log fθ,y fθ, yy for every θ i K, so θ E θ log fθ, Y θ. 4.7 Sice ˆθ θ, we have ˆθ is a ierior oi of Θ o eves of robabiliy aroachig oe, so Qˆθ. Alyig he mea value heorem, θ we have Qθ + A ˆθ θ 4.8 θ where  is he marix of seco erivaives of Q evaluae a a mea value θ j o he lie segme bewee θ a ˆθ. For he firs comoe, we have Q θ θ i log fθ, Y i θ N, iθ 4.9 by he ceral limi heorem. log fθ,y For he seco comoe, we show A E, θθ T which we o comoe-wise. We have A kj i h kjθ j, Y i, where h jk is he seco mixe arial erivaive of log fθ, Y i, a we seek o show each h jk E hjk θ, Y. This follows by h jk θ j, Y i E h jk θ, Y 4. i su h θ K jk θ, Y i E h jk θ, Y + E h jk θ j, Y E h jk θ, Y i 4. he by he uiform law of large umbers, he firs erm coverges o zero, a he fac ha θ j θ i robabiliy imlies he seco erm coverges o zero. Hece, A θ Eθ log fθ,y θθ T Σθ. As he limi is iverible we have ha A is iverible o ses wih meure aroachig oe, so we ca rewrie he revious resul ˆθ θ A Q θ N, Σ θ iθ Σ θ. θ 4. from Slusky s lemma. Fially, we show Σθ iθ. This follows from ierchagig iegraio a iffereiaio o show θ T fθ, y fθ, y y θ for all θ i K. The, use he chai rule o show log fθ, y θθ T a usig his ieiy a θ. fθ, y fθ, y θθ T y 4.3 θθt log fθ, y θ T log fθ, y θ 4.4 Theorem. I he framework of he revious heorem wih a for N fixe, le θ θy,..., Y, be ay ubie esimaor of θ ha is, i saisfies E θ θ θ for all θ Θ. The for all θ iθ. V θ θ iθ 4.5 log fθ,y roof. Cauchy-Swarz a E θ. Secifically, leig lθ, Y θ i θ log fθ, Y i, V θ θ Cov θ θ, l θ, Y V θ l θ, Y iθ 4.6 sice Cov θ θ, l θ, Y θyl θ, y fθ, y i y 4.7 i θy θ fθ, yy θ E θ θ θ. 4.8 θ

3 STATISTICAL THEORY SUMMARY 3 Theorem Dela Meho. Le Θ be a oe subse of R a le Φ : Θ R m be iffereiable a θ Θ, wih erivaive DΦ θ. Le r be a iverge sequece of osiive real umbers a le X be raom variables akig values i Θ such ha r X θ X. The. If X N, i θ, he r ΦX Φθ DΦ θ X 4.9 DΦ θ X N, ΣΦ, Θ. 4. Defiiio Likelihoo Raio es saisic. Suose we observe Y,..., Y from fθ,, a cosier he esig roblem H : θ Θ agais H : θ Θ, where Θ Θ R. The Neyma-earso heory suggess o es hese hyohesis by he likelihoo raio es saisic Λ Θ, Θ log su i fθ, Y i su i fθ, Y 4. i which i erms of he maximum likelihoo esimaors ˆθ, ˆθ, of he moels Θ, Θ is Λ Θ, Θ log fˆθ,, Y i log fˆθ, Y i. 4. i Theorem. Cosier a arameric moel fθ, y, θ Θ R ha saisfies he sumios of he heorem o ymoic ormaliy of he MLE. Cosier he simle ull hyohesis Θ {θ }, θ Θ. The uer H, he likelihoo raio es saisic is ymoically chi-square isribue, so uer θ. Λ Θ, Θ χ 4.3 roof. Sice Λ Θ, Θ Q θ Q ˆθ, we ca exa his i a Taylor series arou ˆθ, obaiig Qˆθ T θ ˆθ + θ ˆθ T Qθ θ θθ T θ ˆθ 4.4 for some vecor θ o he lie segme bewee ˆθ a θ. As i he roof of he revious heorem, we show ha A coverges o iθ i robabiliy. Thus, by Slusky s lemma a cosisecy, ˆθ θ T A iθ coverges o zero i isribuio a robabiliy he limi is cosa, so we ca reea he argume a obai ˆθ θ T A iθ ˆθ θ θ 4.5 a so Λ Θ, Θ h he same limi isribuio he raom variable ˆθ θ T iθ ˆθ θ. 4.6 By coiuiy of x x T iθ x, we obai he limiig isribuio is X T iθ X, wih X N, i θ, which is he square Eucliea orm of he MVN N, I, which h a χ isribuio. 4.. Local Asymoic Normaliy a Coiguiy. Defiiio Local Asymoic Normaliy. Cosier a arameric moel fθ fθ,, θ Θ R a le qθ, y log fθ, y. Suose θ qθ, y a he Fisher iformaio iθ exis a he ierior oi θ Θ. We say ha he moel {fθ : θ Θ} is locally ymoically ormal a θ if for every coverge sequece h h a for Y,..., Y ii fθ, we have,, log fθ + h Y i fθ i i h T qθ, Y i θ ht iθ h + o θ. 4.7 We say ha he moel {fθ : θ Θ} is locally ymoically ormal if i is locally ymoically ormal for every θ i Θ. Theorem. Cosier a arameric moel {fθ, θ Θ}, Θ R, ha saisfies he sumios of he heorem o he ymoic ormaliy of he MLE. The {fθ : θ Θ } is locally ymoically ormal for every oe subse Θ of Θ. roof. We rove for h fixe. As before, we ca exa log fθ + h abou log fθ u o seco orer, a obai h T qθ, Y i qθ, Y i θ ht θθ i i T h 4.8 for some vecor θ o he lie segme bewee θ a θ + h. By he uiform law of large umbers, we have. ht i qθ, Y i θθ T h ht iθ h θ 4.9 Defiiio Coiguiy. Le, Q be wo sequeces of robabiliy meures. We say ha Q is coiguous wih resec o if for every sequece of meurable ses A, he hyohesis A imlies Q A, a wrie Q. The sequeces are muually coiguous if boh Q a Q, a wrie Q. Theorem LeCam s Firs Lemma. Le, Q be robabiliy meures o meurable saces Ω, A. The he followig are equivale: i Q. ii If Q Q U alog a subsequece of, he U >. iii If Q V alog a subsequece of, he EV. iv For ay sequece of saisics meurable fucios T : Ω R k, we have T imlies T Q. roof. i iv: follows by akig A { T > ɛ}, so Q A imlies T Q. Coversely, ake T A. i ii Com Theorem. Le, Q be sequeces of robabiliy meures o meurable saces Ω, A such ha Q Q e X where X N σ, σ, for some σ >. The Q. roof. Sice e X >, so Q from ii a sice E e Nµ,σ µ σ, his follows from iii. Theorem. If {fθ : θ Θ} is locally ymoically ormal a if h h R, he he rouc meures a θ+ h θ corresoig o samles X,..., X from esiies fθ + h a fθ, resecively, are muually coiguous. I aricular, if a saisic T Y,..., Y coverges o zero i robabiliy uer θ he i also coverges o zero i - θ+ h robabiliy. roof. This follows from he fac ha he ymoic exasio coverges o N ht iθh, h T iθh uer θ. The we ca aly iv. 4.. Bayesia Iferece. Defiiio. Q T V su B BR B QB is he oal variaio isace o he se of robabiliy meures o he Borel σ-algebra BR of R. Theorem Bersei-vo Mises Theorem. Cosier a arameric moel {fθ, θ Θ}, Θ R, ha saisfies he sumios of he heorem o he ymoic ormaliy of he MLE. Suose he moel amis a uiformly cosise esimaor T. Le X,..., X be ii from esiy fθ, le ˆθ be he MLE be o ha samle, sume he rior meure Π is efie o he Borel ses of R a ha Π ossesses a Lebesgue-esiy π ha is coiuous a osiive i a eighborhoo of θ. The, if Π X,..., X is he oserior isribuio give he samle, we have. Π X,..., X Nˆθ, i θ T V θ High Dimesioal Liear Moels Here, we cosier he moel Y Xθ ɛ, ɛ N, σ I, θ Θ R, σ >, where X is a esig marix, a ɛ is a saar Gaussia oise vecor i R. Throughou, we eoe he resulig Gram marix ˆΣ XT X which is symmeric a osiive semiefiie. Wrie a < b for a Cb for some fixe ieally harmless cosa C >. Theorem. I he ce, he clsical le squares esimaor irouce by Gauss solves he roblem Y Xθ mi θ R 5.

4 STATISTICAL THEORY SUMMARY 4 wih he soluio ˆθ X T X X T Y N, σ X T X where we rely o X havig full colum rak so X T X is iverible. Assumig XT X I, we have E θ Xˆθ θ e E θ ˆθ θ σ ri σ. 5. Defiiio. θ B k {θ R, a mos k ozero eries}. For θ B k, call S {j : θ j } he acive se of θ. 5.. The LASSO. Defiiio The LASSO. The θ θ LASSO arg mi θ R Y Xθ + λ θ. Theorem The LASSO erforms almos well he LS esimaor. Le θ B k be a k-sarse vecor i R wih acive se S. Suose Y Xθ + ɛ where ɛ N, I, a le θ be he LASSO esimaor wih ealizaio arameer λ 4σ + log, σ max ˆΣ jj, 5.3 j,..., a sume he marix X is such ha, for some r >, θ S θ kr θ θ T ˆΣ θ θ 5.4 o a eve of robabiliy a le β. The wih robabiliy a le β ex we have X θ θ + λ θ θ 4λ k kr < log. 5.5 roof. Noe ha by efiiio we have Y X θ + λ θ Y Xθ + λ θ 5.6 Xθ θ + λ θ ɛt X θ θ + λ θ. 5.7 by iserig he moel equaio. Usig he ail bou o he ex heorem, we have o a eve A, ɛt X θ θ λ θ θ. 5.8 a hus combiig wih he above resul obai Xθ θ + λ θ \ θ θ + λ θ. 5.9 Usig θ θ S + θ S c θs θ S θs + θ S c we obai o his eve, oig θs c by efiiio of S Xθ θ + λ θ S c 3λ θ S θ S + λ θ S c 5. usig he revious iequaliies a 4ab a + 4b. Theorem. Le λ λ. The for all >, max j,..., ɛt X j λ ex. 5.6 roof. Noe ha ɛt X are N, ˆΣ isribue. We he have he robabiliy i quesios excees oe mius max j,..., j ɛ T X > σ + log Z > + log e ex log e Coherece Coiios for Desig Marices. The criical coiio is θ S θ kr θ θ T ˆΣ θ θ 5. holig rue wih high robabiliy. Theorem. The heorem o LASSO hols rue wih he crucial coiio 5. relace wih he followig coiio: For S, he acive se of θ B k, k, sume he marix X saisfies, for all θ i a some uiversal cosa r, {θ R : θ S c 3 θ S θ S } 5. θ S θ kr θ θ T θ. 5. Theorem. Le he marix X have eries X ij ii N, a le ˆΣ XT X. Suose mi,. The for every log k N fixe a every < C <, here exiss large eough such ha θ T θ θ B k ex Ck log. roof. For θ, he resul is rivial. Thus, i suffices o bou θ T θ θ B k\{} 5.3 θ T θ θ B k\{} 5.4 θ T su θ T θ 5.5 θ B k, θ from below by ex Ck log. We ca he o his over each k- imesioal subsace R S for each S {,..., } wih S k, he use θ T su θ B k, θ θ T θ 5.6 θ T su S {,...,} θ R, θ θ T θ. 5.7 S The we jus ee a bou of e C+k log e Ck log k a sum over he k k subses. Usig he below resul a akig C + k log is he sufficie. a so Theorem. Uer he coiios of he revious heorem, we have for Xθ θ + λ θ S c 3λ θ S θs some uiversal cosa c >, every S {,..., } such ha S k 5. a every >, hols o he eve. The we have su θt + c k X θ θ + λ θ θ θ R θ T c k e. 5.8 θ S, θ X θ θ + λ θ S θs + λ θ S c roof Norivial!. Noe 5. 4λ θ S θs θ T 5.3 su θ R, θ θ T θ su θ T ˆΣ θ 5.9 S θ R S, θ kr 4λ X θ θ 5.4 By comacess, we ca cover he ui ball BS {θ R S : θ } by a e of ois θ l such ha for every θ BS here exiss l wih X θ θ + 4λ kr 5.5 θ θ l δ. The wih Φ ˆΣ I, we have θ T Φθ θ θ l Φθ θ l + λ l T Φθ l + θ θ l T Φθ l. 5.3 The seco erm is boue by δ su v BS v T Φv. The hir erm is boue by δ su v BS v T Φv. Thus, su θ T Φθ max θ BS l,...,θ θl Φθ l + δ + δ su v T Φv 5.3 v BS

5 STATISTICAL THEORY SUMMARY 5 which gives he bou su θ T Φθ 9 max θ BS l,...,nδ θl Φθ l 5.3 a δ 3. A θ l BS fixe, we have θ l T Φθ l Xθ l i Xθ E l i 5.33 i a he raom variables Xθ l i are IID N, θ l isribue wih variace θ l. Thus for g iiin,, we have 9 max θ l T Φθ l > 8 l,...,n 3 N 3 l N 3 l θ l T Φθ l > 4 θ l + c k + c k + + c k + + c k gi > 4 + c k + + c k i N 3 e e c k e 5.38 where we aly he ex iequaliy wih z + c k, a rely o he coverig umbers of he ui ball i k-imesioal Eucliea sace saisfyig Nδ A δ k for some uiversal A >. Theorem. Le g i,i,..., be ii N,, a se X i g i. The for all a N, X ex roof. For λ < e, we ca comue he MGF of E λg e λ λ ex log λ λ. The akig Taylor exasios, we have [ log λ λ] λ + 3 λ + + k + λk +... λ λ 5.4 a by IID, log E e λx λ λ. The by Markov s iequaliy, X > E e λx λ ex λ λ λ ex, whe akig λ The akig 4 z + z, we obai he require resul. Refereces