10/ Statistical Machine Learning Homework #1 Solutions

Transcription

1 Caregie Mello Uiversity Departet of Statistics & Data Sciece 0/36-70 Statistical Macie Learig Hoework # Solutios Proble [40 pts.] DUE: February, 08 Let X,..., X P were X i [0, ] ad P as desity p. Let p be te istogra estiator usig bis. Let /. Recall tat te L error is ( p(x) p(x)) p (x)dx p(x)p(x)dx + p (x)dx. As usual, we ay igore te last ter so we defie te loss to be L() p (x)dx p(x)p(x)dx. (a) Suppose we used te direct estiator of te loss, aely, we replace te itegral wit te average to get L() p (x)dx p(x i ). i Sow tat tis fails i te sese tat it is iiized by takig 0. (b) Recall tat te leave-oe-out estiator of te risk is L() p (x)dx i p (i) (X i ), Sow tat L() were Z j is te uber of observatios i bi j. (c) Sow tat L() L() P 0. ( ) + ( ) j Z j Solutio. Defie for j,...,. θ j i (X i B j ) ad Z j θ j

2 0/36-70 Statistical Macie Learig: Hoework (a) (5 pts.) L() p (x)dx i p(x i ) θ j (x B j) dx k i θ j θk (x B j B k ) dx θ j (x B j ) dx θ j (x B j )dx θ j θ j θ j Zj θ j θ j θ j (X i B j ) θ j i (X i B j ) Z j is bouded below by, so L() as 0. Terefore, tis loss is iiized by takig 0. (b) (0 pts.) Fro part (a) we ave p (x)dx θ j. () Ad te secod ter i te leave-oe-out loss is i p ( i) (X i ) ( ) ( ) ( ) i (X i B j ) (X k B j ) k i (X i B j )( θ j (X i B j )) i ( θ j θ j ). ()

3 0/36-70 Statistical Macie Learig: Hoework Takig te differece of () ad (), we get L() θ j ( ) ( ) θ j + ( ) + ( ) θ j ( ) + ( ) ( θ j θ j ) ( ) θ j Z j. (c) (5 pts.) It suffices to sow i p ( i) (X i ) p(x)p(x)dx P 0. (3) Caveat: It is ot a valid lie of reasoig to directly say (3) olds by te weak law of large ubers sice te p ( i) are fuctios of te saple size. p(x)p(x)dx Bj θ j (x B j)p(x)dx θ j p(x)dx θ j p(x)dx B j θ j P (X B j) θ j θ j θ j by Fubii/Toelli teore Ad followig fro () we ave i p ( i) (X i ) ( ) ( ) ( θ j θ j ) θ j ( ) θ j θ j + ( ) θ j + O( ). θ j ( ) 3

4 0/36-70 Statistical Macie Learig: Hoework Terefore, i p ( i) (X i ) p(x)p(x)dx θ j P 0 θ j θ j + O( ) θ j ( θ j θ j ) + O( ) sice θ j by te weak law of large ubers. i (X i B j ) P P (X B j ) θ j 4

5 0/36-70 Statistical Macie Learig: Hoework Proble Let p be te kerel desity estiator (i oe diesio) wit badwidt. Let s (x) Var( p (x)). (a) Sow tat were p (x) E[ p (x)]. p (x) p(x) s (x) N(0, ) Hit: Recall tat te Lyapuov cetral liit teore says te followig: Suppose tat Y, Y,... are idepedet. Let µ i E[Y i ] ad σ i Var(Y i). Let s i σ i. If li s +δ i for soe δ > 0. Te s i (Y i µ u ) N(0, ). E[ Y i µ i +δ ] 0 (b) Assue tat te sootess is β. Suppose tat te badwidt is cose optially. Sow tat p (x) p(x) N(b(x), ) s (x) for soe costat b(x) wic is, i geeral, ot 0. Solutio. (a) Caveat: Te classical Cetral Liit Teore caot be applied ere, as is a fuctio of ad tus te K( x X i ) are ot idetically distributed. However, as te it suggests, te Lyapuov CLT still olds for o-idetically distributed rado variables. Clai. Let p >. Te p E K x X i p (x) Θ p. Proof. See appedix Now E K x X i p (x) +δ +δ +δ E K x X i p (x) Θ +δ +δ, 5

6 0/36-70 Statistical Macie Learig: Hoework ad Terefore, s +δ i E K s i E K E K Θ. x X i p (x) x X i p (x) x X i p (x) +δ Θ(() + δ ) Θ +δ +δ Θ(() δ ) 0, as ad, for ay δ > 0. So, by te Lyapuov CLT, p (x) p (x) s (x) N(0, ). (b) First ote p (x) p(x) s (x) p (x) p (x) s (x) p (x) p (x) s (x) + p (x) p(x) s (x) + Bias(p (x)) Var( p (x)). Fro Teore 5, te optial badwidt is Θ( /5 ). Now fro part (a), we ave ad fro Lea 3, Terefore, p (x) p(x) s (x) Var( p (x)) Θ Bias(p (x)) O( ). p (x) p (x) s (x) p (x) p (x) s (x) p (x) p (x) s (x) + Bias(p (x)) Var( p (x)) O( ) + Θ( () / ) + O( /5 ) Θ( /5 ) p (x) p (x) + O() s (x) N(0,) N(b(x), ). 6

7 0/36-70 Statistical Macie Learig: Hoework Proble 3 Let X,..., X P were X i [0, ]. Assue tat P as desity p wic as bouded cotiuous derivative. Let p (x) be te kerel desity estiator. Sow tat E[ p (0)] p(0) + O(). Solutio. E[ p(0)] E E K 0 0 i K X i X i 0 K(u )p(u)du / / p(0) p(0) K(t)p(t)dt K(t) p(0) + t +p(0) + t 0 / + O(), K(t)dt + O() 0 / +p(0) + o( ) dt / tk(t)dt + O( ) t K(t)dt 0 σk /< let t u were we assued K( ) is supported o [, ],, ad / 0 tk(t)dt is bouded. 7

8 0/36-70 Statistical Macie Learig: Hoework Proble 4 Let p be a desity o te real lie. Assue tat p is -ties cotiuously differetiable ad tat p () <. Let K be a iger order kerel. Tis eas tat K(y)dy, y j K(y)dy 0 for j, y K(y)dy < ad K (y)dy <. Sow tat te kerel estiator wit badwidt satisfies E ( p(x) p(x)) dx C + for soe C > 0. Wat is te optial badwidt ad wat is te correspodig rate of covergece (usig tis badwidt)? Solutio. See Capter 6 of []. 8

9 0/36-70 Statistical Macie Learig: Hoework Proble 5 [0 pts.] Let X,..., X P were X i [0, ] ad P as desity p. Let φ, φ,... be a ortooral basis for L [0, ]. Hece 0 φ j (x)dx for all j ad 0 φ j(x)φ k (x)dx 0 for j k. Assue tat te basis is uiforly bouded, i.e. sup j sup 0 x φ j (x) C <. We ay expad p as p(x) β jφ j (x) were β j φ j (x)p(x)dx. Defie k p(x) β j φ j (x) were β j (/) i φ j(x i ). (a) Sow tat te risk is bouded by for soe costat c > 0. ck + jk+ β j (b) Defie te Sobolev ellipsoid E(, L) of order as te set of desities of te for p(x) β jφ j (x) were β j j < L. Sow tat te risk for ay desity i E(, L) is bouded by c[(k/)+(/k) ]. Usig tis boud, fid te optial value of k ad fid te correspodig risk. Solutio. (a) (0 pts.) First ote, E[ β j ] E φ j (X i ) i E[φ j (x)] p(x)φ j(x)dx 0 β j. 9

10 0/36-70 Statistical Macie Learig: Hoework So β j is ubiased. Now, R( p(x)) E ( p(x) p(x)) dx E k β j φ j (x) β j φ j (x) dx E k ( β j β j )φ j (x) β j φ j (x) dx jk+ k E ( β j β j ) + β j jk+ k Var( β j ) + k Var(φ j(x i )) + C k + βj. jk+ βj jk+ βj jk+ sice φ i φ j δ ij (b) (0 pts.) sup R( p(x)) C k p E(,L) + C k C k C k βj jk+ + k jk+ β j k + jk+ β j j k + L k ax{c, L } k + k fro part (a) 0

11 0/36-70 Statistical Macie Learig: Hoework Proble 6 [40 pts.] Recall tat te total variatio distace betwee two distributios P ad Q is TV(P, Q) sup A P (A) Q(A). I soe sese, tis would be te ideal loss fuctio to use for desity estiatio. We oly use L because it is easier to deal wit. Here you will explore soe properties of TV. (a) Suppose tat P ad Q ave desities p ad q. Sow tat TV(P, Q) (/) p(x) q(x) dx. (b) Let T be ay appig. Let X ad Y be rado variables. Te sup A P (T (X) A) P (T (Y ) A) sup P (X A) P (Y A). A (c) Let K be a kerel. Recall tat te covolutio of a desity p wit K is (p K)(x) p(z)k(x z)dz. Sow tat Hece, sootig reduces L distace. p K q K K p q. Solutio. (a) (5 pts.) For ay easurable B R, p q p(x) q(x) dx (p(x) q(x))dx + B (q(x) p(x))dx R/B B p(x)dx B q(x)dx + R/B q(x)dx R/B p(x)dx B p(x)dx B q(x)dx + ( B q(x)dx) ( B p(x)dx) ( B p(x)dx B q(x)dx) P (B) Q(B) p q P (B) Q(B) for ay easurable B R. By otig, parallel reasoig sows p q q p, So togeter we ave, p q Q(B) P (B) for ay easurable B R. p q P (B) Q(B)

12 0/36-70 Statistical Macie Learig: Hoework ad tus for ay easurable B R. Now cosider te set B is easurable ad p q p(x) q(x) dx p q sup P (B) Q(B), (4) B R B {x R p(x) > q(x)}. B (p(x) q(x))dx + p(x))dx R/B (q(x) B p(x)dx B q(x)dx + R/B q(x)dx R/B p(x)dx B p(x)dx B q(x)dx + ( B q(x)dx) ( B p(x)dx) ( B p(x)dx B q(x)dx) P (B ) Q(B ). P (B ) Q(B ). We ave foud a set B R suc tat terefore, Cobiig (4) ad (5), we ave p q P (B ) Q(B ), p q sup P (B) Q(B). (5) B R T V (P, Q) p q. (b) (0 pts.) Let F be te σ-field geerated by te sets A o te saple space Ω, ad C T (F) {T (A) A F}. Defie T (C) {ω Ω T (ω) C}, i.e. te pre-iage appig. By defiitio, Te, T (C) {T (C) C C} F. sup P (T (X) C) P (T (Y ) C) sup P (X A) P (Y A) C C A T (C) sup P (X A) P (Y A). A F

13 0/36-70 Statistical Macie Learig: Hoework (c) (5 pts.) p K q K p(z)k(x z)dz q(z)k(x z)dz dx (p(z) q(z))k(x z)dz dx p(z) q(z) K(x z) dzdx p(z) q(z) K(x z) dxdz Fubii s teore ( p(z) q(z) K(x z) dx)dz ( p(z) q(z) K(x) dx)dz ivariat to traslatio K(x) dx p(z) q(z) dz K p q 3

14 0/36-70 Statistical Macie Learig: Hoework Refereces [] L. Wassera, All of Noparaetric Statistics, Spriger Texts i Statistics, 006. Appedix Proof of Clai. Fro p a p b p a b p p a p + p b p, we ave Te, p E[ Z i p ] p (x) p E[ Z i p (x) p ] p E[ Z i p ] + p p (x) p. E[ Z i p ] p K p x u p(u)du p K p ( v )p(x + v)dv. So as 0, coose ay [a, b] suc tat K p ( v ) > 0 for soe v [a, b], te K p ( v )p(x+v)dv b a K p ( v )p(x + v)dv b a K p ( v )p(x)dv > 0 by te Bouded Covergece Teore. Also, K p ( v )p(x + v)dv K p ( v ) sup x p(x)dv <, ece K p ( v )p(x + v)dv Θ(), ad accordigly, E[ Z i p ] Θ p. Te Hece p (x) E[Z i ] E[ Z i ] O(). Θ p p E[ Z i p ] p (x) p E[ Z i p (x) p ] p E[ Z i p ] + p p (x) p Θ p wic iplies E[ Z i p (x) p ] Θ p. 4