18.657: Mathematics of Machine Learning

Transcription

1 8.657: Mathematcs of Mache Learg Lecturer: Phlppe Rgollet Lecture 3 Scrbe: James Hrst Sep. 6, Learg wth a fte dctoary Recall from the ed of last lecture our setup: We are workg wth a fte dctoary H = {h,...,h M }of estmators, adwewouldlketouderstadthescalg ofths problem wth respect to M ad the sample sze. Gve H, oe dea s to smply try to mmze the emprcal rsk based o the samples, ad so we defethe emprcal rsk mmzer, ˆh erm, by hˆ erm argmrˆ (h). h H I what follows, we wll smply wrte hˆ stead of hˆ erm whe possble. Also recall the defto of the oracle, h, whch (somehow) mmzes the true rsk ad s defed by h argmr(h). h H The followg theorem shows that, although hˆ caot hope to do better tha h geeral, the dfferece should ot be too large as log as the sample sze s ot too small compared to M. Theorem: The estmator hˆ satsfes R(h) ˆ R(h)+ 2log(2M/δ) wth probablty at least δ. I expectato, t holds that ˆ 2log(2M) IER(h) R(h)+. Proof. From the defto of h, ˆ we have Rˆ (h) ˆ Rˆ (h), whch gves ˆ ˆ ˆ ˆ ˆ R(h) R(h)+R (h) R(h)+R(h) R (h). The oly term here that we eed to cotrol s the secod oe, but sce we do t have ay real formato about h, we wll boud t by a maxmum over H ad the apply Hoeffdg: ˆ ˆ ˆ ˆ ˆ log(2m/δ) R (h) R(h)+R(h) R (h) 2max R (h ) R(h ) 2 2 wth probablty at least δ, whch completes the frst part of the proof.

2 To obta the boud expectato, we start wth a stadard trck from probablty whch bouds a max by ts sum a slghtly more clever way. Here, let {Z } be cetered radom varables, the ( ) IE max Z = logexp sie max Z log IE exp s s ( smax Z ), where the last equalty comes from applyg Jese s equalty to the covex fucto exp( ). Now we boud the max by a sum to get 2M ) log ( ( s 2 )) log(2m s IEexp(sZ ) log 2M exp = +, s s 8 s 8 = where we used Z = Rˆ (h ) R(h ) our case ad the appled Hoeffdg s Lemma. Balacg terms by mmzg over s, ths gves s = 2 2log(2M) ad pluggg produces ˆ log(2m) IE max R (h ) R(h ), 2 whch fshes the proof. 2. CONCENTRATION INEQUALITIES Cocetrato equaltes are results that allow us to boud the devatos of a fucto of radom varables from ts average. The frst of these we wll cosder s a drect mprovemet to Hoeffdg s Iequalty that allows some depedece betwee the radom varables. 2. Azuma-Hoeffdg Iequalty Gve a fltrato {F } of our uderlyg space X, recall that { } are called martgale dffereces f, for every, t holds that F ad IE F = 0. The followg theorem gves a very useful cocetrato boud for averages of bouded martgale dffereces. Theorem (Azuma-Hoeffdg): Suppose that { } are marggale dffereces wth respect to the fltrato {F }, ad let A,B F satsfy A B almost surely for every. The IP ( 2 2 t 2 ) > t exp = B A 2. I comparso to Hoeffdg s equalty, Azuma-Hoeffdg affords ot oly the use of o-uform boudedess, but addtoally requres o depedece of the radom varables. Proof. We start wth a typcal Cheroff boud. IP > t IEe s e st = IE IE e s F e st 2

3 = IE 2 2 e s IEe s F e st IEe s e s (B A) /8 e st, where we have used the fact that the, <, are all F measureable, ad the appled Hoeffdg s lemma o the er expectato. Iteratvely solatg each lke ths ad applyg Hoeffdg s lemma, we get ( ) s 2 B IP > t exp A 2 e st. 8 = Optmzg over s as usual the gves the result. 2.2 Bouded Dffereces Iequalty Although Azuma-Hoeffdg s a powerful result, ts full geeralty s ofte wasted ad ca be cumbersome to apply to a gve problem. Fortuately, there s a atural choce of the {F } ad { }, gvg a smlarly strog result whch ca be much easer to apply. Before we get to ths, we eed oe defto. Defto (Bouded Dffereces Codto): Let g : X IR ad costats c be gve. The g s sad to satsfy the bouded dffereces codto (wth costats c ) f for every. sup g(x,...,x ) g(x,...,x,...,x ) c x,...,x,x Itutvely, g satsfes the bouded dffereces codto f chagg oly oe coordate of g at a tme caot make the value of g devate too far. It should ot be too surprsg that these types of fuctos thus cocetrate somewhat strogly aroud ther average, ad ths tuto s made precse by the followg theorem. Theorem (Bouded Dffereces Iequalty): If g : X IR satsfes the bouded dffereces codto, the ( 2t 2 ) IP g(x,...,x ) IEg(X,...,X ) > t 2exp. c2 Proof. Let {F } be gve by F = σ(x,...,x ), ad defe the martgale dffereces { } by = IEg(X,...,X ) F IEg(X,...,X ) F. The IP > t = IP g(x,...,x ) IEg(X,...,X ) > t, exactly the quatty we wat to boud. Now, ote that IE supg(x,...,x,...,x ) F IEg(X,...,X ) F x 3

4 Smlarly, = IE supg(x,...,x,...,x ) g(x,...,x ) F =: B. x IE fg(x,...,x,...,x ) g(x,...,x ) F =: A. x At ths pot, our assumpto o g mples that B A c for every, ad sce A B wth A,B F, a applcato of Azuma-Hoeffdg gves the result. 2.3 Berste s Iequalty Hoeffdg s equalty s certaly a powerful cocetrato equalty for how lttle t assumes about the radom varables. However, oe of the maor lmtatos of Hoeffdg s ust ths: Sce t oly assumes boudedess of the radom varables, t s completely oblvous to ther actual varaces. Whe the radom varables questo have some kow varace, a deal cocetrato equalty should capture the dea that varace cotrols cocetrato to some degree. Berste s equalty does exactly ths. Theorem (Berste s Iequalty): Let X,...,X be depedet, cetered radom varables wth X c for every, ad wrte σ 2 = Var(X ) for the average varace. The IP ( ) t 2 X > t exp 2σ tc 3 Here, oe should thk of t as beg fxed ad relatvely small compared to, so that stregth of the equalty deed depeds mostly o ad /σ 2. Proof. The dea of the proof s to do a Cheroff boud as usual, but to frst use our assumptos o the varace to obta a slghtly better boud o the momet geeratg fuctos. To ths ed, we expad (s k X ) s k c k 2 IEe sx = +IEsX +IE +Var(X ), k! k! k=2 k=2 where we have used IEX k IEX2 X k 2 Var(X k 2 )c. Rewrtg the sum as a expoetal, we get e sc sx 2 sc IEe s Var(X )g(s), g(s) :=. c 2 s 2 The Cheroff boud ow gves IP ( X > t exp fs 2 ( ) Var(X ))g(s) st ( = exp s>0 s>0 ) fs 2 σ 2 g(s) st, ad optmzg ths over s (a fu calculus exercse) gves exactly the desred result. 4

5 3. NOISE CONDITIONS AND FAST RATES To measure the effectveess of the estmator h, ˆ we would lke to obta a upper boud o the excess rsk E(h) ˆ = R(h) R(h ˆ ). It should be clear, however, that ths must deped sgfcatly o the amout of ose that we allow. I partcular, f η(x) s detcally equal to /2, the we should ot expect to be able to say aythg meagful about E(h) ˆ geeral. Uderstadg ths trade-off betwee ose ad rates wll be the ma subect of ths chapter. 3. The Noseless Case A atural (albet somewhat aïve) case to exame s the completely oseless case. Here, we wll have η(x) {0,} everywhere, Var(Y X) = 0, ad E(h) = R(h) R(h ) = IE 2η(X) I(h(X) = h (X)) = IPh(X) = h (X). Let us ow deote Z = I(h(X ) = Y ) I(h(X ˆ ) = Y ), ad wrte Z = Z IEZ. The otce that we have Z = I(h(X ˆ ) = h(x )), ad Var(Z ) IEZ 2 ˆ = IPh(X ) = h(x ). For ay classfer h H, we ca smlarly defez (h ) (by replacg ˆhwth h throughout). The, to set up a applcato of Berste s equalty, we ca compute Var(Z (h )) IPh (X ) = h(x ) =: σ 2. = At ths pot, we wll make a (farly strog) assumpto about our dctoary H, whch s that h H, whch further mples that h = h. Sce the radom varables Z compare to h, ths wll allow us to use them to boud E(h), ˆ whch rather compares to h. Now, applyg Berste (wth c = 2) to the {Z (h )} for every gves ( ) t 2 δ IP Z (h ) > t exp = 2σ 2 = + 4 :, 3t M ad a smple computato here shows that t s eough to take t max 2σ 2log(M/δ) 4, log(m/δ) =: t 0 () 3 for ths to hold. From here, we may use the assumpto h = h to coclude that IP E(h) ˆ > t 0 (ĵ) δ, ˆ hˆ = h. 5

6 However, we also kow that σ ˆ 2ˆ E(h), whch mples that E(h) ˆ max 2E(h) ˆ log(m/δ) 4, log(m/δ) 3 ˆ wth probablty δ, ad solvg for E(h) gves the mproved rate ˆ log(m/δ) E(h) 2. 6

7 MIT OpeCourseWare Mathematcs of Mache Learg Fall 205 For formato about ctg these materals or our Terms of Use, vst: