1 Review and Overview

Transcription

1 CS9T/STATS3: Statstcal Learg Teory Lecturer: Tegyu Ma Lecture #7 Scrbe: Bra Zag October 5, 08 Revew ad Overvew We wll frst gve a bref revew of wat as bee covered so far I te frst few lectures, we stated ad proved asymptotcs o te maxmum lkelood estmator I partcular, te well-specfed case, we sowed tat L(ĥ) L( ) p + o ( ) were ĥ s te emprcal MLE, s te groud trut fucto, p s te umber of parameters te model, s te umber of trag examples, ad L s te expected egatve log lkelood (test error) We te trastoed to o-asymptotcs results, ad smultaeously we dropped te assumpto of well-specfcty We frst observed tat L(ĥ) L( ) ˆL() L() wc allows us to use uform covergece results to establs bouds o te geeralzato error Our atteto tus tured to boudg te rgt ad sde of te above equato I te case of a fte ypotess class H, usg Hoeffdg s equalty ad te a uo boud, we sowed tat, wt probablty at least δ, log H + log(/δ) ˆL() L() I te case of a fte ypotess class H parametrzed by a bouded parameter θ R p, we cocluded tat, wt probablty O ( p 0), p log ˆL() L() as log as te loss fuctos are suffcetly ce (Lpsctz ad bouded) Last week, we defed te oto of Rademacer complexty ad used t to smplfy our dscusso of uform covergece bouds I partcular, we saw tat log(/δ) ˆL() L() R S (F ) + were R S (F ) s te sample Rademacer complexty o a sample S of sze, ad F s a famly of loss fuctos We ote tat R S (F ) ca be replaced wt te expected Rademacer complexty R (F ) = E R S (F ), but tat t s usually easer to reaso about R S (F ) because tere s oe fewer expectato to worry about If we use te γ-marg loss, te R S (F ) R S(H) γ

2 I ts lecture, we wll focus o boudg R S (H), wc by Talagrad s Lemma ad te above dscusso wll lead to bouds o L(ĥ) L( ) We wll start wt te case of a lear fucto wt bouded parameter, ad move to te more geeral settg of a eural etwork wt a sgle dde layer We ote tat focusg o R S (H) stead of R S (F ) carres aoter advatage: t allows us to gore te labels ad look at oly te puts, wc ofte smplfes te aalyss Rademacer Complexty of Lear Models Teorem Let H = { x w T x : w B } Assume tat E x p x C Te R S (H) B x () ad Proof We ave R S (H) = E R (H) BC () w B = E w B w T x w T ( = B E x sce te L -orm s self-dual = B E x ) x by Jese s equalty: E z = E z E z for oegatve z, sce s cocave B E 0 / x + j x T x j were te secod sum vases because te s are depedet wt mea 0 / j = B x Ts proves Eq () Eq () follows from takg a expectato: R (H) = E R S (H) = B E x B E x BC were te frst equalty s aoter applcato of Jese, ad te secod follows by defto of C

3 Teorem Let H = { x w T x : w B } Assume tat x C for all Let d be te dmeso of te put; e x R d for all Te R (H) BC log d We ote frst te smlartes ad dffereces betwee Teorems ad Teorem looks somewat weaker: we ave to make a stroger assumpto, amely, x C for all, stead of E x C, ad te addto of a factor of log d We also remark tat t s possble eve lkely tat Teorems lke ad apply for oter pars of orm-dual orm pars However, sce te most commo orms are L, L, L, we wll restrct our atteto to tese for ts class We wll ow gve a sketc of te proof of Teorem Te full proof s left as a omework problem Proof Sketc We wll prove a slgtly dfferet statemet, amely tat wt probablty d O() over te radom coce of, we ave w B w T x BC log d (3) Ts looks smlar eoug to our teorem tat te teorem souds belevable as log as te tal of te dstrbuto of te LHS s ot too eavy ( ) w T x = w T x = B x w B w B Let v = x We clam tat, wt probablty d O() over te radom coce of, v C log d, from wc (3) follows mmedately To see ts, fx a dex j Te v j = x j Sce te x j are all bouded by C ad te are depedet, Hoeffdg s equalty mples tat ( ( )) ε Pr[ v j ε] exp O C Takg ε = O(C log d), we coclude tat Pr[ v j ε] d O() Now a uo boud over j =,, d gves tat Pr [ v j C log d ] d O(), as desred Notce tat te costat dde by te O() ca be made small by creasg te costat dde te expresso for ε; we wll tus ot cocer ourselves wt fgurg out te costat factors sce t s uecessary 3 Rademacer Complexty of Neural Networks Somewat surprsgly, t as bee emprcally observed [] tat, a eural etwork wt a sgle dde layer ad o regularzato, creasg te umber of dde uts mproves (or, at least, does ot urt) te geeralzato loss eve past te pot we te umber of dde uts suffces to aceve zero trag error A recet paper [] gves some teoretcal bouds tat justfy tese practcal observatos I te ext two lectures, we wll see some of tese bouds Trougout ts secto, we wll use te followg otato: 3

4 Θ = (w, U) are te parameters of te model f Θ (x) = w T φ(ux) s te model fucto m s te umber of dde uts (e te umber of rows of U, ad te umber of elemets of w) φ(x) = max(x, 0) s te (elemet-wse) ReLU fucto {x } = Rd s te trag set Te ma teorem we wll seek to prove as te form Teorem 3 (Teorem 3 [], formally) Suppose we tra a two-layer eural etwork to mmze a logstc loss fucto Te te geeralzato error decreases as te trag error creases Te proof ad precse statemet of ts result wll be deferred to te ext lecture We wll frst warm up by sowg a smpler statemet Teorem 4 (Specal case of Teorem 43, Secto 64 of Percy Lag s otes) Let H = { f Θ : w B ad u B for all } were u s te t colum of U Te R (H) BB C m Proof Let g U (x) φ(ux) for matrces U Te f Θ (x) = w T g U (x) Tus: R S (H) = E Θ = E Θ w T g U (x ) w T ( = E w Θ ) g U (x ) g U (x ) sce we ca always pck w to pot te same drecto as g U (x ) = B E g U: u j B j U (x ) B m E max φ(u T j x ) j u j B by symmetry = B m E u B φ(u T x ) 4

5 ({ Te expectato looks very suspcously lke R S x φ(u T x) : u B }) ; we oly dffer from te defto of Rademacer complexty by a absolute value sg I fact, some sources, cludg te orgal paper o Rademacer complexty [3] defe t wt te absolute value sg A aalogue of Talagrad s Lemma for ts defto s stated Teorem (Secto 3) of [3], wc we lose a extra factor of Usg ts, sce φ s -Lpsctz, we ca cotue R S (H) B m E u T x u B = B m E u T x u B = B ({ mr S x u T x : u B }) R (H) B ({ mr x u T x : u B }) m BB C were te last le follows from Teorem Teorem 4 s weaker ta te teorem we would lke to prove, sce tere s a m te umerator wc we d lke to get rd of We wll refe ts boud te ext lecture Refereces [] Beam Neysabur, Srad Bojaapall, Davd McAllester, ad Nat Srebro Explorg geeralzato deep learg I Advaces Neural Iformato Processg Systems, pages , 07 [] C We, J D Lee, Q Lu, ad T Ma O te Marg Teory of Feedforward Neural Networks ArXv e-prts, October 08 [3] Peter L Bartlett ad Saar Medelso Rademacer ad gaussa complextes: Rsk bouds ad structural results Joural of Mace Learg Researc, 3(Nov):463 48, 00 5