1 Proving the Fundamental Theorem of Statistical Learning

Transcription

1 THEORETICAL MACHINE LEARNING COS 5 LECTURE #7 APRIL 5, 6 LECTURER: ELAD HAZAN NAME: FERMI MA ANDDANIEL SUO oving te Fundaental Teore of Statistical Learning In tis section, we prove te following: Teore. Fundaental Teore of Statistical Learning). Te ypotesis class H is learnable if and only if te VC-diension of H, denoted d, is less tan. If H is learnable, ten te saple coplexity is given by, ) d log. We refer to te Boolean apping proble i.e., learn soe concept C : X! {, }) trougout te proof. Recall te following definitions fro previous lectures: Definition. VC-diension). Define a ypotesis class H as a class of functions fro a doain X to {, } and C = {c,...,c } X. We say tat te restriction of H to C, H C, is te set of functions fro C to {, } we can derive fro H. In oter words, H C = {c ),...,c )) : H} or te set of vectors if we evaluate eac of H on eac eleent in C. If H C is te set of all functions fro C to {, }, we say tat H satters C. In a previous lecture, we sowed tat tis equivalent to H C = C i.e., ave a ypotesis for all possible configurations of C, were eac c i {, }. Te VC-diension of a ypotesis class H is te largest set C X tat can be sattered by H. If H can satter any size C, we say tat te VC-diension of H is infinity. Definition.3 Growt function H )). Define a function H ) :N! N for ypotesis class H as follows: H ) = ax H C C X : C = Te growt function H ) counts te nuber of different appings C! {, } we can generate if we restrict H to C if C =. Notice tat if VC-DIMENSIONH) =d, ten for all less tan d, we ave H ) = because H satters all C X of size less tan or equal to d. Sauer s lea, wic we proved in te previous lecture, sows tat if is greater tan d, H ) =O d ).. Overall strategy. We already proved one direction of Teore. via te No-Free-Lunc teore: if te VC-diension of our ypotesis class H is infinite, ten H is not learnable. In oter words, if H is learnable, ten VC-DIMENSIONH) is finite.. To sow tat a ypotesis class H wit finite VC-diension is learnable, we first proved Sauer s Lea. Ten, we use te two-saple trick because we cannot take infinitely-sized saples) to sow

2 tat te error on one saple cannot be substantially different fro te error on te oter, rater tan rely on generalization arguents. We will ake use of te following two events: a) Let A be te event tat given a saple S D, tere exists soe H suc tat te error on te saple err S ) is and te generalization error err) is greater tan soe >. In oter words: [A] = [9 H s.t. err S ) =^ err) > S D ] We will prove tat [A] apple H )e. b) Let B be te event tat given two saples, S, S D, tere exists soe H suc tat te error on te first saple, S is, and te error on te second saple S is / for soe >. In oter words: [B] = 9 H s.t. err S ) =^err S ) > S, S D i Clai.4. [A] apple [B]. oof: By te law of total probability, we can write [B] =[B A][A]+[B A][ A] [B A][A] To prove te clai, it is sufficient to sow [B A] /. Let S = {x,...,x } D. Using te ypotesis H defined for event A, we know tat err) > and err S ) =by definition. Ten, wit loss, xi )= z i = o/w and Y = X z i = err S ) i= # X EY z i [err S )] by definition of saple error i= = err) > we see tat [B A] apple [ Y EY > ] because in order for Y EY >, Y ust deviate fro its ean, te generalization error, by ore tan /. Using te Cernoff bound, we see tat Y EY > i apple e n for any we igt coose note tat O/)), tus copleting te proof of Clai.4. Continuing our proof tat [A] apple H )e, we now seek to sow [B] apple H )e /. We eploy te following syetry arguent tat states tat te probability of two saples S and S being so different tat soe ypotesis perfors uc better on one versus te oter is sall. Construct two new sets T and T by randoly partitioning S [ S into equal sets note, tis proof still works if S \ S 6= ;, but need to use ultisets; in practice, owever, S and S are nearly always disjoint because

3 te doain is very large). Now, define a distribution T over coices of T and T and B T as te event B, but wit T and T instead of S and S : [B T ] = 9 H s.t. err T ) =^err T ) > T,T D i apple We clai tat S,S [B] S,S T,T [B T S, S ]. We take tis detour because it is uc easier to analyze T,T [B T S, S ] i.e., wat is te probability tat one set as all errors and te oter as none). T,T [B T S, S ]= 9 H s.t. err T ) =^err T ) > T,T S, S D i apple H S[S ax apple H ) ax err T ) =^err T ) > i T,T err T ) =^ err T ) > i T,T by union bound If we let k be te nuber of errors tat akes on S [ S and k< errors to go around! In tis case,, ten tere siply aren t enoug T,T [err T ) =^ err T ) > / S, S ]= If k, ten tis probability is bounded above by k because all errors ust land in T so tat err T ) = probability of k balls landing in te first of bins). Collecting tese results gives us [A] apple [B] apple E[ H ) k ],k Fro tis last expression, we ave H )e /, wic we set to be less tan for = log H) log. If we use te fact tat log H ) d log as given in Sauer s Lea), ten we ave = d log H) log d = log d A less naive approac can reove te d fro witin te log. Tis final step copletes te proof of Teore.. Metods for bounding generalization error So far, we ave learned about two ways to bound generalization error. tird. Today, we will learn about a 3

4 . VC-diension: te topic of discussion for te last two lectures. Tis te ore general of te two approaces we ave seen so far.. OnlineBatc: tis approac is less general because it requires a convex structure for te proble, but typically uc ore efficient. 3. Radeacer coplexity: we will introduce tis etod today. Note tat te coputation of Radeacer coplexity is NP-ard for soe ypotesis classes. Definition. Radeacer variables). Let be a vector wose eleents are cosen independently and uniforly fro {, +}. Tat is, wit probability / a given eleent is eiter or. Definition. Epirical Radeacer coplexity). Given a saple S = {x,...,x } cosen fro D, define te epirical Radeacer coplexity ˆR S H) as # X ˆR S H) =E ilossx i )) Definition.3 Radeacer coplexity). For soe, let te Radeacer coplexity of H be te expectation of te epirical Radeacer over all saples S of size drawn fro soe distribution D. i R H) ˆRS H) S D To give soe intuition, we consider te value of ˆR S H) wen H satters S. Because we ave H tat can generate any apping of S to {, +}, select te one tat axiizes te su i.e., wen i is, wen i is ). Tis way, regardless of wat we select for, we ave inside te expectation. Tis easure captures te diensionality of a ypotesis class very well because in a way, it is proportional to te VC-diension. We can sow tat R H) is sort of bounded by te VC-diension or H)/ it could be uc saller since te ajority of te probability ass igt not be over te sattered set). Teore.4. Wit probability at least, we ave for all and for all H, r log/ ) err) apple err S )+R H)+3 Tis relation olds for agnostic learning as well since we do not assue realizability. oof: First, define a function S) = {err) err S )}. If S and S differ on only one saple, x i S and x i S, ten i= S) S ) = {err) err S )} {err) err S )} by definition apple {err S ) err S )} sub-additivity of reu lossx = i )) lossx i )) only one saple different = By McDiarid s inequality, for any >, wit probability at least, we get te following bounds: s S) E[ S)] < log 4

5 Terefore, to prove te teore, we need only sow tat E[ S)] apple R H). E[ S)] [{err) err S )}] by definition S apple E [err S S S ) err S)] expectation of i.i.d. saple error apple apple E {err S,S S ) err S )} Jensen s and convexity of reu S,S S,S, apple E S, =E S, E S, X loss, x i ) loss, x i ) )# by definition i= X i= X i= X =R H) X i= i= i loss, x i ) loss, x i ) )# doesn t cange E i loss, x i ) )# i [loss, x i )] i [loss, x i )] )# )# i and sub-additivity of reu i distributed sae way Using te Massart s Lea, we ave R H) apple r log H ) Plugging tis into Teore.4, we get te desired bound. However, tis is a uc stronger clai because Radeacer coplexity is a clai about averages wereas VC-diension is a clai about worst cases. 5