1 Definition of Rademacher Complexity

Transcription

1 COS 511: Theoretcal Machne Learnng Lecturer: Rob Schapre Lecture #9 Scrbe: Josh Chen March 5, 2013 We ve spent the past few classes provng bounds on the generalzaton error of PAClearnng algorths for the cases of consstent and nconsstent hypotheses selected fro fnte and nfnte hypothess spaces. In partcular, last te, we proved bounds for the case of nconsstent hypotheses selected fro nfnte hypothess spaces. However, recall that each te we encountered the proble of an nfnte hypothess space, we had to resort to technques lke usng ghost saples or the VC-denson of a concept class. In ths lecture, we ntroduce a ore odern and elegant approach, usng a concept called Radeacher coplexty. Ths approach turns out to nclude each of the bounds we ve proved n the past few lectures as specal cases. 1 Defnton of Radeacher Coplexty 1.1 Soe usual defntons Before gettng nto the defnton of Radeacher coplexty, we rend ourselves of the usual setup: Let the saple S = ((x 1, y 1 ),..., (x, y )) where, unlke before, y = 1, +1} Let the hypothess h : X 1, +1} To easure how well h fts S, let the tranng error err(h) ˆ = 1 =1 1 h(x ) y Note that, snce we are usng y = 1, +1} nstead of y = 0, 1} as n prevous lectures (for splcty), we can provde an alternatve defnton of tranng error: err(h) ˆ = 1 1h(x ) y } (1) =1 1 f (h(x ), y ) = (1, 1) or ( 1, 1) = 1 0 f (h(x =1 ), y ) = (1, 1) or ( 1, 1) (2) = 1 1 y h(x ) 2 =1 (3) = y h(x ) 2 (4) =1 The ter 1 =1 y h(x ) can be nterpreted as the correlaton of the predctons h(x ) wth the labels y. We see that correlaton s related to tranng error as correlaton = 1 2err(h). ˆ To fnd a hypothess h that nzes tranng error, we can thus equvalently seek to fnd the h satsfyng: 1 arg ax y h(x ) (5) h H

2 1.2 Playng wth correlaton Iagne, now, an experent where we replace a saple s true labels y wth the Radeacher rando varables σ : +1 wth prob. 1/2 σ = (6) 1 wth prob. 1/2 Ths gves a odfed expresson for correlaton: arg ax h H 1 σ h(x ) (7) Instead of selectng the hypothess n H that correlates best wth the labels, ths now selects the hypothess h n H that correlates best wth the rando nose varables σ. Snce h s dependent on the rando varables σ, however, to easure how well H can correlate wth rando nose, we take the expectaton of ths correlaton over the rando varables σ and fnd: E σ [ax h H 1 σ h(x )] (8) Ths ntutvely easures the expressveness of H. We can bound ths expresson usng two extree cases: H = 1 where we only have one choce for a hypothess, and H = 2 where H shatters S. In the frst case, our expectaton equals 0 snce the ax ter dsappears; n the second case our expectaton equals 1 snce there always exsts a hypothess atchng any set of σ s. Thus our easure, as defned above, ust fall between 0 and Generalzng correlaton Instead of workng wth hypotheses h : X 1, +1}, let s generalze our class of functons to the set of all real-valued functons. Replace H wth F, whch we defne to be any faly of functons f : Z R. Now, gven saple S = (z 1,..., z ) wth z Z, f we apply our expresson fro above to F, we arrve at the eprcal Radeacher coplexty of a faly of functons F wth respect to a saple S: 1 1 ˆR S (F) := E σ [sup σ f(z )] (9) Agan, ths expresson easures how well, on average, the functon class F correlates wth rando nose over the saple S. However, often we want to easure the correlaton of F wth respect to a dstrbuton D over X, rather than wth respect to a saple S over X. To fnd ths, we take the expectaton of ˆR S (F) over all saples of sze drawn accordng to D: R (F) := E[ ˆR S (F)] (10) Ths s the Radeacher coplexty, or for clarty, the expected Radeacher coplexty, of F. We now have the defntons we need, and are fnally ready to present our frst generalzaton bounds based on Radeacher coplexty. 1 Note: Snce F can be the faly of all real-valued functons, ax ay not exst. Thus we use sup nstead, whch s defned as the least upper bound on the eleents n a set. For exaple, the sup of the set.9,.99,.999,...} s 1. 2

3 2 Generalzaton bounds based on Radeacher coplexty 2.1 Bounds for general functon classes F The followng theore wll serve as a very general tool for provng unfor convergence bounds va the concept of Radeacher coplexty: Theore 1. Let F be a faly of functons appng fro Z to [0, 1], and let saple S = (z 1,..., z ) where z D for soe dstrbuton D over Z. Defne E[f] := E Z D [f(z)], and defne ÊS[f] := 1 =1 f(z ). Wth probablty 1 δ, for all f F: 2 ( ) E[f] ÊS[f] + 2R (F) + O ( ) E[f] ÊS[f] + 2 ˆR S (F) + O Proof. We derve a bound for E[f] ÊS[f] for all f F, or equvalently, bound sup (E[f] Ê S [f]). Note that ths expresson s a rando varable that depends on S. So we want to bound the followng rando varable: (11) (12) Φ(S) = sup(e[f] ÊS[f]) (13) Step 1: We show, wth probablty 1 δ, Φ(S) E S [Φ(S)] + 2. Ths step allows us to go fro workng wth Φ(S) to workng wth E S [Φ(S)]. then: Recall that McDard s nequalty states that, f: f(x 1,..., x,..., x ) f(x 1,..., x,..., x ) c (14) P r[f(x 1,..., x ) E[f(X 1,..., X )] + ɛ] exp( 2ɛ 2 / Fro the defnton of Φ(S), we have: c 2 ) (15) Φ(S) = sup(e[f] ÊS[f]) (16) = sup (E[f] 1 f(z )) (17) Snce f(z ) [0, 1] for all z, changng any one exaple z to z n the tranng set S wll change 1 f(z ) by at ost 1. Thus ths changng of any one exaple affects Φ(S) by at ost ths aount, plyng that Φ((z 1,..., z,..., z )) Φ((z 1,..., z,..., z )) 1. Ths fts the condton of McDard s nequalty (see (14)) wth c = 1, so we can apply McDard s nequalty and arrve at the bound shown. 2 Note that the Bg-Oh ters n the two expressons have dfferent constants. =1 3

4 Step 2: Defne a ghost saple S = (z 1,..., z ), z D. We show that E S [Φ(S)] E S,S [sup (ÊS [f] ÊS[f])]: E S [Φ(S)] = E S [sup(e[f] ÊS[f])] (18) = E S [sup(e S [ÊS [f]] ÊS[f])] (19) = E S [sup(e S [ÊS [f] ÊS[f]])] (20) E S,S [sup(ês [f] ÊS[f])] (21) Note that we arrve at (19) snce the expected Radeacher coplexty E[f] s equal to the expectaton over all saples S of the eprcal Radeacher coplexty over those S, or E S [ÊS [f]]. We also arrve at (21) by ovng the expectaton over S n (20) outsde of the sup; ths can be done snce the expectaton of a ax over soe functon s at least the ax of that expectaton over that functon. Step 3: We show E S,S [sup (ÊS [f] ÊS[f])] = E S,S,σ[sup σ (f(z ) f(z ))] We use the ghost saplng technque for ths step. In partcular, for each par of eleents z, z n S, S respectvely, swap the two wth probablty 1/2. Let the resultng two sets of exaples be T, T. Snce S, S each ntally represented d saples fro D, we have that T, T S, S. Ths ples: Ê S [f] ÊS[f] ÊT [f] ÊT [f] (22) = 1 f(z ) f(z ) wth prob. 1/2 f(z ) f(z (23) ) wth prob. 1/2 = 1 σ (f(z ) f(z )) (24) Thus the expressons sup (ÊS [f] ÊS[f]) and sup σ (f(z ) f(z )) are equally dstrbuted. The latter depends on an addtonal set of rando varables σ, however, so we ust take the expectaton of the latter over σ as well as S, S. Takng the expectaton of the forer over S, S, as well, we arrve at the expresson shown. Step 4: We show E S,S,σ[sup σ (f(z ) f(z ))] 2R (F) E S,S,σ[sup σ (f(z ) f(z ))] E S,S,σ[sup σ f(z ) + sup ( σ )f(z ))] (25) E S,σ[sup σ f(z )] + E S,σ [sup ( σ )f(z ))] (26) = R (F) + R (F) (27) where we arrve at (27) because σ has the sae dstrbuton as σ. Concluson: Cobnng all the peces together, we fnally have that, wth probablty 1 δ, for all f F: E[f] ÊS[f] 2R (F) + (28) 2 4

5 To derve the bound nvolvng ˆR S (F), we use McDard s nequalty agan. Recall the defnton of ˆR 1 S (F) := E σ [sup σ f(z )]. Snce f [0, 1], changng one eleent n S changes ˆR S (F) by at ost 1. We can apply McDard s nequalty agan, fndng, wth probablty 1 δ: ˆR S (F) R (F) + (29) 2 Usng a δ = δ/2 and applyng the unon bound to (28) and (29), we have our result. Wth probablty 1 δ, for all f F: E[f] ÊS[f] + 2 ˆR S (F) + O( ) (30) 2.2 Bounds for hypothess spaces H To get fro ths generalzaton bound on classes of all real-valued functons to classes of hypotheses, defne the followng: Z = X 1, +1} (31) f h (x, y) = 1h(x) y} (32) F H = f h : h H} (33) Note that, due to (33), each f h F H corresponds to soe h H. Also note that, by these defntons, we have: err(h) = E (x,y) D [1h(x) y}] = E[f h ] (34) err(h) ˆ = 1 1h(x ) y } = h ] (35) Evdently we can use our bound fro Theore 1 to bound err(h) err(h): ˆ 1 ˆR S (F H ) = E σ [ sup σ f h (x, y )] (36) f h F H 1 = E σ [sup σ ( 1 y h(x ) )] (37) h H 2 = E σ [ 1 1 σ + sup ( y σ )h(x )] (38) 2 h H 2 = 1 2 E 1 σ[sup h H = 1 2 E 1 σ[sup h H ( y σ )h(x )] (39) σ h(x )] (40) = 1 2 ˆR S (H) (41) Note that we arrve at (40) snce ( y σ ) has the sae dstrbuton as σ. Now, cobnng (30), (34), (35), and (41), we have: err(h) err(h) ˆ + ˆR S (H) + O( ) (42) 5