Vapnik Chervonenkis classes

Transcription

1 Vapk Chervoeks classes Maxm Ragsky September 23, 2014 A key result o the ERM algorthm, proved the prevous lecture, was that P( f ) L (F ) + 4ER (F (Z )) + 2log(1/δ) wth probablty at least 1 δ. The quatty R (F (Z )) appearg o the rght-had sde of the above boud s the Rademacher average of the radom set F (Z ) = ( f (Z 1 ),..., f (Z ) ) : f F }, ofte referred to as the projecto of F oto the sample Z. From ths we see that a suffcet codto for the ERM algorthm to produce ear-optmal hypotheses wth hgh probablty s that the expected Rademacher average ER (F (Z )) = Õ(1/ ), where the Õ( ) otato dcates that the boud holds up to polylogarthmc factors,.e., there exsts some postve polyomal fucto p( ) such that ER (F (Z )) O p(log ). Hece, a lot of effort statstcal learg theory s devoted to obtag tght bouds o ER (F (Z )). Oe way to guaratee a Õ(1/ ) boud o ER s f the effectve sze of the radom set F (Z ) s fte ad grows polyomally wth. The the Fte Class Lemma wll tell us that R (F (Z )) = O log. I geeral, a reasoable oto of effectve sze s captured by varous coverg umbers (see, e.g., the lecture otes by Medelso [Me03] or the recet moograph by Talagrad [Tal05] for detaled expostos of the relevat theory). I ths lecture, we wll look at a smple combatoral oto of effectve sze for classes of bary-valued fuctos. Ths partcular oto has orgated wth the work of Vapk ad Chervoeks [VC71], ad was hstorcally the frst such oto to be troduced to statstcal learg theory. It s ow kow as the Vapk Chervoeks (or VC) dmeso. 1 Vapk Chervoeks dmeso: defto Defto 1. Let C be a class of (measurable) subsets of some space Z. We say that a fte set S = z 1,..., z } Z s shattered by C f for every subset S S there exsts some C C such that S = S C. 1

2 I other words, S = z 1,..., z } s shattered by C f for ay bary -tuple b = (b 1,...,b ) 0,1} there exsts some C C such that 1z1 C },...,1 z C } = b or, equvaletly, f } 1z1 C },...,1 z C } : C C = 0,1}, where we cosder ay two C 1,C 2 C as equvalet f 1 z C 1 } = 1 z C 2 } for all 1. Defto 2. The Vapk Chervoeks dmeso (or the VC dmeso) of C s V (C ) max N : S Z such that S = ad S s shattered by C }. If V (C ) <, we say that C s a VC class (of sets). We ca express the VC dmeso terms of shatter coeffcets of C : Let S (C ) sup S Z, S = S C : C C } deote the th shatter coeffcet of C, where for each fxed S we cosder ay two C 1,C 2 C as equvalet f S C 1 = S C 2. The V (C ) = max N : S (C ) = 2 }. The VC dmeso V (C ) may be fte, but t s always well-defed. Ths follows from the followg lemma: Lemma 1. If S (C ) < 2, the S m (C ) < 2 m for all m >. Proof. Suppose S (C ) < 2. Cosder ay m >. We wll suppose that S m (C ) = 2 m ad derve a cotradcto. By our assumpto that S m (F ) = 2 m, there exsts S = z 1,..., z m } Z m, such that for every bary -tuple b = (b 1,...,b ) we ca fd some C C satsfyg 1z1 C },...,1 z C }, 1 z+1 C },...,1 zm C } = (b1,...,b,0,...,0). (1) From (1) t mmedately follows that 1z1 C },...,1 z C } = (b1,...,b ). (2) Sce b = (b 1,...,b ) was arbtrary, we see from (2) that S (C ) = 2. Ths cotradcts our assumpto that S (C ) < 2, so we coclude that S m (C ) < 2 m wheever m > ad S (F ) < 2. There s a oe-to-oe correspodece betwee bary-valued fuctos f : Z 0,1} ad subsets of Z: f : Z 0,1} let C f z : f (z) = 1} C Z let f C 1 C }. Thus, we ca exted the cocept of shatterg, as well as the defto of the VC dmeso, to ay class F of fuctos f : Z 0,1}: 2

3 Defto 3. Let F be a class of fuctos f : Z 0,1}. We say that a fte set S = z 1,..., z } Z s shattered by F f t s shattered by the class C F 1 f =1} : f F }, where 1 f =1} s the dcator fucto of the set C f z Z : f (z) = 1}. The th shatter coeffcet of F s S (F ) = S (C F ), ad the VC dmeso of F s defed as V (F ) = V (C F ). I lght of these deftos, we ca equvaletly speak of the VC dmeso of a class of sets or a class of bary-valued fuctos. 2 Examples of Vapk Chervoeks classes 2.1 Sem-fte tervals Let Z = R ad take C to be the class of all tervals of the form (, t] as t vares over R. We wll prove that V (C ) = 1. I vew of Lemma 1, t suffces to show that (1) ay oe-pot set S = a} s shattered by C, ad (2) o two-pot set S = a,b} s shattered by C. Gve S = a}, choose ay t 1 < a ad t 2 > a. The (, t 1 ] S = ad (, t 2 ] S = S. Thus, S s shattered by C. Ths holds for every oe-pot set S, ad therefore we have proved (1). To prove (2), let S = a,b} ad suppose, wthout loss of geeralty, that a < b. The there exsts o t R such that (, t] S = b}. Ths follows from the fact that f b (, t] S, the t b. Sce b > a, we must have t > a, so that a (, t] S as well. Sce a ad b are arbtrary, we see that o two-pot subset of R ca be shattered by C. 2.2 Closed tervals Aga, let Z = R ad take C to be the class of all tervals of the form [s, t] for all s, t R. The V (C ) = 2. To see ths, we wll show that (1) ay two pot set S = a,b} ca be shattered by C ad that (2) o threepot set S = a,b,c} ca be shattered by C. For (1), let S = a,b} ad suppose, wthout loss of geeralty, that a < b. Choose four pots t 1, t 2, t 3, t 4 R such that t 1 < t 2 < a < t 3 < b < t 4. There are four subsets of S:, a}, b}, ad a,b} = S. The [t 1, t 2 ] S =, [t 2, t 3 ] S = a}, [t 3, t 4 ] S = b}, [t 1, t 4 ] S = S. Hece, S s shattered by C. Ths holds for every two-pot set R, whch proves (1). To prove (2), let S = a,b,c} be a arbtrary three-pot set wth a < b < c. The the tersecto of ay [t 1, t 2 ] C wth S cotag a ad c must ecessarly cota b as well. Ths shows that o three-pot set ca be shattered by C, so by Lemma 1 we coclude that V (C ) = Closed halfspaces Let Z = R 2, ad let C cosst of all closed halfspaces,.e., sets of the form z = (z 1, z 2 ) R 2 : w 1 z 1 + w 2 z 2 b} for all choces of w 1, w 2,b R such that (w 1, w 2 ) (0,0). The V (C ) = 3. 3

4 Fgure 1: Impossblty of shatterg a affely depedet four-pot set R 2 by closed halfspaces. To see that S 3 (C ) = 2 3 = 8, t suffces to cosder ay set S = z 1, z 2, z 3 } of three o-collear pots. The t s ot hard to see that for ay S S t s possble to choose a closed halfspace C C that would cota S, but ot S. To see that S 4 (C ) < 2 4, we must look at all four-pot sets S = z 1, z 2, z 3, z 4 }. There are two cases to cosder: 1. Oe pot S les the covex hull of the other three. Wthout loss of geeralty, let s suppose that z 1 cov(s ) wth S = z 2, z 3, z 4 }. The there s o C C such that C S = S. The reaso for ths s that every C C s a covex set. Hece, f S C, the ay pot cov(s ) s cotaed C as well. 2. No pot S s the covex hull of the remag pots. Ths case, whe S s a affely depedet set, s show Fgure 1. Let us partto S to two dsjot subsets, S 1 ad S 2, each cosstg of opposte pots. I the fgure, S 1 = z 1, z 3 } ad S 2 = z 2, z 4 }. The t s easy to see that there s o halfspace C whose boudary could separate S 1 from ts complemet S 2. Ths s, fact, the ()famous XOR couterexample" of Msky ad Papert [MP69], whch has demostrated the mpossblty of uversal cocept learg by oe-layer perceptros. Sce ay four-pot set R 2 falls uder oe of these two cases, we have show that o such set ca be shattered by C. Hece, V (C ) = 3. More geerally, f Z = R d ad C s the class of all closed halfspaces z R d : } d w j z j b for all w = (w 1,..., w d ) R d such that at least oe of the w j s s ozero ad all b R, the V (C ) = d + 1 [WD81]; we wll see a proof of ths fact shortly. 2.4 Axs-parallel rectagles Let Z = R 2, ad let C cosst of all axs-parallel rectagles,.e., sets of the form C = [a 1,b 1 ] [a 2,b 2 ] for all a 1,b 1, a 2,b 2 R. The V (C ) = 4. 4

5 Fgure 2: Impossblty of shatterg a fve-pot set by axs-parallel rectagles. Frst we exhbt a four-pot set S = z 1, z 2, z 3, z 4 } that s shattered by C. It suffces to take z 1 = ( 2, 1), z 2 = (1, 2), z 3 = (2,1), z 4 = ( 1,2). To show that o fve-pot set s shattered by C, cosder a arbtrary S = z 1, z 2, z 3, z 4, z 5 }. Of these, pck ay oe pot wth the smallest frst coordate ad ay oe pot wth the largest frst coordate, ad lkewse for the secod coordate (refer to Fgure 2), for a total of at most four. Let S deote the set cosstg of these pots; Fgure 2, S = z 1, z 2, z 3, z 4 }. The t s easy to see that ay C C that cotas the pots S must cota all the pots S\S as well. Hece, o fve-pot set R 2 ca be shattered by C, so V (C ) = 5. The same argumet also works for axs-parallel rectagles R d,.e., all sets of the form C = [a 1,b 1 ] [a 2,b 2 ]... [a d,b d ], leadg to the cocluso that the VC dmeso of the set of all axs-parallel rectagles R d s equal to 2d. 2.5 Sets determed by fte-dmesoal fucto spaces The followg result s due to Dudley [Dud78]. Let Z be arbtrary, ad let G be a m-dmesoal lear space of fuctos g : Z R, whch meas that each g G has a uque represetato of the form g = m c j ψ j, where ψ 1,...,ψ m : Z R form a fxed learly depedet set ad c 1,...,c m are real coeffcets. Cosder the class } C = z Z : g (z) 0} : g G. The V (C ) m. To prove ths, we eed to show that o set of m + 1 pots Z ca be shattered by C. To that ed, let us fx m + 1 arbtrary pots z 1,..., z m+1 Z ad cosder the mappg L : G R m+1 defed by L(g ) ( g (z 1 ),..., g (z m+1 ) ). It s easy to see that because G s a lear space, L s a lear mappg,.e., for ay g 1, g 2 G ad ay c 1,c 2 R we have L(c 1 g 1 +c 2 g 2 ) = c 1 L(g 1 )+c 2 L(g 2 ). Sce dmg = m, the mage of G uder L,.e., the set L(G ) = (g (z 1 ),..., g (z m+1 )) R m+1 : g G }, 5

6 s a lear subspace of R m+1 of dmeso at most m. Ths meas that there exsts some ozero vector v = (v 1,..., v m+1 ) R m+1 orthogoal to L(G ),.e., for every g G v 1 g (z 1 ) v m+1 g (z m+1 ) = 0. (3) Wthout loss of geeralty, we may assume that at least oe compoet of v s strctly egatve (otherwse we ca take v stead of v ad stll get (3)). Hece, we ca rearrage the equalty (3) as v g (z ) = v g (z ), g G. (4) :v 0 :v <0 Now let us suppose that S m+1 (C ) = 2 m+1 ad derve a cotradcto. Cosder a bary (m + 1)-tuple b = (b 1,...,b m+1 ) 0,1} m+1, where b j = 1 f ad oly f v j 0, ad 0 otherwse. Sce we assumed that S m+1 (C ) = 2 m+1, there exsts some g G such that 1g (z1 ) 0},...,1 g (zm+1) 0} = b. By our defto of b, ths meas that the left-had sde of (4) s oegatve, whle the rght-had sde s egatve, whch s a cotradcto. Hece, S m+1 (C ) < 2 m+1, so V (C ) m. Ths result ca be used to boud the VC dmeso of may classes of sets: Let C be the class of all closed halfspaces R d. The ay C C ca be represeted the form C = z : g (z) 0} for g (z) = w, z b wth some ozero w R d ad b R. The set G of all such affe fuctos o R d s a lear space of dmeso d +1, so by the above result we have V (C ) d +1. I fact, we kow that ths holds wth equalty [WD81]. Ths ca also be see from the followg result, due to Cover [Cov65]: Let G be the lear space of fuctos spaed by fuctos ψ 1,...,ψ m, ad let z 1,..., z } Z be such that the vectors Ψ(z ) = (ψ 1 (z ),...,ψ m (z )),1, form a learly depedet set. The for the class of sets C = z : g (z) 0} : z Z} we have C z 1,..., z } : C C = m 1 =0 1. The codtos eeded for Cover s result are see to hold for dcators of halfspaces, so lettg = m = d + 1 we see that S d (C ) = d =0 ( d ) = 2 d. Hece, V (C ) = d + 1. Let C be the class of all closed balls R d,.e., sets of the form C = z R d : z x 2 r 2} where x R d s the ceter of C ad r R + s ts radus. The we ca wrte C = z : g (z) 0}, where g (z) = r 2 z x 2 = r 2 Expadg the secod expresso for g (5), we get d z j x j 2. (5) d d d g (z) = r 2 x 2 j + 2 x j z j z 2 j, whch ca be wrtte the form g (z) = d+2 k=1 c kψ k (z), where ψ 1 (z) = 1, ψ k (z) = z k for k = 2,...,d + 1, ad ψ d+2 = d z2 j. It ca be show that the fuctos ψ k} d+2 are learly depedet. Hece, k=1 V (C ) d + 2. Ths boud, however, s ot tght; as show by Dudley [Dud79], the class of closed balls R d has VC dmeso d

7 2.6 VC dmeso vs. umber of parameters Lookg back at all these examples, oe may get the mpresso that the VC dmeso of a set of baryvalued fuctos s just the umber of parameters. Ths s ot the case. Cosder the followg oeparameter famly of fuctos: g θ (z) s(θz), θ R. However, the class of sets } C = z R : g θ (z) 0} : θ R has fte VC dmeso. Ideed, for ay, ay collecto of umbers z 1,..., z R, ad ay bary strg b 0,1}, oe ca always fd some θ R such that sg(s(θz )) = +1, f b = 1 1, f b = 0. 3 Growth of shatter coeffcets: the Sauer Shelah lemma The mportace of VC classes learg theory arses from the fact that, as teds to fty, the fracto of subsets of ay z 1,..., z } Z that are shattered by a gve VC class C teds to zero. We wll prove ths fact ths secto by dervg a sharp boud o the shatter coeffcets S (C ) of a VC class C. Ths boud have bee (re)dscovered at least three tmes, frst a weak form by Vapk ad Chervoeks [VC71] 1971, the depedetly ad dfferet cotexts by Sauer [Sau72] ad Shelah [She72] I strct accordace wth Stgler s law of epoymy 1, t s kow the statstcal learg lterature as the Sauer Shelah lemma. Before we state ad prove ths result, we wll collect some prelmares ad set up some otato. Gve tegers,d 1, let d, f > d φ(,d) =0 2, f d If we adopt the coveto that ( ) = 0 for >, we ca wrte φ(,d) = d for all,d 1. We wll fd the followg recursve relato useful: =0 Lemma 2. φ(,d) = φ( 1,d) + φ( 1,d 1). Proof. We have 1 1 ( 1)! + = 1 ( 1)!( )! + ( 1)!!( 1)!. 1 No scetfc dscovery s amed after ts orgal dscoverer" ( s_law_of_epoymy) 7

8 Multplyg both sdes by!( )!, we obta [ ] 1 1!( )! + = ( 1)! + ( )( 1)! =! 1 Hece, 1 1! + = 1!( )! =. (6) Usg the defto of φ(,d), as well as (6), we get ad the lemma s proved. Now for the actual result: d d d 1 d 1 φ(,d) = = 1 + = =0 =1 =1 =1 1 }}}} =φ( 1,d) =φ( 1,d 1) Theorem 1 (Sauer Shelah lemma). Let C be a class of subsets of some space Z wth V (C ) = d <. The for all, S (C ) φ(,d). (7) Proof. There are several dfferet proofs the lterature; we wll use a ductve argumet followg Blumer et al. [BEHW89]. We ca assume, wthout loss of geeralty, that > d, for otherwse S (C ) = 2 = φ(,d). For a arbtrary fte set S Z, let S(S,C ) S C : C C }, where, as before, we cout oly the dstct sets of the form S C. By defto, S (C ) = sup S: S = S(S,C ). Thus, t suffces the prove the followg: For ay S Z wth S = > d, S(S,C ) φ(,d). For the purpose of computg S(S,C ), ay two C 1,C 2 C such that S C 1 = S C 2 are deemed equvalet. Hece, let A A S : A = S C for some C C }. The we may wrte S(S,C ) = S C : C C } = A S : A = S C for some C C } = A. Moreover, t s easy to see that V (A ) V (C ) = d. Thus, the desred result s equvalet to sayg that f A s a collecto of subsets of a -elemet set S (whch we may, wthout loss of geeralty, take to be [] 1,...,}) wth V (A ) d <, the A φ(,d). We wll prove ths statemet by double ducto o ad d. Frst of all, the statemet (7) holds for all 1 ad d = 0. Ideed, f V (A ) = 0, the A = 1 2. Now assume that (7) holds for all ad all A wth V (A ) d 1, ad for all tegers up to 1 ad all A wth V (A ) d. Now let S = [], ad let A be a collecto of subsets of [] wth V (A ) = d <. We wll show that A φ(, d). 8

9 To prove ths clam, let us choose a arbtrary S ad defe A \ A\ } : A A } A A A : A, A } A } Observe that both A \ ad A are classes of subsets of S\ }. Moreover, sce A ad A } map to the same elemet of A \, whle A s the umber of pars of sets A that map to the same set A \, we have A = A \ + A. (8) Sce A \ A, we have V (A \ ) V (A ) d. Also, every set A \ s a subset of S\ }, whch has cardalty 1. Therefore, by the ductve hypothess A \ φ( 1,d). Next, we show that V (A ) d 1. Suppose, to the cotrary, that V (A ) = d. The there must exst some T S\ } wth T = d that s shattered by A. But the T } s shattered by A. To see ths, gve ay T T choose some A A such that T A = T (ths s possble sce T s shattered by A ). But the A } A (by defto of A ), ad (T }) (A }) = (T A) } = T }. Sce ths s possble for a arbtrary T T, we coclude that T } s shattered by A. Now, sce T S\ }, we must have T, so T } = T +1 = d +1, whch meas that there exsts a (d +1)-elemet subset of S = [] that s shattered by A. But ths cotradcts our assumpto that V (A ) d. Hece, V (A ) d 1. Sce A s a collecto of subsets of S\ }, we must have A φ( 1,d 1) by the ductve hypothess. Hece, from (8) ad from Lemma 2 we have A = A \ + A φ( 1,d) + φ( 1,d 1) = φ(,d). Ths completes the ducto argumet ad proves (7). Corollary 1. If C s a collecto of sets wth V (C ) d <, the Moreover, f d, the where e s the base of the atural logarthm. S (C ) ( + 1) d. ( e S (C ) d ) d, Proof. For the frst boud, wrte d d! d φ(,d) = =!( )!! =0 =1 =1 d d! d!(d )! = d = ( + 1) d, =0 =0 where the last step uses the bomal theorem. O the other had, f d/ 1, the d d d d d d d d ( φ(,d) = = 1 + d ) e d, =0 =1 where we aga used the bomal theorem. Dvdg both sdes by (d/) d, we get the secod boud. =1 9

10 Let C be a VC class of subsets of some space Z. From the above corollary we see that S (C ) ( + 1) V (C ) lm sup 2 lm 2 = 0. I other words, as becomes large, the fracto of subsets of a arbtrary -elemet set z 1,..., z } Z that are shattered by C becomes eglgble. Moreover, combg the bouds of the corollary wth the Fte Class Lemma for Rademacher averages, we get the followg: Theorem 2. Let F be a VC class of bary-valued fuctos f : Z 0,1} o some space Z. Let Z be a..d. sample of sze draw accordg to a arbtrary probablty dstrbuto P P (Z). The ER (F (Z V (F )log( + 1) )) 2. A more refed chag techque [Dud78] ca be used to remove the logarthm the above boud: Theorem 3. There exsts a absolute costat C > 0, such that uder the codtos of the precedg theorem ER (F (Z V (F ) )) C. Refereces [BEHW89] A. Blumer, A. Ehrefeucht, D. Haussler, ad M. K. Warmuth. Learablty ad the Vapk Chervoeks dmeso. Joural of the ACM, 36(4): , [Cov65] T. M. Cover. Geometrcal ad statstcal propertes of systems of lear equaltes wth applcatos patter recogto. IEEE Trasactos o Electroc Computers, 14: , [Dud78] R. M. Dudley. Cetral lmt theorems for emprcal measures. Aals of Probablty, 6: , [Dud79] [Me03] R. M. Dudley. Balls R k do ot cut all subsets of k + 2 pots. Advaces Mathematcs, 31(3): , S. Medelso. A few otes o statstcal learg theory. I S. Medelso ad A. J. Smola, edtors, Advaced Lectures Mache Learg, volume 2600 of Lecture Notes Computer Scece, pages [MP69] M. Msky ad S. Papert. Perceptros: A Itroducto to Computatoal Geometry. MIT Press, [Sau72] N. Sauer. O the desty of famles of sets. Joural of Combatoral Theory, Seres A, 13: , [She72] S. Shelah. A combatoral problem: stablty ad order for models ad theores fty laguages. Pacfc Joural of Mathematcs, 41: ,

11 [Tal05] [VC71] [WD81] M. Talagrad. Geerc Chag: Upper ad Lower Bouds of Stochastc Processes. Sprger, V. N. Vapk ad A. Ya. Chervoeks. O the uform covergece of relatve frequeces of evets to ther probabltes. Theory of Probablty ad Its Applcatos, 16: , R. S. Wecour ad R. M. Dudley. Some specal Vapk Chervoeks classes. Dscrete Mathematcs, 33: ,