MACHINE LEARNING. Mistake and Loss Bound Models of Learning

Transcription

1 Iowa State Unvesty MACHINE LEARNING Vasant Honava Bonfomatcs and Computatonal Bology Pogam Cente fo Computatonal Intellgence, Leanng, & Dscovey Iowa State Unvesty Iowa State Unvesty Mstake and Loss Bound Models of Leanng Outlne Machne leanng and theoes of leanng Mstake bound model of leanng Mstake bound analyss of conjunctve concept leanng Weghted majoty and elated multplcatve update algothms Applcatons

2 Iowa State Unvesty Computatonal Models of Leanng Model of the Leane: Computatonal capabltes, sensos, effectos, knowledge epesentaton, nfeence mechansms, po knowledge, etc. Model of the Envonment: Tasks to be leaned, nfomaton souces (teache, quees, expements), pefomance measues Key questons: Can a leane wth a cetan stuctue lean a specfed task n a patcula envonment? Can the leane do so effcently? If so, how? If not, why not? Iowa State Unvesty Models of Leanng: What ae they good fo? To make explct elevant aspects of the leane and the envonment To dentfy easy and had leanng poblems (and the pecse condtons unde whch they ae easy o had) To gude the desgn of leanng systems To shed lght on natual leanng systems To help analyze the pefomance of leanng systems 2

3 Iowa State Unvesty Mstake Bound Analyss Example Leanng Conjunctve Concepts Gven an abtay, nose-fee sequence of labeled examples (X,C(X )),(X 2,C(X 2 ))...(X m,c(x m )) of an unknown bnay conjunctve concept C ove {0,} N, the leane's task s to pedct C(X) fo a gven X. Theoem: Exact onlne leanng of conjunctve concepts can be accomplshed wth at most (N+) pedcton mstakes. Iowa State Unvesty Onlne leanng of conjunctve concepts Algothm A. Intalze L={X, ~X,... X N ~X N } Pedct accodng to match between an nstance and the conjuncton of lteals n L Wheneve a mstake s made on a postve example, dop the offendng lteals fom L Example (0, ) wll esult n L = {~X, X 2,X 3, X 4 } (0, ) wll yeld L = {X 2,X 3 } 3

4 Iowa State Unvesty Mstake bound analyss of conjunctve concept leanng Poof Sketch No lteal n C s eve elmnated fom L Each mstake elmnates at least one lteal fom L The fst mstake elmnates N of the 2N lteals Conjunctve concepts can be leaned wth at most (N+) mstakes Concluson Conjunctve concepts ae easy to lean n the mstake bound model Iowa State Unvesty Optmal Mstake Bound Leanng Algothms Defnton: An optmal mstake bound mbound(c) fo a concept class C s the lowest possble mstake bound n the wost case (consdeng all concepts n C, and all possble sequences D of examples). mbound ( C ) = mn max max leanes L c* C example se quences D mstakes whee mstakes (c*,l,d) s the numbe of mstakes made by L n ts attempt to lean c* based on the sequence of examples povded. ( c*,l,d ) 4

5 Iowa State Unvesty Mstake Bounds and optmal mstake bounds mbound( C) 2 mbound( C) C N mbound( C) log (why?) (why?) tval bounds ( C ) } we wll pove ths Defnton: An optmal leanng algothm fo a concept class C (n the mstake bound famewok) s one that s guaanteed to exactly lean any concept n C, usng any nose-fee example sequence, wth at most O(mbound(C)) mstakes. Iowa State Unvesty V V ξ ξ 0 0 = Fo Veson space and Halvng algothm { c C c s consstent wth the fst examples } = C ( C, X ) = { c C : c( X ) = 0} ( C, X ) = { c C : c( X ) = } ξ 0 ( V, X ) f c * ( X ) > 0, V = ξ ( V, X ) f c * ( X ) = 0 = Halvng ξ Algothm On ( V, X ) ξ ( V, X ) 0 : nput X, Pedct and 0 othewse. Elmnate the concepts (majoty o mnoty) that wee wong f 5

6 Iowa State Unvesty V V ξ ξ 0 0 = Fo Halvng Algothm Defnton: The halvng algothm pedcts accodng to the majoty of concepts n the cuent veson space and a mstake esults n elmnaton of all the offendng concepts fom the veson space. { c C c s consstent wth the fst examples} = C ( C, X ) = { c C : c( X ) = 0} ( C, X ) = { c C : c( X ) = } ξ0 ( V, X ) f c * ( X ) > 0, V = ξ ( V, X ) f c * ( X ) = 0 = Iowa State Unvesty The Halvng Algothm ( ) Theoem: mbound( C) log C Poof: The halvng algothm pedcts accodng to majoty of concepts n the veson space Each mstake elmnates at least half of the canddate hypotheses n the veson space The numbe of mstakes s bounded by log( C ) The halvng algothm can be computatonally feasble f thee s a way to compactly epesent and effcently manpulate the veson space. Othewse t s not computatonally feasble. 6

7 Iowa State Unvesty The Halvng Algothm The halvng algothm s not optmal wth espect to the numbe of mstakes. In ode to mnmze the numbe of mstakes, the leane has to guess accodng to the subset of the veson space that s expected to yeld the fewest mstakes. The optmal mstake bound algothm has to pedct f ( ξ ( V X )) mbound( ( V X )) mbound,, ξ0 and 0 othewse. Howeve ths algothm s even less effcent. Iowa State Unvesty The Halvng Algothm Queston: Ae thee any effcently mplementable leanng algothms wth mstake bounds compaable to that of the halvng algothm? Answe: Lttlestone's algothm fo leanng monotone dsjunctons of at most k of n lteals usng the hypothess class of theshold functons wth at most (k lg n) mstakes. Moe on ths late. 7

8 Iowa State Unvesty Randomzed Halvng Algothm The pedctons made by the halvng algothm may not be based on any concept n C. Thee may not exst n C a concept that s consstent wth the majoty vote. The andomzed halvng algothm due to Maass pedcts accodng to a andomly selected concept c C All concepts n C that ae nconsstent wth the example ae elmnated fom futhe consdeaton. Theoem: The expected numbe of mstakes made by the andomzed halvng algothm s at most log C + O() Iowa State Unvesty Randomzed Halvng Algothm WLOG assume that the ode of pesentaton of the examples s ndependent of the leane s actons Suppose the concepts n the veson space ae odeed by when they ae gong to be elmnated by examples. Let c.c be the ode (so = V ) Let M be the expected numbe of mstakes 8

9 9 Iowa State Unvesty Randomzed Halvng Algothm The andomzed halvng algothm pcks one of the concepts at andom wth pobablty equal to /. Suppose c s chosen c s the taget concept and hence thee can be no futhe mstakes. One of the othe concepts s chosen wth pobablty (-)/ In ths case, Thee wll be one mstake (at least) Plus the expected numbe of mstakes fo the emanng concepts Iowa State Unvesty Randomzed Halvng Algothm ( ) ( ) ( ) ( ) () ln 2) ( ; ) ( O M M M M M M M M M M M M M M M + = = + = + = + + = + = + = + + = = = = = = =

10 Iowa State Unvesty Leanng monotone dsjunctons when elevant attbutes abound C = { x x... x {... N}; j {... k} } N N N C = k k 0 lg C = Θ( k lg N) 2 k j How can we desgn an algothm that acheves ths mstake bound? Iowa State Unvesty Wnnow Algothm Obsevaton Monotone dsjunctons ae a subset of theshold functons) Idea Use theshold neuons to lean monotone dsjunctons N Intalzeθ = ; W = (...) 2 Pedct y( X) = ff W.X > θ othewse pedct y( X) = 0 If c( X) = but y( X) = 0, double all w whee x = If c( X) = 0 but y( X) =, zeo out all w whee x = Theoem Wnnow makes O(k lg N) mstakes 0

11 Iowa State Unvesty w u k u k v θ Wnnow Algothm N 0 w = w 2θ u lg w k ( N + uθ vθ) ( lg w ) ( lg θ + ) ( lgθ + ) N + k( lgθ + ) ( u + v) 2 + 2k lg N u numbe of tmes weghts ae doubled v numbe of tmes weghts ae zeoed out Each weght doublng adds at most θ to the sum of weghts and each zeoed out weght subtacts at least θ fom the sum of weghts No weght that s geate than θ s eve doubled Each weght doublng has to affect at least one of the weghts. Each weght doublng adds at least to the logathm of the weght that got doubled Theoem Wnnow makes O(k lg N) mstakes Iowa State Unvesty Genealzatons of Wnnow Wnnow algothm and ts vaants and genealzatons can be used to lean concepts fom moe expessve concept classes by pepocessng the nput pattens e.g. by tansfomng an n-bt patten nto an O(n k ) bt patten that encodes all conjunctons of at most k lteals (negated o un negated) Wnnow algothm and ts vaants can be made to genealze bette by ncopoatng egulazaton

12 Iowa State Unvesty Weghted majoty leanng algothm (WML) Motvatons Robust algothm fo leanng monotone dsjunctons Suppose we have a pool of pedctos featues, expets, algothms The optmal pedcto that s, the pedcto that makes the fewest mstakes depends on the data and s not known a po Basc dea make pedctons based on a weghted majoty of pedctons of all pedctos n the pool f a mstake s made, adjust the weghts Iowa State Unvesty Weghted majoty leanng algothm (WML) Intalze W = Fo If If If c( X) y( X), ( )... ; θ θ + w Pedct y( X) = f Pedct y( X) = 0 f Pedct 0 0 θ θ + w θ = θ = 0 each tanng example ( X, c( X)) x = 0, x =, y( X) = Random ({,0}) f 0 θ > θ 0 θ < θ w βw 0 θ = θ ( 0 < β < ) 0 2

13 Iowa State Unvesty Weghted majoty leanng algothm (WML) Theoem: Let D be any sequence of tanng examples. Let A={A A n } be any pool of n pedctos. Let k be the numbe of mstakes made by the best pedcto n the pool A on the sequence of examples D. Then m = mbound WML k log + log n β 2 log + β Iowa State Unvesty W W afte afte Weghted majoty leanng algothm (WML) Poof: The total weght assocated wth the n pedctos at any tme s W = (θ 0 + θ ) whee θ 0 and θ ae as defned n WML Consde an example on whch a mstake s made. WLOG, assume that the pedcton was 0. Then the total weght afte the update s θ0 + θ ( θ0 + θ ) ( β) (why?) 2 + β + β ( θ0 + θ ) = Wbefoe 2 2 The pedctos n the weghted majoty must have held at least half the total weght. Afte update, ths poton of the weght gets educed by a facto (-β) Each mstake causes sum of weghts afte an update to be no moe than + β tmes the value befoe the update 2 3

14 4 Iowa State Unvesty Weghted majoty leanng algothm (WML) k fnal j β w =, n β W β W W β W m nt m fnal befoe afte + = The best pedcto (say w j ) makes k mstakes and hence undegoes k weght updates. So Clealy, the weght of the best pedcto can be no geate than the sum of weghts n β β m k + 2 whee m s the numbe of updates Iowa State Unvesty Weghted majoty leanng algothm (WML) ( ) β k n m β m k n β m β k n β β n β n β n β m k m k m k 2 log log log 2 log log log 2 log log log β β β Implcaton: WML makes almost as few mstakes as the optmal leane n the pool We can use weghted majoty when t s unclea whch leane n the pool s optmal

15 Iowa State Unvesty Some Vaatons on WML Selecton fom countably nfnte pool of pedctos No algothm (that halts) can obtan pedctons fom an nfnte numbe of pedctos Howeve.. We can modfy WML so that t consdes successvely lage pools of pedctos The modfed algothm behaves vey much lke WML wth a degadaton n mstake bound of the ode of (log ) whee th pedcto n the pool s the optmal pedcto Iowa State Unvesty Some Vaatons on WML Randomzed pedctons Pedct wth pobablty Pedct 0 wth pobablty θ θ θ + 0 θ0 θ θ + 0 The update equatons can be modfed so that the ate of mstakes appoaches abtaly close to the ate of mstakes of the best pedcto n the pool 5

16 Iowa State Unvesty Some Vaatons on WML Balanced Wnnow Inputs and outputs ae bpola Balanced Wnnow 0 < β < If sgn If y =, + ( W X W X) If y =, w + w w + + ( W, W,( X, y) ) + w + β + y x + x + β w w ; w + ; w w w x + β w + β x w Iowa State Unvesty Some Vaatons on WML Balanced Wnnow Balanced Wnnow s equvalent to keepng a scaled sum of updates as pecepton does but wth a scale facto of η = log β Balanced Wnnow 0 < β < If sgn ( W X) Z Z + ylog X β W 2snh y ( W, Z, ( X, y) ) z ( ) (That s, 2snh z Z w z = e e ) 6

17 Iowa State Unvesty Genealzed Pecepton algothms Genealzed Pecepton If sgn ( W X) Z Z + ηyx W f y ( W, Z, f,( X, y) ) ( Z) (That s, w f ( )) z See Gove, Lttlestone, Schuumans Iowa State Unvesty Quas addtve update algothms fo functon appoxmaton Genealzed Gadent Descent y W X Z Z + ηyx W f ( Z) (That s, w f ( )) z ( W, Z, f,( X, y) ) By choosng the leanng ate η and f appopately, we can obtan gadent-based leanng algothms that wok well n the pesence of elevant attbutes (See Kvnen and Wamuth) 7

18 Iowa State Unvesty Applcatons of Multplcatve Update Algothms Multplcatve update algothms consttute an example of theoetcal analyss of smple algothms leadng to a new poweful famly of algothms that ae useful n pactce Spellng coecton Text pocessng (SPAM fltes) Face ecognton Potfolo selecton Leanng n game-theoetc settngs Iowa State Unvesty Pobably Appoxmately Coect (PAC) Leanng Dstbuton-fee models of leanng Pobably Appoxmately Coect (PAC) Leanng Sample Complexty Analyss of Concept Classes Effcent PAC Leanes polynomal sample leanng, polynomal tme leanng Vapnk-Chevonenks (VC) dmenson and Sample Complexty Occam s azo Leanng unde smple dstbutons Bef tou of othe key esults 8

19 Iowa State Unvesty The Leanng Game H Example Oacle m m Examples S={(x, c(x))} Leane L Concept c C Tanng Samples x D Instance Dstbuton hypothess h H c C D Test Samples x D h h(x) Iowa State Unvesty The Leanng Game We assume An nstance space X A concept space { c : { 0,} } C = X A hypothess space H { h : { 0, } = X An unknown, abtay, not necessaly computable, statonay pobablty dstbuton D ove the nstance space X 9

20 Iowa State Unvesty Rules of the Game An advesay selects a dstbuton D ove a gven nstance space X and a taget concept c fom a gven concept class C An oacle samples the nstance space accodng to D and povdes a set S of labeled examples of an unknown concept c to the leane The leane's task s to output a hypothess h fom H that closely appoxmates the unknown concept c based on the examples t has encounteed The leane s tested on samples dawn fom the nstance space accodng to the same pobablty dstbuton D Iowa State Unvesty Measung the eo of a hypothess The eo of a hypothess h wth espect to a concept c and dstbuton D eo c, D ( h) = Px D ( c( x) h( x) ) c h X eo 20

21 Iowa State Unvesty Pobably Appoxmately Coect Leanng Why? Impossblty of leanng wth 0% eo Because nstances ae sampled accodng to an unknown, abtay pobablty dstbuton D ove the nstance space, thee s no way to be cetan that the leane wll see all the necessay examples to exactly lean an unknown concept exact leanng s mpossble! Impossblty of appoxmate leanng wth 00% confdence Appoxmate leanng (wth a specfed eo ε) cannot be guaanteed hunded pecent of the tme because of the vagaes of the samplng pocess Iowa State Unvesty ε-appoxmaton of a concept c We say that a hypothess h s an ε-appoxmaton of a concept c, wth espect to an nstance dstbuton D f and only f the pobablty that h and c dsagee on an nstance fom the nstance space dawn at andom accodng to the dstbuton D s less than ε. That s, eo c, D ( h) < ε 2

22 Iowa State Unvesty PAC Leanng A pelmnay defnton A concept class C s sad to be PAC-leanable usng a hypothess class H f thee exsts a leanng algothm L such that fo all concepts c C, fo all dstbutons D on an nstance space X, εδ, ( 0< εδ, < ), L, when allowed access to the Example oacle (that s, a fnte set S of labeled examples of a taget concept c), outputs wth pobablty at least ( δ ), a hypothess h H whch s an ε-appoxmaton of c. That s, D P ove X, c C, ε, δ : 0 < ε <, 0 < δ <, ( eo ( ) < ) ( δ ) S D c, D h Such a leanng algothm L s called a PAC leanng algothm fo the concept class C Iowa State Unvesty Notes on the defnton of PAC Leanablty The defnton of PAC leanablty of a specfed concept class C eques that thee be a leanng algothm L that poduces an ε-appoxmaton of any concept n the concept class C, unde any nstance dstbuton, and any choce of the eo (ε) and confdence (δ) paametes. Specfyng a leanng algothm eques the choce of an nstance epesentaton the choce of a hypothess (concept) epesentaton,and an algothm fo detemnng the membeshp of an nstance n a hypothess (concept). Moe on ths late 22

23 Iowa State Unvesty How can we show that a concept class s PAC Leanable? In ode to pove the PAC leanablty of a concept class we have to demonstate the exstence of a leanng algothm whch meets the necessay ctea specfed n the defnton of PAC leanablty. It s even bette f we can offe a constuctve poof that s, povde an algothm that meets the PAC ctea. It tuns out that we can often get away wth usng a athe dumb leanng algothm one that smply outputs a hypothess that s consstent wth the tanng examples. (We assume that H s expessve enough to guaantee the exstence of a consstent hypothess). Iowa State Unvesty PAC Leanablty of Fnte Concept Classes Defnton: A consstent leane s one that etuns some hypothess h H that s consstent wth a tanng set S of cadnalty m. Theoem: A consstent leane L s a PAC leane. That s, gven a suffcently lage numbe (m) of examples of c, the hypothess poduced by L s guaanteed, wth pobablty at least -δ, to be an ε-appoxmaton of c fo any choce of c C, any nstance dstbuton D, and any choce of ε, δ such that 0< ε, δ <. Specfcally, t suffces f H m > ln ε δ 23

24 Iowa State Unvesty V H S A consstent leane { h H h s wth examples n S }, = consstent S V L H,S h V H,S H V H,S Iowa State Unvesty Poof that a consstent leane s a PAC leane Poof sketch Thee ae two knds of hypothess n H, and hence n the veson space V H,S good (ε-appoxmatons of the taget concept) bad (not ε-appoxmatons of the taget concept). Gven a suffcently lage numbe of examples of a taget concept c, a suffcently lage facton of the bad hypotheses get elmnated fom the veson space mantaned by a consstent leane. Consequently, a andomly selected hypothess fom V H,S has a hgh pobablty (at least -δ) of beng an ε-appoxmaton of the taget concept 24

25 Iowa State Unvesty A consstent leane s a PAC leane Defnton: A veson space V H,S s sad to be ε-exhausted wth espect to an nstance dstbuton D and a concept c f evey hypothess h V H,S s an ε-appoxmaton of c. That s, h VH, S eoc, D( h) < Ou goal s to make the tanng set S lage enough to ensue that the pobablty that the veson space s not ε- exhausted wth espect to c and D s suffcently small (less than δ) egadless of the choce of c C and nstance dstbuton D by an advesay. Iowa State Unvesty A consstent leane s a PAC leane Theoem: Suppose H s a fnte hypothess space, and S a set of m (m ) examples of some c C. Then fo any ε (0< ε ), the pobablty that the veson space V H,S s not ε-exhausted wth espect to an nstance dstbuton D and a concept c s at most ε m H e Poof: Let H Bad be the subset of hypothess n V H,S that ae not ε- appoxmatons of c. h H, eo, ( h) ε Bad c D 25

26 Iowa State Unvesty A consstent leane s a PAC leane The pobablty that a hypothess h H Bad agees wth c on a andom nstance dawn accodng to D s at most ( ε ) The pobablty that a hypothess h H Bad s consstent m ndependently dawn andom examples s at most ( ε) m The pobablty that some hypothess n V H,S suvves m ndependently dawn andom examples s at most H Bad m m ( ε ) H ( ε ) snce H Bad H PAC leanng eques that the pobablty of L etunng a bad hypothess s small. That s, H m ( ε ) < δ Iowa State Unvesty A consstent leane s a PAC leane PAC leanng eques that the pobablty of L etunng a bad hypothess s small (at most δ). That s, H m ( ε ) δ ε ε { ε } ( 0 ) ( ) < e Hence, to ensue that a consstent leane s a PAC leane, t suffces that εm H e δ H εm e δ εm H e δ H εm ln δ m ln H + ln ε δ 26

27 Iowa State Unvesty Sample complexty of PAC Leanng fo fnte hypothess classes The smallest ntege m that satsfes the nequalty H m > ln ε δ s called the sample complexty of H. Iowa State Unvesty PAC- Easy and PAC-Had Concept Classes fo Consstent Leanes Conjunctve concepts ae easy to lean Use the same algothm as the one used n the mstake bound settng Sample complexty O N ln 3 + ln ε δ Tme complexty s polynomal n the elevant paametes of nteest The class of all Boolean concepts s had to lean (Why?) Remak: Polynomal sample complexty s necessay but not suffcent fo effcent (polynomal tme) PAC leanng poducng a consstent hypothess may be NP-Had 27

28 Iowa State Unvesty Repesentaton Dstncton between a concept and ts epesentaton A concept s smply a set of nstances extensonal defnton A epesentaton of a concept s a symbolc encodng of that set ntensonal defnton Example A concept can be epesented as a Boolean fomula φ, o a Boolean fomula ϕ that s logcally equvalent to φ, o a tuth table Iowa State Unvesty Repesentaton Dffeent epesentatons of the same concept may dffe adcally n sze Example Boolean paty functon f ( x, x2... xn ) = x x2... xn whee denotes the exclusve OR can be computed by a ccut of,, and gates whose sze s bounded by a fxed polynomal n n but a DNF (dsjuncton of conjunctons) epesentaton of the same functon has sze that s exponental n n. 28

29 Iowa State Unvesty Repesentaton A gven taget concept has many epesentatons The leane s oblvous to whch, f any, epesentaton s beng used by the teache o advesay to encode the taget concept Yet t mattes a geat deal whch of the many epesentatons of hypotheses that the leane chooses the sze of the epesentaton of a hypothess h s a lowe bound on the unnng tme of an algothm that outputs h Iowa State Unvesty Repesentaton A epesentaton scheme fo a concept class C s a functon R : Σ * Cwhee Σ s a fnte alphabet of symbols. * Any stng σ Σ such that R ( σ) = c s called a epesentaton of c unde R Thee may be many epesentatons fo a concept c unde epesentaton R When we need to use eal numbes to epesent concepts, we may allow * R : Σ R ( ) C 29

30 Iowa State Unvesty sze: Σ * ℵ Repesentaton sze R : Σ * C assgns a natual numbe sze(σ) to each epesentaton σ The esults obtaned unde a patcula defnton of sze ae meanngful only f the defnton s natual. Example Σ={0,} sze(σ) s the length of σ n bts If eal numbes ae used to encode a concept, we may chage one unt of sze to each eal numbe cannot tanslate ths measue of sze nto sze n bts unless the eal numbes ae fnte pecson Iowa State Unvesty Sze of a concept c unde a epesentaton R sze () c ( ) { sze( σ )} = mn R σ = c Sze of a concept c C unde a epesentaton scheme R fo C s the sze of the smallest epesentaton of c unde R The lage the value of sze(c), the moe complex the concept c unde the chosen epesentaton Fom now on, when we speak of leanng a concept class C, we wll mean leanng C unde a chosen epesentaton R 30

31 Iowa State Unvesty Sze of nstances { } n In a Boolean nstance space X the sze of n = 0, each nstance s n n In X n = R the sze of each nstance may be taken to be n (wth the usual caveat). n In X n = Α whee A s a fnte alphabet, the sze of an nstance s the length of the coespondng stng (wth maxmum sze beng n) Iowa State Unvesty Effcent (Polynomal Tme) PAC Leanng Defnton: Let C n be a concept class (actually a epesentaton class) ove X n. Let X = and n X n C s sad to be effcently PAC-leanable f C s PACleanable usng a leanng algothm L whch uns n tme that s polynomal n n (sze of the nstance epesentaton), sze(c) (sze of the epesentaton of the taget concept c), C = n C n δ and ε We assume that the leane s gven n and sze(c) as nput howeve, these assumptons can be elaxed 3

32 Iowa State Unvesty Effcent (Polynomal Tme) PAC Leanng Necessay: Sample complexty must be polynomal n the elevant paametes Suffcent: Polynomal sample complexty and a polynomal tme consstent leane Moe examples allowed to acheve lowe eo Moe examples allowed fo achevng hghe confdence Moe examples allowed fo leanng moe complex concepts Moe examples allowed fo leanng fom bgge nstances Iowa State Unvesty Conjunctve Concepts ae Effcently PAC Leanable Conjunctve concepts ae effcently PAC-leanable unde a natual epesentaton of conjunctons Sample complexty O ε N ln 3 + ln δ Tme complexty O ε N ln 3 + ln δ 32

33 Iowa State Unvesty 3-Tem DNF concepts ae not effcently PAC leanable unless P=RP Theoem: 3-tem DNF concept class (dsjunctons of at most 3 conjunctons) ae not effcently PAC-leanable usng the same hypothess class (although t has polynomal sample complexty) unless P=RP. Poof: By polynomal tme educton of gaph 3-coloablty (a well-known NP-complete poblem) to the poblem of decdng whethe a gven set of labeled examples s consstent wth some 3-tem DNF fomula. Iowa State Unvesty Tansfomng Had Poblems to Easy ones Theoem: 3-tem DNF concepts ae effcently PACleanable usng 3-CNF (conjuncton of dsjunctons (clauses) wth at most 3 lteals pe clause) hypothess class. Poof: 3 - tem DNF 3 - CNF Tansfom each example ove N boolean vaables nto a coespondng example ove N 3 vaables (one fo each possble clause n a 3-CNF fomula). T T2 T3 = ( u v w) u T, v T 2, w T The poblem educes to leanng a conjunctve concept ove the tansfomed nstance space. 3 33

34 Iowa State Unvesty Tansfomng Had Poblems to Easy ones Theoem Fo any k 2 k-tem DNF ae effcently PACleanable usng the k-cnf hypothess class. Remak: In ths case, enlagng the seach space by usng a hypothess class that s lage than stctly necessay, actually makes the poblem easy! Remak: No, we have not poved that P=NP. Summay: Conjunctve Easy k - temdnf k -CNF CNF Had Easy Had Iowa State Unvesty Occam Leanng Algothm Defnton: Let α 0 & 0 β < be constants. A leanng algothm L s sad to be an α β Occam algothm fo a concept class C usng a hypothess class H f L, gven a set S of m andom examples of an unknown concept c C outputs a hypothess h H such that h s consstent wth S and sze ( h) α { Nsze ( c) } m β Effectve hypothess space sze ( Nsze(c) ) α m β H 2 nm 34

35 Iowa State Unvesty Occam Leanng Algothm outputs succnct hypothess m α { Nsze ( c } sze ( h ) ) When >> N, m sze ( h ) = O α β ( sze ( c )) m ) m labels have to be compessed nto O(m)β bts A mld equement because we can always obtan a consstent hypothess that s O(mn) bts long (why?) We have to allow sze(h) to depend lnealy on sze(c) n the event the shotest hypothess n H may n fact be the taget concept c. We allow a geneous dependence on m whch often makes t ease to fnd a consstent hypothess fndng the shotest hypothess s often computatonally ntactable β Iowa State Unvesty Occam Leanng Algothm An Occam leanng algothm L fo a concept class C s sad to be an effcent α β Occam leanng algothm fo C f ts unnng tme s bounded by a polynomal n n, m, and sze(c). The smple algothm we consdeed fo leanng conjunctve concepts s an effcent Occam leanng algothm (Pove ths!). 35

36 Iowa State Unvesty Sample complexty of an Occam Algothm Theoem: An Occam algothm s guaanteed to be PAC f the numbe of samples m= O lg + ε δ ε ( Nsze() c ) α β Poof: Left as an execse. Iowa State Unvesty k-decson lsts k decson lst ove Boolean vaables x x N s an odeed sequence l = ( c, b )...( c, b b) ( l l ), Whee each c s a conjuncton of at most k lteals chosen fom x x N (and the negatons) and each b and b s 0 o. On a gven N-bt nput, l s evaluated lke a nested fthen-else statement wth b coespondng to the default output. 36

37 Iowa State Unvesty Occam algothm s PAC fo K-decson lsts Theoem: Fo any fxed k, the concept class of k-decson lsts s effcently PAC-leanable usng the same hypothess class. Algothm Geedly fnd conjunctons of at most k lteals that cove the lagest subset of examples wth the same class label. Remak: k-decson lsts consttute the most expessve Boolean concept class ove the Boolean nstance space {0,} N that ae known to be effcently PAC leanable. Iowa State Unvesty PAC Leanablty of Infnte Concept Classes How many andom examples does a leane need to daw befoe t has suffcent nfomaton to lean an unknown taget concept chosen fom a concept class C? Sample complexty esults deved pevously answe ths queston fo the case of fnte concept classes. Ae thee any non-tval nfnte concept classes that ae PAC leanable fom a fnte set of examples? Can we quantfy the complexty of an nfnte concept class? yes, usng Vapnk-Chevonenks Dmenson! 37

38 Iowa State Unvesty Vapnk-Chevonenks (VC) Dmenson Let C be a concept class ove an nstance space X. BothC and X may be nfnte. We need a way to descbe the behavo of C on a fnte set of ponts S X. S = { X, X 2... X m } Fo any concept class C ove X, and any S X, Π S = c S : c C Equvalently, wth a lttle abuse of notaton, we can wte C C ( ) { } ( S ) = {( c( X )... c( X )) c C} Π : Π C (S) s the set of all dchotomes o behavos on S that ae nduced o ealzed by C m Iowa State Unvesty Vapnk-Chevonenks (VC) Dmenson If Π ( ) { } m whee, o equvalently, C S = 0, S = m m Π ( ) = we say that S s shatteed by C. C S 2 A set S of nstances s sad to be shatteed by a hypothess class H f and only f fo evey dchotomy of S, thee exsts a hypothess n H that s consstent wth the dchotomy. 38

39 Iowa State Unvesty VC Dmenson of a hypothess class Defnton: The VC-dmenson V(H), of a hypothess class H defned ove an nstance space X s the cadnalty d of the lagest subset of X that s shatteed by H. If abtaly lage fnte subsets of X can be shatteed by H, V(H)= How can we show that V(H) s at least d? Fnd a set of cadnalty at least d that s shatteed by H. How can we show that V(H) = d? Show that V(H) s at least d and no set of cadnalty (d+) can be shatteed by H. Iowa State Unvesty VC Dmenson of a Hypothess Class - Examples Example: Let the nstance space X be the 2-dmensonal Eucldan space. Let the hypothess space H be the set of lnea -dmensonal hypeplanes n the 2-dmensonal Eucldan space. Then V(H)=3 (a set of 3 ponts can be shatteed by a hypeplane as long as they ae not co-lnea but a set of 4 ponts cannot be shatteed). Fo the concept class of lnea hypeplanes, VC dmenson s n+ 39

40 Iowa State Unvesty VC Dmenson and Sample complexty A concept class C 2 X s tval f t contans a sngle concept o 2 dsjont concepts whch patton X. Theoem: Let C be a non tval concept class. Then C s PAC leanable f and only f V(C) s fnte. If V(C)=d and d<, then the bounds on sample complexty of C ae gven by d m = Ο lg + lg ε δ ε ε d m = Ω ε Poof: See Readngs Iowa State Unvesty Some Useful Popetes of VC Dmenson ( C C ) V ( C ) V ( C ) If ( C = { X c : c C} ) V ( C) = V ( C ) ( C = C C ) V ( C) V ( C ) + V ( C ) If C If V C s a fnte concept class, V ( C) lgc l concepts fom C, V ( C ) ( C) = d, Π ( m) = max{ Π ( S) : S = m}, Π Φ s fomed by a unon o ntesecton of l C d 2 2 C ( m) Φ ( m) m d ( m) = 2 f m d and Φ ( m) = O( m ) f m < d Poof: Left as an execse d 2 l whee = O( V( C) l lgl) C d

41 Iowa State Unvesty Sample complexty of a multlaye pecepton Acyclc, layeed mult-laye netwoks of s theshold logc unts, each wth nputs, has VC dmenson d = O( + ) s lg( s) Hence, we have: d m = Ο lg + lg ε δ ε ε + s = O lg + slg ε δ ε ε Iowa State Unvesty Leanng when the sze of the taget concept s unknown Results on effcent PAC leanablty of concept classes ae deved unde the assumpton that the sze of the taget concept s one of the nputs to the leanng algothm Can we guaantee effcent PAC leanablty when the sze of the taget concept s unknown? Yes, usng the doublng tck and hypothess testng 4

42 Iowa State Unvesty Hoffdng Bounds Let X.X m be outcomes of ndependent Benoull tals each wth pobablty of success p. Let S = m = X So E ( S ) = pm P ( S pm + t ) e 2 mt 2 P 2 m ( α p ) ( S αm ) e 2 whee α p P 2 m ( α p ) ( S αm ) e 2 whee α p Iowa State Unvesty Chenoff Bounds Let X.X m be ndependent outcomes of ndependent Benoull tals each wth pobablty of success p. Let LE GE S ( p, m, ( α) pm) ( p, m, ( + α) pm) = m = LE ( p, m, ) = P and ( S ) GE ( p, m, ) = P ( S ) e e X 2 α mp / 2 2 α mp / 3 0 α Chenoff Bounds ae tghte than Hoffdng Bounds when p < /4 42

43 Iowa State Unvesty How to detemne f a hypothess s ε-good We cannot dstngush wth cetanty between an ε-good hypothess and one that has eo slghtly geate than ε by testng the hypotheses on a fnte set of examples Howeve, we can dstngush between an (ε/2)-good hypothess and an ε-bad hypothess wth hgh confdence Iowa State Unvesty How to detemne f a hypothess s ε-good Algothm Test (h,n,ε,δ). Make 32 2 m = nln2 + ln calls to Example(c,D) ε δ (n s the sze of the nstances) 2. Accept h f t msclassfes at most 3ε m examples; Othewse, eject h 4 Test (h,n,ε,δ) has the popety: If eo If eo c, D c, D ( h) ε, then P( h s accepted) ε 2 δ n 2 + δ n 2 ( h), then P( h s ejected) + 43

44 Iowa State Unvesty Leanng when the sze of the taget concept s unknown A eques the taget concept sze as a paamete B woks fo an unknown taget concept sze Iowa State Unvesty Leanng n the pesence of nose Types of nose Random msclassfcaton nose Random attbute nose unfom, non unfom Malcous nose examples selected and coupted by an omnpotent advesay who may have access to the ntenal state of the leane 44

45 Iowa State Unvesty Leanng n the pesence of andom msclassfcaton nose Random msclassfcaton nose wth pobablty η the nstance s coectly labeled. Wth pobablty (- η ), the label s flpped Example (x, c(x)) Example η Concept c C Tanng Samples x D η (-η ) ( x,c( x) ) ( x,c( x) ) Iowa State Unvesty Leanng n the pesence of andom msclassfcaton nose Assume WLOG that 0 η η 0 </2 Daw m ε C ln ( ) 2 2η δ 0 examples fom Example η Output a hypothess h C that mnmzes the tanng eo The method can be adapted to the case of unknown η 0 Altenatve methods ae avalable fo specfc concept classes 45

46 Iowa State Unvesty PAC leanng usng weak leanes Weak leane Confdence lowe than (-δ ) Boost confdence Eo geate than ε Boost accuacy Eo geate than ε and Confdence lowe than (-δ ) Boost accuacy and confdence We can tun weak leanes nto stong (PAC) leanes usng accuacy and confdence boostng algothms Iowa State Unvesty Confdence Boostng Run the algothm seveal tmes on ndependently dawn tanng sets to obtan a set of hypotheses The numbe of ndependent uns s chosen to be lage enough to ensue that the pobablty that at least one of the esultng hypothess has eo less than ε s at least (-δ/2) Use hypothess testng to select the best hypothess n the pool wth hgh confdence altenatvely use weghted majoty classfcaton 46

47 Iowa State Unvesty Accuacy Boostng Lean a sequence of hypotheses The fst hypothess s based on the ognal tanng set Each subsequent hypothess s based on a samplng of the tanng set accodng to a dstbuton whch assgns hghe pobablty to tanng examples that wee msclassfed by the pevously leaned hypotheses and pehaps a dffeent eo paamete Classfcaton s based on majoty o weghted majoty of the hypotheses Moe on Accuacy Boostng Late.. Iowa State Unvesty Leanng unde helpful dstbutons PAC Leanng eques success unde all possble pobablty dstbutons Some concept classes ae had to lean unde all dstbutons e.g., egula languages o detemnstc fnte state automata (DFA), yet they ae eadly leaned by humans Queston can natual settngs be modeled by moe bengn o helpful dstbutons? E.g., can DFA be leaned unde helpful dstbutons? What pecsely ae helpful dstbutons? 47

48 Iowa State Unvesty Dgesson Kolmogoov Complexty Kolmogoov complexty K (x ) s a machne ndependent.e. unvesal measue of the complexty of descpton of an object K (x ) = the numbe of bts n the shotest unvesal Tung machne pogam fo x Example Object = (0) 500 Pogam Pnt tmes Example 2 Object (andom stng) Pogam Pnt Smple objects have low Kolmogoov complexty Iowa State Unvesty Unvesal dstbuton We fx a unvesal Tung machne U { length ( π ) ( π = α } K ( α ) = mn U ) π K ( α β ) = mn { length ( π ) U ( π, β ) = α } π K ( α β ) K ( α ) Unvesal dstbuton M assgns hghe pobabltes to smple objects M K ( x ) ( x) 2 M K ( x α ) ( x α ) 2 48

49 Iowa State Unvesty Leanng Unde Unvesal dstbuton Unvesal dstbuton M multplcatvely domnates all enumeable dstbutons Enumeable dstbutons nclude fnte pecson Posson, Gaussan, and many othe dstbutons Theoem: A concept class s Pobably appoxmately Leanable unde each enumeable dstbuton ff t s Pobably appoxmately leanable unde the unvesal dstbuton assumng dung leanng examples ae dawn accodng to M (x) Iowa State Unvesty Leanng unde unvesal dstbuton L and Vtany (99) showed that log n -tem DNF ae leanable unde M (x c) whee c s the taget concept Paekh and Honava (999, 200) showed that Smple DFA (wth encodng of sze O ( log N ) whee N s the numbe of states) ae effcently leanable unde the unvesal dstbuton M (x) DFA ae effcently leanable wth a helpful teache examples ae dawn accodng to M (x c ) whee c s the taget concept Dens (200) showed that DFA ae effcently leanable fom postve examples alone unde M (x c ) 49

50 Iowa State Unvesty Addtonal Possbltes PAC leanng model assumes that taget concepts ae selected unfomly at andom fom C Bengn teache How about f taget concepts ae selected accodng to unvesal dstbuton ove the concept class, namely M (c)? Occam Leane Impose a pefeence bas ove the set of consstent hypotheses Select hypothess h accodng to M ( h ) Bayesan leane Assume pos gven by M ( h ) Iowa State Unvesty Summay of Man Results n Dstbuton- Independent Leanng Theoy PAC-Easy leanng poblems lend themselves to a vaety of effcent algothms. PAC-Had leanng poblems can often be made PAC-easy though appopate nstance tansfomaton and choce of hypothess space Occam's azo often helps Weak leanng algothms can be tuned nto stong PAC leanes though accuacy and confdence boostng Leanng unde estcted classes of nstance dstbutons (e.g., unvesal dstbuton) and pos offes new possbltes 50