6.867 Machine learning, lecture 14 (Jaakkola)

Transcription

1 6.867 Machie learig, lecture 14 (Jaakkla) 1 Lecture tpics: argi ad geeralizati liear classifiers esebles iture dels Margi ad geeralizati: liear classifiers As we icrease the uber f data pits, ay set f classifiers we are csiderig ay lger be able t label the pits i all pssible ways. Such eergig cstraits are critical t be able t predict labels fr ew pits. This tivates a key easure f cpleity f the set f classifiers, the Vapik-Chervekis diesi. The VC-diesi is defied as the aiu uber f pits that a classifier ca shatter. The VC-diesi f liear classifiers the plae is three (see previus lecture). Nte that the defiiti ivlves a aiu ver the pssible pits. I ther wrds, we ay fid less tha d V C pits that the set f classifiers cat shatter (e.g., liear classifiers with pits eactly a lie i 2 d) but there cat be ay set f re tha d V C pits that the classifier ca shatter. The VC-diesi f the set f liear classifiers i d diesis is d+1, i.e., the uber f paraeters. This relati t the uber f paraeters is typical albeit certaily t always true (e.g., the VC-diesi ay be ifiite fr a classifier with a sigle real paraeter!). The VC-diesi iediately geeralizes ur previus results fr budig the epected errr fr a fiite uber f classifiers. There are a uber f techical steps ivlved that we w t get it, hwever. Lsely speakig, d V C takes the place f the lgarith f the uber f classifiers i ur set. I ther wrds, we are cutig the uber f classifiers the basis f hw they ca label pits, t based their idetities i the set. Mre precisely, we have fr ay set f classifiers F: with prbability at least 1 δ ver the chice f the traiig set, R(f) R (f) + ɛ(, d V C, δ), uifrly fr all f F (1) where the cpleity pealty is w a fucti f d V C = d V C (F): d V C (lg(2/d V C ) + 1) + lg(1/(4δ)) ɛ(, d V C, δ) = (2) Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

2 6.867 Machie learig, lecture 14 (Jaakkla) 2 The result is prbleatic fr kerel ethds. Fr eaple, the VC-diesi f kerel classifiers with the radial basis kerel is. We ca, hwever, icrprate the ti f argi i the classifier diesi. Oe such defiiti is V γ diesi that easures the VC-diesi with the cstrait that distict labeligs have t be btaied with a fied argi γ. Suppse all the eaples fall withi a eclsig sphere f radius R. The, as we icrease γ, there will be very few eaples we ca classify i all pssible ways with this cstrait (especially whe γ R; cf. Figure 1). Put ather way, the VCdiesi f a set f liear classifiers required t attai a prescribed argi ca be uch lwer (decreasig as a fucti f the argi). I fact, V γ diesi fr liear classifiers is buded by R 2 /γ 2, i.e., iversely prprtial t the squared argi. Nte that this result is idepedet f the diesi f iput eaples, ad is eactly the istake bud fr the perceptr algrith! Figure 1: The set f liear classifiers required t btai a specific geetric argi has a lwer VC-diesi whe the eaples reai withi a eclsig sphere. The previus geeralizati guaratees ca be used with V γ diesi as well s lg as we replace the traiig errr with argi vilatis, i.e., we cut the fracti f eaples that cat be separated with argi at least γ. Margi ad geeralizati: esebles A eseble classifier ca be writte as a cve cbiati f sipler base classifiers h () = α j h(; θ j ) (3) j=1 Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

3 6.867 Machie learig, lecture 14 (Jaakkla) 3 where α j 0 ad j=1 α j = 1. Bstig geerates such esebles but des t ralized the cefficiets α j t su t e. The ralizati ca be easily perfred after the fact. We are iterested i uderstadig the cpleity f such esebles ad hw geeralizati guaratees depeds the vtig argi achieved, e.g., thrugh bstig algrith. Nte that ur discussi here will t refer t hw such esebles are geerated. Let s start by defiig what the esebles are t. They are t decisi trees. A decisi (classificati) tree is a ethd f recursively partitiig the set f eaples it regis such that withi each regi the eaples wuld have as uifr labels as pssible. The partitiig i a decisi tree culd be based the sae type f decisi stups as we have used fr the eseble. I the eseble, hwever, the dai fr all the stups is the whle space. I ther wrds, yu cat restrict the applicati f the stup withi a specific partiti. I the eseble, each stup ctributes t the classificati f all the eaples. Hw pwerful are esebles based the decisi stups? T uderstad this further let s shw hw we ca shatter ay pits with 2 stups eve i e diesis. It suffices t shw that we ca fid a eseble with 2 stups that reprduces ay specific labelig y 1,..., y f pits 1,..., (w real ubers). T d s, we will cstruct a eseble f tw stups t reprduce the label y t fr t but withut affectig the classificati f ther traiig eaples. If ɛ is less tha the sallest distace betwee ay tw traiig eaples, the 1 1 h pair (; t, y t ) = sig (y t ( t + ɛ)) + sig ( y t ( t ɛ)) (4) 2 2 has value y t withi iterval [ t ɛ, t + ɛ] is zer everywhere else. Thus, settig α t = 1/, h 2 () = α t h pair (; t, y t ) (5) t=1 has the crrect sig fr all the traiig eaples. The eseble f 2 cpets therefre has VC-diesi at least. Esebles are pwerful as classifiers i this sese ad their VC-diesi prly eplais their success i practice. Each eaple i the abve cstructi ly has a very lw vtig argi 1/, hwever. Perhaps we ca siilarly refie the aalysis t icrprate the vtig argi as we did abve with liear classifiers ad the geetric argi. The key idea is t reduce a eseble with ay cpets t a carse eseble with few cpets but e that evertheless classifies the eaples i the sae way. Whe the rigial eseble achieves a large vtig argi this is ideed pssible, ad the size f the carse appriati that Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

4 6.867 Machie learig, lecture 14 (Jaakkla) 4 we eed decreases with icreasig vtig argi. I ther wrds, if we achieve a large vtig argi, we culd have slved the sae classificati prble with uch saller eseble isfar as we ly pay atteti t the classificati errr. Based this ad ther re techical ideas we ca shw that with prbability at least 1 δ ver the chice f the traiig data, ( ) d V C /ρ R (ĥ) R (ĥ; ρ) + Õ 2, (6) where the Õ( ) tati hides cstats ad lgarithic ters, R (ĥ; ρ) cuts the uber f traiig eaples with vtig argi less tha ρ, ad d V C is the VC-diesi f the base classifiers. Nte that the result des t deped the uber f base classifiers i the eseble ĥ. Nte als that the effective diesi d V C /ρ 2 that the uber f traiig eaples is cpared t has a siilar fr as befre, decreasig with the argi ρ. See Schapire et al. (1998) fr details. The paper is available fr the curse website as ptial readig. Miture dels There are ay prbles i achie learig that are t siple classificati prbles but rather delig prbles (e.g., clusterig, diagsis, cbiig ultiple ifrati surces fr sequece atati, ad s ). Mrever, eve withi classificati prbles, we fte have ubserved variables that wuld ake a differece i ters f classificati. Fr eaple, if we are iterested i classifyig tissue saples it specific categries (e.g., tur type), it wuld be useful t kw the cpsiti f the tissue saple i ters f cells that are preset ad i what prprtis. While such variables are t typically bserved, we ca still del the ad ake use f their presece i predicti. Miture dels are siple prbability dels that try t capture abiguities i the available data. They are siple, widely used ad useful. As the ae suggests, a iture del ies differet predictis the prbability scale. The iig is based alterative ways f geeratig the bserved data. Let be a vectr f bservatis. A iture del ver vectrs is defied as fllws. We assue each culd be f pssible types. If we kew the type, j, we wuld del with a cditial distributi P ( j) (e.g., Gaussia with a specific ea). If the verall frequecy f type j i the data is P (j), the the Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

5 6.867 Machie learig, lecture 14 (Jaakkla) 5 iture distributi ver is give by P () = P ( j)p (j) (7) j=1 I ther wrds, culd be geerated i pssible ways. We iagie the geerative prcess t be as fllws: saple j fr the frequecies P (j), the fr the crrespdig cditial distributi P ( j). Sice we d t bserve the particular way the eaple was geerated (assuig the del is crrect), we su ver the pssibilities, weighted by the verall frequecies. We have already ecutered iture dels. Take, fr eaple, the Naive Bayes del P ( y)p (y) ver the feature vectr ad label y. If we pl tgether eaples labeled +1 ad thse labeled 1, ad thrw away the label ifrati, the the Naive Bayes del predicts feature vectrs accrdig t [ ] d P () = P ( y)p (y) = P ( i y) P (y) (8) y=±1 y=±1 i=1 I ther wrds, the distributi P () assues that the eaples ce i tw differet varieties crrespdig t their label. This type f ubserved label ifrati is precisely what the itures ai t capture. Let s start with a siple tw cpet iture f Gaussias del (i tw diesis): P ( θ) = P (1)N(; µ 1, σ 1 2 I) + P (2)N(; µ 2, σ 2 2 I) (9) The paraeters θ defiig the iture del are P (1), P (2), µ 1, µ 2, σ 2 2 1, ad σ 2. Figure 2 shws data geerated fr such a del. Nte that the frequecies P (j) (a.k.a. iig prprtis) ctrl the size f the resultig clusters i the data i ters f hw ay eaples they ivlve, µ j s specify the lcati f cluster ceters, ad σ 2 j s ctrl hw spread ut the clusters are. Nte that each eaple i the figure culd i priciple have bee geerated i tw pssible ways (which iture cpet it was sapled fr). There are ay ways f usig itures. Csider, fr eaple, the prble f predictig fial ea scre vectrs fr studets i achie learig. Each bservati is a vectr with cpets that specify the pits the studet received i a particular questi. We wuld epect that differet types f studets succeed i differet types f questis. This studet type ifrati is t available i the ea scre vectrs, hwever, but we ca del it. Suppse there are studets takig the curse s that we have vectr Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

6 6.867 Machie learig, lecture 14 (Jaakkla) 6 Figure 2: Saples fr a iture f tw Gaussias bservatis 1,...,. Suppse there are uderlyig types f studets (the uber f types ca be iferred fr the data; this is a del selecti prble). We als d t kw hw ay studets takig the curse are f particular type, i.e., we have t estiate the iig prprtis P (j) as well. The iture distributi ver a sigle eaple scre vectr is w give by P ( θ) = P ( j)p (j) (10) j=1 We w t ccer urselves at this pit with the prble f decidig hw t paraeterize the cditial distributis P ( j). Suffice it t say that it wuld t be ureasable t assue that P ( j) = d i=1 P ( i j) as i the Naive Bayes del but each i wuld take values i the rage f scres fr the crrespdig ea questi. Nw, ur iture del assues that each studet is f particular type. If see gave us this ifrati, i.e., j t fr t, the we wuld del the bservatis with the cditial distributi P ( 1,..., j 1,..., j, θ) = P ( t j t ) (11) assuig each studet btais their scre idepedetly fr thers. But the type ifrati is t preset i the data s we will have t su ver the pssible values f j t fr each studet, weighted by the prir prbabilities f types, P (j t ) (sae fr all studets): [ ] [ ] P ( 1,..., θ) = P ( t j t )P (j t ) = P ( t j)p (j) (12) t=1 j t=1 t=1 t=1 j=1 Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].

7 6.867 Machie learig, lecture 14 (Jaakkla) 7 This is the likelihd f the bserved data accrdig t ur iture del. It is iprtat t uderstad that the del wuld be very differet if we echaged the prduct ad the su i the abve epressi, i.e., defie the del as [ ] P ( 1,..., θ) = P ( t j) P (j) (13) j=1 t=1 This is als a iture del but e that assues that all studets i the class are f specific sigle type, we just d t kw which e, ad are suig ver the pssibilities (i the previus del there were pssible assigets f types ver studets). Cite as: Ti Jaakkla, curse aterials fr Machie Learig, Fall MIT OpeCurseWare ( Massachusetts Istitute f Techlgy. Dwladed [DD Mth YYYY].