Tasty Coffee example

Transcription

1 Lecure Slides for (Binary) Classificaion: Learning a Class from labeled Examples ITRODUCTIO TO Machine Learning ETHEM ALPAYDI The MIT Press, 00 (modified by dph, 0000) CHAPTER : Supervised Learning Things represened by a feaure vecor x and a label r (also called y), ofen r in {,0} or {,} Domain D is se of all possible feaure vecors A Hypohesis (someimes called a Concep) is a pariioning of he Domain ino and regions (or he region, or funcion from domain X o {,}) A Hypohesis Class H is a se of hypoheses (someimes called Concep Class) alpaydin@boun.edu.r hp:// Assumpion: iid Examples Tasy Coffee example More Terminology Disribuion of hings and measuremens defines some unknown (bu fixed) P(x) on domain D Targe concep C gives he correc labels, C(x), as a funcion of he feaures, x Find an h in H from examples ha is close o C A loss funcion l(r, r ) measures error of predicions, in classificaion usually: l(r,r )=0 if r=r and l(r,r )= oherwise Wan o minimize P(x) l(c(x), h(x)) probabiliy of error for usual loss Objecs are cups of coffee Measure srengh and sugar Each measuremen is a feaure or aribue Oher feaures (cream, emperaure, roas) Feaures numeric (precision Accuracy) Label (or class) is (asy) or (no) Example is (x,r) pair, x in R, y in{,} Domain: se of all possible x vecors Concep: a boolean funcion on domain, a mapping from x s o and 0 ; or T and F ; or and, or a subse of he domain Targe: he concep o be learned Hypohesis class/space: is he se of hypoheses (conceps) ha can be oupu by a given learning algorihm

2 Srengh and sugar measured 0 o 0 Domain has differen insances How o predic from hese examples srengh sugar 0 label Version space: all conceps in hypoheses space consisen wih raining se. If hypoh. Space is all conceps, hen version space evenly spli on every unseen insance eed inducive bias (smaller hypohesis space), oherwise generalizaion is hopeless Assume asy coffee is a recangle in R Recangles in R are conceps C for which here exiss c,c,c,c so ha C(x) = iff c! x! c and c! x! c Hypohesis class H of recangles # if h classifies x as posiive h( x) = "! 0 if h classifies x as negaive Error of h on daa X! ( h( )" r ) E ( h X ) = x = Bad use of ( ) 7 9 0,0 0,0 0,0 srengh sugar label srengh sugar label srengh sugar label s u g a r 0,0 Srengh 0 s u g a r 0,0 Srengh 0 s u g a r 0,0 Srengh 0 0

3 Key inerplay Key inerplay Triple TradeOff Underlying paern being learned Feaures available Hypohesis space umber of examples available The rick is finding he righ mix, bu Underlying paern being learned Feaures available Hypohesis space umber of examples available The rick is finding he righ mix, bu The smaller he hypohesis space, he luckier we have o be o cach he paern There is a radeoff beween hree facors (Dieerich, 00): Complexiy of H, c (H), Training se size,, Generalizaion error, E, on new daa As, E As c (H), firs E and hen E Model Selecion & Generalizaion Example Example Learning is an illposed problem; daa is no sufficien o find a unique soluion The need for inducive bias, assumpions abou H Generalizaion: How well a model performs on new daa Wha we are really inersed in! Overfiing: H more complex han arge C or f Underfiing: H less complex han arge C or f o a recangle in x,x How o make i a recangle o a recangle in x, x How o make i a recangle I is a recangle in hree dimensions: x, x, and x *x 7

4 VC Dimension Shaering VC Dimension oise poins can be labeled in ways as / H shaers a se if: for each labeling of he se here is an h H consisen wih he labeling VC(H ) = size of a larges shaered se An axisaligned recangle shaers poins only! VapnikChervonenkis dimension is a measure of hypohesis space capaciy VCdim of recangles in in plane is PAC (Probably approximaely correc) bounds: Hypohesis consisen wih O(VCdim ln(/a) / a) examples usually has error a Daa no always perfec Aribue noise Label noise oise can model hypohesis space approximaions arge Domain 9 0 oise Daa no always perfec Aribue noise Label noise oise can model hypohesis space approximaions arge oise Daa no always perfec Aribue noise Label noise oise can model hypohesis space approximaions Targecircle hypohesisrecangles Muliple Classes, C i i=,...,k X = {x,r } = h! $ if x & Ci ri = #!" 0 if x & C j, j % i Train hypoheses h i (x), i =,...,K:! $ ( x ) = #!" i if x & C 0 if x & C, j % i i j Domain Domain

5 Regression X = { x,r } = r! " r = f x ( ) = & E g X ( ) # E w,w 0 X = ( ) " # r! g x $ % ( ) = & = ( x) = wx w0 g ( x) = wx wx w0 g ( ) " r! w x w $ # 0 % Esimaing Errors To esimae generalizaion error, we need daa unseen during raining. We spli he daa as Training se (0%) Validaion se (%) (is raining good) Tes (publicaion) se (%) Supervised Learning as parameer esimaion. Model (hypohesis class) : g( x!). Loss funcion: E! X. Opimizaion procedure: ( ( )) ( ) = " L r, g x!!* " arg min E! X! ( ) 7 Feaure selecion reduces he dimensionaliy (number of feaures) used o describe he insances Why Reduce Dimensionaliy Reduces ime complexiy: Less compuaion Reduces space complexiy: Less parameers Saves he cos of observing he feaure Simpler models are more robus on small daases More inerpreable; simpler explanaion Daa visualizaion (srucure, groups, ouliers, ec) if ploed in or dimensions Feaure Selecion vs Exracion Feaure selecion: Choosing k<d imporan feaures, ignoring he remaining d k Subse selecion algorihms Feaure exracion: Projec he original x i, i =,...,d dimensions o new k<d dimensions, z j, j =,...,k Principal componens analysis (PCA), linear discriminan analysis (LDA), facor analysis (FA) (also cluseringbased approaches) 9 0

6 Feaure Ranking (see GuyonElisseeff) Find feaures wih a high score : Correlaion wih labels (regression) Predicive power of aribue (aribue classifier) Muual informaion beween labels and arges) Relaively quick and simple Subse Selecion There are d subses of d feaures Forward search: Add he bes feaure a each sep Se of feaures F iniially Ø. A each ieraion, find he bes new feaure j = argmin i E ( F x i ) Add x j o F if E ( F x j ) < E ( F ) This is Hillclimbing O(d ) O(dk) runs of algorihm (o pick k feaures) Backward search: Sar wih all feaures and remove one a a ime, if possible. Floaing search (Add k, remove l) onparameric echnique: eares eighbor Does no fi a parameerized model Variable hypohesis complexiy From daa lazy Hypohesis (no gradien descen, opimizaion, or search) (under \lazy in Weka) eares eighbor Algorihm Insances (x s) are vecor of real Sore he n raining examples (x, y ),, (x n, y n ) To predic on new x, find x i closes o x and predic wih y i Commens: o jus simple able lookup Can avoid by minimizing squared disance Decision Boundaries Vornoi diagram, very flexible, ges more complicaed wih addiional poins eares eighbor Applicaions Asronomy (classifying objecs) Medicine diagnosis Objec deecion Characer recogniion (shape maching) Many ohers (basic heory from90 s and 0 s)

7 7 Disance meric imporan Irrelevan aribue example Irrelevan aribue example Consider expensive houses wih feaures: umber of bedrooms ( o ) Lo size in acres (/ o / plus ail) House square fee (00 o 000) Difference in square fee dominaes Irrelevan aribues (e.g. how far away was owner born ) add variabiliy Correlaed aribues also bad Le x [0,] deermine class, y= iff x > 0. Consider predicing on (0,0) given daa (0., x ) labeled 0 (0., x ) labeled where x, x random draws from [0,] (draw picure) wha is s error rae Le x [0,] deermine class, y= iff x > 0. Consider predicing on (0,0) given daa (0., x ) labeled 0 (0., x ) labeled where x, x random draws from [0,] Chance of error ~ %! 7 9 Some ricks Rescale aribues o mean 0 variance Use w j on j h componen: Dis(x, x ) = j w j (x j x j ) w j = I(x j, y) ( muual informaion ) Mahalanobis Disance (covariance Σ) Dis(x,x )= (xx ) T Σ (xx ) A Curse of Dimensionaliy As number of aribues (d) goes up so does volume Consider 000 raining poins in [0,] d where does each poin predic When d=, inerval per poin ~0.00 When d=, area per poin ~0.00, lengh of side abou 0.0 When d=0, volume per poin ~0.00, lengh of side ~ 0. eed exponenially many poins (in d) o ge good coverage Kd rees Grealy speed up finding neares neighbor Like binary search ree, bu organized around dimensions Each node ess single dimension agains hreshold (median) Can use highes variance dimension or cycle hrough dimensions Growing a good Kd ree can be expensive 0

8 oise can cause problems oise example Kneares neighbor Assume ha rue labels always, bu noise randomly corrups labels 0% of he ime (making hem 0) Bayes opimal: predic, es error is 0% eares eighbor: use closes raining poin, 90% of he ime predic, 0% of hese predicions wrong 0% of he ime predic 0, 90% of hese predicions wrong Overall wrong % of he ime Algorihm: Find he closes k poins and predic wih heir majoriy voe K is Bayes opimal in limi as k and raining se size go o (known since 90 s) Edied Key Idea: Reduce memory and compuaion by only soring imporan poins Heurisic: Discard hose poins correcly prediced by ohers (or ake incorrecly prediced poins) Remaining poins concenraed on he decision boundary Finding a smalles subse of poins correcly labeling ohers is Pcomplee.