Linear discriminants. Nuno Vasconcelos ECE Department, UCSD

Transcription

1 Lnear dscrmnants Nuno Vasconcelos ECE Department UCSD

2 Classfcaton a classfcaton problem as to tpes of varables e.g. X - vector of observatons features n te orld Y - state class of te orld X R 2 fever blood pressure Y {dsease no dsease} X Y related b a unknon functon f. f goal: desgn a classfer : X Y suc tat t f 2

3 Loss functons and Rsk usuall. s parametrc α and cannot appromate f. arbtrarl ell tere s a loss L[ α] goal: to fnd te set of parameters tat mnmzes te epected value of te loss or rsk { L[ } R α E X Y α] P Y X L[ α] dd classfcaton: - loss s common f α L [ α ] f α 3

4 Loss functons and Rsk because ] [ ] [ α α α P P R + Q t t f t t t t k? ] [ ] [. ] [. α α α α P P P R Y X Y X Y X + Q: at s te functon tat mnmzes te rsk? snce { } { } { } L E E L E R X Y X Y X ] [ mn ] [ mn * te optmal functon s { } L E ] [ * 4 { } L E X Y ] [ arg mn *

5 Baes classfer under te - loss ts becomes [ ] * arg mn P arg mn Y X arg ma [ ] P P Y X Y X [ ] and snce s n a dscrete set * arg map Y X [ ] te optmal decson s to pck te class of largest posteror probablt ts s te Baes decson rule Baes classfer 5

6 Baes decson rule t carves up te observaton space X assgnng g a label to eac regon clearl * depends on te class denstes * arg mapy P arg ma X arg map [ ] [ ] PY [ ] PX [ ] [ ] P [ ] X Y X Y [ ] + log P [ ] { log P } arg ma + X Y Y Y 6

7 Baes decson rule ts s problematc snce e don t kno at tese denstes are n SLI e ave seen tat denst estmaton s a trck busness ke dea of dscrmnant learnng: estmatng te denstes to ten derve te boundar s a bad strateg denst estmaton s an ll-posed problem slgt cange n problem condtons can lead to arbtrarl large cange n te soluton denst estmaton alas as an nfnte number of solutons tnk of a Gaussan as a mture of Gaussans Vapnk s rule: en solvng a problem avod solvng a more general problem as an ntermedate step! 7

8 Dscrmnant learnng ork drectl t te decson functon postulate a parametrc faml of decson boundares pck te element n ts faml tat produces te best classfer Q: at s a good faml of decson boundares? to get some nsgt let s stck t te BDR a lttle longer assume e ave to Gaussan classes equal covarance equal probablt P Y ½ n {}. notaton: a Gaussan of mean and covarance s G d 2 π e 2 8

9 Dscrmnant learnng for to equal probablt Gaussans of equal covarance [ ] [ ] { } log log arg ma * Y X Y G P P + { } arg mn 2 log log arg ma G + c means { } ag f f > < * 9

10 Lnear dscrmnants te decson boundar s te set of ponts c after some algebra becomes ts s te equaton of te per-plane 2 + t + b 2 b and e ave a lnear dscrmnant

11 Lnear dscrmnants te per-plane equaton can also be rtten as + b + 2 b t bb 2 n 2 3 geometrc nterpretaton plane of normal tat passes troug

12 Lnear dscrmnants under ts notaton te decson functon f < b f f > < * becomes < > f f * g g - t < f g g θ.cosθ. g n 2 2 g > f s on te sde ponts to ponts to te postve sde 3 2

13 Lnear dscrmnants fnall note tat s g te projecton of - onto te unt vector n te drecton of lengt of te component of - ortogonal to te plane n 2 3 b - θ g.e. g/ s te perpendcular dstance from to te plane smlarl b/ s te dstance from te plane to te orgn snce bb 2 3

14 Geometrc nterpretaton n summar te decson rule f g > * f g < as te propertes t t dvdes X nto to alf-spaces boundar s te plane t: normal dstance to te orgn b/ g/ s te dstance from pont to te boundar g for ponts on te plane g g > on te sde ponts to postve sde g < on te negatve sde + b b g 4

15 Lnear dscrmnants s ts a good decson functon? just seen t s optmal for Gaussan classes equal class probablt and covarance sounds too muc as a to problem also optmal f data s lnearl separable tere s a plane c as all s on one sde all s on te oter note: ts oldng on te tranng set onl guarantees optmalt on te ERM sense se not n te sense of mnmzng te true rsk 5

16 Alternatves use a ger-order decson functon e.g. a quadratc boundar W + + s te optmal soluton for an Gaussan problem 2 Gaussan classes no constrants looks lke e are gong to need a ver g-order polnomal l n general! lots of parameters too muc complet ere to stop? can e do sometng else to keep te smplct of te lnear boundar? 6

17 Alternatves 2 transform te space: ntroduce a mappng Φ: X Z 2 o o o o o o o o o o o o suc tat dmz > dmx Φ learnng a lnear boundar n Z s equvalent to learnng a non-lnear boundar n X e.g. to scalar Gaussans zero mean dfferent varances 2 3 n o o o o o o o o o o o o 7

18 Feature transformaton snce P usng te BDR leads to ts G σ X Y * arg map [ ] P [ ] X Y cannot be mplemented t a lnear dscrmnant but becomes feasble b mappng to 2D Y Φ Φ : 2 R R R 2 R 2 8

19 Feature transformaton note tat te problem as not reall canged e stll ave a D set but no embedded n a 2D space a lot more space: e can alas arrange tngs so tat te boundar s lnear te BDR tself tells us o to do ts but once agan requres te denstes easer as te dmz gros usuall feasble n te ERM sense as dmz te problem s tat evaluatng Φ becomes arder and arder 9

20 Back to lnear dscrmnants e ll see o to do ts kernels for no te goal s to eplore te smplct of te lnear dscrmnant let s assume lnear separablt one and trck s to use {-} nstead of {} ere for ponts on te postve sde - for ponts on te negatve sde te decson functon becomes - f g > * * sgn g f g < [ g ] 2

21 Back to lnear dscrmnants also e ave a classfcaton error f and g < or - and g >.e.g < and a correct classfcaton f and g > or - and g <.e.g > note tat snce te data s lnearl separable gven a tranng set D {... n n } e can ave zero emprcal rsk te necessar and suffcent condton s tat + b > 2

22 e margn s te dstance from te boundar to te closest pont γ mn + b tere ll be no error f t s strctl greater tan zero + b > γ > note tat ts s ll-defned n te sense tat γ does not cange f bot and b are scaled b λ e need a normalzaton b - g 22

23 e margn ts s smlar to at e ave seen for Fser dscrmnants natural normalzaton s oever t ntroduces a quadratc constrant and complcates optmzaton a more convenent normalzaton s to make g for te closest pont.e. - mn + b under c γ b g 23

24 Support Vector Macnes under ts normalzaton emprcal error s zero f and onl f b b b sgn b + te SVM s te classfer tat mamzes te margn under ts set of constrants 2 b b + to subject mn 2 24

25 Relatonsp to SRM te SRM prncple: start from a nested collecton of famles of functons S L S k ere S { α for all α} } for eac S fnd te functon set of parameters tat mnmzes te emprcal rsk n Remp mn L [ k k α ] α n k select te functon class suc tat * R mn { R + Φ } emp ere Φ s a functon of te VC dmenson complet of te faml S 25

26 Relatonsp to SRM ere: mn b 2 + b subject to S s te faml of perplanes suc tat <λ te constrants guarantee tat R emp and te VC dmenson s upper-bounded b λ more on ts later.e. te SVM mnmzes an upper-bound of Φ le mantanng R emp zero snce R Remp + Φ ts provdes guarantees on te rsk more later 26

27 Intutvel ts s penalzng complet e.g. te smaller te te larger te number of components set to zero ts s searcng for te more stable perplane among te ones tat ave zero tranng error s te one tat as most room for dscrepances beteen tranng and testng te margn as a securt gap tere are man detals c e ave not flled ts sould gve ou a roug understandng to start on projects * 27

28 Homeork net class e ll go over te Perceptron c s a good classfer to gan nsgt on te role of te margn dualt optmzaton b g lke evertng e ll do n ts course t ll requre a ver good understandng of ts pcture e ll also use epressons lke all te tme + b > ou sould make ourself famlar t tese!!! 28

29 29