CHAPTER 10: LINEAR DISCRIMINATION

Transcription

1 HAPER : LINEAR DISRIMINAION

2 Dscmnan-based lassfcaon 3 In classfcaon h K classes ( k ) We defned dsmnan funcon g () = K hen gven an es eample e chose (pedced) s class label as f g () as he mamum among g () g () g k () In pevous chapes e have Used g ()= P( ) hs s called lkelhood classfcaon Whee e used mamum lkelhood esmae echnque fo esmae class lkelhood P( )

3 4 Lkelhood- vs. Dscmnan-based lassfcaon Lkelhood-based: Assume a model fo p( ) use Baes ule o calculae P( ) g () = P( ) hs eques esmang class condonal denses P( ) Fo hgh-dmensonal daa (man abues/feaues) esmang class condonal denses self s a dffcul ask Dscmnan-based: Assume a model fo g (Φ ); no dens esmaon Paamees Φ descbe he class bounda Esmang he class bounda s enough fo pefomng classfcaon no need o accuael esmae he denses nsde he boundaes

4 Lnea Dscmnan 5 Lnea dscmnan: g Advanages: Smple: O(d) space/compuaon (d s he numbe of feaues) Knoledge eacon: Weghed sum of abues; posve/negave eghs magnudes (ced scong) Opmal hen p( ) ae Gaussan h shaed cov ma; useful hen classes ae (almos) lneal sepaable d

5 Quadac dscmnan: Hghe-ode (poduc) ems: Map fom o z usng nonlnea bass funcons and use a lnea dscmnan n z-space Genealzed Lnea Model z z z z z g W W k g

6 Genealzed Lnea Model 7 Eample of non-lnea bass funcons: sn() ep(-( -m) /c) ep(--m /c) Log( ) ( >c) (a +b >c)

7 o lasses g g g ohese f choose g 8

8 9 Geome

9 Undesandng he geome Le he dscmnan funcon s gven b g()= + + = + hee =( ) ake an o pons lng on he decson suface (bounda) g()= g( )=g( )= + = + => ( - )= Noe ha ( - ) s a veco lng on he decson suface (hpeplane) hch means s nomal o an veco lng on he decson suface

10 Undesandng he geome An daa pon can be en as a sum of o vecos as follos = p +(/) p s nomal poecon of on o decson hpe plane ( p les on he decson hpeplane) s dsance of o he hpeplane g()= + = ( p +(/)+= ( p + )+( )/= /= => =g()/ Smlal f = ll denoe dsance of he hpeplane fom he ogn g()= = => = /

11 Mulple lasses g hoos e g K mag f lasses ae lneal sepaable

12 Mulple classes 3 Dung esng gven deall e should have onl one g () = K geae han zeo and all ohes should be less han Hoeve hs s no alas he case Posve half spaces of he hpeplane s ma ovelap O e ma have all g ()< hese ma be aken as eec case Remembeng ha g ()/ s he dsance fom he npu pon o he decson hpeplane assumng all have smla lengh hs assgns pon o he class (among all g ()>) o hose decson hpeplane he pon s mos dsan

13 Pase Sepaaon g g don' cae f f ohese choos e g f If classes ae no lneal sepaable bu pase lneal sepaable use K(K-)/) pase 4 dscmnans

14 If he class denses ae Gaussan and shae a common covaance ma he dscmnan funcon s lnea hen p ( ) ~ N ( μ ) Fo he specal case hen hee ae o classes e defne (/(-) s knon as ansfomaon of odds of Fom Dscmnans o Poseos 5 P g μ μ μ ohese and f choose and 5 P P / /.

15 P P P P P p P p P d / / ep / μ μ d / / ep / μ μ hee μ μ μ μ μ μ he nvese of s sc o sgmod funcon P P sgmod P ep P P 6

16 Sgmod (Logsc) Funcon 7 alculaeg alculae sgmod andchoose o andchoose f. 5 f g

17 Gaden-Descen 8 E(X) s eo h paamees on sample X *=ag mn E( X) Gaden E E E E... Gaden-descen: Sas fom andom and updaes eavel n he negave decon of gaden d

18 Gaden-Descen 9 E E ( ) E ( + ) + η

19 Gaden-Descen

20 Gaden-Descen

21 Logsc Dscmnaon P P P P P p p P P P p p o o ep hee ˆ o classes: Assume lkelhood ao s lnea

22 anng: o lasses 3 E l E l P ep Benoull X X X ~

23 anng: Gaden-Descen 4 E d E da d E... sgmoda If X

24 5

25 6

26 K> lasses 7 K o K K E l K P p p... ep ep ˆ ~ Mul X X X sofma

27 8

28 Quadac: Sum of bass funcons: hee φ() ae bass funcons. Eamples: Hdden uns n neual neoks (hapes and ) Kenels n SVM (hape 3) Genealzng he Lnea Model 3 K p p W K p p

29 Dscmnaon b Regesson 3 E l X X N ep ep sgmod hee ~ lasses ae NO muuall eclusve and ehausve

30 Leanng o Rank 3 Rankng: A dffeen poblem han classfcaon o egesson Le us sa u and v ae o nsances e.g. o moves We pefe u o v mples ha g( u )>g( v ) hee g() s a scoe funcon hee lnea: g()= Fnd a decon such ha e ge he desed anks hen nsances ae poeced along