CS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 4

Transcription

1 CS434a/54a: Paern Recognon Prof. Olga Veksler Lecure 4

2 Oulne Normal Random Varable Properes Dscrmnan funcons

3 Why Normal Random Varables? Analycally racable Works well when observaon comes form a corruped sngle prooype Is an opmal dsrbuon of daa for many classfers used n pracce 3

4 The Unvarae Normal Densy s a scalar has dmenson p ep π, Where: mean or epeced value of varance 4

5 5

6 Several Feaures Wha f we have several feaures,,, d each normally dsrbued may have dfferen means may have dfferen varances may be dependen or ndependen of each oher How do we model her jon dsrbuon? 6

7 The Mulvarae Normal Densy Mulvarae normal densy n d dmensons s: p d π d / deermnan of d d covarance of and d / ep nverse of [,,, d ] [,,, d ] Each s N, o prove hs, negrae ou all oher feaures from he jon densy 7

8 More on d d d plays role smlar o he role ha plays n one dmenson From we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen j have posve correlaon j > have negave correlaon j < 8

9 9 The Mulvarae Normal Densy If s dagonal hen he feaures,, j are ndependen, and d p ep π 3

10 The Mulvarae Normal Densy p π d / / ep p c ep normalzng consan [ ] scalar s sngle number, he closer s o he larger s p Thus P s larger for smaller

11 s posve sem defne > If for nonzero hen de. Ths case s no neresng, p s no defned. one feaure vecor s a consan has zero varance. or wo componens are mulples of each oher so we wll assume s posve defne > If s posve defne hen so s

12 Posve defne mar of sze d by d has d dsnc real egenvalues and s d egenvecors are orhogonal Thus f Φ s a mar whose columns are normalzed egenvecors of, hen Φ - Φ Φ ΦΛ where Λ s a dagonal mar wh correspondng egenvalues on he dagonal Thus ΦΛΦ and ΦΛ Φ Thus f Λ / denoes mar s.. Λ / / Λ / / Λ ΦΛ ΦΛ ΦΛ ΦΛ ΜΜ

13 Thus Thus where MM M M M M M Φ Λ Pons whch sasfy le on an ellpse cons M roaon mar scalng mar

14 usual Eucledan dsance beween and p decreases Mahalanobs dsance beween and p decreases fas egenvecors of pons a equal Eucledan dsance from le on a crcle p decreases slow pons a equal Mahalanobs dsance from le on an ellpse: sreches crcles o ellpses

15 -d Mulvarae Normal Densy Can you see much n hs graph? A mos you can see ha he mean s around [,], bu can really ell f and are correlaed 5

16 -d Mulvarae Normal Densy How abou hs graph? 6

17 -d Mulvarae Normal Densy Level curves graph p s consan along each conour opologcal map of 3-d surface Now we can see much more and are ndependen and are equal 7

18 -d Mulvarae Normal Densy [,] [,] 4 [,] 4 [,] 8

19 -d Mulvarae Normal Densy [, ]

20 The Mulvarae Normal Densy If X has densy N, hen AX has densy NΑ,Α Α Thus X can be ransformed no a sphercal normal varable covarance of sphercal densy s he deny mar I wh whenng ransform X AX A w ΦΛ.9.9

21 Dscrmnan Funcons Classfer can be vewed as nework whch compues m dscrmnan funcons and selecs caegory correspondng o he larges dscrmnan selec class gvng mamm dscrmnan funcons g g g m feaures 3 d g can be replaced wh any monooncally ncreasng funcon, he resuls wll be unchanged

22 Dscrmnan Funcons The mnmum error-rae classfcaon s acheved by he dscrmnan funcon g Pc P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrmnan funcon s g P c Pc For normal densy, convnen o ake logarhms. Snce logarhm s a monooncally ncreasng funcon, he equvalen dscrmnan funcon s g ln P c + ln Pc

23 Dscrmnan Funcons for he Normal Densy ep / / d c p π ln ln ln c P d g + π Plug n p c and Pc ge Dscrmnan funcon g ln P c + ln Pc Suppose we for class c s class condonal densy p c s N, ln ln c P g + consan for all

24 4 Tha s Case I In hs case, feaures,.,, d are ndependen wh dfferen means and equal varances

25 Case I Dscrmnan funcon g ln + lnp c De d and - / I Can smplfy dscrmnan funcon I g ln + d lnp c consan for all g + ln P c + lnp c 5

26 Case I Geomerc Inerpreaon If lnpc lnpc g j, hen g If lnpc lnpc lnpc + j, hen decson regon for c decson regon for c decson regon for c 3 3 vorono dagram: pons n each cell are closer o he mean n ha cell han o any oher mean decson regon for c n c 3 decson regon for c 3 decson regon for c 3

27 7 Case I + ln c P g lnpc + + consan for all classes + + lnpc g w w g + lnpc + + dscrmnan funcon s lnear

28 Case I consan n g w + w w lnear n : d Thus dscrmnan funcon s lnear, Therefore he decson boundares g g j are lnear lnes f has dmenson planes f has dmenson 3 hyper-planes f has dmenson larger han 3 w

29 Case I: Eample 3 classes, each -dmensonal Gaussan wh Prors c P c and 4 c P Dscrmnan funcon s g 3 P 3 Plug n parameers for each class [ ] [ ] g g 3 [ ] lnpc g

30 Case I: Eample Need o fnd ou when g < g j for,j,,3 Can be done by solvng g g j for,j,,3 Le s ake g g frs [ ] [ ] Smplfyng, [ ] lne equaon

31 Case I: Eample Ne solve g g Almos fnally solve g g And fnally solve g g g 3.4 and 4.8 3

32 Case I: Eample Prors c P c and P 4 c P 3 c 3 g g 3 g g 3 c c g g lnes connecng means are perpendcular o decson boundares 3

33 Case Covarance marces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen

34 Case Dscrmnan funcon g ln + lnp c Dscrmnan funcon becomes g consan for all classes lnpc squared Mahalanobs Dsance + Mahalanobs Dsance y y y If I, Mahalanobs Dsance becomes usual Eucledan dsance y I y y

35 Eucledan vs. Mahalanobs Dsances egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crles o ellpses

36 Case Geomerc Inerpreaon If lnpc g lnpc j, hen decson regon for c decson regon g If lnpc lnpc lnpc, + j hen decson regon for c decson regon for c 3 for c 3 decson regon for c 3 pons n each cell are closer o he mean n ha cell han o any oher mean under Mahalanobs dsance decson regon for c 3

37 Case Can smplfy dscrmnan funcon: + ln c P g + + ln c P + + ln c P consan for all classes ln c P + + Thus n hs case dscrmnan s also lnear w w + + lnpc

38 Case : Eample 3 classes, each -dmensonal Gaussan wh c P c c P P Agan can be done by solvng g g j for,j,,3

39 scalar row vecor Case : Eample + + j j j j lnpc lnpc + j j j j lnpc lnpc + j j j j Pc Pc ln Le s solve n general frs g g j Le s regroup he erms We ge he lne where g g j

40 Case : Eample ln + j Now subsue for,j, [ ] Now subsue for,j,3 [ 3.4.4]. 4 Now subsue for,j,3 Pc Pc j [ ] j.4.4 j

41 Case : Eample c Prors c P c and P 4 P 3 c c 3 c lnes connecng means are no n general perpendcular o decson boundares 4

42 General Case are arbrary Covarance marces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 j

43 43 From prevous dscusson, General Case are arbrary Ths can be smplfed, bu we can rearrange : ln ln c P g + ln ln c P g ln ln c P g w w W g + +

44 General Case are arbrary lnear n g W + w + w consan n quadrac n snce W d d j w j j d, j w j j Thus he dscrmnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 44

45 3 classes, each -dmensonal Gaussan wh General Case are arbrary: Eample c P c c Prors: and P P 3 Agan can be done by solvng g g j for,j,,3 g + + ln + lnp c Need o solve a bunch of quadrac nequales of varables

46 General Case are arbrary: Eample c P c c P 4 P c c c 3 c

47 Imporan Pons The Bayes classfer when classes are normally dsrbued s n general quadrac If covarance marces are equal and proporonal o deny mar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Eucledan dsance classfer If covarance marces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opmal only f dsrbuon of daa s approprae normal