CS434a/54a: Paern Recognon Prof. Olga Veksler Lecure 4
Oulne Normal Random Varable Properes Dscrmnan funcons
Why Normal Random Varables? Analycally racable Works well when observaon comes form a corruped sngle prooype Is an opmal dsrbuon of daa for many classfers used n pracce 3
The Unvarae Normal Densy s a scalar has dmenson p ep π, Where: mean or epeced value of varance 4
5
Several Feaures Wha f we have several feaures,,, d each normally dsrbued may have dfferen means may have dfferen varances may be dependen or ndependen of each oher How do we model her jon dsrbuon? 6
The Mulvarae Normal Densy Mulvarae normal densy n d dmensons s: p d π d / deermnan of d d covarance of and d / ep nverse of [,,, d ] [,,, d ] Each s N, o prove hs, negrae ou all oher feaures from he jon densy 7
More on d d d plays role smlar o he role ha plays n one dmenson From we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen j have posve correlaon j > have negave correlaon j < 8
9 The Mulvarae Normal Densy If s dagonal hen he feaures,, j are ndependen, and d p ep π 3
The Mulvarae Normal Densy p π d / / ep p c ep normalzng consan [ ] 3 3 3 scalar s sngle number, he closer s o he larger s p 3 3 3 3 3 3 Thus P s larger for smaller
s posve sem defne > If for nonzero hen de. Ths case s no neresng, p s no defned. one feaure vecor s a consan has zero varance. or wo componens are mulples of each oher so we wll assume s posve defne > If s posve defne hen so s
Posve defne mar of sze d by d has d dsnc real egenvalues and s d egenvecors are orhogonal Thus f Φ s a mar whose columns are normalzed egenvecors of, hen Φ - Φ Φ ΦΛ where Λ s a dagonal mar wh correspondng egenvalues on he dagonal Thus ΦΛΦ and ΦΛ Φ Thus f Λ / denoes mar s.. Λ / / Λ / / Λ ΦΛ ΦΛ ΦΛ ΦΛ ΜΜ
Thus Thus where MM M M M M M Φ Λ Pons whch sasfy le on an ellpse cons M roaon mar scalng mar
usual Eucledan dsance beween and p decreases Mahalanobs dsance beween and p decreases fas egenvecors of pons a equal Eucledan dsance from le on a crcle p decreases slow pons a equal Mahalanobs dsance from le on an ellpse: sreches crcles o ellpses
-d Mulvarae Normal Densy Can you see much n hs graph? A mos you can see ha he mean s around [,], bu can really ell f and are correlaed 5
-d Mulvarae Normal Densy How abou hs graph? 6
-d Mulvarae Normal Densy Level curves graph p s consan along each conour opologcal map of 3-d surface Now we can see much more and are ndependen and are equal 7
-d Mulvarae Normal Densy [,] [,] 4 [,] 4 [,] 8
-d Mulvarae Normal Densy [, ].5.5.5.5.9.9.9 4.9.9.9.9 4.9
The Mulvarae Normal Densy If X has densy N, hen AX has densy NΑ,Α Α Thus X can be ransformed no a sphercal normal varable covarance of sphercal densy s he deny mar I wh whenng ransform X AX A w ΦΛ.9.9
Dscrmnan Funcons Classfer can be vewed as nework whch compues m dscrmnan funcons and selecs caegory correspondng o he larges dscrmnan selec class gvng mamm dscrmnan funcons g g g m feaures 3 d g can be replaced wh any monooncally ncreasng funcon, he resuls wll be unchanged
Dscrmnan Funcons The mnmum error-rae classfcaon s acheved by he dscrmnan funcon g Pc P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrmnan funcon s g P c Pc For normal densy, convnen o ake logarhms. Snce logarhm s a monooncally ncreasng funcon, he equvalen dscrmnan funcon s g ln P c + ln Pc
Dscrmnan Funcons for he Normal Densy ep / / d c p π ln ln ln c P d g + π Plug n p c and Pc ge Dscrmnan funcon g ln P c + ln Pc Suppose we for class c s class condonal densy p c s N, ln ln c P g + consan for all
4 Tha s Case I In hs case, feaures,.,, d are ndependen wh dfferen means and equal varances
Case I Dscrmnan funcon g ln + lnp c De d and - / I Can smplfy dscrmnan funcon I g ln + d lnp c consan for all g + ln P c + lnp c 5
Case I Geomerc Inerpreaon If lnpc lnpc g j, hen g If lnpc lnpc lnpc + j, hen decson regon for c decson regon for c decson regon for c 3 3 vorono dagram: pons n each cell are closer o he mean n ha cell han o any oher mean decson regon for c n c 3 decson regon for c 3 decson regon for c 3
7 Case I + ln c P g lnpc + + consan for all classes + + lnpc g w w g + lnpc + + dscrmnan funcon s lnear
Case I consan n g w + w w lnear n : d Thus dscrmnan funcon s lnear, Therefore he decson boundares g g j are lnear lnes f has dmenson planes f has dmenson 3 hyper-planes f has dmenson larger han 3 w
Case I: Eample 3 classes, each -dmensonal Gaussan wh 4 6 3 4 Prors c P c and 4 c P Dscrmnan funcon s g 3 P 3 Plug n parameers for each class [ ] [ ] g 3 5 +.38 6 g 3 [ ] 4 3 3 3 + + lnpc g 4 6 3 +.69 6 5 +.38 6 9
Case I: Eample Need o fnd ou when g < g j for,j,,3 Can be done by solvng g g j for,j,,3 Le s ake g g frs [ ] [ ] 3 5 +.38 6 3 4 3 Smplfyng, [ ] 4 6 3 5 +.38 6 47 6 4 3 lne equaon 47 6 3
Case I: Eample Ne solve g g 3 + 3 6. Almos fnally solve g g 3 3.8 And fnally solve g g g 3.4 and 4.8 3
Case I: Eample Prors c P c and P 4 c P 3 c 3 g g 3 g g 3 c c g g lnes connecng means are perpendcular o decson boundares 3
Case Covarance marces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 33
Case Dscrmnan funcon g ln + lnp c Dscrmnan funcon becomes g consan for all classes lnpc squared Mahalanobs Dsance + Mahalanobs Dsance y y y If I, Mahalanobs Dsance becomes usual Eucledan dsance y I y y
Eucledan vs. Mahalanobs Dsances egenvecors of pons a equal Eucledan dsance from le on a crcle pons a equal Mahalanobs dsance from le on an ellpse: sreches crles o ellpses
Case Geomerc Inerpreaon If lnpc g lnpc j, hen decson regon for c decson regon g If lnpc lnpc lnpc, + j hen decson regon for c decson regon for c 3 for c 3 decson regon for c 3 pons n each cell are closer o he mean n ha cell han o any oher mean under Mahalanobs dsance decson regon for c 3
Case Can smplfy dscrmnan funcon: + ln c P g + + ln c P + + ln c P consan for all classes ln c P + + Thus n hs case dscrmnan s also lnear w w + + lnpc
Case : Eample 3 classes, each -dmensonal Gaussan wh 5 3 4 4 3 c P c c P P 3.5.5 4 Agan can be done by solvng g g j for,j,,3
scalar row vecor Case : Eample + + j j j j lnpc lnpc + j j j j lnpc lnpc + j j j j Pc Pc ln Le s solve n general frs g g j Le s regroup he erms We ge he lne where g g j
Case : Eample ln + j Now subsue for,j, [ ] Now subsue for,j,3 [ 3.4.4]. 4 Now subsue for,j,3 Pc Pc j 3.4 +.4 [ 5.4.43]. 4 5.4 +.43 j.4.4 j
Case : Eample c Prors c P c and P 4 P 3 c c 3 c lnes connecng means are no n general perpendcular o decson boundares 4
General Case are arbrary Covarance marces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 j.9.9 4 4
43 From prevous dscusson, General Case are arbrary Ths can be smplfed, bu we can rearrange : ln ln c P g + ln ln c P g + + + + + ln ln c P g w w W g + +
General Case are arbrary lnear n g W + w + w consan n quadrac n snce W d d j w j j d, j w j j Thus he dscrmnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 44
3 classes, each -dmensonal Gaussan wh General Case are arbrary: Eample 3 6 3 4.5.5 7 3.5.5 3 4 c P c c Prors: and P P 3 Agan can be done by solvng g g j for,j,,3 g + + ln + lnp c Need o solve a bunch of quadrac nequales of varables
General Case are arbrary: Eample 3 6 3 4 c P c c P 4 P 3.5.5 7 3.5.5 3 c c c 3 c
Imporan Pons The Bayes classfer when classes are normally dsrbued s n general quadrac If covarance marces are equal and proporonal o deny mar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Eucledan dsance classfer If covarance marces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he mnmum Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opmal only f dsrbuon of daa s approprae normal