Noral Rando Varable and s dscrnan funcons
Oulne Noral Rando Varable Properes Dscrnan funcons
Why Noral Rando Varables? Analycally racable Works well when observaon coes for a corruped snle prooype 3
The Unvarae Noral Densy s a scalar has denson p ep, Where: = ean or epeced value of = varance 4
5
Several Feaures Wha f we have several feaures,,, d each norally dsrbued ay have dfferen eans ay have dfferen varances ay be dependen or ndependen of each oher How do we odel her jon dsrbuon? 6
The Mulvarae Noral Densy Mulvarae noral densy n d densons s: p d d / deernan of d d covarance of and d / Each s N, ep nverse of = [,,, d ] = [,,, d ] 7
More on plays role slar o he role ha d d plays n one denson d Fro we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen j = have posve correlaon j > have neave correlaon j < 8
9 The Mulvarae Noral Densy If s daonal hen he feaures,, j are ndependen, and d p ep 3
scalar s snle nuber, he closer s o he larer s p noralzn consan ep p / / d The Mulvarae Noral Densy 3 3 3 3 3 3 3 3 3 ep c p Thus P s larer for saller
s posve se defne >= If = for nonzero hen de=. Ths case s no neresn, p s no defned. one feaure vecor s a consan has zero varance. or wo coponens are ulples of each oher so we wll assue s posve defne > If s posve defne hen so s
Eenvalues/eenvecors fro Wk Gven a lnear ransforaon A, a non-zero vecor s defned o be an eenvecor of he ransforaon f sasfes he eenvalue equaon A = λ for soe scalar λ. where λ s called an eenvalue of A, correspondn o he eenvecor.
Eenvalues/eenvecors fro Wk Geoercally, eans ha under he ransforaon A, eenvecors only chane n anude and sn he drecon of A s he sae as ha of. The eenvalue λ s sply he aoun of "srech" or "shrnk" o whch a vecor s subjeced when ransfored by A. For eaple, an eenvalue of + eans ha he eenvecor s doubled n lenh and pons n he sae drecon. An eenvalue of + eans ha he eenvecor s unchaned, whle an eenvalue of eans ha he eenvecor s reversed n sense. 3
Eenvalues/eenvecors fro Wk In hs shear appn he red arrow chanes drecon bu he blue arrow does no. Therefore he blue arrow s an eenvecor, wh eenvalue as s lenh s unchaned. 4
Posve defne ar of sze d by d has d dsnc real eenvalues and s d eenvecors are orhoonal Thus f s a ar whose coluns are noralzed eenvecors of, hen - = = where s a daonal ar wh correspondn eenvalues on he daonal Thus = and = Thus f / denoes ar s.. / /
Thus Thus where MM M M M M M Pons whch sasfy le on an ellpse cons M roaon ar scaln ar
usual Eucledan dsance beween and Mahalanobs dsance beween and eenvecors of pons a equal Eucledan dsance fro le on a crcle pons a equal Mahalanobs dsance fro le on an ellpse: sreches crcles o ellpses
-d Mulvarae Noral Densy Level curves raph p s consan alon each conour opolocal ap of 3-d surface and are ndependen and are equal 8
-d Mulvarae Noral Densy,, 4, 4, 9
-d Mulvarae Noral Densy,.5.5.9.9.9.9 4.5.5.9.9.9.9 4
The Mulvarae Noral Densy If X has densy N, hen AX has densy NA,A A Thus X can be ransfored no a sphercal noral varable covarance of sphercal densy s he deny ar I wh whenn ransfor X AX A w.9.9 whose rows are eenvecors of Σ daonal ar wh eenvalues of Σ
Dscrnan Funcons Classfer can be vewed as nework whch copues dscrnan funcons and selecs caeory correspondn o he lares dscrnan selec class vn a dscrnan funcons feaures 3 d can be replaced wh any onooncally ncreasn funcon of, he resuls wll be unchaned
Dscrnan Funcons The nu error-rae classfcaon s acheved by he dscrnan funcon = Pc =P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrnan funcon s = P c Pc For noral densy, convnen o ake loarhs. Snce loarh s a onooncally ncreasn funcon, he equvalen dscrnan funcon s = ln P c + ln Pc 3
Dscrnan Funcons for he Noral Densy ep / / d c p ln ln ln c P d Plu n p c and Pc e Dscrnan funcon = ln P c + ln Pc Suppose for class c s class condonal densy p c s N, ln ln c P consan for all
5 Tha s Case = I In hs case, feaures,.,, d are ndependen wh dfferen eans and equal varances
6 Dscrnan funcon Case = I ln ln c P Can splfy dscrnan funcon ln ln d c P I consan for all ln c P ln c P De = d and - =/ I
Case = I Geoerc Inerpreaon If ln Pc ln Pc j, hen If ln Pc ln Pc, hen ln Pc j decson reon for c decson reon for c decson reon for c 3 3 vorono dara: pons n each cell are closer o he ean n ha cell han o any oher ean decson reon for c n c 3 decson reon for c 3 decson reon for c 3
8 Case = I ln c P ln Pc consan for all classes ln Pc w w ln Pc dscrnan funcon s lnear
Case = I w w consan n w lnear n : d Thus dscrnan funcon s lnear, Therefore he decson boundares = j are lnear lnes f has denson planes f has denson 3 hyper-planes f has denson larer han 3 w
Case = I: Eaple 3 classes, each -densonal Gaussan wh Prors 4 6 c 3 4 4 P c and c P Dscrnan funcon s 3 P 3 3 3 ln Pc Plu n paraeers for each class 5 4 6 5.38.38 3 3 6 3 4 6 3.69 6 3
Case = I: Eaple Need o fnd ou when < j for,j=,,3 Can be done by solvn = j for,j=,,3 Le s ake = frs 3 5 6 Splfyn,.38 3 4 3 4 6 3 47 6 5 6.38 4 3 lne equaon 47 6 3
Case = I: Eaple Ne solve = 3 3 6. Alos fnally solve = 3 3.8 And fnally solve = = 3.4 and 4.8 3
Case = I: Eaple Prors c Pc P 4 and P c 3 c 3 c lnes connecn eans are perpendcular o decson boundares c 33
Case = Covarance arces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 34
squared Mahalanobs Dsance Dscrnan funcon Case = ln ln c P consan for all classes Dscrnan funcon becoes ln Pc Mahalanobs Dsance y y y If =I, Mahalanobs Dsance becoes usual Eucledan dsance y y y I
Eucledan vs. Mahalanobs Dsances eenvecors of pons a equal Eucledan dsance fro le on a crcle pons a equal Mahalanobs dsance fro le on an ellpse: sreches crles o ellpses
Case = Geoerc Inerpreaon If ln Pc ln Pc j, hen decson reon for c decson reon If ln Pc ln Pc ln Pc, j hen decson reon for c decson reon for c 3 for c 3 decson reon for c 3 pons n each cell are closer o he ean n ha cell han o any oher ean under Mahalanobs dsance decson reon for c 3
Case = Can splfy dscrnan funcon: ln c P ln c P ln c P consan for all classes ln c P Thus n hs case dscrnan s also lnear w w ln Pc
Case = : Eaple 3 classes, each -densonal Gaussan wh 5 3 4 4 3 c Pc c P P 3.5.5 4 Aan can be done by solvn = j for,j=,,3
scalar row vecor Case = : Eaple j j j j ln Pc ln Pc j j j j ln Pc ln Pc j j j j Pc Pc ln Le s solve n eneral frs j Le s reroup he ers We e he lne where j
Case = : Eaple j ln j j Pc j Now subsue for,j=, Now subsue for,j=,3 3.4.4. 4 Now subsue for,j=,3 Pc 5.4.43. 4 3.4.4 5.4.43.4.4
Case = : Eaple Prors c 4 P c and c P P 3 c c 3 c lnes connecn eans are no n eneral perpendcular o decson boundares 4
General Case are arbrary Covarance arces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 j.9.9 4 43
44 Fro prevous dscusson, General Case are arbrary Ths can be splfed, bu we can rearrane : ln ln c P ln ln c P ln ln c P w w W
General Case are arbrary W w w quadrac n snce W d lnear n d j w j j consan n d, j w j j Thus he dscrnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 45
General Case are arbrary: Eaple 3 classes, each -densonal Gaussan wh 3 6 3 4.5.5 7 3.5.5 3 Prors: 4 c Pc and c P P 3 Aan can be done by solvn = j for,j=,,3 ln lnp c Need o solve a bunch of quadrac nequales of varables
General Case are arbrary: Eaple 3 6 3 4 c Pc c P 4 P 3.5.5 7 3.5.5 3 c c c 3 c
Iporan Pons The Bayes classfer when classes are norally dsrbued s n eneral quadrac If covarance arces are equal and proporonal o deny ar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he nu Eucledan dsance classfer If covarance arces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he nu Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opal only f dsrbuon of daa s approprae noral