Normal Random Variable and its discriminant functions

Transcription

1 Noral Rando Varable and s dscrnan funcons

2 Oulne Noral Rando Varable Properes Dscrnan funcons

3 Why Noral Rando Varables? Analycally racable Works well when observaon coes for a corruped snle prooype 3

4 The Unvarae Noral Densy s a scalar has denson p ep, Where: = ean or epeced value of = varance 4

5 5

6 Several Feaures Wha f we have several feaures,,, d each norally dsrbued ay have dfferen eans ay have dfferen varances ay be dependen or ndependen of each oher How do we odel her jon dsrbuon? 6

7 The Mulvarae Noral Densy Mulvarae noral densy n d densons s: p d d / deernan of d d covarance of and d / Each s N, ep nverse of = [,,, d ] = [,,, d ] 7

8 More on plays role slar o he role ha d d plays n one denson d Fro we can fnd ou. The ndvdual varances of feaures,,, d. If feaures and j are ndependen j = have posve correlaon j > have neave correlaon j < 8

9 9 The Mulvarae Noral Densy If s daonal hen he feaures,, j are ndependen, and d p ep 3

10 scalar s snle nuber, he closer s o he larer s p noralzn consan ep p / / d The Mulvarae Noral Densy ep c p Thus P s larer for saller

11 s posve se defne >= If = for nonzero hen de=. Ths case s no neresn, p s no defned. one feaure vecor s a consan has zero varance. or wo coponens are ulples of each oher so we wll assue s posve defne > If s posve defne hen so s

12 Eenvalues/eenvecors fro Wk Gven a lnear ransforaon A, a non-zero vecor s defned o be an eenvecor of he ransforaon f sasfes he eenvalue equaon A = λ for soe scalar λ. where λ s called an eenvalue of A, correspondn o he eenvecor.

13 Eenvalues/eenvecors fro Wk Geoercally, eans ha under he ransforaon A, eenvecors only chane n anude and sn he drecon of A s he sae as ha of. The eenvalue λ s sply he aoun of "srech" or "shrnk" o whch a vecor s subjeced when ransfored by A. For eaple, an eenvalue of + eans ha he eenvecor s doubled n lenh and pons n he sae drecon. An eenvalue of + eans ha he eenvecor s unchaned, whle an eenvalue of eans ha he eenvecor s reversed n sense. 3

14 Eenvalues/eenvecors fro Wk In hs shear appn he red arrow chanes drecon bu he blue arrow does no. Therefore he blue arrow s an eenvecor, wh eenvalue as s lenh s unchaned. 4

15 Posve defne ar of sze d by d has d dsnc real eenvalues and s d eenvecors are orhoonal Thus f s a ar whose coluns are noralzed eenvecors of, hen - = = where s a daonal ar wh correspondn eenvalues on he daonal Thus = and = Thus f / denoes ar s.. / /

16 Thus Thus where MM M M M M M Pons whch sasfy le on an ellpse cons M roaon ar scaln ar

17 usual Eucledan dsance beween and Mahalanobs dsance beween and eenvecors of pons a equal Eucledan dsance fro le on a crcle pons a equal Mahalanobs dsance fro le on an ellpse: sreches crcles o ellpses

18 -d Mulvarae Noral Densy Level curves raph p s consan alon each conour opolocal ap of 3-d surface and are ndependen and are equal 8

19 -d Mulvarae Noral Densy,, 4, 4, 9

20 -d Mulvarae Noral Densy,

21 The Mulvarae Noral Densy If X has densy N, hen AX has densy NA,A A Thus X can be ransfored no a sphercal noral varable covarance of sphercal densy s he deny ar I wh whenn ransfor X AX A w.9.9 whose rows are eenvecors of Σ daonal ar wh eenvalues of Σ

22 Dscrnan Funcons Classfer can be vewed as nework whch copues dscrnan funcons and selecs caeory correspondn o he lares dscrnan selec class vn a dscrnan funcons feaures 3 d can be replaced wh any onooncally ncreasn funcon of, he resuls wll be unchaned

23 Dscrnan Funcons The nu error-rae classfcaon s acheved by he dscrnan funcon = Pc =P c Pc /P Snce he observaon s ndependen of he class, he equvalen dscrnan funcon s = P c Pc For noral densy, convnen o ake loarhs. Snce loarh s a onooncally ncreasn funcon, he equvalen dscrnan funcon s = ln P c + ln Pc 3

24 Dscrnan Funcons for he Noral Densy ep / / d c p ln ln ln c P d Plu n p c and Pc e Dscrnan funcon = ln P c + ln Pc Suppose for class c s class condonal densy p c s N, ln ln c P consan for all

25 5 Tha s Case = I In hs case, feaures,.,, d are ndependen wh dfferen eans and equal varances

26 6 Dscrnan funcon Case = I ln ln c P Can splfy dscrnan funcon ln ln d c P I consan for all ln c P ln c P De = d and - =/ I

27 Case = I Geoerc Inerpreaon If ln Pc ln Pc j, hen If ln Pc ln Pc, hen ln Pc j decson reon for c decson reon for c decson reon for c 3 3 vorono dara: pons n each cell are closer o he ean n ha cell han o any oher ean decson reon for c n c 3 decson reon for c 3 decson reon for c 3

28 8 Case = I ln c P ln Pc consan for all classes ln Pc w w ln Pc dscrnan funcon s lnear

29 Case = I w w consan n w lnear n : d Thus dscrnan funcon s lnear, Therefore he decson boundares = j are lnear lnes f has denson planes f has denson 3 hyper-planes f has denson larer han 3 w

30 Case = I: Eaple 3 classes, each -densonal Gaussan wh Prors 4 6 c P c and c P Dscrnan funcon s 3 P ln Pc Plu n paraeers for each class

31 Case = I: Eaple Need o fnd ou when < j for,j=,,3 Can be done by solvn = j for,j=,,3 Le s ake = frs Splfyn, lne equaon

32 Case = I: Eaple Ne solve = Alos fnally solve = And fnally solve = = 3.4 and 4.8 3

33 Case = I: Eaple Prors c Pc P 4 and P c 3 c 3 c lnes connecn eans are perpendcular o decson boundares c 33

34 Case = Covarance arces are equal bu arbrary In hs case, feaures,.,, d are no necessarly ndependen

35 squared Mahalanobs Dsance Dscrnan funcon Case = ln ln c P consan for all classes Dscrnan funcon becoes ln Pc Mahalanobs Dsance y y y If =I, Mahalanobs Dsance becoes usual Eucledan dsance y y y I

36 Eucledan vs. Mahalanobs Dsances eenvecors of pons a equal Eucledan dsance fro le on a crcle pons a equal Mahalanobs dsance fro le on an ellpse: sreches crles o ellpses

37 Case = Geoerc Inerpreaon If ln Pc ln Pc j, hen decson reon for c decson reon If ln Pc ln Pc ln Pc, j hen decson reon for c decson reon for c 3 for c 3 decson reon for c 3 pons n each cell are closer o he ean n ha cell han o any oher ean under Mahalanobs dsance decson reon for c 3

38 Case = Can splfy dscrnan funcon: ln c P ln c P ln c P consan for all classes ln c P Thus n hs case dscrnan s also lnear w w ln Pc

39 Case = : Eaple 3 classes, each -densonal Gaussan wh c Pc c P P Aan can be done by solvn = j for,j=,,3

40 scalar row vecor Case = : Eaple j j j j ln Pc ln Pc j j j j ln Pc ln Pc j j j j Pc Pc ln Le s solve n eneral frs j Le s reroup he ers We e he lne where j

41 Case = : Eaple j ln j j Pc j Now subsue for,j=, Now subsue for,j=, Now subsue for,j=,3 Pc

42 Case = : Eaple Prors c 4 P c and c P P 3 c c 3 c lnes connecn eans are no n eneral perpendcular o decson boundares 4

43 General Case are arbrary Covarance arces for each class are arbrary In hs case, feaures,.,, d are no necessarly ndependen.5.5 j

44 44 Fro prevous dscusson, General Case are arbrary Ths can be splfed, bu we can rearrane : ln ln c P ln ln c P ln ln c P w w W

45 General Case are arbrary W w w quadrac n snce W d lnear n d j w j j consan n d, j w j j Thus he dscrnan funcon s quadrac Therefore he decson boundares are quadrac ellpses and parabollods 45

46 General Case are arbrary: Eaple 3 classes, each -densonal Gaussan wh Prors: 4 c Pc and c P P 3 Aan can be done by solvn = j for,j=,,3 ln lnp c Need o solve a bunch of quadrac nequales of varables

47 General Case are arbrary: Eaple c Pc c P 4 P c c c 3 c

48 Iporan Pons The Bayes classfer when classes are norally dsrbued s n eneral quadrac If covarance arces are equal and proporonal o deny ar, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he nu Eucledan dsance classfer If covarance arces are equal, he Bayes classfer s lnear If, n addon he prors on classes are equal, he Bayes classfer s he nu Mahalanobs dsance classfer Popular classfers Eucldean and Mahalanobs dsance are opal only f dsrbuon of daa s approprae noral