Multivariate Methods. Matlab Example. Principal Components Analysis -- PCA

Transcription

1 Multivariate Methos Xiaoun Qi Principal Coponents Analysis -- PCA he PCA etho generates a new set of variables, calle principal coponents Each principal coponent is a linear cobination of the original variables All the principal coponents are orthogonal to each other so there is no reunant inforation he principal coponents as a whole for an orthogonal basis for the space of the ata he first principal coponent is a single axis in space hen you proect each observation on that axis, the resulting values for a new variable An the variance of this variable is the axiu aong all possible choices of the first axis he secon principal coponent is another axis in space, perpenicular to the first Proecting the observations on this axis generates another new variable he variance of this variable is the axiu aong all possible choices of this secon axis 3 4 Matlab Exaple he full set of principal coponents is as large as the original set of variables But it is coonplace for the su of the variances of the first few principal coponents to excee 80% of the total variance of the original ata By exaining plots of these few new variables, researchers often evelop a eeper unerstaning of the riving forces that generate the original ata 5 ata his ata has 0 rows cols

2 variability in the two coluns boxplot(ata,0 [pc, newata, variances, t] princop(ata ; variance pc newata figure(; plot(ata(:,, ata(:,,'' xlabel('st Diension'; ylabel('n Diension'; gnae; figure( ; plot(newata(:,,'' ylabel('st Principal Coponent'; gnae ; figure(3 ; plot(newata(:,,newata(:,,'' xlabel('st Principal Coponent'; ylabel('n Principal Coponent'; gnae; 9 0

3 PCA Illustration PCA Illustration Explanation Given an nxn atrix that oes have eigenvectors, there are n of the Scale the vector by soe aount before ultiplying it, the sae ultiple of it will be obtaine All the eigenvectors of a atrix are perpenicular, ie, at right angles to each other, no atter how any iensions you have You can express the ata in ters of these perpenicular eigenvectors, instea of expressing the in ters of the x an y axes 3 4 Possible Use of PCA Diensionality reuction he eterination of linear cobinations of variables Feature selection: the choice of the ost useful variables Visualization of ultiiensional ata Ientification of unerlying variables Ientification of groups of obects or of outliers Linear Discriinant Analysis -- LDA PCA sees irections that are efficient for representation; Discriinant analysis sees irections that are efficient for iscriination o obtain goo separation of the proecte ata, we really want the ifference between the eans to be large relative to soe easure of the stanar eviation for each class 5 6 LDA Illustration -- Ba Separation B w A 7 LDA Illustration -- Goo Separation B A w 8

4 LDA Illustration -- -class case Variants of the LDA If the LDA is a class epenent type, for L- class, L separate optiizing criteria are require for each class he optiizing factors in case of class epenent type are copute as: Criterion inv(cov * B For the class inepenent transfor, the optiizing criterion is copute as: Criterion - * B LDA on Expane Basis Expan input space to inclue XX, XX an XX if the input has a -D feature (X, X In this case, the input has 5-D instea of -D hat is: X (X, X, XX, XX, XX Quaratic Discriinant Analysis (QDA -- Bayes Classifier Probability consierations becoe iportant in pattern recognition because of the ranoness uner which pattern classes norally are generate It is possible to erive a classification approach that is optial in the sense that, on average, its use yiels the lowest probability of coitting classification errors Bayes Rules -- Conitional Probability he conitional probability of an event B in relationship to an event A is the probability that event B occurs given that event A has alreay occurre he notation for conitional probability is: B A A B B A B B B A A A B B B A A A B B B A A B A A 3 Founation he probability that a particular pattern X coes fro class wi is enote p(wi/x If the pattern classifier ecies that x cae fro w when it actually cae fro wi, it incurs a loss, enote Li As pattern x ay belong to any one of classes uner consieration, the average loss incurre in assigning x to class w is: r L p( x his equation often is calle the conitional average ris or loss in ecision-theory terinology Equation 4

5 Fro basic probability theory, we now that B A A A B B B A A B A A Using this expression, we write Equation in the for: r Equation L p( x p( x here p(x/w is the probability ensity function of the patterns fro class w an w is the probability of occurrence of class w 5 Because /p(x is positive an coon to all the r(x,,,,, it can be roppe fro Equation without affecting the relative orer of these functions fro the sallest to the largest value he expression for the average loss then reuces to: r L p( x Equation 3 6 he classifier has possible classes to choose fro for any given unnown pattern If it coputes r(x, r(x,, rw(x for each pattern x an assigns the pattern to the class with the sallest loss, the total average loss with respect to all ecisions will be iniu he classifier that iniizes the total average loss is calle the Bayes classifier hus the Bayes classifier assigns an unnown pattern x to class wi if ri(x < r(x for,,, w; i In other wors, x is assigne to class wi if i for,,, w; i L p( x < L p( x Equation 4 q q q q 7 he loss for a correct ecision generally is assigne a value of zero, an the loss for any incorrect ecision usually is assigne the sae nonzero value (say Uner these conitions, the loss function becoes L i δ i Equation 5 where δ i if i, an δi 0 if i Equation 5 inicates a loss of unity for incorrect ecisions an a loss of zero for correct ecisions Substituting Equation 5 into Equation 3 yiels r ( δ p( x Equation 6 p( x p( x 8 he Bayes classifier then assigns a pattern x to class wi if for all i p x p( x < p( x p( x ( i i Equation 7 Or equivalently, if p( x i i > p( x,, K, ; i Equation 8 he Bayes classifier for a 0- loss function is nothing ore than coputation of ecision function of the for: p( x,, K Equation 9, here a pattern vector x is assigne to the class whose ecision function yiels the largest nuerical value 9 he ecision functions in Equation 9 are optial in the sense that they iniize the average loss in isclassification For this optiality to hol, the probability ensity functions of the patterns in each class, as well as the probability of occurrence of each class ust be nown he latter requireent usually is not a proble since it can generally be inferre fro nowlege of the proble Estiation of the probability ensity function p(x/w is another atter If the pattern vectors, x, are n iensional, then p(x/w is a function of n variables, which requires ethos fro ultivariate probability theory for its estiation 30

6 hese ethos are ifficult to apply in practice, especially if the nuber of representative patterns fro each class is not large, or if the unerlying for of the probability ensity functions is not well behave Use of the Bayes classifier generally is base on the assuption of an analytic expression for the various ensity functions an then an estiation of the necessary paraeters fro saple patterns fro each class By far, the ost prevalent for assue for p(x/w is the Gaussian probability ensity function he closer this assuption is to the reality, the closer the Bayes classifier approaches the iniu average loss in classification 3 Bayes Classifier for Gaussian Pattern Classes Let us consier a -D proble (n involving two pattern classes ( governe by Gaussian ensities, with eans an an stanar eviations σ an σ, respectively Fro Equation 9, the Bayes ecision functions have the for: p( x ( x Equation 0 σ e, πσ where the patterns are now scalars 3 his figure shows a plot of the probability ensity functions for the two classes he bounary between the classes are a single point, enote x0 such that (x0 (x0 If the two classes are equally liely to occur, then w w ½, an the ecision bounary is the value of x0 for which p(x0/wp(x0/w his point is the intersection of the two probability ensity functions 33 Any pattern (point to the right of x0 is classifie as belonging to class w Siilarly, any pattern to the left of x0 is classifie as belonging to class w hen the classes are not equally liely to occur, x0 oves to the left if class w is ore liely to occur or, conversely, to the right if class w is ore liely to occur Since the classifier is trying to iniize the loss of isclassification For instance, in the extree case, if class w never occurs, the classifier woul never ae a istae by always assigning all patterns to class w (that is, x0 woul ove to negative infinity 34 In the n-iensional case, the Gaussian ensity of the vectors in the th pattern class has the for: ( x C ( x p( x / e Equation n (π C here each ensity is specifie copletely by its ean vector an covariance atrix C, which are efine as E {x} x N C {( x ( x } C E N x x xx 35 Because of the exponential for of the Gaussian ensity, woring with the natural logarith of this ecision function is ore convenient hat is: ln[ p( x ] ln p( x ln It is equivalent to Equation 9 in ters of classification perforance because the logarith is onotonically increasing function In other wors, the nuerical orer of the ecision function in Equation 9 an Equation is the sae Equation 36

7 Substituting Equation into Equation yiels n ln ln π ln C [( x C ( x ] Equation 3 he ter (n/lnπ is the sae for all classes, so it can be eliinate fro Equation 3: [ ] ln ln C ( x C ( x for,,, Equation 4 he Equation 4 represents the Bayes ecision functions for Gaussian pattern classes uner the conition of a 0- loss function his equation is also calle a quaratic iscriinant function 37 Several Cases -- Case he features are statistically inepenent with the sae variance for all classes Ci σ ln ln C ( x C ( x ( x ln σ Since the iscriinant is linear, the ecision bounaries will be hyper-planes [ ] 38 Case Exaple Case he classes have the sae covariance atrix, but the features are allowe to have ifferent variances N N x[ ] [ ] [ ] ln log σ σ his iscriinant is linear So the ecision bounaries will be hyper-planes Case Exaple Case 3 All the classes have the sae covariance atrix, but this is no longer iagonal ln x C C his iscriinant is linear, so the ecision bounaries will also be hyper-planes 4 4

8 Case 3 Exaple Case 4 Each class has a ifferent covariance atrix, which is proportional to the ientity atrix ln N lnσ ( x σ ( x [ ] his expression cannot be reuce further so: he ecision bounaries are quaratic: hyperellipses Case 4 Exaple 45 Case 5 -- he Most General Case he covariance atrices of ifferent classes are not the sae ln ln C [( x C ( x ] Reorganizing ters in a quaratic for yiels x x w x w 0 C w C w 0 C ln C ln 46 Case 5 Exaple 47 A New Approxiation Metho of the Quaratic Discriinant Function -- By S Oachi, F Sun, an H Aso In orer to avoi the isavantages of the quaratic iscriinant function, they have propose a new approxiation etho of the quaratic iscriinant function his approxiation is one by replacing the values of sall eigenvalues by a constant which is estiate by the axiu lielihoo estiation By applying this approxiation, a new iscriinant function, siplifie quaratic iscriinant function, or SQDF, has been efine his function not only reuces the coputational cost but also iproves the classification accuracy 48

9 A Quaratic Discriinant Function Base on Bias Rectification of Eigenvalues -- By M Saai, M Yonea, H Hase, et al hey propose a new quaratic iscriinant function hey show that the eigenvalues of a covariance atrix obtaine fro saples are biase hey escribe how to rectify the an how to use the rectifie eigenvalues 3 In orer to erive the relation between saple eigenvalues an true eigenvalues, they analyze the biases of the expecte saple eigenvalues by using the perturbation etho an obtain approxiate siultaneous linear equations to rectify saple eigenvalues in ultiiensional noral cases 4 hey show by a Monte Carlo etho that their iscriinant function wors effectively in eightiensional noral cases especially in the case of a sall saple size 49 Suary he various iscriinant ethos are evelope to have optial properties uner various istributional assuptions However, the purpose of iscriinant analysis is to iscriinate herefore, the proper assessent of a iscriinant proceure for a particular ata set is not how well the ata fit the assuptions, but how well the proceure wors on a valiation ata set 50