Pairwise-Covariance Linear Discriminant Analysis

Transcription

1 Proceedigs of the Twety-Eighth AAAI Coferece o Artificial Itelligece Pairwise-Covariace Liear Discrimiat Aalysis Deguag Kog ad Chris Dig Departmet of Computer Sciece & Egieerig Uiversity of Texas, Arligto, 500 UTA Blvd, TX doogkog@gmail.com; chqdig@uta.edu Abstract I machie learig, liear discrimiat aalysis (LDA) is a popular dimesio reductio method. I this paper, we first provide a ew perspective of LDA from a iformatio theory perspective. From this ew perspective, we propose a ew formulatio of LDA, which uses the pairwise averaged class covariace istead of the globally averaged class covariace used i stadard LDA. This pairwise (averaged) covariace describes data distributio more accurately. The ew perspective also provides a atural way to properly weigh differet pairwise distaces, which emphasizes the pairs of class with small distaces, ad this leads to the proposed pairwise covariace properly weighted LDA (pclda). The kerel versio of pclda is preseted to hadle oliear projectios. Efficiet algorithms are preseted to efficietly compute the proposed models. Itroductio I the big data era, a large umber of high-dimesioal data (i.e., DNA microarray, social blog, image scees, etc) are available for data aalysis i differet applicatios. Liear Discrimiat Aalysis (LDA) (Hastie, Tibshirai, ad Friedma 2001) is oe of the most popular methods for dimesio reductio, which has show state-of-the-art performace. The key idea of LDA is to fid a optimal liear trasformatio which projects data ito a low-dimesioal space, where the data achieves maximum iter-class separability. The optimal solutio to LDA is geerally achieved by solvig a eigevalue problem. Despite the popularity ad effectiveess of LDA, however, i stadard LDA model, istead of emphasizig the pairwise-class distaces, it simply takes a average of metrics computed i differet pairs (i.e., computatio of betwee-class scatter matrix S b or withi-class scatter matrix S w ). Thus, some pairwise class distaces are depressed, especially for those pairs whose origial class distaces are relatively large. To overcome this issue, i this paper, we preset a ew formulatio for pairwise liear discrimiat aalysis. To obtai a discrimiat projectio, the proposed method cosiders all the pairwise betwee-class ad with-class distaces. We call it pairwise-covariace LDA (pclda). The, the Copyright c 2014, Associatio for the Advacemet of Artificial Itelligece ( All rights reserved. pclda problem is cast ito solvig a optimizatio problem, which maximizes the class separability computed from pairwise distace. A efficiet algorithm is proposed to solve the resultat problem, ad experimetal results idicate the good performace of the proposed method. A ew perspective of LDA The stadard liear discrimiat aalysis (LDA) is to seek a projectio G =(g 1,, g K 1 ) 2< p (K 1) which maximizes the class separability by solvig, max Tr( GT S b G G G T S )=max wg G Tr(GT S b G)(G T S wg) 1, (1) where S w is the withi-class scatter matrix, ad S b is the betwee-class scatter matrix, ad give by S b = 1 KX k (µ k µ)(µ k µ) T, S w = = 1 KX k k, k, 1 X (x i k x i 2C k µ k )(x i µ k ) T, where k is the umber of data i class C k, µ k 2< p 1 is the mea for the data from class C k, µ is the global mea for all the data. I the history of LDA (Hastie, Tibshirai, ad Friedma 2001), the objective fuctio of LDA is evolved from Fisher s iitial 2-class LDA: max g g T S b g g T S wg. (2) For multi-class LDA, this ca be geeralized to either the trace-of-ratio of Eq.(1), or the followig ratio-of-traces objective: max G Tr(G T S b G) Tr(G T S wg). (3) Mathematically, both geeralizatio are atural; there is o clear differece i terms of machie learig. The trace-ofratio objective Eq.(1) is the most widely used oe. However, the ratio-of-trace objective of Eq.(3) has bee used by may researches, e.g., (Wag et al. 2007), (Kog ad Dig 2012), etc. To our kowledge, there exist o clear explaatios of the differeces betwee these two differet LDA objectives. I this paper, we bridge this gap, by providig theoretical support to the LDA objective of Eq.(1) from KL-divergece perspective, which is described i Theorem 1 below. 1925

2 (a) 2-dim Data distributio (b) LDA ad pclda (c) Elarged LDA ad pclda Figure 1: A sythetic data set of 150 data poits, 50 data of each class. (a) data distributio; (b) 1-dimesioal projectio of LDA ad pclda. Note that both the subspaces (lies) pass through (0,0). We shift them to avoid clutter. (c) Elarged 1-dim LDA ad pclda. From the KL-divergece to classic LDA LDA assumes that data poits of each class k are a Gaussia distributio. The covariace matrix of this class k is called the withi-class scatter matrix S k w. I this paper, we use covariace or averaged covariace istead of the usual withi-class scatter matrix S w to emphasize the ew perspective. The withi-class scatter matrix defied i Eq.(2) is the globally averaged (i.e., averaged over all k classes) covariace matrix. Furthermore, we propose the pairwise averaged covariace as a better formulatio which is used i pclda. We start with the KL-divergece betwee two Gaussia distributios N k (µ k, k ), N l (µ l, l ) with the same covariaces: k = l = kl. The KL-divergece of N k ad N l is: D KL(N k N l )= 1 2 (µ k µ l ) T 1 kl (µ k µ l ). (4) KL-divergece is used as a measure of distace betwee two classes. Whe the data are trasformed usig projectio G, i.e., we project x i to the subspace y i = G T x i, or Y = G T X, the KL-divergece i Y-space is D Y KL (N k N l )= 1 2 (µ k µ l ) T G(G T kl G) 1 G T (µ k µ l ). (5) We have the followig results. Theorem 1. Whe the covariaces of all K classes are idetical, i.e., k =,k =1 K, the sum of all pairwise KL-divergeces: J Y 0 = X k l DKL(N Y k N l ) (6) is idetical to the objective fuctio of stadard LDA of Eq.(1), where P = P K P K l=k+1. Proof: Note that (µ k µ l ) T 1 (µ k µ l ) =Tr[(µ k µ l )(µ k µ l ) T 1 ] =Tr[(µ k µ T k + µ l µ T l µ k µ T l µ l µ T k ) 1 ], we have J X 0 K = X KX k l Tr[(µ k µ T k + µ lµ T l µ k µ T l µ l µ T k ) 1 ] k=l l=1 KX =2Tr[ k (µ k µ)(µ k µ) T 1 ]=2Tr[S b 1 ]. (a) Stadard LDA. (b) Pairwise-covariace LDA. Figure 2: Results o Iris dataset with 3 classes, each class has 50 data poits. Origial 4-dimesioal data are projected ito 2 dimesios. (a) Results of stadard LDA; (b) Results of pairwisecovariace LDA. Now we project x i to the subspace y i = G T x i. The covariace i Y-space is Y = G T G ad the betwee-class scatter matrix becomes: S Y b = GT S b G. Thus J 0(G) =2Tr(G T S b G)(G T G) 1 (7) is idetical to the LDA objective fuctio of Eq.(1) aside from the uimportat costat 2. u Pairwise-covariace LDA Motivatio I stadard LDA, covariaces k of all K classes are assumed to be exactly idetical. This results i a stadard LDA of Eq.(1), as we ca see from Theorem 1. I practice, data covariace for each class is ofte differet. For 2-class problem, whe 1 6= 2, the quadratic discrimiat aalysis (QDA) (Hastie, Tibshirai, ad Friedma 2001) ca be used. However, i QDA, the boudary betwee differet classes is a quadratic surface, ad the discrimiat space ca ot be represeted by G T X explicitly. For multi-class, oe ca directly solve it usig the Gaussia mixture desity fuctio with Bayes rules. I this paper, we seek a discrimiat subspace that ca be obtaied by the liear trasformatio G T X, which has ot bee studied before. Illustrative example I most datasets, data variace for each class is geerally differet, stadard LDA uses the pooled (i.e., the global averaged) withi-class scatter matrices of all classes. However, the global averaged covariace S w could differ from each idividual covariace sig- 1926

3 ificatly. A simple example is show i Fig.1, where a 2- dimesioal data from three classes are show i Fig.(1(a)). Each class has 50 data poits. The covariace for data from each class is 1, 2, 3 : 1 = 3 = , 2 = ; 123 = ,. These idividual covariaces are very differet. I stadard LDA, we average all the classes ad obtai S w = 123. I this paper, we propose a formulatio of LDA that uses pairwise classes. The three pairwise averaged class-covariace: 12, 13 ad 23 are 12 = 23 = , 13 = , We see that the pairwise averaged covariace are much closer to the two idividual covariaces as compared to the global average. Formulatio For simplicity, we defie the distace d k,l (G) betwee two classes k, l as d k,l (G) =2D Y KL(N k, N l ), where DKL Y (N k, N l ) is defied i Eq.(5), ad kl is a pairwise covariace matrix (average of the pair of classes) ad defied as kl = k k + l l +(1 ). (8) k + l Here we use the globally averaged covariace = S w as a regularizatio. Parameter 0 apple apple 1 cotrols the balace of global covariace matrix ad local pairwise covariace matrix k, l. The pairwise-covariace LDA is defied the same as that i Theorem 1: max G J1(G) =X k l d kl (G), (9) where G 2< p (K 1) is the projectio. The objective i Eq.(9) is similar to stadard LDA (except that we use pairwise covariace istead of global averaged covariace). The proposed ew model Back to the form of Eq.9, it is easy to see that we ca defie a better objective. I maximizig J 1, all pairs of distaces are treated equally. However, i classificatio, we wish the pair of classes with smaller distaces to be give more weight, i.e., after projectig to Y = G T X subspace, they are more separated (as compared to other pairs of classes). O the other had, if two classes are already well-separated, i.e., their distaces are large, they ca have less weight i the objective fuctio. Therefore, we propose the followig pairwise covariace properly weighted objective fuctio: mi J2(G) =X k l G [d kl (G)], s.t. q GT G = I, (10) where q 1 is a hyper-parameter. I this objective fuctio, the pair of classes with smaller distaces cotribute more (a) Results of stadard LDA. (c) Covergece of algorithm. (b) Results of pclda. Figure 3: Data: 45 data poits (images) from 3 classes o mist dataset. Origial 784-dimesioal data are projected ito 2- dimesio. (a) Results of stadard LDA; (b) Results of pclda; (c) Covergece of algorithm o mist. Show are objective fuctio vs. iteratios. tha the pair of classes with larger distaces. Parameter q cotrols how much the pair of classes with smaller distaces are weighted. The larger q is, the stroger that pair of classes is weighted. I practice, we foud that q = {1, 2} are good choices. This model is our fial proposed model. For simplicity, we call it pairwise-covariace LDA (pclda) with the proper weightig implicit. As defied i Eq.(10), the objective is ivariat uder ay o-sigular trasformatio usig A 2< (K 1) (K 1), i.e., J 2 (GA) =J 2 (G). To fix this ucertaity, we require G T G = I. Illustratios of pclda We illustrate pclda o sythetic ad real data. I Fig.1, LDA ad pclda results o a sythetic 2D dataset of 150 data poits (50 data of each class) are show. We show the data distributio ad 1-dimesioal projectio results usig LDA ad pclda. The poit here is that the globally averaged covariace S w is a poor represetatio of the idividual covariaces, but the pairwise-covariace approach seems to give a better represetatio such that a sigle pclda dimesio ca clearly separate the 3 classes, while stadard LDA eeds 2-dimesios to separate data from differet classes (results ot show). I Fig.2, we show the results o the widely used iris data 1. Iris has 150 data poits with K=3 classes. Thus LDA project to K-1=2 dimesios. Fig.2 idicates that pclda gives clear discrimiatio betwee classes 2 ad 3 while stadard LDA has strog mixig betwee classes 2 ad 3. I Fig.3, we show results o 45 images (from K=3 classes) from mist hadwritte digits image dataset. LDA projectios to 2-dimesio are show. Result of pclda shows that

4 the 3 classes cotract strogly ad become more separated as compared to the LDA results. These results demostrate the beefits of the pairwise-covariace properly weighted LDA. More experimets ad comparisos with related methods are reported i 7. Algorithm to solve Pairwise-covariace LDA The key idea of our approach is to use gradiet descet algorithm to solve pclda of Eq.(10). The gradiet of J 2 (G) is = X For otatioal simplicity, we write q k kl (G) [d kl (G)] (11) B kl =(µ k µ l )(µ k µ l ) T, d kl (G) =Tr(G T B kl G)(G T kl G) 1. (12) Usig Eq.(12), the derivative of d kl (G) kl =2[B klg(g T kl G) 1 kl G(G T kl G) 1 (G T B kl G)(G T kl G) 1 ]. (13) Note that (G T kl G) 1 is a iverse of a small (K-1)-by- (K-1) matrix. rj 2 ca be efficietly computed usig Algorithm 1. Algorithm 1 Computatio of rj 2(G) (i.e., Eq.11) or rj 2(A) (i.e., gradiet of Eq.21). Iput: G, { k,µ k }, q Output: rj 2 Algorithm: 1: F =0 2: for l =1to K do 3: for k = l +1to K do 4: Compute µ kl = µ k µ l. 5: Compute b = G T µ kl. 6: Compute kl accordig to Eq.(8). % kl accordig to Eq.(23) 7: Compute B = kl G. 8: Compute b =(G T B) 1 b. 9: Compute a = k l (µ kl Bb)/(µ T kl Gb)q+1. 10: Compute F = F + a b T % cross-product betwee vectors a, b 11: ed for 12: ed for 13: rj 2 = 2qF. 14: Output: rj 2. The costrait G T G = I eforces G o the Stiefel maifold. Variatios of G o this maifold is parallel trasport, which gives some restrictio to the gradiet. This has bee bee worked out i (Edelma, Arias, ad Smith 1998). The gradiet that preserves the maifold structure is rj 2 G[rJ 2] T G. (14) Thus the algorithm computes the ew G as follows, G G (rj 2 G[rJ 2] T G) (15) The step size is usually chose as, = kgk 1k/krJ 2 G(rJ 2) T Gk 1, = (16) where kak 1 = P ij A ij. Occasioally, due to the loss of umerical accuracy, we do the projectio: G G(G T G) 1 2 to restore G T G = I. Startig with the stadard LDA solutio of G, this algorithm is iterated util the algorithm coverges to a local optimal solutio. Fig. 3(c) shows the covergece of algorithm o dataset mist. Pairwise-covariace Kerel LDA Kerel LDA (Mika et al. 1999; Tao et al. 2004) is oliear geeralizatio of LDA. We ca derive the kerel versio of pclda. Let x i! (x i ) or X! (X) = ( (x 1 ),, (x ). For 2-class LDA, the projectio vector is g = P i=1 i (x i ) = (X), where = ( P 1 ) T. For K-class LDA, the projectio vector g k = i=1 ik (x i )= (X) k, thus, G =(g 1 g K 1 )= (X)A, where A =( 1 K 1 ). Uder the trasformatio X! (X), G! (X)A, it is easy to see that the LDA objective of Eq.(1) trasforms ito Tr(G T S b G)(G T S w G) 1! Tr(A T S b A)(A T S w A) 1 (17) where the kerel withi-class scatter matrix is: ( k ) ij = (x i) T [ 1 k X (x s) (x s) T (x j) = 1 k X K isk sj, S w = 1 KX k ( k )= 1 K2,(18) ad the kerel betwee-class scatter matrix is: (S b ) ij = (x i) T 1 KX k ( k )( k )T (x j) = 1 KX k K i k K i K kj K j, (19) where we use the shorthad otatios: 1 X = (x s), k = 1 X (x s), k s=1 K i = K i = 1 X K is, K ki = K i k = 1 X k s=1 K is. (20) The solutio of kerel LDA is give by the largest k eigevectors of the eige-equatio S b v = S w v. Whe K =2, this reduces to the familiar 2-class kerel LDA (Tao et al. 2004). Efficiet computatio of S b is give i the ed of 5.1. We are ow ready to preset the pairwise-covariace kerel LDA. We apply the same trasformatio to the pairwisecovariace LDA. We have Theorem 2. Uder the trasformatio X! (X), G! (X)A, the pairwise-covariace LDA of J 2 (G) becomes J 2 (A): mi A X k l [Tr(A T B kl A)(A T kl A) 1 ] q,s.t.at A = I. (21) 1928

5 where (B kl ) ij = (x i) T ( k l)( k l) T (x j) = K i k K i l K kj K lj (22) where shorthad otatios are defied i Eq.(20), k is defied i Eq.(18), ad kl = k k + l l k + l +(1 ). (23) Algorithm for Kerel PC-LDA We solve J 2 (A) of Eq.(21) usig the same algorithm i computig pclda usig J 2 (G) of Eq.(10). The derivative is the same as Eqs.(19,20) except B kl is replaced by B kl, kl replaced by kl, G by A. The costrait A T A = I is hadled i same way as G T G = I i Eqs.(20,21). The step size is give i Eq.(22). The remaiig part is the efficiet computatio of the gradiet rj 2 (A). First, we ote that {B kl }, { k } of Eqs.(22,23) ca be efficietly computed. Let V k be a -by- k matrix cosistig of k colums of K belogig to class k. It is ready to see that i Eq.(21), k = 1 k V k V T k, u k = 1 k V k e, (24) where e =(1 1) T. Here for clarity, we use u k to represet the vector K i, k,i =1. Clearly, B kl =(u k u l )(u k u l ) T. Now rj 2 (A) is computed usig Algorithm 1, with the replacemet µ k u k, k k. (25) S b ca be efficietly computed as S b =(1/) P k k(u k v)(u k v) T, v =(1/) (X)e. Related Work A detailed survey of recet LDA works ca be foud i (Ye ad Ji 2008). Other LDA formulatio There exist earlier works (Li, Jiag, ad Zhag 2003), (Ya et al. 2004) which maximize the differece of traces, a.k.a maximum margi criteria (MMC). Several LDA formulatios with differet costraits ad overfit aalysis are give i (Luo, Dig, ad Huag 2011), (Ya et al. 2004). To solve the well-kow sigularity or uder-sampled problem, there are may extesios of LDA methods proposed, such as Regularized LDA (RLDA) (Hastie, Tibshirai, ad Friedma 2001), ucorrelated LDA (ULDA) (Ye 2005b), orthogoal LDA (OLDA) (Ye 2005a) ad orthogoal cetroid method (OCM) (Park, Jeo, ad Z 2003), etc. Amog these, ULDA extracts the feature vectors which are mutually ucorrelated i lowdimesioal space. Coectio with metric learig David et.al. (Alipaahi, Biggs, ad Ghodsi 2008) showed a strog relatioship betwee distace metric learig methods ad the Fisher Discrimiat Aalysis. Our pairwise-covariace LDA formulatio of Eq.(10) ad kerel pclda of Eq.(21) ca serve for distace metric learig purpose, which ca be used for may applicatios (e.g., (Kog ad Ya 2013), (Kog et al. 2012), etc). Table 1: Characteristics of datasets Dataset data dimesio Class MSRCv Umist Mist Bialpha There are also works discussig local discrimiative Gaussia (LDG) dimesioality reductio (Parrish ad Gupta 2012), local fisher discrimiat aalysis (Sugiyama 2006). Sparsity i the LDA solutio (Clemmese et al. 2011), (Zhag ad Chu 2013) is also desirable for iterpretatio purpose, because it is robustess to the oise ad will lead to efficiet computatio i predictio. However, to our kowledge, oe of the above works cosider the pairwise covariace by computig distace of the projectio i a pairwise way, which is the focus of this paper. Experimet results Dataset We evaluate the proposed pairwise-covariace LDA usig four data sets (see Table 1) for multi-class classificatio experimets, icludig oe face dataset umist, two digit datasets mist (Lecu et al. 1998), bialpha, oe image scee dataset MSRCv1 (Lee ad Grauma 2009) 2. Due to space limit, we omit more details of datasets. Table 1 summarizes the datasets. Methods & Parameter Settigs I our experimet, we use 5-roud 5-fold cross validatio to evaluate the classificatio performace. Each dataset is evely partitioed ito 5 parts. Oly oe part is used as testig ad the other 4 parts are used for traiig. We report the average results for 5 rouds. Next, we give a overview of the dimesio reductio ad classificatio methods used i our experimet. The compared methods ca be divided ito several groups. (1) LDA ad MMC (Li, Jiag, ad Zhag 2003; Ya et al. 2004), kerel LDA (KLDA) For LDA, maximum margi criterio(mmc) ((Li, Jiag, ad Zhag 2003; Ya et al. 2004)), kerel-lda of Eq.(17) method, we project origial data ito LDA-subspace, ad k(k=3) earest eighbor classifier is used for classificatio. For kerel LDA, we use RBF kerel to costruct the pairwise similarity W ij = kx e i x jk 2, where badwidth is searched i the grid {10 4, 10 3,, 10 3, 10 4 }. (2) Regularized LDA (RLDA) (Hastie, Tibshirai, ad Friedma 2001), ucorrelated LDA (ULDA) (Ye 2005b), orthogoal LDA (OLDA) (Ye 2005a) ad orthogoal cetroid method (OCM) (Park, Jeo, ad Z 2003). We compare our method agaist four methods of geeralized LDA. It has bee show (Ye ad Ji 2008) that these four LDAextesios ca be described i a uified framework for geeralized LDA. However, there still exist subtle differeces amog them. The parameter µ i regularized LDA is determied by cross validatio. (3) Proposed pairwise-covariace LDA model of

6 Table 2: Multi-class Classificatio Accuracy o 4 datasets usig 9 differet dimesio reductio methods: LDA, kerel LDA(KLDA), pclda, kerel pclda (pcklda), ad 5 other methods: MMC, RLDA, ULDA, OLDA, OCM. Data LDA MMC RLDA ULDA OLDA OCM pclda ( =1) KLDA pcklda( =1) MSRC Bialpha Mist Umist Average classificatio accuracy Average classificatio accuracy LDA MMC RegLDA ULDA OLDA OCM pclda (β=0.1) pclda (β=0.5) pclda (β=1) KLDA pcklda (β=0.1) pcklda (β=0.5) pcklda (β=1) 0 MSRC Dataset Bialpha 0 mist Dataset umist (a) Classificatio results o MSRC, Bialpha (b) Classificatio results o mist, umist Figure 4: Classificatio results comparisos o 4 datasets, icludig our methods: pclda, pcklda at methods: LDA, KLDA, MMC, RLDA, ULDA, OLDA, OCM. = {0.1, 0.5, 1} ad seve other β (a) pclda result o MSRC β (b) pclda result o mist β (c) pclda result o umist Figure 5: Classificatio accuracy w.r.t differet parameter for our model of Eq.(10) o dataset MSRC, mist, umist. Red lie gives LDA results, ad blue lie draws pclda results at = {0, 0.1,, 0.9, 1.0}. Eq.(10)(pcLDA) ad kerel pairwise-covariace LDA model (pcklda) of Eq.(21) We set q = 1 for Eq.(10), Eq.(21) i our experimets. The parameter is set to be {0.1, 0.5, 1}. To make a fair compariso, we project all origial data to (C-1) dimesio, ad k(k=3) earest eighbor classifier is used for classificatio purpose. Classificatio Performace Aalysis Table 2 ad Fig.4 preset the classificatio performace usig differet dimesio reductio methods. We make several importat observatios from experimet results. (1) As compared to stadard LDA, MMC ad other dimesio reductio methods, pclda cosistetly provides better classificatio performace at differet values (e.g., = {0.1, 0.5, 1}). For example, there is early 5% performace improvemet o bialpha dataset whe compared with stadard LDA method. Note bialpha dataset is composed of data from K=36 classes, this idicates that the proposed pairwise pairwise-covariace LDA method gives much performace improvemet at large class umbers. (2) I kerel space, kerel versio of LDA ad pclda do ot improve the classificatio performace quite a bit (sometimes eve worse). However, pcklda still outperforms stadard KLDA i kerel space. (3) cotrols the complexity of our model, i.e., whe approaches 1, pclda uses local pairwise covariace matrix, ad whe approaches 0, pclda uses global covariace matrix which is equivalet to stadard LDA. Fig.(5) shows the classificatio results o three datasets: MSRC, mist ad umist. The experimet results suggest that, geerally, we ted to get better classificatio results for larger values of. This further cofirms our ituitio, the pairwise covariace really helps to capture the data distributio as compared to globally averaged variace, ad thus the projectio ad classificatio results are improved. Moreover, rather tha maximizig the sum of iter-class distaces, we miimize the sum of iverse iter-class distaces. This choice makes classes that are close together have more ifluece o the LDA fit tha those classes that are well-separated. Coclusio We preset a pairwise-covariace model for liear discrimiat aalysis. The proposed model computes the projectio by utilizig the pairwise class iformatio. A efficiet algorithm is preset to solve the proposed model. Proposed method ca be easily exteded i kerel space. Experimet results idicate the good performace of proposed method. 1930

7 Ackowledgemet. This research is partially supported by NSF-CCF ad NSF-DMS grats. Refereces Alipaahi, B.; Biggs, M.; ad Ghodsi, A Distace metric learig vs. fisher discrimiat aalysis. I AAAI. Clemmese, L.; Hastie, T.; Wiite, D.; ad Ersboll, B Sparse discrimiat aalysis. Techometrics. Edelma, A.; Arias, T. A.; ad Smith, S. T The geometry of algorithms with orthogoality costraits. SIAM J. MATRIX ANAL. APPL 20(2): Hastie, T.; Tibshirai, R.; ad Friedma, J The Elemets of Statistical Learig: Data Miig, Iferece, ad Predictio. Spriger. Hoi, S. C. H.; Liu, W.; Lyu, M. R.; ad Ma, W.-Y Learig distace metrics with cotextual costraits for image retrieval. I CVPR. Kog, D., ad Dig, C. H. Q A semi-defiite positive liear discrimiat aalysis ad its applicatios. I ICDM, Kog, D., ad Ya, G Discrimiat malware distace learig o structural iformatio for automated malware classificatio. I KDD, Kog, D.; Dig, C. H. Q.; Huag, H.; ad Zhao, H Multi-label relieff ad f-statistic feature selectios for image aotatio. I CVPR, Lecu, Y.; Bottou, L.; Begio, Y.; ad Haffer, P Gradiet-based learig applied to documet recogitio. I Proceedigs of the IEEE, Lee, Y. J., ad Grauma, K Foregroud focus: Usupervised learig from partially matchig images. Iteratioal Joural of Computer Visio 85(2): Li, H.; Jiag, T.; ad Zhag, K Efficiet ad robust feature extractio by maximum margi criterio. I Proceedigs of Advaces i Neural Iformatio Processig Systems(NIPS 2003). Luo, D.; Dig, C.; ad Huag, H Liear discrimiat aalysis: New formulatios ad overfit aalysis. I AAAI2011. Mika, S.; Ratsch, G.; Westo, J.; Scholkopf, B.; ad Muller, K Fisher discrimiat aalysis with kerels. Park, H.; Jeo, L. M.; ad Z, J. B. R Lower dimesioal represetatio of text data based o cetroids ad least squares. BIT 43:2003. Parrish, N., ad Gupta, M Dimesioality reductio by local discrimiative gaussias. I ICML. Sugiyama, M Local fisher discrimiat aalysis for supervised dimesioality reductio. I ICML, Tao, X.; Ye, J.; Li, Q.; Jaarda, R.; ad Cherkassky, V Efficiet kerel discrimiat aalysis via qr decompositio. I The Eighteeth Aual Coferece o Neural Iformatio Processig Systems (NIPS 2004), Wag, H.; Ya, S.; Xu, D.; Tag, X.; ad Huag, T Trace ratio vs. ratio trace for dimesioality reductio. I CVPR. Xiag, S.; Nie, F.; ad Zhag, C Learig a mahalaobis distace metric for data clusterig ad classificatio. Ya, J.; Zhag, B.; Ya, S.; Yag, Q.; ad Li, H Immc: icremetal maximum margi criterio. I Proceedigs of the Teth ACM SIGKDD Iteratioal Coferece o Kowledge Discovery ad Data Miig. Ye, J., ad Ji, S Discrimiat Aalysis for Dimesioality Reductio: A Overview of Recet Developmets, Biometrics: Theory, Methods & Applicatios. IEEE/Wiley. Ye, J. 2005a. Characterizatio of a family of algorithms for geeralized discrimiat aalysis o udersampled problems. The Joural of Machie Learig Research 6. Ye, J. 2005b. Characterizatio of a family of algorithms for geeralized discrimiat aalysis o udersampled problems. Joural of Machie Learig 6: Zhag, X., ad Chu, D Sparse ucorrelated liear discrimiat aalysis. I ICML,