HIGH-DIMENSIONAL data are common in many scientific

Transcription

1 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER Kerel Ucrrelated ad Regularized Discrimiat Aalysis: A heretical ad Cmputatial Study Shuiwag Ji ad Jiepig Ye, Member, IEEE Abstract Liear ad kerel discrimiat aalyses are ppular appraches fr supervised dimesiality reducti. Ucrrelated ad regularized discrimiat aalyses have bee prpsed t vercme the sigularity prblem ecutered by classical discrimiat aalysis. I this paper, we study the prperties f kerel ucrrelated ad regularized discrimiat aalyses, called KUDA ad KRDA, respectively. I particular, we shw that uder a mild cditi, bth liear ad kerel ucrrelated discrimiat aalysis prject samples i the same class t a cmm vectr i the dimesiality-reduced space. his implies that ucrrelated discrimiat aalysis may suffer frm the verfittig prblem if there are a large umber f samples i each class. We shw that as the regularizati parameter i KRDA teds t zer, KRDA appraches KUDA. his shws that KUDA is a special case f KRDA ad that regularizati ca be applied t vercme the verfittig prblem i ucrrelated discrimiat aalysis. As the perfrmace f KRDA depeds the value f the regularizati parameter, we shw that the matrix cmputatis ivlved i KRDA ca be simplified, s that a large umber f cadidate values ca be crss-validated efficietly. Fially, we cduct experimets t evaluate the prpsed theries ad algrithms. Idex erms Discrimiat aalysis, kerel methds, regularizati, mdel selecti, sigular value decmpsiti. Ç 1 INRODUCION HIGH-DIMENSIONAL data are cmm i may scietific ad egieerig fields, icludig face recgiti [2], [23], [30], [31], text ifrmati retrieval [13], gee expressi patter image classificati [35], ad micrarray data aalysis [5], [25], [26], [29]. Learig frm such data is challegig due t the curse-f-dimesiality [3]. Oe cmmly attempted apprach is t first reduce the dimesiality f data befre learig algrithms are applied. Pricipal Cmpet Aalysis (PCA) [17] is a classical apprach fr usupervised dimesiality reducti. PCA captures the maximum variace f data, ad it is ptimal i terms f miimum recstructi errr. Liear Discrimiat Aalysis (LDA) [8] is a supervised apprach, ad it pursues prjecti subspace with maximum betweeclass separati ad miimum withi-class dispersi. Bth PCA ad LDA have bee applied i varius tasks [2], [19], [20], [23], [31], [40]. I cases where class labels are available, LDA usually utperfrms PCA i terms f classificati accuracy. Ideed, LDA has bee shw t be effective i dealig with high-dimesial data [11], [18], [21], [30]. Meawhile, a umber f drawbacks exist fr classical LDA whe applied t small sample size ad cmplex data sets. First, whe the dimesiality f data exceeds the sample size, all scatter matrices i LDA are sigular ad. he authrs are with the Departmet f Cmputer Sciece ad Egieerig, Ariza State Uiversity f empe, 699 Suth Mill Aveue, empe, AZ {shuiwag.ji, jiepig.ye}@asu.edu. Mauscript received 16 July 2007; revised 17 Feb. 2008; accepted 4 Mar. 2008; published lie 14 Mar Fr ifrmati btaiig reprits f this article, please sed t: tkde@cmputer.rg, ad referece IEEECS Lg Number KDE Digital Object Idetifier /KDE classical LDA cat be applied. his is kw as the sigularity r udersampled prblem i LDA, ad may techiques have bee prpsed t vercme this prblem [2], [4], [7], [13], [30], [34], [39], [40]. See Secti 2.2 fr a verview. Furthermre, features btaied by LDA are liear cmbiatis f the rigial iputs, ad hece, it cat capture liear structures i cmplex data sets. Kerel methd [28] prvides pwerful ad tractable extesis f liear mdels t liear cases, ad it has bee used t exted LDA t the kerel-iduced feature space [1], [19], [22], [33]. Ucrrelated LDA (ULDA) [16], [36] has bee develped i the ctext f face recgiti, ad it extracts features that are ucrrelated i the dimesiality-reduced space. It has bee shw that ULDA vercmes the sigularity prblem by ptimizig a geeralized Fisher criteri [34]. I ctrast, Regularized LDA (RLDA) [7] vercmes the sigularity prblem by regularizig the ttal scatter matrix, thus makig it sigular. I this paper, we study the relatiship betwee ucrrelated ad regularized discrimiat aalysis i the kerel-iduced feature space. We als develp algrithm fr efficiet mdel selecti i regularized discrimiat aalysis. We shw that uder a mild cditi, which teds t hld fr high-dimesial data, ULDA prjects all samples i the same class t e cmm vectr i the dimesiality-reduced space. If there are a large umber f samples i each class, ULDA may verfit ise i the data. his result is als applicable t Null space LDA (NLDA) [4], which maximizes the betwee-class distace i the ull space f withi-class scatter matrix. Mrever, sice kerel methds first prject the iput data it sme highdimesial feature space ad ULDA teds t verfit fr /08/$25.00 ß 2008 IEEE Published by the IEEE Cmputer Sciety

2 1312 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER 2008 high-dimesial data, we ivestigate the behavir f ULDA i the kerel-iduced feature space. I particular, we shw that whe the kerel matrix is sigular, ULDA i the feature space, called Kerel Ucrrelated Discrimiat Aalysis (KUDA), als prjects samples i the same class t e cmm vectr. We further shw that as the regularizati parameter i the Kerel Regularized Discrimiat Aalysis (KRDA) teds t zer, KRDA appraches KUDA. his prvides theretical evidece that regularizati ca be applied t vercme the verfittig prblem i ucrrelated discrimiat aalysis. he perfrmace f KRDA depeds critically the value f the regularizati parameter, as a large value may sigificatly disturb the ifrmati i the scatter matrix, while a small value may t be effective eugh t slve the sigularity ad verfittig prblems. Crss validati is cmmly used t estimate the ptimal value frm a fiite set f cadidates. Hwever, the cmputatial cst f mdel selecti [12] fr KRDA usig crss validati ca be high, especially whe the cadidate set is large, sice it requires expesive matrix cmputatis fr each cadidate. Based the theretical aalysis f KRDA i this paper, we prpse a mdel selecti algrithm that ca chse the ptimal regularizati value frm a large umber f cadidate values efficietly. he key ctributis f this paper are highlighted as fllws:. We study the prperties f liear ad kerel ucrrelated discrimiat aalysis theretically. We shw that uder a mild cditi, ucrrelated discrimiat aalysis prjects samples i the same class t a cmm vectr i the dimesialityreduced space.. We shw that as the regularizati parameter i KRDA teds t zer, KRDA appraches KUDA. his elucidates the iheret relatiship betwee these tw differet algrithms ad prvides theretical evidece that regularizati ca be used t vercme the verfittig prblem i a ucrrelated discrimiat aalysis.. We shw that the matrix cmputatis ivlved i KRDA ca be simplified, ad we develp a efficiet mdel selecti algrithm t tue the regularizati parameter i KRDA usig crss validati.. We cduct extesive experimets t evaluate the prpsed theries ad algrithms. he rest f this paper is rgaized as fllws: Secti 2 reviews LDA ad its geeralizatis. We study the prperties f ULDA ad KUDA i Secti 3. he relatiship betwee KUDA ad KRDA is established i Secti 4. Secti 5 presets the mdel selecti algrithm fr KRDA. We reprt the experimetal results i Secti 6. Fially, Secti 7 ccludes this paper with discussis ad future wrk. 2 BACKGROUND 2.1 Overview f Liear Discrimiat Aalysis LDA is a supervised dimesiality reducti techique. It prjects high-dimesial data t a lwer dimesial subspace by maximizig the separati f data frm differet classes ad miimizig the dispersi f data frm the same class simultaeusly, thus achievig maximum discrimiati i the dimesiality-reduced subspace. Give a data matrix X ¼½x 1 ; ;x Š2IR m, where x i 2 IR m, fr i ¼ 1; ;, is the ith data pit, LDA fids a trasfrmati matrix G 2 IR m that trasfrms x i i the m-dimesial space t a vectr i the -dimesial space as fllws: x i 2 IR m! G x i 2 IR : Assume that the data matrix X is gruped as X ¼½X 1 ;...; Š; where X i 2 IR m i csists f the i data pits frm the ith class ad P k i ¼ : I LDA, three matrices, called withiclass, betwee-class, ad ttal scatter matrices are defied as fllws: S w ¼ 1 S b ¼ 1 S t ¼ 1 X X x2x i ðx c i Þðx c i Þ ; i ðc i cþðc i cþ ; ðx i cþðx i cþ ; where c i is the cetrid f the ith class, ad c is the glbal cetrid. It ca be verified that S t ¼ S w þ S b. Defie three matrices as fllws: H w ¼ 1 pffiffiffi X 1 c 1 e ð1þ ;...;Xk c k e ðkþ ð1þ ð2þ ð3þ ; ð4þ H b ¼ 1 pffiffiffiffiffi pffiffiffiffiffi pffiffiffi ½ 1ðc1 cþ;...; kðck cþš; ð5þ H t ¼ p 1 ffiffiffi ðx ce Þ; ð6þ where e ðiþ 2 IR i ad e 2 IR are vectrs f all es. he, the three scatter matrices i (1)-(3) ca be expressed as S w ¼ H w H w ; S b ¼ H b H b ; S t ¼ H t H t : ð7þ It fllws frm the prperties f matrix trace that traceðs w Þ¼ 1 traceðs b Þ¼ 1 X x2x i kx c i k 2 2 ; i kc i ck 2 2 : Hece, the trace f S w measures the distace betwee data pit t its class cetrid, ad the trace f S b measures the distace betwee each class cetrid t the glbal cetrid. ð8þ ð9þ

3 JI AND YE: KERNEL UNCORRELAED AND REGULARIZED DISCRIMINAN ANALYSIS: A HEOREICAL AND COMPUAIONAL SUDY 1313 he three scatter matrices i the dimesiality-reduced space trasfrmed by G, deted as Sw L, SL b, ad SL t, ca be expressed as Sw L ¼ G S w G; Sb L ¼ G S b G; St L ¼ G S t G: ð10þ LDA trasfrmati aims at maximizig the discrimiative structures f the data i the dimesiality-reduced space. his ca be achieved by miimizig traceðsw LÞ ad maximizig traceðsb L Þ simultaeusly, resultig i the fllwig ptimizati prblem emplyed i classical LDA [8]: 1S G ¼ argmax trace Sw L L b : ð11þ G It is kw [8] that the sluti t the ptimizati prblem i (11) ca be btaied by slvig the fllwig geeralized eigevalue prblem: S b y i ¼ i S w y i ; ð12þ whse eigevectrs crrespdig t the k 1 largest eigevalues frm clums f G. Whe S w is sigular, this prblem reduces t a eigevalue prblem as Sw 1 S by i ¼ i y i : ð13þ Whe the dimesiality f data is larger tha the umber f samples, all f the three scatter matrices are sigular ad classical LDA cat be applied. his is the well-kw sigularity r udersampled prblem i LDA. Nte that sice S t ¼ S w þ S b, S t is cmmly used i place f S w i the ptimizati i (11) [34]. 2.2 Extesis f LDA May differet appraches have bee prpsed t vercme the sigularity prblem f classical LDA. PCAþLDA [2], [30], [40] wrks by first reducig the dimesiality f data usig PCA befre classical LDA is applied. Sice the matrix S t is psitive semidefiite, RLDA [7] vercmes the sigularity prblem by addig a multiple f the idetity matrix t S t,ass t þ I m, fr sme regularizati parameter >0. LDA/GSVD [13] avids the sigularity prblem by diagalizig the scatter matrices simultaeusly usig Geeralized Sigular Value Decmpsiti (GSVD) [10]. A family f algrithms fr geeralized discrimiat aalysis (GDA) has bee prpsed i [34]. ULDA ad Orthgal LDA (OLDA) have bee shw t be special cases i this framewrk. Che et al. [4] bserved that the ull space f S w ctais critical discrimiative ifrmati, ad they prpse t maximize the betwee-class distace withi the ull space f withi-class scatter matrix. his methd is kw as NLDA. It has bee shw [38] that uder a mild cditi, NLDA is equivalet t OLDA. Direct LDA (DLDA) [39] restricts the sluti space t the rage space f S b, resultig i a efficiet algrithm. Meawhile, LDA has bee exteded t the liear case by usig the kerel trick [28]. Mika et al. [22] exteded LDA t its kerel versi i the biary-class case. Fllwig the wrk i [27], Baudat ad Auar [1] prpsed the GDA fr multiclass prblems. Lu et al. [19] exteded the DLDA [39] t its kerel versi. 3 LINEAR AND KERNEL UNCORRELAED DISCRIMINAN ANALYSIS ULDA [16], [36], [37] extracts features that are ucrrelated i the dimesiality-reduced space. It has bee shw [34] that the sluti t ULDA ca be fud by slvig the fllwig ptimizati prblem: þg G ¼ argmax trace G S t G S b G ; ð14þ G where ðg S t GÞ þ detes the pseudiverse [10] f G S t G. It has bee bserved experimetally that the perfrmace f ULDA may degrade as the umber f traiig samples icreases. prvide theretical explaatis fr this pheme, we shw that uder a mild cditi, which teds t hld fr high-dimesial data, ULDA prjects all data pits frm the same class t e cmm vectr i the dimesiality-reduced space. Fr ULDA i the kerel-iduced feature space, i.e., KUDA, we shw that if the kerel matrix is sigular, KUDA prjects data pits belgig t the same class t e cmm vectr. herefre, whe the umber f data pits i each class is large, ULDA ad KUDA may verfit ise i the data. his prvides theretical explaatis fr the behavirs f ULDA ad KUDA whe the umber f data pits i each class is large. 3.1 Prperties f ULDA We shw that if the trasfrmati f LDA is i the ull space f the withi-class scatter matrix, it will prject all samples i the same class t e cmm vectr i the dimesiality-reduced space. his is summarized i the fllwig prpsiti. Prpsiti 1. Let G be a trasfrmati matrix, ad let x be a data pit frm the ith class. Assume that G S w ¼ 0. he, G x ¼ G c ðiþ, where c ðiþ is the cetrid f the ith class. hat is, all data pits frm the ith class X i are mapped t a cmm vectr G c ðiþ. Prf. Frm (1), we have 0 ¼ G S w G ¼ 1 X G x c ðiþ x c ðiþ ð15þ G: x2x i It fllws that G x c ðiþ ¼ 0, fr all x 2 Xi. Hece, G x ¼ G c ðiþ. his cmpletes the prf. tu I NLDA [4], the betwee-class distace is maximized i the ull space f withi class scatter matrix. It fllws frm Prpsiti 1 that NLDA als prjects samples i the same class t e cmm vectr. Nte that ur result is a geeralizati f the e i [23], where all classes are assumed t have the same umber f samples. It was shw i [38] that uder a mild cditi C1 : rakðs t Þ¼rakðS b ÞþrakðS w Þ; ð16þ which hlds fr may high-dimesial data, the trasfrmati matrix f ULDA lies i the ull space f S w. Hece, uder this cditi, ULDA prjects all samples i the same class t e cmm vectr. I [38], a sufficiet cditi fr C1 has bee established as fllws:

4 1314 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER 2008 Prpsiti 2. Assume that the cditi C2, which requires that the data pits i the data matrix X 2 IR m t be liearly idepedet, hlds, the the cditi C1 i (16) hlds. 3.2 Kerel Ucrrelated Discrimiat Aalysis It fllws frm Prpsiti 2 that the cditi C1 teds t hld fr high-dimesial data. Kerel methds wrk by mappig the iput data t sme high-dimesial feature space. hus, it is iterestig t ivestigate the behavir f ULDA i the kerel-iduced feature space. I KUDA, the iput data are first mapped t a highdimesial feature space F thrugh a liear mappig. Perfrmig ULDA i F crrespds t perfrmig KUDA [32] i the rigial iput space. Fr simplicity, we assume F2IR M, where M ca be ifiite. Assume is a kerel fucti, ad is the crrespdig feature mappig. We use superscript t dete quatities i the feature space trasfrmed by. I KUDA, three scatter matrices are defied i the feature space F as fllws: S w ¼ 1 S t S b ¼ 1 X ðxþc i x2x i ¼ 1 x2x i ðxþc i ; ð17þ i c i c c i c ; ð18þ X ðxþc ðxþc ; ð19þ S b i ¼ i S t i; ð25þ crrespdig t the k 1 largest eigevalues. Direct cmputati f the geeralized eigevalue prblem i (25) ca be avided via the kerel trick. First, we have the fllwig lemma. Lemma 1. he discrimiat vectrs that slve the geeralized eigevalue prblem i (25) lie i the spa f the images f traiig samples i the feature space, i.e., G¼ðXÞB; fr sme matrix B 2 IR ðk1þ. Prf. Ay vectr 2F ca be represeted as ¼ s þ? ; ð26þ ð27þ where s 2 spafðxþg ad? 2 spafðxþg?, where? detes the rthgal cmplemet. Fr ay? 2 spafðxþg?, we have S w? ¼ 0 ad S b? ¼ 0. herefre, fr ay vectr that satisfies (25), we have S b s ¼ S b ¼ S t ¼ S t s: ð28þ his shws that we ca restrict the sluti t (25) i spafðxþg. his cmpletes the prf f the lemma. tu We ext shw hw t slve the geeralized eigevalue prblem i (25), which is summarized belw. Lemma 2. Assume that the data i the feature space are cetered. Let G be expressed as i (26) ad K be the kerel matrix, i.e., K ij ¼ ðx i ;x j Þ: ð29þ he, the sluti t the geeralized eigevalue prblem i (25) ca be btaied by slvig the fllwig prblem: where c i is the cetrid f the ith class, ad c is the glbal cetrid i the feature space. As i (7), they ca als be expressed as Sw ¼ H w H w ;S b ¼ Hb Hb ;S t ¼ Ht Ht ; ð20þ where where S K b Sb k i ¼ i St k i; ¼ KHH K ad H 2 IR k is defied as 8 qffiffiffi qffiffiffi >< j j ; if y i ¼ j; H ij ¼ qffiffiffi >: j ; therwise; ð30þ ð31þ Hw ¼ 1 pffiffiffi ðx 1 Þc 1 e 1 ;...; ðx kþc k e k ; ð21þ H b ¼ 1 pffiffiffiffiffi pffiffiffi 1 c 1 c pffiffiffiffiffi ;...; k c k c ; ð22þ Ht ¼ 1 pffiffiffi ðx 1 Þc e 1 ;...; ðx kþc e k ; ð23þ e i 2 IR i is a vectr f all es, ad ðx i Þ is the data matrix f ith class i the feature space. It fllws frm these defiitis that the trasfrmati G f KUDA ca be cmputed by slvig the fllwig ptimizati prblem: G¼argmax trace G St G þg Sb G : ð24þ G Similar t ULDA, the clums f G are geeralized eigevectrs f S K t ¼ K 2. he geeralized eigevectrs i defied i (25) ca be btaied as i ¼ ðxþ i. Prf. It fllws frm the defiitis i (18) ad (19) that Sb ¼ ðxþhh ðxþ. Whe the data are cetered, St ca be expressed as St ¼ 1 ðxþðxþ. It fllws frm Lemma 1 that G¼ðXÞB fr sme matrix B ¼½ 1 ; ; k1 Š2IR ðk1þ. Substitutig these it the bjective fucti i (24), we get G St G þg Sb G¼ 1 þ B K 2 B B ðkhh KÞB ¼ 1 þ B St k B B Sb k B: It fllws that the matrix B ca be btaied by slvig the fllwig ptimizati prblem: B ¼ argmax trace B St k B þb Sb k B ; ð32þ B

5 JI AND YE: KERNEL UNCORRELAED AND REGULARIZED DISCRIMINAN ANALYSIS: A HEOREICAL AND COMPUAIONAL SUDY 1315 ad i, fr i ¼ 1; ; ðk 1Þ ca be btaied by slvig the geeralized eigevalue prblem i (30). his cmpletes the prf f the lemma. tu I summary, KUDA cmputes the cefficiet matrix: B ¼ ½ 1 ;...; k1 Š 2 IR ðk1þ ; ð33þ by the geeralized eigevalue prblem: S k b i ¼ i S k t i: ð34þ Fr ay test data pit x 2 IR m i the rigial iput space, its prjecti is give by G ðxþ ¼B ½ðx 1 ;xþ; ;ðx ;xþš : ð35þ It fllws frm Prpsiti 2 that if the iput data pits are liearly idepedet, the cditi C1 i (16) always hlds. I KUDA, if the kerel matrix K is sigular, cditi C2, defied i Prpsiti 2, always hlds i the feature space. his is stated i the fllwig prpsiti. Prpsiti 3. If the kerel matrix K, defied i (29), is sigular, the KUDA prjects all data pits i the same class t e cmm pit i the dimesiality-reduced space. Prf. It fllws frm the defiitis f Sb K ad St K that they are the betwee-class ad ttal scatter matrices f K whe each clum i K is csidered as a data pit i the -dimesial space. hus, KUDA is equivalet t applyig ULDA t the kerel matrix K, where each clum f K is csidered as a -dimesial data pit. Whe K is sigular, the data pits are liearly idepedet, ad hece, cditi C1 hlds. It fllws frm Prpsiti 1 that KUDA prjects all data pits i the same class t e cmm vectr i the dimesiality-reduced space. tu he abve discussi shws that ULDA ad KUDA may suffer frm the verfittig prblem. We shw i the ext secti that regularizati ca be used t vercme this prblem. his is further cfirmed by the experimetal results i Secti 6. 4 KERNEL REGULARIZED DISCRIMINAN ANALYSIS Regularizati has bee used widely t vercme the verfittig prblem i machie learig algrithms [6]. It has als bee used t deal with the sigularity prblem i discrimiat aalysis. We shw i this secti that as the regularizati parameter i KRDA teds t zer, KRDA appraches KUDA. his prvides theretical evidece that regularizati is t ly a alterative t cpe with the sigularity prblem but als essetial t prevet verfittig i ucrrelated discrimiat aalysis. Similar t RLDA, KRDA cmputes the trasfrmati matrix G by slvig the fllwig prblem: G¼argmax trace G 1G St þ I G Sb G ; ð36þ G fr sme regularizati parameter >0. By fllwig similar derivatis i the prf f Lemma 2, the ptimizati prblem i (36) ca be refrmulated as B ¼ argmax trace B St k þ K 1B B Sb k B : ð37þ B It fllws that the clums f B are give by the tp ðk 1Þ eigevectrs f the fllwig prblem: St k þ K þs k b i ¼ i i : ð38þ Next, we establish a prperty f KRDA, that is, regularizig St K i KRDA is equivalet t regularizig its zer eigevalues. Let K ¼ UU be the Sigular Value Decmpsiti (SVD) [10] f the kerel matrix K, where U 2 IR is rthgal, ¼ diagð r ; 0Þ, r 2 IR rr is diagal, ad r ¼ rakðkþ. Assume that U is partitied as U ¼ðU 1 ;U 2 Þ, where U 1 2 IR r ad U 2 2 IR ðrþ. It is clear that þ K ¼ Uð 2 þ ÞU ¼ U 1 2 r þ r U 1 : S K t hus, we have the fllwig result. Lemma 3. Let St K, SK b, U 1, r ad r be defied as abve. he, St k þ K þs k b ¼ U 1 2 r þ 1U r 1 Sb k : ð39þ Lemma 3 implies that i KRDA, regularizig the ttal scatter matrix St K is equivalet t regularizig its zer eigevalues. his result ca be used t establish a cecti betwee KUDA ad KRDA. It is kw that KUDA is develped t extract features that are ucrrelated i the dimesiality-reduced space. I ctrast, KRDA is mtivated t avid the sigularity prblem by regularizig the ttal scatter matrix. Hwever, we have frm Lemma 3 that lim!0 Sk t þ K þs k b ¼ St k þs k b : ð40þ It fllws frm (32) ad (37) that the discrimiat vectrs f KUDA ad KRDA are the tp eigevectrs f matrices ðst KÞþ Sb K ad ðst K þ KÞ þ Sb K, respectively. hus, (40) establishes that KRDA appraches KUDA as the regularizati parameter teds t zer. his als prvides theretical evidece that regularizati ca be used t vercme the verfittig prblem i KUDA, which is further cfirmed by the experimetal results i Secti 6. 5 EFFICIEN MODEL SELECION FOR KRDA he perfrmace f KRDA depeds the value f the regularizati parameter, as a large value may sigificatly disturb the ifrmati i the scatter matrix, while a small value may t be effective eugh t slve the sigularity ad verfittig prblems. Crss validati is cmmly used t estimate the ptimal value frm a fiite set f cadidates. raditially, the sluti t the KRDA algrithm is btaied by cmputig the eigedecmpsiti f

6 1316 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER 2008 the matrix ðst K þ KÞ þ Sb K defied i (38). he time cmplexity f the resultig algrithm is Oð 3 Þ, which ca be expesive fr large data sets. Whe v-fld crss validati is perfrmed fr chsig the best, the abve algrithm eeds t be repeated v times. his prcedure is cmputatially extesive, ad hece, it is limited t applicatis with a small sample size. Als, the size f the cadidate set fr is cmmly limited t a small umber, sice the whle prcedure eeds t be repeated fr each cadidate value f. O the ther had, a large set f cadidate values fr is usually ecessary i practice t achieve satisfactry perfrmace. Based the theretical aalysis f KRDA i Secti 4, we shw i this secti that the matrix cmputatis ivlved i KRDA ca be simplified. his leads t a efficiet mdel selecti algrithm fr KRDA that ca chse a ptimal value f frm a large set f cadidates efficietly. he prpsed algrithm is based the fllwig prpsiti. Prpsiti 4. Let y be a eigevectr f ðst K þ KÞ þ Sb K crrespdig t a zer eigevalue, the y ¼ U 1 x, where x is a eigevectr f ð 2 r þ rþ 1 U1 SK b U 1. Prf. Let y be a eigevectr f ðst K þ KÞ þ Sb K crrespdig t a zer eigevalue. It fllws frm Lemma 3 that y ¼ 1 U 1 2 r þ 1U r 1 Sb k y ¼ U 1x; fr sme x. Next, we shw that x is a eigevectr f ð 2 r þ rþ 1 U1 SK b U 1. Multiplyig bth sides f U 1 2 r þ 1U r 1 Sb k y ¼ y; by U1, we get 2 r þ 1U r 1 Sb k ðu 1xÞ ¼U1 ðu 1xÞ ¼x: his cmpletes the prf f the prpsiti. tu It is wrth tig that cmputig U 1 is idepedet f the regularizati parameter. hus, we cmpute the SVD f K i the first step, ad it eeds t be cmputed ly ce regardless f the size f the cadidate set fr. Dete ~ ¼ 2 r þ r. Let Sb K ¼ KHH K ¼ Hb KðHK b Þ, where Hb K ¼ KH 2 IR k, ad k is the umber f classes. he secd step eeds t cmpute the eigevectrs f matrix ~ 1 U1 SK b U 1. Sice this eeds t be cmputed fr every, it may frm the bttleeck f the mdel selecti algrithm. I the fllwig, we describe a efficiet way f cmputig the eigevectrs f ~ 1 U1 SK b U 1, which ca be expressed as ~ 1=2 ~ 1=2 U1 ~ Hk b 1=2 U1 ~ Hk b 1=2 : Let U b b Vb be the SVD f ~ 1=2 U1 HK b, the, we have ~ 1 U1 Sk b U 1 ¼ ~ 1=2 U b 2 b U ~ b 1=2 ¼ð~ 1=2 U b Þ 2 b ð ~ 1=2 U b Þ 1 : hat is, ~ 1=2 U b diagalizes the matrix ~ 1 U1 SK b U 1. hus, the clums f ~ 1=2 U b frm the eigevectrs f ~ 1 U1 SK b U 1. he abve cmputati is mre efficiet tha applyig a eigedecmpsiti t ~ 1 U1 SK b U 1 directly, sice the size f matrix ~ 1=2 U1 HK b is much smaller. he abve discussi leads t a tw-step prcedure fr cmputig the eigevectrs f ðst K þ KÞ þ Sb K as fllws:. Cmpute the skiy SVD f K as K ¼ U 1 r U 1 :. Cmpute the SVD f ~ 1=2 U 1 HK b as ~ 1=2 U1 Hk b ¼ U b b Vb ; ad the eigevectrs are give by the clums f matrix U 1 ~ 1=2 U b. Let ¼f 1 ; ; jj g be the cadidate set fr the regularizati parameter. Iv-fld crss validati, the data is divided it v subsets f apprximately equal size, ad i the ith fld, the ith subset is held ut fr testig ad all ther subsets are used i traiig. Fr each j, j ¼ 1; ; jj, we cmpute the crss validati accuracy, AccuðjÞ, defied as the mea f the accuracies fr all flds. he best regularizati value j is the e with j ¼ argmax AccuðjÞ: j he pseudcde fr KRDA mdel selecti algrithm is give i Algrithm 1. Nte that the classifier i lie 12 ca be ay geeral classificati algrithm. I ur experimets, we used the 1-Nearest Neighbr (1NN) classifier. Algrithm 1. KRDA mdel selecti algrithm. 1. Fr i ¼ 1:v // v-fld crss validati 2. Cstruct A i ad A^i ; // A i : traiig set // A^i : validati set 3. Cstruct K i ad Hb K i usig A i ; Cstruct K^i usig A i ad A^i 4. Cmpute skiy SVD f K i : K i ¼ U 1 r U1 ; 5. H b;l U1 HK i b, r ¼ rakðk i Þ; 6. KL i U 1 Ki ; K^i L U1 K^i ; 7. Fr j ¼ 1:jj // jj chices fr 8. ~ ð 2 r þ j r Þ 1=2 ; 9. Cmpute SVD f ~K b;l ¼ U b b Vb ; 10. G ~U b ; 11. KL i G KL i ; K^i L G K^i L ; 12. Ru 1NN KL i ;K^i L ad cmpute the accuracy, deted as Accuði; jþ; 13. EdFr 14. EdFr P AccuðjÞ v v Accuði; jþ; 16. j arg max j AccuðjÞ; 17. Output j as the best parameter. I Algrithm 1, lie 4 takes Oð 3 Þ time fr the SVD cmputati. Lies 5 ad 6 take OðrkÞ ad Oðr 2 Þ time, respectively, fr the matrix multiplicatis. Fr each chice j, lies 9 ad 10 take Oðrk 2 Þ time fr the

7 JI AND YE: KERNEL UNCORRELAED AND REGULARIZED DISCRIMINAN ANALYSIS: A HEOREICAL AND COMPUAIONAL SUDY 1317 ABLE 1 Statistics f the Data Sets Used i the Experimets eigedecmpsiti ad matrix multiplicati. Lie 11 takes OðkrÞ time fr the matrix multiplicati. he cmputati f the classificati accuracy by 1NN i lie 12 takes Oð 2 kþ time. hus, the ttal time cmplexity, ðjjþ, fr estimatig the best parameter frm the cadidate set is ðkk Þ ¼ Ov 3 þ rk þ r 2 þkkðrk 2 þ kr þ 2 kþ ¼ Ovð 3 þkk 2 kþ ¼ Ov 2 ð þkkkþ : We ca cmpare ðjjþ with ð1þ, where jj ¼1, ad btai ðkkþ v2 ð þkkkþ ð1þ v 2 ð þ kþ 1 þ kkk : ð41þ Fr mst applicatis, the umber f classes k is much smaller tha the sample size, i.e., k. hus, the verhead f estimatig the ptimal regularizati parameter frm a large cadidate set is small. 6 EXPERIMENAL EVALUAION I this secti, we cduct experimets t evaluate the prpsed theries ad algrithms. We first cmpare the relative perfrmace f KUDA, KRDA, ad SVM i terms f classificati accuracy. he cditi C1, defied i (16), is als examied fr each data set. I rder t visualize the effect f regularizati i KRDA, we prject e data set with three classes t tw-dimesial (2D) plaes with differet regularizati parameters. he verfittig prblem f KUDA ad the regularizati effect f KRDA ca be elucidated by this experimet. Fially, the efficiecy f the prpsed mdel selecti algrithm is evaluated. 6.1 Experimetal Setup We use a ttal f 12 data sets i the experimets. Amg them, the sar, cacer, isphere, wavefrm, wie, iris, glass, letter, ad vwel are frm the UCI Machie Learig Repsitry [24]. he vehicle is frm the StatLg prject. 1 he svmguide2 is frm a biifrmatics applicati i [9], ad it has bee used i [14]. he USPS data set was rigially described i [15]. Fr the letter ad USPS data sets, we radmly sample 100 data pits frm each f the 10 classes. All attribute values i the data sets are liearly scaled t the iterval [1, 1]. he statistics f all data sets are summarized i able 1. Bth the KUDA ad KRDA algrithms are implemeted i Matlab. 2 he 1NN classifier is used t btai the classificati accuracy i the dimesiality-reduced space. he LIBSVM tlbx 3 is used as the SVM classifier. 6.2 Classificati Perfrmace We first cmpare the classificati perfrmace f KUDA, KRDA, ad SVM. he Gaussia kerel is used i the experimets, ad the parameter is tued usig crss validati. Fr KRDA ad SVM, the regularizati parameter ad C als eed t be specified. We use duble crss validati t chse the kerel ad the regularizati parameter jitly i the experimets. All f the data sets are radmly partitied it traiig ad test sets with a rati 1:1, ad this prcess is repeated 10 times fr each data set. he mea accuracies ad the stadard deviatis ver 10 radm partitis are reprted i Fig. 1 ad able 2. test the statistical differece f the three algrithms, we perfrm paired, tw-sided siged rak tests fr three pairs f algrithms, ad the P values are reprted i able 3. he ull hypthesis is that the cmpared algrithms are t statistically differet, ad a P value less tha 0.05 is usually used t reject the ull hypthesis. We ca bserve that KRDA ad SVM utperfrm KUDA seve data sets (cacer, isphere, wie, iris, svmguide2, vehicle, ad glass). his implies that the regularizati emplyed i KRDA is effective t vercme the verfittig prblem i KUDA. check the C1 cditi, we als recrd the raks f the three scatter matrices i the feature space. Results shw that the cditi C1 is satisfied fr all partitis i ie ut f the 12 data sets. Fr the cacer data, six f the partitis vilate the C1 cditi. Fr this data, the sample size is relatively large (683), ad the dimesiality is lw (10). his implies that fr large sample size ad lw-dimesial data, the C1 cditi may t be satisfied. he C1 cditi is als t satisfied i the ith partiti f the glass data ad the secd partiti f the letter data. Nte that fr the glass ad letter data sets, the umber f classes k is 6 ad 10, respectively. hus, the raks f betwee-class scatter matrices cat be larger tha k 1, i.e., 5 ad 9, respectively. Hwever, Matlab reprts raks f 6 ad 10. Hece, we expect that umerical prblems cause this iaccuracies i cmputig the matrix rak. 6.3 Effect f Regularizati We ivestigate the verfittig prblem f KUDA ad shw hw regularizati emplyed i KRDA ca be used t vercme this prblem. he svmguide2 data set has three classes, ad we radmly partiti it it traiig ad test sets f equal size. he traiig set is the used t lear a trasfrmati usig KRDA with differet

8 1318 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER 2008 Fig. 1. Cmparis f mea accuracies ver 10 radm partitis btaied by KUDA, KRDA, ad SVM twelve data sets. All the parameters are tued by crss validati. ABLE 2 Mea Accuracies ad Stadard Deviatis ver 10 Radm Partitis f the Data Achieved by KUDA, KRDA, ad SVM SA detes test set accuracy ad SD detes stadard deviati. ABLE 3 P Values Obtaied frm Wilcx Siged est Cmparig the Relative Perfrmace f KUDA, KRDA, ad SVM A p-value less tha 0.05 idicates that the perfrmace differece f the cmpared algrithms is statistically sigificat. values f the regularizati parameter, ad bth sets are prjected t 2D plaes fr visualizati. Fig. 2 presets the plts fr the traiig ad test pits with fur differet values f (0, 0.01, 0.03, 0.2). Whe ¼ 0, KRDA is equivalet t KUDA. I this case, the traiig samples frm the same class are prjected t a cmm pit the 2D plae. I ctrast, the test pits frm differet classes verlap, ad the test accuracy is ly percet. Whe icreases t 0.01, the traiig pits spread ut ad becme verlapped, while the test pits frm differet classes becme better separated. A test accuracy f percet is achieved with this value f. We ra the prpsed KRDA mdel selecti algrithm ad the ptimal value btaied is With this value, the traiig pits still shw sme verlappig while the test pits are well separated fr differet classes. he classificati accuracy achieved fr this ptimal is percet. Whe we further icrease beyd the ptimal value t 0.2, bth the traiig ad test pits shw icreased verlappig ad the test accuracy decreases t percet. his experimet clearly shws the verfittig prblem f KUDA. Meawhile, the regularizati effect f KRDA t alleviate this prblem is demstrated. We ca als bserve frm the experimet that the perfrmace f KRDA is critically depedet the value f the regularizati parameter. We shw i Secti 6.2 that the prpsed KRDA mdel selecti algrithm ca tue this parameter s that the achieved perfrmace is cmpetitive with that f SVM. 6.4 Mdel Selecti fr KRDA We cduct experimets t evaluate the scalability f the prpsed mdel selecti algrithm. I particular, we start by settig jj ¼1 ad duble its size each time util it reaches 1,024, which is sufficietly large fr mst applicatis. he rutime fr each cadidate set size is reprted i able 4 fr all the 12 data sets. It ca be see i able 4 that whe jj is relatively small, the rutime icreases very slwly as jj dubles. his is because at this rage f jj, cmputati i the first stage (lies 1 6 i Algrithm 1) f ur prpsed tw-step mdel selecti algrithm

9 JI AND YE: KERNEL UNCORRELAED AND REGULARIZED DISCRIMINAN ANALYSIS: A HEOREICAL AND COMPUAIONAL SUDY 1319 Fig. 2. (a) Visualizati f the traiig ad (b) test pits after prjecti t the 2D plae via KRDA with differet values f (0, 0.01, 0.03, 0.2) fr the svmguide2 data set. he test accuracy fr each value f is als reprted. dmiates. Sice the secd step eeds t be cmputed jj times, cmputati spet at this step will icrease as jj dubles. able 4 als reprts the rati ð1;024þ= ð1þ fr ease f cmparis. Whe jj is icreased frm 1 t 1,024, the largest icrease i rutime is fr the glass data set. Fr mst ther data sets, the icrease is less tha 10. his shws that the cmputati time icreases slwly as the size f cadidate set icreases. 7 CONCLUSION AND FUURE WORK We preset a theretical ad cmputatial study f kerel ucrrelated ad regularized discrimiat aalysis i this paper. We shw that uder a mild cditi, the ucrrelated discrimiat aalysis prjects all samples i the same class t a cmm vectr i the dimesiality-reduced space. hus, it may suffer frm the verfittig prblem. We further establish a equivalece relatiship betwee

10 1320 IEEE RANSACIONS ON KNOWLEDGE AND DAA ENGINEERING, VOL. 20, NO. 10, OCOBER 2008 ABLE 4 Ruig ime (i Secds) f KRDA Mdel Selecti Algrithm as the Size f Cadidate Set Dubles he last clum presets the rati betwee the rutime whe jj is 1,024 ad 1, respectively. KUDA ad KRDA. hat is, KRDA appraches KUDA as the value f the regularizati parameter teds t zer. his prvides theretical evidece that regularizati ca be used t vercme the verfittig prblem i ucrrelated discrimiat aalysis. Based the theretical study, we shw that the matrix cmputatis ivlved i KRDA ca be simplified, resultig i a efficiet mdel selecti algrithm fr KRDA t tue the regularizati parameter. Extesive experimets have bee cducted t evaluate the prpsed theries ad algrithms. Evaluati f the prpsed theries ad algrithms i this paper maily fcuses stadard bechmark data sets. We pla t apply the prpsed algrithm t sme challegig real-wrld applicatis, such as the stage rage classificati f gee expressi patter images [35]. ACKNOWLEDGMENS his research is spsred i part by the Ariza State Uiversity ad by the US Natial Sciece Fudati Grat IIS REFERENCES [1] G. Baudat ad F. Auar, Geeralized Discrimiat Aalysis Usig a Kerel Apprach, Neural Cmputati, vl. 12,. 10, pp , [2] P.N. Belhumeur, J.P. Hespaha, ad D.J. Kriegma, Eigefaces versus Fisherfaces: Recgiti Usig Class Specific Liear Prjecti, IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 19,. 7, pp , July [3] R.E. Bellma, Adaptive Ctrl Prcesses: A Guided ur. Pricet Uiv. Press, [4] L.F. Che, H.Y.M. Lia, J.C. Li, M.D. Ka, ad G.J. Yu, A New LDA-Based Face Recgiti System Which Ca Slve the Small Sample Size Prblem, Patter Recgiti, vl. 33,. 10, [5] S. Dudit, J. Fridlyad, ad.p. Speed, Cmparis f Discrimiati Methds fr the Classificati f umrs Usig Gee Expressi Data, J. Am. Statistical Assc., vl. 97,. 457, pp , [6]. Evgeiu, M. Ptil, ad. Pggi, Regularizati Netwrks ad Supprt Vectr Machies, Advaces i Cmputatial Math., vl. 13,. 1, pp. 1-50, [7] J.H. Friedma, Regularized Discrimiat Aalysis, J. Am. Statistical Assc., vl. 84,. 405, pp , [8] K. Fukuaga, Itrducti t Statistical Patter Recgiti, secd ed. Academic Press Prfessial, [9] J.L. Gardy, C. Specer, K. Wag, M. Ester, G.E. usady, I. Sim, S. Hua, K. defays, C. Lambert, K. Nakai, ad F.S.L. Brikma, PSOR-B: Imprvig Prtei Subcellular Lcalizati Predicti fr Gram-Negative Bacteria, Nucleic Acids Research, vl. 31,. 13, pp , [10] G.H. Glub ad C.F. Va La, Matrix Cmputatis, third ed. Jhs Hpkis Uiv. Press, [11] Y. Gu,. Hastie, ad R. ibshirai, Regularized Liear Discrimiat Aalysis ad Its Applicati i Micrarrays, Bistatistics, vl. 8,. 1, pp , [12]. Hastie, R. ibshirai, ad J.H. Friedma, he Elemets f Statistical Learig : Data Miig, Iferece, ad Predicti. Spriger, [13] P. Hwlad, M. Je, ad H. Park, Structure Preservig Dimesi Reducti fr Clustered ext Data Based the Geeralized Sigular Value Decmpsiti, SIAM J. Matrix Aalysis ad Applicatis, vl. 25,. 1, pp , [14] C. Hsu, C. Chag, ad C. Li, A Practical Guide t Supprt Vectr Classificati, techical reprt, Dept. f Cmputer Sciece, Nat l aiwa Uiv., [15] J.J. Hull, A Database fr Hadwritte ext Recgiti Research, IEEE ras. Patter Aalysis Machie Itelligece, vl. 16,. 5, pp , May [16] Z. Ji, J. Yag, Z. Hu, ad Z. Lu, Face Recgiti Based the Ucrrelated Discrimiat rasfrmati, Patter Recgiti, vl. 34, pp , [17] I.. Jlliffe, Pricipal Cmpet Aalysis. Spriger, [18] A. Krai ad J.M. Richards, Liear Discrimiat ext Classificati i High Dimesi, Hybrid Ifrmati Systems, A. Abraham ad M. Keppe, eds., pp , Physica Verlag, [19] J. Lu, K.N. Plataitis, ad A.N. Veetsapuls, Face Recgiti Usig Kerel Direct Discrimiat Aalysis Algrithms, IEEE ras. Neural Netwrks, vl. 14,. 1, pp , [20] A.M. Martiez ad A.C. Kak, PCA versus LDA, IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 23,. 2, pp , Feb [21] A.M. Martiez ad M. Zhu, Where Are Liear Feature Extracti Methds Applicable? IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 27,. 12, pp , Dec [22] S. Mika, G. Rätsch, J. West, B. Schölkpf, ad K.-R. Müller, Fisher Discrimiat Aalysis with Kerels, Neural Netwrks fr Sigal Prcessig IX, Y.H. Hu, J. Larse, E. Wils, ad S. Duglas, eds., pp , IEEE, 1999.

11 JI AND YE: KERNEL UNCORRELAED AND REGULARIZED DISCRIMINAN ANALYSIS: A HEOREICAL AND COMPUAIONAL SUDY 1321 [23] M. Neamtu, H. Cevikalp, M. Wilkes, ad A. Barkaa, Discrimiative Cmm Vectrs fr Face Recgiti, IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 27,. 1, pp. 4-13, Ja [24] D.J. Newma, S. Hettich, C.L. Blake, ad C.J. Merz, UCI Repsitry f Machie Learig Databases, ics.uci.edu/ml/, [25] S.L. Pmery, P. amay, M. Gaasebeek, L.M. Sturla, M. Agel, M.E. McLaughli, J.Y.H. Kim, L.C. Gumerva, P.M. Black, J.C. Alle, D. Zagzag, J.M. Ols,. Curra, C. Wetmre, J.A. Biegel,. Pggi, S. Mukherjee, R. Rifki, A. Califa, G. Stlvitzky, D.N. Luis, J.P. Mesirv, E.S. Lader, ad.r. Glub, Predicti f Cetral Nervus System Embryal umur Outcme Based Gee Expressi, Nature, vl. 415,. 6870, pp , [26] S. Ramaswamy, P. amay, R. Rifki, S. Mukherjee, C.-H. Yeag, M. Agel, C. Ladd, M. Reich, E. Latulippe, J.P. Mesirv,. Pggi, W. Gerald, M. Lda, E.S. Lader, ad.r. Glub, Multiclass Cacer Diagsis Usig umr Gee Expressi Sigatures, Prc. Nat l Academy f Scieces (PNAS 01), vl. 98,. 26, pp , [27] B. Schölkpf, A.J. Smla, ad K-R. Müller, Nliear Cmpet Aalysis as a Kerel Eigevalue Prblem, Neural Cmputati, vl. 10,. 5, pp , [28] S. Schölkpf ad A. Smla, Learig with Kerels: Supprt Vectr Machies, Regularizati, Optimizati ad Beyd. MI Press, [29] A.I. Su, J.B. Welsh, L.M. Sapis, S.G. Ker, P. Dimitrv, H. Lapp, P.G. Schultz, S.M. Pwell, C.A. Mskaluk, H.F. Friers Jr., ad G.M. Hampt, Mlecular Classificati f Huma Carcimas by Use f Gee Expressi Sigatures, Cacer Research, vl. 61,. 20, pp , [30] D.L. Swets ad J. Weg, Usig Discrimiat Eigefeatures fr Image Retrieval, IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 18,. 8, pp , Aug [31] M. urk ad A. Petlad, Eigefaces fr Recgiti, J. Cgitive Neursciece, vl. 3,. 1, pp , [32]. Xig, J. Ye, ad V. Cherkassky, Kerel Ucrrelated ad Orthgal Discrimiat Aalysis: A Uified Apprach, Prc. IEEE CS Cf. Cmputer Visi ad Patter Recgiti (CVPR 06), pp , [33] J. Yag, A.F. Fragi, J. Yag, D. Zhag, ad Z. Ji, KPCA Plus LDA: A Cmplete Kerel Fisher Discrimiat Framewrk fr Feature Extracti ad Recgiti, IEEE ras. Patter Aalysis ad Machie Itelligece, vl. 27,. 2, pp , Feb [34] J. Ye, Characterizati f a Family f Algrithms fr Geeralized Discrimiat Aalysis Udersampled Prblems, J. Machie Learig Research, vl. 6, pp , [35] J. Ye, J. Che, Q. Li, ad S. Kumar, Classificati f Drsphila Embryic Develpmetal Stage Rage Based Gee Expressi Patter Images, Prc. Cmputatial Systems Biifrmatics Cf. (CSB 06), pp , [36] J. Ye, R. Jaarda, Q. Li, ad H. Park, Feature Extracti via Geeralized Ucrrelated Liear Discrimiat Aalysis, Prc. 21st It l Cf. Machie Learig (ICML 04), p. 113, [37] J. Ye,. Li,. Xig, ad R. Jaarda, Usig Ucrrelated Discrimiat Aalysis fr issue Classificati with Gee Expressi Data, IEEE/ACM ras. Cmputatial Bilgy ad Biifrmatics, vl. 1,. 4, pp , Oct.-Dec [38] J. Ye ad. Xig, Cmputatial ad heretical Aalysis f Null Space ad Orthgal Liear Discrimiat Aalysis, J. Machie Learig Research, vl. 7, pp , July [39] H. Yu ad J. Yag, A Direct LDA Algrithm fr High- Dimesial Data with Applicatis t Face Recgiti, Patter Recgiti, vl. 34, pp , [40] W. Zha, R. Chellappa, ad P. Phillips, Subspace Liear Discrimiat Aalysis fr Face Recgiti, echical Reprt CAR-R-914, Ceter fr Autmati Research, Uiv. f Marylad, Cllege Park, Shuiwag Ji is a PhD studet i the Departmet Cmputer Sciece ad Egieerig, Ariza State Uiversity. He is als affiliated with the Ceter fr Evlutiary Fuctial Gemics, the Bidesig Istitute, Ariza State Uiversity. His research iterests iclude machie learig, data miig, ad biifrmatics. Jiepig Ye received the PhD degree i cmputer sciece frm the Uiversity f Miesta- wi Cities i He is curretly a assistat prfessr i the Departmet f Cmputer Sciece ad Egieerig, Ariza State Uiversity. He has bee a cre faculty member f the Ceter fr Evlutiary Fuctial Gemics, the Bidesig Istitute, Ariza State Uiversity, sice August He was awarded the Guidat Fellwship i I 2004, his paper geeralized lw rak apprximatis f matrices received the utstadig studet paper award at the 21st Iteratial Cferece Machie Learig. His research iterests iclude machie learig, data miig, ad biifrmatics. He has published extesively i these areas. He is a member f the IEEE ad the ACM.. Fr mre ifrmati this r ay ther cmputig tpic, please visit ur Digital Library at