Semi-supervised Discriminant Analysis

Transcription

1 Sem-supervsed Dscrmnnt Anlyss Deng C UIUC dengc2@cs.uuc.edu Xofe He Yhoo! hex@yhoo-nc.com Jwe Hn UIUC hnj@cs.uuc.edu Abstrct Lner Dscrmnnt Anlyss (LDA) hs been populr method for extrctng fetures whch preserve clss seprblty. The projecton vectors re commonly obtned by mxmzng the between clss covrnce nd smultneously mnmzng the wthn clss covrnce. In prctce, when there s no suffcent trnng smples, the covrnce mtrx of ech clss my not be ccurtely estmted. In ths pper, we propose novel method, clled Semsupervsed Dscrmnnt Anlyss (SDA), whch mkes use of both lbeled nd unlbeled smples. The lbeled dt ponts re used to mxmze the seprblty between dfferent clsses nd the unlbeled dt ponts re used to estmte the ntrnsc geometrc structure of the dt. Specfclly, we m to lern dscrmnnt functon whch s s smooth s possble on the dt mnfold. Expermentl results on sngle trnng mge fce recognton nd relevnce feedbck mge retrevl demonstrte the effectveness of our lgorthm. 1. Introducton In mny vsul nlyss pplctons, such s mge retrevl, fce recognton, etc., one s often confronted wth hgh-dmensonl dt. However, there mght be reson to suspect tht the nturlly generted hgh-dmensonl dt probbly resdes on lower dmensonl mnfold. Ths leds one to consder methods of dmensonlty reducton tht llow one to represent the dt n lower dmensonl spce. Two of the most populr technques for ths purpose re Prncpl Component Anlyss (PCA) [16] nd Lner Dscrmnnt Anlyss (LDA) [9]. PCA s n unsupervsed method. It performs dmensonlty reducton by projectng the orgnl n-dmensonl The work ws supported n prt by the U.S. Ntonl Scence Foundton NSF IIS , NSF BDI nd MIAS ( DHS Insttute of Dscrete Scence Center for Multmodl Informton Access nd Synthess). Any opnons, fndngs, nd conclusons or recommendtons expressed here re those of the uthors nd do not necessrly reflect the vews of the fundng gences. dt onto the d( n)-dmensonl lner subspce spnned by the ledng egenvectors of the dt s covrnce mtrx. Its gol s to fnd set of mutully orthogonl bss functons tht cpture the drectons of mxmum vrnce n the dt so tht the prwse Euclden dstnces cn be best preserved. If the dt s embedded n lner subspce, PCA s gurnteed to dscover the dmensonlty of the subspce nd produces compct representton. LDA s supervsed method. It serches for the project xes on whch the dt ponts of dfferent clsses re fr from ech other whle requrng dt ponts of the sme clss to be close to ech other. When lbel nformton vlble, e.g., for clssfcton tsk, LDA cn cheve sgnfcnt better performnce thn PCA [1]. However, when there s no suffcent trnng smples reltve to the number of dmensons, the covrnce mtrx of ech clss my not be ccurtely estmted. In ths cse, the generlzton cpblty on testng smples cn not be gurnteed. A possble soluton to del wth nsuffcent trnng (lbeled) smples could be lernng on both lbeled nd unlbeled dt (sem-supervsed nd trnsductve lernng). It s nturl nd resonble snce n relty we usully hve only prt of nput dt lbeled, long wth lrge number of unlbeled dt. In the lst decdes, sem-supervsed lernng (or trnsductve lernng) hs ttrcted n ncresng mount of ttenton. Two welnown lgorthms re Trnsductve SVM (TSVM) [23] nd Co-Trnng. Recently, there re consderble nterest nd succuss on grph bsed sem-supervsed lernng lgorthms [3, 20, 26, 27], whch consder the grph over ll the smples s pror to gude the decson mkng. All these lgorthms consdered the problem of clssfcton, ether trnsductve or nductve. In ths pper, we m t dmensonlty reducton n the sem-supervsed cse. We proposed sem-supervsed dmensonlty reducton lgorthm, clled Sem-supervsed Dscrmnnt Anlyss (SDA). SDA ms to fnd projecton whch respects the dscrmnnt structure nferred from the lbeled dt ponts, s well s the ntrnsc geometrcl structure nferred from both lbeled nd unlbeled dt ponts. Specfclly, the lbeled dt ponts, combned wth

2 the unlbeled dt ponts, re used to buld grph ncorportng neghborhood nformton of the dt set. The grph provdes dscrete pproxmton to the locl geometry of the dt mnfold. Usng the noton of grph Lplcn, smoothness penlty on the grph cn be ncorported nto the objectve functon. In ths wy, our SDA lgorthm cn optmlly preserves the mnfold structure. The rest of ths pper s orgnzed s follows. In Secton 2, we provde bref revew of LDA. We ntroduce our Sem-supervsed Dscrmnnt Anlyss (SDA) lgorthm for dmensonlty reducton n Secton 3. The expermentl results re presented n Secton 4. Fnlly, we conclude the pper nd provde suggestons for future work n Secton Grph Perspectve of LDA Lner Dscrmnnt Anlyss (LDA) seeks drectons on whch the dt ponts of dfferent clsses re fr from ech other whle requrng dt ponts of the sme clss to be close to ech other [9]. Suppose we hve set of l smples x 1, x 2,, x l R n, belongng to c clsses. The objectve functon of LDA s s follows: S w = S b = opt = rg mx T S b T S w, (1) (µ (k) µ)(µ (k) µ) T, (2) ( lk =1 (x (k) µ (k) )(x (k) µ (k) ) T ), (3) where µ s the totl smple men vector, s the number of smples n the k-th clss, µ (k) s the verge vector of the k- th clss, nd x (k) s the -thsmplenthek-th clss. We cll S w the wthn-clss sctter mtrx nd S b the between-clss sctter mtrx. Defne the totl sctter mtrx S t = l =1 (x µ)(x µ) T,wehveS t = S b + S w [9]. The objectve functon of LDA n Eqn. (1) s equvlent to opt = rg mx T S b T S t. (4) The optml s re the egenvectors correspondng to the non-zero egenvlue of egen-problem: S b = λs t. (5) Snce the rnk of S b s bounded by c 1, there re t most c 1 egenvectors correspondng to non-zero egenvlues [9]. Wthout loss of generlty, we ssume µ = 0 1.Wehve S b = = = (µ (k) )(µ (k) ) T ( 1 =1 x (k) )( 1 X (k) W (k) (X (k) ) T =1 x (k) ) T where W (k) s mtrx wth ll the elements equl to 1/ nd X (k) =[x (k) 1,, x(k) ] denote the dt mtrx of k-th clss. Let the dt mtrx X =[X (1),,X (c) ] nd defne l l mtrx W l l s: We hve W (1) W (2) 0 W l l = W (c) S b = (6) X (k) W (k) (X (k) ) T = XW l l X T. (7) Thus, the objectve functon of LDA n Eqn. (4) cn be rewrtten s opt = rg mx T S b T = rg mx S t T XW l l X T T XX T. (8) Ths formulton of LDA objectve functon wll be very helpful n developng our lgorthm. It s frst ntroduced n [14]. 3. Sem-supervsed Dscrmnnt Anlyss LDA consders seekng the optml projectons purely on the trnng (lbeled) set. In relty, t s possble to cqure lrge set of unlbeled dt. In ths secton, we re tryng to extend LDA model to ncorporte the mnfold structure llustrted by unlbeled dt The Objectve Functon LDA ms to fnd projecton vector such tht the rto between T S b nd T S t s mxmzed. When there s no suffcent trnng smple, overfttng my hppen. A typcl wy to prevent overfttng s to mpose regulrzer 1 Ths cn be cheved by centerng the dt,.e., subtrct the men vector from ll the smple vectors.

3 [11]. The optmzton problem of the regulrzed verson of LDA cn be wrtten s follows: mx T S b T S t + αj() where J() controls the lernng complexty of the hypothess fmly, nd the coeffcent α controls blnce between the model complexty nd the emprcl loss. One of the most populr regulrzers s the Tkhonov regulrzer [21]: J() = 2. LDA model wth Tkhonov regulrzer s usully referred s Regulrzed Dscrmnnt Anlyss (RDA) [8]. The regulrzer term J() provdes us the flexblty to ncorporte our pror knowledge on some prtculr pplctons. When set of unlbeled exmples vlble, we m to construct J() ncorportng the mnfold structure. The key to sem-supervsed lernng lgorthm s the pror ssumpton of consstency. For clssfcton, t mens nerby ponts re lkely to hve the sme lbel [26]. For dmensonlty reducton, t cn be nterpreted s nerby ponts wll hve smlr embeddngs (low-dmensonl representtons). Gven set of exmples {x } m =1, we cn use p-nerest neghbor grph G to model the reltonshp between nerby dt ponts. Specfclly, we put n edge between nodes nd j f x nd x j re close,.e., x nd x j re mong p nerest neghbors of ech other. Let the correspondng weght mtrx be S, defned by { 1, f x N S j = p (x j ) or x j N p (x ) (10) 0, otherwse. where N p (x ) denotes the set of p nerest neghbors of x. In generl, the mppng functon should be s smooth s possble on the grph. Specfclly, f two dt ponts re lnked by n edge, they re lkely to be n the sme clss. Moreover, the dt ponts lyng on densely lnked subgrph re lkely to hve the sme lbel. Thus, nturl regulrzer cn be defned s follows: J() = ( T x T ) 2 x j Sj (11) j Ths formulton s motvted from spectrl dmensonlty reducton [2, 13], whch lso plys key role n spectrl clusterng [17] nd vrous knds of grph bsed semsupervsed lernng lgorthms [3, 6, 20]. Let X =[x 1, x 2,, x m ].Wehve J() = ( T x T x j ) 2 S j j = 2 T x D x T 2 j = 2 T X(D S)X T = 2 T XLX T T x S j x T j (9) where D s dgonl mtrx; ts entres re column (or row, snce S s symmetrc) sum of S, D =Σ j S j. L = D S s the Lplcn mtrx [7]. Wth ths dt dependent regulrzer, we get the objectve functon of our sem-supervsed dscrmnnt nlyss: mx T S b T ( S t + αxlx T ). (12) The projectve vector tht mxmzes the objectve functon s gven by the mxmum egenvlue soluton to the generlzed egenvlue problem: 3.2. The Algorthm S b = λ(s t + αxlx T ) (13) Gven lbeled set {(x,y )} l =1 belongng to c clsses nd n unlbeled set {x } m =l+1.thek-th clss hve smples, c = l. Wthout loss of generlty, we ssume tht the dt ponts n {x 1,, x l } re ordered ccordng to ther lbels. The lgorthmc procedure of sem-supervsed dscrmnnt nlyss s stted below: 1. Construct the djcency grph: Construct the p- nerest neghbors grph mtrx S s n Eqn. (10) nd clculte the grph Lplcn L = D S. 2. Construct the lbeled grph: Construct the weght mtrx W R m m for lbeled grph s [ ] Wl l 0 W = 0 0 where W l l R l l s defned n Eqn. (6). Defne [ ] I 0 Ĩ = 0 0 where I s n dentty mtrx of sze l l. 3. Egen-problem: Compute the egenvectors wth respect to the non-zero egenvlues for the generlzed egenvector problem: XWX T = λx(ĩ + αl)xt, (14) where X =[x 1,, x l, x l+1,, x m ]. It s esy to check tht W s of rnk c nd we wll hve c egenvectors wth respect to non-zero egenvlues 2 [10]. We denote them s 1,, c. 4. SDA Embeddng: Let A =[ 1, 2,, c ], A s n c trnsformton mtrx. The smples cn be embedded nto c dmensonl subspce by x z = A T x 2 We consder the cse tht the number of fetures n>c.

4 Let X l =[x 1,, x l ] be the lbeled dt mtrx. It s esy to check tht XWX T = X l W l l X T l = S b nd XĨXT = X l Xl T = S t. Thus, the egen-problem n Eqn. (14) s sme s the egenproblem n Eqn. (13). To get stble soluton of the egen-problem n Eqn. (14), the mtrx X(Ĩ + αl)xt s requred to be nonsngulr [10] whch s not true when the number of fetures s lrger thn the number of smples. In ths cse, we cn pply the Tkhonov regulrzton de s the wy n regulrzed dscrmnnt nlyss [8]. Thus, our generlzed egen-problem becomes: ( ) XWX T = λ X(Ĩ + αl)xt + βi (15) For β>0, the mtrx X(Ĩ +αl)xt +βi s certnly nonsngulr. We cn lso use the spectrl regresson technque to solve ths sngulrty problem, plese see [5] for detls Kernel SDA The lgorthm descrbed bove s lner method. It my fl to dscover the ntrnsc geometry when the dt mnfold s hghly nonlner. In ths subsecton, we dscuss how to perform SDA n Reproducng Kernel Hlbert Spce (RKHS), whch gves rse to kernel SDA. The pproch used here s essentlly smlr to [13]. We consder the problem n feture spce F nduced by some nonlner mppng φ : R n F For proper chosen φ, n nner product, cn be defned on F whch mkes for so-clled reproducng kernel Hlbert spce (RKHS). More specfclly, φ(x),φ(y) = K(x, y) holds where K(.,.) s postve sem-defnte kernel functon. Severl populr kernel functons re: Gussn kernel K(x, y) = exp( x y 2 /σ 2 ); polynomernel K(x, y) = (1 + x, y ) d ; Sgmod kernel K(x, y) = tnh( x, y + α). Gven set of vectors {v F =1, 2,,d} whch re orthonorml ( v, v j = δ,j ), the projecton of φ(x ) F to these v 1,, v d leds to mppng from R n to Euclden spce R d through y = ( v 1,φ(x ), v 2,φ(x ),, v d,φ(x ) ) T We look for such {v F = 1, 2,,d} tht helps {y =1,,m} preserve locl geometrcl nd dscrmnnt structure of the dt mnfold. Let Φ denote the dt mtrx n RKHS: Φ=[φ(x 1 ),φ(x 2 ),,φ(x m )] Now, the egenvector problem of Eqn. (14) n RKHS cn be wrtten s follows: ΦW Φ T v = λφ ( Ĩ + αl ) Φ T v (16) Becuse the egenvector of (16) re lner combntons of φ(x 1 ),φ(x 2 ),,φ(x m ), there exst coeffcents α, = 1, 2,, msuch tht v = m α φ(x )=Φα =1 where α =(α 1,α 2,,α m ) T R m. Followng some lgebrc formultons, we get: ΦW Φ T v = λφ ( Ĩ + αl ) Φ T v ΦW Φ T Φα = λφ ( Ĩ + αl ) Φ T Φα Φ T ΦW Φ T Φα = λφ T Φ ( Ĩ + αl ) Φ T Φα KWKα = λk ( Ĩ + αl ) Kα (17) where K s the kernel mtrx, K j = K(x, x j ). Let the column vectors α 1,α 2,,α c be the egenvectors wth respect to the non-zero egenvlues of egen-problem n Eqn. (17) nd the m c trnsformton mtrx Θ=[α 1,,α c ]. A dt pont cn be embedded nto c dmensonl subspce by x z =Θ T K(:, x) (18) where K(:, x) =[K(x. 1, x),, K(x m, x)] T 4. Expermentl Results In ths secton, severl experments re performed to test our lgorthm. We choose two scenros n whch semsupervsed lernng s nturl nd necessry. They re sngle trnng mge fce recognton [4] nd relevnce feedbck mge retrevl [18]. Mny of proposed grph bsed sem-supervsed lernng lgorthms [26, 27] cn only work on trnsductve settng. Tht s, both the trnng nd test set (wthout lbel nformton) re vlble durng the lernng process. In relty (e.g., fce recognton), more nturl settng for sem-supervsed lernng s s follows. The vlble trnng set contns both lbeled nd unlbeled exmples, nd the testng set s not vlble durng the trnng phrse, whch we refer here s sem-supervsed settng. Toths end, mnfold regulrzton [3, 20] s one of the most successful pproches tht ddress both two settngs. Mnfold regulrzton extends mny of the exstng nductve lgorthms (e.g., SVM, Regresson) to sem-supervsed lernng by ddng geometrclly bsed regulrzton term.

5 Fgure 1. Smple fce mges from the CMU PIE fce dtbse. For ech subject, there re 43 fce mges under dfferent llumnton wth fxed pose nd expresson. The SDA lgorthm essentlly shres the smlr de whle focuses on dmensonlty reducton. In the SDA subspce, ny ordnry clssfer cn then be used. In our experments, we smply choose the nerest centrod method Sngle Trnng Imge Fce Recognton One of the most successful nd well-studed technques to fce recognton s the ppernce-bsed method [22]. Prevous works hve demonstrted tht the fce recognton performnce cn be mproved sgnfcntly n lower dmensonl lner subspces [1, 14, 22]. Two of the most populr ppernce-bsed methods nclude Egenfce [22] (bsed on PCA) nd Fsherfce [1] (bsed on LDA). In generl, fce ppernce does not depend solely on dentty. It s lso nfluenced by llumnton nd vewpont. Chnges n pose nd llumnton wll cuse lrge chnges n the ppernce of fce. Thus, ppernce-bsed methods need number of trnng mges for ech subject, n order to cope wth pose nd llumnton vrbltes. One of the clsscl chllenges n fce recognton s recognton from sngle trnng mge [4]. In ths settng, the ordnry ppernce-bsed methods (e.g., Egenfce nd Fsherfce) tend to fl. Actully, wth sngle trnng smple per clss, t s esy to check tht the between-clss sctter mtrx wll be sme s the totl sctter mtrx. Thus, LDA cn not be ppled. Recent studes show tht the fce mges re smpled from nonlner low-dmensonl mnfold whch s embedded n the hgh-dmensonl mbent spce [14]. If we hve lrge set of unlbeled fce mges (whch s possble due to the fst growth of dgtl photogrphy ndustry), the ntrnsc mge mnfold cn stll be estmted even wth sngle lbeled fce mge per subject. In ths experment, we test our SDA lgorthm n ths sngle trnng mge fce recognton settng. The CMU PIE fce dtbse [19] s used n ths experment. It contns 68 subjects wth 41,368 fce mges s whole. The fce mges were cptured under vryng pose, llumnton nd expresson. In ths experment, we choose the frontl pose (C27) wth vryng lghtng nd llumnton whch leves us 43 mges per subject. The sze of ech cropped fce mge s pxels, wth 256 grey levels per pxel. Fgure 1 shows some smple mges for certn subject. For ech subject, 30 mges re rndomly selected s the trnng set. Among these 30 mges, 1 m- Tble 1. Recognton error rtes on PIE (men±std-dev%) Unlbeled error Test error Bselne 74.7± ±1.6 Egenfce (PCA) [22] 74.7± ±1.6 Lplcnfce (LPP) [14] 43.9± ±2.4 Consstency [26] 48.0±1.8 LpSVM [3] 43.5± ±2.6 LpRLS [3] 42.5± ±2.6 SDA 41.0± ±2.7 ge s rndomly selected nd lbeled whch leves other 29 mges unlbeled. We verge the results over 20 rndom splt. Tble 1 shows the performnce comprson of dfferent lgorthms. The Bselne pproch s smply the nerest neghbor clssfcton on the orgnl mge spce. For other pproches, ll the trnng mges (lbeled nd unlbeled) re used to lern ether subspce or clssfer. The nerest neghbor clssfer s then performed n the subspce 3. The Bselne nd Egenfce pproches do not consder the mnfold structure nd get very poor performnce due to the llumnton chnge. All the other sem-supervsed lernng pproches mke use of the mnfold structure nd cheved sgnfcnt mprovements. Prtculrly, our SDA method cheved the best performnce mong ll the compred lgorthms Relevnce Feedbck Imge Retrevl Relevnce feedbck s well estblshed nd effectve frmework for nrrowng down the gp between low-level vsul fetures nd hgh-level semntc concepts n Content- Bsed Imge Retrevl (CBIR) [18]. Due to the lmtton of the user s feedbcks nd the hgh dmensonlty of the feture spce, one hopes to fnd subspce wth certn dmensonlty reducton lgorthms. The semntc reltonshp between mges cn be better reveled n ths subspce. The relevnce feedbck settng s certnly semsupervsed settng, wth lrge number of unlbeled dt (mges n the dtbse) nd smll number of lbeled dt (feedbcks provded by the user). Recently, there re consderble nterests on developng sem-supervsed dmensonlty reducton lgorthms for CBIR. Some populr ones nclude ncrementl Locl- 3 Snce we hve only one lbeled smple per clss, nerest neghbor clssfer nd the nerest centrod method re the sme.

6 Precson Bselne LPP ARE SSP SDA Precson Bselne LPP ARE SSP SDA Precson Bselne LPP ARE SSP SDA Scope Scope Scope () Feedbck Iterton 1 (b) Feedbck Iterton 2 (c) Feedbck Iterton 4 (d) (e) (f) Fgure 2. Compre the retrevl performnce of dfferent lgorthms. ()-(c) V llustrtng wth the precson-scope curves, we plot the results n the 1st, 2nd, nd 4th feedbck terton, respectvely. Our SDA lgorthm performs the best on the entre scope for ll the three feedbck tertons. (d) (f) Smple mges from ctegory 24, 25, nd 30, respectvely. ty Preservng Projecton (LPP) [12], Augmented Relton Embeddng (ARE) [15] nd Semntc Subspce Projecton (SSP) [25]. In ths experment, we compre our SDA wth these three lgorthms for relevnce feedbck mge retrevl Imge Dtbse nd Low Level Fetures The COREL dt set s wdely used n mny CBIR systems, such s [12, 15, 25]. For the ske of evlutons, we lso choose ths dt set for testng. 79 ctegores of color mges were selected, where ech conssts of 100 mges. Some smple mges re shown n Fgure 2. We combne 64-dmensonl color hstogrm nd 64- dmensonl Color Texture Moment (CTM, [24]) to represent the mges. The color hstogrm s clculted usng bns n HSV spce. The Color Texture Moment s proposed by Yu et l. [24], whch ntegrtes the color nd texture chrcterstcs of the mge n compct form. CTM dopts locl Fourer trnsform s texture representton scheme nd derves eght chrcterstc mps for descrbng dfferent spects of co-occurrence reltons of mge pxels n ech chnnel of the (SVcosH, SVsnH, V) color spce. Then CTM clcultes the frst nd second moment of these mps s representton of the nturl color mge pxel dstrbuton. Plese see [24] for detls Evluton Settngs To exhbt the dvntges of usng our pproch, we need relble wy of evlutng the retrevl performnce nd the comprsons wth other systems. Dfferent spects of the expermentl desgn re descrbed below. Evluton Metrcs: We use precson-scope curve [15] to evlute the effectveness of the mge retrevl lgorthms. The scope s specfed by the number (N) of top-rnked mges presented to the user. The precson s the rto of the number of relevnt mges presented to the user to the scope N. The precson-scope curve descrbes the precson wth vrous scopes nd thus gves n overll performnce evluton of the lgorthms. In rel mge retrevl system, query mge s usully not n the mge dtbse. To smulte such envronment, we use fve-fold cross vldton to evlute the lgorthms whch s lso dopted n the pper [15]. More precsely, we dvde the whole mge dtbse nto fve subsets wth equl sze. Thus, there re 20 mges per ctegory n ech subset. At ech run of cross vldton, one subset s selected s the query set, nd the other four subsets re used s the dtbse for retrevl. The precson-scope curve nd precson rte re computed by vergng the results from the fve-fold cross vldton. Automtc Relevnce Feedbck Scheme: We desgned n utomtc feedbck scheme to model the retrevl process. For ech submtted query, our system retreves nd rnks the mges n the dtbse. The top 10 rnked mges were selected s the feedbck mges, nd ther lbel nformton (relevnt or rrelevnt) s used for re-rnkng. Note tht, the mges whch hve been selected t prevous tertons re excluded from lter selectons. For ech query, the utomtc relevnce feedbck mechnsm s performed for four tertons. The smlr scheme ws used n [12], [15], [25].

7 4.2.3 Imge Retrevl Results In rel world, t s not prctcl to requre the user to provde mny rounds of feedbcks. The retrevl performnce fter the frst severl rounds of feedbcks s the most mportnt. Fgure 2 shows the verge precson-scope curves of the dfferent lgorthms for the 1st, 2nd nd 4th feedbck tertons. The bselne curve descrbes the ntl retrevl result wthout feedbck nformton. Specfclly, t the begnnng of retrevl, the Euclden dstnces n the orgnl 128-dmensonl spce re used to rnk the mges n the dtbse. After the user provdes relevnce feedbcks, the LPP, ARE, SSP, nd SDA lgorthms re then ppled to rernk the mges n the dtbse. Our SDA lgorthm sgnfcntly outperforms the other three lgorthms on the entre scope. ARE performs better thn the other two, especlly wth smll scope. All these four lgorthms re sgnfcntly better thn the bselne, whch ndctes tht the user provded relevnce feedbcks re very helpful for mprovng the retrevl performnce. 5. Concluson In ths pper, we propose new lner dmensonlty reducton lgorthm clled Sem-supervsed Dscrmnnt Anlyss. It cn mke effcent use of both lbeled nd unlbeled dt ponts. The lbeled dt ponts re used to mxmze the dscrmntng power, whle the unlbeled dt ponts re used to mxmze the loclty preservng power. Expermentl results on sngle trnng mge fce recognton nd relevnce feedbck mge retrevl demonstrte the effectveness of our lgorthm. References [1] P. N. Belhumeur, J. P. Hepnh, nd D. J. Kregmn. Egenfces vs. fsherfces: recognton usng clss specfc lner projecton. IEEE Trnsctons on Pttern Anlyss nd Mchne Intellgence, 19(7): , [2] M. Belkn nd P. Nyog. Lplcn egenmps nd spectrl technques for embeddng nd clusterng. In Advnces n Neurl Informton Processng Systems [3] M. Belkn, P. Nyog, nd V. Sndhwn. Mnfold regulrzton: A geometrc frmework for lernng from exmples. Journl of Mchne Lernng Reserch, [4] D. Beymer nd T. Poggo. Fce recognton from one exmple vew. In Proceedngs of the Ffth Interntonl Conference on Computer Vson (ICCV 95), [5] D. C, X. He, nd J. Hn. Spectrl regresson: A unfed subspce lernng frmework for content-bsed mge retrevl. In Proceedngs of the ACM Conference on Multmed, [6] O. Chpelle, J. Weston, nd B. Schölkopf. Cluster kernels for sem-supervsed lernng. In Advnces n Neurl Informton Processng Systems 16, [7] F. R. K. Chung. Spectrl Grph Theory, volume 92 of Regonl Conference Seres n Mthemtcs. AMS, [8] J. H. Fredmn. Regulrzed dscrmnnt nlyss. Journl of the Amercn Sttstcl Assocton, 84(405): , [9] K. Fukung. Introducton to Sttstcl Pttern Recognton. Acdemc Press, 2nd edton, [10] G. H. Golub nd C. F. V. Lon. Mtrx computtons. Johns Hopkns Unversty Press, 3rd edton, [11] T. Hste, R. Tbshrn, nd J. Fredmn. The Elements of Sttstcl Lernng: Dt Mnng, Inference, nd Predcton. New York: Sprnger-Verlg, [12] X. He. Incrementl sem-supervsed subspce lernng for mge retrevl. In Proceedngs of the ACM Conference on Multmed, New York, October [13] X. He nd P. Nyog. Loclty preservng projectons. In Advnces n Neurl Informton Processng Systems 16. MIT Press, Cmbrdge, MA, [14] X. He, S. Yn, Y. Hu, P. Nyog, nd H.-J. Zhng. Fce recognton usng lplcnfces. IEEE Trnsctons on Pttern Anlyss nd Mchne Intellgence, 27(3): , [15] Y.-Y. Ln, T.-L. Lu, nd H.-T. Chen. Semntc mnfold lernng for mge retrevl. In Proceedngs of the ACM Conference on Multmed, Sngpore, November [16] K. V. Mrd, J. T. Kent, nd J. M. Bbby. Multvrte Anlyss. Acdemc Press, [17] A. Y. Ng, M. Jordn, nd Y. Wess. On spectrl clusterng: Anlyss nd n lgorthm. In Advnces n Neurl Informton Processng Systems 14, pges MIT Press, Cmbrdge, MA, [18] Y. Ru, T. S. Hung, M. Orteg, nd S. Mehrotr. Relevnce feedbck: A power tool for ntertve content-bsed mge retrevl. IEEE Trnsctons on Crcuts nd Systems for Vdeo Technology, 8(5), [19] T. Sm, S. Bker, nd M. Bst. The CMU pose, llumnlton, nd expresson dtbse. IEEE Trnsctons on PAMI, 25(12): , [20] V. Sndhwn, P. Nyog, nd M. Belkn. Beyond the pont cloud: from trnsductve to sem-supervsed lernng. In Proc Int. Conf. Mchne Lernng (ICML 05), [21] A. N. Tkhonov. Regulrzton of ncorrectly posed problems. Sovet Mth., (4), 1963 (Englsh Trnslton). [22] M. Turk nd A. Pentlnd. Egenfces for recognton. Journl of Cogntve Neuroscence, 3(1):71 86, [23] V. N. Vpnk. Sttstcl lernng theory. John Wley & Sons, [24] H. Yu, M. L, H.-J. Zhng, nd J. Feng. Color texture moments for content-bsed mge retrevl. In Interntonl Conference on Imge Processng, pges 24 28, [25] J. Yu nd Q. Tn. Lernng mge mnfolds by semntc subspce projecton. In Proceedngs of the ACM Conference on Multmed, Snt Brbr, October [26] D. Zhou, O. Bousquet, T. Ll, J. Weston, nd B. Schölkopf. Lernng wth locl nd globl consstency. In Advnces n Neurl Informton Processng Systems 16, [27] X. Zhu, Z. Ghhrmn, nd J. Lfferty. Sem-supervsed lernng usng gussn felds nd hrmonc functons. In Proc. of the twenteth Internton Conference on Mchne Lernng, 2003.