General Averaged Divergence Analysis

Transcription

1 General Averaged Dvergence Analyss Dacheng ao, Xuelong 2, Xndong u 3,, and Stephen J Maybank 2 Departent of Coputng, Hong Kong Polytechnc Unversty, Hong Kong 2 Sch Coputer Scence & Inforaton Systes, Brkbeck, Unversty of ondon, ondon, UK 3 Departent of Coputer Scence, Unversty of Veront, USA csdct@coppolyueduhk; {xuelong, saybank}@dcsbbkacuk; xwu@csuvedu Abstract Subspace selecton s a powerful tool n data nng An portant subspace ethod s the Fsher Rao lnear dscrnant analyss (DA), whch has been successfully appled n any felds such as boetrcs, bonforatcs, and ulteda retreval However, DA has a crtcal drawback: the proecton to a subspace tends to erge those classes that are close together n the orgnal feature space If the separated classes are sapled fro Gaussan dstrbutons, all wth dentcal covarance atrces, then DA axzes the ean value of the Kullback ebler (K) dvergences between the dfferent classes e generalze ths pont of vew to obtan a fraework for choosng a subspace by ) generalzng the K dvergence to the Bregan dvergence and 2) generalzng the arthetc ean to a general ean he fraework s naed the general averaged dvergence analyss (GADA) Under ths GADA fraework, a geoetrc ean dvergence analyss (GMDA) ethod based on the geoetrc ean s studed A large nuber of experents based on synthetc data show that our ethod sgnfcantly outperfors DA and several representatve DA extensons Introducton he Fsher-Rao lnear dscrnant analyss (DA) has a proble n ergng classes that are close together n the orgnal feature space, as shown n Fg hs s referred to as the class separaton proble n ths paper As ponted out by Mcachlan [4], oog et al [], and u et al [3], ths ergng of classes sgnfcantly reduces the recognton rate Fg shows an exaple n whch DA does not select the optal subspace for pattern classfcaton o prove ts perforance, a weghted DA (DA) [8] s ntroduced However, the recognton rate of DA s senstve to the selecton of the weghtng functon oog et al [] developed another weghtng ethod for DA, naely the approxate parwse accuracy crteron (apac) he advantage of apac s that the proecton atrx can be obtaned by the egenvalue decoposton DA Proecton Drecton Fg here are three classes (naed, 2, and 3) of saples, whch are drawn fro a Gaussan dstrbuton n each class DA fnds a proecton drecton, and erges class and class 2 One of the reasonable proecton drectons for classfcaton trades the dstance between class and class 2 off the dstance between the classes, 2 and class 3 In ths paper, to further reduce the class separaton proble, we frst generalze DA to obtan a general averaged dvergence analyss (GADA) If dfferent classes are assued to be sapled fro Gaussan denstes wth dfferent expected values but dentcal covarances, then DA axzes the ean value of the Kullback ebler (K) dvergences [3] between the dfferent pars of denstes Our generalzaton of DA has two aspects: ) the K dvergence s replaced by the Bregan dvergence [2]; and 2) the arthetc ean s replaced by a general ean functon By choosng dfferent optons n ) and 2) a seres of subspace selecton algorths s obtaned, wth DA ncluded as a specal case Under the general averaged dvergence analyss, we nvestgate the effectveness of the geoetrc ean and K dvergence based subspace selecton for solvng the class separaton proble he geoetrc ean aplfes the effects of the sall dvergences and at the sae te reduces the effects of the large dvergences he ethod s naed the geoetrc ean dvergence analyss (GMDA)

2 2 near Dscrnant Analyss he a of DA [8] s to fnd n the feature space a low densonal subspace n whch the dfferent classes of easureents are well separated he subspace s spanned by a set of vectors, w,, whch fors the coluns of a atrx = [ w,, w ] It s assued that a tranng set of easureents s avalable he tranng set s dvded nto c classes he th class contans n easureents x ; ( n ), and has a ean value n = = of ( n ) ; µ x he between class scatter atrx S b and the wthn class scatter atrx S w are defned by where S µ µ µ µ c b = n ( )( ) n = c n w ( ; )( ; ) n = = c n = S = x µ x µ () n= s the sze of the tranng set and c n ( ) = n ; µ x s the ean vector of the total = = tranng set he proecton atrx * of DA s defned by (( w ) b ) * = argaxtr S S (2) he proecton atrx * s coputed fro the egenvectors of S ws b, under the assupton that S w s nvertble If c equals to 2, DA reduces to Fsher dscrnant analyss [9]; otherwse DA s known as Rao dscrnant analyss [6] 3 General Averaged Dvergence Analyss If the dfferent classes are assued to be sapled fro Gaussan denstes wth dfferent expected values but dentcal covarances, then DA axzes the ean value of the K dvergences between the dfferent pars of denstes e propose a fraework, the General Averaged Dvergence Analyss, for choosng a dscrnatve subspace by: ) generalzng the dstorton easure fro the K dvergence to the Bregan dvergence, and 2) generalzng the arthetc ean to a general ean functon 3 Bregan Dvergence [2] Defnton (Bregan Dvergence): et U : S R be a C convex functon defned on a closed convex set S R + he frst dervatve of U s U, whch s a onotone functon he nverse functon of U s ( U ) ( ) ξ = he saple probablty of the th class s p = p y = ξ p between the functon U and the tangent lne to U at ξ p, U ξ p s gven by: x he dfference at ( ( )) s ( ξ, ξ ) ( ) ( ) d p p { U ξ p U ξ p } p{ ξ( p) ξ( p) } = Based on (3), the Bregan dvergence for ( ) (, ) p and (3) D p p = d ξ p ξ p dµ, (4) where dµ s the ebesgue easure he rght-hand sde of (4) s also called the U dvergence [5] Because U s a convex functon, d( ξ( p), ξ ( q) ) s non negatve Consequently, the Bregan dvergence s non negatve Because d( ξ( p), ξ ( q) ) s n general not syetrc, the Bregan dvergence s also not syetrc Detaled nforaton about the Bregan dvergence can be found n [5] If U( x) = exp( x), then the Bregan dvergence reduces to the usual K-dvergence, p D( p q) = p p p log dµ p (5) p = p log dµ = K( p q) p Further exaples can be found n [5] For Gaussan probablty densty functons, p N( xµ ;, Σ), where µ s the ean vector of the th class saples and Σ s the wthn class covarance atrx of the th class, the K dvergence [3] s N ( xµ ;, Σ) K ( p p ) = dxn ( x ; µ, Σ) ln N ( xµ ;, Σ ) (6) = ln Σ ln Σ + tr Σ Σ + tr Σ D, ( ) ( ) where D = ( µ µ ) ( µ µ ) and Σ det ( Σ) o splfy the notaton we denote the K dvergence p x y = and between the proected denstes p ( x y = ) by D ( p p) = D( p( y = ) p( y = ) ) p x x (7) 32 General Averaged Dveregnces Analyss e replace the arthetc ean by the followng general ean,

3 ϕ ( ( )) qq n qq D p p c Vϕ ( ) = ϕ (8) n c where ϕ () s a strct onotonc real-valued ncreasng functon defned on ( 0, + ) ; () functon of ϕ () ; ϕ s the nverse q s the pror probablty of the th class (usually, we can set q = n n or sply q = c); p s the condtonal dstrbuton of the th n class; x R n where R s the feature space contanng the tranng n k saples; and R ( n k ) s the proecton atrx he general averaged dvergence functon easures the average of all dvergences between pars of classes n the subspace e obtan the proecton atrx * by axzng the general averaged dvergence functon V ϕ ( ) over for a fxed ϕ () he general optzaton algorth for subspace selecton based on (8) s gven n able Note that, usually, the concavty of V ϕ ( ) cannot be guaranteed o reduce the effects of local axa [], we choose a nuber of dfferent ntal proecton atrces, perfor optzatons on the, and then select the best one If V ϕ ( ) depends only on the subspace of R n spanned by the coluns of, then can be replaced by M where M s an atrx n whch the coluns of M are orthogonal On settng ϕ ( x) = x, we use the followng arthetc ean based ethod for choosng a subspace, qq D ( p p) * = argax c qqn n c (9) = arg ax qq D p p c Observaton : DA axzes the arthetc ean of the K dvergences between all pars of classes, under the assupton that the Gaussan dstrbutons for the dfferent classes all have the sae covarance atrx he proecton atrx * n DA can be obtaned by axzng a partcular V ϕ ( ) Proof Accordng to (6) and (7), the K dvergence between the th class and the th class n the proected subspace wth the assupton of equal covarance atrces ( Σ = Σ = Σ ) s as follows: D p p = tr Σ D + constant (0) hen, we have able General Averaged Dvergence Maxzaton for Subspace Selecton Input: ranng saples x ;, where denotes the th class ( c ) and s the th saple n the th class ( n ), the denson of selected features k < n (n s the denson of x ; ), and M s the axu nuber of dfferent ntal values for the proecton atrx Output: Optal lnear proecton atrx * for = : M { 2 Randoly ntalze t ( t = ), e, all entres of are rando nubers 3 whle V ( t ) V ( t ) * = argax qq D p p = arg ax c c (( Σ) D ) ( qq tr ) = arg ax tr ( Σ) qq D c c c = arg ax tr ( Σ) qq D = = + c c Because Sb = qq = = + D, as proved by oog [0], and St = Sb + Sw = Σ (see [8]), we have arg ax qq D p p ( ) c (( S w ) S b ) = arg ax tr ϕ ϕ > ε, do{ Conduct the gradent steepest ascent algorth to axze the averaged dvergences defned n (8): 4 t t + κ V ϕ ( t ) where κ s a sall value (eg, 000) 5 t t+ 6 }//whle on lne 3 7 }//for on lne * argax 8 V ϕ ( t ) () It follows fro () that a soluton of DA can be obtaned by the generalzed egenvalue decoposton Exaple: Decell and Mayekar [5] axzed the arthetc ean of all syetrc K dvergences between all pars of classes n the proected subspace he syetrc K dvergences are gven by

4 SK( p p) = K( p p) + K( p p) 2 2 = tr + + tr + ( Σ Σ Σ Σ) ( Σ Σ )( µ µ )( µ µ ) (2) In essence, there s no dfference between [5] and axzng the arthetc ean of all K dvergences De la orre and Kanade [4] developed the orented dscrnant analyss (ODA) based on the sae obectve functon used n [5], but used teratve aorzaton to obtan a soluton Iteratve aorzaton speeds up the tranng stage Furtherore, they generalzed ODA for a ultodal case as the ultodal ODA (MODA) by cobnng t wth Gaussan Mxture Models (GMM) learnt by a noralzed cut for the ultodal case Each class s odelled by a GMM 33 How to Deal wth Multodal Case Up to ths pont t has been assued that the easureent vectors n a gven class are sapled fro a sngle Gaussan dstrbuton hs assupton often fals n large real world data sets, such as those used for ult vew face/gat recognton, natural age classfcaton or texture classfcaton o overcoe ths ltaton, each class can be odeled by a GMM Many ethods for obtanng GMMs are descrbed n the lterature Exaples nclude KMeans [6], GMM wth expectaton axzaton (EM) [6], graph cut [7], and spectru clusterng Unfortunately, these ethods are not adaptve, n that the nuber of subclusters ust be specfed, and soe of the (eg, EM and KMeans) are senstve to the ntal values In our algorth we use a recently ntroduced GMM EM lke algorth proposed by Fgueredo and Jan [7], whch was naed the GMM FJ ethod he reasons for choosng GMM-FJ are as follows: t fnds the nuber of subclusters; t s less senstve to the choce of the ntal values of the paraeters than EM; and t can avod the boundary of the paraeter space e assue that the easureents n each class are sapled fro a GMM and the proecton atrx can be obtaned by axzng the general averaged dvergences, whch easure the averaged dstorton between any par of subclusters n dfferent classes, e, k l k l qq ϕ ( D ( p p) ) c k C l C Vϕ ( ) = ϕ s t, qq n n c s C t C (3) k where q s the pror probablty of the k th subcluster of the th class; p k s the saple probablty of the k th k l D p p s the dvergence subcluster n the th class; ( ) between the k th subcluster n the th class and the l th subcluster n the th class 34 Geoetrc Mean based Subspace Selecton In DA and ODA the arthetc ean of the dvergences s used to fnd a sutable subspace to proect the feature vectors he an beneft of usng the arthetc ean s that the proecton atrx can be obtaned by the generalzed egenvalue decoposton However, DA s not optal for ultclass classfcaton [4] because of the class separaton proble entoned n Secton I herefore, t s useful to nvestgate other choces of ϕ n (8) he log functon s a sutable choce for ϕ because t ncreases the effects of the sall dvergences and at the sae te reduces the effects of the large dvergences On settng ϕ ( x) = log ( x) n (9) the generalzed geoetrc ean of the dvergences s obtaned he requred subspace * s gven by c ( ) qq * = argax D p p qqn (4) n c It follows fro the ean nequalty that the generalzed geoetrc ean s upper bounded by the arthetc ean of the dvergences, e, c ( ) D p p qq qqn n c qq D ( p p) c qq n n c Furtherore, (4) ephaszes the total volue of all dvergences For exaple, n the specal case of q = q for all,, c c c ( ) arg ax D p p ( ) ( ) = arg ax D p p = arg ax D p p 35 K Dvergence qq qqn n c qq In ths paper, we cobne the K dvergence and the geoetrc ean as an exaple for practcal applcatons Replacng D ( p p) wth the K dvergence and optzng the logarth of (4), we have

5 * = argax c ( ) = arg ax log K p p, and K ( p p) (5) s the K dvergence between the th class and the th class n the proected subspace, K ( p p ) = log Σ log Σ 2 + tr Σ Σ (6) (( ) ( )) ( Σ ) D + tr o obtan the optzaton procedure for the geoetrc ean and the K dvergence based subspace selecton algorth based on able, we need the frst order dervatve of ( ), ( ) = K( p p ) K( p p ) (7) and c ( ) ( ) ( ) + ( Σ + D) ( Σ ) Σ ( Σ ) ( Σ D) ( Σ ) K p p = Σ Σ Σ Σ + (8) he ultodal extenson of (5) can be drectly obtaned fro (3) 4 Coparatve Studes Usng Synthetc Data In ths secton, we denote the proposed ethod as the geoetrc ean dvergence analyss (GMDA), and copare GMDA wth DA [8], apac [], DA (slar to apac, but wth a dfferent weghtng functon), HDA [2], ODA [4], and MODA [4] e use the 3 weghtng functon for DA d 4 Heteroscedastc Proble o exane the classfcaton ablty of these subspace selecton ethods for the heteroscedastc proble [2], we generate two classes such that each class has 500 saples, drawn fro a Gaussan dstrbuton he two classes have dentcal ean values but dfferent covarances As shown n Fg 2, DA, apac, and DA separate class eans wthout takng the dfferences between covarances nto account In contrast, HDA, ODA, and GMDA consder both the dfferences between class eans and the dfferences between class covarances, so they have less tranng errors, as shown n Fg 2 42 Multodal Proble In any applcatons t s useful to odel the dstrbuton of a class usng a GMM, because saples n the class ay be drawn fro a ultodal dstrbuton o deonstrate the classfcaton ablty of the ultodal extenson of GMDA, we generate two classes; each class has two subclusters, and saples n each subcluster are drawn fro a Gaussan Fg 3 shows the selected subspaces of dfferent ethods In ths case DA, DA, and apac do not select the sutable subspace for classfcaton However, the ultodal extensons of ODA and GMDA can fnd the sutable subspace Furtherore, although HDA does not take account of ultodal classes, t can select the sutable subspace hs s because n ths case the two classes have slar class eans but sgnfcantly dfferent class covarance atrces when each class s odeled by a sngle Gaussan For coplex cases, eg, when each class conssts of ore than 3 subclusters, HDA wll fal to fnd the optal subspace for classfcaton 43 Class Separaton Proble he ost pronent advantage of GMDA s that t can sgnfcantly reduce the classfcaton errors caused by the too strong effects of the large dvergences between certan classes o deonstrate ths pont, we generate three classes and the saples n each class are drawn fro a Gaussan dstrbuton wo classes are close together and the thrd s far away In Fg 4, t s deonstrated that GMDA shows a good ablty to separate the last two classes of saples However, DA, HDA, and ODA do not gve good results he apca and DA algorths are better than DA but nether of the gves the sutable proecton drecton he results obtaned fro apca are better than those obtaned fro DA, because apac uses a better weghtng strategy than DA 5 Statstcal Experents In ths secton, we utlze a synthetc data odel, whch s a generalzaton of the data generaton odel used by orre and Kanade [4], to evaluate MGMKD n ters of accuracy and robustness he accuracy s easured by the average error rate and the robustness s easured by the standard devaton of the classfcaton error rates In ths data generaton odel, there are fve classes In our experents, for each of the tranng/testng sets, the data generator gves 200 saples for each of the fve classes (therefore,,000 saples n total) Moreover, the saples n each class are obtaned fro a Gaussan Each Gaussan densty s a lnear transforaton of a standard noral dstrbuton he

6 lnear transforatons are defned by x ; = z + µ + n, where x R, z ~ N 0, I R, ; ~,2, R 20 n N 0 I R, denotes the th class, denotes the th saple n ths class, and µ s the ean value of the correspondng noral dstrbuton he µ are assgned as follows: µ = 2N ( 0,) , µ 2 = 020, µ 3 = ( 2N ( 0,) 4 )[ 00, 0], 4 = ( 2N ( 0,) + 4 )[ 0, 0] and 5 = ( 2N ( 0,) + 4 )[ 5, 5, 5, 5] µ 0, µ 0 0 he proecton atrx s a rando atrx Each of ts eleents s Based on ths data generaton sapled fro N ( 0,5) odel, 800 groups (each group wth the tranng and testng saples) of synthetc data are generated fro the odel For coparson, the subspace selecton ethods are frst utlzed to select a gven nuber of features hen the Mahalanobs dstance [6] and the nearest neghbour rule are used to exane the accuracy and robustness of GMDA n coparson wth DA and ts extensons he baselne algorths are DA, apac, DA, HDA, and ODA e conducted the above desgned experents 800 tes based on randoly generated data sets he experental results are reported n ables 2-5 ables 2 and 4 show the average error rates of DA, apac, DA, HDA, ODA, and GMDA based on the Mahalanobs dstance and the nearest neghbour rule, respectvely Heren arthetc ean values are coputed on dfferent feature densons fro to 6 (by colun) Correspondngly, the standard devatons under each condton, whch easure the robustness of the classfers, are gven n ables 3 and 5 e have twenty feature densons for each saple and all the saples are dvded nto fve classes herefore, the axal feature nuber for DA, apac, and DA s 5 =4; n contrast, HDA, ODA, and GMDA can extract ore features than DA, apac, and DA Based on ables 2-5, t can be concluded that GMDA outperfors DA, apac, DA, HDA, and ODA, consstently e now deonstrate why GMDA s a sutable subspace selecton ethod for classfcaton et us frst study the relatonshp between ( ), whch s defned n (5), and the tranng error rate Experents are done on a randoly selected data set fro 800 data sets generated at the begnnng of ths secton e set tranng teratons as 200 In Fg 5, the left shows that the classfcaton error rates decrease wth the ncrease of the tranng teratons and the rght shows that the obectve functon values ( ) ncrease wth the ncrease of the tranng teratons onotoncally herefore, the classfcaton error rates decrease wth the ncrease of ( ) hs eans that axzng wll be useful to acheve a low classfcaton error rate It s also portant to nvestgate how K dvergences between dfferent classes change wth the ncreasng nuber of tranng teratons, because t s helpful to deeply understand how and why GMDA reduces the class separaton proble In Fg 6, we show how K dvergences change n GMDA over the st, 2 nd, 5 th, 0 th, 20 th, and 200 th tranng teratons, respectvely he sall K dvergences, whch are less than 2, are arked wth rectangles here are 5 classes, so we can ap K dvergences to a 5 5 atrx wth zero dagonal values he entry of the th colun and the th row eans the K dvergence between the th class and the th class e denote t as K ( t ), where t eans the t th tranng teraton Because the K dvergence s not syetrc, the atrx s not syetrc, e, K ( t ) K ( t ) Accordng to Fg 6, n the st tranng teraton (the top left 5 5 atrx), there are 8 values less than 2 In the 2 nd teraton (the top rght 5 5 atrx), there are only 4 values less than 2 Copared wth the st teraton, 6 out of 8 have ncreased In the 5 th teraton (the ddle left 5 5 atrx), there are only 2 values less than 2 and they have ncreased n coparson wth the 2 nd teraton However, these two dvergences have decreased to 0439 and 0366 n the 200 th teraton (the botto rght 5 5 atrx) n coparng wth the 20 th teraton (the botto left 5 5 atrx) to guarantee the ncrease of ( ) hs s not sutable to separate classes, because the dvergences between the are very sall 6 Concluson If separate classes are sapled fro Gaussan dstrbutons, all wth dentcal covarance atrces, then the Fsher Rao lnear dscrnant analyss (DA) axzes the ean value of the Kullback ebler (K) dvergences between the dfferent classes e have generalzed ths pont of vew to obtan a fraework for choosng a subspace by ) generalzng the K dvergence to the Bregan dvergence and 2) generalzng the arthetc ean to a general ean he fraework s naed the general averaged dvergence analyss (GADA) Under ths fraework, the geoetrc ean and K dvergence based subspace selecton s then studed DA has a crtcal drawback n that the proecton to a subspace tends to erge those classes that are close together n the orgnal feature space A large nuber of experents based on synthetc data have shown that our ethod sgnfcantly outperfors DA and several representatve DA extensons n overcong ths drawback

7 References [] S Boyd and Vandenberghe, Convex Optzaton, Cabrdge Unversty Press, 2004 [2] M Bregan, he Relaxaton Method to Fnd the Coon Ponts of Convex Sets and Its Applcaton to the Soluton of Probles n Convex Prograng, USSR Copt Math and Math Phys, no 7, pp , 967 [3] M Cover and J A hoas, Eleents of Inforaton heory New York: ley, 99 [4] F De la orre and Kanade Multodal Orented Dscrnant Analyss, Int l Conf Machne earnng, 2005 [5] H P Decell and S M Mayekar, Feature Cobnatons and the Dvergence Crteron, Coputers and Math th Applcatons, vol 3, pp 7 76, 977 [6] RO Duda, PE Hart, and DG Stork, Pattern Classfcaton John ley and Sons Inc 200 [7] M Fgueredo and AK Jan, Unsupervsed learnng of fnte xture odels, IEEE rans Pattern Analyss and Machne Intellgence, vol 24, no 3, pp , 2002 [8] K Fukunaga, Introducton to statstcal pattern recognton (Second Edton) Acadec Press 990 [9] R A Fsher, he Statstcal Utlzaton of Multple Measureents, Ann Eugencs, vol 8 pp , 938 [0] M oog, Approxate Parwse Accuracy Crtera for Multclass near Denson Reducton: Generalzatons of the Fsher Crteron, Delft Unv Press, 999 [] M oog, R P Dun, and R Haeb Ubach, Multclass near Denson Reducton by eghted Parwse Fsher Crtera, IEEE rans Pattern Analyss Machne Intellgence, vol 23, no 7, pp , July 200 [2] M oog and R P Dun, near Densonalty Reducton va a Heteroscedastc Extenson of DA: he Chernoff Crteron, IEEE rans Pattern Analyss Machne Intellgence, vol 26, no 6, pp , June 2004 [3] J u, KN Platanots, and AN Venetsanopoulos, Face Recognton Usng DA Based Algorths, IEEE rans Neural Networks, vol 4, no, pp , 2003 [4] GJ Mcachlan, Dscrnant Analyss and Statstcal Pattern Recognton, ley, New York, 992 [5] N Murata, akenouch, Kanaor, and S Eguch, Inforaton Geoetry of U Boost and Bregan Dvergence, Neural Coputaton, vol 6, no 7, pp,437,48, 2004 [6] C R Rao, he Utlzaton of Multple Measureents n Probles of Bologcal Classfcaton, J Royal Statstcal Soc, B, vol 0, pp , 948 [7] J Sh and J Malk, Noralzed Cuts and Iage Segentaton, IEEE rans Pattern Analyss and Machne Intellgence, vol 22, no 8, pp , Aug 2000 Fg 2 Heteroscedastc proble: n ths fgure, fro left to rght, fro top to botto, there are sx subfgures showng the proecton drectons obtaned usng DA, HDA, apac, DA, ODA, and GMDA he tranng errors of these ethods, as easured by Mahanalobs dstance, are 0340, 02880, 0340, 0340, 02390, and ODA and GMDA fnd the best proecton drecton for classfcaton

8 Fg 3 Multodal proble: n ths fgure, fro left to rght, fro top to botto, there are sx subfgures to descrbe the optal proecton drectons by usng DA, HDA, apac, DA, MODA, and a ultodal extenson of GMDA (M-GMDA) he tranng errors easured by Mahalanobs dstance of these ethods are 0097, 0067, 0097, 0097, 00083, and MODA and M-GMDA fnd the best proecton drecton for classfcaton

9 Fg 4 arge class dvergence proble: n ths fgure, fro left to rght, fro top to botto, there are nne subfgures to descrbe the proecton drectons (ndcated by lnes n each subfgure) by usng DA, HDA, apac, DA, ODA, and GMDA he tranng errors easured by Mahalanobs dstance of these ethods are 0300, 0300, 02900, 03033, 0300, and 067 GMDA fnds the best proecton drecton for classfcaton 05 Classfcaton Error Rate vs ranng Iteratons GMDA 55 Obectve Functon Value vs ranng Iteratons Classfcaton Error Rate Obectve Functon Value ranng Iteratons 20 GMDA ranng Iteratons Fg 5 he consstency of the GMDA obectve functon ( ) and the classfcaton error rate ( ) = ( ) 2 = 3637 ( ) ( ) 5 = = ( ) 20 = ( ) 200 = 5962 Fg 6 he K dvergences n GMDA over st, 2 nd, 5 th, 0 th, 20 th, and 200 th tranng teratons

10 able 2: Average error rates (ean for 800 experents) of DA, apac, DA, HDA, ODA, and GMDA (Mahalanobs dstance) Bass DA apac DA HDA ODA GMDA able 3: Standard devatons of error rates (for 800 experents) of DA, apac, DA, HDA, ODA, and GMDA (Mahalanobs dstance) Bass DA apac DA HDA ODA GMDA able 4: Average error rates (ean for 800 experents) of DA, apac, DA, HDA, ODA, and GMDA (Nearest neghbor rule) Bass DA apac DA HDA ODA GMDA able 5: Standard devatons of error rates (for 800 experents) of DA, apac, DA, HDA, ODA, and GMDA (Nearest neghbor rule) Bass DA apac DA HDA ODA GMDA