MULTI-REGULARIZATION PARAMETERS ESTIMATION FOR GAUSSIAN MIXTURE CLASSIFIER BASED ON MDL PRINCIPLE

Transcription

1 MULI-REGULARIZAIO PARAMEERS ESIMAIO FOR GAUSSIA MIXURE CLASSIFIER BASED O MDL PRICIPLE Xulng Zou,, Png Guo an C L Plp Cen 3 e Laboratory of Image Processng an Pattern Recognton, Beng ormal Unversty, Beng, 875, Cna Artfcal Intellgence Insttute, Beng Cty Unversty, Beng, Cna 3 e Faculty of Scence & ecnology, Unversty of Macau, Macau SAR, Cna zxlmouse@bcueucn, {pguo, plpcen}@eeeorg Keywors: Abstract: Gaussan classfer, Covarance matrx estmaton, Mult-regularzaton parameters selecton, Mnmum escrpton lengt Regularzaton s a soluton to solve te problem of unstable estmaton of covarance matrx wt a small sample set n Gaussan classfer An mult-regularzaton parameters estmaton s more ffcult tan sngle parameter estmaton In ts paper, KLIM_L covarance matrx estmaton s erve teoretcally base on MDL (mnmum escrpton lengt) prncple for te small sample problem wt g menson KLIM_L s a generalzaton of KLIM (Kullbac-Lebler nformaton measure) wc consers te local fference n eac menson Uner te framewor of MDL prncple, mult-regularzaton parameters are selecte by te crteron of mnmzaton te KL vergence an estmate smply an rectly by pont estmaton wc s approxmate by two-orer aylor expanson It costs less computaton tme to estmate te mult-regularzaton parameters n KLIM_L tan n RDA (regularze scrmnant analyss) an n LOOC (leave-one-out covarance matrx estmate) were cross valaton tecnque s aopte An ger classfcaton accuracy s aceve by te propose KLIM_L estmator n experment IRODUCIO Gaussan mxture moel (GMM) as been wely use n real pattern recognton problem for clusterng an classfcaton, were te maxmum leloo crteron s aopte to estmate te moel parameters wt te tranng samples (Bsop, 7) (Evertt, 98) However, t often suffers from small sample sze problem wt g mensonal ata In ts case, for -mensonal ata, f less tan + tranng samples from eac class s avalable, te sample covarance matrx estmate n Gaussan classfer s sngular An ts can lea to lower classfcaton accuracy Regularzaton s a soluton to ts n of problem Srnage an regularze covarance estmators are examples of suc tecnques Srnage estmators are a wely use class of estmators wc regularze te covarance matrx by srnng t towar some postve efnte target structures, suc as te entty matrx or te agonal of te sample covarance (Freman, 989); (Hoffbec, 996); (Scafer, 5); (Srvastava, 7) More recently, a number of metos ave been propose for regularzng te covarance estmate by constranng te estmate of te covarance or ts nverse to be sparse (Bcel, 8); (Freman, 8); (Cao, ) e above regularzaton metos manly concern varous mxture of sample covarance matrx, common covarance matrx an entty matrx or constrant te estmate of te covarance or ts nverse to be sparse In tese metos, te regularzaton parameters are requre to be etermne by cross valaton tecnque Altoug te regularzaton metos ave been successfully apple for classfyng small-number ata wt some eurstc approxmatons (Freman, 989); (Hoffbec, 996), te selecton of regularzaton parameters by cross valaton tecnque s very computaton-expensve Moreover, cross-valaton performance s not always well n te selecton of lnear moels n some cases (Rvals, 999) Recently, a covarance matrx estmator calle Kullbac-Lebler nformaton measure (KLIM) s evelope base on mnmum escrpton lengt

2 MULI-REGULARIZAIO PARAMEERS ESIMAIO FOR GAUSSIA MIXURE CLASSIFIER BASED O MDL PRICIPLE (MDL) prncple for small number samples wt g menson ata (Guo, 8) e KLIM estmator s erve teoretcally by KL vergence An a formula for fast estmaton te regularzaton parameter s erve However, snce multparameters optmzaton s more ffcult tan sngle parameter optmzaton, only a specal case tat te regularzaton parameters are taen te same value for all mensons s consere n KLIM oug estmaton of regularzaton parameter becomes smple, te accuracy of covarance matrx estmaton s ecrease by gnore te local fference n eac menson s wll result n ecreasng te classfcaton accuracy of Gaussan classfer fnally In ts paper, KLIM s generalze to KLIM_L wc consers te local fference n eac menson Base on MDL prncple, te KLIM_L covarance matrx estmaton s erve for te small sample problem wt g menson Multregularzaton parameters n eac menson are selecte by te crteron of mnmzaton te KL vergence an estmate effcently by two-orer aylor expanson e feasblty an effcency of KLIM_L are sown by te experments HEOREICAL BACKGROUD Gaussan Mxture Classfer Gven a ata set D { x } wc wll be classfe Assume tat te ata pont n D s sample from a Gaussan mxture moel wc as component: were p( x, ) G( x, m, ) wt,, G x x x (, m, ) ( ) / exp{ ( m ) ( m )} / () () s a general multvarate Gaussan ensty functon x s a ranom vector, te menson of x s an {,, } m s a parameter vector set of Gaussan mxture moel In te case of Gaussan mxture moel, te Bayesan ecson rule arg max p( x, ) s aopte to classfy te vector x nto class wt te largest posteror probablty p( x, ) After moel parameters estmate by te maxmum leloo (ML) meto wt expectatonmaxmzaton (EM) algortm (Rener, 984), te posteror probablty can be wrtten n te form: ˆ (, ˆ, ˆ ) (, ˆ G x m p x ) p( x, ˆ ),,, An te classfcaton rule becomes: were, (3) arg mn ( x ),,,,, (4) ˆ ˆ ˆ x x (5) ( ) ( ˆ ) ( ˆ X m m ) ln ln s equaton s often calle te scrmnant functon for te class (Aeberar, 994) Snce clusterng s more general tan classfcaton n te mxture moel analyss case, we conser te general case n te followng Covarance Matrx Estmaton e central ea of te MDL prncple s to represent an entre class of probablty strbutons as moels by a sngle unversal representatve moel, suc tat t woul be able to mtate te beavour of any moel n te class e best moel class for a set of observe ata s te one wose representatve permts te sortest cong of te ata e MDL estmates of bot te parameters an ter total number are consstent; e, te estmates converge an te lmt specfes te ata generatng moel (Rssanen, 978); (Barron, 998) e coelengt (probablty strbuton or a moel) crteron of MDL nvolves n te KL vergence (Kullbac, 959) ow conserng a gven sample ata set D { x } generate from an unnown ensty, t can be moelle by a fnte Gaussan, were s te parameter p( X ) mxture ensty p( x, ) set In te absence of nowlege of p( x ), t may be estmate by an emprcal ernel ensty estmate p ( x ) obtane from te ata set Because tese two probablty enstes escrbe te same unnown ensty p( x ), tey soul be best matce wt proper mxture parameters an smootng parameters Accorng to MDL prncple, te moel parameters soul be estmate wt mnmze KL vergence KL(, ) base on te gven ata rawn from te unnown ensty p( x ) (Kullbac, 959), 3

3 CA - Internatonal Conference on eural Computaton eory an Applcatons wt p p ( x) KL(, ) p ( x) ln x p( x, ) ( x) (,, ) G x x W exp ( ) ( ) / / x x W x x ( ) W (6) (7) Here W s a mensonal agonal matrx wt a general form, W ag(,,, ), (8) were,,,, are smootng parameters (or regularzaton parameters) n te nonparametrc ernel ensty In te followng ts set s enote as { } e Eq (6) equals to zero f an only f p ( x) p( x, ) If te lmt, te ernel ensty functon p ( x ) becomes a functon, ten Eq (6) reuces to te negatve log leloo functon So te ornary EM algortm can be re-erve base on te mnmzaton of ts KL vergence functon wt te lmt e ML-estmate parameters are sown as follows: mˆ p(, ˆ x ) x, n m m ˆ p( x, ˆ)( x ˆ )( x ˆ ) n (9) e covarance matrx estmaton for te lmt s sown as follows By mnmzng Eq (6) wt respect to, e, settng KL(, )/, te followng covarance matrx estmaton formula can be obtane: ˆ p x x ˆ x x x ( ) p(, )( mˆ )( mˆ ) p ( x) p( x, ˆ )x () In ts case, te aylor expanson s use for p( x, ) at x x wt respect to x an t s expane to frst orer approxmaton: ˆ ˆ ˆ x p( x, ) p( x, ) ( x x ) p( x, ) () wt p( x, ˆ) p( x, ˆ) x x x x On substtutng te above equaton nto Eq () an accorng to te propertes of probablty ensty functon, te followng approxmaton s fnally erve: ( ) W ˆ () e estmaton n Eq () s calle as KLIM_L n te paper, were ˆ s te ML estmaton wen, tang te form of Eq (9) 3 MULI- REGULARIZAIO PARAMEERS ESIMAIO BASED O MDL 3 Regularzaton Parameters Selecton e regularzaton parameter set n te Gaussan ernel ensty plays an mportant role n estmatng te mxture moel parameter Dfferent wll generate fferent moels So selectng te regularzaton parameters s a moel selecton problem In te paper te smlar meto as n (Guo, 8) s aopte base on MDL prncple to select te regularzaton parameters n KLIM_L Accorng to te prncple of MDL, t soul be wt te sortest coelengt to select a moel Wen, te regularzaton parameters can be estmate wt te mnmze KL vergence regarng wt ML estmate parameter ˆ, * arg mn J( ), J( ) KL(, ˆ ) (3) ow a secon orer approxmaton for estmatng te regularzaton parameter s aopte ere Rewrte te J ( ) as: J ( ) J ( ) J ( ), (4) were J ( ) p ( x) ln p( x, )x, J e( ) p ( ) ln p ( ) e x x x Replacng ln p( x, ) wt te secon orer term of aylor expanson nto te ntegral of J ( ) an resultng n te followng approxmaton of J ( ), J ( ) ln p( x, ) trace( ( ln p( x, ))) W x x (5) For very sparse ata strbuton, te followng approxmaton can be use: 4

4 MULI-REGULARIZAIO PARAMEERS ESIMAIO FOR GAUSSIA MIXURE CLASSIFIER BASED O MDL PRICIPLE p ln p G(, ) ln G(, ) ( x) ( x),, x x W x x W G( xx,, W)[ ln ln ln W ( x x ) W ( x x )] Substtutng te Eq (6) nto J ( ), t can be got: e (6) J ( ) ( ) ln ( ) e p x p x x (7) ln ln ln W So far, te approxmaton formula of J ( ) s obtane: J( ) trace( W ( ln p( x, ))) x x, (8) ln W C were C s a constant rrelevant to Let H xx ln p( x, ) ang partal ervatve of J ( ) to W an lettng t be equal to zero, te roug approxmaton formula of s obtane as follows: W ag( H ) (9) 3 Approxmaton for Regularzaton Parameters e Eq (9) can be rewrtten as follows: W ag( H ) ag( xxln p( x, )) wt x xln p( x, ) { p( x, )[ ( x m )( x m ) ] p x x m p x x m (, ) ( ) (, )( ) } Conserng ar-cut case ( p( x, ) or ) an usng te approxmatons p( x, )( x )( x ) n ˆ, m m p( x, ) n, ˆ I an t can be obtane:, W () ag( ) Suppose te egenvalues an egenvectors of te common covarance matrx are an,, were (,,, ), ten tere exsts:, () Substtutng Eq () nto Eq () an usng te average egenvalue ( )/ trace( )/ to substtute eac egenvalue of matrx (only n te enomnator), t can be obtane: W trace ( ) ag( ) wt [ ] Fnally, te regularzaton parameters can be approxmate by te followng equaton: trace ( ) W ag(,, ) () 33 Comparson of KLIM_L wt Regularzaton Metos e four metos (KLIM_L, KLIM, RDA (Regularze Dscrmnant Analyss, Freman, 989) an LOOC (Leave-one-out covarance matrx estmate, Hoffbec, 996)) are all regularzaton metos to estmate te covarance matrx for small sample sze problem ey all conser ML estmate covarance matrx wt te atonal extra matrces KLIM_L s erve by te smlar way as KLIM uner te framewor of MDL prncple Meanwle, te estmaton of regularzaton parameters s smlar to KLIM base on MDL prncple KLIM_L s a generalzaton of KLIM Mult-regularzaton parameters are nclue an estmate n KLIM_L wle one regularzaton parameter s estmate n KLIM For every ( ), f t s taen by trace( ), ten W (trace( ) / ) I It wll reuce to te case of W I n KLIM, were trace( ) / KLIM_L s erve base on MDL prncple wle RDA an LOOC are eurstcally propose ey ffer n mxtures of covarance matrx consere an te crteron use to select te regularzaton parameters 5

5 CA - Internatonal Conference on eural Computaton eory an Applcatons Dfferent computaton tme costs are requre n te four regularzaton scrmnant metos e tme costs of tem are sorte ecreasng n te followng orer: KLIM, KLIM_L, LOOC an RDA s wll be valate by te followng experments 4 EXPERIME RESULS In ts secton, te classfcaton accuracy an tme cost of KLIM_L are compare wt LDA (Lnear Dscrmnant Analyss, Aeberar, 994), RDA, LOOC an KLIM on COIL- obect ata (ene, 996) COIL- s a atabase of gray-scale mages of obects e obects were place on a motorze turntable aganst a blac bacgroun e turntable was rotate troug 36 egrees to vary obect pose wt respect to a fx camera Images of te obects were taen at pose ntervals of 5 egrees, wc correspons to 7 mages per obect e total number of mages s 44 an te sze of eac mage s 8 8 In te experment, te regularzaton parameter of KLIM s estmate by trace( ) / e parameter matrx W of KLIM_L s estmate by Eq () In RDA, te values of an are sample n a coarse gr, (, 5, 5, 75, ), resultng n 5 ata ponts In LOOC, te four parameters are taen accorng to te table n (Hoffbec, 996) Sx mages are ranomly selecte as tranng samples from eac class to estmate te mean an covarance matrx An te remanng mages are employe as testng samples to verfy te classfcaton accuracy Snce te menson of mage ata s very g of 8 8, PCA s aopte ere to reuce te ata menson Experments are performe wt fve fferent numbers of mensons Eac experment runs 5 tmes, an te mean an stanar evaton of classfcaton accuracy are reporte as results e results of experment are sown n table table In te experment, te classfcaton accuracy of KLIM_L s te best among te fve compare metos, wle te classfcaton accuracy of KLIM s te secon best e classfcaton accuracy of LOOC s te worst among te compare metos except n menson 8, were te classfcaton accuracy of LOOC s ger tan tat of LDA Conserng te tme cost of regularzaton parameters estmatng, KLIM_L nees a lttle more tme to estmate te regularzaton parameters tan KLIM nees, wle RDA an LOOC nee muc more tme tan KLIM_L nees e expermental results are consstent wt te teoretcal analyss 5 COCLUSIOS In ts paper, te KLIM_L covarance matrx estmaton s erve base on MDL prncple for te small sample problem wt g menson Uner te framewor of MDL prncple, multregularzaton parameters are estmate smply an rectly by pont estmaton wc s approxmate by two-orer aylor expanson KLIM_L s a generalzaton of KLIM Wt te KL nformaton measure, total samples can be use to estmate te regularzaton parameters n KLIM_L, mang t less computaton-expensve tan usng leave-one-out cross-valaton meto n RDA an LOOC able : Mean classfcaton accuracy on COIL- obect atabase classfer LDA RDA LOOC KLIM KLIM_L 8 86(6) 87() 8(8) 877(8) 9() 7 836() 864(8) 8(8) 87(7) 879(8) 6 85() 868() 88(3) 877(4) 88(6) 5 85(5) 863(4) 83(9) 875() 878(3) 4 869(7) 869(7) 86(3) 876(3) 876(6) able : me cost (n secons) of estmatng regularzaton parameters on COIL- obect atabase classfer RDA LOOC KLIM KLIM_L e-5 866e e-5 788e e e e e e e-5 6

6 MULI-REGULARIZAIO PARAMEERS ESIMAIO FOR GAUSSIA MIXURE CLASSIFIER BASED O MDL PRICIPLE KLIM_L estmator aceves ger classfcaton accuracy tan LDA, RDA, LOOC an KLIM estmators on COIL- ata set In te future wor, te ernel meto combne wt tese regularzaton scrmnant metos wll be stue for small sample problem wt g menson an te selecton of ernel parameters wll be nvestgate uner some crteron ACKOWLEDGEMES e researc wor escrbe n ts paper was fully supporte by te grants from te atonal atural Scence Founaton of Cna (Proect o 98, 69353) Prof Guo s te autor to wom te corresponence soul be aresse, s e-mal aress s pguo@eeeorg small sample set case base on MDL prncple wt applcaton to spectrum recognton, Pattern Recognton, Vol 4, 84~854 Rener, R A, Waler, H F, 984 Mxture enstes, maxmum leloo an te EM algortm, SIAM Rev 6, Aeberar, S, Coomans, e Vel, D, O, 994 Comparatve analyss of statstcal pattern recognton metos n g mensonal settngs, Pattern Recognton 7 (8), Rssanen, J, 978 Moelng by sortest ata escrpton, Automatca 4, Barron, A, Rssanen, J, Yu, B, 998 e mnmum escrpton lengt prncple n cong an moelng, IEEE rans Inform eory 44 (6), Kullbac, S, 959 Informaton eory an Statstcs, Wley, ew Yor ene, S A, ayar, S K an Murase, H, 996 Columba Obect Image lbrary(coil-) ecncal report CUCS-5-96 REFERECES Bsop, C M, 7 Pattern recognton an macne learnng, Sprnger-Verlag ew Yor, Inc Secaucus, J, USA Evertt, B S, Han, D, 98 Fnte Mxture Dstrbutons, Capman an Hall, Lonon Freman, J H, 989 Regularze scrmnant analyss, Journal of te Amercan Statstcal Assocaton, vol 84, no 45, Hoffbec, J P an Langrebe, D A, 996 Covarance matrx estmaton an classfcaton wt lmte tranng ata, IEEE ransactons on Pattern Analyss an Macne Intellgence, vol 8, no 7, Scafer, J an Strmmer, K, 5 A srnage approac to large-scale covarance matrx estmaton an mplcatons for functonal genomcs, Statstcal Applcatons n Genetcs an Molecular Bology, vol 4, no Srvastava, S, Gupta, M R, Frgy, B A, 7 Bayesan quaratc scrmnant analyss, J Mac Learnng Res 8, Bcel, P J an Levna, E, 8 Regularze estmaton of large covarance matrces, Annals of Statstcs, vol 36, no, 99 7 Freman, J, Haste,, an bsran, R, 8 Sparse nverse covarance estmaton wt te grapcal lasso, Bostatstcs, vol 9, no 3, Cao, G, Bacega, L R, Bouman, C A, e Sparse Matrx ransform for Covarance Estmaton an Analyss of Hg Dmensonal Sgnals IEEE ransactons on Image Processng, Volume, Issue 3, Rvals, I, Personnaz, L, 999 On cross valaton for moel selecton, eural Comput, Guo, P, Ja, Y, an Lyu, M R, 8 A stuy of regularze Gaussan classfer n g-menson 7