Boosting Margin Based Distance Functions for Clustering

Transcription

1 Boostn Mrn Bs Dstn Funtons or Clustrn Tomr Hrtz Aron Br-Hlll Dpn Wnsll Sool o Computr Sn n Ennrn n t Cntr or Nurl Computton, T Hrw Unvrsty o Jruslm, Jruslm, Isrl Astrt T prormn o rp s lustrn mtos rtlly pns on t qulty o t stn unton, us to omput smlrts twn prs o norn nos. In ts ppr w lrn stn untons y trnn nry lssrs wt mrns. T lssrs r n ovr t prout sp o prs o ponts n r trn to stnus wtr two ponts om rom t sm lss or not. T sn mrn s us s t stn vlu. Our mn ontruton s stn lrnn mto (DstBoost), w omns oostn ypotss ovr t prout sp wt wk lrnr s on prttonn t ornl tur sp. E wk ypotss s Gussn mxtur mol omput usn sm-suprvs onstrn EM lortm, w s trn usn ot unll n ll t. W lso onsr SVM n son trs oostn s mrn s lssrs n t prout sp. W xprmntlly ompr t mrn s stn untons wt otr xstn mtr lrnn mtos, n wt xstn tnqus or t rt norporton o onstrnts nto vrous lustrn lortms. Clustrn prormn s msur on som nmrk tss rom t UCI rpostory, smpl rom t MNIST ts, n t st o olor ms o nmls. In most ss t DstBoost lortm snntly n roustly outprorm ts ompttors. 1. Introuton Grp s lustrn mtos v n wly n sussully us n mny omns su s omputr vson, onormts n xplortory t nlyss. Ts tory spns w rn o lortms, rom lssl lomr- Prlmnry work. Unr rvw y t Intrntonl Conrn on Mn Lrnn (ICML). Do not strut. tv mtos su s vr lnk (Du t l., 2001), to t rntly vlop n mor sopstt sptrl mtos (S & Mlk, 2000) n stost ormultons (Bltt t l., 1997; Glyu t l., 2001). T ntl rprsntton n ll ts mtos s mtrx (or rp) o stns twn ll prs o tponts. T omputton o ts stn mtrx s onsr prprossn stp, n typlly on uss som norm on t tur sp (or rlt vrnt). Dspt t mportnt rns twn t vrous rp-s lustrn lortms, t s wly knowl tt lustrn prormn rtlly pns on t qulty o t stn unton us. Otn t qulty o t stn unton s mor mportnt tn t sps o t lustrn lortm. In ts ppr w ous on t quston o ow to lrn oo stn unton, w wll l to mprov lustrn. Our mn ontruton s DstBoost - novl sm-suprvs lortm or lrnn stn untons. W onsr sm-suprvs lustrn snro n w t t s umnt y som sprs s normton, n t orm o quvln onstrnts. Equvln onstrnts r rltons twn prs o t ponts, w nt wtr t ponts lon to t sm tory or not. W trm onstrnt postv wn t ponts r known to rom t sm lss, n ntv otrws. Su onstrnts rry lss normton tn xplt lls on t ornl tponts, sn lrly quvln onstrnts n otn rom xplt lls ut not v vrs. Mor mportntly, t s n sust tt n som ss quvln onstrnts r sr to otn, splly wn t ts s vry lr n ontns lr numr o tors wtout pr-n nms (Hrtz t l., 2003). In rnt yrs tr s n rown ntrst n sm suprvs lustrn snros, ln to two rnt (n rlt) lns o rsr. In t rst, t onstrnts r norport rtly nto t lustrn lortm, lmtn t lustrn solutons onsr to tos tt omply wt t vn onstrnts. Exmpls r t onstrn omplt lnk lortm (Kln t l., 2002), onstrn K-mns (Wst t l., 2001) n on-

2 strn EM o Gussn mxtur (Sntl t l., 2003). T son ln o rsr, to w ts work lons, uss t onstrnts to lrn n normtv stn unton (pror to lustrn). Most o t work n ts r s ous on t lrnn o Mlnos stn untons o t orm (Sntl t l., 2002; Xn t l., 2002). In ts pprs t prmtr Mlnos mtr ws us n omnton wt som sutl prmtr lustrn lortm, su s K-mns or EM o mxtur o Gussns. In ontrst, w vlop n ts ppr mto tt lrns non-prmtr stn unton, w n mor nturlly us n non-prmtr rp s lustrn. Mor ormlly, lt not t ornl t sp, n ssum tt t t s smpl rom srt lls. Our ol s to lrn stn unton!. 1 Our ky osrvton s tt w n lrn su unton, y posn rlt nry lsston prolm ovr t prout sp "#, n solvn t usn mrn s lsston tnqus. T nry prolm s t prolm o stnusn twn prs o ponts tt lon to t sm lss n prs o ponts tt lon to rnt lsss. 2 T trnn t nlu st o quvln onstrnts, w n ormlly rr s nry lls on ponts n $%. I w ll prs o ponts rom t sm lss y n prs o ponts lonn to rnt lsss y, w n ntrprt t lssr s mrn s t rqur stn unton. Hvn ru stn lrnn to nry lsston wt mrns, w n now ttmpt to solv ts prolm usn stnr powrul mrn s lssrs. W v xplor ot support vtor mns (SVM s) n oostn lortms. Howvr, xprmnts wt svrl SVM vrnts n son trs(c4.5) oostn v l us to ronz tt t sp lsston prolm w r ntrst n s som unqu turs w rqur spl trtmnt: 1. T prout sp nry unton w ws to lrn s som unqu strutur w my l to unnturl prttons o t sp twn t lls. T onpt w ws to lrn s n ntor o n quvln rlton ovr t ornl sp. Tus t proprts o trnstvty n symmtry o t rlton pl omtrl onstrnts on t nry ypotss. Ovously, trtonl mls o ypotss, su s lnr sprtors or son trs, r not lmt to 1 Not tt ts unton s not nssrly mtr, s t trnl nqulty my not ol. 2 Not tt ts prolm s losly rlt to t mult lss lsston prolm: w n orrtly nrt nry prtton o t t n prout sp, w mpltly n mult-lss lssr n t ornl vtor sp &. quvln rlton ntors, n t s not sy to nor ts onstrnts wn su lssrs r us. 2. In t lrnn sttn w v sr ov, w r prov wt ' tponts n n wt sprs st o quvln onstrnts (or lls n prout sp) ovr som prs o ponts n our t. W ssum tt t numr o quvln onstrnts prov s mu smllr tn t totl numr o quvln onstrnts () *',+-, n s o orr () '. W tror v ss to lr mounts o unll t, n n sm-suprvs lrnn sms lk n ttrtv opton. Howvr, lssl nry lssrs lk SVM n oostn mtos r trn usn ll t only. Ts onsrtons l us to vlop t DstBoost lortm, w s our mn ontruton n ts ppr. DstBoost s stn lrnn lortm w ttmpts to rss t ssus suss ov. It lrns stn unton w s s on oostn nry lssrs wt onn ntrvl n prout sp, usn wk lrnr tt lrns n t ornl tur sp (n not n prout sp). W sust oostn sm tt norports unll t ponts. Ts unll ponts prov nsty pror, n tr wts rply y urn t oostn pross. T wk lrnr w us s s on onstrn Exptton Mxmzton (EM) lortm, w omputs Gussn mxtur mol, n n provs prtton o t ornl sp. T onstrn EM prour uss unll t n quvln onstrnts to n Gussn mxtur tt ompls wt tm. A wk prout sp ypotss s tn orm s t quvln rlton o t omput prtton. W v xprmnt wt DstBoost n onut svrl mprl omprsons o ntrst. T rst s omprson o DstBoost to otr mrn s stn untons otn usn t mor trtonl lortms o SVM n son tr oostn. Anotr omprson s twn DstBoost n prvously sust stn lrnn lortms w r s on Mlnos mtr stmton. Fnlly, lustrn usn t stn unton lrnt y DstBoost s ompr to prvously sust mtos o norportn quvln onstrnts rtly nto lustrn lortms. Durn t omprtv ssssmnt DstBoost ws vlut wt svrl lomrtv lustrn lortms n wt rnt mounts o quvln onstrnts normton. W us svrl tsts rom t UCI rpostory (Blk & Mrz, 1998), A smpl rom t MNIST tst (LCun t l., 1998), n tst o nturl ms otn rom ommrl m CD. In most o our xprmnts t DstBoost mto outprorm ts ompttors.

3 2. Boostn ornl sp prttons usn DstBoost T DstBoost lortm uls stn untons s on t wt mjorty vot o st o ornl sp sot prttons. T wk lrnr s tsk n ts rmwork s to n plusl prttons o t sp, w omply wt t vn quvln onstrnts. In ts tsk, t unll t n o onsrl lp, s t llows to n pror on wt r plusl prttons. In orr to norport t unll t nto t oostn pross, w umnt t Aoost wt onn ntrvls prsnt n (Spr & Snr, 1999). T tls o ts umntton r prsnt n Ston 2.1. T tls o t wk lrnr w us r prsnt n Ston Sm suprvs oostn n prout sp Our oostn sm s n xtnson o t Aoost lortm wt onn ntrvls (Spr & Snr, 1999; Spr t l., 1997) to nl unsuprvs t ponts. As n Aoost, w us t oostn pross to mxmz t mrns o t ll ponts. T unll ponts only prov yn nsty pror or t wk lrnr. T lortm w us s skt n F. 1. Gvn prtlly ll tst wr, t lortm srs or ypotss w mnmzs t ollown loss unton:! "$#%'&(*),+ * - (1) Not tt t unll ponts o not ontrut to t mnmzton ojtv (1). Rtr, t oostn roun ty r vn to t wk lrnr n supply t wt som (opully usul) normton rrn t omn s nsty. T unll ponts tvly onstrn t sr sp urn t wk lrnr stmton, vn prorty to ypotss w ot omply wt t prws onstrnts n wt t nsty normton. Sn t wk lrnr s tsk oms rr n ltr oostn rouns, t oostn lortm slowly rus t wt o t unll ponts vn to t wk lrnr. Ts s ompls n stp 4 o t lortm (s F. 1). In prout sp tr r () '+- unll ponts, w orrspon to ll t possl prs o ornl ponts, n t numr o wts s tror () *',+. Howvr, t upt ruls or t wt o unll pont r ntl, n so ll t unll ponts n sr t sm wt. Hn t numr o upts w tvly o n roun s proportonl to t numr o ll prs only. T wt o t unll prs s urnt to Alortm 1 Boostn wt unll t Gvn / *2 43$ 5 6 Intlz7 8 :9:;<8= >.. '; For? >.. '@ 1. Trn wk lrnr usn struton7ba 2. Gt wk ypotss A C3! wt D A 1 7EA 8 A 8 GF. I no su ypotss n oun, trmnt t loop n st@hi?. 3. Coos A= NMPO +KJ0L #QO 4. Upt: 7RA M! 8 TS 7RAU8 (*),+ A A 5 7RAU8 (*),+ A V 5. Normlz: 7EA M 8 KH7RA M! 8 W9>XYA M! wrx=a M 1 7EA MU8 6. Output t nl ypotss A A A y t lst s st s t wt o ny ll pr. Ts mmtly ollows rom t upt rul n stp 4 o t lortm (F. 1), s unll pr s trt s ll pr wt mxml mrn o 1. W not n pssn tt t s possl to norport unll t nto t oostn pross tsl, s s n sust n ( Al Bu t l., 2002; Grnvlt t l., 2001). In ts work t mrn onpt ws xtn to unll t ponts. T mrn or su pont s postv numr rlt to t onn t ypotss s n lssyn ts pont. T lortm tn trs to mnmz t totl (ot ll n unll) mrn ost. T prolm wt ts rmwork s tt ypotss n vry rtn out t lsston o unll ponts, n n v low mrn osts, vn wn t lsss ts ponts norrtly. In t sm suprvs lustrn ontxt t totl mrn ost my omnt y t mrns o unonstrn pont prs, n n mnmzn t osn t nssrly l to ypotss tt omply wt t onstrnts. In, w v mprlly tst som vrnts o ts lortms n oun tt ty l to nror prormn Mxturs o Gussns s wk ypotss T wk lrnr n DstBoost s s on t onstrn EM lortm prsnt y (Sntl t l., 2003). Ts l-

4 X 1 $ ortm lrns mxtur o Gussns ovr t ornl t sp, usn unll t n st o postv n ntv onstrnts. Blow w rly rvw t s lortm, n tn sow ow t n mo to norport wts on smpl t ponts. W lso sr ow to trnslt t oostn wts rom prout sp ponts to ornl t ponts, n ow to xtrt prout sp ypotss rom t sot prtton oun y t EM lortm. A Gussn mxtur mol (GMM) s prmtr sttstl mol w ssums tt t t ornts rom wt sum o svrl Gussn sours. Mor ormlly, GMM s vn y, wr nots t wt o Gussn, ts rsptv prmtrs, n nots t numr o Gussn sours n t GMM. EM s wly us mto or stmtn t prmtr st o t mol ( ) usn unll t (Dmpstr t l., 1977). In t onstrn EM lortm quvln onstrnts r ntrou nto t E (Exptton) stp, su tt t xptton s tkn only ovr ssnmnts w omply wt t vn onstrnts (nst o summn ovr ll possl ssnmnts o t ponts to sours). Assum w r vn st o unll... smpl ponts3 < N, n st o prws onstrnts ovr ts ponts. Dnot t nx prs o postvly onstrn ponts y + 6 n t nx prs o ntvly onstrn ponts y U; '; +. T GMM mol ontns st o srt n vrls, wr t Gussn sour o pont Q s trmn y t n vrl. T onstrn EM lortm ssums t ollown jont struton o t osrvls 3 n t ns : 3 = (2) 0!! T lortm sks to mxmz t t lkloo, w s t mrnl struton o (2) wt rspt to. T quvln onstrnts rt omplx pnns twn t n vrls o rnt t ponts. Howvr, t jont struton n xprss usn Mrkov ntwork, s sn n F. 1. In t E stp o t lortm t prolts 3 r omput y pplyn stnr nrn lortm to t ntwork. Su nrn s sl t numr o ntv onstrnts s () *',, n t ntwork s sprsly onnt. T mol prmtrs r tn upt s on t omput prolts. T upt o t Gussn prmtrs n on n los orm, usn ruls smlr to t stnr EM upt ruls. T upt o t lustr wts! s mor omplt, sn ts prmtrs ppr n t normlzton onstnt X n (2), n t soluton s oun Fur 1. A Mrkov ntwork rprsntton o t onstrn mxtur sttn. E osrvl t no s srt n no s ts nstor. Postvly onstrn nos v t sm n no s tr nstor. Ntv onstrnts r xprss usn s twn t n nos o ntvly onstrn ponts.hr ponts 2,3,4 r onstrn to totr, n pont 1 s onstrn to rom rnt lss. wt rnt snt prour. T lortm ns lol mxmum o t lkloo, ut t prtton oun s not urnt to stsy ny sp onstrnt. Howvr, sn t oostn prour nrss t wts o ponts w lon to unsts quvln onstrnts, t s most lkly tt ny onstrnt wll sts n on or mor prttons. W v norport wts nto t onstrn EM prour orn to t ollown smnts: T lortm s prsnt wt vrtul smpl o sz '#". A trnn pont wt wt $ pprs $ '%" tms n ts smpl. All t rpt tokns o t sm pont r onsr to postvly onstrn, n r tror ssn to t sm sour n vry vluton n t E stp. In ll o our xprmnts w v st ' " to t tul smpl sz. Wl t wk lrnr pts struton ovr t ornl sp ponts, t oostn pross sr n 2.1 nrts struton ovr t smpl prout sp n roun. T prout sp struton s onvrt to struton ovr t smpl ponts y smpl mrnlzton. Splly, not y $ t wt o pru8 '& ; t wt $)( o pont s n to $ ( (3) In roun, t mxtur mol omput y t onstrn EM s us to ul nry unton ovr t prout sp n onn msur. W rst rv prtton o t t rom t Mxmum A Postror (MAP) ssnmnt o ponts. A nry prout sp ypotss s tn n y vn t vlu to prs o ponts rom t sm Gussn sour, n to prs o ponts rom rnt sours. Ts vlu trmns t sn o t ypotss output. Ts sttn urtr supports nturl onn msur - t prolty o t pr s

5 ) ) MAP ssnmnt w s: H8 + I8 + wr + r t n vrls tt to t two ponts. T wk ypotss output s t sn onn msur n!, n so t wk ypotss n vw s wk stn unton. 3. Lrnn n t prout sp usn trtonl lssrs W v tr to solv t stn lrnn prolm ovr t prout sp usn two mor trtonl mrn s lssrs. T rst s support vtor mn, tt trs to n lnr sprtor twn t t xmpls n mnsonl tur sp. T son s t ABoost lortm, wr t wk ypotss r son trs lrnt usn t C4.5 lortm. Bot lortms to sltly pt to t tsk o prout sp lrnn, n w v mprlly tst possl pttons usn t sts rom t UCI rpostory. Splly, w to l wt t ollown tnl ssus: Prout sp rprsntton: A pr o ornl sp ponts must onvrt nto snl pont, w rprsnts ts pr n t prout sp. T smplst rprsntton s t ontnton o t two ponts. Anotr ntutv rprsntton s t ontnton o t sum n rn vtors o t two ponts. Our mprl tsts nt tt wl SVM works ttr wt t rst rprsntton, t C4.5 oostn vs ts st prormn wt t sum n rn rprsntton. Enorn symmtry: I w wnt to lrn symmtr stn unton stsyn C, w v to xpltly nor ts proprty. Wt t rst rprsntton ts n on smply y ouln t numr o trnn ponts, ntroun onstrn pr tw: s t pont! n s t pont!. In ts sttn t SVM lortm ns t lol optmum o symmtr Lrnn n t soluton s urnt to symmtr. Wt t son rprsntton w oun tt moyn t rprsntton to symmtrlly nvrnt v t st rsults. Splly, w rprsnt pr o ponts /8 ;!, wr usn t vtor Q r t rst oornts o t ponts. W onsr two lnr prprossn trnsormtons o t ornl t or rtn t prout sp ponts: t wtnn trnsormton, n t RCA trnsormton (Br-Hll t l., 2003) w uss postv quvln onstrnts. In nrl w oun tt pr-prossn wt RCA ws most nl or ot t SVM n C4.5 oostn lortms. Prmtr tunn: or t SVM w us t polynoml krnl o orr 4, n tr-o onstnt o 1 twn rror n mrn. T oostn lortm ws run or rouns (pnn on t tst), n t son trs wr ult wt stoppn rtron o trn rror smllr tn 0.05 n l. T lustrn prormn otn usn ts two vrnts s ompr to DstBoost n ston 4. T sn ssus mnton ov wr s on t prormn ovr t UCI tsts, n t sttns rmn x or t rst o t xprmnts. 4. Exprmntl Rsults W ompr our DstBoost lortm wt otr tnqus or sm-suprvs lustrn usn quvln onstrnts. W us ot stn lrnn tnqus, nlun our two smplr vrnts or lrnn n prout sp (SVM n oostn son trs), n onstrn lustrn tnqus. W n y ntroun our xprmntl stup n t vlut mtos. Tn w prsnt t rsults o ll ts mtos on svrl tsts rom t UCI rpostory, sust o t MNIST lttr ronton tst, n n nml m ts Exprmntl stup Gtrn quvln onstrnts: Follown (Hrtz t l., 2003), w smult strut lrnn snro, wr lls r prov y numr o unoornt npnnt trs. Aornly, w rnomly os smll susts o t ponts rom t tst n prtton o t susts nto quvln lsss. T onstrnts otn rom ll t susts r tr n us y t vrous lortms. T sz o sust n ts xprmnts ws osn to, wr s t numr o lsss n t t. In xprmnt w us susts, n t mount o prtl normton ws ontroll y t onstrnt nx ; ts nx msurs t mount o ponts w prtpt n t lst on onstrnt. In our xprmnts w us.. Howvr, t s mportnt to not tt t numr o quvln onstrnts tus prov typlly nlus only smll sust o ll possl prs o tponts, w s ',+-. Evlut Mtos: w ompr t lustrn prormn o t ollown tnqus: 1. Our propos oostn lortm (DstBoost).

6 2. Mlnos stn lrnn wt Rlvnt Componnt Anlyss (RCA) (Br-Hll t l., 2003). 3. Mlnos stn lrnn wt non-lnr optmzton (Xn) (Xn t l., 2002). 4. Mrn s stn lrnn usn SVM s prout sp lrnr (SVM) (sr n Ston 3). 5. Mrn s stn lrnn usn prout sp son trs oostn (DToost). 6. Constrn EM o Gussn Mxtur Mol (Constrn EM) (Sntl t l., 2003). 7. Constrn Complt Lnk (Constrn Complt Lnk) (Kln t l., 2002). 8. Constrn K-mns (COP K-mns) (Wst t l., 2001). Mtos 1-5 omput stn unton, n ty r vlut y pplyn stnr lomrtv lustrn lortm (Wr) to t stn rp ty nu. Mtos 6-8 norport quvln onstrnts rtly nto t lustrn pross. All mtos wr vlut y lustrn t t n msurn t sor n s (4) wr nots prson n nots rll. For t stn lrnn tnqus w lso sow umultv nor purty urvs. Cumultv nor purty msurs t prnt o orrt nors up to t nor, vr ovr ll t tponts. In xprmnt w vr t rsults ovr 50 or mor rnt quvln onstrnt rlztons. Bot DstBoost n t son tr oostn lortms wr run or onstnt numr o oostn (pnn on t tst). In rlzton ll t lortms wr vn t xt sm quvln onstrnts. Dmnsonlty ruton: t onstrn LDA lortm Som o t tsts rs n mnsonl sp, w must ru n orr to prorm prmtr stmton rom trnn t. W us two mtos or mnsonlty ruton: stnr Prnpl Componnts Anlyss (PCA), n onstrn Lnr Dsrmnnt Anlyss (LDA) lortm w s s on quvln onstrnts. Clssl LDA (lso ll FDA, (Fukun, 1990)) omputs projton rtons tt mnmz t wtn-lss sttr n mxmz t twn-lss sttr. Mor ormlly, vn ll tst wr -t >.. n G, LDA s vn y t mtrx tt mxmzs QA wr A nots t totl sttr mtrx ( # nots t wtn- s t t s mprl mn) n lss sttr mtrx ( s t mprl mn o t & -t lss). Sn n our sm-suprvs lrnn snro w v ss to quvln onstrnts nst o lls, w n wrt own onstrn LDA lortm. Tus w stmt t wtn lss sttr mtrx usn postv quvln onstrnts nst o lls. Splly, vn st o postv quvln onstrnts, w us trnstv losur ovr ts st to otn smll susts o ponts tt r known to lon to t sm lss. Dnot ts susts y #%, wr sust s ompos o vrl numr o t ponts - + >.. 1. W us ts susts to stmt s ollows # 0! (5) (6) wr r nots t mn o sust Rsults on UCI tsts W slt svrl tsts rom t UCI t rpostory n us t xprmntl stup ov to vlut t vrous mtos. F. 2 sows lustrn sor plots or svrl t sts usn Wr s lomrtv lustrn lortm. Clrly DstBoost vs snnt mprovmnts ovr Mlnos s stn msurs n otr prout sp lrnn mtos. Comprn DstBoost to mtos w norport onstrnts rtly, lrly t only tru ompttor o DstBoost s ts own wk lrnr, t onstrn EM lortm. Stll, n t vst mjorty o ss DstBoost vs n tonl snnt mprovmnt ovr t EM Rsults on t MNIST lttr ronton tst W ompr ll lustrn mtos on sust o t MNIST lttr ronton tst (LCun t l., 1998). W rnomly slt trnn smpls ( rom o t lsss). T ornl t mnson ws!, w ws projt y stnr PCA to t rst prnpl mnsons. W tn urtr projt t t usn t onstrn LDA lortm to! mnsons. Clustrn n nor purty plots r prsnt on t lt s o F 3. T lustrn prormn o t DstBoost lortm s snntly ttr tn t otr mtos. T

7 protn onospr 1 ln oston Fur 2. Clustrn sor ovr 4 t sts rom t UCI rpostory usn Wr s lustrn lortm. Mtos sown r: () Euln, () RCA, () onstrn EM, () SVM, () DToost, () DstBoost, () Xn, () Constrn Complt Lnk, () Constrn K-mns. T rsults wr vr ovr 100 rlztons o onstrnts, n 1-st rror rs r sown. T onstrnt nx ws n ll ss. umultv purty urvs sust tt ts suss my rlt to t slowr y o t nor purty sors or DstBoost Rsults on Anml m tst W rt n m ts w ontn ms o nmls tkn rom ommrl m CD, n tr to lustr tm s on olor turs. T lustrn tsk n ts s s mu rr tn n t prvous ppltons. T ts ontn 10 lsss wt totl o 565 ms. F. 3 sows w xmpls o ms rom t ts. T ornl ms wr vly omprss jp ms. T ms wr rprsnt usn Color Corn Vtors (Pss t l., 1996) (CCV s). Ts rprsntton xtns t olor storm rprsntton, y pturn som ru sptl proprts o t olor struton n n m. Splly, n CCV vtor storm n s v nto two ns, rprsntn t numr o Cornt n Non-Cornt pxls rom olor. Cornt pxls r pxls wos noroo ontns mor tn nors w v t sm olor. W rprsnt t ms n HSV olor sp, quntz t ms to R olor ns, n omput t CCV o m -!. 3 mnsonl vtor - usn In orr to ru t mnson o our t, w rst rmov ll zro mnsons n tn us t rst PCA mnsons, ollow y Constrn LDA to urtr ru t mnson o t t to!. T lustrn rsults n nor purty rps r prsnt on t rt s o F 3. 4 T ulty o t tsk s wll rlt n t low lustrn sors o ll t mtos. 3 T stnr stn msur us on CCV turs s C-squr stn (lso ommonly us to msur stn twn storms). W lso tr to lustr t t usn t C-squr stns, n t sor otn ws. 4 On ts tst t COP k-mns lortm only onvr o ts runs. on Howvr, DstBoost stll outprorms ts ompttors, s t n ll prvous xmpls. 5. Dsusson In ts ppr, w v sr DstBoost - novl lortm w lrns stn untons tt nn lustrn prormn usn sprs s normton. Our xtnsv omprsons sow t vnt o our mto ovr mny omptn mtos or lrnn stn untons n or lustrn usn quvln onstrnts. Anotr pplton w w v not xplor r, s nrst nor lsston. Nrst nor lsston lso rtlly pns on t stn unton twn tponts; our op s tt stn untons lrn rom quvln onstrnts n lso us or mprovn nrst nor lsston. Rrns Br-Hll, A., Hrtz, T., Sntl, N., & Wnsll, D. (2003). Lrnn stn untons usn quvln rltons. Blk, C., & Mrz, C. (1998). UCI rpostory o mn lrnn tss. Bltt, M., Wsmn, S., & Domny, E. (1997). Dt lustrn usn mol rnulr mnt. Nurl Computton, 9, Al Bu, F., Grnvlt, Y., & Amros, C. (2002). Sm-suprvs mrnoost. Dmpstr, A. P., Lr, N. M., & Run, D. B. (1977). Mxmum lkloo rom nomplt t v t EM lortm. JRSSB, 39, Du, R. O., Hrt, P. E., & Stork, D. G. (2001). Pttrn Clsston. Jon Wly n Sons In. Fukun, K. (1990). Sttstl pttrn ronton. Sn Do: Am Prss. 2n ton.

8 0.9 Mnst CCV Mnst 0.9 CCV % o orrt nors 0.9 % o orrt nors Numr o nors Numr o nors Fur 3. Top Clustrn rsults usn Wr s lortm on sust o t MNIST tst (500 tponts, 10 lsss) n on t nml olor m ts (565 ms, 10 lsss). Bottom: Cumultv nor purty rps on t sm tsts. Mtos sown r: () Euln, () RCA, () onstrn EM, () SVM, () DToost, () DstBoost, () Xn, () Constrn Complt Lnk, () Constrn K-mns. Rsults wr vr ovr 50 rlztons. T onstrnt nx s 1 n ll ss. Glyu, Y., Wnsll, D., & Wrmn., M. (2001). Sl ornzton n vson: stost lustrn or m smntton, prptul roupn, n m ts ornzton. Grnvlt, Y., Al Bu, F., & Amros, C. (2001). Boostn mxtur mols or sm suprvs lrnn. Hrtz, T., Br-Hlll, A., Sntl, N., & Wnsll, D. (2003). Ennn m n vo rtrvl: Lrnn v quvln onstrnts. IEEE Con. on Computr Vson n Pttrn Ronton, Mson WI, Jun Kln, D., Kmvr, S., & Mnnn, C. (2002). From nstn-lvl onstrnts to sp-lvl onstrnts: Mkn t most o pror knowl n t lustrn. LCun, Y., Bottou, L., Bno, Y., & Hnr, P. (1998). Grnt-s lrnn ppl to oumnt ronton. Prons o t IEEE, 86, Pss, G., Z, R., & Mllr, J. (1996). Comprn ms usn olor orn vtors. ACM Multm (pp ). Spr, R. E., Frun, Y., Brtltt, P., & L, W. S. (1997). Boostn t mrn: nw xplnton or t tvnss o votn mtos. Pro. 14t Intrntonl Conrn on Mn Lrnn (pp ). Morn Kumnn. Spr, R. E., & Snr, Y. (1999). Improv oostn usn onn-rt prtons. Mn Lrnn, 37, Sntl, N., Hrtz, T., Br-Hll, A., & Wnsll, D. (2003). Computn ussn mxtur mols wt EM usn quvln onstrnts. Sntl, N., Hrtz, T., Wnsll, D., & Pvl, M. (2002). Ajustmnt lrnn n rlvnt omponnt nlyss. Computr Vson - ECCV. S, J., & Mlk, J. (2000). Normlz uts n m smntton. IEEE Trnstons on Pttrn Anlyss n Mn Intlln, 22, Wst, K., Cr, C., Rors, S., & Srol, S. (2001). Constrn K-mns lustrn wt kroun knowl. Pro. 18t Intrntonl Con. on Mn Lrnn (pp ). Morn Kumnn, Sn Frnso, CA. Xn, E., N, A., Jorn, M., & Russll, S. (2002). Dstn mtr lrnn wt pplton to lustrn wt s-normton. Avns n Nurl Inormton Prossn Systms. T MIT Prss.