Research Article ISSN:

Transcription

1 Research Artcle [Dave, 1(2): Aprl, 2012] IJESRT INTERNATIONAL JOURNAL OF ENGINEERING SCIENCES & RESEARCH TECHNOLOGY Emprcal Study On Error Correctng Output Code Based On Multclass Classfcaton Ms. Devangn Dave 1, Prof. M. Samvatsar 2 3, Prof. P. K. Bhanoda devangpdave@yahoo.co.n Abstract A common way to address a mult-class classfcaton problem s to desgn a model that conssts of hand pcked bnary classfers and to combne them so as to solve the problem. Error-Correctng Output Codes () s one such framework that deals wth mult-class classfcaton problems. Recent works n the C doman has shown promsng results demonstratng mproved performance. Therefore, framework s a powerful tool to deal wth mult-class classfcaton problems. The error correctng ablty mproves and enhances the generalzaton ablty of the base classfers. Ths paper ntroduces state-of-the-art codng (one-versus-one, one-versus-all, dense random, sparse random, D, forest-, and -ONE) and decodng desgns (hammng, Eucldean, nverse hammng, laplacan, β-densty, attenuated, loss-based, probablstc kernel-based, and loss weghted) perspectves along wth emprcal study of C followng comparson of varous methods n the above context. Towards the end, our paper consoldates detals relatng to comparson of varous classfcaton methods wth Error Correctng Output Code method avalable n wake, after carryng out experments wth weak tool as a fnal supplement to our studes. Index Terms Codng, Decodng, Error Correctng Output Codes, Multclass Classfcaton.. Introducton The task of supervsed machne learnng can be seen as the problem of fndng an unknownn functon C(x) gven the tranng set of example pars < x ; C (x ) >. C(x) s usually a set of dscrete labels. For example, n face detecton, C(x) s a bnary functon C(x) {face, nonface}, n optcal dgt recognton C(x) {0,, 9}. In order to address the bnary classfcaton task many technques and algorthms have been proposed: decson trees, neural networks, large margn classfcaton technques, etc. Some of those methods can be easly extended to multclass problems. However, some other powerful and popular classfers, such as AdaBoost [4] and Support Vector machnes [3], do not extend to multclass easly. In those stuatons, the usual way to proceed s to reduce the complexty of the multclass problem nto multple smpler bnary classfcaton problems. There are many dfferent approaches for reducng multclass to bnary classfcaton problems. The smplest approach consders the comparson between each class aganst all the others. Ths produces N c bnary problems, where N c s the number of classes. Other researchers suggested the comparson of all possble pars of classes [5], resultng n an N c (N c - 1)/2 set of bnary problems. Detterch and Bakr [7] presented a general framework n whch the classfcaton s performed accordng to a set of bnary error correctng output codes (). In ths approach, the problem s transformed n n bnary classfcaton sub problems, where n s the error correctng output code length n ε {N c,, }. Then, the output of all classfers must be combned tradtonally usng Hammng dstance. The approach of Detterch and Bakr was mproved by Allwen et al. [6] by ntroducng an uncertanty value n the desgn and explorng alternatves for mxng the resultng outputs of the classfers. In partcular, they ntroduced loss-based decodng as a way of mergng the classfers. Recently, Passern et al. [2] proposed a new decodng functon that combnes the margns through an estmate of the class condtonal probabltes. strateges have been proven to be qute compettve wth/better than other multclass extensons of SVM and Adaboost [8], [9]. Although most of the mprovements n error correctng output codes have been made n the (C) Internatonal Journal of Engneerng Scences & Research Technology[52-60]

2 decodng process, lttle attenton has been pad to the desgn of the codes themselves. Crammer and Snger n [1] were the frst to report mprovements n the desgn of the codes. However, the results were rather pessmstc snce they proved that the problem of fndng the optmal dscrete codes s computatonally ntractable snce t s NP-complete. Error Correctng Output Codes Gven a set of N c classes, the bass of the framework conssts of desgnng a codeword for each of the classes. 1) These code words encode the membershp nformaton of each class for a gven bnary problem. 2) Arrangng the code words as rows of a matrx, we obtan a codng matrx M c, where M c {-1, 0, +1} Nc n, beng n the length of the code words codfyng each classes. 3) From the pont of vew of learnng, M c s constructed by consderng n bnary problems each one correspondng to a column of the matrx M c. 4) Each of these bnary problems splts the set of classes n two parttons (coded by +1 or -1 n M c accordng to ther class set membershp or 0 f the class s not consdered by the current bnary problem). 5) Then at the decodng step, applyng the n traned bnary classfers, a code s obtaned for each data pont n the test set. 6) Ths code s compared to the base code words of each class defned n the matrx M c, and the data pont s assgned to the class wth the closest codeword. Below fgure show codng desgn for a 4- class problem. Whte, black and grey postons corresponds to the symbols +1, -1 and 0, respectvely. Once the four bnary problems are learnt, at the decodng step a new test sample X s tested by the n classfers. Then the new codeword x={x 1,, xn} s compared wth the class code words {C 1,, C4}, classfyng the new sample by the class C whch codeword mnmzes the decodng measure. Codng Desgn Here the codng desgn covers the state-of-the art of codng strateges, manly dvded n two man groups: problem-ndependent approaches, whch do not take nto account the dstrbuton of the data to defne the codng matrx, and the problem-dependent desgns, where nformaton of the partcular doman s used to gude the codng desgn. Problem-Independent Codng Desgns One-versus-all (Rfkn and Klautau, 2004): N c dchotomzers are learnt for Nc classes, where each one splts one class from the rest of classes. One-versus-one (Nlsson, 1965): n = N c (N c 1)/2 dchotomzers are learnt for N c classes, splttng each possble par of classes. Dense Random (Allwen et al., 2002): n = 10 logn c dchotomzers are suggested to be learnt for N c classes, where P ( 1) = 1 P (+1), beng P ( 1) and P (+1) the probablty of the symbols -1 and +1 to appear, respectvely. Then, from a set of defned random matrces, the one whch maxmzes a decodng measure among all possble rows of M c s selected. Sparse Random (Escalera et al., 2009): n = 15 logn c dchotomzers are suggested to be learnt for N c classes, where P (0) = 1 P ( 1) P (+1), defnng a set of random matrces M c and selectng the one whch maxmzes a decodng measure among all possble rows of M c. Problem-Dependent Codng Desgns D (Puol et al., 2006): problemdependent desgn that uses n = N c 1 dchotomzers. The parttons of the problem are learnt by means of a bnary tree structure usng exhaustve search or a SFFS crteron.

3 Fnally, each nternal node of the tree s embedded as a column n M c. Forest- (Escalera et al., 2007): problemdependent desgn that uses n = (N c 1) T dchotomzers, where T stands for the number of bnary tree structures to be embedded. Ths approach extends the varablty of the classfers of the D desgn by ncludng extra dchotomzers. -ONE (Puol et al., 2008): problemdependent desgn that uses n = 2 N c suggested dchotomzers. A valdaton sub-set s used to extend any ntal matrx M c and to ncrease ts generalzaton by ncludng new dchotomzers that focus on dffcult to splt classes. Decodng Desgn The notaton used refers to that used n (Escalera et al., 2008): Hammng decodng: = n = HD( x, y ) 1(1 sgn( x y ))/ 2, beng x a test codeword and y a codeword from M c correspondng to class C. Inverse Hammng decodng: IHD(x, y) = max ( 1 D T ), where ( 1, 2 ) = HD (y 1, y 2 ), and D s the vector of Hammng decodng values of the test codeword x for each of the base code words y. Eucldean decodng: ED( x, y ) n 2 = = 1( x y ) Attenuated Eucldean decodng: AED( x, y ) n = = Loss-based decodng: n 1 y x ( x y ) LB( ρ, v ) = = 1 L( y. f ( ρ)), Where ρ s a test sample, L s a loss functon, and f s a real-valued functon f: R n R. Probablstc-based decodng: PD( y, x P( x ) = log( π [1,..., n] : M = M (, ) f c ) + K), c 2 (, ) 0 Where K s a constant factor that collects the probablty mass dspersed on the nvald codes, and the probablty P(x = Mc (, ) f ) s estmated by means of P y f ey ( = ) = 1/1 + v f w, + ) ( x where vectors υ and ω are obtaned by solvng an optmzaton problem (Passern et al., 2004). Laplacan decodng: α + 1 LAP(x, y ) =, where α s the number α + β + K of matched postons between x and y, β s the number of mss-matches wthout consderng the postons coded by 0, and K s an nteger value that codfes the number of classes consdered by the classfer. Pessmstc β-densty Dstrbuton decodng: v 1 Accuracy s : v sψ( v, α, β ) dv =, where 3 1 α β ψ ( v, α, β ) = v (1 v), ψ s the β- K Densty Dstrbuton between a codeword x and a class codeword y for class c, and v R :[0,1]. Loss-Weghted decodng: LW ( ρ, ) n 1M W (, ) L( y. f ( ρ, )), Where = = M (, ) = H(, ) / = 1 H(, ), w m k H(, ) = 1/ m = 1 ϕ ( h ( ρ ),, ), n = = x x y 1, ϕ (,, ) 0, otherwse m s the number of tranng samples from class C, and ρ k s the k th sample from class C. k Outlne Of Ecoc Algorthm Tranng Load tranng data and parameters,.e., the length of code L and tranng class K. 1. Create a L-bt code for the K classes usng a knd of codng algorthm. 2. For each bt, tran the base classfer usng the bnary class (0 and 1) over the total tranng data. Testng 1. Apply each of the L classfers to the test example.

4 2. Assgn the test example the class wth the largest votes. What Makes A Good Ecoc? The key problem for approach s how to desgn the codng matrx M. Many studes [10, 11, 12, 13, and 14] have shown that the fnal classfer wll have good dscrmnate ablty f the codng matrx M has the followng characterstcs: Characterstc 1: Row separaton Each codeword (a row n the codng matrx M) should be well-separated n Hammng dstance from each of the other code words. Characterstc 2: Column separaton Each column should be uncorrelated wth one another. Ths means that the bnary classfers of dfferent columns have low correlatons among them. Characterstc 3: Bnary classfers have low Errors Whle for recognton of a large number of classes, besdes classfcaton accuracy, the effcency s also qute mportant. To make a quck decson, t s expected to evaluate as few bnary classfers as possble. Ths requres the codeword to be effcent (.e. contans a small number of bts). As explaned n [15], for a code to be effcent, dfferent bts should be ndependent of each other, and each bt has a 50% chance of beng one or zero. In desgn, ndependent bts can be relaxed as uncorrelated columns (.e. property 2 mentoned above). And 50% chance of frng for each bt requres. Characterstc 4: Balanced column For each column, the numbers of 1 and 1 are equal,.e., M ( r, ) = 0. R Fndng an satsfyng the above characterstcs s a NP-hard problem [16]. So we can say that for effcent and accurate recognton of a large number of classes, a good s expected to have the followng characterstcs: Effcent - requres a small number of bts. Good dversty - the codng matrx has good row and column separaton. The resultng bnary classfers are accurate. What s So Good About Ecoc? 1. Improves classfcaton accuracy. 2. Can be used wth many dfferent classfers. 3. Commonly used n many areas. 4. Not prone to over fttng. 5. Possbly try a varant. Practcal Advantages Of Ecoc 1. It s fast, smple and easy to program 2. It s flexble can combne wth any learnng algorthm 3. Able to reduce the bas and varance produced by the learnng algorthm. So t wdely used to deal wth mult-class categorzaton problems. 4. Low computatonal cost. 5. Outperforms the drect multclass method. 6. Can use wth data that s textual, numerc, dscrete, etc. 7. General learnng scheme - can be used for varous learnng tasks. 8. Good generalzaton. Dsadvantages 1. s not effectve f each ndvdual codeword s not separated from each of the other code words wth a large Hammng dstance. 2. only succeed f the errors made n the ndvdual bt postons are relatvely uncorrelated, so that the numbers of smultaneous errors n many bt postons s small. If there are many smultaneous errors, the wll not able to correct them (Peterson & Weldon, 1972). 3. support vector machnes are not always superor to one-aganst-all fuzzy support vector machnes. 4. One-versus-all schemes are more stable than other schemes. 5. Sometmes decomposton of mult-class problem nto multple bnary problems we are dong n ncurs consderable bas for centrod classfer, whch results n notceable degradaton of performance for centrod classfer. 6. Fndng the optmal s NP hard. Comparson Of Some Ecoc Methods. One-Versus-All strategy. The most well-known bnary codng strateges are the one-versus-all strategy [17], where each class s dscrmnated aganst the rest of classes. In Fg. 1a,

5 the one-versus-all desgn for a four-class problem s shown. The whte regons of the codng matrx M correspond to the postons coded by 1 and the black regons to -1. Thus, the code word for class C 1 s {1,-1,-1,-1}. Each column of the codng matrx codfes a bnary problem learned by ts correspondng dchotomzer h. For nstance, dchotomzer h 1 learns C 1 aganst classes C 2, C 3, and C 4, dchotomzer h 2 learns C 2 aganst classes C 1, C 3, and C 4, etc. The Dense Random Strategy. The dense random strategy [10], where a random matrx M s generated, maxmzng the rows and columns separablty n terms of the Hammng dstance [7]. An example of a dense random matrx for a four class problem s shown n Fg. 1c. One-Versus-One and Random Sparse Strategy. It was when Allwen et al. [10] ntroduced a thrd symbol (the zero symbol) n the codng process when the codng step receved specal attenton. Ths symbol ncreases the number of parttons of classes to be consdered n a ternary framework by allowng some classes to be gnored. Then, the ternary codng matrx becomes M { 1,0,1} N n.in ths case, the symbol zero means that a partcular class s not consdered by a certan bnary classfer. Thanks to ths, strateges such as one-versus-one [20] and random sparse codng [10] can be formulated n the framework. Fg. 1b shows the one-versus-one confguraton for a four-class problem. In ths case, the gray postons correspond to the zero symbol. A possble sparse random matrx for a four-class problem s shown n Fg. 1d. Fg. 1. (a) One-versus-all, (b) one-versus-one, (c) dense random, and (d) sparse random desgns. Spectral Error Correctng Output Codes for Effcent Multclass Recognton. Algorthm: Input: Gven the class set C = {c 1, c 2,..., c n } 1. Tran a SVM classfer f for each class par {c, c } 2. Construct the smlarty graph G. Set each class c as a vertex and the weght w. 3. Compute the normalzed Laplacan Lsym of G. 4. Compute the egenvectors v 1, v 2,..., v n of Lsym. 5. Transform each v, 2, to a partton ndcator vector m 6. Generate an matrx M l wth code length l: M l = [m 2, m3,, ml +1 ] { f l = 1 7. Tran bnary classfers } to form code predcton functon f l (.) = [f 1 (.), f 2 (.),..., f l (.)] 8. Search the optmal code length l*. Output: M l * and f l * (.) TABLE I EFFICIENCY COMPARISON ON FACE RECOGNITION AND FLOWER CLASSIFICATION DATA. Dataset Methods Accuracy Code length face one-aganst- 99.5% 300

6 recognton wth k = 18 flower classfcaton wth k=18 all Spectral 99.1% 30 Random 98.9% 120 dense Random 98.8% 100 sparse Class Map 97.3% 150 Dscrmnate 69.0% 200 one-aganstall 54.6% 102 Spectral 54.7% 15 Random 54.0% 83 dense Random 54.1% 44 sparse Class Map 50.2% 35 Dscrmnate 35.2% 100 Conclusons In ths paper the dfferent codng and decodng methods for Error Correctng Output Code have been studed. Advantages and dsadvantages of some codng methods are dscussed. From ths study on one can conclude that compare to other methods, better performance can be acheved by usng Error Correctng Output Code. TABLE II performance parameters (based on results obtaned usng weka on dataset contact-lenses wth 10 folds cross valdaton) Method Name Tme Taken (Seconds) Correctly Classfed Instances (%) Incorrect ly Classfe d Instances (%) Kappa Statst c Mean Absolu te Error Root Mean Squar e Error Relatv e Absolut e Error (%) Root relatve square d error (%) Multclass Classfcato n usng

7 SVM(funct ons.smo ) ANN (functons. Multlayer Perceptron) Meta Baggng Meta AdaboostM Nested Dchotomes Stackng Reference [1] K. Crammer and Y. Snger, On the Learnablty and Desgn of Output Codes for Multclass Problems, Machne Learnng, vol. 47, no. 2-3, pp , [2] A. Passern, M. Pontl, and P. Frascon, New Results on Error Correctng Codes of Kernel Machnes, IEEE Trans. Neural Networks, vol. 15, no. 1,pp , [3] V.N. Vapnk, The Nature of Statstcal Learnng Theory. Sprnger [4] Y. Freund and R.E. Shapre, A Decson- Theoretc Generalzaton of On-Lne Learnng and an Applcaton to Boostng, J. Computer

8 and System Scences, vol. 55,no. 1, pp , [5] T. Haste and R. Tbshran, Classfcaton by Parwse Couplng, Annals of Statstcs, vol. 26, no. 2, pp , [6] E.L Allwen, R.E Shapre, and Y. Snger, Reducng Multclass to Bnary: A Unfyng Approach for Margn Classfers, J. Machne Learnng Research, vol. 1, pp , [7] T.G. Detterch and G. Bakr, Solvng Multclass Learnng Problems va Error- Correctng Output Codes J. Artfcal Intellgence Research, vol. 2, pp ,1995. [8] R.E. Schapre, Usng Output Codes to Boost Multclass Learnng Problems, Machne Learnng: Proc. 14th Int l Conf., pp , [9] C. Hsu and C. Ln, A Comparson of Methods for Mult-Class Support Vector Machnes, IEEE Trans. Neural Networks, vol. 13, no. 2, pp , Mar [10] E. L. Allwen and R. E. Schapre. Reducng multclass to bnary: A unfyng approach for margn classfers. Journal of Machne Learnng Research, 1: , [11] N. Garc ıa-pedraas and C. Fyfe. Evolvng output codes for multclass problems. IEEE Trans. Evolutonary Computaton, 12(1):93 106, [12] R. Ghader and T.Wndeatt. Crcular ecoc: A theoretcal and expermental analyss. In ICPR, pages , [13] O. Puol and P. Radeva. Dscrmnant ecoc: A heurstc method for applcaton dependent desgn of error correctng output codes. PAMI, 28(6): , [14] R. Schapr and Y. Snger. Solvng multclass learnng problems va error-correctng output codes. Journal of Artfcal Intellgence Research, 2: , [15] Y. Wess, A. Torralba, and R. Fergus. Spectral hashng. In NIPS, [16] K. Crammer and Y. Snger. On the learnablty and desgn of output codes for multclass problems. Machne Learnng, 47(2-3): , [17] N.J. Nlsson, Learnng Machnes. McGraw- Hll, [18] T. Haste and R. Tbshran, Classfcaton by Parwse Groupng, Proc. Neural Informaton Processng Systems Conf., vol. 26,pp , 1998.