Study of Classification Methods Based on Three Learning Criteria and Two Basis Functions

Transcription

1 Study of Classfcaton Methods Based on hree Learnng Crtera and wo Bass Functons Jae Kyu Suhr Abstract - hs paper nvestgates several classfcaton ethods based on the three learnng crtera and two bass functons. he three learnng crtera are the least squares error, the total error rate, and the area under the recever operatng characterstc curve. he two bass functons are reduced ultal odel and sngle-hdden layer feedforward neural networks. In the experent, fve classfcaton ethods were evaluated by usng the UCI database and sutable data ralzaton procedures for each bass functon were dscussed.. Introducton Pattern classfcaton has been a key coponent for decson akng n any research felds. In pattern classfcaton, eprcal learnng consttutes a aor paradg []. Under ths paradg, a classfer s desgned to nze a certan cost functon (crteron) wth a tranng set. Least Squares Error (LSE) has been wdely used as a cost functon for eprcal classfer learnng. he reasons for the popularty of LSE are ts splcty, clear physcal eanng, and tractablty for analyss. he ebedent of nlneartes nto lnear odels has wdened ts applcaton. Recently, two effcent bass functons were proposed: Reduced ultal Model () [] and Extree Learnng Machne () [3]. uses a reduced verson of full polyal and uses a Sngle-hdden Layer Feedforward Neural networks (SLFNs). However, the an proble when usng the LSE cost functon s that t tres to nze the fttng error rather than the classfcaton error durng the learnng process. herefore, three an approaches have been adopted to overcoe ths drawback of the LSE cost functon. hese approaches are the dscrnant approach (Fsher Dscrnant Analyss and Generalzed Dscrnant Analyss), the structural approach (Support Vector Machne), and the classfcaton-error approach. In the thrd approach, two cost functons were recently proposed. One s otal Error Rate (ER) [4,5] and the other s Area under the recever operatng characterstcs curve (AUC) [6]. he an breakthrough of these three papers s a sooth approxate forulaton for calculatng ER and AUC. he step functon used for the countng process s approxated by a quadratc functon nstead of a sgod functon. hs akes t possble to have closed-for soluton. In ths paper, the above entoned fve ethods (,,,, and ) were nvestgated. Fve two-class probles n the UCI database were used for the experent snce can only be appled to two-class proble so far. he paper s organzed as follows: In Secton, the nvestgated classfcaton ethods are brefly descrbed. In Secton 3, experental setup, data ralzaton ssue, and evaluaton results are presented. Fnally, ths paper s concluded wth a suary n Secton 4. Method descrpton. LSE-based ethod Least Squares Error (LSE) s a wdely used cost functon for eprcal classfer learnng due to the several reasons as enton n the prevous secton. Consder an l-densonal nput x and the followng paraetrc odel adoptng a bass expanson ter: K g(, x) = p ( x) = p( x) () k = k k where pk ( x ) corresponds to the kth bass ter of the row vector p( x) = [ p( x), p( x),..., pk ( x )], and = [,,... ] K s a colun paraeter vector to be estated. When we have learnng data pars p( x ) vector can be K extended to P ( ) and a kwn label can be deted by y ( ). In ths case, the LSE cost functon becoes b J ( ) = y P () where b controls the weghtng of the regularzaton factor. he estated tranng output s gven by yˆ = P ˆ, where the soluton for ˆ, whch nzes J s ˆ = ( PPbI) Py (3) hs LSE cost functon has been wden ts applcaton by ebeddng nlneartes nto lnear odels. Recently, two effcent nlnear bass functons were ntroduced. One s the Reduced ultvarate Model () [] and the other s the Extree Learnng Machne () [3]. uses reduced verson of full polyal as a bass functon whch can be expressed as r l r ˆ k f (, x) = x ( x x x ) k rl l k= = = r x x x xl = l r ( )( ),,. uses a Sngle-hdden Layer Feedforward Neural networks (SLFNs) as a bass functon. It randoly chooses hdden des and analytcally deternes the output weghts of SLFNs. Gven arbtrary dstnct saples ( x, y ), =,, where x s a d-densonal nput vector and y s a (4)

2 q-densonal output target vector. he standard SLFNs wth p hdden des and actvaton functon φ can be odeled as p p β φ ( x ) = β φ ( w x b ) = g, =,..., = = where w s the weght vector connectng the th hdden de to the nput des, β s the weght vector connectng the th hdden de to the output des, b s the threshold of the th hdden de, and g s the q-densonal network output. he equatons above can be wrtten ore copactly as Hβ = y (6) where φ( w x b) φ( wp xp bp) H = φ( b) φ( p bp) w x w x p and y s a kwn label. Snce H can be treated as a P atrx n Eq. (), a least-squares soluton can be obtaned for the output weghng paraeters as ˆ β = Hy= ( HH) Hy (8). ER-based ethod Although, LSE -based ethods (for exaple and ) has been wdely used, ts ltaton becoes apparent when hgh accuracy s requred. It s anly because the LSE cost functon nzes the fttng error rather than the classfcaton error whch s ostly desred to be nzed n classfcaton tasks. o overcoe ths drawback, the otal Error Rate (ER) cost functon was proposed [4,5]. ER can be calculated by sung the false postve rate (FP) and the false negatve rate (FN). FP and FN can be expressed as Eq. (9) by detng the postve and negatve exaples of varables by the superscrpts and -, respectvely. FP = Lg ( ( ) τ ), FN Lg ( ( ) τ ) x = x (9) = = where L s a zero-one step loss functon. Let g( x) = g(, x ) wth adustable paraeters operatng on the feature vector x, then the ER can be wrtten as ER(, x, x ) () = Lg ( (, ) τ) L( τ g(, )) x x = = A natural choce to approxate a step functon s a sgod functon [7]. However, ths leads two aor probles. Frst, the forulaton becoes nlnear wth respect to the learnng paraeters. hs akes t dffcult to fnd the optal paraeters. Second, the obectve functon can becoe llcondtoned due to the uch local plateaus. hs causes the optzaton procedure to be te-consung. herefore, a quadratc functon was proposed to approxate a step functon. hs s very effectve snce t results a closedfor classfcaton-error-based soluton. By usng ths, (5) (7) approxaton and g(, x) = p( x), the ER n Eq. () can be rewrtten as b ER(, x, x ) = ( ) τ η px = () τ ( ) η px = where η s a postve offset of a quadratc functon. he LSE soluton of Eq. () calculated by lettng the dervatve of ER(, x, x ) wth respect to the paraeter be zero s ( τ η) ( τ η) = b I p p p p p p () where I s and dentty atrx of K K sze. In a ore copact atrx for, Eq. () can be wrtten as = bi PP P P (3) ( τ η) ( τ η) P P where P and P are the sae as P n Eq. () except that they are produced by usng the postve and negatve saples, respectvely, and = [,...,], = [,...,]. hs ER cost functon based on the quadratc approxaton was appled to the and bass functon n [4,5]. hey are called as and, respectvely..3 AUC-based ethod he recever operatng characterstcs (ROC) curve has been extensvely adopted for evaluatng the classfer perforance. However, the processes of classfer desgn optzaton and the fnal ROC perforance evaluaton are usually conducted separately. hs s anly because the ROC does t have a well-posed structure due to the error countng pont of vew. o overcoe ths drawback, a sooth approxate forulaton by usng a quadratc functon was proposed to calculate the Area under ROC curve (AUC) [6]. hs enables a drect optzaton of the AUC wth respect to the classfer paraeters. he AUC [8] for the gven tranng exaples can be expressed as AUC( x, x ) = (4) g( ) > g( ) x x = = where the ter corresponds to a whenever the g( x ) > g( x ) eleents g( x ) > g( x ), and otherwse. Let g( x) = g(, x ) wth adustable paraeters operatng on the feature vector x, then the goal to optze the classfer s dscrnaton perforance can be treated as to axze the AUC: arg ax AUC(, x, x ) = arg ax u g(, ) g(, ) x x = = (5) where u s a unt step functon. Maxzng the AUC s equal to nzng the Area Above ROC curve (AAC) n Eq. (6).

3 argn AAC(, x, x ) = arg n u g(, ) g(, ) x x = = (6) Snce the approxaton by usng a sgod functon can cause soe probles as entoned n the prevous secton, a quadratc functon was used to approxate a step functon. he approxated AAC after lettng g(, x) = p( x) s argn AAC(, x, x ) b (7) arg n ( ( ) ( )) η px px = = where η s a postve offset of a quadratc functon. he optal paraeter can be obtaned as Eq. (8) by lettng the dervatve of Eq. (7) wth respect to be zero. = bi ( ) ( ) p p p p = = η ( ) p p = = (8) where I s and dentty atrx of K K sze. he optal threshold n the sense of the total error rate crteron [9] can be calculated as τ = ( ) ( ) px px (9) = = 3. Experents 3. Experental setup In the experent, we appled fve dfferent classfcaton ethods to fve two-class probles n the UCI database []. Suares of classfcaton ethods and data sets are shown n able and able, respectvely. he experental setup s as follows. Mn-ax ralzaton was appled to P atrx n case of -based ethods and to the orgnal feature vector n case of based ethods. he reason for ths procedure wll be dscussed later. runs of -fold cross valdaton were perfored. ~ orders for -based ethods and ~ nuber of hdden neurons for -based ethods. A sgod functon was used for -based ethods as an actvaton functon due to ts popularty and effectveness. We fxed τ = η =.5 for and and η = for. he regularzaton factor b was set to -4 for all ethods. o evaluate and copare each ethod, two crtera were used. One s the test classfcaton error rate and the other s the (= -log AUC). For both crtera, a lower percentage eans better perforance. able. Suary of Classfcaton Methods Cost func. LSE ER AUC Bass func. [] [4] [6] SLFNs [3] [5] - able. Suary of Data Sets DB nae #case #feature #class #ss Pa-dabetes No SPEC-heart 67 No StatLog-heart 7 3 No c-tac-toe No No 3. Experental results he experental results consst of two parts. Frst, the feature vector ralzaton procedure s dscussed and then the results of fve dfferent classfcaton ethods are dscussed. For ralzng the feature vectors, we used the n-ax ralzaton ethod whch s kwn as a sple and effectve technque []. Snce several ralzaton ethods have been already copared n that paper, we tred to analyze how to use t when usng dfferent bass functons ( and ). We appled the n-ax ralzaton technque n three dfferent ways: ralzaton, ralzaton before akng P or H atrx, and ralzaton after akng P or H atrx before P atrx after P atrx order(~) sngular value rato (log scale) order(~) Fg.. Coparson of three ralzaton procedures for n ters of classfcaton error and sngular value rato. Fg. shows the classfcaton error rates of wth three dfferent ralzaton procedures by usng data set. In Fg., ralzaton after akng P atrx produces the best perforance and ralzaton produces the worst perforance. he reason that the results of ralzaton before and after akng P atrx are better than the result of ralzaton s the nstablty of the paraeter estaton caused by a sngularty of P P atrx whch should be nverted. hs sngularty coes when generatng P atrx. Snce P atrx s produced by ultplyng and addng any s, the coponent values of ths atrx are qute unbalanced f s are t ralzed. hs akes P P atrx close to sngular n case of ralzaton. One ore portant thng we can tce fro Fg. s that the result of ralzaton after akng P atrx produces a better perforance than the result of ralzaton before akng P atrx. hs s also due to the sngularty of 5 5 before P atrx after P atrx

4 P P atrx. Even though s are ralzed before akng P atrx, the ralzed s are ultpled and added each other whle akng P atrx. hs also produces unbalanced coponent values n P atrx whch causes a sngularty of P P atrx. Fg. shows how uch sngular P P atrx s n each ralzaton procedure and each order. he degree of sngularty was easured by usng a rato between the largest and sallest sngular values of P P atrx. Snce a sngular value ndcates an portance of correspondng sngular vector, t can be sad that a larger sngular value rato eans ore sngularty. Sngular value ratos are presented n log scale. In cases of ralzaton and ralzaton before akng P atrx, sngular value ratos ncrease wth order. hs s because a hgher order leads ore unbalanced coponents. However, ths rato s alost constant n case of ralzaton after P atrx. hs s the reason that ralzaton after akng P atrx produces a good and stable perforance n all orders before H atrx after H atrx hdden neuron (~) hdden neuron (~) Fg.. Coparson of three ralzaton procedures for n ters of classfcaton error and kurtoss value. Fg. shows classfcaton error rates of wth three dfferent ralzaton procedures by usng data set. In Fg., the results of ralzaton and ralzaton before akng H atrx are alost the sae and the result of ralzaton before akng H atrx s the best. he reason that ralzaton after akng H atrx and ralzaton produce slar results are due to the nature of a sgod functon used as an actvaton functon. Fg. 3 shows hstogras of s at each step of n case of ralzaton after akng H atrx. Fg. 3 shows a hstogra of the orgnal s. hese feature values are n the range of about ~35. Fg. 3 shows a hstogra of s after the rando weght and bas. Most of the values are n the range of about -~. Fg. 3 (c) shows a hstogra of s after applyng a sgod functon. Snce a sgod functon stretches out the values around zero and akes negatve sall values zeros and postve large values ones, ost of the s are concentrated on zero and one. herefore, t s alost eanngless to ralze s after akng H atrx. In Fg., the result of ralzaton before akng H atrx s better than the results of others. hs s because the pre-ralzaton akes the s bounded near zero before applyng a sgod functon. Fg. 4,, and (c) show a hstogra of the orgnal s, a hstogra kurtoss value 4 3 before H atrx after H atrx of ralzed s before akng H atrx and a hstogra of s after the rando weght and bas, respectvely. It can be seen that the s after the rando weght and bas are n the range of about -4~5. herefore, after applyng a sgod functon, the feature values becoe uch rcher (Fg. 4 (d)) than those of the ralzaton after akng H atrx (Fg. 3 (d)). hs s the reason that ralzaton before akng H atrx outperfors than the others. Fg. shows kurtoss of the s of H atrx n each ralzaton procedure and each order. hs value s near 3 for Gaussan dstrbuton and for sub-gaussan dstrbuton. Snce the dstrbuton n cases of ralzaton and ralzaton after akng H atrx are shaped lke sub-gaussan, the kurtoss values are close to. In case of ralzaton before akng H atrx, the dstrbuton s shaped lke Gaussan, so the kurtoss values are close to 3. nuber of occurrence nuber of occurrence (c) (d) Fg. 3. Hstogras of s at each step of n case of ralzaton after akng H atrx. nuber of occurrence nuber of occurrence (c) nuber of occurrence nuber of occurrence nuber of occurrence nuber of occurrence (d)

5 Fg. 4. Hstogras of s at each step of n case of ralzaton before akng H atrx order(~), hdden neuron(~) wdbc Pa-dabetes order(~), hdden neuron(~) SPEC-heart order(~), hdden neuron(~) StatLog-heart order(~), hdden neuron(~) tc-tac-toe order(~), hdden neuron(~) order(~), hdden neuron(~) wdbc order(~), hdden neuron(~) Fg. 5. ER and of,,,, and n fve data sets. and coluns show ER and n each data set, respectvely. he experental results of fve ethods by usng fve twoclass probles are shown n Fg. 5. and coluns n ths fgure show the test classfcaton error rates and values n each data set, respectvely. he perforances of all ethods are very slar. Especally, and shows alost the sae perforance n ters of test error rate and. hs s because fnds the optal order(~), hdden neuron(~) SPEC-heart order(~), hdden neuron(~) StatLog-heart order(~), hdden neuron(~) tc-tac-toe Pa-dabetes paraeters wth a fxed threshold to nze the total error rate and fnd the optal threshold wth the fxed paraeters to nze the total error rate. est error rate and value show very slar trend. It ght be sad that there s a strong correlaton between test error rate and. 4. Conclusons In ths paper, fve dfferent classfcaton ethods based on three learnng crtera and two bass functons were nvestgated. Approprate ralzaton procedures for and -based ethods were also dscussed. For two-class probles, fve ethods showed slar perforances, especally, and were qute slar. he results also showed that the classfcaton error rate and are hghly correlated. For data ralzaton, t was shown that the ralzaton should be appled after akng P atrx n case of and before akng H atrx n case of. hs can be generalzed as follows: the data ralzaton procedure should be chosen wth a consderaton of a bass functon propertes. References [] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classfcaton, seconded. John Wley & Sons,. [] K.-A. oh, Q.-L. ran, and D. Srnvasan, Bencharkng a reduced ultvarate polyal pattern classfer, IEEE rans. Pattern Analyss and Machne Intellgence, vol. 6,. 6, pp , 4. [3] Huang, G.-B., Zhu, Q.-Y., & Sew, C.-K. (6). Extree learnng achne: heory and applcatons. Neurocoputng, 7, [4] K.-A. oh and H.-L. Eng, Between classfcaton-error approxaton and weghted least-squares learnng, IEEE rans. Pattern Analyss and Machne Intellgence, vol. 3,. 4, pp , 8. [5] K.-A. oh, Deternstc Neural Classfcaton, Neural Coputaton, 8. [6] K.-A. oh, J. K and S. Lee, Maxzng Area Under ROC Curve for Boetrc Scores Fuson, Pattern Recognton, 8. [7] K.-A. oh, Learnng fro arget Kwledge Approxaton, Proc. Frst IEEE Conf. Industral Electroncs and Applcatons, pp. 85-8, May 6. [8] J.A. Hanley, B.J. McNel, he eanng and use of the area under a recever operatng characterstc (ROC) curve, Radology 43 (98) [9] K.-A. oh, Between AUC Based and Error Rate Based Learnng, he 3rd IEEE Conference on Industral Electroncs and Applcatons (ICIEA), Sngapore, June 8. [] D.J. Newan, S. Hettch, C.L. Blake, and C.J. Merz, UCI Repostory of Machne Learnng Databases, Unv. of Calforna, Dept. of Inforaton and Coputer Scences,

6 [] A. K. Jan, K. Nandakuar and A. Ross, Score Noralzaton n Multodal Boetrc Systes, Pattern Recognton, Vol. 38, No., pp. 7-85, Deceber 5.