Normalized Ordinal Distance; A Performance Metric for Ordinal, Probabilistic-ordinal or Partial-ordinal Classification Problems

Transcription

1 Normalize rinal Distance; A Performance etric for rinal, Probabilistic-orinal or Partial-orinal Classification Problems ohamma Hasan Bahari, Hugo Van hamme Center for rocessing seech an images, KU Leuven, Belgium {mohamahasan.bahari,hugo.vanhamme}@esat.kuleuven.be Abstract In many forensic scenarios, we eal with roblems of orinal nature, where there is intrinsic orering between the categories. For examle, in human age grou recognition from seech or images, the categories can be chil, young, mile-age an senior. In etecting the level of intoxication, the categories can be low, meium an high. In this aer, a novel alication-ineenent erformance metric for orinal, robabilistic-orinal an artial-orinal classification roblems is introuce. Conventional erformance metrics for orinal classification roblems, such as mean absolute error of consecutive integer labels an ranke robability score, are ifficult to interret an can be misleaing. In this aer, first, the orinal istance between two arbitrary vectors in Eucliean sace is introuce. Then, a new erformance metric, namely normalize orinal istance, is roose base on the introuce orinal istance. This erformance metric is concetually simle, comutationally inexensive an alicationineenent. The avantages of the roose metho over the conventional aroaches an its ifferent characteristics are shown using several numerical examles. 1. Introuction A large number of real worl classification roblems are orinal, where there is intrinsic orering between the categories. For examle, in quality reiction systems, the task is to categorize the quality of a rouct into ba, goo an excellent [1]. In human age grou recognition from seech or images, the categories can be chil, young, mile-age an senior [2, 3]. In the classification of the theraeutic success, the classes are goo recovery, moerate isability, severe isability, an fatal outcome [4]. In all orinal classification roblems (C ), the class labels are orinal numbers, i.e. there is intrinsic orering between the categories. Probabilistic-rinal an Partial-rinal Classification roblems, labele as C Pr an CPa resectively, are well-known generalizations of the C. In C Pr, for a test ataoint, the classifier calculates the robability of belonging to each category. In C Pa, instea of the cris class labels each ataoint has a egree of membershi to every class [5]. These tyes of roblems, exlaine in sections 2.2 an 2.3 in etail, can be foun in many omains, such as natural language rocessing, social network analysis, bioinformatics an agriculture [5]. Scientists have roose ifferent methos to solve C, C Pr an CPa [5, 6, 7, 8, 9, 10]. For examle, ccullagh introuce an orinal classifier, namely the roortional os moel (P), base on logistic regression [6]. In [7], C is aresse using a generalization of suort vector machines (SV) namely suort vector orinal regression (SVR). A neural network aroach for the C is suggeste in [8]. [9] suggeste Gaussian rocesses for C. In [5], kernel-base roortional os moels is introuce to solve the C Pa. To measure the erformance of these classifiers, ifferent aroaches have been suggeste. For examle, mean zero-one error (E mzo ) an mean absolute error of consecutive integer labels (Ema) cil are wiely alie to measure the erformance of the classifiers in C [7, 8, 9, 10]. However, non of these methos are alicable to C Pr an CPa. Percentage of correctly fuzzy classifie instances (P cfci ) an Average Deviation (E a ) have been suggeste to measure the classifier erformance in C Pr an CPa [5, 11, 12, 13]. The main rawback of P cfci is that it oes not consier the orer of categories [11, 12]. The E a suggests a simle iea to solve this roblem [12, 13]. Although the E a is attractive from several asects, the interretation of its results is ifficult, because the range of its outut eens on the alication. The same ifficulty is observe in Ema. cil Alication eenency makes the interretation of Ema cil an E a very challenging. The average of ranke robability scores (E rs ), is also alie as a erformance metric in C Pr an CPa [14, 15]. In this metho, the orer of categories is imortant an the range of the outut is fixe between 0 an 1. This metho can be alie to C, C Pr an CPa. However, analysis reveals that E rs over estimates the erformance of classifiers in many situations. This issue, which leas to a erroneous interretation of classifier erformance, is illustrate by some numerical examles in section 5. In this aer, we investigate ifferent characteristics of these erformance metrics an finally a new alicationineenent erformance metric, namely Normalize r-

2 inal Distance (E no ), is introuce. The atlab coe of the suggeste aroach, which can be alie to all three tyes of consiere roblems C, C Pr an CPa, can be ownloae from our website 1. This aer is organize as follows. In section 2, the mathematical formulations of C, C Pr an CPa are resente. In section 3, five ifferent conventional erformance metrics are exlaine. The roose erformance metric is elaborate in section 4. In section 5, the effectiveness of the roose aroach is illustrate using some numerical examles. The aer enes with a conclusion in section Problem Formulation In this section, the orinal, robabilistic-orinal an artial-orinal roblems are formulate rinal Classification Assume that we are given a training ata set S tr = {(X 1,Y 1 ),...,(X n,y n ),...,(X N,Y N )}, where X n = [x n,1,...,x n,i,...,x n,i ] enotes a vector of observe characteristics of the ata item an Y n = [ y n,1,...,y n,,...,y n,d ] enotes a label vector. The label vector is efine as follows if X n belongs to class C : y n, j = δ j,. (1) where δ enotes the Kronecker elta. In orinal roblems, there is an intrinsic orering between the classes, which is enote as C 1... C... C D like low, meium an high [5]. The goal is to aroximate a classifier function (G), such that for the m th unseen observation Xm tst, Ŷ m = G(Xm tst ) is as close as ossible to the true label. For a cris classifier Ŷ m is efine as follows if the th class is chosen for Xm tst ŷ m, j = δ j,. (2) 2.2. Probabilistic-rinal classification Probabilistic-orinal classification roblem (C Pr ) is a generalization of the C, where each element of the classifier outut vector (Ŷ ) reresents the robability of belonging to the corresoning category. In this tye of classification, Y n is efine by relation (1). However, Ŷ m is efine as follows ] {[ŷm,1,...,ŷ m,,...,ŷ } m,d R D Ŷ m = ŷ m, 0; D, (3) =1 ŷm, = 1 where R enotes the set of real numbers Partial-rinal Classification Partial-orinal classification roblem (C Pa ) is another generalization of C [5]. In orinal roblems, each ata object 1 htt:// is limite to belong to a single category, i.e. out of all D elements of Y n, only one is nonzero. However, this is too conservative in the case of non-cris or fuzzy classes. This limitation is relaxe in C Pa by rehrasing Y n as follows {[ ] yn,1,...,y n,,...,y } n,d R D Y n = y n, 0; D =1 y. (4) n, = 1 Therefore, each ataoint has a egree of membershi to all classes. Like in orinal roblems, the final goal is to aroximate a classifier function (G), such that for an unseen observation X tst, Ŷ m = G(Xm tst ) is as close as ossible to the true label. In this tye of classification Ŷ m is also efine by relation Conventional Performance etrics In this section, five wiely-use conventional metrics, namely E mzo, Ema, cil P cfci, E a an E rs are introuce [5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17] ean Zero-ne Error (E mzo ) Performance metric E mzo is the fraction of incorrect reictions, which is calculate as follows [7, 8, 9, 10] E mzo = 1 δŷm,y m ), (5) where is the total number of test set ataoints, ŷ m is the reicte label of the m th test set ataoint an y m is the true label of the m th test set ataoint. The main avantage of E mzo is its simlicity. However, it oes not consier the orer of the categories. Furthermore, it is not alicable to measure the erformance in C Pr or CPa ean Absolute Error of Consecutive Integer Labels (E cil ma) To calculate the Ema, cil first, both true labels an reicte labels of the test set ataoints are transforme into consecutive integers so that if the th column of the label vector is 1 then the transforme label is equal to [7, 8, 9, 10]. After label transformation the Ema cil is calculate as follows Ema cil = 1 Û m U m, (6) where Û m is the transforme reicte label of the m th test set ataoint an U m is the transforme true label of the m th test set ataoint. The Ema cil enjoys the avantage of consiering the orer of categories into account. However, it cannot be alie to evaluate the classifiers in C Pr or CPa oreover, the range of its outut is alication-eenent. Therefore, the interretation of this metric is challenging. This is shown in section 5 using some numerical examles.

3 3.3. Percentage of Correctly Fuzzy Classifie Instances (P cfci ) Performance metric P cfci has been alie to measure the erformance of robabilistic or fuzzy classifiers [11, 12]. It is calculate as follows P cfci = D =1 ŷ m, y m, ). (7) As it can be inferre from the above relation, the orer of the categories is not consiere in P cfci Average Deviation (E a ) Performance metric E a was originally introuce by Van Broekhoven [12] to evaluate the classifiers in fuzzy orere classification roblems. It was also alie in ifferent alications with other names [5, 13]. The E a is calculate as follows E a = 1 { =1 ŷ m,i y m,i }. (8) It can be interrete from the above relation that the orer of categories is imortant in E a. It is also useful for classifier evaluation in C Pr or CPa. However, similar to Ema, cil the range of E a is alication-eenent an hence ifficult to interret Average Ranke Probability Scores (E rs ) The ranke robability score was originally introuce to score the outut of robabilistic classifiers [14, 15]. It is efine as follows ) RPS Y (Ŷ ) = 1 2 D 1 ŷ i y i. (9) =1 ( This scoring rule can be easily extene to measure the erformance of classifiers in C, C Pr an CPa using the following relation. 1 E rs = (D 1) =1 ( ŷ m,i y m,i ) 2 (10) As it can be interrete from the above relation, the orer an the number of categories are imortant in E rs. It is assume that the maximum of the nominator of E rs is (D 1). Therefore, to fix the range of E rs between 0 an 1 the nominator is ivie to its maximum ossible value (D 1). However, this assumtion is very conservative so that in many ractical cases the maximum of the nominator of E rs is less than (D 1). Consequently, this assumtion may lea to an erroneous interretation of the classifier erformance. Numerical examles of Section 5 reveal this issue clearly. 4. Proose Performance etric In this section, first, rinal Distance (D) of two vectors in Eucliean sace is introuce. Then, a new erformance metric, namely normalize orinal istance (E no ), is eveloe base on the orinal istance rinal Distance (D) In this section, the efinition of a istance function is recature. Then, the inkowski istance is escribe an finally, the orinal istance is introuce as an extension of the inkowski istance Distance By efinition, a istance function of two oints A = [a 1,...,a,...,a D ] an B = [b 1,...,b,...,b D ] is a function D : R D R D R, which satisfies the following three conitions [18]: 1. D(A,B) 0 an D(A,B) = 0 A = B 2. D(A,B) = D(B,A) 3. D(A,C) D(A,B) + D(B,C) A variety of istance functions have been introuce by scientists for ifferent alications such as inkowski istance, ahalanobis istance, Chebyshev istance an Hamming istance [18] The inkowski istance of orer The inkowski istance of orer or -norm is a istance function, which satisfies all conitions of a istance function. A B = ( D ) 1/ a b (11) =1 where is a real number not less than 1. As in can be interrete from relation (11), in -norm, the orer of the elements of two oints A an B, is not imortant The rinal istance of orer The notion of orinal istance is reviously use to measure the ifferences of two strings [19] or two histograms [20]. In this aer, an orinal istance of two vectors in Eucliean sace is introuce. The rinal Distance of orer between two oints A an B is efine as follows A B D = ā = b = ( D =1 ) 1/ ā b a i (12) b i,

4 where is a real number not less than 1. Since (12) is a inkowski istance between Ā = [ā 1...ā...ā D ] an B = [ b 1... b... b D ], it follows that the orinal istance of orer satisfies the conitions of section Normalize rinal Distance (E no ) In this section, a new erformance metric, namely normalize orinal istance (E no ), is introuce to measure the erformance classifiers in C, C Pr an CPa. E no = Ym Ŷ m D ψ, (13) Y m where ψ Y m is the uer boun of Y Ŷ D for any ossible Ŷ in its efine range. ψ Y is efine as follows ψ Y = max T Y T D, (14) where T = {t 1,...,t,...,t D } is an arbitrary vector with the same secifications of Ŷ mentione in relation (2), I.e., T lies on a simlex. ψ Y can be calculate using theorem 1. Theorem 1: The uer boun of Y Ŷ D for any ossible Ŷ can be obtaine as follows ψ Y = max( Y L 1 D,..., Y L D,..., Y L D D ) (15) or equivalently ψ Y = max( Y L 1 D, Y L D D ), (16) where L is a vector of size Y. The th element of L is equal to 1 an the rest of elements are zero. As it can be interrete from relations (15) an (16), although the latter one is more restrictive, it rovies an easier way to calculate ψ Y. Proof: We first rove the relation (15), which hel us to show the correctness of relation (16). Proof of relation (15): By efinition Y T D = Λ(Y T ), (17) where Λ is a lower triangular matrix of size D D with all iagonal an lower iagonal elements equal to 1. Since (Y T ) is a convex function of T an a convex function remains convex uner an affine transformation, Λ(Y T ) is also convex. n the other han, a convex function on a comact convex set attains its maximum at an extreme oint of the set [21]. In this roblem T {[t 1,...,t,...,t D ] R D t 0; D =1 t = 1}, i.e., T lies on a simlex. The extreme oints of this comact convex set are L with {1,...,D}. Therefore, max T Λ(Y T ) = max ( Λ(Y L 1 ),..., Consequently, Λ(Y L ),..., Λ(Y L D ) ) (18) max Y T D = max ( Y L 1 D,..., T Y L D,..., Y L D D ) (19) Proof of relation (16): Relation (16) is now shown by contraiction. Suose relation (15) is not equivalent with relation (16), then there must be a k {2,...,D 1} such that both Relations 20 na 21 hol. Y L k D Y L k D Exansion of relation (20) an (21) is k 1 ( =1 k 1 ( =1 y i ) + =k y i ) + =k > Y L 1 D (20) > Y L D D (21) y i ) > =1 y i ) > =1 ( After some maniulations (22) an (23) lea to [ k 1 =1 =k [ ( y i ) y i ) ( y i ) (22) y i ). (23) y i ) ] > 0 (24) y i ) ] ] > 0. (25) If relation (24) hols, ( y i) > y i) hence ( y i) > 0.5 for at least one between 1 an k 1. Likewise, from (25), ( y i) < 0.5 for at least one between k an D 1. This is imossible, since y i is an increasing function of an hence (16) hols Relation to E rs There is a close relationshi between E rs an E no secially for = 2. In both E rs an E no, enominators are assume to be the uer boun of the nominator an are use to kee the range of erformance metric between 0 an 1. In E rs, it is assume that the uer boun of the nominator is (D 1) [15, 22]. However, this is a conservative boun in many situations. This is illustrate by some numerical examles in section 5. We will also show this conservative assumtion can result in a misleaing or erroneous interretation of the classifiers erformance. In E no, this uer boun is exlicitly efine by relation (14) an calculate by relation (16).

5 Table 1: The erformance of two classifiers measure by E mzo, E a, Ema, cil P cfci, E rs, Eno 1, E2 no, an E no in examle 1. Performance etric Problem 1 Problem 2 E mzo E a Ema cil P cfci E rs E 1 no E 2 no E no Results an Discussion In this section, the behaviors of E no an five conventional erformance metrics, namely E mzo, P cfci, E a, E rs, an Ema cil are analyze using a number of numerical examles. In examle 1, it is shown that P cfci an E mzo are not suitable for measuring the erformance of orinal classifiers, because these methos o not consier the orer of categories. Examles 2 illustrates that the interretation of E a an Ema cil eens on the number of categories, hence these metrics are alication-eenent. In examle 3, the eficiency of E rs is emonstrate. Examles 4 shows the avantages of E no over P cfci, E rs an E a in measuring the erformance of the classifiers in C Pa, where other conventional aroaches are not alicable Examle 1 In this examle, the avantage of E no over P cfci an E mzo in measuring the erformance of the classifiers in a tyical C is shown. For an orinal three-class classification roblem, classifier 1 an classifier 2 result in confusion matrix 1, labele as C 1 an C 2 resectively. In these matrices each column reresents the instances in a reicte class an each row shows the instances in an actual class C 1 = C 2 = (26) Table 1 shows the erformance of two classifiers measure by E mzo, E a, Ema, cil P cfci, E rs, Eno 1, E2 no, an E no. As it can be interrete from this table, E mzo, P cfci an Eno fail to reflect the egraation of erformance from the classifier 1 to the classifier 2. However, Eno 1, E2 no, E a, E rs an Ema cil erfectly show that classifier 1 outerforms classifier Examle 2 In this examle, we show that the number of categories in the classification roblem influences the interretation of E a an Ema. cil Consier the following two orinal classification roblems. Problem 1: For a test ataoint, the true label an the estimate label are Y 1 = [1 0] an Ŷ 1 = [0 1] resectively. Problem 2: For a test ataoint, the true label an the estimate label are Y 1 = [ ] an Ŷ 1 = [ ] resectively. Table 2 shows the erformance of classifiers in these roblems obtaine using E mzo, E a, Ema, cil P cfci, E rs, Eno 1, Eno 2, an E no in examle 2. As it can be interrete from Table 2, E mzo, P cfci, E a, Ema cil an Eno treate classifiers of both roblems in the same manner. However, the estimate label of the first roblem is comletely incorrect, while the estimate label in the secon roblem is very near to the true label. Performance metrics E rs, Eno 1 an E2 no reflect the higher erformance of the secon classifier comare to the first one Examle 3 In this examle, the main eficiency of E rs an ifficulties in interretation of E a an Ema cil are exemlifie. Consier the following two cases. Case 1: For an orinal three-class classification roblem, a comletely useless classifier is alie which result in C C 3 = (27) Case 2: For another orinal three-class classification roblem, con- Table 2: The erformance of two classifiers measure by E mzo, E a, Ema, cil P cfci, E rs, Eno 1, E2 no, an E no in examle 2. Performance etric Problem 1 Problem 2 E mzo 1 1 E a 1 1 Ema cil 1 1 P cfci 0 0 E rs E 1 no E 2 no E no 1 1

6 sier a classifier with C C 4 = (28) The erformance of classifiers in case 1 an 2 calculate by the E mzo, P cfci, E a, E rs, E no an Ecil ma are liste in Table 3. As it can be seen from the Table 3, the erformance of the alie classifier in case 1 measure by E rs is 0.50, while all estimate labels are incorrect an the classifier is totally useless. The oututs of E no an P cfci are 1 an 0 resectively, which arooriately reflect that the alie classifier is useless in this case. The table also inicates that E rs, E a an Ema cil result in the same values for both cases, while we know that the alie classifier in the secon case is much more effective than the first one. This is arooriately reflecte by Eno 1, E2 no an E no Examle 4 In this examle, P cfci, E a, E rs, an E no are evaluate in measuring the erformance of classifiers in C Pa. Consier an eight-class C Pa. In this roblem, the test ataoint label is Y = [ ]. Two classifiers are alie in this roblem. Table 4 shows the outut of the alie classifiers. The measure erformance of these classifiers using P cfci, E a, E rs, Eno 1, E2 no an Eno is resente in Table 5. As it can be unerstoo from Table 4, the estimate label of the secon classifier is more similar to the true label comare to that of first classifier. However, the outut of the P cfci is the same for both of them. This is ue to the fact that the orer of categories has no effect on the outut of P cfci. In this examle, E a, E rs, Eno 1, E2 no an E no reflect the erformance imrovement from the first classifier to the secon one. Table 3: The erformance of two classifiers measure by E mzo, E a, Ema, cil P cfci, E rs, Eno 1, E2 no, an E no in examle 3. Performance etric Case 1 Case 2 E mzo E a 1 1 Ema cil 1 1 P cfci 0 50 E rs E 1 no E 2 no E no Conclusion In this aer, the orinal istance between two arbitrary vectors in Eucliean sace has been introuce. Then, Normalize rinal Distance (E no ) as an alicationineenent erformance metric for orinal, robabilisticorinal or artial-orinal classification roblems has been resente. Different avantages of the E no over conventional erformance metrics such as mean absolute error of consecutive integer labels Ema, cil mean zero-one error (E mzo ), correctly fuzzy classifie instances (P cfci ), average eviation (E a ), or ranke robability score (E rs ) have been shown using a number of numerical examles. 7. Acknowlegements This work is suorte by the Euroean Commission as a arie-curie ITN-roject (FP7-PEPLE-ITN-2008), namely Bayesian Biometrics for Forensics (BBfor2), uner Grant Agreement number References [1] S. Erural, A metho for robust esign of roucts or rocesses with categorical resonse, ETU, Ankara, [2]. H. Bahari an H. Van hamme, Seaker age estimation an gener etection base on suervise non-negative matrix factorization, in Proc. IEEE Worksho on Biometric easurements an Systems for Security an eical Alications (BIS), 2011, [3].H. Bahari an H. Van hamme, Seaker age estimation using hien markov moel weight suervectors, in Proc. 11th International Conference on Information Science, Signal Processing an their Alications (ISSPA), 2012, [4] J.S. Caroso an J.F.P. a Costa, Learning to classify orinal ata: the ata relication metho, Journal of achine Learning Research, vol. 8, no ,. 6, [5] J. Verwaeren, W. Waegeman, an B. De Baets, Learning artial orinal class membershis with kernel-base roortional os moels, Comutational Statistics an Data Analysis, vol. 56, no. 4, , Table 4: The outut of alie classifiers in examle 4. Classifier utut (Ŷ ) Classifier Classifier

7 Table 5: The erformance of two classifiers measure by P cfci, E a, E rs, Eno 1, E2 no an E no in examle 4. Performance etric Classifier 1 Classifier 2 E a P cfci E rs E 1 no E 2 no E no [6] P. ccullagh, Regression moels for orinal ata, Journal of the royal statistical society. Series B (ethoological), , [7] W. Chu an S.S. Keerthi, Suort vector orinal regression, Neural Comutation, vol. 19, no. 3, , [8] J. Cheng, Z. Wang, an G. Pollastri, A neural network aroach to orinal regression, in Neural Networks, IJCNN 2008.(IEEE Worl Congress on Comutational Intelligence). IEEE International Joint Conference on, 2008, [9] W. Chu an Z. Ghahramani, Gaussian rocesses for orinal regression, Journal of achine Learning Research, vol. 6, , [15] A.H. urhy, n the ranke robability score, J. Alie eteorology, vol. 8, , [16] J. Kohonen an J. Suomela, Lessons learne in the challenge: making reictions an scoring them, Lecture Notes in Artificial Intelligence, , [17]. Toa, easurement of subjective robability istributions, Tech. Re., DTIC Document, [18].. Deza an E. Deza, Encycloeia of istances, Sringer, [19] J. orovic, J. Shaw, an P.L. Sun, A fast, noniterative an exact histogram matching algorithm, Pattern Recognition Letters, vol. 23, no. 1, , [20] J. Luxenburger, oeling an Exloiting User Search Behavior for Information Retrieval, Ph.D. thesis, PhD thesis, Universität es Saarlanes, [21] D.R. Kincai an E.W. Cheney, Numerical analysis: mathematics of scientific comuting, vol. 2, Amer athematical Society, [22]. Déqué, JF Royer, R. Stroe, an. France, Formulation of gaussian robability forecasts base on moel extene-range integrations, Tellus A, vol. 46, no. 1, , [10] SK Shevae an W. Chu, inimum enclosing sheres formulations for suort vector orinal regression, in Data ining, ICD 06. Sixth International Conference on. IEEE, 2006, [11] S. anel, H.C. Williams, an S.J. rmero, Evaluating resence absence moels in ecology: the nee to account for revalence, Journal of Alie Ecology, vol. 38, no. 5, , [12] E. Van Broekhoven, V. Ariaenssens, an B. De Baets, Interretability-reserving genetic otimization of linguistic terms in fuzzy moels for fuzzy orere classification: An ecological case stuy, International Journal of Aroximate Reasoning, vol. 44, no. 1, , [13] A.. outon, B. De Baets, E. Van Broekhoven, an P.L.. Goethals, Prevalence-ajuste otimisation of fuzzy moels for secies istribution, Ecological oelling, vol. 220, no. 15, , [14] P. Bougeault, The wgne survey of verification methos for numerical reiction of weather elements an severe weather events, Toulouse: étéo-france, 2003.