FEATURE SELECTION BASED ON SURVIVAL CAUCHY-SCHWARTZ MUTUAL INFORMATION

Transcription

1 04 IEEE International Conferene on Aousti, peeh and ignal Proessing (ICAP FEATURE ELECTIO BAED O URVIVAL CAUCHY-CHWARTZ MUTUAL IFORMATIO Badong Chen, iaohan Yang, Hua Qu, Jihong Zhao, anning Zheng, Jose C. Prinipe. hool of Eletroni and Inforation Engineering, i an Jiaotong University, i an, China. Departent of Eletrial and Coputer Engineering, University of Florida, Gainesville, UA (henbd@ail.xjtu.edu.n, prinipe@nel.ufl.edu ABTRACT Feature seletion tehniques play a ruial role in ahine learning tasks suh as regression and lassifiation. Many filter ethods of feature seletion are based on the utual inforation (e.g. MIF, MIF-U, MIF, and RMR ethods. In this work, a new utual inforation is defined based on the ross survival inforation potential (CIP and Cauhy-hwartz divergene (CD, alled the survival Cauhy-hwartz utual inforation (C-MI. We apply this new utual inforation to selet an inforative subset of features for a VM lassifier. Experiental results illustrate the desirable perforane of the new ethod. Index Ters Feature seletion, survival inforation potential, Cauhy-hwartz divergene, lassifiation. ITRODUCTIO eleting an inforative subset of andidate features is very iportant in ahine learning sine it has a ruial ipat on the oputational ost and generalization perforane of the learning algoriths. The feature seletion tehniques in lassifiation an be, in general, divided into approahes that are lassifier-dependent ("wrapper" or "ebedded" ethods and lassifier-independent ("filter" ethods. The filter ethods define a heuristi soring riterion to evaluate the relevane of the data independently of any partiular lassifier. Many feature seletion riteria in the literature are designed based on the fundaental onept of utual inforation (MI [-6]. everal typial exaples are listed in Table, where C denotes the lass label, denotes the set of urrently seleted features, f denotes a andidate feature that is not seleted so far, β is a ontrol paraeter, H and I denote, respetively, hannon's entropy and utual inforation [7]: H ( p ( xlog p ( x ( Table everal utual inforation based feature seletion riteria This work was supported by ational atural iene Foundation of China (o Criterion Criterion funtion J ( f MIF [] I( C; f β ( I( s; f s RMR [4] I( C; f ( I( s; f s MIF-U [3] I( f; s I( C; f β I( C; s s H( s MIF-U [5] I( f; s I( C; f ax I( C; s s H( s MIF [6] I( f; s IC ( ; f s in{ H ( f, H( s} py ( x, y I( Y ; = py ( xy, log dy ( p( x py( y where p, p Y, and p Y denote the orresponding arginal and joint probability densities (or the probability asses for disrete variables. In reent years, soe new definitions of entropy and utual inforation are proposed based on uulative distribution funtions or survival funtions of the rando variables, suh as the uulative residual entropy (CRE [8,9], ross uulative residual entropy (CCRE [8,9], survival exponential entropy [0], and survival inforation potential (IP []. Copared with the traditional definitions, these new definitions have soe erits suh as the validity in a wide range of distributions, robustness, and the sipliity in oputation. The CCRE has been suessfully used in iage registration [8,9], and the IP finds appliations in adaptive systes training []. In this paper, we define a new utual inforation, alled the survival Cauhy-hwartz utual inforation (C-MI, based on the ross IP (CIP and Cauhy-hwartz divergene (CD []. The proposed utual inforation an be easily estiated fro saples (just by oparing the data values and arrying out a ultipliation, without the hoie of any free paraeters. This new utual inforation is then applied to selet an inforative subset of features for a VM lassifier, and the experiental results onfir its good perforane in feature seletion /4/$ IEEE 676

2 . URVIVAL CAUCHY-CHWARTZ MI.. Definitions Before presenting the definition of C-MI, we give the definitions of the ross survival inforation potential (CIP and the survival Cauhy-hwartz divergene (CD. Let andy be two non-negative rando variables with the sae diension, Y,. Denote F (. and G (. respetively, the survival funtions of and Y, F ( x = P( > x = P(,, > x > x (3 where = (,,,, and x = ( x,, x. Then the CIP between and Y (or F and G is defined by ( Y, = ( FG, = F( xg ( x (4 If the distributions of andy are idential, the CIP equals the quadrati IP (QIP []: ( Y, = (, = F ( x (5 In addition, the following equality holds: ( Y, = E in ( i, Yi (6 i= where E denotes the expetation operator. The above equality an be easily proved as follows: F( x G( x = P( > x,, > x P( Y > x,, Y > x = P in, Y > x,, in, Y > x And hene Y, = ( ( ( ( F( xgx ( = ( in (, >,, in (, > P Y x Y x = E I ( in ( i, Yi > xi i= = E I ( in ( i, Yi > xi i= = E in ( i, Yi i= where I (. denotes the indiator funtion. Based on the CIP, we define the survival Cauhy- hwartz divergene (CD between andy as (7 ( Y, DC (, Y log (, ( Y, Y F( x G( x (8 log F ( x G ( x whih is in for idential to the Cauhy-hwartz divergene (CD defined in []. By Cauhy-hwartz inequality, we have DC (, Y 0 (9 where equality holds if and only if F ( x = γ G( x for a onstant salar γ. As F( 0 = G( 0 =, we onlude that DC (, Y = 0if and only if F ( x = G( x. uppose now, Y. Based on the CD, we define the survival Cauhy-hwartz utual inforation (C-MI between andy as C (, = C (, I Y D H FG ( H, FG ( H, H ( FG, FG log (0 H( z F( x G( y dz log H ( z dz F ( x G ( y dz where Z = (, Y, =, and H (. is the survival funtion of Z. Clearly, we have IC (, Y 0, with equality if and only if and Y are independent (i.e. H = FG... Estiators We an easily estiate the CIP, CD, and C-MI fro the saple data. Given saple data of, { x(, x(,, x( }, the survival funtion of an be estiated as ˆ F ( x = δ ( τ x( i dτ Ω( x i= ( = ( u( x x( i i= where Ω ( x = { ξ R : ξ > x,, ξ > x}, δ (. is the ultivariate Dira δ funtion, and u( x xi ( = δ ( τ xi ( dτ ( Ω( x 676

3 Iˆ C (, Y log in ( xk( i, xk( j in ( yk( i, yk( l i= j= l= k= k= i= j= k= k= i= j= k= in ( xk( i, xk( j in ( yk( i, yk( j in ( xk( i, xk( j in ( yk( i, yk( j i= j= k= (7 ubstituting ( into (, ˆ ˆ (, = F ( x, we obtain = u( x x( i i= (3 = ( u( x x( i u( x x( j i= j= ( a = in ( xk( i, xk( j i= j= k = where (a follows fro ( u( x x( i u( x x( j = in ( xk( i, xk( j k = (4 By siilar derivation, we obtain ˆ ( YY, and ˆ ( Y,, and hene, the CD an be estiated as ˆ ˆ ( Y, DC (, Y log ˆ ( ˆ, ( YY, in ( xk( i, yk( j i= j= k= log in ( xk( i, xk( j in ( yk( i, yk( j i= j= k= i= j= k= (5 Then, the C-MI between and Y an be siply estiated as Iˆ Y, = Dˆ HFG, (6 C ( C ( The detailed expression is shown in (7 at the top of this page. Reark: The C-MI has soe erits: it has onsistent definition in the ontinuous and disrete doains; it an be oputed fro saple data without density estiation and the hoie of free paraeters; 3 it is a ore robust easure sine the survival funtion is ore regular than the density funtion (note that the density is oputed as the derivative of the distribution. The C-MI is defined only for non-negative rando variables, but this will not prohibit its pratial appliability, sine in ost pratial situations, the saple data are always bounded, and one an easily obtain positive data by siple translation. 3. APPLICATIO TO FEATURE ELECTIO The C-MI has any potential appliations in areas where traditional utual inforation is applied. In this work, we fous only on the feature seletion proble. peifially, one an design soe new feature seletion riterion based on the C-MI. For exaple, a seletion riterion siilar to the MIF-U an be designed as IC ( f; s J ( f = IC ( C; f β IC ( C; s s (, ss (8 whih we refer to as the "C- MIF-U" riterion. In the following, we evaluate the perforane of the seleted features indued by different riteria through onduting experients on two data sets: Pia Indians Diabetes Data et [3], and Heart Disease Data et [4], whih are set in the UC-Irvine repository. Table lists the brief inforation of the two data sets. The perforane of the C- MIF-U riterion is opared with the results of MIF, MIF-U, RMR, MIF-U, and MIF (see Table for details. In all ases, the ontrol paraeter β for MIF, MIF-U and C-MIF-U was experientally set at 0.8. Table Brief Inforation of the Data ets Used Dataset Feature nu aple nu Classes Pia Heart We onsider the upport Vetor Mahine (VM as the lassifier to evaluate the seleted feature subsets and show the effetiveness of the new riterion. In the experients we use the LIBVM pakage, whih supports both -lass and ultilass lassifiation. Both data sets used were split into two disjoint sets: training (70%, and testing(30%. Pia Indians Diabetes Data et First of all, we noralized every input feature of this data set to have the values in [0, ]. Table 3 shows the rates of orret lassifiation obtained by VM. It opares the perforane of C- MIF-U for the entire range of feature seletion with the perforane of other five riteria. The bold nubers in the table indiate that this riterion perfors better than the rest of the riteria. As one an see learly, the C-MIF-U gets slightly higher lassifiation auray than other riteria exept only when the nuber of the seleted features is

4 Table 3 Corret Classifiation Rates for Pia Indians Diabetes Data et (% u of eleted Feature MIF MIF-U RMR MIF MIF_U C-MIF-U ALL( Table 4 Corret Classifiation Rates for Heart Disease Data et (% u of eleted Feature MIF MIF-U RMR MIF MIF_U C-MIF-U ALL(3 Heart Disease Data et Table 4 reports the orret lassifiation rates for different nubers of the seleted features on Heart Disease data set. It shows that C- MIF-U perfors better than other riteria in ost ases. 4. COCLUIO Mutual inforation as a heuristi easure of relevane has been broadly used in feature seletion. In this paper, a new utual inforation, alled the survival Cauhy-hwartz utual inforation (C-MI, is define based on the ross survival inforation potential (CIP and Cauhy-hwartz divergene (CD. This utual inforation an be diretly estiated fro saple data without density estiation and hoie of free paraeters. Experiental results on two data sets with VM lassifier suggest that the feature seletion riteria based on C-MI ay perfor very well in lassifiation tasks. REFERECE [] G. Brown, A. Pook, M. J. Zhao, and M. Luján, Conditional likelihood axiization: A unifying fraework for inforation theoreti feature seletion, The Journal of Mahine Learning Researh, vol. 3, pp. 7 66, 0. [] R. Battiti, Using utual inforation for seleting features in supervised neural net learning, IEEE Trans. eural etw., vol. 5, no. 4, pp , Jul [3]. Kwak and C.-H. Choi, Input feature seletion for lassifiation probles, IEEE Trans. eural etw., vol. 3, no., pp , Jan. 00. [4] H. Peng, F. Long, and C. Ding, Feature seletion based on utual inforation: Criteria of ax-dependeny, ax-relevane and in-redundany, IEEE Trans. Pattern Anal. Mah. Intell., vol. 7, no. 8, pp. 6 38, Aug [5] J. ovoviova, P. ool, M. Haindl, P. Pudil, Conditional Mutual Inforation Based Feature eletion for Classifiation Task, Progress in Pattern Reognition, Iage Analysis and Appliations, Leture otes in Coputer iene, vol. 4756, pp ,

5 [6] P. A. Estevez, M. Teser, C. A. Perez, and J. M. Zurada, oralized Mutual Inforation Feature eletion, IEEE Trans. eural etw., vol. 0, no., pp. 89 0, 009. [7] Cover, T. M., Thoas, J. A., Eleent of Inforation Theory, Chihester, U.K.: Wiley, 99. [8] M. Rao, Y. Chen, B. C. Veuri, and F. Wang, Cuulative residual entropy: A new easure of inforation, IEEE Trans. Inf. Theory, vol.50, no. 6, pp. 0 8, 004. [9] F. Wang, B. C. Veuri, on-rigid ulti-odal iage registration using ross-uulative residual entropy, International Journal of Coputer Vision, vol. 74, no., pp. 0 5, 007. [0] K. Zografos and. adarajah, urvival exponential entropies, IEEE Trans. Inf. Theory, vol. 5, no. 3, pp , 005. [] B. Chen, P. Zhu, and J. C. Prinipe, urvival Inforation Potential: A ew Criterion for Adaptive yste Training, IEEE Trans. ignal Proess., vol. 60, no. 3, pp , 0. [] Prinipe, J. C., Inforation Theoreti Learning: Renyi s Entropy and Kernel Perspetives, pringer, ew York, 00. [3] [4]