Estimating Mutual Information Using Gaussian Mixture Model for Feature Ranking and Selection

Transcription

1 Estiating utual Inforation Using Gaussian ixture odel for Feature Ranking and Seletion Tian Lan, Deniz Erdogus, Uut Ozerte, Yonghong Huang Abstrat Feature seletion is a ritial step for pattern reognition and any other appliations. Typially, feature seletion strategies an be ategorized into wrapper and filter approahes. Filter approah has attrated uh attention beause of its flexibility and oputational effiieny. Previously, we have developed an ICA-I fraework for feature seletion, in whih the utual Inforation (I) between features and lass labels was used as the riterion. However, sine this ethod depends on the linearity assuption, it is not appliable for an arbitrary distribution. In this paper, exploiting the fat that Gaussian ixture odel (G) is generally a suitable tool for estiating probability densities, we propose ethod for feature ranking and seletion. We will disuss the details of algorith and deonstrate the experiental results. We will also opare the ethod with the ICA-I ethod in ters of perforane and oputational effiieny. F I. ITRODUCTIO EATURE seletion and diensionality redution is an iportant proble for pattern reognition and any other appliations. For exaple, in ouniation, transitting low diensional data that ontains ost of the inforation is ore desirable than diretly sending the original high diensional ounterpart due to bandwidth liitations. In pattern reognition ontext, feature seletion and diensionality redution an address the salient features, and eliinate the irrelevant features; hene, inrease the robustness and iprove the generalization perforane of the lassifiation syste. Speifially, in geographial and bioedial signal proessing, the diension of feature spae an be hundreds or thousands, and it is ipratial to analyze these data diretly without a diensionality redution that iproves generalization. Diensionality redution an be ahieved by subspae projetion or feature seletion. In subspae projetion, the Tian Lan is with the Bioedial Engineering Departent, OGI Shool of Siene and Engineering, Oregon Health and Siene University, Beaverton, OR USA, (phone: ; e-ail: lantian@be.ogi.edu). Deniz Erdogus is with the Coputer Siene and Eletrial Engineering Departent and Bioedial Engineering Departent, OGI Shool of Siene and Engineering, Oregon Health and Siene University, Beaverton, OR USA, (e-ail: derdogus@ieee.org). Uut Ozerte is with the Coputer Siene and Eletrial Engineering Departent, OGI Shool of Siene and Engineering. Oregon Health and Siene University, Beaverton, OR USA. (e-ail: ozerteu@see.ogi.edu). Yonghong Huang is with the Coputer Siene and Eletrial Engineering Departent, OGI Shool of Siene and Engineering. Oregon Health and Siene University, Beaverton, OR USA. (e-ail: huang@see.ogi.edu). original features are projeted linearly or non-linearly to a low diensional spae, whih represents ajor statistis of the data. There are any existing subspae projetion ethods, suh as PCA, ICA and LDA [1-5]. However, the projetions that PCA and ICA seek are not neessarily related to the lassifiation perforane, hene are not neessarily useful in pattern reognition. LDA overoes this shortoing by finding the projetions that axiize lass separability. On the other hand, this ethod relies on a Gaussian distribution assuption; so it is not appliable for an arbitrary distribution. Lan et al. developed a subspae projetion fraework, whih applies linear ICA transforation and utual inforation axiization for diensionality redution in EEG signal lassifiation [6]. This ethod exhibits several advantages, suh as it is oputationally effiient and flexible, and it is also suitable for high diensional data; however, the linearity assuption essentially resulting fro ICA liits its appliations. Although subspae projetion an effetively reove the redundant features, the relationship between the projeted features and the original features beoes vague. In soe appliations, suh as ulti-sensor array target detetion, and EEG signal proessing, the syste an only transit and proess signals fro a ertain nuber of sensors in real-tie, due to the liitation of bandwidth and oputation apaity. In these partiular ases, feature seletion is ore suitable, whih selets a subset fro the original feature spae. It is widely aepted that soe lassifiation algoriths, suh as deision tree, ulti-layer pereptron neural networks have inherent ability to fous on relevant features and ignore irrelevant ones [7]. In general, feature seletion is ahieved by a feature ranking proedure. Feature seletion ethods an be divided into wrapper and filter approahes. Wrapper approah uses lassifiation auray as riterion oupled with a speifi lassifier, whih requires re-training the lassifier for different obinations of feature sets; hene, it is slow and inflexible. Filter approah, on the other hand, ranks and selets features by optiizing soe riteria independent of the lassifier, and is ore flexible and suitable for adaptive learning. In the filter approah, it is iportant to optiize a riterion that is relevant to Bayes risk, whih is typially easured by the probability of error. A suitable riterion is the I between the seleted features and the lass labels, otivated by lower and upper bounds in inforation theory that relate this quantity to probability of error [8,9]. As opposed to linear and seond-order statistis suh as orrelation and ovariane, I

2 easures non-linear dependenies between a set of rando variables taking into aount higher order statistial strutures existing in the data. any feature seletion ethods have been developed in the past years [10-12]. Guyon & Elisseeff also reviewed several approahes used in ahine learning ontext [13]. In our previous work, we have proposed an ICA-I ethod for feature seletion, and applied it on EEG hannel seletion [14]. This ethod exploits the fat that an invertible linear transforation does not hange the I, and assues that linear ICA transforation yields independent features, so that the I between feature vetors and lass labels an be onveniently estiated by the diret suation of I between eah independent projeted feature vetor and lass labels. However, sine the auray of this ethod highly depends on the perforane of linear transforation, it is not appliable for arbitrary distribution. Atually, if we know the distribution of the feature vetors, we an diretly estiate the I between feature vetors and lass labels by definition. There are several density estiation ethods, suh as histogra, G, and Kernel density estiation (KDE). G is widely used beause (1) it is a ore powerful tool as opared to paraetri estiators that only an estiate a faily of density funtions; (2) it results in a ontinuously differentiable estiation, whih is appropriate for gradient based adaptive learning approahes; (3) it is less oputationally intensive as opared with KDE. This otives us to use G estiating I for feature seletion. In this paper, we propose a ethod for feature ranking and seletion. In the next setion, we will disuss the algorith in detail. In the experiental result part, we apply this ethod on several datasets of UCI ahine learning repository, and EEG dataset olleted by Honeywell for the AugCog projet. We also opare this ethod with the previous ICA-I ethod in ters of auray and speed. II. ALGORITH The goal of feature seletion is to iprove the generalization perforane of the lassifiation syste by seleting the inforative features, without oproising lassifiation auray by throwing away oponents. Therefore the feature seletion riterion ust iniize the Bayes risk, whih typially is lassifiation error in pattern reognition proble. The average probability of error has been shown to be related to I between the feature vetors and the lass labels. Speifially, Fano s and Hellan & Raviv s bounds deonstrate that probability of error is bounded fro below and above by quantities that depend on the Shannon I between these variables [8, 9]. axiizing this I redues both bounds, therefore, fores the probability of error to derease. A siilar result was also obtained by Erdogus & Prinipe using Renyi s I; a paraetri faily of lower and upper bounds for the probability of error was provided [15,16]. Hellan & Raviv [7] showed that the upper bound on Bayes error is given by (H S ()-I S (x,))/2, where H S () is the Shannon entropy of the a priori probabilities of the lasses and I S (x,) is the Shannon I between the ontinuous-valued feature vetors and the disrete-valued lass labels. Consequently, axiizing the I between the seleted features and the lass labels potentially iproves lassifiation perforane, and has drawn uh attention [17, 18]. Shannon I between feature vetors x and is defined in ters of the entropies of the overall data and individual lasses as I S ( x; ) = H S ( x ) p H S ( x ) (1) where p are the prior lass probabilities. The entropy is given by H S ( x) = p( x) log p( x) dx (2) H S ( x ) = p( x ) log p( x ) dx where p(x ) are the lass onditional distributions and the overall data distribution is p ( x) = p p( x ) (3) Above equations show that the ritial step for this feature seletion ethod is the entropy estiation. Previously in ICA-I ethod, we estiate entropy by an indiret ethod. While in ethod, sine one an approxiate an arbitrary distribution by liited nuber of Gaussian oponents with suffiient aount of data, one an estiate entropy diretly by definition (2). The G density estiation an be written as: (4) p ( x ) = α G ( x ) = 1 where G (x) is the distribution of eah Gaussian oponent, and α is the orresponding oponent prior. So the estiation of overall entropy an be written as: H S ( x ) = log α G ( xi ) (5) i = 1 = 1 where the lass onditional entropy is given by: (6) H S ( x ) = log α G ( xi ) i = 1 = 1 where is the overall data saples and C is the data saples for lass C, x is the data saples fro lass C. Cobining (1), (5) and (6), the I estiation an be written as: I ( x; ) = log α G ( xi ) (7) i= 1 = 1 + p log α G ( x i ) i= 1 = 1 Using the I estiation given by (5)-(7), the feature ranking algorith an be desribed as Proedure 1: A. Estiate both lass densities and overall density for eah feature vetor using G separately. B. Estiate the I between eah feature vetor and lass labels. Find the feature with axiu I, and ark it as opt-sub1 (optial subset of 1 feature). C. Selet one of the reaining feature vetors, obine it with opt-sub1 to for sub2 (subset of 2 features). Estiate both lass density and overall density of sub2, and then estiate I between sub2 and lass

3 labels. Repeat this proess for all reaining features, find the features with axiu I, and ark the as opt-sub2. D. Repeat Step C by inreasing one feature at a tie, until all features are ranked in the sense of I axiization. This proedure results in an ordering of features suh that the first d features have axial I with lass labels. The hoie of d to be used in the appliation is dependent on the requireent for lassifiation perforane and oputational ost. Using this searh strategy, the oputational oplexity is (n+1)n/2 (n is the total nuber of features) instead of the 2 n of exhaustive evaluation. In Proedure 1, G with ertain nuber of oponents is fitted to the data fro using the Expetation-axiization algorith [19]. This G fitting is required for all obinations of feature vetors. What s ore, to deterine the optial nuber of oponents, we apply ross-validation and use several restarts to ahieve axiu likelihood. Therefore, Proedure 1 is tie-onsuing. We ipleented Proedure 1 with atlab on Dell Preision 370 with single P4 2.8G CPU, 1GB eory. The training datasets ontain 30 diensional EEG signals, with about 300 saples. The whole feature ranking proedure took about 125 hours, whih akes algorith alost ipratiable in real world appliations. To overoe this diffiulty, we first use spherial ovariane atrix, and assue that the nuber of optial oponents for all features is idential to that for different obinations of feature subsets. In this way, one only needs to do the ross-validation one for all features at the beginning, and pik rows and oluns fro the ean vetors and ovariane atries for the orresponding features. Under this assuption, the algorith is revised as proedure 2: A. Use ross-validation to deterine the optial nuber of oponents for eah lass and overall data for all feature vetors. Estiate both lass densities and overall density for all feature vetors using G. Get the ean vetors and ovariane atries for eah oponent. B. Pik the orresponding rows and oluns fro ean vetors and ovariane atries (generated in A), and estiate density for eah feature vetor. Estiate the I between eah feature vetor and lass labels. Find the feature with axiu I, and ark it as opt-sub1 (optial subset of 1 feature). C. Pik one in the reaining feature vetors, obine it with opt-sub1 to for sub2 (subset of 2 features). Find the orresponding rows and oluns fro ean vetors and ovariane atries (generated in A), for the new ean vetors and ovariane atries, and estiate both lass densities and overall density of sub2, and then estiate I between sub2 and lass labels. Repeat this proess for all reaining features, find the features with axiu I, and ark it as opt-sub2. Repeat Step C by inreasing one feature at a tie, until all features are ranked in the sense of I axiization. We ipleented Proedure 2 on the sae software and hardwire platfor. The training datasets are idential to above experient. Resulting in a better oputational effiieny, the whole feature ranking proedure took about 10 inutes for this siulation. In the next setion, we will show the different feature ranking results fro both proedures on the sae training set. III. EXPERIETS AD RESULTS In this setion, we will show experiental results by applying the proposed algorith on Iris data and Wisonsin breast aner data fro UCI ahine learning repository [20]. We also ipleented the algorith on EEG data olleted by Honeywell in AugCog projet. As a oparison, we also ipleented the previous ICA-I algorith on the sae datasets. A. UCI ahine learning repository In this experient, we applied both and ICA-I on Iris and Wisonsin breast aner datasets. For experients using, the applied proedure that an be desribed as: 1. Randoly split data into training and testing sets. 2. Use training data to fit G by 5-fold ross-validation. Use the trained G odel, together with training data to do feature ranking (If not entioned, proedure 2 is used throughout the siulations). 3. Use the trained G odel to for a paraetri Bayes lassifier, and use this lassifier to do the lassifiation on the ranked features. 4. Repeat 1-3 for 10 onte Carlo runs, reord the feature ranking indies for eah tie, and average the lassifiation auray over 10 ties. For experients using ICA-I, the only differene is the seond step, in whih the ICA-I algorith was applied to rank the features. The feature ranking results for both datasets are shown in Table I and II, and the lassifiation auraies in Fig. 1 and 2. As a oparison, we also apply the Bayes error based wrapper approah for feature ranking. The results are shown in Fig. 1 and 2. Experiental results show that in Iris and Wisonsin breast aner data, exhibits ore auray than ICA-I. Visualizing eah feature in iris data, feature 3 and 4 exhibit uh higher separability than feature 1 and 2. Table I shows that reflets this fat without any proble; while ICA-I yields inaurate results (rank feature 4 as the least iportant feature). The ranking results of Wisonsin breast aner (Table II) are onsistent with the separability of the eah feature. The lassifiation auray urve in Fig. 1 and 2 also deonstrates that works better than ICA-I for these two datasets. Despite the better perforane, is very slow opared with ICA-I (about 100 ties slower aording to Table I and II). Fig. 1 and 2 also show the feature ranking results using Bayes error as riterion. Obviously, the results are better than and ICA-I ethods, beause this ethod is optial to seleted G

4 TABLE I FEATURE RAKIG RESULTS FOR IRIS DATA Average CPU tie (seond) Ranking indies (9 ties) (1 tie) ICA-I (10 ties) The seond row shows the feature ranking results by ethod; the third row shows feature ranking results by ICA-I ethod. The seond olun shows the run-tie for two ethods (based on P4 2.8G CPU). These nubers only give the reader a onept about the effiieny of two ethods. The third olun shows the feature indies for 10 onte Carlo runs. Classifiation rate Classifiation rate vs. nuber of best features ICA-I Bayes error based TABLE II FEATURE RAKIG RESULTS FOR WISCOSI BREAST CACER DATA Average CPU tie (seond) Ranking indies ICA-I The seond row shows the feature ranking results by ethod; the third row shows feature ranking results by ICA-I ethod. The seond olun shows the run-tie for two ethods (based on P4 2.8G CPU). These nubers only give the reader a onept about the effiieny of two ethods. The third olun shows the feature indies for 10 onte Carlo runs n best features Fig. 1. Classifiation auray for Iris data by, ICA-I and Bayes error based algoriths. The lassifiation auray is the average over 10 onte Carlo siulations Classifiation rate vs. nuber of best features lassifier. However, sine it requires obinational learning, it is uh slower than both and ICA-I. And the lassifiation results obtained by ranked feature have only slight differene opared with error based approah. B. EEG dataset We also applied the proposed ethod on EEG dataset olleted by Honeywell in Augented Cognition (AugCog) projet. The ai of AugCog projet is to enhane the task-related perforane of a huan user through oputer ediated assistane based on assessents of ognitive state fro EEG signals. In this experient, the subjet exeuted the predefined tasks, whih orrespond to the different level of brain ativities (high and low). The EEG data was pre-proessed by reoving the usular artifats, filtering out irrelevant frequeny bands. Power Spetru Density features are extrated in 30 diensions (for ore inforation about AugCog projet, please refer to [6, 14, 21]). The experiental proedure is siilar to that for Iris and Wisonsin breast aner datasets, exept for two differenes: 1) in AugCog projetion, we have two data files, one is used as training, the other is used as testing; 2) we did not use onte Carlo proedure. The ranking results and lassifiation auray are shown in Table III and Fig. 3. As referene, we also list the ranking results by Proedure 1 entioned in setion II. 1) The experiental results on EEG dataset also validate that ICA-I is uh faster than. Fro perforane point of view, both and ICA-I Classifiation rate ICA-I Bayes error based n best features Fig. 2. Classifiation auray for Wisonsin breast aner data by, ICA-I and Bayes error based algoriths. The lassifiation auray is the average over 10 onte Carlo siulations. exhibit ertain degree of onsisteny. However, none of the is superior to the other: if we only want to selet 3-5 features, ICA-I yields better perforane. If we want to selet features, yields better perforane. For the ICA-I algorith, as we entioned before, linear assuption ight degrade the auray of I estiation. For algorith, there ould be two reasons: 1) by using Proedure 2 to replae Proedure 1, we assue that the optial nubers of oponents are idential for different obination of features, this ight not hold in soe ases; 2) we do not have enough data. Consider we are working on 30 diensions, but we only use about 300 data saples to fit G odel

5 TABLE III FEATURE RAKIG RESULTS FOR AUGCOG EEG DATA Average CPU Ranking indies tie (seond) (Proedure 2) ICA-I 2.86 (Proedure 1) About 125 hours The seond row shows the feature ranking results by ethod; the third row shows feature ranking results by ICA-I ethod. the fourth row also show the feature ranking results by using ethod proedure 1. The seond olun shows the run-tie for two ethods (based on P4 2.8G CPU). These nubers only give the reader a onept about the effiieny of two ethods. The third olun shows the feature indies for 10 onte Carlo runs. Classifiation rate Classifiation rate vs. nuber of best features n best features IV. COCLUSIO ICA-I Fig. 3. Classifiation auray for EEG data by and ICA-I algoriths. In this paper, we proposed a feature ranking and seletion ethod: algorith. This ethod exploits the fat that G is a widely aepted density estiator for an arbitrary distribution, and an be used for entropy estiator. In setion II, this ethod is desribed in detail, and an approxiation is used for reduing the oputational requireent. Experients on Iris, Wisonsin breast aner, and EEG data show that an partly overoe the non-linear proble that ICA-I has. However, beause it is uh slower than ICA-I, and its auray will be ipaired in higher diension when there is no enough training data, this ethod an be used as a suppleent to our existing ethod. We also show the feature ranking results by using Bayse error riterion for Iris and Wisonsin breast aner data, experiental results show that this ethod is optial to seleted G lassifier; however, it is too slow to use in real tie AugCog appliation. Knowing the shortoings of both and ICA-I algoriths, the future work will fous on two aspets: 1) iproving the perforane of ICA transforation, in order to inrease the auray of I estiation; 2) aelerating the G algorith to redue the oputational requireent for G fitting. ACKOWLEDGET This work was supported by DARPA under ontrat DAAD C-0054 and by SF under grant ECS The EEG data was olleted at the Huan-Centered Systes Lab., Honeywell, inneapolis, innesota. REFERECES [1] E. Oja, Subspae ethods of Pattern Reognition, Wiley, ew York, [2] P.A. Devijver, J. Kittler, Pattern Reognition: A Statistial Approah, Prentie Hall, London, [3] K. Fukunaga. Introdution to Statistial Pattern Reognition, 2 nd ed., Aadei Press, ew York, [4] R. Everson, S. Roberts. Independent Coponent Analysis: A Flexible onlinearity and Deorrelating anifold Approah, eural Coputation, vol. 11, no. 8, pp , [5] A. Hyvärinen, E. Oja, P. Hoyer, J. Hurri, Iage Feature Extration by Sparse oding and Independent Coponent Analysis,Proeedings of ICPR 98, pp , [6] T. Lan, D. Erdogus, A. Adai,. Pavel, Feature Seletion by Independent Coponent Analysis and utual Inforation axiization in EEG Signal Classifiation, Proeedings of IJC 05, ontreal, Canada, pp , Aug [7] W. Duh, T. Wiezorek, J. Biesiada,. Blahnik, Coparison of feature ranking ethods based on inforation entropy, Pro. of International Joint Conferene on eural etworks (IJC), Budapest 2004, IEEE Press, pp [8] R.. Fano, Transission of Inforation: A Statistial Theory of Couniations. Wiley, ew York, [9].E. Hellan, J. Raviv, Probability of Error, Equivoation and the Chernoff Bound, IEEE Transations on Inforation Theory, vol. 16, pp , [10] Battiti, R., Using utual Inforation for Seleting Features in Supervised eural et Training, IEEE Trans eural etworks, vol 5, no 4, pp July [11] Kira, K. and Rendell,L., The feature seletion proble: Traditional ethods and a new algorith, In Proeedings of the Tenth ational Conferene on Artifiial Intelligene (AAAI-92), pages , enlo Park, CA, USA, AAAI Press. [12] John,G.H., Kohavi,R., & Pfleger,K., Irrelevant features and the subset seletion proble. In Proeedings of the 11th International Conferene on ahine Learning, pp , San ateo, CA, organ Kaufann, [13] Guyon,I., and Elisseeff, A., An Introdution to Variable and Feature Seletion, Journal of ahine Learning Researh (Speial Issue on Variable and Feature Seletion), [14] T. Lan, D. Erdogus, A. Adai,. Pavel, S. athan, Salient EEG Channel Seletion in Brain Coputer Interfaes by utual Inforation axiization, Proeedings of EBC 05, Shanghai, China, Sept [15] D. Erdogus, Inforation Theoreti Learning: Renyi s Entropy and its appliations to Adaptive Syste Training, PhD Dissertation, University of Florida, [16] D. Erdogus, J.C. Prinipe, Lower and Upper Bounds for islassifiation Probability Based on Renyi s Inforation, Journal of VLSI Signal Proessing Systes, vol. 37, no. 2/3, pp , [17] K. Torkkola, Feature Extration by on-paraetri utual Inforation axiization, Journal of ahine Learning Researh, vol. 3, pp , [18] R. Battiti, Using utual Inforation for Seleting Features in Supervised eural etworks learning, IEEE Trans. eural etworks, vol. 5, no. 4, pp , 1994.

6 [19] A.P. Depster,.. Laird, D.B. Rubin, axiu Likelihood fro Inoplete Data via the E Algorith, Journal of the Royal Statistial Soiety, vol. 39, pp. 1-38, [20] [21] T. Lan, A. Adai, D. Erdogus,. Pavel, Estiating Cognitive State Using EEG Signals, Proeedings of EUSIPCO 05, Sep 2005.