Feature Selection by Independent Component Analysis and Mutual Information Maximization in EEG Signal Classification

Transcription

1 Feature Seletion by Independent Component Analysis and Mutual Information Maximization in EEG Signal Classifiation Tian Lan, Deniz Erdogmus, Andre Adami, Mihael Pavel BME Department, Oregon Health & Siene University, Beaverton, Oregon 97006, USA Abstrat Feature seletion and dimensionality redution are important steps in pattern reognition. In this paper, we propose a sheme for feature seletion using linear independent omponent analysis and mutual information maximization method. The method is theoretially motivated by the fat that the lassifiation error rate is related to the mutual information between the feature vetors and the lass labels. The feasibility of the priniple is illustrated on a syntheti dataset and its performane is demonstrated using EEG signal lassifiation. Experimental results show that this method works well for feature seletion. Keywords Feature Seletion, Independent Component Analysis, Mutual Information, Entropy Estimation, EEG, Brain-Computer Interfae I. INTRODUCTION Feature seletion and dimensionality redution are important steps in pattern reognition tasks and many other appliations. In pratie, the relevant information about the data struture an often be represented by a lower dimensional manifold embedded in the original Eulidian data spae. Speifially, in pattern reognition, a high dimensional feature vetor is available, but usually the lassifiation task an be ahieved equally well by a feature vetor of redued dimensionality. Furthermore, reduing the number of features will also help the lassifier learn a more robust solution and ahieve a better generalization performane. This is due to the fat that irrelevant feature omponents are eliminated by the optimal subspae projetion. Dimensionality redution by subspae projetion is typially ahieved by feature transformation methods. This transformation generates either a new feature spae, or a subset of the original feature spae, whih an be treated as a speial ase of the former situation. The transformation an be linear or non-linear. Linear transformations have been widely used due to their simpliity. While nonlinear transformations attrat inreasingly more attention due to their ability to apture the nonlinear relationships within the data, the omplexity of finding robust regularized nonlinear transformations makes them a seond hoie in most of appliations. In this paper, we will fous on linear transformations leaving the nonlinear transformations for future study. There are many existing linear transformation methods for dimensionality redution. Priniple omponent analysis (PCA) is a widely used dimensionality redution tehnique [,2]. However, sine the projetions it finds are not neessarily related to the lass labels, it is not partiularly useful in pattern reognition. Linear disriminant analysis (LDA) attempts to eliminate this shortoming of PCA by finding linear projetions that maximize lass separability under the Gaussian distribution assumption [3]. The LDA projetions are optimized based on the means and the ovariane matries of lasses, whih are not desriptive of an arbitrary probability density funtion (pdf). Independent omponent analysis (ICA) has also been used as a tool to find linear transformations that maximize the statistial independene of random variables [4,5]. However, like PCA, the projetion that ICA finds also has no neessary relationship with lass labels, and it is not able to enhane lass separability [6]. Optimal feature seletion oupled with a speifi lassifier topology, namely the wrapper approah, results in a ombinatorial omputational requirement; thus, is unsuitable for adaptive learning of feature projetions. On the ontrary, the filter approah, whih selets features by optimizing some riterion is independent of the lassifier, hene is more flexible. In the filter approah, it is important to optimize a riterion that is relevant to Bayes risk, whih is typially measured by the probability of error. A suitable riterion is mutual information (MI) between the projeted features and the lass labels, whih is motivated by lower and upper bounds in information theory that relate this quantity to probability of error [7,8]. In priniple, MI measures nonlinear dependenies between a set of random variables taking into aount higher order statistial strutures existing in the data, as opposed to linear and seond-order statistial measures suh as orrelation and ovariane. Several MI based methods have been developed for feature seletion [9-3]. Estimating MI requires the knowledge of joint pdf of the data in feature spae. Evaluating the MI between two salar random variables (one being the disrete lass labels) using histograms is studied in literature [9,3]. However, this approah fails when dealing

2 Original Feature x ICA Projetion (W) Independent Features y Class Labels MI-Based Sorting I(y i ;) with high dimensional variables. Torkkola [6] proposed an approah using a quadrati divergene measure to find an optimal transformation that maximizes the MI between features and lass labels. This approah, being dependent on Parzen density estimation, is ineffiient for subspae projetions to high dimensionalities due to the joint density estimation requirement. A shortoming of existing MI-based feature seletion methods is that, sine features are generally mutually dependent, feature seletion in this manner is typially suboptimal in the sense of maximum joint mutual information priniple. In pratie, the mutual information must be estimated nonparametrially from the training samples [4]. Although this is a hallenging problem for multiple ontinuous-valued random variables, the lass labels are disrete-valued in the feature transformation setting. This redues the problem to just estimating entropies of ontinuous random vetors. Furthermore, if the omponents of the random vetor are independent, the joint entropy beomes the sum of marginal entropies. Thus, the joint mutual information of a feature vetor with the lass labels is equal to the sum of marginal mutual information of eah individual feature with the lass labels, provided that the features are independent. In this paper, we exploit this fat by ombining independent omponent analysis (ICA) preproessing with a sample-spaing based entropy estimator [5] for feature seletion (see Fig. ). The ontributions of this paper are: (i) theoretial motivation of feature seletion using ICA preproessing and marginal mutual information sorting, (ii) a omputationally effiient training algorithm for this paper that employs a fast analytial solution for ICA and simple and onsistent samplespaing estimators for mutual information, (iii) the appliation of this tehnique to the lassifiation of EEG signals for ognitive load assessment. II. THEORETICAL BACKGROUND Low-Dim Features f Fig.. Feature seletion using ICA preproessing and mutual information sorting The goal of feature subspae projetions is to improve lassifier robustness by reduing data dimensionality in order to failitate better generalization, as well as reduing the learning and operating omplexity of the lassifiers. While doing so, lassifiation performane must not be ompromised by throwing away omponents that provide useful information regarding the lass labels. Theoretially, optimal feature projetions should minimize the Bayes risk funtion for the given problem; the average probability of error is a widely used and aepted risk funtion and merits speial attention. For different risk funtions, the following theoretial and pratial results an be easily modified. The average probability of error has been shown to be related to MI between the feature vetors and the lass labels. Speifially, Fano s and Hellman & Raviv s bounds demonstrate that probability of error is bounded from below and above by quantities that depend on the Shannon MI between these variables [7,8]. Maximizing this MI redues both bounds, therefore, fores the probability of error to derease. A similar result was also obtained by Erdogmus & Prinipe using Renyi s MI; a parametri family of lower and upper bounds for the probability of error was provided [6,7]. Hellman & Raviv [7] showed that the upper bound on Bayes error is given by (H S (C)-I S (Y,C))/2, where H S (C) is the Shannon entropy of the a priori probabilities of the lasses and I S (Y,C) is the Shannon MI between the ontinuousvalued feature vetor and the disrete-valued lass label. Consequently, maximizing the MI between the projeted features and the lass labels potentially improves lassifiation performane, and has drawn muh attention [6,9-2]. Mutual information was first introdued by Shannon in the ontext of digital ommuniations between disrete random variables and was generalized to ontinuous random variables. In feature extration, we are interested in the MI between the ontinuous-valued feature vetor y and the disrete-valued lass labels. Shannon MI between y and is defined in terms of the entropies of the overall data and the individual lasses as I S ( y ; ) = H S ( y) p H S ( y ) () where p are the prior lass probabilities. The entropy is given by H S ( y) = y) log y) dy (2) H S ( y ) = y ) log y ) dy where y ) are the lass onditional distributions and the overall data distribution is p ( y ) = p y ) (3) Assume that features y are mutually independent, we have: n I S (y; ) = I S ( yi ; ) (4) i= where I S ( yi ; ) = H S ( yi ) H S ( yi ), and y i is the i th omponent of feature spae. There exist a number of entropy estimators for onedimensional variables. Here, we will use the sample-spaing estimator for its simpliity. The independene assumption an be aquired by ICA transformation. After that, the MI between eah feature and the lass labels, I S (y,),,i S (y n,) an be estimated by the

3 projeted data samples. We rank I S (y i,) aording to the value, and hoose the m features with largest MI that aount for the majority of the total MI between the feature vetor and the lass label. In priniple, any ICA algorithm followed by any MI estimator ould be employed during the feature seletion proedure desribed above. In the next setion, we disuss the speifi ICA transformation and MI estimator that are employed in our experiments. III. ICA TRANSFORMATION AND MI ESTIMATION ICA Using Generalized Eigenvalue Deomposition: The square linear ICA problem is expressed in (5), where X is the n N observation matrix, A is the n n mixing matrix, and S is the n N independent soure matrix. X = AS (5) Eah olumn of X and S represents one sample of data. If we onsider eah olumn as a sample in time, (5) beomes: x ( t) = As( t) (6) Many effetive and effiient algorithms based on a variety of assumptions inluding maximization of nongaussianity, minimization of mutual information, nonstationarity of the soures, et. exist to solve this ICA problem [4,5,8]. All these ould be ompatly formulated in the form of a generalized eigendeomposition problem that gives the ICA solution in an analytial form [9]. Therefore, this formulation reviewed by Parra & Sajda in [9] will be employed in this paper. Aording to this formulation, one possible assumption set that leads to an ICA solution utilizes the higher order statistis (speifially fourth-order umulants). Under this set of assumptions, the separation matrix W is the solution to the following generalized eigendeomposition problem: R x W = Q x WΛ (7) where R x is the ovariane matrix and Q x is the umulant matrix estimated using sample averages: Q x =E[x T xxx T ]- R x tr(r x )-E[xx T ]E[xx T ]-R x R x. Given the estimates for these matries, the ICA solution an be easily determined using effiient generalized eigendeomposition algorithms (or using the eig ommand in Matlab). Estimating MI Using Sample-Spaings: Reall that in the ase of feature seletion for lassifiation, the mutual information estimation redues to the sum of marginal and onditional entropies as shown in () and (4). Therefore, we only need to estimate marginal entropies. There exist many entropy estimators in the literature for single-dimensional variables. Here, we use an estimator based on samplespaings, whih stems from order statistis. This estimator is seleted beause of its onsisteny, rapid asymptoti onvergene, and simpliity. Consider a one dimensional random variable Y. Given a set of iid samples of Y {y,,y N }, first these samples are sorted in inreasing order suh that y () y (N). The m- spaing entropy estimator is given by: N m N + y i+ m y i H ˆ ( )( ( ) ( ) ) ( Y ) = log N m i= m (8) This estimator uses two assumptions: the true density y) is approximated by a pieewise uniform density determined by m-neighbor distanes and outside of the sample range, the true density has the same mean log probability density as the rest of the distribution. The seletion of the parameter m is determined by a biasvariane trade-off and typially m = N. In general, for asymptoti onsisteny the sequene m(n) should satisfy lim m( N) = lim m( N) / N = 0 (9) N N IV. EXPERIMENTS AND RESULTS Syntheti Dataset: In order to illustrate the feasibility and the performane of the proposed feature seletion method, we apply it to a simple syntheti dataset. This problem onsists of lassifying two one-dimensional Lapalaian lasses where a seond onfusing irrelevant Gaussian feature is introdued. Speifially, The 2-dimensional feature vetor x is a random linear ombination (determined by a matrix A) of the 2 independent features s and s 2, where s obeys the distribution given in (0) determining the lass labels ompletely and s 2 is redundant zero-mean unit-variane Gaussian noise independent from the lass label. s ~ p f( s) + ( p) f 2 ( s ) (0) The lass distributions are f ( s) = ex 2 s ± / σ ) () 2σ A Monte Carlo experiment is performed with p=0.5, σ varying from 0. to 2, and the number of training samples seleted as 0 2, 0 3, and 0 4. For eah ombination the following proess is repeated 00 times: a random mixing matrix A is seleted (eah entry uniform in [0,]), a new training set is generated with the speified number of samples, and a new testing set of 0 6 samples is generated. The ICA solution and MI-based feature seletion are performed using the training data, and a simple threshold lassifier (also determined from the training data) is employed on the test data. For referene, the true optimal Bayes lassifier (simple threshold of zero on s ) is also applied to the test data in every ase. The results averaged over the 00 Monte Carlo runs are shown in Fig. 2. As expeted, the performane approahes the theoretial optimal as the training set size inreases. In order to evaluate the performane of seleted ICA transformation and the feature seletion method, we introdue a parameter: osα, whih is defined as: osα = α T e / α (2)

4 where α T =e T WA is the atual seletion matrix, and e T is the ideal seletion matrix with value [,0] T or [0,] T. Ideally, we expet the value of osα to be as lose as when number of training samples inreases. Fig. 3 shows the osα value averaged over 00 Monte Carlo runs. As we expeted, the value of osα keeps unhanged for different σ. However, unexpetedly, it does not inrease as the size of the training set inreases. Cognitive State Classifiation Using EEG Signals: In this example, the proposed method for feature seletion is applied to the lassifiation of ognitive state using features extrated from EEG signals olleted while the subjet performs a mental task. The data is olleted as part of an augmented ognition projet, in whih the estimated ognitive state is used to assess the mental load of the subjet in order to modify the interation of the subjet with a omputer system with the goal of inreasing user performane. In this experimental setup, the EEG signals measured at 256Hz by seven eletrodes loated at salient sites (CZ, P3, P4, PZ O2, P04, F7) are used to generate power-spetral features (- seond sliding window integrated over 5 frequeny bands: 4 8Hz, 8 2Hz, 2 6Hz, 6 30Hz, 30 44Hz). The novelty in this appliation is that the subjets are freely moving around in ontrast to the typial brain-omputer interfae (BCI) experimental setups where the subjets are in a stritly ontrolled setting. The assessment of ognitive state in ambulatory subjets is partiularly diffiult, sine the movements introdue strong artifats irrelevant to the mental task/load. The mobility of the operator inreases the omplexity of the design, beause the measurement of the physiologial states is extremely diffiult in situations where the body of the subjet is in motion. Feature seletion beomes important in this task due to its abilities to keep the useful information and eliminate the irrelevant information for lassifiation, in order to inrease the robustness of the lassifiation performane of the system. During data olletion, the subjet is outfitted with the suite of sensors and performs a predetermined set of tasks: slow walking, navigating and ounting, ommuniating with radio, and studying mission map. The EEG data is olleted for training and testing. The whole lassifiation system ontains four parts: preproessing, feature extration and seletion, lassifiation, and postproessing. Preproessing is used to filter out noise and remove the artifats. Feature extration and seletion generates features from the lean EEG signal, and selets useful features using the proposed method. For lassifiation, the K-Nearest-Neighbor (KNN) lassifier is utilized. The postproessing uses the assumption that the variations in ognitive state for a given ontinuous task will be slowly varying in time. A median filter operating on a window of 2-seond deisions reently generated by the lassifier is used to eliminate a portion of erroneous deisions made by the lassifiation system. Using the first /3 of the olleted data for training and the remaining 2/3 for testing, the orret lassifiation rate of the system on the test data over four lasses is shown in Fig. Fig. 2. Classifiation errors vs. σ for different sizes of training sets ompared with the theoretially optimal lassifier. Fig. 3. Cosα vs. σ for different sizes of training sets ompared with the ideal value of. Fig. 4. Corret lassifiation rate (vertial axis) vs. dimensionality of optimally seleted features (horizontal axis).

5 4 for different feature subspae projetion dimensions. An auray of 80% is ahieved with 2 dimensions, while the remaining 23 dimensions do not signifiantly ontribute to the lassifiation auray. These results demonstrate that the proposed method of feature dimensionality redution is able to apture the low-dimensional relevant omponents of the feature vetor. V. CONCLUSIONS In this paper, we presented a feature seletion method based on the maximum mutual information priniple. The tehnique ombines the analytial solution for linear ICA transformations and a omputationally effiient mutual information estimator, by taking into aount the fat that minimization of Bayes lassifiation error an be approximately ahieved by maximizing the mutual information between the features and the lass labels. The linear ICA transformation is used to separate the mixed features into approximately independent features so that single-dimensional mutual information estimation an be onveniently employed. The ICA transformation is determined by solving a generalized eigendeomposition problem, whih is also omputationally effiient and effetive. The urrent method relies on linear ICA, whih does not neessarily yield independent features, whih violates the assumption of additive deomposition of mutual information that we have employed. Future work will expand this tehnique using nonlinear ICA in order to improve performane. Another alternative researh diretion is to use linear independent subspae analysis ombined with effiient joint entropy estimators. Experiments using syntheti and real (EEG) data demonstrate the validity and the effetiveness of the proposed tehnique. The results on the EEG data set reveal the fat that this method is able to determine relevant low-dimensional strutures in data in the broad ontext of brain omputer interfaes. The method exhibits the following appealing properties: Intuitively motivated by information theory. Easy to implement with low omputational requirements. Robust and aurate in lassifier design due to the seletion of salient features. Aknowledgment: This work was supported by DARPA under ontrat DAAD-6-03-C The EEG data was olleted at the Human-Centered Systems Lab., Honeywell, Minneapolis, Minnesota. REFERENCES [] E. Oja, Subspae Methods of Pattern Reognition, Wiley, New York, 983. [2] P.A. Devijver, J. Kittler, Pattern Reognition: A Statistial Approah, Prentie Hall, London, 982. [3] K. Fukunaga. Introdution to Statistial Pattern Reognition, 2nd ed., Aademi Press, New York, 990. [4] R. Everson, S. Roberts, Independent Component Analysis: A Flexible Nonlinearity and Deorrelating Manifold Approah, Neural Computation, vol., no. 8, pp , [5] A. Hyvärinen, E. Oja, P. Hoyer, J. Hurri, Image Feature Extration by Sparse Coding and Iomponent Analysis, Proeedings of ICPR 98, pp , 998. [6] K. Torkkola, Feature Extration by Non-Parametri Mutual Information Maximization, Journal of Mahine Learning Researh, vol. 3, pp , [7] R. M. Fano, Transmission of Information: A Statistial Theory of Communiations. Wiley, New York, 96. [8] M.E. Hellman, J. Raviv, Probability of Error, Equivoation and the Chernoff Bound, IEEE Transations on Information Theory, vol. 6, pp , 970. [9] R. Battiti, Using Mutual Information for Seleting Features in Supervised Neural Networks learning, IEEE Trans. Neural Networks, vol. 5, no. 4, pp , 994. [0] A. Ai-ani, M. Derihe, An Optimal Feature Seletion Tehnique Using the Conept of Mutual Information, Proeedings of ISSPA, pp , 200. [] N. Kwak, C-H. Choi, Input Feature Seletion for Classifiation Problems, IEEE Transations on Neural Networks, vol. 3, no., pp , [2] H.H. Yang, J. Moody, Feature Seletion Based on Joint Mutual Information, in Advanes in Intelligent Data Analysis and Computational Intelligent Methods and Appliation, 999. [3] H.H. Yang, J. Moody, Data Visualization and Feature Seletion: New Algorithms for Nongaussian Data, Advanes in NIPS, pp , [4] K.E. Hild II, D. Erdogmus, J.C. Prinipe, Blind Soure Separation Using Renyi's Mutual Information, IEEE Signal Proessing Letters, vol. 8, no. 6, pp , 200. [5] E.G. Learned-Miller, J.W. Fisher III, ICA Using Spaings Estimates of Entropy,.Journal of Mahine Learning Researh, vol. 4, pp , [6] D. Erdogmus, Information Theoreti Learning: Renyi s Entropy and its appliations to Adaptive System Training, PhD Dissertation, University of Florida, [7] D. Erdogmus, J.C. Prinipe, Lower and Upper Bounds for Mislassifiation Probability Based on Renyi s Information, Journal of VLSI Signal Proessing Systems, vol. 37, no. 2/3, pp , [8] A. Hyvärinen, E. Oja, A Fast Fixed Point Algorithm for Independent Component Analysis, Neural Computation, vol. 9, no. 7, pp , 997. [9] L. Parra, P. Sajda, Blind Soure Separation via Generalized Eigenvalue Deomposition, Journal of Mahine Learning Researh, vol. 4, pp , 2003.