A Hybrid Approach for Modeling High Dimensional Medical Data

Transcription

1 A Hybri Approach for Moeling High Dimensional Meical Data Alok Sharma 1, Gofrey C. Onwubolu 1 1 University of the South Pacific, Fii sharma_al@usp.ac.f, onwubolu_g@usp.ac.f Abstract. his work presents the application of hybri PCA an LDA to moeling high imensional meical ata, which is a real-life problem. For moeling an classifying meical ata, we aopte this combination of two stage PCA an LDA proceure which is also known as Fisherface technique. During the training phase we applie this combination for etracting features from meical ata. In the classification stage we introuce weighting ratio which is use with the conventional Eucliean istance measure to classify a given sample. For brevity we call this technique the weighte istance Fisherface technique. he presente technique shows promising results for meical ata when compare with stanar GMDH technique; in the two problems taken from the machining learning atabases, the presente approach performe better than the stanar GMDH. Keywors Dimensionality reuction, inuctive moeling, classification, PCA, LDA. 1 Introuction Moeling an pattern classification plays crucial role in everyay life. he evolving computational eman makes this fiel very challenging an thus open for research. When high imensional feature vectors are involve in the computation it makes the implementation of moel an/or pattern classifier quite ifficult an sometimes impossible. his limitation is usually referre as the curse of imensionality. Efforts are unergoing to reuce the compleity in an efficient manner an at the same time achieve sufficient level of classification accuracy. he two well known linear techniques for imensionality reuction are principal component analysis (PCA) [6] an linear iscriminant analysis (LDA) [4]. he goal of PCA is to fin a parsimonious ata space from the original ata space such that the representation information is maimally preserve. he features in the reuce imensional plane are transforme from higher imensional space such that the mean square error is minimum. On the other han, LDA provies a reuce imensional space such that the features of ifferent classes or categories are maimally iscriminate. In the case of solving high imensional problem several authors applie PCA technique prior to the LDA technique [1][8][10][11]. In the work reporte in this paper, we aopte this combination of two stage PCA an LDA proceure which is also known as Fisherface technique for moeling an classifying meical ata. During the training phase we applie this combination for etracting features from meical ata. In the classification stage, we introuce weighting ratio which is use with the conventional Eucliean istance measure to classify a given sample. For brevity we call this technique the weighte istance Fisherface technique. he presente technique shows promising results for meical ata when compare with Group Metho of Hanling Data (GMDH) technique. 39

2 Moel Description Inuctive moeling aims at constructing an efficient an effective moel of high imensional ata. In a given set of inputs, system state, an outputs, the thir component is always eucible with the other two at han (see Fig. 1). A training ataset of inputs, X, an system states, S, can be use to estimate the ensuing outputs, Y, in a preiction or forecast moel. It is a moeling or esign problem to obtain a system, S, for given inputs an outputs X an Y. A control problem is to seek the optimal inputs, X for a given system states, S, that can be use to estimate the ensuing outputs, Y. hese concepts are well escribe by Eler an Brown [5]. Inputs (X) System (S) Outputs (Y) Estimate Output (Preiction) Estimate System State (Moeling) Estimate Input (Control) Fig. 1 Inuctive moeling he system presente here is the weighte istance Fisherface moel which has been applie for meical ata analysis. he basic block iagram of the moel is shown in Fig.. he moel has an output variable or target variable y which epens on -imensional input vector 1,,..., ] = [ an the moel parameters. he moel parameters can be estimate by using training ata where the output y an input are known quantities. Once the moel is estimate then it can be use to provie output y for any unknown. 1 f y Fig.. Basic block iagram of the moel he target variable can be represente in the form of function f an the input as y = f () = f (,, K, ) (1) 1 he function f is a combination of two linear functions namely PCA an LDA. We conclue that our moeling approach in Fig. follows the inuctive moeling architecture in Fig. 1. PCA fins a linear transformation Φ which reuces -imensional ata to h-imensional feature vectors (where h < ) in such a way that the information is maimally preserve in minimum mean square error sense. his linear transformation is known as PCA transform or Karhunen-Loéve transform (KL) [6]. Since the transformation is from -imensional feature space to h-imensional feature space the size of Φ is h. he h column vectors of the matri Φ are the basis vectors. he first basis vector is in the irection of maimum variance of the given feature vectors. he remaining basis vectors are mutually orthogonal an, in orer, maimize the remaining variances subect to the orthogonal conition. Each basis vector represents a principal ais. hese principal aes are those 40

3 orthonormal aes onto which the remaining variances uner proection are maimum. hese orthonormal aes are given by the ominant/leaing eigenvectors (i.e. those with the largest associate eigenvalues) of the measure covariance matri. In PCA, original feature space is characterize by these basis vectors an the number of basis vectors use for characterization is usually less than the imensionality of the feature space [8][9]. In LDA, the imensional embeings are reuce in such a way that the orientation of the proecte ata of classes on an arbitrary line or space is well separate from each other. he transformation vectors are taken so that the criterion J is maimum, where J is the ratio of betweenclass scatter matri (S B ) an within-class scatter matri (S W ) [4]. In a c-class problem the LDA proects from -imensional space to c 1 or less imensional space (R R c-1 ). here are some limitations in applying LDA irectly viz. matri S W can become singular ue to high imensionality of original feature vectors in comparison with low number of training vectors available. o overcome this limitation, a number of authors have propose the use of PCA prior to the application of LDA [1][8][10][11] in the feature etraction stage. In meical ata analysis the PCA technique is applie for two main reasons (i) (ii) the basis vectors that are of less importance can be iscare which woul help in reucing the noise that coul be present in meical ata. to overcome the singularity issue relate with the irect application of LDA. he LDA is applie after the application of PCA to give such feature space in which ifferent class ata are maimally separate uner iscriminant criterion. In aition, the weighte istance measure has been applie for classifying input vectors which was not applie previously in the literature. he following section briefly illustrates the mathematical etails of the moel..1 Principal Component Analysis he PCA transform can be foun by minimizing mean square error. o see this, let the feature vector h be R (-imensional space), reuce imensional feature vector be z R an reconstructe feature vector be ˆ R. hen the mean square error can be represente as MSE = E [ ˆ ] where E [ ] is the epectation operation with respect to an is the norm square value. We know that PCA transformation Φ is of size h an it is use to o imensionality reuction from - imensional space to h-imensional feature space, i.e. Φ : z or z = Φ () he PCA transformation Φ can be obtaine by minimizing mean square error E[ ˆ ] which turns out to be a generalize eigenvalue problem i.e.: Σ φ = λ φ (3) where Φ = {φ : = 1,,..., h}, φ R, an Σ is covariance matri of all input -imensional vectors. he epression λ enotes eigenvalues corresponing to φ. he eigenvectors (φ1,...,φh ) of Φ shoul be arrange such that their corresponing eigenvalues are in escening orer λ 1 > λ >... > λh. his arrangement is, however, not a necessary step for PCA but it is mentione here since the moel conucts this arrangement process prior to the application of LDA. 41

4 . Linear Discriminant Analysis he LDA technique can be illustrate for two-class problem an multi-class problem. he LDA technique for multi-class problem is briefly escribe here. Let a c-class problem (assuming c > ) is given with c unique class lables ω 1,K,ωc. In LDA the proection is from h-imensional feature space to k-imensional feature space where k < h such that the samples or patterns of classes are wellseparate. For a c-class problem the transformation can be given as: s = W z where s R k an z is from equation. (4) he transformation matri W is compute by maimizing Fisher s criterion J(W) = W S B W / W SW W. he computation of between-class scatter matric S B an withinclass scatter matri S W can be compute from the vectors z. See Dua an Hart [4] for etails of computing these matrices. he transformation matri W is given by [4]: S B w i = λ S w (5) i W i where W = {wi : i = 1,,..., k}. he eigenvectors w i (columns of W) correspon to the eigenvalues λ i. Since the rank of between-class scatter matri S B is c 1 or less, k c 1. he net section elaborates the training phase of the moel..3 raining Phase of the Weighte Distance Fisherface Moel he training phase of the moel is epicte in Fig. 3. he training ata is processe through the PCA raining Data PCA z s Centroi µ LDA Computation Fig. 3. raining phase of the moel block to give parsimonious ata space. he feature vectors z are obtaine after PCA process. hese features are then sent to the LDA block which prouces iscriminant features s. he obtaine features s of similar classes will be sent to the Centroi Computation block which will give respective centroi for each class. he training process can be summarize as follows: 1. Compute PCA transformation Φ for input vectors using equation 3.. Proect the samples on lower imensional space R h using equation. his will give z feature vectors. 3. Compute transformation W using eigenvalue ecomposition (equation 5). 4. Proect the z feature vectors on to k-imensional space by using equation 4. his will give s feature vectors. 1 1 It can be observe from equations an 4 that feature vector s is the linear combination of the elements of input vector an can be represente as sm = nvnm for m = 1,K, k n= 1 where v nm w m = h = 1 φ n, n is the element of, sm is the element of s, element of W an φ n (nth row an th column) is the element of Φ. w m (th row an mth column) is the 4

5 5. Compute the centroi of each class. his will give centroi vector µ for = 1,K, c. he parameters that are require to store for testing phase are Φ, W an µ. he classification phase or testing phase is illustrate in the following section..4 Classification Phase of the Weighte Distance Fisherface Moel he classification phase of the moel is epicte in Fig. 4. he input vector with unknown class label is entere in PCA block where feature vector z is compute using PCA transformation Φ. he transforme vector z is then processe through the LDA block which prouce feature vector s using LDA transformation W at its output. he feature vector s then sent via the weighte istance measure block where weighte istance δ (for = 1,K, c ) is compute. All the istance are evaluate in the comparison block an the class label is associate to the input vector for which the istance is minimum. he classification phase is summarize as follows: 1. Compute z for input vector using equation where Φ is a known quantity.. Compute s from the transformation W an input z using equation Compute weighte istance δ between a test vector s an the centroi δ = ( 1 / wt ) s µ for = 1,K, c he term by wt = number of samples in class / total wt is the weighting ratio which reflects a priori probability an can be given µ number of samples in the training ata 4. Fin the argument for which the weighte istance δ is minimum.. c k = arg minδ =1 Assign the class label ωk to the test pattern. he class label is the output of the moel or the target variable i.e. y = ω. k Unknown class labelle input vector PCA Φ z s Weighte δ LDA istance measure W µ comparison y Recognize class Fig. 4. Classification phase of the moel 3 Heart Disease Meical Database he Long Beach [] heart isease meical atabase has been use in the eperimentation. he atabase is provie by Robert Detrano of V.A. Meical Center, Long Beach an Clevelan Clinic Founation. he ataset contains 00 cases an 75 attributes, but all publishe eperiments [3][7] refer to using a subset of 13 of them. here are five classes in this ataset. he target variable iscriminates between five levels of heart isease where label 0 inicates no presence of heart isease an 1,, 3, 4 represent the presence of heart isease at a graually increase level. herefore there are two basic types of classification problem (i) a binary classification problem that 43

6 will etect the eistence of heart isease (level 1-4) or no heart isease (level 0) an (ii) a five class classification problem that ientifies all the five levels accurately. he system is moele using the original 75 attribute Long Beach ata an reuce Long Beach ata of 13 attributes. he ata is ranomly split into 180 cases for learning the moel an 0 cases for appraising the moel. he eperimentation has been escribe in the following section. 4 Eperimentation on Meical Database his section emonstrates the performance of the propose classifier in comparison with GMDH moel. he GMDH moel was applie on heart isease meical ataset by Lemke an Mueller [7]. For two-class problem the target variable woul be either has or has not that signify the presence of heart isease an absence of heart isease respectively. For five-class problem the target variable will be 0,1,,3 an 4 that signify no heart isease an four levels of heart isease. he classification accuracy in percentage an the number of false classifie cases are epicte in ab. 1 for the testing ata on Long Beach 75. est False classifie Accuracy [%] ab. 1. Classification results for Long Beach 75 ataset GMDH [7] Weighte istance Fisherface Has/has not Class 0-4 Has/has not Class It can be observe from ab. 1. that the weighte istance Fisherface moel prouces 100% accuracy for two-class problem whereas GMDH prouces only 90% in this case. On the other han, the propose moel is giving 80% accuracy for five-class problem whereas GMDH is giving only 75% accuracy. he improvement is significant in both the cases. In the case of two-class problem the PCA imension (h) was 3 an LDA imension (k) was 1 for the propose moel an for five-class problem h = 7 an k = 4. able epicts the classification accuracy in percentage an the number of false classifie cases for the testing ata on Long Beach 13. est False classifie Accuracy [%] ab.. Classification results for Long Beach 13 ataset GMDH [7] Weighte istance Fisherface Has/has not Class 0-4 Has/has not Class It is evient from ab.. that the weighte istance Fisherface moel prouces 75% an 50% accuracy for two-class problem an five-class problem respectively. On the other han, GMDH prouces only 65% an 0% accuracy for two-class problem an five-class problem respectively. he PCA imension was h = 3 an LDA imension was k = 1 for two class problem an h = 11 an k = 1 for five-class problem. he values of h an k are selecte for which the moel is proucing the best results. hus the results obtaine epicte are the best results that can be achieve by the moel. 5 Conclusion his paper has presente an inuctive moeling approach base on combination of two stage PCA an LDA proceure which is also known as Fisherface technique for moeling an classifying meical ata. For etracting features from meical ata we have applie this combination of techniques an in 44

7 the classification stage we ae weighting ratio which was use with the conventional Eucliean istance measure to classify a given sample. We referre this technique as the weighte istance Fisherface technique. We conclue that our moeling approach in Fig. follows the inuctive moeling architecture in Fig. 1. For the two problems taken from the machining learning atabases, the presente approach performe better than the stanar GMDH. It can be sai from the obtaine results that the weigthe istance Fisherface moel is proucing better results than the GMDH moel while eperimente on heart isease meical atabase. he possible future research work woul be to hybriize the techniques with GMDH-like networks. References [1] Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Eigenfaces vs. fisherfaces: recognition using class specific linear proection. IEEE rans. on Pattern Analysis an Machine Intelligence, 19(7), p , [] Blake, C.L., Merz, C.J.: UCI repository of machine learning atabases. Irvine, CA, University of Calif., Dept. of Information an Comp. Sci., [3] Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmi, J., Sanhu, S., Guppy, K., Lee, S., Froelicher, V.: International application of a new probability algorithm for the iagnosis of coronary artery isease. American Journal of Cariology, 64, p , [4] Dua, R.O., Hart, P.E.: Pattern classification an scene analysis. John Wiley an Sons, New York, [5] Eler IV, J.F., Brown, D.E.: Inuctive an polynomial networks. echnical Report IPC-R-9-009, Institute for parallel computation an epartment of systems engineering, University of Virginia, Charlottesville, VA, USA, 199. [6] Fukunaga, K.: Introuction to statistical pattern recognition. Acaemic Press Inc., Hartcourt Brace Jovanovich, Publishers, 1990 [7] Lekme, F., Mueller, J.-A.: Meical ata analysis using self-organizing ata mining technologies. Systems Analysis Moelling Simulation, 43(10), p , 003. [8] Sharma, A., Paliwal, K.K., Onwubolu, G.C.: Class-epenent PCA, LDA an MDC: a combine classifier for pattern classification. Pattern Recognition, 39(7), p , 006. [9] Sharma, A., Paliwal, K.K.: Fast Principal Component Analysis using Fie-Point Algorithm, Pattern Recognition Letters, 8, p , 007. [10] Swets, D.L., Weng, J.: Using iscriminative eigenfeatures for image retrieval. IEEE rans. on Pattern Analysis an Machine Intelligence, 18(8), p , [11] Zhao, W., Chellappa, R., Phillips, P.J.: Subspace linear iscriminant analysis for face recognition. ech. Rep. CAR-R-914, Center for Automation Research, University of Marylan, College Park,