WEIGHTED PRINCIPAL COMPONENT ANALYSIS BASED ON FUZZY CLUSTERING MIKA SATO-ILIC. Received May 12, 2003

Transcription

1 Sientiae Mathematiae Japoniae Online, Vol. 10, (2004), WEIGHTED PRINCIPAL COMPONENT ANALYSIS BASED ON FUZZY CLUSTERING MIKA SATO-ILIC Reeied May 12, 2003 Abtrat. In thi paper, we propoe a weighted prinipal omponent analyi (WPCA) uing the reult of fuzzy lutering [4]. The prinipal omponent analyi (PCA) [1], [7] i one widely ued and well-known data analyi method. Howeer there i a problem, when the data doe not hae a truture that the PCA an apture we annot obtain any atifatory reult. For the mot part, thi i due to the uniformity of the data truture, whih mean we annot find any ignifiant proportion or aumulated proportion for the obtained prinipal omponent. In order to ole thi problem, we ue the luter truture and degree of belongingne of objet to luter, whih i obtained a the fuzzy lutering reult. By the introdution of the pre-laifiation and the degree of belongingne to the data, we an tranform the data into a learer trutured data, o aoiding the noie in the data. 1 Introdution Reently, PCA i the method of multiariate analyi whih i the mot widely ued. Due to the partiular feature of PCA whih an repreent the main tendeny of an obered data ompatly, PCA ha been ued a a method to follow up a lue when we an not ee any ignifiant truture in the data. Beide the importane of PCA a an exploratory method, eeral extended method of PCA hae been propoed with the iewpoint that PCA an how the ability to the full, by bringing additional onideration of the obered data into the analyi. For example, ontrained PCA (CPCA) [11], [12], nonlinear PCA [10], and the time dependent prinipal omponent analyi [2] are typial example of thi. The propoed method in thi paper alo take the poition that the method i baed on the idea that we try to bring PCA ability into full play by introduing pre-information of the luter truture of an obered data. The pre-information of the data i repreented a weight baed on a fuzzy lutering reult. That i, the main differene between onentional PCA and WPCA baed on fuzzy lutering i the introdution of luter degree a weight on luter of the oberation pae. The weight are repreented by the produt of the degree of belongingne of objet to fuzzy luter with repet to the fuzzy luter when an objet i fixed. Due to a property of the algebrai produt and the ondition of the degree of belongingne, thee weight an how how muh the data ha the lutering truture. Aording to weighted linear ombination, we an how that the etimate of prinipal omponent for WPCA baed on fuzzy lutering are obtained in a imilar way a in onentional PCA. The time dependent prinipal omponent analyi [2] alo introdue weight to PCA, howeer, the analyi doe not ue the luter truture of the data. Moreoer, a poibiliti regreion model [13], the withing regreion model [6], and the weighted regreion analyi baed on fuzzy lutering [9] are among eeral example of reearh that introdued the onept of fuzzine to the regreion model Mathemati Subjet Claifiation. 62H25, 62H30. Key word and phrae. Fuzzy luter, Prinipal omponent, Weight.

2 360 M. SATO-ILIC Thi paper i organized a follow: in etion 2, we explain onentional prinipal omponent analyi. A fuzzy lutering method ued in thi paper i mentioned in etion 3. In etion 4, we propoe the weighted prinipal omponent analyi baed on the fuzzy lutering. Setion 5 tate eeral numerial example for howing the alidity and the better performane of the propoed weighted prinipal omponent analyi. The onluion i gien in etion 6. 2 Prinipal Component Analyi (PCA) When we obtain the oberation of objet with repet to ariable, PCA aume that there are eeral main omponent (fator) aued by the relationhip of the ariable. The purpoe of PCA i to obtain the omponent from the data and apture the feature of the objet. That i, PCA abtrat the data to the main omponent and find the feature of the obtained data. The obered data whih are alue of p ariable with repet to n objet are denoted by the following: (1) X =(x ia ), i =1,,n, a=1,,p. Suppoe the firt prinipal omponent z 1 whih i defined a the following linear ombination: (2) z 1 = Xl 1, where l 1 =(l 11,l 12,,l 1p ). The purpoe of PCA i to etimate l 1 whih maximize the ariane of z 1 under the ondition of l 1l 1 = 1. The ariane of z 1 i a follow: (3) V {z 1 } = V {Xl 1 } = l 1V {X}l 1 = l 1Σl 1, where, V { } how the ariane of and Σ i a ariane-oariane matrix of X. Uing the Lagrange method of indeterminate multiplier, the following ondition i needed for whih l 1 ha non-triial olution. (4) Σ λi =0, where λ i an indeterminate multiplier and I i a unit matrix. (4) i the harateriti equation, o λ i obtained a an eigen-alue of Σ. Uing (4) and the ondition l 1 l 1 =1,we obtain (5) l 1 Σl 1 = λl 1 l 1 = λ. From (3) and (5), V {z 1 } = λ. So, l 1 i determined a the orreponding eigen-etor for the maximum eigen-alue of Σ. In order to obtain the eond prinipal omponent z 2, we define the following linear ombination: (6) z 2 = Xl 2, where l 2 =(l 21,l 22,,l 2p ). We need to etimate l 2 whih maximize the ariane of z 2 under the ondition of l 2l 2 = 1 and oariane of z 1 and z 2 i 0, that i, z 1 and z 2 are mutually unorrelated. If we denote the oariane between z 1 and z 2 a o{z 1, z 2 }, then thi ondition i repreented a follow: o{z 1, z 2 } =o{xl 1,Xl 2 } = l 1 o{x, X}l 2 = l 1 Σl 2 = λl 1 l 2 =0.

3 WPCA BASED ON FUZZY CLUSTERING 361 That i, we need to etimate l 2 whih atify (Σ λi)l 2 = 0 under the ondition l 2 l 2 =1 and l 1l 2 = 0. From the fat that the eigen-etor orreponding to the different eigenalue are mutually orthogonal, l 2 i found a the eigen-etor orreponding to the eond larget eigen-alue of Σ. If we get k (k p) different eigen-alue of Σ, aording to the aboe, we find the k prinipal omponent. A an indiator whih an how how many prinipal omponent are needed to explain the data atifatory, the proportion ha been propoed. The proportion of the α-th prinipal omponent C α i defined a: (7) C α = λ α tr(σ). From V {z α } = λ α and p α=1 λ α = tr(σ), C α an explain the importane of the α-th prinipal omponent. Alo, the aumulated proportion until k-th prinipal omponent i defined a: (8) P = k C α. α=1 In order to get the interpretation of the obtained prinipal omponent, the fator loading r α,j whih i defined a a orrelation oeffiient between the α-th prinipal omponent z α and the j-th ariable x j ha been propoed a following: (9) r α,j = o{z α, x j } V {zα }V {x j } = λα l αj σjj, where σ jj i ariane of x j. 3 Fuzzy Clutering Conentional lutering mean laifying the gien oberation into exluie ubet (luter). That i, we an diriminate learly if an objet belong to a luter or not. Howeer, uh a partition i hardly enough to repreent many real ituation. Then a fuzzy lutering method i offered to ontrat luter with ague boundarie, namely thi method allow that one objet belong to ome oerlapping luter with ome grade. In other word, the eene of fuzzy lutering i to onider not only the belonging tatu to the aumed luter, but alo to onider how muh the objet belong to the luter. So, there i a merit to repreenting the omplex data ituation of real data. The tate of fuzzy lutering i repreented by a partition matrix U =(u ik ) whoe element how the grade of belongingne of the objet to the luter, u ik,i=1,,n, k = 1,,K, where n i number of objet and K i number of luter. In general, u ik atifie the following ondition: (10) u ik [0, 1], u ik =1. Fuzzy -mean (FCM) [3] i one of the method of fuzzy lutering. FCM i the method whih minimize the weighted within-la um of quare: (11) J(U, 1,, K )= i=1 (u ik ) m d 2 (x i, k ), where k =( ka ),,,K, a=1,,pdenote the alue of the entroid of luter k, x i =(x ia ),i=1,,n, a=1,,p i i-th objet, and d 2 (x i, k ) i the quare Eulidean

4 362 M. SATO-ILIC ditane between x i and k. The exponent m whih determine the degree of fuzzine of the lutering i hoen from [1, ) in adane. The purpoe i to obtain the olution U and 1,, K whih minimize (11), and the olution are obtained by Piard iteration of the following expreion: u ik = j=1 1 {d(x i, k )/d(x i, j )} 2 m 1, k = (u ik ) m x i i=1, i =1,,n; k =1,,K. (u ik ) m i=1 From (11), we an rewrite a: J(U, 1,, K ) = = = = (u ik ) m d 2 (x i, k ) i=1 i=1 i=1 i=1 (u ik ) m (x i k, x i k ) (u ik ) m (x i h k + h k k, x i h k + h k k ) (u ik ) m [(x i h k, x i h k )+2(x i h k, h k k ) +(h k k, h k k )], where (, ) denote real inner produt. If we aume h k = (u ik ) m x i i=1, (u ik ) m i=1 then minimizer of (11) i hown a: (12) J(U) = ((u ik ) m (u jk ) m d ij )/(2 (u lk ) m ), uing 2 i=1 j=1 n (u lk ) m (u ik ) m (x i h k, x i h k )= l=1 i=1 i=1 l=1 l=1 (u ik ) m (u lk ) m (x i x l, x i x l ), and d ij = d(x i, x j ). When m = 2, (12) i the objetie funtion of the FANNY algorithm. The detail of the FANNY algorithm i hown in [8]. 4 Weighted Prinipal Component Analyi (WPCA) baed on Fuzzy Clutering We apply a fuzzy lutering method to the data matrix X hown in (1) and obtain the

5 WPCA BASED ON FUZZY CLUSTERING 363 degree of belongingne U =(u ik ),i=1,,n, k =1,,K. Uing the obtained U, we define the weight matrix W a follow: K u 1 1k 0 0 K w u 1 2k 0 0 W = 0 w w K n 0 0 u 1 nk Then we introdue the following weighted matrix WX: (13) w w x 11 x 1p WX = x 0 0 w n1 x np n = In order to aoid 0 1, we replae (10) a the following ondition: (14) u ik (0, 1), u ik =1. w 1 x 11 w 1 x 1p... w n x n1 w n x np From the property of the algebrai produt and the ondition (14), we an ee that if u ik = 1 K, for i, k, then u 1 ik K (= w i), ( i) take minimum alue. And if u ik i loe to 1 for k, i, then K u 1. ik (= w i), ( i) i loe to maximum alue. That i, the weight w i how that if the belonging tatu of the objet i to the luter i learer, then the weight beome larger. Otherwie, if the belonging tatu of the objet i i more ague ituation, that i, the luter truture of the data i loe to uniformity, then the weight beome mall. So, WX how that learer objet under the luter truture hae larger alue and that the objet whih are aguely ituated under the lutering are aoided uh a thoe whih do not hae any ignifiant relation to the lutering for example oberation whih are treated a noie. Suppoe z 1 i the firt prinipal omponent of the tranformed data WX hown in (13). z 1 i defined a: (15) z 1 = WX l 1, where l 1 =( l 11, l 12,, l 1p ). The purpoe of the WPCA baed on fuzzy lutering i to etimate l 1 whih maximize the ariane of z 1 under the ondition of l 1 l 1 = 1. The ariane of z 1 i a follow: (16) V { z 1 } = V {WX l 1 } = V { X l 1 } = l 1 Σ l 1, where, WX X, Σ = X X =(WX) (WX)=X WWX.

6 364 M. SATO-ILIC Uing the Lagrange method of indeterminate multiplier, the following ondition i needed for whih l 1 ha non-triial olution. (17) Σ λi =0, where λ i an indeterminate multiplier and I i a unit matrix. (17) i the harateriti equation, o λ i obtained a an eigen-alue of Σ. Uing (17) and the ondition l 1 l 1 =1, we obtain (18) l 1 Σ l 1 = λ l 1 l 1 = λ. From (16) and (18), V { z 1 } = λ. So, l 1 i determined a the orreponding eigen-etor for the maximum eigen-alue of Σ. In order to obtain the eond prinipal omponent z 2 for X, we define the following linear ombination: (19) z 2 = X l 2, where l 2 =( l 21, l 22,, l 2p ). We need to etimate l 2 whih maximize the ariane of z 2 under the ondition of l 2 l 2 = 1 and oariane of z 1 and z 2 i 0, that i, z 1 and z 2 are mutually unorrelated and whih i repreented a follow: o{ z 1, z 2 } =o{x l 1, X l 2 } = l 1o{ X, X} l 2 = l 1 Σ l 2 = λ l 1 l 2 =0, where o{ z 1, z 2 } how oariane between z 1 and z 2. That i, we need to etimate l 2 whih atify ( Σ λi) l 2 = 0 under the ondition l 2 l 2 = 1 and l 1 l 2 = 0. From the fat that the eigen-etor orreponding to the different eigen-alue are mutually orthogonal, l2 i found a the eigen-etor orreponding to the eond larget eigen-alue of Σ. If we get k (k p) different eigen-alue of Σ, aording to the aboe, we find the k prinipal omponent for X. The proportion of α-th prinipal omponent for WPCA baed on fuzzy lutering, Cα, i propoed a follow: (20) Cα = λ α tr( Σ), where V { z α } = λ α and p α=1 λ α = tr( Σ). The aumulated proportion until k-th prinipal omponent for WPCA baed on fuzzy lutering i defined a: k (21) P = C α. α=1 The fator loading r α,j between the α-th prinipal omponent z α and the j-th ariable x j i propoed a the following: (22) r α,j = o{ z α, x j } V { zα }V { x j } = λα lαj σjj, where σ jj i ariane of x j.

7 WPCA BASED ON FUZZY CLUSTERING Numerial Example The data i Fiher iri data [5]. The data onit of 150 ample of iri flower with repet to four ariable, epal length, epal width, petal length, and petal width. The ample are obered from three kind of iri flower, iri etoa, iri eriolor, and iri irginia. Figure 1 how the reult of PCA for the iri data. In thi figure, the abia how the alue of firt prinipal omponent hown in (2) and the ordinate i the alue of the eond prinipal omponent hown in (6). mean iri etoa, mean iri eriolor, and i iri irginia. From thi figure, we an ee iri etoa (the ymbol i ) i learly diided from iri eriolor and iri irginia (the ymbol are and ). Figure 2 how the proportion (hown in (7)) and aumulated proportion (hown in (8)) of the four omponent. In thi figure, the abia how eah omponent and the ordinate how the alue of the proportion of eah omponent. Alo the alue on the barplot how the aumulated proportion. From thi figure, we an ee that it an almot be explained by the firt and the eond prinipal omponent. Howeer, we an not ignore the third prinipal omponent ompletely. Figure 3 how the reult of WPCA baed on fuzzy lutering. A a weight W in (13), we ue a fuzzy lutering reult whih i obtained by uing the FANNY algorithm whoe objetie funtion i (12) when m = 2. In figure 3, the abia how the alue of the firt prinipal omponent and the ordinate i the alue of the eond prinipal omponent. From thi figure, we an ee a lear ditintion between iri etoa (the ymbol i ) and iri eriolor, iri irginia (the ymbol are and ), omparing the reult hown in figure 1. Figure 4 how the proportion (hown in (20)) and aumulated proportion (hown in (21)) of WPCA baed on fuzzy lutering reult. In thi figure, the abia how eah prinipal omponent and the ordinate i the alue of proportion for eah prinipal omponent. The alue on the barplot i the aumulated proportion for eah prinipal omponent. Through a omparion between figure 2 and figure 4, we an ee the higher alue of the aumulated proportion until the eond prinipal omponent in WPCA baed on fuzzy lutering (0.991) than the alue of the aumulated proportion until the eond prinipal omponent of PCA (0.958). Thi how the higher dirimination ability with WPCA baed on fuzzy lutering. Moreoer, we an not ee any ignifiant meaning for the third prinipal omponent from the reult of the proportion of WPCA baed on fuzzy lutering hown in figure 4. So, we an aoid the noie of the data by the introdution of the weight baed on the fuzzy lutering reult. eond prinipal omponent firt prinipal omponent Figure 1 Reult of PCA for Iri Data

8 366 M. SATO-ILIC Variane x Comp. 1 Comp. 2 Comp. 3 Comp. 4 Figure 2 Barplot of Proportion for PCA eond prinipal omponent firt prinipal omponent Figure 3 Reult of WPCA baed on Fuzzy Clutering for Iri Data x Variane Comp. 1 Comp. 2 Comp. 3 Comp. 4 Figure 4 Reult of Proportion for WPCA baed on Fuzzy Clutering

9 WPCA BASED ON FUZZY CLUSTERING 367 Table 1 and 2 how the quared alue of λ α hown in (5) and the quared alue of λ α hown in (18), repetiely. In thee table omp.1 how the firt prinipal omponent, omp.2 how the eond prinipal omponent, omp. 3 i the third prinipal omponent, and omp. 4 i the forth prinipal omponent. From the omparion between the reult of table 1 and 2, we an ee the higher alue for the eond prinipal omponent in table 2 than the alue for the eond prinipal omponent in table 1. Moreoer, we an ee the maller alue for the third and the forth prinipal omponent in table 2 a ompared to the alue for the third and the forth prinipal omponent in table 1. From thi again, we an ee that the WPCA baed on fuzzy lutering ha a high apability to apture the ignifiane of the latent data truture learly and tend to aoid the noie of the data, although the alue for the firt prinipal omponent beome maller in table 2. Table 1 Standard Deiation for Prinipal Component in PCA Comp. Comp. 1 Comp. 2 Comp. 3 Comp. 4 SD Table 2 Standard Deiation for Prinipal Component in WPCA baed on Fuzzy Clutering Comp. Comp. 1 Comp. 2 Comp. 3 Comp. 4 SD Moreoer, table 3 and 4 how the reult of fator loading whih are hown in (9) and (22), repetiely. The alue of thee table how the alue of the fator loading whih an how the relationhip between eah prinipal omponent and eah ariable. So, from thee reult, we an ee how eah omponent i explained by the ariable. It i related with the interpretation of eah omponent. In thee table, Sepal L. how epal length, Sepal W. how epal width, Petal L. how petal length, and Petal W. i the petal width. Table 3 Fator Loading in PCA Comp. Comp. 1 Comp. 2 Comp. 3 Comp. 4 Sepal L Sepal W Petal L Petal W Table 4 Fator Loading in WPCA baed on Fuzzy Clutering Comp. Comp. 1 Comp. 2 Comp. 3 Comp. 4 Sepal L Sepal W Petal L Petal W From the omparion between the reult of table 3 and 4, we an ee quiet different reult. For example, in table 3, the firt prinipal omponent i mainly explained by the ariable, epal length, petal length, and petal width. Howeer, in table 4, we an ee

10 368 M. SATO-ILIC that the firt prinipal omponent i explained by the ariable, epal length, epal width, and petal length. Moreoer, for the eond prinipal omponent, we an ee the differene, that i, in table 3, the eond prinipal omponent i repreented by the trong relationhip of the ariable epal width, but in table 4, we an ee the high orrelation between the eond prinipal omponent and the ariable petal width. From thi, we an ee that we an apture the omponent whih hae different meaning from onentional PCA by uing WPCA baed on fuzzy lutering. In other word, the ability to apture the different latent fator of the data i hown by introduing the luter truture of the data. 6 Conluion We propoed a weighted prinipal omponent analyi uing the reult of fuzzy lutering. If the luter truture whih are obtained by the fuzzy lutering and the prinipal omponent truture are the ame, then the two reult are eentially the ame. Howeer, normally both truture are different, we an obtain the different reult from the reult of onentional PCA by uing the propoed method. From eeral numerial example, we howed higher dirimination ability of the propoed method ompared with onentional PCA. Referene [1] T. W. Anderon, An Introdution to Multiariate Statitial Analyi, Seond ed., John Wiley & Son, (1984). [2] Y. Baba and T. Nakamura, Time Dependent Prinipal Component Analyi. Meaurement and Multiariate Analyi, S. Nihiato et al. ed., Springer, (2002), [3] J. C. Bezdek, Pattern Reognition with Fuzzy Objetie Funtion Algorithm, Plenum Pre, (1987). [4] J. C. Bezdek, J. Keller, R. Krinapuram, and N. R. Pal, Fuzzy Model and Algorithm for Pattern Reognition and Image Proeing, Kluwer Aademi Publiher, (1999). [5] R. A. Fiher, The Ue of Multiple Meaurement in Taxonomi Problem, Ann. Eugeni, 7(2), (1936), [6] R. J. Hathaway and J. C. Bezdek, Swithing Regreion Model and Fuzzy Clutering, IEEE Tran. Fuzzy Syt., 1(3), (1993), [7] I. T. Jolliffe, Prinipal Component Analyi, Springer Verlag, (1986). [8] L. Kaufman and P. J. Roueeuw, Finding Group in Data, John Wiley & Son, (1990). [9] M. Sato-Ili and T. Matuoka, On an Appliation of Fuzzy Clutering for Weighted Regreion Analyi, The 4th Conferene of Aian Regional Setion of the International Aoiation for Statitial Computing, (2002), [10] B. Sholkopf, A. Smola, and K. Muller, Nonlinear Component Analyi a a Kernel Eigenalue Problem, Neural Computation, 10, (1998), [11] Y. Takane and T. Shibayama, Prinipal Component Analyi with External Information on Both Subjet and Variable, Pyhometrika, 56, (1991), [12] Y. Takane, Seiyakutuki Syueibun Buneki, Aakura Syoten, (1995) (in Japanee). [13] H. Tanaka and J. Watada, Poibiliti Linear Sytem and Their Appliation to the Linear Regreion Model, Fuzzy Set and Sytem, 27, (1988), Faulty of Sytem and Information Engineering Unierity of Tukuba Tennodai 1-1-1, Tukuba, Ibaraki , Japan mika@k.tukuba.a.jp