On Utilization of K-Means for Determination

Transcription

1 Jounal of Softwae Engineeing and Applications, 07, 0, ISSN Online: ISSN Pint: On Utilization of K-Means fo Detemination of q-paamete fo Tsallis-Entopy-Maximized-FCM Maoto Yasuda Depatment of Electical and Compute Engineeing, National Institute of Technology, Gifu College, Motosu, Japan How to cite this pape: Yasuda, M. (07) On Utilization of K-Means fo Detemination of q-paamete fo Tsallis-Entopy-Maximized- FCM. Jounal of Softwae Engineeing and Applications, 0, Received: Apil 4, 07 Accepted: June 0, 07 Published: June 3, 07 Copyight 07 by autho and Scientific Reseach Publishing Inc. This wo is licensed unde the Ceative Commons Attibution Intenational License (CC BY 4.0). Open Access Abstact In this pape, we conside a fuzzy c-s (FCM) clusteing algoithm combined with the deteministic annealing method and the Tsallis entopy imization. The Tsallis entopy is a q-paamete extension of the Shannon entopy. By imizing the Tsallis entopy within the famewo of FCM, membeship functions simila to statistical mechanical distibution functions can be deived. One of the majo consideations when using this method is how to detemine appopiate q values and the est annealing tempeatue, T, fo a given data set. Accodingly, in this pape, a method fo detemining these values simultaneously without intoducing any additional paametes is pesented. In ou appoach, the membeship function is appoximated by a seies of expansion methods and the K-s clusteing algoithm is utilized as a pepocessing step to estimate a adius of each data distibution. The esults of expeiments indicate that the poposed method is effective and both q and T can be detemined automatically and algebaically fom a given data set. Keywods Fuzzy c-means, K-Means, Tsallis Entopy, Entopy Maximization, Entopy Regulaization, Deteministic Annealing. Intoduction Techniques fom statistical mechanics can be used fo the investigation of the macoscopic popeties of a physical system consisting of many elements. Recently, eseach activities utilizing statistical mechanical models o techniques fo infomation pocessing have become inceasingly popula. DOI: 0.436/jsea June 3, 07

2 Rose et al. [] [] poposed deteministic annealing (DA) as a deteministic vaiant of simulated annealing (SA) [3]. In DA, the minimization poblem fo an objective function is teated as the minimization of the fee enegy of a system. The DA appoach tacs the function s minimum with deceasing the system tempeatue, thus allowing the deteministic optimization of the objective function at each tempeatue. Hence, DA is moe efficient than SA, but does not guaantee that the solution is the global optimal solution. Fom the viewpoint of statistical mechanics, the membeship functions of the fuzzy c-s (FCM) clusteing [4] with imum entopy o entopy egulaization methods [5] [6] can be seen as distibution functions fom statistical mechanics. Fo example, FCM imized with the Shannon entopy gives a membeship function simila to the Boltzmann distibution function []. Tsallis [7], inspied by multi-factal, non-extensively extended the Boltzmann Gibbs statistics by postulating a genealized fom of the entopy (the Tsallis entopy) with a genealization paamete q. The Tsallis entopy is poved to be applicable to the numeous systems [8] [9]. In the field of fuzzy clusteing, a membeship function was deived by imizing the Tsallis entopy within the famewo of FCM [0] [] []. This membeship function has a simila fom to the statistical mechanical distibution function, and is suitable fo use with annealing methods because it contains a paamete coesponding to the system tempeatue. Accodingly, the Tsallis entopy imized FCM was successfully combined with the DA method as Tsallis-DAFCM in [3]. One of the majo challenges with using Tsallis-DAFCM is the detemination of an appopiate value fo q and the est (o initial) annealing tempeatue, T, fo a given data set. Especially, the detemination of a suitable q value is a fundamental poblem fo systems whee the Tsallis entopy is applied. Even in physics, quite a few systems ae nown in which q is calculable. In the pevious study [3], the values wee expeimentally detemined, and only oughly optimized. Accodingly, we pesented a method that can detemine both q and T simultaneously fom a given data set without intoducing additional paametes [4]. The membeship function of Tsallis-DAFCM was appoximated by a seies expansion to simplify the function. Based on this simplified fomula, both q and T could be estimated along with the membeship function fo a given data set. Howeve, it was also found that the esults fom this method depend on the estimation of the adius of the distibution of the data o the location of clustes. To ovecome this difficulty, in this study, we popose a method that utilizes K-s [5] as a pepocessing step of the appoximation method. That is, a data set is clusteed by K-s oughly. We then estimate the adius of the distibution of the data set, and apply the appoximation method to detemine q and T. Expeiments ae pefomed on numeical data and the Iis Data Set [6], and the esults show that the poposed method can be used to detemine q and 606

3 T automatically and algebaically fom a data set. It is also confimed that the data can be patitioned into clustes appopiately using these paametes.. FCM with Tsallis Entopy Maximization ( ) ( ) Let {,, } (,, p p X = x xn x = x x ) R space, and let {,, } (,, p V = v vc vi = vi vi ) i be a data set in p-dimensional eal [ 0,](,, ;,, ) be the c distinct clustes. Let u i= c= n be the membeship function, and let n c m i i ( ) () = i= J = u d < m be the objective function of FCM, whee d i = x v i. On the othe hand, the Tsallis entopy is defined as q Sq = pi q () i whee p i is the pobability of the iith event and q is a eal numbe [7]. The Tsallis entopy eaches the Shannon entopy as q. Next, we apply the Tsallis entopy imization method to FCM [] [3]. Fist, Equation () is ewitten as S q n c q = ui q (3) = i= Then, the objective function in Equation () is ewitten as Unde the nomalization constaint of J n c q q uidi = i= c i= the Tsallis entopy functional becomes = (4) ( ) ui = (5) q ( ) n c n c δs αδ u β δ u d (6) q i i i = i= = i= whee α and β ae the Lagange multiplies. By applying the vaiational method, the stationay condition fo the Tsallis entopy functional yields the following membeship function fo Tsallis-FCM []: whee u i { ( ) } β q d j q = (7) Z { ( ) } β q d j c q (8) j= Z = Fom Equation (7), the expession fo v i becomes n q u id = i i = n q u = i v (9) 607

4 3. Appoximation of Membeship Function The pefomance of Tsallis-DAFCM is supeio to those of othe entopy-based- FCM methods []. Howeve, it is still unnown how to detemine an appopiate q value and a est annealing tempeatue T fo a given data set. To tacle this poblem, we fist simplify the membeship function using a seies expansion. 3.. Seies Expansion of u i u i in Equation (7) can be expanded to a powe of β as follows: u i n= 0 ( 0) n ui n = β (0) n n! β When the tempeatue is enough, if the seies expansion up to the thid ode tems is used, Equation (0) becomes whee u cd + L c qd cql cd L + L c c c i i i i = + β + β 3 () L L = = c j= c j= d d j, j. () 3.. Detemination of q and T Based on the esults in Section 3., we popose a method fo detemining both q and β simultaneously. Fist, to ensue the convegence of Equation (0), we use the following expession fo β : L = n = β = N N n L c ( l) l= L,, (3) whee N and ( l ) denote the imum numbe of iteations, and the numbe of iteations to be used in the calculation of calculated as T = β. L i, espectively. T can be Then, setting v i = 0 and eplacing x with the continuous vaiable x, Equation () becomes whee ( c+ L ( x) ) qβ 4 β β u ( x) = x x + c c c + ( cl( x) cql( x) β + L ( x) ) c 3 β β (4) 608

5 L L ( ) ( ) c x = v x j= c x = v x j= j j 4,. (5) Fom this equation, q can be detemined as follows. By designating the ange of the dataset as (,, p R = R R ), the imum ange of the distibution R is defined as Θ R R ag R θ = Θ= (6) θ p Futhemoe, by assuming that the adius of each cluste is between R c R, and u ( x ) tends to u 0 at x = ( x = 0,, x Θ =,, x p = 0), Equation (4) can be solved fo q. Consequently, we have the following fomula fo q. ϑ( ) = { c( c+ L ( x ) β) 4 } c c L x β q = ϑ ( ) ( ( )) { L( x ) + L ( x ) β}, c c β. 4 ( L ( x )) (7) It should be noted that in this equation, fo simplicity, u 0 is set to u0 = (8) c because Equation (7) tends to c as 4. Poposed Algoithm x goes to. By combining the method pesented in the pevious section with Tsallis- DAFCM, we poposed the following fuzzy c-s clusteing algoithm [4]. In this algoithm, the numbe of clustes in the data is assumed to be nown in advance. In the fist algoithm shown in Figue, the paametes q and β = T fo a given data set ae detemined ( N is the imum numbe of iteation. In Equation (7), L ( x ) and L ( x ) ae appoximated by L and L, espectively.). The second algoithm is the conventional Tsallis-DAFCM algoithm []. ) Set the tempeatue eduction ate T, and the thesholds fo convegence δ and δ. ) Geneate c initial clustes at andom locations. Set the cuent tempeatue T to β. 3) Calculate u i using Equation (7). 4) Calculate the cluste centes using Equation (9). 5) Compae the diffeence between the cuent centes and the centes of the pevious iteation obtained using the same tempeatue v i. If the convegence condition i c v i v i < δ is satisfied, then go to Step.6. Othewise e- 609

6 Figue. Pocessing flow of the conventional method. tun to Step.3. 6) Compae the diffeence between the cuent centes and the centes of the pevious iteation obtained using a lowe tempeatue v. If the convegence condition i c v i v i < δ is satisfied, then stop. Othewise decease the tempeatue; T = T T, and etun to Step.3. The expeimental esults in [4] confimed that the fist algoithm can detemine β desiably. Howeve, they also evealed that q fom this algoithm stongly depends on the estimation of the adius in Equation (7). Accodingly, as shown in Figue, the fist algoithm is divided in two pats. The fist one detemines β. In the second pat, the K-s algoithm is utilized to calculate by assuming that each data point belongs to its neaest cluste. 5. Expeiments To examine the effectiveness of the poposed algoithm, we conducted two expeiments. 60

7 5.. Expeiment Figue. Pocessing flow of the poposed method. The fist expeiment examined whethe appopiate q and β = T values can be detemined fo a given data set, and the elation between the numbe of iteations N and the paametes q and β. In this expeiment, data sets containing (a) thee clustes and (b) five clustes wee used, as shown in Figue 3. Each cluste follows a nomal distibution, and contains, 50 data points. Dependencies of the imum, minimum, and standad deviation of β, a adius of the data distibution and q fo Figue 3(a) on the numbe of iteations N ae summaized in Table, Table and Table 7. Figue 4 shows the plots of the imum, minimum, and of q. In these tables, and denote R and the of R c and R, espectively. and, on the othe hand, denote the imum and adius of the distibution obtained by K-s, espectively. In Table 7, the value of q fo fo example is calculated using Equation (7) as q = ϑ ( ). Based on the esults in Table, the value of q was calculated by 6

8 (a) Figue 3. Numeical data ( C denotes the cluste numbe). (a) c = 3 ; (b) c = 5. x (b) Figue 4. Maximum, minimum, and of q ( c = 3, β = 5.35e 06 ). 6

9 Table. Maximum, minimum,, and standad deviation of β ( c = 3 ). N Maximum Minimum Mean Std. deviation e e e e e e e e e e e e 07 0, e e e e 07 Table. Maximum, minimum,, and standad deviation of and ( c = 3 ). N Maximum Minimum Mean Std. deviation , , fixing β to its value 5.35e 06. R,, and fo Figue 3(a) ae 860.0, and 86.7, espectively. Fom Table, it can be seen that the imum of β tends to incease and the minimum of β tends to decease with inceasing N. Howeve, when N become 00 o moe, the of β does not depend on N. Fom Table, it can be seen that the of and hadly depends on N, though the standad deviation becomes lage when N become, 000 o moe. This is caused by a vey seldom misclassification of K-s. Compaing the esults in Table 7, it can be found that, when is set to o, q has smalle standad deviations, and the magnitude of the change in the values of q is compaatively small. This shows that q can be calculated stably by pefoming K-s fist. It is also can be found that the imum of q inceases with inceasing N, because of the andom locations of clustes. Even though oveestimates the adius of the clustes, clusteing can be pefomed popely in this case. Accodingly, q has little impact on clusteing in this expeiment. Dependencies of the imum, minimum, and standad deviation of β, a adius of the data distibution and q fo Figue 3(b) on the numbe of iteations N ae summaized in Table 3, Table 4 and Table 8. Figue 5 shows the plots of the imum, minimum, and of q. 63

10 Figue 5. Maximum, minimum, and of q ( c = 5, β = 3.608e 06 ). Table 3. Maximum, minimum,, and standad deviation of β ( c = 5 ). N Maximum Minimum Mean Std. deviation e e e e e e e e e 06.36e e e 07 0, e 06.8e e e 07 Table 4. Maximum, minimum,, and standad deviation of and ( c = 5 ). N Maximum Minimum Mean Std. deviation , , R,, and fo Figue 3(b) ae 860.0, and 58.0, espectively. Based on the esults in Table 3, the value of q was calculated by fixing β to 3.608e 06. Compaing these esults with those in Table, Table and Table 7, it can be found that q fo c = 5 has lage standad deviations than those fo c = 3. This is caused by an incease in the numbe of combinations of data points and clustes. 64

11 In Table 8, it can be seen that q fo has the lagest standad deviations. This is consideed to be caused by the significant standad deviations of shown in Table, suggesting a vaiation of the estimation of the adius of the distibution. On the othe hand, q fo has the smallest standad deviations. Substituting the values of β and q in Table 3 and Table 8 diectly, Figue 6 and Figue 7 compae the membeship function fo the cluste C, u ( x) = { β ( q) x v } q { β ( q) x v j= j } 5 q (9) with ( c+ L( x v) ) qβ 4 β β u ( x) = x v x v + c c c ( cl( x v) cql( x v) + L ( x v) β) β +. 3 c fo c = 5, = and, and N = 0 and 0,000. In the equations, v j is set to each of the cluste coodinates in Figue 3(b). The data pojections on the xz and yz planes ae also plotted. u x and The figues show no significant diffeence between u ( x ) and ( ) between and. (0) Figue 6. Compaisons of the membeship functions calculated by Equations (9) and (0) ( c = 5, =, N =, 00, 000 ). 65

12 Figue 7. Compaisons of the membeship functions calculated by Equations (9) and (0) ( c = 5, =, N =, 00, 000 ). Compaed with the clustes in Figue 3(a), those in Figue 3(b) ae not aligned in a staight line. Howeve, the esults fo c = 5 ae as accuate as those fo c = 3. As a esult, the imum eo facto is consideed to be β. Since the clustes in Figue 3(a) ae aligned in a staight line, β cannot be detemined optimally by locating clustes andomly as does in the algoithm in Figue. Fom these esults, it can be confimed that N = 00 is sufficient to detemine both β and q fo the data sets in Figue. 5.. Expeiment In this expeiment, the Iis Data Set [6], which compises data fom 50 iis flowes with fou-dimensional vectos, is used. The thee clustes to be detected ae Vesicolo, Viginia and Setosa, and the paametes in the algoithm in Figue ae set as follows: δ = 0.0, and δ = 0.0, and T = 0.8. R,, and ae 5.90,.95 and.97, espectively Detemination of Paametes The imum, minimum,, and standad deviation of β,, and q ae summaized in Table 5, Table 6 and Table 9. Figue 8 shows the plots of the imum, minimum, and of q. Based on the esults in Table 5, the value of q was calculated by fixing β to.076e-0. Fom Table 5, it can be seen that a dependency of β on N is same as those in Table and Table 3. Table 6 shows that the of and 66

13 Figue 8. Maximum, minimum, and of q fo the Iis data set ( β =.076e 0 ). Table 5. Maximum, minimum,, and standad deviation of β fo the Iis data set. N Maximum Minimum Mean Std. deviation 0.455e e 0.097e 0.554e e e 0.08e 0.765e e e 0.075e 0.87e 0 0, e e 0.076e 0.949e 0 Table 6. Maximum, minimum,, and standad deviation of Iis data set. and fo the N Maximum Minimum Mean Std. deviation , , can be calculated egadless of the value of N. Table 9 shows that the standad deviations of q fo and ae smalle than those of and showing the effectiveness of the poposed method. It can be found that these tables show that the poposed method gives simila esults to those in the Section 5., and N = 5 to 0 is sufficient to detemine β,,, and q. In the algoithm shown in Figue, it is unnecessay to 67

14 epeat the calculations of the s of β and q the same numbe of times N. It is also found that not only the estimations of the adius ae impotant to impove the accuacy because gives supeio esult compaed with those of. Fo this eason, a pepocessing method that can estimate the location of clustes quicly, such as the Canopy method [7] might be suitable fo the poposed method to be moe effective Clusteing Accuacy The imum and numbe of data points misclassified by the pevious method [4], the poposed method, and Tsallis-DAFCM in, 000 tials ae summaized in Table 0 and Figue 9. T = β is fixed to /.076e 0 = In Tsallis-DAFCM, as a typical value, q is changed fom. to.8. Even though the expeiment was epeated 000 times, the esults obtained with the poposed method wee almost identical. By compaing the numbe of misclassified data points of the poposed method with those of the pevious method, it can be confimed the esults fom both methods ae not significantly diffeent when = and = o when = and =. By compaing the numbe of misclassified data points of the poposed method with those of Tsallis-DAFCM, it can be confimed the poposed method can get slightly bette esults. By examining the imum numbe of misclassified, we see that Tsallis-DAFCM misclassifies data moe often than does the poposed method. These esults confim that appopiate values of q and T = β fo the Iis Data Set can be estimated by the poposed method. Setting = is most suitable fo this data set Computational Time Figue 0 compaes the of computational times of β and q, and clus- Figue 9. Maximum, minimum, and numbes of misclassified data points fo the Iis Data Set of the pevious method, the poposed method and Tsallis-DAFCM ( β =.076e 0, T = 9.94 ). 68

15 (a) (b) (c) Figue 0. Mean of computational times of β,q, and clusteing fo the Iis Data Set. (a) β ; (b) q ; (c) Clusteing. teing in 000 tials (Executions wee conducted on an Intel(R) Coe(TM) Duo CPU GHz). Figue 0(a) shows that the computational time of β does not depend on and inceases popotionally to N because, as can be seen fom Equations 69

16 () and (3), the value of β is detemined independently of and L is calculated N times. Figue 8(b), on the othe hand, shows that the calculation of q fo sometimes taes time suggesting that, in this case, give an appopiate q value. Figue 0(c) shows that that when is set to o be conducted quicly and stably Evaluation of the Poposed Algoithm becomes too lage to, clusteing can Fom the expeimental esults in 5. and 5., the effectiveness of the poposed algoithm using K-s can be evaluated as follows: ) and can be obtained with vey small vaiances without consuming much computational time; ) q can be detemined with a vey small vaiance using without consuming; 3) Much computational time; 4) The numeical data sets and the Iis Data Set can be clusteed desiably using. 6. Conclusions The Tsallis entopy is a q-paamete extension of the Shannon entopy. FCM with the Tsallis entopy imization has a pope chaacteistic fo clusteing, especially when it is combined with DA as Tsallis-DAFCM. The extent of its membeship function stongly depends on the paamete q and the initial annealing tempeatue T. In this study, we poposed a method fo appoximating the membeship function of Tsallis-DAFCM which, by using the K-s method as a pepocessing step, detemines q and β = T automatically and algebaically fom a given data set. Expeiments wee pefomed on the numeical data sets and the Iis Data Set, and showed that the poposed method can moe accuately and stably detemine q and β algebaically than the pevious method without consuming much computational time. It was also confimed that the data can be patitioned into clustes appopiately using these paametes. In the futue, as descibed in 5., we fist intend to exploe ways to impove the accuacy of the estimates fo β and q by using othe ough clusteing methods. We then intend to examine the effectiveness of the method using vey complicated eal wold data set [8]. Refeences [] Rose, K. (998) Deteministic Annealing fo Clusteing, Compession, Classification, Regession, and Related Optimization Poblems. Poceedings of the IEEE, 86, [] Rose, K., Guewitz, E. and Fox, B.C. (990) A Deteministic Annealing Appoach to 60

17 Clusteing. Patten Recognition Lettes,, [3] Kipatic, S., Gelatt, C.D. and Vecchi, M.P. (983) Optimization by Simulated Annealing. Science, 0, [4] Bezde, J.C. (98) Patten Recognition with Fuzzy Objective Function Algoithms. Penum Pess, New Yo. [5] Honda, K. and Ichihashi, H. (007) A Regulaization Appoach to Fuzzy Clusteing with Nonlinea Membeship Weights. Jounal of Advanced Computational Intelligence and Intelligent Infomatics,, [6] Kanzawa, Y. (0) Entopy-Based Fuzzy Clusteing fo Non-Euclidean Relational Data and Indefinite Kenel Data. Jounal of Advanced Computational Intelligence and Intelligent Infomatics, 6, [7] Tsallis, C. (988) Possible Genealization of Boltzmann-Gibbs Statistics. Jounal of Statistical Physics, 5, [8] Abe, S. and Oamoto, Y. (00) Nonextensive Statistical Mechanics and Its Applications. Spinge, Belin. [9] Gell-Mann, M. and Tsallis, C. (004) Nonextensive Entopy Intedisciplinay Applications. Oxfod Univesity Pess, New Yo. [0] Menad, M., Couboulay, V. and Dadignac, P. (003) Possibilistic and Pobabilistic Fuzzy Clusteing: Unification within the Famewo of the Non-Extensive Themostatistics. Patten Recognition, 36, [] Menad, M., Dadignac, P. and Chibelushi, C.C. (004) Non-Extensive Themostatistics and Exteme Physical Infomation fo Fuzzy Clusteing. Intenational Jounal of Computational Cognition,, -63. [] Yasuda, M. (00) Deteministic Annealing Appoach to Fuzzy C-Means Clusteing Based on Entopy Maximization. Advances in Fuzzy Systems, 0, Aticle ID: [3] Yasuda, M. and Oito, Y. (04) Multi-Q Extension of Tsallis Entopy Based Fuzzy C-Means Clusteing. Jounal of Advanced Computational Intelligence and Intelligent Infomatics, 8, [4] Yasuda, M. (06) Appoximate Detemination of Q-Paamete fo FCM with Tsallis Entopy Maximization. Poceedings of the Joint 8th Intenational Confeence on Soft Computing and 7th Intenational Symposium on Advanced Intelligent Systems, [5] MacQueen, J. (967) Some Methods fo Classification and Analysis of Multivaiate Obsevations. Poceedings of the 5th Beeley Symposium on Mathematical Statistics and Pobability, Vol., [6] UCI Machine Leaning Repositoy (998) Iis Data Set. [7] McCallum, A., Nigam, K. and Unga, L.H. (000) Efficient Clusteing of High Dimensional Data Sets with Application to Refeence Matching. Poceedings of the 6th ACM SIGKDD Intenational Confeence on Knowledge Discovey and Data Mining, [8] De, T., Bunet, D.F. and Chattopadhyay, A.K. (06) Clusteing Lage Numbe of Extagalactic Specta of Galaxies and Quasas though Canopies. Communication in Statistics Theoy and Methods, 45,

18 Appendix Table 7. Maximum, minimum,, and standad deviation of q ( c = 3, β = 5.35e 06 ). N Maximum Minimum Mean Std. deviation , , , , Table 8. Maximum, minimum,, and standad deviation of q ( c = 5, β = 3.608e 06 ). N Maximum Minimum Mean Std. deviation , , , ,

19 Table 9. Maximum, minimum,, and standad deviation of q fo the Iis Data Set (.076e 0 β = ). N Maximum Minimum Mean Std. deviation , , , , Table 0. Maximum, minimum, and numbes of misclassified data points fo the Iis Data Set of the pevious method, the poposed method and Tsallis-DAFCM ( β =.076e 0, T = 9.94 ). Method N q Maximum Minimum Mean Pevious method ( ) , Pevious method ( ) ,

20 Continued Poposed method ( ) , Poposed method ( ) , Tsallis-DAFCM Table. Computational times of β, q, and clusteing fo the Iis data set. N β q Clusteing , , , ,

21 Submit o ecommend next manuscipt to SCIRP and we will povide best sevice fo you: Accepting pe-submission inquiies though , Faceboo, LinedIn, Twitte, etc. A wide selection of jounals (inclusive of 9 subjects, moe than 00 jounals) Poviding 4-hou -quality sevice Use-fiendly online submission system Fai and swift pee-eview system Efficient typesetting and poofeading pocedue Display of the esult of downloads and visits, as well as the numbe of cited aticles Maximum dissemination of you eseach wo Submit you manuscipt at: O contact jsea@scip.og