A Decision Model for Fuzzy Clustering Ensemble

Transcription

1 A Decision Model for Fuzzy Clusering Enseble Yanqiu Fu Yan Yang Yi Liu School of Inforaion Science & echnology, Souhwes Jiaoong Universiy, Chengdu 6003, China Absrac Algorih Recen researches and experiens have showed ha clusering enseble approaches can enhance he robusness and sabiliies of unsupervised learning grealy. Mos of he focused on crisp clusering cobinaion. However, in his paper, we offer a decision odel based on fuzzy se heory for fuzzy clusering enseble. Firsly, obain he opial pariion called exper fro H individual fuzzy pariions generaed by fuzzy c-eans algorih. Then, use fuzzy voing schee o generae he ajoriy judger. Finally, he wo arixes are cobined by Decision Model. Experienal resuls show he effeciveness of he proposed ehod coparing o he resuls based on crisp clusering. Keywords: Fuzzy Clusering Enseble, Fuzzy Valid Funcion, Siilariy Measure, Coprehensive Decision Model. Inroducion Clusering algorihs provide an iporan ean o explore srucure wihin he unlabelled daa by organizing i ino groups or clusers. Many clusering algorihs have exised, bu no single algorih can adequaely handle all sors of cluser shapes and srucures []. The exploraory naure of clusering asks deand efficien ehods which would benefi fro he srenghs of any individual clusering algorihs. Inspired by he success of cobining of uliple classifiers in iproving he qualiy of daa classificaions, he sae idea applied in inegraion of uliple clusering, which is considered o be an exaple o furher broaden and siulae new progress in he area [], he basic process of clusering cobinaion is illusraed in Figure. In a recenly review, here are any approaches proposed o cope wih hese probles []-[6], and os of researches ineres in wo ain fields: generae differen uliple cluserings and design he differen consensus funcion [3]. Be noed of he relaed lieraures above, few approaches focused on he cobinaion of fuzzy clusering. Here below, we show soe researches on his filed. X Daa ses Algorih Algorih 3. Algorih H H Fig.: The process of clusering cobinaion. Finding a fuzzy consensus clusering or generae fuzzy co-associaion arix are boh feasible way in fuzzy clusering enseble [7]. Fuzzy co-associaion arix based on hree fuzzy siilariy beween daa poins is generaed in Ref[7], Ref[8] inegraes fuzzy c-eans (FCM) algorih and fuzzy k-neares neighbors (FKNN) algorih hrough uilizing seleced good resuls produced by FCM o each he FKNN algorih. Oher relaed works are ineresing in he applicaion of fuzzy clusering ensebles [9], [0]. In his paper, we ry o presen a novel way for fuzzy consensus clusering, ha is called Decision Model. In secion, we inroduce fuzzy valid funcions, fuzzy voing schee, siilariy easure and he coplee decision process for cobining fuzzy cluserings. Secion 3 is experienal analysis. Finally in secion 4, we give he conclusion and discuss he fuure work.. A Dicision Model 3 Consensus funcion In he field of fuzzy clusering enseble, we can use he general clusering enseble ehod direcly because of he vagueness an objec belongs o any cluser. Thus, we should find an appropriae way o sui for fuzzy clusering cobinaion. Here, we presen a novel decision odel for fuzzy clusering enseble. To express he core idea, we ake copeiion in our life for exaple. Soeies as we know boh expers and ajoriy judgers opinions should be aken ino consideraion for selecing he bes ones. In his paper we exend he sae concep o our fuzzy clusering enseble. Firs, we choose an opial pariion as exper fro H pariions generaed by FCM algorih, hen produce ajoriy judger by using fuzzy ajoriy voing schee. Finally, he clusering resuls are given hrough our Decision Model. The whole process is T

2 illusraed in Table. The deailed algorih will be posed in.,. and.3 secions:.. FCM algorih and Opion of fuzzy validiy funcions Suppose X= { X, X X } is a se of N daa saples (objecs) in d-diensional space, where X i { x i, x i x } = represens he ih saple for i=,,,n. We use FCM algorih o generae H fuzzy pariions arix = { π,, π } by randoly H selecing c cluser ceners and iniial cluser prooype odel. For he ih pariion π i, here is a ebership degree ij ( [0,])indicaing wih wha degree he saple X i belongs o he cluser cener. These H pariions have soe diversiy fro each oher since he differen seings of he iniial condiions and paraeers. As for cluser validiy funcions which are ofen used o evaluae he perforance of clusering in differen indexes and even wo differen clusering ehods, a lo of he were proposed during he las 0 years. Aong he funcions, here are wo iporan ypes for FCM. One is based on he fuzzy pariion of saple se, whose represenaive funcions are pariion coefficien (F pc ) and pariion enropy ( F pe ), he oher is on he geoeric srucure of saple se, he represenaive funcions for his ype are N.Zahid-M.Liouri funcion ( F FS ) and he Xie Beni funcion ( F xb ). A brief suary of hese 4 funcions are illusraed in Table. Inpu: n d-diensional saples; H-nuber of pariions; Oupu: Decision resuls of clusering cobinaion Iniializaion: Se c>=, >, e>0; Seps:. Produce H daa pariions: For j = o H do. Randoly choose c cluser ceners and iniial cluser prooype odel;. Run he FCM Algorih and produce H daa pariions;. Obain he opial pariion U ij* called exper fro he H pariions: 3. Generae ajoriy judgerfv ij: For each fuzzy pariion arix, coun he ies one saple belongs o he sae cluser using cuse heory. 4. Decision Model: Design a fuzzy coprehensive decision odel: M ij = U ij*? FV ij: If he exper and he ajoriy judger have consisen resuls, join he saples in he decided cluser, else he saples will be assigned o heir appropriae cluser using siilariy easure. Table : A Decision Model for Fuzzy Clusering Ensebles. I is expeced o choose an opial clusering which os closes o he naural srucure of he given daa se fro hese differen H-pariions. To cope wih his proble, we inroduce he definiion of cluser-validiy analysis. The process of choosing a pariion fis wih he unknown srucure of he inpu daa se os using soe kind of cluser validiy funcions is called cluser-validiy analysis []. Table : A brief suary of four seleced validiy funcions. Based on he soe experiences of oher lieraures, we selec N.Zahid-M.Liouri funcion ( F FS ) o be our validiy funcion. [0,] In fac, F akes ino accoun siulaneously he properies of he fuzzy ebership degrees and he srucure of he daa iself, i is defined as: FS c n F (, ; ) [( ) FS U V x = x k V i ] i= k= c n [( ) V V i ] i= k = = J ( UV, ; X ) K( U, V; X) () where V is he ean of he whole daa se, J is a copacness easure and K is a separaion easure beween clusers ceners and he ean V. The arix U * ij which has he iniu value of F FS is recognized o be he opial pariion and called exper in his paper. Since he exper has been seleced fro all he pariions in his secion, one iporan judger is

3 generaed. However, considering he inforaion only fro he exper is no fully ideal and unilaeral, fuzzy ajoriy voing schee of clusering individuals is needed and will be proposed in he nex secion... Fuzzy ajoriy voing schee The idea behind fuzzy ajoriy voing is he judgen of a group is superior o hose of individuals. This concepion has been successfully explored in cobining hard clusers o accoun he co-associaion values, However, in fuzzy clusering, we can' accoun values le in he general cluser enseble ehod by using saisic of how ofen each pair of saples appears in he sae cluser because we jus obain he differen degrees of ebership ha every saple belongs o is own cluser. As a soluion of his proble, he concepion of cu-se level based on fuzzy se heory is inroduced in his paper. The process is described as following: Le F(X) be a fuzzy se, if A F( X) and α [0,], he cu-se level is a disinc se defined by A = { xx X, ( x) α}.for any pariions α A c π = c, =,,,H, ij n n nc copue each row s cu-se level, ha is A = { xx π, ( x) α} (k=,,,c), α can be fixed by 0.5. α k ij According o he characerisic funcion which is defined as, x A ( x ) i χ A i = 0, x i A, he probabiliy of each saple belongs o a cerain cluser can be couned by he equaion of H j χ A ( x i ) fv = ij H =, i=,,,n; j=,,,c, () which resuls in he fuzzy voing arix FV ij, where Each (i, j) is herefore a voe owards heir gahering in a cluser. Now, anoher easure called ajoriy judger is also produced..3. Decision odel To ake ino accoun siulaneously he opinion of he exper and he aiude of ajoriy judger, a decision odel based on fuzzy se heory has been proposed in his secion. A brief specificaion of he deciding process is firsly given as follows: As he daa se X= { X, X X } given before, for each saple X i ( i=,,,n), is cluser is fixed by he following descripion ha is for each row, obain he ax value of exper judger fv i = ax { fv ij } j n = ax { ij} j n and he ajoriy, if k=, join he X i o he kh (or h) cluser, else he cluser label is uncerain, which sees le a conradicion exiss beween wo judgers. In his case, soe ore discussion should be aken as he nex sep. Noing ha he exper is consan since i s one of he daa pariions while he ajoriy judger is variable because i depends on he perforance of pariions, he exper considered o be he auhoriy when he resul of he ajoriy judger is no ideal, for insance, every cell in ajoriy judger arix is equal, which suggess he voing is fail or he diversiy of ajoriy judger value for soe saples is low, which ax fv in fv <0.. Soe can be represened by { ij} { ij} j n j n oher exaples should be furher sudied in he fuure work. As for he reaining unlabeled saples, we can use he siilariy easure o deal wih his proble. Now, here are any ways o define he siilariy easure, such as he relaed coefficien and Euclidean disance, ec. Moivaed by sipliciy and easyanipulaion, we choose he relaed coefficien easure because i s os frequenly used [4]. The siilariy easure rij is defined as follows: where ( x x i)( x jk x j) k= r ij = ( ) x x ( x x ) i jk j k= k= xi = x k =,, xj = x jk k = (3) Observing he equaion, i is easy o see he siilariy easure well express he associaion beween each pair of saples, wih he idea ha a higher value indicaes greaer siilariy. Since siilariy easure r is called for each x can ij uniforly disribue in [0, ]. However, os real daa ses ay no ee he requireen of unifor disribuion in [0, ]. To ee he need of fuzzy siilariy easure, he following ransfor is necessary: in { x } i n { } in { } x ' x = ax x x i n i n (4) where k=,,,, i=,,,n, each x is a cell in he given daa se X= { X, X X }. We assue ha neighboring saples wihin a naural cluser easured by siilariy are very lely o be locaed in he sae group, which iplies ha he saples wihin one cluser are ore siilar (r ij? ) and dissiilar saples are ore separae (r ij? 0), Therefore, we can join he unlabeled saple o he ij

4 sae cluser in which one labeled saple who has axiizaion of siilariy degrees wih i. Repea his process unil all he unlabeled saples are assigned. 3. Experienal resuls A reasonable approach o easure he perforance of he proposed decision odel is o copue he F- easure values on various daases since our experiens were conduced wih daases where rue naural clusers are known. The daases include arificial, UCI and web ypes, soe characerisics of he daases are showed in Table 3. Daases 3D-3C 3D-C Iris Zoo Web Web Types Arificial Arificial UCI UCI WEB WEB No.of feaures No.of poins Table 3: Characerisics of he daases. No.of clusers The wo arificial daases of 3-diensional poins are arificially generaed fro Gaussian disribuions wih differen eans and co-variance arices, and he sizes of he are as follows (00, 00, 00) and (00, 00). Fro Figure, we can easily see he disribuions of he wo daases have saisfacory feaure ha is he saples wihin one cluser are siilar and differen ceners are separae. Iris and Zoo daases are available fro he UCI achine learning reposiory a [3]. The Iris daase consiss of hree classes (Seosa, Versicolor and Virginica), wih 50 insances per class, represened by 4 feaures. The zoo daase consiss of 0 insances represened by 6 nueric aribues. Finally, we esed wih wo preprocessed web daases which obained fro news docuens downloaded fro Yahoo English Websie by RSS paern.? 3D-3C Fig : Disribuions of he wo arificial daases (3D-C and 3D-3C). For coparison fuzzy clusering and crisp clusering, we perfor FCM algorih and K-eans algorih based on Euclidean Disance. However, FCM algorih is used o generae H pariions while K-eans algorih is used o produce he final resuls direcly. For sake of unificaion of coparison and based on Ref [], we se H=0 for all he daases in each of which he generaed H pariions algorih are cobined by running he Decision Model independenly. Then copue F-easure values of all he given daases wih he wo ehods respecively, which are repored in Table 4 and he coparaive resuls are illusraed in Figure 3. Table 4: F-easure values of K-eans algorih and he Decision Model. Figure 4 includes six pie chars of six daases clusering resuls produced by he proposed ehod, which specifically show which saples belong o heir given clusers. Fro he resuls, i is easy o see ha he resuls fro decision odel are generally beer han crisp one in he sae condiions. In he case of Web and Web daases, he F-easure values iprove by bigger argin han oher daases, which suggess ha he proposed ehod in his paper is ore suiable for WEB daases. The reason can be considered ha he proposed ehod include ore inforaion han he crisp one since i consis of wo judgers. (i)3d-c

5 (? ) Web Fig 3: he coparaive resuls of K-eans algorih and he Decision Model.? 3D-C (? ) Web Fig 4: Six pie chars of six daases clusering resuls produced by he Decision Model. 4. Conclusions and fuure work (? )3D-3C (? ) Iris (? )Zoo The purpose of his paper is o presen a novel decision odel for fuzzy clusering enseble. In our life, especially in soe copeiive aciviies, o selec he opial ones, boh expers and he ajoriy s opinion should be aken ino consideraion. Once he expers and he ajoriy have consensus, final decision can be ade, or soe exra discussion should be aken. In his paper he sae concep has been exended o cobine daa pariions which produced by fuzzy clusering algorihs. As we know, differen clusering algorihs or a single clusering algorih wih differen iniializaions will in general produce differen pariioning resuls. In crisp clusering pariions, i is ipossible o copare differen crisp pariions wih he validiy funcions based on he fuzziness of pariions, since he fuzziness of any crisp pariion is zero, no ideal pariion can be available. Bu in fuzzy clusering analysis, he opial pariion which called exper in his aricle can be easily chosen by he fuzzy cluser-validiy crierion and opinions of he ajoriy could be obained hrough fuzzy voing schee, finally, synhesizing he wo arixes resuls in he uliae decision. If exper and he ajoriy have differing opinions, fuzzy siilariy easure would be adoped o solve his proble.

6 Based on heoreical analysis and experiens, we can conclude ha he novel decision odel is suiable for fuzzy clusering enseble and i is beer han he ehod which crisp clusering enseble adops. However, soe cases when conradicion beween ajoriy judgers and exper happened should be discussed in he nex sep and ongoing work will be focused on designing a ehod or exrapolaion of his ehodology o fi for dealing wih large daa se in fuzzy clusering enseble. References [] A. Fred, Finding Consisen Clusers in Daa Pariions, Proceedings of he nd Inernaional Workshop on Muliple Classifier Syses, Volue 096 of Lecure Noes in Copuer Science, Springe, pp , 00. [] A Srehl, J Ghosh. Cluser Ensebles A Knowledge Reuse Frae work for Cobining Muliple Pariions. Journal of Machine Learning Research, 3:583-67, 003. [3] B. Minaei-Bidgoli, A Topchy, W F Punch. A Coparison of Resapling Mehods for Clusering Ensebles. Inl Conf on Machine Learning. Models, Technologies and Applicaions( MLMTA 004), pp , 004. [4] T. Alexander, Anil K. Jain, and Willia Punch, Cobining Muliple Weak Cluserings, Proceedings of he Third IEEE Inernaional Conference on Daa Mining (ICDM 03), 003. [5] H. Ayad, M. Kael, Finding Naural Clusers Using Muli-Cluserer Cobiner Based on Shard Nears Neighbors. Proceedings of he 4 h Inernaional Workshop on Muliple Classifier Syses(MCS 03), Volue 709 of Lecure Noes in Copuer Science, Springer, pp.66-75, 003 [6] B. Minaei-Bidgoli, A Topchy, W F Punch, A Coparison of Resapling Mehods for Clusering Ensebles. Inl Conf on Machine Learning Models, Technologies and Applicaions( MLMTA 004), pp , 004 [7] L. Y. Yang, H.R. Lv and W.Y. Wang, Sof Cluser Enseble Based on Fuzzy Siilariy Measure, IMACS Muliconferenceon CESA, pp , 006. [8] K. Josien, T. Warren Liao, Siulaneous grouping of pars and achines wih an inegraed fuzzy clusering ehod Fuzzy Ses and Syses, 6:-, 00. [9] Julinda Gllavaa, Erir Qeli, and Bernd Freisleben, Deecing Tex in Videos Using Fuzzy Clusering Ensebles, Proceedings of he Eighh IEEE Inernaional Syposiu on Muliedia (ISM'06), 006. [0] J. C. Bezdek, Cluser validiy wih fuzzy ses, J. Cyberni. 3:58-7, 974. [] J. C. Bezdek, Nuerical axonoy wih fuzzy ses, J. Mah.Biol. :57-7, 974 [] G. Xinbo, Fuzzy Cluser Analysis and Applicaion, Xidian Universiy Press, pp.0-3, 004. [3] hp:// l. [4] Y.Fukuyaa, M. Sugeno, A new ehod of choosing he nuber of clusers for he fuzzy c- eans ehod, Proc.5h, Fuzzy Sys. Syp, pp.47-50, 989. [5] J.M.F. Salido and S. Murakai, Rough se analysis of a general ype of fuzzy daa using ransiive aggregaions of fuzzy siilariy relaions. Fuzzy Ses and Syses, 39: , 003. [6] hp://racr.shu.edu.cn/en_index.asp. [7] H.R. Berenji and P.S. Khedkar, Using fuzzy logic for perforance evaluaion in reinforceen learning. Inernaional Journal of Approxiae Reasoning, 8(3):3-44, 998.