Problems of Fuzzy c-means and similar algorithms with high dimensional data sets Roland Winkler Frank Klawonn, Rudolf Kruse

Transcription

1 Problems of Fuzzy c-means and similar algorithms with high dimensional data sets Roland Winkler Frank Klawonn, Rudolf Kruse July 20, 2010

2 2 / 44 Outline 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

3 3 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

4 4 / 44 Prototype based algorithms Objective function that needs to be minimized: J = c n i=1 j=1 f (u ij )d 2 ij c: number of prototypes, n: number of data objects, u ij membership value, f : R R: fuzzifier function, d ij = y i x j, y i : prototype, x j : data object, 1 = c i=1 u ij artificial generated data set 1: 2 dimensions, 4 clusters, 1000 data objects per cluster, 10% noise artificial generated data set 2: 50 dimensions, 100 clusters, 100 data objects per cluster, 10% noise

5 HCM f (u) = u Data set 1 Data set 2, found clusters: 40 5 / 44

6 6 / 44 FCM f (u) = u ω, ω = 2 Data set 1 Data set 2, found clusters: 0

7 NFCM f (u) = u ω, ω = 2, noise cluster at dnoise = 0.3 Data set 1 Data set 2, found clusters: 0 7 / 44

8 PFCM f (u) = 1 β 2 1+β u + 2β 1+β u, β = 0.5 in a 2 cluster environment: u1j = 1 Data set 1 d1j d2j <β Data set 2, found clusters: 90 8 / 44

9 9 / 44 PNFCM f (u) = ( 1 β 1+β u2 + 2β 1+β u ), β = 0.5, d noise = 0.3 Data set 1 Data set 2, found clusters: 0

10 10 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

11 11 / 44 Test setup Generate a data set with n = c data objects representing c clusters of infinite density Place the data objects on a D-dimensional hypersphere surface of radios 1 such that the pairwise distances are maximised Place prototypes in the centre of gravity (cog) and gradually move them to the data objects Observe the objective function

12 12 / 44 Tests Applying the algorithms y i = (1 α) cog + αx i for α [0, 1] Normalise the objective function by the value of the objective function at α = 0 Let α gradually increase from 0 to 1 and monitor the normalized objective function value 2 Test setups Setup 1: 100 prototypes, between 2 and 200 dimensions Setup 2: 50 dimensions, between 2 and 500 prototypes

13 13 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

14 14 / 44 Objective function of HCM 100 prototypes 50 Dimensions various dimensions various prototypes

15 15 / 44 Objective function of FCM f (u) = u ω, ω = prototypes 50 Dimensions various dimensions various prototypes

16 16 / 44 Objective function of NFCM f (u) = u ω, ω = 2, d noise = prototypes 50 Dimensions various dimensions various prototypes

17 17 / 44 Objective function of PFCM f (u) = ( 1 β 1+β u2 + 2β 1+β u ), β = prototypes 50 Dimensions various dimensions various prototypes

18 18 / 44 Objective function of PNFCM f (u) = ( 1 β 1+β u2 + 2β 1+β u ), β = 1 2, d noise = prototypes 50 Dimensions various dimensions various prototypes

19 19 / 44 Tweak the fuzzy based algorithms to work at high dimensions For FCM, a dimension dependent fuzzifier can be used to counter the dimension effect: ω = D With a tweaked fuzzifier, NFCM behaves like PNFCM and needs a dimension dependent noise distance: d noise = 0.5 log 2 (D) PNFCM requires also requires an dimension dependent noise distance: d noise = 0.5 log 2 (D)

20 20 / 44 FCM with adjusted parameters f (u) = u ω, ω = D 100 prototypes 50 Dimensions various dimensions various prototypes

21 21 / 44 FCM with adjusted parameters 50 Dimensions, 100 clusters, found clusters: 92

22 22 / 44 NFCM with adjusted parameters f (u) = u ω, ω = D, d noise = 0.5 log 2 (D) 100 prototypes 50 Dimensions various dimensions various prototypes

23 23 / 44 NFCM with adjusted parameters 50 Dimensions, 100 clusters, found clusters: 91, correct clustered noise: 1000, incorrect clustered as noise: 900

24 24 / 44 PNFCM with adjusted parameters f (u) = ( 1 β 1+β u2 + 2β 1+β u ), β = 0.5, d noise = 0.5 log 2 (D) 100 prototypes 50 Dimensions various dimensions various prototypes

25 25 / 44 PNFCM with adjusted parameters 50 Dimensions, 100 clusters, found clusters: 90, correct clustered noise: 991, incorrect clustered as noise: 0

26 26 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

27 27 / 44 Scoring the algorithms 50 Dimensions, 100 cluster with 100 data objects each, 1000 noise data objects Algorithm Found clusters Correctly clustered noise Incorrect clustered as noise HCM FCM NFCM PFCM PNFCM Adjusted Parameter FCM AP NFCM AP PNFCM AP

28 28 / 44 Effects of high dimensions on the feature space Let S be a hypersphere of D dimensions with radius R. S has a volume of V = C D R D with C D = π D 2 Γ( D +1) only 2 depending on D Let S be a hypersphere of D dimensions with radius R and V = C D R D = 1 2 V Then R = ( ) 1 1 D 2 R Example: D = 2: R = R D = 10: R = R D = 100: R = R D = 1000: R = R

29 29 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

30 Literature Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is nearest neighbor meaningful? In Database Theory - ICDT 99, volume 1540 of Lecture Notes in Computer Science, pages Springer Berlin / Heidelberg, F. Höppner, F. Klawonn, R. Kruse, and T. Runkler. Fuzzy Cluster Analysis. John Wiley & Sons, Chichester, England, Frank Klawonn and Frank Höppner. What is fuzzy about fuzzy clustering? understanding and improving the concept of the fuzzifier. In Cryptographic Hardware and Embedded Systems - CHES 2003, volume 2779 of Lecture Notes in Computer Science, pages Springer Berlin / Heidelberg, / 44

31 Thank you for your attention 31 / 44

32 32 / 44 Current Section 1 FCM (and similar) Algorithms in high dimensions 2 High Dimension Test Environment 3 Effects of high dimensional data sets 4 Conclusions 5 Literature 6 Appendix

33 33 / 44 Hard c-means (HCM) Clustering Problem: data objects X = {x 1,..., x n }, prototypes Y = {y 1,..., y c } n Objective Function: J HCM = min dij 2 S j=1 j S i with d ij = y i x j, S = {S 1,..., S c }, S i N and S i S k = for i k. updated with: S i = { j : d ij d ik k } and y i = 1 x j S i j S i

34 HCM membership values example 34 / 44

35 35 / 44 Fuzzy c-means (FCM) Clustering Problem: data objects X = {x 1,..., x n }, prototypes Y = {y 1,..., y c } Objective Function: J FCM = c n uij ωd ij 2 i=1 j=1 with d ij = y i x j, 1 = c u ij, u ij > 1 and ω > 1. updated with: 2 u ij = d 1 ω ij c 2 d k=1 1 ω kj i=1 and y i = n uij ω x j j=1 n uij ω j=1.

36 FCM fuzzified membership values example 36 / 44

37 37 / 44 Noise Fuzzy c-means (NFCM) Objective Function: J NFCM = c n i=0 j=1 u ω ij d 2 ij with d 0j = d noise 0, d ij = y i x j for i = 1... c, 1 = c u ij, u ij > 1 and ω > 1. i=1 updated with: 2 u ij = d 1 ω ij c 2 d k=0 1 ω kj and y i i=1...c = n uij ω x j j=1 n uij ω j=1

38 NFCM fuzzified membership values example 38 / 44

39 39 / 44 Fuzzy c-means with polynomial fuzzifier (PFCM) J PFCM = c n i=1 j=1 ( ) 1 β 1+β u2 ij + 2β 1+β u ij dij 2 with d ij = y i x j, 1 = c u ij, u ij > 1 and 0 β 1. i=1 with ϕ is a permutation that sorts the prototypes ascending by their distance and ĉ is the number of clusters that have a positive membership value: 1 u ij = 1 β 1+(ĉ 1)β β iff ϕ(i) ĉ ĉ d ij 2 k=1 d ϕ(k)j 2 0 otherwise in a 2 cluster environment: u 1j = 1 d 1j d 2j < β

40 PFCM fuzzified membership values example 40 / 44

41 41 / 44 Noise fuzzy c-means with polynomial fuzzifier (PNFCM) [3] J PNFCM = c n i=0 j=1 ( ) 1 β 1+β u2 ij + 2β 1+β u ij dij 2 with d 0j = d noise 0, d ij = y i x j for i = 1... c, 1 = c u ij, u ij > 1 and 0 β 1. with ϕ and ĉ as for PFCM: 1 u ij = 1 β 1+(ĉ 1)β β iff ϕ(i) ĉ ĉ d ij 2 k=0 d ϕ(k)j 2 0 otherwise i=1

42 PNFCM fuzzified membership values example 42 / 44

43 43 / 44 Maximal separated data objects on a hypersphere surface Sample many D-dimensional normal distributed data objects and project them to length 1 The resulting data objects follow a rotation symmetric probability distribution Run Hard k-means on the random generated data set Project prototypes on the hypersphere surface as new data objects

44 44 / 44 Data set properties Dim Clusters Data per Cluster Angle [ ] Distance Objects (Min - Max) (Min) (Min - Max)