Defining Clustering:! Challenges and Previous Work. Part II. Margareta Ackerman

Transcription

1 Defining Clustering: Challenges and Previous Work Part II Margareta Ackerman

2 Kleinberg s axioms Scale Invariance: A(X,d) = A(X,cd) for all (X,d) and all strictly positive c. Consistency: If d equals d, except for shrinking distances within clusters of A(X,d) or stretching between-cluster distances, then A(X,d ) = A(X,d). Richness: For any clustering C of X, there exists a distance function d over X so that A(X,d) = C.

3 Theorem [Kleinberg, 02]: These axioms are inconsistent. Namely, no function can satisfy these three axioms. Why are axioms that seem to capture our intuition about clustering inconsistent?? It turns out the formalism is stronger than the intuition that it intends to capture.

4 Problem: Consistent changes to the underlying distance should not create any new contenders for the best clustering of the data. C C 0 A clustering function that satisfies Kleinberg s Consistency cannot output C 0.

5 Overcoming the impossibility result Consistency is too strong. Therefore, Kleinberg s impossibility doesn t imply that clustering is impossible to axiomatize. There are several ways to achieve consistency of the three axioms by changing the underlying framework.

6 Clustering quality measures How good is this clustering? Clustering-quality measures quantify the quality of clusterings.

7 Defining clustering quality measures A clustering-quality measure is a function m(dataset, clustering) 2 R satisfying some properties that make this function a meaningful clustering quality measure. What properties should it satisfy?

8 Rephrasing Kleinberg s axioms for clustering quality measures Scale Invariance m(c, d) =m(c, d) for all C, d and strictly positive. Richness For any clustering C of X, there exists a distance function d over X so that C = argmax C m(c, d)

9 Consistency: If d 0 equals d, except for shrinking distances within clusters of C or stretching between-cluster distances, then m(c, d) apple m(c, d 0 ). d d 0 C C

10 Major gain - consistency of new axioms Theorem [Ackerman & Ben-David, NIPS 08]: Consistency, scale invariance, and richness for clustering quality measures form a consistent set of requirements. Dunn s index ( 73): min x6 C y d(x, y) max x C y d(x, y) This clustering quality measure satisfies consistency, scale-invariance, and richness.

11 Consistency of new axioms Proof: Consider Dunn s index ( 73): min x6 C y d(x, y) max x C y d(x, y) This clustering quality measure satisfies consistency, scale-invariance, and richness.

12 Consistency of new axioms Proof: Consider Dunn s index ( 73): min x6 C y d(x, y) max x C y d(x, y) Scale invariance is satisfied because the constant cancels out.

13 Consistency of new axioms Proof: Consider Dunn s index ( 73): min x6 C y d(x, y) max x C y d(x, y) Consistency is satisfied because, following a consistent change, the numerator can only increase, and denominator can only decrease. As such, the quality of the clustering (using Dunn s index) can only increase.

14 Consistency of new axioms Proof: Consider Dunn s index ( 73): min x6 C y d(x, y) max x C y d(x, y) To show that richness holds, consider any clustering C. Set d so that in-cluster dissimilarities are all 1, and between cluster dissimilarities are 2. Then Dunn s index of C is 2, and no other clustering has a larger index.

15 Additional measures satisfying our axioms C-index (Dalrymple-Alford, 1970) Gamma (Baker & Hubert, 1975) Adjusted ratio of clustering (Roenker et al., 1971) D-index (Dalrymple-Alford, 1970) Modified ratio of repetition (Bower, Lesgold, and Tieman, 1969) Variations of Dunn s index (Bezdek and Pal, 1998) Strict separation (Balcan, Blum, and Vempala, 2008) And many more...

16 Why is the quality measure formulation more faithful to intuition? In the earlier setting of clustering functions, consistent changes to the underlying distance should not create any new contenders for the best clustering of the data. d C d 0 C 0 A clustering function that satisfies Kleinberg s Consistency cannot output C 0.

17 Why is the quality measure formulation more faithful to intuition? d In the setting of clustering-quality measures, consistency requires only that the quality of clustering Cnot get worse. d 0 C 0 C A different clustering can have better quality than the original.

18 Axioms of clustering? The above result showed that Kleinberg s axioms are consistent in the setting of clustering quality measures. Next class, we will show that there is an even simpler way to overcome Kleinberg s impossibility result: Allow the user to specify the number of clusters. So do we have axioms of clustering? :-) No. :-(

19 Property VS Axioms A set of axioms should be 1. Sound: Every object in the class should satisfy all of the axioms. 2.Complete: Any object outside the class should fail at least one of these axioms. Since we don t have a definition of clustering, how would we know if a set of axioms if sound and complete?

20 Property VS Axioms Minimum requirement: 1. Relaxed soundness: Existing clustering algorithms (especially those useful in practice) should satisfy all the axioms 2. Relaxed completeness: Any object we can come up with that should not be called a clustering algorithm should fail at least one of the axioms. Most clustering axioms proposed in the literature are properties, and not axioms. This leaves open the problem of defining, or axiomatizing, clustering.