Data Exploration and Unsupervised Learning with Clustering

Transcription

1 Data Exploration and Unsupervised Learning with Clustering Paul F Rodriguez,PhD San Diego Supercomputer Center Predictive Analytic Center of Excellence

2 Clustering Idea Given a set of data can we find a natural grouping? X 2 Essential R commands: D =rnorm(12,0,1) #generate 12 #random normal X1 =matrix(d,6,2) #put into 6x2 matrix X1[,1]=X1[,1]+4; #shift center X1[,2]=X1[,2]+2; #repeat for another set of points #bind data points and plot plot(rbind(x1,x2), xlim=c(-10,10),ylim=c(-10,10)); X 1

3 Why Clustering A good grouping implies some structure In other words, given a good grouping, we can then: Interpret and label clusters Identify important features Characterize new points by the closest cluster (or nearest neighbors) Use the cluster assignments as a compression or summary of the data

4 Clustering Objective Objective: find subsets that are similar within cluster and dissimilar between clusters Similarity defined by distance measures Euclidean distance Manhattan distance Mahalanobis (Euclidean w/dimensions rescaled by variance)

5 Kmeans Clustering A simple, effective, and standard method Start with K initial cluster centers Loop: Assign each data point to nearest cluster center Calculate mean of cluster for new center Stop when assignments don t change Issues: How to choose K? How to choose initial centers? Will it always stop?

6 Kmeans Example For K=1, using Euclidean distance, where will the cluster center be? X 2 X 1

7 Kmeans Example For K=1, the overall mean minimizes Sum Squared Error (SSE), aka Euclidean distance Essential R commands: Kresult = kmeans(x,1,10,1) #choose 1 data point as initial K centers #10 is max loop iterations #1 is number of initial sets to try #Kresult is an R object with subfields Kresult$cluster #cluster assignments Kresults$tot.withinss # tot within SSE

8 Kmeans Example K=1 K=2 Essential R commands: inds=which(kresult$cluster==k) plot(x[inds,],col2use= red ); K=3 K=4 As K increases individual points get a cluster

9 Choosing K for Kmeans Essential R commands: Total Within Cluster SSE for (num_k in 1:10) { Kres=kmeans(X,num_k,10,1); Save and then plot Kres$tot.withinss K=1 to 10 - Not much improvement after K=2 ( elbow )

10 Kmeans Example more points How many clusters should there be?

11 Choosing K for Kmeans Total Within Cluster SSE K=1 to 10 - Smooth decrease at K 2, harder to choose - In general, smoother decrease => less structure

12 Kmeans Guidelines Choosing K: Elbow in total-within-cluster SSE as K=1 N Cross-validation: hold out points, compare fit as K=1 N Choosing initial starting points: take K random data points, do several Kmeans, take best fit Stopping: may converge to sub-optimal clusters may get stuck or have slow convergence (point assignments bounce around), 10 iterations is often good

13 Kmeans Example uniform K=1 K=2 K=3 K=4

14 Choosing K - uniform Total Within Cluster SSE K=1 to 10 - Smooth decrease across K => less structure

15 Kmeans Clustering Issues Scale: Dimensions with large numbers may dominate distance metrics Outliers: Outliers can pull cluster mean, K-mediods uses median instead of mean

16 Soft Clustering Methods Fuzzy Clustering Use weighted assignments to all clusters Find min weighted SSE Expectation-Maximization: Mixture of multivariate Gaussian distributions find best cluster means & variances that maximize likelihood

17 Kmeans unequal cluster variance

18 Choosing K unequal distributions Total Within Cluster SSE K=1 to 10 - Smooth decrease across K => less structure

19 EM clustering Selects K=2 (Bayesian Information Criterion) Handles unequal variance R: library( mclust ) em_fit=mclust(x); plot(em_fit);

20 Kmeans computations Distance of each point to each cluster center For N points, D dimensions: each loop requires N*D*K operations Update Cluster centers only track points that change, get change in cluster center On HPC: Distance calculations can be partitioned data across dimension

21 R Kmeans Performance 1 Gordon compute node, normal random matrices R: system.time(kmeans()) Wall Time (secs) 60 min 30 min 8000 pts Number of Points (i.e. rows) 1K 4K 8K 16K 32K Number of Dimensions (i.e. columns in data matrix) 4000 pts 2000 pts 1000 pts

22 Kmeans vs EM performance 1 Gordon compute node, normal random matrices R: system.time(mclust()) Wall Time (secs) 15 min 10 min EM: 1000 pts Number of Points (i.e. rows) Kmeans:1000 pts 1K 4K 8K Number of Dimensions (i.e. columns in data matrix)

23 Kmeans big data example 45,000 NYTimes articles, 102,000 unique words (UCI Machine Learning repository) Full Data Matrix: 45Kx102K ~ 40Gb article 1 article 2 article Cell i,j is count of i th -word in j th -article article 45K

24 Matlab original script Distance calculation to cluster center 1. Take difference of column Subtract articles X NxP Cluster_Means MxP square and sum across columns Works better for large N small P

25 Matlab Script Altered 1. take difference of row Subtract X NxP Cluster_Means MxP use dot product Works better for large P and dot( ) will use threads

26 Kmeans Matlab Runtime Matlab Kmeans (original) ~ 50 hours Matlab Kmeans (distributed) ~ 10 hours, 8 threads

27 Kmeans results cluster means shown with coordinates determining fontsize 7 viable clusters found

28 Incremental & Hierarchical Clustering Start with 1 cluster (all instances) and do splits OR Start with N clusters (1 per instance) and do merges Can be greedy & expensive in its search some algorithms might merge & split algorithms need to store and recalculate distances Need distance between groups in constrast to K-means

29 Incremental & Hierarchical Clustering Result is a hierarchy of clusters displayed as a dendrogram tree Useful for tree-like interpretations syntax (e.g. word co-occurences) concepts (e.g. classification of animals) topics (e.g. sorting Enron s) spatial data (e.g. city distances) genetic expression (e.g. possible biological networks) exploratory analysis

30 Incremental & Hierarchical Clustering Clusters are merged/split according to distance or utility measure Euclidean distance (squared differences) conditional probabilities (for nominal features) Options to choose which clusters to Link single linkage, mean, average (w.r.t. points in clusters) (may lead to different trees, depending on spreads) Ward method (smallest increase within cluster variance) change in probability of features for given clusters

31 Linkage options e.g. single linkage (closest to any cluster instance) Cluster1 Cluster2 e.g. mean (closest to mean of all cluster instances) Cluster1 Cluster2

32 Linkage options (cont ) e.g. average (mean of pairwise distances) Cluster1 Cluster2 e.g. Ward s method (find new cluster with min. variance) Cluster1 Cluster2

33 Hierarchical Clustering Demo 3888 Interactions among 685 proteins From Hu et.al. TAP dataset b0009 b b0009 b b0014 b b0014 b b0014 b b0014 b b0014 b b0015 b

34 Hierarchical Clustering Demo Essential R commands: >d=read.table("hu_tap_ppi.txt"); Note: strings read as factors >str(d) #show d structure 'data.frame': 3888 obs. of 3 variables: $ V1: Factor w/ 685 levels "b0009","b0014",..: $ V2: Factor w/ 536 levels "b0011","b0014",..: $ V3: num >fs =c(d[,1],d[,2]); #combine factor levels >str(fs) int [1:7776]

35 Hierarchical Clustering Demo Essential R commands: C =matrix(0,p,p); #Connection matrix (aka Adjacency matrix) IJ =cbind(d[,1],d[,2]) #factor level is saved as Nx2 list of i-th,j-th protein for (i in 1:N) {C[IJ[i,1],IJ[i,2]]=1;} #populate C with 1 for connections install.packages('igraph') library('igraph') gc=graph.adjacency(c,mode="directed") plot.graph(gc,vertex.size=3,edge.arrow.size=0,vertex.label=na) or just plot(.

36 Hierarchical Clustering Demo hclust with single distance: chaining d2use=dist(c,method="binary") fit <- hclust(d2use, method= single") plot(fit) the cluster distance when 2 are combined Items that cluster first

37 Hierarchical Clustering Demo hclust with Ward distance: spherical clusters

38 Hierarchical Clustering Demo Where height change looks big, cut off tree groups <- cutree(fit, k=7) rect.hclust(fit, k=7, border="red")

39 Hierarchical Clustering Demo Kmeans vs Hierarchical: Lots of overlap despite that Kmeans not have binary distance option Kmeans cluster assigment Hierarchical Group Assigment groups <- cutree(fit, k=7) ; Kresult=kmeans(d2use,7,10,1); table(kresult$cluster,groups)

40 Dimensionality Reduction via Principle Components Idea: Given N points and P features (aka dimensions), can we represent data with fewer features: Yes, if features are constant Yes, if features are redundant Yes, if features only contribute noise (conversely, want features that contribute to variations of the data)

41 PCA: Dimensionality Reduction via Principle Components Find set of vector (aka factors) that describe data in alternative way First component is the vector that maximizes the variance of data projected onto that vector K-th component is orthogonal to all k-1 previous components

42 PCA on 2012 Olympic Althetes Height by Weight scatter plot Height- cm (mean centered) PC2 PC1 Projection of (145,5) to PCs Total Variance Conserved: Var in Weight + Var in Height = Var in PC1 + Var in PC2 In general: Var in PC1> Var in PC2> Var in PC3 Weight- Kg (mean centered)

43 PCA on Height by Weight scatter plot Height- cm (mean centered) Essential R: X = X-mean; #mean center data S <- svd(x); #returns matrices U,D,V #cols in V are 2D vectors # S = U*D*V #plot each V col as a line in X s space (use 7 th grade geometry) points(.., type= l ); #line type #get the 1 st coord. point in X s space S$u[,1]*S$v[,1]*S$d[1] #repeat for 2 nd coord. point and plot Weight- Kg (mean centered) #PCA is eigen()

44 Principle Components Can choose k heuristically as approximation improves, or choose k so that 95% of data variance accounted aka Singular Value Decomposition PCA on square matrices only SVD gives same vectors on square matrices Works for numeric data only In contrast, clustering reduces to categorical groups In some cases, k PCs k clusters

45 Summary Having no label doesn t stop you from finding structure in data Unsupervised methods are somewhat related

46 End