Clustering. CS4780/5780 Machine Learning Fall Thorsten Joachims Cornell University

Size: px

Start display at page:

Download "Clustering. CS4780/5780 Machine Learning Fall Thorsten Joachims Cornell University"

Benjamin Higgins
5 years ago
Views:

1 Clusterng CS4780/5780 Mahne Learnng Fall 2012 Thorsten Joahms Cornell Unversty Readng: Mannng/Raghavan/Shuetze, Chapters 16 (not 16.3) and 17 (

2 Outlne Supervsed vs. Unsupervsed Learnng Herarhal Clusterng Herarhal Agglomeratve Clusterng (HAC) Non-Herarhal Clusterng K-means Mxtures of Gaussans and EM-Algorthm

3 Supervsed Learnng vs. Unsupervsed Learnng Supervsed Learnng Classfaton: partton examples nto groups aordng to pre-defned ategores Regresson: assgn value to feature vetors Requres labeled data for tranng Unsupervsed Learnng Clusterng: partton examples nto groups when no pre-defned ategores/lasses are avalable Novelty deteton: fnd hanges n data Outler deteton: fnd unusual events (e.g. hakers) Only nstanes requred, but no labels

4 Clusterng Partton unlabeled examples nto dsjont subsets of lusters, suh that: Examples wthn a luster are smlar Examples n dfferent lusters are dfferent Dsover new ategores n an unsupervsed manner (no sample ategory labels provded).

5 Applatons of Clusterng Cluster retreved douments to present more organzed and understandable results to user dversfed retreval Detetng near duplates Entty resoluton E.g. Thorsten Joahms == Thorsten B Joahms Cheatng deteton Exploratory data analyss Automated (or sem-automated) reaton of taxonomes e.g. Yahoo-style Compresson

6 Applatons of Clusterng

7 Clusterng Example

8 Clusterng Example

9 Clusterng Example

10 Clusterng Example

11 Clusterng Example

12 Smlarty (Dstane) Measures Euldan dstane (L 2 norm): L 1 norm: L 2 x, x N = x x 2 Cosne smlarty: =1 L 1 x, x N = x x =1 os x, x = x x x x Kernels

13 Herarhal Clusterng Buld a tree-based herarhal taxonomy from a set of unlabeled examples. anmal vertebrate fsh reptle amphb. mammal nvertebrate worm nset rustaean Reursve applaton of a standard lusterng algorthm an produe a herarhal lusterng.

14 Agglomeratve vs. Dvsve Clusterng Agglomeratve (bottom-up) methods start wth eah example n ts own luster and teratvely ombne them to form larger and larger lusters. Dvsve (top-down) separate all examples mmedately nto lusters. anmal vertebrate nvertebrate fsh reptle amphb. mammal worm nset rustaean

15 Herarhal Agglomeratve Clusterng (HAC) Assumes a smlarty funton for determnng the smlarty of two lusters. Starts wth all nstanes n a separate luster and then repeatedly jons the two lusters that are most smlar untl there s only one luster. The hstory of mergng forms a bnary tree or herarhy. Bas algorthm: Start wth all nstanes n ther own luster. Untl there s only one luster: Among the urrent lusters, determne the two lusters, and j, that are most smlar. Replae and j wth a sngle luster j

16 Cluster Smlarty How to ompute smlarty of two lusters eah possbly ontanng multple nstanes? Sngle lnk: Smlarty of two most smlar members. Complete lnk: Smlarty of two least smlar members. Group average: Average smlarty between members.

17 Sngle-Lnk HAC Can result n straggly (long and thn) lusters due to hanng effet. When omputng luster smlarty, use maxmum smlarty sm of ( pars:, ) max sm( x, y ) j x, y j

18 Complete-Lnk HAC Makes more tght, spheral lusters. When omputng luster smlarty, use mnmum smlarty sm of ( pars:, ) mn sm( x, y ) j x, y j

19 Computatonal Complexty of HAC In the frst teraton, all HAC methods need to ompute smlarty of all pars of n ndvdual nstanes whh s O(n 2 ). In eah of the subsequent O(n) mergng teratons, t must ompute the dstane between the most reently reated luster and all other exstng lusters. In order to mantan the smlarty matrx n O(n 2 ) overall, omputng the smlarty to any other luster must eah be done n onstant tme. Mantan Heap to fnd smallest par O(n 2 log n)

20 Sngle-Lnk Example x1 x2 x3 x4 x5 x x x x x Merge x3,x4 replae wth max x1 x2 1 x5 x x x x5 Merge x1,x2 replae wth max 3 x x Merge 1,2 replae wth max x

21 Computng Cluster Smlarty After mergng and j, the smlarty of the resultng luster to any other luster, k, an be omputed by: Sngle Lnk: sm(( j ), k ) max( sm(, k ), sm( j, k )) Complete Lnk: sm(( j ), k ) mn( sm(, k ), sm( j, k ))

22 Group Average Agglomeratve Clusterng Use average smlarty aross all pars wthn the merged luster to measure the smlarty of two lusters. Compromse between sngle and omplete lnk. ) ( : ) ( ), ( 1) ( 1 ), ( j j x x y y j j j y x sm sm

23 Computng Group Average Smlarty Assume osne smlarty and normalzed vetors wth unt length. Always mantan sum of vetors n eah luster. s( j ) x Compute smlarty of lusters n onstant tme: sm(, j ) ( s( ) s( ( x j j )) ( s( )( ) s( j )) ( 1) )

24 Non-Herarhal Clusterng K-means lusterng ( hard ) Mxtures of Gaussans and tranng va Expetaton maxmzaton Algorthm ( soft )

25 Clusterng Crteron Evaluaton funton that assgns a (usually real-valued) value to a lusterng Clusterng rteron typally funton of wthn-luster smlarty and between-luster dssmlarty Optmzaton Fnd lusterng that maxmzes the rteron Global optmzaton (often ntratable) Greedy searh Approxmaton algorthms

26 Centrod-Based Clusterng Assumes nstanes are real-valued vetors. Clusters represented va entrods (.e. average of ponts n a luster) : μ() 1 x x Reassgnment of nstanes to lusters s based on dstane to the urrent luster entrods.

27 K-Means Algorthm Input: k = number of lusters, dstane measure d Selet k random nstanes {s 1, s 2, s k } as seeds. Untl lusterng onverges or other stoppng rteron: For eah nstane x : Assgn x to the luster j suh that d(x, s j ) s mn. For eah luster j //update the entrod of eah luster s j = ( j )

28 K-means Example (k=2) Pk seeds Reassgn lusters Compute entrods Reassgn lusters x x x x Compute entrods Reassgn lusters Converged!

29 Tme Complexty Assume omputng dstane between two nstanes s O(N) where N s the dmensonalty of the vetors. Reassgnng lusters for n ponts: O(kn) dstane omputatons, or O(knN). Computng entrods: Eah nstane gets added one to some entrod: O(nN). Assume these two steps are eah done one for teratons: O(knN). Lnear n all relevant fators, assumng a fxed number of teratons, more effent than HAC.

30 Bukshot Algorthm Problem Results an vary based on random seed seleton, espeally for hgh-dmensonal data. Some seeds an result n poor onvergene rate, or onvergene to sub-optmal lusterngs. Idea: Combne HAC and K-means lusterng. Frst randomly take a sample of nstanes of sze Run group-average HAC on ths sample n 1/2 Use the results of HAC as ntal seeds for K-means. Overall algorthm s effent and avods problems of bad seed seleton.

31 Clusterng as Predton Setup Learnng Task: P(X) Tranng Sample: S = (x 1,, x n ) Hypothess Spae: H = h 1,, h H eah desrbes P(X h ) where h are parameters Goal: learn whh P(X h ) produes the data What to predt? Predt where new ponts are gong to fall

32 Gaussan Mxtures and EM Gaussan Mxture Models Assume k P X = x h = j=1 P X = x Y = j, h P(Y = j) where P X = x Y = j, h = N(X = x μ j, Σ j ) and h = (μ 1,, μ k, Σ 1,, Σ k ). EM Algorthm Assume P(Y) and k known and Σ = 1. REPEAT μ j = n =1 n =1 P Y=j X=x,μ j x P Y=j X=x,μ j P Y = j X = x, μ j = P X=x Y=j,μ j )P(Y=j) k l=1 P X=x Y=l,μ j )P(Y=l) = e 0.5 x μ 2 j k l=1 P(Y=j) e 0.5 x μ l 2 P(Y=l)

Outline. Clustering: Similarity-Based Clustering. Supervised Learning vs. Unsupervised Learning. Clustering. Applications of Clustering

Outline. Clustering: Similarity-Based Clustering. Supervised Learning vs. Unsupervised Learning. Clustering. Applications of Clustering Clusterng: Smlarty-Based Clusterng CS4780/5780 Mahne Learnng Fall 2013 Thorsten Joahms Cornell Unversty Supervsed vs. Unsupervsed Learnng Herarhal Clusterng Herarhal Agglomeratve Clusterng (HAC) Non-Herarhal