Outline. Clustering: Similarity-Based Clustering. Supervised Learning vs. Unsupervised Learning. Clustering. Applications of Clustering

Transcription

1 Clusterng: Smlarty-Based Clusterng CS4780/5780 Mahne Learnng Fall 2013 Thorsten Joahms Cornell Unversty Supervsed vs. Unsupervsed Learnng Herarhal Clusterng Herarhal Agglomeratve Clusterng (HAC) Non-Herarhal Clusterng K-means Mtures of Gaussans and EM-Algorthm Outlne Readng: Mannng/Raghavan/Shuetze, Chapters 16 (not 16.3) and 17 ( Supervsed Learnng vs. Unsupervsed Learnng Supervsed Learnng Classfaton: partton eamples nto groups aordng to pre-defned ategores Regresson: assgn value to feature vetors Requres labeled data for tranng Unsupervsed Learnng Clusterng: partton eamples nto groups when no pre-defned ategores/lasses are avalable Novelty deteton: fnd hanges n data Outler deteton: fnd unusual events (e.g. hakers) Only nstanes requred, but no labels Clusterng Partton unlabeled eamples nto dsont subsets of lusters, suh that: Eamples wthn a luster are smlar Eamples n dfferent lusters are dfferent Dsover new ategores n an unsupervsed manner (no sample ategory labels provded). Applatons of Clusterng Applatons of Clusterng Cluster retreved douments to present more organzed and understandable results to user dversfed retreval Detetng near duplates Entty resoluton E.g. Thorsten Joahms == Thorsten B Joahms Cheatng deteton Eploratory data analyss Automated (or sem-automated) reaton of taonomes e.g. Yahoo, DMOZ Compresson 1

2 Smlarty (Dstane) Measures Euldan dstane (L 2 norm): L 1 norm: L 1, N = =1 Cosne smlarty: Kernels L 2, N = 2 os, = =1 2

3 Herarhal Clusterng Buld a tree-based herarhal taonomy from a set of unlabeled eamples. vertebrate fsh reptle amphb. mammal anmal nvertebrate worm nset rustaean Agglomeratve vs. Dvsve Clusterng Agglomeratve (bottom-up) methods start wth eah eample n ts own luster and teratvely ombne them to form larger and larger lusters. Dvsve (top-down) separate all eamples mmedately nto lusters. anmal Reursve applaton of a standard lusterng algorthm an produe a herarhal lusterng. vertebrate fsh reptle amphb. mammal nvertebrate worm nset rustaean Herarhal Agglomeratve Clusterng (HAC) Assumes a smlarty funton for determnng the smlarty of two lusters. Starts wth all nstanes n a separate luster and then repeatedly ons the two lusters that are most smlar untl there s only one luster. The hstory of mergng forms a bnary tree or herarhy. Bas algorthm: Start wth all nstanes n ther own luster. Untl there s only one luster: Among the urrent lusters, determne the two lusters, and, that are most smlar. Replae and wth a sngle luster Cluster Smlarty How to ompute smlarty of two lusters eah possbly ontanng multple nstanes? Sngle lnk: Smlarty of two most smlar members. Complete lnk: Smlarty of two least smlar members. Group average: Average smlarty between members. Sngle-Lnk HAC Complete-Lnk HAC Can result n straggly (long and thn) lusters due to hanng effet Makes more tght, spheral lusters. When omputng luster smlarty, use mamum smlarty sm of ( pars:, ) ma sm(, y ), y When omputng luster smlarty, use mnmum smlarty sm of ( pars:, ) mn sm(, y ), y 3

4 Computatonal Complety of HAC In the frst teraton, all HAC methods need to ompute smlarty of all pars of n ndvdual nstanes whh s O(n 2 ). In eah of the subsequent O(n) mergng teratons, must fnd smallest dstane par of lusters Mantan heap O(n 2 log n) In eah of the subsequent O(n) mergng teratons, t must ompute the dstane between the most reently reated luster and all other estng lusters. Can ths be done n onstant tme suh that O(n 2 log n) overall? Computng Cluster Smlarty After mergng and, the smlarty of the resultng luster to any other luster, k, an be omputed by: Sngle Lnk: sm(( ), k ) ma( sm(, k ), sm(, k )) Complete Lnk: sm(( ), k ) mn( sm(, k ), sm(, k )) Sngle-Lnk Eample Merge 3,4 replae wth ma Group Average Agglomeratve Clusterng Use average smlarty aross all pars wthn the merged luster to measure the smlarty of two lusters Merge 1,2 replae wth ma sm(, ) ( 1) ( ) y( ): y sm(, y) Merge 1,2 replae wth ma Compromse between sngle and omplete lnk. Computng Group Average Smlarty Assume osne smlarty and normalzed vetors wth unt length. Always mantan sum of vetors n eah luster. s( ) Compute smlarty of lusters n onstant tme: ( s( ) s( )) ( s( ) s( )) ( ) sm(, ) ( )( 1) Non-Herarhal Clusterng K-means lusterng ( hard ) Mtures of Gaussans and tranng va Epetaton mamzaton Algorthm ( soft ) 4

5 Clusterng Crteron Evaluaton funton that assgns a (usually real-valued) value to a lusterng Clusterng rteron typally funton of wthn-luster smlarty and between-luster dssmlarty Optmzaton Fnd lusterng that mamzes the rteron Global optmzaton (often ntratable) Greedy searh Appromaton algorthms Centrod-Based Clusterng Assumes nstanes are real-valued vetors. Clusters represented va entrods (.e. average of ponts n a luster) : μ() Reassgnment of nstanes to lusters s based on dstane to the urrent luster entrods. 1 K-Means Algorthm K-means Eample (k=2) Input: k = number of lusters, dstane measure d Selet k random nstanes {s 1, s 2, s k } as seeds. Untl lusterng onverges or other stoppng rteron: For eah nstane : Assgn to the luster suh that d(, s ) s mn. For eah luster //update the entrod of eah luster s = ( ) Pk seeds Reassgn lusters Compute entrods Reassgn lusters Compute entrods Reassgn lusters Converged! Tme Complety Assume omputng dstane between two nstanes s O(N) where N s the dmensonalty of the vetors. Reassgnng lusters for n ponts: O(kn) dstane omputatons, or O(knN). Computng entrods: Eah nstane gets added one to some entrod: O(nN). Assume these two steps are eah done one for teratons: O(knN). Lnear n all relevant fators, assumng a fed number of teratons, more effent than HAC. Bukshot Algorthm Problem Results an vary based on random seed seleton, espeally for hgh-dmensonal data. Some seeds an result n poor onvergene rate, or onvergene to sub-optmal lusterngs. Idea: Combne HAC and K-means lusterng. Frst randomly take a sample of nstanes of sze Run group-average HAC on ths sample n 1/2 Use the results of HAC as ntal seeds for K-means. Overall algorthm s effent and avods problems of bad seed seleton. 5

6 Clusterng as Predton Setup Learnng Task: P(X) Tranng Sample: S = ( 1,, n ) Hypothess Spae: H = h 1,, h H eah desrbes P(X h ) where h are parameters Goal: learn whh P(X h ) produes the data What to predt? Predt where new ponts are gong to fall Gaussan Mtures and EM Gaussan Mture Models Assume k P X = h = =1 P X = Y =, h P(Y = ) where P X = Y =, h = N(X = μ, Σ ) and h = (μ 1,, μ k, Σ 1,, Σ k ). EM Algorthm Assume P(Y) and k known and Σ = 1. REPEAT μ = n =1 P Y= X=,μ n =1 P Y= X=,μ P Y = X =, μ = P X= Y=,μ )P(Y=) k P X= Y=l,μ)P(Y=l) l=1 = e 0.5 μ 2 P(Y=) e 0.5 μ 2 k l l=1 P(Y=l) 6