Clustering. Outline. Supervised vs. Unsupervised Learning. Clustering. Clustering Example. Applications of Clustering

Transcription

1 Clusteng CS478 Mahne Leanng Spng 008 Thosten Joahms Conell Unvesty Outlne Supevsed vs. Unsupevsed Leanng Heahal Clusteng Heahal Agglomeatve Clusteng (HAC) Non-Heahal Clusteng K-means EM-Algothm Readng: Mannng/Shuetze Chapte 4 (not 4..3, 4..4) Based on sldes fom Pof. Clae Cade, Pof. Ray Mooney, Pof. Ymng Yang Supevsed vs. Unsupevsed Leanng Supevsed Leanng Classfaton: patton eamples nto goups aodng to pe-defned ategoes Regesson: assgn value to featue vetos Reques labeled data fo tanng Unsupevsed Leanng Clusteng: patton eamples nto goups when no pe-defned ategoes/lasses ae avalable Novelty deteton: fnd hanges n data Outle deteton: fnd unusual events (e.g. hakes) Only nstanes equed, but no labels Clusteng Patton unlabeled eamples nto dsont subsets of lustes, suh that: Eamples wthn a luste ae smla Eamples n dffeent lustes ae dffeent Dsove new ategoes n an unsupevsed manne (no sample ategoy labels povded). Applatons of Clusteng Clusteng Eample Cluste eteved douments (e.g. Teoma) to pesent moe oganzed and undestandable esults to use Detetng nea duplates Entty esoluton E.g. Thosten Joahms == Thosten B Joahms Cheatng deteton Eploatoy data analyss Automated (o sem-automated) eaton of taonomes e.g. Yahoo-style Compesson

2 Smlaty (Dstane) Measues Heahal Clusteng Euldan dstane (L nom): m L (, ') = ( ' L nom: Cosne smlaty: Kenels ) = m (, ') = ' = L os(, ' ) = ' ' Buld a tee-based heahal taonomy fom a set of unlabeled eamples. vetebate fsh eptle amphb. mammal anmal nvetebate wom nset ustaean Reusve applaton of a standad lusteng algothm an podue a heahal lusteng. Agglomeatve vs. Dvsve Clusteng Agglomeatve (bottom-up) methods stat wth eah eample n ts own luste and teatvely ombne them to fom lage and lage lustes. Dvsve (top-down) sepaate all eamples mmedately nto lustes. anmal vetebate fsh eptle amphb. mammal nvetebate wom nset ustaean Heahal Agglomeatve Clusteng (HAC) Assumes a smlaty funton fo detemnng the smlaty of two lustes. Stats wth all nstanes n a sepaate luste and then epeatedly ons the two lustes that ae most smla untl thee s only one luste. The hstoy of megng foms a bnay tee o heahy. Bas algothm: Stat wth all nstanes n the own luste. Untl thee s only one luste: Among the uent lustes, detemne the two lustes, and, that ae most smla. Replae and wth a sngle luste Cluste Smlaty How to ompute smlaty of two lustes eah possbly ontanng multple nstanes? Sngle lnk: Smlaty of two most smla membes. Complete lnk: Smlaty of two least smla membes. Goup aveage: Aveage smlaty between membes. Sngle-Lnk Agglomeatve Clusteng When omputng luste smlaty, use mamum smlaty of pas: sm(, ) = ma sm(,, y Can esult n staggly (long and thn) lustes due to hanng effet.

3 Sngle Lnk Eample Complete Lnk Agglomeatve Clusteng When omputng luste smlaty, use mnmum smlaty of pas: sm(, ) = mn sm(,, y Makes moe tght, spheal lustes. Computatonal Complety of HAC In the fst teaton, all HAC methods need to ompute smlaty of all pas of n ndvdual nstanes whh s O(n ). In eah of the subsequent n megng teatons, t must ompute the dstane between the most eently eated luste and all othe estng lustes. In ode to mantan the smlaty mat n O(n ) oveall, omputng the smlaty to any othe luste must eah be done n onstant tme. Mantan e.g. Heap to fnd smallest pa Computng Cluste Smlaty Afte megng and, the smlaty of the esultng luste to any othe luste, k, an be omputed by: Sngle Lnk: sm (( ), k ) = ma( sm(, k ), sm(, k )) Complete Lnk: sm (( ), k ) = mn( sm(, k ), sm(, k )) Goup Aveage Agglomeatve Clusteng Use aveage smlaty aoss all pas wthn the meged luste to measue the smlaty of two lustes. sm(, ) = sm ( ) ( ) y ( ) : y Compomse between sngle and omplete lnk. (, Computng Goup Aveage Smlaty Assume osne smlaty and nomalzed vetos wth unt length. Always mantan sum of vetos n eah luste. s( ) = Compute smlaty of lustes n onstant tme: ( s( ) + s( )) ( s( ) + s( )) ( + ) sm(, ) = ( + )( + ) 3

4 Non-Heahal Clusteng Sngle-pass lusteng K-means lusteng ( had ) Epetaton mamzaton ( soft ) Clusteng Cteon Evaluaton funton that assgns a (usually ealvalued) value to a lusteng Clusteng teon typally funton of wthn-luste smlaty and between-luste dssmlaty Optmzaton Fnd lusteng that mamzes the teon Global optmzaton (often ntatable) Geedy seah Appomaton algothms Centod-Based Clusteng Assumes nstanes ae eal-valued vetos. Clustes epesented va entods (.e. mean of ponts n a luste) : µ() = Reassgnment of nstanes to lustes s based on dstane to the uent luste entods. K-Means Algothm Input: k = numbe of lustes, dstane measue d Selet k andom nstanes {s, s, s k } as seeds. Untl lusteng onveges o othe stoppng teon: Fo eah nstane : Assgn to the luste suh that d(, s ) s mn. Fo eah luste //update the entod of eah luste s = µ( ) K-means Eample (k=) Pk seeds Reassgn lustes Compute entods Reasssgn lustes Compute entods Reassgn lustes Conveged! Tme Complety Assume omputng dstane between two nstanes s O(m) whee m s the dmensonalty of the vetos. Reassgnng lustes fo n ponts: O(kn) dstane omputatons, o O(knm). Computng entods: Eah nstane gets added one to some entod: O(nm). Assume these two steps ae eah done one fo teatons: O(knm). Lnea n all elevant fatos, assumng a fed numbe of teatons, moe effent than HAC. 4

5 Bukshot Algothm Poblem Results an vay based on andom seed seleton, espeally fo hgh-dmensonal data. Some seeds an esult n poo onvegene ate, o onvegene to sub-optmal lustengs. Idea: Combne HAC and K-means lusteng. Fst andomly take a sample of nstanes of sze n Run goup-aveage HAC on ths sample Use the esults of HAC as ntal seeds fo K-means. Oveall algothm s effent and avods poblems of bad seed seleton. 5