Clustering Techniques

Transcription

1 Clusteng Tehnques Refeenes: Beln Chen Moden Infomaton Reteval, haptes 5, 7 2. Foundatons of Statstal Natual Language Poessng, Chapte 4

2 Clusteng Plae smla obets n the same goup and assgn dssmla obets to dffeent goups Wod lusteng Neghbo ovelap: wods ou wth the smla left and ght neghbos (suh as n and on) Doument lusteng Douments wth the smla tops o onepts ae put togethe But lusteng annot gve a ompehensve despton of the obet How to label obets shown on the vsual dsplay Clusteng s a way of leanng 2

3 Clusteng vs. Classfaton Classfaton s supevsed and eques a set of labeled tanng nstanes fo eah goup (lass) Clusteng s unsupevsed and leans wthout a teahe to povde the labelng nfomaton of the tanng data set Also alled automat o unsupevsed lassfaton 3

4 Types of Clusteng Algothms Two types of stutues podued by lusteng algothms Flat o non-heahal lusteng Heahal lusteng Flat lusteng Smply onsstng of a etan numbe of lustes and the elaton between lustes s often undetemned Heahal lusteng A heahy wth usual ntepetaton that eah node stands fo a sublass of ts mothe s node The leaves of the tee ae the sngle obets Eah node epesents the luste that ontans all the obets of ts desendants 4

5 Had Assgnment vs. Soft Assgnment Anothe mpotant dstnton between lusteng algothms s whethe they pefom soft o had assgnment Had Assgnment Eah obet s assgned to one and only one luste Soft Assgnment Eah obet may be assgned to multple lustes An obet has a pobablty dstbuton P ( ) ove lustes whee P ( ) s the pobablty that s a membe of Is somewhat moe appopate n many tasks suh as NLP, IR, 5

6 Had Assgnment vs. Soft Assgnment Heahal lusteng usually adopts had assgnment whle n flat lusteng both types of lusteng ae ommon 6

7 Summazed Attbutes of Clusteng Algothms Heahal Clusteng Pefeable fo detaled data analyss Povde moe nfomaton than flat lusteng No sngle best algothm (eah of the algothms only optmal fo some applatons) Less effent than flat lusteng (mnmally have to ompute n n mat of smlaty oeffents) Flat lusteng Pefeable f effeny s a onsdeaton o data sets ae vey lage K-means s the oneptually method and should pobably be used on a new data beause ts esults ae often suffent K-means assumes a smple Euldean epesentaton spae, and so annot be used fo many data sets, e.g., nomnal data lke olos The EM algothm s the most hoe. It an aommodate defnton of lustes and alloaton of obets based on omple pobablst models 7

8 Heahal Clusteng 8

9 Heahal Clusteng Can be n ethe bottom-up o top-down mannes Bottom-up (agglomeatve) Stat wth ndvdual obets and goupng the most smla ones E.g., wth the mnmum dstane apat sm (, y ) (, y ) The poedue temnates when one luste ontanng all obets has been fomed Top-down (dvsve) Stat wth all obets n a goup and dvde them nto goups so as to mamze wthn-goup smlaty + d 9

10 Heahal Agglomeatve Clusteng (HAC) A bottom-up appoah Assume a smlaty measue fo detemnng the smlaty of two obets Stat wth all obets n a sepaate luste and then epeatedly ons the two lustes that have the most smlaty untl thee s one only luste suvved The hstoy of megng/lusteng foms a bnay tee o heahy 0

11 Heahal Agglomeatve Clusteng (HAC) Algothm luste numbe

12 Dstane Mets Euldan dstane (L 2 nom) L L nom m 2 2 (, y) ( y ) L (, y) m y Cosne Smlaty (tansfom to a dstane by subtatng fom ) y y 2

13 Measues of Cluste Smlaty Espeally fo the bottom-up appoahes Sngle-lnk lusteng The smlaty between two lustes s the smlaty of the two losest obets n the lustes Seah ove all pas of obets that ae fom the two dffeent lustes and selet the pa wth the geatest smlaty sm (, ) ma sm (,y ),y Complete-lnk lusteng The smlaty between two lustes s the smlaty of the two most dssmla membes Sphee-shaped lustes ae aheved Pefeable fo most IR and NLP applatons sm (, ) mn sm (,y ),y C u C u C v geatest smlaty least smlaty C v 3

14 Measues of Cluste Smlaty 4

15 Measues of Cluste Smlaty Goup-aveage agglomeatve lusteng A ompomse between sngle-lnk and omplete-lnk lusteng The smlaty between two lustes s the aveage smlaty between membes If the obets ae epesented as length-nomalzed vetos and the smlaty measue s the osne Thee ests an fast algothm fo omputng the aveage smlaty y sm (, y ) os (, y ) y y 5

16 6 Measues of Cluste Smlaty Goup-aveage agglomeatve lusteng (ont.) The aveage smlaty SIM between vetos n a luste s defned as The sum of membes n a luste : Epess n tems of ( ) ( ) ( ) y y y sm SIM, ( ) s ( ) SIM ( ) s ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) + + y s s SIM SIM SIM y s s s

17 Measues of Cluste Smlaty Goup-aveage agglomeatve lusteng (ont.) -As megng two lustes and, the luste sum vetos s ( ) and s ( ) ae known n advane s s + s, + The aveage smlaty fo the unon wll be SIM ( ) ( ) ( ) New New ( ) ( s ( ) + s ( )) s ( ) + s ( ) ( ) ( + ) ( + )( + ) 7

18 An Eample 8

19 Dvsve Clusteng A top-down appoah Stat wth all obets n a sngle luste At eah teaton, selet the least oheent luste and splt t Contnue the teatons untl a pedefned teon (e.g., the luste numbe) s aheved The hstoy of lusteng foms a bnay tee o heahy 9

20 Dvsve Clusteng To selet the least oheent luste, the measues used n bottom-up lusteng an be used agan hee Sngle lnk measue Complete-lnk measue Goup-aveage measue How to splt a luste Also s a lusteng task (fndng two sub-lustes) Any lusteng algothm an be used fo the splttng opeaton, e.g., Bottom-up algothms Non-heahal lusteng algothms (e.g., K-means) 20

21 Dvsve Clusteng Algothm : 2

22 Non-Heahal Clusteng 22

23 Non-heahal Clusteng Stat out wth a patton based on andomly seleted seeds (one seed pe luste) and then efne the ntal patton In a mult-pass manne Poblems assoated non-heahal lusteng When to stop What s the ght numbe of lustes MI, goup aveage smlaty, lkelhood k- k k+ Algothms ntodued hee The K-means algothm The EM algothm Heahal lusteng also has to fae ths poblem 23

24 The K-means Algothm A had lusteng algothm Defne lustes by the ente of mass of the membes Intalzaton A set of ntal luste entes s needed Reuson Assgn eah obet to the luste whose ente s loset Then, e-ompute the ente of eah luste as the entod o mean of ts membes Usng the medod as the luste ente? 24

25 The K-means Algothm Algothm luste entod luste assgnment alulaton of new entod 25

26 Eample The K-means Algothm 26

27 Eample 2 The K-means Algothm govenment fnane spots eseah name 27

28 The K-means Algothm Choe of ntal luste entes (seeds) s mpotant Pk at andom O use anothe method suh as heahal lusteng algothm on a subset of the obets Poo seeds wll esult n sub-optmal lusteng 28

29 The EM Algothm A soft veson of the K-mean algothm Eah obet ould be the membe of multple lustes Clusteng as estmatng a mtue of (ontnuous) pobablty dstbutons l π π ( X Θ ) log n ma P ( ) P π ( ) k k P n ( Θ ) log P P P ( ) 2 2 k P P ( ) ( ) 2 ( ) k ( ; Θ) ( ; Θ ) P ( Θ ) ma P ma ( 2π ) ( Θ ) k P( ; Θ ) P( Θ ) Contnuous ase: P ep µ m Σ 2 T ( µ ) Σ ( ) 29

30 30 The EM Algothm E step (Epetaton) The epetaton h of the hdden vaable z M-step (Mamzaton) ( ) ( ) ( ) ( ) ( ) Θ Θ Θ Θ Θ k l l l P P P P z E h ; ; ; n n h h u ( )( ) Σ n n T h h µ µ k n n h h π

31 The EM Algothm The ntal luste dstbutons an be estmated usng the K-means algothm The poedue temnates when the lkelhood funton l ( X Θ ) s onveged o mamum numbe of teatons s eahed 3