K means B d ase Consensus Cluste i r ng Dr. Dr Junjie Wu Beihang University

Transcription

1 K means Based dconsensus Clusterng Dr. Junje Wu Dr. Junje Wu Behang Unversty

2 Outlne Motvatons Pont to Centrod to Dstance Utlty Functons for KCC Expermental Results Concludng remarks

3 Cluster Analyss

4 Clusterng Algorthms Prototype based: K means, FCM Densty based: DBSCAN, CLIQUE Graph based: AHC, MnMaxCut Hybrd: K means + AHC

5 Problems wth Sngle Clusterer No perfect one! Senstve to data factors Hard to set proper parameters May converge to bad saddle ponts Orjust too heurstc Can we fnd a new way?

6 Consensus Clusterng To fnd a best parttonng from multple basc parttonngs (an ensemble classfer thnkng)

7 Consensus Clusterng, cont d r Γ(, π Π ) = wu (, ππ) = 1 U: utlty functon w : weght of π Г: consensus functon X = { x, x, x, x, x, x, x} π = 1 2 T ( 1,1,1, 2, 2,3,3 ) T π = ( 2,2,2,3,3,1,1 ) T π3 = ( 1,1,2,2,2,3,3 ) T π = ( ,1,1,1,2,2,2 2 2 ) 4 Π= { ππππ,,, }

8 Consensus Clusterng, cont d Advantages Robust: lower the rsks from werd data, mproper p algorthms and parameters, etc. Novelty: may help fnd a better structure Concurrency: run n parallel A Must: only have parttonng epsodes Challenge NP complete problem

9 Related Work Graph based algorthms (Strehl et al., JMLR, 2002) Co assocaton matrx method (Fred and Jan, PAMI, 2005) Bnary matrx method (Topchy et al., ICDM, 2003) Other heurstcs (Lu et al., AAAI, 2008; )

10 Why K means Consensus Clusterng Smple Robust Effcent NP-complete U? Roughly lnear (K-means)

11 Man Contrbutons Theory for KCC utlty functons KCC algorthm for nconsstent samples Some emprcal fndngs U H s a good KCC utlty functon RFS strategy s useful n some crcumstances Some mutual funds have unethcal behavors

12 Outlne Motvatons Pont to Centrod to Dstance Utlty Functons for KCC Expermental Results Concludng remarks

13 K means Clusterng Objectve functon K mn w f ( xm, ) k= 1 x C Two phase teratons Assgn x to the nearest centrod Update centrods k x The arthmetc h mean centrod s preferred k

14 Pont to Centrod to Dstance What f fx centrod type to arthmetc mean? A defnton A theorem f ( x, y ) = φ ( x ) φ ( y ) ( x y ) T φ ( y). K means

15 Examples f ( xy, ) = φ( x) φ( y) ( x y) T φ( y) φ(x) f(x,y) Name x 2 x-y 2 Sqrt Euc. Dst. -H(x) D(x y) KL dvergence x x - Cosne Dst. x t y/ y Interestngly, f s not a metrc!

16 Outlne Motvatons Pont to Centrod to Dstance Utlty Functons for KCC Expermental Results Concludng remarks

17 Defnton and Smplfcaton r K max b wu( π, π ) mn f ( xl, mk) = 1 k= 1 b x C l k

18 A Crtcal Fact The contngency table for π and π n p The centrod ds rght htthe normalzed row!

19 A Suffcent Condton r k + k = 1 k= 1 K wu( ππ, ) = p φ( m ) K P2C D K K b b f xl mk φ xl p l k+ φ mk pk+ φ mk k= 1 b x C k= 1 k= 1 mn (, ) mn ( ) ( ) max ( ) l k

20 A Suffcent Condton, cont d Theorem 2 r φ( m ) = wϕ( m ),1 k K k k, = 1 Theorem 2 K () () () k + ϕ k1 k + kj k + kk k + k = 1 U( ππ, ) = p ( < p / p,, p / p,, p / p > )

21 Examples ϕ ( ) m k, U ( π, π ) b f ( x, m ) l k U C K K K K K 2 ( ) 2 () 2 () 2 mkj, ( p + j) pk+ ( pkj / pk+ ) ( p+ j) j= 1 j = 1 k= 1 j= 1 j= 1 r ( b) 2 w xl, mk, =11 K K U () () H () m log log MI CC kj, m kj, p + j p + j j= 1 j= 1 ( b) (, ) wd ( xl, mk, ) r =1 U cos K K K () () r m k, < p+ 1,, p+ K > () 2 () 2 ( b) pk+ ( pkj / pk+ ) ( p+ j) w xl, mk, k= 1 j= 1 j= 1 = 1 (1 cos(, )) U () () Lp mk, p < p+ 1,, p+ K > p K K K p () p () ( / ) p p pk+ pkj pk+ ( p+ j) k= 1 j= 1 j= 1 K ( b) p 1 r x 1 lj, ( m, ) j= kj w p 1 = 1 ( mk, p) (1 ) UC U H U U cos L 5 U L8

22 Propertes Non unqueness of U ϕ () () ϕs( mk, ) = ϕ( mk, ) ϕ ( < p+ 1,, p+ K > ) K () () k + s k1 k + kk k + k = 1 α Utlty functon for KCC Uϕ ( ππ, ) = p ϕ( < ( p / p ),,( p / p ) > ) Many to one s = Uϕ ( ππ, ) ϕ ( < p+,, p+ > ) () () 1 K standard form, or utlty gan P2C dstance

23 Propertes, cont d Non negatvty of Utlty Gan K Uϕ ( ππ, ) = p ϕ( m ) ϕ( p m ) k+ k, k+ k, k= 1 k= 1 K () () () () = ϕ( < pk1,, pkk > ) = ϕ ( < p+ 1,, p+ K > ) k = 1 Uϕ ( ππ, ) = U ( ππ, ) ϕ ( < p+,, p+ > ) 0 s () () ϕ 1 K K

24 Propertes, cont d Utlty Gan Rato ϕ ϕ ϕ + + () () n( mk, ) = s( mk, )/ ( < p 1,, p K > ) K () () k + n k1 k + kk k + k = 1 Uϕ ( ππ, ) = p ϕ( < ( p / p ),,( p / p ) > ) n Uϕ ( π, π) ϕ( < p,, p > ) = ϕ( < p,, > ) () () K () () + 1 p+ K normalzed form, or utlty gan rato

25 Inconsstent Samples Often we have basc parttonngs from nconsstent sample sets Adjustments for KCC m p b ( ), l k k x C C kj, = () n k nk n n n n + () () k k k + = () x b lj K () () () () () k + ϕ k1 k + kk k + k = 1 U( ππ, ) = p ( < p / p,, p / p > )

26 Inconsstent Samples, cont d The proof for convergence r K r K Δ = b b b f ( x, y) f ( x, m ) ( ), b ( ) x,, l Ck C l k xl C l k k Ck = 1 k= 1 = 1 k= 1 r K K b b = b ( ) f ( x ( ) l,, y) b f( x l,, m, ) x l Ck C k xl Ck C k k = 1 k= 1 k= 1 r K b = b ( ) φ( mk, ) φ( y) ( xl, y) φ( y) x C C l k k = 1 k= 1 r K () ( nk n k ) f( mk,, y) = 1 k= 1 = 0 P2C dstance

27 Outlne Motvatons Pont to Centrod to Dstance Utlty Functons for KCC Expermental Results Concludng remarks

28 Data

29 Strateges for Basc Clusterngs For UCI data sets: Random Parameter Selecton (RPS) wth K means clusterng: Random Feature Selecton (RFS) wth K means clusterng: two features for a basc clusterng For text tdt data sets: Multple Clusterng Algorthms wth CLUTO (5 clusterng methods 5 objectve functons) Valdaton measure: normalzed Rand ndex R n

30 Expermental Results, #1 Convergence property of KCC

31 Expermental Results, #1 Convergence property of KCC

32 Expermental Results, #2 Clusterng accuracy of KCC dataset U NU U U U NU NU NU U NU H H cos L5 L8 cos L5 L8 C C

33 Expermental Results, #2 Clusterng accuracy of KCC

34 Expermental Results, #3 Effects of RFS strategy

35 Expermental Results, #3 Effects of RFS strategy

36 Outlne Motvatons Pont to Centrod to Dstance Utlty Functons for KCC Expermental Results Concludng remarks

37 Conclusons Study K means based consensus clusterng Gve a suffcent condton Handle the nconsstent samples Gvesome emprcal results for practcal use Future work Gve the necessary condton (almost done!) Applcatons

38 Thank You!