Machine Learning in Simple Networks. Lars Kai Hansen

Transcription

1 Machine Learning in Simple Networs Lars Kai Hansen

2 Outline Communities and lin prediction Modularity Modularity as a combinatorial optimization problem Gibbs sampling Detection threshold a phase transition? Learning community parameters The Hofman-Wiggins generative model Is there a threshold for detection when you learn the parameters and complexity?

3 Muzeeer Wiipedia based common sense Wiipedia used as a proxy for the music users mental model Implementation: Filter retrieval using Wiipedia s article/ categories Muzeeer.com LINK PREDICTION to complete the ontological quality of Wiipedia

4 Networ models Nodes/vertices and lins/edges Directed / undirected Weighted / un-weighted Lin distributions Random Long tail Hubs and authorities Lin induced correlations The Rich club Communities Lin prediction

5 Motivation for community detection Community structure may mar a non-stationary lin distribution with high and low density sub-networs, hence summarizing with a single model could be misleading

6 Modularity can be predictive for dynamics M.E.J. Newman and M. Girvan, Finding and evaluating community structure in networs, Phys. Rev. E 69, (2004).

7 Modularity objective function The modularity is expressed as a sum over lins, such that we penalize missing lins in communities - missing is measured relative to a null distribution P 0 ij. Aij Q = PP i j δ ( ci, cj ) ij 2m C i is the community assignment of node j and 2m = Σ ij A ij, i = Σ j A ij The null is a baseline distribution P ij = i j /(2m) 2 The value of the modularity lies in the range [ 1,1]. It is positive if the number of edges within groups exceeds the number expected on the basis of chance M.E.J. Newman and M. Girvan. Finding and evaluating community structure in networs. Physical Review E, 69:026113, 2004, cond-mat/

8 Potts representation Introduce 0,1 binary variables S j coding the community assignment: node j is member of community δ ( c, c ) = S S i j i j Aij P( j, i) = 2m Aij Aij Q = (, ) ij PP c c PP S S ij 2m δ = 2m 1 Tr( SBS ') Q = B ij ijsisj = 2m 2m i j i j i j i j

9 Spectral optimization Newman relaxes the optimization problem to the simplex Q = 1 Tr( SBS ') B ijs is ij j = 2m 2m L = Tr( SBS ') + Tr( Λ S ) 2 m B S = S Λ

10 Combinatorial optimization We can use a physics analogy Simulated Annealing (Kirpatric et al. 1983) QS ( ) TrSBS ( ') PS ( AT, ) exp( ) = exp( ) T 2mT Gibbs sampling is a Monte Carlo realization of a Marov process in which each variable is randomly assigned according to its marginal distribution PS ( S, AT, ) j j PS ( AT, ) = PS ( AT, ) S j S Geman,D Geman, "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images". IEEE Transactions on Pattern Analysis and Machine Intelligence 6 (6): (1984)

11 Potts model 1-node Discrete probability distribution on states = 1,,K ( ) 1 ' ' (, ) exp, (, ) exp exp K S S PS AT T PS AT r T S r T ϕ ϕ ϕ = = = =

12 Gibbs sampling ϕ Bij Aij i j = S = S S j j j 2m 2m 2m 2m i j j j r i = exp( ϕi / T ) exp( ϕ / T ) ' i ' S i = potts( r) i

13 Deterministic annealing Instead of drawing Gibbs samples according to the marginals we can average instead, this provides a set of self-consistent equations for the means (for 0,1 Bernoulli variables the mean is the probability μ i =P(S i )) r i = exp( ϕi / T ) exp( ϕ / T ) ' i ' ϕ Bij Aij = r = r PPr j j j 2m 2m i j j i j j S. Lehmann, L.K. Hansen: Deterministic modularity optimization European Physical Journal B 60(1) (2007).

14 Experimental evaluation Create a simple testbed with lin probability and noise S. Lehmann, L.K. Hansen: Deterministic modularity optimization European Physical Journal B 60(1) (2007).

15 S. Lehmann, L.K. Hansen: Deterministic modularity optimization European Physical Journal B 60(1) (2007).

16 Generative community model (Hofman & Wiggins, 2008) PASpq (,, ) = p(1 p) q(1 q) c d = j i, = (1 A ) S S j i, j i ij j i c d e f ij j i ( 1 ) ij j i ( ) f = (1 A ) 1 S S j i A S S e= A S S ij j i

17 Learning parameters of the generative model Hofman & Wiggins (2008) Here Variational Bayes Dirichlets/beta prior and posterior distributions for the probabilities Very well determined (over ill) Independent binomials for the assignment variables (misses correlation) Maximum lielihood for the parameters Gibbs sampling for the assignments Jae M. Hofman and Chris H. Wiggins, Bayesian Approach to Networ Modularity Phys. Rev. Lett. 100, (2008),

18 The community detection threshold how many lins are needed to detect the structure? P in p SNR = = qc ( 1) C 1 Jorg Reichardt and Michele Leone, Un)detectable Cluster Structure in Sparse Networs Phys. Rev. Lett. 101, (2008),

19 Experimental design Planted solution N = 1000 nodes C true = 5 Quality: Mutual information between planted assignments and the best identified Gibbs sampling No annealing Burn-in 200 iterations Averaging 800 iterations Parameter learning Q = 10 iterations

20 Community Detection fully informed on number of communities and probabilities MUTUAL INF. PLANTED COMMUNITY MUTUAL INF. PLANTED COMMUNITY COMMUNITY DETECTION (N =1000, C = 10, SNR = 50) INTRA COMMUNITY LINK PROB (P) COMMUNITY DETECTION (N =1000, C = 5, SNR = 50) INTRA COMMUNITY LINK PROB (P) MUTUAL INF. PLANTED COMMUNITY MUTUAL INF. PLANTED COMMUNITY COMMUNITY DETECTION (N =1000, C = 5, SNR = 5) INTRA COMMUNITY LINK PROB (P) COMMUNITY DETECTION (N =1000, C = 5, SNR = 10) INTRA COMMUNITY LINK PROB (P)

21 Now what happens to the phase transition if we learn the parameters with a too complex model (C > C true = 5)? MUTUAL INF. PLANTED COMMUNITY COMMUNITY DETECTION (N =1000, C = 10, SNR = 10) INTRA COMMUNITY LINK PROB (P) MUTUAL INF. PLANTED COMMUNITY COMMUNITY DETECTION (N =1000, C = 10, SNR = 5) INTRA COMMUNITY LINK PROB (P) 200 MEMBERSHIPS COMMUNITY

22 Conclusions Community detection can be formulated as an inference problem (Hofman & Wiggins, 2008) The sampling process for fixed SNR has a phase transition lie detection threshold (Richard & Leone, 2008) The phase transition remains (sharpens?) if you learn the parameters of a generative model with unnown complexity

23