Estimating Clustering Coefficients and Size of Social Networks via Random Walk

Transcription

1 Estimating Clustering Coefficients and Size of Social Networks via Random Walk Stephen J. Hardiman* Capital Fund Management France Liran Katzir Advanced Technology Labs Microsoft Research, Israel *Research was conducted while the author was unaffiliated

2 Motivation: Social Networks Qzone Netlog Google+ Bebo Twitter Facebook Classmates.com Sina Weibo Sonico.com Orkut Renren Habbo Flixster MyLife Tagged Friendster hi5 LinkedIn Vkontakte Plaxo

3 Motivation: External access Social Analytics The online social network v 3 v 5 v 7 v 1 v 2 v 9 Privacy Disk Space Communication v 4 v 6 v 8

4 Task: Estimate parameters Global Clustering Coefficient Network Average CC Number of Registered Users Predicting Social Products Potential. Business development/ advertisement/ market size.

5 Global Clustering Coefficient Global CC = 3 x number of triangles number of connected triplet v 3 v 5 v 7 v 1 v 2 v 9 Triangle v 4 v 6 v 8 Connected Triplet

6 Global Clustering Coefficient Exact: [Alon et al, 1997] Estimation input is read at least once: Random Access: [Avron, 2010] Streaming Model: [Buriol et al, 2006] Estimation sampling: Random Access: [Schank et al, 2005] External Access: This work.

7 Local Clustering Coefficient C i = #connections between vi s neighbors d i (d i 1)/2 C 2 =1/3 v 3 v 5 d i degree of node i d 1 = 1 d 2 = 3 d 9 = 2 v 7 v 1 v 2 v 9 Network Average CC = average local CC v 4 v 6 v 8

8 Network Average CC Exact: Naïve. Estimation input is read at least once: Streaming Model: [Becchetti et al, 2010] Estimation sampling: Random Access: [Schank et al, 2005] External Access: [Ribeiro et al 2010], [Gjoka et al, 2010], This work Improved accuracy.

9 Number of Registered Users Exact: trivial Estimation sampling: External Access: [Hardiman et al 2009], [Katzir et al, 2011], This work Improved accuracy.

10 Random Walk Sampled Nodes: v 1 v 2 v 3 v 4 v 5 Stationary Distribution = d i 1 3 v 3 v d i v v 1 v v 4 v 6 v 8 v 9

11 Random Walk - Summary Sampled Nodes Visible Nodes Invisible Nodes Visible Edges Invisible Edges v 3 v 5 v 7 v 1 v 2 v 9 v 4 v 6 v 8

12 Global CC Algorithm The estimated global clustering coefficient: c g = Φ g Ψ g 1. Ψ g Sampled nodes average degree if there is an edge v k 1 v k+1, φ k = 1 iff v k 1, v k, v k+1 is a triangle 0 Otherwise. 2. Φ g Sampled nodes average φ k d k.

13 Global CC Example Φ g = = 2 3 Ψ g = = 7 5 φ 2 = 0 φ 3 = 1 v 1 v 2 v 3 v 5 v 7 c g = c g = φ 4 = 0 v 4 v 6

14 E φ k d k = Expectation of φ k = = n i=1 n i=1 n i=1 d i D E φ kd k x k = v i d i D 2l i D d i The degree of node v i. 2l i d i d i d i l i The number of triangles contain v i. n The number of nodes. Total expectation d i d i combinations. 2l i yield φ k =1 D = n i=1 d i

15 Global CC Proof n n E Φ g = E φ k d k = 2 D l i E Ψ g = 1 D d i d i 1 i=1 i=1 c g = Φ g concentration bounds E Φg Ψ g concentration bounds E Ψg n i=1 n 2 i=1 l i d i d i 1 = c g d i The degree of node v i. l i The number of triangles contain v i. n The number of nodes. D = n i=1 d i

16 Guarantees For any ε 1 and δ 1, we have 8 Prob 1 ε c g c g 1 + ε c g 1 δ when the number of samples, r, satisfies r r g = O mixing time(ε)

17 Network Average CC Algorithm The estimated network average CC: c l = Φ l Ψ l 1. Ψ l Sampled nodes average 1/degree. φ k = 1 if there is an edge v k 1 v k+1, 0 Otherwise. 2. Φ l Sampled nodes average φ k 1 d k 1.

18 Evaluations Network n (size) D/n c l c g DBLP 977, Orkut 3,072, Flickr 2,173, Live Journal 4,843, DBLP facts: Paper with most co-authors: has 119 listed authors. Most prolific author: Vincent Poor with 798 entries.

19 Relative estimation value Global CC DBLP Network Percentage of mined nodes Gjoka et al* Ribeiro et al* This work Relative improvement ranges between 300% and 500% depending on the network.

20 Relative estimation value Network Average CC Orkut Network Ribeiro et al Gjoka et al Random walk Relative improvement ranges between 50% and 400% depending on the network Percentage of mined nodes

21 Conclusions 1. New external access estimator from Global Clustering Coefficient. 2. Improved estimator for Network Average Clustering Coefficient. 3. Improved estimator for number of registered users.

22 Estimating Sizes of Social Networks via Biased Sampling Liran Katzir Yahoo! Labs, Haifa, Israel Edo Liberty Yahoo! Labs, Haifa, Israel Oren Somekh Yahoo! Labs, Haifa, Israel

23 The Birthday Paradox The expected number of collisions in a list of r i.i.d. samples from a set of n elements is A collision is a pair of identical samples. Example: Samples: X = (d, b, b, a, b, e). Total 3 collisions, (x 2, x 3 ), (x 2, x 5 ), and (x 3, x 5 ) r r 1 2n.

24 Cardinality estimation uniform When C collisions are observed r r 1 n 2C Needs r = O n samples to converge. Used by [Ye et al, 2010] to estimate the size.

25 Stationary distribution sampling Sampled Nodes: v 5 v 2 v 5 v 4 v 2 Stationary Distribution = d i 1 3 v 3 v d i v v 1 v v 4 v 6 v 8 v 9

26 Cardinality estimation stationary When C collisions are observed 1 d x d n x 2C 4 Needs r = O n log n samples to converge when d i ~zipf( n, 2).

27 Example: d x = d x = n = v 5 v 2 v 5 v 4 v 2 v 3 v 5 v 7 v 1 v 2 v 9 v 4 v 6 v 8

28 Global CC Proof E d x = n i=1 d i D d i E 1 d x = n i=1 d i D 1 d i = n D E C = n i=1 d i D d i D n = d x 1 d x concentration bounds E dx E 2C concentration bounds 2E C 1 d x d i D d n i D d i d i D D = n d i The degree of node v i. n The number of nodes. D = n i=1 d i

29 Improvements 1. Using all samples (Hardiman et al 2009). 2. Using Conditional Monte Carlo (This work).

30 All Samples Restrict computation to indexes m steps apart, I = k, l k l m A collision is only be considered within I. Φ = x k = x l k, l I Ratio of degrees is similarly defined Ψ = k,l I d xk d xl

31 Conditional Monte Carlo A collision between x k and x l, is replaced by the conditional collision is steps k+1 and l+1 respectively. Common Neighbors E 1 xk+1 =x l+1 x k, x l = d xk d xl

32 Conditional Monte Carlo The pair v 4, v 7 is not a collision, but it contributes 1 12 to the collision counter. v 3 v 5 v 7 v 1 v 2 v 9 v 4 v 6 v 8

33 Relative estimation value Size Estimation DBLP Network Priot art This work Percentage of mined nodes

34 Thanks