Overlapping Community Structure & Percolation Transition of Community Network. Yong-Yeol Ahn CCNR, Northeastern University NetSci 2010

Transcription

1 Overlapping Community Structure & Percolation Transition of Community Network Yong-Yeol Ahn CCNR, Northeastern University NetSci 2010

2 Sune Lehmann James P. Bagrow

3 Communities

4 Communities

5

6

7 Overlap

8 G. Palla, I. Derényi, I. Farkas & T. Vicsek, Nature, 2005

9

10 Colleagues Family Friends

11

12

13 Pervasive overlap

14 Pervasive overlap Consequences

15 Simple local structure

16 Complex global structure

17 Complex global structure

18

19

20 Example:

21 What is this?

22 What the xxxx is this?

23 What the xxxx is this?

24 Word association network: Network of commonly associated English words G. Palla, I. Derényi, I. Farkas & T. Vicsek, Nature, 2005

25 COMBINE FRUIT BLENDER JOIN INTEGRATE JUICE BLEND MIXTURE MIX LOOK APPEAR DISAPPEAR SEE VANISH REAPPEAR SHOW ATTEND BROOM PAINT SWEEP PAINTER GROOM HAIR BRUSH PAINTING COMB TOOTHBRUSH HAIRSPRAY TOOTHPASTE

26 Simple Complex BROOM PAINT SWEEP PAINTER GROOM HAIR BRUSH PAINTING COMB TOOTHBRUSH HAIRSPRAY TOOTHPASTE Local Global

27

28

29 ?

30 Quantitative Evaluation Framework

31 Quantitative Evaluation Framework Community quality How homogeneous each community is?

32 Quantitative Evaluation Framework Community quality How homogeneous each community is? Overlap quality How accurate the # of overlap is?

33 Quantitative Evaluation Framework Community quality How homogeneous each community is? Overlap quality How accurate the # of overlap is? Community coverage How many nodes are covered?

34 Quantitative Evaluation Framework Community quality How homogeneous each community is? Overlap quality How accurate the # of overlap is? Community coverage How many nodes are covered? Overlap coverage How many memberships are assigned?

35 Metadata

36 Quantitative Evaluation Framework Community quality Amazon.com Community coverage no membership Subjects Africa - General Africa History Subjects HIV / AIDS Medical Africa Subjects HIV / AIDS Medical Nonfiction / General Infectious Diseases high coverage low coverage Overlap quality Metabolic network Overlap coverage community Acetyl-CoA 1. Glycolysis / Gluconeogenesis 2. TCA cycle 3. Fatty acid biosynthesis memberships high overlap Many pathway Memberships IDP (Inosine diphosphate) 1. Purine metaboilsm low overlap Few pathway Memberships high overlap coverage low overlap coverage

37 metadata network description N k community overlap PPI (Y2H) PPI (AP/MS) PPI network of S. cerevisiae obtained by yeast two-hybrid (Y2H) experiment [3] Affinity purification mass spectrometry (AP/MS) experiment Set of each protein s known functions (GO terms) a The number of GO terms GO terms GO terms PPI (LC) Literature curated (LC) GO terms GO terms PPI (all) Metabolic Phone Actor US Congress Philosopher Word Assoc. Amazon.com Union of Y2H, AP/MS, and LC GO terms GO-terms PPI networks b Metabolic network (metabolites connected by reactions) of E. coli Social contacts between mobile phone users [15, 16, 17] Film actors that appear in the same movies during [18] Congressmen who co-sponsor bills during the 108th US Congress [19, 20] Philosophers and their philosophical influences, from the English Wikipedia e English words that are often mentally associated [23] Products that users frequently buy together Set of each metabolite s pathway annotations (KEGG) c Each user s most likely geographic location Set of plot keywords for all of the actor s films Political ideology, from the common space score [21, 22] Set of (wikipedia) hyperlinks exiting in the philosopher s page Set of each word s senses, as documented by WordNet f g Set of each product s user tags (annotations) The number of KEGG pathway annotations Call activity (number of phone calls d ) Length of career (year of first role) Seniority (number of congresses served) Number of wikipedia subject categories Number of senses Number of product categories

38

39 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

40 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

41 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

42 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

43

44

45

46 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

47 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

48 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

49 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

50 1*'/2-*/ 3Q9:D<5(23Q9:<G9 2344O=78P(23Q9:<G9 3Q9:D<5(RO<D78P 2344O=78P(RO<D78P -. / / / / / / 0 1 BC3=9 >283: EF(/3=G:966 >4<?3=%234!!"#!# $%&' 0#&"'$()*+,#-./ $,'))!%#* &#* &!%#" )+)# #%!* 5+3*-()*+,#-./ "*)! ++%*+ )!)'+ "%*# 1*+3#4/ -. / 0 1( S S S S S -7=T6.3A96(H23=8:3DI /D7RO9(B9:23D<873= 0:99AP(K3AOD<:78P( 1=;34<5

51 Still works well even in the social network!

52 Still works well even in the social network!

53 Why?

54 One thing we know: They re incomplete.

55 Hypothesis: Sampling quickly breaks down community overlap.

56 How to quantify community overlap?

57 How to quantify community overlap? Percolation transition of community graph

58 community graph

59 community graph

60 Communities Nodes

61 Community overlap Links

62 Community overlap Links G. Palla, I. Derényi, I. Farkas & T. Vicsek, Nature, 2005 Figure 3 Network of the 82 communities in the DIP core list of the protein protein interactions of S. cerevisiae for k 5 4. The areas of the circles and the widths of the links are proportional to the size of the corresponding com ov

63 Hypothesis: Sampling quickly breaks down community overlap.

64 Sampling quickly breaks down community overlap. Size of giant component network community graph p

65 Sampling quickly breaks down community overlap. Size of giant component network community graph p

66 Problem: we don t know the true community structure

67 Solution 1: pre-assign true community structure Solution II: use existing community detection methods

68 Solution 1: pre-assign true community structure community (a) people A B C D E F G H I J K M.E.J. Newman, PRE 68, (2003) M.E.J. Newman and J. Park, PRE 68, (2003)

69 3 community (a) A B C D E F G H I J K H A B E I people F C D G K J

70 (a) A B C D E F G H I J K f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν s n z n, m=0 n=0 mr m z m 1, ns n z n 1, r m the prob. of belonging m distinct groups s n. the prob. that a group has n people µ = m mr m of groups pe g functions. ν = n ns n

71 People graph

72 (a) A B E H I F C D G?J A B C D E F G H I J K K f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν s n z n, m=0 n=0 mr m z m 1, ns n z n 1,

73 (a) A B E H I F C D G?J A B C D E F G H I J K K f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν s n z n, m=0 n=0 mr m z m 1, ns n z n 1, h n (z) = = = n k=1 n k=1 n 1 k =0 P (k n)z k 1 n 1 p k 1 (1 p) n k z k 1 k 1 n 1 (zp) k (1 p) n 1 k k = (1+(z 1)p) n 1 h(z) = 1 ns n h n (z) ν n=0 = g 1 (1 + (z 1)p)

74 (a) A B E H I F C D G?J A B C D E F G H I J K K f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν h(z) = 1 ν s n z n, m=0 n=0 n=0 mr m z m 1, ns n z n 1, ns n h n (z) = g 1 (1 + (z 1)p) G 0 (z) =f 0 (h(z)). G 1 (z) =f 1 (h(z)).

75 (a) A C? E B D G H F I A B C D E F G H I J K J K f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν h(z) = 1 ν s n z n, m=0 n=0 n=0 mr m z m 1, ns n z n 1, ns n h n (z) = g 1 (1 + (z 1)p) G 0 (z) =f 0 (h(z)). G 1 (z) =f 1 (h(z)). There is gcc if pf 1(1)g 1(1) > 1

76 Community graph

77 (a) A B C D E F G H I J K? f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν s n z n, m=0 n=0 mr m z m 1, ns n z n 1,

78 (a) A B C D E F G H I J K? f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν s n z n, m=0 n=0 mr m z m 1, ns n z n 1, q The probability that a community still looks like a community after samping

79 q The probability that a community still looks like a community after samping e.g. Single link cannot be identified as a community

80 (a) A B C D E F G H I J K? f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν q s n z n, m=0 n=0 mr m z m 1, ns n z n 1, j m (z) = (1 p)+ j(z) = 1 µ m m 1 p q k 1 (1 q) m k z k 1 k 1 k=1 = 1 p + p(1 + (z 1)q) m 1 m=0 = 1 p + p mr m j m (z) 1 µ m=0 = 1 p + pf 1 (1 + (z 1)q) mr m (1 + (z 1)q) m 1

81 (a) A B C D E F G H I J K? f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν q s n z n, m=0 n=0 mr m z m 1, ns n z n 1, = pqf 1(1)g 1(1) > 1,

82 (a) A B C D E F G H I J K? f 0 (z) = r m z m, g 0 (z) = m=0 n=0 f 1 (z) = 1 µ g 1 (z) = 1 ν q s n z n, m=0 n=0 mr m z m 1, ns n z n 1, = pqf 1(1)g 1(1) > 1, cf. for people graph: pf 1(1)g 1(1) > 1

83 Critical point is different!

84 1 people graph Size of giant component community graph p

85 Solution 1: pre-assign true community structure Solution II: use existing community detection methods

86 Solution 1: pre-assign true community structure :) Solution II: use existing community detection methods

87 Solution II: use existing community detection methods

88 Word association network

89

90 1 Size of giant component p

91 Summary Sampling quickly destroy community overlap Possible origin of apparent nonoverlapping community structure: incomplete data Link sampling will only emphasize this effect