Sequential Change-Point Approach for Online Community Detection

Transcription

1 Sequential Change-Point Approach for Online Community Detection Yao Xie Joint work with David Marangoni-Simonsen H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Presented at DMA Workshop, INFORMS 2014

2 Community Collaboration network Facebook Protein interaction network Word association

3 Enron data set Some networks might be structurally suspicious, even if none of the content passing on it looks that way... diagnose bad acting within a large organization. The Atlantic, 2011

4 500,000 s involving 151 unknown employees and more than 75,000 distinct addresses; each with time stamp, sender and receiver between the years 1998 and 2002, record for 1,177 days legal project : many people are connected to many others on a project, and information is widely distributed illicit : information concentrated in a few hands

5 Emergence of a community Starting from a certain time, anomalous discussion topics arise between a small group of people. Sequential anomaly detection in the presence of noise and limited feedback, Raginsky et al., 2013.

6 Online detection of community emergence a network with N nodes observe a sequence of independent adjacency matrices X 1, X 2,... X i R N N : interaction of nodes at time i there may exits an unknown time s.t. after that an unknown subset of nodes interact with higher frequency offline version [Arias-Castro-Verzelen2014]

7 Sequential change-point detection approach H 0 : X i : Erdős-Renyi random graph [X t ] ij = { 1 w. p. p0 0 otherwise (i, j). H 1 : there exists an unknown time κ such that afterwards unknown subset of nodes S interact more frequenctly [X t ] ij = { 1 w. p. p1 0 otherwise i, j S, t > κ, [X t ] ij = { 1 w. p. p0 0 otherwise i / S or j / S, t > κ. p 0 < p 1

8 Goal: detect emergence of an unknown community as quickly as possible define a stopping rule T for sequential data such that rarely raise false alarm when there is no change raise alarm quickly after the change (small detection delay)

9 Classic change-point detection In statistics and quality-control Min-max formulation: Page (54), Lorden (71) Bayesian: Shiryayev (63), Roberts (66) a sequence i.i.d. observations y 1, y 2, R unknown change-point κ > 0. unknown change-point κ > 0 y t 5 0 H 0 :y t f 0, t = 1, 2,... H 1 :y t f 0, t = 1,..., κ, y t f 1, t = κ + 1, f 0 f 1 apple change-point

10 Likelihood ratio based procedure for a hypothesized κ = k: l(t, k) = log k i=1 f 0(y i ) t i=k+1 f 1(y i ) t i=1 f = 0(y i ) likelihood ratio based change-point detection: T = inf{t 1 : maxl(t, k) b} k<t t i=k+1 log f 1(y i ) f 0 (y i ) CUSUM 40 max k<t

11 Normal distributions f 0 = N (0, 1), f 1 = N (µ, 1), µ > 0 CUSUM procedure T = inf{t : max k<t t i=k+1 (µy i µ2 2 ) b} when µ is unknown: ˆµ(k) = ( t i=k+1 y i)/(t k) GLR procedure T = inf{t : max k<t ( t i=k+1 y i) 2 t k b}

12 Likelihood ratio based statistic for edge (i, j), assumed change-point location κ = k, observation up to time t, likelihood ratio statistic given by l(κ = k p 1, S) = t (i,j) S m=k+1 ( p1 [X m ] ij log ) ( ) 1 p1 + (1 [X m ] ij ) log p 0 1 p 0 } {{ } U (i,j) k,t,p 1 typically, we can assume p 0 known since it can estimated from historic data p 1 is usually unknown since it represents anomaly

13 Exhaustive Search (ES) method Approach 1: assume unknown p 1 = δ δ: nominal value that would be important to detect T ES,1 = inf{t : max max U (i,j) k,t,δ b}, t m 1 k t m 0 S [N]: S =s (i,j) S exist a recursive implementation (similar to CUSUM) for each possible S, calculate W S,t+1 = max{w S,t + T ES,1 = inf{t : (i,j) S U (i,j) t,t+1,δ, 0}, max W S,k b}. S [N]: S =s

14 Exhaustive Search (ES) method (cont.) Approach 2: estimate p 1 for each hypothesize parameter values k and S p 1 (S) = T ES,2 = inf{t : 2 S ( S 1)(t k) max no recursive implementation max t (i,j) S m=k+1 t m 1 k t m 0 S [N]: S =s (i,j) S [X m ] ij, U (i,j) k,t, p 1 (S) b}. limitation of ES: S unknown, have to search all possible subsets of {1,, N}. Number of possible subsets Ω = 2 N, exponential in N.

15 Mixture method exploit structure: typically community is a small subset assume two nodes (i, j) both in community with probability α α can be a guess for S /N introduce indicator variable Q ij = { 1 w. p. α 0 otherwise i, j S. l(κ = k p 1, S) = Q ij t m=k+1 1 i<j N log { E Qij [(1 Q ij )+ } p [Xm] ij 1 (1 p 1 ) 1 [Xm] ij p [Xm] ] = ij 0 (1 p 0 ) 1 [Xm] ij 1 i<j N h(u (i,j) k,t,p 1 ).

16 Mixture method (cont.) h(x) log{1 α + α exp(x)} f(x) = log(1 p0 + p0e x ) log(1 p 0 + p 0 e x ) p 0 =1 p 0 = 0.01 p 0 = 0.1 p 0 = 1 p 0 = x T Mix = inf{t : max t m 1 k t m 0 1 i<j N h(u (i,j) k,t,δ ) b}, No search over subset max S.

17 Drawback of Mixture method statistics of Mixture method can be gathered from false community can increase false alarm rate Hierarchical Mixture method (H-Mix) solves this problem by introducing dendrogram decomposition of the graph

18 Hierarchical Mixture method (H-Mix)

19

20 Complexity Table : Complexities of algorithms under various conditions regarding k and N. S N/2 S N/2 S N/2 Exhaustive O(N N S ) O(N S ) O(2 S 2 ) Search Mixture Model O(N 2 ) O(N 2 ) O(N 2 ) Hierarchical Mixture O(N 3 ) O(N 4 ) O(N 4 )

21 Choice of b Choice of threshold b involves a tradeoff between ARL and EDD: ARL (average run length) (captures false-alarm-rate) usually choose b to make ARL large 5000, for large N simulating ARL via Monte Carlo is hard accurate theoretical approximation for ARL is highly valuable EDD (expected detection delay) a relatively small number 10 theoretical approximation provides useful insight

22 Performance metrics detection delay 20 run length threshold b apple average run length (ARL): expected detection delay (EDD): E {T } sup ess sup E k {T k T > k} k

23 Theoretical results We obtain analytical expression for ARL of Mixture method Theorem When b, an upper approximation to the ARL E [T mix ] of the Mixture method with known p 1 is given by: [ 2N/m0 yν 2 (y 1 γ(θ y )) ARL UA = dy], (1) 2N/m1 H(N, θ y ) and a lower approximation to the ARL is given by: [ m1 ] 1 ARL LA = 2Nν2 (2N γ(θτ )/τ 2 ) τ 2, (2) H(N, θ τ=m τ ) 0

24 expressions can be evaluated explicitly no Monte Carlo simulation needed 10 x integrand τ only a few τ values play role in the summation

25 accurate approximation can be used to determine threshold b for given ARL

26 Table : Theoretical vs. simulated thresholds for p 0 = 0.3, p 1 = 0.8, and N = 6. The threshold b calculated using theory is very close to the corresponding threshold obtained using simulation. ARL Theory b Simulated b

27 g probability Proof techniques ols: change-of-measure or exponential tilting lculate EDD max Z k,t has positive drift, calculate expected k<t of first hitting detection statistic forms a random walk max k<t Z k,t ols: 20 composition k,t = drift + variance + bounded w.h.p.terms ncentration of measure ald s lemma n-linear renewal theory max k<t Z k,t to calculate ARL = calculate boundary hitting probability of a noise-like random walk the moment of detection of a stopping time, asymptotically exponentially distributed when b, approximately P {max k<t Z k,t b} mλ t

28 Numerical performance analysis Detect emergence of a community interact with p 0 interact with p 1 size of community is S = s

29 Table : Comparison of detection delays for various cases when N = 6. The numbers inside the brackets are the threshold b such that ARL = s = 3, p 0 = 0.2, p 1 = 0.9 s = 3, p 0 = 0.3, p 1 = 0.7 s = 4, p 0 = 0.3, p 1 = 0.7 T ES,1 δ = p (9.96) 9.5 (10.17) 5.0 (8.48) T Mix T H Mix T Mix δ = p 1 δ = p (6.71) 12.8 (6.77) 6.7 (6.88) 3.8 (9.95) 10.8 (10.18) 6.4 (10.17) 6.0 (6.71) 23.3 (6.77) 11.0 (6.88)

30 Robustness against false communities Table : ARL and DD for each algorithm under the conditions p 0 = 0.2, p 1 = 0.9, k = 3, and N = 6 where the ARL = Threshold Detection Delay T ES, T Mix T H Mix Mixture method incorrectly reacts to false community very quickly.

31 Table : Comparison of detection delays for a larger network N = 50. The numbers inside the brackets are the threshold b such that ARL = T Mix, δ = p 1 p 0 = 0.3, p 1 = 0.7, s = (-7.44) p 0 = 0.3, p 1 = 0.7, s = (-7.41)

32 Summary detect emergence of a community in sequential data present a new change-point detection approach three methods: exhaustive search (ES), mixture (Mix), and hierarchical mixture (H-Mix) methods, all able to detect the community quickly in different settings complexity: Mix < H-Mix ES robustness: Mix < H-Mix ES accurate theoretically characterize performance of Mix method future: apply on real data and larger networks: Enron data set