Distributed detection of topological changes in communication networks. Riccardo Lucchese, Damiano Varagnolo, Karl H. Johansson

Transcription

1 1 Distributed detection of topological changes in communication networks Riccardo Lucchese, Damiano Varagnolo, Karl H. Johansson

2 Thanks to... 2

3 The need: detecting changes in topological networks 3

4 The need: detecting changes in topological networks 3

5 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines

6 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Transmit tables of IDs Pros: perfect reconstruction / detection Cons: not scalable

7 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

8 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

9 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

10 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

11 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

12 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

13 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Random-Walks schemes Pros: scalable Cons: not entirely distributed.

14 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

15 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

16 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

17 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

18 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

19 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

20 4 Literature review I.e., potential solutions Main idea: iterate topology estimation routines Exploit Capture-Recapture schemes Pros: scalable Cons: not entirely distributed.

21 5 Our contributions A topology change detector* that is: a Generalized Likelihood Ratio (GLR) test truly distributed scalable and fast (based on max-consensus) In this presentation: algorithm statistical characterization experiments

22 5 Our contributions A topology change detector* that is: a Generalized Likelihood Ratio (GLR) test truly distributed scalable and fast (based on max-consensus) In this presentation: algorithm statistical characterization experiments

23 Preliminaries 6

24 7 Preliminaries Notation on Hypothesis Testing H 0 = {f p θ with θ Ω 0 } H 1 = {f p θ with θ Ω 1 } Ω 1 = Ω c 0 g (f ) : range(f ) {0, 1} g(f ) = 1 under H 0 : error of type I (false positive) g(f ) = 0 under H 1 : error of type II (false negative) R := {f s.t. g (f ) = 0} R c := {f s.t. g (f ) = 1} β g (θ) := P [f R c ; θ] α 0 (g) := sup θ Ω 0 β g (θ)

25 7 Preliminaries Notation on Hypothesis Testing H 0 = {f p θ with θ Ω 0 } H 1 = {f p θ with θ Ω 1 } Ω 1 = Ω c 0 g (f ) : range(f ) {0, 1} g(f ) = 1 under H 0 : error of type I (false positive) g(f ) = 0 under H 1 : error of type II (false negative) R := {f s.t. g (f ) = 0} R c := {f s.t. g (f ) = 1} β g (θ) := P [f R c ; θ] α 0 (g) := sup θ Ω 0 β g (θ)

26 7 Preliminaries Notation on Hypothesis Testing H 0 = {f p θ with θ Ω 0 } H 1 = {f p θ with θ Ω 1 } Ω 1 = Ω c 0 g (f ) : range(f ) {0, 1} g(f ) = 1 under H 0 : error of type I (false positive) g(f ) = 0 under H 1 : error of type II (false negative) R := {f s.t. g (f ) = 0} R c := {f s.t. g (f ) = 1} β g (θ) := P [f R c ; θ] α 0 (g) := sup θ Ω 0 β g (θ)

27 7 Preliminaries Notation on Hypothesis Testing H 0 = {f p θ with θ Ω 0 } H 1 = {f p θ with θ Ω 1 } Ω 1 = Ω c 0 g (f ) : range(f ) {0, 1} g(f ) = 1 under H 0 : error of type I (false positive) g(f ) = 0 under H 1 : error of type II (false negative) R := {f s.t. g (f ) = 0} R c := {f s.t. g (f ) = 1} β g (θ) := P [f R c ; θ] α 0 (g) := sup θ Ω 0 β g (θ)

28 Preliminaries Notation on Graphs Very important assumption: synchronous communications time Considered concepts: k-steps neighbors links among k-steps neighbors i

29 Preliminaries Notation on Graphs Very important assumption: synchronous communications time Considered concepts: k-steps neighbors links among k-steps neighbors i

30 9 Problem Formulation (simplified) S (i) k (t) := size of k-steps neighborhood of node i at time t H 0 : S (i) k (t N) =... = S(i) (t 1) = S, S(i) (t) σs k k H 1 : S (i) k (t N) =... = S(i) (t 1) = S, S(i) (t) < σs k k Parameters: σ (relative amplitude of change) N (horizon)

31 Algorithms 10

32 11 Size Estimation with Max Consensus i.i.d. local generation max consensus log (y max ) exp ( S ) estimate S with statistical inference.

33 11 Size Estimation with Max Consensus i.i.d. local generation y 5 U[0, 1] max consensus y 2 U[0, 1] log (y max ) exp ( S ) y 3 U[0, 1] y 1 U[0, 1] estimate S with statistical inference. y 4 U[0, 1]

34 11 Size Estimation with Max Consensus i.i.d. local generation y 5 max {y i } i max consensus log (y max ) exp ( S ) y 2 max {y i } i y 3 max {y i } i y 1 max {y i } i estimate S with statistical inference. y 4 max {y i } i

35 11 Size Estimation with Max Consensus i.i.d. local generation y max max consensus y max log (y max ) exp ( S ) y max y max estimate S with statistical inference. y max

36 11 Size Estimation with Max Consensus i.i.d. local generation y max max consensus y max log (y max ) exp ( S ) y max y max estimate S with statistical inference. y max

37 11 Size Estimation with Max Consensus i.i.d. local generation y max max consensus y max log (y max ) exp ( S ) y max y max estimate S with statistical inference. y max

38 11 Size Estimation with Max Consensus i.i.d. local generation max consensus log (y max ) exp ( S ) estimate S with statistical inference. y max (1). y max (M) y max (1). y max (M) y max (1). y max (M) y max (1). y max (M) y max (1). y max (M)

39 12 Characteristics (under no-quantization assumptions) ( ML estimator: Ŝ = 1 ) 1 M log (y max (m)) M m=1 Ŝ SM Inv-Gamma( M, 1 ) [Ŝ ] E = M S M 1 (Ŝ ) S var 1 S M M trades off performance vs. communications

40 Characteristics (under no-quantization assumptions) ( ML estimator: Ŝ = 1 ) 1 M log (y max (m)) M m=1 Ŝ SM Inv-Gamma( M, 1 ) [Ŝ ] E = M S M 1 (Ŝ ) S var 1 S M M trades off performance vs. communications

41 13 Extension: Size Estimation of k-steps Neighborhoods i t

42 13 Extension: Size Estimation of k-steps Neighborhoods i t

43 13 Extension: Size Estimation of k-steps Neighborhoods i t

44 13 Extension: Size Estimation of k-steps Neighborhoods i t

45 13 Extension: Size Estimation of k-steps Neighborhoods i t

46 13 Extension: Size Estimation of k-steps Neighborhoods i t

47 13 Extension: Size Estimation of k-steps Neighborhoods i t

48 13 Extension: Size Estimation of k-steps Neighborhoods i t

49 13 Extension: Size Estimation of k-steps Neighborhoods i t

50 13 Extension: Size Estimation of k-steps Neighborhoods i t

51 13 Extension: Size Estimation of k-steps Neighborhoods i t

52 14 Extension: k-steps Neighborhood Size Change-Detection t χ := 1 log (y max (m)) = M Ŝ 1 m.

53 14 Extension: k-steps Neighborhood Size Change-Detection t χ := 1 log (y max (m)) = M Ŝ 1 m. 0.33

54 14 Extension: k-steps Neighborhood Size Change-Detection t χ := 1 log (y max (m)) = M Ŝ 1 m

55 14 Extension: k-steps Neighborhood Size Change-Detection t χ := 1 log (y max (m)) = M Ŝ 1 m

56 14 Extension: k-steps Neighborhood Size Change-Detection t χ := 1 log (y max (m)) = M Ŝ 1 m

57 15 k-steps Neighborhood Size Change-Detection H 0 : S(t N) =... = S(t 1) = S S(t) σs H 1 : S(t N) =... = S(t 1) = S S(t) < σs used to estimate S under H 0 χ(t 5) χ(t 4) χ(t 3) χ(t 2) χ(t 1) χ(t) used to estimate S(t) under no hypotheses or H 0

58 15 k-steps Neighborhood Size Change-Detection H 0 : S(t N) =... = S(t 1) = S S(t) σs H 1 : S(t N) =... = S(t 1) = S S(t) < σs used to estimate S under H 0 χ(t 5) χ(t 4) χ(t 3) χ(t 2) χ(t 1) χ(t) used to estimate S(t) under no hypotheses or H 0

59 15 k-steps Neighborhood Size Change-Detection H 0 : S(t N) =... = S(t 1) = S S(t) σs H 1 : S(t N) =... = S(t 1) = S S(t) < σs used to estimate S under H 0 χ(t 5) χ(t 4) χ(t 3) χ(t 2) χ(t 1) χ(t) used to estimate S(t) under no hypotheses or H 0

60 k-steps Neighborhood Size Change-Detection 1 (estimation of the pre-change value) ( 1 t 1 S = N τ=t N 2 (estimation of the post-change value) χ(τ) ) 1 Ŝ(t) = χ(t) 1 { Ŝ(t) if Ŝ(t) σs Ŝ 0 (t) = 3 (computation of the log-glr) ) (Ŝ0 (t) Λ = Mlog Ŝ(t) σs otherwise ) (Ŝ0 (t) Ŝ(t) χ(t) 4 (decision between H 0 and H 1 ) g ( f ) { 0 if Λ λ (how to compute λ in 2 slides) = 1 otherwise

61 17 Neighborhood Size Change-Detection General χ(t 5) χ(t 4) χ(t 3) χ(t 2) χ(t 1) χ(t) H 0 : S(t N) =... = S(t T ) = S τ {t T + 1,..., 0} S(t τ) σs H 1 : S(t N) =... = S(t T ) = S τ {t T + 1,..., 0} s.t. S(t) < σs Parameters: σ (relative amplitude of change) N (outer horizon) T (inner horizon) Algorithm: parallelize the previous one!

62 17 Neighborhood Size Change-Detection General χ(t 5) χ(t 4) χ(t 3) χ(t 2) χ(t 1) χ(t) H 0 : S(t N) =... = S(t T ) = S τ {t T + 1,..., 0} S(t τ) σs H 1 : S(t N) =... = S(t T ) = S τ {t T + 1,..., 0} s.t. S(t) < σs Parameters: σ (relative amplitude of change) N (outer horizon) T (inner horizon) Algorithm: parallelize the previous one!

63 Characterization 18

64 Computation of the Thresholds - General Case Γ (M, M) 1 set set q 1 = 1 with Γ (a, b) = upper incomplete Γ (M) Gamma function, Γ (a) = Gamma function 2 set p 1 ( ) = Gamma ( M, M 1) 3 evaluate the Lambert W -function in the interval ( 1 e, 0) 4 set p 2 (a) = p 1 ( W ( e a 1 )) W ( e a 1) e a 1 5 set p ν (a) as the mixed probability density and mass function of { 1 with mass q1 ν = a (0, 1) with density p 2 (a) /q 1 6 compute the mixed probability density and mass function of ω as T times {}}{ p ω ( ) = p ν ( ) p ν ( ) 7 compute the quantile function of ω, F 1 ω ( ) 8 set λ T = F 1 ω (α 0 )

65 20 Computation of the Power - General Case [ βg r (κ, M) := P f R c ; [ ]] S,..., S, κσs,..., κσs 1 compute q 1 = 1 as before 2 set p 1 (a) = Gamma (M, (κm) 1) 3 compute F ω (λ T ) as before 4 compute Γ (M, κm) Γ (M) β r g (κ, M) = F ω (λ T ) no UMP test exists for this problem!

66 Computation of the Power - General Case [ βg r (κ, M) := P f R c ; [ ]] S,..., S, κσs,..., κσs 1 compute q 1 = 1 as before 2 set p 1 (a) = Gamma (M, (κm) 1) 3 compute F ω (λ T ) as before 4 compute Γ (M, κm) Γ (M) β r g (κ, M) = F ω (λ T ) no UMP test exists for this problem!

67 Computation of the Power - General Case [ βg r (κ, M) := P f R c ; [ ]] S,..., S, κσs,..., κσs 1 compute q 1 = 1 as before 2 set p 1 (a) = Gamma (M, (κm) 1) 3 compute F ω (λ T ) as before 4 compute Γ (M, κm) Γ (M) β r g (κ, M) = F ω (λ T ) no UMP test exists for this problem!

68 21 1 β g(κ, r M) 1 β g(κ, r M) α 0 = 0.01, T = M α 0 = 0.05, T = M 1 β g(κ, r M) 1 β g(κ, r M) α 0 = 0.01, T = M α 0 = 0.05, T = M κ = 0.7 κ = 0.8 κ = 0.9

69 22 Experiments video

70 23 Conclusions using max-consensus (fast scheme!) is meaningful for topology change detection purposes main tradeoff = performance vs. communication requirements characterization can be used parameters selection future direction: adapt for traffic management purposes

71 24 Distributed detection of topological changes in communication networks Riccardo Lucchese, Damiano Varagnolo, Karl H. Johansson entirely written in L ATEX 2ε using Beamer and Tik Z licensed under the Creative Commons BY-NC-SA 2.5 European License: