Sparsification of Graphs and Matrices

Transcription

1 Sparsification of Graphs and Matrices Daniel A. Spielman Yale University joint work with Joshua Batson (MIT) Nikhil Srivastava (MSR) Shang- Hua Teng (USC) HUJI, May 21, 2014

2 Objective of Sparsification: Approximate any (weighted) graph by a sparse weighted graph.

3 Objective of Sparsification: Approximate any (weighted) graph by a sparse weighted graph. Spanners - Preserve Distances [Chew 89] Cut- Sparsifiers preserve wt of edges leaving every set S V [Benczur- Karger 96] S S

4 Spectral Sparsification [S- Teng] Approximate any (weighted) graph by a sparse weighted graph. Graph G =(V,E,w) Laplacian X w u,v (x(u) x(v)) 2 (u,v)2e

5 Laplacian Quadratic Form, examples All edge- weights are 1 1 x :

6 Laplacian Quadratic Form, examples All edge- weights are 1 1 x : x T L G x = = X (u,v)2e (x(u) x(v)) 2 Sum of squares of differences across edges

7 Laplacian Quadratic Form, examples All edge- weights are 1 1 x : x T L G x = = =1 X 1 (u,v)2e Sum of squares of differences across edges 1 (x(u) x(v)) 2

8 Laplacian Quadratic Form, examples When x is the characteristic vector of a set S, sum the weights of edges on the boundary of S x T L G x = X w u,v (u,v)2e u2s v62s S

9 Learning on Graphs (Zhu- Ghahramani- Lafferty 03) Infer values of a function at all vertices from known values at a few vertices. Minimize X (a,b)2e Subject to known values (x(a) x(b)) 2 1 0

10 Learning on Graphs (Zhu- Ghahramani- Lafferty 03) Infer values of a function at all vertices from known values at a few vertices. Minimize X (a,b)2e Subject to known values (x(a) x(b))

11 Graphs as Resistor Networks View edges as resistors connecting vertices Apply voltages at some vertices. Measure induced voltages and current flow. 1V 0V

12 Graphs as Resistor Networks View edges as resistors connecting vertices Apply voltages at some vertices. Measure induced voltages and current flow. 0.5V 0.5V 1V 0V 0.375V 0.625V

13 Graphs as Resistor Networks View edges as resistors connecting vertices Apply voltages at some vertices. Measure induced voltages and current flow. Induced voltages minimize X (v(a) v(b)) 2 (a,b)2e Subject to fixed voltages (by battery)

14 Graphs as Resistor Networks Effective Resistance between s and t = potential difference of unit flow s t Re (s, t) = s 0 t Re (s, t) =

15 Laplacian Matrices x T L G x = X (u,v)2e w u,v (x(u) x(v)) 2 X L G = w u,v L u,v (u,v)2e 1 1 E.g. L 1,2 = = 1 1 1

16 Laplacian Matrices x T L G x = L G = X (u,v)2e X (u,v)2e = X (u,v)2e w u,v (x(u) x(v)) 2 w u,v L u,v w u,v (b u,v b T u,v) where b u,v = u v Sum of outer products

17 Laplacian Matrices x T L G x = L G = X (u,v)2e X (u,v)2e = X (u,v)2e w u,v (x(u) x(v)) 2 w u,v L u,v w u,v (b u,v b T u,v) Positive semidefinite If connected, nullspace = Span(1)

18 Inequalities on Graphs and Matrices For matrices M and f M M 4 f M if x T Mx apple x T f Mx for all x For graphs G =(V,E,w) and H =(V,F,z) G 4 H if L G 4 L H

19 Inequalities on Graphs and Matrices For matrices M and f M M 4 f M if x T Mx apple x T f Mx for all x For graphs G =(V,E,w) and H =(V,F,z) G 4 H G 4 kh if L G 4 L H if L G 4 kl H

20 Approximations of Graphs and Matrices M M f 1 if 1+ M 4 M f 4 (1 + )M For graphs G =(V,E,w) and H =(V,F,z) G H if L G L H

21 Approximations of Graphs and Matrices M M f 1 if 1+ M 4 M f 4 (1 + )M For graphs G =(V,E,w) and H =(V,F,z) G H That is, for all if L G L H 1 1+ apple P(u,v)2F z u,v(x(u) x(v)) 2 P(u,v)2E w apple 1+ u,v(x(u) x(v)) 2

22 Implications of Approximation G H Boundaries of sets are similar. Effective resistances are similar. L H and L G have similar eigenvalues L + G L + H Solutions to systems of linear equations are similar.

23 Spectral Sparsification [S- Teng] For an input graph G with n vertices, find a sparse graph H having Õ(n) edges so that G H

24 Why? Solving linear equations in Laplacian Matrices key part of nearly- linear time algorithm use for learning on graphs, maxflow, PDEs, Preserve Eigenvectors, Eigenvalues and electrical properties Generalize Expanders Certifiable cut- sparsifiers

25 Approximations of Complete Graphs are Expanders Expanders: d- regular graphs on n vertices (n grows, d fixed) every set of vertices has large boundary random walks mix quickly incredibly useful

26 Approximations of Complete Graphs are Expanders Expanders: d- regular graphs on n vertices (n grows, d fixed) weak expanders: eigenvalues bounded from 0 strong expanders: all eigenvalues near d

27 Example: Approximating a Complete Graph For G the complete graph on n verts. all non- zero eigenvalues of L G are n. For, x? 1 kxk =1 x T L G x = n

28 Example: Approximating a Complete Graph For G the complete graph on n verts. all non- zero eigenvalues of L G are n. For, x? 1 kxk =1 x T L G x = n For H a d- regular strong expander, all non- zero eigenvalues of L H are close to d. For, x? 1 kxk =1 x T L H x d

29 Example: Approximating a Complete Graph For G the complete graph on n verts. all non- zero eigenvalues of L G are n. For, x? 1 kxk =1 x T L G x = n For H a d- regular strong expander, all non- zero eigenvalues of L H are close to d. For, x? 1 n d H kxk =1 x T L H x d is a good approximation of G

30 Best Approximations of Complete Graphs Ramanujan Expanders [Margulis, Lubotzky- Phillips- Sarnak] d 2 p d 1 apple (L H ) apple d +2 p d 1

31 Best Approximations of Complete Graphs Ramanujan Expanders [Margulis, Lubotzky- Phillips- Sarnak] d 2 p d 1 apple (L H ) apple d +2 p d 1 Cannot do better if n grows while d is fixed [Alon- Boppana]

32 Best Approximations of Complete Graphs Ramanujan Expanders [Margulis, Lubotzky- Phillips- Sarnak] d 2 p d 1 apple (L H ) apple d +2 p d 1 Can we approximate every graph this well?

33 Example: Dumbbell G Complete graph on n vertices 1 Complete graph on n vertices H d- regular Ramanujan, times n/d 1 d- regular Ramanujan, times n/d 33

34 Example: Dumbbell K n 1 K n d- regular Ramanujan, times n/d 1 d- regular Ramanujan, times n/d 34

35 Example: Dumbbell G 2 G 1 1 G 3 K n K n H 1 d- regular Ramanujan, times n/d H 2 1 d- regular Ramanujan, times n/d H 3 G = G 1 + G 2 + G 3 H = H 1 + H 2 + H 3 G 1 4 (1 + )H 1 G 2 = H 2 G 4 (1 + )H G 3 4 (1 + )H 3 35

36 Cut- Sparsifiers [Benczur- Karger 96] For every G, is an H with for all x {0, 1} n 1 1+ apple xt L H x x T L G x O(n log n/ 2 ) apple 1+ edges S S

37 Cut- approximation is different m 1 k

38 Cut- approximation is different m 1 k m 3

39 Cut- approximation is different m 1 k m 3 x T L G x = km + m 2 Need long edge if k<m

40 Main Theorems for Graphs For every G =(V,E,w), there is a H =(V,F,z) s.t. G H and F V (2 + ) 2 / 2

41 Main Theorems for Graphs For every G =(V,E,w), there is a H =(V,F,z) s.t. G H and F V (2 + ) 2 / 2 Within a factor of 2 of the Ramanujan bound

42 Main Theorems for Graphs For every G =(V,E,w), there is a H =(V,F,z) s.t. G H and F V (2 + ) 2 / 2 By careful random sampling, get (S- Srivastava 08) In time O( E log 2 V log(1/ )) (Koutis- Levin- Peng 12)

43 Sparsification by Random Sampling Assign a probability p u,v to each edge (u,v) Include edge (u,v) in H with probability p u,v. If include edge (u,v), give it weight w u,v /p u,v E [ L H ]= X (u,v)2e p u,v (w u,v /p u,v )L u,v = L G

44 Sparsification by Random Sampling Choose p u,v to be w u,v times the effective resistance between u and v. Low resistance between u and v means there are many alternate routes for current to flow and that the edge is not critical. Proof by random matrix concentration bounds (Rudelson, Ahlswede- Winter, Tropp, etc.)

45 Matrix Sparsification M = B B T fm = 1 (1 + ) M 4 f M 4 (1 + )M

46 Matrix Sparsification M = B B T = P e b eb T e fm = = P e s eb e b T e 1 (1 + ) M 4 f M 4 (1 + )M most s e =0

47 Main Theorem (Batson- S- Srivastava) For M = P e b eb T e, there exist s e so that for fm = P e s eb e b T e M f M and at most n(2 + ) 2 / 2 s e are non- zero

48 Simplification of Matrix Sparsification 1 (1 + ) M 4 M f 4 (1 + )M is equivalent to 1 (1 + ) I 4 M 1/2 MM f 1/2 4 (1 + )I

49 Simplification of Matrix Sparsification 1 (1 + ) I 4 M 1/2 f MM 1/2 4 (1 + )I Set v e = M 1/2 b e X e v e v T e = I Decomposition of the identity X hu, v e i 2 = kuk 2 e

50 Simplification of Matrix Sparsification 1 (1 + ) I 4 M 1/2 MM f 1/2 4 (1 + )I Set v e = M 1/2 b e We need X v e ve T = I e X s e v e ve T I e

51 Simplification of Matrix Sparsification 1 (1 + ) I 4 M 1/2 MM f 1/2 4 (1 + )I Set v e = M 1/2 b e We need X v e ve T = I e X s e v e ve T I e Random sampling sets p e = kv e k 2

52 What happens when we add a vector?

53 Interlacing

54 More precisely Characteristic Polynomial:

55 More precisely Characteristic Polynomial: Rank- one update: p A+vv T = 1+ P i <v,u i > 2 i x p A Where Au i = i u i

56 More precisely Characteristic Polynomial: Rank- one update: p A+vv T = 1+ P i <v,u i > 2 i x p A are zeros of

57 Physical model of interlacing λ i = positive unit charges resting at barriers on a slope

58 Physical model of interlacing u i is eigenvector v hv, u i i 2 is added vector charge on barrier

59 Physical model of interlacing Barriers repel eigs. u i is eigenvector v hv, u i i 2 is added vector charge on barrier

60 Physical model of interlacing Barriers repel eigs. Inverse law repulsion gravity

61 Examples

62 v = u 1 Ex1: All weight on u 1

63 v = u 1 Ex1: All weight on u 1

64 v = u 1 Ex1: All weight on u 1

65 Ex1: All weight on u 1 Gravity keeps resting on barrier Pushed up against next barrier v = u 1

66 Ex2: Equal weight on u1, u2

67 Ex2: Equal weight on u1, u2

68 Ex2: Equal weight on u1, u2

69 Ex3: Equal weight on all u 1, u 2,, u n +1/n +1/n +1/n

70 Ex3: Equal weight on all u 1, u 2,, u n +1/n +1/n +1/n

71 Adding a random v e Because v e are decomposition of identity, h i E hv e,u i i 2 =1/m e E e PA+ve v T e 1 = 1+ m X i i 1 x! P A 1 = P A m d dx P A

72 Many random v e E PA+ve v T (x) 1 d =1 e e m dx P A(x) h i E P ve1 v e 1,...,e e T + +v ek v T (x) = 1 k 1 e k 1 m k d x n dx

73 Many random v e E PA+ve v T (x) 1 d =1 e e m dx P A(x) h i E P ve1 v e 1,...,e e T + +v ek v T (x) = 1 k 1 e k 1 m k d x n dx Is an associated Laguerre polynomial! For k = n/ 2, roots lie between (1 ) 2 n and m 2 (1 + ) 2 n m 2

74 Matrix Sparsification Proof Sketch Have X v e v T e = I Want X s e v e v T e I e e Will do with {e : s e 6=0} apple 6n All eigenvalues between 1 and 13, 2.6

75 Broad outline: moving barriers - n 0 n

76 Step 1 - n 0 n

77 Step 1 - n 0 n - n 0 n

78 Step 1 - n 0 n +1/ n+1/3 0 n+2

79 Step 1 - n tighter constraint looser constraint 0 n +1/ n+1/3 0 n+2

80 Step i+1 0

81 Step i+1 +1/3 +2 0

82 Step i+1 0

83 Step i+1 +1/3 +2 0

84 Step i+1 0

85 Step i+1 +1/3 +2 0

86 Step i+1 0

87 Step i+1 0

88 Step i+1 0

89 Step i+1 0

90 Step 6n 0 n 13n

91 Step 6n 0 n 13n 2.6-approximation with 6n vectors.

92 Problem need to show that an appropriate always exists.

93 Problem need to show that an appropriate always exists. Is not strong enough for induction

94 Problems If many small eigenvalues, can only move one Bunched large eigenvalues repel the highest one

95 The Lower Barrier Potential Function (A) = P i 1 i =Tr (A I) 1

96 The Lower Barrier Potential Function (A) = P i 1 i =Tr (A I) 1 (A) apple 1 =) min (A) +1

97 The Lower Barrier Potential Function (A) = P i 1 i =Tr (A I) 1 No λ i within dist. 1 No 2 λ i within dist. 2 No 3 λ i within dist. 3.. No k λ i within dist. k (A) apple 1 =) min (A) +1

98 The Upper Barrier Potential Function u (A) = P i 1 u i =Tr (ui A) 1

99 The Beginning -n 0 n

100 The Beginning -n 0 n

101 Step i+1 0

102 Step i+1 +1/ Lemma. can always choose +s e v e ve T so that potentials do not increase

103 Step i+1 0

104 Step i+1 0

105 Step i+1 0

106 Step i+1 0

107 Step 6n 0 n 13n

108 Step 6n 0 n 13n 2.6-approximation with 6n vectors.

109 Goal Lemma. can always choose +s e v e ve T so that potentials do not increase +1/3 +2 +s e v e v T e

110 Upper Barrier Update +2 Add svv T and set

111 Upper Barrier Update +2 Add svv T and set u 0 (A + svv T ) =Tr (u 0 I A svv T ) 1

112 Upper Barrier Update +2 Add svv T and set u 0 (A + svv T ) =Tr (u 0 I A svv T ) 1 = u0 (A)+ sv T (u 0 I A) 2 v 1 sv T (u 0 I A) 1 v By Sherman- Morrison Formula

113 Upper Barrier Update +2 Add svv T and set u 0 (A + svv T ) =Tr (u 0 I A svv T ) 1 = u0 (A)+ sv T (u 0 I A) 2 v 1 sv T (u 0 I A) 1 v Need apple u (A)

114 How much of Rearranging: vv T can we add? u 0 (A + svv T ) apple u (A) iff 1 sv T (u 0 I A) 2 u (A) u 0 (A) +(u0 I A) 1 v

115 How much of Rearranging: vv T can we add? u 0 (A + svv T ) apple u (A) iff 1 sv T (u 0 I A) 2 u (A) u 0 (A) +(u0 I A) 1 v Write as 1 sv T U A v

116 Lower Barrier Similarly: l 0 (A + svv T ) apple l (A) iff 1 apple sv T (A l 0 I) 2 l 0 (A) l(a) (A l 0 I) 1 v Write as 1 apple sv T L A v

117 Goal Show that we can always add some vector while respecting both barriers. +1/3 +2 +svv T Need: sv T U A v apple 1 apple sv T L A v

118 Two expectations Need: sv T U A v apple 1 apple sv T L A v Can show: E v T e U A v e apple 3/2m e E v T e L A v e 2/m e

119 Two expectations Need: sv T U A v apple 1 apple sv T L A v Can show: So: E v T e U A v e apple 3/2m e E v T e L A v e 2/m e E e v T e U A v e apple Ee v T e L A v e And, exists e : v T e U A v e apple v T e L A v e

120 Two expectations Need: sv T U A v apple 1 apple sv T L A v Can show: So: E v T e U A v e apple 3/2m e E v T e L A v e 2/m e E e v T e U A v e apple Ee v T e L A v e And, exists e : v T e U A v e apple v T e L A v e And s that puts 1 between them

121 Two expectations Need: sv T U A v apple 1 apple sv T L A v Can show: So: E v T e U A v e apple 3/2m e E v T e L A v e 2/m e E e v T e U A v e apple Ee v T e L A v e And, exists e : v T e U A v e apple v T e L A v e And s that puts 1 between them

122 Bounding expectations v T U A v =Tr U A vv T E e Tr UA v e v T e =Tr U A E e ve v T e =Tr(U A I) /m =Tr(U A )/m

123 Bounding expectations Tr (U A )= Tr (u0 I A) 2 u (A) u 0 (A) +Tr (u0 I A) 1 = u0 (A) apple apple 1 u (A) As barrier function is monotone decreasing

124 Bounding expectations Tr (U A )= Tr (u0 I A) 2 u (A) u 0 (A) +Tr (u0 I A) 1 Numerator is derivative of barrier function. As barrier function is convex, = u0 (A) apple u (A) apple 1 apple 1 u 0 u Tr(U A ) apple 1/2+1=3/2

125 Bounding expectations Similarly, Tr (L A ) l 0 1 l 1 = 1 1/3 1 =2

126 Step i+1 +1/ Lemma. can always choose so that potentials do not increase

127 Twice- Ramanujan Sparsifiers Fixing steps and tightening parameters gives ratio max(a) min(a) p d +1+2 d apple d +1 2 p d Less than twice as many edges as used by Ramanujan Expander of same quality

128 Open Questions The Ramanujan bound Properties of vectors from graphs? Faster algorithm union of random Hamiltonian cycles?