Algorithms, Graph Theory, and Linear Equa9ons in Laplacians. Daniel A. Spielman Yale University

Transcription

1 Algorithms, Graph Theory, and Linear Equa9ons in Laplacians Daniel A. Spielman Yale University

2 Shang- Hua Teng

3 Carter Fussell TPS David Harbater UPenn Pavel Cur9s Todd Knoblock CTY

4 Serge Lang Gian- Carlo Rota

5 Algorithms, Graph Theory, and Linear Equa9ons in Laplacians Daniel A. Spielman Yale University

6 Solving Linear Equa9ons Ax = b, Quickly Goal: In 9me linear in the number of non- zeros entries of A Special case: A is the Laplacian Matrix of a Graph

7 Solving Linear Equa9ons Ax = b, Quickly Goal: In 9me linear in the number of non- zeros entries of A Special case: A is the Laplacian Matrix of a Graph

8 Solving Linear Equa9ons Ax = b, Quickly Goal: In 9me linear in the number of non- zeros entries of A Special case: A is the Laplacian Matrix of a Graph 1. Why 2. Classic Approaches 3. Recent Developments 4. Connec9ons

9 Graphs Set of ver9ces V. Set of edges E of pairs {u,v} V

10 Graphs Set of ver9ces V. Set of edges E of pairs {u,v} V Erdös Alon Lovász Karp

11 Laplacian Quadra9c Form of G = (V,E) For x : V IR x T L G x = (u,v) E (x (u) x (v)) 2

12 Laplacian Quadra9c Form of G = (V,E) For x : V IR x T L G x = (u,v) E (x (u) x (v)) 2 x :

13 Laplacian Quadra9c Form of G = (V,E) For x : V IR x T L G x = (u,v) E (x (u) x (v)) 2 x : x T L G x =

14 Laplacian Quadra9c Form for Weighted Graph G =(V,E,w) w : E IR + assigns a posi9ve weight to every edge x T L G x = (u,v) E w (u,v) (x (u) x (v)) 2 Matrix L G is posi9ve semi- definite nullspace spanned by const vector, if connected

15 Laplacian Matrix of a Weighted Graph L G (u, v) = w(u, v) if (u, v) E d(u) if u = v 0 otherwise d(u) = (v,u) E w(u, v) the weighted degree of u 1! ! 1! 5 1! 3 3! ! ! ! ! !

16 Laplacian Matrix of a Weighted Graph L G (u, v) = w(u, v) if (u, v) E d(u) if u = v 0 otherwise d(u) = (v,u) E w(u, v) 3 2 1! ! 1! 5 1! 3 3! the weighted degree of u combinatorial degree is # of aaached edges ! ! ! ! !

17 Networks of Resistors Ohm s laws gives i = v/r In general, i = L G v with w (u,v) =1/r (u,v) Minimize dissipated energy v T L G v 1V 0V

18 Networks of Resistors Ohm s laws gives i = v/r In general, i = L G v with w (u,v) =1/r (u,v) Minimize dissipated energy By solving Laplacian v T L G v 0.5V 0.5V 1V 0V 0.375V 0.625V

19 Learning on Graphs Infer values of a func9on at all ver9ces from known values at a few ver9ces. Minimize x T L G x = Subject to known values 0 (u,v) E w (u,v) (x (u) x (v)) 2 1

20 Learning on Graphs Infer values of a func9on at all ver9ces from known values at a few ver9ces. Minimize x T L G x = Subject to known values (u,v) E 0.5 w (u,v) (x (u) x (v)) By solving Laplacian

21 Spectral Graph Theory Combinatorial proper9es of G are revealed by eigenvalues and eigenvectors of L G Compute the most important ones by solving equa9ons in the Laplacian.

22 Solving Linear Programs in Op9miza9on Interior Point Methods for Linear Programming: network flow problems Laplacian systems Numerical solu9on of Ellip9c PDEs Finite Element Method

23 How to Solve Linear Equa9ons Quickly Fast when graph is simple, by elimina9on. Fast when graph is complicated*, by Conjugate Gradient (Hestenes 51, S9efel 52)

24 Cholesky Factoriza9on of Laplacians 1! ! 1! 5 1! 3 1! ! ! ! ! ! When eliminate a vertex, connect its neighbors. Also known as Y- Δ

25 Cholesky Factoriza9on of Laplacians 1! !.33! 1! 5.33! 1!.33! 3 1! ! ! ! ! ! When eliminate a vertex, connect its neighbors. Also known as Y- Δ ! ! ! ! !

26 Cholesky Factoriza9on of Laplacians ! 5.33! 1!.33! 3 1! ! ! ! ! ! When eliminate a vertex, connect its neighbors. Also known as Y- Δ ! ! ! ! !

27 1! ! 1! 5 1! ! ! 1! 3 1!.33! 1! !.4! 1.2! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

28 1! ! 1! 1 1! 4 The order maaers 1! ! ! ! ! ! 1! ! 1 1! ! 1! 4 1! 1! ! ! ! ! ! ! ! ! ! !

29 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree (connected, no cycles) #ops ~ O( V )

30 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree (connected, no cycles) #ops ~ O( V )

31 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V )

32 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V )

33 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V )

34 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V )

35 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V ) Planar #ops ~ O( V 3/2 ) Lipton- Rose- Tarjan 79

36 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V ) Planar #ops ~ O( V 3/2 ) Lipton- Rose- Tarjan 79

37 Complexity of Cholesky Factoriza9on #ops ~ Σ v (degree of v when eliminate) 2 Tree #ops ~ O( V ) Planar #ops ~ O( V 3/2 ) Lipton- Rose- Tarjan 79 Expander like random, but O( V ) edges #ops Ω( V 3 ) Lipton- Rose- Tarjan 79

38 Conductance and Cholesky Factoriza9on Cholesky slow when conductance high Cholesky fast when low for G and all subgraphs For S V S Φ(S) = #edgesleavings sum degrees on smaller side, S or V S Φ G =min S V Φ(S)

39 Conductance S

40 Conductance S Φ(S) =3/5

41 Conductance S

42 Cheeger s Inequality and the Conjugate Gradient Cheeger s inequality (degree- d unwted case) 1 λ 2 2 d Φ G 2 λ 2 d λ 2 = second- smallest eigenvalue of L G ~ d/mixing 9me of random walk

43 Cheeger s Inequality and the Conjugate Gradient Cheeger s inequality (degree- d unwted case) 1 λ 2 2 d Φ G 2 λ 2 d λ 2 = second- smallest eigenvalue of L G ~ d/mixing 9me of random walk Conjugate Gradient finds - approx solu9on to L G x = b in O( d/λ 2 log 1 ) mults by L G in O( E d/λ 2 log 1 ) ops

44 Fast solu9on of linear equa9ons CG fast when conductance high. Elimina9on fast when low for G and all subgraphs.

45 Fast solu9on of linear equa9ons CG fast when conductance high. Planar graphs Elimina9on fast when low for G and all subgraphs. Want speed of extremes in the middle

46 Fast solu9on of linear equa9ons CG fast when conductance high. Planar graphs Elimina9on fast when low for G and all subgraphs. Want speed of extremes in the middle Not all graphs fit into these categories!

47 Precondi9oned Conjugate Gradient Solve L G x = b by Approxima9ng L G by L H (the precondi9oner) In each itera9on solve a system in L H mul9ply a vector by L G - approx solu9on ater O( κ(l G,L H ) log 1 ) itera9ons

48 Precondi9oned Conjugate Gradient Solve L G x = b by Approxima9ng L G by L H (the precondi9oner) In each itera9on solve a system in L H mul9ply a vector by L G - approx solu9on ater O( κ(l G,L H ) log 1 ) itera9ons rela6ve condi6on number

49 Inequali9es and Approxima9on L H L G L G L H if is positive semi-definite, " i.e. for all x, x T L H x x T L G x Example: if H is a subgraph of G x T L G x = (u,v) E w (u,v) (x (u) x (v)) 2

50 Inequali9es and Approxima9on L H L G L G L H if is positive semi-definite, " i.e. for all x, κ(l G,L H ) t x T L H x x T L G x if iff L H L G tl H cl H L G ctl H for some c

51 Inequali9es and Approxima9on L H L G L G L H if is positive semi-definite, " i.e. for all x, κ(l G,L H ) t x T L H x x T L G x if iff L H L G tl H cl H L G ctl H for some c Call H a t- approx of G if κ(l G,L H ) t

52 Other defini9ons of the condi9on number (Golds9ne, von Neumann 47) κ(l G,L H )= max x Span(L H ) x T L G x x T L H x max x Span(L G ) x T L H x x T L G x κ(l G,L H )= λ max(l G L + H ) λ min (L G L + H ) pseudo-inverse min non-zero eigenvalue

53 Vaidya s Subgraph Precondi9oners Precondi9on G by a subgraph H L H L G so just need t for which L G tl H Easy to bound t if H is a spanning tree H And, easy to solve equa9ons in L H by elimina9on

54 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E

55 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E

56 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E path- len 3

57 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E path- len 5

58 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E path- len 1

59 The Stretch of Spanning Trees Boman- Hendrickson 01: L G st G (T )L T Where st T (G) = path-length T (u, v) (u,v) E In weighted case, measure resistances of paths

60 Low- Stretch Spanning Trees For every G there is a T with st T (G) m 1+o(1) (Alon- Karp- Peleg- West 91) where m = E st T (G) O(m log m log 2 log m) (Elkin- Emek- S- Teng 04, Abraham- Bartal- Neiman 08) Solve linear systems in 9me O(m 3/2 log m)

61 Low- Stretch Spanning Trees For every G there is a T with st T (G) m 1+o(1) (Alon- Karp- Peleg- West 91) where m = E st T (G) O(m log m log 2 log m) (Elkin- Emek- S- Teng 04, Abraham- Bartal- Neiman 08) If G is an expander st T (G) Ω(m log m)

62 Expander Graphs Infinite family of d- regular graphs (all degrees d) satsifying λ 2 const > 0 Spectrally best are Ramanujan Graphs (Margulis 88, Lubotzky- Phillips- Sarnak 88) all eigenvalues inside d ± 2 d 1 Fundamental examples Amazing proper9es

63 Expanders Approximate Complete Graphs Let G be the complete graph on n ver9ces (having all possible edges) All non- zero eigenvalues of L G are n x T L G x = n for all x 1, x =1

64 Expanders Approximate Complete Graphs Let G be the complete graph on n ver9ces x T L G x = n for all x 1, x =1 Let H be a d- regular Ramanujan Expander (d 2 d 1) x T L H x (d +2 d 1) κ(l G,L H ) d +2 d 1 d 2 d 1 1

65 Sparsifica9on Goal: find sparse approxima9on for every G S- Teng 04: For every G is an H with O(n log 7 n/ 2 ) edges and κ(l G,L H ) 1+

66 Sparsifica9on S- Teng 04: For every G is an H with O(n log 7 n/ 2 ) edges and κ(l G,L H ) 1+ Conductance high λ 2 high (Cheeger) random sample good (Füredi- Komlós 81) Conductance not high can split graph while removing few edges

67 Fast Graph Decomposi9on by local graph clustering Given vertex of interest find nearby cluster, small Φ(S), in 9me

68 Fast Graph Decomposi9on by local graph clustering Given vertex of interest find nearby cluster, small Φ(S), in 9me S- Teng 04: Lovász- Simonovits Andersen- Chung- Lang 06: PageRank Andersen- Peres 09: evoloving set Markov chain

69 Sparsifica9on Goal: find sparse approxima9on for every G S- Teng 04: For every G is an H with O(n log 7 n/ 2 ) edges and κ(l G,L H ) 1+ S- Srivastava 08: with O(n log n/ 2 ) edges proof by modern random matrix theory Rudelson s concentra9on for random sums

70 Sparsifica9on Goal: find sparse approxima9on for every G S- Teng 04: For every G is an H with O(n log 7 n/ 2 ) edges and κ(l G,L H ) 1+ S- Srivastava 08: with O(n log n/ 2 ) edges Batson- S- Srivastava 09 dn edges and κ(l G,L H ) d +1+2 d d +1 2 d

71 Sparsifica9on by Linear Algebra edges vectors Given vectors v 1,...,v m IR n s.t. v e v T e = I e Find a small subset S {1,...,m} and coefficients c e s.t. e S c e v e v T e I

72 Sparsifica9on by Linear Algebra Given vectors v 1,...,v m IR n s.t. e v e v T e = I Find a small subset S {1,...,m} and coefficients c e s.t. c e v e ve T I e S Rudelson 99 says can find S O(n log n/ 2 ) if choose S at random* * = Pr [e] 1/ v e 2 In graphs, are effec9ve resistances

73 Sparsifica9on by Linear Algebra Given vectors v 1,...,v m IR n s.t. e v e v T e = I Find a small subset S {1,...,m} and coefficients c e s.t. c e v e ve T I e S Batson- S- Srivastava: can find S 4n/ 2 by greedy algorithm with S9eltjes poten9al func9on

74 Rela9on to Kadison- Singer, Paving Conjecture Would be implied by the following strong version of Weaver s conjecture KS 2 Exists constant α s.t. for Exists v e 2 = n m α S {1,...,m}, S = m/2 v 1,...,v m IR n v e ve T = I e v e ve T 1 2 I 1 4 e S s.t. for

75 Rela9on to Kadison- Singer, Paving Conjecture Exists constant α s.t. for v e 2 = n m α Exists S {1,...,m}, S = m/2 v 1,...,v m IR n s.t. for v e ve T = I e v e ve T 1 2 I 1 4 e S Rudelson 99: α = const/(log n) Batson- S- Srivastava 09: true, but with coefficients in sum

76 Rela9on to Kadison- Singer, Paving Conjecture Exists constant α s.t. for v e 2 = n m α Exists S {1,...,m}, S = m/2 v 1,...,v m IR n s.t. for v e ve T = I e v e ve T 1 2 I 1 4 e S Rudelson 99: α = const/(log n) Batson- S- Srivastava 09: true, but with coefficients in sum S- Srivastava 10: Bourgain- Tzafriri Restricted Invertability

77 Sparsifica9on and Solving Linear Equa9ons Can reduce any Laplacian to one with non- zero entries/edges. O( V ) S- Teng 04: Combine Low- Stretch Trees with Sparsifica9on to solve Laplacian systems in 9me O(m log c n log 1 ) m = E n = V

78 Sparsifica9on and Solving Linear Equa9ons S- Teng 04: Combine Low- Stretch Trees with Sparsifica9on to solve Laplacian systems in 9me O(m log c n log 1 ) m = E n = V Kou9s- Miller- Peng 10: 9me O(m log 2 n log 1 ) Kolla- Makarychev- Saberi- Teng 09: O(m log n log 1 ) ater preprocessing

79 What s next Other families of linear equa9ons: from directed graphs from physical problems from op9miza9on problems Solving Linear equa9ons as a primi9ve Decomposi9ons of the iden9ty: Understanding the vectors we get from graphs the Ramanujan bound Kadison- Singer?

80 Conclusion It is all connected