Randomized Gossip Algorithms for Solving Laplacian Systems

Transcription

1 Randomized Gossip Algorithms for Solving Laplacian Systems Anastasios Zouzias and Nikolaos M. reris Abstract We consider the problem of solving a Laplacian system of equations Lx = b in a distributed fashion, where L is the Laplacian of the communication graph. Solving Laplacian systems arises in a number of applications including consensus, distributed control, clock synchronization, localization and calculating effective resistances, to name a few. We leverage our analysis on a randomized variant of Kaczmarz s algorithm to propose a distributed asynchronous gossip algorithm with expected exponential convergence. We quantify the convergence rate depending solely on properties of the network topology, and further propose an accelerated version that scales favorably for larger networks. Our approach naturally extends to leastsquares estimation of general linear systems where each row/column is assigned to nodes of a given network. Last but not least, we show that average consensus is a special case in our framework. Keywords: Gossip algorithms, Laplacian systems, Consensus, Distributed algorithms, Cyberphysical systems, Randomized algorithms, Clock synchronization. I. INTRODUCTION We are entering the era of cyberphysical systems, i.e., very large networks in which collaborating intelligent agents possessing sensing, communication and computation capabilities are interconnected typically via wireless) for performing complicated real-time operations and controlling physical systems [1]. There is a wealth of potential applications such as in transportation systems e.g., highway, railway and air-traffic management), power systems such as smart grids), healthcare such as mobile health), smart houses, defense e.g., military sensor networks), operations research e.g., supply chain management) and many more. An archetypal paradigm is Wireless Sensor Networks WSNs) []. In these largescale complex systems centralized management is not an option for various reasons, most notably: a) fault-tolerant computations are needed to handle the time-varying network topology due to the unreliability of the wireless medium and mobility of agents, b) portable devices have limited energy and computational resources which disallows excessive communication required for message- Anastasios Zouzias is with the department of Mathematical and Computational Sciences at IBM-Research Zürich, Switzerland. zouzias@gmail.com. Nikolaos M. reris is with the Engineering Division of the New York University Abu Dhabi. nfreris@gmail.com passing. Therefore, distributed collaborative actions are indispensable so that the network perform efficiently as an entity; that is, each agent is responsible for acquiring measurements and taking actions in a predefined area, while communicating with others in its proximity via wireless. Moreover, in the typical absence of networkwide synchronization, actions have to be taken asynchronously in fact distributed asynchronous algorithms are used for clock synchronization [3], [4], [5]. A popular class of distributed asynchronous algorithms is that of gossip algorithms [6], [7], [8], widely popularized by the average consensus paradigm, in which each node initially has a value and seeks to compute the average over the entire network, by means of local averaging. Applications are ubiquitous such as in distributed formation control of autonomous vehicles [9], [10], distributed optimization [11], [1], clock synchronization [4], [5], [13], and sensor network monitoring [], to name a few. A. The Model of Computation: Gossip algorithms The gossip model of computation is also known as the asynchronous time model [6], [14]. Here, we focus on the randomized gossip model in which each node can activate itself randomly at a fixed mainly pre-decided) rate. More formally, each node i has a clock which ticks at the times of a Poisson process with rate γ i, where all processes are assumed independent. Therefore, the inter-tick times of each node are rate γ i exponential random variables, independent over all nodes and over time. Equivalently, using properties of the Poisson distribution, this corresponds to activating nodes i V at each tick i.i.d. with probability mass function { γi γ } i V. It becomes evident that if {γ i } are pre-)defined based solely on information accessible to agent i the activation scheme is distributed. To conclude, the gossip model is a special case of the asynchronous model of computation where each node performs a predefined computation infinitely often independent of others); randomization in the gossip model helps to provide bounds on the rate of convergence of the algorithms.

2 B. Our contributions In this paper we leverage our analysis on Randomized Extended Kaczmarz REK) [15] for solving linear systems in the least-squares sense. We showcase that under the gossip model REK gives rise to a new point of view for the design and analysis of gossip algorithms, i.e., solving linear systems Ax = b, where A is adapted to a graph V, E), in the sense that a ij 0 only if i, j) E. We focus on the particular case where A = L, the graph Laplacian, which features many interesting applications such as consensus, clock synchronization and localization without GPS, and calculating effective resistances in a resistance network. Parallel to the wealth of literature in gossip algorithms and average consensus, we also provide algorithms with exponential convergence with rate depending solely on network characteristics via the second smallest eigenvalue of the Laplacian. This paper extends and expands the ideas in [5] which treated the case of smoothing relative measurements; we generalize the analysis to Laplacian systems for which we exploit the structure to develop a gossip algorithm with accelerated convergence over a direct application of REK on L. C. Related Work The analysis of classical synchronous and asynchronous) distributed algorithms for the averaging problem can be traced back to the seminal work of Tsitsiklis [6]. The analysis of randomized gossip-based averaging algorithms for an arbitrary network topology was studied in [7]; Although their results are stated for computing the average function, their theoretical framework can be easily extended to the computation of other functions including maximum, minimum or product. Laplacian systems are encountered in many interesting applications: a) in the normal equations for the smoothing problem with promiment applications in distributed clock synchronization and in-door localization [13], [16], [5], b) in calculating effective resistances in networks [17] which also relates to estimating the variance of MMSE estimates [16], and c) several notable graph problems such as estimating multi-commodity flows, graph partitioning, graph sparsification and random walks on graphs the interested reader is encouraged to read the recent monograph on the subject [17]. Numerical algorithms for Laplacian systems have been a very active field of study most notably by Spielman et al. [18], [17], and the success story is the derivation of primitives with nearly linear in the number of graph edges) complexity. However, all these approaches are naturally centralized as opposed to the distributed model of computation that we focus on here. Solving Laplacian systems in a distributed manner is a fundamental computational primitive. Laplacian solvers have been successfully analyzed in both the synchronous and asynchronous model of computation [6]. However, to the best of our knowledge, solving Laplacian systems under the gossip model of computation remains by and large ill-studied. That said, the techniques of [7] have been applied in [19], [0], [1] to provide a naive solution to the problem, albeit with many drawbacks that we list in the following paragraph. or additional references on the least squares estimation problem, see the survey paper of Dimakis et al. [, IV]. The approach of [19], [0], [1] for solving Laplacian systems applies average consensus algorithms as the main building block towards solving the least-squares estimation problem; in the first distributed) step, each node applies a consensus-based algorithm to approximate all entries of the normal equations matrix of the least squares problem together with the corresponding right hand side vector of the normal equations; in the second step, each node has all the required information to solve the least squares estimation problem individually. The main drawback of this approach is that each node has to compute in parallel) a quadratic number of instances of the averaging problem and additionally is required to solve a linear system; such solution clearly does not scale to large networks. Moreover, these approaches are able to only provide asymptotic convergence, without quantifying the rate. In contrast, our results provide an efficient distributed gossip algorithm, and further establish exponential convergence as well as bounds on the convergence rate, cf. Cor. 5. A. Notation II. PRELIMINARIES The communication network is modeled by an undirected graph G = V, E), where two nodes i, j can exchange packets only if i, j) E. We let n := V be the number of agents and m := E be the number of communication links. or each i V, we define its neighborhood, N i := {j V : i, j) E}. The degree of the node is d i := N i. rom now on we implicitly assume that the input graph is connected 1. or matrix representations, we label the nodes as 1,..., n and write the undirected edge e = i, j) with 1 We assume, without any loss of generality a connected graph, otherwise gossip algorithms may operate independently on each connected component, but no information exchange is possible among components.

3 i < j. The edge-vertex incidence matrix of the graph B R m n has entries: 1, if k = i; B ek := 1, if k = j; 1) 0, otherwise. Let L be the unnormalized) Laplacian matrix of G, L := D A where A R n n is the adjacency matrix of G a ij = 1 if i, j) E, otherwise a ij = 0) and D is the diagonal matrix whose i, i)-th entries is the degree of node i. It is easy to verify that L = B B. inally, for an integer m 1 we let [m] := {1,..., m}. B. Problem ormulation Consider an unknown vector x R n of node variables, where variable x i corresponds to node i. The goal of this paper is to design distributed iterative algorithms under the gossip model of computation that solve Lx = b, where each node i V has access only to the values b i and b j for all j N i, as well as the estimates ˆx i and ˆx j for j N i, corresponding to the solution of the system. Since L is singular see Sec. II-C below) the goal is to compute, in a distributed manner, the entries of the minimum l -norm least squares solution, x LS := L b, where L denotes the pseudo-inverse of L [3]. More precisely, each node i V seeks to compute actually sufficiently approximate) the i-th coefficient of x LS. C. Tools from graph theory Let λ 1 λ... λ n be the eigenvalues of L. or a connected graph we have that λ 1 = 0, and 0 < λ d max, for i =,..., n. The second smallest eigenvalue of L denoted by λ G) is called the iedler value or algebraic connectivity of G; this depends solely on the network topology and can be lower-bounded via Cheeger s inequality [4]. We will show that the rate of convergence of the presented random gossip algorithms depends on λ G). Let B = UΣV be the truncated) singular value decomposition of B, i.e., U and V are m n 1) and n n 1) matrices with orthonormal columns respectively, and Σ is a diagonal matrix of size n 1) with positive elements. Since L = B B, it holds that L = VΣ V. D. Basics from Linear Algebra We use boldface lower case for vectors and upper case for matrices. We denote the rows and columns of A by A 1),..., A m) and A 1),..., A n), respectively both viewed as column vectors). We denote by 0, 1, the vectors with all entries being 0, respectively 1, where the dimensions m are made clear from the context. n We use A := i=1 j=1 a ij and A := max x 0 Ax / x to denote the robenius norm and spectral norm, respectively. Given any b R m, we can uniquely write it as b RA) + b RA), where b RA) is the projection of b onto the column-space of A, RA) := {Ax x R n }, and RA) denotes the orthogonal complement of the column-space. Let σ 1 σ... σ min be the non-zero singular values of A. The Moore-Penrose pseudo-inverse of A is denoted by A [3]. Recall that A = 1/σ min. We denote the inner product between vectors x and y of the same dimension by x, y := i x iy i. or any non-zero real matrix A, we define κ A) := A A = i σ i A). ) A) E. Randomized Extended Kaczmarz σ min In this section we summarize the key results on Randomized Extended Kaczmarz REK) from [15]. The Randomized Kaczmarz RK) algorithm calculates the minimum l norm solution of a consistent linear system Ax = b, by iteratively randomly selecting a row of A and projecting the current estimate to the corresponding solution halfspace { A i), x = b i }, if row i is selected); cf. Alg. 1. Algorithm 1 Randomized Kaczmarz RK) 1: procedure A, b, T ) A R m n, b R m : Set x 0) to be any vector 3: for k = 0, 1,,..., T { 1 do 4: Pick i k [m] w.p. A i) } / A i [m] 5: Set x k+1) = x k) + bi k xk), A i k ) A A i k ) i k) 6: end for 7: Output x T ) 8: end procedure The following theorem is a generalization of the result in [15]; the proof, which we omit for length considerations, is analogous. Theorem 1 RK convergence): Assume that Ax = b has a solution and denote x LS := A b. RK converges to x := x LS + x 0) in the mean-square where x0) := I A A)x 0). or any k > 0 E x k) x 1 1 κ A) ) k x 0) x. 3) Note that if RK is initialized with a vector in the column space of A, e.g., x0) = 0, then it converges to x LS.

4 In the case of an inconsistent linear system, RK does not converge. The following result applies. Lemma RK on inconsistent system): Assume that the system Ax = y has a solution for some y R m ; denote by x := A y. Let ˆx k) denote the k-th iterate of RK applied to the linear system Ax = b with b := y + w, for any fixed w R m Then E k 1 ˆx k) x 1 1 ) ˆx k 1) x + w κ A) A, 4) where E k 1 denotes conditional expectation given the selected rows until k 1. or an inconsistent linear system Ax = b has no solution) we may consider a solution in the least-squares sense, which amounts to solving the normal equations A Ax = A b. This is a consistent linear system whence we can apply RK directly the issue is, however, that convergence may be too slow since κ A A) κ A) 1 see also [5]). To tackle this, we note that solving the normal equations is equivalent to solving Ax = b RA) recall b RA) denotes the projection of b to the range space of A) and apply the following Randomized Orthogonal Projection ROP) algorithm to iteratively approximate b RA). Algorithm Randomized Orthogonal Projection ROP) 1: procedure A, b, T ) A R m n, b R m : Initialize z 0) = b 3: for k = 0, 1,,..., T 1 do 4: Pick j k [n] with probability p j := Aj) / A, j [n] ) 5: Set z k+1) = I m A j k )A j k ) z A jk ) k) 6: end for 7: Output b z T ) 8: end procedure Theorem 3 Covergence of ROP): or any k > 0 it holds that: E z k) b RA) 1 1 ) k κ b RA) A). 5) In effect, combining ROP with RK approximates the minimum l norm least-squares solution A b whether Ax = b is consistent or not. To avoid the delay in obtaining meaningful estimates when running ROP as a pre-processing step to RK, we can combine them together i.e., use steps 4,5 of ROP inside the RK loop, as well as b ik z k) i k in the place of of b ik ). A close look at the analysis of [15] reveals that the random selection of rows/columns need not be independent for a given iteration k. Algorithm 3 Randomized Extended Kaczmarz REK) 1: procedure A, b, T ) A R m n, b R m, : Initialize x 0) = 0 and z 0) = b 3: for k = 0, 1,,..., T { 1 do A i) } 4: Pick i k [m] w.p. / A { i [m] Aj) } 5: Pick j k [n] w.p. / A j [n] 6: Set z k+1) = z k) A j k ), z k) A A jk ) jk ), and 7: x k+1) = x k) + bi k zk) 8: end for 9: end procedure i x k), A i k ) k A A i k ) i k) Theorem 4 Convergence of REK): or each k 0, the estimates of REK with input A possibly rankdeficient) satify: E x k) x LS 1 1 ) k/ 1 + κ A) ) x LS κ A), where κa) := σmaxa) σ mina) is the condition number. Remark 1: The sampling Steps 4 and 5 of Algorithm 3 can be arbitrarility correlated by repeating the proof of Theorem 4.1 of [15] and using a linearity of expectation argument on Equation 4.3 in p.78 of [15]. Remark : All algorithms described here are amenable to a distributed implementation, since activation probabilities, inner products and vector multiplications require, for each node, information from its 1-hop neighborhood. III. RANDOMIZED GOSSIPING VIA RANDOMIZED EXTENDED KACZMARZ We may obtain a randomized gossip algorithm that exponentially converges to the least-squares solution of the Laplacian system corresponding to the underlying communication graph, by directly applying REK Algorithm 3) on the system Lx = b. We call this the Randomized Gossip Laplacian Solver RGLS). The algorithm is fully distributed and asynchronous; to see this note that Lj) = d j + d j, and for any z R n we have z, L j) = dj z j l N j z l, whence activation probabilities and inner product computation can be performed in a decentralized fashion. We skip the exact description of the algorithm for length considerations, as the goal is to obtain an accelerated version in the sequel. or completeness, we show the convergence rate in the following corollary: 6)

5 Corollary 5 Convergence rate of RGLS): The updates of estimates produced by RGLS satisfy: E x k) x LS λ 1 G) ) k/ m κ L) ) x LS i V d. i In particular, for any ε > 0, if k 4m + i V d i 1 + κ A) ) ) x LS λ G) ln ε, then E x k) x LS ε. Proof: The proof is based on the fact that RGLS is an instantiation of Algorithm 3 applied on Lx = b Notice that L = i V d i + d i) = m + i V d i. A. Improved Gossiping for Laplacian Systems The ad-hoc RGLS algorithm requires, roughly speaking, Õm+ i V d i )/λ G)) number 3 of rounds for convergence to a vector arbitrarily close to the least squares solution with high probability. Here we improve the above bound to Om/λ G)) iterations whenever Lx = b has a solution. 4 The main idea is based on the special decomposition of the Laplacian matrix, i.e., L = B B and the following elementary result. Lemma 6: Let x LS be the minimum l -norm least squares solution of Lx = b. Then x LS can be obtained after the following two-step procedure: a) Compute the minimum l -norm least squares solution of B y = b, i.e., y LS := B ) b b) Compute and return the minimum l -norm least squares solution of Bx = y LS, i.e., B y LS Proof: We have B y LS = B B ) b = VΣ 1 U UΣ 1 V b = VΣ V b = L b = x LS, where we used B = UΣV, U U = I m and VΣ V = L. We apply the procedure described in Lemma 6 to solve the Laplacian system. Assuming the notation of Lemma 6, observe that y LS is in the column span of B, hence the linear system Bx = y LS is consistent. The above lemma suggests that we can utilize the Randomized Kaczmarz RK) algorithm Alg. 1) to compute an approximation to y LS, and then again invoke in parallel) RK to solve the linear system of Step b) of Lemma 6. The rationale behind this approach is based on the fact that we are solving two linear systems with coefficient 3 Õ ) is notation for O ) when hiding logarithmic factors depending on ε). 4 The general setting can be handled analogously, i.e., by using REK twice via Lemma 6 but we skip the details for clarity of exposition. Note also that b RL) = b 1 n n i=1 b i1, whence one can equally utilize average consensus [7] for distributedly computing b RL) with rate of convergence 1 λ G)/n). matrix B instead of L = B B, so the modified condition number κ ) is roughly squared-rooted compared to the original system L, i.e., κ B) = m/λ L). We call the resulting algorithm the Accelerated Randomized Gossip Laplacian Solver ARGLS), cf. Alg. 4. The algorithm is again fully distributed, asynchronous and has exponential convergence in the mean square, with rate characterized by the following theorem: Algorithm 4 Accelerated Randomized Gossip Laplacian Solver 1: procedure : for all nodes i V do Initialization step 3: Set x 0) i = 0 and detect neighbors N i 4: Node i obtains b j and sets y 0) i,j) = 0 for all j N i 5: end for 6: for k = 0, 1,,... each clock tick) do 7: Pick a node s k [n] w.p. d sk 8: Pick an edge i k, j k ) uniformly from the edges adjacent to s k i k or j k equals to s k ) 9: Set y k+1) = y k) + b sk y k), B sk ) /d sk d sk B sk ) Run RK on B y = b 10: Set x k+1) = x k) + y k) i k,j k ) xk), B i k,j k )) B i k,j k )) Run RK on Bx = y k) 11: end for 1: end procedure Theorem 7 Convergence rate of ARGLS): or every k > 0, it holds that E x k+1) x LS 1 1 ) k+1 ) κ B) x LS + yls /σ min B). In particular, the algorithm converges in k = Ωm/λ G)) iterations. Proof sketch: irst, note as a plain application of Thm. 1, that E y l) y LS 1 1 ) l κ y LS B). Second, we have: E x k+1) x LS 1 1 ) k+1 ) κ B) x LS + yls /σ min B). This can be established by viewing the evolution of the algorithm as an application of RK applied on a noisy linear system, and invoking the analysis of Lemma with time-varying noise, w k) := y k) y LS. The last portion follows the same lines as the proof of Thm. 4 [15]; we skip the lenghty details.

6 IV. CONNECTION WITH AVERAGE CONSENSUS i V x i. A simple In a network where each node i initially holds a scalar x i the extension to the vector case being trivial), the goal of average consensus is for each node to approx- 1 imately compute the average V scheme to achieve consensus is the gossip algorithm of [7]: an edge e = i, j) is picked uniformly at random and nodes i and j exchange estimates ˆx i and ˆx j, and update their value to ˆx i + ˆx j )/. The analysis of pairwise averaging algorithm used results from Markov chains and random matrices [7]. Now, we demonstrate that the averaging problem can be cast as an instance of RK; apply Algorithm 1 with x 0) the vector of node input values, and set b = 0 and A to be the edge-vertex incidence matrix B of the network. n i=1 x0) i )1 and x LS = Notice that I B B)x 0) = 1 n 0, hence RK asymptotically reaches consensus to the average see Thm. 1). Indeed, B has kernel equal to the span of 1 and hence I B B is the projection matrix to the span of 1. Moreover, Steps 4 and 5 of Algorithm 1 correspond to selecting an edge uniformly at random and applying pairwise averaging on the selected edge, respectively. To conclude, we retrieve the exact same method as in [7] as a special case of our algorithm. V. CONCLUSIONS We have proposed a generalization of randomized gossip algorithms for solving linear systems with expected exponential convergence. or the special class of Laplacian systems, we have leveraged the structure to provide accelerated convergence compared to an ad hoc approach. Our results generalize consensus algorithms and provide new design tools based on random projections. There is a wealth of applications in distributed control and optimization, solving graph partitioning problems as well as distributed clock synchronization and localization in cyberphysical systems. REERENCES [1] K.-D. Kim and P. R. Kumar, Cyber-physical systems: A perspective at the centennial, Proceedings of the IEEE, vol. 100, no. Special Centennial Issue, pp , May 01. [] N. reris, H. Kowshik, and P. R. Kumar, undamentals of Large Sensor Networks: Connectivity, Capacity, Clocks and Computation, Proceedings of the IEEE, vol. 98, no. 1, pp , Nov [3] N. reris, S. Graham, and P. R. Kumar, undamental Limits on Synchronizing Clocks over Networks, IEEE Transactions on Automatic Control, vol. 56, no. 4, pp , Jun [4] N. reris, V. Borkar, and P. R. Kumar, A model-based Approach to Clock Synchronization, in Proceedings of the 48th IEEE Conference on Decision and Control, Sanghai, Dec. 009, pp [5] N. reris and A. Zouzias, ast distributed smoothing of relative measurements, in 51st IEEE Conference on Decision and Control CDC), Dec. 01, pp [6] J. N. Tsitsiklis, D. P. Bertsekas, and M. Athans, Distributed Asynchronous Deterministic and Stochastic Gradient Optimization Algorithms, IEEE Transactions on Automatic Control, vol. 31, no. 9, pp , Sep [7] S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, Randomized Gossip Algorithms, IEEE/ACM Trans. Netw., vol. 14, no. SI, pp , 006. [8] J. Liu, B. D. Anderson, M. Cao, and A. S. Morse, Analysis of accelerated gossip algorithms, Automatica, vol. 49, no. 4, pp , 013. [9] W. Ren, R. Beard, and E. Atkins, Information consensus in multivehicle cooperative control, IEEE Control Systems, vol. 7, no., pp. 71 8, April 007. [10] A. Jadbabaie, J. Lin, and A. Morse, Coordination of groups of mobile autonomous agents using nearest neighbor rules, IEEE Transactions on Automatic Control, vol. 48, no. 6, pp , June 003. [11] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, ound. Trends Mach. Learn., vol. 3, no. 1, pp. 1 1, Jan [1] S. Lee and A. Nedich, Distributed random projection algorithm for convex optimization, Available at eb [13] P. Barooah, N. da Silva, and J. Hespanha, Distributed Optimal Estimation from Relative Measurements for Localization and Time Synchronization, Distributed Computing in Sensor Systems, Lecture Notes in Computer Science, vol. 406, pp , Jun [14] D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation. Prentice Hall, [15] A. Zouzias and N. reris, Randomized Extended Kaczmarz for Solving Least Squares, SIAM Journal on Matrix Analysis and Applications, vol. 34, no., pp , 013. [16] A. Giridhar and P. R. Kumar, The Spatial Smoothing Method of Clock Synchronization in Wireless Networks, Theoretical Aspects of Distributed Computing in Sensor Networks, pp. 7 56, 011. [17] N. K. Vishnoi, Lx = b: Laplacian solvers and their algorithmic applications, Theoretical Computer Science, vol. 8, pp , 01. [18] D. A. Spielman, Algorithms, graph theory, and linear equations in laplacian matrices, Proceedings of the International Congress of Mathematicians, vol. 4, pp , 010. [19] S. Bolognani, S. Del avero, L. Schenato, and D. Varagnolo, Consensus-based Distributed Sensor Calibration and Least- Square Parameter Identification in WSNs, International Journal of Robust and Nonlinear Control, vol. 0, no., pp , 010. [0] L. Xiao, S. Boyd, and S. Lall, A Scheme for Robust Distributed Sensor usion based on Average Consensus, in Proceedings of the Symposium on Information Processing in Sensor Networks IPSN), 005. [1], A Space-Time Diffusion Scheme for Peer-to-Peer Least- Squares Estimation, in Proceedings of the Symposium on Information Processing in Sensor Networks IPSN), 006, pp [] A. G. Dimakis, S. Kar, J. M.. Moura, M. G. Rabbat, and A. Scaglione, Gossip Algorithms for Distributed Signal Processing, Proceedings of the IEEE, vol. 98, no. 11, pp , Nov [3] G. H. Golub and C.. V. Loan, Matrix Computations, 3rd ed. The Johns Hopkins University Press, [4]. Chung, Spectral Graph Theory. American Mathematical Society, 1997.