arxiv: v1 [cs.lg] 26 Jul 2017

Transcription

1 Updating Singular Value Decomposition for Rank One Matrix Perturbation Ratnik Gandhi, Amoli Rajgor School of Engineering & Applied Science, Ahmedabad University, Ahmedabad , India arxiv: v [cslg] 26 Jul 207 Abstract An efficient Singular Value Decomposition (SVD) algorithm is an important tool for distributed and streaming computation in big data problems It is observed that update of singular vectors of a rank- perturbed matrix is similar to a Cauchy matrix-vector product With this observation, in this paper, we present an efficient method for updating Singular Value Decomposition of rank- perturbed matrix in O(n 2 log( ɛ )) time The method uses Fast Multipole Method (FMM) for updating singular vectors in O(n log( ɛ )) time, where ɛ is the precision of computation Keywords: Multipole Method Introduction Updating SVD, Rank- perturbation, Cauchy matrix, Fast SVD is a matrix decomposition technique having wide range of applications, such as image/video compression, real time recommendation system, text mining (Latent Semantic Indexing (LSI)), signal processing, pattern recognition, etc Computation of SVD is rather simpler in case of centralized system where entire data matrix is available at a single location It is more complex when the matrix is distributed over a network of devices Further, increase in complexity is attributed to the real-time arrival of new data Processing of data addresses: ratnikgandhi@ahdunieduin (Ratnik Gandhi), amolirajgor@ietahdunieduin (Amoli Rajgor)

2 over distributed stream-oriented systems require efficient algorithms that generate results and update them in real-time In streaming environment the data is continuously updated and thus the output of any operation performed over the data must be updated accordingly In this paper, a special case of SVD updating algorithm is presented where the updates to the existing data are of rank- Organization of the paper is as follows: Section 3 motivates the problem of updating SVD for a rank- perturbation and presents a characterization to look at the problem from matrix-vector product point of view An existing algorithm for computing fast matrix-vector product using interpolation is discussed in Section 4 Section 5 introduces Fast Multipole Method (FMM) In Section 6 we present an improved algorithm based on FMM for rank- SVD update that runs in O(n 2 log ɛ ) time, where real ɛ > 0 is a desired accuracy parameter Experimental results of the presented algorithms are given in Section 7 For completeness we have explained in details matrix factorization (Appendix A), solution to Sylvester Matrix (Appendix B), FAST algorithm for Cauchy matrix vector product (Appendix C) and Fast Multipole Method (Appendix D) at the end of this paper 2 Related Work In a series of work Gu and others [, 2, 3, 4] present work on rank-one approximate SVD update Apart from the low-rank SVD update, focus of their work is to discuss numerical computations and related accuracies in significant details This leads to accurate computation of singular values, singular vectors and Cauchy matrix-vector product Our work differs from this work as follows: we use matrix factorization that is explicitly based on solution of Sylvester equation [5] to reach Eq (20) that computes updated singular vectors (see Section 32) Subsequently, we reach an equation, Eq (24) (similar to equation (33) in [2]) Further, we show that with this new matrix decomposition we reach the computational complexity O(n 2 log ɛ ) for updating rank- SVD 2

3 3 SVD of Rank-One Perturbed Matrices Let SVD of a m n matrix A = UΣV Where, U R m m, Σ R m n and V R n n, where, without loss of generality, we assume, m n Let there be a rank-one update ab to matrix A given by, Â = A + ab () and let Û ˆΣ ˆV denote the new(updated) SVD, where a R m, b R n, Thus, ÂÂ = Û ˆΣˆΣ Û (2) An algorithm for updating SVD of a rank- perturbed matrix is given in Bunch and Nielsen [6] The algorithm updates singular values using characteristic polynomial and computes the updated singular vectors explicitly using the updated singular values From (), ÂÂ = (UΣV + ab )(UΣV + ab ) = UΣV V Σ U + UΣV }{{ } b a + a } b V Σ {{ U } +a }{{} b b b b β ÂÂ = UΣΣ U + ba + a b + βaa (3) Where, b = UΣV b, b = b V Σ U and β = b b From (3) it is clear that the problem of rank- update () is modified to problem of three rank- updates (that is further converted to two rank- updates in (4)) From (2) and (3) we get, a Û } ˆΣˆΣ {{ } Û = U ΣΣ }{{ } U + ba + a b + βaa ˆD D Û ˆDÛ = UDU + ρ a a +ρ 2 b b (4) }{{} Ũ DŨ We consider the model and characterization as in [5] A detailed factorization of this matrix is given in Appendix A 3

4 Refer Appendix Appendix A Eq (A6) for more details Similar computation for right singular vectors is required The following computation is to be done for each rank- update, ie, the procedure below will repeat four times, two times each for updating left and right singular vectors From (4) we have, where ā = U a From (6) we have, From (8) and (6) we get, Ũ DŨ = UDU + ρ a a (5) = U (D + ρ āā ) U, }{{} (6) B B := D + ρ āā (7) B = C D C (Schur-decomposition) (8) Ũ DŨ = U( C }{{} Ũ D C )U }{{} (9) Ũ After adding the rank- perturbation to UDU in (5), the updated singular vector matrix is given by matrix-matrix product Ũ = U C (0) Stange [5], extending the work of [6], presents an efficient way of updating SVD by exploring the underlying structure of the matrix-matrix computations of (0) 3 Updating Singular Values An approach for computing singular values is through eigenvalues Given a matrix Ŝ = S + ρuu, its eigenvalues d can be computed in O(n 2 ) numerical operations by solving characteristic polynomial of (S + ρuu )x = dx, where S = diag(d i ) Golub [7] has shown that the above characteristic polynomial has following form 4

5 w( d) n u 2 i = + ρ () d i d i= Note that in the equation above d is an unknown and thus, though the polynomial function is structurally similar to Eq (24), we can not use FMM for solving it Recall, in order to compute singular values ˆD of updated matrix  we need to update D twice as there are two symmetric rank- updates, ie, for Ũ DŨ = UDU + ρ a a we will update B = D + ρ āā Eq (6) and similarly for Û ˆDÛ = Ũ DŨ + ρ 2 b b we will update B = D + ρ 2 b b At times, while computing eigen-system, some of the eigenvalues and eigenvectors are known (This happens when there is some prior knowledge available about the eigenvalues or when the eigenvalues are approximated by using methods such as power iteration) In such cases efficient SVD update method should focus on updating unknown eigen values and eigen vectors Matrix Deflation is the process of eliminating known eigenvalue from the matrix Bunch, Nielsen and Sorensen [8] extended Golub s work by bringing the notion of deflation They presented a computationally efficient method for computing singular values by deflating the system for cases: () when some values of ā and b are zero, (2) ā = and b = and (3) B and B has eigenvalues with repetition (multiplicity more than one) After deflation, the singular values of the updated matrix ˆD can be obtained by Eq () 32 Updating Singular Vectors In order to update singular vectors the matrix-matrix product of (0) is required A naive method for matrix multiplication has complexity O(n 3 ) We 5

6 exploit matrix factorization in Stange [5] that shows the structure of matrix C to be Cauchy From (7) and (8) we get, C D C = D + ρ āā D C C D = ρ āā C (2) Equation (2) is Sylvester equation with solution, (I n D + ( D) I n ) vec C = vec( ρ āā C) (3) Simplifying LHS of (3) for C we get, ā C = Where, C = [ of the form, (µ λ ) (µ n λ ) ā c (µ λ n ) (µ n λ n ) ā c n ] (4) c c n and from RHS of (3), c i R n are columns ā n Simplifying (5) further we get, c i = ρ ā µ i λ ā n µ i λ n ā c i (5) c i = ρ ā µ i λ ā n µ i λ n [ c i = ρ (ā c i + ā 2 c i2 ) ] ā ā 2 c i ā λ µ i ā n λ n µ i c i2 6

7 Thus, denoting ˆα as ρ (ā c i + ā 2 c i2 ) we get, [ c i = ˆα ā λ µ i Placing (6) in (5) and we have, ˆα ā λ µ i ā n λ n µ i Therefore for each element of c i, = ˆαρ ā µ i λ ā n µ i λ n ] ā n λ (6) n µ i n k= ā 2 k λ k µ i ˆα āj ā j n ā 2 k = ˆαρ λ j µ i µ i λ j λ k µ i k= ( ) n ā 2 k 0 = ˆα + ρ λ k µ i k= }{{} =0 (7) Equation inside the bracket of (7) is the characteristic equation () used for finding updated eigenvalues µ i of D by using eigenvalues of D ie λ As µ i are the zeros of this equation Placing λ k and µ i will make the term in the bracket (7) zero Due to independence of the choice of scalar ˆα, any value of ˆα can be used to scale the matrix C In order to make the final matrix orthogonal, each column of C is scaled by inverse of Euclidean norm of the respective column (8) From Eq (6) matrix notation for C can be written as {}} { ā λ µ λ µ n C = ā n λ n µ λ n µ n }{{} [ ] C= c c n C c c n (8) Where, c i is the Euclidean norm of c i From (8) it is evident that the C matrix is similar to Cauchy matrix and C is the scaled version of Cauchy matrix C In 7

8 order to update singular vectors we need to calculate matrix-matrix product as given in (0) From (8) and (0) we get, ā U C = U ā n λ µ λ µ n λ n µ λ n µ n c c n U C c [ ] = û û n C c n Where, û i = u i ā i and i =,, n c U C = U C c n c U C = U 2 c n [ ] Where, U = û û n and U 2 = U C (9) (20) 32 Trummer s Problem: Cauchy Matrix-Vector Product In (9) there are n vectors in U which are multiplied with Cauchy matrix C The problem of multiplying a Cauchy matrix with a vector is called Trummer s problem As there are n vectors in (9), matrix-matrix product in it can be represented as n matrix-vector products, ie, it is same as solving Trummer s problem n times Section 4 describes an algorithm given in [9] which efficiently computes such matrix-vector product in O(n log 2 (n)) time 4 FAST: Method based on Polynomial Interpolation and FFT Consider the the matrix - (Cauchy) matrix product U 2 = U C of Section 32 This product can be written as 8

9 U 2 = U 2 = [ ] û û n C (2) u u n λ µ λ µ n (22) u n u nn λ n µ λ n µ n The dot product of each row vector of U and the coulmn vector of the Cauchy matrix can be represented in terms of a function of eigenvalues µ i, i =,, n, f(µ ) = = f(µ ) = Hence in general, f(x) = [ ] u u n u u n + + λ µ λ n µ n u j λ j µ j= n j= λ µ λ n µ (23) u j λ j x (24) Equation (24) can be shown as the ratio of two polynomials f(x) = h(x) g(x), h(x) = g(x) h(x) = j= n j= n where, g(x) = (λ j x) (25) u j λ j x n n u j (λ j x) λ j x j= j= (26) (27) The FAST algorithm [9] represents the a matrix-vector product as (24) and (25) It then finds solutions to this problem by the use of interpolation The FAST algorithm of Gerasoulis [9] has been reproduce in Appendix C 9

10 5 Matrix-vector product using FMM The FAST algorithm in [9] has complexity O(n log 2 n) It computes the function f(x) using FFT and interpolation We observe that these two methods are two major procedures that contributes to the overall complexity of FAST algorithm To reduce this complexity we present an algorithm that uses FMM for finding Cauchy matrix-vector product that updates the SVD of rank- perturbed matrix with time complexity O(n 2 log( ɛ )) Recall from Section 32, update of singular vectors require matrix-(cauchy) matrix product U 2 = U C, This product is represented as the function below f(µ i ) = n j= We use FMM to compute this function u j µ i λ j 5 Fast Multipole Method (FMM) An algorithm for computing potential and force of n particles in a system is given in Greengard and Rokhlin [0] This algorithm enables fast computation of forces in an n-body problem by computing interactions among points in terms of far-field and local expansions It starts with clustering particles in groups such that the inter-cluster and intra-cluster distance between points make them wellseparated Forces between the clusters are computed using Multipole expansion Dutt et al [] describes the idea of FMM for particles in one dimension and presents an algorithm for evaluating sums of the form, f(x) = N α k φ(x x k ) (28) k= Where, {α,, α N } is a set of complex numbers and φ(x) can be a function that is singular at x = 0 and smooth every where Based on the choice of function, (28) can be used to evaluate sum of different forms for example: f(x) = N k= α k x x k (29) 0

11 where, φ(x) = x Dutt et al [] presents an algorithm for computing summation (29) using FMM that runs in O(n log( ɛ )) 52 Summation using FMM Consider well separated points {x, x 2,, x N } and {y,, y M } in R such that for points x 0, y 0 R and r > 0, r R x i x 0 < r i =,, N y i y 0 < r i =,, M and x 0 y 0 > 4r For a function defined as f : R C such that f(x) = N k= α k x x k (30) where, {α,, α k } is a set of complex numbers Given f(x), the task is to find f(y ),, f(y M ) 6 Rank-One SVD update In this section, we present Algorithm 6 that uses FMM and updates Singular Value Decomposition in O(n 2 log( ɛ )) time 6 Algorithm: Update SVD of rank- modified matrix using FMM INPUT A R m n = UΣV T, a R m and b R n OUTPUT U n, Σ n, and V n STEP Compute b = UΣV T b, ã = V Σ T U T a, β = b T b, α = a T a, D u = ΣΣ T and D v = Σ T Σ STEP 2 Compute the Schur decomposition of [ a b ] [ Q u = a b ] β 0 = Q u ρ 0 0 ρ 2 Q T u and

12 STEP 3 Compute the Schur decomposition of [ b ã ] [ Q v = a 2 b 2 ] α 0 = Q v ρ 0 0 ρ 2 Q T v STEP 4 Compute updated left singular vector Ũ and D by calling the procedure RankOneUpdate(U, a, D, ρ ) - Algorithm 62 STEP 5 Compute left singular vector of Â, ie, U n and D n by calling the procedure RankOneUpdate(Ũ, b D, ρ 2 ) STEP 6 Compute updated right singular vector Ṽ and D v by calling the procedure RankOneUpdate(V, a 2, D v, ρ 3 ) STEP 7 Compute right singular vector of  ie V n and D vn by calling the procedure RankOneUpdate(Ṽ, b 2, Dv, ρ 4 ) STEP 8 Find singular values by computing square root of the updated eigenvalues Σ n = D n Note that the Schur decomposition of Steps 2 and 3 are computed over constant size matrices and thus the decomposition will take constant time 62 Algorithm: RankOneUpdate INPUT U, a, D, ρ OUTPUT Ũ, D STEP Compute ā = U T a STEP 2 Compute µ as zeros of the equation w(µ) = + ρ λ µ λ µ n STEP 3 Compute C = λ n µ STEP 4 Compute U = U STEP 5 Compute C = ā ā ā n λ n µ n ā n 2 n i= λ µ λ µ n λ n µ λ n µ n ā 2 i λ i µ and

13 STEP 6 Compute U 2 = U C as n matrix-vector product Where each row-column dot product is represented as a function for each µ i f(µ i ) = n j= U 2 = FMM(λ, µ, U ) - (Appendix D) u j µ i λ j STEP 7 Form Ũ by dividing each column of U 2 by norm of respective column of C 63 Complexity for Rank-One SVD update Theorem Given a matrix A R m n such that m n and precision of computation parameter ɛ > 0, the complexity of computing SVD of a rank- update of A with Algorithm 6 is O(n 2 log( ɛ )) Proof Follows from Algorithm 6 and Table Section Complexity Operation 3 O(n 2 ) Compute Û ˆΣˆΣ Û and ˆV ˆΣ ˆΣ ˆV 3 O(n 2 ) Update singular values solving w(t) = + ρ n z 2 j λ = 0 j t j= 5 O(n log( ɛ )) Update singular vectors as Ũ = U C using FMM O(n 2 log( ɛ )) Total Complexity Table : Rank- SVD update Complexity 7 Numerical Results All the computations for the algorithm were performed on MATLAB over a machine with Intel i5, quad-core, 7 GHz, 64-bit processor with 8 GB RAM 3

14 Matrices used in these experiments are square and generated randomly with values ranging from [, 9] The sample size varies from 2 2 to For computing the matrix vector product we use FMM Algorithm (instead of FAST Algorithm) with machine precision ɛ =5 0 In order to update singular vectors we need to perform two rank- updates (A6) For each such rank- update we use FMM Û ˆDÛ T = UDU T + ρ a a T T +ρ 2 b b (3) }{{} Ũ DŨ T Figure shows the time taken by FAST Algorithm and FMM Algorithm to compute the first rank- update Figure : (A6) Run-time for rank- update Figure 2: Extrapolated run-time for rank- update (A6) 7 Choice of ɛ In earlier computations we fixed the machine precision parameter ɛ for FMM based on the order of the Chebyshev polynomials, ie, ɛ = 5 p, where p is the order of Chebyshev polynomial As we increase the order of polynomials - we expect reduction in the error and increase in updated vectors accuracy This increase in accuracy comes with the cost of higher computational time To show this we fix the input matrix dimension to and generate values of these matrices randomly in the range [0,] Figure 3 shows the error between updated singular vectors generated by Algorithm 6 and exact computation 4

15 of singular vector updates machine precision p = 20 The results in figure justifies our choice of fixed Figure 3: Error with varying p Error is computed using the equation (32) [5], where  is the perturbed matrix, Û ˆΣ ˆV T is the approximation computed using FMM-SVDU and max ˆσ is the directly computed maximum eigenvalue of  T Â Û ˆΣ ˆV Error = max max ˆσ (32) Table 2 shows the accuracy of rank- SVD update using FMM- SVDU for varying sample size Figure 4 shows plot of the accuracy of FMM-SVDU with varying sample size 5

16 Table 2: rank- SVD update accuracy No Dimension Error n n (Eqn 32) Figure 4: Accuracy of FMM-SVDU 6

17 8 Conclusion In this paper we considered the problem of updating Singular Value Decomposition of a rank- perturbed matrix We presented an efficient algorithm for updating SVD of rank- perturbed matrix that uses the Fast Multipole method for improving Cauchy matrix- vector product computational efficiency An interesting and natural extension of this work is to consider updates of rank-k References References [] M Gu, Studies in numerical linear algebra, PhD thesis, Yale University (993) [2] M Gu, S C Eisenstat, A stable and fast algorithm for updating the singular value decomposition, Tech Rep YALEU/DCS/RR-966, Dept of Computer Science, Yale University (993) [3] M Gu, S C Eisenstat, A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem, SIAM journal on Matrix Analysis and Applications 5 (4) (994) [4] M Gu, S C Eisenstat, Downdating the singular value decomposition, SIAM Journal on Matrix Analysis and Applications 6 (3) (995) [5] P Stange, On the efficient update of the singular value decomposition, PAMM 8 () (2008) [6] J R Bunch, C P Nielsen, Updating the singular value decomposition, Numerische Mathematik 3 (2) (978) 29 [7] G H Golub, Some modified matrix eigenvalue problems, Siam Review 5 (2) (973)

18 [8] J R Bunch, C P Nielsen, D C Sorensen, Rank-one modification of the symmetric eigenproblem, Numerische Mathematik 3 () (978) 3 48 [9] A Gerasoulis, A fast algorithm for the multiplication of generalized Hilbert matrices with vectors, Mathematics of Computation 50 (8) (988) [0] L Greengard, V Rokhlin, A fast algorithm for particle simulations, Journal of computational physics 73 (2) (987) [] A Dutt, M Gu, V Rokhlin, Fast algorithms for polynomial interpolation, integration, and differentiation, SIAM Journal on Numerical Analysis 33 (5) (996) Appendix A Matrix Factorization We consider the matrix factorization in [5] Let SVD of a m n matrix A = UΣV Where, U R m m, Σ R m n and V R n n Here it is assumed that m n Let there be a rank-one update ab to matrix A Eq (A) and let Û ˆΣ ˆV denote the new(updated) SVD, where a R m, b R n  = A + ab (A)  = Û ˆΣ ˆV (A2) In order to find SVD of this updated matrix  we need to compute  and   because left singular vector Û of  is orthonormal eigenvector of  and right singular vector ˆV of  is orthonormal eigenvector of   Â = (Û ˆΣ ˆV )(Û ˆΣ ˆV ) (A3) = Û ˆΣ ˆV ˆV ˆΣ Û Â = Û ˆΣˆΣ Û Where, ˆV ˆV = I (A4) 8

19 ÂÂ = (UΣV + ab )(UΣV + ab ) = (UΣV + ab )(V Σ U + ba ) = UΣV V Σ U + UΣV }{{ } b a + a } b V Σ {{ U } +a }{{} b b b b β ÂÂ = UΣΣ U + ba + a b + βaa Where, V V = I (A5) Where, b = UΣV b, b = b V Σ U and β = b b From Eq (A5) it is clear that the problem of rank- update Eq (A) is modified to problem of three rank- updates that is further converted to two rank- updates in Eq (A6) Equating Eq (A4) and Eq (A5) we get, a Û } ˆΣˆΣ {{ } Û = U ΣΣ }{{ } U + ba + a b + βaa ˆD D Û ˆDÛ = UDU + ba + a b + βaa = UDU + βaa + ba + a b = UDU + (βa + b)a + a b [ = UDU + βa + b a ] a b Û ˆDÛ [ = UDU + a b ] β 0 a b Û ˆDÛ [ = UDU + a b ] Q ρ 0 0 ρ 2 Q a b Where, β 0 = Q ρ 0 0 ρ 2 Q (Schur-decomposition) 9

20 Û ˆDÛ [ = UDU + Û ˆDÛ [ = UDU + Ũ DŨ ] Q a b }{{ [ } ] a b ρ 0 0 ρ 2 ] a b ρ 0 0 ρ 2 Q a ] b } [ {{ } a b a b Û ˆDÛ = UDU + ρ a a +ρ 2 b b (A6) }{{} Similarly for computing right singular vectors do the following   = (UΣV + ab ) (UΣV + ab ) = (V Σ U + ba )(UΣV + ab )   = V Σ U UΣV + V } Σ {{ U a } b + b } a UΣV {{ } +b }{{} a a ã ã α ( ˆV ˆΣ Û )(Û ˆΣ ˆV ) = V Σ ΣV + ãb + bã + αbb Where, U U = I ˆV ˆΣ ˆΣ ˆV = V Σ ΣV + ρ 3 a 2 a 2 + ρ 4 b 2 b 2 (A7) b The following computation is to be done for each rank- update in Eq (A6) and Eq (A7) ie the below procedure will repeat four times, two times each for updating left and right singular vectors From Eq (A6) we have, Ũ DŨ = UDU + ρ a a (A8) = UDU + ρ (UU a )(UU a ) = UDU + ρ (Uā)(Uā) Where, ā = U a = UDU + ρ (Uāā U ) Ũ DŨ = U (D + ρ āā ) U }{{} B From Eq (A9) we have, (A9) B := D + ρ āā (A0) B = C D C (Schur-decomposition) (A) Placing Eq (A) in Eq (A9) we get, 20

21 Ũ DŨ = U( C }{{} Ũ D C )U }{{} (A2) Ũ After the rank- update to UDU Eq (A8) the updated singular vector matrix is given by matrix-matrix product Ũ = U C (A3) Appendix B Solution to Sylvester Equation In Section 32 we discussed method for updating singular vectors For the same we derived solutions to (2) using Sylvester equation In this section we present details of how this solution can be obtained For simplicity consider the case where the dimension of all the matrices (C, D and D) is 2 2 LHS = (I n D + ( D) I n ) vec C = 0 λ 0 µ λ 2 0 µ 2 0 λ µ c 0 λ = µ 0 0 c λ µ 2 0 c λ µ 2 c 22 λ µ c 0 λ = 2 µ 0 0 c λ µ 2 0 c λ 2 µ 2 c 22 vec c c 2 c 2 c 22 2

22 RHS = vec( ρ āā C) = ρ vec = ρ vec ā ā 2 = ρ vec ā ā 2 = ρ vec ā ā 2 ā ā 2 [ ā ā 2 ] c c 2 c 2 c 22 [ ā c + ā 2 c 2 ā c 2 + ā 2 c 22 ] [ ] ā ā 2 c [ ] ā ā 2 c 2 c 2 [ ā c ā c 2 ] [ ] Where, C = c c 2 and c = c c 2 = = ρ vec c ā ā c 2 ā 2 ā c ā 2 ā c 2 ā ā c ā = ρ 2 ā c ā ā c 2 ā 2 ā c 2 c 2 c 2 c 22 c 22 (B) Equating LHS and RHS we get, λ µ λ 2 µ λ µ 2 0 c c 2 c 2 = ρ ā ā c ā 2 ā c ā ā c λ 2 µ 2 c 22 ā 2 ā c 2 22

23 Therefore, Where, c c 2 c 2 c 22 c c 2 c 2 c 22 c c 2 c 2 c 22 c c 2 c 2 c 22 c c 2 λ µ ā ā c 0 λ = ρ 2 µ 0 0 ā 2 ā c 0 0 λ µ 2 0 ā ā c λ 2 µ 2 ā 2 ā c 2 λ µ ā ā c 0 = ρ λ 2 µ 0 0 ā 2 ā c 0 0 λ µ 2 0 ā ā c λ 2 µ 2 ā 2 ā c 2 = ρ = ρ ā ā c λ µ ā 2ā c λ 2 µ ā ā c 2 λ µ 2 ā 2ā c 2 λ 2 µ 2 ā ā c µ λ ā 2ā c µ λ 2 ā ā c 2 µ 2 λ ā 2ā c 2 µ 2 λ 2 = ρ āā c µ λ ā 2ā c µ λ 2 ie c = ρ In general c i = ρ ā µ λ ā 2 µ λ 2 ā µ i λ ā 2 µ i λ n ā c and ā c c 2 c 22 = ρ āā c 2 µ 2 λ ā 2ā c 2 µ 2 λ 2 and c 2 = ρ ā µ 2 λ ā 2 µ 2 λ 2 By placing c and c 2 in C [ ] = c c 2 we get C as below ā c 2 (B2) 23

24 [ C = C = c c 2 ] āā c ā ā c 2 µ λ µ 2 λ ā ā c ā ā c 2 µ λ 2 µ 2 λ 2 C = ā 0 0 ā 2 C = ā 0 0 ā 2 µ λ µ 2 λ ā c ā c 2 ā c µ λ 2 ā c 2 µ 2 λ 2 µ λ µ 2 λ µ λ 2 µ 2 λ 2 ā c 0 0 ā c 2 Appendix C FAST Algorithm In this section we present FAST Algorithm [9] that computes functions of the form (24) using polynomial interpolation in time O(n log 2 n) Compute the coefficients of g(x) in its power form, by using FFT polynomial multiplication, in O(n(log n) 2 ) time Decription: Uses FFT for speedy multiplication which reduces complexity of multiplication to O(n log(n)) from O(n 2 ) Input: Function g(x) and eigenvalues of D ie [µ,, µ n ] Output: Coefficients of g(x) ie [a 0, a,, a n ] Complexity: O(n log 2 n) 2 Compute the coefficients of g (x) in O(n) time Description: Differentiate the function g(x) and then compute its coefficients Input: Function g(x) Output: Coefficients of g (x) ie [b 0, b,, b n ] Complexity: O(n) 24

25 3 Evaluate g(λ i ), g (λ i ) and g(µ i ) Description: For the functions g(x) and g (x) evaluate their values at λ i and µ i Input: Function g(x) and g (x), eigenvalues [λ, λ 2,, λ n ] of D and [µ, µ 2,, µ n ] of D Output: g(λ i ), g (λ i ) and g(µ i ) Complexity: O(n log 2 n) 4 Compute h j = u j g (λ j ) Description: For each eigenvalue λ j compute its function value h(λ j ) ie find the points (λ j, u j g (λ j )) Input: [λ, λ 2,, λ n ] Output: h j Complexity: O(n) 5 Find interpolation polynomial h(x) for the points (λ j, h j ) Description: Given the function values and input ie points (λ j, h j ) find interpolation polynomial for n points Input: Points (λ j, h j ) Output: h(x) Complexity: O(n log 2 n) 6 Compute v i = h (µ i) g(µ i) Description: Compute the ratio v i = h (µ i) g(µ i) for each µ i where, v i is value of f(µ i ) at each µ i Input: Function h(x) and g(x) and [µ, µ 2,, µ n ] 25

26 Output:f(µ i ) = v i Complexity: O(n) Appendix D Fast Multipole Method Appendix D Interpolation and Chebyshev Nodes Chebyshev nodes are the roots of the Chebyshev polynomials and they are used as points for interpolation These nodes lie in the range [, ] A polynomial of degree less than or equal to n can fit over n Chebyshev nodes For Chebyshev nodes the approximating polynomial is computed using Lagrange interpolation Expansions are used to quantify interactions among points Expansions are only computed for points which are well-separated from each other No Expansion Description 2 Φ l,i Ψ l,i Far-field expansion is computed for points withing the cluster/sub-interval i of level l Local expansion is computed for points which are well seperated from sub-interval i of level l Table D3: Expansions used in FMM Appendix D2 FMM Algorithm STEP Description: Decide the size of Chebyshev expansion ie, p = log 5 (ɛ) = log 5 ( ɛ ) where ɛ > is the precision of computation or machine accuracy parameter Input: ɛ Output: p Complexity: O() 26

27 STEP 2 Description: Set s as the number of points in the cell of finest level (s 2p) and level of finest division nlevs = log 2 ( N s ) where, N is the number of points Input: N and p Output: s and nlevs Complexity: O() STEP 3 Description: Consider p Chebyshev nodes defined over an interval [, ] of the form below Where, i =,, p ( 2i t i = cos p π ) 2 (D) STEP 4 Description: Consider Chebyshev polynomials of the form Where, j =,, p u j (t) = p k= k j t t k t j t k (D2) STEP 5 Description: Calculate the far-field expansion Φ l,i at i th subinterval of level l For i th interval of level l (nlevs) far-field expansion due to points in interval [x 0 r, x 0 + r] about center of the interval x 0 is defined by a vector of size p as Φ nlevs,i = N t i α k 3r t i (x k x 0 ) k= Input: l = nlevs, α k, x k, x 0, r, t i for i =,, 2 nlevs, Output: Φ nlevs,i Complexity: O(N p) (D3) STEP 6 [Bottom-up approach] Description: Compute the far-field expansion of individual subintervals 27

28 in terms of far-field expansion of their children These are represented by p p matrix defined as below M L (i, j) = u j ( ti 2 + t i ) M R (i, j) = u j ( ti 2 t i ) (D4) (D5) Where, i =,, p and j =,, p {u,, u p } and {t,, t p } are as defined in Eq (D2) and Eq (D) respectively Far-field expansion for i th subinterval of level l due to far-field expansion of its children is computed by shifting children s far-field expansion by M L or M R and adding those shifts as below Φ l,i = M L Φ l+,2i l + M R Φ l+,2i (D6) Input: M L, M R and Φ Output: Φ l,i Complexity: O( 2Np2 s ) STEP 7 Description: Calculate the local expansion Ψ l,i at i th subinterval of level l For i th interval of level l local expansion due to points outside the interval [y 0 r, y 0 + r] about center of the interval y 0 is defined by a vector of size p as Ψ i = Input: t i, α k, r, x k,and x 0 Output: Ψ i Complexity: O(N p) N α k rt i (x k x 0 ) k= (D7) STEP 8 [Top-down approach] Description: Compute the local expansion of individual subintervals in 28

29 terms of local expansion of their parents These are represented by p p matrix defined as below ( ti ) S L (i, j) = u j 2 ( ti + ) S R (i, j) = u j 2 (D8) (D9) Where, i =,, p and j =,, p {u,, u p } and {t,, t p } are as defined in Eq (D2) and Eq (D) respectively Compute local expansions using far-field expansion using p p matrix defined as below ( 3 ) T (i, j) = u j t i 6 ( 3 ) T 2 (i, j) = u j t i 4 ( 3 ) T 3 (i, j) = u j t i + 4 ( 3 ) T 4 (i, j) = u j t i + 6 (D0) (D) (D2) (D3) Local expansion for each subinterval at finer level is computed using local expansion of their parents For this, first the local expansion of parent interval are shifted by S L or S R and then the result is added with interactions of subinterval with other well separated subintervals (which were not considered at the parent level) Ψ l+,2i = S L Ψ l,i +T Φ l+,2i 3 +T 3 Φ l+,2i+ +T 4 Φ l+,2i+2 (D4) Ψ l+,2i = S R Ψ l,i + T Φ l+,2i 3 + T 2 Φ l+,2i 2 + T 4 Φ l+,2i+2 (D5) Input: S L, S R, T, T 2, T 3, T 4, Φ and Ψ Output: Ψ l+,2i and Ψ l+,2i Complexity: O( 8Np2 s ) 29

30 STEP 9 Description: Evaluate local expansion Ψ nlevs,i at some of {y j } (which falls in subinterval i of level nlevs) to obtain a p size vector Complexity: O(N p) STEP 0 Description: Add all the remaining interactions which are not covered by expansions Compute interactions of each {y k } in subinterval i of level nlevs with all {x j } in subinterval i, i, i + Add all these interactions with the respective local expansions Complexity: O(3N s) Total Complexity of FMM O( + + Np + (2 2Np2 s = O(2 + 3Np + 3Ns + (0Np 2 /s)) = O(2 + 3Np + 6Np + (0Np 2 /2p)) = O(2 + 9Np + (5Np)) = O(2 + 4Np) = O(N p) ( ( )) ( ) = O N log, Where p = log ɛ ɛ ) + Np + (8 2Np2 ) + Np + 3Ns) s 30