Fast Structured Spectral Methods

Spectral methods HSS structures Fast algorithms Conclusion Fast Structured Spectral Methods Yingwei Wang Department of Mathematics, Purdue University Joint work with Prof Jie Shen and Prof Jianlin Xia Workshop on Fast Direct Solvers Nov 13, 2016 Yingwei Wang Fast Structured Spectral Methods 1 / 32

Spectral methods HSS structures Fast algorithms Conclusion Talk Outline 1 Introduction to spectral methods 2 Low-rank property and structured matrices 3 Fast direct solvers and transforms 4 Concluding remarks and future work Yingwei Wang Fast Structured Spectral Methods 2 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices Outline 1 Introduction to spectral methods Finite elements vs. spectral methods Where do the dense matrices come from 2 Low-rank property and structured matrices 3 Fast direct solvers and transforms 4 Concluding remarks and future work Yingwei Wang Fast Structured Spectral Methods 3 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices Numerical PDEs Many successful numerical methods for differential equations aim at approximating the unknown function by a sum u N (x) = N u n ϕ n (x), (1) n=0 where {ϕ n (x)} N n=0 are prescribed basis functions and {u n} N n=0 are unknown coefficients to be determined. Two ways to choose polynomial basis set {φ n (x)} N n=0 : 1 Local basis sets: piecewise polynomials of low degrees (e.g., piecewise constant, linear, quadratic, etc). 2 Global basis sets: (orthogonal) polynomials of high degrees defined in the whole interval. Yingwei Wang Fast Structured Spectral Methods 4 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices 1.2 1 φ 0 (x) φ 1 (x) φ 2 (x) 0.8 0.6 0.4 0.2 0 1 0.5 0 0.5 1 1.5 φ 0 (x) 1 0.5 0 φ 2 (x) 0.5 φ 1 (x) 1 1 0.5 0 0.5 1 Figure : Basis functions: finite elements vs. spectral methods (3) Yingwei Wang Fast Structured Spectral Methods 5 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 1 0.5 0 0.5 1 1.5 1 0.5 0 0.5 1 1 0.5 0 0.5 1 Figure : Basis functions: finite elements vs. spectral methods (9) Yingwei Wang Fast Structured Spectral Methods 6 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices Spectral methods vs. finite element methods Finite element methods: Local basis: piecewise linear functions in P 1 FE. Rate of convergence: O(h) = O(N 1 ). Most widely used tools. u u h H 1 h u H 1. (2) Spectral methods: Global basis: orthogonal polynomials/trigonometric functions Significant advantage: high-order rate of convergence. u u N X N σ(m) u H m, (3) where σ(m) > 0 is the so-called order of convergence in terms of the regularity index m. Main difficulties: irregular domains, singular solutions, high dimensional problems, nonlinear equations, unbounded domains, dense linear systems, etc. Yingwei Wang Fast Structured Spectral Methods 7 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices My ongoing research topics Table : Challenges of spectral methods and our strategies Challenges Strategies Irregular domain Petrov-Galerkin Unbounded domain Mapped Chebyshev functions Corner singularity Müntz-Galerkin methods High dimensionality Sparse grids Nonlinear equations Two-level algorithm Dense matrices Fast Structured Spectral Methods Yingwei Wang Fast Structured Spectral Methods 8 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices Where do the dense matrices come from? (1) Spectral Galerkin approximation to the elliptic equation with variable coefficients leads to α(x)u (β(x) u) = f (x) (4) (M α + S β )u = f, (5) where the mass M and stiffness S matrices could be banded or totally dense. (2) The transforms between coefficients of orthogonal polynomials expansion and function values are also dense. (3) The k-th order differential matrices D (k) in spectral collocation methods are full matrices. Yingwei Wang Fast Structured Spectral Methods 9 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices Goal of this project When the order of the matrix N is large, the O(N 3 ) operations and O(N 2 ) storage required by common dense numerical linear algebra become overwhelmingly expensive. The main goal of this project is to reduce the computational cost of high order spectral methods from O(N 3 ) to nearly optimal complexity O(N log N) as other low order methods. Yingwei Wang Fast Structured Spectral Methods 10 / 32

Spectral methods HSS structures Fast algorithms Conclusion Introduction Dense matrices How to deal with dense linear systems? Iterative methods. An effective approach is to use a low-order method, such as finite-difference, finite-element, or integral operator as a preconditioner. [Canuto & Quarteroni 1985, Deville & Mund 1985, Greengard 1991, Coutsias & Hagstrom & Hesthaven & D. Torres 1996, Kim & Parter 1996, Parter 2001, Viswanath 2015]. Truncating the full matrix to a banded matrix, based on an essential assumption that the variable coefficients can be accurately approximated by low-order polynomials. [Olver & Townsend 2013]. Our strategy: low-rank property and hierarchically semiseparable (HSS) representation. Yingwei Wang Fast Structured Spectral Methods 11 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Outline 1 Introduction to spectral methods 2 Low-rank property and structured matrices Low-rank property: Galerkin, transform, collocation Hierarchically semiseparable (HSS) structures 3 Fast direct solvers and transforms 4 Concluding remarks and future work Yingwei Wang Fast Structured Spectral Methods 12 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Fourier- and Chebyshev-Galerkin methods We restrict our attention to the 1D elliptic equation: α(x)u (β(x)u x ) x = f (x), x (a, b), (6) with Dirichlet or periodical boundary conditions. The Fourier- or Chebyshev-Galerkin formulation of the above problem leads to the following linear system where Au := (M α + S β )u = f, (7) M α = F D α F, (8) S β = D k F D β F D k. (9) Yingwei Wang Fast Structured Spectral Methods 13 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Fourier- and Chebyshev-Galerkin methods We restrict our attention to the 1D elliptic equation: α(x)u (β(x)u x ) x = f (x), x (a, b), (6) with Dirichlet or periodical boundary conditions. The Fourier- or Chebyshev-Galerkin formulation of the above problem leads to the following linear system where Au := (M α + S β )u = f, (7) M α = F D α F, (8) S β = D k F D β F D k. (9) A matrix has the low-rank property if all its off-diagonal blocks have small ranks or numerical ranks. Yingwei Wang Fast Structured Spectral Methods 13 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Low-rank property for smooth variable coefficients If both α and β are constants, then 1 Fourier : A is a diagonal matrix; 2 Chebyshev : rank(a 1:n,n+1:N ) 4 and rank(a n+1:n,1:n ) 4. Theorem (Shen,W & Xia, 2016) If α(x) can be approximated by an r-term Fourier (or Chebyshev) series within a tolerance τ, then the numerical rank (with respect to the tolerance O(Nτ)) of any off-diagonal block of the following matrix is bounded by r: C F D α F (or F D α F ), where α = (α(x j )) N 1 j=0, D α = diag(α), F and F represent forward and backward disctrete Fourier (or Chebyshev) transforms. The above theorem immediately indicates the low rank property of the matrix A in (7). Yingwei Wang Fast Structured Spectral Methods 14 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Low-rank property for coefficients with steep gradients If the variable coefficient α(x) has steep gradients, then the decay of the entries away from the diagonal is very slow. Fortunately, the low-rank property still holds. 1 0.8 0.6 0.4 10 0 10 5 10 10 10 0 10 5 10 10 0.2 0 1 0.5 0 0.5 1 x (i) Plot of α 4 (x) 10 15 10 15 0 200 400 600 800 1000 1200 0 50 100 150 200 Index Index (ii) Entries of F α 4 (iii) Off-diagonal singular values 1 Figure : Plots of α 4(x) = + 1 + 1, absolute 1000(x 0.5) 2 +1 1000x 2 +1 1000(x+0.5) 2 +1 values of the Chebyshev coefficients F α 4, and the first 200 singular values of the off-diagonal block ( F D α4 F ) 1: N 2, N +1:N, where N = 1280. 2 Yingwei Wang Fast Structured Spectral Methods 15 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Low-rank property for Chebyshev-Legendre transforms Consider two expansions of f (x) P N as follows N N f (x) = fn L L n (x) = fn T T n (x), x [ 1, 1], (10) n=0 n=0 where {L n } N n=0 and {T n} N n=0 are the Legendre and Chebyshev polynomials, respectively. The connection coefficients between Legendre and Chebyshev expansion are Theorem (Shen,W & Xia, 2016) f L = K T L f T, f T = K L T f L. (11) The backward and forward Cheyshev-Legendre transform matrices K L T = (κi,j L T ) and K T L = (κ T i,j L ) are of low numerical rank. More precisely, for given tolerance ɛ, the numerical HSS rank is r l = O ( log 2 (1/ɛ) log(log 1/ɛ) log N l ). Yingwei Wang Fast Structured Spectral Methods 16 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Low-rank property for Chebyshev-Legendre transforms 50 40 log(nl) log(log(nl)) Chebyshev-Legendre 50 40 log(nl) log(log(nl)) Chebyshev-Legendre r l 30 r l 30 20 20 10 10 1 10 2 10 3 10 4 10 10 1 10 2 10 3 10 4 N N l l (i) K T L (ii) K L T Figure : Numerical HSS ranks of Chebyshev-Legendre transforms The low rank property still holds for Chebyshev-Jacobi case and Jaocbi-Jacobi case. Yingwei Wang Fast Structured Spectral Methods 17 / 32

Spectral methods HSS structures Fast algorithms Conclusion Low Rank Property Low-rank property for differential matrices r r r D 1 D 1 D 1 D 2 D 2 D 2 35 O(log(N)) 35 O(log(N)) 35 O(log(N)) 25 25 25 15 15 15 5 10 1 10 2 10 3 10 4 N 5 10 1 10 2 10 3 10 4 N 5 10 1 10 2 10 3 10 4 N (i)d (1) LGL and D(2) LGL (ii) D (1) CGL and D(2) CGL (iii) D (1) JGL and D(2) JGL Figure : The HSS ranks of Legendre, Chebyshev and Jacobi (α = 1/2, β = 2) differentiation matrices Yingwei Wang Fast Structured Spectral Methods 18 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Outline 1 Introduction to spectral methods 2 Low-rank property and structured matrices 3 Fast direct solvers and transforms HSS construction, factorization and solution Jacobi transforms with general indices 4 Concluding remarks and future work Yingwei Wang Fast Structured Spectral Methods 19 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Fast direct solver based on HSS structures Three stages of fast direct solver for Ax = f based on HSS. [Chandrasekaran et al 2005, 2006, Xia et al 2010,2012, 2013, 2016, Halko et al 2011, Martinsson 2016] 1 Construction. The storage required in HSS form is O(rN). Randomized scheme based on randomized sampling technique and matrix-vector multiplication. (Fourier- and Chebyshev Galerkin methods, Jacobi-Jacobi transforms). Analytic scheme based on explicit formula. (Differential matrices in spectral collocation methods). 2 Factorization. The factorization costs O(r 2 N) flops. 3 Solution. The solution stage costs O(rN) flops. Yingwei Wang Fast Structured Spectral Methods 20 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Fast direct solver based on HSS structures Three stages of fast direct solver for Ax = f based on HSS. [Chandrasekaran et al 2005, 2006, Xia et al 2010,2012, 2013, 2016, Halko et al 2011, Martinsson 2016] 1 Construction. The storage required in HSS form is O(rN). Randomized scheme based on randomized sampling technique and matrix-vector multiplication. (Fourier- and Chebyshev Galerkin methods, Jacobi-Jacobi transforms). Analytic scheme based on explicit formula. (Differential matrices in spectral collocation methods). 2 Factorization. The factorization costs O(r 2 N) flops. 3 Solution. The solution stage costs O(rN) flops. Applications in finite difference, finite element, boundary integral methods [Bebendorf 2003,2005, Martinsson & Rokhlin 2005, Martinsson 2009, Kong et al 2011, Schmitz & Ying 2012] and spectral methods [Lyons et al 2005, Gillman & Martinsson 2014]. Yingwei Wang Fast Structured Spectral Methods 20 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Example (Fourier-Galerkin method for problem with periodic BC) Consider the problem where α(x)u (β(x)u x ) x = f (x), x (0, π), (12) u (m) (0) = u (m) (2π), m = 0, 1, 2,..., (13) α(x) = cos(sin(x)), β(x) = 1 (tanh(γ(2x 3π)) + tanh( γ(2x π))), 2 f (x) = 1. Note that β(x) is a a square-wave -like function, which leads to two cusp points in the numerical solution. Yingwei Wang Fast Structured Spectral Methods 21 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Plot of β(x): a square-wave-like function 1 10 5 10 0 0.8 0.6 0.4 10 0 10 5 10 5 10 10 0.2 0 0 2 4 6 x (i) Plot of β(x) 10 10 10 15 10 15 0 100 200 300 400 500 600 0 50 100 150 Index (ii) Entries of F β(x) (iii) Off-diagonal singular values Figure : Plots of β(x) = 1 2 (tanh(10 (2x 3π)) + tanh(10 (2x π))), absolute values of the Fourier coefficients F β, and singular values of the off-diagonal block (F D β F ) 1: N 2, N 2 +1:N, where N = 320. Yingwei Wang Fast Structured Spectral Methods 22 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms 1.8 1.6 1.4 1.2 1 0 2 4 6 x Relative residue 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 10 9 10 10 No preconditioner Jacobi preconditioner 200 400 600 800 1000 Iteration number Flops 10 11 10 10 10 9 10 8 10 7 10 6 10 5 10 2 10 3 10 4 N (i) Numerical solution (ii) BiCGstab (iii) FSFGM Construction Factorization Solution Figure : Numerical results (γ = 10, N = 1280) Yingwei Wang Fast Structured Spectral Methods 23 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Example (Degenerate coefficients and Neumann BC) Consider the problem e x u (sin 2 (2πx)u x ) x = 1, x ( 1, 1), (14) u ( 1) = u (1) = 0, (15) 1 0.8 0.6 0.4 0.2 0 1 0.5 0 0.5 1 x (i) Plot of β(x) = sin 2 (2πx) 2.5 2 1.5 1 0.5 0 1 0.5 0 0.5 1 x (ii) Numerical solution Yingwei Wang Fast Structured Spectral Methods 24 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Numerical results Table : Backslash in Matlab Relative residue 10 1 10 0 No preconditioner Jacobi preconditioner 200 400 600 800 1000 Iteration number Figure : BiCGstab methods Flops 10 11 10 10 10 9 10 8 10 7 10 6 10 5 Construction Factorization Solution 10 2 10 3 10 4 N Figure : Our method Time (s) N Error T B T B f T B s 320 0.0968 0.0942 0.0026 3.6737e-13 640 0.5419 0.5270 0.0149 1.9300e-12 1280 5.0057 4.8662 0.1395 1.0611e-11 2560 38.7493 37.9495 0.7999 5.4067e-11 Table : Our FSCGM Time (s) N Error T F T F c T F f T F s T F it 320 0.3913 0.3471 0.0091 0.0028 0.0323 3.8080e-13 640 0.9580 0.8628 0.0215 0.0053 0.0684 2.0439e-12 1280 2.4628 2.2375 0.0636 0.0114 0.1502 9.6406e-12 2560 5.5811 5.2159 0.1480 0.0147 0.2025 3.2873e-11 5120 14.5403 13.7799 0.2965 0.0386 0.4252 3.4661e-10 Yingwei Wang Fast Structured Spectral Methods 25 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Fast structured Jacobi transforms Fast Structured Chebyshev-Jacobi transform (FSCJT): 1 Initialization. Construct HSS approximation to connection coefficients K with cost O(N 2 ) and store the generators in O(rN) memory; 2 Multiplication. Perform the matrix-vector product K v for any vector v with linear cost O(N). Table : HSS construction cost and accuracy for Chebyshev-Legendre transform K T L K L T N t c e c t c e c 160 0.080 2.2984e-13 0.005 2.4553e-13 320 0.016 1.1482e-12 0.018 3.2789e-13 640 0.067 1.3069e-12 0.060 3.6378e-13 1280 0.266 1.5614e-12 0.243 1.0479e-12 2560 1.230 2.3320e-12 1.048 1.8469e-12 5120 4.828 1.0110e-11 5.344 2.3042e-12 Yingwei Wang Fast Structured Spectral Methods 26 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Numerical tests: efficiency Flops 10 9 10 8 10 7 10 6 Direct FSCLT CLTAF O(N) O(N(logN) 2 /loglogn) Flops 10 9 10 8 10 7 10 6 Direct FSCLT CLTAF O(N) O(N(logN) 2 /loglogn) 10 5 10 4 10 1 10 2 10 3 10 4 10 5 N (i) K L T 10 5 10 4 10 1 10 2 10 3 10 4 10 5 N (ii) K T L Figure : Flops of Chebyshev-Legendre transforms FSCLT : Fast Structured Chebyshev-Legendre transform CLTAF : Cheyshev-Legendre transforms using an asymptotic formula [Hale & Townsend 2013] Yingwei Wang Fast Structured Spectral Methods 27 / 32

Spectral methods HSS structures Fast algorithms Conclusion HSS solver Jacobi Transforms Spectral collocation method: differential matrices Table : HSS construction errors of D (1) and D (2) (N = 640) r Legendre Chebyshev D (1) D (2) D (1) D (2) 10 1.5648e-06 5.4788e-10 4.3322e-07 4.0308e-07 15 2.4280e-08 7.3837e-12 5.6335e-09 9.8430e-09 20 6.4620e-10 2.5887e-13 1.3916e-10 3.5713e-10 25 1.7818e-11 8.8575e-15 3.7001e-12 1.2671e-11 30 5.0248e-13 2.9827e-16 1.0142e-13 1.9011e-12 35 1.4391e-14 9.9313e-18 2.8386e-15 1.8550e-12 40 4.1714e-16 3.2790e-19 8.0837e-17 1.8550e-12 45 1.2064e-17 1.0749e-20 2.5503e-18 1.8550e-12 50 2.7127e-18 3.4747e-22 1.4556e-18 1.8550e-12 55 2.8614e-18 5.0990e-23 1.7396e-18 1.8550e-12 60 2.9109e-18 4.3920e-23 1.7205e-18 1.8550e-12 Yingwei Wang Fast Structured Spectral Methods 28 / 32

Spectral methods HSS structures Fast algorithms Conclusion FSSM framework Outline 1 Introduction to spectral methods 2 Low-rank property and structured matrices 3 Fast direct solvers and transforms 4 Concluding remarks and future work The framework of our fast structured spectral methods Yingwei Wang Fast Structured Spectral Methods 29 / 32

Spectral methods HSS structures Fast algorithms Conclusion FSSM framework The framework of FSSM The framework of Fast Structured Spectral Methods (FSSM) contains fast structured spectral Galerkin methods, fast structured spectral collocation methods and fast structured Jacobi transforms. The ongoing work for two-dimensional elliptic equations. 1 Using HSS approximation plus eigen decomposition to solve the equation in separate form (a(x)u x ) x (b(y)u y ) y = f (x, y), (x, y) ( 1, 1) 2. (16) 2 Employing the solution of separate equation as the preconditioner to the non-separable equation (a(x, y) u(x, y)) = f (x, y), (x, y) ( 1, 1) 2. (17) J. Vogel, J. Xia, S. Cauley, and V. Balakrishnan, Superfast divide-and-conquer method and perturbation analysis for structured eigenvalue solutions, SIAM Journal on Scientific Computing, 38 (2016), pp. A1358 A1382. Yingwei Wang Fast Structured Spectral Methods 30 / 32

Spectral methods HSS structures Fast algorithms Conclusion FSSM framework Preliminary results: 2D problems Flops 10 11 10 10 10 9 10 8 10 7 10 6 Eigen solver Mat Vec O(N 2 ) 200 400 600 N (i) Separable equation Relative residuals 10 2 10 0 10 2 10 4 10 6 10 8 10 10 No preconditioner Preconditioned 10 0 10 1 10 2 10 3 Number of iterations (ii) Non-separable equation Figure : Fast structured collocation methods for 2D problems The coefficient in non-separable equation is chosen as a(x, y; δ) = tanh ((x + y)/δ) tanh ((x y)/δ) + 2, (x, y) ( 1, 1) 2, δ = 0.01. Yingwei Wang Fast Structured Spectral Methods 31 / 32

Spectral methods HSS structures Fast algorithms Conclusion FSSM framework References Jie Shen, Yingwei Wang, and Jianlin Xia, Fast Structured Direct Spectral Methods for Differential Equations with Variable Coefficients, I. The One-Dimensional Case, SIAM Journal on Scientific Computing, 38 (2016), pp. A28 A54. Jie Shen, Yingwei Wang, Jianlin Xia, Fast structured Jacobi-Jacobi transforms, submitted to Math Comp. Jie Shen, Yingwei Wang, Jianlin Xia, Fast structured spectral collocation methods, preprint. Questions? Yingwei Wang Fast Structured Spectral Methods 32 / 32