Continuous analogues of matrix factorizations NASC seminar, 9th May 2014

Transcription

1 Continuous analogues of matrix factorizations NSC seminar, 9th May 2014 lex Townsend DPhil student Mathematical Institute University of Oxford (joint work with Nick Trefethen) Many thanks to Gil Strang, MIT Work supported by supported by EPSRC grant EP/P505666/1

2 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] lex Oxford 2/24

3 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] lex Oxford 2/24

4 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] = square matrix f(x, y) chebfun2 [T & Trefethen, 13] lex Oxford 2/24

5 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] = square matrix f(x, y) chebfun2 [T & Trefethen, 13] v f(s, y)v(s) ds chebop [Driscoll, Bornemann, & Trefethen, 08] lex Oxford 2/24

6 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] = square matrix f(x, y) chebfun2 [T & Trefethen, 13] v f(s, y)v(s) ds chebop [Driscoll, Bornemann, & Trefethen, 08] SVD, QR, LU, Chol? cmatrix [T & Trefethen, 14] lex Oxford 2/24

7 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] = square matrix f(x, y) chebfun2 [T & Trefethen, 13] v f(s, y)v(s) ds chebop [Driscoll, Bornemann, & Trefethen, 08] SVD, QR, LU, Chol? cmatrix [T & Trefethen, 14] Interested in continuous analogues rather than infinite analogues lex Oxford 2/24

8 Introduction Discrete vs continuous v = column vector f(x) chebfun [Battles & Trefethen, 04] = tall skinny matrix [ f 1 (x) f n (x) ] quasimatrix [Stewart, 98] = square matrix f(x, y) chebfun2 [T & Trefethen, 13] v f(s, y)v(s) ds chebop [Driscoll, Bornemann, & Trefethen, 08] SVD, QR, LU, Chol? cmatrix [T & Trefethen, 14] Interested in continuous analogues rather than infinite analogues side: Infinite analogues are Schmidt, Wiener Hopf, infinite-dimensional QR, etc lex Oxford 2/24

9 Introduction Matrices, quasimatrices, cmatrices matrix quasimatrix cmatrix m n [a, b] n [a, b] [c, d] cmatrix is a continuous function of (y, x) [a, b] [c, d] lex Oxford 3/24

10 Introduction Matrices vs cmatrices n m n matrix: entries indexed by {1,, m} {1,, n} n [a, b] [c, d] cmatrix: entries indexed by [a, b] [c, d] {1,, m} subset of R Question Well-ordered Not well-ordered by < What is the 1st column? Successor No successor What is the next column? null set Null subsets What sparsity makes sense? Finite Infinite Convergence? lex Oxford 4/24

11 Introduction Matrices vs cmatrices n m n matrix: entries indexed by {1,, m} {1,, n} n [a, b] [c, d] cmatrix: entries indexed by [a, b] [c, d] {1,, m} subset of R Question Well-ordered Not well-ordered by < What is the 1st column? Successor No successor What is the next column? null set Null subsets What sparsity makes sense? Finite Infinite Convergence? Three heroes: lex Oxford 4/24

12 Introduction Matrices vs cmatrices n m n matrix: entries indexed by {1,, m} {1,, n} n [a, b] [c, d] cmatrix: entries indexed by [a, b] [c, d] {1,, m} subset of R Question Well-ordered Not well-ordered by < What is the 1st column? Successor No successor What is the next column? null set Null subsets What sparsity makes sense? Finite Infinite Convergence? Three heroes: Smoothness lex Oxford 4/24

13 Introduction Matrices vs cmatrices n m n matrix: entries indexed by {1,, m} {1,, n} n [a, b] [c, d] cmatrix: entries indexed by [a, b] [c, d] {1,, m} subset of R Question Well-ordered Not well-ordered by < What is the 1st column? Successor No successor What is the next column? null set Null subsets What sparsity makes sense? Finite Infinite Convergence? Three heroes: Smoothness pivoting lex Oxford 4/24

14 Introduction Matrices vs cmatrices n m n matrix: entries indexed by {1,, m} {1,, n} n [a, b] [c, d] cmatrix: entries indexed by [a, b] [c, d] {1,, m} subset of R Question Well-ordered Not well-ordered by < What is the 1st column? Successor No successor What is the next column? null set Null subsets What sparsity makes sense? Finite Infinite Convergence? Three heroes: Smoothness pivoting ɛ mach lex Oxford 4/24

15 Singular value decomposition Matrix factorization = UΣV T, Σ = diagonal, U, V = orthonormal columns U Σ V T lex Oxford 5/24

16 Singular value decomposition Matrix factorization = UΣV T, Σ = diagonal, U, V = orthonormal columns U Σ V T Exists: SVD exists and is (almost) unique lex Oxford 5/24

17 Singular value decomposition Matrix factorization = UΣV T, Σ = diagonal, U, V = orthonormal columns U Σ V T Exists: SVD exists and is (almost) unique pplication: best rank r approx is r = 1st r terms (in 2- & F-norm) lex Oxford 5/24

18 Singular value decomposition Matrix factorization = UΣV T, Σ = diagonal, U, V = orthonormal columns U Σ V T Exists: SVD exists and is (almost) unique pplication: best rank r approx is r = 1st r terms (in 2- & F-norm) Separable model: = n j=1 σ ju j v T is a sum of outer products j lex Oxford 5/24

19 Singular value decomposition Matrix factorization = UΣV T, Σ = diagonal, U, V = orthonormal columns U Σ V T Exists: SVD exists and is (almost) unique pplication: best rank r approx is r = 1st r terms (in 2- & F-norm) Separable model: = n j=1 σ ju j v T is a sum of outer products j Computation: Bidiagonalize then iterate [Golub & Kahan (1965)] lex Oxford 5/24

20 Singular value decomposition Continuous analogue = UΣV T, Σ = diagonal, U, V = orthonormal columns σ 1 v T 1 u 1 u 2 σ 2 v T 2 t least formally U Σ V T lex Oxford 6/24

21 Singular value decomposition Continuous analogue = UΣV T, Σ = diagonal, U, V = orthonormal columns σ 1 v T 1 u 1 u 2 σ 2 v T 2 t least formally U Σ V T Exists: SVD exists if is continuous and is (almost) unique [Schmidt 1907] lex Oxford 6/24

22 Singular value decomposition Continuous analogue = UΣV T, Σ = diagonal, U, V = orthonormal columns σ 1 v T 1 u 1 u 2 σ 2 v T 2 t least formally U Σ V T Exists: SVD exists if is continuous and is (almost) unique [Schmidt 1907] pplication: best rank r approx is f r = 1st r terms (L 2 -norm) [Weyl 1912] lex Oxford 6/24

23 Singular value decomposition Continuous analogue = UΣV T, Σ = diagonal, U, V = orthonormal columns σ 1 v T 1 u 1 u 2 σ 2 v T 2 t least formally U Σ V T Exists: SVD exists if is continuous and is (almost) unique [Schmidt 1907] pplication: best rank r approx is f r = 1st r terms (L 2 -norm) [Weyl 1912] Separable model: = j=1 σ ju j v T is a sum of outer products j lex Oxford 6/24

24 Singular value decomposition Continuous analogue = UΣV T, Σ = diagonal, U, V = orthonormal columns σ 1 v T 1 u 1 u 2 σ 2 v T 2 t least formally U Σ V T Exists: SVD exists if is continuous and is (almost) unique [Schmidt 1907] pplication: best rank r approx is f r = 1st r terms (L 2 -norm) [Weyl 1912] Separable model: = j=1 σ ju j v T is a sum of outer products j Computation: void bidiagonalization lex Oxford 6/24

25 Singular value decomposition bsolute and uniform convergence of the SVD Theorem Let be an [a, b] [c, d] cmatrix that is (uniformly) Lipschitz continuous in both variables Then the SVD of exists, the singular values are unique with σ j 0 as j, and = j=1 σ j u j v T j, where the series is uniformly and absolutely convergent to Proof See [Schmidt 1907], [Hammerstein 1923], and [Smithies 1937] If satisfies the assumptions of the theorem, then = UΣV T lex Oxford 7/24

26 Singular value decomposition lgorithm 1 Compute = Q R 2 Compute quasimatrix QR, R T = Q RR R (Householder triangularization of a quasimatrix [Trefethen 08]) R T = Q Q R R R R 3 Compute SVD RR = U Σ V T = (Q V)Σ(Q R U) T This is a continuous analogue of a discrete algorithm [Ipsen 90] lex Oxford 8/24

27 Singular value decomposition Related work Erhard Schmidt James Mercer Carl Eckart & Gail Young utonne, Bateman, Hammerstein, Kellogg, Picard, Smithies, Weyl izerman, Braverman, König, Rozonoer Golub, Hestenes, Kahan, Kogbetliantz, Reinsch lex Oxford 9/24

28 LU decomposition Matrix factorization = P 1 LU, P = permutation, L = unit lower-triangular, U = upper-triangular P 1 L U P 1 L = psychologically lower-triangular Exists: It (almost) exists and with extra conditions is (almost) unique pplication: Used to solve dense linear systems x = b Separable model: = n j=1 l ju T j is a sum of outer products [Pan 2000] Computation: Gaussian elimination with pivoting lex Oxford 10/24

29 LU decomposition Continuous analogue = LU, L = unit lower-triangular, U = upper-triangular u T 1 u T 2 l 1 l 2 L U Exists: It (usually) exists and with extra conditions is (almost) unique pplication: Can be used to solve integral equations Separable model: = j=1 l ju T j is a sum of outer products Computation: Continuous analogue of GECP (GE with complete pivoting) lex Oxford 11/24

30 LU decomposition Computation The standard point of view: P 1 L U lex Oxford 12/24

31 LU decomposition Computation The standard point of view: different point of view: P 1 L U (j, :)(:, k)/(j, k) (GE step for matrices) (y 0, :)(:, x 0 )/(y 0, x 0 ) (GE step for functions) Each step of GE is a rank-1 update We use complete pivoting lex Oxford 12/24

32 LU decomposition Computation The standard point of view: different point of view: P 1 L U (j, :)(:, k)/(j, k) (GE step for matrices) (y 0, :)(:, x 0 )/(y 0, x 0 ) (GE step for functions) Each step of GE is a rank-1 update We use complete pivoting Pivoting orders the columns and rows lex Oxford 12/24

33 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? lex Oxford 13/24

34 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 lex Oxford 13/24

35 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 y 2 lex Oxford 13/24

36 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 y 3 y 2 lex Oxford 13/24

37 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 y 3 y 2 y 4 lex Oxford 13/24

38 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 y 3 y 2 y 4 y 5 lex Oxford 13/24

39 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? y 1 y 3 y 2 y 4 y 5 lex Oxford 13/24

40 LU decomposition What is a triangular quasimatrix? u T 1 u T 2 l 1 l 2 L = unit lower-triangular U = upper-triangular L U What is a lower-triangular quasimatrix? Red dots = 0 s, blue squares = 1 s Position of 0 s is determined by pivoting strategy Forward substitution has a continuous analogue More precisely, L is lower-triangular wrt y 1, y 2, y 1 y 3 y 2 y 4 y 5 lex Oxford 13/24

41 LU decomposition bsolute and uniform convergence of LU Theorem Let be an [a, b] [c, d] continuous cmatrix Suppose (, x) is analytic in the stadium of radius 2ρ(b a) about [a, b] for some ρ > 1 where it is bounded in absolute value by M (uniformly in x) Then = l j u T j, j=1 where the series is uniformly and absolutely convergent to Moreover, k l j u T j Mρ k j=1 a stadium b 2ρ(b a) lex Oxford 14/24

42 LU decomposition Chebfun2 application Low rank function approximation = chebfun2(@(x,y) cos(10*(xˆ2+y))+sin(10*(x+yˆ2))); contour(, ) = pivot location (y, x ) k j =1 lex Oxford Rank = 125 Rank = 65 Rank = 5 Rank = 33 d b (y, x )dydx `j (y )uj (x ), c a Rank = 28 Rank = 2 k b j =1 a d `j (y )dy uj (x )dx c 15/24

43 LU decomposition Chebfun2 application SVD is optimal, but GE can be faster 2D Runge function: Relative error in L (y, x) = γ(x 2 + y 2 ) γ=1 γ= Rank of approximant γ=100 SVD GE Wendland s CSRBFs: s (y, x) = φ 3,s ( x y 2 ) C 2s Relative error in L φ 3,3 C 6 φ 3,1 C Rank of approximant SVD GE φ 3,0 C 0 lex Oxford 16/24

44 LU decomposition Related work Eugene Tyrtyshnikov Mario Bebendorf Keith Geddes Petros Drineas Goreinov, Oseledets, Savostyanov, Zamarashkin Gesenhues, Griebel, Hackbusch, Rjasanow Carvajal, Chapman Candes, Greengard, Mahoney, Martinsson, Rokhlin Moral of the story: Iterative GE is everywhere, under different guises Many others: Halko, Liberty, Martinsson, O Neil, Tropp, Tygert, Woolfe, etc lex Oxford 17/24

45 Cholesky factorization Matrix factorization = R T R, R = upper-triangular R T R Exists: Exists and is unique if is a positive-definite matrix pplication: numerical test for a positive-definite matrix Separable model: = n j=1 r jr T is a sum of outer products j Computation: Cholesky algorithm, ie, GECP on a positive definite matrix lex Oxford 18/24

46 Cholesky factorization Continuous analogue = R T R, R = upper-triangular quasimatrix r T 1 r 1 r 2 r T 2 t least formally Pivoting: Essential Continuous analogue of pivoted Cholesky Exists: Exists and is essentially unique for nonnegative definite functions Definition n [a, b] [a, b] continuous symmetric cmatrix is nonnegative definite if v T v = b b a a v(y)(y, x)v(x)dxdy 0, v C[a, b] lex Oxford 19/24

47 Cholesky factorization Convergence Theorem Let be an [a, b] [a, b] continuous, symmetric, and nonnegative definite cmatrix Suppose that (, x) is analytic in the closed Bernstein ellipse E 2ρ(b a) with foci a and b with ρ > 1 and bounded in absolute value by M, uniformly in y Then = j=1 r j r T j, where the series is uniformly and absolutely convergent to Moreover, k r j r T j 32Mkρ k 4ρ 1 j=1 a stadium E 2ρ(b a) b lex Oxford 20/24

48 Cholesky factorization Computation Pivoted Cholesky = GECP on nonnegative definite function Pivots in Cholesky Pivot size Step Each step is a rank 1 update: (:, x 0 )(x 0, :)/(x 0, x 0 ) 1 lways take the absolute maximum on the diagonal even if there is a tie with an off-diagonal entry lex Oxford 21/24

49 Cholesky factorization Chebfun2 application test for symmetric nonnegative definite functions = chebfun2(@(x,y) cos(10*x*y) + y + xˆ2 + sin(10*x*y)); B = * ; chol(b) 1 Inverse multiquadric ll the pivots are nonnegative and on the y = x line nonnegative definite lex Oxford 22/24

50 Demo Demo lex Oxford 23/24

51 References Z Battles & L N Trefethen, n extension of MTLB to continuous functions and operators, SISC, 25 (2004), pp T Driscoll, F Bornemann, & L N Trefethen, The chebop system for automatic solution of differential equations, BIT, 48 (2008), pp C Eckart & G Young, The approximation of one matrix by another of lower rank, Psychometrika, 1 (1936), pp N J Higham, ccuracy and Stability of Numerical lgorithms, 2nd edition, SIM, 2002 E Schmidt, Zur Theorie der linearen und nichtlinearen Integralgleichungen I Teil Entwicklung willkürlichen Funktionen nach System vorgeschriebener, Math nn, 63 (1907), pp G W Stewart, fternotes Goes to Graduate School, Philadelphia, SIM, 1998 T & L N Trefethen, Gaussian elimination as an iterative algorithm, SIM News, March 2013 T & L N Trefethen, n extension of Chebfun to two dimensions, to appear in SISC, 2013 lex Oxford 24/24