Low Rank Approximation Lecture 7. Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL
|
|
- Mervin Cooper
- 5 years ago
- Views:
Transcription
1 Low Rank Approximation Lecture 7 Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL daniel.kressner@epfl.ch 1
2 Alternating least-squares / linear scheme General setting: Solve optimization problem min f (X), X where X is a (large) matrix or tensor and f is simple (e.g., convex). Constrain X to M r, set of rank-r matrices or tensors and aim at solving min X M r f (X), Set X = i(u 1, U 2,..., U d ). (e.g., X = U 1 U2 T ). Low-rank formats are multilinear there is hope that optimizing for each component is simple: min U µ f (i(u 1, U 2,..., U d )). 2
3 Alternating least-squares / linear scheme Set f (U 1,..., U d ) := f (i(u 1,..., U d )). ALS: 1: while not converged do 2: U 1 arg min U1 f (i(u 1, U 2,..., U d )) 3: U 2 arg min U1 f (i(u 1, U 2,..., U d )) 4:... 5: U d arg min U1 f (i(u 1, U 2,..., U d )) 6: end while Examples: ALS for fitting CP decomposition Subspace iteration. Closely related: Block Gauss-Seidel, Block Coordinate Descent. Difficulties: Representation (U 1, U 2,..., U d ) often non-unique, parameters may become unbounded. M r not closed Convergence (analysis) 3
4 Subspace iteration and ALS Given A R m n, consider computation of best rank-r approximation: min f (U, V ), U R m r,v R n r f (U, V ) := A UV T 2 F Representation UV T is unique for each U, V individually if U, V have rank r. f is convex wrt U and V individually. Hence, U f (U, V ), H = f (U + H, V ) f (U, V ) + O( H 2 2) = 2 AV UV T V, H. 0 = U f (U, V ) = 2(AV UV T V ) U = AV (V T V ) 1. For stability it is advisable to choose V such that it has orthonormal columns. 4
5 Subspace iteration and ALS ALS for low-rank matrix approximation: 1: while not converged do 2: Compute economy size QR factorization: V = QR and update V Q. 3: U AV 4: Compute economy size QR factorization: U = QR and update U Q. 5: V A T U 6: end while Returns an approximation A UV T. This is the subspace iteration from Lecture 1! EFY. Develop an ALS method for solving the weighted low-rank approximation problem with square and invertible matrices W L, W R. min U,V W L (A UV T )W R F 5
6 Linear matrix equations For linear operator L : R m n R m n, consider linear system Examples: 1 Sylvester matrix equation: L(X) = C, C, X R m n. AX + XB = C, A R m m, B R n n, C, X R m n. Applications: Discretized 2D Laplace on rectangle, stability analysis, optimal control, model reduction of linear control systems. Special case Lyapunov equations: m = n, A = B T, C symmetric (and often negative semi-definite) Stochastic Galerkin methods in uncertainty quantification. Stochastic control. 1 See [V. Simoncini, Computational methods for linear matrix equations, SIAM Rev., 58 (2016), pp ] for details and references. 6
7 Linear matrix equations Using the matrix M L representing L in canonical bases, we can rewrite L(X) = B as linear system M L (vec(x)) = vec(c). Assumption: M L has low Kronecker rank: M L = B 1 A B R A R, R m, n. Equivalently, L(X) = A 1 XB T A R XB T R EFY. Develop a variant of ACA (from Lecture 3) that aims at approximating a given sparse matrix A by a matrix of low Kronecker rank for given m, n. EFY. Show that if m = n, M L is symmetric and has Kronecker rank R, one can find symmetric matrices A 1,..., A R, B 1,..., B R such that L(X) = A 1 XB A R XB R. Is it always possible to choose all A k, B k positive semi-definite if M L is positive definite? 7
8 Linear matrix equations Two ways of turning L(X) = C into optimization problem: 1. If M L is symmetric positive definite: 2. General L: min L(X), X X, B. 2 X 1 Will focus on spd M L in the following. min L(X) B 2 F X 8
9 Linear matrix equations Low-rank approximation of L(X) = B obtained by solving min f (U, V ) for f (U, V ) = 1 U,V 2 L(UV T ), UV T UV T, C. Let L have Kronecker rank R. Then L(UV T ), UV T = R R A k UV T B k, UV T = A k UV T B k V, U. k=1 k=1 This shows that arg min U f (U, V ) is solution of linear matrix equation A 1 U(V T B 1 V ) + + A R U(V T B R V ) = CV. EFY. Show that this linear matrix equation always has a unique solution under the assumption that L is symmetric positive definite. For R = 2, can be reduced to R linear systems of size n n. For R > 2, need to solve Rn Rn system. 9
10 Linear matrix equations ALS for linear matrix equations: 1: while not converged do 2: Compute economy size QR factorization: V = QR and update V Q. 3: Solve A 1 U(V T B 1 V ) + + A R U(V T B R V ) = CV for U. 4: Compute economy size QR factorization: U = QR and update U Q. 5: Solve (U T A 1 U)V T B (U T A R U)V T B R = U T C for V. 6: end while Returns an approximation X UV T. For R = 2, there are better alternatives: ADI, Krylov subspace methods,... [Simoncini 2016]. 10
11 2D eigenvalue problem u(x) + V (x)u = λu(x) in Ω = [0, 1] [0, 1] with Dirichlet b.c. and Henon-Heiles potential V Regular discretization Reshaped ground state into matrix Ground state 10 0 Singular values Excellent rank-10 approximation possible 11
12 Rayleigh quotients wrt low-rank matrices Consider symmetric n 2 n 2 matrix A. Then We now... x, Ax λ min (A) = min x 0 x, x. reshape vector x into n n matrix X; reinterpret Ax as linear operator A : X A(X). 12
13 Rayleigh quotients wrt low-rank matrices Consider symmetric n 2 n 2 matrix A. Then X, A(X) λ min (A) = min X 0 X, X with matrix inner product,. We now... restrict X to low-rank matrices. 13
14 Rayleigh quotients wrt low-rank matrices Consider symmetric n 2 n 2 matrix A. Then λ min (A) min X=UV T 0 X, A(X). X, X Approximation error governed by low-rank approximability of X. Solved by Riemannian optimization techniques or ALS. 14
15 ALS for eigenvalue problem ALS for solving X, A(X) λ min (A) min. X=UV T 0 X, X Initially: fix target rank r U R m r, V n r randomly, such that V is ONB λ λ = residual =
16 ALS for eigenvalue problem ALS for solving Fix V, optimize for U. λ min (A) min X=UV T 0 X, A(X). X, X X, A(X) = vec(uv T ) T A vec(uv T ) = vec(u) T (V I) T A(V I)vec(U) Compute smallest eigenvalue of reduced matrix (rn rn) matrix (V I) T A(V I). Note: Computation of reduced matrix benefits from Kronecker structure of A. 16
17 ALS for eigenvalue problem ALS for solving Fix V, optimize for U. λ min (A) min X=UV T 0 X, A(X). X, X λ λ = residual =
18 ALS for eigenvalue problem ALS for solving λ min (A) min X=UV T 0 Orthonormalize U, fix U, optimize for V. X, A(X). X, X X, A(X) = vec(uv T ) T A vec(uv T ) = vec(v T )(I U) T A(I U)vec(V T ) Compute smallest eigenvalue of reduced matrix (rn rn) matrix (I U) T A(I U). Note: Computation of reduced matrix benefits from Kronecker structure of A. 18
19 ALS for eigenvalue problem ALS for solving λ min (A) min X=UV T 0 Orthonormalize U, fix U, optimize for V. X, A(X). X, X λ λ = residual =
20 ALS ALS for solving λ min (A) min X=UV T 0 Orthonormalize V, fix V, optimize for U. X, A(X). X, X λ λ = residual =
21 ALS for eigenvalue problem ALS for solving λ min (A) min X=UV T 0 Orthonormalize U, fix U, optimize for V. X, A(X). X, X λ λ = residual =
22 Extension of ALS to TT Recall interface matrices X µ 1 R n 1n 2 n µ r µ 1, X µ R n µ+1n µ+2 n d r µ 1 yielding factorization X <µ> = X µ 1 X µ, T µ = 1,..., d 1. Combined with recursion X µ+1 T = Uµ R (X µ T I nµ ), this yields X <µ> = X µ 1 Uµ R X µ+1, T µ = 1,..., d 1. Hence, vec(x ) = (X µ+1 X µ 1 ) vec(u µ ) This formula allows us to pull out µth core! 22
23 Extension of ALS to TT A TT decomposition is called µ-orthogonal if and (U L ν) T U L ν = I rν, X T νx ν = I rν for ν = 1,..., µ 1. U R ν (U R ν ) T = I rν, X ν X T ν = I rµ for ν = µ + 1,..., d. This implies that X µ+1 X µ 1 has orthonormal columns! Consider eigenvalue problem Optimizing for µth core X, A(X ) λ min (A) = min X =0 X, X X, A(X ) vec U µ, A µ vec U µ min = min U µ 0 X, X U µ 0 vec U µ, vec U µ with r µ 1 n µ r µ r µ 1 n µ r µ matrix A µ = (X µ+1 X µ 1 ) T A(X µ+1 X µ 1 ) 23
24 Extension of ALS to TT U µ is obtained as eigenvector belonging to smallest eigenvalue of A µ. Computation of A µ for large d only feasible if A has low operator TT ranks (and is in operator TT decomposition). One microstep of ALS optimizes U µ and prepares for next core, by adjusting orthogonalization. One sweep of ALS consists of processing cores twice: once from left to right and once from right to left. 24
25 Extension of ALS to TT Input: X in right-orthogonal TT decomposition. 1: for µ = 1, 2,..., d 1 do 2: Compute A µ and replace core U µ by an eigenvector belonging to smallest eigenvalue of A µ. 3: Compute QR decomposition U L µ = QR. 4: Set U L µ Q. 5: Update U µ+1 R 1 U µ+1. 6: end for 7: for µ = d, d 1,..., 2 do 8: Compute A µ and replace core U µ by an eigenvector belonging to smallest eigenvalue of A µ. 9: Compute QR decomposition (U R µ ) T = QR. 10: Set U R µ Q T. 11: Update U µ 1 R 3 U µ 1. 12: end for 25
26 Extension of ALS to TT Remarks: Small matrix A µ quickly gets large as TT ranks increase Need to use iterative methods (e.g., Lanczos, LOBPCG), possibly combined with preconditioning [Kressner/Tobler 2011] for solving eigenvalue problems. In ALS TT ranks of X need to be chosen a priori. Adaptive choice of rank by merging neighbouring cores, optimizing for the merged core, and split the optimized merged core DMRG, modified ALS. Cheaper: AMEn [White 2005, Dolgov/Savostyanov 2013]. Principles of ALS easily extend to other optimization problems, e.g., linear systems [Holtz/Rohwedder/Schneider 2012]. 26
27 Numerical Experiments - Sine potential, d = 10 ALS err_lambda res nr_iter Execution time [s] Size = Maximal TT rank 40. See [Kressner/Steinlechner/Uschmajew 2014] for details. 27
28 Numerical Experiments - Henon-Heiles potential, d = 20 ALS err_lambda res nr_iter Execution time [s] Size = Maximal TT rank
29 Numerical Experiments - 1/ ξ 2 potential, d = 20 ALS err_lambda res nr_iter Execution time [s] Size = Maximal TT rank
Numerical tensor methods and their applications
Numerical tensor methods and their applications 8 May 2013 All lectures 4 lectures, 2 May, 08:00-10:00: Introduction: ideas, matrix results, history. 7 May, 08:00-10:00: Novel tensor formats (TT, HT, QTT).
More informationPreconditioned low-rank Riemannian optimization for linear systems with tensor product structure
Preconditioned low-rank Riemannian optimization for linear systems with tensor product structure Daniel Kressner Michael Steinlechner Bart Vandereycken July 29, 2015 Abstract he numerical solution of partial
More informationLecture 4. Tensor-Related Singular Value Decompositions. Charles F. Van Loan
From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 4. Tensor-Related Singular Value Decompositions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010
More informationAPPLIED NUMERICAL LINEAR ALGEBRA
APPLIED NUMERICAL LINEAR ALGEBRA James W. Demmel University of California Berkeley, California Society for Industrial and Applied Mathematics Philadelphia Contents Preface 1 Introduction 1 1.1 Basic Notation
More informationarxiv: v2 [math.na] 13 Dec 2014
Very Large-Scale Singular Value Decomposition Using Tensor Train Networks arxiv:1410.6895v2 [math.na] 13 Dec 2014 Namgil Lee a and Andrzej Cichocki a a Laboratory for Advanced Brain Signal Processing,
More informationLow Rank Approximation Lecture 3. Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL
Low Rank Approximation Lecture 3 Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL daniel.kressner@epfl.ch 1 Sampling based approximation Aim: Obtain rank-r approximation
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationExploiting off-diagonal rank structures in the solution of linear matrix equations
Stefano Massei Exploiting off-diagonal rank structures in the solution of linear matrix equations Based on joint works with D. Kressner (EPFL), M. Mazza (IPP of Munich), D. Palitta (IDCTS of Magdeburg)
More informationLecture 4. CP and KSVD Representations. Charles F. Van Loan
Structured Matrix Computations from Structured Tensors Lecture 4. CP and KSVD Representations Charles F. Van Loan Cornell University CIME-EMS Summer School June 22-26, 2015 Cetraro, Italy Structured Matrix
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationComputational methods for large-scale linear matrix equations and application to FDEs. V. Simoncini
Computational methods for large-scale linear matrix equations and application to FDEs V. Simoncini Dipartimento di Matematica, Università di Bologna valeria.simoncini@unibo.it Joint work with: Tobias Breiten,
More informationIn this section again we shall assume that the matrix A is m m, real and symmetric.
84 3. The QR algorithm without shifts See Chapter 28 of the textbook In this section again we shall assume that the matrix A is m m, real and symmetric. 3.1. Simultaneous Iterations algorithm Suppose we
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More information5. Solving the Bellman Equation
5. Solving the Bellman Equation In the next two lectures, we will look at several methods to solve Bellman s Equation (BE) for the stochastic shortest path problem: Value Iteration, Policy Iteration and
More informationDomain decomposition on different levels of the Jacobi-Davidson method
hapter 5 Domain decomposition on different levels of the Jacobi-Davidson method Abstract Most computational work of Jacobi-Davidson [46], an iterative method suitable for computing solutions of large dimensional
More informationKrylov subspace methods for linear systems with tensor product structure
Krylov subspace methods for linear systems with tensor product structure Christine Tobler Seminar for Applied Mathematics, ETH Zürich 19. August 2009 Outline 1 Introduction 2 Basic Algorithm 3 Convergence
More informationPreconditioned Locally Minimal Residual Method for Computing Interior Eigenpairs of Symmetric Operators
Preconditioned Locally Minimal Residual Method for Computing Interior Eigenpairs of Symmetric Operators Eugene Vecharynski 1 Andrew Knyazev 2 1 Department of Computer Science and Engineering University
More informationOn the convergence of higher-order orthogonality iteration and its extension
On the convergence of higher-order orthogonality iteration and its extension Yangyang Xu IMA, University of Minnesota SIAM Conference LA15, Atlanta October 27, 2015 Best low-multilinear-rank approximation
More informationLecture 18 Classical Iterative Methods
Lecture 18 Classical Iterative Methods MIT 18.335J / 6.337J Introduction to Numerical Methods Per-Olof Persson November 14, 2006 1 Iterative Methods for Linear Systems Direct methods for solving Ax = b,
More informationLarge-scale eigenvalue problems
ELE 538B: Mathematics of High-Dimensional Data Large-scale eigenvalue problems Yuxin Chen Princeton University, Fall 208 Outline Power method Lanczos algorithm Eigenvalue problems 4-2 Eigendecomposition
More informationIndex. for generalized eigenvalue problem, butterfly form, 211
Index ad hoc shifts, 165 aggressive early deflation, 205 207 algebraic multiplicity, 35 algebraic Riccati equation, 100 Arnoldi process, 372 block, 418 Hamiltonian skew symmetric, 420 implicitly restarted,
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline
More informationIterative methods for symmetric eigenvalue problems
s Iterative s for symmetric eigenvalue problems, PhD McMaster University School of Computational Engineering and Science February 11, 2008 s 1 The power and its variants Inverse power Rayleigh quotient
More information1 Non-negative Matrix Factorization (NMF)
2018-06-21 1 Non-negative Matrix Factorization NMF) In the last lecture, we considered low rank approximations to data matrices. We started with the optimal rank k approximation to A R m n via the SVD,
More informationIntroduction to Iterative Solvers of Linear Systems
Introduction to Iterative Solvers of Linear Systems SFB Training Event January 2012 Prof. Dr. Andreas Frommer Typeset by Lukas Krämer, Simon-Wolfgang Mages and Rudolf Rödl 1 Classes of Matrices and their
More informationLecture 3: Inexact inverse iteration with preconditioning
Lecture 3: Department of Mathematical Sciences CLAPDE, Durham, July 2008 Joint work with M. Freitag (Bath), and M. Robbé & M. Sadkane (Brest) 1 Introduction 2 Preconditioned GMRES for Inverse Power Method
More informationLecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm
CS 622 Data-Sparse Matrix Computations September 19, 217 Lecture 9: Krylov Subspace Methods Lecturer: Anil Damle Scribes: David Eriksson, Marc Aurele Gilles, Ariah Klages-Mundt, Sophia Novitzky 1 Introduction
More informationMATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors.
MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v
More informationNotes on singular value decomposition for Math 54. Recall that if A is a symmetric n n matrix, then A has real eigenvalues A = P DP 1 A = P DP T.
Notes on singular value decomposition for Math 54 Recall that if A is a symmetric n n matrix, then A has real eigenvalues λ 1,, λ n (possibly repeated), and R n has an orthonormal basis v 1,, v n, where
More informationSolving Symmetric Indefinite Systems with Symmetric Positive Definite Preconditioners
Solving Symmetric Indefinite Systems with Symmetric Positive Definite Preconditioners Eugene Vecharynski 1 Andrew Knyazev 2 1 Department of Computer Science and Engineering University of Minnesota 2 Department
More informationMaths for Signals and Systems Linear Algebra in Engineering
Maths for Signals and Systems Linear Algebra in Engineering Lecture 18, Friday 18 th November 2016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE LONDON Mathematics
More informationQR-decomposition. The QR-decomposition of an n k matrix A, k n, is an n n unitary matrix Q and an n k upper triangular matrix R for which A = QR
QR-decomposition The QR-decomposition of an n k matrix A, k n, is an n n unitary matrix Q and an n k upper triangular matrix R for which In Matlab A = QR [Q,R]=qr(A); Note. The QR-decomposition is unique
More informationIntroduction to Arnoldi method
Introduction to Arnoldi method SF2524 - Matrix Computations for Large-scale Systems KTH Royal Institute of Technology (Elias Jarlebring) 2014-11-07 KTH Royal Institute of Technology (Elias Jarlebring)Introduction
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationInexact Inverse Iteration for Symmetric Matrices
Inexact Inverse Iteration for Symmetric Matrices Jörg Berns-Müller Ivan G. Graham Alastair Spence Abstract In this paper we analyse inexact inverse iteration for the real symmetric eigenvalue problem Av
More informationarxiv: v1 [hep-lat] 2 May 2012
A CG Method for Multiple Right Hand Sides and Multiple Shifts in Lattice QCD Calculations arxiv:1205.0359v1 [hep-lat] 2 May 2012 Fachbereich C, Mathematik und Naturwissenschaften, Bergische Universität
More informationMatrix Equations and and Bivariate Function Approximation
Matrix Equations and and Bivariate Function Approximation D. Kressner Joint work with L. Grasedyck, C. Tobler, N. Truhar. ETH Zurich, Seminar for Applied Mathematics Manchester, 17.06.2009 Sylvester matrix
More informationMATHEMATICS. Course Syllabus. Section A: Linear Algebra. Subject Code: MA. Course Structure. Ordinary Differential Equations
MATHEMATICS Subject Code: MA Course Structure Sections/Units Section A Section B Section C Linear Algebra Complex Analysis Real Analysis Topics Section D Section E Section F Section G Section H Section
More informationS N. hochdimensionaler Lyapunov- und Sylvestergleichungen. Peter Benner. Mathematik in Industrie und Technik Fakultät für Mathematik TU Chemnitz
Ansätze zur numerischen Lösung hochdimensionaler Lyapunov- und Sylvestergleichungen Peter Benner Mathematik in Industrie und Technik Fakultät für Mathematik TU Chemnitz S N SIMULATION www.tu-chemnitz.de/~benner
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationBlock Bidiagonal Decomposition and Least Squares Problems
Block Bidiagonal Decomposition and Least Squares Problems Åke Björck Department of Mathematics Linköping University Perspectives in Numerical Analysis, Helsinki, May 27 29, 2008 Outline Bidiagonal Decomposition
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 3: Iterative Methods PD
More informationEIGIFP: A MATLAB Program for Solving Large Symmetric Generalized Eigenvalue Problems
EIGIFP: A MATLAB Program for Solving Large Symmetric Generalized Eigenvalue Problems JAMES H. MONEY and QIANG YE UNIVERSITY OF KENTUCKY eigifp is a MATLAB program for computing a few extreme eigenvalues
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationThe rational Krylov subspace for parameter dependent systems. V. Simoncini
The rational Krylov subspace for parameter dependent systems V. Simoncini Dipartimento di Matematica, Università di Bologna valeria.simoncini@unibo.it 1 Given the continuous-time system Motivation. Model
More informationIterative Methods for Sparse Linear Systems
Iterative Methods for Sparse Linear Systems Luca Bergamaschi e-mail: berga@dmsa.unipd.it - http://www.dmsa.unipd.it/ berga Department of Mathematical Methods and Models for Scientific Applications University
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationPreconditioned GMRES Revisited
Preconditioned GMRES Revisited Roland Herzog Kirk Soodhalter UBC (visiting) RICAM Linz Preconditioning Conference 2017 Vancouver August 01, 2017 Preconditioned GMRES Revisited Vancouver 1 / 32 Table of
More informationMath 671: Tensor Train decomposition methods II
Math 671: Tensor Train decomposition methods II Eduardo Corona 1 1 University of Michigan at Ann Arbor December 13, 2016 Table of Contents 1 What we ve talked about so far: 2 The Tensor Train decomposition
More informationNUMERICAL METHODS WITH TENSOR REPRESENTATIONS OF DATA
NUMERICAL METHODS WITH TENSOR REPRESENTATIONS OF DATA Institute of Numerical Mathematics of Russian Academy of Sciences eugene.tyrtyshnikov@gmail.com 2 June 2012 COLLABORATION MOSCOW: I.Oseledets, D.Savostyanov
More informationNumerical Methods I: Eigenvalues and eigenvectors
1/25 Numerical Methods I: Eigenvalues and eigenvectors Georg Stadler Courant Institute, NYU stadler@cims.nyu.edu November 2, 2017 Overview 2/25 Conditioning Eigenvalues and eigenvectors How hard are they
More informationSimple iteration procedure
Simple iteration procedure Solve Known approximate solution Preconditionning: Jacobi Gauss-Seidel Lower triangle residue use of pre-conditionner correction residue use of pre-conditionner Convergence Spectral
More informationLecture Note 7: Iterative methods for solving linear systems. Xiaoqun Zhang Shanghai Jiao Tong University
Lecture Note 7: Iterative methods for solving linear systems Xiaoqun Zhang Shanghai Jiao Tong University Last updated: December 24, 2014 1.1 Review on linear algebra Norms of vectors and matrices vector
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationThe amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.
AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative
More informationSummary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method
Summary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method Leslie Foster 11-5-2012 We will discuss the FOM (full orthogonalization method), CG,
More informationLecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University
Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Rayleigh Quotient Iteration Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Solving Eigenvalue Problems All
More information18.06SC Final Exam Solutions
18.06SC Final Exam Solutions 1 (4+7=11 pts.) Suppose A is 3 by 4, and Ax = 0 has exactly 2 special solutions: 1 2 x 1 = 1 and x 2 = 1 1 0 0 1 (a) Remembering that A is 3 by 4, find its row reduced echelon
More informationFEM and sparse linear system solving
FEM & sparse linear system solving, Lecture 9, Nov 19, 2017 1/36 Lecture 9, Nov 17, 2017: Krylov space methods http://people.inf.ethz.ch/arbenz/fem17 Peter Arbenz Computer Science Department, ETH Zürich
More informationStructure preserving Krylov-subspace methods for Lyapunov equations
Structure preserving Krylov-subspace methods for Lyapunov equations Matthias Bollhöfer, André Eppler Institute Computational Mathematics TU Braunschweig MoRePas Workshop, Münster September 17, 2009 System
More informationNumerical Methods for Solving Large Scale Eigenvalue Problems
Peter Arbenz Computer Science Department, ETH Zürich E-mail: arbenz@inf.ethz.ch arge scale eigenvalue problems, Lecture 2, February 28, 2018 1/46 Numerical Methods for Solving Large Scale Eigenvalue Problems
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 19: More on Arnoldi Iteration; Lanczos Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 17 Outline 1
More informationMATH 423 Linear Algebra II Lecture 10: Inverse matrix. Change of coordinates.
MATH 423 Linear Algebra II Lecture 10: Inverse matrix. Change of coordinates. Let V be a vector space and α = [v 1,...,v n ] be an ordered basis for V. Theorem 1 The coordinate mapping C : V F n given
More informationOrthogonal tensor decomposition
Orthogonal tensor decomposition Daniel Hsu Columbia University Largely based on 2012 arxiv report Tensor decompositions for learning latent variable models, with Anandkumar, Ge, Kakade, and Telgarsky.
More informationScientific Computing with Case Studies SIAM Press, Lecture Notes for Unit VII Sparse Matrix
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 1: Direct Methods Dianne P. O Leary c 2008
More informationSolvability of Linear Matrix Equations in a Symmetric Matrix Variable
Solvability of Linear Matrix Equations in a Symmetric Matrix Variable Maurcio C. de Oliveira J. William Helton Abstract We study the solvability of generalized linear matrix equations of the Lyapunov type
More informationFinal Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson
Final Exam, Linear Algebra, Fall, 2003, W. Stephen Wilson Name: TA Name and section: NO CALCULATORS, SHOW ALL WORK, NO OTHER PAPERS ON DESK. There is very little actual work to be done on this exam if
More informationMathematical Optimisation, Chpt 2: Linear Equations and inequalities
Mathematical Optimisation, Chpt 2: Linear Equations and inequalities Peter J.C. Dickinson p.j.c.dickinson@utwente.nl http://dickinson.website version: 12/02/18 Monday 5th February 2018 Peter J.C. Dickinson
More information1 Inner Product and Orthogonality
CSCI 4/Fall 6/Vora/GWU/Orthogonality and Norms Inner Product and Orthogonality Definition : The inner product of two vectors x and y, x x x =.., y =. x n y y... y n is denoted x, y : Note that n x, y =
More informationPreconditioned inverse iteration and shift-invert Arnoldi method
Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationReal Eigenvalue Extraction and the Distance to Uncontrollability
Real Eigenvalue Extraction and the Distance to Uncontrollability Emre Mengi Computer Science Department Courant Institute of Mathematical Sciences New York University mengi@cs.nyu.edu May 22nd, 2006 Emre
More informationInexact inverse iteration for symmetric matrices
Linear Algebra and its Applications 46 (2006) 389 43 www.elsevier.com/locate/laa Inexact inverse iteration for symmetric matrices Jörg Berns-Müller a, Ivan G. Graham b, Alastair Spence b, a Fachbereich
More informationStabilization and Acceleration of Algebraic Multigrid Method
Stabilization and Acceleration of Algebraic Multigrid Method Recursive Projection Algorithm A. Jemcov J.P. Maruszewski Fluent Inc. October 24, 2006 Outline 1 Need for Algorithm Stabilization and Acceleration
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationLecture 4 Orthonormal vectors and QR factorization
Orthonormal vectors and QR factorization 4 1 Lecture 4 Orthonormal vectors and QR factorization EE263 Autumn 2004 orthonormal vectors Gram-Schmidt procedure, QR factorization orthogonal decomposition induced
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Power iteration Notes for 2016-10-17 In most introductory linear algebra classes, one computes eigenvalues as roots of a characteristic polynomial. For most problems, this is a bad idea: the roots of
More informationMotivation: Sparse matrices and numerical PDE's
Lecture 20: Numerical Linear Algebra #4 Iterative methods and Eigenproblems Outline 1) Motivation: beyond LU for Ax=b A little PDE's and sparse matrices A) Temperature Equation B) Poisson Equation 2) Splitting
More informationM.A. Botchev. September 5, 2014
Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev
More informationBoundary Value Problems - Solving 3-D Finite-Difference problems Jacob White
Introduction to Simulation - Lecture 2 Boundary Value Problems - Solving 3-D Finite-Difference problems Jacob White Thanks to Deepak Ramaswamy, Michal Rewienski, and Karen Veroy Outline Reminder about
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More informationUniversity of Maryland Department of Computer Science TR-5009 University of Maryland Institute for Advanced Computer Studies TR April 2012
University of Maryland Department of Computer Science TR-5009 University of Maryland Institute for Advanced Computer Studies TR-202-07 April 202 LYAPUNOV INVERSE ITERATION FOR COMPUTING A FEW RIGHTMOST
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationDELFT UNIVERSITY OF TECHNOLOGY
DELFT UNIVERSITY OF TECHNOLOGY REPORT 16-02 The Induced Dimension Reduction method applied to convection-diffusion-reaction problems R. Astudillo and M. B. van Gijzen ISSN 1389-6520 Reports of the Delft
More informationCLASSICAL ITERATIVE METHODS
CLASSICAL ITERATIVE METHODS LONG CHEN In this notes we discuss classic iterative methods on solving the linear operator equation (1) Au = f, posed on a finite dimensional Hilbert space V = R N equipped
More informationOn Lagrange multipliers of trust-region subproblems
On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008
More informationOrder reduction numerical methods for the algebraic Riccati equation. V. Simoncini
Order reduction numerical methods for the algebraic Riccati equation V. Simoncini Dipartimento di Matematica Alma Mater Studiorum - Università di Bologna valeria.simoncini@unibo.it 1 The problem Find X
More informationFast iterative solvers for fractional differential equations
Fast iterative solvers for fractional differential equations Tobias Breiten a, Valeria Simoncini b, Martin Stoll c a Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstr 36,
More informationMatrix Algorithms. Volume II: Eigensystems. G. W. Stewart H1HJ1L. University of Maryland College Park, Maryland
Matrix Algorithms Volume II: Eigensystems G. W. Stewart University of Maryland College Park, Maryland H1HJ1L Society for Industrial and Applied Mathematics Philadelphia CONTENTS Algorithms Preface xv xvii
More informationApproximate Low Rank Solution of Generalized Lyapunov Matrix Equations via Proper Orthogonal Decomposition
Applied Mathematical Sciences, Vol. 4, 2010, no. 1, 21-30 Approximate Low Rank Solution of Generalized Lyapunov Matrix Equations via Proper Orthogonal Decomposition Amer Kaabi Department of Basic Science
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationLinear Algebra. Brigitte Bidégaray-Fesquet. MSIAM, September Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble.
Brigitte Bidégaray-Fesquet Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble MSIAM, 23 24 September 215 Overview 1 Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning
More informationON THE GLOBAL KRYLOV SUBSPACE METHODS FOR SOLVING GENERAL COUPLED MATRIX EQUATIONS
ON THE GLOBAL KRYLOV SUBSPACE METHODS FOR SOLVING GENERAL COUPLED MATRIX EQUATIONS Fatemeh Panjeh Ali Beik and Davod Khojasteh Salkuyeh, Department of Mathematics, Vali-e-Asr University of Rafsanjan, Rafsanjan,
More informationENGG5781 Matrix Analysis and Computations Lecture 9: Kronecker Product
ENGG5781 Matrix Analysis and Computations Lecture 9: Kronecker Product Wing-Kin (Ken) Ma 2017 2018 Term 2 Department of Electronic Engineering The Chinese University of Hong Kong Kronecker product and
More information