Numerical Linear Algebra

Transcription

1 Numerical Linear Algebra Lecture Notes 2014 Bärbel Janssen October 15, 2014 Department of High Performance Computing School of Computer Science and Communication Royal Institute of Technology, KTH

2

3 Contents 1 Introduction 5 2 Condition and Stability Errors and machine precision Errors Floating-point representation Arithmetic operations an cancellation Conditioning Stability Linear Algebra and Analysis Key definitions and notation Examples Linear systems Vector norms Matrix norms Orthogonal systems Non-quadratic linear systems Eigenvalues and eigenvectors Geometric and algebraic multiplicity Singular Value Decomposition Conditioning of linear algebraic systems Conditioning of eigenvalue problems Direct Solution Methods Triangular linear systems Gaussian elimination An example for why pivoting is necessary Conditioning of Gaussian elimination Symmetric matrices Cholesky decomposition Orthogonal decomposition Householder Transformation Direct determination of eigenvalues

4 Contents 5 Iterative Solution Methods Fixed point iteration and defect correction Construction of iterative methods Descent methods Gradient method Conjugate gradient method (cg method) Iterative Methods for Eigenvalue Problems Methods for the partial eigenvalue problem The Power method The inverse iteration Krylov space methods Reduction by Galerkin approximation Lanczos and Arnoldi method

5 1 Introduction These lecture notes are based on a course given at Royal Institute of Technology, KTH in autum During a prestudy week problems are solve theoretically. Even small programs are implemented in Matlab. Then a compact course follows in which each day a new topic of numerical linear algebra is considered. The course ends with a week for solving bigger projects with Matlab. 5

6

7 2 Condition and Stability Numerical methods require use of computers and therefore precision is limited. methods we use have to be analyzed in view of the finite precision. The 2.1 Errors and machine precision In approximations, we would like to decide how close we are to a (possibly) exact solution. This usually means we are interested in the difference between the approximated and the real solution Errors Definition 2.1 (Absolute error) Let x be an approximation to x. The absolute error e is defined as e = x x, x R, e = x x, x R n. The absolute error e might be big just because x or x is big. This gives rise to the following definition. Definition 2.2 (Relative error) For x 0, we define the relative error between x and its approximation x as e R = x x, x R, e R = x x x x, x Rn. Errors may have different reasons. But one of them is the already mentioned finite precision. This is connected with the computer s represenatation of numbers Floating-point representation In scientific computuations, floating-point numbers are typically used. The three digit floating-point representation of π = is Definition 2.3 (Decimal floating-point number) The fraction.314 is called the mantissa, 10 is called the base, and 1 is called the exponent. Generally, n-digit decimal floating-point numbers have the form ±.d 1 d 2 d n 10 p, 7

8 2 Condition and Stability where the digits d 1 through d n are integers between 0 and 9 and d 1 is never zero except in the case d 1 = d 2 = = d n = 0. Generally, a floating-point number can be represented using a base b as ±.d 1 d 2 d n b p, where each digit is between 0 and b 1. For the decimal system, b = 10, while b = 2 in the inary system, where the digits are 0 or 1. If a number can be written in that sense, we call it representable. Before a computation with any number x, we have to convert it into a representable number x: x = fl(x). The relation between the exact value x and its representable version fl(x) has the form where δ may be positive, negative or zero. fl(x) = x(1 + δ), Arithmetic operations an cancellation Errors in finite arithmetic do not only occur in representing numbers but also in performing arithmetic operations. The result after an operation may be not representable even if the input numbers both were representable. The result needs to be rounded or truncated to be representable. The result of a floating point operation satifies fl(a op b) = (a op b)(1 + δ) where op is one of the basic operations +,, :. Usually, properties of exact mathematic operations are not valid in arithmetic operations as fl(fl(x + y) + z) fl(x + fl(y + z)). A well know example for loss of precision occurs in the computation of the exponential of an arbitrary number x. Since the convergence radius of the exponential series is infinity which means it converges for all numbers x R the power-series expansion exp(x) = k=0 x k k! = 1 + x x ! x3 + (2.1) is a good candidate for computations. However, for negative values of x, the convergence is very bad with a high relative error. When the expansion eq. (2.1) is applied to any negative argument, the terms alternate in sign. Subtraction is required to compute the approximation. The large relative error is caused by rounding in combination with subtraction. When close floating-point numbers are subtracted, the leading digits of their mantissas are subtracted (hence the term cancellation). With exact arithmetic, these low-order digits would in general be nonzero. However, this information has been discarded before the subtraction in order to represent the numbers. Remark 2.1 Mathematically equivalent methods may differ dramatically in their sensitivity to rounding errors. 8

9 2.2 Conditioning 2.2 Conditioning When analyzing algorithms for mathematical problems we also have to know about the conditioning of the problem itself. Definition 2.4 (Condition) A problem is well conditioned if small changes in problem parameters produce small changes in the outcome. Thus to decide whether a problem is well conditioned, we make a list of the problem s parameters, we change each parameter slightly, and we observe the outcome. If the new outcome deviates from the old one just a little, the problem is well conditioned. Remark 2.2 The property to be well or ill conditioned is a property of the problem itself. It does not have to do anything with an algorithm applied to solve a certain problem. Especially, it is independent of computation and rounding errors. Example 2.1 Let us consider the problem of determining the roots of the polynomial (x 1) 4 = 0, (2.2) whose four routs are all exactly equal to 1. Now we suppose that the right-hand side is perturbed by 10 8 such that the perturbed problem of eq. (2.2) now reads and it can be equivalently written as ( (x 1) = x (x 1) 4 = 10 8, (2.3) ) ( x ) ( x 1 + i ) ( x i ), 100 where i C is the imaginary number such that i 2 = 1. That means one root has changed by 10 2 compared to the root of eq. (2.2). The change in the solution is six orders larger in comparison to the size of the perturbation. So we say that this problem is ill conditioned. 2.3 Stability Let us turn to the anlysis of algorithms. We expect them to give a correct answer, even if small errors are made. Definition 2.5 (Stability) An algorithm is stable if small changes in algorithm parameters have a small effect on the algorithm s output. Example 2.2 Consider the example of solving the equation x = e x or equivalently x = ln(x). 9

10 2 Condition and Stability The iteration procedure x k+1 = ln(x k ) (2.4) produces approximations which diverge from the root except if you choose the solution x as the starting guess x 1 = x. This algorithm is unstable since each iteration amplifies the error. An interation cobweb in fig. 2.1 shows a few iteration steps beginning with x 1 = 1 /2. As contrast to the unstable iteration in eq. (2.4) let us look at a stable iteration f(x) f(x) = x f(x 3 ) f(x 1 ) f(x 2 ) f(x 4 ) x x 3 x 1 x 2 x 4 which is given as Figure 2.1: Cobweb for the iteration eq. (2.4). x k+1 = exp( x k ). (2.5) Beginning with the starting value x 1 = 1, this algorithm produces convergent iterates to the root. The first few iterates are shown in a cobweb in fig This is an example for a stable algorithm. 10

11 2.3 Stability f(x) f(x) = x f(x 2 ) f(x 4 ) f(x 3 ) f(x 1 ) x x 2 x 4 x 5 x 3 x 1 Figure 2.2: Cobweb for the iteration eq. (2.5). 11

12

13 3 Linear Algebra and Analysis In this chapter we introduce the notation which is used throughout this lecture notes. For convinience, we recall the most important definitions and results. However, we refer to standard literature for proofs. 3.1 Key definitions and notation The elements of finite dimensional real vector spaces are denoted by x 1 x =. x n R n with x i R. If we want to refer to this vector as a row vector, we simply write x T = (x 1,..., x n ). For two vectors x, y R addition and scalar multiplication is defined as x + y := (x 1 + y 1,..., x n + y n ), αx := (αx 1,..., αx n ). Definition 3.1 (Linear independence) A set of vectors v 1,..., v k R n is called linearly independent if k c i v i = 0 c i = 0, for i = 1,..., k. i=1 The rank of A, rank(a) is the number of linearly independent columns of A. This is the ingredient for the next definition of a basis. Definition 3.2 (Basis) A set of n vectors v 1,..., v n R n is called a basis of R n if they are linearly independent. We say that the basis spans R n. Any element of R n can be uniquely written as n x = c i v i, c i R, for i = 1,..., n. i=1 Using Zorn s Lemma, one can show that each vector space possesses a basis. A special basis of R n is formed by the unit vectors {e 1,..., e n }, where δ 1i { 1 for i = j, e i =. δ ni, δ ij = The symbol δ ij is called Kronecker sybol. 0 for i j. 13

14 3 Linear Algebra and Analysis Definition 3.3 (Euclidean scalar product) The function (, ) : R n R n R defined by n (x, y) = y T x = x T y = c i d i, where x = n i=1 c iv i, y = n i=1 d iv i is called Euclidean scalar product. Two vectors x and y of R n are orthogonal if (x, y) = 0. This whole lecture deals with linear transformation. Its definition is the following: Definition 3.4 (Linearity) Let A be a transformation between two vector spaces as We call A linear if A : R n R m. i=1 A(αx + βy) = A(αx) + A(βy) = αa(x) + βa(y). holds true for every α, β R and for every x, yr n. In the next definition we turn to linear transformations and matrices. Definition 3.5 (Linear transformation and matrices) Let {v 1,..., v n } and {w 1,..., w m } be bases of R n and R m respectively. Relative to these bases, a linear transformation A : R n R m is represented by the matrix A having m rows and n columns: a 11 a a 1n a 21 a a 2n A =......, a m1 a m2... a mn the coefficients a ij of the matrix A are uniquely defined by the relations Av j = m a ij w i, j = 1,..., n. i=1 A matrix with elements a ij is written as A = (a ij ). Given a matrix A, (A) ij denotes the element in the ith row and the jth column. The transpose of A is uniquely defined by the relations (Ax, y) = ( x, A T y ) for every x R n, y R m 14

15 3.1 Key definitions and notation which imply that ( A T ) ij = a ji. It is called symmetric if A = A T. For (square) matrices, addition and scalar multiplication is defined elementwise. Let then we have A = (a ij ), B = (b ij ), α R αa + B = (αa ij + b ij ) as well as multiplications between a matrix and a vector or between two matrices as Examples Ax = n a ik x k R n, AB = k=1 n a ik b kj R n n. In order to better understand the definitions, let us look at some examples. Example 3.1 (Transpose) Given a matrix A = k=1 ( ) 1 2 3, then its transpose is A T = , 1 6 Only matrices of the same dimensions can be added as the next example shows. Example 3.2 (Addition of matrices) ( ) ( ) = ( 1 8 ) In the case of matrix vector multiplication the dimensions have to be suitable. Example 3.3 (Matrix vector multiplication) a 11 a 12 ( ) a 11 x 1 + a 12 x 2 a 21 a 22 x1 = a x 21 x 1 + a 22 x 2, a 31 a 2 32 a 31 x 1 + a 32 x 2 or a more concrete example if A = ( 3 1 ) and x = 2, 3 then Ax = ( ) = ( ) In the same manner two matrices of suitable dimensions are multiplied. 15

16 3 Linear Algebra and Analysis Example 3.4 (Matrix matrix multiplication) Let A = ( ) , B = 2 2, then AB = ( ) Linear systems One focus of this course is to solve linear systems like a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2.. a m1 x 1 + a m2 x a mn x n = b m (3.1) numerically. Direct and iterative methods will be presented and explained to numerically compute a solution. Before applying sophisticated methods, we have to make sure that a solution exists. Therfor, we recall the known results from basic Linear Algebra. We present criteria of solvability for rectangular as well as quadratic systems. The system eq. (3.1) is solvable if and only if rank(a) = rank(a, b) with the composed matrix a ( ) 11 a 1n b 1 A, b =......, a mn a mn b m In case the matrix A is quadratic, the following criteria for solvability of eq. (3.1) are equivalent: Ax = 0 implies x = 0. rank(a) = n. det(a) 0. All eigenvalues of A are nonzero. The inverse A 1 of A exists and it holds A 1 A = AA 1 = I. For the definition of an eigenvalue, see section

17 3.3 Vector norms 3.3 Vector norms Using numerical methods with limited precision and roundoff errors, we have to decide how far away from the exact solution the computed solution is. This means we have to measure distances between vectors for example. In this section we provide the necessary tools. Definition 3.6 (Norm) A mapping : R n R is a (vector) norm if it fulfills the following properties: (i) x 0 for any vector x R n, x = 0 x = 0. (ii) For any scalar α R and any vector x R n, it holds αx = α x. (iii) For any two vectors x R n and y R n, the triangle inequality holds: x + y x + y. A vector space equipped with a norm is called a normed vector space. Example 3.5 (Vector norms) A standard example for a vector norm is the euclidian norm ( n x 2 := i=1 x i 2 ) 1/2 Other examples are the maximum norm and the l 1 norm or the general l p norms for 1 p < x := max 1 i n x i, ( n ) 1/p x p := x i p. Let us compute different norms of the same vector in different vector norms. Example 3.6 Consider the vector ( ) 1 x =, then x 2 1 = 3, x 2 = 5 and x = 2. i=1 Different norms for the same vector can give different results. However, their norms can not vary arbitrarily as the following therorem shows. Theorem 3.1 (Equivalence of norms) All norms on the finite dimensional vector space R n are equivalent to the maximum norm, i. e., there exist positive constants m and M such that for each norm m x x M x, x R n. 17

18 3 Linear Algebra and Analysis In the following we state some of the most important inequalities used throuout this course. Lemma 3.1 (Hölder s inequality) For 1 < p, q < and 1 /p + 1 /q = 1 and x, y R n, the inequality ( n n ) 1/p ( n ) 1/q x i y i x i p y i q i=1 i=1 i=1 holds. In the case p = q = 2 this inequality is called Cauchy-Schwarz inequality. Lemma 3.2 (Cauchy-Schwarz inequality) ( n n ) 1/2 ( n ) 1/2 x i y i x i 2 y i 2 i=1 i=1 i=1 In other references, you can find this inequality also written like this (x, y) 2 (x, x)(y, y) = x 2 2 y 2 2, or like this x T y x 2 y 2. Lemma 3.3 (Minkowski inequality) For 1 p the inequality x + y p x p + y p, x, y R n. For scalars the following inequality is often used. Lemma 3.4 (Young inequality) For 1 < p, q < the inequality xy x p p + y q, x, y R. q 3.4 Matrix norms Definition 3.7 (Matrix norm) A mapping : R n n R is a matrix norm if it fulfills the following properties: (i) A 0 for any matrix A R n n, A = 0 A = 0. (ii) For any scalar α R and any matrix A R n n, it holds αa = α A. (iii) For any two matrices A R n n and B R n n, the triangle inequality holds: A + B A + B. 18

19 3.5 Orthogonal systems If a matrix norm in addition also satisfies it is called submultiplicative or consistent. AB A B, A matrix norm for any A R n can be created through any vector norm on R n by Ax A := sup x R n x. (3.2) x 0 From this definition, it follows directly that All matrix norm generated like this are called natural matrix norms. These norms all comply with (i) Ax A x for every x R n, which follows directly from eq. (3.2). (ii) AB A B for every A, B R n n. (iii) the norm of the identity matrix I = 1. Theorem 3.2 (Natural matrix norms) Let A = (a ij ) be a square matrix. Then Ax A 1 = sup x Rn 1 x 0 x 1 Ax A 2 = sup x Rn 2 x 0, x 2 Ax A = sup x Rn x 0 x = max 1 j n = max 1 i n n a ij, i=1 n a ij, j=1 3.5 Orthogonal systems We introduced the definition of orthogonal. Two vectors x and y R n are orthogonal if (x, y) = 0. For two subspaces N, M R n we define orthogonality as (x, y) = 0, x N, y M. Definition 3.8 (Orthogonal projection) Let M R n be a nontrivial subset. For any vector x R n the orthogonal projection P M x M is defined by x P M x 2 = min x y. y M This best approximation property is equivalent to the relation which can be used to compute P M x. (x P M x, y) = 0 y M, 19

20 3 Linear Algebra and Analysis A set of vectors {v 1,..., v m }, v i 0 of R n which are mutually orthogonal (v k, v l ) = 0 for k l is necessarily independent. Definition 3.9 (Orthogonal system) A set of vectors {v 1,..., v m }, v i 0 of R n which are mutually orthogonal (v k, v l ) = 0 for k l is called orthogonal system and in the case m = n orthogonal basis. If in addation also (v k, v k ) = 1 for k = 1,..., m the system is called orthonormal system or orthonormal basis, respectively. Often it is necessary to transform an existing system of vectors into an orthogonal or orthonormal one. The following theorem states how. Theorem 3.3 (Gram-Schmidt algorithm) Let {v 1,..., v m } be any basis of R n. Then b 1 := v1 v 1, bk := v k k (v k, b j )b j, b k := b k, k = 2,..., n, b k j=1 yields an orthonormal basis {b 1,..., b n } of R n. This algorithm is called Gram-Schmidt algorithm. Definition 3.10 (Orthonormal matrix) A matrix Q R m n is called orthogonal or orthonormal if its column vectors foram an orthogonal or orthonormal system, respectively. In the case m = n such a matrix is called unitary. Lemma 3.5 A unitary matrix Q R n n is regular and its inverse is Q 1 = Q T. Further it holds (Qx, Qy) = (x, y), x, y R n, Qx 2 = x 2, x R n. (3.3) Definition 3.11 (Range and null space) For a general matrix A R m n, we define its range and null space (or kernel) as respectively. Definition 3.12 A quadratic matrix is called normal if A T A = AA T. is called positive semi-definite if and positive definite if range(a) := {y R m : y = Ax, x R n }, kern(a) := {x R n : Ax = 0}, (Ax, x) R, (Ax, x) 0, x R n, (Ax, x) R, (Ax, x) > 0, x R n \ {0}, 20

21 3.6 Non-quadratic linear systems Remark 3.1 In view of (3.3), we remark that euclidian scalar product and euclidian vector norm are invariant under unitary transformations. Example 3.7 (Unitary matrix) The real unitary matrix Q (ij) θ = i j cos θ 0 sin θ 0 i sin θ 0 cos θ 0 j describes a rotation in the (x i, x j )-plane about the origin with angle θ [0, 2π). 3.6 Non-quadratic linear systems Let A R n m be a not necessarily quadratic matrix and b R m a right-hand side vector. In this section we consider solving the non-quadractic system Ax = b (3.4) for x R m, where m n. We need to define a new notion of a solution to the linear system eq. (3.4) as the following example shows: 1 1 A = 1 and b = There exists no vector x such that eq. (3.4) is exactly fulfilled. Instead of looking for an exact solution we seek a vector x such that Ax is as close to b as possible. In mathematical terms, this means we look for a vector x such that b Ax 2 is minimized. In the case that x is a exact solution to eq. (3.4) it also minimizes b Ax 2. Theorem 3.4 (Least squares) There exists a solution x of eq. (3.4) in the sense that A x b 2 = min x R n Ax b 2. This is equivalent to x being a solution of the normal equation A T A x = A T b. 21

22 3 Linear Algebra and Analysis 3.7 Eigenvalues and eigenvectors In the following we consider quadratic matrices A R n n. Definition 3.13 (Eigenvalue) A number λ C is called eigenvalue of the matrix A R n n if there exists a vector w 0, w R n such that Aw = λw. The vector w is called eigenvector. Eigenvalues can also be obtain as zeros of the characteristic polynomial χ A (z) := det(a zi). (3.5) I denotes the identity matrix of the same size as the matrix A. The set of all eigenvalues of a matrix A is called its spectrum and denoted by σ(a) C. Any matrix A R n n has n eigenvalues {λ 1,..., λ n } which are not necessarily distinct. Definition 3.14 (Similar matrices) Two matrices A, B R n n are called similar (to each other) if there is a regular matrix T R n n such that B = T 1 AT. Lemma 3.6 For two similar matrices A and B R n n there holds: (i) det(a) = det(b). (ii) σ(a) = σ(b). (iii) trace(a) = trace(b), where the trace of A is defined as trace(a) := n i=1 a ii. Let λ i, i = 1,..., n denote the eigenvalues to the matrix A R n n and let u i, i = 1,..., n denote the corresponding normalized eigenvectors so that Au i = λ i u i, i = 1,..., n. (3.6) Writing u i as the column vectors of the matrix U, and let Λ = diag(λ 1,..., λ n ) be a diagonal matrix with the eigenvalues on the diagonal. With these matrices, the eq. (3.6) can be written in compact form as AU = UΛ A = UΛU T (3.7) since U is orthonormal and therefor U T = U 1. The form on the right of eq. (3.7) is called the spectral decomposition of A. On the other hand, if a spectral decomposition with an orthonormal matrix U is available, the entries of the diagonal matrix Λ are the eigenvalues of A. In general not all matrices are diagonalizable in that way. If for 22

23 3.7 Eigenvalues and eigenvectors example A does not have n linearly independent eigenvectors it is not diagonalizable. However, there exists an invertible matrix P such that P 1 AP = J 1... where each submatrix J i of order n i is of the form λ i 1.. J i = λ i I, if n i = 1, J i =.... λ i 1, if n i 2. (3.8) J r λ i This is the Jordan normal form of A. decomposition. Another normal form is given by the Schur then there exists an or- Theorem 3.5 (Real Schur decomposition) If A R n n thogonal Q R n n such that R 11 R 12 R 1m Q T 0 R 22 R 2m AQ =......, 0 0 R mm where each R ii is either a 1-by-1 matrix or a 2-by-2 matrix having complex conjugate eigenvalues. For a proof, we refer to [2]. Lemma 3.7 (Spectral norm) For an arbitrary square matrix A R n n the product matrx A T A R n n is always symmetric and positive semi-definite. For the spectral norm of A there holds { } A 2 = max λ, λ σ(a T A). If A is symmetric, then there holds A 2 = max { λ, λ σ(a)}. Let be an arbitrary vector norm with a corresponding matrix norm. Then, with a normalized eigenvector u = 1 of a matrix A corresponding the eigenvalue λ, there holds λ = λ u = λu = Au A u = A, 23

24 3 Linear Algebra and Analysis e.g., all eigenvalues of A are contained in a circle in C with center at the origin and radius A. Especially with A, we obtain the eigenvalue bound n max λ A = max a ij. (3.9) λ σ(a) i=1,...,n Since the eigenvalues of A T and A are the same σ(a) = σ(a T ) and using the bound eq. (3.9) simultaneously for A T and for A, we get the refinded bound max λ min ( ( ) ) n n A, A T = min max a ij, max a ij. λ σ(a) i=1,...,n j=1,...,n Theorem 3.6 (Gerschgorin-Hadamard theorem) All eigenvalues of a matrix A R n n are contained in the union of the corresponding Gerschgorin circles { } n K j := z C : z a jj a jk, j = 1,..., n. k=1,k j Example 3.8 (Eigenvalues and eigenvectors) Let us consider the following matrices and vectors ( ) ( ) ( ) ( ) A =, B =, u =, v =, then we have Au = ( ) 3 = 3u, Bv = 3 j=1 j=1 ( ) 3 = 3v. 0 This shows that λ = 3 is an eigenvalue of both A and B, u is an eigenvector of A and v is an eigenvector of B Geometric and algebraic multiplicity The set of eigenvectors corresponding to a single eigenvalue, together with the zero vector, forms a subspace of C n. This subspace is called eigenspace. If λ is an eigenvector of A we denote the corresponding eigenspace by E λ. An eigenspace is an example of an invariant subspace of A, i.e., AE λ E λ. The dimension of E λ can be interpreted as the maximum number of linearly independent eigenvectors to the same eigenvalue λ. We call the dimension of E λ the geometric multiplicity of λ. The geometric multiplicity can also be described as the dimension of the nullspace A λi, since that nullspace is again E λ. By the fundamental theorem of algebra, we can write χ A in the form χ A (z) = (z λ 1 )(z λ 2 ) (z λ n ) for some numbers λ i C. By definition 3.13, each λ i is an eigenvalues of A, and all eigenvalues of A appear somewhere in the list. In general, an eigenvalue might appear more than once. The algebraic multiplicity of an eigenvalue λ of A is its multiplicity as a root of χ A. An eigenvalue is simple if its algebraic multiplicity is 1. i=1 24

25 3.7 Eigenvalues and eigenvectors Example 3.9 Consider the matrices A = 2, B = Both matrices A and B have the same characteristic polynomial χ A (z) = χ B (z) = (z 2) 3, so there is a single eigenvalue λ = 2 of algebraic multiplicity 3. In case of A we can choose three independent eigenvectors, for example e 1, e 2 and e 3, so the geometric multiplicity is also 3. For B, on the other hand, we can find only a single independent eigenvector (a scalar multiple of e 1 ), so the geometric multiplicity of the eigenvalue is only Singular Value Decomposition Theorem 3.7 (Singular value decomposition) Let A R m n be an arbitrary matrix. There exist orthogonal matrices U R m m and V R n n such that A = USV T, S = diag(σ 1,..., σ p ) R m n, p = min(m, n). (3.10) where σ 1 σ 2 σ p 0. Depending on whether m n or m n the matrix S has the form σ 1 σ , or σ n. 0 σ m 0 The nonnegative values {σ i } are called the singular values of A and eq. (3.10) is called the singular value decomposition (SVD) of A. From eq. (3.10) we observe that there holds Av i = σ i u i, A T u i = σ i v i, i = 1,..., min(m, n) with u i and v i the columns vectors of U and V, respectively. Moreover, we obtain A T Av i = σ 2 i v i, AA T u i = σ 2 i u i, which shows that the values σ i, i = 1, min(m, n) are the square roots of eigenvalues of the symmetric positive definite matrices A T A R n n and AA T R m m corresponding to the eigenvectors v i and u i, respectively. (i) In the case m n the matrix A T A R n n has the p = n eigenvalues {σ 2 i, i = 1,..., n}, while the matrix AA T R m m has the m eigenvalues {σ 2 1,..., σ 2 n, 0 n+1,..., 0 m }. 25

26 3 Linear Algebra and Analysis (ii) In the case m n the matrix A T A R n n has the n eigenvalues {σ 2 1,..., σ 2 m, 0 m+1,..., 0 n }, while the matrix AA T R m m has the p = m eigenvalues {σ 2 i, i = 1,..., m}. The existence of a decomposition eq. (3.10) is a consequence of the fact that A T A is orthonormally diagonalizable as Q T (A T A)Q = diag(σ 2 i ). Important consequences of the decomposition in eq. (3.10). Suppose that the singular values are ordered like then there holds (i) rank(a) = r, σ 1 σ r > σ r+1 = = σ p = 0, p = min(m, n) (ii) kern(a) = span{v r+1,..., v n }, (iii) range(a) = span{u 1,..., u r }, (iv) A = r i=1 σ iu i v it, (v) A 2 = σ 1 = σ max, (vi) A F = (σ σ 2 r) (Frobenius norm). 3.8 Conditioning of linear algebraic systems Let us consider a linear system to be solved which is given as Ax = b where A R n n and b R n. Let us further assume that this system is perturbed by small errors δa and δb, such that the actually system to be solved is à x = b with à = A + δa, b = b + δb and x = x + δx. We are interested in the impact of these perturbations on the solution. Theorem 3.8 (Perturbation theorem) Let the matrix A R n n be regular and the perturbation satisfy δa < A 1 1. Then the perturbed matrix à = A + δa is also regular and for the resulting relative error in the solution there holds ( δx x cond(a) δb 1 cond(a) δa b + δa ), (3.11) A A with the so called condition number cond(a) := A A 1 of the matrix A. 26

27 3.9 Conditioning of eigenvalue problems The condition number cond(a) depends on the chosen vector norm in the estimate eq. (3.11). Most often, the max-norm or the euclidian norm 2 are used. In the first case, there holds cond (A) := A A 1 Especially for hermitian matrices we have cond 2 (A) := A 2 A 1 2 = λ max λ min, with the eigenvalues λ max and λ min of A with largest and smallest modulus, respectively. Example 3.10 Consider the following matrix A and its inverse A 1 ( ) ( ) A =, A 1 = We derive and therefore, we obtain Hence, this matrix is very ill-conditioned. A = , A 1 = cond (A) Conditioning of eigenvalue problems The most natural way of computing eigenvalues of a matrix A R n n appears to go via its definition of zeros of the characteristic polynomial, see eq. (3.5). The computation of corresponding eigenvectors could be achieved by solving the singular system (A λi)w = 0. This approach is not advisable in general since the determination of zeros of a polynomial may be highly ill-conditioned. The computation of zeros of the Wilkinson polynomial is a famous example, see [1, chapter 3.3]. Theorem 3.9 (Stability theorem) Let A R n n be a diagonalizable matrix, and let {w 1,..., w n } be n linearly independent eigenvectors of A and let B R n n be an arbitrary second matrix. Then, for each eigenvalue λ(b) of B there is a corresponding eigenvalue λ(a) of A such that with the matrix W = (w 1,..., w n ) there holds λ(a) λ(b) cond 2 (W ) A B 2. (3.12) For symmetric matrices A R n n there exists an ONB in R n of eigenvectors so that W in eq. (3.12) can be assumed unitary, W W T = I. That leads to cond 2 (W ) = W 2 W T 2 = 1. In conclusion, the eigenvalue problem of symmetric (or more general normal) matrices is well conditioned. For general matrices, the conditioning of the eigenvalue problem may be arbitrarily bad. 27

28

29 4 Direct Solution Methods In this chapter, we study direct methods for the solution of a system of linear equations. In contrast to iterative methods which we will later turn to in chapter 5, direct methods deliver a solution in a finite number of steps if we assume an exact arithmetic. This could be seen as an advantage over iterative methods. However, in order to get useful result from a direct method it has to be carried out to the last step. Iterative methods which theroretically converge after an infinite number of steps might produce a satisfying solution in a few number of iterations. Direct methods are known to be very robust. They are well applicable to moderate sized problems but become infeasible for large systems due to their usually high storage requirements and high computational complexity. Today, moderate size systems are of dimensions up to n We will consider to solve systems of the form Ax = b (4.1) with a quadratic matrix A and vectors x and b of suitable sizes. 4.1 Triangular linear systems Systems which have a triangular form are particularly easy to solve. In the case of an upper triangular system, the matrix A has the coefficient matrix a 11 a a 1n a a 2n A =.... and the coresponding linear system looks like a nn a 11 x 1 + a 12 x a 1n x n = b 1 a 22 x a 2n x n = b a nn x n = b n For a nn 0, we obtain x n = bn a nn and plug this into the second last equation to compute x n 1. If we progress in that manner and in case a jj 0, j = 1,..., n, we derive the solution to (4.1) as ( n ) x n = b n, x j = 1 a jk x k, j = n 1, a nn a jj k=j+1 29

30 4 Direct Solution Methods The same operations to obtain a solution can be applied to a lower triangular system in opposite order. Let us take a look at the following example: Example 4.1 x 1 + x 2 + 2x 3 = 3 x 2 3x 3 = 4. 19x 3 = 19 In the first step we obtain x 3 = 1. After inserting it in the second equation we derive x 1 + x = 3 x 2 3 = 4 x 3 = 1 and find that x 2 = 1. In the last step we solve the system and get 4.2 Gaussian elimination x 1 = 2 x 2 = 1. x 3 = 1 The elimination method of Gauss transforms the system Ax = b in several elimination steps into an upper triangular form and then applies the procedure described in section 4.1. Only elimination steps that do not change the solution are allowed. These are the following: permutation of two rows of the matrix, permutation of two colums of the matrix, inlcuding according renumbering of the unkowns x i, addition of a scalar multiple of a row to another row of the matrix. The procedure is applied to the composed system ( A, b ). For the realization, we assume that the matrix A is regular. We start by setting A (0) = A, b (0) = b. Since we assumed the matrix to be regular, there exists an element a (0) r1 0, 1 r n. In the first step, we permute the first and the rth row. We call the result ( Ã (0), b (0)). Each remaining row subtracts q j1 = ã(0) j1 /ã (0) 11 times the first row. After this first step, we arrive at ( A (1), b (1)) = ã (0) 11 ã (0) 12 ã (0) 1n b(0) 1 0 a (1) 22 a (1) 2n b (1) a (1) n2 a (1) nn b (1) n, 30

31 4.2 Gaussian elimination where the new entries are given as a (1) j1 = ã(0) ji q j1 ã (0) 1i, b(0) j q j1 b(0) 1, 2 i, j n. Again, the submatrix (a (1) jk ) is regular and we can repeat the same steps which we applied to the matrix A (0). After n 1 of these elimination steps we arrive at the desired upper triangular form ã (0) 11 ã (0) 12 ã (0) 13 ã (0) 1n b(0) 1 0 ã (1) 22 ã (1) 23 ã (1) 2n b(1) 2 ( A (n 1), b (n 1)) = 0 0 ã (2) 33 ã (2) 3n a (n 1) nn b(2) 3. b (n 1) n. (4.2) For the solution of this upper triangular system we refer to section 4.1. understanding, we present the following eample: For better Example A = 2 1 3, b = The nonzero element from the first column to be determined is a (0) 11 = 3 which means that we do not have to permute any rows at the beginning. For the second and third row, we compute In the next step, we get q 21 = ã21 ã (0) 11 = 2 3, q 31 = ã31 ã (0) 11 = 1 3. ( A (1), b (1)) = 0 1 / /3, 0 2 / /3 and again no swapping of rows is required since the entry a (1) With one more elimination step, we get the upper triangualar form ( A (1), b (1)) = 0 1 / / For an example of how to finally derive the solution, have a look at example 4.1. Since all the permitted actions are linear manipulations they can be described in terms of matrices: (Ã(0), b (0)) ( = P 1 A (0), b (0)) (, A (1), b (1)) = G (Ã(0) 1, b (0)), 31

32 4 Direct Solution Methods where P 1 is a permutation matrix and G 1 is a Frobenius matrix given by 1 r P 1 = r q 21 1 G 1 =.... q n1 1 Both matrices P 1 and G 1 are regular with determinants det(p 1 ) = det(g 1 ) = 1 and there holds 1 P1 1 = P 1, G 1 q =..... q n1 1 Since there holds Ax = b A (1) x = G 1 P 1 Ax = G 1 P 1 b = b (1), the systems Ax = b and A (1) x = b (1) have the same solution. After n 1 elimination steps, we arrive at (A, b) ( A (1), b (1)) ( A (n 1), b (n 1)) =: (R, c), where ( A (i), b (i)) = G i P i ( A (i 1), b (i 1)), ( A (0), b (0)) := (A, b), with permutation matrics P i and Frobenius matrics G i of the following form 1 P i =... 1 i r 0 1 i r G i = i... 1 i q i+1,i q ni 1 32

33 4.2 Gaussian elimination The end result (R, c) = G n 1 P n 1 G 1 P 1 (A, b) is an upper triangular system which has the same solution as the original system An example for why pivoting is necessary To illustrate a potential pitfall of the Gaussian elimination, we solve the system.001x 1 + x 2 = 1, x 1 + x 2 = 2. (4.3) If we assume to use exact arithmetic, we obtain the upper triangular system which has the solution x 2 = x 1 + x 2 = 1, 999x 2 = 998, x 1 = 1 x = 1 998/ , = However, if we assume to use two-digit decimal floating-point arithmetic with rounding, we would derive the following upper triangular system.001x 1 + x 2 = 1, (1 1000)x 2 = (4.4) Evaluating the first difference, the computer produces fl(1 1000) = fl( 999) = fl( ) = For the second difference, the computer derives fl(2 1000) = fl( 998) = fl( ) = The upper triangular system that results from two-digit decimal floating-point arithmetic is.001x 1 + x 2 = 1, (4.5) 1000x 2 = By backward substitution, the solution is given as x 2 = = 1, x 1 = 1 x = 0. 33

34 4 Direct Solution Methods The error in x 1 is 100% since the computed x 1 is zero while the correct solution for x 1 is close to 1. As you observe, the upper triangular systems in eq. (4.3) and eq. (4.5) do not differ a lot from each other. The crucial step is the backward substitution where we compute 1 x 2. Here, cancellation happens due to the fact that x is close to one. A way to avoid this dilemma is changing the two equations in eq. (4.4) to x 1 + x 2 = 2,.001x 1 + x 2 = 1. Applying two-digit decimal floating-point arithmetic to this system yields the upper triangular system x 1 + x 2 = 2, x 2 = 1. With backward substitution the solution x 2 = x 1 = 1 is derived. This is in much better agreement with the exact solution. This process is called pivoting. Definition 4.1 (Pivot element) The element a r1 = ã (0) 11 in eq. (4.2) is called pivot element and the whole substep of its determination is called pivot search. For reasons of numerical stability, usually the chioce a r1 = max j=1,...,n a j1 is made. The whole process inluding permuation of rows is called column pivoting. If the elements of the matrix A are of very different size, total pivoting is advisable. This consists in the choice a rs = max k,j=1,...,n a jk and subsequent permutation of the 1st row with the rth row and the 1st column with the sth column. According to the column permutation also the unknowns x k have to be renumbered. Total pivoting is costly so that column pivoting is usually preferred. Let us recall the end result after a Gaussian elimination (with pivoting). We can use the zero entries in the lower triangular matrix to store the elements q k+1,k..., q nk which will be renamed as λ k+1,k..., λ nk after possible permuatation of rows due to pivoting: r 11 r 12 r 13 r 1n c 1 λ 21 r 22 r 23 r 2n c 2 λ 31 λ 32 r 33 r 3n c 3. (4.6) λ n1 λ n2 λ n,n 1 r nn c n 34

35 4.2 Gaussian elimination Theorem 4.1 (LR factorization) The matrices 1 0 r 11 r 12 r 1n l 21 1 L = , R = r 22 r 2n.... l n1 l n,n r nn are factors in the (multiplicative) decomposition of the matrix P A, P A = LR, P := P n 1 P 1. If there exists such a decomposition with P = I, then it is uniquely determined. Once an LR decomposition is computed, the solution of the linear system Ax = b can be achieved by successively solving two triangular systems Ly = P b, Rx = y by forward and backward substitution, respectively. We revisit the example 4.2 to illustrate the LR decomposition. Example pivoting elimination 2/3 1/ /3 pivoting /3 2/ / /3 1/ /3 premutation 1/3 2/ /3 elimination 1/3 2/3 1 10/3 1/3 2/ /3 2/3 1/ /3 2/3 1/2 1 /2 4 We obtain the solution x 3 = 8, x 2 = 3 2 and the LR decomposition ( ) x 3 = 7, x 1 = 1 3 (2 x 2 6x 3 ) = 19, P 1 = I, P 2 = 0 0 1, P A = = LR = 1/ / /3 1/ /2 35

36 4 Direct Solution Methods Conditioning of Gaussian elimination For any (regular) matrix A there exists an LR decomposition like P A = LR. We derive R = L 1 P A, R 1 = (P A) 1 L for R and its inverse R 1. Due to column pivoting, all the elements of L and L 1 are less or equal one and there holds Therefore, cond (L) = L L 1 n 2. cond (R) = R R 1 = L 1 P A (P A) 1 L L 1 P A (P A) 1 L n 2 cond (P A). Applying the perturmation theorem, theorem 3.8, we obtain the estimate for the solution of the equation LRx = P b (considering only perturbation of the right-hand side b, δa = 0) δx x cond (L) cond (R) δp b P b n4 cond (P A) δp b P b. Hence, the conditioning of the original system Ax = b by the LR decomposition is amplified by n 4 in the worst case. However, this is an extremely pessimistic estimate which can be significantly improve, for example, see [4, theorem 2.2]. 4.3 Symmetric matrices For symmetric matrices A = A T we would like that the symmetry is also reflected in the LR factorization. It should be only half the work to achive this. Let us assume that there exists a decomposition such that A = LR, where L is the a unit lower triangular matrix and R is a upper triangular matrix. From the symmerty of A it follows that LR = A = A T = (LR) T = (LD R) T = R T DL T, where 1 r 12 /r11 r 1n/r11.. R = r n 1,n/rn 1,n 1 r 11 0, D = r nn The uniqueness of the LR decomposition implies that L = R T and R = DL T such that A may be written as A = LDL T. 36

37 4.3.1 Cholesky decomposition Symmetric positive definite matrices allow for a Cholesky decomposition A = LDL T = L L T 4.3 Symmetric matrices with the matrix L := LD 1 /2. For computing the Cholesky decomposition it suffices to compute the matrices D and L. This reduces the required work to half the work for the LR decomposition. The algorithm starts from the relation A = L L T which is l 11 0 l 11 ln ln1 lnn 0 lnn = and yields equations to determine the first column of L: a 11 a nn..... a n1 a nn l2 11 = a 11, l21 l11 = a 21,..., ln1 l11 = a n1, from which l11 = a 11, for j = 2,..., n : l21 = a 21 l11 = a 21 a11. is obtained. Let now for some i {2,..., n} the elements l jk, k = 1,..., i 1, j = k,..., n be already computed. Then, via l2 i1 + l 2 i l 2 ii = a ii, lii > 0, lj1 li1 + l j2 li l ji lii = a ji, the next elements l ii and l ji, j = i + 1,..., n can be obtained as lii = lji = a ii l i1 2 l i l i,i 1 2 {a, ji l j1 li1 l j2 li2... l } j,i 1 li,i 1, l 1 ii Example 4.4 The matrix A = has the uniquely determined Cholesky decomposition A = LDL = L L T given as A = =

38 4 Direct Solution Methods 4.4 Orthogonal decomposition Let A R m n be a not necessarily quadratic matrix and b R m a right-hand side vector. We consider the system Ax = b for x R n. As discussed in section 3.6, we seek a vector x R n with minimal defect norm b Ax. A solution is obtained by solving the normal equation A T Ax = A T b. Theorem 4.2 (QR decomposition) Let A R m n be a rectangular matrix with m n and rank(a) = n. Then there exists a uniquely determined othonormal matrix Q R m n with the property Q T Q = I and a uniquely determined upper triangular matrix R R n n with diagonal r ii > 0, i = 1..., n, such that A = QR. In theorem 3.3 we introduced an algorithm which transforms a basis into an orthonormal one. However, in practical applications this algorithm is unfeasible because of its instability. Due to round-off effects the orthonormality of the columns of Q is quickly lost already after a few orthonomailzation steps. A more stable algorithm for this poouse is the Householder transformation described below Householder Transformation Definition 4.2 (Householder transformation) For any normalized vector v R m, v 2 = 1, the matrix S = I 2vv T R m m is called Householder transformation and the vector v is called Householder vector. Householder transformations are symmetric S = S T and orthonormal S T S = SS T = I. For the geometric interpretation of the Householder transformation S, let us consider an arbitrary normed vector v R 2, v 2 = 1. The two vectors { v, v } form a basis of R 2, where v T v = 0. For an arbitrary vector u = αv + βv R 2, there holds Su = ( I 2vv ) ( T αv + βv ) = αv + βv 2α(v v T )v 2β(v v T )v = αv + βv. }{{}}{{} =1 =0 Starting from a matrix A R m n the Householder algorithm generates a sequence of matrices A := A (0) A (i) A (n) = R, 38

39 4.4 Orthogonal decomposition where A (i) has the form i.... A (i) = i In the ith step the Householder transformation S i K m m is determined such that After n steps the result is S i A (i 1) = A (i). R = A (n) = S n S n 1 s q A = Q T A, where Q R m m as product of unitary matrices is also unitary and R R m m has the form r 11 r 1n.... R = n m. 0 r nn 0 0 Therefore we have the representation A = S T 1 S T n R = Q R. We obtain the desired QR decomposition of A by removing the last m n columns in Q and the last m n rows in R: R A = Q n = QR. 0 }{{} n }{{} m n m n In the following, we describe the elimination process in more detail. Step 1: As a first step of this process, we construct a Householder matrix S 1 which transforms a 1 (the first column of A) into a multiple of e 1, the first coordinate vector. This results in a vector which has just a first component. Depending on the sign of a 11 39

40 4 Direct Solution Methods we choose one of the axes span{a 1 + a 1 e 1 } or span{a 1 a 1 e 1 } in order to reduce round-off errors. In case a 11 0 we choose v 1 = a 1 + a 1 2 e 1 a 1 + a 1 2 e 1 2, v 1 = a 1 a 1 2 e 1 a 1 a 1 2 e 1 2. Then the matrix A (1) = (I 2v 1 v T 1 )A has the column vectors a (1) 1 = (I 2v 1 v T 1 )a 1 = a 1 2 e 1, a (1) k = (I 2v 1 v T 1 )a k = a k 2(a k, v 1 )v 1, k = 2,..., n. In contrast to the first step of the Gaussian elimination, the first row of the resulting matrix is changed. Step i: Let the transformed matrix A (i 1) be already computed. The Householder transformation is given as I 0 S i = I 2v i vi T =, v i = 0. i 1 m. 0 0 I 2ṽ i ṽ T }{{} i 1 The application of the (orthonormal) matrix S i to A (i 1) leaves the first i 1 rows and columns of A (i 1) unchanged. For the construction of v i, we use the considerations of step 1 for the submatrix Thus, it follows that ṽ i = Ã (i 1) = ã(i 1) ii ã (i 1) ii ã (i 1) ii ã (i 1) in..... = ã (i 1) mi ã (i 1) mn ã (i 1) ii 2 ẽ i ã (i 1) ii 2 ẽ i 2, ṽ i = ã and the matrix A (i) has the column vectors a (i) k a (i) i = a (i) k ṽ i ( ) ã (i 1) 1,..., ã (i 1) n. (i 1) ii ã (i 1) ii + ã (i 1) ii 2 ẽ i + ã (i 1) ii 2 ẽ i 2, = a (i 1) k, k = 1,..., i 1, ( T a (i 1) 1i,..., a (i 1) i 1,i, ã(i 1) i 2, 0,..., 0), ( ) = a (i 1) k 2 ã (i 1) k, ṽ i v i, k = i + 1,..., n. For a quadratic matrix A R n n the costs for a QR decomposition are about double the costs for a LR decomposition. 40

41 4.5 Direct determination of eigenvalues 4.5 Direct determination of eigenvalues Let us consider a general square matrix A R n n. The direct way of computing eigenvalues of A would be to follow the definition of what an eigenvalues is and compute the zeros of the characteristic polynomial χ A (z) = det(zi A) by a suitable method, e.g. the Newton method. We already discussed that the determination of zeros of polynomials, especially in monomial expasion can be highly ill-conditioned. In general the eigenvalues cannot be computed via the characteristic polynomial. Only in special cases where the polynomial does not need to be built up explicitly as for tri-diagonal or Hessenberg matrices. Triagonal matrix a 1 b 1 c bn 1 Hessenberg matrix a 11 a 12 a 1n a an 1,n c n a n 0 a n,n 1 a nn Reduction methods Let us recall some properties of similar matrices. Two matrices A, B C n n are called similar (A B) if there holds A = T 1 BT with a regular matrix T C n n. In view of det(a zi) = det(t 1 (B zi)t ) = det(t 1 ) det(b zi) det(t ) = det(b zi) similar matrices have the same characteristic polynomial and therefore the same eigenvalues. For any eigenvalue λ of A with a corresponding eigenvector w there holds Aw = T 1 BT w = λw, i.e., T w is an eigenvector of B corresponding to the same eigenvalue λ. Further, algebraic and geometric multiplicity of eigenvalues of similar matrices are the same. A reduction method reduces a given matrix A C n n by a sequence of similarity transformations to a simply structured matrix for which the eigenvalue problem is easier to solve. In general, the direct transformation of a given matrix into Jordan normal form (see eq. (3.8)) or the real Schur decomposition, see theorem 3.5, in finitely many steps is possible only if all its eigenvectors are a priori known. Therefore, one transforms the matrix in finitely many steps into a matrix of a simpler structure, e.g. Hessenberg form. The classical method for computing the eigenvalues of a tri-diagonal or Hessenberg matrix is Hyman s method. This method computes the characteristic polynomial of a Hessenberg matrix without explicitly determining the coefficients in its monomial expansion. With Sturm s method, one can then determine the zeros of the characteristic polynomial. For further reading on Hyman s and Sturm s method, we refer to [4]. 41