Lecture Notes for Inf-Mat 3350/4350, Tom Lyche

Transcription

1 Lecture Notes for Inf-Mat 3350/4350, 2007 Tom Lyche August 5, 2007

2 2

3 Contents Preface vii I A Review of Linear Algebra 1 1 Introduction Notation Vectors Vector Spaces Linear Independence and Bases Operations on Subspaces Sums and intersections of subspaces The quotient space Norms Convergence of Vectors Convergence of series of vectors Inner Products Orthogonality Projections and Orthogonal Complements Matrices Arithmetic Operations and Block Multiplication The Transpose Matrix Linear Systems The Inverse matrix Rank, Nullity, and the Fundamental Subspaces Linear Transformations and Matrices Orthonormal and Unitary Matrices Determinants Permutations Basic Properties of Determinants The Adjoint Matrix and Cofactor Expansion i

4 ii Contents 4.4 Computing Determinants Some Useful Determinant Formulas Gaussian Elimination Gaussian Elimination and LU-factorization Algoritms Operation count Pivoting Permutation matrices Gaussian elimination works mathematically Pivot strategies The PLU-Factorization An Algorithm for Finding the PLU-Factorization Eigenvalues and Eigenvectors The Characteristic Polynomial The characteristic equation Similarity Transformations Linear Independence of Eigenvectors Left Eigenvectors II Some Linear Systems with a Special Structure 73 7 Examples of Tridiagonal Systems Cubic Spline Interpolation Splines Cubic spline interpolation LU-factorization of a tridiagonal system A Two-Point Boundary Value Problem Numerical example An Eigenvalue Problem The buckling of a beam The eigenpairs of the matrix tridiag( 1, 2, 1) LU Factorization and Positive Definite Matrices Algebraic Properties of Triangular Matrices The LU and LDL T -Factorizations Positive Definite Matrices Definitions and examples When is a matrix positive definite? Algorithms Some Bivariate Problems and the Kronecker Product The Poisson Problem Block LU-factorization of a block-triangular matrix 106

5 Contents iii 9.2 The Kronecker Product Fast Direct Solution of a Large Linear System A Fast Poisson Solver based on Diagonalization A Fast Poisson Solver based on the Discrete Sine and Fourier Transforms The discrete sine transform (DST) The discrete Fourier transform (DFT) The fast Fourier transform (FFT) A Poisson solver based on the FFT Problems III Some Matrix Theory Orthonormal Eigenpairs and the Schur Form The Schur Form Hermitian and Normal Matrices The Spectral Theorem The Rayleigh Quotient and Minmax Theorems The Rayleigh Quotient Minmax Theorems The Hoffman-Wielandt Theorem Proof of the Real Schur Form The Singular Value Decomposition Singular Values and Singular Vectors Singular values The singular value decomposition Three forms of the SVD The SVD of A H Singular vectors The SVD of A H A and AA H Singular values of normal and positive definite matrices A geometric interpretation The Pseudo-inverse and Orthogonal Projections The pseudo-inverse Orthogonal projections The Minmax Theorem for Singular Values and the Hoffman- Wielandt Theorem Matrix Norms Matrix Norms Consistent and Subordinate Matrix Norms Operator norms The p-norms

6 iv Contents Unitary invariant matrix norms Absolute and monotone norms The Condition Number with Respect to Inversion Determining the Rank of a Matrix IV Iterative Methods for Linear Systems The Classical Iterative Methods Convergence and Spectral Radius Convergence in R m,n and C m,n The spectral radius Classical Iterative Methods; Component form Classical Iterative Methods; Matrix Form Convergence Convergence of SOR Numerical Example The Conjugate Gradient Method The Conjugate Gradient Algorithm Numerical Example Derivation and Basic Properties Convergence Minimization and Preconditioning Minimization Preconditioning Preconditioning Example A banded matrix Preconditioning V Orthonormal Transformations and Least Squares Orthonormal Transformations The QR decomposition and QR factorization QR and Gram-Schmidt The Householder Transformation Householder Triangulation Givens rotations Least Squares Examples Numerical Solution using the Normal Equations Numerical Solution using the QR-factorization QR and linear systems Numerical Solution using the Singular Value Factorization

7 Contents v 18.5 Perturbation Theory for Least Squares Perturbing the right hand side Perturbing the matrix Perturbation Theory for Singular Values VI Eigenvalues and Eigenvectors Numerical Eigenvalue Problems Perturbation of Eigenvalues Gerschgorin s theorem Unitary Similarity Transformation of a Matrix into Upper Hessenberg or Tridiagonal Form Computing a Selected Eigenvalue of a Symmetric Matrix The inertia theorem Approximating λ m Perturbation Proofs The Power and QR methods The Power method The inverse power method The QR-Algorithm The relation to the power method A convergence theorem The shifted QR-algorithms VII Appendix 259 A Computer Arithmetic 261 A.1 Absolute and Relative Errors A.2 Floating Point Numbers A.3 Rounding and Arithmetic Operations A.3.1 Rounding A.3.2 Arithmetic operations A.4 Backward Rounding-Error Analysis A.4.1 Computing a sum A.4.2 Computing an inner product A.4.3 Computing a matrix product B Differentiation of Vector Functions 271 C Some Inequalities 275 C.1 Convexity C.2 Inequalities D The Jordan Form 279

8 vi Contents D.1 The Jordan Form D.1.1 The minimal polynomial Bibliography 285 Index 287

9 Preface These lecture notes are the result of an ongoing project to write a text for a course in matrix analysis and numerical linear algebra given at the advanced undergraduate and beginning graduate level at the University of Oslo. In the first 6 chapters we give a quick review of basis linear algebra. Some of the results in this first part are not found in most introductory linear algebra courses. In particular we discuss algorithmic aspects of Gaussian elimination. The lectures can start with chapter 7 and then each of the remaining chapters 8-20 should correspond to one week of lectures. Oslo, 05 August, 2007 Tom Lyche vii

10 viii Preface

11 List of Algorithms 5.5 lufactor forwardsolve backsolve PLU-factorization Forward Substitution (column oriented) Backward Substitution (column oriented) locatesubinterval trifactor trisolve bandcholesky bandforwardsolve bandbacksolve Fast Poisson Solver Recursive FFT Stationary Iterative Methods Conjugate Gradient Iteration Testing Conjugate Gradient Preconditioned Conjugate Gradient Algorithm Generate a Householder transformation Upper Hessenberg linear system Solving least squares by Householder QR Householder reduction to Hessenberg form Assemble Householder transformations The Power Method Rayleigh quotient iteration ix

12 x List of Algorithms

13 List of Exercises xi

14 xii List of Exercises

15 List of Exercises xiii

16 xiv List of Exercises

17 List of Exercises xv A A A B D D D D D D D

18 xvi List of Exercises

19 Part I A Review of Linear Algebra 1

20

21 Chapter 1 Introduction 1.1 Notation The following sets will be used throughout these notes. 1. The set of natural numbers, integers, rational numbers, real numbers, and complex numbers are denoted by N, Z, Q, R, C, respectively. 2. R n is the set of n-tuples of real numbers which we will represent as column vectors. Thus x R n means x 1 x 2 x =.., where x i R for i = 1,..., n. Row vectors are normally identified using the transpose operation. Thus if x R n then x is a column vector and x T is a row vector. 3. R m,n is the set of m n matrices with real entries represented as a 11 a 12 a 1n a 21 a 22 a 2n A =.... a m1 a m2 a mn The entry in the ith row and jth column of a matrix A will be denoted by a i,j, a ij, A(i, j) or (A) i,j. We use the notations a 1j a 2j a.j =., a T at 2. i. = [a i1, a i2,..., a in ], A = [a.1, a.2,... a.n ] =.. a mj a T m. 3 x n a T 1.

22 4 Chapter 1. Introduction for the columns a.j and rows a T i. of A. We often drop the dots and write a j and a T i when no confusion can arise. If m = 1 then A is a row vector, if n = 1 then A is a column vector, while if m = n then A is a square matrix. In this text we will denote matrices by boldface capital letters A, B, C and vectors most often by boldface lower case letters x, y, z,. 4. The imaginary unit 1 is denoted by i. The complex conjugate and the modulus of a complex number z is denoted by z and z, respectively. Thus if z = x + iy = re iφ = r(cos φ + i sin φ) is a complex number then z = x iy = re iφ = cos φ i sin φ and z = x 2 + y 2 = r. Re(z) := x and Im(z) := y denote the real and imaginary part of the complex number z. 5. For matrices and vectors with complex entries we use the notation A C m,n and x C n. We identify complex row vectors using either the transpose T or the Hermitian (conjugate) transpose operation x H := x T = [x 1,..., x n ]. 6. The unit vectors in R n and C n are denoted by e 1 = 0, e 2 = 0, e 3 = 1,, e n = 0, while I n = I =: [δ ij ] n i,j=1, where δ ij :=. 0 { 1 if i = j, 0 otherwise, is the identity matrix of order n. Both the columns and the transpose of the rows of I are the unit vectors e 1, e 2,..., e n. 7. We use the following notations for diagonal- and tridiagonal n n matrices d d 2 0 diag(d i ) = diag(d 1,..., d n ) := = d d n d 1 c 1 a 2 d 2 c 2 B = tridiag(a i, d i, c i ) = tridiag(a, d, c) := d n, a n 1 d n 1 c n 1 a n d n Here b ii = d i for i = 1,..., n, b i+1,i = a i+1, b i,i+1 = c i for i = 1,..., n 1, and b ij = 0 otherwise. 8. We use the colon equal symbol v := e to indicate that the symbol v is defined by the expression e..

23 Chapter 2 Vectors This chapter contains a review of vector space concepts that will be useful in this text. we start by introducing a vector space. To define a vector space we need a field F, a set of vectors V, a way to combine vectors called vector addition, and a way to combine elements of F and V called scalar multiplication. In the first part of this section F will be an arbitrary field, but later the field will be the set of real or complex numbers with the usual arithmetic operations. 2.1 Vector Spaces Definition 2.1 A field is a set F together with two operations +, : F F F such that for all a, b, c F the following arithmetic rules hold (A0) there exists an element 0 F such that a + 0 = a. (Am) there exists an element ( a) F such that a + ( a) = 0. We define subtraction as a b := a + ( b). (Aa) a + (b + c) = (a + b) + c. (Ac) a + b = b + a. (M1) there exists an element 1 F such that a 1 = a. (Mi) if a 0 then there exists an element a 1 F such that a a 1 = 1. (Ma) a (b c) = (a b) c. (Mc) a b = b a. (D) a (b + c) = a b + a c. The requirements (A0), (Am), (Aa) are the axioms for a group. They state that (F, +) is a group, and since in addition (Ac) holds then (F, +) is by defintion an 5

24 6 Chapter 2. Vectors abelian group. The axioms (M1), (Mi), (Ma), (Mc) state that (F\{0}, ) is an abelian group. Often we drop the dot and write ab for the product a b. Examples of fields are R or C with ordinary addition and multiplication. Definition 2.2 A vector space over a field F is a set V together with two operations vector addition, + : V V V and scalar multiplication, : F V V such that for all a, b F and v, w V the following hold (V) (V, +) is an abelian group. (Va) (a b) v = a (b v). (Vd1) (a + b) v = a v + b v. (Vd2) a (v + w) = a v + a w. (M1) 1 v = v. We denote a vector space by (V, F) or by V if the underlying field is clear from the context. Definition 2.3 Let (V, F) be a vector space and S a nonempty subset of V. Then (S, F) is a subspace of (V, F) if (S, F) is itself a vector space. It follows that (S, F) is a subspace of (V, F) if S is closed under vector addition and scalar multiplication, i.e. as 1 + bs 2 S for all a, b F and all s 1, s 2 S. For any vector space (V, F) the two sets {0}, consisting only of the zero element in V, and V itself are subspaces. They are called the trivial subspaces. Here are some examples of vector spaces. Example 2.4 (The Vector Spaces R n and C n ) In the following chapters we will deal almost exclusively with the vector spaces R n = (R n, R), C n = (C n, C) and their subspaces. Addition and scalar multiplication are defined by v + w = v 1 + w 1.. v n + w n, av = av 1... av n Example 2.5 (Subspaces of R 2 and R 3 ) For a given vector x R n let S = {tx : t R}. Then S is a subspace of R n, in fact it represents a straight line passing through the origin. For n = 2 it can be shown that all nontrivial subspaces of R 2 are of this form. For n = 3 the nontrivial subspaces are all lines and all planes containing {0}.

25 2.1. Vector Spaces 7 Example 2.6 (The Vector Space C(I)) Let F = R and let C(I) be the set of all real valued functions f : I R which are defined and continuous on an interval I R. Here the vectors are functions in C(I). Vector addition and scalar multiplication are defined for all f, g C(I) and all a R by (f + g)(x) := f(x) + g(x), (af)(x) := af(x), for all x I. C(I) = ( C(I), R ) is a vector space since the sum of two continuous functions is continuous, a constant times a continuous function is continuous vector addition and scalar multiplication are defined point-wise, so the axioms for a vector space follows from properties of real numbers. Example 2.7 (The Vector Space Π n ) Let Π n (I) be the set of all polynomials of degree at most n defined on a subset I R or I C. We write simply Π n if I = R or I = C. With pointwise addition and scalar multiplication defined as in Example 2.6 the set (Π n (I), R) is a subspace of (C(I), R). Definition 2.8 (Linear Combinations) The sum c 1 v 1 + c 2 v c n v n with c i F and v i V for i = 1,..., n is called a linear combination of v 1,..., v n. We say that the linear combination is nontrivial if at least one of the c i s is nonzero. The set span{v 1,..., v n } := {c 1 v c n v n : c i F, i = 1,..., n} spanned by v 1,..., v n V is a subspace of (V, F). A vector space V is called finite dimensional if it has a finite spanning set; i.e. there exist n N and {v 1,..., v n } in V such that V = span{v 1,..., v n }. Exercise 2.9 Show that the 0 of vector addition is unique and that {0} is a subspace. Exercise 2.10 Show that 0 x = 0 for any x V. Exercise 2.11 Show that span{v 1,..., v n } is a subspace. Exercise 2.12 Show that span{v 1,..., v n } is the smallest subspace containing the vectors v 1,..., v n.

26 8 Chapter 2. Vectors 2.2 Linear Independence and Bases Definition 2.13 Let X := {v 1,..., v n } be a set of vectors in a vector space (V, F). We say that X is linearly dependent if we can find a nontrivial linear combination which is equal to zero. We say that X is linearly independent if it is not linearly dependent. In other words c 1 v c n v n = 0 for some c 1,..., c n F = c 1 = = c n = 0. The elements in a set of linearly independent vectors must all be nonzero and we have Lemma 2.14 Suppose v 1,..., v n span a vector space V and that w 1,..., w k are linearly independent vectors in V. Then k n. Proof. Suppose k > n. Write w 1 as a linear combination of elements from the set X 0 := {v 1,..., v n }, say w 1 = c 1 v 1 + +c n v n. Since w 1 0 not all the c s are equal to zero. Pick a nonzero c, say c i1. Then v i1 can be expressed as a linear combination of w 1 and the remaining v s. So the set X 1 := {w 1, v 1,..., v i1 1, v i1+1,..., v n } must also be a spanning set for V. We repeat this for w 2 and X 1. In the linear combination w 2 = d i1 w 1 + j i 1 d j v j, we must have d i2 0 for some i 2. Moreover i 2 i 1 for otherwise w 2 = d 1 w 1 contradicting the linear independence of the w s. So the set X 2 consisting of the v s with v i1 replaced by w 1 and v i2 replaced by w 2 is again a spanning set for V. Repeating this process n 2 more times we obtain a spanning set X n where all the v s have been replaced by w 1,..., w n. Since k > n we can then write w k as a linear combination of w 1,..., w n contradicting the linear independence of the w s. We conclude that k n. Definition 2.15 A finite set of vectors {v 1,..., v n } in a vector space (V, F) is a basis for (V, F) if 1. span{v 1,..., v n } = V. 2. {v 1,..., v n } is linearly independent. Theorem 2.16 Suppose (V, F) is a vector space and that S := {v 1,..., v n } is a spanning set for V. Then we can find a subset {v i1,..., v ik } of S that forms a basis for V. Proof. If {v 1,..., v n } is linearly dependent we can express one of the v s as a nontrivial linear combination of the remaining v s and drop that v from the spanning set. Continue this process until the remaining v s are linearly independent. They still span the vector space and therefore form a basis. Corollary 2.17 A vector space is finite dimensional if and only if it has a basis.

27 2.2. Linear Independence and Bases 9 Proof. Let V = span{v 1,..., v n } be a finite dimensional vector space. By Theorem 2.16 V has a basis. Conversely, if V = span{v 1,..., v n } and {v 1,..., v n } is a basis then it is by defintion a finite spanning set. Theorem 2.18 Every basis for a vector space V has the same number of elements. This number is called the dimension of the vector space and denoted dim V. Proof. Suppose X = {v 1,..., v n } and Y = {w 1,..., w k } are two bases for V. By Lemma 2.14 we have k n. Using the same Lemma with X and Y switched we obtain n k. We conclude that n = k. The set of unit vectors {e 1,..., e n } form a basis for both R n and C n. The dimension of the trivial subspace {0} is defined to be zero. Theorem 2.19 Every linearly independent set of vectors {v 1,..., v k } in a finite dimensional vector space V can be enlarged to a basis for V. Proof. If {v 1,..., v k } does not span V we can enlarge the set by one vector v k+1 which cannot be expressed as a linear combination of {v 1,..., v k }. The enlarged set is also linearly independent. Continue this process. Since the space is finite dimensional it must stop after a finite number of steps. It is convenient to introduce a matrix transforming a basis in a subspace into a basis for the space itself. Lemma 2.20 Suppose S is a subspace of a finite dimensional vector space (V, F) and let {s 1,..., s n } be a basis for S and {v 1,..., v m } a basis for V. Then each s j can be expressed as a linear combination of v 1,..., v m, say s j = m a ij v i for j = 1,..., n. (2.1) i=1 If x S then x = n j=1 c js j = m i=1 b iv i for some coefficients b := [b 1,..., b m ] T, c := [c 1,..., c n ] T. Moreover b = Ac, where A = [a ij ] C m,n. The matrix A has linearly independent columns. Proof. (2.1) holds since s j V and {v 1,..., v n } spans V. Since {s 1,..., s n } is a basis for S and {v 1,..., v m } a basis for V every x S can be written x = n j=1 c js j = m i=1 b iv i for some scalars (c j ) and (b i ). But then x = n j=1 (2.1) c j s j = n ( m ) m ( n ) m c j a ij v i = a ij c j vi = b i v i. j=1 i=1 Since {v 1,..., v m } is linearly independent it follows that b i = n j=1 a ijc j for i = 1,..., m or b = Ac. Finally, to show that A has linearly independent columns i=1 j=1 i=1

28 10 Chapter 2. Vectors suppose b := Ac = 0 for some c = [c 1,..., c n ] T. Define x := n j=1 c js j. Then x = m i=1 b iv i and since b = 0 we have x = 0. But since {s 1,..., s n } is linearly independent we have c = 0. The matrix A in Lemma 2.20 is called a change of basis matrix. Exercise 2.21 Show that the elements in a linearly independent set must be nonzero. Exercise 2.22 Show that the set of unit vectors {e 1,..., e n } form a basis both for R n and for C n. Why does this show that the dimension of R n and C n is n? 2.3 Operations on Subspaces Let R and S be two subsets of a vector space (V, F) and let a be a scalar. The sum, multiplication by scalar, union, and intersection of R and S are defined by R + S := {r + s : r R and s S}, (2.2) as := {as : s S}, (2.3) R S := {x : x R or x S}. (2.4) R S := {x : x R and x S}. (2.5) Exercise 2.23 Let R = {(x, y) : x 2 + y 2 1} be the unit disc in R 2 and set S = {(x, y) : (x 1 2 )2 + y 2 1}. Find R + S, 2S, R S,, and R S Sums and intersections of subspaces In many cases R and S will be subspaces. Then as = S and both the sum and intersection of two subspaces is a subspace of (V, F). Note however that the union R S of two subspaces is not necessarily a subspace. Exercise 2.24 Let R and S be two subspaces of a vector space (V, F). Show that as = S and that both R + S and R S are subspaces of (V, F). Example 2.25 For given vectors x, y R n with x and y linearly independent let R = span{x} and S = span{y}. Then R and S are subspaces of R n. For n = 2 we have R + S = R 2, while for n = 3 the sum represents a plane passing through the origin. We also see that R S = {0} and that R S is not a subspace. Exercise 2.26 Show the statements made in Example Theorem 2.27 Let R and S be two subspaces of a vector space (V, F). Then dim(r + S) = dim(r) + dim(s) dim(r S). (2.6)

29 2.3. Operations on Subspaces 11 Proof. Let {u 1,..., u p } be a basis for R S, where {u 1,..., u p } =, the empty set, in the case R S = {0}. We use Theorem 2.19 to extend {u 1,..., u p } to a basis {u 1,..., u p, r 1,..., r q } for R and a basis {u 1,..., u p, s 1,..., s t } for S. Every x R + S can be written as a linear combination of {u 1,..., u p, r 1,..., r q, s 1,..., s t } so these vectors span R + S. We show that they are linearly independent and hence a basis. Suppose u + r + s = 0, where u := p j=1 α ju j, r := q j=1 ρ jr j, and s := t j=1 σ js j. Now r = (u + s) belongs to both R and to S and hence r R S. Therefore r can be written as a linear combination of u 1,..., u p say r := p j=1 β ju j and at the same time as a linear combination of r 1,..., r q. But then 0 = p j=1 β ju j q j=1 ρ jr j and since {u 1,..., u p, r 1,..., r q } is linearly independent we must have β 1 = = β p = ρ 1 = = ρ q = 0 and hence r = 0. We now have u + s = 0 and by linear independence of {u 1,..., u p, s 1,..., s t } we obtain α 1 = = α p = σ 1 = = σ t = 0. We have shown that the vectors {u 1,..., u p, r 1,..., r q, s 1,..., s t } constitute a basis for R + S. The result now follows from a simple calculation dim(r + S) = p + q + t = (p + q) + (p + t) p = dim(r) + dim(s) dim(r S). From this theorem it follows that dim(r + S) = dim(r) + dim(s) provided R S = {0}. Definition 2.28 (Direct Sum) Let R and S be two subspaces of a vector space (V, F). If R S = {0} then the subspace R + S is called a direct sum and denoted R S. The subspaces R and S are called complementary in the subspace R S. Theorem 2.29 Let R and S be two subspaces of a vector space (V, F) and assume R S = {0}. Every x R S can be decomposed uniquely in the form x = r + s, where r R and s S. If {r 1,..., r k } is a basis for R and {s 1,..., s n } is a basis for S then {r 1,..., r k, s 1,..., s n } is a basis for R S. Proof. To show uniqueness, suppose we could write x = r 1 + s 1 = r 2 + s 2 for r 1, r 2 R and s 1, s 2 S. Then r 1 r 2 = s 2 s 1 and it follows that r 1 r 2 and s 2 s 1 belong to both R and S and hence to R S. But then r 1 r 2 = s 2 s 1 = 0 so r 1 = r 2 and s 2 = s 1. Thus uniqueness follows. Suppose {r 1,..., r k } is a basis for R and {s 1,..., s n } is a basis for S. Since dim(r + S) = dim(r) + dim(s) the vectors {r 1,..., r k, s 1,..., s n } span R + S. To show linear independence suppose k j=1 ρ jr j + n j=1 σ js j = 0. The first sum belongs to R and the second to S and the sum is a decomposition of 0. By uniqueness of the decomposition both sums must be zero. But then ρ 1 = = ρ k = σ 1 = = σ n = 0 and linear independence follows.

30 12 Chapter 2. Vectors The quotient space For the sum of two sets we write x + S := {x + s : s S} when one of the sets is a singleton set {x}. Suppose S is a subspace of a vector space (X, F). Since as = S we have The set a(x + S) + b(y + S) = (ax + by) + S, for all a, b F and all x, y S. is a vector space if we define X /S := {x + S : x X } (2.7) a(x + S) + b(y + S) := (ax + by) + S, for all a, b F and all x, y S. The space X /S is called the quotient space of X by S. The zero element in X /S is S itself. Moreover, if x + S = y + S then x y S. Exercise 2.30 Show that X /S is a vector space. Theorem 2.31 Suppose S is a subspace of a finite dimensional vector space (X, F). Then dim(s) + dim ( X /S ) = dim(x ). (2.8) Proof. Let n := dim(x ), k = dim(s), and let {s 1,..., s k } be a basis for S. By Theorem 2.19 we can extend it to a basis {s 1,..., s k, t k+1,..., t n } for X. The result will follow if we can show that {t k+1 + S,..., t n + S} is a basis for X /S. Recall that the zero element in X /S is S. To show linear independence suppose n j=k+1 a j(t j + S) = S for some a k+1,..., a n in F. Since n j=k+1 a js = S and the zero element in X /S is unique we must have n j=k+1 a jt j = 0 which implies that a k+1 = = a n = 0 by linear independence of the t s. It remains to show that span{t k+1 + S,..., t n + S} = X /S. Suppose x + S X /S. For some a 1,..., a n we have x = x 1 + x 2, where x 1 = k j=1 a js j and x 2 = n j=k+1 a jt j. Since x 1 + S = S we have x + S = x 2 + S = n j=k+1 a jt j + S = n j=k+1 a j(t j + S) X /S. 2.4 Norms To measure the size of a vector in a vector space (V, F) we use norms. Definition 2.32 (Norm) A norm in a vector space (V, F), where F = R or F = C, is a function : V R that satisfies for all x, y in V and all a in F 1. x 0 with equality if and only if x = 0. (positivity) 2. ax = a x. (homogeneity) 3. x + y x + y. (subadditivity)

31 2.4. Norms 13 The triple (V, F, ) is said to be a normed vector space and the inequality 3. is called the triangle inequality. Since x = x y + y x y + y we obtain x y x y. By symmetry x y = y x y x and we obtain the inverse triangle inequality x y x y, x, y V. (2.9) Consider now some specific vector norms. For the vector spaces R n and C n we define for p 1 the p-norms by The most important cases are: x p := ( n x j p) 1/p, (2.10) j=1 1. x 1 = n j=1 x j, (the one-norm or l 1 -norm) x := max 1 j n x j. (2.11) 2. x 2 = ( n j=1 x j 2) 1/2, the two-norm, l2 -norm, or Euclidian norm) 3. x = max 1 j n x j, (the infinity-norm, l -norm, or max norm.) The infinity norm is related to the other p-norms by lim x p = x for all x C n. (2.12) p This clearly holds for x = 0. For x 0 we write x p := x ( n j=1 ( x j x ) p ) 1/p. Now each term in the sum is not greater than one and at least one term is equal to one and we obtain x x p n 1/p x, p 1. (2.13) Since lim p n 1/p = 1 for any n N we see that (2.12) follows. It can be shown (cf. Appendix C) that the p-norm are norms in R n and in C n for any p with 1 p. The triangle inequality x + y p x p + y p is called Minkowski s inequality. To prove it one first establishes Hölder s inequality n x j y j x p y q, j=1 1 p + 1 = 1. (2.14) q The relation 1 p + 1 q = 1 means that if p = 1 then q = and if p = 2 then q = 2. (2.13) shows that the infinity norm and any other p-norm can be bounded in terms of each other. We define

32 14 Chapter 2. Vectors Definition 2.33 Two norms and in a finite dimensional vector space (V, F) are equivalent if there are positive constants m and M (depending only on the dimension of V) such that for all vectors x V we have The following result is proved in Appendix C. m x x M x. (2.15) Theorem 2.34 All vector norms in a finite dimensional vector space are equivalent. The inverse triangle inequality (2.9) shows that a norm is a continuous function V R. Exercise 2.35 Show that p is a vector norm in R n for p = 1, p =. Exercise 2.36 The set S p = {x R n : x p = 1} is called the unit sphere in R n with respect to p. Draw S p for p = 1, 2, for n = 2. Exercise 2.37 Let 1 p. Produce a vector x l such that x l = x l p and another vector x u such that x u p = n 1/p x u p. Thus the inequalities in (2.12) are sharp. Exercise 2.38 If 1 q p then x p x q n 1/q 1/p x p, x C n. Hint: For the rightmost inequality use Jensen s inequality Cf. Theorem C.2 with f(z) = z p/q and z i = x i q. For the left inequality consider first y i = x i / x, i = 1, 2,..., n. 2.5 Convergence of Vectors Consider an infinite sequence {x k } = x 0, x 1, x 2,... of vectors in R n. This sequence converges to zero if and only if each component sequence x k (j) converges to zero for j = 1,..., n. In terms of the natural basis we have x k = n j=1 x k(j)e k and another way of stating convergence to zero is that in terms of the basis {e 1,..., e n } for R n each coefficient x k (j) of x k converges to zero. Consider now a more general vector space. Definition 2.39 Let {v 1,..., v n } be a basis for a finite dimensional vector space (V, F), where F = R or F = C, and let {x k } be an infinite sequence of vectors in V with basis coefficients {c k }, i.e. x k = n j=1 c kjv j for each k. We say that {x k } converges to zero, or have the limit zero, if lim k c kj = 0 for j = 1,..., n. We say that {x k } converge to the limit x in V if x k x converges to zero. We write this as lim k x k = x or x k x (as k ).

33 2.5. Convergence of Vectors 15 This definition is actually independent of the basis chosen. If {w 1,..., w n } is another basis for V and x k = n j=1 b kjw j for each k then from Lemma 2.20 b k = Ac k for some nonsingular matrix A independent of k. Hence c k 0 if and only if b k 0. If {a k } and {b k } are sequences of scalars and {x k } and {y k } are sequences of vectors such that {a k } a, {b k } b, {x k } x, and {y k } y then {a k x k + b k y k } ax + by. This shows that scalar multiplication and vector addition are continuous functions with respect to this notion of limit. Corresponding to a basis {v 1,..., v n }, we define x c := max 1 j n c j where x = n c j v j. We leave as an exercise to show that this is a norm on V. Recall that any two norms on V are equivalent. This implies that for any other norm on V there are positive constants α, β such that any x = n j=1 c jv j satisfy x α max 1 j n c j and c j β x for j = 1,..., n. (2.16) Suppose now (V, F, ) is a normed vector space with F = R or F = C. The notion of limit can then be stated in terms of convergence in norm. Theorem 2.40 In a normed vector space we have x k x if and only if lim k x k x = 0. Proof. Suppose {v 1,..., v n } is a basis for the vector space and assume x k, x V. Then x k x = n j=1 c kjv j for some scalars c kj By (2.16) we see that j=1 1 β max k,j c kj x k x α max k,j c kj and hence x k x 0 lim k c kj 0 for each j x k x. Since all vector norms are equivalent we have convergence in any norm we can define on a finite dimensional vector space. Definition 2.41 Let (V, F, ) be a normed vector space and let {x k } in V be an infinite sequence. 1. {x k } is a Cauchy sequence if lim k,l (x k x l ) = 0 or equivalently lim k,l x k x l = 0. More precisely, for each ɛ > 0 there is an integer N N such that for each k, l N we have x k x l ɛ. 2. The normed vector space is said to be complete if every Cauchy sequence converges to a point in the space. 3. {x k } is called bounded if there is a positive number M such that x k M for all k.

34 16 Chapter 2. Vectors 4. {x nk } is said to be a subsequence of {x k } k 0 if 0 n 0 < n 1 < n 2. Theorem 2.42 In a finite dimensional vector space V the following hold: 1. A sequence in V is convergent if and only if it is a Cauchy sequence. 2. V is complete. 3. Every bounded sequence in V has a convergent subsequence. Proof. 1. Suppose x k x. By the triangle inequality x k x l x k x + x l x and hence x k x l 0. Conversely, let {v 1,..., v n } be a basis for V and {x k } a Cauchy sequence with x k = n j=1 c kjv j for each k. Then x k x l = n j=1 (c kj c lj )v j and since lim k,l (x k x l ) = 0 we have by definition of convergence lim k,l (c kj c lj ) = 0 for j = 1,..., n. Thus for each j we have a Cauchy-sequence {c kj } C and since C is complete {c kj } converges to some c j C. But then x k x := n j=1 c jv j V. 2. V is complete since we just showed that every Cauchy sequence converges to a point in the space. 3. Let {v 1,..., v n } be a basis for V and {x k } be a bounded sequence with x k = n j=1 c kjv j for each k. By (2.16) each coefficient sequence {c kj } k is a bounded sequence of complex numbers and therefore, by a well known property of complex numbers, has a convergent subsequence. In particular the sequence of v 1 coefficients {c k1 } has a convergent subsequence c ki,1. For the second component the sequence {c ki,2} has a convergent subsequence, say c li,2. Continuing with j = 3,..., n we obtain integers 0 m 0 < m 1 < such that {c mi,j} is a convergent subsequence of c kj for j = 1,..., n. But then {x mi } is a convergent subsequence of {x k } Convergence of series of vectors Consider now an infinite series m=0 y m of vectors in a vector space (V, F) with F = R or F = C. We say that the series converges if the sequence of partial sums {x k } given by x k = k m=0 y m converges. A sufficient condition for convergence is that m=0 y m converges for some vector norm. We say that the series converges absolutely if this is the case. Note that m=0 y m m=0 y m, and absolute convergence in one norm implies absolute convergence in any norm by Theorem In an absolute convergent series we may change the order of the terms without changing the value of the sum. Problems for Section 2.5 Exercise 2.43 Show that if {a k } a, {b k } b, {x k } x, and {y k } y then {a k x k + b k y k } ax + by. Exercise 2.44 Show that c is a norm.

35 2.6. Inner Products Inner Products An inner product or scalar product in a vector space (V, F), where F = R or F = C, is a function, mapping pairs of vectors into a scalar. We consider first the case where F = R. Definition 2.45 An inner product in a vector space (V, R) is a function V V R satisfying for all x, y, z V and all a, b R the following conditions: 1. x, x 0 with equality if and only if x = 0. (positivity) 2. x, y = y, x (symmetry) 3. ax + by, z = a x, z + b y, z. (linearity) The triple (V, R,,, ) is called a real inner product space The standard inner product in V = R n is given by x, y := x T y. It is clearly an inner product in R n. When the field of scalars is C the inner product is complex valued and properties 2. and 3. are altered as follows: Definition 2.46 An inner product in a vector space (V, C) is a function V V C satisfying for all x, y, z V and all a, b C the following conditions: 1. x, x 0 with equality if and only if x = 0. (positivity) 2. x, y = y, x (skew symmetry) 3. ax + by, z = a x, z + b y, z. (linearity) The triple (V, C,,, ) is called a complex inner product space Note the complex conjugate in 2. and that (Cf. Exercise 2.52) x, ay + bz = a x, y + b x, z. (2.17) The standard inner product in C n is given by x, y := x H y = n j=1 x jy j. It is clearly an inner product in C n. Suppose now (V, F,,, ) is an inner product space with F = R or F = C. We define the inner product norm by x := x, x, x V. For any vectors x, y V and scalar a F we have (Cf. Exercises 2.51 and 2.52) by linearity and symmetry the expansion x + ay 2 = x 2 + 2a x, y + a 2 y 2 (real case), (2.18) = x 2 + 2Re x, ay + a 2 y 2 (complex case), (2.19) where Re z and Im z denotes the real- and imaginary part of the complex number z.

36 18 Chapter 2. Vectors In the complex case we can write the inner product of two vectors as a sum of inner product norms. For any x, y V it follows from (2.19) that 4 x, y = x + y 2 x y 2 + i x iy 2 i x + iy 2, (2.20) where i = 1 and we used that Im(z) = Re( iz) for any z C. To show that the inner product norm is a norm in (V, R) we need the triangle inequality. To show it we start with a famous inequality. Theorem 2.47 (Cauchy-Schwarz inequality) For any x, y in a real or complex inner product space x, y x y with equality if and only if x and y are linearly dependent. Proof. The inequality is trivial if x, y = 0 so assume x, y = 0. Suppose first x, y R. We define the scalar a := x,y y, and use (2.18) to obtain 0 2 x + ay 2 = x 2 ( x, y ) 2 / y 2. Thus the inequality follows in the real case. Suppose next x, y is complex valued, say x, y = re iφ. We define b := e iφ and observe that b x, y = r is real valued and b = 1. Using the real case of the Cauchy-Schwarz inequality we find x, y = b x, y = bx, y bx y = x y which proves the inequality also in the complex case. We have equality if and only if x + ay = 0 which means that x and y are linearly dependent. Theorem 2.48 (Triangle Inequality) For any x, y in a real or complex inner product space x + y x + y. Proof. From the Cauchy-Schwarz inequality it follows that Re x, y x y. Using this on the inner product term in (2.19) with a = 1 we get x + y 2 x x y + y 2 = ( x + y ) 2. Taking square roots completes the proof. Theorem 2.49 (Parallelogram Identity) For all x, y in a real or complex inner product space x + y 2 + x y 2 = 2 x y 2. Proof. We set a = ±1 in the inner product expansion (2.19) and add the two equations.

37 2.7. Orthogonality 19 In the real case the Cauchy-Schwarz inequality implies that 1 x,y x y 1 for nonzero x and y so there is a unique angle θ in [0, π] such that cos θ = x, y x y. (2.21) This defines the angle between vectors in a real inner product space. Exercise 2.50 Suppose A R m,n has linearly independent columns. Show that x, y := x T A T Ay defines an inner product on R n. F Exercise 2.51 Show (2.18) Exercise 2.52 Show (2.17) and (2.19). Exercise 2.53 Show (2.20) Exercise 2.54 Show that in the complex case there is a unique angle θ in [0, π/2] such that x, y cos θ = x y. (2.22) 2.7 Orthogonality As in the previous section we assume that (V, F,,, ) is an inner product space with F = R or F = C. Also denotes the inner product norm. Definition 2.55 (Orthogonality) Two vectors x, y in a real or complex inner product space are called orthogonal or perpendicular, denoted as x y, if x, y = 0. The vectors are orthonormal if in addition x = y = 1. For orthogonal vectors it follows from (2.19) that the Pythagorean theorem holds x + y 2 = x 2 + y 2, if x y. Definition 2.56 (Orthogonal- and Orthonormal Bases) A set of vectors {v 1,..., v k } in a subspace S of a real or complex inner product space is called an orthogonal basis for S if it is a basis for S and v i, v j = 0 for i j. It is an orthonormal basis for S if it is a basis for S and v i, v j = δ ij for all i, j. A basis for an inner product space can be turned into an orthogonal- or orthonormal basis for the subspace by the following construction.