Numerical Linear Algebra A Solution Manual. Georg Muntingh and Christian Schulz

Transcription

1 Numerical Linear Algebra A Solution Manual Georg Muntingh and Christian Schulz

2

3 Contents Chapter A Short Review of Linear Algebra Exercise 5: The inverse of a general matrix Exercise 6: The inverse of a matrix Exercise 7: Sherman-Morrison formula Exercise 9: Cramer s rule; special case Exercise 3: Adjoint matrix; special case Exercise 3: Determinant equation for a plane Exercise 3: Signed area of a triangle Exercise 33: Vandermonde matrix 3 Exercise 34: Cauchy determinant 3 Exercise 35: Inverse of the Hilbert matrix 5 Chapter Diagonally dominant tridiagonal matrices; three examples 7 Exercise : The shifted power basis is a basis 7 Exercise 5: LU factorization of nd derivative matrix 7 Exercise 6: Inverse of nd derivative matrix 8 Exercise 7: Central difference approximation of nd derivative 8 Exercise 8: Two point boundary value problem 9 Exercise 9: Two point boundary value problem; computation 9 Exercise 3: Approximate force Exercise 38: Matrix element as a quadratic form Exercise 39: Outer product expansion of a matrix Exercise 4: The product A T A Exercise 4: Outer product expansion Exercise 4: System with many right hand sides; compact form Exercise 43: Block multiplication example Exercise 44: Another block multiplication example Chapter Gaussian eliminations and LU Factorizations Exercise 8: Column oriented backsolve Exercise : Computing the inverse of a triangular matrix Exercise 3: Finite sums of integers 4 Exercise 4: Multiplying triangular matrices 5 Exercise 3: Row interchange 6 Exercise 4: LU and determinant 6 Exercise 5: Diagonal elements in U 6 Exercise 3: Making a block LU into an LU 7 Exercise 36: Using PLU of A to solve A T x = b 8 Exercise 37: Using PLU to compute the determinant 8 Exercise 38: Using PLU to compute the inverse 8 i

4 Chapter 3 LDL* Factorization and Positive definite Matrices 9 Exercise 3: Positive definite characterizations 9 Chapter 4 Orthonormal and Unitary Transformations Exercise 44: The A T A inner product Exercise 45: Angle between vectors in complex case Exercise 48: What does Algorithm housegen do when x = e? Exercise 49: Examples of Householder transformations Exercise 4: Householder transformation Exercise 48: QR decomposition Exercise 49: Householder triangulation Exercise 43: QR using Gram-Schmidt, II 3 Exercise 434: Plane rotation 3 Exercise 435: Solving upper Hessenberg system using rotations 4 Chapter 5 Eigenpairs and Similarity Transformations 5 Exercise 59: Idempotent matrix 5 Exercise 5: Nilpotent matrix 5 Exercise 5: Eigenvalues of a unitary matrix 5 Exercise 5: Nonsingular approximation of a singular matrix 5 Exercise 53: Companion matrix 6 Exercise 57: Find eigenpair example 6 Exercise 5: Jordan example 6 Exercise 54: Properties of the Jordan form 7 Exercise 55: Powers of a Jordan block 7 Exercise 57: Big Jordan example 8 Exercise 53: Schur decomposition example 8 Exercise 534: Skew-Hermitian matrix 8 Exercise 535: Eigenvalues of a skew-hermitian matrix 8 Exercise 549: Eigenvalue perturbation for Hermitian matrices 9 Exercise 55: Hoffman-Wielandt 9 Exercise 554: Biorthogonal expansion 9 Exercise 557: Generalized Rayleigh quotient 9 Chapter 6 The Singular Value Decomposition 3 Exercise 67: SVD examples 3 Exercise 68: More SVD examples 3 Exercise 66: Counting dimensions of fundamental subspaces 3 Exercise 67: Rank and nullity relations 3 Exercise 68: Orthonormal bases example 3 Exercise 69: Some spanning sets 33 Exercise 6: Singular values and eigenpair of composite matrix 33 Exercise 66: Rank example 33 Exercise 67: Another rank example 34 Chapter 7 Norms and Perturbation theory for linear systems 36 Exercise 77: Consistency of sum norm? 36 Exercise 78: Consistency of max norm? 36 Exercise 79: Consistency of modified max norm? 36 Exercise 7: The sum norm is subordinate to? 37 ii

5 Exercise 7: The max norm is subordinate to? 38 Exercise 79: Spectral norm 38 Exercise 7: Spectral norm of the inverse 38 Exercise 7: p-norm example 39 Exercise 74: Unitary invariance of the spectral norm 39 Exercise 75: AU rectangular A 39 Exercise 76: p-norm of diagonal matrix 39 Exercise 77: Spectral norm of a column vector 4 Exercise 78: Norm of absolute value matrix 4 Exercise 735: Sharpness of perturbation bounds 4 Exercise 736: Condition number of nd derivative matrix 4 Exercise 747: When is a complex norm an inner product norm? 43 Exercise 748: p-norm for p = and p = 44 Exercise 749: The p-norm unit sphere 45 Exercise 75: Sharpness of p-norm inequality 45 Exercise 75: p-norm inequalities for arbitrary p 45 Chapter 8 Least Squares 47 Exercise 8: Fitting a circle to points 47 Exercise 87: The generalized inverse 48 Exercise 88: Uniqueness of generalized inverse 48 Exercise 89: Verify that a matrix is a generalized inverse 48 Exercise 8: Linearly independent columns and generalized inverse 49 Exercise 8: The generalized inverse of a vector 49 Exercise 8: The generalized inverse of an outer product 49 Exercise 83: The generalized inverse of a diagonal matrix 5 Exercise 84: Properties of the generalized inverse 5 Exercise 85: The generalized inverse of a product 5 Exercise 86: The generalized inverse of the conjugate transpose 5 Exercise 87: Linearly independent columns 5 Exercise 88: Analysis of the general linear system 5 Exercise 89: Fredholm s Alternative 5 Exercise 83: Condition number 5 Exercise 835: Problem using normal equations 53 Chapter 9 The Kronecker Product 54 Exercise 9: Poisson matrix 54 Exercise 95: Properties of Kronecker products 54 Exercise 99: nd derivative matrix is positive definite 55 Exercise 9: D test matrix is positive definite? 55 Exercise 9: Eigenvalues for D test matrix of order 4 56 Exercise 9: Nine point scheme for Poisson problem 56 Exercise 93: Matrix equation for nine point scheme 57 Exercise 94: Biharmonic equation 58 Chapter Fast Direct Solution of a Large Linear System 6 Exercise 5: Fourier matrix 6 Exercise 6: Sine transform as Fourier transform 6 Exercise 7: Explicit solution of the discrete Poisson equation 6 iii

6 Exercise 8: Improved version of Algorithm 6 Exercise 9: Fast solution of 9 point scheme 6 Exercise : Algorithm for fast solution of 9 point scheme 63 Exercise : Fast solution of biharmonic equation 63 Exercise : Algorithm for fast solution of biharmonic equation 64 Exercise 3: Check algorithm for fast solution of biharmonic equation 64 Chapter The Classical Iterative Methods 66 Exercise : Richardson and Jacobi 66 Exercise 3: Convergence of the R-method when eigenvalues have positive real part 66 Exercise 6: Example: GS converges, J diverges 66 Exercise 7: Divergence example for J and GS 67 Exercise 8: Strictly diagonally dominance; The J method 67 Exercise 9: Strictly diagonally dominance; The GS method 68 Exercise 3: Convergence example for fix point iteration 68 Exercise 4: Estimate in Lemma can be exact 69 Exercise 5: Slow spectral radius convergence 69 Exercise 3: A special norm 7 Exercise 33: When is A + E nonsingular? 7 Chapter The Conjugate Gradient Method 7 Exercise : A-norm 7 Exercise : Paraboloid 7 Exercise 5: Steepest descent iteration 7 Exercise 8: Conjugate gradient iteration, II 73 Exercise 9: Conjugate gradient iteration, III 74 Exercise : The cg step length is optimal 74 Exercise : Starting value in cg 74 Exercise 7: Program code for testing steepest descent 75 Exercise 8: Using cg to solve normal equations 77 Exercise 3: Krylov space and cg iterations 78 Exercise 6: Another explicit formula for the Chebyshev polynomial 79 Exercise 8: Maximum of a convex function 79 Chapter 3 Numerical Eigenvalue Problems 8 Exercise 35: Nonsingularity using Gerschgorin 8 Exercise 36: Gerschgorin, strictly diagonally dominant matrix 8 Exercise 38: Continuity of eigenvalues 8 Exercise 3: -norm of a diagonal matrix 8 Exercise 35: Number of arithmetic operations 8 Exercise 37: Number of arithmetic operations 8 Exercise 38: Tridiagonalize a symmetric matrix 8 Exercise 3: Counting eigenvalues 8 Exercise 33: Overflow in LDL T factorization 83 Exercise 34: Simultaneous diagonalization 83 Exercise 35: Program code for one eigenvalue 84 Exercise 36: Determinant of upper Hessenberg matrix (TODO) 85 Chapter 4 The QR Algorithm 86 iv

7 Exercise 44: Orthogonal vectors 86 v

8 CHAPTER A Short Review of Linear Algebra Exercise 5: The inverse of a general matrix A straightforward computation yields d b a b = ad bc c a c d ad bc ad bc = ad bc showing that the two matrices are inverse to each other Exercise 6: The inverse of a matrix, By Exercise 5, and using that cos θ + sin θ =, the inverse is given by cos θ sin θ sin θ cos θ Exercise 7: Sherman-Morrison formula A direct computation yields (A + BC T ) ( A A B(I + C T A B) C T A ) = I B(I + C T A B) C T A + BC T A BC T A B(I + C T A B) C T A = I + BC T A B(I + C T A B)(I + C T A B) C T A = I + BC T A BC T A = I, showing that the two matrices are inverse to each other Exercise 9: Cramer s rule; special case Cramer s rule yields x = 3 6 / = 3, x = 3 6 / = Exercise 3: Adjoint matrix; special case We are given the matrix A =

9 Computing the cofactors of A gives ( ) ( ) ( ) adj T A = ( ) ( ) ( ) ( ) ( ) ( ) T 4 4 = One checks directly that adj A A = det(a)i, with det(a) = 343 Exercise 3: Determinant equation for a plane Let ax + by + cz + d = be an equation for a plane through the points (x i, y i, z i ), with i =,, 3 There is precisely one such plane if and only if the points are not colinear Then ax i + by i + cz i + d = for i =,, 3, so that x y z a x y z b x y z c = x 3 y 3 z 3 d Since the coordinates a, b, c, d of the plane are not all zero, the above matrix is singular, implying that its determinant is zero Computing this determinant by cofactor expansion of the first row gives the equation y z + y z y 3 z 3 x x z x z x 3 z 3 y + x y x y x 3 y 3 z x y z x y z x 3 y 3 z 3 = of the plane Exercise 3: Signed area of a triangle Let T denote the triangle with vertices P, P, P 3 Since the area of a triangle is invariant under translation, we can assume P = A = (, ), P = (x, y ), P 3 = (x 3, y 3 ), B = (x 3, ), and C = (x, ) As is clear from Figure, the area A(T ) can be expressed as A(T ) = A(ABP 3 ) + A(P 3 BCP ) A(ACP ) = x 3y 3 + (x x 3 )y + (x x 3 )(y 3 y ) x y = x x 3 y y 3, which is what needed to be shown

10 For any n =,,, let x x x n x x x n D n := x 3 x 3 x n 3 x n x n x n Exercise 33: Vandermonde matrix n be the determinant of the Vandermonde matrix in the Exercise Clearly the formula ( ) D N = (x i x j ) j<i N holds for N = (in which case the product is empty and defined to be ) and N = Let us assume ( ) holds for N = n > Since the determinant is an alternating multilinear form, adding a scalar multiple of one column to another does not change the value of the determinant Subtracting x k n times column k from column k + for k = n, n,,, we find x x n x x x n x n x n x n x x n x x x n x n x n x n D n = x 3 x n x 3 x 3 x n x3 n x3 n x n x n x n x n x n x n xn n xn n x n Next, by cofactor expansion along the last row and by the multilinearity in the rows, x x n x x x n x n x n x n x D n = ( ) n x n x x x n x n x n x n x n x n x n x n x n xn x n = ( ) n (x x n )(x x n ) (x n x n )D n = (x n x )(x n x ) (x n x n ) (x i x j ) = j<i n (x i x j ) j<i n By induction, we conclude that ( ) holds for any N =,, n x n Exercise 34: Cauchy determinant (a) Let [α,, α n ] T, [β,, β n ] T R n and let α ( ) +β α +β α +β n α A = (a i,j ) i,j = = +β α +β α +β n α i + β j i,j α n+β α n+β α n+β n 3

11 Multiplying the ith row of A by n k= (α i + β k ) for i =,,, n gives a matrix C = (c i,j ) i,j, c i,j = n (α i + β k ) k= k j The determinant of an n n matrix is a homogeneous polynomial of degree n in the entries of the matrix Since each entry of C is a polynomial of degree n in the variables α i, β j, the determinant of C must be a homogeneous polynomial of degree n(n ) in α i, β j By the multilinearity of the determinant, det C = n i,j= (α i + β j ) det A Since A vanishes whenever α i = α j or β i = β j for i j, the homogeneous polynomial det C contains factors (α i α j ) and (β i β j ) for i < j n As there are precisely (n ) = (n )n such factors, necessarily ( ) det C = k (α i α j ) (β i β j ) i<j n i<j n for some constant k To determine k, we can evaluate det C at a particular value, for instance any {α i, β j } i,j satisfying α + β = = α n + β n = In that case C becomes a diagonal matrix with determinant det C = n i= n (α i +β k ) = k= k i n i= n (α i α k ) = k= k i i<k n (α i α k ) i<k n (α k α i ) Comparing with ( ) shows that k = We conclude that ( ) det A = (α i α j ) (β i β j ) i<j n i<j n n (α i + β j ) i,j= (b) Deleting row l and column k from A, results in the matrix A l,k associated to the vectors [α,, α l, α l+,, α n ] and [β,, β k, β k+,, β n ] By the adjoint 4

12 formula for the inverse A = (b k,l ) and by ( ), b k,l := ( ) k+l det A l,k det A n (α i + β j ) = ( ) k+l = (α l + β k ) = (α l + β k ) i,j= n (α i + β j ) i,j= i l j k n (α s + β k ) s= s l n (α s α l ) s= s l n s= s l α s + β k α s α l which is what needed to be shown i<j n i,j l i<j n (α i α j ) (α i α j ) n (β s + α l ) s= s k n (β s β k ) s= s k n s= s k β s + α l β s β k, i<j n i,j k i<j n (β i β j ) (β i β j ) If we write Exercise 35: Inverse of the Hilbert matrix α = [α,, α n ] = [,,, n], β = [β,, β n ] = [,,, n ], then the Hilbert matrix matrix is of the form H n = (h i,j ) = ( /(α i + β j ) ) By Exercise 34(b), its inverse T n = (t n i,j) := H n is given by t n i,j = (i + j ) We wish to show that n s= s j s + i s j n s= s i s + j, i, j n s i ( ) t n i,j = f(i)f(j), i, j n, i + j where f : N Q is the sequence defined by ( ) i n f() = n, f(i + ) = f(i), for i =,, i 5

13 Clearly ( ) holds when i = j = Suppose that ( ) holds for some (i, j) Then n t n s + + i n s + j i+,j = (i + j) s j s i s= s j = (i + j) (i + j) s= s i+ n+ (s + i ) s= n (s j) s= s j = (i + j ) (n + i)(n i) (i + j)i( i) = i n i + j i (i + j ) = i n f(i)f(j) i + j i f(i + )f(j) = (i + ) + j, n s= s j n (s + j ) s= n (s i) s= s i n (s + i ) s= s j n (s j) s= s j s + i s j n (s + j ) s= s i n (s i) s= s i (i + j ) n s= s i s + j s i so that ( ) holds for (i +, j) Carrying out a similar calculation for (i, j + ), or using the symmetry of T n, we conclude by induction that ( ) holds for any i, j 6

14 CHAPTER Diagonally dominant tridiagonal matrices; three examples Exercise : The shifted power basis is a basis We know that the set of polynomials of degree n is a vector space of dimension n + : They are spanned by {x k } n k=, and these are linearly independent (if a linear combination of these is zero, then it has in particular n + zeros (since every x is a zero), and it follows from the fundamental theorem of algebra that the linear combination must be zero) Since the shifted power basis also has n + vectors which are polynomials, all we need to show is that they are linearly independent Suppose then that n a j (x x i ) j = j= In particular we can then pick n + distinct values z k for x so that this is zero But then the polynomial n j= a jx j has the n + different zeros z k x i Since the {x k } n k= are linearly independent, it follows that all a j =, so that the shifted power basis also is a basis Exercise 5: LU factorization of nd derivative matrix Let L = (l ij ) ij, U = (r ij ) ij and T be as in the exercise Clearly L is unit lower triangular and U is upper triangular We compute the product LU by separating cases for its entries There are several ways to carry out and write down this computation, some more precise than others For instance, (LU) = = ; (LU) ii = i + i + =, i i for i =,, m; (LU) i,i = i i =, i i for i =,, m; (LU) i,i = =, for i =,, m; (LU) ij =, for i j It follows that T = LU is an LU factorization One can also show this by induction using the trifactor-algorithm Since T and U have the same super-diagonal, we must have c m = for all m Assume now that L m U m = T m, and that l m = (m )/m and u m = (m + )/m From the trifactor-algorithm, l m = a m /u m = /((m + )/m) = m/(m + ) u m+ = d m+ l m c m = m/(m + ) = (m + )/(m + ) This shows that the trifactor-algorithm produces the desired terms in L m+ and U m+ as well 7

15 Another way to show this by induction is as follows For m =, one has L U = = T Now let m > be arbitrary and assume that L m U m = T m With a := [,,, m m + ]T, b := [,,, ] T, block multiplication yields Lm Um b L m+ U m+ = a T m+ = m+ Tm L m b a T U m a T b + m+ = m+ By induction, we can then conclude that T m = L m U m for all m Let S = (s ij ) ij be defined by Exercise 6: Inverse of nd derivative matrix s ij = s ji = j(m + i) = m + Tm b b T = T m+ ( i ) j, for j i m m + In order to show that S = T, we multiply S by T and show that the result is the identity matrix To simplify notation we define s ij := whenever i =, i = m +, j =, or j = m + With j < i m, we find ( m ST )i,j = s i,k T k,j = s i,j + s i,j s i,j+ k= ( = i ) ( j + + j j ) =, m + ( m ST )j,i = s j,k T k,i = s j,i + s j,i s j,i+ k= ( = i ) ( j + i ) ( j i + ) j m + m + m + = j + j j + j i i + i + =, m + ( m ST )i,i = s i,k T k,i = s i,i + s i,i s i,i+ k= = ( i ) ( (i ) + i ) ( i i + ) i = m + m + m + which means that ST = I Moreover, since S, T, and I are symmetric, transposing this equation yields TS = I We conclude that S = T Exercise 7: Central difference approximation of nd derivative If all h i equal to the same number h, then λ i = µ i = h h + h =, δ i = y i+ y i, β i = 3(δ i + δ i ) = 3 y i+ y i, h h which is what needed to be shown 8

16 Exercise 8: Two point boundary value problem (a) For j =,, m, we get when we gather terms that ( h f(x j ) = h ) r(x j) v j + ( + h q(x j ))v j + ( + h ) r(x j) v j+ From this we get the desired formula for a j, c j, and d j, and the right hand sides b j for j m For j =, since v is known we have to move ( h r(x ) ) v = a g over to the right hand side, so that we obtain b = h f(x ) a g For j = m, since v m+ is known we have to move ( + h r(x m) ) v m+ = c m g over to the right hand side, so that we obtain b m = h f(x m ) c m g This leads to the tridiagonal system Av = b in the exercise (b) One has When h r(x) / < for all x [a, b], we see that a, j, c j (, ) It follows that a j + c j = + h r(x m) + + h r(x m) = Since q(x j ), d j = d j, so that A is weakly diagonally dominant Since c j = + h r(x j) <, and d j > it follows in particular that d > c Clearly also all a j > since h r(x) / <, and since also d j >, in particular d n, so that all the conditions in the theorem are fulfilled (c) We can use the method trisolve to find the v,, v m Note that the indexing of the a j should be shifted with one in this exercise, to be compatible with the notation used in tridiag(a j, d j, c j ) (a j and d j have the same index when they are in the same column of the matrix In this exercise they have the same index when they are in the same row) Exercise 9: Two point boundary value problem; computation (a) and (c) The provided values for r, f, q give that a j = c j =, d j = + h The initial conditions are g =, g =, so that b = (h +, h,, h ) The code can look as follows for m = [ , 59] h = /(m+); x = h*(:m) ; [l, u] = trifactor( -ones(, m - ), (+hˆ)*ones(, m), -ones(, m - )); b = hˆ*ones(m, ); b() = b() + ; v = trisolve(l, u, -ones(, m - ), b); err = max(abs( (-sinh(x)/sinh()) - v)) log(err)/log(h) end The code also solves (c); If the error is proportional to h p, then err = Ch p for some C But then p = (log(err) log C)/ log h log(err)/ log h for small h, which is the quantity computed inside the for-loop It seems that this converges to 3, so that one would guess that the error is proportional to h 3 (b) m = 9 h = /(m+); x = h*(:m) ; 9

17 [l, u] = trifactor( -ones(, m - ), (+hˆ)*ones(, m), -ones(, m - )); b = hˆ*ones(m, ); b() = b() + ; v = trisolve(l, u, -ones(, m - ), b); plot(x, (-sinh(x)/sinh()), x, v) legend( Exact solution, Estimated solution ) Exercise 3: Approximate force Since sin x has Taylor series x x 3 /3! + x 5 /5!, We have that sin(πh/) = πh/ + O(h 3 ) If we square both sides we obtain sin (πh/) = π h /4 + O(h 4 ) From this we obtain that 4 sin (πh/)r/(h L ) = π R/L + O(h ) Exercise 38: Matrix element as a quadratic form Write A = (a ij ) ij and e i = (δ ik ) k, where { if i = k, δ ik = otherwise, is the Kronecker delta Then, by the definition of the matrix product, ( ) e T i Ae j = e T i (Ae j ) = e T i a lk δ jk = e T i (a lj ) l = δ il a lj = a ij k l l Exercise 39: Outer product expansion of a matrix Clearly e i e T j is the matrix E i,j with at entry (i, j), and zero elsewhere Clearly also A = i,j a i,je i,j = i,j a i,je i e T j Exercise 4: The product A T A A matrix product is defined as long as the dimensions of the matrices are compatible More precisely, for the matrix product AB to be defined, the number of columns in A must equal the number of rows in B Let now A be an n m matrix Then A T is an m n matrix, and as a consequence the product B := A T A is well defined Moreover, the (i, j)-th entry of B is given by (B) ij = ( A T A ) n = a ij ki a kj = a Ṭ ia j = a i, a j, k= which is what was needed to be shown Exercise 4: Outer product expansion Recall that the matrix product of A C m,n and B T = C C n,p is defined by n n (AC) ij = a ik c kj = a ik b jk k= k=

18 For the outer product expansion of the columns of A and B, on the other hand, we find ( ) a :k b T :k = a ij ikb jk It follows that ( ) n n AB T = ( a ij ik b jk = a:k b:k) T k= k= Exercise 4: System with many right hand sides; compact form Let A, B, and X be as in the Exercise (= ): Suppose AX = B Multiplying this equation from the right by e j yields Ax j = b j for j =,, p ( =): Suppose Ax j = b j for j =,, p Let I = I p denote the identity matrix Then AX = AXI = AX[e,, e p ] = [AXe,, AXe p ] ij = [Ax,, Ax p ] = [b,, b p ] = B Exercise 43: Block multiplication example The product AB of two matrices A and B is defined precisely when the number of columns of A is equal to the number of rows of B For both sides in the equation AB = A B to make sense, both pairs (A, B) and (A, B ) need to be compatible in this way Conversely, if the number of columns of A equals the number of rows of B and the number of columns of A equals the number of rows of B, then there exists integers m, p, n, and s with s p such that Then A C m,p, B C p,n, A C m,s, A C m,p s, B C s,n (AB) ij = p a ik b kj = k= s a ik b kj + k= p k=s+ a ik = (A B ) ij Exercise 44: Another block multiplication example Since the matrices have compatible dimensions, a direct computation gives T λ a T T λ a T T λ a CAB = = = T B C A B C A B C A B

19 CHAPTER Gaussian eliminations and LU Factorizations Exercise 8: Column oriented backsolve If A is upper triangular, suppose that we after n k steps of the algorithm have reduced our system to one of the form a, a, a,k a, a,k a k,k x x x k = Clearly then x k = b k /a k,k (this explains the first statement inside the for-loop) Eliminating the x k -variable we obtain the system a, a, a,k a, a,k a k,k x x x k b b b k = b b b k x k a,k a,k a k,k This means that the right hand side b should be updated by subtracting A(:(k-),k)*x(k) If A is d-banded, A,k = = A k d,k =, so that this is the same as subtracting A(lk:(k-),k)*x(k) with lk being the maximum of and k d This explains the second part inside the for-loop Finally we end up with a -matrix, so to find x we only need to divide with a, Exercise : Computing the inverse of a triangular matrix This exercise introduces an efficient method for computing the inverse B of a triangular matrix A Let us solve the problem for an upper triangular matrix (the lower triangular case is similar) By the rules of block multiplication, [Ab,, Ab n ] = A[b,, b n ] = AB = I = [e,, e n ] The kth column in this matrix equation says that Ab k = e k Let b k = (b k,, b nk ) T Since the last n k components of e k are, back subsitution yields that b k+,k = = b n,k =, [ so that B ] is upper triangular (as stated also by Lemma 35) Splitting A into A A blocks where A A has size k k (A and A are then upper triangular),

20 we get A A A b k b kk = A b k b kk [ = ek so that we need to solve b k a a,k b k () A = = e k b kk a k,k b kk This yields () for solving for the kth column of B (note that the Matlab notation I( : k, k) yields e k ) Let us consider the number of arithmetic operations needed to compute the inverse In finding b k we need to solve a k k triangular system Solving for x we need to compute k multiplications, k additions, and one division This gives a total number of k arithmetic operations Solving for x needs k 4 operations, and so on, all the way down to x k which needs operations Solving for x k = /a k,k needs an additional division, so that we need to perform k + r = + (k )k r= operations Since we solve a triangular system for any k n, we end up with a total of n n ( + (k )k) = n + (k )k = n + 3 (n )n(n + ) = 3 n(n + ) k= k= arithmetic operations Here we used the formulas we deduced in Exercise 3 Usually we are just interesting in the leading term for the number of operations (here n 3 /3) This can be obtained more simply by approximating the sums with integrals as in the book: solving the k k triangular system can be solved in + k r= r k r (k r= ) k operations, and adding together the number of operations for all k we obtain n k= k n k= k dk n 3 /3 operations Performing this block multiplication for k = n, n,,, we see that the computations after step k only use the first k leading principal submatrices of A It follows that the column b k computed at step k can be stored in row (or column) k of A without altering the remaining computations A Matlab implementation which stores the inverse (in-place) in A can thus look as follows: n = 8; A = rand(n); A = triu(a); U=A; for k=n:-: U(k,k) = /U(k,k); for r=k-:-: 3 ],

21 U(r, k) = -U(r,r+:k)*U(r+:k,k)/U(r,r); end end U*A A Python implementation can look as follows: from numpy import * n = 8 A = matrix(randomrandom((n,n))) A=triu(A) U=Acopy() for k in range(n-,-,-): U[k,k] = /U[k,k] for r in range(k-,-,-): U[r, k] = -U[r,(r+):(k+)]*U[(r+):(k+),k]/U[r,r] print U*A In the code, r and k are row- and column indices, respecively Inside the for-loop we compute x r for the system in Equation () The contribution from x r+,, x k can be written as a dot product, which here is computed as a matrix product (the minus sign comes from that we isolate x r on the left hand side) Note that k goes from n and downwards If we did this the other way we would overwrite matrix entries needed for later calculations Exercise 3: Finite sums of integers There are many ways to prove these identities The quickest is perhaps by induction We choose instead an approach based on what is called a generating function This approach does not assume knowledge of the sum-expressions we want to derive, and the approach also works in a wide range of other circumstances It is easily checked that the identities hold for m =,, 3 So let m 4 and define Then P m (x) := + x + + x m = xm+ x P m(x) = (m + )xm + mx m+ (x ), P m(x) = + (m + m)x m + ( m )x m + (m m)x m+ (x ) 3 4

22 Applying l Hôpital s rule twice, we find m = P m() = lim x (m + )x m + mx m+ (x ) = lim x m(m + )x m + m(m + )x m (x ) = m(m + ), establishing () In addition it follows that m m = (k ) = m + k= m k = m + m(m + ) = m, which establishes (4) Next, applying l Hôpital s rule three times, we find that (m ) m = P m() is equal to k= + (m + m)x m + ( m )x m + (m m)x m+ lim x (x ) 3 = lim x (m )(m + m)x m + m( m )x m + (m + )(m m)x m 3(x ) = lim x (m )(m )(m + m)x m 3 + (m )m( m )x m + m(m + )(m m)x m 6(x ) = (m )m(m + ), 3 establishing (5) Finally, m = m k = k= m ( ) m m (k )k + k = (k )k + k k= k= k= = 3 (m )m(m + ) + m(m + ) = 3 (m + )(m + )m, which establishes (3) Exercise 4: Multiplying triangular matrices Computing the (i, j)-th entry of the matrix AB amounts to computing the inner product of the ith row a T i: of A and the jth column b :j of B Because of the triangular nature of A and B, only the first i entries of a T i: can be nonzero and only the first j entries of b :j can be nonzero The computation a T i:b :j therefore involves min{i, j} multiplications and min{i, j} additions Carrying out this calculation for all i and j, amounts to a total number of ( n n n i ) n ( min{i, j} ) = (j ) + (i ) = i= j= n ( i + (n i)(i ) ) = i= i= j= j=i+ n ( i + ni n + i ) i= 5

23 n = n + (n + ) i i= n i= i = n + n(n + )(n + ) n(n + )(n + ) 6 = n + 3 n(n + )(n + ) = 3 n3 + 3 n = 3 n(n + ) arithmetic operations A similar calculation gives the same result for the product BA Suppose we are given an LU factorization u u = l u Exercise 3: Row interchange Carrying out the matrix multiplication on the right hand side, one finds that u u =, l u l u + u implying that u = u = It follows that necessarily l = and u =, and the pair L =, U = is the only possible LU factorization of the matrix One directly checks that this is indeed an LU factorization Exercise 4: LU and determinant Suppose A has an LU factorization A = LU Then, by Lemma 6, A [k] = L [k] U [k] is an LU factorization for k =,, n By induction, the cofactor expansion of the determinant yields that the determinant of a triangular matrix is the product of its diagonal entries One therefore finds that det(l [k] ) =, det(u [k] ) = u u kk and for k =,, n det(a [k] ) = det(l [k] U [k] ) = det(l [k] ) det(u [k] ) = u u kk Exercise 5: Diagonal elements in U From Exercise 4, we know that det(a [k] ) = u u kk for k =,, n Since A is nonsingular, its determinant det(a) = u u nn is nonzero This implies that det(a [k] ) = u u kk for k =,, n, yielding a = u for k = and a well-defined quotient for k =,, n det(a [k] ) det(a [k ] ) = u, u k,k u k,k u, u k,k = u k,k, 6

24 Exercise 3: Making a block LU into an LU We can write a block LU factorization of A as I U U U m L A = LU = I U U m L m L m I U mm (ie the blocks are denoted L ij, U ij ) We now assume that U ii has an LU factorization L ii Ũ ii ( L ii unit lower triangular, Ũ ii upper triangular), and define ˆL = Ldiag( L ii ), Û = diag( L ii )U We get that I L L ˆL = Ldiag( L ii ) = I L L m L m I Lmm L L L L = L m L L m L Lmm This shows that ˆL has the blocks L ii on the diagonal, and since these are unit lower triangular, it follows that also ˆL is unit lower triangular Also, L U U U m Û = diag( L ii )U = L U U m L U mm mm L U L U L U m L U L U m = L mmu mm L L Ũ L U L U m L L Ũ L U m = L L mm mm Ũ mm Ũ L U L U m Ũ L U m = Ũ mm where we inserted U ii = L ii Ũ ii This shows Û has the blocks Ũii on the diagonal, and since these are upper triangular, it follows that also Û is upper triangular 7

25 Exercise 36: Using PLU of A to solve A T x = b If A = PLR, then A T = R T L T P T The matrix L T is upper triangular and the matrix R T is lower triangular, implying that R T L T is an LU factorization of A T P Since A is nonsingular, the matrix R T must be nonsingular, and we can apply Algorithms 6 and 7 to economically solve the systems R T z = b, L T y = z, and P T x = y, to find a solution x to the system R T L T P T x = A T x = b If A = PLU, then Exercise 37: Using PLU to compute the determinant det(a) = det(plu) = det(p) det(l) det(u) and the determinant of A can be computed from the determinants of P, L, and U Since the latter two matrices are triangular, their determinants are simply the products of their diagonal entries The matrix P, on the other hand, is a permutation matrix, so that every row and column is everywhere, except for a single entry (where it is ) Its determinant is therefore quickly computed by cofactor expansion Exercise 38: Using PLU to compute the inverse Solving an n n-triangular system takes n operations, as is clear from the rforwardsolve and rbacksolve algorithms From Exercise it is thus clear that inverting an upper /lower triangular matrix takes n k= k n 3 /3 operations (see Exercise 3) Inverting both L and U thus takes n 3 /3 G n operations According to Exercise 4, it takes approximately G n arithmetic operations to multiply an upper and a lower triangular matrix It thus takes approximately G n + G n = G n operations to compute U L 8

26 CHAPTER 3 LDL* Factorization and Positive definite Matrices Exercise 3: Positive definite characterizations We check the equivalent statements of Theorem 38 for the matrix A = Obviously A is symmetric In addition A is positive definite, because [ x x y = x ] y + xy + y = (x + y) + x + y > for any nonzero vector [x, y] T R The eigenvalues of A are the roots of the characteristic equation = det(a λi) = ( λ) = (λ )(λ 3) Hence the eigenvalues are λ = and λ = 3, which are both positive 3 The leading principal submatrices of A are [] and A itself, which both have positive determinants 4 If we assume as in a Cholesky factorization that B is lower triangular we have that BB T = [ b b b b = b b b b b b b b + b ] = Since b = we can choose b = b b = then gives that b = /, and b + b = finally gives b = / = 3/ (we chose the positive square root) This means that we can choose [ ] B = / 3/ This could also have been obtained by writing down an LDL-factorization (as in the proof for its existence), and then multiplying in the square root of the diagonal matrix 9

27 CHAPTER 4 Orthonormal and Unitary Transformations Exercise 44: The A T A inner product Assume that A R m n has linearly independent columns We show that, A : (x, y) x T A T Ay satisfies the axioms of an inner product on a real vector space V, as described in Definition 4 Let x, y, z V and a, b R, and let, be the standard inner product on V Positivity One has x, x A = x T A T Ax = Ax, Ax, with equality holding if and only if Ax = Since Ax is a linearly combination of the columns of A with coefficients the entries of x, and since the columns of A are assumed to be linearly independent, one has Ax = if and only if x = Symmetry One has x, y A = x T A T Ay = (x T A T Ay) T = y T A T Ax = y, x A Linearity One has ax + by, z A = (ax + by) T A T Az = ax T A T Az + by T A T Az = a x, z A + b y, z A Exercise 45: Angle between vectors in complex case By the Cauchy-Schwarz inequality for a complex inner product space, x, y x y Note that taking x and y perpendicular yields zero, taking x and y equal yields one, and any value in between can be obtained by picking an appropriate affine combination of these two cases Since the cosine decreases monotonously from one to zero on the interval [, π/], there is a unique argument θ [, π/] such that cos θ = x, y x y Exercise 48: What does Algorithm housegen do when x = e? If x = e, then the algorithm yields ρ =, and a = e = We then get z = e, and u = z + e = e = e + z

28 and H = I uu T = Exercise 49: Examples of Householder transformations (a) Let x and y be as in the exercise As x = y, we can apply what we did in Example 45 to obtain a vector v and a matrix H, v = x y =, H = I vvt 4 v T v = 3 4, such that Hx = y As explained in the text above Example 45, this matrix H is a Householder transformation with u := v/ v (b) Let x and y be as in the exercise As x = y, we can apply what we did in Example 45 to obtain a vector v and a Householder transformation H, v = x y =, such that Hx = y H = I vvt v T v = 3, Exercise 4: Householder transformation Let H = I uu T R, be any Householder transformation Then u = [u u ] T R is a vector satisfying u +u = u =, implying that the components of u are related via u = u Moreover, as u, u u =, one has u = u, and there exists an angle φ [, π) such that cos(φ ) = u = u For such an angle φ, one has u u = ± + cos φ cos φ = ± cos φ = sin(±φ ) We thus find an angle φ := ±φ for which u H = u u cos(φ u u u = ) sin(±φ ) cos(φ) sin(φ) sin(±φ ) cos(φ = ) sin(φ) cos(φ) Furthermore, we find cos φ cos φ sin φ cos φ H = = sin φ sin φ cos φ sin φ sin φ cos φ = sin φ cos φ cos(φ) sin(φ) When applied to the vector [cos φ, sin φ] T, therefore, H doubles the angle and reflects the result in the y-axis

29 Exercise 48: QR decomposition That Q is orthonormal, and therefore unitary, can be shown directly by verifying that Q T Q = I A direct computation shows that QR = A Moreover, R = =: R,, where R is upper triangular It follows that A = QR is a QR decomposition A QR factorization is obtained by removing the parts of Q and R that don t contribute anything to the product QR Thus we find a QR factorization A = Q R, Q :=, R := Exercise 49: Householder triangulation (a) Let A = [a, a, a 3 ] = be as in the Exercise We wish to find Householder transformations H, H that produce zeros in the columns a, a, a 3 of A Applying Algorithm 47 to the first column of A, we find first that a = 3, z = (/3, /3, /3) T, and then u = 3 3, H A := (I u u T )A = Next we need to map the bottom element (H A) 3, of the second column to zero, without changing the first row of H A For this, we apply Algorithm 47 to the vector (, ) T to find a =, z = (, ) T, and then [ u = and H ] := I u u T =, which is a Householder transformation of size Since 3 H H A := H H A =, it follows that the Householder transformations H and H bring A into upper triangular form (b) Clearly the matrix H 3 := I is orthogonal and R := H 3 H H A is upper triangular with positive diagonal elements It follows that A = QR, Q := H T H T H T 3 = H H H 3, is a QR factorization of A of the required form

30 Let Exercise 43: QR using Gram-Schmidt, II 3 A = [a, a, a 3 ] = Applying Gram-Schmidt orthogonalization, we find v = a =, q =, a T v =, v v T = a at v v v v T = v, q =, a T 3 v v T v = 3, a T 3 v v T v = 5 4, v 3 = a 3 at 3 v v T v v at 3 v v T v v = 3 3 3, q 3 = 3 Since (R ) = v =, (R ) = v = 4, (R ) 33 = v 3 = 6, and since also (R ) ij = (a j ) T q i = v i (a T j v i )/(v T i v i ) for i > j we get that (R ) = =, (R ) 3 = 3 = 3, (R ) 3 = = 5, so that and Q = [ q q q 3 ] = A = Q R = R = Suppose x = r cos α, P = r sin α Exercise 434: Plane rotation cos θ sin θ sin θ cos θ 3

31 Using the angle difference identities for the sine and cosine functions, we find cos(θ α) = cos θ cos α + sin θ sin α, sin(θ α) = sin θ cos α cos θ sin α, Px = r cos θ cos α + sin θ sin α = sin θ cos α + cos θ sin α r cos(θ α) r sin(θ α) Exercise 435: Solving upper Hessenberg system using rotations To determine the number of arithmetic operations of Algorithm 436, we first consider the arithmetic operations in each step Initially the algorithm stores the length of the matrix and adds the right hand side as the (n + )-th column to the matrix Such copying and storing operations do not count as arithmetic operations The second big step is the loop Let us consider the arithmetic operations at the k-th iteration of this loop First we have to compute the norm of a two dimensional vector, which comprises 4 arithmetic operations: two multiplications, one addition and one square root operation Assuming r > we compute c and s each in one division, adding arithmetic operations to our count Computing the product of the Givens rotation and A includes multiplications and one addition for each entry of the result As we have (n + k) entries, this amounts to 6(n + k) arithmetic operations The last operation in the loop is just the storage of two entries of A, which again does not count as an arithmetic operation The final step of the whole algorithm is a backward substitution, known to require O(n ) arithmetic operations We conclude that the Algorithm uses n ( ) n O(n ) (n + k) = O(n ) + 6 (n + k) k= = O(n ) + 3n + 9n = O(4n ) arithmetic operations k= 4

32 CHAPTER 5 Eigenpairs and Similarity Transformations Exercise 59: Idempotent matrix Suppose that (λ, x) is an eigenpair of a matrix A satisfying A = A Then λx = Ax = A x = λax = λ x Since any eigenvector is nonzero, one has λ = λ, from which it follows that either λ = or λ = We conclude that the eigenvalues of any idempotent matrix can only be zero or one Exercise 5: Nilpotent matrix Suppose that (λ, x) is an eigenpair of a matrix A satisfying A k = for some natural number k Then = A k x = λa k x = λ A k x = = λ k x Since any eigenvector is nonzero, one has λ k =, from which it follows that λ = We conclude that any eigenvalue of a nilpotent matrix is zero Exercise 5: Eigenvalues of a unitary matrix Let x be an eigenvector corresponding to λ Then Ax = λx and, as a consequence, x A = x λ To use that A A = I, it is tempting to multiply the left hand sides of these equations, yielding λ x = x λλx = x A Ax = x Ix = x Since x is an eigenvector, it must be nonzero Nonzero vectors have nonzero norms, and we can therefore divide the above equation by x, which results in λ = Taking square roots we find that λ =, which is what needed to be shown Apparently the eigenvalues of any unitary matrix reside on the unit circle in the complex plane Exercise 5: Nonsingular approximation of a singular matrix Let λ,, λ n be the eigenvalues of the matrix A As the matrix A is singular, its determinant det(a) = λ λ n is zero, implying that one of its eigenvalues is zero If all the eigenvalues of A are zero let ε := Otherwise, let ε := min λi λ i be the absolute value of the eigenvalue closest to zero By definition of the eigenvalues, det(a λi) is zero for λ = λ,, λ n, and nonzero otherwise In particular det(a εi) is nonzero for any ε (, ε ), and A εi will be nonsingular in this interval This is what we needed to prove 5

33 Exercise 53: Companion matrix (a) To show that ( ) n f is the characteristic polynomial π A of the matrix A, we need to compute q n λ q n q q λ π A (λ) = det(a λi) = det λ By the rules of determinant evaluation, we can substract from any column a linear combination of the other columns without changing the value of the determinant Multiply columns,,, n by λ n, λ n,, λ and adding the corresponding linear combination to the final column, we find q n λ q n q f(λ) λ π A (λ) = det = ( )n f(λ), where the second equality follows from cofactor expansion along the final column Multiplying this equation by ( ) n yields the statement of the Exercise (b) Similar to (a), by multiplying rows, 3,, n by λ, λ,, λ n and adding the corresponding linear combination to the first row Exercise 57: Find eigenpair example As A is a triangular matrix, its eigenvalues correspond to the diagonal entries One finds two eigenvalues λ = and λ =, the latter with algebraic multiplicity two Solving Ax = λ x and Ax = λ x, one finds (valid choices of) eigenpairs, for instance (λ, x ) = (, ), (λ, x ) = (, ) It follows that the eigenvectors span a space of dimension, and this means that A is defective Exercise 5: Jordan example This exercise shows that it matters in which order we solve for the columns of S One would here need to find the second column first before solving for the other two The matrices given are A = 3 4, J =, 4 we are asked to find S = [s, s, s 3 ] satisfying [As, As, As 3 ] = AS = SJ = [s, s, s 3 ]J = [ s, s + s, s 3 ] 6

34 The equations for the first and third columns say that s and s 3 are eigenvectors for λ =, so that they can be found by row reducing A I: A I = 4 4 (,, ) T and (,, ) T thus span the set of eigenvectors for λ = s can be found by solving As = s + s, so that (A I)s = s This means that (A I) s = (A I)s =, so that s ker(a I) A simple computation shows that (A I) = so that any s will do, but we must also choose s so that (A I)s = s is an eigenvector of A Since A I has rank one, we may choose any s so that (A I)s is nonzero In particular we can choose s = e, and then s = (A I)s = (, 4, 4) T We can also choose s 3 = (,, ) T, since it is an eigenvector not spanned by the s and s which we just defined All this means that we can set S = 4 4 Exercise 54: Properties of the Jordan form Let J = S AS be the Jordan form of the matrix A as in Theorem 59 Items 3 are easily shown by induction, making use of the rules of block multiplication in and 3 For Item 4, write E m := J m (λ) λi m, with J m (λ) the Jordan block of order m By the binomial theorem, J m (λ) r = (E m + λi m ) r = r k= Since E k m = for any k m, we obtain min{r,m } ( ) r J m (λ) r = λ r k E k k m k= ( ) r E k k m(λi m ) r k = r k= ( ) r λ r k E k k m Exercise 55: Powers of a Jordan block Let S be as in Exercise 5 J is block-diagonal so that we can write ( ) J n = n n = = n, n where we used property 4 in exercise 54 on the upper left block It follows that A = (SJS ) = SJ S = = 4 4 =

35 Exercise 57: Big Jordan example The matrix A has Jordan form A = SJS, with 3 3 J =, S = Exercise 53: Schur decomposition example The matrix U is unitary, as U U = U T U = I One directly verifies that R := U T AU = 4 Since this matrix is upper triangular, A = URU T is a Schur decomposition of A Exercise 534: Skew-Hermitian matrix By definition, a matrix C is skew-hermitian if C = C = : Suppose that C = A + ib, with A, B R m,m, is skew-hermitian Then A ib = C = C = (A + ib) = A T ib T, which implies that A T = A and B = B T (use that two complex numbers coincide if and only if their real parts coincide and their imaginary parts coincide) In other words, A is skew-hermitian and B is real symmetric = : Suppose that we are given matrices A, B R m,m such that A is skew- Hermitian and B is real symmetric Let C = A + ib Then C = (A + ib) = A T ib T = A ib = (A + ib) = C, meaning that C is skew-hermitian Exercise 535: Eigenvalues of a skew-hermitian matrix Let A be a skew-hermitian matrix and consider a Schur triangularization A = URU of A Then R = U AU = U ( A )U = U A U = (U AU) = R Since R differs from A by a similary transform, their eigenvalues coincide (use the multiplicative property of the determinant to show that det(a λi) = det(u ) det(uru λi)) det(u) = det(r λi)) As R is a triangular matrix, its eigenvalues λ i appear on its diagonal From the equation R = R it then follows that λ i = λ i, implying that each λ i is purely imaginary 8

36 Exercise 549: Eigenvalue perturbation for Hermitian matrices Since a positive semidefinite matrix has no negative eigenvalues, one has β n It immediately follows from α i + β n γ i that in this case γ i α i Exercise 55: Hoffman-Wielandt The matrix A has eigenvalues and 4, and the matrix B has eigenvalue with algebraic multiplicity two Independently of the choice of the permutation i,, i n, the Hoffman-Wielandt Theorem would yield n n n 6 = µ ij λ j a ij b ij =, j= i= j= which clearly cannot be valid The Hoffman-Wielandt Theorem cannot be applied to these matrices, because B is not normal, B H B = = BB H Exercise 554: Biorthogonal expansion The matrix A has characteristic polynomial det(a λi) = (λ 4)(λ ) and right eigenpairs (λ, x ) = (4, [, ] T ) and (λ, x ) = (, [, ] T ) Since the right eigenvectors x, x are linearly independent, there exists vectors y, y satisfying y i, x j = δ ij The set {x x } forms a basis of C, and the set {y, y } is called the dual basis How do we find such vectors y, y? Any vector [x, x ] T is orthogonal to the vector [αx, αx ] T for any α Choosing α appropriately, one finds y = 3 [, ]T, y = 3 [, ]T By Theorem 553, y and y are left eigenvectors of A For any vector v = [v, v ] T C, Equation (5) then gives us the biorthogonal expansions v = y, v x + y, v x = 3 (v v )x + 3 (v + v )x = x, v y + x, v y = (v + v )y + (v v )y Exercise 557: Generalized Rayleigh quotient Suppose (λ, x) is a right eigenpair for A, so that Ax = λx Then the generalized Rayleight quotient for A is R(y, x) := y Ax y x = y λx y x = λ, which is well defined whenever y x On the other hand, if (λ, y) is a left eigenpair for A, then y A = λy and it follows that R(y, x) := y Ax y x = λy x y x = λ 9

37 CHAPTER 6 The Singular Value Decomposition Exercise 67: SVD examples (a) For A = [3, 4] T we find a matrix A T A = 5, which has the eigenvalue λ = 5 This provides us with the singular value σ = + λ = 5 for A Hence the matrix A has rank and a SVD of the form A = [ U U ] [ 5 ] [V ], with U, U R,, V = V R The eigenvector of A T A that corresponds to the eigenvalue λ = 5 is given by v =, providing us with V = [ ] Using part 3 of Theorem 65, one finds u = [3, 5 4]T Extending u to an orthonormal basis for R gives u = [ 4, 5 3]T A SVD of A is therefore A = (b) One has A =, A T =, A T A = The eigenvalues of A T A are the zeros of det(a T A λi) = (9 λ) 8, yielding λ = 8 and λ =, and therefore σ = 8 and σ = Note that since there is only one nonzero singular value, the rank of A is one Following the dimensions of A, one finds 8 Σ = The normalized eigenvectors v, v of A T A corresponding to the eigenvalues λ, λ are the columns of the matrix V = [v v ] = Using part 3 of Theorem 65 one finds u, which can be extended to an orthonormal basis {u, u, u 3 } using Gram-Schmidt Orthogonalization (see Theorem 49) The vectors u, u, u 3 constitute a matrix U = [u u u 3 ] = 3 3

38 A SVD of A is therefore given by A = 8 3 Exercise 68: More SVD examples (a) We have A = e and A T A = e T e = [ ] This gives the eigenpair (λ, v ) = (, ) of A T A Hence σ = and Σ = e = A As Σ = A and V = I we must have U = I m yielding a singular value decomposition A = I m e I (b) For A = e T n, the matrix A T A = has eigenpairs (, e j ) for j =,, n and (, e n ) Then Σ = e T R,n and V = e n, e n,, e R n,n Using part 3 of Theorem 65 we get u =, yielding U = [ ] A SVD for A is therefore given by A = e T n = [ ] e T en, e n,, e (c) In this exercise A =, A 3 T = A, A T A = 9 The eigenpairs of A T A are given by (λ, v ) = (9, e ) and (λ, v ) = (, e ), from which we find 3 Σ =, V = Using part 3 of Theorem 65 one finds u = e and u = e, which constitute the matrix U = A SVD of A is therefore given by 3 A = Exercise 66: Counting dimensions of fundamental subspaces Let A have singular value decomposition UΣV By parts and 3 of Theorem 65, span(a) and span(a ) are vector spaces of the same dimension r, implying that rank(a) = rank(a ) This statement is known as the rank-nullity theorem, and it follows immediately from combining parts and 4 in Theorem 65 3 As rank(a ) = rank(a) by, this follows by replacing A by A in 3

39 Exercise 67: Rank and nullity relations Let A = UΣV be a singular value decomposition of a matrix A C m n By part 5 of Theorem 64, rank(a) is the number of positive eigenvalues of AA = UΣV VΣ U = UDU, where D := ΣΣ is a diagonal matrix with real nonnegative elements Since UDU is an orthogonal diagonalization of AA, the number of positive eigenvalues of AA is the number of nonzero diagonal elements in D Moreover, rank(aa ) is the number of positive eigenvalues of AA (AA ) = AA AA = UΣΣ ΣΣ V = UD U, which is the number of nonzero diagonal elements in D, so that rank(a) = rank(aa ) From a similar argument for rank(a A), we conclude that rank(a) = rank(aa ) = rank(a A) Let r := rank(a) = rank(a ) = rank(aa ) = rank(a A) Applying Theorem 64, parts 3 and 4, to the singular value decompositions A = UΣV, A = VΣU, AA = UΣΣ U, A A = VΣ ΣV, one finds that {v r+,, v n } is a basis for both ker(a) and ker(a A), while {u r+, u m } is a basis for both ker(a ) and ker(aa ) In particular it follows that dim ker(a) = dim ker(a A), dim ker(a ) = dim ker(aa ), which is what needed to be shown Given is the matrix A = Exercise 68: Orthonormal bases example From Example 66 we know that B = A T and hence A = UΣV T and B = VΣ T U T, with V =, Σ =, U = From Theorem 65 we know that V forms an orthonormal basis for span(a T ) = span(b), V an orthonormal basis for ker(a) and U an orthonormal basis for ker(a T ) = ker(b) Hence span(b) = αv + βv, ker(a) = γv 3 and ker(b) = 3