Chapter 1. Matrix Calculus

Transcription

1 Chapter 1 Matrix Calculus 11 Definitions and Notation We assume that the reader is familiar with some basic terms in linear algebra such as vector spaces, linearly dependent vectors, matrix addition and matrix multiplication (Horn and Johnson [30], Laub [39]) Throughout we consider matrices over the field of complex numbers C or real number R Let z C with z = x + iy and x, y R Then z = x iy In some cases we restrict the underlying field to the real numbers R The matrices are denoted by A, B, C, D, X, Y The matrix elements (entries) of the matrix A are denoted by a jk For the column vectors we write u, v, w The zero column vector is denoted by 0 Let A be a matrix Then A T denotes the transpose and Ā is the complex conjugate matrix We call A the adjoint matrix, where A := ĀT A special role is played by the n n matrices, ie the square matrices In this case we also say the matrix is of order n I n denotes the n n unit matrix (also called identity matrix) The zero matrix is denoted by 0 Let V be a vector space of finite dimension n, over the field R of real numbers, or the field C of complex numbers If there is no need to distinguish between the two, we speak of the field F of scalars A basis of V is a set { e 1, e 2,, e n } of n linearly independent vectors of V, denoted by (e j ) n j=1 Every vector v V then has the unique representation n v = v j e j j=1 the scalars v j, which we will sometimes denote by (v) j, being the components of the vector v relative to the basis (e) j As long as a basis is fixed 1

2 2 Matrix Calculus and Kronecker Product unambiguously, it is thus always possible to identify V with F n In matrix notation, the vector v will always be represented by the column vector v 1 v 2 v = while v T and v will denote the following row vectors v T = (v 1, v 2,, v n ), v = ( v 1, v 2,, v n ) where ᾱ is the complex conjugate of α The row vector v T is the transpose of the column vector v, and the row vector v is the conjugate transpose of the column vector v Definition 11 Let C n be the familiar n dimensional vector space Let u, v C n Then the scalar (or inner) product is defined as Obviously and (u, v) := v n n ū j v j j=1 (u, v) = (v, u) (u 1 + u 2, v) = (u 1, v) + (u 2, v) Since u and v are considered as column vectors, the scalar product can be written in matrix notation as (u, v) u v Definition 12 Two vectors u, v C n are called orthogonal if (u, v) = 0 Example 11 Let u = ( ) ( ) 1 1, v = 1 1 Then (u, v) = 0

3 Matrix Calculus 3 The scalar product induces a norm of u defined by u := (u, u) In section 114 a detailed discussion of norms is given Definition 13 A vector u C n is called normalized if (u, u) = 1 Example 12 The vectors u = 1 2 ( 1 1 ), v = 1 ( ) are normalized and form an orthonormal basis in the vector space R 2 Let V and W be two vector spaces over the same field, equipped with bases (e j ) n j=1 and (f i) m i=1, respectively Relative to these bases, a linear transformation A : V W is represented by the matrix having m rows and n columns a 11 a 12 a 1n a 21 a 22 a 2n A = a m1 a m2 a mn The elements a ij of the matrix A are defined uniquely by the relations m Ae j = a ij f i, j = 1, 2,, n i=1 Equivalently, the jth column vector a 1j a 2j a mj of the matrix A represents the vector Ae j relative to the basis (f i ) m i=1 We call (a i1 a i2 a in ) the ith row vector of the matrix A A matrix with m rows and n columns is called a matrix of type (m, n), and the vector space over the field F consisting of matrices of type (m, n) with elements in F is denoted by A m,n

4 4 Matrix Calculus and Kronecker Product A column vector is then a matrix of type (m, 1) and a row vector a matrix of type (1, n) A matrix is called real or complex according whether its elements are in the field R or the field C A matrix A with elements a ij is written as A = (a ij ) the first index i always designating the row and the second, j, the column Definition 14 A matrix with all its elements 0 is called the zero matrix or null matrix Definition 15 Given a matrix A A m,n (C), the matrix A A n,m (C) denotes the adjoint of the matrix A and is defined uniquely by the relations (Au, v) m = (u, A v) n for every u C n, v C m which imply that (A ) ij = ā ji Definition 16 Given a matrix A = A m,n (R), the matrix A T A n,m (R) denotes the transpose of a matrix A and is defined uniquely by the relations (Au, v) m = (u, A T v) n for every u R n, which imply that (A T ) ij = a ji v R m To the composition of linear transformations there corresponds the multiplication of matrices Definition 17 If A = (a ij ) is a matrix of type (m, l) and B = (b kj ) of type (l, n), their matrix product AB is the matrix of type (m, n) defined by l (AB) ij = a ik b kj We have k=1 (AB) T = B T A T, (AB) = B A Note that AB BA, in general, where A and B are n n matrices Definition 18 A matrix of type (n, n) is said to be square, or a matrix of order n if it is desired to make explicit the integer n; it is convenient to speak of a matrix as rectangular if it is not necessarily square Definition 19 If A = (a ij ) is a square matrix, the elements a ii are called diagonal elements, and the elements a ij, i j, are called off-diagonal elements

5 Matrix Calculus 5 Definition 110 The identity matrix square matrix I := (δ ij ) (also called unit matrix) is the Definition 111 A square matrix A is invertible if there exists a matrix (which is unique, if it does exist), written as A 1 and called the inverse of the matrix A, which satisfies AA 1 = A 1 A = I Otherwise, the matrix is said to be singular Recall that if A and B are invertible matrices then (AB) 1 = B 1 A 1, (A T ) 1 = (A 1 ) T, (A ) 1 = (A 1 ) Definition 112 A square matrix A is symmetric if A is real and A = A T The sum of two symmetric matrices is again a symmetric matrix Definition 113 A square matrix A is skew-symmetric if A is real and A = A T Every square matrix A over R can be written as sum of a symmetric matrix S and a skew-symmetric matrix T, ie A = T + S Thus S = 1 2 (A + AT ), T = 1 2 (A AT ) Definition 114 A square matrix A over C is Hermitian if A = A The sum of two Hermitian matrices is again a Hermitian matrix Definition 115 A square matrix A over C is skew-hermitian if A = A The sum of two skew-hermitian matrices is again a skew-hermitian matrix Definition 116 A Hermitian n n matrix A is positive semidefinite if for all nonzero u C n u Ax 0 Example 13 The 2 2 matrix is positive semidefinite A = ( )

6 6 Matrix Calculus and Kronecker Product Let B be an arbitrary m n matrix over C Then the n n matrix B B is positive semidefinite Definition 117 A square matrix A is orthogonal if A is real and AA T = A T A = I Thus for an orthogonal matrix A we have A 1 = A T The product of two orthogonal matrices is again an orthogonal matrix The inverse of an orthogonal matrix is again an orthogonal matrix The orthogonal matrices form a group under matrix multiplication Definition 118 A square matrix A is unitary if AA = A A = I The product of two unitary matrices is again a unitary matrix The inverse of a unitary matrix is again a unitary matrix The unitary matrices form a group under matrix multiplication Example 14 Consider the 2 2 matrix ( ) 0 i σ y = i 0 The matrix σ y is Hermitian and unitary We have σy = σ y and σy = σy 1 Furthermore σy 2 = I 2 The matrix is one of the Pauli spin matrices Definition 119 A square matrix is normal if AA = A A Example 15 The matrix is normal, whereas A = B = ( ) 0 i i 0 ( ) 0 i 0 0 is not a normal matrix Note that B B is normal Normal matrices include diagonal, real symmetric, real skew-symmetric, orthogonal, Hermitian, skew-hermitian, and unitary matrices Definition 120 A matrix A = (a ij ) is diagonal if a ij = 0 for i j and is written as A = diag(a ii ) = diag(a 11, a 22,, a nn )

7 Matrix Calculus 7 The matrix product of two n n diagonal matrices is again a diagonal matrix Definition 121 Let A = (a ij ) be an m n matrix over a field F The columns of A generate a subspace of F m, whose dimension is called the column rank of A The rows generate a subspace of F n whose dimension is called the row rank of A In other words: the column rank of A is the maximum number of linearly independent columns, and the row rank is the maximum number of linearly independent rows The row rank and the column rank of A are equal to the same number r Thus r is simply called the rank of the matrix A Example 16 The rank of the 2 3 matrix ( ) A = is r(a) = 2 The rank of the matrix product of two matrices cannot exceed the rank of either factors Definition 122 The kernel (or null space) of an m n matrix A is the subspace of vectors x in C n for which Ax = 0 The dimension of this subspace is the nullity of A Example 17 Consider the matrix ( ) 1 1 A = 1 1 Then from the linear equation Ax = 0 we obtain x 1 + x 2 = 0 The null space of A is the set of solutions to this equation, ie a line through the origin of R 2 The nullity of A is equal to 1 Definition 123 Let A, B be n n matrices Then the commutator of A and B is defined by [A, B] := AB BA Obviously we have [A, B] = [B, A] and if C is another n n matrix [A, B + C] = [A, B] + [A, C] Let A, B be n n matrices Then the anticommutator of A and B is defined by [A, B] + := AB + BA

8 8 Matrix Calculus and Kronecker Product Exercises (1) Let A, B be n n upper triangular matrices Can we conclude that AB = BA? (2) Let A be an arbitrary n n matrix Let B be a diagonal matrix Is AB = BA? (3) Let A be a normal matrix and U be a unitary matrix Show that U AU is a normal matrix (4) Show that the following operations, called elementary transformations, on a matrix do not change its rank: (i) The interchange of the i-th and j-th rows (ii) The interchange of the i-th and j-th columns (5) Let A and B be two square matrices of the same order Is it possible to have AB + BA = 0? (6) Let A k, 1 k m, be matrices of order n satisfying m A k = I k=1 Show that the following conditions are equivalent (i) A k = (A k ) 2, 1 k m (ii) A k A l = 0 for k l, 1 k, l m m (iii) r(a k ) = n k=1 where r(a) denotes the rank of the matrix A (7) Prove that if A is of order m n, B is of order n p and C is of order p q, then A(BC) = (AB)C (8) Let S be an invertible n n matrix Let A be an arbitrary n n matrix and à = SAS 1 Show that Ã2 = SA 2 S 1

9 Matrix Calculus 9 12 Matrix Operations Let F be a field, for example the set of real numbers R or the set of complex numbers C Let m, n be two integers 1 An array A of numbers in F a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n = (a ij) a m1 a m2 a m3 a mn is called an m n matrix with entry a ij in the ith row and jth column A row vector is a 1 n matrix A column vector is an n 1 matrix We have a zero matrix, in which a ij = 0 for all i, j Let A = (a ij ) and B = (b ij ) be two m n matrices We define A + B to be the m n matrix whose entry in the i-th row and j-th column is a ij + b ij The m n matrices over a field F form a vector space Matrix multiplication is only defined between two matrices if the number of columns of the first matrix is the same as the number of rows of the second matrix If A is an m n matrix and B is an n p matrix, then the matrix product AB is an m p matrix defined by n (AB) ij = a ir b rj for each pair i and j, where (AB) ij denotes the (i, j)th entry in AB Let A = (a ij ) and B = (b ij ) be two m n matrices with entries in some field Then their Hadamard product is the entry-wise product of A and B, that is the m n matrix A B whose (i, j)th entry is a ij b ij r=1 Example 18 Let Then A = ( ) 1 i, B = 1 i ( ) ( ) 3 i 1 A + B = 0 i ( ) 2 i A B = 1 0 ( ) i AB = i 2 1

10 10 Matrix Calculus and Kronecker Product 13 Linear Equations Let A be an m n matrix over a field F Let b 1,, b m be elements of the field F The system of equations a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2 a m1 x 1 + a m2 x a mn x n = b m is called a system of linear equations We also write Ax = b, where x and b are considered as column vectors The system is said to be homogeneous if all the numbers b 1,, b m are equal to 0 The number n is called the number of unknowns, and m is called the number of equations The system of homogeneous equations also admits the trivial solution x 1 = x 2 = = x n = 0 A system of homogeneous equations of m linear equations in n unknowns with n > m admits a nontrivial solution An under determined linear system is either inconsistent or has infinitely many solutions An important special case is m = n Then for the system of linear equations Ax = b we investigate the cases A 1 exists and A 1 does not exist If A 1 exists we can write the solution as x = A 1 b If m > n, then we have an overdetermined system and it can happen that no solution exists One solves these problems in the least-square sense Example 19 Consider the system of linear equations 3x 1 x 2 = 1 x 2 x 3 = 0 x 1 + x 2 + x 3 = 1 These equations have the matrix representation x x 2 = x 3 1 with solution x 1 = 1 7, x 2 = 4 7 and x 3 = 4 7

11 Matrix Calculus Trace and Determinant In this section we introduce the trace and determinant of a n n matrix and summarize their properties Definition 124 The trace of a square matrix A = (a jk ) of order n is defined as the sum of its diagonal elements n tr(a) := a jj j=1 Example 110 Let A = ( ) Then tr(a) = 0 The properties of the trace are as follows Let a, b C and let A, B and C be three n n matrices Then tr(aa + bb) = atr(a) + btr(b) tr(a T ) = tr(a) tr(ab) = tr(ba) tr(a) = tr(s 1 AS) tr(a A) = tr(aa ) tr(abc) = tr(cab) = tr(bca) S nonsingular n n matrix Thus the trace is a linear functional From the third property we find that tr([a, B]) = 0 where [, ] denotes the commutator The last property is called the cyclic invariance of the trace Notice, however, that tr(abc) tr(bac) in general An example is given by the following matrices ( ) ( ) ( ) A =, B =, C = We have tr(abc) = 1 but tr(bac) = 0

12 12 Matrix Calculus and Kronecker Product If λ j, j = 1, 2,, n are the eigenvalues of the n n matrix A (see section 15), then n n tr(a) = λ j, tr(a 2 ) = λ 2 j j=1 j=1 More generally, if p designates a polynomial of degree r r p(x) = a j x j then Moreover we find tr(p(a)) = j=0 n p(λ k ) k=1 tr(aa ) = tr(a A) = Thus tr(aa ) is a norm of A Example 111 Let A = n j,k=1 ( ) 0 i i 0 a jk 2 0 Then AA = I 2, where I 2 is the 2 2 identity matrix Thus A = 2 Let x, y be column vectors in R n Then x T y = tr(xy T ) = tr(yx T ) Let A an n n matrix over R Then we have x T Ay = tr(ayx T ) Next we introduce the definition of the determinant of an n n matrix Then we give the properties of the determinant Definition 125 The determinant of an n n matrix A is a scalar quantity denoted by det(a) and is given by det(a) := p(j 1, j 2,, j n )a 1j1 a 2j2 a njn j 1,j 2,,j n where p(j 1, j 2,, j n ) is a permutation equal to ±1 and the summation extends over n! permutations j 1, j 2,, j n of the integers 1, 2,, n For an n n matrix there exist n! permutations Therefore p(j 1, j 2,, j n ) = sign (j r j s ) 1 s<r n

13 Matrix Calculus 13 Example 112 For a matrix of order (3,3) we find p(1, 2, 3) = 1, p(1, 3, 2) = 1, p(3, 1, 2) = 1 p(3, 2, 1) = 1, p(2, 3, 1) = 1, p(2, 1, 3) = 1 Then the determinant for a 3 3 matrix is given by a 11 a 12 a 13 det a 21 a 22 a 23 = a 11 a 22 a 33 a 11 a 23 a 32 + a 13 a 21 a 32 a 31 a 32 a 33 a 13 a 22 a 31 + a 12 a 23 a 31 a 12 a 21 a 33 Definition 126 We call a square matrix A a nonsingular matrix if det(a) 0 whereas if det(a) = 0 the matrix A is called a singular matrix If det(a) 0, then A 1 exists Conversely, if A 1 exists, then det(a) 0 Example 113 The matrix ( ) is nonsingular since its determinant is 1, and the matrix ( ) is singular since its determinant is 0 Next we list some properties of determinants 1 Let A be an n n matrix and A T the transpose Then det(a) = det(a T ) Example det = det =

14 14 Matrix Calculus and Kronecker Product Remark Let Then Obviously A T = A = ( ) 0 i 1 0 ( ) 0 1, A i 0 ĀT = det(a) det(a ) 2 Let A be an n n matrix and α R Then det(αa) = α n det(a) ( ) 0 1 i 0 3 Let A be an n n matrix If two adjacent columns are equal, ie A j = A j+1 for some j = 1, 2,, n 1, then det(a) = 0 4 If any vector in A is a zero vector then det(a) = 0 5 Let A be an n n matrix Let j be some integer, 1 j < n If the j-th and (j + 1)-th columns are interchanged, then the determinant changes by a sign 6 Let A 1,, A n be the column vectors of an n n matrix A If they are linearly dependent, then det(a) = 0 7 Let A and B be n n matrices Then det(ab) = det(a) det(b) 8 Let A be an n n diagonal matrix Then det(a) = a 11 a 22 a nn 9 (d/dt) det(a(t)) = sum of the determinants where each of them is obtained by differentiating the rows of A with respect to t one at a time, then taking its determinant Proof we find Since det(a(t)) = p(j 1,, j n )a 1j1 (t) a njn (t) j 1,,j n

15 Matrix Calculus 15 d dt det(a(t)) = p(j 1,, j n ) da 1j1(t) dt j 1,,j n + p(j 1,, j n )a 1j1 (t) da 2j2(t) dt j 1,,j n a 2j2 (t) a njn (t) + a njn (t) + + p(j 1,, j n )a 1j1 (t) a n 1jn 1 (t) da njn(t) dt j 1,,j n Example 115 We have ( ) ( ) ( ) d e t dt det cos t e t sin t e t cos t 1 sin t 2 = det 1 sin t 2 + det 0 2t cos t 2 10 Let A be an invertible n n symmetric matrix over R Then for every vector v R n v T A 1 v = det(a + vvt ) det(a) 1 Example 116 Let A = Then A 1 = A and therefore Since and det(a) = 1 we obtain ( ) 0 1, v = 1 0 v T A 1 v = 2 vv T = det(a + vv T ) det(a) ( ) ( ) = 2 11 Let A be an n n matrix Then det(exp(a)) exp(tr(a)) 12 The determinant of a diagonal matrix or triangular matrix is the product of its diagonal elements

16 16 Matrix Calculus and Kronecker Product 13 Let A be an n n matrix Let λ 1, λ 2,, λ n be the eigenvalues of A (see section 15) Then det(a) = λ 1 λ 2 λ n 14 Let A be a Hermitian matrix Then det(a) is a real number 15 Let U be a unitary matrix Then for some ϕ R Thus det(u) = 1 det(u) = e iϕ 16 Let A, B, C be n n matrices and let 0 be the n n zero matrix Then ( ) A 0 det = det(a) det(b) C B 17 The determinant of the matrix b 1 a b 2 a A n :=, n = 1, 2, b n 1 a n b n satisfies the recursion relation det(a n ) = b n det(a n 1 ) + a n det(a n 2 ), det(a 0 ) = 1, det(a 1 ) = b 1 where n = 2, 3, 18 Let A be a 2 2 matrix Then det(i 2 + A) 1 + tr(a) + det(a) 19 Let A be an invertible n n matrix, ie det(a) 0 Then the inverse of A can be calculated as (A) 1 kj = ln(det(a)) (A) jk

17 Matrix Calculus 17 Exercises (1) Let X and Y be n n matrices over R Show that (X, Y ) := tr(xy T ) defines a scalar product, ie prove that (X, X) 0, (X, Y ) = (Y, X), (cx, Y ) = c(x, Y ) (c R), (X + Y, Z) = (X, Z) + (Y, Z) (2) Let A and B be n n matrices Show that tr([a, B]) = 0 where [, ] denotes the commutator (ie [A, B] := AB BA) (3) Use (2) to show that the relation [A, B] = λi, λ C for finite dimensional matrices can only be satisfied if λ = 0 For certain infinite dimensional matrices A and B we can find a nonzero λ (4) Let A and B be n n matrices Suppose that AB is nonsingular Show that A and B are nonsingular matrices (5) Let A and B be n n matrices over R Assume that A is skewsymmetric, ie A T = A Assume that n is odd Show that det(a) = 0 (6) Let ( ) A11 A A = 12 A 21 A 22 be a square matrix partitioned into blocks Assuming the submatrix A 11 to be invertible, show that det(a) = det(a 11 ) det(a 22 A 21 A 1 11 A 12) (7) A square matrix A for which A n = 0, where n is a positive integer, is called nilpotent Let A be a nilpotent matrix Show that det(a) = 0 (8) Let A be an n n skew-symmetric matrix over R Show that if n is odd then det(a) = 0 Hint Apply det(a) = ( 1) n det(a) (9) Let A be an n n matrix with A 2 = I n Calculate det(a)

18 18 Matrix Calculus and Kronecker Product 15 Eigenvalue Problem The eigenvalue problem plays a central role in theoretical and mathematical physics (Steeb [60; 61]) We give a short introduction into the eigenvalue calculation for finite dimensional matrices In section 26 we study the eigenvalue problem for Kronecker products of matrices Definition 127 A complex number λ is said to be an eigenvalue (or characteristic value) of an n n matrix A, if there is at least one nonzero vector u C n satisfying the eigenvalue equation Au = λu, u 0 Each nonzero vector u C n satisfying the eigenvalue equation is called an eigenvector (or characteristic vector) of A with eigenvalue λ The eigenvalue equation can be written as (A λi)u = 0 where I is the n n unit matrix and 0 is the zero vector This system of n linear simultaneous equations in u has a nontrivial solution for the vector u only if the matrix (A λi) is singular, ie det(a λi) = 0 The expansion of the determinant gives a polynomial in λ of degree equal to n, which is called the characteristic polynomial of the matrix A The n roots of the equation det(a λi) = 0, called the characteristic equation, are the eigenvalues of A Definition 128 Let λ be an eigenvalue of an n n matrix A The vector u is a generalized eigenvector of A corresponding to λ if (A λi) n u = 0 The eigenvectors of a matrix are also generalized eigenvectors of the matrix Theorem 11 Every n n matrix A has at least one eigenvalue and corresponding eigenvector Proof We follow the proof in Axler [2] Suppose v C n \ {0} Then {v, Av,, A n v} must be a linearly dependent set of vectors, ie there exist c 0,, c n C such that c 0 v + c 1 Av + + c n A n v = 0

19 Matrix Calculus 19 Let m {0, 1,, n} be the largest index satisfying c m 0 Consider the following polynomial in x and its factorization over C: c 0 + c 1 x + + c m x m = c m (x x 0 )(x x 1 ) (x x m ) for some x 0,, x m C (ie the roots of the polynomial) Then c 0 v + c 1 Av + + c n A n v = c m (A x 0 I n )(A x 1 I n ) (A x m I n )v = 0 It follows that there is a largest j {0, 1,, m} satisfying (A x j I n ) [(A x j+1 I n ) (A x m I n )v] = 0 This is a solution to the eigenvalue equation, the eigenvalue is x j and the corresponding eigenvector is (A x j+1 I n ) (A x m I n )v Definition 129 The spectrum of the matrix A is the subset n sp(a) := { λ i (A) } i=1 of the complex plane The spectral radius of the matrix A is the nonnegative number defined by If λ sp(a), the vector subspace ϱ(a) := max{ λ j (A) : 1 j n } { v V : Av = λv } (of dimension at least 1) is called the eigenspace corresponding to the eigenvalue λ Example 117 Let Then A = ( ) 0 i i 0 det(a λi 2 ) λ 2 1 = 0 Therefore the eigenvalues are given by λ 1 = 1, λ 2 = 1 To find the eigenvector of the eigenvalue λ 1 = 1 we have to solve ( ) ( ) ( ) 0 i u1 u1 = 1 i 0 u 2 u 2

20 20 Matrix Calculus and Kronecker Product Therefore u 2 = iu 1 and the eigenvector of λ 1 = 1 is given by ( ) 1 u 1 = i For λ 2 = 1 we have and hence ( ) ( ) 0 i u1 = 1 i 0 u 2 u 2 = ( ) 1 i ( u1 u 2 ) We see that (u 1, u 2 ) u 2u 1 = 0 Both eigenvectors are not normalized A special role in theoretical physics is played by the Hermitian matrices In this case we have the following theorem Theorem 12 Let A be a Hermitian matrix, ie A = A, where A ĀT The eigenvalues of A are real, and two eigenvectors corresponding to two different eigenvalues are mutually orthogonal Proof The eigenvalue equation is Au = λu, where u 0 Now we have the identity (Au) u u A u u (A u) u (Au) since A is Hermitian, ie A = A Inserting the eigenvalue equation into this equation yields (λu) u = u (λu)b or λ(u u) = λ(u u) Since u u 0, we have λ = λ and therefore λ must be real Let Now Au 1 = λ 1 u 1, Au 2 = λ 2 u 2 λ 1 (u 1, u 2 ) = (λ 1 u 1, u 2 ) = (Au 1, u 2 ) = (u 1, Au 2 ) = (u 1, λ 2 u 2 ) = λ 2 (u 1, u 2 ) Since λ 1 λ 2, we find that (u 1, u 2 ) = 0 Theorem 13 The eigenvalues λ j of a unitary matrix U satisfy λ j = 1

21 Matrix Calculus 21 Proof Since U is a unitary matrix we have U = U 1, where U 1 is the inverse of U Let Uu = λu be the eigenvalue equation It follows that (Uu) = (λu) or u U = λu Thus we obtain u U Uu = λλu u Owing to U U = I we obtain u u = λλu u Since u u 0 we have λλ = 1 Thus the eigenvalue λ can be written as λ = exp(iα), α R Thus λ = 1 Theorem 14 If x is an eigenvalue of the n n normal matrix A corresponding to the eigenvalue λ, then x is also an eigenvalue of A corresponding to the eigenvalue λ Proof Since (A λi n )x = 0 and AA = A A we find ((A λi n )x, (A λi n )x) = [ (A λi n )x ] [ (A λi n )x ] = x (A λi n )(A λi n )x = x (A λi n )(A λi n )x = x (A λi n )0 = 0 Consequently (A λi n )x = 0 Theorem 15 The eigenvalues of a skew-hermitian matrix (A = A) can only be 0 or (purely) imaginary The proof is left as an exercise to the reader Example 118 The skew-hermitian matrix ( ) 0 i A = i 0 has the eigenvalues ±i

22 22 Matrix Calculus and Kronecker Product Now consider the general case Let λ be an eigenvalue of A with the corresponding eigenvector x, and let µ be an eigenvalue of A with the corresponding eigenvector y Then and x A y = (x A )y = (Ax) y = (λx) y = λx y It follows that x y = 0 or λ = µ Example 119 Consider x A y = x (A y) = µx y A = ( ) Both eigenvalues of A are zero We have ( ) 0 0 A = 1 0 Both eigenvalues of A are also zero The eigenspaces corresponding to the eigenvalue 0 of A and A are {( ) } t : t C, t 0 0 and {( ) } 0 : t C, t 0 t respectively Obviously both conditions above are true The eigenvalues of A A and AA are given by 0 and 1

23 Matrix Calculus 23 Exercises (1) Let A be an n n matrix over C Show that the eigenvectors of A corresponding to distinct eigenvalues are linearly independent (2) Show that tr(a) = n λ j (A), det(a) = j=1 n λ j (A) (3) Let U be a Hermitian and unitary matrix What can be said about the eigenvalues of U? (4) Let A be an invertible matrix whose elements, as well as those of A 1, are all nonnegative Show that there exists a permutation matrix P and a matrix D = diag (d j ), with d j positive, such that A = P D (the converse is obvious) (5) Let A and B be two square matrices of the same order Show that the matrices AB and BA have the same characteristic polynomial (6) Let a, b, c R Find the eigenvalues and eigenvectors of the symmetric 4 4 matrix a b 0 0 A = c a b 0 0 c a b 0 0 c a (7) Let a 1, a 2,, a n R Show that the eigenvalues of the matrix a 1 a 2 a 3 a n 1 a n a n a 1 a 2 a n 2 a n 1 a n 1 a n a 1 a n 3 a n 2 A = a 3 a 4 a 5 a 1 a 2 a 2 a 3 a 4 a n a 1 called a circulant matrix, are of the form j=1 λ l+1 = a 1 + a 2 ξ l + a 3 ξ 2 l + + a n ξ n 1 l, l = 0, 1,, n 1 where ξ l := e 2iπl/n

24 24 Matrix Calculus and Kronecker Product 16 Cayley-Hamilton Theorem The Cayley-Hamilton theorem states that the matrix A satisfies its own characteristic equation, ie (A λ 1 I n ) (A λ n I n ) = 0 n n where 0 n n is the n n zero matrix Notice that the factors commute In this section we follow Axler [2] Definition 130 An n n matrix A is upper triangular with respect to a basis {v 1,, v m } C n, where m n, if Av j span{v 1,, v j }, j = 1,, m Theorem 16 For every n n matrix A there exists a basis V for C n such that A is upper triangular with respect to V Proof Let v 1 be an eigenvector of A The proof that every matrix A is upper triangular is by induction The case n = 1 is obvious Consider the subspace For all u U U = {(A x j I n )x : x C n } Au = (A x j I n )u + x j u U since (A x j I n )u U by definition Since x j is an eigenvalue of A we have det(a x j I n ) = 0 so that dim(u) < n The induction hypothesis is that any square matrix is upper triangular with respect to a basis for a (sub)space with dimension less than n Consequently A has a triangular representation on U Let {v 1,, v dimu } be a basis for U and {v 1,, v n } be a basis for C n We have for k {dimu + 1,, n} where Av k = (A x j I n )v k + x j v k span{v 1,, v dimu, v k } span{v 1,, v dimu, v k } span{v 1,, v k } and (A x j I n )v k U by definition It follows that A is upper triangular with respect to V = {v 1,, v n }

25 Matrix Calculus 25 If A has a triangular representation with respect to some basis we can find a triangular representation with respect to an orthonormal basis by applying the Gram-Schmidt orthonormalization process (see section 117) Theorem 17 Every n n matrix A satisfies its own characteristic equation (A λ 1 I n ) (A λ n I n ) = 0 n n where 0 n n is the n n zero matrix, and λ 1,, λ n are the eigenvalues of A Proof Let A be triangular with respect to the basis {v 1,, v n } C n for C n Thus v 1 is an eigenvector of A corresponding to an eigenvalue, say λ j1 Consequently (A λ j1 I n )v 1 = 0 Now suppose ( k ) (A λ j1 I n )(A λ j2 I n ) (A λ jk I n )v k = (A λ jk I n ) v k = 0 for k = 1, 2,, r Now Av r+1 span{v 1, v 2,, v r+1 } so that Av r+1 = u + αv r+1 for some u span{v 1,, v r }, α C The supposition above (induction hypothesis) implies ( k ) (A λ jk I n ) u = 0 so that p=1 p=1 ( k ) ( k ) (A λ jk I n ) Av r+1 = α (A λ jk I n ) p=1 p=1 which simplifies to ( k ) (A αi n ) (A λ jk I n ) v r+1 = 0 p=1 since A commutes with (A ci n ) (c C) Thus either ( k ) (A λ jk I n ) v r+1 = 0 p=1 v r+1 or α is an eigenvalue, say λ jr+1, of A If the first case holds a re-ordering of the basis {v 1,, v n } postpones the arbitrary choice of λ jr+1 In either case, we have shown ( by induction that n ) (A λ p I n ) v k = 0, k = 1, 2,, n p=1 Since {v 1,, v r } is a basis we must have n (A λ p I n ) = (A λ 1 I n ) (A λ n I n ) = 0 n n p=1

26 26 Matrix Calculus and Kronecker Product 17 Projection Matrices First we introduce the definition of a projection matrix and give some of its properties Projection matrices (projection operators) play a central role in finite group theory in the decomposition of Hilbert spaces into invariant subspaces (Steeb [60; 61; 57]) Definition 131 An n n matrix Π is called a projection matrix if and Π = Π Π 2 = Π The element Πu (u C n ) is called the projection of the element u Example 120 Let n = 2 and ( ) 1 0 Π 1 =, Π 2 = 0 0 ( ) Then Π 1 = Π 1, Π 2 1 = Π 1, Π 2 = Π 2 and Π 2 2 = Π 2 Furthermore Π 1 Π 2 = 0 and ( ) ( ) ( ) ( ) u1 u1 u1 0 Π 1 =, Π 2 = u 2 0 u 2 u 2 Theorem 18 Let Π 1 and Π 2 be two n n projection matrices Assume that Π 1 Π 2 = 0 Then (Π 1 u, Π 2 u) = 0 Proof We find (Π 1 u, Π 2 u) = (Π 1 u) (Π 1 u) = (u Π 1)(Π 2 u) = u (Π 1 Π 2 )u = 0 Theorem 19 Let I n be the n n unit matrix and Π be a projection matrix Then I n Π is a projection matrix Proof Since (I n Π) = In Π = I n Π and (I n Π) 2 = (I n Π)(I n Π) = I n Π Π + Π = I n Π we find that I n Π is a projection matrix

27 Matrix Calculus 27 Theorem 110 The eigenvalues λ j of a projection matrix Π are given by λ j { 0, 1 } Proof From the eigenvalue equation Πu = λu we find Π(Πu) = (ΠΠ)u = λπu Using the fact that Π 2 = Π we obtain Πu = λ 2 u Thus λ = λ 2 since u 0 and hence λ { 0, 1 } Theorem 111 (Projection Theorem) Let U be a nonempty, convex, closed subset of the vector space C n Given any element w C n, there exists a unique element Πw such that This element Πw U satisfies Πw U and w Πw = inf w v v U (Πw w, v Πw) 0 and, conversely, if any element u satisfies then u = Πw Furthermore for every v U u U and (u, v u) 0 for every v U For the proof refer to Ciarlet [13] Πu Πv u v Let u be a nonzero normalized column vector Then uu is a projection matrix, since (uu ) = uu and (uu )(uu ) = u(u u)u = uu If u is the zero column vector then uu is the square zero matrix which is also a projection matrix

28 28 Matrix Calculus and Kronecker Product Exercises (1) Show that the matrices Π 1 = 1 ( ) 1 1, Π 2 = are projection matrices and that Π 1 Π 2 = 0 ( ) (2) Is the sum of two n n projection matrices an n n projection matrix? (3) Let A be an n n matrix with A 2 = A Show that det(a) is either equal to zero or equal to 1 (4) Let and Show that (5) Consider the matrices ( ) 2 1 A =, I = Π = u = 0 0, v = Πu Πv u v ( ) 1 0, C = 0 1 ( ) Show that [A, I 2 ] = 0 and [A, C] = 0 Show that I 2 C = C, CI = C, CC = I A group theoretical reduction (Steeb [60; 61]) leads to the projection matrices Π 1 = 1 2 ( ) 1 1, Π 2 = ( ) Apply the projection operators to the standard basis to find a new basis Show that the matrix A takes the form ( ) 3 0 à = 0 1 within the new basis Notice that the new basis must be normalized before the matrix à can be calculated

29 Matrix Calculus Fourier and Hadamard Matrices Fourier and Hadamard matrices play an important role in spectral analysis (Davis [15], Elliott and Rao [18], Regalia and Mitra [47]) We give a short introduction to these types of matrices In sections 28 and 316 we discuss the connection with the Kronecker product Let n be a fixed integer 1 We define ( ) 2πi w := exp cos n ( 2π n ) + i sin ( ) 2π n where i = 1 w might be taken as any primitive n-th root of unity It can easily be proved that and w n = 1 w w = 1 w = w 1 w k = w k = w n k 1 + w + w w n 1 = 0 where w is the complex conjugate of w Definition 132 By the Fourier matrix of order n, we mean the matrix F (= F n ) where F := 1 (w (i 1)(j 1) ) 1 1 w w 2 w n 1 1 w 2 w 4 w 2(n 1) n n 1 w n 1 w 2(n 1) w (n 1)(n 1) where F is the conjugate transpose of F The sequence w k, k = 0, 1, 2, is periodic with period n Consequently there are only n distinct elements in F Therefore F can be written as F = 1 1 w w 2 w n 1 1 w 2 w 4 w n 2 n 1 w n 1 w n 2 w

30 30 Matrix Calculus and Kronecker Product The following theorem can easily be proved Theorem 112 F is unitary, ie Proof F F = F F = I n F 1 = F This is a result of the geometric series identity n 1 w r(j k) 1 { wn(j k) n if j = k 1 w j k = 0 if j k r=0 A second application of the geometrical identity yields F 2 F F = F This means F 2 is an n n permutation matrix Corollary 11 F 4 = I n, F 3 = F 4 (F ) 1 = I n F = F Corollary 12 The eigenvalues of F are ±1, ±i, with appropriate multiplicities The characteristic polynomials f(λ) of F (= F n) are as follows n 0 modulo 4, f(λ) = (λ 1) 2 (λ i)(λ + 1)(λ 4 1) (n/4) 1 n 1 modulo 4, f(λ) = (λ 1)(λ 4 1) (1/4)(n 1) n 2 modulo 4, f(λ) = (λ 2 1)(λ 4 1) (n/4)(n 2) n 3 modulo 4, f(λ) = (λ i)(λ 2 1)(λ 4 1) (1/4)(n 3) Definition 133 Let Z = (z 1, z 2,, z n ) T and Ẑ = (ẑ 1, ẑ 2,, ẑ n ) T where z j C The linear transformation Ẑ = F Z where F is the Fourier matrix is called the discrete Fourier transform

31 Matrix Calculus 31 Its inverse transformation exists since F 1 exists and is given by Let Z = F 1 Ẑ F Ẑ p(z) = a 0 + a 1 z + + a n 1 z n 1 be a polynomial of degree n 1 It will be determined uniquely by specifying its values p(z n ) at n distinct points z k, k = 1, 2,, n in the complex plane C Select these points z k as the n roots of unity 1, w, w 2,, w n 1 Then so that nf a 0 a 1 a n 1 a 0 a 1 a n 1 p(1) = p(w) p(w n 1 ) p(1) = 1 p(w) F n p(w n 1 ) These formulas for interpolation at the roots of unity can be given another form Definition 134 By a Vandermonde matrix V (z 0, z 1,, z n 1 ) is meant a matrix of the form z 0 z 1 z n 1 V (z 0, z 1,, z n 1 ) := z0 2 z1 2 zn 1 2 z0 n 1 z1 n 1 zn 1 n 1 It follows that Furthermore V (1, w, w 2,, w n 1 ) = n 1/2 F V (1, w, w 2,, w n 1 ) = n 1/2 F = n 1/2 F p(z) = (1, z,, z n 1 )(a 0, a 1,, a n 1 ) T = (1, z,, z n 1 )n 1/2 F (p(1), p(w),, p(w n 1 )) T = n 1/2 (1, z,, z n 1 )V (1, w,, w n 1 )(p(1), p(w),, p(w n 1 )) T

32 32 Matrix Calculus and Kronecker Product Let F 2 denote the Fourier matrices of order n 2n whose rows have been permuted according to the bit reversing permutation Definition 135 A sequence in natural order can be arranged in bitreversed order as follows: For an integer expressed in binary notation, reverse the binary form and transform to decimal notation, which is then called bit-reversed notation Example 121 The number 6 can be written as 6 = Therefore in binary Reversing the binary digits yields 011 Since 3 = we have 6 3 Since the sequence 0, 1 is the bit reversed order of 0, 1 and 0, 2, 1, 3 is the bit reversed order of 0, 1, 2, 3 we find that the matrices F 2 and F 4 are given by F 2 = 1 ( ) 1 1 = F F 4 = i 1 i 1 i 1 i Definition 136 By a Hadamard matrix of order n, H ( H n ), is meant a matrix whose elements are either +1 or 1 and for which HH T = H T H = ni n where I n is the n n unit matrix Thus, n 1/2 H is an orthogonal matrix Example 122 The 1 1, 2 2 and 4 4 Hadamard matrices are given by

33 Matrix Calculus 33 H 1 = (1) H 2 = ( ) 1 1 2F 2 = H 4,1 = H 4,2 = Note that the columns (or rows) considered as vectors are orthogonal to each other Sometimes the term Hadamard matrix is limited to the matrices of order 2 n These matrices have the property so that H 2 n = H T 2 n H 2 2 n = 2n I A recursion relation to find H 2 n using the Kronecker product will be given in sections 28 and 316 Definition 137 The Walsh-Hadamard transform is defined as Ẑ = HZ where H is an Hadamard matrix, where Z = (z 1, z 2,, z n ) T Since H 1 exists we have Z = H 1 Ẑ For example the inverse of H 2 is given by H 1 2 = 1 2 ( )

34 34 Matrix Calculus and Kronecker Product Exercises (1) Show that F = F T, F = (F ) T = F, F = F This means F and F are symmetric (2) Show that the sequence is the bit reversed order of 0, 8, 4, 12, 2, 10, 6, 14, 1, 9, 5, 13, 3, 11, 7, 15 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 (3) Find the eigenvalues of F4 = 1 1 ω ω 2 ω ω 2 ω ω 2 1 ω 3 ω 2 ω Derive the eigenvalues of F 4 (4) The discrete Fourier transform in one dimension can also be written as ˆx(k) = 1 N N 1 n=0 x(n) exp( ik2πn/n) where N N and k = 0, 1, 2,, N 1 Show that Let x(n) = N 1 k=0 ˆx(k) exp(ik2πn/n) x(n) = cos(2πn/n) where N = 8 and n = 0, 1, 2,, N 1 Find ˆx(k) (5) Find all 8 8 Hadamard matrices and their eigenvalues

35 Matrix Calculus Transformation of Matrices Let V be a vector space of finite dimension n and let A : V V be a linear transformation, represented by a (square) matrix A = (a ij ) relative to a basis (e i ) Relative to another basis (f i ), the same transformation is represented by the matrix B = Q 1 AQ where Q is the invertible matrix whose jth column vector consists of the components of the vector f j in the basis (e i ) Since the same linear transformation A can in this way be represented by different matrices, depending on the basis that is chosen, the problem arises of finding a basis relative to which the matrix representing the transformation is as simple as possible Equivalently, given a matrix A, that is to say, those which are of the form Q 1 AQ, with Q invertible, those which have a form that is as simple as possible Definition 138 If there exists an invertible matrix Q such that the matrix Q 1 AQ is diagonal, then the matrix A is said to be diagonalizable In this case, the diagonal elements of the matrix Q 1 AQ are the eigenvalues λ 1, λ 2,, λ n of the matrix A The jth column vector of the matrix Q consists of the components (relative to the same basis as that used for the matrix A) of a normalized eigenvector corresponding to λ j In other words, a matrix is diagonalizable if and only if there exists a basis of eigenvectors Example 123 The 2 2 matrix is diagonalizable with We find The 2 2 matrix cannot be diagonalized A = ( ) Q = 1 ( ) Q 1 AQ = B = ( ) ( )

36 36 Matrix Calculus and Kronecker Product For nondiagonalizable matrices Jordan s theorem gives the simplest form among all similar matrices Definition 139 A matrix A = (a ij ) of order n is upper triangular if a ij = 0 for i > j and lower triangular if a ij = 0 for i < j If there is no need to distinguish between the two, the matrix is simply called triangular Theorem 113 (1) Given a square matrix A, there exists a unitary matrix U such that the matrix U 1 AU is triangular (2) Given a normal matrix A, there exists a unitary matrix U such that the matrix U 1 AU is diagonal (3) Given a symmetric matrix A, there exists an orthogonal matrix O such that the matrix O 1 AO is diagonal The proof of this theorem follows from the proofs in section 111 The matrices U satisfying the conditions of the statement are not unique (consider, for example, A = I) The diagonal elements of the triangular matrix U 1 AU of (1), or of the diagonal matrix U 1 AU of (2), or of the diagonal matrix of (3), are the eigenvalues of the matrix A Consequently, they are real numbers if the matrix A is Hermitian or symmetric and complex numbers of modulus 1 if the matrix is unitary or orthogonal It follows from (2) that every Hermitian or unitary matrix is diagonalizable by a unitary matrix The preceding argument shows that if, O is an orthogonal matrix, there exists a unitary matrix U such that D = U OU is diagonal (the diagonal elements of D having modulus equal to 1), but the matrix U is not, in general, real, that is to say, orthogonal Definition 140 The singular values of a square matrix A are the positive square roots of the eigenvalues of the Hermitian matrix A A (or A T A, if the matrix A is real) Example 124 Let ( ) A = Then A A = Obviously A A is a positive semidefinite matrix The eigenvalues of A A are 0, 4, 9 Thus the singular values are 0, 2, 3

37 Matrix Calculus 37 They are always nonnegative, since from the relation A Au = λu, u 0, it follows that (Au) Au = λu u The singular values are all strictly positive if and only if the matrix A is invertible In fact, we have Au = 0 A Au = 0 u A Au = (Au) Au = 0 Au = 0 Definition 141 Two matrices A and B of type (m, n) are said to be equivalent if there exists an invertible matrix Q of order m and an invertible matrix R of order n such that B = QAR This is a more general notion than that of the similarity of matrices In fact, it can be shown that every square matrix is equivalent to a diagonal matrix Theorem 114 If A is a real, square matrix, there exist two orthogonal matrices U and V such that U T AV = diag(µ i ) and, if A is a complex, square matrix, there exist two unitary matrices U and V such that U AV = diag(µ i ) In either case, the numbers µ i 0 are the singular values of the matrix A The proof of this theorem follows from the proofs in section 111 If A is an n n matrix and U is an n n unitary matrix, then (m N) since UU = I n UA m U = (UAU ) m

38 38 Matrix Calculus and Kronecker Product Exercises (1) Find the eigenvalues and normalized eigenvectors of the matrix ( ) 2 1 A = 1 2 Then use the normalized eigenvectors to construct the matrix Q 1 such that Q 1 AQ is a diagonal matrix (2) Consider the skew-symmetric matrix A = Find the eigenvalues and the corresponding normalized eigenvectors Can one find an invertible 3 3 matrix S such that SAS 1 is a diagonal matrix? (3) Show that the matrix is not diagonalizable (4) Let O be an orthogonal matrix Show that there exists an orthogonal matrix Q such that Q 1 OQ is given by ( ) ( ) cos θ1 sin θ (1) (1) ( 1) ( 1) 1 cos θr sin θ r sin θ 1 cos θ 1 sin θ r cos θ r where denotes the direct sum (5) Let A be a real matrix of order n Show that a necessary and sufficient condition for the existence of a unitary matrix U of the same order and of a real matrix B (of the same order) such that U = A + ib (in other word, such that the matrix A is the real part of the matrix U) is that all the singular values of the matrix A should be not greater than 1

39 Matrix Calculus Permutation Matrices In this section we introduce permutation matrices and discuss their properties The connection with the Kronecker product is described in section 24 By a permutation σ of the set N := { 1, 2,, n } is meant a one-to-one mapping of N onto itself Including the identity permutation there are n! distinct permutations of N We indicate a permutation by σ(1) = i 1, σ(2) = i 2,, σ(n) = i n which is written as ( ) 1 2 n σ : i 1 i 2 i n The inverse permutation is designated by σ 1 Thus σ 1 (i k ) = k Let e T j,n denote the unit (row) vector of n components which has a 1 in the j-th position and 0 s elsewhere e T j,n := (0,, 0, 1, 0,, 0) Definition 142 By a permutation matrix of order n is meant a matrix of the form e T i 1,n e T i 2,n P = P σ = e T i n,n The i-th row of P has a 1 in the σ(i)-th column and 0 s elsewhere The j-th column of P has a 1 in the σ 1 (j)-th row and 0 s elsewhere Thus each row and each column of P has precisely one 1 in it We have P σ x 1 x 2 = x n x σ(1) x σ(2) x σ(n)

40 40 Matrix Calculus and Kronecker Product Example 125 Let σ : ( ) Then the permutation matrix is P σ = Example 126 The set of all 3 3 permutation matrices are given by the 6 matrices It can easily be proved that P σ P τ = P στ where the product of the permutations σ, τ is applied from left to right Furthermore, Hence (P σ ) = P σ 1 (P σ ) P σ = P σ 1P σ = P I = I n where I n is the n n unit matrix It follows that (P σ ) = P σ 1 = (P σ ) 1 Consequently, the permutation matrices form a group under matrix multiplication We find that the permutation matrices are unitary, forming a finite subgroup of the unitary group (see section 118 for more details on group theory) The determinant of a permutation matrix is either +1 or 1 The trace of an n n permutation matrix is in the set { 0, 1,, n 1, n}

41 Matrix Calculus 41 Exercises (1) Show that the number of n n permutation matrices is given by n! (2) Find all 4 4 permutation matrices (3) Show that the determinant of a permutation matrix is either +1 or 1 (4) A 3 3 permutation matrix P has tr(p ) = 1 and det(p ) = 1 What can be said about the eigenvalues of P? (5) Show that the eigenvalues λ j of a permutation matrix are λ j { 1, 1 } (6) Show that the rank of an n n permutation matrix is n (7) Consider the set of all n n permutation matrices How many of the elements are their own inverses, ie P = P 1? (8) Consider the 4 4 permutation matrix P = Find all the eigenvalues and normalized eigenvectors From the normalized eigenvectors construct an invertible matrix Q such that Q 1 P Q is a diagonal matrix (9) Let P 1 and P 2 be two n n permutation matrices Is [P 1, P 2 ] = 0, where [, ] denotes the commutator The commutator is defined by [P 1, P 2 ] := P 1 P 2 P 2 P 1 (10) Is it possible to find v R n such that vv T is an n n permutation matrix? (11) Let I n be the n n identity matrix and 0 n be the n n zero matrix Is the 2n 2n matrix ( ) 0n I n a permutation matrix? I n 0 n

42 42 Matrix Calculus and Kronecker Product 111 Matrix Decompositions Normal matrices have a spectral decomposition Let A be upper triangular with respect to an orthonormal basis {v 1,, v n } C n Then Av j = j a j,k v k k=1 for some a 1,1, a 2,1, a 2,2,, a n,n C Since {v 1,, v n } is orthonormal we have a j,k = v k Av j It follows that or equivalently We have and However ie a k,j = v j A v k, A v k = n a k,j v j j=k (Av j ) (Av j ) = (A v j ) (A v j ) = j a j,k 2 k=1 n a k,j 2 k=j (A v 1 ) (A v 1 ) = v 1 AA v 1 = v 1 A Av 1 = (Av j ) (Av j ) j n = a j,k 2 = a k,j 2, k=1 j 1 a j,k 2 = k=1 k=j n k=j+1 a k,j 2 For j = 1 we find a 2,1 = a 3,1 = = 0, for j = 2 we find a 3,2 = a 4,2 = = 0, etc Thus A is diagonal with respect to {v 1,, v n } As a consequence {v 1,, v n } form an orthonormal set of eigenvectors of A (and span C n ) Thus we can write (spectral theorem) n A = λ j v j vj j=1

43 Matrix Calculus 43 where Av j = λ j v j Since ( n ) k=1 k=1 v k e k,n e j,n = v j it follows that [( n ) ] ( n ) v k e k,n e j,n = e j,n v k e l,n = vk and consequently we write ( n A = v k ek,n) ( n n ) λ j e j,n e j,n v k e k,n = V DV where k=1 j=1 V = k=1 n v k e k,n k=1 is a unitary matrix The columns are the orthonormal eigenvectors of A and n D = λ k e k,n e k,n k=1 is a diagonal matrix of corresponding eigenvalues, ie the eigenvalue λ k in the k-th entry on the diagonal of D corresponds the the eigenvector v k which is the k-th column of V This decomposition is known as a spectral decomposition or diagonalization of A k=1 Example 127 Let A = ( ) 0 i i 0 The eigenvalues are λ 1 = 1, λ 2 = 1 with the corresponding normalized eigenvectors u 1 = 1 ( ) 1, u 2 = 1 ( ) 1 2 i 2 i Then

44 44 Matrix Calculus and Kronecker Product A λ 1 u 1 u 1 + λ 2 u 2 u 2 1 ( ) 1 (1, i) 1 ( ) 1 (1, i) 2 i 2 i 1 ( ) 1 i 1 ( ) 1 i 2 i 1 2 i 1 ( ) 0 i i 0 Example 128 Let A = Then the eigenvalues are λ 1 = 1, λ 2 = 1, λ 3 = 10 This means the eigenvalue λ = 1 is twofold The eigenvectors are 1 u 1 = 2, 1 u 2 = 0, 2 u 3 = We find that (u 1, u 3 ) = 0, (u 2, u 3 ) = 0, (u 1, u 2 ) = 1 However, the two vectors u 1 and u 2 are linearly independent Now we use the Gram-Schmidt algorithm to find orthogonal eigenvectors (see section 117) We choose such that Then u 1 = u 1, u 2 = u 2 + αu 1 α := (u 1, u 2 ) (u 1, u 1 ) = 1 5 u 2 = The normalized eigenvectors are u 1 = 1 1 2, u 2 = , u 3 =

45 Matrix Calculus 45 Consequently A = λ 1 u 1 u 1 + λ 2 u 2u 2 + λ 3 u 3 u 3 From the normalized eigenvectors we obtain the orthogonal matrix Therefore O = O T AO = where O T = O 1 and the eigenvalues of A are 1, 1 and 10 If the normal matrix A has multiple eigenvalues with corresponding nonorthogonal eigenvectors, we proceed as follows Let λ be an eigenvalue of multiplicity m Then the eigenvalues with their corresponding eigenvectors can be ordered as λ, λ,, λ, λ m+1,, λ n, u 1, u 2,, u m, u m+1,, u n The vectors u m+1,, u n are orthogonal to each other and to the rest What is left is to find a new set of orthogonal vectors u 1, u 2,, u m each being orthogonal to u m+1,, u n together with each being an eigenvector of A The procedure we use is the Gram-Schmidt in section 117 Let u 1 = u 1, u 2 = u 2 + αu 1 Then u 2 is an eigenvector of A, for it is a combination of eigenvectors corresponding to the same eigenvalue λ Also u 2 is orthogonal to u m+1,, u n since the latter are orthogonal to u 1 and u 2 What remains is to make u 2 orthogonal to u 1 ie to u 1 We obtain Next we set α = (u 1, u 2 ) (u 1, u 1 ) u 3 = u 3 + αu 1 + βu 2