Matrices A brief introduction

Matrices A brief introduction Basilio Bona DAUIN Politecnico di Torino Semester 1, 2014-15 B. Bona (DAUIN) Matrices Semester 1, 2014-15 1 / 41

Definitions Definition A matrix is a set of N real or complex numbers organized in m rows and n columns, with N = mn a 11 a 12 a 1n a A = 21 a 22 a 2n a ij [ ] a ij i = 1,...,m j = 1,...,n a m1 a m2 a mn A matrix is always written as a boldface capital letter viene as in A. To indicate matrix dimensions we use the following symbols A m n A m n A F m n A F m n where F = R for real elements and F = C for complex elements. B. Bona (DAUIN) Matrices Semester 1, 2014-15 2 / 41

Transpose matrix Given a matrix A m n we define a transpose matrix the matrix obtained exchanging rows and columns a 11 a 21 a m1 A T n m = a 12 a 22 a m2...... a 1n a 2n a mn The following property holds (A T ) T = A B. Bona (DAUIN) Matrices Semester 1, 2014-15 3 / 41

Square matrix A matrix is said to be square when m = n A square n n matrix is upper triangular when a ij = 0, i > j a 11 a 12 a 1n 0 a A n n = 22 a 2n...... 0 0 a nn If a square matrix is upper triangular its transpose is lower triangular and viceversa a 11 0 0 A T n n = a 12 a 22 0...... a 1n a 2n a nn B. Bona (DAUIN) Matrices Semester 1, 2014-15 4 / 41

Symmetric matrix A real square matrix is said to be symmetric if A = A T, or A A T = O In a real symmetric matrix there are at least n(n+1) independent 2 elements. If a matrix K has complex elements k ij = a ij +jb ij (where j = 1) its conjugate is K with elements k ij = a ij jb ij. Given a complex matrix K, an adjoint matrix K is defined, as the conjugate transpose K = K T = K T A complex matrix is called self-adjoint or hermitian when K = K. Some textbooks indicate this matrix as K or K H B. Bona (DAUIN) Matrices Semester 1, 2014-15 5 / 41

Diagonal matrix A square matrix is said to be diagonal if a ij = 0 for i j a 1 0 0 0 a A n n = diag(a i ) = 2 0...... 0 0 a n A diagonal matrix is always symmetric. B. Bona (DAUIN) Matrices Semester 1, 2014-15 6 / 41

Matrix Algebra Matrices form an algebra, i.e., a vector space endowed with the product operator. The main operations are: product by a scalar, sum, matrix product Product by a scalar a 11 a 12 a 1n αa 11 αa 12 αa 1n a αa = α 21 a 22 a 2n...... = αa 21 αa 22 αa 2n...... a m1 a m2 a mn αa m1 αa m2 αa mn Sum a 11 +b 11 a 12 +b 12 a 1n +b 1n a A+B = 21 +b 21 a 22 +b 22 a 2n +b 2n...... a m1 +b m1 a m2 +b m2 a mn +b mn B. Bona (DAUIN) Matrices Semester 1, 2014-15 7 / 41

Sum Properties A+O = A A+B = B+A (A+B)+C = A+(B+C) (A+B) T = A T +B T The neutral element O is called null or zero matrix. The matrix difference is defined introducing the scalar α = 1: A B = A+( 1)B. B. Bona (DAUIN) Matrices Semester 1, 2014-15 8 / 41

Matrix Product Matrix product The operation follows the rule row by column : the generic c ij element of the product matrix C m p = A m n B n p is The following identity holds: c ij = n a ik b kj k=1 α(a B) = (αa) B = A (αb) B. Bona (DAUIN) Matrices Semester 1, 2014-15 9 / 41

Product Properties A B C = (A B) C = A (B C) A (B+C) = A B+A C (A+B) C = A C+B C (A B) T = B T A T In general: the matrix product is NOT commutative: A B B A, except some particular case; A B = A C does not imply B = C, except some particular case; A B = O does not imply A = O or B = O, except some particular case. B. Bona (DAUIN) Matrices Semester 1, 2014-15 10 / 41

Identity Matrix The neutral element wrt the matrix product is calledidentity matrix usually written as I n or I when there are no ambiguities on the dimension. Identity matrix 1 0 0 0 0 I =...... 0 0 1 Given a rectangular matrix A m n the following hold A m n = I m A m n = A m n I n B. Bona (DAUIN) Matrices Semester 1, 2014-15 11 / 41

Matrix Power Given a square matric A R n n, the k-th power is k A k = A l=1 One matrix is said to be idempotent iff A 2 = A A k = A. B. Bona (DAUIN) Matrices Semester 1, 2014-15 12 / 41

Matrix Trace Trace The trace of a square matrix A n n is the sum of its diagonal elements tr(a) = Trace satisfy the following properties tr(αa+βb) = αtr(a)+β tr(b) tr(ab) = tr(ba) tr(a) = tr(a T ) tr(a) = tr(t 1 AT) for T non-singular (see below for explanation) n k=1 a kk B. Bona (DAUIN) Matrices Semester 1, 2014-15 13 / 41

Row/column cancellation Given the square matrix A R n n, we call A (ij) R (n 1) (n 1) the matrix obtained deleting the la i-the row and the j-the columns of A. Example: given A = deleting row 2, column 3 we obtain A (23) = 1 5 3 2-6 4 9-7 7 4-8 2 0 9-2 3 1 5 2 7 4 2 0 9 3 B. Bona (DAUIN) Matrices Semester 1, 2014-15 14 / 41

Minors and Determinant A minor of order p of a generic matrix A m n is defined as the il determinant D p of a square sub-matrix obtained selecting any p rows and p columns of A m n There exist as many minors as the possible choices of p on m rows and p on n columns The definition of determinant will be given soon. Given a matrix A m n its principal minors of order k are the determinants D k, with k = 1,,min{m,n}, obtained selecting the first k rows and k columns of A m n. B. Bona (DAUIN) Matrices Semester 1, 2014-15 15 / 41

Example Given the 4 3 matrix 1 3 5 7 2 4 A = 1 3 2 8 1 6 we compute a generic minor D 2, for example that obtained selecting the first and rows 1 and 3 and columns 1 and 2 (in red). First we form the submatrix D = and then we compute the determinant [ ] 1 3 1 3 D 2 = det(d) = 3 1 ( 3 1) = 0 B. Bona (DAUIN) Matrices Semester 1, 2014-15 16 / 41

Example Given the 4 3 matrix 1 3 5 7 2 4 A = 1 3 2 8 1 6 we compute the principal minors minors D k,k = 1,2,3, D 1 = 1 [ ] 1 3 D 2 = det = 23 7 2 D 3 = det 1 3 5 7 2 4 = 161 1 3 2 B. Bona (DAUIN) Matrices Semester 1, 2014-15 17 / 41

Complement We call the complement C rc of a generic (r,c) element of a square matrix A n n the determinant of the matrix obtained deleting its r-the row and the c-th column, i.e., deta (rc) D rc = deta (rc). The cofactor of the (r,c) element of a square matrix A n n is the signed product C rc = ( 1) r+c D rc B. Bona (DAUIN) Matrices Semester 1, 2014-15 18 / 41

Example Given the 3 3 matrix some of the cofactors are A = 1 2 3 4 5 6 7 8 9 C 11 = ( 1) 2 (45 48) = 3 C 12 = ( 1) 3 (36 42) = 6 C 31 = ( 1) 4 (12 15) = 3 B. Bona (DAUIN) Matrices Semester 1, 2014-15 19 / 41

Adjugate/Adjunct/Adjoint The cofactor matrix of A is the n n matrix C whose (i,j) entry C ij is the (i,j) cofactor of A C ij = ( 1) i+j D ij The adjugate or adjunct or adjoint of a square matrix A is the transpose of C, that is, the n n matrix whose (i,j) entry is the (j,i) cofactor of A, A adj ij = C ji = ( 1) i+j D ji The adjoint matrix of A is the matrix X that satisfies the following equality AX = XA = det(a)i B. Bona (DAUIN) Matrices Semester 1, 2014-15 20 / 41

Example Given the 3 3 matrix its adjoint is A = 1 3 2 4 6 5 7 9 8 A adj = 3 6 3 3 6 3 6 12 6 B. Bona (DAUIN) Matrices Semester 1, 2014-15 21 / 41

Determinant The determinant of a square matrix A x n can be computed in different ways. Choosing any row i, the definition by row is: det(a) = n a ik ( 1) i+k det(a (ik) ) = k=1 n a ik A ik k=1 Choosing any column j, the definition by column is:: det(a) = n a kj ( 1) k+j det(a (kj) ) = k=1 n a kj A kj Since these definitions are recursive, involving the determinants of increasingly smaller minors, we define the determinant of a 1 1 matrix A = a, simply as deta = a. k=1 B. Bona (DAUIN) Matrices Semester 1, 2014-15 22 / 41

Properties The determinant has the following properties: det(a B) = det(a)det(b) det(a T ) = det(a) det(ka) = k n det(a) if one exchanges s rows or columns of A, obtaining A s, then det(a s ) = ( 1) s det(a) if A has two or more rows/columns equal or proportional, then det(a) = 0 if A has a row/column that can be obtained as a linear combination of other rows/columns, then det(a) = 0 if A is triangular, then det(a) = n i=1 a ii if A is block-triangular, with p blocks A ii on the diagonal, then det(a) = p i=1 deta ii B. Bona (DAUIN) Matrices Semester 1, 2014-15 23 / 41

Rank and Singularity A matrix A is singular if det(a) = 0. The rank (or characteristic) of a matrix A m n is the number ρ(a m n ), defined as the largest integer p for which at least a minor D p is non-zero. The following properties hold: ρ(a) min{m,n} if ρ(a) = min{m,n}, A is said to be full rank if ρ(a) < min{m,n}, the rank of the matrix is said to drop if A n n and deta < n the matrix is not full rank, or is rank deficient ρ(a B) min{ρ(a),ρ(b)} ρ(a) = ρ(a T ) ρ(a A T ) = ρ(a T A) = ρ(a) B. Bona (DAUIN) Matrices Semester 1, 2014-15 24 / 41

Invertible Matrix A square matrix A R n n it is said to be invertible or non singular if the inverse A 1 n n exists, such that AA 1 = A 1 A = I n A matrix is invertible iff ρ(a) = n, i.e., it is full-rank; this is equivalent to have a non zero determinant det(a) 0. The inverse is computed as: A 1 = 1 det(a) Aadj The following properties hold: (A 1 ) 1 = A; (A T ) 1 = (A 1 ) T. The inverse, if exists, allows to solve the following matrix equation with respect to the unknown x, as y = Ax x = A 1 y. B. Bona (DAUIN) Matrices Semester 1, 2014-15 25 / 41

Matrix derivative If a square matrix A n n (t) has elements function of a variable (e.g., the time t) a ij (t), then the matrix derivative is [ ] d d A(t) = Ȧ(t) = dt dt a ij(t) = [ȧ ij (t)] If A(t) rank is full, ρ(a(t)) = n for every t, then the derivative of the inverse is d dt A(t) 1 1 = A (t)ȧ(t)a(t) 1 Notice that, since the inverse is a nonlinear function, the derivative of the inverse is in general different from the inverse of the derivative [ da(t) dt ] 1 d [ A(t) 1 ] dt B. Bona (DAUIN) Matrices Semester 1, 2014-15 26 / 41

Example Given the square matrix we have The inverse of A is A(t) = d A(t) = Ȧ(t) = dt A(t) 1 = [ ] cosθ(t) sinθ(t) sin θ(t) cos θ(t) [ ] sinθ(t) cosθ(t) θ cosθ(t) sinθ(t) [ ] cosθ(t) sinθ(t) = A(t) sinθ(t) cosθ(t) T and in this particular case the two inverses are equal [ da(t) dt ] 1 = d [ A(t) 1 ] = dt [ ] cosθ(t) sinθ(t) sinθ(t) cosθ(t) B. Bona (DAUIN) Matrices Semester 1, 2014-15 27 / 41

Matrix Decomposition Given a real matrix A R m n, the following products give symmetric matrices A T A R n n AA T R m m Given a square matrix A, it is always possible to decompose it in a sum of two matrices where is symmetric, and A = A s +A ss A s = 1 2 (A+AT ) A ss = 1 2 (A AT ) is skew-symmetric. B. Bona (DAUIN) Matrices Semester 1, 2014-15 28 / 41

Similarity Transformation Similarity transformation Given a square matrix A R n n and a square nonsingular matrix T R n n, the matrix B R n n obtained as B = T 1 AT or B = TAT 1 is called similar to A, and the transformation T is called similarity transformation. B. Bona (DAUIN) Matrices Semester 1, 2014-15 29 / 41

Eigenvalues and Eigenvectors If it is possible to find a nonsingular matrix U such that A is similar to the diagonal matrix Λ = diag(λ i ) A = UΛU 1 AU = UΛ and if we call u i the i-th column of U, we have U = [ u 1 u 2 u n ] Au i = λ i u i This relation is the well known formula defining eigenvectors and eigenvalues of A. The λ i are the eigenvalues of A and the u i are the eigenvectors of A. B. Bona (DAUIN) Matrices Semester 1, 2014-15 30 / 41

Eigenvalues and Eigenvectors Eigenvalues and eigenvectors Given a square matrix A n n, the matrix eigenvalues λ i are the (real or complex) solutions of the characteristic equation det(λi A) = 0 det(λi A) is a polynomial in λ, called the characteristic polynomial of A. If the eigenvalues are all distinct, we call eigenvectors the vectors u i that satisfy the following identity Au i = λ i u i B. Bona (DAUIN) Matrices Semester 1, 2014-15 31 / 41

Geometrical interpretation If the eigenvalues are not all distinct, we obtain the generalized eigenvectors, whose characterization goes beyond the scope of this presentation. From a geometrical point of view, the eigenvectors represent those particular directions in the R n space, i.e., the domain of the linear transformation represented by A, that remain invariant under the transformation, while the eigenvalues give the scaling constants along these same directions. The set of the matrix eigenvalues is usually indicated as Λ(A) or {λ i (A)}; the set of the matrix eigenvectors is indicated as {u i (A)}. In general, they are normalized, i.e., {u i (A)} = 1 B. Bona (DAUIN) Matrices Semester 1, 2014-15 32 / 41

Eigenvalues Properties Given a square matrix A and its eigenvalues, {λ i (A)}, the following properties hold true {λ i (A+cI)} = {(λ i (A)+c)} {λ i (ca)} = {(cλ i (A)} Given a triangular matrix a 11 a 12 a 1n 0 a 22 a 2n......, 0 0 a nn a 11 0 0 a 21 a 22 0...... a n1 a n2 a nn its eigenvalues are the elements on the main diagonal {λ i (A)} = {a ii }; the same is true for a diagonal matrix. B. Bona (DAUIN) Matrices Semester 1, 2014-15 33 / 41

Other properties Given a square matrix A n n and its eigenvalues {λ i (A)}, the following hold true n det(a) = and tr(a) = i=1 n i=1 Given any invertible similarity transformation T, B = T 1 AT the eigenvalues of A are invariant to it, i.e., {λ i (B)} = {λ i (A)} λ i λ i B. Bona (DAUIN) Matrices Semester 1, 2014-15 34 / 41

Modal matrix If we build a matrix M whose columns are the normalized eigenvectors u i (A) M = [ u 1 u n ] then the similarity transformation with respect to M results in the diagonal matrix λ 1 0 0 0 λ Λ = 2 0...... = M 1 AM 0 0 λ n M is the modal matrix. If A is symmetric, all its eigenvalues are real and we have In this case M is orthonormal. Λ = M T AM B. Bona (DAUIN) Matrices Semester 1, 2014-15 35 / 41

Singular Value decomposition (SVD) Given a matrix A R m n, having rank r = ρ(a) s, with s = min{m,n}, it can be decomposed (factored) in the following way: A = UΣV T = s σ i u i v T i (1) i=1 The decomposition is characterized by three elements: σ i u i v i as follows. B. Bona (DAUIN) Matrices Semester 1, 2014-15 36 / 41

SVD Characterization σ i (A) 0 are the singular values and are equal to the non-negative square roots of the eigenvalues of the symmetric matrix A T A: {σ i (A)} = { λ i (A T A)} σ i 0 ordered in decreasing order σ 1 σ 2 σ s 0 if r < s there are r positive singular values; the remaining ones are zero σ 1 σ 2 σ r > 0; σ r+1 = = σ s = 0 U is a orthonormal (m m) matrix U = [ u 1 u 2 u m ] containing the eigenvectors u i of AA T B. Bona (DAUIN) Matrices Semester 1, 2014-15 37 / 41

SVD Characterization V is a (n n) orthonormal matrix V = [ v 1 v 2 v n ] whose columns are the eigenvectors v i of the matrix A T A Σ is a (m n) matrix, with the following structure if m < n Σ = [ Σ s O ] if m = n Σ = Σ s if m > n Σ = [ Σs O where Σ s = diag(σ i ) is diagonal with dimension s s, having the singular values on the diagonal. ]. B. Bona (DAUIN) Matrices Semester 1, 2014-15 38 / 41

Example Given its SVD is where A = [ ] 1 3 2 4 6 5 A = UΣV T ρ(a) = 2 [ ] 0.3863 0.9224 U = 0.9224 0.3863 [ ] 9.5080 0 0 Σ = 0 0.7729 0 0.4287 0.8060 V = 0.7039 0.5812 0.4082 0.5663 0.1124 0.8165 B. Bona (DAUIN) Matrices Semester 1, 2014-15 39 / 41

Alternative SVD Decomposition Alternately, we can decompose the A matrix as follows: where A = [ P P ] [ ] Σr O }{{} O O U }{{} Σ [ Q T Q T ] }{{} V T P is an orthonormal m r matrix, P is an orthonormal m (m r) matrix; = PΣ r Q T (2) Q is an orthonormal n r matrix, Q T is an orthonormal n (n r) matrix; Σ r is an diagonal r r matrix, whose diagonal elements are the positive singular values σ i > 0, i = 1,,r. B. Bona (DAUIN) Matrices Semester 1, 2014-15 40 / 41

Rank The rank r = ρ(a) of A is equal to the number r s of nonzero singular values. Given any matrix A R m n, both A T A and AA T are symmetric, with identical positive singular values and differ only for the number of zero singular values. B. Bona (DAUIN) Matrices Semester 1, 2014-15 41 / 41