Linear Algebra and Matrices

Linear Algebra and Matrices 4 Overview In this chapter we studying true matrix operations, not element operations as was done in earlier chapters. Working with MAT- LAB functions should now be fairly routine. The challenge we are now faced with is understanding linear algebra concepts and the significance of linear algebra as applied to engineering problem solving. Since linear algebra is not a prerequisite for this course, we will proceed with caution as we introduce the various definitions and matrix properties. Matrix Operations In this section the fundamental matrix properties of transpose, dot product, multiplication, matrix power, and matrix polynomials are introduced. Transpose Consider a general rectangular matrix of size by columns) N M (rows Chapter 4: Overview 4 1

A [ ] a ij a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 a 41 a 42 a 43 The matrix transpose is the interchanging of the rows and columns of A and is denoted with superscript T In particular the element a ij is now located in the a ji position, that is a ij a ji a 11 a 21 a 31 a 41 A T a 12 a 22 a 32 a 42 a 13 a 23 a 33 a 43 As specific numerical case is B 12 34, B T 135 246 56 In MATLAB a matrix B is transposed via B» B [1 2 3;4 5 6] B 1 2 3 4 5 6 Chapter 4: Matrix Operations 4 2

» B' ans 1 4 2 5 3 6 Dot Product The dot product operator should be familiar from analytic geometry kˆ Angle between Plane Formed By Vectors A and B A θ B ˆ j A and B in the plane formed by AB ˆ i A B A B cosθ If the vectors A and B are defined as follows: A a 1 î + a 2 ĵ + a 3 kˆ B b 1 î + b 2 ĵ + b 3 kˆ the dot product A B dot product is the scalar A B 3 i 1 a i b i Chapter 4: Matrix Operations 4 3

In general, for an N-dimensional vector, dot product The MATLAB operator for dot product is dot(a,b) Matrix Multiplication A B i 1 a i b i For the matrix multiplication AB C to make sense we first must make sure the number of columns in B matches the number of rows in B We then proceed to multiply each row of A by the columns of B to form a new matrix of dimension rows of A by columns of B, that is A N Note: We might be able to form AB but not BA Mathematically matrix multiplication is such that each element in C is of the form of a row-column dot product Note: For i and j fixed k spans the ith row of A and the jth column of B, thus forming a dot product between a row and column of A and B respectively N M B M P C N P c ij N k 1 a ik b kj Chapter 4: Matrix Operations 4 4

A pictorial view of matrix multiplication as row column dot products R 1 A B R 2 R N C 1 C 2 C P Columns Rows M R 1 C 1 R 1 C 2 R 1 C P C R 2 C 1 R 2 C 2 R i C j R N C 1 R N C P Example: 1 0 2 A 25 1, B 1 4 2 03 1 5 2 1 and form AB C The dimensions conform so we may proceed The c 11 element of C is Chapter 4: Matrix Operations 4 5

c 11 251 11 2 1+ 5 ( 1 ) + 1 5 2 5 The c 22 is 0 c 22 03 1 4 0 0+ 3 4 + ( 1 ) 2 10 2 Solving for all of the remaining entries in like fashion we obtain In MATLAB matrix multiplication follows naturally, except we no not use * instead of.*» A [2 5 1; 0 3-1];» B [1 0 2; -1 4-2; 5 2 1];» C A*B C 2 22-5 -8 10-7 Matrix Powers We are familiar with the concept of scalar powers, e.g., a 5 a 3 What about matrix powers? C AB 2 22 5 8 10 7 3 3 3 3 3 243 Chapter 4: Matrix Operations 4 6

In MATLAB we know that 2 a 11 2 a 12 A.^2 2 a 21 2 a 33 that is each element of the matrix is squared To raise a matrix to a power using the rules of matrix algebra is A 2 A A A 4 A A A A For this operation work the matrix must be square (same number of rows as columns)» A [1 2; 3 4] A 1 2 3 4» [A^2 A^4] ans 7 10 199 290 15 22 435 634 Chapter 4: Matrix Operations 4 7

Matrix Polynomials Extending the idea of a matrix power a bit further we can talk about matrix polynomials For a scalar x we have previously defined the Nth degree polynomial fx ( ) as fx ( ) a 0 x N + a 1 x N 1 + + a N 1 x+ a N If x is a matrix, then the corresponding matrix polynomial involves matrix power rather than scalar powers MATLAB has a function, polyvalm(a,x) that evaluates a matrix polynomial much like polyval(a,x) evaluates a polynomial Note: The matrix x must be a square matrix A significant difference between the two is that while polyval returns a vector or a scalar depending upon what the input x is, polyvalm will always return a matrix of the same size as the input matrix x What good are they? They find use in the study of feedback control systems, in particular approximating the state transition matrix (see ECE 3510 (Linear Systems) and ECE 4510 (Feedback & Control) Chapter 4: Matrix Operations 4 8

Example: Variants on Practice! p. 130 Use MATLAB to define the following matrices. Then compute the specified matrices, if they exist. Also check your work via a hand calculation. A 2 1 0 1 3 0, B 1 3 1 5 C 3 2 1 2, D 12 0 2» A [2 1; 0-1; 3 0]; B [1 3; -1 5];» C [3 2; -1-2; 0 2]; D [1 2]; 1. Find CB 2» C*B^2 ans -18 98 14-62 -12 44 Chapter 4: Matrix Operations 4 9

Check this! CB 2 3 2 1 3 1 3 1 2 1 5 1 5 0 2 3 2 1 2 0 2 1 3 3+ 15 1 5 3 + 25 3 2 1 2 0 2 2 18 6 22 6 12 54+ 44 2+ 12 18 44 12 44 18 98 14 62 12 44 2. Find C T A» C'*A ans 6 4 10 4 Check this! C T 2 1 A 3 1 0 0 1 2 2 2 3 0 6 3+ 1 4+ 6 2 + 2 6 4 10 4 Chapter 4: Matrix Operations 4 10

Problems Solving Applied: Protein Molecular Weights (text p. 130 134) Here we wish to compute the molecular weight of a particular protein molecule Each protein molecule is composed of different amounts of up to 20 amino acids, each having a specific molecular weight An instrument called a protein sequencer can extract the order of the amino acids in a given protein molecule The protein sequencer output is a row vector containing 20 integers, say protein in MATLAB or when read from a data file is a vector or perhaps a matrix whose rows correspond to many sequencer outputs prot1 [0 0 0 1 0 2 0 0 0 0 0 1 1 0 0 1 0 0 0 0]; The value of each integer is the number of molecules of a corresponding amino acid (the list is alphabetical) If we put all of the corresponding amino acids weights in another vector of length 20, say amino_weight, we can find the weight of a particular protein molecule by forming the dot product between the sequencer output vector (matrix) and the amino acids weight vector prot1_weight dot(prot1,amino_weight); For more details see the text Chapter 4: Problems Solving Applied: Protein Molecular Weights (text p. 130 134) 4 11

Matrix Functions The first section of this chapter discussed matrix transpose and multiplication. The next level of complexity in matrix manipulations are matrix inverse, determinant, and finally eigenvectors and eigenvalues. Matrix Inverse and Rank The inverse of a matrix, A, if it exists, is a matrix, that A 1, such AA 1 I 1 0 0 1 0 0 1 0 0 1 Note: If AA 1 I, then also A 1 A I The true inverse of a matrix only makes sense if A is a square matrix, that is A is N N In general a special numerical algorithm is needed to find the inverse of a matrix For a 2 2 matrix we can write out the equations without too much trouble Chapter 4: Matrix Functions 4 12

Consider ab cd AB CD 10 01 (4.1) We must have aa+ bc 1 ab+ bd 0 ca+ dc 0 cb+ dd 1 (4.2) Solving for ABCand,,, D in terms of abcand,,, d by first eliminating C and D C D c --A d a --B b (4.3) Upon substitution aa cb b c d --A 1 A d a b --B 1 B ----------------- d ad bc and inserting (4.4) and (4.5) into (4.3) results in C c ad -----------------, bc D ----------------- b ad bc a ----------------- ad bc (4.4) (4.5) (4.6) Chapter 4: Matrix Functions 4 13

In summary, if A ab cd (4.7) A 1 1 ad ----------------- d b bc c a, for ad bc 0 (4.8) To check (4.8) we take (4.7) times (4.8) AA 1 ab ----------------- d ----------------- b ad bc ad bc cd ----------------- c ----------------- a ad bc ad bc ad ----------------- bc ---------------------- ab + ab ad bc ad bc ----------------- cd cd ---------------------- bc + ad ad bc ad bc 10 01 (4.9) For the case of the ad bc 0 2 2 we see that the inverse only exists if If a matrix is singular or ill-conditioned it does not have an inverse A related quantity to the inverse is the rank of a matrix If we view a matrix as corresponding to a system of equations (row-wise), the number of independent equations is the rank of the matrix Chapter 4: Matrix Functions 4 14

If the rank of a matrix is the number of rows in the matrix, then the matrix is nonsingular and the matrix inverse exists A special algorithm can be used to determine numerically the rank of a matrix MATLAB has functions for computing the inverse and rank of a square matrix rank(a) computes the rank of a matrix inv(a) computes the inverse of a matrix if it exists; an error message is printed if the inverse does not exist Example: Consider» A [2 1; 4 3];» rank(a) ans 2» inv(a) ans 1.5000-0.5000-2.0000 1.0000» A*inv(A) ans 1 0 0 1 A 21 43 Chapter 4: Matrix Functions 4 15

Check to see if the inverse agrees with the formula in (4.8) ----------- ----------- 1 6 4 6 4 4 ----------- ----------- 2 6 4 6 4 3 A 1 3 -- 1 ----- 2 2 2 1 (checks) The text author suggests that we first check to see if the inverse exists via the rank test, before attempting to compute the matrix inverse Determinants The determinant is a scalar quantity that arises in matrix problems The determinant of matrix A is denoted by The determinant often appears in problems dealing with the matrix inverse For a 2 2 matrix with A A a 11 a 12 A a 11 a 22 a 12 a 21 a 21 a 22 Note: for the 2 2 A 1 1 ----- A a 22 a 12 a 21 a 11 Chapter 4: Matrix Functions 4 16

A determinant of zero means the matrix is singular (inverse does not exist) For a 3 3 the determinant is computed according to the following rule A a 11 a 12 a 13 a 21 a 22 a 23 Add the product of the solid line linked entries and subtract the product of the dashed line linked entries (3 of each type) a 31 a 32 a 33 a 11 a 22 a 33 + a 12 a 23 a 31 + a 21 a 32 a 13 -a 31 a 22 a 13 a 21 a 12 a 33 a 32 a 23 a 11 The MATLAB function for computing the determinant of a square matrix is det(a) Example: A 2 2 and a 3 3» A [1 2; 3 4];» det(a) ans -2 A 12 A 4 6 2 34 Chapter 4: Matrix Functions 4 17

B 123 456 789 B 15 ( )9 ( ) + 26 ( )7 ( ) + 48 ( )3 ( ) -75 ( )3 ( ) 42 ( )9 ( ) 86 ( )1 ( ) 0» B [1 2 3; 4 5 6; 7 8 9];» det(b) ans 0» rank(b) ans 2 % Must be three to be invertable!» inv(b) Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND 2.055969e- 018. ans 1.0e+016 * -0.4504 0.9007-0.4504 0.9007-1.8014 0.9007-0.4504 0.9007-0.4504 Example: Variants on Practice! p. 136. Use MATLAB to define the following matrices. Then compute he specified matrices and determinants if they exist. For the case of 2 2 and 3 3 determinants check your answers by hand. Do not worry about calculating the 3 3 inverse by hand. Chapter 4: Matrix Functions 4 18

E 210 572, F 610 1 0 4 3 2 1 0 1 5» E [2 1 0; 5 7 2; 6 1 0];» F [-1 0 4; 3 2 1; 0-1 5]; 1. Find ( EF T ) 1» rank(e*f') ans 3 # Inverse exits» E*F' ans -2 8-1 3 31 3-6 20-1» inv(e*f') ans 0.4946 0.0652-0.2989 0.0815 0.0217-0.0163-1.3370 0.0435 0.4674 2. Find EF T» E*F' ans -2 8-1 3 31 3-6 20-1» det(e*f') ans -184 Check the determinant by hand calculating using the given above EF T Chapter 4: Matrix Functions 4 19

EF T 231 ( )( 1) + 83 ( )( 6) + 320 ( )( 1) -( 6) ( 31) ( 1) 38 ( )( 1) 20( 3 )( 2) 184 Problem Solving Applied: A Simple Pattern Recognition Model In the field of pattern recognition we typically have as input to the recognition scheme a set of measurements in vector form, say corresponding to a collection of sensor outputs or pattern features we observe By training the pattern classifier using representative samples from each pattern class, we then have a system that can take a new measurement vector and decide which class it belongs to Applications include image recognition (hand writing, finger prints, radar targets), voice recognition, etc. A statistical model for the measurement vectors of each class is that they are normally (Gaussian) distributed of dimension n, with a particular mean vector and covariance matrix C The covariance matrix tells us how each of the components in the measurement vectors are statistically related to one another for a given class Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 20

C c 11 c 12 c 1n c 21 c 22 c 2n c n1 c n2 c nn Using the model described above the pattern recognition scheme computes a recognition function r ij ( x) for each input vector x The so called maximum likelihood ratio test results in r ij ( x) x T C 1 1 ( m i m j ) --m ( 2 i + m j ) T C 1 ( m i m j ) where the and are mean vectors associated with each m i m j class, C is the covariance matrix, here assumed to be the same for all classes; there are M different classes to choose from By sequencing through all the ij pairs, except i j, we chose the largest and say if a particular observation r kj ( x) is largest then x belongs to class k { 12,,, M} Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 21

Eigenvectors and Eigenvalues Consider A an n n (square) matrix which satisfies AX λx where X is a column vector ( n 1) and λ is a scalar (4.10) Assuming X is nonzero we call X an eigenvector (proper values) of matrix A and λ the corresponding eigenvalue We can rewrite (4.10) as a set of homogeneous equations where I is an n n ( A λi)x 0 identity matrix (4.11) Since X is nonzero, we can only obtain zero on the left side of (4.11) if and only if det( A λi) A λi 0 (4.12) When multiplied out (4.12) becomes what is known as the characteristic equation corresponding to matrix A The roots of the characteristic equation are the eigenvalues of A Since the characteristic polynomial is nth degree, there are n total eigenvalues (roots) The MATLAB function eig(a) returns a column vector containing the eigenvalues of A Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 22

Example: Consider A 1 1 2 4 We must solve 1 1 2 4 λ 10 1 λ 1 01 2 4 λ ( 1 λ)4 ( λ) + 2 λ 2 5λ + 6 The eigenvalues are Check using MATLAB» A [1 1; -2 4];» eig(a) ans 2 3 5 + 5 2 4 16 ( ) λ 1 ------------------------------------------- 3 2 λ 5 25 24 2 ------------------------------ 2 2 Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 23

We obtain the corresponding eigenvectors by solving 1 λ i 1 2 4 λ i x 1 0 x 2 0, for i 1,2 Case 1: λ 1 3 which implies 2 1 2 1 x 1 0 x 2 0 or x 2 2x 1 Thus the eigenvector corresponding to X 1 α 1 1 2 λ 1 is of the form where α 1 can be any number λ2 2 Case 2: For 1 1 2 2 x 1 0 x 2 0 or x 2 x 1 Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 24

Thus the eigenvector corresponding to X 2 α 1 2 1 λ 1 is of the form where α 2 can be any number Symmetric Matrices with Distinct Eigenvalues: If the matrix A is n n and symmetric, that is A A T, and the corresponding eigenvalues are distinct, then the eigenvectors will be orthogonal. Orthogonal eigenvectors means that the dot product of say X 1 with X 2 is zero, while the dot product of X 1 with itself gives use the length of squared X 1 Orthonormal eigenvectors means that they are orthogonal, with the additional feature being that they have unit length Note: We can normalize an eigenvector to unit have length as follows: Now, x X 1 i x 2 X i X i X i 1 1 -------------------- x 1 2 2 x 1 + x 2 x 2 Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 25

An expanded version of the MATLAB function eig() is [Q,d] eig(a); where Q is a matrix whose columns are the eigenvectors of A normalized to unity length and d is a diagonal matrix whose entries are the corresponding eigenvalues of A If A is symmetric and has distinct eigenvalues, then Example: Variant to Practice! p. 139 Consider the matrix 1. Determine the eigenvalues of A» A [0 0-2; 0-2 0; -2 0 3];» eig(a) ans -2-1 4 Q Q T I Identity Matrix Thus λ 1 2, λ 2 1, and λ 3 4 A 0 0 2 0 2 0 2 0 3 Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 26

2. Determine the corresponding set of orthonormal eigenvectors Note: The eigenvalues are distinct and A is symmetric, so we expect to have orthogonal (with MATLAB orthonormal) eigenvectors» [Q,d] eig(a);» Q Q 0-0.8944-0.4472-1.0000 0 0 0-0.4472 0.8944 Thus Are they orthonormal? If so» Q*Q' ans 0.8944 0.4472 X 1 01, X 2 0, X 3 0 0 0.4472 0.8944 QQ T 1.0000 0 0 0 1.0000 0 0 0 1.0000 3. Compute the determinant A λi and verify that for each eigenvalue it is zero» det(a - (-2)*eye(3)) To be more precise we may ans 0 use d(1,1) λ 1 I Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 27

» det(a - (-1)*eye(3)) ans 0» det(a - (4)*eye(3)) ans 0 4. Show that AQ Ad where Q is the matrix containing the eigenvectors as columns and d is the matrix containing the corresponding eigenvalues on the main diagonal and zeros elsewhere» A*Q ans» Q*d 0 0.8944-1.7889 2.0000 0 0 0 0.4472 3.5777 ans 0 0.8944-1.7889 2.0000 0 0 0 0.4472 3.5777 % They agree! Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 28

Decompositions Matrices, square or rectangular, can be decomposed or factored into products of special matrices Triangular Factorization: For A a square matrix we can always write it as a product of a lower triangular matrix and an upper triangular matrix (LU factorization). A LU. This is accomplished by performing row and column operations on A Assuming U is computed first it may be that interchanging rows is required, thus L may not be in strict lower triangular form The MATLAB function [L,U] lu(a); takes A and returns the upper triangular factor matrix U and the lower triangular factor matrix L such that LU A Example: A 1 2 3 0 4 6 1 4 7» A [1 2 3; 0 4 6; 1-4 7];» [L,U] lu(a);» L Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 29

L 1.0000 0 0 0-0.6667 1.0000 1.0000 1.0000 0» U U» L*U 1.0000 2.0000 3.0000 0-6.0000 4.0000 0 0 8.6667 ans 1 2 3 0 4 6 1-4 7 QR Factorization: Here factorization of matrix A produces an orthonormal matrix (matrix Q is orthonormal if QQ T I) times an upper triangular matrix, R. A QR The matrix A does not have to be square The MATLAB function [Q,R] qr(a) returns orthonormal matrix Q and upper triangular matrix R such that A QR Note: for A m n, Q is m m and R is m n Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 30

Example: Use A from the above example.» [Q,R] qr(a);» Q Q -0.7071-0.5145-0.4851 0-0.6860 0.7276-0.7071 0.5145 0.4851» R R» Q*R -1.4142 1.4142-7.0711 0-5.8310-2.0580 0 0 6.3059 ans 1.0000 2.0000 3.0000 0 4.0000 6.0000 1.0000-4.0000 7.0000 rectangu- Singular Value Decomposition (SVD): For an m lar matrix A, the SVD is the triple matrix product A USV where both U and V are orthogonal matrices and S is a diagonal matrix composed of the singular values of A. n Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 31

The MATLAB function [U,S,V} svd(a) returns U as a m m matrix, S as a m n matrix, and V as a n n matrix Chapter 4: Problem Solving Applied: A Simple Pattern Recognition Model 4 32