Review of Linear Algebra

Review of Linear Algebra Dr Gerhard Roth COMP 40A Winter 05 Version

Linear algebra Is an important area of mathematics It is the basis of computer vision Is very widely taught, and there are many resources and books available We only focus on the basics Need to understand matrices, vectors and their basic operations

Linear Equations A system of linear equations, eg 4 4 can be written in matri form, m rows and n columns: 4 4 or in general: A b

Matri A matri is an m n array of numbers ij mn m m n n a a a a a a a a a a A Eample: 0 7 0 9 3 4 4 5 3 A

Matri Arithmetic Matri addition A mn B mn a ij b ij m n Matri multiplication A Matri transpose n mnbn p C c m p ij k T A a T ji T T A B A B T T AB B A T a ik b kj

Review of Vectors and Matrices Matrices An m n matri has m rows and n columns (height by width, y by ) Matri A = Matri B = Matri C = Q: Epress A,B,C as m n

Review of Vectors and Matrices Matrices An m n matri has m rows and n columns (height by width, y by ) Matri A = Matri B = Matri C = Q: Epress A,B,C as m n A: Matri A is 3 3 Matri B is 4 Matri C is 4 3

Review of Vectors and Matrices Matrices An m n matri has m rows and n columns (height by width, y by ) Matri A = Matri B = Matri C = Q: Epress A,B,C as m n A: Matri A is 3 3 Matri B is 4 Matri C is 4 3 Q: Which matrices can be multiplied by each other?

Review of Vectors and Matrices Matrices An m n matri has m rows and n columns (height by width, y by ) Matri A = Matri B = Matri C = Q: Epress A,B,C as m n A: Matri A is 3 3 Matri B is 4 Matri C is 4 3 Q: Which matrices can be multiplied by each other? Hint: # cols of first matri = # rows of second matri

Review of Vectors and Matrices Matrices An m n matri has m rows and n columns (height by width, y by ) Matri A = Matri B = Matri C = Q: Epress A,B,C as m n A: Matri A is 3 3 Matri B is 4 Matri C is 4 3 Q: Which matrices can be multiplied by each other? Hint: # cols of first matri = # rows of second matri Answer: B C C A

Review of Vectors and Matrices Calculate CA Matri C Matri A =?

Review of Vectors and Matrices Calculate CA Matri C Matri A =? 4 3 3 3 matri matri Q: But first What will size of result be?

Review of Vectors and Matrices Calculate CA Matri C Matri A =? 4 3 3 3 matri matri Q: But first What will size of result be? HINT: st matri must have as many columns as nd matri has rows to allow matri multiplication

Review of Vectors and Matrices Calculate CA Matri C Matri A =? 4 3 3 3 matri matri Q: But first What will size of result be? HINT: st matri must have as many columns as nd matri has rows to allow matri multiplication A: Result has same number of rows as first matri, and same number of columns as second matri

Review of Vectors and Matrices Calculate CA Matri C Matri A =? 4 3 3 3 matri matri Q: But first What will size of result be? HINT: st matri must have as many columns as nd matri has rows to allow matri multiplication A: Result has same number of rows as first matri, and same number of columns as second matri A: Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 3b + 4e + 5h 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 3b + 4e + 5h rc + sf + tl 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 3b + 4e + 5h rc + sf + tl uc + vf + yl 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 3b + 4e + 5h rc + sf + tl uc + vf + yl zc + f + l 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate CA Matri C Matri A = ra + sd + tg ua + vd + yg za + d + g 3a + 4d + 5g rb + se + th ub + ve + yh zb + e + h 3b + 4e + 5h rc + sf + tl uc + vf + yl zc + f + l 3c + 4f + 5l 4 3 3 3 matri matri Result is 4 3

Review of Vectors and Matrices Calculate BC Matri B Matri C =? 4 4 3 matri matri Q: But first What will size of result be? HINT: st matri must have as many columns as nd matri has rows to allow matri multiplication A: Result has same number of rows as first matri, and same number of columns as second matri

Review of Vectors and Matrices Calculate BC Matri B Matri C =? 4 4 3 matri matri Q: But first What will size of result be? HINT: st matri must have as many columns as nd matri has rows to allow matri multiplication A: Result has same number of rows as first matri, and same number of columns as second matri A: Result is 3

Review of Vectors and Matrices Calculate BC Matri B Matri C = 4 4 3 matri matri Result is 3 jr + ku + lz + m3 nr + ou + pz + q3 js + kv + l + m4 ns + ov + p + q4 jt + ky + l + m5 nt + oy + p + q5

Multiplication not commutative AB BA Matri multiplication is not commutative Eample: 9 9 7 5 6 3 5 0 7 3 8 3 5 5 6

Symmetric Matri We say matri A is symmetric if A T A Eample: A 4 4 5 A symmetric matri has to be a square matri

Inverse of matri If A is a square matri, the inverse of A, written A - satisfies: AA I A A I Where I, the identity matri, is a diagonal matri with all s on the diagonal I 0 0 I 3 0 0 0 0 0 0

Vectors matrices with one column n 5 3 eg The length or the norm of a vector is n 38 5 3 eg A vector is an n by matri, it is a column in our book

Vector Arithmetic v u v u v v u u v u v u v u v v u u v u Vector addition Vector subtraction Multiplication by scalar u u u u u

Dot Product (inner product) 5 3 a 3 4 b 9 5 3) ( 3 4 3 4 5 3 b a b a T n n T b a b a a b b a b a

Vectors: Dot Product (Inner product) T A B A B a b c e ad be cf d f Think of the dot product as a matri multiplication T A A A aa bb cc The magnitude is the dot product of a vector with itself A B A B cos() The dot product is also related to the angle between the two vectors

Vectors: Outer product Here u is an m by column, v is n by column so the outer product is m by n

Trace of Matri The trace of a matri: Tr( A) n i a ii

Orthogonal Matri A matri A is orthogonal if A T A I or A T A Eample: A cos sin sin cos

Matri Transformation (and projections) A matri-vector multiplication transforms one vector to another m n n m b A 6 6 4 7 3 4 3 5 Eample:

Coordinate Rotation r r y ' ' cos sin sin r cos r y

Vectors and Points Two points in a Cartesian coordinate system define a vector P(,y) Q(,y) y v y y v P(,y) Q(,y) y v A point can also be represented as a vector, defined by the point and the origin (0,0) P Q v 0 0 y y P 0 0 y y Q or v P Q Note: point and vector are different; vectors do not have positions

Least Squares When m>n for an m-by-n matri A, A b In this case, we look for an approimate solution We look for vector such that has no solution A b is as small as possible This is called the least squares solution

Least Squares Least squares solution of linear system of equations Normal equation: A T A A A b T A A is square and symmetric T b The Least Square solution makes A b minimal ( A A) T A T b

Least Square Fitting of a Line y d m y m c y d c y d c m y m y y d c The best solution c, d is the one that minimizes: ) ( ) ( m d m c y d c y A y E Line equations: y A

Least Square Fitting - Eample y P=(-,), P=(,), P3=(,3) Problem: find the line that best fit these three points: Solution: 3 d c d c d c 3 d c b A A A T T 6 5 6 3 d c or is The solution is and best line is 7 4 7 9, d c y 7 4 7 9

Homogeneous System m linear equations with n unknowns A = 0 Assume that m >= n- and rank(a) = n- Trivial solution is = 0 but there are more If we have a given solution, st A = 0 then c * is also a solution since A(c* ) = 0 Need to add a constraint on, Usually make a unit vector T Can prove that the solution of A = 0 satisfying this constraint is the eigenvector corresponding to the only zero eigenvalue of that matri A T A

Homogeneous System This solution can be computed using the eigenvector or SVD routine Find the zero eigenvalue (or the eigenvalue almost zero) Then the associated eigenvector is the solution And any scalar times is also a solution

Linear Independence A set of vectors is linear dependant if one of the vectors can be epressed as a linear combination of the other vectors A set of vectors is linearly independent if none of the vectors can be epressed as a linear combination of the other vectors n n k k k k k v v v v v

Eigenvalue and Eigenvector We say that is an eigenvector of a square matri A if A is called eigenvalue and is called eigenvector The transformation defined by A changes only the magnitude of the vector Eample: 5 5 5 4 3 4 4 3 and 5 and are eigenvalues, and and are eigenvectors

Properties of Eigen Vectors If,,, q are distinct eigenvalues of a matri, then the corresponding eigenvectors e,e,,e q are linearly independent A real, symmetric matri has real eigenvalues with eigenvectors that can be chosen to be orthonormal

SVD: Singular Value Decomposition T A UDV An mn matri A can be decomposed into: U is mm, V is nn, both of them have orthogonal columns: I U U T D is an mn diagonal matri I V V T Eample: 0 0 0 0 3 0 0 0 0 0 0 0 0 0 0 3 0 0

Singular Value Decomposition Any m by n matri A can be written as product of three matrices A = UDV T The columns of the m by m matri U are mutually orthogonal unit vectors, as are the columns of the n by n matri V The m by n matri D is diagonal, and the diagonal elements, are called the singular values i It is the case that n A matri is non-singular if and only all of the singular values are not zero The condition number of the matri is If the condition number is large, then then matri is almost singular and is called ill-conditioned 0 n

Singular Value Decomposition The rank of a square matri is the number of linearly independent rows or columns For a square matri (m = n) the number of non-zero singular values equals the rank of the matri If A is a square, non-singular matri, it s inverse can be written as T T where A VD The squares of the non zero singular values are the non-zero eigenvalues of both the n by n matri A T A T and of the m by m matri The columns of U are the eigenvectors of The columns of V are the eigenvectors of U AA A UDV T AA A A T