Lecture 3 Linear Algebra Background Dan Sheldon September 17, 2012
Motivation Preview of next class: y (1) w 0 + w 1 x (1) 1 + w 2 x (1) 2 +... + w d x (1) d y (2) w 0 + w 1 x (2) 1 + w 2 x (2) 2 +... + w d x (2) d... y (N) w 0 + w 1 x (N) 1 + w 2 x (N) 2 +... + w d x (N) d After linear algebra y Xw
Linear Algebra in ML Linear Algebra Succinct notation for models and algorithms Numerical tools (save coding!) w = (X T X) 1 X T y Inspiration for new models and problems: Netflix
Netflix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob 5 2 Carol 5 David 5 5 Eve 5 4 Matrix completion problem, matrix factorization
Today s Topics Matrices Vectors Matrix-Matrix multiplication (and special cases) Tranpose Inverse
Matrices A matrix is an rectangular array of numbers 101 10 A = 54 13 10 47 When A has m rows and n columns, we say that: A is an m n matrix A R m n The entry in row i and column j is denoted A ij
Matrices Example A R 3 2 A 11 = 101 A 32 = A 22 = A 23 = A = 101 10 54 13 10 47
Vectors A vector is an n 1 matrix: x = 8 2.4 1 10 We write x R n (instead of x R n 1 ) The ith entry is x i
Vectors Example x = 8 2.4 1 10 x R 4 x 1 = x 4 =
Addition If two matrices have the same size, we can add them by adding corresponding elements [ ] [ ] [ ] 1 2 3 5 4 7 + = 3 4 1 0 2 4 Subtraction is similar Matrices of different sizes cannot be added or subtracted
Scalar Multiplication A scalar x R is a real number (i.e., not a vector) Scalar times matrix: e.g., 2, 3, π, 2, 1.843,... 2 [ ] 1 3 = 2 0 (multiply each entry by the scalar) [ 2 ] 6 4 0
Matrix-Matrix Multiplication Can multiply two matrices if their inner dimensions match A R m n, B R n p The product has entries C = AB R m p n C ij = A ik B kj k=1
Matrix-Matrix Multiplication C ij = n A ik B kj k=1 Move along ith row of A and jth row of B. Multiply corresponding entries, then add. c 11 c 12 c 13 a 11 a 12 [ ] c 21 c 22 c 23 c 31 c 32 c 33 = a 21 a 22 b11 b 12 b 13 a 31 a 32 b 21 b 22 b 23 c 41 c 42 c 43 a 41 a 42 c 32 = a 31 b 12 + a 32 b 22
Matrix-Matrix Multiplication Example [ ] [ ] 1 1 3 2 A =, B = 0 3 1 0 [ ] [ ] [ ] 1 1 3 2 4 2 AB = = 0 3 1 0 3 0
Multiplication Properties Associative Distributive Not commutative (AB)C = A(BC) A(B + C) = AB + AC (B + C)D = BD + CD AB BA
Matrix-Vector Multiplication A (worthy) special case of matrix-matrix multiplication: A R m n, x R n y = Ax R m Definition n y i = A ij x j j=1
Matrix-Vector Multiplication y i = n A ij x j j=1 y 1 a 11 a 12 y 2 y 3 = a 21 a 22 a 31 a 32 y 4 a 41 a 42 [ x1 x 2 ] y 3 = a 31 x 1 + a 32 x 2
Matrix-Vector Multiplication Example A = [ 1 1 Ax = 0 3 [ ] 6.5 Az = 4.5 [ ] [ ] 1 1 1, x = 0 3 1 ] [ 1 1 ] = [ ] 2 3 z = [ ] 8 1.5
Transpose Transposition of a matrix swaps the rows and columns [ ] [ ] 1 1 A =, A T 1 0 =. 0 3 1 3 Definition: Let A R m n The transpose A T R n m has entries (A T ) ij = A ji.
Transpose Example 3 2 A = 1 0 A T = 1 4 [ 3 1 ] 1 2 0 4 Example 1 x = 3 x T = [ 1 3 2 ] 2
Dot product A special special-case of matrix-matrix multiplication Let x, y be vectors of same size (x, y R n ). Their dot product is x T y = n x i y i i=1 y 1 = [ ] y 2 x 1 x 2... x n. y n
Vector Norm The norm of a vector x = x 2 1 + x2 2 +... + x2 n = x T x Geometric interpretation: length of the vector
Transpose Properties Transpose of transpose (A T ) T = A Transpose of sum Transpose of product (A + B) T = A T + B T (AB) T = B T A T
Identity The identity matrix I R n n has entries { 1 i = j I ij = 0 i j, I 1 1 = [1], I 2 2 = For any A, B of appropriate dimensions [ ] 1 0 0 1 0, I 0 1 3 3 = 0 1 0. 0 0 1 IA = A BI = B
Inverse The inverse A 1 R n n of a square matrix A R n n satisfies AA 1 = I = A 1 A Compare to division of scalars xx 1 = 1 = x 1 x Not all matrices are invertible [ ] 0 0 E.g., A not square, A = [0], A =, many more 0 0
Inverse Example A = Is B the inverse of A? [ ] 1 0, B = 0 2 [ ] 1 0 1 0 2 Example A = Verify on your own. [ ] 1 1 0 3 A 1 = [ ] 1 1 3 1 0 3
Inverse Properties Inverse of inverse Inverse of product (A 1 ) 1 = A (AB) 1 = B 1 A 1 Inverse of transpose (A 1 ) T = (A T ) 1 := A T
What You Should Know Definitions of matrices and vectors Meaning of matrix multiplication Systems of equations matrix-vector equations Properties of multiplication Properties of inverse, transpose Get familiar with these as course goes on