Lecture 3 Linear Algebra Background

Transcription

1 Lecture 3 Linear Algebra Background Dan Sheldon September 17, 2012

2 Motivation Preview of next class: y (1) w 0 + w 1 x (1) 1 + w 2 x (1) w d x (1) d y (2) w 0 + w 1 x (2) 1 + w 2 x (2) w d x (2) d... y (N) w 0 + w 1 x (N) 1 + w 2 x (N) w d x (N) d After linear algebra y Xw

3 Linear Algebra in ML Linear Algebra Succinct notation for models and algorithms Numerical tools (save coding!) w = (X T X) 1 X T y Inspiration for new models and problems: Netflix

4 Netflix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice Bob 5 2 Carol 5 David 5 5 Eve 5 4 Matrix completion problem, matrix factorization

5 Today s Topics Matrices Vectors Matrix-Matrix multiplication (and special cases) Tranpose Inverse

6 Matrices A matrix is an rectangular array of numbers A = When A has m rows and n columns, we say that: A is an m n matrix A R m n The entry in row i and column j is denoted A ij

7 Matrices Example A R 3 2 A 11 = 101 A 32 = A 22 = A 23 = A =

8 Vectors A vector is an n 1 matrix: x = We write x R n (instead of x R n 1 ) The ith entry is x i

9 Vectors Example x = x R 4 x 1 = x 4 =

10 Addition If two matrices have the same size, we can add them by adding corresponding elements [ ] [ ] [ ] = Subtraction is similar Matrices of different sizes cannot be added or subtracted

11 Scalar Multiplication A scalar x R is a real number (i.e., not a vector) Scalar times matrix: e.g., 2, 3, π, 2, 1.843,... 2 [ ] 1 3 = 2 0 (multiply each entry by the scalar) [ 2 ] 6 4 0

12 Matrix-Matrix Multiplication Can multiply two matrices if their inner dimensions match A R m n, B R n p The product has entries C = AB R m p n C ij = A ik B kj k=1

13 Matrix-Matrix Multiplication C ij = n A ik B kj k=1 Move along ith row of A and jth row of B. Multiply corresponding entries, then add. c 11 c 12 c 13 a 11 a 12 [ ] c 21 c 22 c 23 c 31 c 32 c 33 = a 21 a 22 b11 b 12 b 13 a 31 a 32 b 21 b 22 b 23 c 41 c 42 c 43 a 41 a 42 c 32 = a 31 b 12 + a 32 b 22

14 Matrix-Matrix Multiplication Example [ ] [ ] A =, B = [ ] [ ] [ ] AB = =

15 Multiplication Properties Associative Distributive Not commutative (AB)C = A(BC) A(B + C) = AB + AC (B + C)D = BD + CD AB BA

16 Matrix-Vector Multiplication A (worthy) special case of matrix-matrix multiplication: A R m n, x R n y = Ax R m Definition n y i = A ij x j j=1

17 Matrix-Vector Multiplication y i = n A ij x j j=1 y 1 a 11 a 12 y 2 y 3 = a 21 a 22 a 31 a 32 y 4 a 41 a 42 [ x1 x 2 ] y 3 = a 31 x 1 + a 32 x 2

18 Matrix-Vector Multiplication Example A = [ 1 1 Ax = 0 3 [ ] 6.5 Az = 4.5 [ ] [ ] 1 1 1, x = ] [ 1 1 ] = [ ] 2 3 z = [ ] 8 1.5

19 Transpose Transposition of a matrix swaps the rows and columns [ ] [ ] 1 1 A =, A T 1 0 = Definition: Let A R m n The transpose A T R n m has entries (A T ) ij = A ji.

20 Transpose Example 3 2 A = 1 0 A T = 1 4 [ 3 1 ] Example 1 x = 3 x T = [ ] 2

21 Dot product A special special-case of matrix-matrix multiplication Let x, y be vectors of same size (x, y R n ). Their dot product is x T y = n x i y i i=1 y 1 = [ ] y 2 x 1 x 2... x n. y n

22 Vector Norm The norm of a vector x = x x x2 n = x T x Geometric interpretation: length of the vector

23 Transpose Properties Transpose of transpose (A T ) T = A Transpose of sum Transpose of product (A + B) T = A T + B T (AB) T = B T A T

24 Identity The identity matrix I R n n has entries { 1 i = j I ij = 0 i j, I 1 1 = [1], I 2 2 = For any A, B of appropriate dimensions [ ] , I = IA = A BI = B

25 Inverse The inverse A 1 R n n of a square matrix A R n n satisfies AA 1 = I = A 1 A Compare to division of scalars xx 1 = 1 = x 1 x Not all matrices are invertible [ ] 0 0 E.g., A not square, A = [0], A =, many more 0 0

26 Inverse Example A = Is B the inverse of A? [ ] 1 0, B = 0 2 [ ] Example A = Verify on your own. [ ] A 1 = [ ]

27 Inverse Properties Inverse of inverse Inverse of product (A 1 ) 1 = A (AB) 1 = B 1 A 1 Inverse of transpose (A 1 ) T = (A T ) 1 := A T

28 What You Should Know Definitions of matrices and vectors Meaning of matrix multiplication Systems of equations matrix-vector equations Properties of multiplication Properties of inverse, transpose Get familiar with these as course goes on