Linear Algebra. Introduction. Marek Petrik 3/23/2017. Many slides adapted from Linear Algebra Lectures by Martin Scharlemann

Linear Algebra Introduction Marek Petrik 3/23/2017 Many slides adapted from Linear Algebra Lectures by Martin Scharlemann

Midterm Results Highest score on the non-r part: 67 / 77 Score scaling: Additive in order for at least of 20% class gets over 95% Independent scaling for graduate and undergraduate students Not like grading on a curve5 Questions that were most often wrong Some of the same questions will be on the final (practice tests)

Methods Covered Until Now Supervised learning 1. Linear regression 2. Logistic regression 3. LDA, QDA 4. Naive Bayes 5. Lasso 6. Ridge regression 7. Subset selection Unsupervised learning 1. PCA 2. K-means clustering 3. Hierarchical clustering 4. Expectation maximization

Important Concepts We Have Covered Training and test sets Cross-validation and leave-one-out Maximum likelihood Maximum a posteriori

Remainder of the Course In ISL: 1. Support vector machines 2. Boosting and bagging 3. Advanced nonlinear features Not in ISL: 1. Recommender systems 2. Methods for time series analysis 3. Reinforcement learning 4. Deep learning 5. Graphical models

Today: Linear Algebra Crucial in many machine learning algorithms Which ones? 1. Linear regression 2. Logistic regression 3. LDA, QDA 4. Naive Bayes 5. Lasso 6. Ridge regression 7. Subset selection 8. PCA 9. K-means clustering 10. Hierarchical clustering 11. Expectation maximization

Suggested Linear Algebra Books Strang, G. (2016). Introduction to linear algebra (5th ed.) http://math.mit.edu/ gs/linearalgebra/ Watch online lectures: https://ocw.mit.edu/courses/mathematics/ 18-06-linear-algebra-spring-2010/ video-lectures/ Hefferon, J. (2017). Linear algebra (3rd ed.). Free PDF: http://joshua.smcvt.edu/linearalgebra/

Linear Equation Equation of a line: y m 1 } b x

Linear Equation Equation of a line: y y = mx + b m 1 } b x

Linear Equation Equation of a line: y m 1 } b x y = mx + b y mx = b

Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b

Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b mx 1 + x 2 = b

Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b mx 1 + x 2 = b ( a 2 0) a 1 x 1 + a 2 x 2 = b

Linear Equation Linear equation in variables x 1,..., x n : a 1 x 1 + a 2 x 2 +... + a n x n = b where a 1,..., a n and maybe b are all known in advance.

Linear Equation Linear equation in variables x 1,..., x n : a 1 x 1 + a 2 x 2 +... + a n x n = b where a 1,..., a n and maybe b are all known in advance. Solution is a list s 1,..., s n of numbers so a 1 s 1 + a 2 s 2 +... + a n s n = b Example 1: For the equation a 1 x 1 + a 2 x 2 = b of a line, given above: The pair s 1, s 2 is a solution the point (s 1, s 2 ) is on the line.

Example 2 Converting grades to the standard scale: F = 0 grade 4 = A, let x 1 be your first midterm grade x 2 be your second midterm grade x 3 be your grade on the final x 4 be your homework & quiz grade x 5 be your i-clicker grade

System of Linear Equations A system of linear equations is a bunch of linear equations:

System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x 2 +... + a 1n x n = b 1 a 21 x 1 + a 22 x 2 +... + a 2n x n = b 2. a m1 x 1 + a m2 x 2 +... + a mn x n = b m

System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x 2 +... + a 1n x n = b 1 a 21 x 1 + a 22 x 2 +... + a 2n x n = b 2. a m1 x 1 + a m2 x 2 +... + a mn x n = b m A solution to the system is a list s 1,..., s n R that is simultaneously a solution to all m equations.

System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x 2 +... + a 1n x n = b 1 a 21 x 1 + a 22 x 2 +... + a 2n x n = b 2 a m1 x 1 + a m2 x 2 +... + a mn x n = b m A solution to the system is a list s 1,..., s n R that is simultaneously a solution to all m equations. That is, all m equations are true when x 1 = s 1, x 2 = s 2,..., x n = s n..

Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2

Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Solution two lines intersect: x 2 x 1

Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Solution two lines intersect: x 2 Example: 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 (x 1, x 2 ) = (1, 1) x 1

Line Configurations Other possibilities: x 2 x 1 x 2 x 1

Line Configurations Other possibilities: x 2 1x 1 + 2x 2 = 3 1x 1 + 2x 2 = 4 x 1 inconsistent x 2 x 1

Line Configurations Other possibilities: x 2 1x 1 + 2x 2 = 3 1x 1 + 2x 2 = 4 x 1 inconsistent x 2 1x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 x 1 redundant

3 Possible Line Configurations Upshot: There are exactly three possibilities

3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution

3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution

3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions

3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions Some goals: Figure out which possibility applies. Write down the solution if it s unique.

Matrix The critical information is in the a ij, b i.

Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x 1 +... + a 1n x n = b 1 a 21 x 1 +... + a 2n x n = b 2. a m1 x 1 +... + a mn x n = b m

Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x 1 +... + a 1n x n = b 1 a 21 x 1 +... + a 2n x n = b 2 a m1 x 1 +... + a mn x n = b m. a 11 a 12... a 1n a 21 a 22... a 2n...... a m1 a m2... a mn m n coefficient matrix

Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x 1 +... + a 1n x n = b 1 a 21 x 1 +... + a 2n x n = b 2 a 11 a 12... a 1n a 21 a 22... a 2n...... a m1 a m2... a mn m n coefficient matrix a m1 x 1 +... + a mn x n = b m. a 11 a 12... a 1n b 1 a 21 a 22... a 2n b 2....... a m1 a m2... a mn b m m (n + 1) augmented matrix

Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations

Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Multiply an equation by c 0

Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Multiply an equation by c 0 Replace one equation by itself plus a multiple of another equation

Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Matrix operations: Reorder the rows Multiply an equation by (interchange) c 0 Replace one equation by itself plus a multiple of another equation

Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Matrix operations: Reorder the rows (interchange) Multiply an equation by c 0 Replace one equation by itself plus a multiple of another equation Multiply a row by c 0 (scaling) Replace one row by itself plus a multiple of another row (replacement) Grand strategy: Do this until the equations are easy to solve.

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 [ 1 2 3 2 1 3 ] Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 [ 1 2 3 2 1 3 ] [ 1 2 3 0 3 3 ] Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1 [ 1 2 3 2 1 3 ] [ 1 2 3 0 3 3 [ 1 0 1 0 3 3 [ 1 0 1 0 1 1 ] ] ]

Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5

Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5 Subtract 2 first from second Add 3 first to third: L1 : x 1 3x 2 2x 3 = 6 L2 2L1 : 2x 2 + x 3 = 4 3L1 + L3 : 3x 2 + 2x 3 = 13

Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5 1 3 2 6 2 4 3 8 3 6 8 5 Subtract 2 first from second Add 3 first to third: L1 : x 1 3x 2 2x 3 = 6 L2 2L1 : 2x 2 + x 3 = 4 3L1 + L3 : 3x 2 + 2x 3 = 13

Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13

Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13 Add 3 2 L2 to L1 and L3: L1 + 3 2 L2 : x 1 1 2 x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 + 3 2 L2 : 7 2 x 3 = 7

Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13 Add 3 2 L2 to L1 and L3: L1 + 3 2 L2 : x 1 1 2 x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 + 3 2 L2 : 7 2 x 3 = 7 1 3 2 6 0 2 1 4 0 3 2 13 1 0 1 2 0 0 2 1 4 0 0 7 2 7 Multiply L3 by 2 7 1 0 1 2 0 0 2 1 4 0 0 1 2 Move on to Step three! - the third column

Step three: L1 : x 1 1 2 x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 : x 3 = 2 Subtract L3 from L2; add 1 2 L3 to L1: L1 + 1 2 L3 : x 1 = 1 L2 1 2 L3 : 2x 2 = 6 L3 : x 3 = 2

Step three: L1 : x 1 1 2 x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 : x 3 = 2 Subtract L3 from L2; add 1 2 L3 to L1: L1 + 1 2 L3 : x 1 = 1 L2 1 2 L3 : 2x 2 = 6 L3 : x 3 = 2 1 0 1 2 0 0 2 1 4 0 0 1 2 1 0 0 1 0 2 0 6 0 0 1 2 Multiply L2 by 1 2 1 0 0 1 0 1 0 3 0 0 1 2 We re done: (x 1, x 2, x 3 ) = (1, 3, 2)

What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4

What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION

What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION

What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 Subtract 2L1 from L2: x 1 + 2x 2 = 3 0= 0 redundant SOLUTIONS INFINITE

Configurations If the system has 3 variables, an equation determines a plane in R 3.

Configurations If the system has 3 variables, an equation determines a plane in R 3. So a solution will be the intersection of 3 planes: P 1, P 2, P 3 :

Configurations If the system has 3 variables, an equation determines a plane in R 3. So a solution will be the intersection of 3 planes: P 1, P 2, P 3 : P 1 P 3 P 3 P2 P 1 P 2 inconsistent equations (no solution)

Question? George tells you the system of equations has exactly three solutions. 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10

Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George. B) George is probably wrong.

Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. a m

Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. Can add two m-vectors in the obvious way, or multiply a vector by a real number: a 1 b 1 a 1 + b 1 a 2 b 2 a 1 + b 2 a 3 + b 3 = a 1 + b 3 ;... a m b m a 1 + b m a m

Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. Can add two m-vectors in the obvious way, or multiply a vector by a real number: a 1 b 1 a 1 + b 1 a 1 ca 1 a 2 b 2 a 1 + b 2 a 2 ca 2 a 3 + b 3 = a 1 + b 3 ; c a 3 = ca 3..... a m b m a 1 + b m a m ca m Do not multiply two vectors together like this. a m

Vector Operations Examples: 1 4 5 2 + 5 = 7 ; 3 6 9

Vector Operations Examples: 1 4 5 1 1 2 + 5 = 7 ; 1 2 = 2 3 6 9 3 3

Vector Operations Examples: 1 4 5 1 1 2 + 5 = 7 ; 1 2 = 2 3 6 9 3 3 1 0 1 3 2 7 2 + 2 1 = 3 4 0

Vector Operations Examples: 1 4 5 1 1 2 + 5 = 7 ; 1 2 = 2 3 6 9 3 3 1 0 1 3 2 7 2 + 2 1 = 3 4 0 3 1 7 0 + 2 1 3 2 7 2 + 2 ( 1) = 3 3 7 4 + 2 0

Vector Operations Examples: 1 4 5 1 1 2 + 5 = 7 ; 1 2 = 2 3 6 9 3 3 1 0 1 3 2 7 2 + 2 1 = 3 4 0 3 1 7 0 + 2 1 5 3 2 7 2 + 2 ( 1) = 10 3 3 7 4 + 2 0 19

Question: 1 0 1 7 2 3 2 2 1 = 3 4 0

Question: 1 0 1 7 2 3 2 2 1 = 3 4 0 5 5 5 5 5 A) 10 B) 9 C) 10 D) 19 E) 10 19 10 9 10 19

Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 u + v v u x 1

Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; parallelogram rule; x 2 u + v v u x 1

Application Gives more flexible way to describe a line.

Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + t d, t R Argument: x 2 p d x 1

Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + 1 d, t = 1 Argument: x 2 p p + d d x 1

Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + 1 d, t = 1 Argument: x 2 p p - d d x 1

Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + t d, t R Argument: x 2 p p + td d x 1

Question: Pictured are points u, v R 2.

Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 u v x 1

Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 E D C B A u v x 1

Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 u v x 1

Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2-3v -v u v x 1

Properties of Vectors For all w, x, y R n and s, t R, we have the following.

Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative)

Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative)

Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z

Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0

Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0 t( x + y) = t x + t y (distributive law)

Linear Combination Definition Suppose {t 1, t 2... t k } are all real numbers. The vector y = t 1 v 1 + + t k v k is called a linear combination of the vectors { v 1, v 2... v k }.

Linear Combination Definition Suppose {t 1, t 2... t k } are all real numbers. The vector y = t 1 v 1 + + t k v k is called a linear combination of the vectors { v 1, v 2... v k }. Sample problem: Given vectors { a 1, a 2... a n, b} in R m, find real numbers {t 1, t 2... t n } so that t 1 a 1 + + t n a n = b.

Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a 1 + + t n a n = b :

Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a 1 + + t n a n = b : Let b 1 b 2 b = b 3 ; a j =. a 1j a 2j a 3j b m a mj. for 1 j n

Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a 1 + + t n a n = b : Let b 1 b 2 b = b 3 ; a j =. a 1j a 2j a 3j b m a mj. for 1 j n Then for any 1 i m, the i th row of the equation becomes: t 1 a i1 + t 2 a i2 + + t n a in = b i or a i1 t 1 + a i2 t 2 + + a in t n = b i

Linear Combination In other words, solving t 1 a 1 + + t n a n = b : is the same as solving this system of m linear equations: a 11 t 1 +... + a 1n t n = b 1 a 21 t 1 +... + a 2n t n = b 2. a m1 t 1 +... + a mn t n = b m We just learned how to do this!

Linear Combination Solving t 1 a 1 + + t n a n = b : is the same as solving a system of m linear equations.

Linear Combination Solving t 1 a 1 + + t n a n = b : is the same as solving a system of m linear equations. The system has augmented matrix a 11 a 12... a 1n b 1 a 21 a 22... a 2n b 2....... a m1 a m2... a mn b m

Example Suppose 0 0 6 a 1 = 2 4 a 2 = 2 4 a 3 = 1 1 a 4 = 8 8 1 and want to find c 1, c 2, c 3, c 4 so that 12 c 1 a 1 + c 2 a 2 + c 3 a 3 + c 4 a 4 = 4 13 23 0 6 10 26

Example This translates to the system of linear equations whose augmented matrix is 0 0 6 0 12 2 2 1 6 4 4 4 1 10 13 8 8 1 26 23

Example This translates to the system of linear equations whose augmented matrix is 0 0 6 0 12 2 2 1 6 4 4 4 1 10 13 8 8 1 26 23 which reduces to: 1 1 0 0 3 2 0 0 1 0 2 0 0 0 1 1 2 0 0 0 0 0

Example This translates to the system of linear equations whose augmented matrix is 0 0 6 0 12 2 2 1 6 4 4 4 1 10 13 8 8 1 26 23 which reduces to: 1 1 0 0 3 2 0 0 1 0 2 0 0 0 1 1 2 0 0 0 0 0 and so has general solution c 2 = anything, c 1 = 3 2 c 2, c 3 = 2, c 4 = 1 2

Span Definition Given a collection { v 1, v 2,..., v k } of vectors in R m, the set of all linear combinations of these vectors, that is all vectors that can be written as c 1 v 1 + + c k v k for some c 1,..., c k R is denoted Span { v 1,..., v k } and is called the span of { v 1,..., v k }. Easy example: If k = 1 so there is only one vector v, then Span{ v} is just all vectors that are multiples of v. That is, Span{ v} = {c v c R}

Span Picturing the span when m = 2, 3:

Span Picturing the span when m = 2, 3: When there is only one vector v then Span{ v} = {c v c R} is just the line that contains both 0 (take c = 0) and v (take c = 1).

Span Picturing the span when m = 2, 3: When there is only one vector v then Span{ v} = {c v c R} is just the line that contains both 0 (take c = 0) and v (take c = 1). With two vectors u and v, Span{ u, v} = {c 1 u + c 2 v} pictured via the parallelogram rule (Span = entire plane; c i 0 highlighted): u (3u+5v)/2 v

Span So we can think of the set of all solutions as 3 2 1 0 + Span 1 0 0 2 1 2

Span So we can think of the set of all solutions as 3 2 1 0 + Span 1 0 0 2 1 2 So we can picture the solution as a line in the direction of the second vector, going through the point given by the first vector (but in R 4!) x 2 p p + td d x 1

Matrix Multiplication Definition The linear combination is abbreviated. x 1 a 1 + + x k a k x 1 [ ] x 2 a1 a 2 a 3... a k x 3. Here [ a 1 a 2 a 3... a k ] is the matrix A with i th column a i. x k

Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b 2 +... a k b k.. b k

Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b 2 +... a k b k.. so b k 7 [ ] 1 2 3 4 3 1 = 2

Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b 2 +... a k b k.. so b k 7 [ ] 1 2 3 4 3 1 = 2 1 7 + ( 2) 3 + 3 1 + ( 4) 2 = 7 6 + 3 8 = 4

Question 3 [ ] 1 2 3 4 7 2 = 1 A) -3 B) -6 C) -9 D) 6 E) 9

Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] 2 3 1 2 2 3 = 2 + 1 + 4 4 2 5 4 2 1 4 [ ] 1 5

Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] 2 3 1 2 2 3 = 2 + 1 + 4 4 2 5 4 2 so [ 2 3 1 4 2 5 1 4 ] 2 1 = 4 [ ] 1 5 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26

Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] 2 3 1 2 2 3 = 2 + 1 + 4 4 2 5 4 2 so [ 2 3 1 4 2 5 1 4 ] 2 1 = 4 [ ] 1 5 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26 Think of doing the simple case on each row of the matrix A: R 2.

Matrix-vector Multiplication Apply simple case to first row: [ ] 2 3 1 2 = 4 2 5 1 4 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26

Matrix-vector Multiplication Apply simple case to first row: [ ] 2 3 1 2 = 4 2 5 1 4 Apply simple case to second row: [ 2 3 1 4 2 5 ] 2 1 = 4 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26

Matrix-vector Multiplication Apply simple case to first row: [ ] 2 3 1 2 = 4 2 5 1 4 Apply simple case to second row: [ 2 3 1 4 2 5 ] 2 1 = 4 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26 [ ] [ ] 2 2 + 3 1 1 4 3 = 4 2 2 1 + 5 4 26 R 2.

Matrix-vector Multiplication Further abbreviation: Use vector notation: then x 1 x 2 x 3 = x. x k

Matrix-vector Multiplication Further abbreviation: Use vector notation: then x 1 x 2 x 3 = x. x k x 1 a 1 + + x k a k = [ ] x 2 a 1 a 2 a 3... a k x 3 = A x. x 1 x k

Matrix-vector Multiplication Summary: For A an m n matrix, and a vector x R n, multiplication A x is defined and gives a vector in R m. Multiplication has two important properties: For any vectors u, v R n, A( u + v) = A u + A v For any vector u R n and any c R, A(c u) = c(a u). For example: u 1 u 2 u 3 A( u + v) = [ ] v 2 a 1 a 2 a 3... a n ( + v 3 ) =.. u n v 1 v n

Question [ ] 3 1 2 3 4 7 2 1 4 3 2 = 1 A) [ ] 7 9 B) [ ] 9 4 C) [ ] 9 1 D) [ ] 4 9 E) π e

Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a 1 + + x n a n = b.

Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a 1 + + x n a n = b. Might arise from the question: Is b in Span{ a 1, a 2... a n }? New: translation to matrix equation: Given m n matrix A and b R m find a vector x R n so that A x = b where A = [ a 1 a 2 a 3... a n ].

Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line.

Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line. On the other hand, if they don t lie on the same line, then Span{ x 1, x 2 } consists of an entire plane. (The plane determined by the heads of the vectors and 0.) So the span of two vectors may be a plane, or it could be something simpler: either a line, or even just 0 in the case that x 1 = 0 = x 2.

Background Similarly, if three non-trivial vectors x 1, x 2, x 3 all lie on the same line, then Span{ x 1, x 2, x 3 } is just that line.

Background Similarly, if three non-trivial vectors x 1, x 2, x 3 all lie on the same line, then Span{ x 1, x 2, x 3 } is just that line. If they don t all lie on the same line, but lie on the same plane, then Span{ x 1, x 2, x 3 } is just that plane.

Linear Independence How do we put these ideas into math lingo, so we can be precise? Definition A set of vectors { v 1,..., v k } in R m is linearly independent if and only if the only solution to the equation x 1 v 1 + + x k v k = 0 is the solution x i = 0 for 1 i k. Conversely, the set of vectors { v 1,..., v k } is linearly dependent if there are real numbers c 1,..., c k, not all zero, such that c 1 v 1 + + c k v k = 0. Idea: if the set is linearly independent, then span is big as possible.

In pictures: v 2 v 1 v 3 { v 1, v 2, v 3 } linearly dependent.

v v 1 2 v 3 { v 1, v 2, v 3 } linearly independent.

Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set.

Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set. Argument: Suppose, say, { v 1, v 2, v 3 } { v 1,..., v k } is linearly dependent.

Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set. Argument: Suppose, say, { v 1, v 2, v 3 } { v 1,..., v k } is linearly dependent. This means that there are c 1, c 2, c 3 not all 0 so that But then, c 1 v 1 + c 2 v 2 + c 3 v 3 = 0 c 1 v 1 + c 2 v 2 + c 3 v 3 + 0 v 4 +... + 0 v n = 0 so the whole set is linearly dependent.

Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent:

Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a 1 + + x n a n = 0.

Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a 1 + + x n a n = 0. But this is a homogeneous system of linear equations - we can answer that question! Construct associated matrix of column vectors: a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn Reduce to echelon form and see if there are any free variables.

Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent.

Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent. For if n > m then the associated matrix of column vectors a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn is longer than it is high. When reduced to echelon form, there must be free variables: $ 0 $ 0 0 0 $ 0 0 0 0 $ 0 0 0 0 0 $

Question Question: Is this set of vectors linearly dependent, or linearly independent? 2 3 3, 4 4 5 A) Dependent since they are 2 vectors in R 3 and 2 < 3. B) Independent since they are 2 vectors in R 3 and 2 < 3. C) Dependent because one is a multiple of the other. D) Independent because neither is a multiple of the other. E) Independent because one is a multiple of the other.

Question Question: Is this set of vectors linearly dependent, or linearly independent? 1 2 3 2, 4, 5 5 10 13

Question Question: Is this set of vectors linearly dependent, or linearly independent? 1 2 3 2, 4, 5 5 10 13 A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. Answer: The second is a multiple of the first, so that pair alone is linearly dependent.

Question Question: Is this set of vectors linearly dependent, or linearly independent? 1 1 4 1, 3, 9 0 2 5

Question Question: Is this set of vectors linearly dependent, or linearly independent? 1 1 4 1, 3, 9 0 2 5 A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. E) I can t tell.

Answer: For this triple of vectors 1 1 4 1, 3, 9 0 2 5 we want to determine whether the homogeneous system of linear equations: 1 1 4 x 1 1 3 9 x 2 = 0 0 2 5 x 3 has a non-trivial solution.

Answer: For this triple of vectors 1 1 4 1, 3, 9 0 2 5 we want to determine whether the homogeneous system of linear equations: 1 1 4 x 1 1 3 9 x 2 = 0 0 2 5 x 3 has a non-trivial solution. Consider the associated column matrix (no need to augment): 1 1 4 1 1 4 1 3 9 0 2 5 0 2 5 0 2 5

Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. For example, suppose v 1 Span{ v 2, v 3,..., v k }. That means there are c 2, c 3,..., c k so that v 1 = c 2 v 2 + c 3 v 3 +... + c k v k But this vector equation can be rewritten 1 v 1 + c 2 v 2 + c 3 v 3 +... + c k v k = 0.

Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v 3 +... + c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get v 1 + c 2 c 1 v 2 + c 3 c 1 v 3 +... + c k c 1 v k = 0

Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v 3 +... + c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get and this can be rewritten v 1 + c 2 c 1 v 2 + c 3 c 1 v 3 +... + c k c 1 v k = 0 v 1 = c 2 c 1 v 2 c 3 c 1 v 3... c k c 1 v k

Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v 3 +... + c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get and this can be rewritten v 1 + c 2 c 1 v 2 + c 3 c 1 v 3 +... + c k c 1 v k = 0 v 1 = c 2 c 1 v 2 c 3 c 1 v 3... c k c 1 v k Hence v 1 Span{ v 2, v 3,..., v k }.

Connection with Span In pictures: v 2 v 1 v 3 v 1 Span{ v 2, v 3 } and { v 1, v 2, v 3 } linearly dependent.

Connection with Span v v 1 2 v 3 v 1 / Span{ v 2, v 3 } and { v 1, v 2, v 3 } linearly independent.