Linear Algebra. Introduction. Marek Petrik 3/23/2017. Many slides adapted from Linear Algebra Lectures by Martin Scharlemann

Transcription

1 Linear Algebra Introduction Marek Petrik 3/23/2017 Many slides adapted from Linear Algebra Lectures by Martin Scharlemann

2 Midterm Results Highest score on the non-r part: 67 / 77 Score scaling: Additive in order for at least of 20% class gets over 95% Independent scaling for graduate and undergraduate students Not like grading on a curve5 Questions that were most often wrong Some of the same questions will be on the final (practice tests)

3 Methods Covered Until Now Supervised learning 1. Linear regression 2. Logistic regression 3. LDA, QDA 4. Naive Bayes 5. Lasso 6. Ridge regression 7. Subset selection Unsupervised learning 1. PCA 2. K-means clustering 3. Hierarchical clustering 4. Expectation maximization

4 Important Concepts We Have Covered Training and test sets Cross-validation and leave-one-out Maximum likelihood Maximum a posteriori

5 Remainder of the Course In ISL: 1. Support vector machines 2. Boosting and bagging 3. Advanced nonlinear features Not in ISL: 1. Recommender systems 2. Methods for time series analysis 3. Reinforcement learning 4. Deep learning 5. Graphical models

6 Today: Linear Algebra Crucial in many machine learning algorithms Which ones? 1. Linear regression 2. Logistic regression 3. LDA, QDA 4. Naive Bayes 5. Lasso 6. Ridge regression 7. Subset selection 8. PCA 9. K-means clustering 10. Hierarchical clustering 11. Expectation maximization

7 Suggested Linear Algebra Books Strang, G. (2016). Introduction to linear algebra (5th ed.) gs/linearalgebra/ Watch online lectures: linear-algebra-spring-2010/ video-lectures/ Hefferon, J. (2017). Linear algebra (3rd ed.). Free PDF:

8 Linear Equation Equation of a line: y m 1 } b x

9 Linear Equation Equation of a line: y y = mx + b m 1 } b x

10 Linear Equation Equation of a line: y m 1 } b x y = mx + b y mx = b

11 Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b

12 Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b mx 1 + x 2 = b

13 Linear Equation Equation of a line: x 2 m 1 } b x 1 y = mx + b y mx = b x 2 mx 1 = b mx 1 + x 2 = b ( a 2 0) a 1 x 1 + a 2 x 2 = b

14 Linear Equation Linear equation in variables x 1,..., x n : a 1 x 1 + a 2 x a n x n = b where a 1,..., a n and maybe b are all known in advance.

15 Linear Equation Linear equation in variables x 1,..., x n : a 1 x 1 + a 2 x a n x n = b where a 1,..., a n and maybe b are all known in advance. Solution is a list s 1,..., s n of numbers so a 1 s 1 + a 2 s a n s n = b

16 Linear Equation Linear equation in variables x 1,..., x n : a 1 x 1 + a 2 x a n x n = b where a 1,..., a n and maybe b are all known in advance. Solution is a list s 1,..., s n of numbers so a 1 s 1 + a 2 s a n s n = b Example 1: For the equation a 1 x 1 + a 2 x 2 = b of a line, given above: The pair s 1, s 2 is a solution the point (s 1, s 2 ) is on the line.

17 Example 2 Converting grades to the standard scale: F = 0 grade 4 = A, let x 1 be your first midterm grade x 2 be your second midterm grade x 3 be your grade on the final x 4 be your homework & quiz grade x 5 be your i-clicker grade

18 Example 2 Converting grades to the standard scale: F = 0 grade 4 = A, let x 1 be your first midterm grade x 2 be your second midterm grade x 3 be your grade on the final x 4 be your homework & quiz grade x 5 be your i-clicker grade Then translates to 0.2x x x x x 5 = your course grade

19 System of Linear Equations A system of linear equations is a bunch of linear equations:

20 System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2. a m1 x 1 + a m2 x a mn x n = b m

21 System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2. a m1 x 1 + a m2 x a mn x n = b m A solution to the system is a list s 1,..., s n R that is simultaneously a solution to all m equations.

22 System of Linear Equations A system of linear equations is a bunch of linear equations: a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2 a m1 x 1 + a m2 x a mn x n = b m A solution to the system is a list s 1,..., s n R that is simultaneously a solution to all m equations. That is, all m equations are true when x 1 = s 1, x 2 = s 2,..., x n = s n..

23 Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2

24 Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Solution two lines intersect: x 2 x 1

25 Example Conceptual example: a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Solution two lines intersect: x 2 Example: 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 (x 1, x 2 ) = (1, 1) x 1

26 Line Configurations Other possibilities: x 2 x 1 x 2 x 1

27 Line Configurations Other possibilities: x 2 1x 1 + 2x 2 = 3 1x 1 + 2x 2 = 4 x 1 inconsistent x 2 x 1

28 Line Configurations Other possibilities: x 2 1x 1 + 2x 2 = 3 1x 1 + 2x 2 = 4 x 1 inconsistent x 2 1x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 x 1 redundant

29 3 Possible Line Configurations Upshot: There are exactly three possibilities

30 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution

31 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution

32 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions

33 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions Some goals: Figure out which possibility applies.

34 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions Some goals: Figure out which possibility applies. Write down the solution if it s unique.

35 3 Possible Line Configurations Upshot: There are exactly three possibilities 1. There could be no solution 2. There could be exactly one solution 3. There could be infinitely many solutions Some goals: Figure out which possibility applies. Write down the solution if it s unique. if there are infinitely many solutions, figure out a way to describe them all.

36 Matrix The critical information is in the a ij, b i.

37 Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x a 1n x n = b 1 a 21 x a 2n x n = b 2. a m1 x a mn x n = b m

38 Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x a 1n x n = b 1 a 21 x a 2n x n = b 2 a m1 x a mn x n = b m. a 11 a a 1n a 21 a a 2n a m1 a m2... a mn m n coefficient matrix

39 Matrix The critical information is in the a ij, b i. So extract just those numbers into a matrix a 11 x a 1n x n = b 1 a 21 x a 2n x n = b 2 a 11 a a 1n a 21 a a 2n a m1 a m2... a mn m n coefficient matrix a m1 x a mn x n = b m. a 11 a a 1n b 1 a 21 a a 2n b a m1 a m2... a mn b m m (n + 1) augmented matrix

40 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations

41 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Multiply an equation by c 0

42 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Multiply an equation by c 0 Replace one equation by itself plus a multiple of another equation

43 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Matrix operations: Reorder the rows Multiply an equation by (interchange) c 0 Replace one equation by itself plus a multiple of another equation

44 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Matrix operations: Reorder the rows (interchange) Multiply an equation by c 0 Replace one equation by itself plus a multiple of another equation Multiply a row by c 0 (scaling)

45 Operations Allowed These don t change the set of solutions for a system of linear equations: Reorder the equations Matrix operations: Reorder the rows (interchange) Multiply an equation by c 0 Replace one equation by itself plus a multiple of another equation Multiply a row by c 0 (scaling) Replace one row by itself plus a multiple of another row (replacement) Grand strategy: Do this until the equations are easy to solve.

46 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3

47 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3

48 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3

49 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

50 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 [ ] Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

51 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 [ ] [ ] Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1

52 Example 1x 1 + 2x 2 = 3 2x 1 + 1x 2 = 3 Subtract twice 1 st from 2 nd : 1x 1 + 2x 2 = 3 3x 2 = 3 Add 2 3 second to first: 1x 1 = 1 3x 2 = 3 Multiply second by 1 3 x 1 = 1 x 2 = 1 [ ] [ [ [ ] ] ]

53 Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5

54 Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5 Subtract 2 first from second Add 3 first to third: L1 : x 1 3x 2 2x 3 = 6 L2 2L1 : 2x 2 + x 3 = 4 3L1 + L3 : 3x 2 + 2x 3 = 13

55 Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = Subtract 2 first from second Add 3 first to third: L1 : x 1 3x 2 2x 3 = 6 L2 2L1 : 2x 2 + x 3 = 4 3L1 + L3 : 3x 2 + 2x 3 = 13

56 Step one: Use first x 1 term [upper left matrix entry] to eliminate all other x 1 terms [change rest of first column to 0]: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 1 4x 2 3x 3 = 8 L3 : 3x 1 + 6x 2 + 8x 3 = 5 Subtract 2 first from second Add 3 first to third: L1 : x 1 3x 2 2x 3 = 6 L2 2L1 : 2x 2 + x 3 = 4 3L1 + L3 : 3x 2 + 2x 3 = Move on to Step two! - the second column

57 Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13

58 Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13 Add 3 2 L2 to L1 and L3: L L2 : x x 3 = 0 L2 : 2x 2 + x 3 = 4 L L2 : 7 2 x 3 = 7

59 Step two: L1 : x 1 3x 2 2x 3 = 6 L2 : 2x 2 + x 3 = 4 L3 : 3x 2 + 2x 3 = 13 Add 3 2 L2 to L1 and L3: L L2 : x x 3 = 0 L2 : 2x 2 + x 3 = 4 L L2 : 7 2 x 3 = Multiply L3 by Move on to Step three! - the third column

60 Step three: L1 : x x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 : x 3 = 2 Subtract L3 from L2; add 1 2 L3 to L1: L L3 : x 1 = 1 L2 1 2 L3 : 2x 2 = 6 L3 : x 3 = 2

61 Step three: L1 : x x 3 = 0 L2 : 2x 2 + x 3 = 4 L3 : x 3 = 2 Subtract L3 from L2; add 1 2 L3 to L1: L L3 : x 1 = 1 L2 1 2 L3 : 2x 2 = 6 L3 : x 3 = Multiply L2 by We re done: (x 1, x 2, x 3 ) = (1, 3, 2)

62 What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4

63 What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION

64 What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION

65 What can go wrong? x 1 + 2x 2 = 3 x 1 + 2x 2 = 4 Subtract L1 from L2: x 1 + 2x 2 = 3 0= 1 inconsistent NO SOLUTION x 1 + 2x 2 = 3 2x 1 + 4x 2 = 6 Subtract 2L1 from L2: x 1 + 2x 2 = 3 0= 0 redundant SOLUTIONS INFINITE

66 Configurations If the system has 3 variables, an equation determines a plane in R 3.

67 Configurations If the system has 3 variables, an equation determines a plane in R 3. So a solution will be the intersection of 3 planes: P 1, P 2, P 3 :

68 Configurations If the system has 3 variables, an equation determines a plane in R 3. So a solution will be the intersection of 3 planes: P 1, P 2, P 3 : P 1 P 3 P 3 P2 P 1 P 2 inconsistent equations (no solution)

69 Configurations If the system has 3 variables, an equation determines a plane in R 3. So a solution will be the intersection of 3 planes: P 1, P 2, P 3 : P 1 P 3 P 3 P2 P 1 P 2 inconsistent equations (no solution) line of solutions solution P 2 P3 1P 1P P 2 3 P single solution redundant

70 Question? George tells you the system of equations has exactly three solutions. 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10

71 Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George.

72 Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George. B) George is probably wrong.

73 Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George. B) George is probably wrong. C) George is definitely right.

74 Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George. B) George is probably wrong. C) George is definitely right. D) George is definitely wrong.

75 Question? George tells you the system of equations 5x 1 + 2x 2 3x 3 = 4 12x 1 7x 2 + 2x 3 = 8 3x 1 + 4x 2 + 5x 3 = 10 has exactly three solutions. A) George is probably right, since he s Honest George. B) George is probably wrong. C) George is definitely right. D) George is definitely wrong. E) My brain is full.

76 Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. a m

77 Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. Can add two m-vectors in the obvious way, or multiply a vector by a real number: a 1 b 1 a 1 + b 1 a 2 b 2 a 1 + b 2 a 3 + b 3 = a 1 + b 3 ;... a m b m a 1 + b m a m

78 Vector An m-vector [column vector, vector in R m ] is an m 1 matrix: a 1 a 2 a 3 a =. Can add two m-vectors in the obvious way, or multiply a vector by a real number: a 1 b 1 a 1 + b 1 a 1 ca 1 a 2 b 2 a 1 + b 2 a 2 ca 2 a 3 + b 3 = a 1 + b 3 ; c a 3 = ca a m b m a 1 + b m a m ca m Do not multiply two vectors together like this. a m

79 Vector Operations Examples: = 7 ; 3 6 9

80 Vector Operations Examples: = 7 ; 1 2 =

81 Vector Operations Examples: = 7 ; 1 2 = = 3 4 0

82 Vector Operations Examples: = 7 ; 1 2 = = ( 1) =

83 Vector Operations Examples: = 7 ; 1 2 = = ( 1) =

84 Question: = 3 4 0

85 Question: = A) 10 B) 9 C) 10 D) 19 E)

86 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

87 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

88 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

89 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

90 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 v u x 1

91 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; x 2 u + v v u x 1

92 Picturing Vectors 2- and 3-vectors: think of vector as arrow from 0 multiply number with vector via scaling. add vectors head-to-tail; parallelogram rule; x 2 u + v v u x 1

93 Application Gives more flexible way to describe a line.

94 Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + t d, t R Argument: x 2 p d x 1

95 Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + 1 d, t = 1 Argument: x 2 p p + d d x 1

96 Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + 1 d, t = 1 Argument: x 2 p p - d d x 1

97 Application Gives more flexible way to describe a line. For a line through a point p, in direction d, use x = p + t d, t R Argument: x 2 p p + td d x 1

98 Question: Pictured are points u, v R 2.

99 Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 u v x 1

100 Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 E D C B A u v x 1

101 Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2 u v x 1

102 Question: Pictured are points u, v R 2. Which point represents u 3 v? x 2-3v -v u v x 1

103 Properties of Vectors For all w, x, y R n and s, t R, we have the following.

104 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative)

105 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative)

106 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z

107 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0

108 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0 t( x + y) = t x + t y (distributive law)

109 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0 t( x + y) = t x + t y (distributive law) (s + t) x = s x + t x (distributive law)

110 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0 t( x + y) = t x + t y (distributive law) (s + t) x = s x + t x (distributive law) s(t x) = (st) x (associative)

111 Properties of Vectors For all w, x, y R n and s, t R, we have the following. x + y = y + x (commutative) ( x + y) + w = x + ( y + w) (associative) z + 0 = 0 + z = z x + ( x) = x + x = 0 t( x + y) = t x + t y (distributive law) (s + t) x = s x + t x (distributive law) s(t x) = (st) x (associative) 1 x = x

112 Linear Combination Definition Suppose {t 1, t 2... t k } are all real numbers. The vector y = t 1 v t k v k is called a linear combination of the vectors { v 1, v 2... v k }.

113 Linear Combination Definition Suppose {t 1, t 2... t k } are all real numbers. The vector y = t 1 v t k v k is called a linear combination of the vectors { v 1, v 2... v k }. Sample problem: Given vectors { a 1, a 2... a n, b} in R m, find real numbers {t 1, t 2... t n } so that t 1 a t n a n = b.

114 Linear Combination Definition Suppose {t 1, t 2... t k } are all real numbers. The vector y = t 1 v t k v k is called a linear combination of the vectors { v 1, v 2... v k }. Sample problem: Given vectors { a 1, a 2... a n, b} in R m, find real numbers {t 1, t 2... t n } so that t 1 a t n a n = b. Off hand, could have any number of {t 1, t 2... t n } solutions.

115 Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a t n a n = b :

116 Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a t n a n = b : Let b 1 b 2 b = b 3 ; a j =. a 1j a 2j a 3j b m a mj. for 1 j n

117 Linear Combination How to think about solving for {t 1, t 2... t n } in the equation t 1 a t n a n = b : Let b 1 b 2 b = b 3 ; a j =. a 1j a 2j a 3j b m a mj. for 1 j n Then for any 1 i m, the i th row of the equation becomes: t 1 a i1 + t 2 a i2 + + t n a in = b i or a i1 t 1 + a i2 t a in t n = b i

118 Linear Combination In other words, solving t 1 a t n a n = b : is the same as solving this system of m linear equations: a 11 t a 1n t n = b 1 a 21 t a 2n t n = b 2. a m1 t a mn t n = b m We just learned how to do this!

119 Linear Combination Solving t 1 a t n a n = b : is the same as solving a system of m linear equations.

120 Linear Combination Solving t 1 a t n a n = b : is the same as solving a system of m linear equations. The system has augmented matrix a 11 a a 1n b 1 a 21 a a 2n b a m1 a m2... a mn b m

121 Linear Combination Solving t 1 a t n a n = b : is the same as solving a system of m linear equations. The system has augmented matrix a 11 a a 1n b 1 a 21 a a 2n b a m1 a m2... a mn b m Since each a j is a column of i numbers, can just write [ a 1 a 2... a n b ]

122 Example Suppose a 1 = 2 4 a 2 = 2 4 a 3 = 1 1 a 4 = and want to find c 1, c 2, c 3, c 4 so that 12 c 1 a 1 + c 2 a 2 + c 3 a 3 + c 4 a 4 =

123 Example This translates to the system of linear equations whose augmented matrix is

124 Example This translates to the system of linear equations whose augmented matrix is which reduces to:

125 Example This translates to the system of linear equations whose augmented matrix is which reduces to: and so has general solution c 2 = anything, c 1 = 3 2 c 2, c 3 = 2, c 4 = 1 2

126 Span Definition Given a collection { v 1, v 2,..., v k } of vectors in R m, the set of all linear combinations of these vectors, that is all vectors that can be written as c 1 v c k v k for some c 1,..., c k R is denoted Span { v 1,..., v k } and is called the span of { v 1,..., v k }.

127 Span Definition Given a collection { v 1, v 2,..., v k } of vectors in R m, the set of all linear combinations of these vectors, that is all vectors that can be written as c 1 v c k v k for some c 1,..., c k R is denoted Span { v 1,..., v k } and is called the span of { v 1,..., v k }. Easy example: If k = 1 so there is only one vector v, then Span{ v} is just all vectors that are multiples of v. That is, Span{ v} = {c v c R}

128 Span Picturing the span when m = 2, 3:

129 Span Picturing the span when m = 2, 3: When there is only one vector v then Span{ v} = {c v c R} is just the line that contains both 0 (take c = 0) and v (take c = 1).

130 Span Picturing the span when m = 2, 3: When there is only one vector v then Span{ v} = {c v c R} is just the line that contains both 0 (take c = 0) and v (take c = 1). With two vectors u and v, Span{ u, v} = {c 1 u + c 2 v} pictured via the parallelogram rule (Span = entire plane; c i 0 highlighted): u (3u+5v)/2 v

131 Span So we can think of the set of all solutions as Span

132 Span So we can think of the set of all solutions as Span So we can picture the solution as a line in the direction of the second vector, going through the point given by the first vector (but in R 4!) x 2 p p + td d x 1

133 Matrix Multiplication Definition The linear combination is abbreviated. x 1 a x k a k x 1 [ ] x 2 a1 a 2 a 3... a k x 3. Here [ a 1 a 2 a 3... a k ] is the matrix A with i th column a i. x k

134 Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b a k b k.. b k

135 Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b a k b k.. so b k 7 [ ] = 2

136 Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b a k b k.. so b k 7 [ ] = ( 2) ( 4) 2 = = 4

137 Matrix Multiplication Simplest example: each a i R 1, i. e. each column in the matrix is just a number: b 1 [ ] b 2 a1 a 2 a 3... a k b 3 = a 1 b 1 + a 2 b a k b k.. so b k 7 [ ] = ( 2) ( 4) 2 = = 4 R 1.

138 Question 3 [ ] = 1 A) -3 B) -6 C) -9 D) 6 E) 9

139 Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] = [ ] 1 5

140 Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] = so [ ] 2 1 = 4 [ ] 1 5 [ ] [ ] =

141 Matrix-vector Multiplication More complicated example: For a i R 2, i = 1, 2, 3: [ ] [ ] [ ] = so [ ] 2 1 = 4 [ ] 1 5 [ ] [ ] = Think of doing the simple case on each row of the matrix A: R 2.

142 Matrix-vector Multiplication Apply simple case to first row: [ ] = [ ] [ ] =

143 Matrix-vector Multiplication Apply simple case to first row: [ ] = Apply simple case to second row: [ ] 2 1 = 4 [ ] [ ] = [ ] [ ] =

144 Matrix-vector Multiplication Apply simple case to first row: [ ] = Apply simple case to second row: [ ] 2 1 = 4 [ ] [ ] = [ ] [ ] = R 2.

145 Matrix-vector Multiplication Further abbreviation: Use vector notation: then x 1 x 2 x 3 = x. x k

146 Matrix-vector Multiplication Further abbreviation: Use vector notation: then x 1 x 2 x 3 = x. x k x 1 a x k a k = [ ] x 2 a 1 a 2 a 3... a k x 3 = A x. x 1 x k

147 Matrix-vector Multiplication Summary: For A an m n matrix, and a vector x R n, multiplication A x is defined and gives a vector in R m. Multiplication has two important properties: For any vectors u, v R n, A( u + v) = A u + A v

148 Matrix-vector Multiplication Summary: For A an m n matrix, and a vector x R n, multiplication A x is defined and gives a vector in R m. Multiplication has two important properties: For any vectors u, v R n, A( u + v) = A u + A v For any vector u R n and any c R, A(c u) = c(a u).

149 Matrix-vector Multiplication Summary: For A an m n matrix, and a vector x R n, multiplication A x is defined and gives a vector in R m. Multiplication has two important properties: For any vectors u, v R n, A( u + v) = A u + A v For any vector u R n and any c R, A(c u) = c(a u). For example: u 1 u 2 u 3 A( u + v) = [ ] v 2 a 1 a 2 a 3... a n ( + v 3 ) =.. u n v 1 v n

150 Matrix-vector Multiplication Summary: For A an m n matrix, and a vector x R n, multiplication A x is defined and gives a vector in R m. Multiplication has two important properties: For any vectors u, v R n, A( u + v) = A u + A v For any vector u R n and any c R, A(c u) = c(a u). For example: u 1 u 2 u 3 A( u + v) = [ ] v 2 a 1 a 2 a 3... a n ( + v 3 ) =.. (u 1 +v 1 ) a 1 + +(u n +v n ) a n = (u 1 a 1 + +u n a n )+(v 1 a 1 + +v n a n ) = A u + A v u n v 1 v n

151 Question [ ] = 1 A) [ ] 7 9 B) [ ] 9 4 C) [ ] 9 1 D) [ ] 4 9 E) π e

152 Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a x n a n = b.

153 Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a x n a n = b. Might arise from the question: Is b in Span{ a 1, a 2... a n }?

154 Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a x n a n = b. Might arise from the question: Is b in Span{ a 1, a 2... a n }? New: translation to matrix equation: Given m n matrix A and b R m find a vector x R n so that A x = b

155 Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a x n a n = b. Might arise from the question: Is b in Span{ a 1, a 2... a n }? New: translation to matrix equation: Given m n matrix A and b R m find a vector x R n so that A x = b where A = [ a 1 a 2 a 3... a n ].

156 Matrix Equation Sample problem from before: Given vectors { a 1, a 2... a n } and b in R m, find real numbers {x 1, x 2... x n } so that x 1 a x n a n = b. Might arise from the question: Is b in Span{ a 1, a 2... a n }? New: translation to matrix equation: Given m n matrix A and b R m find a vector x R n so that A x = b where A = [ a 1 a 2 a 3... a n ]. Note: the Matrix-vector multiplication A x makes sense only if the number of columns in A matches the number of entries in x.

157 Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line.

158 Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line. On the other hand, if they don t lie on the same line, then Span{ x 1, x 2 } consists of an entire plane.

159 Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line. On the other hand, if they don t lie on the same line, then Span{ x 1, x 2 } consists of an entire plane. (The plane determined by the heads of the vectors and 0.)

160 Background Background thoughts: If two non-trivial vectors x 1, x 2 both lie on the same line, then Span{ x 1, x 2 } is just that line. On the other hand, if they don t lie on the same line, then Span{ x 1, x 2 } consists of an entire plane. (The plane determined by the heads of the vectors and 0.) So the span of two vectors may be a plane, or it could be something simpler: either a line, or even just 0 in the case that x 1 = 0 = x 2.

161 Background Similarly, if three non-trivial vectors x 1, x 2, x 3 all lie on the same line, then Span{ x 1, x 2, x 3 } is just that line.

162 Background Similarly, if three non-trivial vectors x 1, x 2, x 3 all lie on the same line, then Span{ x 1, x 2, x 3 } is just that line. If they don t all lie on the same line, but lie on the same plane, then Span{ x 1, x 2, x 3 } is just that plane.

163 Background Similarly, if three non-trivial vectors x 1, x 2, x 3 all lie on the same line, then Span{ x 1, x 2, x 3 } is just that line. If they don t all lie on the same line, but lie on the same plane, then Span{ x 1, x 2, x 3 } is just that plane. If they don t all lie in the same plane, then Span{ x 1, x 2, x 3 } looks like space.

164 Linear Independence How do we put these ideas into math lingo, so we can be precise? Definition A set of vectors { v 1,..., v k } in R m is linearly independent if and only if the only solution to the equation x 1 v x k v k = 0 is the solution x i = 0 for 1 i k. Conversely, the set of vectors { v 1,..., v k } is linearly dependent if there are real numbers c 1,..., c k, not all zero, such that c 1 v c k v k = 0.

165 Linear Independence How do we put these ideas into math lingo, so we can be precise? Definition A set of vectors { v 1,..., v k } in R m is linearly independent if and only if the only solution to the equation x 1 v x k v k = 0 is the solution x i = 0 for 1 i k. Conversely, the set of vectors { v 1,..., v k } is linearly dependent if there are real numbers c 1,..., c k, not all zero, such that c 1 v c k v k = 0. Idea: if the set is linearly independent, then span is big as possible.

166 Linear Independence How do we put these ideas into math lingo, so we can be precise? Definition A set of vectors { v 1,..., v k } in R m is linearly independent if and only if the only solution to the equation x 1 v x k v k = 0 is the solution x i = 0 for 1 i k. Conversely, the set of vectors { v 1,..., v k } is linearly dependent if there are real numbers c 1,..., c k, not all zero, such that c 1 v c k v k = 0. Idea: if the set is linearly independent, then span is big as possible. If the set is linearly dependent then span is thinner than it has to be;

167 Linear Independence How do we put these ideas into math lingo, so we can be precise? Definition A set of vectors { v 1,..., v k } in R m is linearly independent if and only if the only solution to the equation x 1 v x k v k = 0 is the solution x i = 0 for 1 i k. Conversely, the set of vectors { v 1,..., v k } is linearly dependent if there are real numbers c 1,..., c k, not all zero, such that c 1 v c k v k = 0. Idea: if the set is linearly independent, then span is big as possible. If the set is linearly dependent then span is thinner than it has to be; you could even throw some away and not change the span.

168 In pictures: v 2 v 1 v 3 { v 1, v 2, v 3 } linearly dependent.

169 v v 1 2 v 3 { v 1, v 2, v 3 } linearly independent.

170 Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set.

171 Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set. Argument: Suppose, say, { v 1, v 2, v 3 } { v 1,..., v k } is linearly dependent.

172 Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set. Argument: Suppose, say, { v 1, v 2, v 3 } { v 1,..., v k } is linearly dependent. This means that there are c 1, c 2, c 3 not all 0 so that But then, c 1 v 1 + c 2 v 2 + c 3 v 3 = 0 c 1 v 1 + c 2 v 2 + c 3 v v v n = 0 so the whole set is linearly dependent.

173 Linear Independence If any subset of { v 1,..., v k } is linearly dependent, so is the whole set. Argument: Suppose, say, { v 1, v 2, v 3 } { v 1,..., v k } is linearly dependent. This means that there are c 1, c 2, c 3 not all 0 so that But then, c 1 v 1 + c 2 v 2 + c 3 v 3 = 0 c 1 v 1 + c 2 v 2 + c 3 v v v n = 0 so the whole set is linearly dependent. Equivalently: if { v 1,..., v k } is linearly independent then so is every subset of vectors from { v 1,..., v k }.

174 Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent:

175 Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a x n a n = 0.

176 Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a x n a n = 0. But this is a homogeneous system of linear equations - we can answer that question!

177 Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a x n a n = 0. But this is a homogeneous system of linear equations - we can answer that question! Construct associated matrix of column vectors: a 11 a a 1n a 21 a a 2n A = a m1 a m2... a mn Reduce to echelon form and see if there are any free variables.

178 Determining Independence How to check whether a set of vectors { a 1,..., a n } is linearly dependent: It s equivalent to saying there is non-zero solution to x 1 a x n a n = 0. But this is a homogeneous system of linear equations - we can answer that question! Construct associated matrix of column vectors: a 11 a a 1n a 21 a a 2n A = a m1 a m2... a mn Reduce to echelon form and see if there are any free variables. There are no free variables if and only if linearly independent

179 Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent.

180 Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent. For if n > m then the associated matrix of column vectors a 11 a a 1n a 21 a a 2n A = a m1 a m2... a mn is longer than it is high.

181 Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent. For if n > m then the associated matrix of column vectors a 11 a a 1n a 21 a a 2n A = a m1 a m2... a mn is longer than it is high. When reduced to echelon form, there must be free variables: $ 0 $ $ $ $

182 Determining Independence Example 4: Any set of more than m vectors in R m is linearly dependent. For if n > m then the associated matrix of column vectors a 11 a a 1n a 21 a a 2n A = a m1 a m2... a mn is longer than it is high. When reduced to echelon form, there must be free variables: $ 0 $ $ $ $ Here 6 > 5 and x 3 is the free variable.

183 Question Question: Is this set of vectors linearly dependent, or linearly independent? 2 3 3, A) Dependent since they are 2 vectors in R 3 and 2 < 3. B) Independent since they are 2 vectors in R 3 and 2 < 3. C) Dependent because one is a multiple of the other. D) Independent because neither is a multiple of the other. E) Independent because one is a multiple of the other.

184 Question Question: Is this set of vectors linearly dependent, or linearly independent? 2 3 3, A) Dependent since they are 2 vectors in R 3 and 2 < 3. B) Independent since they are 2 vectors in R 3 and 2 < 3. C) Dependent because one is a multiple of the other. D) Independent because neither is a multiple of the other. E) Independent because one is a multiple of the other. Answer: It s a pair of vectors and neither is a multiple of the other. Hence linearly independent.

185 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 4,

186 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 4, A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent.

187 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 4, A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. Answer: The second is a multiple of the first, so that pair alone is linearly dependent.

188 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 4, A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. Answer: The second is a multiple of the first, so that pair alone is linearly dependent. Since this subset is linearly dependent, so is the entire set.

189 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 4, A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. Answer: The second is a multiple of the first, so that pair alone is linearly dependent. Since this subset is linearly dependent, so is the entire set. Both B) and C) are used to show linear dependence.

190 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 3,

191 Question Question: Is this set of vectors linearly dependent, or linearly independent? , 3, A) Independent since they are 3 vectors in R 3. B) Dependent because one is a multiple of the other. C) Dependent because a subset is dependent. D) Independent because a subset is independent. E) I can t tell.

192 Answer: For this triple of vectors , 3, we want to determine whether the homogeneous system of linear equations: x x 2 = x 3 has a non-trivial solution.

193 Answer: For this triple of vectors , 3, we want to determine whether the homogeneous system of linear equations: x x 2 = x 3 has a non-trivial solution. Consider the associated column matrix (no need to augment):

194 Answer: For this triple of vectors , 3, we want to determine whether the homogeneous system of linear equations: x x 2 = x 3 has a non-trivial solution. Consider the associated column matrix (no need to augment):

195 Answer: For this triple of vectors , 3, we want to determine whether the homogeneous system of linear equations: x x 2 = x 3 has a non-trivial solution. Consider the associated column matrix (no need to augment): Since there is a free variable (namely x 3 ) there are non-trivial solutions, so linearly dependent.

196 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. For example, suppose v 1 Span{ v 2, v 3,..., v k }.

197 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. For example, suppose v 1 Span{ v 2, v 3,..., v k }. That means there are c 2, c 3,..., c k so that v 1 = c 2 v 2 + c 3 v c k v k

198 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. For example, suppose v 1 Span{ v 2, v 3,..., v k }. That means there are c 2, c 3,..., c k so that v 1 = c 2 v 2 + c 3 v c k v k But this vector equation can be rewritten 1 v 1 + c 2 v 2 + c 3 v c k v k = 0.

199 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. For example, suppose v 1 Span{ v 2, v 3,..., v k }. That means there are c 2, c 3,..., c k so that v 1 = c 2 v 2 + c 3 v c k v k But this vector equation can be rewritten 1 v 1 + c 2 v 2 + c 3 v c k v k = 0. Since at least one of the coefficients (namely 1) is not zero, this shows the set of vectors is linearly dependent.

200 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent.

201 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v c k v k = 0.

202 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v c k v k = 0. For concreteness, say c 1 0.

203 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get v 1 + c 2 c 1 v 2 + c 3 c 1 v c k c 1 v k = 0

204 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get and this can be rewritten v 1 + c 2 c 1 v 2 + c 3 c 1 v c k c 1 v k = 0 v 1 = c 2 c 1 v 2 c 3 c 1 v 3... c k c 1 v k

205 Connection with Span Theorem A set of vectors { v 1,..., v k } in R m is linearly dependent if and only if at least one of the vectors is in the span of all the others. Here s the argument in the other direction: Suppose { v 1,..., v k } is linearly dependent. That means there are c 1, c 2,..., c k, not all zero, so that c 1 v 1 + c 2 v 2 + c 3 v c k v k = 0. For concreteness, say c 1 0. Divide by c 1 to get and this can be rewritten v 1 + c 2 c 1 v 2 + c 3 c 1 v c k c 1 v k = 0 v 1 = c 2 c 1 v 2 c 3 c 1 v 3... c k c 1 v k Hence v 1 Span{ v 2, v 3,..., v k }.

206 Connection with Span In pictures: v 2 v 1 v 3 v 1 Span{ v 2, v 3 } and { v 1, v 2, v 3 } linearly dependent.

207 Connection with Span v v 1 2 v 3 v 1 / Span{ v 2, v 3 } and { v 1, v 2, v 3 } linearly independent.