Lecture: Linear algebra. 1. Subspaces. 2. Orthogonal complement. 3. The four fundamental subspaces 4. Solutions of linear equation systems The fundamental theorem of linear algebra 5. Determining the fundamental subspaces Lecture LA 1 Linear algebra I - Per Enqvist
Vector Spaces A vector space V is a set on which vector addition + and scalar multiplication are such that V1 For all v 1,v 2,v 3 V, v 1 +(v 2 +v 3 ) = (v 1 +v 2 )+v 3 V2 Exists a zero vector V, such that v+ = +v = v for all v V V3 For each v V there exists a unique v V such that v+( v) = ( v)+v = V4 For all v 1,v 2 V, v 1 +v 2 = v 2 +v 1 V5 For all v V, 1 v = v V6 For all α,β R and all v V, α (β v) = (αβ) v V7 For all α,β R and all v V, (α+β) v = α v+β v V8 For all α R and all v 1,v 2 V, α (v 1 +v 2 ) = α v 1 +α v 2 Lecture LA 2 Linear algebra I - Per Enqvist
Subspaces Definition 1. A subset M R n is called a subspace in R n if for all v 1,v 2 M and α 1,α 2 R it holds that α 1 v 1 + α 2 v 2 M Note that it always holds that M. x 3 M v 1 x 2 v 3 x 1 v 4 v 2 Lecture LA 3 Linear algebra I - Per Enqvist
Basis for a subspace The vectors v 1,...,v k are linearly independent if there exists no scalars α 1,,α k (not all zero) such that k l=1 α lv l =. Definition 2. M is spanned by v 1,...,v k if for each v M there are α 1,...,α k R such that k α l v l = v l=1 If v 1,...,v k are linearly independent, then they form a basis for M and the dimension of M is k, and it is denoted dim M = k. Notation: M = span{v 1,...,v k } = { k l=1 α lv l : α l R; l = 1,...,k}. Lecture LA 4 Linear algebra I - Per Enqvist
Sums of subspaces and orthogonal subspaces Given two subspaces M 1 and M 2, the subset M = {v 1 + v 2 : v 1 M 1,v 2 M 2 } is a subspace and we write M = M 1 + M 2. Two vectors v 1 and v 2 are orthogonal if v T 1 v 2 = v 1 v 2 cos θ =, i.e., the angle θ between them is π/2, and we write v 1 v 2. Two subspaces M 1 and M 2 are orthogonal if v 1 v 2 for all v 1 M 1 and v 2 M 2, and we write M 1 M 2. If dim M = dim M 1 + dim M 2, then every v M can be uniquely written on the form v = v 1 + v 2, where v 1 M 1 and v 2 M 2. Lecture LA 5 Linear algebra I - Per Enqvist
Direct sums of subspaces A linear space M (for example R n ) is a direct sum of the subspaces M 1, M 2 (i.e. M 1, M 2 M) if M 1 M 2 and M 2 + M 2 = R n. The direct sum is denoted M = M 1 M 2, and every v M can be written uniquely v = v 1 + v 2 for some v 1 M 1 and v 2 M 2. Example 1. Assume M 1 = span{v 1,v 2,...,v k } M 2 = span{v k+1,...,v l } M = span{v 1,v 2,...,v k,v k+1,...,v l } where v 1,...,v l are linearly independent vectors. Then it holds that M = M 1 M 2 if v i v j for all i = 1,...,k and j = k + 1,...,l. Lecture LA 6 Linear algebra I - Per Enqvist
Orthogonal complement The orthogonal complement to a subspace M R n is defined by M = {w R n : w T v =, v M} The following holds M is a subspace R n = M M. If dim M = k, then dim M = n k. (M ) = M. Lecture LA 7 Linear algebra I - Per Enqvist
The four fundamental subspaces Consider the linear operator A : R n R m, A = a 11. a m1... a 1n.... a mn [ ] = â 1 â 2... â n = ā T 1 ā T 2. ā T m The Range space: R(A) = {Ax x R n } = { n } = â j x j x j R n, j = 1,...,n j=1 The Null space: N(A) = {x R n Ax = } = { x R n ā T i x =, i = 1,...,m } Lecture LA 8 Linear algebra I - Per Enqvist
The Row space: R(A T ) = { A T u u R m} { m } = ā i u i u i R n, i = 1,...,m i=1 The left Null space: N(A T ) = { u R m A T u = } = { u R m â T j u =, j = 1,...,n } = { u R m u T A = } R(A) and N(A T ) are subspaces in R m R(A T ) and N(A) are subspaces in R n Lecture LA 9 Linear algebra I - Per Enqvist
The orthogonal complement of the fundamental subspaces Theorem 25.1 The following orthogonality relations hold (i) R(A) = N(A T ). (ii) R(A T ) = N(A). (iii) N(A) = R(A T ). (iv) N(A T ) = R(A). Lecture LA 1 Linear algebra I - Per Enqvist
The Fundamental Theorem of Linear algebra Theorem Let A R m n. Then the following direct sums hold (a) R n = R(A T ) N(A) (b) R m = R(A) N(A T ) where R(A T ) N(A) and R(A) N(A T ). Furthermore, it holds that dim R(A) = dim(a T ). Proof: Since N(A), R(A T ) R n according to the previous theorem, R(A T ) = N(A) it holds that R n = R(A T ) N(A). The proof of (b) follows from the same argument. Lecture LA 11 Linear algebra I - Per Enqvist
Solution of linear equation systems Let A R m n and b R n and consider the linear equation system Ax = b (1) (A) When does there exist a solution? (B) Is the solution unique? Which solution is chosen otherwise? (C) How do we construct the solution? The fundamental theorem of linear algebra answers the questions above. Lecture LA 12 Linear algebra I - Per Enqvist
Geometric illustration of the fundamental theorem of linear algebra N(A) N(A T ) R n A R m x n x x r R(A T ) Ax r R(A) The picture answers the questions (A) (C) above. The equation system Ax = b has a solution iff b R(A). The solution is unique if, and only if, N(A) =. Otherwise, every solution can be decomposed into two components x = x r + x n where x r R(A T ) satisfies Ax r = b and x n N(A). Often we select the smallest norm solution, i.e., x n =. Lecture LA 13 Linear algebra I - Per Enqvist
The following three slides dicusses the questions (A) (C) in more detail. (A) There is a solution to (1) if b R(A) (B) The solution is unique iff N(A) =. This follows since every x R n uniquely can be written x = x r + x n where x r R(A T ) and x n N(A). We have Ax = Ax r + Ax n = Ax r. The component x n N(A) is thus arbitrary and the solution is unique iff N(A) =. Which solution do we chose if N(A)? One choice is to take the solution that has the shortest length measured in the Euclidean norm x = x T x. It follows that we should choose x n =. Lecture LA 14 Linear algebra I - Per Enqvist
(C) Assume b R(A). There are three cases for the construction of solutions (i) N(A) = and m = n (quadratic matrix). Then the solution is determined by x = A 1 b. (ii) If N(A), then we choose the minimum norm solution solution given by x = A T u, where u R m solves AA T u = b. If AA T is invertible then the closed form is x = A T (AA T ) 1 b. (iii) N(A) = and m > n (overdetermined system). Then the solution is obtained by x = (A T A) 1 A T b. Lecture LA 15 Linear algebra I - Per Enqvist
Determining the four fundamental subspaces Two methods for determining the fundamental subspaces. Singular Value Decomposition. The best method from a numerical point of view. Gauss-Jordan s method. We focus on this method Lecture LA 16 Linear algebra I - Per Enqvist
Minimal rank factorization Theorem 26.1 Let A R m n and A = BC, where B R m r has linearly independent columns and C R r n has linearly independent rows. Then A and A T both have ranges of dimension r. Furthermore, N(A) = N(C) N(A T ) = N(B T ) R(A) = R(B) R(A T ) = R(C T ) Lecture LA 17 Linear algebra I - Per Enqvist
Summary of Gauss-Jordan s method Step 1 Perform elementary row operations to transform A to staircase form. PA = T = U (2) O where P is a product of elementary row operation matrices, while U is a staircase matrix on the form U = 1 * * 1 * 1 β 1 β 2 β 3 * * ** * * 1 * β r * * * * * * * * * * * Lecture LA 18 Linear algebra I - Per Enqvist
Step 1 (cont.) The staircase columns have indices β 1,...,β r, where r = rang(a) = dim R(A) (rang of the matrix). The other columns have the indices ν 1,...,ν l, l = n r. Step 2 Define A β = U β = U ν = [â β1... â βr ] [u β1... u βr ] [u ν1... u νl ] = I r S = P 1 Note that U β (the staircase columns in U) is an identity matrix. Lecture LA 19 Linear algebra I - Per Enqvist
Step 2 (cont.) Let S = P 1, then an equivalent expression for (2) is ] A = ST = [S 1 S 2 U = S 1 U O If we consider the columns with indices β 1,...,β r we have and then A = A β U. A β = S 1 U β = S 1, Lecture LA 2 Linear algebra I - Per Enqvist
Step 3 The factorization A = A β U is of the form A = BC where m rows = A B C r linearly independent rows n columns r linearly independent columns According to ch. 26 R(A) = R(A β U) = R(A β ) = span{â β1,...,â βr } N(A) = N(A β U) = N(U) = {x R n : Ux = } = {x R n : U β x β + U ν x ν = } = {x R n : x β = U ν x ν ; x ν R l } Lecture LA 21 Linear algebra I - Per Enqvist
Step 3 (cont.) Furthermore, using the previous factorization R(A T ) = R(U T A T β) = R(U T ) Finally, N(A T ) = R(A) = {y R m : A T βy = } which corresponds to solving an equation system. Lecture LA 22 Linear algebra I - Per Enqvist
Solving linear equation systems Existence and uniqueness of a linear equation system can be determined using Ax = b 1. the fundamental Theorem of linear algebra Gives theoretical insight and solution formulas. 2. Gauss-Jordan s method Practical method for calculations by hand. Lecture LA 23 Linear algebra I - Per Enqvist
Gauss-Jordan s method for solving equation systems Transform the equation system using elementary row operations Ax = b PAx = Pb U x = b r b n where U = 1 * * 1 * 1 * * ** * * * * * * * * * * * 1 * * * β 1 β 2 β 3 β r Lecture LA 24 Linear algebra I - Per Enqvist
Observe that 1. A solution does not exist, if and only if, b n 2. If b n =, there exists solutions, and they can all be written on the form x = x r + x n where x n N(A) and x βl, for coefficients corresponding to staircase columns x =, otherwise where x β = x β1. = b r x βk Lecture LA 25 Linear algebra I - Per Enqvist
Example Determine the nullspace and rangespace for the matrix 1 1 1 1 1 2 7 3 1 A = 2 1 8 1 4 1 3 1 3 Perform elementary row operations to transform A into staircase form. Lecture LA 26 Linear algebra I - Per Enqvist
Example A 1 = 1 1 1 1 3 6 2 1 3 6 1 4 1 2 2 3 = P 1 A, where P 1 = 1 1 1 2 1 +1 1 We note that β 1 = 1. 1 1 1 1 1 2 2/3 1/3 A 2 = 3 6 1 4 1 2 2 3 = P 2 A 1, where P 2 = 1 1/3 1 1 Lecture LA 27 Linear algebra I - Per Enqvist
Example A 3 = 1 3 5/3 1/3 1 2 2/3 1/3 3 3 8/3 8/3 = P 3 A 2, where P 3 = 1 +1 1 3 1 +1 1 We note that β 2 = 2. 1 3 5/3 1/3 1 2 2/3 1/3 A 4 = 1 1 8/3 8/3 = P 4 A 3, where P 4 = 1 1 1/3 1 Lecture LA 28 Linear algebra I - Per Enqvist
Example A 5 = 1 3 2 1 2 1 1 1 = P 5 A 4, where P 4 = 1 5/3 1 2/3 1 8/3 1 We note that β 3 = 4, and ν 1 = 3, ν 2 = 5. Rank(A) = 3. Then P = P 5 P 4 P 3 P 2 P 1, and A = P 1 T = A β U where A β = 1 1 1 1 2 3 2 1 1 1 1 1 3 2 and U = 1 2 1 1 1 Lecture LA 29 Linear algebra I - Per Enqvist
Example The range space is now given by R(A) = R(A β ) where we know that A β has linearly independent columns. 1 1 1 1 2 3 R(A) = R( ) = span(â 2 1 1 1,â 2,â 4 ) 1 1 Lecture LA 3 Linear algebra I - Per Enqvist
Example The nullspace is given by N(A) = N(U) = I x 1 x 2 x 4 3 2 = 2 1 x 3 x 5 1 N(A) = x 1 x 2 x 3 x 4 x 5 = 3 2 1 x 3 + 2 1 1 1 x 5 3 2 2 1 = span 1, 1 1 Lecture LA 31 Linear algebra I - Per Enqvist
Example We also know that R(A T ) = R(U T ) = span 1 3 2 1 2 1 1 1 and N(A T ) = N(A T β ) = = span 2 5 8 9 follows by similar calculations as for N(A). Lecture LA 32 Linear algebra I - Per Enqvist
Reading instructions Chapter 23-26. Lecture LA 33 Linear algebra I - Per Enqvist