MATH 167: APPLIED LINEAR ALGEBRA Chapter 2

MATH 167: APPLIED LINEAR ALGEBRA Chapter 2 Jesús De Loera, UC Davis February 1, 2012

General Linear Systems of Equations (2.2).

Given a system of m equations and n unknowns. Now m n is OK! Apply elementary row operations to simplify! An m n matrix is in echelon form if it has an staircase structure. Use it to solve systems of equations!!!???????...? 0 0?????...? 0 0 0 0 0 0?...?............ 0 0 0 0 0 0 0...? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 The first r rows contain a non-zero pivot, the first non-zero element of that row. The last m r rows are full of zeros. Below each pivot there is a column of zeros. Each pivot lies to the right of the pivot in the row above.

Theorem Any matrix can be reduced to the echelon form by a finite sequence of multiplications with elementary matrices and permutation matrices. Thus, for any m n matrix there exist a permutation P, an upper-triangular matrix U, and a lower-triangular matrix L such PA = LU. Definition In a linear system in echelon form, the variables corresponding to the columns containing a pivot are basic or pivot variables, while the variables corresponding to variables without a pivot are called free variables. The rank of A is the number of basic variables. From the echelon form, the solution to a system of equations proceeds by back substitution procedure. Each non-zero equation is solved for the basic variable associated to its pivot, the result is substituted into the preceding before they are solved. Solution gives all basic variable as combinations of the free variables. Free variables are allowed to take any value and parametrize all solutions of the system.

1 3 2 1 0 a 2 6 1 4 3 b 1 3 3 3 1 c 3 9 8 7 2 d 1 3 2 1 0 a 0 0 3 6 3 b 2 a 0 0 0 0 4 d + 2/3 b 13/3 a 0 0 0 0 0 c 1/3 b + 5/3 a This system only has a solution when c 1/3 b + 5/3 a = 0, in that case the free variables are 2nd and 4rd columns (y,u)

For example when a = 0, b = 3, c = 1, d = 1 x y z u v 1 3 2 1 0 a = 0 0 0 3 6 3 b 2 a = 3 0 0 0 0 4 d + 2/3 b 13/3 a = 3 0 0 0 0 0 c 1/3 b + 5/3 a = 0 The general solution is v = 3/4, z = 1 + 2u + v = 1/4 + 2u, x = 3y 2z + u = 1/2 3y 3u. WE OBTAIN INFINITELY MANY SOLUTIONS! QUESTION Can you select a, b, c, d to make it have a SINGLE solution??

Theorem A system Ax = b of m linear equations and n unknowns has either 1 Exactly one solution, 2 Infinitely many solutions, or 3 no solution at all. Geometric meaning: Each equation represents a (hyper) plane, they either intersect at a single point, share a line, or do not intersect at all. Not true for non-linear equations!. KEY POINT Ax = b has a solution means that the vector b is a linear combination of the columns of A! The set of all linear combinations of columns of A is the Column space, denoted C(A). IMPORTANT CASE: What happens if the right-hand side vector b = ZERO?? HOMOGENEOUS SYSTEM Definition: The set of all solutions to Ax = 0 is called the NULLSPACE of A, denoted N(A). Culture: The rest of the world calls this the kernel of A.

Go further! Make reduced row echelon form (RREF): All pivots are 1 s and zero above the pivot too! For the matrix we had before the RREF is 1 3 0 3 0 0 0 1 2 0 0 0 0 0 1 0 0 0 0 0 The RREF is very useful to determine the solutions to the homogeneous system Ax = 0: 1 Identify basic and free variables. 2 Given a free variable, set its value to 1 and the other free variables to zero. That provides one special solution in the nullspace. 3 Each free variables produces one special solution.

Example: In the previous matrix the special solutions are ( 3, 1, 0, 0, 0) and ( 3, 0, 2, 1, 0). Lemma Any solution of the nullspace of A is a linear combination of the special solutions. Theorem A homogeneous system Ax = 0 of m equations in n unknowns 1 It always has a trivial solution! 2 It has a non-trivial solution ( 0) rank(a) is r < n. 3 If m < n then the system has always a non-trivial solution. 4 If m = n then the system has a non-trivial solution if and only if A is singular. WHY should I care? Homogeneous systems look too special to be of use! Lemma Every solution of Ax = b can be written in the form x 1 + x where x in the nullspace of A.

Vector Spaces and Subspaces (2.1)

A vector space V is a set with two operations addition and scalar multiplication. the scalars are members of a field K, in which case is called a vector space over K. The following conditions must hold for all elements and any scalars : 1 V must be closed under both operations (no escaping to a different space). 2 Commutativity of vector addition 3 Associativity of vector addition. 4 Existence of additive identity. 5 Existence of additive inverse. 6 Associativity of scalar multiplication. 7 Distributivity of scalar sums. 8 Distributivity of vector sums. 9 Existence of Scalar multiplication identity. Example is d-dimensional Euclidean space, where every element is represented by an ordered list of d real numbers, scalars are real numbers, addition is componentwise, and scalar multiplication is multiplication on each term separately.

The space M n,m (C) of n m matrices with complex coefficients. Polynomials with real coefficients. Space of all functions defined on the interval [0, 1]. Space of sample functions on n given points. Definition A subspace of a vector space is a non-empty subset that satisfies all the requirements above, linear combinations stay in the subspace. QUESTIONS Which of the following are subspaces? 1 Set of all solutions to the equation z = x y, 2 Set of all solutions to z = xy. 3 Set of all vectors in R n whose first component is zero. 4 Set of all even polynomials (p(x) = p( x)). 5 The set of all continuous functions with 1 f (t)dt = f (1/2). 0

FOUR KEY SUBSPACES Say we are given an m n matrix A Column Space: Also known as the range of A, denoted as C(A); this is subspace of R m. Row Space: Contains all combinations of the rows of A; same as the Column Space of A T. It is a subspace of R n. Nullspace N(A), vectors elements that (right) multiply to zero with A Left Nullspace: Same as the Nullspace of A T. IMPORTANCE?: Most algorithms and applications of linear algebra are understood by moving these 4 subspaces. C(A), C(A T ) and N(A), N(A T ) behave well under elementary row operations!

What are these 4 subspaces? (continued) Ax = a 11 a 12... a 1n a 21 a 22... a 2n... a m1 a m2... a mn x 1 x 2. x n = x 1 (column 1) +x 2 (column 2) +... + x n (column n) = b. Therefore b C(A), a linear combination of the columns of A

What are these 4 subspaces? (continued) Nullspace: nullspace N(A), a subspace of R n. Solutions to the system Ax = 0 are not changed by elimination operations. row 1 x 1 0 Ax = row 2 x 2 = 0 row m x m 0 where x N(A). Left Nullspace: nullspace of A T, a subspace of R m ; all solutions to A T y = 0.

Linear Independence, Basis, and Dimension (2.3)

Definition Let v 1, v 2,..., v k be vectors on vector space V. A linear combination is a sum of the form c 1 v 1 + c 2 v 2 +... c k v k with scalars c 1,..., c k. The span of the vectors is the set of all possible linear combinations. Lemma The span of v 1, v 2,..., v k is a subspace. In fact, it is the smallest subspace that contains the vectors! Example: Consider the trigonometric functions f 0 (x) = 1, f 1 (x) = cos(x), f 2 (x) = sin(x), f 3 (x) = cos 2 (x), f 4 (x) = cos(x)sin(x), f 5 (x) = sin 2 (x). The span of f 0, f 1, f 2,..., f 5 we can ask. Is this the smallest set of spanning set for that subspace? No! Suffices with f 0, f 1, f 2 and g 1 (x) = cos(2x), g 2 (x) = sin(2x). Why?

cos 2 (x) = 1 2 cos(2x) + 1/2, sin2 (x) = 1 cos(2x) + 1/2, 2 cos(x)sin(x) = 1 2 sin(2x) So not all vectors f 0,..., f 5 are essential to span the same subspace! f 6 is redundant. How to deal with this? Definition The vectors v 1, v 2,..., v k are linearly dependent if there exist scalars c 1,..., c k not all zero such that c 1 v 1 + + c k v k = 0. Vectors that are not dependent are called linearly independent. Example: Are f 0 (x) = 1,f 1 (x) = cos(x), f 2 (x) = sin(x), f 3 (x) = cos 2 (x), f 4 (x) = cos(x)sin(x), f 5 (x) = sin 2 (x) linearly dependent? (YES!)1+0 cos(x)+0 sin(x)+( 1)cos 2 (x)+( 1)sin 2 (x)+0 cos(x)sin How to check this in practice? We can set up a system of equations and Jesúscheck De Loera, whether UC Davis the MATHonly 167: APPLIED solution LINEAR is calgebra i = 0. Chapter 2

Theorem Let v 1,..., v n vectors in K m and A = (v 1... v k ) the corresponding m n matrix. 1 The vectors are linear dependent if there is a non-zero solution to Ax = 0. 2 The vectors are linearly independent the only solution to the homogeneous system Ax = 0 is the trivial one. 3 The vector b lies in the span of v 1, v 2,..., v n the system Ac = b has at least one solution. Say we are given some vectors v 1, v 2,..., v n in K m if n > m then they must be linearly dependent Definition A basis of a vector space (subspace) V is a sequence of vectors which are linearly independent and span V. Examples: The standard vectors e i = (0, 0,..., 1,..., 0) form a basis for R m. The monomials 1, x, x 2, x 3,..., x r are a basis for the vector space of polynomials of degree r.

Important properties Lemma: There is one and only one way to write a vector in V as a combination of basis vectors. Theorem: Any two bases of the same vector space V contain the same number of vectors. Definition: The number of vectors on a basis of the vector (sub)space is called its dimension. This is an ID number for vector spaces! If a basis is finite we say that V is a finite-dimensional vector space. Theorem: Any linearly set of vectors in V can be extended to a basis, by adding more vectors if necessary. Similarly, any spanning set can be reduced to a basis, by discarding unnecessary vectors.

The Four Fundamental Subspaces (2.4)

Recall we talk about 4 important spaces: First we have the column space of linear combinations of columns: a 11 a 12... a 1n x 1 a 21 a 22... a 2n x 2 Ax =... a m1 a m2... a mn. x n = x 1 (column 1) +x 2 (column 2) +... + x n (column n) = b. IMPORTANT TWIN: the row space of linear combinations of the rows o

Nullspace: nullspace N(A), a subspace of R n. Solutions to the system Ax = 0 are not changed by elimination operations. row 1 x 1 0 Ax = row 2 x 2 = 0 row m x m 0 where x N(A). Left Nullspace: nullspace of A T, a subspace of R m ; all solutions to A T y = 0.

Fundamental Theorem Let A be an m n matrix. Then dimensions of the fantastic 4 subspaces obey the laws: dim C(A) = dim C(A T ). Row space and nullspace in R n, and dim C(A T ) = r then dim N(A) = n r. dim(c(a T )) + dim(n(a)) = n. Similarly for the Column space and left nullspace in R m, so if dim C(A) = r, then dim N(A T ) = m r. dim(c(a)) + dim(n(a T )) = m.

Everything is a beautiful by-product of ROW REDUCTION! Say A is an m n matrix. Lemma: The row space of A, C(A T ), has the same dimension as the row space of the echelon form OR the reduced echelon form. Thus dim(c(a T )) = rank(a). Lemma: The Nullspace N(A) has dimension n rank(a) Why? Because the special solutions (a free variable has value 1 while the others free variables are zero) give linearly independent solutions that generate N(A). Lemma: The dimension of the column space C(A) equals rank(a) which also equals the dimension of the row space! WHY? Because the row reduction is same as multiplying A with an invertible square Q. If R is the RREF of A, then the columns corresponding to pivots of R are mapped by Q 1 to independent vectors! WARNING: How do you find bases for each of these 4 spaces?

We have proved that dim(c(a T )) = #of pivots in RREF. On the other hand dim(n(a)) = #of columns #of pivots so we get if dim C(A T ) = n, then dim N(A) = n r. Corollary: Let A be an m n matrix, then 1 If matrix has full row rank r = m, then Ax = b has at least one solution for every b. Thus A has a right-inverse C such that AC = I m. For this m n. 2 If matrix has full column rank r = n, then Ax = b has at most one solution for every b. Thus A has a left-inverse B such that BA = I n. For this to happen m n. Corollary: The only matrix that can have both a right and a left inverse must be square. TO BE CONTINUED LATER: The one sided inverses have explicit formulas B = (A T A) 1 A T, and C = A T (AA T ) 1.

Linear Transformations (2.6)

Definition: Let V, W be vector spaces, a function L : V W is linear if L(v + u) = L(v) + L(u) and L(λv) = λl(v). Exciting linear transformations: 1 Given an m n matrix A then the function from R n into R m given by the rule v is mapped to Av. 2 Integration on the space of continuous functions inside an interval 3 Differentiation on the space of continuous differentiable functions on an interval. 4 From Calculus, the gradient, the divergence are also linear maps. 5 Operations of rotation, reflection, projection, and dilation are linear transformations from R n into itself. Theorem Every linear transformation L from R n into R m is given by matrix multiplication with some m n matrix A. Why? Lemma The composition of linear maps is linear, in the case of R n, linear maps are matrices, and matrix multiplication is the composition of maps.