Lectures -2: Linear Algebra Background Almost all linear and nonlinear problems in scientific computation require the use of linear algebra These lectures review basic concepts in a way that has proven to be helpful for the understanding and development of numerical methods We will also introduce some concepts that you may not have come across in previous linear algebra courses, such as vector norm Let b = be a vector with m scalar entries b j We will say that b is an m-vector All vectors will be columns vectors, unless explicitly specified otherwise The vector entries will be real numbers and we write b R m Throughout this text vectors are represented with lower case boldface letter scalars with lower case letter Sometimes we will use Greek letters (α, β, γ, ) to denote scalars Matrices are arrays of scalars We denote matrices with upper case letters For instance, a, a,2 a,n a 2, a 2,2 a 2,n () a m, a m,2 a m,n denotes an m n-matrix and we will write A R m n When n =, the matrix simplifies to an m-vector If instead m =, then the matrix becomes a row vector with n entries The mapping x Ax is linear, which means that for any vectors x,y R n and any scalar α, we have b b 2 b m A(x + y) = Ax + Ay, A(αx) = α (Ax) Any linear mapping from R n to R m can be expressed with an m n-matrix It is often meaningful to think of a matrix as set of column vectors Many of the properties of a matrix can be inferred by studying its columns We write where Let A R m n and [a,a 2,,a n a j = x = x x 2 x n a,j a 2,j a m,j Rn
We express the matrix-vector product Ax as a linear combination of the columns of A, x x 2 Ax = [a,a 2,,a n = n x j a j x n The range of a matrix A R m n, written range(a), is the set of vectors that can be expressed as a linear combination of the columns of A Thus, range(a) = {Ax : x R n } The rank of A is the dimension of the range Hence, the rank is the number of linearly independent columns of A A matrix is said to be of full rank if its rank is min{m, n} [ 2 2 3 Then rank(a) = 2 Then rank(a) = [ 2 2 4 [ 2 3 2 4 5 Then rank(a) = 2 The null space of A R m n, denoted by null(a), is set of solutions to Ax =, ie, null(a) = {x : Ax = } The dimension of null(a) is n rank(a) [ 2 2 3 Then null(a) = {} This space is said to have dimension zero [ 2 2 4 Then {[ 2 null(a) = span } 2
Thus, dim(null(a)) = We turn to matrix-matrix products Introduce the matrices [a,a 2,,a n R m n, C = [c,c 2,,c l R m l B = [b,b 2,,b l = We express C = AB as c j = Ab j = b, b,2 b,l b 2, b 2,2 b 2,l b n, b n,2 b n,l b k,j a k, j =, 2,, l k= Rn l Let A R n n be of full rank Then there is a unique matrix, denoted by A, such that where A AA = I, I = diag[,,, R n n, denotes the identity matrix The matrix A is referred to as the inverse of A Consider the linear system of equations Ax = b Multiplying left-hand side and right-hand side by A yields x = A b Here A b is the vector of coefficients of the expansion of b in terms of the columns a j of A, e = b = Ax = where x j is the jth entry of the vector x Define the axis vectors in R n,, e 2 = x j a j,, e n = An expansion of b in terms of the axis vectors can be written as b b 2 b = b n n = b j e j 3
The purpose of the solution of linear systems of equations is to carry out a change of basis Instead of expressing b in terms of the axis vectors e j, the solution x yields the coefficients in an expansion of b in terms of the columns a j of A The transpose of an m n- matrix A, denoted by A T, is an n m-matrix, whose jth row contains the entries of the jth column of A Let A be given by () Then A T = a, a 2, a m, a,2 a 2,2 a m,2 a,n a 2,n a m,n Example: 2 3, A T = [ 2 3 A matrix is said to be symmetric if iit is equal to its transpose, ie if A T In particular, symmetric matrices have to be square Moreover, the entries a i,j of a symmetric matrix A satisfy a i,j = a j,i for all indices i, j Let be vectors in R n Then u = u u n, v = (u,v) = u j v j is the inner product between the vectors u and v Thus, the inner product is just a matrix-matrix product between the matrices u T and v An inner product is bilinear, ie, it is linear in each argument u and v, separately Specifically, for any u,v,w R n and α, β R, it holds (2) v v 2 v n (u + v,w) = (u,w) + (v,w), (u,v + w) = (u,v) + (u,w), (αu, βv) = αβ(u,v) The inner product can be used to define the Euclidean length of an n-vector, u = (u,u) = j The Euclidean length is an example of a norm A norm is a real-valued function that can be used to measure the size of a vector Specifically, a norm is a function 4
: R n R that assigns a real-valued length to each vector, such that, for all u,v R n and α R, (3) (4) (5) u, u = implies that u =, u + v u + v triangle inequality, αu = α u Exercise : Show that u = u j is a norm, ie, show that the function satisfies the properties (3), (4), and (5) This norm is often denoted by It has applications in statistics Exercise 2: Let [ u = Draw a picture of the set {u : u }, where is the norm of Exercise Exercise 3: Show that u = max j n u j is a norm This norm is often denoted by Its main advantage is that it is easy to evaluate Exercise 4: Let [ u = Draw a picture of the set {u : u }, where is the norm of Exercise 3 The norm (2) is often denoted by 2 The Cauchy inequality (6) (u,v) u 2 v 2, u,v R n, provides a relation between the Euclidean norm and the inner product Exercise 5: Use the Cauchy inequality to show that (2) is a norm Hint: Express u + v 2 2 = (u + v,u + v) in terms of u 2 2 = (u,u) and v 2 2 = (v,v) using the bilinearity of the inner product when showing the triangle inequality Exercise 5: Let [ u = Draw a picture of the set {u : u 2 } 5