Linear Algebra Review

Chapter 1 Linear Algebra Review It is assumed that you have had a beginning course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc I will review some of these terms here, but quite rapidly Sideline: Linear Algebra is part of Algebra What is Algebra? An algebraic object is usually a set with one or more operations, and some properties that these operations have to satisfy The operations may involve one element, two, or more A typical binary (two-element) operation is written c = a b We usually want associativity a (b c) = (a b) c We may or may not demand commutativity the existence of a unit element e and the existence of an inverse a b = b a, a e = e a = a, a a 1 = a 1 a = e For example, a group has one associative operation, with a unit element and an inverse A commutative group is called Abelian The operation in an Abelian group is usually written as +, the unit element as 0, and the inverse as a A ring has two operations One operation is written + and forms an Abelian group The other operation is written as It has to be distributive a (b + c) = a b + a c (a + b) c = a c + b c Multiplication does not even have to be associative, and there may or may not be a unit element 1 for multiplication 1

2 CHAPTER 1 LINEAR ALGEBRA REVIEW A field F has two operations, addition + and multiplication Under addition, F is an Abelian group has a unit element 0 and inverse a Under multiplication, F \ {0} is an Abelian group with unit element 1 and inverse 1/a The standard examples for fields are Q (the rationals), R (the reals), and C (the complex numbers) For each type of algebraic objects, we are interested in the homomorphisms These are mappings that preserve the algebraic operations For a standard binary operation this means that if f is a mapping between two objects S and T, then f(a b) = f(a) f(b) The mapping is an isomorphism if it is one-to-one, onto, and a homomorphism in both directions Now we get to Linear Algebra as a special case 11 Vector Spaces The standard object in linear algebra is a vector space Definition 11 A vector space V over a field F (the scalars) is a set of vectors with two operations: vector addition vector + vector = vector, which makes V into an Abelian group, and scalar multiplication with properties scalar vector = vector, α (β v) = (α β) v, α (v + w) = α v + α w, (α + β) v = α v + β v I am using Roman letters a, b, c, v, w for vectors, Greek letters α, β, γ for scalars, and later upper case letters A, B, for maps or matrices I will leave out the for scalar multiplication from now on, and just write αv A homomorphism L between vector spaces needs to preserve the two operations This means or equivalently L(v + w) = L(v) + L(w), L(αv) = αl(v), L(αv + βw) = αl(v) + βl(w) Such a mapping is called a linear map There are two standard examples for finite-dimensional vector spaces, which cover everything we need in this course

11 VECTOR SPACES 3 111 The Geometric View: Arrows A vector is an arrow, given by a direction and a length You can move it around to any place you want Examples are forces in physics, or velocities 112 The Analytic View: Columns of Numbers The vector space is V = R n or C n v 1 v 1 + w 1 αv 1 v 2 v =, v + w = v 2 + w 2, αv = αv 2 v n v n + w n Note: vectors are columns of numbers, not rows 113 Linear Independence and Bases αv n Let {v i } be a collection of vectors A linear combination of these vectors is α i v i i The set of all linear combinations of {v i } is called the span The {v i } are called linearly dependent if there is a set of coefficients {α i }, not all zero, for which α i v i = 0 i Otherwise, they are linearly independent A basis of V is a collection {v i } so that every w V can be written uniquely as a linear combination Theorem 12 (a) Every vector space has a basis (usually infinitely many of them) (b) Every basis has the same number of elements This is called the dimension of the space

4 CHAPTER 1 LINEAR ALGEBRA REVIEW Take an arbitrary n-dimensional vector space over F Pick a basis {e 1, e 2,, e n } Then we can equate α 1 w = α i e i w = α 2 α n V F n Thus, every finite-dimensional vector space over F is isomorphic to F n Nevertheless, it is often useful to visualize the vectors as arrows Sideline: As a generalization of F n, the space of infinite sequences {α 1, α 2, α 3, } is also a vector space, with dimension infinity Even more general, the space of functions on an interval [a, b] is a vector space In finite dimensions, we have subscripts 1,, n In the case of sequences, we have subscripts 1, 2, In the case of functions, you can think of the x in f(x) as a subscript Linear algebra for infinite-dimensional spaces is called functional analysis, and is its own topic 12 Subspaces One standard question in algebra is this: if I have a group, ring, field, etc, are there any subsets that are also such objects? The standard rules (commutativity, associativity, etc) are automatic It is just a question of whether the subset is closed under the operations in this object In the case of vector spaces, a subset W of V is a subspace if it is closed under linear combinations The smallest subspace is {0}, the largest is V itself For any collection of vectors, span({v i }) is a subspace The sum of subspaces is given by S + T = {x = v + w : v S, w T } This is again a subspace If the bases of S, T are linearly independent, or equivalenty, if S T = {0}, this is called a direct sum, written S T Now consider a linear map L from one vector space V to another W Definition 13 R(L) = range of L = {Lv : v V } W, N(L) = nullspace of L = kernel of L = {v V : Lv = 0} V

13 MATRICES 5 Both of these are subspaces The dimension of R(L) is called the rank of L Theorem 14 13 Matrices dim(r(l)) + dim(n(l)) = dim(v ) So far, L is just a mapping between vector spaces It maps vectors to other vectors Now we want to introduce coordinates We pick a basis {e i }, i = 1,, n in V, and another basis {f j }, j = 1,, m in W If v = i v ie i is an arbitrary vector in V (expressed in terms of the basis {e i }), then by linearity, w = Lv = v i Le i i We can express Le i in the basis of W : Le i = j l ji f j We collect all these numbers in a matrix L Then L v = w l 11 l 12 l 1n l 21 l 22 l 2n l m1 l m2 l mn v 1 v n = w 1 w m Note the numbering in L: the first subscript always refers to the row, the second to the column Entry l 35 is in row 3, column 5 The map L goes from F n to F m, and is of size m n (which is the opposite order) Sideline: There are two ways to think about matrix times vector multiplication The first one is the dot product interpretation: The jth entry in w is the dot product between the jth row of L, and v v 1 v 2 w j = (l j1, l j2,, l jn) = lj1v1 + lj2v2 + + ljnvn v n The second one is the linear combination interpretation: w is the linear combination of the columns of L, with coefficients from v This is the way we derived it above When you program this on a computer, matrix times vector corresponds to a double loop The loop can be executed in either order, corresponding to the two

6 CHAPTER 1 LINEAR ALGEBRA REVIEW interpretations Depending on the computer architecture, one way may be faster than the other Remark: Technically, I should distinguish between the mapping L and the matrix L, but I won t Just keep in mind that the matrix depends on the choice of basis, the mapping does not 131 Matrix Multiplication At the next level: suppose you have a linear map L from V (dimension n) to W (dimension m), and M from W to X (dimensions p) We can also consider the combined map N = M L (backwards again) You can verify that for the matrices, N = M L (p n) = (p m) (m n) n ij = k m ik l kj In words: the entry n ij in the product is the dot product of row i in M with column j in L Note: The middle dimension has to match, and gets canceled The size of the result comes from the outer numbers Likewise in the sum, the middle index gets canceled Sideline: On a computer, matrix times matrix corresponds to a triple loop, which can be executed in 6 orders, corresponding to 3 different viewpoints (Each one shows up twice, depending on the order in which the product matrix gets filled in) The first two are the dot product and linear combination interpretations from above The third one is the sum of rank one matrices interpretation If w is a row vector (size 1 n), and v is a column vector (size n 1), then w v is a scalar (1 1 matrix), and vw is a matrix of rank 1: v 1w 1 v 1w 2 v 1w n v 2w 1 v 2w 2 v 2w n v nw 1 v nw 2 v nw n

13 MATRICES 7 You can think of matrix multiplication ML as the sum of all rank one matrices produced from all possible products between columns of M and rows of L Matrix multiplication is not commutative in general: LM M L Unless the matrices are square, the two products are not even both defined, or not the same size However, matrix multiplication is associative and distributive: (LM)N = L(MN) L(M + N) = LM + LN (L + M)N = LN + MN One more observation about matrix times matrix computation: If you have N = ML, then then first column of N depends only on the first column of L, not any of the other numbers in L The second column of N depends only on the second column in L, and so on Likewise, the first row in N depends only on the first row in M, and so on This means that if you want to solve a system of matrix equations AX = B, you can treat this as a sequence of matrix-vector problems: where b i, x i are the columns of B, X 132 Special Matrices Ax 1 = b 1 Ax m = b m Let F m n be the set of matrices of size m n This becomes a vector space over F of dimension mn, with entry-by-entry addition and scalar multiplication The unit element for addition is the zero matrix: 0 0 0 0 0 0 O = 0 0 0 In F n n, the identity matrix 1 0 0 I = 0 1 0 0 0 1 satisfies I L = L I = L

8 CHAPTER 1 LINEAR ALGEBRA REVIEW For a given L, the inverse matrix L 1 satisfies L L 1 = L 1 L = I The inverse matrix may or may not exist Example: Take ( ) a b L = C 2 2 c d The inverse is L 1 = 1 ad bc ( d b c a which can be verified by multiplying The inverse exists if and only if ad bc 0 ), Sideline: The term ad bc is the determinant of the matrix There are more complicated formulas for the determinant of a larger matrix, but we won t need them Here is what the determinant means If you have a mapping from V into itself, consider the image of the unit basis vectors e i The unit vectors form a square (in dimension 2) or cube (in dimension 3) of unit area or volume After the mapping, the images form a parallelogram or parellelepiped The determinant is the area (or volume) of that The sign of the determinant has to do with the orientation If the determinant is zero, that means that the square or cube gets flattened into something lower-dimensional, and there is no inverse 14 Basis Change As I pointed out above, the mapping L is independent of the choice of basis, but the matrix L depends on the basis How does the matrix change when you change the basis? Suppose you have the standard basis {e i }, e i = (0,, 0, 1, 0,, 0) T, and a new basis {f i } Let F be the matrix with columns f i Then by definition, f i = F e i, so e i = F 1 f i Let x be an arbitrary vector In the two bases, it is expressed as x = i v i e i = w i f i = w i F e i This works out to v = F w, or w = F 1 v To get from the original representation v in basis {e i } to the new representation w in basis {f i } you have to multiply by F 1 Now assume we have a linear map from V to V In basis {e i } it is represented by a square matrix L In the original basis, consider y = Lx

15 SPECIAL MATRIX SHAPES 9 Convert to the new basis F 1 y = ( F 1 LF ) ( F 1 x ) Thus, the mapping is represented in the new basis as ( F 1 LF ) The matrices L and F 1 LF are called conjugates of each other, or similar matrices Similar matrices can be considered as the same mapping represented in two different bases Any properties of a matrix that are geometric, such as the determinant or eigenvalues, are preserved under conjugation Other properties that are analytic are not preserved, such as the special shapes of matrices listed below 15 Special Matrix Shapes Diagonal 0 0 0 0 0 0 Tridiagonal More generally, a banded matrix has several bands of numbers near the diagonal 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Triangular (upper or lower) 0 0 0 Permutation One 1 in each row or column, the rest are 0 This corresponds to a permutation of the basis vectors 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 0

10 CHAPTER 1 LINEAR ALGEBRA REVIEW 16 Block Matrices You can partition matrices into blocks: 1 2 3 4 5 6 7 8 9 10 11 12 = ( A B C D E F As long as the dimensions all fit together, you can add/multiply block matrices just like regular matrices: ( ) A B C G ( ) H AG + BH + CI = D E F DG + EH + F I I This is used extensively in numerical analysis (dividing matrix computations among multiple processors) Special Case: Suppose V has dimension n, and a there is an m-dimensional subspace S that gets mapped into itself (an invariant subspace) If we choose a basis v 1,, v m for the subspace, and then more basis vectors v m+1,, v n for a basis of all of V, then in this basis the matrix of the mapping is block upper triangular: ( ) L11 L 12 0 L 22 )