Linear algebra This chapter discusses the solution of sets of linear algebraic equations and defines basic vector/matrix operations The focus is upon elimination methods such as Gaussian elimination, and the related LU and Cholesky factorizations Following a discussion of these methods, the existence and uniqueness of solutions are considered Example applications include the modeling of a separation system and the solution of a fluid mechanics boundary value problem The latter example introduces the need for sparse-matrix methods and the computational advantages of banded matrices Because linear algebraic systems have, under well-defined conditions, a unique solution, they serve as fundamental building blocks in more-complex algorithms Thus, linear systems are treated here at a high level of detail, as they will be used often throughout the remainder of the text Linear systems of algebraic equations We wish to solve a system of N simultaneous linear algebraic equations for the N unknowns x, x 2,,x N, that are expressed in the general form a x + a 2 x 2 + +a N x N = b a 2 x + a 22 x 2 + +a 2N x N = b 2 () a N x + a N2 x 2 + +a NN x N = b N a ij is the constant coefficient (assumed real) that multiplies the unknown x j in equation i b i is the constant right-hand-side coefficient for equation i, also assumed real As a particular example, consider the system for which x + x 2 + x 3 = 4 2x + x 2 + 3x 3 = 7 (2) 3x + x 2 + 6x 3 = 2 a = a 2 = a 3 = b = 4 a 2 = 2 a 22 = a 23 = 3 b 2 = 7 (3) a 3 = 3 a 32 = a 33 = 6 b 3 = 2 wwwcambridgeorg
2 Linear algebra It is common to write linear systems in matrix/vector form as Ax = b (4) where a a 2 a 3 a N a 2 a 22 a 23 a 2N A = a N a N2 a N3 a NN x x 2 x = x N b b 2 b = b N (5) Row i of A contains the values a i, a i2,,a in that are the coefficients multiplying each unknown x, x 2,,x N in equation i Column j contains the coefficients a j, a 2 j,,a Nj that multiply x j in each equation i =, 2,,N Thus, we have the following associations, coefficients multiplying rows equations columns a specific unknown in each equation We often write Ax = b explicitly as a a 2 a N x b a 2 a 22 a 2N x 2 = b 2 a N a N2 a NN x N b N (6) For the example system (2), 4 A = 2 3 b = 7 (7) 3 6 2 In MATLAB we solve Ax = b with the single command, x=a\b For the example (2), we compute the solution with the code A=[;23;36]; b = [4; 7; 2]; x=a\b, x= 9-7 -8 Thus, we are tempted to assume that, as a practical matter, we need to know little about how to solve a linear system, as someone else has figured it out and provided us with this handy linear solver Actually, we shall need to understand the fundamental properties of linear systems in depth to be able to master methods for solving more complex problems, such as sets of nonlinear algebraic equations, ordinary and partial wwwcambridgeorg
Review of scalar, vector, and matrix operations 3 differential equations, etc Also, as we shall see, this solver fails for certain common classes of very large systems of equations, and we need to know enough about linear algebra to diagnose such situations and to propose other methods that do work in such instances This chapter therefore contains not only an explanation of how the MATLAB solver is implemented, but also a detailed, fundamental discussion of the properties of linear systems Our discussion is intended only to provide a foundation in linear algebra for the practice of numerical computing, and is continued in Chapter 3 with a discussion of matrix eigenvalue analysis For a broader, more detailed, study of linear algebra, consult Strang (23) or Golub & van Loan (996) Review of scalar, vector, and matrix operations As we use vector notation in our discussion of linear systems, a basic review of the concepts of vectors and matrices is necessary Scalars, real and complex Most often in basic mathematics, we work with scalars, ie, single-valued numbers These may be real, such as 3, 4, 5/7, 3459, or they may be complex, + 2i, /2 i, where i = The set of all real scalars is denoted R The set of all complex scalars we call C For a complex number z C, we write z = a + ib, where a, b Rand a = Re{z} = real part of z b = Im{z} = imaginary part of z (8) The complex conjugate, z = z,of z = a + ib is Note that the product zz is always real and nonnegative, z = z = a ib (9) zz = (a ib)(a + ib) = a 2 iab + iab i 2 b 2 = a 2 + b 2 () so that we may define the real-valued, nonnegative modulus of z, z, as Often, we write complex numbers in polar notation, z = zz = a 2 + b 2 () z = a + ib = z (cos θ + i sin θ) θ = tan (b/a) (2) Using the important Euler formula, a proof of which is found in the supplemental material found at the website that accompanies this book, e iθ = cos θ + i sin θ (3) wwwcambridgeorg
4 Linear algebra e [2] = (,,) v v e [] = (,,) v 3 e [3] = (,,) Figure Physical interpretation of a 3-D vector we can write z as z = z e iθ (4) Vector notation and operations We write a three-dimensional (3-D) vector v (Figure ) as v v = (5) v 3 v is real if v,,v 3 R; we then say v R 3 We can easily visualize this vector in 3- D space, defining the three coordinate basis vectors in the (x), 2(y), and 3(z) directions as to write v R 3 as e [] = e [2] = e [3] = (6) v = v e [] + e [2] + v 3 e [3] (7) We extend this notation to define R N, the set of N-dimensional real vectors, v v = v N (8) where v j Rfor j =, 2,,N By writing v in this manner, we define a column vector; however, v can also be written as a row vector, v = [v v N ] (9) The difference between column and row vectors only becomes significant when we start combining them in equations with matrices wwwcambridgeorg
Review of scalar, vector, and matrix operations 5 We write v R N as an expansion in coordinate basis vectors as v = v e [] + e [2] + +v N e [N] (2) where the components of e [ j] are Kroenecker deltas δ jk, e [ j] e [ j] e [ j] δ j 2 δ j2 = = δ jk = δ jn e [ j] N {, if j = k, if j k Addition of two real vectors v R N, w R N is straightforward, v w v + w v + w = + w 2 = + w 2 v N w N v N + w N as is multiplication of a vector v R N by a real scalar c R, v cv cv = c = c v N cv N (2) (22) (23) For all u, v, w R N and all c, c 2 R, u + (v + w) = (u + v) + w c(v + u) = cv + cu u + v = v + u (c + c 2 )v = c v + c 2 v (24) v + = v (c c 2 )v = c (c 2 v) v + ( v) = v = v where the null vector R N is = (25) We further add to the list of operations associated with the vectors v, w R N the dot (inner, scalar) product, v w = v w + w 2 + +v N w N = v k w k (26) k= wwwcambridgeorg
6 Linear algebra For example, for the two vectors v = 2 4 w = 5 (27) 3 6 v w = v w + w 2 + v 3 w 3 = ()(4) + (2)(5) + (3)(6) = 4 + + 8 = 32 (28) For 3-D vectors, the dot product is proportional to the product of the lengths and the cosine of the angle between the two vectors, where the length of v is v w = v w cos θ (29) v = v v (3) Therefore, when two vectors are parallel, the magnitude of their dot product is maximal and equals the product of their lengths, and when two vectors are perpendicular, their dot product is zero These ideas carry completely into N- dimensions The length of a vector v R N is v = v v = N vk 2 (3) If v w =, v and w are said to be orthogonal, the extension of the adjective perpendicular from R 3 to R N If v w = and v = w =, ie, both vectors are normalized to unit length, v and w are said to be orthonormal The formula for the length v of a vector v R N satisfies the more general properties of a norm v of v R N A norm v is a rule that assigns a real scalar, v R, to each vector v R N such that for every v, w R N, and for every c R,wehave k= v = v = if and only if (iff) v = cv = c v v + w v + w (32) Each norm also provides an accompanying metric, a measure of how different two vectors are d(v, w) = v w (33) In addition to the length, many other possible definitions of norm exist The p-norm, v p, of v R N is [ ] /p v p = v k p (34) k= wwwcambridgeorg
Review of scalar, vector, and matrix operations 7 Table p-norm values for the 3-D vector (, 2, 3) p v p 6 2 4 = 3742 35 5 39 The length of a vector is thus also the 2-normFor v = [ 2 3], the values of the p-norm, computed from (35), are presented in Table v p = [ p + 2 p + 3 p ] /p = [() p + (2) p + (3) p ] /p (35) We define the infinity norm as the limit of v p as p, which merely extracts from v the largest magnitude of any component, v lim p v p = max j [,N] { v j } (36) For v = [ 2 3], v = 3 Like scalars, vectors can be complex We define the set of complex N-dimensional vectors as C N, and write each component of v C N as v j = a j + ib j a j, b j R i = (37) The complex conjugate of v C N, written as v or v *,is a + ib a ib v a 2 + ib 2 a 2 ib 2 = = a N + ib N a N ib N (38) For complex vectors v, w C N, to form the dot product v w, we take the complex conjugates of the first vector s components, v w = vk w k (39) This ensures that the length of any v C is always real and nonnegative, v 2 2 = vk v k = k= k= (a k ib k )(a k + ib k ) = k= For v, w C N, the order of the arguments is significant, k= ( a 2 k + bk 2 ) (4) v w = (w v) w v (4) wwwcambridgeorg
8 Linear algebra Matrix dimension For a linear system Ax = b, a a 2 a N a 2 a 22 a 2N A = a N a N2 a NN x x 2 x = x N b b 2 b = b N (42) to have a unique solution, there must be as many equations as unknowns, and so typically A will have an equal number N of columns and rows and thus be a square matrix A matrix is said to be of dimension M N if it has M rows and N columns We now consider some simple matrix operations Multiplication of an M N matrix A by a scalar c a a 2 a N ca ca 2 ca N a 2 a 22 a 2N ca = c = ca 2 ca 22 ca 2N a M a M2 a MN ca M ca M2 ca MN (43) Addition of an M N matrix A with an equal-sized M N matrix B a a N b b N a 2 a 2N + b 2 b 2N a M a MN b M b MN a + b a N + b N a 2 + b 2 a 2N + b 2N = a M + b M a MN + b MN (44) Note that A + B = B + A and that two matrices can be added only if both the number of rows and the number of columns are equal for each matrix Also, c(a + B) = ca+ cb Multiplication of a square N N matrix A with an N-dimensional vector v This operation must be defined as follows if we are to have equivalence between the coefficient and matrix/vector representations of a linear system: a a 2 a N v a v + a 2 + +a N v N a 2 a 22 a 2N Av = = a 2 v + a 22 + +a 2N v N a N a N2 a NN v N a N v + a N2 + +a NN v N (45) wwwcambridgeorg
Review of scalar, vector, and matrix operations 9 Av is also an N-dimensional vector, whose j th component is (Av) j = a j v + a j2 + +a jn v N = a jk v k (46) k= We compute (Av) j by summing a jk v k along rows of A and down the vector, [ a jk ] v k Multiplication of an M N matrix A with an N-dimensional vector v From the rule for forming Av, we see that the number of columns of A must equal the dimension of v; however, we also can define Av when M N, a a 2 a N v a v + a 2 + +a N v N a 2 a 22 a 2N Av = = a 2 v + a 22 + +a 2N v N a M a M2 a MN v N a M v + a M2 + +a MN v N (47) If v R N, for an M N matrix A, Av R M Consider the following examples: 2 3 4 4 3 2 2 3 3 = 2 2 3 4 3 4 2 3 4 3 2 4 5 6 2 = 32 (48) 3 5 6 4 29 Note also that A(cv) = cav and A(v + w) = Av + Aw Matrix transposition We define for an M N matrix A the transpose A T to be the N M matrix a a 2 a N A T a 2 a 22 a 2N = a M a M2 a MN T a a 2 a M a 2 a 22 a M2 = a N a 2N a NM (49) wwwcambridgeorg
Linear algebra The transpose operation is essentially a mirror reflection across the principal diagonal a, a 22, a 33, Consider the following examples: T [ ] T 4 2 3 4 7 2 3 = = (5) 4 5 6 2 5 3 6 4 5 6 7 8 9 2 5 8 3 6 9 If a matrix is equal to its transpose, A = A T, it is said to be symmetric Then, a ij = ( A T) ij = a ji i, j {, 2,, N} (5) Complex-valued matrices Here we have defined operations for real matrices; however, matrices may also be complexvalued, c c N (a + ib 2 ) (a N + ib N ) c 2 c 2N C = = (a 2 + ib 2 ) (a 2N + ib 2N ) (52) c M c MN (a M + ib M ) (a MN + ib MN ) For the moment, we are concerned with the properties of real matrices, as applied to solving linear systems in which the coefficients are real Vectors as matrices Finally, we note that the matrix operations above can be extended to vectors by considering a vector v R N to be an N matrix if in column form and to be a N matrix if in row form Thus, for v, w R N, expressing vectors by default as column vectors, we write the dot product as v w = v T w = [v v N ] w w N = v w + +v N w N (53) The notation v T w for the dot product v w is used extensively in this text Elimination methods for solving linear systems With these basic definitions in hand, we now begin to consider the solution of the linear system Ax = b, in which x, b R N and A is an N N real matrix We consider here elimination methods in which we convert the linear system into an equivalent one that is easier to solve These methods are straightforward to implement and work generally for any linear system that has a unique solution; however, they can be quite costly (perhaps prohibitively so) for large systems Later, we consider iterative methods that are more effective for certain classes of large systems wwwcambridgeorg