Linear algebra forms the basis for much of modern mathematics theoretical, applied, and computational. The purpose of this book is to provide a broad and solid foundation for the study of advanced mathematics. A secondary aim is to introduce the reader to many of the interesting applications of linear algebra. Detailed outline of the book Chapter 1 is optional reading; it provides a concise exposition of three main emphases of linear algebra: linear equations, best approximation, and diagonalization (that is, decoupling variables). No attempt is made to give precise definitions or results; rather, the intent is to give the reader a preview of some of the questions addressed by linear algebra before the abstract development begins. Most students studying a book like this will already know how to solve systems of linear algebraic equations, and this knowledge is a prerequisite for the first three chapters. Gaussian elimination with back substitution is not presented until Section 3.7, where it is used to illustrate the theory of linear operator equations developed in the first six sections of Chapter 3. The discussion of Gaussian elimination was delayed advisedly; this arrangement of the material emphasizes the nature of the book, which presents the theory of linear algebra and does not emphasize mechanical calculations. However, if this arrangement is not suitable for a given class of students, there is no reason that Section 3.7 cannot be presented early in the course. Many of the examples in the text involve spaces of functions and elementary calculus, and therefore a course in calculus is needed to appreciate much of the material. The core of the book is formed by Chapters 2, 3, 4, and 6. They present an axiomatic development of the most important elements of finite-dimensional linear algebra: vector spaces, linear operators, norms and inner products, and determinants and eigenvalues. Chapter 2 begins with the concept of a field, of which the primary examples are R (the field of real numbers) and C (the field of complex numbers). Other examples are finite fields, particularly Z p, the field of integers modulo p (where p is a prime number). As much as possible, the results in the core part of the book (particularly Chapters 2 4) are phrased in terms of an arbitrary field, and examples are given that involve finite fields as well as the more standard fields of real and complex numbers. Once fields are introduced, the concept of a vector space is introduced, xv
xvi Preface along with the primary examples that will be studied in the text: Euclidean n-space and various spaces of functions. This is followed by the basic ideas necessary to describe vector spaces, particularly finite-dimensional vector spaces: subspace, spanning sets, linear independence, and basis. Chapter 2 ends with two optional application sections, Lagrange polynomials (which form a special basis for the space of polynomials) and piecewise polynomials (which are useful in many computational problems, particularly in solving differential equations). These topics are intended to illustrate why we study the common properties of vector spaces and bases: In a variety of applications, common issues arise, so it is convenient to study them abstractly. In addition, Section 2.8.1 presents an application to discrete mathematics: Shamir s scheme for secret sharing, which requires interpolation in a finite field. Chapter 3 discusses linear operators, linear operator equations, and inverses of linear operators. Central is the fact that every linear operator on finite-dimensional spaces can be represented by a matrix, which means that there is a close connection between linear operator equations and systems of linear algebraic equations. As mentioned above, it is assumed in Chapter 2 that the reader is familiar with Gaussian elimination for solving linear systems, but the algorithm is carefully presented in Section 3.7, where it is used to illustrate the abstract results on linear operator equations. Applications for Chapter 3 include linear ordinary differential equations (viewed as linear operator equations), Newton s method for solving systems of nonlinear equations (which illustrates the idea of linearization), the use of matrices to represent graphs, binary linear block codes, and linear programming. Eigenvalues and eigenvectors are introduced in Chapter 4, where the emphasis is on diagonalization, a technique for decoupling the variables in a system so that it can be more easily understood or solved. As a tool for studying eigenvalues, the determinant function is first developed. Elementary facts about permutations are needed; these are developed in Appendix B for the reader who has not seen them before. Results about polynomials form further background for Chapter 4, and these are derived in Appendix C. Chapter 4 closes with two interesting applications in which linear algebra is key: systems of constant coefficient linear ordinary differential equations and integer programming. Chapter 4 shows that some matrices can be diagonalized, but others cannot. After this, there are two natural directions to pursue, given in Chapters 5 and 8. One is to try to make a nondiagonalizable matrix as close to diagonal form as possible; this is the subject of Chapter 5, and the result is the Jordan canonical form. As an application, the matrix exponential is presented, which completes the discussion of systems of ordinary differential equations that was begun in Chapter 4. A brief introduction to the spectral theory of graphs is also presented in Chapter 5. The remainder of the text does not depend on Chapter 5. Chapter 6 is about orthogonality and its most important application, best approximation. These concepts are based on the notion of an inner prod-
xvii uct and the norm it defines. The central result is the projection theorem, which shows how to find the best approximation to a given vector from a finite-dimensional subspace (an infinite-dimensional version appears in Chapter 10). This is applied to problems such as solving overdetermined systems of linear equations and approximating functions by polynomials. Orthogonality is also useful for representing vector spaces in terms of orthogonal subspaces; in particular, this gives a detailed understanding of the four fundamental subspaces defined by a linear operator. Application sections address weighted polynomial approximation, the Galerkin method for approximating solutions to differential equations, Gaussian quadrature (that is, numerical integration), and the Helmholtz decomposition for vector fields. Symmetric (and Hermitian) matrices have many special properties, including the facts that all their eigenvalues are real, their eigenvectors can be chosen to be orthogonal to one another, and every such matrix can be diagonalized. Chapter 7 develops these facts and includes applications to optimization and spectral methods for differential equations. Diagonalization is an operation applied to square matrices, in which one tries to choose a special basis (a basis of eigenvectors) that results in diagonal form. In fact, it is always possible to obtain a diagonal form, provided two bases are used (one for the domain and another for the co-domain). This leads to the singular value decomposition (SVD) of a matrix, which is the subject of Chapter 8. The SVD has many advantages over the Jordan canonical form. It exists even for non-square matrices; it can be computed in finite-precision arithmetic (whereas the Jordan canonical form is unstable and typically completely obscured by the round-off inherent to computers); the bases involved are orthonormal (which means that operations with them are stable in finiteprecision arithmetic). All of these advantages make the SVD a powerful tool in computational mathematics, whereas the Jordan canonical form is primarily a theoretical tool. As an application of the SVD, Chapter 8 includes a brief study of linear inverse problems. It also includes a discussion of the Smith normal form, which is used in discrete mathematics to study properties of integer matrices. To use linear algebra in practical applications (whether they be to other areas of mathematics or to problems in science and engineering), it is typically necessary to do one or both of the following: Perform linear algebraic calculations on a computer (in finite-precision arithmetic), and introduce ideas from analysis about convergence. Chapter 9 includes a brief survey of the most important facts from numerical linear algebra, the study of computer algorithms for problems in linear algebra. Chapter 10 extends some results from single-variable analysis to Euclidean n-space, with an emphasis on the fact that all norms define the same notion of convergence on a finite-dimensional vector space. It then presents a very brief introduction to functional analysis, which is the study of linear algebra in infinite-dimensional vector spaces. In such settings, analysis is critical.
xviii Preface Exercises Each section in the text contains exercises, which range from the routine to quite challenging. The results of some exercises are used later in the text; these are labeled essential exercises, and the student should at least read these to be familiar with the results. Each section contains a collection of miscellaneous exercises, which illustrate, verify, and extend the results of the section. Some sections contain projects, which lead the student to develop topics that had to be omitted from the text for lack of space. Figures Figures appearing in the text were prepared using MATLAB R. For product information, please contact: The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 USA Tel: 508-647-7000 Fax: 508-647-7001 E-mail: info@mathworks.com Web: www.mathworks.com Applications Twenty optional sections introduce the reader to various applications of linear algebra. In keeping with the goal of this book (to prepare the reader for further studies in mathematics), these applications show how linear algebra is essential in solving problems involving differential equations, optimization, approximation, and combinatorics. They also illustrate why linear algebra should be studied as a distinct subject: Many different problems can be addressed using vector spaces and linear operators. Here is a list of the application sections in the text: 2.8 Polynomial interpolation and the Lagrange basis; includes a discussion of Shamir s method of secret sharing 2.9 Continuous piecewise polynomial functions 3.8 Newton s method 3.9 Linear ordinary differential equations 3.10 Graph theory 3.11 Coding theory 3.12 Linear programming
xix 4.8 Systems of linear ODEs 4.9 Integer programming 5.5 The matrix exponential 5.6 Graphs and eigenvalues 6.8 More on polynomial approximation 6.9 The energy inner product and Galerkin s method 6.10 Gaussian quadrature 6.11 The Helmholtz decomposition 7.3 Optimization and the Hessian matrix 7.4 Lagrange multipliers 7.5 Spectral methods for differential equations 8.4 The SVD and linear inverse problems 8.5 The Smith normal form of a matrix Possible course outlines A basic course includes Sections 2.1 2.7, 3.1 3.7, 4.1 4.6, 6.1 6.7, 7.1 7.2, and either 5.1 5.4 or 8.1 8.3. I cover all the material in these sections, except that I only summarize Sections 4.1 4.4 (determinants) in two lectures to save more time for applications. I otherwise cover one section per day, so this material requires 30 or 31 lectures. Allowing up to five days for exams and review, this leaves about six or seven days to discuss applications (in a 14- week, three-credit course). An instructor could cover fewer applications to allow time for a complete discussion of the material on determinants and the background material in Appendices B and C. In this way, all the material can be developed in a rigorous fashion. An instructor with more class meetings has many more options, including dipping into Chapters 9 and 10. Students should at least be aware of this material. The book s web site (www.math.mtu.edu/~msgocken/fdlabook) includes solutions to selected odd-numbered exercises and an up-to-date list of errors with corrections. Readers are invited to alert me of suspected errors by email. Mark S. Gockenbach msgocken@mtu.edu