# A TOUR OF LINEAR ALGEBRA FOR JDEP 384H

Save this PDF as:

Size: px
Start display at page:

Download "A TOUR OF LINEAR ALGEBRA FOR JDEP 384H" ## Transcription

1 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H Contents Solving Systems 1 Matrix Arithmetic 3 The Basic Rules of Matrix Arithmetic 4 Norms and Dot Products 5 Norms 5 Dot Products 6 Linear Programming 7 Eigenvectors and Eigenvalues 9 Solving Systems Solving linear systems is a fundamental activity, practiced from high school forward Recall that a linear equation is one of the form a 1 x 1 + a 2 x a n x n = b, where x 1, x 2,, x n are the variables and a 1, a 2,, a n, b are known constants A linear system is a collection of linear equations, and a solution to the linear system is a choice of the variables that satises every equation in the system What can we expect? Example 6 Use geometry to answer the question with one or more equations in 2 unknowns Ditto 3 unknowns Basic Fact (Number of Solutions): A linear system of m equations in n unknowns is either inconsistent (no solutions) or consistent, in which case it either has a unique solution or innitely many solutions In the former case, unknowns cannot exceed equations If the number of equations exceeds the number of unknowns, the system is called overdetermined If the number of unknowns exceeds the number of equations, the system is called underdetermined While we're on this topic, let's review some basic linear algebra as well: 1

2 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 2 The general form for a linear system of m equations in the n unknowns x 1, x 2,, x n is a 11 x 1 + a 12 x a 1j x j + a 1n x n = b 1 a 21 x 1 + a 22 x a 2j x j + a 2n x n = b 2 a i1 x 1 + a i2 x a ij x j + a in x n = b i a m1 x 1 + a m2 x a mj x j + a mn x n = b m Notice how the coecients are indexed: in the ith row the coecient of the jth variable, x j, is the number a ij, and the right hand side of the ith equation is b i The statement A = [a ij ] means that A is a matrix (rectangular table of numbers) whose (i, j)th entry, ie, entry in the ith row and jth column, is denoted by a ij Generally, the size of A will be clear from context If we want to indicate that A is an m n matrix, we write A = [a ij ] m,n Similarly, the statement b = [b i ] means that b is a column vector (matrix with exactly one column) whose ith entry is denoted by b i, and c = [c j ] means that c is a row vector (matrix with exactly one row) whose jth entry is denoted by c j In case the type of the vector (row or column) is not clear from context, the default is a column vector How can we describe the matrices of the general linear system described above? First, there is the m n coecient matrix A = a 11 a 12 a 1j a 1n a 21 a 22 a 2j a 2n a i1 a i2 a ij a in a m1 a m2 a mj a mn Notice that the way we subscripted entries of this matrix is really very descriptive: the rst index indicates the row position of the entry and the second, the column position of the entry Next, there is the m 1

3 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 3 right hand side vector of constants b 1 b 2 b = b i b m Finally, stack this matrix and vector along side each other (we use a vertical bar below to separate the two symbols) to obtain the m (n+1) augmented matrix Ã = [A b] = a 11 a 12 a 1j a 1n b 1 a 21 a 22 a 2j a 2n b 2 a i1 a i2 a ij a in b i a m1 a m2 a mj a mn b m Matrix Arithmetic Given any two matrices (or vectors) A = [a ij ] and B = [b ij ] of the same size, we may add them or multiply by a scalar according to the rules and A + B = [a ij + b ij ] ca = [ca ij ] For each size m n we can form a zero matrix of size m n: just use zero for every entry This is a handy matrix for matrix arithmetic Matlab knows all about these arithmetic operations, as we saw in our Matlab introduction Moreover, there is a multiplication of matrices A and B provided that A is m p and B is p n The result is an m n matrix given by the formula [ p ] AB = a ik b kj k=1 Again, Matlab knows all about matrix multiplication

4 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 4 The Basic Rules of Matrix Arithmetic There are many algebra rules for matrix arithmetic We won't review all of them here, but one noteworthy fact is that matrix multiplication is not commutative: AB BA in general Another is that there is no general cancellation of matrices: if AB = 0 Another is that there is an identity matrix for any square size n n: I n is the matrix with ones down the diagonal entries and zeros elsewhere For example, I 3 = We normally just write I and n should be clear from context Algebra fact: if A is m n and B is n m, then AI = A and IB = B One of the main virtues of matrix multiplication is that is gives us a symbolic way of representing the general linear system given above, namely, we can put it in the simple looking form Ax = b, where A and b are given as above, namely where Ax = b, x = is the (column) vector of unknowns Actually, this is putting the cart before the horse because this idea actually inspires the whole notion of matrix multiplication This form suggests a way of thinking about the solution to the system, namely, x 1 x 2 x i x n x = b A The only problem is that this doesn't quite make sense in the matrix world But we can make it sensible by a little matrix algebra: imagine that there were a multiplicative inverse matrix A 1 for A, that is, a matrix such that A 1 A = I = AA 1

5 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 5 Then we could multiply both sides of Ax = b on the left by A 1 and obtain that x = A 1 b Matlab represents this equation by x = A\b In general, a square matrix A may not have an inverse One condition for an inverse to exist is that the matrix have nonzero determinant, where this is a number associated with a matrix such as one saw in high school algebra with Cramer's rule We won't go into details here Of course, Matlab knows all about inverses and we can compute the inverse of A by issuing the commands A^(-1) or inv(a) In fact, Matlab even knows many more subtle matrix constructions For example, try evaluating B= A^(1/2) and follow it with B*B Sets of all matrices or vectors of a given size form number systems with a scalar multiplication and addition operation that satises a list of basic laws Such a system is a vector space For example, for vectors from R 3, we have operations a 1 a 2 a 3 + c b 1 b 2 a 1 a 2 a 3 b 3 = a 1 + b 1 a 2 + b 2 a 3 + b 3 ca 1 ca 2 ca 3 = and similar operations for n-vectors of the vector space R n or the vector space of n n matrices R n n Norms and Dot Products Norms Before we get started, there are some useful ideas that we need to explore We already know how to measure the size of a scalar (number) quantity x: use x as a measure of its size Thus, we have a way of thinking about things like the size of an error Now suppose we are dealing with vectors and matrices Question: How do we measure the size of more complex quantities like vectors or matrices? Answer: We use some kind of yardstick called a norm that assigns to each vector x a non-negative number x subject to the following norm laws for arbitrary vectors x, y and scalar c: For x 0, x > 0 and for x = 0, x = 0 cx = c x x + y x + y

6 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 6 Examples: For vector space R n and vector x = [x 1, x 2 ; ; x n ], 1-norm: x 1 = x 1 + x x n 2-norm (the default norm): x 2 = ( x x x 2 n ) 1/2 -norm: x = max ( x 1, x 2,, x n ) Let's do some examples in Matlab, which knows all of these norms: >x=[1;-3;2;-1], y=[2;0;-1;2] >norm(x) >norm(x,1) >norm(x,2) >norm(x,inf) >norm(-2*x),abs(-2)*norm(x) >norm(x+y),norm(x)+norm(y) Matrix Norms: For every vector norm x of n-vectors there is a corresponding induced norm on the vector space of n n matrices A by the formula A = max x 0 Ax x = max Ax x =1 These induced norms satisfy all the basic norm laws and, in addition: Ax A x (compatible norms) AB A B (multiplicative norm) One can show directly that if A = [a ij ], then n A 1 = max a ij 1 j n i=1 A 2 = ρ (A T A), where ρ(b) is largest eigenvalue of square matrix B in absolute value n A = max a ij 1 i n j=1 Dot Products The 2-norm is special It can be derived from another operation on vectors called the dot product Here is how it works Denition The dot product of vectors u, v in R n is the scalar u v = u T v This operation satises a long list of properties, called the inner product properties: u u 0 with equality if and only if u = 0

7 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 7 u v = v u u (v + w) = u v + u w c (u v) = (cu) v What really makes this idea important are the following facts Here we understand that u means the 2-norm u 2 u = u u If θ is an angle between the vectors represented in suitable coordinate space, then u v = u u cos θ Once we have the idea of angle between vectors, we can speak of orthogonal (perpendicular) vectors, ie, vectors for which the angle between them is π/2 (90 ) Given any two vectors u, v, with v 0, the formula proj v u = u v v v v denes a vector parallel to v such that u proj v u is orthogonal to v Linear Programming Here we understand that a vector inequality like x 0 is satised if the inequality is satised in each coordinate With this notation we can express a general linear programming problem as follows: (LP) Find a value of x that minimizes the linear function f (x) = c T x subject to conditions that Ax b, Bx = c and x 0 A somewhat restricted form is a linear programming problem in canonical form: (CLP) Find a value of x that minimizes the linear function f (x) = c T x subject to conditions that Ax b and x 0 Notes: (1) The problem (CLP) is an optimization problem because its objective is the nd an extremum (maximum or minimum) of a scalar function (2) The problem is in canonical form if it is expressed as above or is a maximization problem or has inequalities the other way: Ax b (3) The function f (x) = c T x is called the objective function (4) The variables that occur in f (x), ie, the coordinates of x, are called decision variables

8 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 8 (5) The set of all vectors x that satisfy the constraints is called the feasible set A linear programming problem is said to be in standard form if the only inequalities are simple ones of the form x 0 and all other constraints on the variables are linear equalities So such a linear programming problem takes the form (SLP) Find a value of x that minimizes the linear function f (x) = c T x subject to conditions that Ax = b and x 0 In a sense all the problems thus far expressed are equivalent because one form can be put into another with the following tricks: (1) A linear inequality like 2x 1 +x 2 4 is turned into an equality by adding in a nonnegative slack variable x 3 to get 2x 1 +x 2 +x 3 = 4 at the price of adding one new variable (2) An inequality going the wrong way like 2x 1 +x 2 4 is straightened out by multiplying by 1 to obtain 2x 1 x 2 4 Now add in a slack variable as in 1 (3) Another way to handle a wrong way linear inequality is to subtract a nonnegative surplus variable x 3 to obtain the equality 2x 1 + x 2 x 3 = 4 (4) An equality like x 1 + x 2 = 3 is turned into an inequality by replacing the equation by two inequalities x 1 + x 2 3 and x 1 x 2 3 (5) A maximization problem like maximizing 006x x 2 is equivalent to minimizing the negative of the objective function, ie, minimize 006x 1 008x 2 In general, it is not clear that a linear programming problem even has a solution If there is a solution, then there is one that occurs on the boundary of the feasible set and more specically, at a corner (called an extreme point) of the feasible set Such a solution is called a basic feasible solution The three possible outcomes are the following: (1) The feasible set is empty In this case there is no solution to the linear programming problem (2) There are feasible points, but no optimum for the objective function If this happens, the feasible set is unbounded and we can make the objective function arbitrarily large (in the case of searching for a maximum) by moving along an unbounded line in the feasible set (3) The problem has a solution This always happens when the feasible set is nonempty and bounded, but it may also happen if the feasible set is unbounded However, there are exactly two possibilities: a unique solution or innitely many solutions

9 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 9 In the latter case, every solution can be expressed as a linear combination of basic feasible solutions Generally, the solution of (LP) or discovery there is none follows two steps if the so-called simplex method is used: Phase I: nd a basic feasible solution If none is found, conclude the feasible set is empty and there is no solution Phase II: nd an optimal basic feasible solution by moving from one to another that reduces the objective function, or conclude that function values are unbounded and there is no solution Other methods for solving (LP) problems that have gained popularity in recent years are the so-called interior point methods They involve ideas from non-linear optimization and are rather more sophisticated than the simplex method In contrast, the simplex method is simple to understand and has stood the test of time fairly well It is still the method of choice for small to medium sized linear programming problems Eigenvectors and Eigenvalues An eigenvalue for a square n n matrix A is a number λ such that for some NONZERO vector x, called an eigenvector for λ, Ax = λx Eigenvalue Facts: (1) The eigenvalues of A are precisely the roots of the nth degree polynomial p (λ) = λi A, where denotes determinant (2) An n n matrix A has exactly n eigenvalues, counting multiplicities and complex numbers (3) The eigenvalues of αi + βa are just α + βλ, where λ runs over the eigenvalues of A (4) More generally, if r(λ) = p(λ)/q(λ) is any rational function, ie, p, q are polynomials, then the eigenvalues of r(a) are r(λ), where λ runs over the eigenvalues of A, provided both expressions make sense (5) The spectral radius of A is just ρ (A) = max{ λ λ is an eigenvalue of A} (6) Suppose that x 0 is an arbitrary initial vector, {b k } is an arbitrary sequence of uniformly bounded vectors, and the sequence

10 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 10 {x k }, k = 0, 1, 2,, is given by the formula x k+1 = Ax k + b k If ρ(a) < 1, then {x k } is a uniformly bounded sequence of vectors Such a matrix A is called a stable matrix (7) If ρ(a) > 1, then {x k } will not be uniformly bounded for some choices of x 0 (8) (Principal Axes Theorem) If A is a (real) symmetric matrix, then there exists an orthogonal matrix P (this means that P 1 = P T ) such that P 1 AP = D, a diagonal matrix with real entries Specically, the diagonal entries of D are the eigenvalues of A and the columns of P form an orthogonal set of unit-length eigenvectors of A (9) (Spectral Decomposition Theorem) If A is a (real) symmetric matrix, with eigenvalues λ 1, λ 2,, λ n and corresponding unit-length orthogonal eigenvectors e 1, e 2,, e n, then A = λ 1 e 1 e T 1 + λ 2 e 2 e T λ n e n e T n Let's do a few calculations with matrices to illustrate the idea of eigenvalues and eigenvectors in Matlab, which knows all about these quantities > A = [3 1 0;-1 3 1;0 3-2] > eig(a) > eig(a) > [V,D]=eig(A) > v = V(:,1),lam = D(1,1) > A*v,lam*v > x = [1;-2;3] > b = [2;1;0] > x = A*x+b % repeat this line Now let's demonstrate boundedness of an arbitrary sequence as above Try the same with a suitable A > A = A/4 > max(abs(eig(a))) > x = [1;-2;3] > x = A*x+b % repeat this line Finally, let's nd the orthogonal diagonalization guaranteed by the principal axes theorem for a symmetric matrix > A = A'*A % need a symmetric matrixthis one's SPD > [P,D] = eig(a) > P' - P^(-1) % test for P'=P^(-1)

11 A TOUR OF LINEAR ALGEBRA FOR JDEP 384H 11 > e = P(:,1) > % confirm the spectral decomposition theorem > lambda1 = D(1,1),lambda2 = D(2,2),lambda3 = D(3,3) > e1 = P(:,1),e2 = P(:,2),e3 = P(:,3) > lambda1*e1*e1'+lambda2*e2*e2'+lambda3*e3*e3' - A

### CS 246 Review of Linear Algebra 01/17/19 1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector

### Review of Linear Algebra Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=

### Math 314 Lecture Notes Section 006 Fall 2006 Math 314 Lecture Notes Section 006 Fall 2006 CHAPTER 1 Linear Systems of Equations First Day: (1) Welcome (2) Pass out information sheets (3) Take roll (4) Open up home page and have students do same

### Introduction to Matrix Algebra Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p

### 2. Matrix Algebra and Random Vectors 2. Matrix Algebra and Random Vectors 2.1 Introduction Multivariate data can be conveniently display as array of numbers. In general, a rectangular array of numbers with, for instance, n rows and p columns

### Linear Algebra: Matrix Eigenvalue Problems CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### Math Linear Algebra Final Exam Review Sheet Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of

### Linear Algebra (part 1) : Vector Spaces (by Evan Dummit, 2017, v. 1.07) 1.1 The Formal Denition of a Vector Space Linear Algebra (part 1) : Vector Spaces (by Evan Dummit, 2017, v. 1.07) Contents 1 Vector Spaces 1 1.1 The Formal Denition of a Vector Space.................................. 1 1.2 Subspaces...................................................

### 3. Vector spaces 3.1 Linear dependence and independence 3.2 Basis and dimension. 5. Extreme points and basic feasible solutions A. LINEAR ALGEBRA. CONVEX SETS 1. Matrices and vectors 1.1 Matrix operations 1.2 The rank of a matrix 2. Systems of linear equations 2.1 Basic solutions 3. Vector spaces 3.1 Linear dependence and independence

### Linear Algebra Massoud Malek CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS KOLMAN & HILL NOTES BY OTTO MUTZBAUER 11 Systems of Linear Equations 1 Linear Equations and Matrices Numbers in our context are either real numbers or complex

### Linear Algebra Highlights Linear Algebra Highlights Chapter 1 A linear equation in n variables is of the form a 1 x 1 + a 2 x 2 + + a n x n. We can have m equations in n variables, a system of linear equations, which we want to

### This appendix provides a very basic introduction to linear algebra concepts. APPENDIX Basic Linear Algebra Concepts This appendix provides a very basic introduction to linear algebra concepts. Some of these concepts are intentionally presented here in a somewhat simplified (not

### Linear Equations in Linear Algebra 1 Linear Equations in Linear Algebra 1.7 LINEAR INDEPENDENCE LINEAR INDEPENDENCE Definition: An indexed set of vectors {v 1,, v p } in n is said to be linearly independent if the vector equation x x x

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### A matrix is a rectangular array of. objects arranged in rows and columns. The objects are called the entries. is called the size of the matrix, and Section 5.5. Matrices and Vectors A matrix is a rectangular array of objects arranged in rows and columns. The objects are called the entries. A matrix with m rows and n columns is called an m n matrix.

### Matrix operations Linear Algebra with Computer Science Application Linear Algebra with Computer Science Application February 14, 2018 1 Matrix operations 11 Matrix operations If A is an m n matrix that is, a matrix with m rows and n columns then the scalar entry in the

### Linear Algebra Review. Vectors Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors

### Appendix A: Matrices Appendix A: Matrices A matrix is a rectangular array of numbers Such arrays have rows and columns The numbers of rows and columns are referred to as the dimensions of a matrix A matrix with, say, 5 rows

### Math Bootcamp An p-dimensional vector is p numbers put together. Written as. x 1 x =. x p Math Bootcamp 2012 1 Review of matrix algebra 1.1 Vectors and rules of operations An p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the

### Chapter 5. Linear Algebra. A linear (algebraic) equation in. unknowns, x 1, x 2,..., x n, is. an equation of the form Chapter 5. Linear Algebra A linear (algebraic) equation in n unknowns, x 1, x 2,..., x n, is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b where a 1, a 2,..., a n and b are real numbers. 1

### Linear Algebra. The analysis of many models in the social sciences reduces to the study of systems of equations. POLI 7 - Mathematical and Statistical Foundations Prof S Saiegh Fall Lecture Notes - Class 4 October 4, Linear Algebra The analysis of many models in the social sciences reduces to the study of systems

### Matrix Arithmetic. a 11 a. A + B = + a m1 a mn. + b. a 11 + b 11 a 1n + b 1n = a m1. b m1 b mn. and scalar multiplication for matrices via. Matrix Arithmetic There is an arithmetic for matrices that can be viewed as extending the arithmetic we have developed for vectors to the more general setting of rectangular arrays: if A and B are m n

### OPERATIONS RESEARCH. Linear Programming Problem OPERATIONS RESEARCH Chapter 1 Linear Programming Problem Prof. Bibhas C. Giri Department of Mathematics Jadavpur University Kolkata, India Email: bcgiri.jumath@gmail.com MODULE - 2: Simplex Method for

### 1 Matrices and Systems of Linear Equations. a 1n a 2n March 31, 2013 16-1 16. Systems of Linear Equations 1 Matrices and Systems of Linear Equations An m n matrix is an array A = (a ij ) of the form a 11 a 21 a m1 a 1n a 2n... a mn where each a ij is a real

### Linear Algebra (part 1) : Matrices and Systems of Linear Equations (by Evan Dummit, 2016, v. 2.02) Linear Algebra (part ) : Matrices and Systems of Linear Equations (by Evan Dummit, 206, v 202) Contents 2 Matrices and Systems of Linear Equations 2 Systems of Linear Equations 2 Elimination, Matrix Formulation

### Knowledge Discovery and Data Mining 1 (VO) ( ) Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory

### MATH.2720 Introduction to Programming with MATLAB Vector and Matrix Algebra MATH.2720 Introduction to Programming with MATLAB Vector and Matrix Algebra A. Vectors A vector is a quantity that has both magnitude and direction, like velocity. The location of a vector is irrelevant;

### AN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES AN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES JOEL A. TROPP Abstract. We present an elementary proof that the spectral radius of a matrix A may be obtained using the formula ρ(a) lim

### Contents. 2.1 Vectors in R n. Linear Algebra (part 2) : Vector Spaces (by Evan Dummit, 2017, v. 2.50) 2 Vector Spaces Linear Algebra (part 2) : Vector Spaces (by Evan Dummit, 2017, v 250) Contents 2 Vector Spaces 1 21 Vectors in R n 1 22 The Formal Denition of a Vector Space 4 23 Subspaces 6 24 Linear Combinations and

### Yinyu Ye, MS&E, Stanford MS&E310 Lecture Note #06. The Simplex Method The Simplex Method Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapters 2.3-2.5, 3.1-3.4) 1 Geometry of Linear

### MAT2342 : Introduction to Applied Linear Algebra Mike Newman, fall Projections. introduction MAT4 : Introduction to Applied Linear Algebra Mike Newman fall 7 9. Projections introduction One reason to consider projections is to understand approximate solutions to linear systems. A common example

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 1 x 2. x n 8 (4) 3 4 2 MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS SYSTEMS OF EQUATIONS AND MATRICES Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### MATH2071: LAB #5: Norms, Errors and Condition Numbers MATH2071: LAB #5: Norms, Errors and Condition Numbers 1 Introduction Introduction Exercise 1 Vector Norms Exercise 2 Matrix Norms Exercise 3 Compatible Matrix Norms Exercise 4 More on the Spectral Radius

### Matrix Multiplication 3.2 Matrix Algebra Matrix Multiplication Example Foxboro Stadium has three main concession stands, located behind the south, north and west stands. The top-selling items are peanuts, hot dogs and soda.

### Linear Algebra Review. Fei-Fei Li Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector

### Linear Algebra March 16, 2019 Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented

### Quantum Computing Lecture 2. Review of Linear Algebra Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces

### Linear Algebra and Matrix Inversion Jim Lambers MAT 46/56 Spring Semester 29- Lecture 2 Notes These notes correspond to Section 63 in the text Linear Algebra and Matrix Inversion Vector Spaces and Linear Transformations Matrices are much

### An Introduction To Linear Algebra. Kuttler An Introduction To Linear Algebra Kuttler April, 7 Contents Introduction 7 F n 9 Outcomes 9 Algebra in F n Systems Of Equations Outcomes Systems Of Equations, Geometric Interpretations Systems Of Equations,

### Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition) Vector Space Basics (Remark: these notes are highly formal and may be a useful reference to some students however I am also posting Ray Heitmann's notes to Canvas for students interested in a direct computational

### Linear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0. Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the

### Linear Algebra: Lecture Notes. Dr Rachel Quinlan School of Mathematics, Statistics and Applied Mathematics NUI Galway Linear Algebra: Lecture Notes Dr Rachel Quinlan School of Mathematics, Statistics and Applied Mathematics NUI Galway November 6, 23 Contents Systems of Linear Equations 2 Introduction 2 2 Elementary Row

### 6.1 Matrices. Definition: A Matrix A is a rectangular array of the form. A 11 A 12 A 1n A 21. A 2n. A m1 A m2 A mn A 22. 61 Matrices Definition: A Matrix A is a rectangular array of the form A 11 A 12 A 1n A 21 A 22 A 2n A m1 A m2 A mn The size of A is m n, where m is the number of rows and n is the number of columns The

### 1 Matrices and Systems of Linear Equations Linear Algebra (part ) : Matrices and Systems of Linear Equations (by Evan Dummit, 207, v 260) Contents Matrices and Systems of Linear Equations Systems of Linear Equations Elimination, Matrix Formulation

### Chapter 9: Systems of Equations and Inequalities Chapter 9: Systems of Equations and Inequalities 9. Systems of Equations Solve the system of equations below. By this we mean, find pair(s) of numbers (x, y) (if possible) that satisfy both equations.

### Introduction to Quantitative Techniques for MSc Programmes SCHOOL OF ECONOMICS, MATHEMATICS AND STATISTICS MALET STREET LONDON WC1E 7HX Introduction to Quantitative Techniques for MSc Programmes SCHOOL OF ECONOMICS, MATHEMATICS AND STATISTICS MALET STREET LONDON WC1E 7HX September 2007 MSc Sep Intro QT 1 Who are these course for? The September

### Chapter 3 Transformations Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

### Basic Concepts in Linear Algebra Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear

### Chap 3. Linear Algebra Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions

### B553 Lecture 5: Matrix Algebra Review B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### Notes on Linear Algebra and Matrix Theory Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = 30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### Review of Basic Concepts in Linear Algebra Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra

### Matrix Operations. Linear Combination Vector Algebra Angle Between Vectors Projections and Reflections Equality of matrices, Augmented Matrix Linear Combination Vector Algebra Angle Between Vectors Projections and Reflections Equality of matrices, Augmented Matrix Matrix Operations Matrix Addition and Matrix Scalar Multiply Matrix Multiply Matrix

### Vectors and Matrices Statistics with Vectors and Matrices Vectors and Matrices Statistics with Vectors and Matrices Lecture 3 September 7, 005 Analysis Lecture #3-9/7/005 Slide 1 of 55 Today s Lecture Vectors and Matrices (Supplement A - augmented with SAS proc

### 3 Matrix Algebra. 3.1 Operations on matrices 3 Matrix Algebra A matrix is a rectangular array of numbers; it is of size m n if it has m rows and n columns. A 1 n matrix is a row vector; an m 1 matrix is a column vector. For example: 1 5 3 5 3 5 8

### MATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y. as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations

### Chapter 1: Systems of linear equations and matrices. Section 1.1: Introduction to systems of linear equations Chapter 1: Systems of linear equations and matrices Section 1.1: Introduction to systems of linear equations Definition: A linear equation in n variables can be expressed in the form a 1 x 1 + a 2 x 2

### Definition 2.3. We define addition and multiplication of matrices as follows. 14 Chapter 2 Matrices In this chapter, we review matrix algebra from Linear Algebra I, consider row and column operations on matrices, and define the rank of a matrix. Along the way prove that the row

### Introduction to Matrices 214 Analysis and Design of Feedback Control Systems Introduction to Matrices Derek Rowell October 2002 Modern system dynamics is based upon a matrix representation of the dynamic equations governing the

### 14 Singular Value Decomposition 14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing

### Linear Algebra Review. Fei-Fei Li Linear Algebra Review Fei-Fei Li 1 / 37 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector

### Properties of Linear Transformations from R n to R m Properties of Linear Transformations from R n to R m MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Topic Overview Relationship between the properties of a matrix transformation

### Quick Tour of Linear Algebra and Graph Theory Quick Tour of Linear Algebra and Graph Theory CS224W: Social and Information Network Analysis Fall 2014 David Hallac Based on Peter Lofgren, Yu Wayne Wu, and Borja Pelato s previous versions Matrices and

### A primer on matrices A primer on matrices Stephen Boyd August 4, 2007 These notes describe the notation of matrices, the mechanics of matrix manipulation, and how to use matrices to formulate and solve sets of simultaneous

### Stat 206: Linear algebra Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two

### Matrices. 1 a a2 1 b b 2 1 c c π Matrices 2-3-207 A matrix is a rectangular array of numbers: 2 π 4 37 42 0 3 a a2 b b 2 c c 2 Actually, the entries can be more general than numbers, but you can think of the entries as numbers to start

### Linear Algebra Basics Linear Algebra Basics For the next chapter, understanding matrices and how to do computations with them will be crucial. So, a good first place to start is perhaps What is a matrix? A matrix A is an array

### Elementary maths for GMT Elementary maths for GMT Linear Algebra Part 2: Matrices, Elimination and Determinant m n matrices The system of m linear equations in n variables x 1, x 2,, x n a 11 x 1 + a 12 x 2 + + a 1n x n = b 1

### Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017 Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

### Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, 2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for

### Linear Algebra Review Chapter 1 Linear Algebra Review It is assumed that you have had a course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc. I will review some of these terms here, but quite

### MAT 610: Numerical Linear Algebra. James V. Lambers MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic

### Foundations of Matrix Analysis 1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the

### What is A + B? What is A B? What is AB? What is BA? What is A 2? and B = QUESTION 2. What is the reduced row echelon matrix of A = STUDENT S COMPANIONS IN BASIC MATH: THE ELEVENTH Matrix Reloaded by Block Buster Presumably you know the first part of matrix story, including its basic operations (addition and multiplication) and row

### The Singular Value Decomposition The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will

### IE 5531: Engineering Optimization I IE 5531: Engineering Optimization I Lecture 7: Duality and applications Prof. John Gunnar Carlsson September 29, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I September 29, 2010 1

### MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP) MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

### 15-780: LinearProgramming 15-780: LinearProgramming J. Zico Kolter February 1-3, 2016 1 Outline Introduction Some linear algebra review Linear programming Simplex algorithm Duality and dual simplex 2 Outline Introduction Some linear

### Chapter Two Elements of Linear Algebra Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to

### Elementary Linear Algebra Matrices J MUSCAT Elementary Linear Algebra Matrices Definition Dr J Muscat 2002 A matrix is a rectangular array of numbers, arranged in rows and columns a a 2 a 3 a n a 2 a 22 a 23 a 2n A = a m a mn We

### 7. Dimension and Structure. 7. Dimension and Structure 7.1. Basis and Dimension Bases for Subspaces Example 2 The standard unit vectors e 1, e 2,, e n are linearly independent, for if we write (2) in component form, then we obtain

### Matrix & Linear Algebra Matrix & Linear Algebra Jamie Monogan University of Georgia For more information: http://monogan.myweb.uga.edu/teaching/mm/ Jamie Monogan (UGA) Matrix & Linear Algebra 1 / 84 Vectors Vectors Vector: A

### Final Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2 Final Review Sheet The final will cover Sections Chapters 1,2,3 and 4, as well as sections 5.1-5.4, 6.1-6.2 and 7.1-7.3 from chapters 5,6 and 7. This is essentially all material covered this term. Watch

### ICS 6N Computational Linear Algebra Matrix Algebra ICS 6N Computational Linear Algebra Matrix Algebra Xiaohui Xie University of California, Irvine xhx@uci.edu February 2, 2017 Xiaohui Xie (UCI) ICS 6N February 2, 2017 1 / 24 Matrix Consider an m n matrix

### Chapter 6: Orthogonality Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products

### Understanding the Simplex algorithm. Standard Optimization Problems. Understanding the Simplex algorithm. Ma 162 Spring 2011 Ma 162 Spring 2011 February 28, 2011 Standard Optimization Problems. A standard maximization problem can be conveniently described in matrix form

### Chap6 Duality Theory and Sensitivity Analysis Chap6 Duality Theory and Sensitivity Analysis The rationale of duality theory Max 4x 1 + x 2 + 5x 3 + 3x 4 S.T. x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1 ~x 4 0 If we

### Mathematics 13: Lecture 10 Mathematics 13: Lecture 10 Matrices Dan Sloughter Furman University January 25, 2008 Dan Sloughter (Furman University) Mathematics 13: Lecture 10 January 25, 2008 1 / 19 Matrices Recall: A matrix is a

### MATRICES. a m,1 a m,n A = MATRICES Matrices are rectangular arrays of real or complex numbers With them, we define arithmetic operations that are generalizations of those for real and complex numbers The general form a matrix of

### Chapter 3. Linear and Nonlinear Systems 59 An expert is someone who knows some of the worst mistakes that can be made in his subject, and how to avoid them Werner Heisenberg (1901-1976) Chapter 3 Linear and Nonlinear Systems In this chapter

### Lecture 15, 16: Diagonalization Lecture 15, 16: Diagonalization Motivation: Eigenvalues and Eigenvectors are easy to compute for diagonal matrices. Hence, we would like (if possible) to convert matrix A into a diagonal matrix. Suppose 1 Chapter 5 Linear Programming (LP) General constrained optimization problem: minimize subject to f(x) x Ω Ω R n is called the constraint set or feasible set. any point x Ω is called a feasible point We Finite Mathematics Chapter 2 Section 2.1 Systems of Linear Equations: An Introduction Systems of Equations Recall that a system of two linear equations in two variables may be written in the general form