B553 Lecture 5: Matrix Algebra Review
|
|
- Esther Sanders
- 6 years ago
- Views:
Transcription
1 B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations of vector quantities. This lecture will present standard matrix notation, conventions, and basic identities that will be used throughout this course. During the course of this discussion we will also drop the boldface notation for vectors, and it will remain this way for the rest of the class. 1 Matrices A matrix A represents a linear transformation of an n-dimensional vector space to an m-dimensional one. It is given by an m n array of real numbers. Usually matrices are denoted as uppercase letters (e.g., A, B, C), with the entry in the i th row and j th column denoted in the subscript i,j, or when it is unambiguous, ij (e.g., A 1,2, A 1p ). A = A 1,1 A 1,n.. A m,n A m,n (1) 1
2 1.1 Matrix-Vector Product An m n matrix A transforms vectors x = (x 1,..., x n ) into m-dimensional vectors y = (y 1,..., y m ) = Ax as follows: y 1 = y m = A 1j x j... A mj x j Or, more concisely, y i = n A ijx j for i = 1,..., m. (Note that matrixvector multiplication is not symmetric, so xa is an invalid operation.) Linearity of matrix-vector multiplication. We can see that matrixvector multiplication is linear, that is A(ax+by) = aax+bay for all a, b, x, and y. It is also linear in terms of component-wise addition and multiplication of matrices, as long as the matrices are of the same size. More precisely, if A and B are both m n matrices, then (aa + bb)x = aax + bbx for all a, b, and x. Identity matrix. One special matrix that occurs frequently is the n n identity matrix I n, which has 0 s in all off-diagonal positions I ij with i j, and 1 s in all diagonal positions I ii. It is significant because I n x = x for all x R n. 1.2 Matrix Product When two linear transformations are performed one after the other, the result is also a linear transformation. Suppose A is m n, B is n p, and x is a p-dimensional vector, and consider the result of A(Bx) (that is, first multiplying by B and then multiplying the result by A). We see that Bx = ( p B 1j x j,..., (2) p B nj x j ) (3) and Ay = ( A 1k y k,..., A mk y k ) (4) 2
3 So ( A(Bx) = A 1k ( p B kj x j ),..., Rearranging the summations, we see that ( p A(Bx) = ( A 1k B kj )x j ),..., A mk ( ) p B kj x j ). (5) ) p ( A mk B kj x j ). (6) In other words, we could have A(Bx) = Cx if we were to form a matrix C such that C ij = A ik B kj (7) This is exactly the definition of the matrix product, and we say C = AB. The entry C ij of can also be obtained taking the dot-product of the i th column of A and the j th column of B. Matrix product is associative but not symmetric. By the above derivation we can drop the parentheses A(Bx) = (AB)x. So, matrix-vector and matrix-matrix multiplication are associative. Note again however that matrix-matrix multiplication is not symmetric, that is AB BA in general. Column and row vectors. Note that if we were to write an n-dimensional vector x stacked in a n 1 matrix x (denoted in lowercase), we can turn the matrix-vector y = Ax into the matrix product y = Ax. Here, if A is an m n matrix, then y is an m 1 matrix. y 1. y m = A 1,1 A 1,n.. A m,n A m,n x 1. x n (8) Hence, there is a one-to-one correspondence between vectors and matrices with one column. These matrices are called column vectors and will be our default notation for vectors throughout the rest of the course. We will occasionally also deal with row vectors, which are matrices with a single row. 1.3 Transpose The transpose A T of a matrix A simply switches A s rows and columns. (A T ) ij = A ji. (9) 3
4 If A is m n, then A T is n m. Symmetric matrix. If A = A T, then A is symmetric. 1.4 Matrix Inverse An inverse A 1 of an n n square matrix A is a matrix that satisfies the following equation: AA 1 = A 1 A = I n (10) where I n is the identity matrix. Not all square matrices have an inverse, in which case we say A is not invertible (or singular). Invertible matrices are significant because the unique solution x to the system of linear equations Ax = b, is simply A 1 b. This holds for any b. If the matrix is not invertible, then such an equation may or may not have a solution. Orthogonal matrix. An orthogonal matrix is a square matrix that satisfies AA T = I n. In other words, its transpose is its inverse. 1.5 Matrix identities Identities involving the transpose: (ca) T = ca T for any real value c. (A + B) T = A T + B T. (AB) T = B T A T. All 1 1 matrices are symmetric, the identity matrix is symmetric, and all uniform scalings of a symmetric matrix are symmetric. A + A T is symmetric. The dot product x y is equal to x T y, with x and y denoting the column vector representations of x and y, respectively. x T Ay = y T A T x, with x and y column vectors. Identities involving the inverse: I 1 n = I n. 4
5 (ca) 1 = 1 c A 1 for any real value c 0. (AB) 1 = B 1 A 1 if both B and A are invertible. If A and B are invertible, then (ABA 1 ) 1 = AB 1 A Common mistakes Matrix expressions are similar to standard expressions regarding real numbers in that addition and subtraction are equivalent, multiplication is nearly equivalent, and inverses give an approximation of division. But, this similarity leads to common pitfalls when manipulating matrix equations. Here are some common mistakes that you should look out for. 1. Swapping the arguments of a matrix product. 2. Propagating transposes or inverses into a matrix product without swapping the order of arguments. 3. Assuming that a matrix is invertible (or worse, assuming a non-square matrix is invertible). 4. Performing operations on matrices of incompatible size. 2 Rank, Null space, and Definiteness If A is not invertible (for instance, it may not be square) then the system of linear equations Ax = b may not have a solution x. Or, it may have an infinite number of solutions. Or, it may have solutions for some b s and not others. We would like to characterize, based on properties of A, when such equations can be solved. 2.1 Matrix rank Consider the columns of A as a list of vectors a 1,..., a n. Recall that if b Span(a 1,..., a n ), then b is a linear combination of a 1,..., a n. If this holds, then it is sufficient to set each component x i to the respective coefficient on a i in order to solve Ax = b. On the other hand, if b / Span(a 1,..., a n ), then 5
6 there is not solution. So, the set of vectors b such that Ax = b has a solution is precisely Span(a 1,..., a n ). Rank. The rank of an m n matrix A is the size of the largest subset of {a 1,..., a n } that is linearly independent. In other words, if A has rank k, then Span(a 1,..., a n ) is an k-dimensional subspace of R m. If k = n, then A is said to have full column rank, and such problems have at most one solution. If k = m, then A is said to have full row rank, and such problems have at least one solution. If k = m = n, then A is invertible. Overdetermined system. Now suppose that the rank of A is k < m. Then there are some possible values of b that are not attainable by linear combinations of a 1,..., a n. Such systems are known as overdetermined because there are more constraints than can be fulfilled by adjusting the values of x. Overdetermined systems are usually not solved exactly, but are more often solved in a least squares sense min x Ax b 2. Underdetermined system. If the rank of A is k < n, then there are an infinite number of solutions x to the equation Ax 0 = Ax. To see this, let some column of A be linearly dependent on the remaining columns. Suppose this column is a 1 without loss of generality. Then, a 1 n i=2 c ia i = 0 for some coefficients c i. So, any multiple of the vector v = (1, c 2,..., c n ) can be added to x 0 without affecting the value of A(x 0 + cv). Such systems are known as underdetermined because they may be solved by multple values of x. A system can be both underdetermined and overdetermined if k < m and k < n. This means there are some values of b for which there is no solution, but for those that do have a solution, there are an infinite number of solutions. 2.2 Null space For underdetermined systems with k < n, we ask how many directions d can we move in to preserve Ad = 0? The space of such directions is known as the null space. These are significant because if we move a point x in any such direction, we leave the value of Ax = A(x + d) = Ax + Ad = Ax unchanged. It turns out that this space can be spanned by n k linearly independent directions, and is therefore a space of dimension n k. (Null spaces will feature prominently in constrained optimization problems.) 6
7 2.3 Positive/Negative Definiteness A symmetric square matrix A is positive semi-definite if for all vectors x, x T Ax 0. It is strictly positive definite if equality holds only for x = 0. It can be shown that positive definite matrices are invertible. The inverse of a positive definite matrix is positive definite as well. Although it is not clear at the moment what this condition means, it will become important in later lectures. Many matrices that we encounter will be shown to be positive definite! For example, the matrix (A T A) for a matrix A of full column rank is positive definite. Also, a local minimum of a scalar field, the Hessian matrix is positive definite. Likewise, a matrix for which x T Ax 0 is called negative semi-definite, and is called strictly negative definite if equality holds only at x = 0. If none of these conditions holds, the matrix is called indefinite. 3 Matrix Factorizations Several matrix factorizations have proven useful in numerical analysis, computer science, and engineering. It is a good idea to familiarize yourself with these factorizations so that you can apply them. 3.1 Eigenvalues and Eigenvectors If there exist a real number λ and vector x such that Ax = λx, then λ and x are known as an eigenvalue and eigenvector of A, respectively. Briefly, here are some facts about eigenvalues. 1. All matrices have at least one and at most n distinct eigenvalues. 2. Symmetric matrices have real eigenvalues. 3. Positive definite matrices have a full set of real, positive eigenvalues. 4. Positive semi-definite matrices have real, nonnegative eigenvalues. 5. Nonsymmetric matrices may have complex eigenvalues and eigenvectors. 7
8 Eigendecomposition. Symmetric matrices A can be decomposed into the form QΛQ T, where Λ is a diagonal matrix and Q is an orthogonal matrix. Λ is related to Q in that the ith entry of Λ is an eigenvalue that corresponds to the i th column of Q, which is its eigenvector. The significance of this decomposition is that multiplication by a symmetric matrix can be represented by a rotation transformation, then an axisaligned scaling, then an inverse rotation. It also gives a convenient form for the inverse, and to test whether an inverse exists. If every element of the diagonal of Λ is nonzero, then A 1 = QΛ 1 Q T. Λ 1 is easy to compute because it simply requires taking the reciprocal of each element on the diagonal. 3.2 Decompositions into Triangular Forms LU decomposition. It can be shown that using the Gaussian elimination procedure, any matrix A can be decomposed into A = P LU, where P is a permutation matrix, L is a lower triangular matrix, and U is an upper triangular matrix. This decomposition is significant because permutation matrices are easily invertible, and triangular matrices are easily invertible if their diagonals are nonzero. (The solution to any invertible triangular matrix equation Lx = b can be found quickly through a backsubstitution procedure) So, if L, and U are invertible, then A is invertible as well! This method is very frequently employed to solve an invertible system of equations. Cholesky decomposition. The special case of the LU decomposition of a symmetric positive-definite matrix is known as a Cholesky decomposition. It can be seen that to be symmetric, U = L T, and hence A = LL T. For symmetric indefinite matrices, there is a related Cholesky decomposition into LDL T, where D is a diagonal matrix. Cholesky decompositions can be be computed in slightly fewer steps than general LU decompositions. 3.3 Singular Value Decomposition The singular value decomposition (SVD) is one of the most useful tools in scientific computing. It gives a similar factorizaton to the eigendecomposition, but can be applied to non-square matrices. It also gives convenient solutions to find a matrix s rank and null space, and to compute pseudoinverses. It is the most common method used to perform principal components analysis (PCA) in statistics and machine learning, and in generalizing New- 8
9 ton s method to higher dimensions. It can also be used to perform robust least-squares fitting in underdetermined systems. The SVD of an m n matrix A takes the form: A = UΣV T (11) where U is an m m orthogonal matrix, V is an n n orthogonal matrix, and Σ is an m n matrix with nonzero entries only on the diagonal. Computing the rank. The rank of A is equal to the number of nonzero elements on the diagonal of Σ. Computing the nullspace. If Σ ii = 0 for some i, then the i th column of V T is in the null space of A. The set of all such columns of V T is an orthogonal basis of the null space. If these vectors are assembled into an n (n k) matrix N, then all solutions to the equation Ax = b can be obtained by finding a single solution x 0, and letting x = x 0 + Ny any arbitrary choice of y R n k. Computing the pseudoinverse. A pseudoinverse is a generalization of the inverse of a matrix that is used when an inverse does not exist. It can also be used when a matrix is not square. The pseudoinverse A + is defined as an n m matrix that has the following properties: 1. AA + A = A 2. A + AA + = A + 3. (AA + ) T = AA + 4. (A + A) T = A + A This matrix can be computed using the SVD. Note that the pseudoinverse Σ + of Σ can be computed by taking the reciprocal of all nonzero diagonal entries of Σ, and leaving the zero entries. Then the pseudoinverse of A is A + = V Σ + U T (convince yourself that this satisfies the properties of the pseudoinverse). Note that if A is invertible, then A + = A 1. Robust least squares. The SVD can be used to solve for all least-squares solutions to a system linear equations, whether the system is full rank, underdetermined, overdetermined, or both! It can be shown that x 0 = A + b is 9
10 a least-squares solution to min x Ax b 2. To see this, take the gradient of this quadratic function at x 0 2A T (Ax 0 b) = A T (AA + b b) = (A T AA + A T )b (12) Now look at the transpose of the matrix above, apply the transpose rule, apply the third property of the pseudoinverse, and then apply the first property of the pseudoinverse: (A T AA + A T ) T = (AA + ) T A A = AA + A A = A A = 0 (13) Hence, the gradient at x 0 is zero. Since we can also compute the null-space matrix N, we see that all vectors of the form x = A + b + Ny, with y arbitrary, are least squares solutions as well. 4 Software considerations 4.1 Software libraries Software libraries for basic matrix operations are available in most languages. Examples include LAPACK, GSL, JAMA for Java, and Numpy for Python. Matlab is a special purpose language devised explicitly to make matrix calculations convenient. Most packages will provide the Cholesky decomposition, LU decomposition, QR decomposition, and SVD. They typically also provide eigenvalue/eigenvector computations for symmetric positive definite matrices, and sometimes for nonsymmetric matrices as well. 4.2 Computational Complexity For square matrices, matrix-vector multiplication is O(n 2 ), while the naive approach to matrix multiplication is O(n 3 ). There are algorithms that achieve a slightly lower exponent, but these are not typically not competitive in practice because of large hidden constants. Matrix inversion is as complex as matrix multiplication, and is typically solved using the O(n 3 ) LU decomposition or the Cholesky decomposition if the matrix is symmetric (also O(n 3 ) but with a smaller constant factor). Eigendecompositions and SVDs are also O(n 3 ) but with yet a larger constant factor. 10
11 4.3 Sparse Matrices Sparse matrices matrices in which most entries are zero arise in many applications including physical simulation and problems on graphs. Sparse matrices can be stored in less than O(n 2 ) space, and many operations (addition, multiplication) can be performed in time proportional to the number of nonzero entries rather than the size of the matrix. Solving a sparse system of equations Ax = b can often be solved efficiently using the conjugate gradient method. See J. Shewchuck (1994) An Introduction to the Conjugate Gradient Method Without the Agonizing Pain for a good (and entertainingly written) reference on this method. 5 Exercises 1. 11
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationLinear Algebra and Matrix Inversion
Jim Lambers MAT 46/56 Spring Semester 29- Lecture 2 Notes These notes correspond to Section 63 in the text Linear Algebra and Matrix Inversion Vector Spaces and Linear Transformations Matrices are much
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationLinear Algebra Primer
Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................
More informationMATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 1 x 2. x n 8 (4) 3 4 2
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS SYSTEMS OF EQUATIONS AND MATRICES Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
More informationThe Singular Value Decomposition
The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13 Review of Key Concepts We review some key definitions and results about matrices that will
More informationLinear Algebra. The analysis of many models in the social sciences reduces to the study of systems of equations.
POLI 7 - Mathematical and Statistical Foundations Prof S Saiegh Fall Lecture Notes - Class 4 October 4, Linear Algebra The analysis of many models in the social sciences reduces to the study of systems
More informationLinear Algebra Review. Vectors
Linear Algebra Review 9/4/7 Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa (UCSD) Cogsci 8F Linear Algebra review Vectors
More informationLinear Algebra for Machine Learning. Sargur N. Srihari
Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationJim Lambers MAT 610 Summer Session Lecture 1 Notes
Jim Lambers MAT 60 Summer Session 2009-0 Lecture Notes Introduction This course is about numerical linear algebra, which is the study of the approximate solution of fundamental problems from linear algebra
More informationElementary maths for GMT
Elementary maths for GMT Linear Algebra Part 2: Matrices, Elimination and Determinant m n matrices The system of m linear equations in n variables x 1, x 2,, x n a 11 x 1 + a 12 x 2 + + a 1n x n = b 1
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationMatrices: 2.1 Operations with Matrices
Goals In this chapter and section we study matrix operations: Define matrix addition Define multiplication of matrix by a scalar, to be called scalar multiplication. Define multiplication of two matrices,
More informationEE731 Lecture Notes: Matrix Computations for Signal Processing
EE731 Lecture Notes: Matrix Computations for Signal Processing James P. Reilly c Department of Electrical and Computer Engineering McMaster University September 22, 2005 0 Preface This collection of ten
More informationMath Linear Algebra Final Exam Review Sheet
Math 15-1 Linear Algebra Final Exam Review Sheet Vector Operations Vector addition is a component-wise operation. Two vectors v and w may be added together as long as they contain the same number n of
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationCS 246 Review of Linear Algebra 01/17/19
1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector
More informationIntroduction to Matrices
POLS 704 Introduction to Matrices Introduction to Matrices. The Cast of Characters A matrix is a rectangular array (i.e., a table) of numbers. For example, 2 3 X 4 5 6 (4 3) 7 8 9 0 0 0 Thismatrix,with4rowsand3columns,isoforder
More informationBasic Elements of Linear Algebra
A Basic Review of Linear Algebra Nick West nickwest@stanfordedu September 16, 2010 Part I Basic Elements of Linear Algebra Although the subject of linear algebra is much broader than just vectors and matrices,
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More informationStat 159/259: Linear Algebra Notes
Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the
More informationANALYTICAL MATHEMATICS FOR APPLICATIONS 2018 LECTURE NOTES 3
ANALYTICAL MATHEMATICS FOR APPLICATIONS 2018 LECTURE NOTES 3 ISSUED 24 FEBRUARY 2018 1 Gaussian elimination Let A be an (m n)-matrix Consider the following row operations on A (1) Swap the positions any
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More informationNumerical Linear Algebra
Chapter 3 Numerical Linear Algebra We review some techniques used to solve Ax = b where A is an n n matrix, and x and b are n 1 vectors (column vectors). We then review eigenvalues and eigenvectors and
More informationLinear Algebra and Eigenproblems
Appendix A A Linear Algebra and Eigenproblems A working knowledge of linear algebra is key to understanding many of the issues raised in this work. In particular, many of the discussions of the details
More informationFoundations of Matrix Analysis
1 Foundations of Matrix Analysis In this chapter we recall the basic elements of linear algebra which will be employed in the remainder of the text For most of the proofs as well as for the details, the
More informationThe Singular Value Decomposition (SVD) and Principal Component Analysis (PCA)
Chapter 5 The Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) 5.1 Basics of SVD 5.1.1 Review of Key Concepts We review some key definitions and results about matrices that will
More informationMaths for Signals and Systems Linear Algebra in Engineering
Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE
More informationMathematical foundations - linear algebra
Mathematical foundations - linear algebra Andrea Passerini passerini@disi.unitn.it Machine Learning Vector space Definition (over reals) A set X is called a vector space over IR if addition and scalar
More informationLinear Algebra V = T = ( 4 3 ).
Linear Algebra Vectors A column vector is a list of numbers stored vertically The dimension of a column vector is the number of values in the vector W is a -dimensional column vector and V is a 5-dimensional
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationNotes on Eigenvalues, Singular Values and QR
Notes on Eigenvalues, Singular Values and QR Michael Overton, Numerical Computing, Spring 2017 March 30, 2017 1 Eigenvalues Everyone who has studied linear algebra knows the definition: given a square
More informationNumerical Linear Algebra
Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix
More informationAppendix C Vector and matrix algebra
Appendix C Vector and matrix algebra Concepts Scalars Vectors, rows and columns, matrices Adding and subtracting vectors and matrices Multiplying them by scalars Products of vectors and matrices, scalar
More information3 (Maths) Linear Algebra
3 (Maths) Linear Algebra References: Simon and Blume, chapters 6 to 11, 16 and 23; Pemberton and Rau, chapters 11 to 13 and 25; Sundaram, sections 1.3 and 1.5. The methods and concepts of linear algebra
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationConceptual Questions for Review
Conceptual Questions for Review Chapter 1 1.1 Which vectors are linear combinations of v = (3, 1) and w = (4, 3)? 1.2 Compare the dot product of v = (3, 1) and w = (4, 3) to the product of their lengths.
More informationA Brief Outline of Math 355
A Brief Outline of Math 355 Lecture 1 The geometry of linear equations; elimination with matrices A system of m linear equations with n unknowns can be thought of geometrically as m hyperplanes intersecting
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization
More informationScientific Computing: Dense Linear Systems
Scientific Computing: Dense Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 February 9th, 2012 A. Donev (Courant Institute)
More informationChapter 4 - MATRIX ALGEBRA. ... a 2j... a 2n. a i1 a i2... a ij... a in
Chapter 4 - MATRIX ALGEBRA 4.1. Matrix Operations A a 11 a 12... a 1j... a 1n a 21. a 22.... a 2j... a 2n. a i1 a i2... a ij... a in... a m1 a m2... a mj... a mn The entry in the ith row and the jth column
More informationSystems of Linear Equations and Matrices
Chapter 1 Systems of Linear Equations and Matrices System of linear algebraic equations and their solution constitute one of the major topics studied in the course known as linear algebra. In the first
More informationMath 4A Notes. Written by Victoria Kala Last updated June 11, 2017
Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 37 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationReview Questions REVIEW QUESTIONS 71
REVIEW QUESTIONS 71 MATLAB, is [42]. For a comprehensive treatment of error analysis and perturbation theory for linear systems and many other problems in linear algebra, see [126, 241]. An overview of
More information1 9/5 Matrices, vectors, and their applications
1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric
More informationBASIC NOTIONS. x + y = 1 3, 3x 5y + z = A + 3B,C + 2D, DC are not defined. A + C =
CHAPTER I BASIC NOTIONS (a) 8666 and 8833 (b) a =6,a =4 will work in the first case, but there are no possible such weightings to produce the second case, since Student and Student 3 have to end up with
More informationAppendix A: Matrices
Appendix A: Matrices A matrix is a rectangular array of numbers Such arrays have rows and columns The numbers of rows and columns are referred to as the dimensions of a matrix A matrix with, say, 5 rows
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationSystems of Linear Equations and Matrices
Chapter 1 Systems of Linear Equations and Matrices System of linear algebraic equations and their solution constitute one of the major topics studied in the course known as linear algebra. In the first
More informationPOLI270 - Linear Algebra
POLI7 - Linear Algebra Septemer 8th Basics a x + a x +... + a n x n b () is the linear form where a, b are parameters and x n are variables. For a given equation such as x +x you only need a variable and
More informationSolution of Linear Equations
Solution of Linear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 7, 07 We have discussed general methods for solving arbitrary equations, and looked at the special class of polynomial equations A subclass
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More informationforms Christopher Engström November 14, 2014 MAA704: Matrix factorization and canonical forms Matrix properties Matrix factorization Canonical forms
Christopher Engström November 14, 2014 Hermitian LU QR echelon Contents of todays lecture Some interesting / useful / important of matrices Hermitian LU QR echelon Rewriting a as a product of several matrices.
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationDeep Learning Book Notes Chapter 2: Linear Algebra
Deep Learning Book Notes Chapter 2: Linear Algebra Compiled By: Abhinaba Bala, Dakshit Agrawal, Mohit Jain Section 2.1: Scalars, Vectors, Matrices and Tensors Scalar Single Number Lowercase names in italic
More informationA Review of Linear Algebra
A Review of Linear Algebra Mohammad Emtiyaz Khan CS,UBC A Review of Linear Algebra p.1/13 Basics Column vector x R n, Row vector x T, Matrix A R m n. Matrix Multiplication, (m n)(n k) m k, AB BA. Transpose
More informationMATH 315 Linear Algebra Homework #1 Assigned: August 20, 2018
Homework #1 Assigned: August 20, 2018 Review the following subjects involving systems of equations and matrices from Calculus II. Linear systems of equations Converting systems to matrix form Pivot entry
More informationBackground Mathematics (2/2) 1. David Barber
Background Mathematics (2/2) 1 David Barber University College London Modified by Samson Cheung (sccheung@ieee.org) 1 These slides accompany the book Bayesian Reasoning and Machine Learning. The book and
More informationIntroduction to Matrix Algebra
Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p
More informationMAT 610: Numerical Linear Algebra. James V. Lambers
MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic
More informationVector Spaces, Orthogonality, and Linear Least Squares
Week Vector Spaces, Orthogonality, and Linear Least Squares. Opening Remarks.. Visualizing Planes, Lines, and Solutions Consider the following system of linear equations from the opener for Week 9: χ χ
More informationMatrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =
30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can
More informationLecture Notes in Linear Algebra
Lecture Notes in Linear Algebra Dr. Abdullah Al-Azemi Mathematics Department Kuwait University February 4, 2017 Contents 1 Linear Equations and Matrices 1 1.2 Matrices............................................
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More information. =. a i1 x 1 + a i2 x 2 + a in x n = b i. a 11 a 12 a 1n a 21 a 22 a 1n. i1 a i2 a in
Vectors and Matrices Continued Remember that our goal is to write a system of algebraic equations as a matrix equation. Suppose we have the n linear algebraic equations a x + a 2 x 2 + a n x n = b a 2
More informationCheng Soon Ong & Christian Walder. Canberra February June 2017
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2017 (Many figures from C. M. Bishop, "Pattern Recognition and ") 1of 141 Part III
More informationMTH 464: Computational Linear Algebra
MTH 464: Computational Linear Algebra Lecture Outlines Exam 2 Material Prof. M. Beauregard Department of Mathematics & Statistics Stephen F. Austin State University February 6, 2018 Linear Algebra (MTH
More informationImage Registration Lecture 2: Vectors and Matrices
Image Registration Lecture 2: Vectors and Matrices Prof. Charlene Tsai Lecture Overview Vectors Matrices Basics Orthogonal matrices Singular Value Decomposition (SVD) 2 1 Preliminary Comments Some of this
More informationMATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
1 MATHEMATICS FOR COMPUTER VISION WEEK 2 LINEAR SYSTEMS Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 2 Linear Systems and solutions Systems of linear
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationNumerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??
Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement
More informationMobile Robotics 1. A Compact Course on Linear Algebra. Giorgio Grisetti
Mobile Robotics 1 A Compact Course on Linear Algebra Giorgio Grisetti SA-1 Vectors Arrays of numbers They represent a point in a n dimensional space 2 Vectors: Scalar Product Scalar-Vector Product Changes
More informationLarge Scale Data Analysis Using Deep Learning
Large Scale Data Analysis Using Deep Learning Linear Algebra U Kang Seoul National University U Kang 1 In This Lecture Overview of linear algebra (but, not a comprehensive survey) Focused on the subset
More informationA Quick Tour of Linear Algebra and Optimization for Machine Learning
A Quick Tour of Linear Algebra and Optimization for Machine Learning Masoud Farivar January 8, 2015 1 / 28 Outline of Part I: Review of Basic Linear Algebra Matrices and Vectors Matrix Multiplication Operators
More information[Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty.]
Math 43 Review Notes [Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty Dot Product If v (v, v, v 3 and w (w, w, w 3, then the
More informationLinear Algebra Review. Fei-Fei Li
Linear Algebra Review Fei-Fei Li 1 / 51 Vectors Vectors and matrices are just collections of ordered numbers that represent something: movements in space, scaling factors, pixel brightnesses, etc. A vector
More informationMatrix Algebra for Engineers Jeffrey R. Chasnov
Matrix Algebra for Engineers Jeffrey R. Chasnov The Hong Kong University of Science and Technology The Hong Kong University of Science and Technology Department of Mathematics Clear Water Bay, Kowloon
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More informationMIT Final Exam Solutions, Spring 2017
MIT 8.6 Final Exam Solutions, Spring 7 Problem : For some real matrix A, the following vectors form a basis for its column space and null space: C(A) = span,, N(A) = span,,. (a) What is the size m n of
More informationLinear Algebra. Min Yan
Linear Algebra Min Yan January 2, 2018 2 Contents 1 Vector Space 7 1.1 Definition................................. 7 1.1.1 Axioms of Vector Space..................... 7 1.1.2 Consequence of Axiom......................
More informationThe study of linear algebra involves several types of mathematical objects:
Chapter 2 Linear Algebra Linear algebra is a branch of mathematics that is widely used throughout science and engineering. However, because linear algebra is a form of continuous rather than discrete mathematics,
More informationLecture 6. Numerical methods. Approximation of functions
Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Least-square method basis functions design matrix residual weighted least squares normal equation
More informationMODELS USING MATRICES WITH PYTHON
MODELS USING MATRICES WITH PYTHON José M. Garrido Department of Computer Science May 2016 College of Computing and Software Engineering Kennesaw State University c 2015, J. M. Garrido Models Using Matrices
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1
More informationSingular Value Decomposition
Singular Value Decomposition (Com S 477/577 Notes Yan-Bin Jia Sep, 7 Introduction Now comes a highlight of linear algebra. Any real m n matrix can be factored as A = UΣV T where U is an m m orthogonal
More informationLinear Algebra, Summer 2011, pt. 2
Linear Algebra, Summer 2, pt. 2 June 8, 2 Contents Inverses. 2 Vector Spaces. 3 2. Examples of vector spaces..................... 3 2.2 The column space......................... 6 2.3 The null space...........................
More informationChapter Two Elements of Linear Algebra
Chapter Two Elements of Linear Algebra Previously, in chapter one, we have considered single first order differential equations involving a single unknown function. In the next chapter we will begin to
More informationLinear Algebra. Chapter Linear Equations
Chapter 3 Linear Algebra Dixit algorizmi. Or, So said al-khwarizmi, being the opening words of a 12 th century Latin translation of a work on arithmetic by al-khwarizmi (ca. 78 84). 3.1 Linear Equations
More informationGlossary of Linear Algebra Terms. Prepared by Vince Zaccone For Campus Learning Assistance Services at UCSB
Glossary of Linear Algebra Terms Basis (for a subspace) A linearly independent set of vectors that spans the space Basic Variable A variable in a linear system that corresponds to a pivot column in the
More informationLinear Algebra March 16, 2019
Linear Algebra March 16, 2019 2 Contents 0.1 Notation................................ 4 1 Systems of linear equations, and matrices 5 1.1 Systems of linear equations..................... 5 1.2 Augmented
More informationThere are six more problems on the next two pages
Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with
More informationMath113: Linear Algebra. Beifang Chen
Math3: Linear Algebra Beifang Chen Spring 26 Contents Systems of Linear Equations 3 Systems of Linear Equations 3 Linear Systems 3 2 Geometric Interpretation 3 3 Matrices of Linear Systems 4 4 Elementary
More informationThis operation is - associative A + (B + C) = (A + B) + C; - commutative A + B = B + A; - has a neutral element O + A = A, here O is the null matrix
1 Matrix Algebra Reading [SB] 81-85, pp 153-180 11 Matrix Operations 1 Addition a 11 a 12 a 1n a 21 a 22 a 2n a m1 a m2 a mn + b 11 b 12 b 1n b 21 b 22 b 2n b m1 b m2 b mn a 11 + b 11 a 12 + b 12 a 1n
More information