# 1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)

Save this PDF as:

Size: px
Start display at page:

Download "1 Vectors. Notes for Bindel, Spring 2017 Numerical Analysis (CS 4220)"

## Transcription

1 Notes for Most of mathematics is best learned by doing. Linear algebra is no exception. You have had a previous class in which you learned the basics of linear algebra, and you will have plenty of practice with these concepts over the semester. This brief refresher lecture is supposed to remind you of what you've already learned and introduce a few things you may not have seen. It also serves to set notation that will be used throughout the class. In addition to these notes, you may nd it useful to go back to a good linear algebra text (there are several listed on the course syllabus) or to look at the linear algebra review chapters in the book. 1 Vectors A vector space (or linear space) is a set of vectors that can be added or scaled in a sensible way that is, addition is associative and commutative and scaling is distributive. We will generally denote vector spaces by script letters (e.g. V, W), vectors by lower case Roman letters (e.g. v, w), and scalars by lower case Greek letters (e.g. α, β). But we feel free to violate these conventions according to the dictates of our conscience or in deference to other conicting conventions. There are many types of vector spaces. Apart from the ubiquitous spaces R n and C n, the most common vector spaces in applied mathematics are dierent types of function spaces. These include P d = {polynomials of degree at most d}; V = {linear functions V R (or C)}; L(V, W) = {linear maps V W}; C k (Ω) = { k-times dierentiable functions on a set Ω}; and many more. We compute with vectors in R n and C n, which we represent concretely by tuples of numbers in memory, usually stored in sequence. To keep a broader perspective, though, we will also frequently describe examples involving the polynomial spaces P d.

2 1.1 Spanning sets and bases We often think of a matrix as a set of vectors V = [ v 1 v 2... v n ]. The range space R(V ) or the span sp{v j } n j=1 is the set of vectors {V c : c R n } (or C n for complex spaces). The vectors are linearly independent if any vector in the span has a unique representation as a linear combination of the spanning vectors; equivalently, the vectors are linearly independent if there is no linear combination of the spanning vectors that equals zero except the one in which all coecients are zero. A linearly independent set of vectors is a basis for a space V if sp{v j } n j=1 = V; the number of basis vectors n is the dimension of the space. Spaces like C k (Ω) do not generally have a nite basis; they are innite-dimensional. We will focus on nite-dimensional spaces in the class, but it is useful to know that there are interesting innitedimensional spaces in the broader world. The standard basis in R n is the set of column vectors e j with a one in the jth position and zeros elsewhere: I = [ e 1 e 2... e n ]. Of course, this is not the only basis. Any other set of n linearly independent vectors in R n is also a basis V = [ v 1 v 2... v n ]. The matrix V formed in this way is invertible, with multiplication by V corresponding to a change from the {v j } basis into the standard basis and multiplication by V 1 corresponding to a map from the {v j } basis to the standard basis. We will use matrix-like notation to describe bases and spanning sets even when we deal with spaces other than R n. For example, a common basis for P d is the power basis X = [ 1 x x 2... x d]. Each column in X is really a function of one variable (x), and matrix-vector multiplication with X represents a map from a coecient vector in R d+1 to

3 a polynomial in P d. That is, we write polynomials p P d in terms of the basis as d p = Xc = c j x j and, we think of computing the coecients from the abstract polynomial via a formal inverse: c = X 1 p. We typically think of a map like Y = X 1 in terms of rows j=0 y 0 Y y 1 =. where each row yj is a linear functional or dual vector (i.e. linear mappings from the vector space to the real or complex numbers). Collectively, {y0,..., yd } are the dual basis to {1, x,..., x d }. The power basis is not the only basis for P d. Other common choices include Newton or Lagrange polynomials with respect to a set of points, which you may have seen in another class such as CS In this class, we will sometimes use the Chebyshev 1 polynomial basis {T j (x)} given by the recurrence T 0 (x) = 1 T 1 (x) = x y d T j+1 (x) = 2xT j (x) T j 1 (x), j 1, and the Legendre polynomial basis {P j (x)}, given by the recurrence P 0 (x) = 1 P 1 (x) = x (j + 1)P j+1 (x) = (2j + 1)xP j (x) jp j 1 (x). 1 Pafnuty Chebyshev ( åáûø¼â) was a nineteenth century Russian mathematician, and his name has been transliterated from the Cyrillic alphabet into the Latin alphabet in several dierent ways. We inherit our usual spelling from one of the French transliterations, but the symbol T for the polynomials comes from the German transliteration Tschebysche.

4 As we will see over the course of the semester, sometimes the obvious choice of basis (e.g. the standard basis in R n or the power basis in P d ) is not the best choice for numerical computations. 1.2 Vector norms A norm measures vector lengths. It is positive denite, homogeneous, and sub-additive: v 0 and v = 0 i v = 0 αv = α v u + v u + v. The three most common vector norms we work with in R n are the Euclidean norm (aka the 2-norm), the -norm (or max norm), and the 1-norm: v 2 = v j 2 v = max v j j v 1 = j j v j Many other norms can be related to one of these three norms. In particular, a natural norm in an abstract vector space will often look strange in the corresponding concrete representation with respect to some basis function. For example, consider the vector space of polynomials with degree at most 2 on [ 1, 1]. This space also has a natural Euclidean norm, max norm, and 1-norm; for a given polynomial p(x) these are 1 p 2 = p(x) 2 dx 1 p = max p 1 = p(x) x [ 1,1] 1 1 p(x) dx. But when we write p(x) in terms of the coecient vector with respect to the power basis (for example), the max norm of the polynomial is not the same

5 as the max norm of the coecient vector. In fact, if we consider a polynomial p(x) = c 0 + c 1 x, then the max norm of the polynomial p is the same as the one-norm of the coecient vector the proof of which is left as a useful exercise to the reader. In a nite-dimensional vector space, all norms are equivalent: that is, if and are two norms on the same nite-dimensional vector space, then there exist constants c and C such that for any v in the space, c v v C v. Of course, there is no guarantee that the constants are small! An isometry is a mapping that preserves vector norms. For R n, the only isometries for the 1-norm and the -norm are permutations. For Euclidean space, though, there is a much richer set of isometries, represented by the orthogonal matrices (matrices s.t. Q Q = I). 1.3 Inner products An inner product, is a function from two vectors into the real numbers (or complex numbers for an complex vector space). It is positive denite, linear in the rst slot, and symmetric (or Hermitian in the case of complex vectors); that is: v, v 0 and v, v = 0 i v = 0 αu, w = α u, w and u + v, w = u, w + v, w u, v = v, u, where the overbar in the latter case corresponds to complex conjugation. A vector space with an inner product is sometimes called an inner product space or a Euclidean space. Every inner product denes a corresponding norm v = v, v The inner product and the associated norm satisfy the Cauchy-Schwarz inequality u, v u v.

6 The standard inner product on C n is x y = y x = n ȳ j x j. j=1 But the standard inner product is not the only inner product, just as the standard Euclidean norm is not the only norm. Just as norms allow us to reason about size, inner products let us reason about angles. In particular, we dene the cosine of the angle θ between nonzero vectors v and w as cos θ = v, w v w. Returning to our example of a vector space of polynomials, the standard L 2 ([ 1, 1]) inner product is p, q L 2 ([ 1,1]) = 1 1 p(x) q(x) dx. If we express p and q with respect to a basis (e.g. the power basis), we ne that we can represent this inner product via a symmetric positive denite matrix. For example, let p(x) = c 0 +c 1 x+c 2 x 2 and let q(x) = d 0 +d 1 x+d 2 x 2. Then where d 0 p, q L 2 ([ 1,1]) = d 1 d 2 a ij = 1 1 x i 1 x j 1 dx = a 00 a 01 a 02 a 10 a 11 a 12 a 20 a 21 a 22 c 0 c 1 c 2 { 2/(i + j 1), = d Ac = c, d A i + j even 0, otherwise The symmetric positive denite matrix A is what is sometimes called the Gram matrix for the basis {1, x, x 2 }. We say two vectors u and v are orthogonal with respect to an inner product if u, v = 0. If u and v are orthogonal, we have the Pythagorean theorem: u + v 2 = u + v, u + v = u, u + v, u + u, v + v, v = u 2 + v 2

7 Two vectors u and v are orthonormal if they are orthogonal with respect to the inner product and have unit length in the associated norm. When we work in an inner product space, we often use an orthonormal basis, i.e. a basis in which all the vectors are orthonormal. For example, the normalized Legendre polynomials 2 2j + 1 P j(x) form orthonormal bases for the P d inner product with respect to the L 2 ([ 1, 1]) inner product. 2 Matrices and mappings A matrix represents a mapping between two vector spaces. That is, if L : V W is a linear map, then the associated matrix A with respect to bases V and W satises A = W 1 LV. The same linear mapping corresponds to dierent matrices depending on the choices of basis. But matrices can reresent several other types of mappings as well. Over the course of this class, we will see several interpretations of matrices: Linear maps. A map L : V W is linear if L(x + y) = Lx + Ly and L(αx) = αlx. The corresponding matrix is A = W 1 LV. Linear operators. A linear map from a space to itself (L : V V) is a linear operator. The corresponding (square) matrix is A = V 1 LV. Bilinear forms. A map a : V W R (or C for complex spaces) is bilinear if it is linear in both slots: a(αu + v, w) = αa(u, w) + a(v, w) and a(v, αu + w) = αa(v, u) + a(v, w). The corresponding matrix has elements A ij = a(v i, w j ); if v = V c and w = W d then a(v, w) = d T Ac. We call a bilinear form on V V symmetric if a(v, w) = a(w, v); in this case, the corresponding matrix A is also symmetric (A = A T ). A symmetric form and the corresponding matrix are called positive semidenite if a(v, v) 0 for all v. The form and matrix are positive denite if a(v, v) > 0 for any v 0. A skew-symmetric matrix (A = A T ) corresponds to a skew-symmetric or anti-symmetric bilinear form, i.e. a(v, w) = a(w, v).

8 Sesquilinear forms. A map a : V W C (where V and W are complex vector spaces) is sesquilinear if it is linear in the rst slot and the conjugate is linear in the second slot: a(αu + v, w) = αa(u, w) + a(v, w) and a(v, αu + w) = ᾱa(v, u) + a(v, w). The matrix has elements A ij = a(v i, w j ); if v = V c and w = W d then a(v, w) = d Ac. We call a sesquilinear form on V V Hermitian if a(v, w) = a(w, v); in this case, the corresponding matrix A is also Hermitian (A = A ). A Hermitian form and the corresponding matrix are called positive semidenite if a(v, v) 0 for all v. The form and matrix are positive denite if a(v, v) > 0 for any v 0. A skew-hermitian matrix (A = A ) corresponds to a skew-hermitian or anti-hermitian bilinear form, i.e. a(v, w) = a(w, v). Quadratic forms. A quadratic form φ : V R (or C) is a homogeneous quadratic function on V, i.e. φ(αv) = α 2 φ(v) for which the map b(v, w) = φ(v + w) φ(v) φ(w) is bilinear. Any quadratic form on a nitedimensional space can be represented as c Ac where c is the coecient vector for some Hermitian matrix A. The formula for the elements of A given φ is left as an exercise. We care about linear maps and linear operators almost everywhere, and most students come out of a rst linear algebra class with some notion that these are important. But apart from very standard examples (inner products and norms), many students have only a vague notion of what a bilinear form, sesquilinear form, or quadratic form might be. Bilinear forms and sesquilinear forms show up when we discuss large-scale solvers based on projection methods. Quadratic forms are important in optimization, physics (where they often represent energy), and statistics (e.g. for understanding variance and covariance). 2.1 Matrix norms The space of matrices forms a vector space; and, as with other vector spaces, it makes sense to talk about norms. In particular, we frequently want norms that are consistent with vector norms on the range and domain spaces; that is, for any w and v, we want w = Av = w A v.

9 One obvious consistent norm is the Frobenius norm, A 2 F = i,j a 2 ij. Even more useful are induced norms (or operator norms) Av A = sup v 0 v = sup Av. v =1 The induced norms corresponding to the vector 1-norm and -norm are A 1 = max a ij (max column sum) j i A = max a ij (max row sum) i j The norm induced by the vector Euclidean norm (variously called the matrix 2-norm or the spectral norm) is more complicated. The Frobenius norm and the matrix 2-norm are both orthogonally invariant (or unitarily invariant in a complex vector space. That is, if Q is a square matrix with Q = Q 1 (an orthogonal or unitary matrix) of the appropriate dimensions QA F = A F, AQ F = A F, QA 2 = A 2, AQ 2 = A 2. This property will turn out to be frequently useful throughout the course. 2.2 Decompositions and canonical forms Matrix decompositions (also known as matrix factorizations) are central to numerical linear algebra. We will get to know six such factorizations well: P A = LU (a.k.a. Gaussian elimination). Here L is unit lower triangular (triangular with 1 along the main diagonal), U is upper triangular, and P is a permutation matrix. A = LL (a.k.a. Cholesky factorization). Here A is Hermitian and positive denite, and L is a lower triangular matrix.

10 A = QR (a.k.a. QR decomposition). Here Q has orthonormal columns and R is upper triangular. If we think of the columns of A as a basis, QR decomposition corresponds to the Gram-Schmidt orthogonalization process you have likely seen in the past (though we rarely compute with Gram-Schmidt). A = UΣV (a.k.a. the singular value decomposition or SVD). Here U and V have orthonormal columns and Σ is diagonal with non-negative entries. A = QΛQ (a.k.a. symmetric eigendecomposition). Here A is Hermitian (symmetric in the real case), Q is orthogonal or unitary, and Λ is a diagonal matrix with real numbers on the diagonal. A = QT Q (a.k.a. Schur form). Here A is a square matrix, Q is orthogonal or unitary, and T is upper triangular (or nearly so). The last three of these decompositions correspond to canonical forms for abstract operators. That is, we can view these decompositions as nding bases in which the matrix representation of some operator or form is particularly simple. More particularly: SVD: For any linear mapping L : V W, there are orthonormal bases for the two spaces such that the corresponding matrix is diagonal Symmetric eigendecomposition: For any Hermitian sesquilinear map on an inner product space, there is an orthonormal basis for the space such that the matrix representation is diagonal. Schur form: For any linear operator L : V V, there is an orthonormal basis for the space such that the matrix representation is upper triangular. Equivalently, if {u 1,..., u n } is the basis in question, then sp({u j } k j=1) is an invariant subspace for each 1 k n. The Schur form turns out to be better for numerical work than the Jordan canonical form that you should have seen in an earlier class. We will discuss this in more detail when we discuss eigenvalue problems.

11 2.3 The SVD and the 2-norm The singular value decomposition is useful for a variety of reasons; we close o the lecture by showing one such use. Suppose A = UΣV is the singular value decomposition of some matrix. Using orthogonal invariance (unitary invariance) of the 2-norm, we have i.e. A 2 = U AV 2 = Σ 2, A 2 = max v 2 =1 j σ j v j 2 vj 2. That is, the spectral norm is the largest weighted average of the singular values, which is the same as just the largest singular value. The small singular values also have a meaning. If A is a square, invertible matrix then A 1 2 = V Σ 1 U 2 = Σ 1 2, i.e. A 1 2 is the inverse of the smallest singular value of A. The smallest singular value of a nonsingular matrix A can also be interpreted as the distance to singularity: if σ n is the smallest singular value of A, then there is a matrix E such that E 2 = σ n and A + E is singular; and there is no such matrix with smaller norm. These facts about the singular value decomposition are worth pondering, as they will be particularly useful in the next lecture when we ponder sensitivity and conditioning. 3 Problems to ponder 1. In terms of the power basis, write the matrix A R 4 3 corresponding to the mapping from P 2 P 3 given by multiplication by x. 2. Repeat the rst problem in terms of the basis of Legendre polynomials. 3. What is the spectral norm of the operator p xp from P 2 P 3 with respect to the L 2 ([ 1, 1]) inner product and norm? Note that it may be helpful to write the matrix in terms of the normalized Legendre polynomials and to use MATLAB's svd command.

12 4. What is the L ([ 1, 1]) norm of p xp from P 2 P 3? That is, we want the norm induced by the vector norm p = max x [ 1,1] p(x). 5. Show that if p(x) = c 0 + c 1 x P 2, then p = c 1, where the rst norm is a function max norm and the second is a norm on the coecient vector. 6. Show the law of cosines in an inner product space: for nonzero vectors u and v, u + v 2 = u 2 + v cos θ u v where cos(θ) is the angle between u and v. 7. Show that the 1-norm and -norm are equivalent, i.e. for all v R n, c v v 1 C v What are the tightest possible constants c and C? 8. Write the dual basis for the power basis {1, x, x 2 } for P 2 in terms of function evaluations for example, y 0p = p(0) gives the constant coecient.

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Lecture 3: Review of Linear Algebra

ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters, transforms,

### Lecture 3: Review of Linear Algebra

ECE 83 Fall 2 Statistical Signal Processing instructor: R Nowak, scribe: R Nowak Lecture 3: Review of Linear Algebra Very often in this course we will represent signals as vectors and operators (eg, filters,

### Lecture 3: QR-Factorization

Lecture 3: QR-Factorization This lecture introduces the Gram Schmidt orthonormalization process and the associated QR-factorization of matrices It also outlines some applications of this factorization

### Linear Algebra in Actuarial Science: Slides to the lecture

Linear Algebra in Actuarial Science: Slides to the lecture Fall Semester 2010/2011 Linear Algebra is a Tool-Box Linear Equation Systems Discretization of differential equations: solving linear equations

### 2. Review of Linear Algebra

2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### HONORS LINEAR ALGEBRA (MATH V 2020) SPRING 2013

HONORS LINEAR ALGEBRA (MATH V 2020) SPRING 2013 PROFESSOR HENRY C. PINKHAM 1. Prerequisites The only prerequisite is Calculus III (Math 1201) or the equivalent: the first semester of multivariable calculus.

### ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS. 1. Linear Equations and Matrices

ELEMENTARY LINEAR ALGEBRA WITH APPLICATIONS KOLMAN & HILL NOTES BY OTTO MUTZBAUER 11 Systems of Linear Equations 1 Linear Equations and Matrices Numbers in our context are either real numbers or complex

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### Stat 159/259: Linear Algebra Notes

Stat 159/259: Linear Algebra Notes Jarrod Millman November 16, 2015 Abstract These notes assume you ve taken a semester of undergraduate linear algebra. In particular, I assume you are familiar with the

### Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition

Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2016 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing

### EXERCISES ON DETERMINANTS, EIGENVALUES AND EIGENVECTORS. 1. Determinants

EXERCISES ON DETERMINANTS, EIGENVALUES AND EIGENVECTORS. Determinants Ex... Let A = 0 4 4 2 0 and B = 0 3 0. (a) Compute 0 0 0 0 A. (b) Compute det(2a 2 B), det(4a + B), det(2(a 3 B 2 )). 0 t Ex..2. For

### Review of some mathematical tools

MATHEMATICAL FOUNDATIONS OF SIGNAL PROCESSING Fall 2016 Benjamín Béjar Haro, Mihailo Kolundžija, Reza Parhizkar, Adam Scholefield Teaching assistants: Golnoosh Elhami, Hanjie Pan Review of some mathematical

### I. Multiple Choice Questions (Answer any eight)

Name of the student : Roll No : CS65: Linear Algebra and Random Processes Exam - Course Instructor : Prashanth L.A. Date : Sep-24, 27 Duration : 5 minutes INSTRUCTIONS: The test will be evaluated ONLY

### The Eigenvalue Problem: Perturbation Theory

Jim Lambers MAT 610 Summer Session 2009-10 Lecture 13 Notes These notes correspond to Sections 7.2 and 8.1 in the text. The Eigenvalue Problem: Perturbation Theory The Unsymmetric Eigenvalue Problem Just

### Inner product spaces. Layers of structure:

Inner product spaces Layers of structure: vector space normed linear space inner product space The abstract definition of an inner product, which we will see very shortly, is simple (and by itself is pretty

### Matrices. Chapter What is a Matrix? We review the basic matrix operations. An array of numbers a a 1n A = a m1...

Chapter Matrices We review the basic matrix operations What is a Matrix? An array of numbers a a n A = a m a mn with m rows and n columns is a m n matrix Element a ij in located in position (i, j The elements

### BASIC ALGORITHMS IN LINEAR ALGEBRA. Matrices and Applications of Gaussian Elimination. A 2 x. A T m x. A 1 x A T 1. A m x

BASIC ALGORITHMS IN LINEAR ALGEBRA STEVEN DALE CUTKOSKY Matrices and Applications of Gaussian Elimination Systems of Equations Suppose that A is an n n matrix with coefficents in a field F, and x = (x,,

### Chapter 3 Least Squares Solution of y = A x 3.1 Introduction We turn to a problem that is dual to the overconstrained estimation problems considered s

Lectures on Dynamic Systems and Control Mohammed Dahleh Munther A. Dahleh George Verghese Department of Electrical Engineering and Computer Science Massachuasetts Institute of Technology 1 1 c Chapter

### Lecture # 3 Orthogonal Matrices and Matrix Norms. We repeat the definition an orthogonal set and orthornormal set.

Lecture # 3 Orthogonal Matrices and Matrix Norms We repeat the definition an orthogonal set and orthornormal set. Definition A set of k vectors {u, u 2,..., u k }, where each u i R n, is said to be an

### Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

### Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition)

Vector Space Basics (Remark: these notes are highly formal and may be a useful reference to some students however I am also posting Ray Heitmann's notes to Canvas for students interested in a direct computational

### Linear Algebra. Workbook

Linear Algebra Workbook Paul Yiu Department of Mathematics Florida Atlantic University Last Update: November 21 Student: Fall 2011 Checklist Name: A B C D E F F G H I J 1 2 3 4 5 6 7 8 9 10 xxx xxx xxx

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2

EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko

### B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

### HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION)

HOMEWORK PROBLEMS FROM STRANG S LINEAR ALGEBRA AND ITS APPLICATIONS (4TH EDITION) PROFESSOR STEVEN MILLER: BROWN UNIVERSITY: SPRING 2007 1. CHAPTER 1: MATRICES AND GAUSSIAN ELIMINATION Page 9, # 3: Describe

### MATH 304 Linear Algebra Lecture 34: Review for Test 2.

MATH 304 Linear Algebra Lecture 34: Review for Test 2. Topics for Test 2 Linear transformations (Leon 4.1 4.3) Matrix transformations Matrix of a linear mapping Similar matrices Orthogonality (Leon 5.1

### Inverses. Stephen Boyd. EE103 Stanford University. October 28, 2017

Inverses Stephen Boyd EE103 Stanford University October 28, 2017 Outline Left and right inverses Inverse Solving linear equations Examples Pseudo-inverse Left and right inverses 2 Left inverses a number

### Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### There are two things that are particularly nice about the first basis

Orthogonality and the Gram-Schmidt Process In Chapter 4, we spent a great deal of time studying the problem of finding a basis for a vector space We know that a basis for a vector space can potentially

### a11 a A = : a 21 a 22

Matrices The study of linear systems is facilitated by introducing matrices. Matrix theory provides a convenient language and notation to express many of the ideas concisely, and complicated formulas are

### Linear Algebra Primer

Linear Algebra Primer David Doria daviddoria@gmail.com Wednesday 3 rd December, 2008 Contents Why is it called Linear Algebra? 4 2 What is a Matrix? 4 2. Input and Output.....................................

### 1 Principal component analysis and dimensional reduction

Linear Algebra Working Group :: Day 3 Note: All vector spaces will be finite-dimensional vector spaces over the field R. 1 Principal component analysis and dimensional reduction Definition 1.1. Given an

### 18.06 Problem Set 10 - Solutions Due Thursday, 29 November 2007 at 4 pm in

86 Problem Set - Solutions Due Thursday, 29 November 27 at 4 pm in 2-6 Problem : (5=5+5+5) Take any matrix A of the form A = B H CB, where B has full column rank and C is Hermitian and positive-definite

### Extra Problems for Math 2050 Linear Algebra I

Extra Problems for Math 5 Linear Algebra I Find the vector AB and illustrate with a picture if A = (,) and B = (,4) Find B, given A = (,4) and [ AB = A = (,4) and [ AB = 8 If possible, express x = 7 as

### Review of Linear Algebra Definitions, Change of Basis, Trace, Spectral Theorem

Review of Linear Algebra Definitions, Change of Basis, Trace, Spectral Theorem Steven J. Miller June 19, 2004 Abstract Matrices can be thought of as rectangular (often square) arrays of numbers, or as

### COMP6237 Data Mining Covariance, EVD, PCA & SVD. Jonathon Hare

COMP6237 Data Mining Covariance, EVD, PCA & SVD Jonathon Hare jsh2@ecs.soton.ac.uk Variance and Covariance Random Variables and Expected Values Mathematicians talk variance (and covariance) in terms of

### The Singular Value Decomposition

The Singular Value Decomposition An Important topic in NLA Radu Tiberiu Trîmbiţaş Babeş-Bolyai University February 23, 2009 Radu Tiberiu Trîmbiţaş ( Babeş-Bolyai University)The Singular Value Decomposition

### Computational math: Assignment 1

Computational math: Assignment 1 Thanks Ting Gao for her Latex file 11 Let B be a 4 4 matrix to which we apply the following operations: 1double column 1, halve row 3, 3add row 3 to row 1, 4interchange

### FINITE-DIMENSIONAL LINEAR ALGEBRA

DISCRETE MATHEMATICS AND ITS APPLICATIONS Series Editor KENNETH H ROSEN FINITE-DIMENSIONAL LINEAR ALGEBRA Mark S Gockenbach Michigan Technological University Houghton, USA CRC Press Taylor & Francis Croup

### MATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y.

as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations

### Inner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:

Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v

### Linear Algebra- Final Exam Review

Linear Algebra- Final Exam Review. Let A be invertible. Show that, if v, v, v 3 are linearly independent vectors, so are Av, Av, Av 3. NOTE: It should be clear from your answer that you know the definition.

### ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA

ALGEBRA QUALIFYING EXAM PROBLEMS LINEAR ALGEBRA Kent State University Department of Mathematical Sciences Compiled and Maintained by Donald L. White Version: August 29, 2017 CONTENTS LINEAR ALGEBRA AND

### Math 443 Differential Geometry Spring Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook.

Math 443 Differential Geometry Spring 2013 Handout 3: Bilinear and Quadratic Forms This handout should be read just before Chapter 4 of the textbook. Endomorphisms of a Vector Space This handout discusses

### MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2.

MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. Diagonalization Let L be a linear operator on a finite-dimensional vector space V. Then the following conditions are equivalent:

### Numerical Methods I Solving Square Linear Systems: GEM and LU factorization

Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,

### The Spectral Theorem for normal linear maps

MAT067 University of California, Davis Winter 2007 The Spectral Theorem for normal linear maps Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (March 14, 2007) In this section we come back to the question

### Matrices and Vectors. Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A =

30 MATHEMATICS REVIEW G A.1.1 Matrices and Vectors Definition of Matrix. An MxN matrix A is a two-dimensional array of numbers A = a 11 a 12... a 1N a 21 a 22... a 2N...... a M1 a M2... a MN A matrix can

### v = v 1 2 +v 2 2. Two successive applications of this idea give the length of the vector v R 3 :

Length, Angle and the Inner Product The length (or norm) of a vector v R 2 (viewed as connecting the origin to a point (v 1,v 2 )) is easily determined by the Pythagorean Theorem and is denoted v : v =

### Properties of Matrices and Operations on Matrices

Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,

### Finite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product

Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )

### Matrices and Matrix Algebra.

Matrices and Matrix Algebra 3.1. Operations on Matrices Matrix Notation and Terminology Matrix: a rectangular array of numbers, called entries. A matrix with m rows and n columns m n A n n matrix : a square

### Functional Analysis. James Emery. Edit: 8/7/15

Functional Analysis James Emery Edit: 8/7/15 Contents 0.1 Green s functions in Ordinary Differential Equations...... 2 0.2 Integral Equations........................ 2 0.2.1 Fredholm Equations...................

### CS168: The Modern Algorithmic Toolbox Lecture #8: How PCA Works

CS68: The Modern Algorithmic Toolbox Lecture #8: How PCA Works Tim Roughgarden & Gregory Valiant April 20, 206 Introduction Last lecture introduced the idea of principal components analysis (PCA). The

### Linear Algebra for Machine Learning. Sargur N. Srihari

Linear Algebra for Machine Learning Sargur N. srihari@cedar.buffalo.edu 1 Overview Linear Algebra is based on continuous math rather than discrete math Computer scientists have little experience with it

### 5.6. PSEUDOINVERSES 101. A H w.

5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and

### Applied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic

Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.3: QR Factorization, SVD 2 / 66 QR Factorization 3 / 66 QR Factorization

### Mathematical Methods wk 2: Linear Operators

John Magorrian, magog@thphysoxacuk These are work-in-progress notes for the second-year course on mathematical methods The most up-to-date version is available from http://www-thphysphysicsoxacuk/people/johnmagorrian/mm

### Math 396. An application of Gram-Schmidt to prove connectedness

Math 396. An application of Gram-Schmidt to prove connectedness 1. Motivation and background Let V be an n-dimensional vector space over R, and define GL(V ) to be the set of invertible linear maps V V

### Numerical Methods. Elena loli Piccolomini. Civil Engeneering. piccolom. Metodi Numerici M p. 1/??

Metodi Numerici M p. 1/?? Numerical Methods Elena loli Piccolomini Civil Engeneering http://www.dm.unibo.it/ piccolom elena.loli@unibo.it Metodi Numerici M p. 2/?? Least Squares Data Fitting Measurement

### Math 290, Midterm II-key

Math 290, Midterm II-key Name (Print): (first) Signature: (last) The following rules apply: There are a total of 20 points on this 50 minutes exam. This contains 7 pages (including this cover page) and

### Definition 1. A set V is a vector space over the scalar field F {R, C} iff. there are two operations defined on V, called vector addition

6 Vector Spaces with Inned Product Basis and Dimension Section Objective(s): Vector Spaces and Subspaces Linear (In)dependence Basis and Dimension Inner Product 6 Vector Spaces and Subspaces Definition

### Math 25a Practice Final #1 Solutions

Math 25a Practice Final #1 Solutions Problem 1. Suppose U and W are subspaces of V such that V = U W. Suppose also that u 1,..., u m is a basis of U and w 1,..., w n is a basis of W. Prove that is a basis

### MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

### Linear algebra 2. Yoav Zemel. March 1, 2012

Linear algebra 2 Yoav Zemel March 1, 2012 These notes were written by Yoav Zemel. The lecturer, Shmuel Berger, should not be held responsible for any mistake. Any comments are welcome at zamsh7@gmail.com.

### Orthogonal iteration revisited

Week 10 11: Friday, Oct 30 and Monday, Nov 2 Orthogonal iteration revisited Last time, we described a generalization of the power methods to compute invariant subspaces. That is, starting from some initial

### 1. The Polar Decomposition

A PERSONAL INTERVIEW WITH THE SINGULAR VALUE DECOMPOSITION MATAN GAVISH Part. Theory. The Polar Decomposition In what follows, F denotes either R or C. The vector space F n is an inner product space with

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Linear Algebra & Geometry why is linear algebra useful in computer vision?

Linear Algebra & Geometry why is linear algebra useful in computer vision? References: -Any book on linear algebra! -[HZ] chapters 2, 4 Some of the slides in this lecture are courtesy to Prof. Octavia

### 13-2 Text: 28-30; AB: 1.3.3, 3.2.3, 3.4.2, 3.5, 3.6.2; GvL Eigen2

The QR algorithm The most common method for solving small (dense) eigenvalue problems. The basic algorithm: QR without shifts 1. Until Convergence Do: 2. Compute the QR factorization A = QR 3. Set A :=

### Linear Algebra Notes. Lecture Notes, University of Toronto, Fall 2016

Linear Algebra Notes Lecture Notes, University of Toronto, Fall 2016 (Ctd ) 11 Isomorphisms 1 Linear maps Definition 11 An invertible linear map T : V W is called a linear isomorphism from V to W Etymology:

### 1 Linear transformations; the basics

Linear Algebra Fall 2013 Linear Transformations 1 Linear transformations; the basics Definition 1 Let V, W be vector spaces over the same field F. A linear transformation (also known as linear map, or

### Problems in Linear Algebra and Representation Theory

Problems in Linear Algebra and Representation Theory (Most of these were provided by Victor Ginzburg) The problems appearing below have varying level of difficulty. They are not listed in any specific

### Algebra Exam Syllabus

Algebra Exam Syllabus The Algebra comprehensive exam covers four broad areas of algebra: (1) Groups; (2) Rings; (3) Modules; and (4) Linear Algebra. These topics are all covered in the first semester graduate

### Basic Concepts in Linear Algebra

Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear

### Scientific Computing: Solving Linear Systems

Scientific Computing: Solving Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 September 17th and 24th, 2015 A. Donev (Courant

### The QR Decomposition

The QR Decomposition We have seen one major decomposition of a matrix which is A = LU (and its variants) or more generally PA = LU for a permutation matrix P. This was valid for a square matrix and aided

### Exercise Sheet 1. 1 Probability revision 1: Student-t as an infinite mixture of Gaussians

Exercise Sheet 1 1 Probability revision 1: Student-t as an infinite mixture of Gaussians Show that an infinite mixture of Gaussian distributions, with Gamma distributions as mixing weights in the following

### Numerical Linear Algebra

Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one

### Applied Numerical Linear Algebra. Lecture 8

Applied Numerical Linear Algebra. Lecture 8 1/ 45 Perturbation Theory for the Least Squares Problem When A is not square, we define its condition number with respect to the 2-norm to be k 2 (A) σ max (A)/σ

### AM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition

AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized

### Kernel Method: Data Analysis with Positive Definite Kernels

Kernel Method: Data Analysis with Positive Definite Kernels 2. Positive Definite Kernel and Reproducing Kernel Hilbert Space Kenji Fukumizu The Institute of Statistical Mathematics. Graduate University

### Comprehensive Introduction to Linear Algebra

Comprehensive Introduction to Linear Algebra WEB VERSION Joel G Broida S Gill Williamson N = a 11 a 12 a 1n a 21 a 22 a 2n C = a 11 a 12 a 1n a 21 a 22 a 2n a m1 a m2 a mn a m1 a m2 a mn Comprehensive

### Computational Methods. Eigenvalues and Singular Values

Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations

### Problem Set # 1 Solution, 18.06

Problem Set # 1 Solution, 1.06 For grading: Each problem worths 10 points, and there is points of extra credit in problem. The total maximum is 100. 1. (10pts) In Lecture 1, Prof. Strang drew the cone

### Supplementary Notes on Linear Algebra

Supplementary Notes on Linear Algebra Mariusz Wodzicki May 3, 2015 1 Vector spaces 1.1 Coordinatization of a vector space 1.1.1 Given a basis B = {b 1,..., b n } in a vector space V, any vector v V can

MTH50 Spring 07 HW Assignment 7 {From [FIS0]}: Sec 44 #4a h 6; Sec 5 #ad ac 4ae 4 7 The due date for this assignment is 04/05/7 Sec 44 #4a h Evaluate the erminant of the following matrices by any legitimate

### Eigenvalues and Eigenvectors

Chapter 1 Eigenvalues and Eigenvectors Among problems in numerical linear algebra, the determination of the eigenvalues and eigenvectors of matrices is second in importance only to the solution of linear

### Symmetric Matrices and Eigendecomposition

Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions

### NONCOMMUTATIVE POLYNOMIAL EQUATIONS. Edward S. Letzter. Introduction

NONCOMMUTATIVE POLYNOMIAL EQUATIONS Edward S Letzter Introduction My aim in these notes is twofold: First, to briefly review some linear algebra Second, to provide you with some new tools and techniques

### 18.06 Problem Set 8 - Solutions Due Wednesday, 14 November 2007 at 4 pm in

806 Problem Set 8 - Solutions Due Wednesday, 4 November 2007 at 4 pm in 2-06 08 03 Problem : 205+5+5+5 Consider the matrix A 02 07 a Check that A is a positive Markov matrix, and find its steady state

### Eigenvectors and Hermitian Operators

7 71 Eigenvalues and Eigenvectors Basic Definitions Let L be a linear operator on some given vector space V A scalar λ and a nonzero vector v are referred to, respectively, as an eigenvalue and corresponding

### Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases

Week Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases. Opening Remarks.. Low Rank Approximation 38 Week. Orthogonal Projection, Low Rank Approximation, and Orthogonal Bases 38.. Outline..

### LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT FALL 2006 PRINCETON UNIVERSITY

LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT 204 - FALL 2006 PRINCETON UNIVERSITY ALFONSO SORRENTINO 1 Adjoint of a linear operator Note: In these notes, V will denote a n-dimensional euclidean vector

### Multilinear Algebra For the Undergraduate Algebra Student

Multilinear Algebra For the Undergraduate Algebra Student Davis Shurbert Department of Mathematics and Computer Science University of Puget Sound April 4, 204 Introduction When working in the field of