A Note on Two Different Types of Matrices and Their Applications
|
|
- Jayson Terry
- 5 years ago
- Views:
Transcription
1 A Note on Two Different Types of Matrices and Their Applications Arjun Krishnan I really enjoyed Prof. Del Vecchio s Linear Systems Theory course and thought I d give something back. So I ve written a little note on a couple of topics which introduce applications of two different kinds of matrices that you might not encounter in the course. Hopefully, you ll find it interesting. There are other reasons I wrote this and they are both purely selfish: I could use some proof writing practise, and I really enjoy working these kinds of things out for myself. If I have inadvertently messed up a proof or ignored a deeper mathematical understanding of the objects under consideration, please let me know. The only excuse I have to offer is that I m an engineer, and that I m more comfortable with nuts and bolts than with tangent bundles and cohomologies. I d appreciate any kind of feedback. 1 Rotation in higher dimensions This material is based on a discussion I had with Prof. V. Balakrishnan when I was at the Indian Institute of Technology Madras. All the essential ideas are his, and I just wrote this stuff down and expanded on a few minor aspects. 1.1 Preliminaries To fix ideas, let us see what defines a rotation in R 3, and then extend this to a general finite dimensional vector space. Rotation has the following properties: 1. It is clear from our intuition that rotation is a linear operator. We can use intuition here because we are still in the process of defining rotation. That is, if we have two vectors a, b and a real constant α, R(αa + b) = αr(a)+r(b). The picture is immediate in two dimensions, and with a little thought one realizes that the picture is equally valid in three dimensions if one looks down the axis of rotation and omits the components of a and b along this axis. All this tells us is that R has a matrix representation in any basis, and we can now talk about matrices instead of operators. 2. Rotation leaves the inner product between two vectors unchanged. That 1
2 is, (Ra, Rb) = (a, b) or (1) a T R T Rb = a T b (2) for every a, b R 3, where (, ) represents the inner product. Choose some orthogonal basis, and select the vectors vectors a, b from the basis set {e 1, e 2,, e n }. By choosing all possible combinations for a and b, and using Eq. 1, we find that R T R = I,where I is the identity matrix. In other words, R T = R 1. It follows that: det(rr 1 ) = det(rr T ) = 1 det(r) = ±1 The property of inner product invariance can be understood in more intuitive terms as the invariance of the angle between two vectors and invariance of the length of vectors under the rotational transformation. It is simple to verify the converse, and show that angle invariance (in R 2 and R 3, where the word angle has meaning) implies inner product invariance. Matrices which preserve the inner product are called orthogonal. 3. Every rotation matrix has an orthogonal generator. I cooked this property up to suit our purposes - we will use this property to show that det(r) = 1 only. By cooked up, I mean that it seems to capture the continuity property of rotations intuitively to me, and so I ll proceed to define it mathematically. Define: for every rotation matrix R and every N R, an orthogonal matrix dr s.t., R = dr N. Perhaps a more talented mathematician can define this carefully in terms of norms and define an infinitesimal generator. Suppose we fix some odd N, we must have det(dr N ) = (det(dr)) N. Then, it follows that since the generator is orthogonal, it must have determinant plus one. 1 In summary, the defining properties of rotation matrices R are, R T R = I, det(r) = +1, and these are the properties required to extend the concept of rotation to higher dimensions. 1 When det(r) = 1 - say in the case of a mirror inversion through the origin where R = I - it is clear that no such infinitesimal generator exists when N is even 2
3 1.2 Axes of rotation In R 3, an axis of rotation is an invariant vector under the rotation operation. That is, a vector v is along an axis of rotation if, Rv = v. In more familiar terms, v is along the axis of rotation if it is an eigenvector corresponding to the eigenvalue +1. Next, notice that the eigenvalues of a rotation matrix lie along the unit circle in the complex plane. Let v be an eigenvector and λ the corresponding eigenvalue. Then, v T v = v T R T Rv = λ 2 v T v λ 2 = 1. So an eigenvalue can be complex or ±1. Suppose the eigenvalues are {λ 1, λ 2,, λ p } with appropriate multiplicities. Then, we know: p λ i = det(r) = +1 (3) i=1 This condition places the constraint that the eigenvalue 1 can appear only with even multiplicities. The complex eigenvalues appear in pairs anyway since R has only real entries and λλ = 1 from Eq. 3. When n = 2, there can be no concept of axis. In fact, the only rotation matrix with a +1 eigenvalue is the identity. 2 Suppose the dimensionality of the space n 2 is even. Then, it follows from the above that the eigenvalue +1 must also appear with an even multiplicity. That is, in an even dimensional space, there can never exist a unique axis of rotation. Now suppose n is odd, then there is the possibility that +1 can appear with multiplicity 1, and it is possible to specify a unique axis of rotation. Of course this is not certain, and we won t attempt to find conditions for this to happen. However, when n = 3, there always is a unique axis of rotation. To paraphrase Prof. Balakrishnan, we are indeed lucky to living in R 3! 3 2 The second partial derivative test We are all familiar with the second partial derivative test to classify the critical points of a function of two variables. There are some nice matrices to study in this regard when we want to generalize the test to higher dimensions - this is the motivation. The result will rely on something called the spectral theorem 2 The student is encouraged to verify this. I always wanted to say that. 3 One might argue that this is baseless rhetoric and that the extension of the concept of rotation to higher dimensions is an artificial construct - I agree. 3
4 (because it says something about the eigenvalues of a matrix) for real symmetric matrices, and there are some neat proofs used here. The discussion is elementary and I pretty much re-derived it from scratch. One problem with all the proofs is that I implicitly assume a basis since we need to use the gradient operator. I believe the concepts are easy to generalize, and I vaguely recollect the terms Fréchet and Gâteaux in being used in this regard The test in two variables In the following, we will use the variable x to represent both a vector and a single variable, and the meaning will be clear from context (hopefully). At a critical point, the gradient vector of a scalar function f : R n R is zero. To further classify the critical points into maxima, minima and saddle points, we have the following test in R 2. Assume that f has continuous second derivatives at the critical point (x 0, y 0 ) and construct the Hessian H of the function at the critical point. This is the matrix of partial derivatives: [ ] fxx f H = xy f yx f yy Then, the test says: det(h) > 0 and f xx > 0 det(h) > 0 and f xx < 0 det(h) < 0 det(h) = 0 Minimum Maximum Saddle point Inconclusive (4) 2.2 In higher dimensions To examine these conditions in greater detail and generalize it to higher dimensions, we need to make a few statements regarding the structure of the problem. R n has the usual inner-product (, ) defined on it and the Euclidean norm induced by the inner product. The cartesian basis is implicitly assumed for the reasons mentioned earlier. Now, let us go back to the definition of a extremal point. A function has a minimum at a critical point x 0 if there exists an ɛ > 0 such that f(x) f(x 0 ) x s.t. x x 0 < ɛ. (5) The condition for a maximum is obvious and we have a saddle if for every ɛ > 0, there are x 1, x 2 s.t., f(x 1 ) > f(x 0 ) and f(x 2 ) < f(x 0 ). To analyze the behavior around the critical point, we need to use Taylor s theorem. We can write Taylor s theorem for a continuous function of several variables in terms of inner products or in matrix notation. It says that around 4
5 some point x 0, f(x) = f(x 0 ) + ( f(x0), x x 0 ) (x x 0, H f(x0)(x x 0 )) + R(x x 0 ) f(x) = f(x 0 ) + (x x 0 ) T f(x0) (x x 0) T H f(x0)(x x 0 ) + R(x x 0 ) where f(x0) represents the gradient vector and H f(x0) is the Hessian matrix of f at x 0. The key property we use here is that the remainder R(x x 0 ) can be made as small as possible by making x x 0 very small. This is such a regularly used property that we hardly ever bother about when it is applicable. For this, we need to make a further assumption on f(x) and state that not only are the second partials continuous, but also differentiable at the critical point. Then there is an ɛ ball around x 0 in which the remainder R(x x 0 ) is smaller in absolute value than the absolute value of other terms if they are both not zero. At the critical point, the gradient is the zero vector and the behavior of f(x) is dictated by the second derivative term. For there to be a minimum, say, then the term x T H f(x0)x must be greater than zero for every x in a ball of radius of ɛ about x 0. It is easy to see that since the Hessian is a linear operator, this condition must be true for every x. To summarize, x T H x0 x > 0 x Minimum x T H x0 x < 0 x Maximum x T H x0 x < 0 x S R and x T H x0 x > 0 x R S Saddle where S is some subset of R n and H is the hessian matrix. Clairaut s theorem tells us that the Hessian matrix is symmetric, i.e., order of partial differentiation does not matter. A symmetric matrix H having the first two property in Eq. 6 is called positive-definite or negative-definite. Now, we must study the properties of symmetric matrices to put the conditions in more useful terms Where does this Hessian come from? We just stated Taylor s theorem in the last section in terms of the Hessian and the gradient - how does this form arise? In one or two dimensions, one can visualize a tangent line or plane z = f(x 0 ). There is a minimum or maximum at a critical point if the function does not cross the tangent plane some small ball of radius ɛ around a critical point. Then, one way of using the more familiar single variable Taylor theorem is to consider the change in the value of f(x) along every line through the critical point. So if we would like a critical point to be a minimum, we would require the second derivative to be positive in every direction. This is most conveniently understood using the concept of gradient. The change in a function of several variables f(x 1,, x n ) can be written as: (6) 5
6 df = n i=1 f x i dx i Then, we can define an operator that gives a vector of partial derivatives, take its dot product with some unit vector û = (u 1,, u 2 ) and find the change in f(x) in the u direction as: df u du = (û. )f(x 0) It follows that the second derivative - the change in change of f - along u is, d 2 f u du 2 = (û. )((û. )f(x 0)). To see how this can be written in terms of the Hessian in Eq. 6, we can use Einstein s summation notation. The gradient operator in a direction u will be written as u i i f(x) where i is short-hand for the partial derivative with respect to x i. The second derivative is just u k k (u i i f(x)). Rearranging, we find u k ( k i f(x))u i which we can write as an explicit sum for clarity as, ( u k i k k i f(x)u i ) = u T Hu 2.3 Properties of Symmetric Matrices To better understand the conditions in Eq. 6, we state and indicate the proof of some results regarding symmetric matrices. We work in the restricted setting of R n, and work only with symmetric matrices, rather than in a more general Hilbert space with symmetric operators to avoid some technical issues. 4 Definition (Tangent Space). Let x R n,. Then the tangent space is defined as T = {t (x, t) = 0} It follows from the linearity of the inner product that T is a subspace and one can show that T span{x} = R n. Lemma 1 (Eigenvalues of Symmetric Matrices). The eigenvalues of a real symmetric matrix are real. Proof. Let λ be an complex eigenvalue of H and let v be the associated eigenvector. Since H is symmetric, (Hv) T = v T H. 4 That basically means I don t know how and don t want to make a mistake. 6
7 We know (from class) that λ and v, the complex conjugates are also an eigenvalue eigenvector pair. Then, and it follows that λ = λ v T H v = λv T v = λv T v, Lemma 2 (Inner Product Property). Let A be a symmetric matrix. (Av, x) = (v, Ax) x, v. Then, Proof. (Av) T = v T A T = v T A (v, Ax) = v T Ax = (Ax) T v = x T Av = (x, Av) Theorem 1 ( Real Spectral Theorem). Real symmetric matrices are diagonalizable and can be written in the form Q T DQ, where Q is an orthogonal matrix and D is diagonal matrix, both with purely real entries. Proof. Note. I st... I mean adapted this proof from somewhere on the net, and the guy who posted it got it from some book (Apostol s Calculus or something). There are no details in the stated proof, and so I expanded it and wrote a proof for engineers. We will prove that we can construct n distinct eigenvectors and that they are orthogonal. For a general diagonalizable matrix A, the similarity transformation to the eigenvector basis is P 1 AP. P has as it s columns the eigenvectors of A, and since the eigenvectors are orthogonal, we can construct an orthogonal P by simply normalizing the eigenvectors. Claim 2. A has at least one eigenvector. Pf. Define a function f = (x, Ax) on the surface of the unit ball S = {x x, x < 1} in R n. Then the gradient of f is given by (see Appendix), f = 2Ax Now, S is a closed and bounded set and f is continuous. Then, there is a generalization of Weierstrass Theorem (I m not sure about the name) that states that f reaches has a maximum at some x 1. Let T x1 be the tangent space at x 1. Then, it is clear that f is orthogonal to T x1 (see Appendix). Hence it follows that, Ax 1 = 0 or Ax 1 = λx 1. Claim 3. T x1 is an invariant subspace of A. 7
8 Pf. Let v T x1. By lemma 2, (Ax 1, v) = (Av, x 1 ). But by claim 2, This implies Av T x1. (Ax 1, v) = 0. Then, all we have to do is restrict A to T x1 (apply it only to vectors in T x1 ) 5, and consider the same function f = (Ax, x) on the unit sphere in T x1 and repeat the procedure until all n eigenvectors are found. It is clear that the eigenvectors are orthogonal. Since the eigenvalues and corresponding eigenvectors are real, it follows that A is diagonalizable in the form specified. 2.4 Conclusion Since the hessian H can be diagonalized in the form specified by the spectral theorem, we can perform a coordinate transformation to the eigenvector basis. The condition for positive definiteness remains identical in the new basis because, x T Hx = (x T Q T )D(Qx). Then, it is clear that positive definiteness requires that all the eigenvalues of H be strictly positive and negative definiteness requires all the eigenvalues to be strictly negative. The saddle point condition requires some of the eigenvalues to be positive and some to be strictly negative. Of course, if even one of the eigenvalues is zero, it means that the second derivative is zero in some direction and we need higher order tests to analyze the behavior of the critical point here. It is easy to understand the conditions for the two-dimensional case given by Eq. 4 now. Pretty neat, no? A Constrained Optimization First, an ugly proof: Let f : R n R be a continuous function with continuously differentiable first partials. Let S be the surface of the unit sphere in R n. Weierstrass theorem states that a maximum exists - call it x 0. We need to show that f(x0) is orthogonal to every vector in the the tangent space T x0. First notice that any vector v can be expressed as a sum αx 0 + βˆt where ˆt is a unit vector in T x0 and β > 0 (because T x0 is a subspace and the T + spanx 0 = R n ). Then suppose the contrary: that is, let x0 f = αx 0 + βˆt so that it is not orthogonal to T x0. Now, Taylor s theorem states that: f(x) = f(x 0 ) + f(x0) (x x 0 ) + R(x x 0 ) (7) 5 There is a theorem called the Invariant Subspace Theorem which tells us that if a linear operator T is invariant on a subspace S, there is another operator T S that defined on the subspace S s.t., T S (x) = T (x) x S. Obvious, isn t it. 8
9 Since the second derivative exists, there exists an δ > 0 s.t. for every x x 0 < δ, R(x x 0 ) < f(x0) (x x 0 ) Now, choose an x = ax 0 + bˆt from S s.t. x x 0 with b > 0 and a has the same sign as α. It is clear that one such exists because one can obtain the following conditions for a and b: 0 < (x, x) = k = a 2 + b 2 1 ((a 1)x 0 + bˆt, (a 1)x 0 + bˆt) = (a 1) 2 + b 2 < ɛ 1 ɛ 2 < a < 1 The inner product in Eq. 7 is α(a 1) + βb. βb can be made positive always, and αa is positive as well if α < 0. Suppose α > 0, then we need, βb > α(1 a). After a little algebraic manipulation, we find that the condition on a > α2 β 2 α 2 + β 2 Since β > 0, it is clear that the RHS of the previous inequality is strictly less than 1 and its obvious that we can select an a which does the job. What we have done here is to just state a general principle in optimization problems called the method of Lagrange multipliers. That is, given a f(x) subject to a constraint equation g(x) = c, the extremal points of f(x) are found at the points where: f = λ g λ R Why should this be the case? The gradient of g(x) = c always points in a direction normal to the curve. When walking along this curve, we find extrema of the function f(x) when the total derivative of df has no change in the direction tangential to the constraint curve. That is, the gradient of f(x) must be normal to the constraint curve or parallel to g. 6 In our case, the constraint equation is g(x) = (x, x) and g = 2x. B Gradient of (x, Ax) We stated that (x, Ax) = 2Ax. The simplest proof is using Einstein s summation notation. In simplest terms, a repeated index in the summation notation tells us we have to sum over that index. Let A = {a ij } and x = {x j }. As an illustrative example, we can write the components of the gradient of f(x) = Ax as, ( (Ax)) k = a ij x j = a ik. x k 6 There is a nice article on Lagrange Multipliers in Wikipedia. 9
10 Then for the function f(x) = (x, Ax), each component of the gradient vector is given by, ( (x, Ax)) k = (x i a ij x j ) x k = a kj x j + x i a ik where we have used the product rule for differentiation. Since A is symmetric, a kj = a jk and the result follows immediately from the above. 10
Transpose & Dot Product
Transpose & Dot Product Def: The transpose of an m n matrix A is the n m matrix A T whose columns are the rows of A. So: The columns of A T are the rows of A. The rows of A T are the columns of A. Example:
More informationTranspose & Dot Product
Transpose & Dot Product Def: The transpose of an m n matrix A is the n m matrix A T whose columns are the rows of A. So: The columns of A T are the rows of A. The rows of A T are the columns of A. Example:
More informationTangent spaces, normals and extrema
Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent
More informationREVIEW OF DIFFERENTIAL CALCULUS
REVIEW OF DIFFERENTIAL CALCULUS DONU ARAPURA 1. Limits and continuity To simplify the statements, we will often stick to two variables, but everything holds with any number of variables. Let f(x, y) be
More informationIdeas from Vector Calculus Kurt Bryan
Ideas from Vector Calculus Kurt Bryan Most of the facts I state below are for functions of two or three variables, but with noted exceptions all are true for functions of n variables..1 Tangent Line Approximation
More informationChapter 7. Extremal Problems. 7.1 Extrema and Local Extrema
Chapter 7 Extremal Problems No matter in theoretical context or in applications many problems can be formulated as problems of finding the maximum or minimum of a function. Whenever this is the case, advanced
More informationFFTs in Graphics and Vision. The Laplace Operator
FFTs in Graphics and Vision The Laplace Operator 1 Outline Math Stuff Symmetric/Hermitian Matrices Lagrange Multipliers Diagonalizing Symmetric Matrices The Laplacian Operator 2 Linear Operators Definition:
More informationImplicit Functions, Curves and Surfaces
Chapter 11 Implicit Functions, Curves and Surfaces 11.1 Implicit Function Theorem Motivation. In many problems, objects or quantities of interest can only be described indirectly or implicitly. It is then
More informationChapter 2. Matrix Arithmetic. Chapter 2
Matrix Arithmetic Matrix Addition and Subtraction Addition and subtraction act element-wise on matrices. In order for the addition/subtraction (A B) to be possible, the two matrices A and B must have the
More informationLecture 5 : Projections
Lecture 5 : Projections EE227C. Lecturer: Professor Martin Wainwright. Scribe: Alvin Wan Up until now, we have seen convergence rates of unconstrained gradient descent. Now, we consider a constrained minimization
More informationCalculus 2502A - Advanced Calculus I Fall : Local minima and maxima
Calculus 50A - Advanced Calculus I Fall 014 14.7: Local minima and maxima Martin Frankland November 17, 014 In these notes, we discuss the problem of finding the local minima and maxima of a function.
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationMath 291-3: Lecture Notes Northwestern University, Spring 2016
Math 291-3: Lecture Notes Northwestern University, Spring 216 Written by Santiago Cañez These are lecture notes for Math 291-3, the third quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,
More informationLinear Algebra, Summer 2011, pt. 3
Linear Algebra, Summer 011, pt. 3 September 0, 011 Contents 1 Orthogonality. 1 1.1 The length of a vector....................... 1. Orthogonal vectors......................... 3 1.3 Orthogonal Subspaces.......................
More informationMath (P)Review Part I:
Lecture 1: Math (P)Review Part I: Linear Algebra Computer Graphics CMU 15-462/15-662, Fall 2017 Homework 0.0 (Due Monday!) Exercises will be a bit harder / more rigorous than what you will do for the rest
More informationA geometric proof of the spectral theorem for real symmetric matrices
0 0 0 A geometric proof of the spectral theorem for real symmetric matrices Robert Sachs Department of Mathematical Sciences George Mason University Fairfax, Virginia 22030 rsachs@gmu.edu January 6, 2011
More informationFaculty of Engineering, Mathematics and Science School of Mathematics
Faculty of Engineering, Mathematics and Science School of Mathematics GROUPS Trinity Term 06 MA3: Advanced Calculus SAMPLE EXAM, Solutions DAY PLACE TIME Prof. Larry Rolen Instructions to Candidates: Attempt
More informationMetric spaces and metrizability
1 Motivation Metric spaces and metrizability By this point in the course, this section should not need much in the way of motivation. From the very beginning, we have talked about R n usual and how relatively
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More information3.7 Constrained Optimization and Lagrange Multipliers
3.7 Constrained Optimization and Lagrange Multipliers 71 3.7 Constrained Optimization and Lagrange Multipliers Overview: Constrained optimization problems can sometimes be solved using the methods of the
More informationA Primer on Three Vectors
Michael Dine Department of Physics University of California, Santa Cruz September 2010 What makes E&M hard, more than anything else, is the problem that the electric and magnetic fields are vectors, and
More informationRecitation 1 (Sep. 15, 2017)
Lecture 1 8.321 Quantum Theory I, Fall 2017 1 Recitation 1 (Sep. 15, 2017) 1.1 Simultaneous Diagonalization In the last lecture, we discussed the situations in which two operators can be simultaneously
More informationFinal Review Sheet. B = (1, 1 + 3x, 1 + x 2 ) then 2 + 3x + 6x 2
Final Review Sheet The final will cover Sections Chapters 1,2,3 and 4, as well as sections 5.1-5.4, 6.1-6.2 and 7.1-7.3 from chapters 5,6 and 7. This is essentially all material covered this term. Watch
More informationThe Derivative. Appendix B. B.1 The Derivative of f. Mappings from IR to IR
Appendix B The Derivative B.1 The Derivative of f In this chapter, we give a short summary of the derivative. Specifically, we want to compare/contrast how the derivative appears for functions whose domain
More informationAnalysis II: The Implicit and Inverse Function Theorems
Analysis II: The Implicit and Inverse Function Theorems Jesse Ratzkin November 17, 2009 Let f : R n R m be C 1. When is the zero set Z = {x R n : f(x) = 0} the graph of another function? When is Z nicely
More informationUNDERSTANDING THE DIAGONALIZATION PROBLEM. Roy Skjelnes. 1.- Linear Maps 1.1. Linear maps. A map T : R n R m is a linear map if
UNDERSTANDING THE DIAGONALIZATION PROBLEM Roy Skjelnes Abstract These notes are additional material to the course B107, given fall 200 The style may appear a bit coarse and consequently the student is
More informationLagrange Multipliers
Optimization with Constraints As long as algebra and geometry have been separated, their progress have been slow and their uses limited; but when these two sciences have been united, they have lent each
More informationSeptember Math Course: First Order Derivative
September Math Course: First Order Derivative Arina Nikandrova Functions Function y = f (x), where x is either be a scalar or a vector of several variables (x,..., x n ), can be thought of as a rule which
More informationNotes on multivariable calculus
Notes on multivariable calculus Jonathan Wise February 2, 2010 1 Review of trigonometry Trigonometry is essentially the study of the relationship between polar coordinates and Cartesian coordinates in
More informationNumerical Linear Algebra
Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix
More informationDot Products, Transposes, and Orthogonal Projections
Dot Products, Transposes, and Orthogonal Projections David Jekel November 13, 2015 Properties of Dot Products Recall that the dot product or standard inner product on R n is given by x y = x 1 y 1 + +
More informationSpectral Theorem for Self-adjoint Linear Operators
Notes for the undergraduate lecture by David Adams. (These are the notes I would write if I was teaching a course on this topic. I have included more material than I will cover in the 45 minute lecture;
More informationLinear Algebra, Summer 2011, pt. 2
Linear Algebra, Summer 2, pt. 2 June 8, 2 Contents Inverses. 2 Vector Spaces. 3 2. Examples of vector spaces..................... 3 2.2 The column space......................... 6 2.3 The null space...........................
More informationEigenvalues and eigenvectors
Roberto s Notes on Linear Algebra Chapter 0: Eigenvalues and diagonalization Section Eigenvalues and eigenvectors What you need to know already: Basic properties of linear transformations. Linear systems
More informationMath (P)Review Part II:
Math (P)Review Part II: Vector Calculus Computer Graphics Assignment 0.5 (Out today!) Same story as last homework; second part on vector calculus. Slightly fewer questions Last Time: Linear Algebra Touched
More information1 Mathematical preliminaries
1 Mathematical preliminaries The mathematical language of quantum mechanics is that of vector spaces and linear algebra. In this preliminary section, we will collect the various definitions and mathematical
More informationConnectedness. Proposition 2.2. The following are equivalent for a topological space (X, T ).
Connectedness 1 Motivation Connectedness is the sort of topological property that students love. Its definition is intuitive and easy to understand, and it is a powerful tool in proofs of well-known results.
More informationQuantum Mechanics- I Prof. Dr. S. Lakshmi Bala Department of Physics Indian Institute of Technology, Madras
Quantum Mechanics- I Prof. Dr. S. Lakshmi Bala Department of Physics Indian Institute of Technology, Madras Lecture - 6 Postulates of Quantum Mechanics II (Refer Slide Time: 00:07) In my last lecture,
More informationLinear Algebra Review
Chapter 1 Linear Algebra Review It is assumed that you have had a course in linear algebra, and are familiar with matrix multiplication, eigenvectors, etc. I will review some of these terms here, but quite
More informationDesign and Optimization of Energy Systems Prof. C. Balaji Department of Mechanical Engineering Indian Institute of Technology, Madras
Design and Optimization of Energy Systems Prof. C. Balaji Department of Mechanical Engineering Indian Institute of Technology, Madras Lecture - 09 Newton-Raphson Method Contd We will continue with our
More informationMath 320-3: Lecture Notes Northwestern University, Spring 2015
Math 320-3: Lecture Notes Northwestern University, Spring 2015 Written by Santiago Cañez These are lecture notes for Math 320-3, the third quarter of Real Analysis, taught at Northwestern University in
More informationEigenvectors and Hermitian Operators
7 71 Eigenvalues and Eigenvectors Basic Definitions Let L be a linear operator on some given vector space V A scalar λ and a nonzero vector v are referred to, respectively, as an eigenvalue and corresponding
More informationRoberto s Notes on Linear Algebra Chapter 9: Orthogonality Section 2. Orthogonal matrices
Roberto s Notes on Linear Algebra Chapter 9: Orthogonality Section 2 Orthogonal matrices What you need to know already: What orthogonal and orthonormal bases for subspaces are. What you can learn here:
More informationMath 291-2: Final Exam Solutions Northwestern University, Winter 2016
Math 29-2: Final Exam Solutions Northwestern University, Winter 206 Determine whether each of the following statements is true or false f it is true, explain why; if it is false, give a counterexample
More informationMath 251 Midterm II Information Spring 2018
Math 251 Midterm II Information Spring 2018 WHEN: Thursday, April 12 (in class). You will have the entire period (125 minutes) to work on the exam. RULES: No books or notes. You may bring a non-graphing
More informationSolutions to Homework 7
Solutions to Homework 7 Exercise #3 in section 5.2: A rectangular box is inscribed in a hemisphere of radius r. Find the dimensions of the box of maximum volume. Solution: The base of the rectangular box
More informationExtreme Values and Positive/ Negative Definite Matrix Conditions
Extreme Values and Positive/ Negative Definite Matrix Conditions James K. Peterson Department of Biological Sciences and Department of Mathematical Sciences Clemson University November 8, 016 Outline 1
More information14 Singular Value Decomposition
14 Singular Value Decomposition For any high-dimensional data analysis, one s first thought should often be: can I use an SVD? The singular value decomposition is an invaluable analysis tool for dealing
More informationPhysics 110. Electricity and Magnetism. Professor Dine. Spring, Handout: Vectors and Tensors: Everything You Need to Know
Physics 110. Electricity and Magnetism. Professor Dine Spring, 2008. Handout: Vectors and Tensors: Everything You Need to Know What makes E&M hard, more than anything else, is the problem that the electric
More information1 Lagrange Multiplier Method
1 Lagrange Multiplier Method Near a maximum the decrements on both sides are in the beginning only imperceptible. J. Kepler When a quantity is greatest or least, at that moment its flow neither increases
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationMath 290-2: Linear Algebra & Multivariable Calculus Northwestern University, Lecture Notes
Math 290-2: Linear Algebra & Multivariable Calculus Northwestern University, Lecture Notes Written by Santiago Cañez These are notes which provide a basic summary of each lecture for Math 290-2, the second
More informationLecture 6: Lies, Inner Product Spaces, and Symmetric Matrices
Math 108B Professor: Padraic Bartlett Lecture 6: Lies, Inner Product Spaces, and Symmetric Matrices Week 6 UCSB 2014 1 Lies Fun fact: I have deceived 1 you somewhat with these last few lectures! Let me
More informationIntroduction to Group Theory
Chapter 10 Introduction to Group Theory Since symmetries described by groups play such an important role in modern physics, we will take a little time to introduce the basic structure (as seen by a physicist)
More informationEigenvalues and Eigenfunctions of the Laplacian
The Waterloo Mathematics Review 23 Eigenvalues and Eigenfunctions of the Laplacian Mihai Nica University of Waterloo mcnica@uwaterloo.ca Abstract: The problem of determining the eigenvalues and eigenvectors
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationChapter 11 - Sequences and Series
Calculus and Analytic Geometry II Chapter - Sequences and Series. Sequences Definition. A sequence is a list of numbers written in a definite order, We call a n the general term of the sequence. {a, a
More information11 a 12 a 21 a 11 a 22 a 12 a 21. (C.11) A = The determinant of a product of two matrices is given by AB = A B 1 1 = (C.13) and similarly.
C PROPERTIES OF MATRICES 697 to whether the permutation i 1 i 2 i N is even or odd, respectively Note that I =1 Thus, for a 2 2 matrix, the determinant takes the form A = a 11 a 12 = a a 21 a 11 a 22 a
More informationMain topics for the First Midterm Exam
Main topics for the First Midterm Exam The final will cover Sections.-.0, 2.-2.5, and 4.. This is roughly the material from first three homeworks and three quizzes, in addition to the lecture on Monday,
More information[Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty.]
Math 43 Review Notes [Disclaimer: This is not a complete list of everything you need to know, just some of the topics that gave people difficulty Dot Product If v (v, v, v 3 and w (w, w, w 3, then the
More informationOptimality Conditions
Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.
More informationMeaning of the Hessian of a function in a critical point
Meaning of the Hessian of a function in a critical point Mircea Petrache February 1, 2012 We consider a function f : R n R and assume for it to be differentiable with continuity at least two times (that
More informationLecture 11: Differential Geometry
Lecture 11: Differential Geometry c Bryan S. Morse, Brigham Young University, 1998 2000 Last modified on February 28, 2000 at 8:45 PM Contents 11.1 Introduction..............................................
More informationGetting Started with Communications Engineering
1 Linear algebra is the algebra of linear equations: the term linear being used in the same sense as in linear functions, such as: which is the equation of a straight line. y ax c (0.1) Of course, if we
More informationMATH Max-min Theory Fall 2016
MATH 20550 Max-min Theory Fall 2016 1. Definitions and main theorems Max-min theory starts with a function f of a vector variable x and a subset D of the domain of f. So far when we have worked with functions
More informationEXAMPLES OF PROOFS BY INDUCTION
EXAMPLES OF PROOFS BY INDUCTION KEITH CONRAD 1. Introduction In this handout we illustrate proofs by induction from several areas of mathematics: linear algebra, polynomial algebra, and calculus. Becoming
More informationARE202A, Fall 2005 CONTENTS. 1. Graphical Overview of Optimization Theory (cont) Separating Hyperplanes 1
AREA, Fall 5 LECTURE #: WED, OCT 5, 5 PRINT DATE: OCTOBER 5, 5 (GRAPHICAL) CONTENTS 1. Graphical Overview of Optimization Theory (cont) 1 1.4. Separating Hyperplanes 1 1.5. Constrained Maximization: One
More informationMath 118, Fall 2014 Final Exam
Math 8, Fall 4 Final Exam True or false Please circle your choice; no explanation is necessary True There is a linear transformation T such that T e ) = e and T e ) = e Solution Since T is linear, if T
More informationMath 5311 Constrained Optimization Notes
ath 5311 Constrained Optimization otes February 5, 2009 1 Equality-constrained optimization Real-world optimization problems frequently have constraints on their variables. Constraints may be equality
More informationIntroduction to gradient descent
6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our
More information(x, y) = d(x, y) = x y.
1 Euclidean geometry 1.1 Euclidean space Our story begins with a geometry which will be familiar to all readers, namely the geometry of Euclidean space. In this first chapter we study the Euclidean distance
More informationRecitation 9: Probability Matrices and Real Symmetric Matrices. 3 Probability Matrices: Definitions and Examples
Math b TA: Padraic Bartlett Recitation 9: Probability Matrices and Real Symmetric Matrices Week 9 Caltech 20 Random Question Show that + + + + +... = ϕ, the golden ratio, which is = + 5. 2 2 Homework comments
More informationa s 1.3 Matrix Multiplication. Know how to multiply two matrices and be able to write down the formula
Syllabus for Math 308, Paul Smith Book: Kolman-Hill Chapter 1. Linear Equations and Matrices 1.1 Systems of Linear Equations Definition of a linear equation and a solution to a linear equations. Meaning
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More information3.5 Quadratic Approximation and Convexity/Concavity
3.5 Quadratic Approximation and Convexity/Concavity 55 3.5 Quadratic Approximation and Convexity/Concavity Overview: Second derivatives are useful for understanding how the linear approximation varies
More informationLAGRANGE MULTIPLIERS
LAGRANGE MULTIPLIERS MATH 195, SECTION 59 (VIPUL NAIK) Corresponding material in the book: Section 14.8 What students should definitely get: The Lagrange multiplier condition (one constraint, two constraints
More informationMath 291-1: Lecture Notes Northwestern University, Fall 2015
Math 29-: Lecture Notes Northwestern University, Fall 25 Written by Santiago Cañez These are lecture notes for Math 29-, the first quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,
More informationCALCULUS III. Paul Dawkins
CALCULUS III Paul Dawkins Table of Contents Preface... iii Outline... iv Three Dimensional Space... Introduction... The -D Coordinate System... Equations of Lines... 9 Equations of Planes... 5 Quadric
More informationWARPED PRODUCTS PETER PETERSEN
WARPED PRODUCTS PETER PETERSEN. Definitions We shall define as few concepts as possible. A tangent vector always has the local coordinate expansion v dx i (v) and a function the differential df f dxi We
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationHW3 - Due 02/06. Each answer must be mathematically justified. Don t forget your name. 1 2, A = 2 2
HW3 - Due 02/06 Each answer must be mathematically justified Don t forget your name Problem 1 Find a 2 2 matrix B such that B 3 = A, where A = 2 2 If A was diagonal, it would be easy: we would just take
More informationThe following definition is fundamental.
1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic
More informationMath 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.
Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,
More informationTopic 15 Notes Jeremy Orloff
Topic 5 Notes Jeremy Orloff 5 Transpose, Inverse, Determinant 5. Goals. Know the definition and be able to compute the inverse of any square matrix using row operations. 2. Know the properties of inverses.
More informationMAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012
(Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More information2 Systems of Linear Equations
2 Systems of Linear Equations A system of equations of the form or is called a system of linear equations. x + 2y = 7 2x y = 4 5p 6q + r = 4 2p + 3q 5r = 7 6p q + 4r = 2 Definition. An equation involving
More informationThis exam will be over material covered in class from Monday 14 February through Tuesday 8 March, corresponding to sections in the text.
Math 275, section 002 (Ultman) Spring 2011 MIDTERM 2 REVIEW The second midterm will be held in class (1:40 2:30pm) on Friday 11 March. You will be allowed one half of one side of an 8.5 11 sheet of paper
More informationNotes on Linear Algebra and Matrix Theory
Massimo Franceschet featuring Enrico Bozzo Scalar product The scalar product (a.k.a. dot product or inner product) of two real vectors x = (x 1,..., x n ) and y = (y 1,..., y n ) is not a vector but a
More informationLecture 7: Positive Semidefinite Matrices
Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.
More informationEigenvalues and Eigenvectors
LECTURE 3 Eigenvalues and Eigenvectors Definition 3.. Let A be an n n matrix. The eigenvalue-eigenvector problem for A is the problem of finding numbers λ and vectors v R 3 such that Av = λv. If λ, v are
More informationMath Linear Algebra II. 1. Inner Products and Norms
Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,
More informationInner product spaces. Layers of structure:
Inner product spaces Layers of structure: vector space normed linear space inner product space The abstract definition of an inner product, which we will see very shortly, is simple (and by itself is pretty
More informationStat 206: Linear algebra
Stat 206: Linear algebra James Johndrow (adapted from Iain Johnstone s notes) 2016-11-02 Vectors We have already been working with vectors, but let s review a few more concepts. The inner product of two
More informationLESSON 25: LAGRANGE MULTIPLIERS OCTOBER 30, 2017
LESSON 5: LAGRANGE MULTIPLIERS OCTOBER 30, 017 Lagrange multipliers is another method of finding minima and maxima of functions of more than one variable. In fact, many of the problems from the last homework
More informationWe have been going places in the car of calculus for years, but this analysis course is about how the car actually works.
Analysis I We have been going places in the car of calculus for years, but this analysis course is about how the car actually works. Copier s Message These notes may contain errors. In fact, they almost
More informationLecture 10: A (Brief) Introduction to Group Theory (See Chapter 3.13 in Boas, 3rd Edition)
Lecture 0: A (Brief) Introduction to Group heory (See Chapter 3.3 in Boas, 3rd Edition) Having gained some new experience with matrices, which provide us with representations of groups, and because symmetries
More informationAlgebra I Fall 2007
MIT OpenCourseWare http://ocw.mit.edu 18.701 Algebra I Fall 007 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms. 18.701 007 Geometry of the Special Unitary
More informationNotes on Linear Algebra I. # 1
Notes on Linear Algebra I. # 1 Oussama Moutaoikil Contents 1 Introduction 1 2 On Vector Spaces 5 2.1 Vectors................................... 5 2.2 Vector Spaces................................ 7 2.3
More information