Functions of Several Variables
|
|
- Kevin Anderson
- 5 years ago
- Views:
Transcription
1 Functions of Several Variables The Unconstrained Minimization Problem where In n dimensions the unconstrained problem is stated as f() x variables. minimize f()x x, is a scalar objective function of vector argument x, and s a column vector of n real R n We can take the gradient of f() x, as f which is a column vector of n functions. f f , ,, f x x n T The n x n matrix of second derivatives is called the Hessian matrix and can be found by taking the gradient again, thus Hx (,, x n ) T f f or more specifically, the operator can be written as.. x n. x j... x n.. x n x j n n where the subscript i denotes the row and j denotes the column. The Hessian matris symmetric because of the properties of the second derivatives of continuous functions. x j fx () x j fx ()
2 Some Definitions Definition a point x ** is said to be the strong global minimum of a function f() f for all x R n, f() x fx ( ** ) > 0. Definition a point is said to be a strong local minimum of a function f() f there exists a δ > 0 x * such that f() x fx ( * ) > 0 for all x such that x x * < δ. If the inequalities in f() x fx ( ** ) > 0 and f() x fx ( * ) > 0 are replaced by < then we have strong global and local maxima. If the inequalities in f() x fx ( ** ) > 0 and f() x fx ( * ) > 0 are replaced by then the minima are called weak global and weak local minima. Taylor Expansion We can expand f() n a Taylor expansion about x as fx ( + x) fx () x T + fx () + -- x T fx () x + O( x 3 ) where the notation O( x 3 ) represents terms of order x 3 and these can be neglected for x sufficiently small. From the definition of local minimum, for f( x * + x) fx ( * ) > 0 for x > 0. x * to be a local minimum this implies that From the Taylor expansion f( x * + x) fx ( * ) x T fx * ( ) + -- x T fx ( * ) x must be greater than 0, where we have neglected O( x 3 ) terms. For this to be true it is obvious that f( x * ) 0 otherwise we could pick x appropriately so that x T fx ( * ) < 0.
3 Necessary Conditions A necessary condition for a local minimum is the analogue of a one dimensional function having a stationary point. Thus we can state the following definition and theorem. Definition If f( x * ) 0 then x * is called a stationary point of f() x. Theorem necessary condition for local minimum based on f() x x * is a local minimum of f() x only if x * is a stationary point of f() x, that is f( x * ) 0. Sufficient Conditions In order to obtain sufficient conditions from the Taylor expansion we need to study the second term: Qx ( ) -- x T fx ( * ) x This is called a quadratic form which can be written more generally as Qz () z T Az where z is an arbitrary column vector, A R n n is an arbitrary matrix and R n the quadratic form is a scalar. The properties of the quadratic form are governed by the properties of the matrix A. Definition The matrix A is called:. positive definite if Qz () > 0, z, z 0,. positive semidefinite if Qz () 0, z, z 0, 3. negative definite if Qz () < 0, z, z 0, 4. negative semidefinite if Qz () 0, z, z 0,and 5. indefinite if the sign of Qz () depends on z. Theorem Sufficient condition based on ( f, f) if f( x * ) 0 and fx ( * ) is positive definite then x * is a strong local minimum of f( x).
4 Note this is not a necessary condition since if fx ( * ) 0 and the O( x 3 ) terms are positive then x * is still a strong local minimum. Therefore a necessary condition involving fx ( * ) can be stated as follows: Theorem Necessary condition based on ( f, f) x * is a local minimum only if fx ( * ) 0 and fx ( * ) is positive semidefinite. Definition a stationary point which is not a local minimum or local maximum is called a saddle point. Example: Consider the function the gradient is easily found as f() x x + x f() x x x T and thus f( x * ) 0 x* x* 0 x * 00 T is a stationary point. The Hessian is independent of x since f(x) is a quadratic function. We now determine whether the Hessian is positive definite by looking at the quadratic form which can easily be written as fx ( * ) z Hx ( * ) H z T Hz z z z z z + z z z z z z + z z z z T Hz [ ( z z ) + z z ] [ ( z + z ) 3z z ]. Now we note that the first form is positive for z and z both negative or positive and that the second form is always positive for z and z of opposite sign. Thus z T Hz > 0, z, z 0 and thus Hx ( * ) H is positive definite x * is a local minimum. z
5 Examples of quadratic forms A quadratic form can always be made so that the matris symmetric, for consider the following examples: a) + x + 3x.5 x.5 3 x b).5 + x + 4 x 3 + x 3 x x x x 3 c) 00 + x + 3x 3 x x x x 3 d) consider the general two dimensional case, Q can always be chosen symmetric: + x + 3x Q b a 3 such that a + b, or the symmetric form Q c c 3 such that c ( a + b) 0.5.
6 Tests for Definiteness Definition the principal minors of a matrix a a a 3 a n A a a a 3 a n a 3 a 3 a 33 a n a n a n a n3 a nn a a a 3 a are written as a, a, 3 a a a 3, n A. a a a 3 a 3 a 33 Definition the eigenvalues of a matrix A are defined as all the scalars, which satisfy the characteristic equation A λ i I 0. λ i An nxn real symmetric matrix will have n real eigenvalues. Theorem Sylvester s Theorem A quadratic form z T Qz is positive definite if and only if all the leading principle determinants of the matrix Q are positive. Table : Tests for Definiteness definiteness eigenvalues principal minors. positive definite all λ i > 0 i > 0, i n. positive semidefinite all λ i 0 i 0, i n 3. negative definite all λ i < 0 < 0, > 0, 3 < 0 4. negative semidefinite all λ i 0 0, 0, indefinite some λ i 0, some λ i 0 none of the above It is also obvious from the definition of positive definite that: Theorem the matrix A is negative definite if A is positive definite.
7 Examples Determine the sign definiteness of the following:. A 30; we have, 5, and therefore since all leading principal determinants are positive definite. A is positive. A 3 0 ; we have, 7 and A is indefinite Example: Static Linear Springs K B A K P x System of linear springs in equilibrium The blocks A and B are two frictionless bodies connected to 3 linear elastic springs having spring constants K, K,. When the force P 0, 0, x 0 defines the natural position. We want to find new positions, x when a nonzero force P is applied. Solution: Recall that for a linear spring Hooke s Law says that F Kd where K is the spring constant and d is the displacement of the spring from equilibrium. The strain energy of a spring is the work done in stretching it, that is x E Kxdx --Kx. 0
8 The work put into the system by the constant force P is given by Px energy of the system is and thus the potential U --K. --K 3 ( x ) --K + + x Px We want to minimize this potential energy U according to the principle of minimum potential energy. Thus we have U * * * K x ( x ) 0 or U * * * K x 3 ( x ) + K x P 0 K + K + * x * 0 P which gives * PK ( K K + K + K ) * PK ( x + ) ( K K + K + K ) as the only stationary point. We now determine the Hessian as Hx () U U x x U x U x K + K + and need to know whether Hx () positive definite? The principal minors are given as K K + > 0, + K + ( K K + K + K ) > 0 and therefore x * *, x is the minimum. The value of the potential at this minimum is given as P ( K + ) U min K + K K + K
9 Quadratic Functions In general we write a scalar quadratic function of n variables, x qx () c b T + x + --x T Ax where c R, b R n, and the matrix A R n n. R n, in matrix notation as We note that from Taylor s expansion any function can be approximated by a quadratic function f( x + x) fx () x T + fx () + -- x T fx () x + O( x 3 ) in a small enough region. Therefore what we now say about quadratic functions qx () can be used later in approximations. In algebraic form we write b T qx () as n qx () c+ b i + -- x j a ij i j i where [ b b n ], and A [ a ij ] n n. Since x T As a quadratic form we can always assume it to be symmetric. n n Definition If A is positive definite then qx ( ) is called a positive definite quadratic function. Question Show that the gradient vector of a quadratic function by q() x Ax + b. qx () c b T + x + --x T Ax is given q Solution: To find the gradient vector we require:, k,, n. Writing qx () algebraically x k and rearranging we have: qx () c+ b i + -- x a x k + kj j Now since A is symmetric i i c+ b i + -- a kk x k a ij a ji j x k j k and we have i k j a ij x j + a kj x j + x k a ik + i k i k j k a ij x j
10 and therefore qx () c + b i + -- a kk x k + x k a kj x j + i j k i k j k a ij x j This result can be written in vactor notation as b k + -- a. kk x k + a kj b k + a kj x j Thus we see that the gradient of a quadratic is easily found by a matrix multiplication and vector addition. q x k j k q() x b + Ax j Question Show that the Hessian matrix of the quadratic function equal to A. qx () c b T + x + --x T Ax is Solution: We know that the Hessian matris given as and from the above we see that Hx () qx ( ()) T qx () x j n n Therefore q x j b j + a jk x k. k qx () a x ji a ij j and in matrix form Hx () H A and the Hessian of a quadratic function is a constant matrix. Thus we see that if qx () is a positive definite quadratic function then H is positive definite at all points which implies that x * will be a minimum. In fact since
11 is the unique minimum of a positive definite quadratic function. Example q( x * ) b + Ax * 0 x * A b Consider the following quadratic function in two variables: f(, x ) K + d + ex + cx + b x + a where: a, b, c, d, e, K. Determine the conditions under which f() s positive definite. Solution: Rewriting f() x in matrix form as f() x K de a b x x x b c x we see that the Hessian is given by A a b. b c Therefore by Sylvester s theorem A is positive definite if and only if: a > 0, 4ac b > 0 c > 0. From geometry we know that if 4ac b > 0 then the contour plots f() x const. define concentric ellipses. unique minimum Concentric ellipses define shape of quadratic function in -D
12 As a specific example if a d K, e, b 0andc,, we have f() x + x + + x f() x x x x 04 x and therefore A 0, b 04 A x * A b is the unique minimum.
Math (P)refresher Lecture 8: Unconstrained Optimization
Math (P)refresher Lecture 8: Unconstrained Optimization September 2006 Today s Topics : Quadratic Forms Definiteness of Quadratic Forms Maxima and Minima in R n First Order Conditions Second Order Conditions
More informationPerformance Surfaces and Optimum Points
CSC 302 1.5 Neural Networks Performance Surfaces and Optimum Points 1 Entrance Performance learning is another important class of learning law. Network parameters are adjusted to optimize the performance
More informationMath 5BI: Problem Set 6 Gradient dynamical systems
Math 5BI: Problem Set 6 Gradient dynamical systems April 25, 2007 Recall that if f(x) = f(x 1, x 2,..., x n ) is a smooth function of n variables, the gradient of f is the vector field f(x) = ( f)(x 1,
More informationLinear Algebra Solutions 1
Math Camp 1 Do the following: Linear Algebra Solutions 1 1. Let A = and B = 3 8 5 A B = 3 5 9 A + B = 9 11 14 4 AB = 69 3 16 BA = 1 4 ( 1 3. Let v = and u = 5 uv = 13 u v = 13 v u = 13 Math Camp 1 ( 7
More informationCE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions
CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura
More informationLinear Algebra And Its Applications Chapter 6. Positive Definite Matrix
Linear Algebra And Its Applications Chapter 6. Positive Definite Matrix KAIST wit Lab 2012. 07. 10 남성호 Introduction The signs of the eigenvalues can be important. The signs can also be related to the minima,
More informationHW3 - Due 02/06. Each answer must be mathematically justified. Don t forget your name. 1 2, A = 2 2
HW3 - Due 02/06 Each answer must be mathematically justified Don t forget your name Problem 1 Find a 2 2 matrix B such that B 3 = A, where A = 2 2 If A was diagonal, it would be easy: we would just take
More informationREVIEW OF DIFFERENTIAL CALCULUS
REVIEW OF DIFFERENTIAL CALCULUS DONU ARAPURA 1. Limits and continuity To simplify the statements, we will often stick to two variables, but everything holds with any number of variables. Let f(x, y) be
More informationChapter 2: Unconstrained Extrema
Chapter 2: Unconstrained Extrema Math 368 c Copyright 2012, 2013 R Clark Robinson May 22, 2013 Chapter 2: Unconstrained Extrema 1 Types of Sets Definition For p R n and r > 0, the open ball about p of
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationOR MSc Maths Revision Course
OR MSc Maths Revision Course Tom Byrne School of Mathematics University of Edinburgh t.m.byrne@sms.ed.ac.uk 15 September 2017 General Information Today JCMB Lecture Theatre A, 09:30-12:30 Mathematics revision
More informationHere each term has degree 2 (the sum of exponents is 2 for all summands). A quadratic form of three variables looks as
Reading [SB], Ch. 16.1-16.3, p. 375-393 1 Quadratic Forms A quadratic function f : R R has the form f(x) = a x. Generalization of this notion to two variables is the quadratic form Q(x 1, x ) = a 11 x
More informationDifferentiable Functions
Differentiable Functions Let S R n be open and let f : R n R. We recall that, for x o = (x o 1, x o,, x o n S the partial derivative of f at the point x o with respect to the component x j is defined as
More informationMath Matrix Algebra
Math 44 - Matrix Algebra Review notes - (Alberto Bressan, Spring 7) sec: Orthogonal diagonalization of symmetric matrices When we seek to diagonalize a general n n matrix A, two difficulties may arise:
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationIntro to Nonlinear Optimization
Intro to Nonlinear Optimization We now rela the proportionality and additivity assumptions of LP What are the challenges of nonlinear programs NLP s? Objectives and constraints can use any function: ma
More informationMATH 5720: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 2018
MATH 57: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 18 1 Global and Local Optima Let a function f : S R be defined on a set S R n Definition 1 (minimizers and maximizers) (i) x S
More informationOptimality Conditions
Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.
More informationNonlinear equations and optimization
Notes for 2017-03-29 Nonlinear equations and optimization For the next month or so, we will be discussing methods for solving nonlinear systems of equations and multivariate optimization problems. We will
More informationReview of Vectors and Matrices
A P P E N D I X D Review of Vectors and Matrices D. VECTORS D.. Definition of a Vector Let p, p, Á, p n be any n real numbers and P an ordered set of these real numbers that is, P = p, p, Á, p n Then P
More informationSymmetry and Properties of Crystals (MSE638) Stress and Strain Tensor
Symmetry and Properties of Crystals (MSE638) Stress and Strain Tensor Somnath Bhowmick Materials Science and Engineering, IIT Kanpur April 6, 2018 Tensile test and Hooke s Law Upto certain strain (0.75),
More informationLecture Notes: Geometric Considerations in Unconstrained Optimization
Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections
More information8 Extremal Values & Higher Derivatives
8 Extremal Values & Higher Derivatives Not covered in 2016\17 81 Higher Partial Derivatives For functions of one variable there is a well-known test for the nature of a critical point given by the sign
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More information5 More on Linear Algebra
14.102, Math for Economists Fall 2004 Lecture Notes, 9/23/2004 These notes are primarily based on those written by George Marios Angeletos for the Harvard Math Camp in 1999 and 2000, and updated by Stavros
More informationSeptember Math Course: First Order Derivative
September Math Course: First Order Derivative Arina Nikandrova Functions Function y = f (x), where x is either be a scalar or a vector of several variables (x,..., x n ), can be thought of as a rule which
More informationOptimal Control. Quadratic Functions. Single variable quadratic function: Multi-variable quadratic function:
Optimal Control Control design based on pole-placement has non unique solutions Best locations for eigenvalues are sometimes difficult to determine Linear Quadratic LQ) Optimal control minimizes a quadratic
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationChapter 7. Extremal Problems. 7.1 Extrema and Local Extrema
Chapter 7 Extremal Problems No matter in theoretical context or in applications many problems can be formulated as problems of finding the maximum or minimum of a function. Whenever this is the case, advanced
More informationComputational Optimization. Mathematical Programming Fundamentals 1/25 (revised)
Computational Optimization Mathematical Programming Fundamentals 1/5 (revised) If you don t know where you are going, you probably won t get there. -from some book I read in eight grade If you do get there,
More informationLecture 6 Positive Definite Matrices
Linear Algebra Lecture 6 Positive Definite Matrices Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Spring 2017 2017/6/8 Lecture 6: Positive Definite Matrices
More information8. Diagonalization.
8. Diagonalization 8.1. Matrix Representations of Linear Transformations Matrix of A Linear Operator with Respect to A Basis We know that every linear transformation T: R n R m has an associated standard
More informationPaul Schrimpf. October 18, UBC Economics 526. Unconstrained optimization. Paul Schrimpf. Notation and definitions. First order conditions
Unconstrained UBC Economics 526 October 18, 2013 .1.2.3.4.5 Section 1 Unconstrained problem x U R n F : U R. max F (x) x U Definition F = max x U F (x) is the maximum of F on U if F (x) F for all x U and
More informationVector Derivatives and the Gradient
ECE 275AB Lecture 10 Fall 2008 V1.1 c K. Kreutz-Delgado, UC San Diego p. 1/1 Lecture 10 ECE 275A Vector Derivatives and the Gradient ECE 275AB Lecture 10 Fall 2008 V1.1 c K. Kreutz-Delgado, UC San Diego
More informationAppendix A Taylor Approximations and Definite Matrices
Appendix A Taylor Approximations and Definite Matrices Taylor approximations provide an easy way to approximate a function as a polynomial, using the derivatives of the function. We know, from elementary
More informationMath 291-2: Final Exam Solutions Northwestern University, Winter 2016
Math 29-2: Final Exam Solutions Northwestern University, Winter 206 Determine whether each of the following statements is true or false f it is true, explain why; if it is false, give a counterexample
More informationRoots and Coefficients Polynomials Preliminary Maths Extension 1
Preliminary Maths Extension Question If, and are the roots of x 5x x 0, find the following. (d) (e) Question If p, q and r are the roots of x x x 4 0, evaluate the following. pq r pq qr rp p q q r r p
More informationThe Derivative. Appendix B. B.1 The Derivative of f. Mappings from IR to IR
Appendix B The Derivative B.1 The Derivative of f In this chapter, we give a short summary of the derivative. Specifically, we want to compare/contrast how the derivative appears for functions whose domain
More informationCalculus 2502A - Advanced Calculus I Fall : Local minima and maxima
Calculus 50A - Advanced Calculus I Fall 014 14.7: Local minima and maxima Martin Frankland November 17, 014 In these notes, we discuss the problem of finding the local minima and maxima of a function.
More informationChapter 2 BASIC PRINCIPLES. 2.1 Introduction. 2.2 Gradient Information
Chapter 2 BASIC PRINCIPLES 2.1 Introduction Nonlinear programming is based on a collection of definitions, theorems, and principles that must be clearly understood if the available nonlinear programming
More informationIn these chapter 2A notes write vectors in boldface to reduce the ambiguity of the notation.
1 2 Linear Systems In these chapter 2A notes write vectors in boldface to reduce the ambiguity of the notation 21 Matrix ODEs Let and is a scalar A linear function satisfies Linear superposition ) Linear
More informationSymmetric matrices and dot products
Symmetric matrices and dot products Proposition An n n matrix A is symmetric iff, for all x, y in R n, (Ax) y = x (Ay). Proof. If A is symmetric, then (Ax) y = x T A T y = x T Ay = x (Ay). If equality
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More informationPositive Definite Matrix
1/29 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Positive Definite, Negative Definite, Indefinite 2/29 Pure Quadratic Function
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationLecture 2 - Unconstrained Optimization Definition[Global Minimum and Maximum]Let f : S R be defined on a set S R n. Then
Lecture 2 - Unconstrained Optimization Definition[Global Minimum and Maximum]Let f : S R be defined on a set S R n. Then 1. x S is a global minimum point of f over S if f (x) f (x ) for any x S. 2. x S
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationA A x i x j i j (i, j) (j, i) Let. Compute the value of for and
7.2 - Quadratic Forms quadratic form on is a function defined on whose value at a vector in can be computed by an expression of the form, where is an symmetric matrix. The matrix R n Q R n x R n Q(x) =
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More information1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )
Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical
More informationMultivariate Newton Minimanization
Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.
More informationLecture Unconstrained optimization. In this lecture we will study the unconstrained problem. minimize f(x), (2.1)
Lecture 2 In this lecture we will study the unconstrained problem minimize f(x), (2.1) where x R n. Optimality conditions aim to identify properties that potential minimizers need to satisfy in relation
More informationIntroduction to gradient descent
6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationMATH529 Fundamentals of Optimization Unconstrained Optimization II
MATH529 Fundamentals of Optimization Unconstrained Optimization II Marco A. Montes de Oca Mathematical Sciences, University of Delaware, USA 1 / 31 Recap 2 / 31 Example Find the local and global minimizers
More informationVectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1. x 2. x =
Linear Algebra Review Vectors To begin, let us describe an element of the state space as a point with numerical coordinates, that is x 1 x x = 2. x n Vectors of up to three dimensions are easy to diagram.
More informationMax-Min Problems in R n Matrix
Max-Min Problems in R n Matrix 1 and the essian Prerequisite: Section 6., Orthogonal Diagonalization n this section, we study the problem of nding local maxima and minima for realvalued functions on R
More informationCHAPTER 2: QUADRATIC PROGRAMMING
CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,
More informationVasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks
C.M. Bishop s PRML: Chapter 5; Neural Networks Introduction The aim is, as before, to find useful decompositions of the target variable; t(x) = y(x, w) + ɛ(x) (3.7) t(x n ) and x n are the observations,
More informationSTAT200C: Review of Linear Algebra
Stat200C Instructor: Zhaoxia Yu STAT200C: Review of Linear Algebra 1 Review of Linear Algebra 1.1 Vector Spaces, Rank, Trace, and Linear Equations 1.1.1 Rank and Vector Spaces Definition A vector whose
More informationLinear Algebra: Linear Systems and Matrices - Quadratic Forms and Deniteness - Eigenvalues and Markov Chains
Linear Algebra: Linear Systems and Matrices - Quadratic Forms and Deniteness - Eigenvalues and Markov Chains Joshua Wilde, revised by Isabel Tecu, Takeshi Suzuki and María José Boccardi August 3, 3 Systems
More informationBasics of Calculus and Algebra
Monika Department of Economics ISCTE-IUL September 2012 Basics of linear algebra Real valued Functions Differential Calculus Integral Calculus Optimization Introduction I A matrix is a rectangular array
More informationCHAPTER 4: HIGHER ORDER DERIVATIVES. Likewise, we may define the higher order derivatives. f(x, y, z) = xy 2 + e zx. y = 2xy.
April 15, 2009 CHAPTER 4: HIGHER ORDER DERIVATIVES In this chapter D denotes an open subset of R n. 1. Introduction Definition 1.1. Given a function f : D R we define the second partial derivatives as
More informationUNCONSTRAINED OPTIMIZATION PAUL SCHRIMPF OCTOBER 24, 2013
PAUL SCHRIMPF OCTOBER 24, 213 UNIVERSITY OF BRITISH COLUMBIA ECONOMICS 26 Today s lecture is about unconstrained optimization. If you re following along in the syllabus, you ll notice that we ve skipped
More information1 Convexity, concavity and quasi-concavity. (SB )
UNIVERSITY OF MARYLAND ECON 600 Summer 2010 Lecture Two: Unconstrained Optimization 1 Convexity, concavity and quasi-concavity. (SB 21.1-21.3.) For any two points, x, y R n, we can trace out the line of
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationEC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline
LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationElements of Continuum Elasticity. David M. Parks Mechanics and Materials II February 25, 2004
Elements of Continuum Elasticity David M. Parks Mechanics and Materials II 2.002 February 25, 2004 Solid Mechanics in 3 Dimensions: stress/equilibrium, strain/displacement, and intro to linear elastic
More information4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:
Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationChapter 1: Systems of linear equations and matrices. Section 1.1: Introduction to systems of linear equations
Chapter 1: Systems of linear equations and matrices Section 1.1: Introduction to systems of linear equations Definition: A linear equation in n variables can be expressed in the form a 1 x 1 + a 2 x 2
More informationMath Camp Notes: Linear Algebra I
Math Camp Notes: Linear Algebra I Basic Matrix Operations and Properties Consider two n m matrices: a a m A = a n a nm Then the basic matrix operations are as follows: a + b a m + b m A + B = a n + b n
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationLinear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.
Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the
More information1 Kernel methods & optimization
Machine Learning Class Notes 9-26-13 Prof. David Sontag 1 Kernel methods & optimization One eample of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions
More informationMatrices and Deformation
ES 111 Mathematical Methods in the Earth Sciences Matrices and Deformation Lecture Outline 13 - Thurs 9th Nov 2017 Strain Ellipse and Eigenvectors One way of thinking about a matrix is that it operates
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More information1. General Vector Spaces
1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule
More informationLecture 8: Maxima and Minima
Lecture 8: Maxima and Minima Rafikul Alam Department of Mathematics IIT Guwahati Local extremum of f : R n R Let f : U R n R be continuous, where U is open. Then f has a local maximum at p if there exists
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationPhysics 403. Segev BenZvi. Parameter Estimation, Correlations, and Error Bars. Department of Physics and Astronomy University of Rochester
Physics 403 Parameter Estimation, Correlations, and Error Bars Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Review of Last Class Best Estimates and Reliability
More informationMathematical Economics: Lecture 16
Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More informationLecture 1 Notes: 06 / 27. The first part of this class will primarily cover oscillating systems (harmonic oscillators and waves).
Lecture 1 Notes: 06 / 27 The first part of this class will primarily cover oscillating systems (harmonic oscillators and waves). These systems are very common in nature - a system displaced from equilibrium
More information(a) Write down the value of q and of r. (2) Write down the equation of the axis of symmetry. (1) (c) Find the value of p. (3) (Total 6 marks)
1. Let f(x) = p(x q)(x r). Part of the graph of f is shown below. The graph passes through the points ( 2, 0), (0, 4) and (4, 0). (a) Write down the value of q and of r. (b) Write down the equation of
More informationApril 26, Applied mathematics PhD candidate, physics MA UC Berkeley. Lecture 4/26/2013. Jed Duersch. Spd matrices. Cholesky decomposition
Applied mathematics PhD candidate, physics MA UC Berkeley April 26, 2013 UCB 1/19 Symmetric positive-definite I Definition A symmetric matrix A R n n is positive definite iff x T Ax > 0 holds x 0 R n.
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 3: Positive-Definite Systems; Cholesky Factorization Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 11 Symmetric
More informationAppendix 3. Derivatives of Vectors and Vector-Valued Functions DERIVATIVES OF VECTORS AND VECTOR-VALUED FUNCTIONS
Appendix 3 Derivatives of Vectors and Vector-Valued Functions Draft Version 1 December 2000, c Dec. 2000, B. Walsh and M. Lynch Please email any comments/corrections to: jbwalsh@u.arizona.edu DERIVATIVES
More informationGeneral Physics I Spring Oscillations
General Physics I Spring 2011 Oscillations 1 Oscillations A quantity is said to exhibit oscillations if it varies with time about an equilibrium or reference value in a repetitive fashion. Oscillations
More informationLinear Algebra. Linear Equations and Matrices. Copyright 2005, W.R. Winfrey
Copyright 2005, W.R. Winfrey Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
More informationPrincipal Components Theory Notes
Principal Components Theory Notes Charles J. Geyer August 29, 2007 1 Introduction These are class notes for Stat 5601 (nonparametrics) taught at the University of Minnesota, Spring 2006. This not a theory
More informationAM 205: lecture 6. Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization
AM 205: lecture 6 Last time: finished the data fitting topic Today s lecture: numerical linear algebra, LU factorization Unit II: Numerical Linear Algebra Motivation Almost everything in Scientific Computing
More information3 (Maths) Linear Algebra
3 (Maths) Linear Algebra References: Simon and Blume, chapters 6 to 11, 16 and 23; Pemberton and Rau, chapters 11 to 13 and 25; Sundaram, sections 1.3 and 1.5. The methods and concepts of linear algebra
More information