, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
|
|
- Abel Perkins
- 6 years ago
- Views:
Transcription
1 Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues of A, the quadratic function f behaves very differently. Note that A is the second derivative of f, i.e., the Hessian matrix. To study basic properties of quadratic forms we first consider the case with a positive definite matrix ( ) 2 A =, b =. (2) 2 The eigenvectors of A corresponding to the eigenvalues λ =, λ 2 = 3 are u = ( ) u 2 = ( ). 2 2 Defining the orthogonal matrix U := [u, u 2 ], we obtain the eigenvalue de Figure : Quadratic form with the positive definite matrix A defined in (2). Left: Contour lines of f(x); the red lines indicate the eigenvector directions of A. Right: Graph of the function. Note that the function is bounded from below and convex. composition of A, i.e., U T AU = Λ = ( ). 3 Note that U T = U. The contour lines of f as well as the eigenvector directions are shown in Figure. Defining the new variables x := U T x, the
2 quadratic form corresponding to (2) can, in the new variables, be written as f( x) = 2 xλ x. Thus, in the variables that correspond to the eigenvector directions, the quadratic form is based on the diagonal matrix Λ, and the eigenvalue matrix U corresponds to the basis transformation. So to study basic properties of quadratic forms, we can restrict ourselves to diagonal matrices A. Note also that the curvature d2 f( x) d x 2 i is simply λ i, so that the ith eigenvalue represents the curvature in the direction of the ith eigenvector. In two dimensions, f( x) is an ellipse with principle radii proportional to λ.5 i Figure 2: Quadratic form with indefinite matrices A (upper row) and A 2 (lower row) defined in (3). Left: Contour lines. Right: Graph of the function. Note that the function is unbounded from above and from below. 2
3 We next consider the quadratic form corresponding to the indefinite matrices ( ) ( ) 2 2 A =, A 2 2 =, (3) 2 and use b =. Visualizations of the corresponding quadratic form are shown in Figure 2. Note that the function corresponding to A coincides with the one from A 2 after interchanging the coordinate axes. The origin is a maximum in one coordinate direction and a minimum in the other direction, which is a consequence of the indefiniteness of matrices A and A 2. The corresponding quadratic functions are bounded neither from above nor from below, and thus do not have a minimum or a maximum. Finally, we study quadratic forms with semi-definite Hessian matrices. We consider the cases A = ( ) 2, b = ( ), b 2 = ( ) For the indefinite case, the choice of b influences whether there exists a minimum or not. Visualizations of the quadratic forms can be seen in Figure 3. In the direction in which the Hessian matrix is singular, the function is determined by the linear term b. The function based on A and b is unbounded from below and, thus, does not have a minimum. On the other hand, the function based on A and b 2 is independent of x 2, and bounded from below. Thus, all points with x 2 = are minima of f. If b i lies in the range space of A (i.e., it can be written as a linear combination of the columns of A), then there exists a minimum, and in fact infinitely many when A is singular, as it is here. On the other hand, when b i does not lie in the range space of A, then there does not exist a minimum. In summary, when the symmetric matrix A is positive definite, the quadratic form has a unique minimum at the stationary point (i.e. the point at which the gradient vanishes). When A is indefinite, the quadratic form has a stationary point, but it is not a minimum. Finally, when A is singular, it has either no stationary points (when b does not lie in the range space of A), or infinitely many (when b lies in the range space). Convergence of steepest descent for increasingly illconditioned matrices We consider the quadratic function (4) f(x, x 2 ) = 2 (c x 2 + c 2 x 2 2) = 2 xt Ax (5) 3
4 Figure 3: Quadratic form with semi-definite matrix A, along with b (upper row) and b 2 (lower row) defined in (4). Note that the case of b corresponds to an inconsistent right side (i.e., b does not belong to the range space of A), and that of b 2 corresponds to a consistent right side (b 2 does belong to the range space of A). Left: Contour lines. Right: Graph of the function. Note that depending on b, the function does not have a minimum (upper row) or has infinitely many minima (lower row). for various c and c 2, where A = diag(c, c 2 ) and x = (x, x 2 ) T. The function is convex and has a global minimum at x = x 2 =. Since A is diagonal, c and c 2 are also the eigenvalues of A. We use the steepest descent method with exact line search to minimize (5). A listing of the simple algorithm is given next. % s t a r t i n g p o i n t x = c2 / s q r t ( c ˆ2 + c2 ˆ 2 ) ; x2 = c / s q r t ( c ˆ2 + c2 ˆ 2 ) ; f o r i t e r = : e r r = s q r t ( x ˆ2 + x2 ˆ 2 ) ; f p r i n t f ( I t e r : %3d : x : %+4.8f, x2 : %+4.8f, e r r o r %4.8 f \n, i t e r, x, x2, e r r ) ; 4
5 i f ( e r r o r < e 2) f p r i n t f ( Converged with e r r o r %2.2 f. \ n, e r r o r ) ; b reak ; end % e x a c t l i n e s e a r c h a l p h a = ( c ˆ2 x ˆ2 + c2 ˆ2 x2 ˆ2) / ( c ˆ3 x ˆ2 + c2 ˆ3 x2 ˆ 2 ) ; g = c x ; g2 = c2 x2 ; x = x a l p h a g ; x2 = x2 a l p h a g2 ; end Running the above script with c = c 2 =, the method terminates after a single iteration. This one-step convergence is a property of the steepest descent when the eigenvalues c, c 2 coincide and thus the contour lines are circles; see Figure Figure 4: Contour lines and iterates for c = c 2 = (left plot) and c = 5, c 2 = (right plot). I t e r a t i o n : x : , x2 : , e r r o r. I t e r a t i o n 2 : x : +., x2 : +., e r r o r. Converged with e r r o r.. For c = 5, c 2 =, the iteration terminates after 36 iterations; the first iterations are as follows: I t e r a t i o n : x : , x2 : , e r r o r. I t e r a t i o n 2 : x :.37449, x2 : , e r r o r I t e r a t i o n 3 : x : , x2 : , e r r o r I t e r a t i o n 4 : x :.58848, x2 : , e r r o r I t e r a t i o n 5 : x : , x2 : , e r r o r
6 I t e r a t i o n 6 : x : , x2 : , e r r o r I t e r a t i o n 7 : x : , x2 : , e r r o r I t e r a t i o n 8 : x :.47822, x2 : , e r r o r I t e r a t i o n 9 : x : , x2 : , e r r o r I t e r a t i o n : x :.543, x2 : , e r r o r Taking coefficients between errors of two consecutive iterations, we observe that err k+ err k = 2 3 = = κ κ +, where κ denotes the condition number of the matrix A, i.e., ( ) c κ = cond = λ max(a) c 2 λ min (A) = c. c 2 The contour lines of f for c = c 2 = and c = 5, c 2 = are shown in Figure 4. Now, we study the function (5) with c 2 = and with different values for c, namely c =, 5,,. The number of iterations required for these cases are 39, 629, 383 and 387, respectively. As can be seen, the number increases significantly with c, and thus with increasing κ. The output of the first iterations for c = are I t e r a t i o n : x : , x2 : , e r r o r. I t e r a t i o n 2 : x :.8423, x2 : , e r r o r I t e r a t i o n 3 : x : , x2 : , e r r o r I t e r a t i o n 4 : x : , x2 : , e r r o r I t e r a t i o n 5 : x : , x2 : , e r r o r I t e r a t i o n 6 : x : , x2 : , e r r o r I t e r a t i o n 7 : x : , x2 : , e r r o r I t e r a t i o n 8 : x : , x2 : , e r r o r I t e r a t i o n 9 : x : , x2 : , e r r o r I t e r a t i o n : x : , x2 : , e r r o r Taking quotients of consecutive errors, we again observe the theoretically expected convergence rate of (κ )/(κ + ) = 9/ =.882. The large number of iterations for the other cases of c can be explained by the increasing ill-conditioning of the quadratic form. The theoretical convergence rates for c = 5,, are.968,.982 and.998, respectively. These are also exactly the rates we observe for all these cases. Contour lines and iterates for c 2 =, 5 are shown in Figure 5. 6
7 Figure 5: Contour lines and iterates for c = (left plot) and c = 5 (right plot). Convergence examples for Newton s method Here, we study convergence properties of Newton s method for various functions. We start by studying the nonlinear function f : R R defined by f(x) = 2 x2 3 x3. (6) The graph of this function, as well as of its derivative are shown in Figure Figure 6: Graph of nonlinear function defined in (6) (left plot) and of its derivative (right plot). We want to find the (local) minimum of f, i.e., the point x =. As expected, at the local minimum, the derivative of f vanishes. However, the derivative 7
8 also vanishes at the local maximum x =. A Newton method to find the local minimum of f uses the stationarity of the derivative at the extremal points (minimum and maximum). At a given point x k, the new point x k+ is computed as (where we use a step length of ) x k+ = x2 k 2x k This expression is plotted in Figure 7. First, we consider a Newton iteration (7) 3 f(x)=x 2 /(2x ) Figure 7: The Newton step (7) plotted as a function. starting from x = 2. We obtain the iterations I t e r a t i o n : x :+2. I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x : I t e r a t i o n : x : +. Thus, the Newton method converges to the local maximum x =. Observe that initially, the convergence rate appears to be linear, but beginning from the 6th iteration we can observe the quadratic convergence of Newton s method, i.e., the number of correct digits doubles at every Newton iteration. Next we use the initialization x = 2, which results in the following iterations: 8
9 I t e r a t i o n : x : 2. I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x :.465 I t e r a t i o n : x :. Now, the iterates converge to the local minimum x =. Again we initially observe linear convergence, and as we get close to the minimum, the convergence becomes quadratic. To show how sensitive Newton s method is to the initial guess, we now initialize the method with values that are very close to each other. Initializing with x =.5 results in convergence to x = : I t e r a t i o n : x : +.5 I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x : I t e r a t i o n : x : I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : +.58 Initialization with x =.499 leads to convergence to the local minimum x = : I t e r a t i o n : x : I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x : I t e r a t i o n : x : I t e r a t i o n 2 : x : I t e r a t i o n 3 : x :
10 I t e r a t i o n 4 : x :.59 Finally, if the method is initialized with x =.5, it diverges since the second derivative (i.e., the Hessian) of f is singular at this point. Next, we study the function f(x) = x4 4 which is shown together with its derivative in Figure 8. This function has a singular Hessian at its minimum x =. Using the Newton method to find the Figure 8: Function f(x) = x 4 /4 (left) and its derivative (right). global minimum with the starting guess x = results in the following iterations (only the first 5 iterations are shown): I t e r a t i o n : x : +. I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x : I t e r a t i o n : x : I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : Note that the iterates converge to the solution x =, but they converge at just a linear rate due to singularity of the Hessian at the solution. For the initial guess x =, the iterates are the negative of the ones shown above.
11 Finally, we consider the negative hyperbolic secant function f(x) = sech(x). The graph of the function and its derivative are shown in Figure 9. This function changes its curvature, and thus Newton s method diverges if the initial guess is too far from the minimum. The Newton iteration to find the minimum x =. sech(x).4 sech(x)*tanh(x) Figure 9: Function f(x) = sech(x) (left) and its derivative (right). of this function computes, at a current iterate x k, the new iterate as x k+ = x k + sinh(2x k) cosh(2x k ) 3. (8) We first study the Newton iterates for starting value x =. and observe quadratic convergence (actually, the convergence is even faster than quadratic, which is due to properties of the hyperbolic secant function): I t e r a t i o n : x : +. I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x :. Starting the iteration from x =.2 or x =.5, the method also converges to the local (and global) minimum. The iterates for starting guess of x =.5 are: I t e r a t i o n : x : +.5 I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : +.
12 However, if the method is initialized with x =.6 (or any value larger than that), the method diverges. The first iterates for a starting guess of x =.6 are: I t e r a t i o n : x : +.6 I t e r a t i o n 2 : x : I t e r a t i o n 3 : x : I t e r a t i o n 4 : x : I t e r a t i o n 5 : x : I t e r a t i o n 6 : x : I t e r a t i o n 7 : x : I t e r a t i o n 8 : x : I t e r a t i o n 9 : x : I t e r a t i o n : x : This divergence is explained by the fact that the function is concave at x =.6. 2
Lecture Notes: Geometric Considerations in Unconstrained Optimization
Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationECE580 Partial Solution to Problem Set 3
ECE580 Fall 2015 Solution to Problem Set 3 October 23, 2015 1 ECE580 Partial Solution to Problem Set 3 These problems are from the textbook by Chong and Zak, 4th edition, which is the textbook for the
More informationECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor.
ECE580 Exam 1 October 4, 2012 1 Name: Solution Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc.
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationLECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION
15-382 COLLECTIVE INTELLIGENCE - S19 LECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION TEACHER: GIANNI A. DI CARO WHAT IF WE HAVE ONE SINGLE AGENT PSO leverages the presence of a swarm: the outcome
More informationIntroduction to Unconstrained Optimization: Part 2
Introduction to Unconstrained Optimization: Part 2 James Allison ME 555 January 29, 2007 Overview Recap Recap selected concepts from last time (with examples) Use of quadratic functions Tests for positive
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationIntroduction to gradient descent
6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationMATH 5720: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 2018
MATH 57: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 18 1 Global and Local Optima Let a function f : S R be defined on a set S R n Definition 1 (minimizers and maximizers) (i) x S
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationDeep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.
Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationECE580 Fall 2015 Solution to Midterm Exam 1 October 23, Please leave fractions as fractions, but simplify them, etc.
ECE580 Fall 2015 Solution to Midterm Exam 1 October 23, 2015 1 Name: Solution Score: /100 This exam is closed-book. You must show ALL of your work for full credit. Please read the questions carefully.
More information1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:
Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion
More informationLecture 6 Positive Definite Matrices
Linear Algebra Lecture 6 Positive Definite Matrices Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Spring 2017 2017/6/8 Lecture 6: Positive Definite Matrices
More informationIntroduction to unconstrained optimization - direct search methods
Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationMath (P)refresher Lecture 8: Unconstrained Optimization
Math (P)refresher Lecture 8: Unconstrained Optimization September 2006 Today s Topics : Quadratic Forms Definiteness of Quadratic Forms Maxima and Minima in R n First Order Conditions Second Order Conditions
More informationNumerical Methods I Solving Nonlinear Equations
Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationA A x i x j i j (i, j) (j, i) Let. Compute the value of for and
7.2 - Quadratic Forms quadratic form on is a function defined on whose value at a vector in can be computed by an expression of the form, where is an symmetric matrix. The matrix R n Q R n x R n Q(x) =
More informationAn Iterative Descent Method
Conjugate Gradient: An Iterative Descent Method The Plan Review Iterative Descent Conjugate Gradient Review : Iterative Descent Iterative Descent is an unconstrained optimization process x (k+1) = x (k)
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationCE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions
CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura
More informationExploring the energy landscape
Exploring the energy landscape ChE210D Today's lecture: what are general features of the potential energy surface and how can we locate and characterize minima on it Derivatives of the potential energy
More informationMath 118, Fall 2014 Final Exam
Math 8, Fall 4 Final Exam True or false Please circle your choice; no explanation is necessary True There is a linear transformation T such that T e ) = e and T e ) = e Solution Since T is linear, if T
More informationPositive Definite Matrix
1/29 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Positive Definite, Negative Definite, Indefinite 2/29 Pure Quadratic Function
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationMATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,
More informationMathematical optimization
Optimization Mathematical optimization Determine the best solutions to certain mathematically defined problems that are under constrained determine optimality criteria determine the convergence of the
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Taylor s Theorem Can often approximate a function by a polynomial The error in the approximation
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationLinear Algebra And Its Applications Chapter 6. Positive Definite Matrix
Linear Algebra And Its Applications Chapter 6. Positive Definite Matrix KAIST wit Lab 2012. 07. 10 남성호 Introduction The signs of the eigenvalues can be important. The signs can also be related to the minima,
More informationThere are six more problems on the next two pages
Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with
More informationHW3 - Due 02/06. Each answer must be mathematically justified. Don t forget your name. 1 2, A = 2 2
HW3 - Due 02/06 Each answer must be mathematically justified Don t forget your name Problem 1 Find a 2 2 matrix B such that B 3 = A, where A = 2 2 If A was diagonal, it would be easy: we would just take
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationPaul Schrimpf. October 18, UBC Economics 526. Unconstrained optimization. Paul Schrimpf. Notation and definitions. First order conditions
Unconstrained UBC Economics 526 October 18, 2013 .1.2.3.4.5 Section 1 Unconstrained problem x U R n F : U R. max F (x) x U Definition F = max x U F (x) is the maximum of F on U if F (x) F for all x U and
More informationTranspose & Dot Product
Transpose & Dot Product Def: The transpose of an m n matrix A is the n m matrix A T whose columns are the rows of A. So: The columns of A T are the rows of A. The rows of A T are the columns of A. Example:
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one
More informationAdvanced Mathematical Programming IE417. Lecture 24. Dr. Ted Ralphs
Advanced Mathematical Programming IE417 Lecture 24 Dr. Ted Ralphs IE417 Lecture 24 1 Reading for This Lecture Sections 11.2-11.2 IE417 Lecture 24 2 The Linear Complementarity Problem Given M R p p and
More informationTwo hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. 29 May :45 11:45
Two hours MATH20602 To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER NUMERICAL ANALYSIS 1 29 May 2015 9:45 11:45 Answer THREE of the FOUR questions. If more
More informationNonlinear Optimization
Nonlinear Optimization (Com S 477/577 Notes) Yan-Bin Jia Nov 7, 2017 1 Introduction Given a single function f that depends on one or more independent variable, we want to find the values of those variables
More informationC&O367: Nonlinear Optimization (Winter 2013) Assignment 4 H. Wolkowicz
C&O367: Nonlinear Optimization (Winter 013) Assignment 4 H. Wolkowicz Posted Mon, Feb. 8 Due: Thursday, Feb. 8 10:00AM (before class), 1 Matrices 1.1 Positive Definite Matrices 1. Let A S n, i.e., let
More informationTranspose & Dot Product
Transpose & Dot Product Def: The transpose of an m n matrix A is the n m matrix A T whose columns are the rows of A. So: The columns of A T are the rows of A. The rows of A T are the columns of A. Example:
More informationOptimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The
More informationMath 411 Preliminaries
Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector
More informationUnconstrained Optimization
1 / 36 Unconstrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University February 2, 2015 2 / 36 3 / 36 4 / 36 5 / 36 1. preliminaries 1.1 local approximation
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationChapter 13. Convex and Concave. Josef Leydold Mathematical Methods WS 2018/19 13 Convex and Concave 1 / 44
Chapter 13 Convex and Concave Josef Leydold Mathematical Methods WS 2018/19 13 Convex and Concave 1 / 44 Monotone Function Function f is called monotonically increasing, if x 1 x 2 f (x 1 ) f (x 2 ) It
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationNumerical Optimization
Unconstrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 01, India. NPTEL Course on Unconstrained Minimization Let f : R n R. Consider the optimization problem:
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More information5 Overview of algorithms for unconstrained optimization
IOE 59: NLP, Winter 22 c Marina A. Epelman 9 5 Overview of algorithms for unconstrained optimization 5. General optimization algorithm Recall: we are attempting to solve the problem (P) min f(x) s.t. x
More informationOptimality Conditions
Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.
More informationChapter 2. Optimization. Gradients, convexity, and ALS
Chapter 2 Optimization Gradients, convexity, and ALS Contents Background Gradient descent Stochastic gradient descent Newton s method Alternating least squares KKT conditions 2 Motivation We can solve
More information4 damped (modified) Newton methods
4 damped (modified) Newton methods 4.1 damped Newton method Exercise 4.1 Determine with the damped Newton method the unique real zero x of the real valued function of one variable f(x) = x 3 +x 2 using
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More information13. Nonlinear least squares
L. Vandenberghe ECE133A (Fall 2018) 13. Nonlinear least squares definition and examples derivatives and optimality condition Gauss Newton method Levenberg Marquardt method 13.1 Nonlinear least squares
More informationCHAPTER 2: QUADRATIC PROGRAMMING
CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,
More informationMultivariate Newton Minimanization
Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationMonotone Function. Function f is called monotonically increasing, if. x 1 x 2 f (x 1 ) f (x 2 ) x 1 < x 2 f (x 1 ) < f (x 2 ) x 1 x 2
Monotone Function Function f is called monotonically increasing, if Chapter 3 x x 2 f (x ) f (x 2 ) It is called strictly monotonically increasing, if f (x 2) f (x ) Convex and Concave x < x 2 f (x )
More informationPenalty and Barrier Methods. So we again build on our unconstrained algorithms, but in a different way.
AMSC 607 / CMSC 878o Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 3: Penalty and Barrier Methods Dianne P. O Leary c 2008 Reference: N&S Chapter 16 Penalty and Barrier
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationReview of Classical Optimization
Part II Review of Classical Optimization Multidisciplinary Design Optimization of Aircrafts 51 2 Deterministic Methods 2.1 One-Dimensional Unconstrained Minimization 2.1.1 Motivation Most practical optimization
More informationIntroduction - Motivation. Many phenomena (physical, chemical, biological, etc.) are model by differential equations. f f(x + h) f(x) (x) = lim
Introduction - Motivation Many phenomena (physical, chemical, biological, etc.) are model by differential equations. Recall the definition of the derivative of f(x) f f(x + h) f(x) (x) = lim. h 0 h Its
More informationTMA4180 Solutions to recommended exercises in Chapter 3 of N&W
TMA480 Solutions to recommended exercises in Chapter 3 of N&W Exercise 3. The steepest descent and Newtons method with the bactracing algorithm is implemented in rosenbroc_newton.m. With initial point
More informationLecture 3: Linesearch methods (continued). Steepest descent methods
Lecture 3: Linesearch methods (continued). Steepest descent methods Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 3: Linesearch methods (continued).
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationChapter 6. Eigenvalues. Josef Leydold Mathematical Methods WS 2018/19 6 Eigenvalues 1 / 45
Chapter 6 Eigenvalues Josef Leydold Mathematical Methods WS 2018/19 6 Eigenvalues 1 / 45 Closed Leontief Model In a closed Leontief input-output-model consumption and production coincide, i.e. V x = x
More informationOR MSc Maths Revision Course
OR MSc Maths Revision Course Tom Byrne School of Mathematics University of Edinburgh t.m.byrne@sms.ed.ac.uk 15 September 2017 General Information Today JCMB Lecture Theatre A, 09:30-12:30 Mathematics revision
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationPerformance Surfaces and Optimum Points
CSC 302 1.5 Neural Networks Performance Surfaces and Optimum Points 1 Entrance Performance learning is another important class of learning law. Network parameters are adjusted to optimize the performance
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationPreliminary Examination, Numerical Analysis, August 2016
Preliminary Examination, Numerical Analysis, August 2016 Instructions: This exam is closed books and notes. The time allowed is three hours and you need to work on any three out of questions 1-4 and any
More informationIPAM Summer School Optimization methods for machine learning. Jorge Nocedal
IPAM Summer School 2012 Tutorial on Optimization methods for machine learning Jorge Nocedal Northwestern University Overview 1. We discuss some characteristics of optimization problems arising in deep
More informationECE 680 Modern Automatic Control. Gradient and Newton s Methods A Review
ECE 680Modern Automatic Control p. 1/1 ECE 680 Modern Automatic Control Gradient and Newton s Methods A Review Stan Żak October 25, 2011 ECE 680Modern Automatic Control p. 2/1 Review of the Gradient Properties
More information