Unconstrained Multivariate Optimization

Size: px
Start display at page:

Download "Unconstrained Multivariate Optimization"

Transcription

1 Unconstrained Multivariate Optimization Multivariate optimization means optimization of a scalar function of a several variables: and has the general form: y = () min ( ) where () is a nonlinear scalar-valued function of the vector variable. Bacground Before we discuss optimization methods, we need to tal about how to characterize nonlinear, multivariable functions such as (). Consider the nd order aylor series epansion about the point 0 : If we let: ( ) ( ) ( ) ( ) ( ) + " # + # " # ( ) ( ) a = ( ) " + " b H = = " " "

2 Unconstrained Multivariate Optimization hen we can re-write the aylor series epansion as a quadratic approimation for (): and the derivatives are: ( ) a + b + H ( ) " b + H ( ) " H Hessian gradient We can describe some of the local geometric properties of () using its gradient and Hessian. If fact there are only a few possibilities for the local geometry, which can easily be differentiated by the eigenvalues of the Hessian (H). Recall that the eigenvalues of a square matri (H) are computed by finding all of the roots (λ i ) of its characteristic equation: I " H = 0

3 Unconstrained Multivariate Optimization he possible geometries are: ) if λ i < 0 ( i =,,n), the Hessian is said to be negative definite. his object has a unique maimum and is what we commonly refer to as a hill (in three dimensions). () ) if λ i > 0 ( i =,,n), the Hessian is said to be positive definite. his object has a unique minimum and is what we commonly refer to as a valley (in three dimensions). ()

4 Unconstrained Multivariate Optimization 3) if λ i < 0 ( i =,,m) and λ i > 0 ( i =m+,,n), the Hessian is said to be indefinite. his object has neither a unique maimum or minimum, and is what we commonly refer to as a saddle (in three dimensions). () 4) if λ i < 0 ( i =,,m) and λ i = 0 ( i =m+,,n), the Hessian is said to be negative semi-definite. his object does not have a unique maimum and is what we commonly refer to as a ridge (in three dimensions). ()

5 Unconstrained Multivariate Optimization 5) if λ i > 0 ( i =,,m) and λ i = 0 ( i =m+,,n), the Hessian is said to be positive semi-definite. his object does not have a unique minimum and is what we commonly refer to as a trough (in three dimensions). () A well posed problem has a unique optimum, so will will limit our discussions to either problems with positive definite Hessians (for minimization) or negative definite Hessians (for maimization). Further, we would prefer to choose units for our decision variables () so that the eigenvalues of the Hessian all have approimately the same magnitude. his will scale the problem so that the profit contours are concentric circles and will condition our optimization calculations.

6 Unconstrained Multivariate Optimization Necessary and Sufficient Conditions For a twice continuously differentiable scalar function (), a point * is an optimum if: and: * * 0 * = is positive definite (a minimum) is negative definite (a maimum) We can use these conditions directly, but it usually involves solving a set of simultaneous nonlinear equations (which is usually just as tough as the original optimization problem). Consider: ( ) A e = A hen: A = ( A) e + A( A) e = A ( A) e (+ A) and stationarity of the gradient requires that: A = A ( A) e (+ A) = 0 his is a set of very nonlinear equations in the variables.

7 Unconstrained Multivariate Optimization Eample Consider the scalar function: ( ) = or: $ ( ) = 3 + [ ] + # " & % where = [ ]. Stationarity of the gradient requires: " % = [ ] + $ = 0 # & " % ( $ " # & # $ % & " % = - = - $ # 0 & Chec the Hessian to classify the type of stationary point: " % " % = $ = 4 $ # & # 4 & he eigenvalues of the Hessian are: I " # 4 = = ( - ) = 0 $ % & # & ( % $ ( " 6 4 ) = 6, - i hus the Hessian is indefinite and the stationary point is a saddle point. his is a trivial eample, but it highlights the general procedure for direct use of the optimality conditions. )

8 Unconstrained Multivariate Optimization Lie the univariate search methods we studied earlier, multivariate optimization methods can be separated into two groups: those which depend solely on function evaluations, those which use derivative information (either analytical derivatives or numerical approimations). Regardless of which method you choose to solve a multivariate optimization problem, the general procedure will be: ) select a starting point(s), ) choose a search direction, multivariate method 3) minimize in chosen search direction, 4) repeat steps & 3 until converged. univariate search Also, successful solution of an optimization problem will require specification of a convergence criterion. Some possibilities include: + " ( ) ( ) " + # " $ % &

9 Sequential Univariate Searches erhaps the simplest multivariable search technique to implement would be a system of sequential univariate searches along some fied set of directions. Consider the two dimensional case, where the chosen search directions are parallel to the coordinate aes: * In this algorithm, you:. select a starting point 0,. select a coordinate direction (e.g. s = [0 ], or [ 0] ), 3. perform a univariate search min ( + s), 4. repeat steps and 3 alternating between search directions, until converged.

10 Sequential Univariate Searches he problem with this method is that a very large number of iterations may be required to attain a reasonable level of accuracy. If we new something about the orientation of the objective function the rate of convergence could be greatly enhanced. Consider the previous two dimensional problem, but using the independent search directions ([ ], [ -] ). * 0 Of course, finding the optimum in n steps only wors for quadratic objective functions where the Hessian is nown and each line search is eact. here are a large number of these optimization techniques which vary only in the way that the search directions are chosen.

11 Nelder-Meade Simple his approach has nothing to do with the SIMLEX method of linear programming. he method derives its name from an n-dimensional polytope (and as a result is often referred to as the polytope method). A polytope is an n-dimensional object that has n+ vertices: n polytope number of vertices line segment triangle 3 3 tetrahedron 4 M M M It is an easily implemented direct search method, that only relies on objective function evaluations. As a result, it can be robust to some types of discontinuities and so forth. However, it is often slow to converge and is not useful for larger problems (>0 variables).

12 Nelder-Meade Simple o illustrate the basic idea of the method, consider the twodimensional minimization problem: * Step : = ma (,, 3 ), define 4 as the reflection of through the centroid of the line joining and 3. 4 ma (,, 3 ), form a new polytope from the points, 3, 4. Steps through 4: repeat the procedure in Step to form new polytopes.

13 Nelder-Meade Simple We can repeat the procedure until the polytope defined by 4, 5, 6. Further iterations will cause us to flip between two polytopes. We can eliminate these problems by introducing two further operations into the method: contraction when the reflection step does not offer any improvement, epansion to accelerate convergence when the reflection step is an improvement *

14 Nelder-Meade Simple For minimization, the algorithm is: h is given by ( h ) = ma{( ),,( n+ )}, l is given by ( l ) = min{( ),,( n+ )}, calculate the centroid of the vertices other than h as: c & n - = $ +. + n %, i= i * ( ) h # " calculate reflection step: r = c - h if ( r ) < ( l ) then attempt epansion, e = c + γ ( r - c ) γ > (usually ~3, or could do a line search) if ( e ) < ( r ) the epansion was successful, form the new polytope replacing h with e. else the epansion step failed, form the new polytope replacing h with r. end if

15 Nelder-Meade Simple else if ( r ) < ma{( i ) i h } use the reflection, form the new polytope replacing h with r. else the reflection step failed, try a contraction, s = # c + ( h - c ), ( r ) " ( h ) $ % c + ( r - c ), ( r ) < ( h ) where 0 <β <, usually β =½. if ( s ) < min{( h ),( r )} use contraction, form the new polytope replacing h with s. else repeat contraction. end if. repeat the entire procedure until termination. he Nelder-Meade Simple method can be slow to converge, but it is useful for functions whose derivatives cannot be calculated or approimated (e.g. some non-smooth functions).

16 Nelder-Meade Simple Eample min r r notes epansion γ=3, failed 0 epansion γ=3, failed epansion γ=3, failed 0 0 contraction β= ½ epansion γ=3, failed

17 Gradient-Based Methods his family of optimization methods use first-order derivatives to determine a descent direction. (Recall that the gradient gives the direction of quicest increase for the objective function. hus, the negative of the gradient gives the direction of quicest decrease for the objective function.) Steepest Descent It would seem that the fastest way to find an optimum would be to always move in the direction in which the objective function decreases the fastest. Although this idea is intuitively appealing, it is very rarely true; however, with a good line search every iteration will move closer to the optimum.

18 Gradient-Based Methods he Steepest Descent algorithm is:. choose a starting point 0,. calculate the search direction: [ ] s = " 3. determine the net iterate for : = + + s by solving the univariate optimization problem: min ( + s ) 4. repeat steps & 3 until termination. Some termination criteria to consider: + ( )- ( ) + - # " $ # where ε is some small positive scalar. # # " " "

19 Gradient-Based Methods Steepest Descent Eample # min + + start at 0 = [3 3] ( = & $ % # " start at 0 = [ ] " ( ) 0 # 3 % & 4 $ 3 ( # $ 4 # % & 0 $ ( # % & $ 0 ( 0 # % & 0 $ ( # $ # % & 0 $ ( # % & $ 0 ( % & (? 8 " ( ) % & (? his eample shows that the method of Steepest Descent is very efficient for well-scaled quadratic optimization problems (the profit contours are concentric circles). his method is not nearly as efficient for non-quadratic objective functions or objective functions with very elliptical profit contours (poorly scaled). 0 0

20 Gradient-Based Methods Steepest Descent Eample # Consider the minimization problem: hen: " 0 99% min $ # 99 0 & # 0 " 99& = % $ " ( = 0 " " o develop an objective function for our line search, substitute + = + λ s into the original objective function: o simplify our wor, let H = [ ] $ 0 # 99 ( - " ) ( - & ) " ) ( + s ) = %# 99 0 ( = * $ 0 # 99 - $ 0 # 99 * $ 0 # 99, & %# ) / (. & %# ), & ( + %# 99 0 ) (, which yields: his epression is quadratic in λ, thus maing it easy to perform an eact line search: " 0 99% $ # 99 0 & 3 [ " + ] s H H H ( + ) = d = - H + H = 0 3 d - /.

21 Gradient-Based Methods Solving for the step length yields: # 0 " 99& H % " 99 0 ( = = $ 3 H 0 99 # " & % $ " 99 0 ( 3 = Starting at 0 = [ ] : # 0, 00 " 9, 998& % $ " 9, 998 0, 00 ( # 4, 000, 004 " 3, 999, 996& % $ " 3, 999, 996 4, 000, 004 ( ( ) λ

22 Gradient-Based Methods Conjugate Gradients Steepest Descent was based on the idea that the minimum will be found provided that we always move in the downhill direction. he second eample showed that the gradient does not always point in the direction of the optimum, due to the geometry of the problem. Fletcher and Reeves (964) developed a method which attempts to account for local curvature in determining the net search direction. he algorithm is:. choose a starting point 0,. set the initial search direction as: [ ] 0 s0 = " 3. determine the net iterate for : = + + s by solving the univariate optimization problem: min ( + s ) 4. calculate the net search direction as: s " " = " s " " 5. repeat steps 3 & 4 until termination. +

23 Gradient-Based Methods he manner in which the successive search directions are calculated is important. For quadratic functions, these successive search directions are conjugate with respect to the Hessian. his means that for the quadratic function: ( ) = a + b + H successive search directions satisfy: s Hs + As a result, these successive search directions have incorporated within them information about the geometry of the optimization problem. = 0 Conjugate Gradients Eample Consider again the optimization problem: " 0 99% min $ # 99 0 & We saw that Steepest Descent was slow to converge on this problem because of its poor scaling. As in the previous eample, we will use an eact line search. It can be shown that the optimal step length is: # 0 " 99& % s " 99 0 ( = - $ 0 99 # " & s % s $ " 99 0 (

24 Gradient-Based Methods Starting at 0 = [ ] : ( ) s " 0 # % & $ ( # & % $ ( # 0 % & $ 0 ( # 03 & % $ ) 97 ( #. 8809& % $ ( # 03 & % $ ) 97 ( #). 977& % $ ) ( Notice the much quicer convergence of the algorithm. For a quadratic objective function with eact line searches, the Conjugate Gradients method ehibits quadratic convergence. he quicer convergence comes at an increased computational requirement. hese include: a more comple search direction calculation, increased storage for maintaining previous search directions and gradients. hese are usually small in relation to the performance improvements.

25 Newton s Method he Conjugate Gradients method showed that there was a considerable performance gain to be realized by incorporating some problem geometry into the optimization algorithm. However, the method did this in an approimate sense. Newton s method does this by using second-order derivative information. In the multivariate case, Newton s method determines a search direction by using the Hessian to modify the gradient. he method is developed directly from the aylor Series approimation of the objective function. Recall that an objective function can be locally approimated as: hen, stationarity can be approimated as: ( ) ( ) ( ) ( ) ( ) + " # + # " # ( ) ( ) " + # = 0 Which can be used to determine an epression for calculating the net point in the minimization procedure: ( ) ( ) + = " " Notice that in this case we get both the direction and the step length for the minimization. Further, if the objective function is quadratic, the optimum is found in a single (Newton Step) without a line search.

26 Newton s Method At a given point, the algorithm for Newton s method is:. calculate the gradient at the current point,. calculate the Hessian at the current point, 3. calculate the Newton step: 4. calculate the net point: " ( ) ( ) = " # # = repeat steps through 4 until termination. Newton s Method Eample Consider once again the optimization problem: min " 0 99% $ # 99 0 & he gradient and Hessian for this eample is: # 0 " 99& # 0 " 99 & = % $ " (, = 99 0 % $ " 99 0 (

27 Newton s Method Starting at 0 = [ ] : ( ) 0 " $ % # & " % $ #( 97 & " 0 $ % 0 # 0& " 0 ( 99% $ #( 99 0 & Comparing the three methods on this problem:.5 Steepest Descent Conjugate Gradients Newton

28 Newton s Method As epected Newton s method was very effective, eventhough the problem was not well scaled. poor scaling was compensated for by using the inverse of the Hessian. Steepest Descent was very good for the first two iterations, when we were quite far from the optimum. But it slowed rapidly as: * Q " 0 he Conjugate Gradients approach avoided this difficulty by including some measure of problem geometry in the algorithm. However, the method started out using the Steepest Descent direction and built in curvature information as the algorithm progressed to the optimum. For higher-dimensional and non-quadratic objective functions, the number of iterations required for convergence can increase substantially. In general, Newton s Method will yield the best convergence properties, at the cost of increased computational load to calculate and store the Hessian. he Conjugate Gradients approach can be effective in situations where computational requirements are an issue. However, there are some other methods where the Hessian is approimated using gradient information and the approimate Hessian is used to calculate a Newton step. hese are called the Quasi-Newton methods.

29 Newton s Method As we saw in the univariate case, care must be taen when using Newton s method for comple functions: 0.5 () Since Newton s method is searching for a stationary point, it can oscillate between such points in the above situation. For such comple functions, Newton s method is often implemented with an alternative to step calculate the net point: + = + " where the step length is determined as: i) 0 < λ < (so we don t tae a full step), ii) perform a line search on λ. Generally, Newton and Newton-lie methods are preferred to other methods, the main difficulty is determining the Hessian. 0

30 Approimate Newton Methods As we saw in the univariate case the derivatives can be approimated by finite differences. In multiple dimensions this can require substantially more calculations. As an eample, consider forward difference approimation of the gradient: + *" ( + i ) # ( = ( % $ )" i & i where δ i is a perturbation in the direction of i. his approach requires additional objective function evaluation per dimension for forward or bacward differencing, and additional objective function evaluations per dimension for central differencing. o approimate the Hessian using only finite differences in the objective function could have the form: + [ ( + ) # ( )]# [ ( + ) # ( )] * " i j = ( % $ () " i" j %& i j Finite difference approimation of the Hessian requires at least an additional objective function evaluations per dimension. Some differencing schemes will give better performance than others; however, the increased computational load required for difference approimation of the second derivatives precludes the use of this approach for larger problems. )

31 Approimate Newton Methods Alternatively, the Hessian can be approimated in terms of gradient information. In the forward difference case the Hessian can be approimated as: # + ( = ) & *" i " + i % hi = i hen, we can form a finite difference approimation to the Hessian as: ~ H h he problem is that often this approimate Hessian is nonsymmetric. A symmetric approimation can be formed as: $ ~ ~ H = H + H Whichever approach is used to approimate the derivatives, the Newton step is calculated from them. In general, Newton s methods based on finite differences can produce results similar to analytical results, providing care is taen in choosing the perturbations δ i. (Recall that as the algorithm proceeds: hen small approimation errors will affect convergence and accuracy.) # [ ] = i [ ] $ # * and " 0. Some alternatives which can require fewer computations, yet build curvature information as the algorithms proceed are the Quasi-Newton family.

32 Quasi-Newton Methods his family of methods are the multivariate analogs of the univariate Secant methods. hey build second derivative (Hessian) information as the algorithm proceeds to solution, using available gradient information. Consider the aylor series epansion for the gradient of the objective function along the step direction s : ( ) = + " + K + runcating the aylor series at the second-order terms an rearranging slightly yields the so-called Quasi-Newton Condition: ( ) H + = " # " = g hus, any algorithm which satisfies this condition can build up curvature information as it proceeds from to + in the direction Δ ; however, the curvature information is gained in only one direction. hen as these algorithms proceed the from iteration-to-iteration, the approimate Hessians can differ by only a ran-one matri: H$ H$ vu Combining the Quasi-Newton Condition and the update formula yields: $ + + = + ( ) $ ( ) ( $ + + ) H = H vu = g

33 Quasi-Newton Methods Which yields a solution for the vector u of: u v and a Hessian update formula: = [ g " H$ ] [" " ] H$ = H$ $ v v g H + + " providing Δ and v are not orthogonal Unfortunately this update formula is not unique with respect to the Quasi-Newton Condition and the step vector s. We could add an additional term of wz to the update formula, where the vector w is any vector from the null space of Δ. (Recall that if w null(δ ), then Δ w = 0). hus there are a family of such Quasi-Newton methods which differ only in this additional term wz. erhaps the most successful member of the family is the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update: H$ H$ " g " g + = + " g " $ H " " H$ " H$ " he BFGS update has several advantages including that it is symmetric and it can be further simplified if the step is calculated according to: $H = " $ #

34 Quasi-Newton Methods In this case the BFGS update formula simplifies to: H$ H$ " g " g # + = + + " g " " General Quasi-Newton Algorithm. Initialization: - choose a starting point 0, - choose an initial Hessian approimation (usually $H I ).. At iteration : - calculate the gradient, - calculate the search direction: - calculate the net iterate: + = + s using a line search on λ. - compute 0 = ( $ ) ( ) s = H " - update the Hessian ( ) using the method of your choice (e.g. BFGS). 3. Repeat step until termination., " g, ". $H +

35 Quasi-Newton Methods BFGS Eample Consider once again the optimization problem: " 0 99% min $ 99 0 # & Recall that the gradient for this eample is: = # 0 " 99 & % $ " 99 0 ( As before starting at 0 = [ ] : $ ( ) H s λ

36 Quasi-Newton Methods It is worth noting that the optimum is found in 3 steps (although we are very close within ). Also, notice that we have a very accurate approimation for the Hessian (to 7 significant figures) after updates. he BFGS algorithm was: not quite as efficient as Newtons method, approimately as efficient as the Conjugate Gradients approach, much more efficient than Steepest Descent. his is what should be epected for quadratic objective functions of low dimension. As the dimension of the problem increases, the Newton and Quasi-Newton methods will usually out perform the other methods. A cautionary note: he BFGS algorithm guarantees that the approimate Hessian remains positive definite (and invertible) providing that the initial matri is chosen as positive definite, in theory. However, round-off errors and sometimes the search history can cause the approimate Hessian to become badly conditioned. When this occurs, the approimate Hessian should be reset to some value (often the identity matri I).

37 Convergence During our discussions we have mentioned convergence properties of various algorithms. here are a number of ways with which convergence properties of an algorithm can be characterized. he most conventional approach is to determine the asymptotic behaviour of the sequence of distances between the iterates ( ) and the optimum ( * ), as the iterations of a particular algorithm proceed. ypically, the distance between an iterate and the optimum is defined as: - We are most interested in the behaviour of an algorithm within some neighbourhood of the optimum; hence, the asymptotic analysis. he asymptotic rate of convergence is defined as: * + lim " - - * * p = # with the asymptotic order p and asymptotic error β. In general, most algorithms show convergence properties which are within three categories: linear (0 β <, p = ), superlinear (β = 0, p = ), quadratic (β 0, p = ),

38 Convergence For the methods we have studied, it can be shown that: ) the Steepest Descent method ehibits superlinear convergence in most cases. For quadratic objective functions the asymptotic convergence properties can be shown to approach quadratic as κ(h) and linear as κ(h). ) the Conjugate Gradients method will generally show superlinear (and often nearly quadratic) convergence properties. 3) Newton s method converges quadratically. 4) Quasi-Newton methods can closely approach quadratic rates of convergence.

39 Unconstrained Multivariate Optimization As in the univariate case, there are two basic approaches: direct search (Nelder-Meade Simple,...), derivative-based (Steepest Descent, Newton s...). Choosing which method to use for a particular problem is a trade-off among: ease of implementation, convergence speed and accuracy, computational limitations. Unfortunately there is no best method in all situations. For nearly quadratic functions Newton s and Quasi-Newton methods will usually be very efficient. For nearly flat or nonsmooth objective functions, the direct search methods are often a good choice. For very large problems (more than 000 decision variables), algorithms which use only gradient information (Steepest Descent, Conjugate Gradients, Quasi- Newton, etc.) can be the only practical methods. Remember that the solution to a multivariable optimization problem requires: a function (and derivatives) that can be evaluated, an appropriate optimization method, a good starting point, a well chosen termination criterion.

40 Unconstrained Optimization roblems. For the Wastewater settling pond problem discussed in class, we found that the outlet concentration could be described as: c0( t) = c i te Assuming τ = 4 hours and c i =00 grams/liter, please do each of the following: t " a) Find the maimum outlet concentration using Matlab s fminbnd and fminunc commands. b) Determine the sensitivity of the maimum outlet concentration and time at which the maimum occurs to the assumed value of τ. c) Determine the sensitivity of the maimum outlet concentration and time at which the maimum occurs to the assumed value of c i.

41 Unconstrained Optimization roblems. he file inhibit.mat contains data from eperiments that relate the consumption rate (µ) of substrate (hr - ) in a wastewater treatment plant to the concentration of the substrate (S) in the feed (grams/litre) to the wastewater treatment plant. he relationship is epected to be: µ = K µ ma m + S + S K S Using the given data, please do each of the following: a) Estimate the model parameters µ, K m, and K using Matlab s lsqcurvefit command. b) he last data point in the supplied data is epected to be an outlier. Determine how sensitive the parameters estimates are to the suspected outlier? Constrained ultivariable NL - Until now we have only been discussing NL that do not have constraints. Most problems that we wish to solve will have some form of constraints. - i) equality constraints (material and energy balances) - ii) inequality constraints (limits on operating conditions) - hese type of problems require techniques that rely heavily on the unconstrained methods.

Chapter III. Unconstrained Univariate Optimization

Chapter III. Unconstrained Univariate Optimization 1 Chapter III Unconstrained Univariate Optimization Introduction Interval Elimination Methods Polynomial Approximation Methods Newton s Method Quasi-Newton Methods 1 INTRODUCTION 2 1 Introduction Univariate

More information

(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x)

(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x) Solving Nonlinear Equations & Optimization One Dimension Problem: or a unction, ind 0 such that 0 = 0. 0 One Root: The Bisection Method This one s guaranteed to converge at least to a singularity, i not

More information

EAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science

EAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one

More information

1 Numerical optimization

1 Numerical optimization Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms

More information

Lecture V. Numerical Optimization

Lecture V. Numerical Optimization Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize

More information

Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence

Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Root-finding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we

More information

1 Numerical optimization

1 Numerical optimization Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................

More information

Gradient Descent. Dr. Xiaowei Huang

Gradient Descent. Dr. Xiaowei Huang Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,

More information

Optimization Methods

Optimization Methods Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available

More information

Today. Introduction to optimization Definition and motivation 1-dimensional methods. Multi-dimensional methods. General strategies, value-only methods

Today. Introduction to optimization Definition and motivation 1-dimensional methods. Multi-dimensional methods. General strategies, value-only methods Optimization Last time Root inding: deinition, motivation Algorithms: Bisection, alse position, secant, Newton-Raphson Convergence & tradeos Eample applications o Newton s method Root inding in > 1 dimension

More information

Unconstrained optimization

Unconstrained optimization Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

More information

Methods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent

Methods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived

More information

Outline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems

Outline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

More information

17 Solution of Nonlinear Systems

17 Solution of Nonlinear Systems 17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m

More information

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality

More information

Quasi-Newton Methods

Quasi-Newton Methods Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications

More information

Numerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09

Numerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09 Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods

More information

Nonlinear Programming

Nonlinear Programming Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

More information

Static unconstrained optimization

Static unconstrained optimization Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R

More information

Lecture Notes: Geometric Considerations in Unconstrained Optimization

Lecture Notes: Geometric Considerations in Unconstrained Optimization Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections

More information

Statistics 580 Optimization Methods

Statistics 580 Optimization Methods Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of

More information

Convex Optimization. Problem set 2. Due Monday April 26th

Convex Optimization. Problem set 2. Due Monday April 26th Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining

More information

Chapter 4. Unconstrained optimization

Chapter 4. Unconstrained optimization Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file

More information

Multivariate Newton Minimanization

Multivariate Newton Minimanization Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.

More information

Algorithms for Constrained Optimization

Algorithms for Constrained Optimization 1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic

More information

Programming, numerics and optimization

Programming, numerics and optimization Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428

More information

Topic 8c Multi Variable Optimization

Topic 8c Multi Variable Optimization Course Instructor Dr. Raymond C. Rumpf Office: A 337 Phone: (915) 747 6958 E Mail: rcrumpf@utep.edu Topic 8c Multi Variable Optimization EE 4386/5301 Computational Methods in EE Outline Mathematical Preliminaries

More information

Optimization II: Unconstrained Multivariable

Optimization II: Unconstrained Multivariable Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1

More information

Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

More information

ECS550NFB Introduction to Numerical Methods using Matlab Day 2

ECS550NFB Introduction to Numerical Methods using Matlab Day 2 ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves

More information

Beyond Newton s method Thomas P. Minka

Beyond Newton s method Thomas P. Minka Beyond Newton s method Thomas P. Minka 2000 (revised 7/21/2017) Abstract Newton s method for optimization is equivalent to iteratively maimizing a local quadratic approimation to the objective function.

More information

v are uncorrelated, zero-mean, white

v are uncorrelated, zero-mean, white 6.0 EXENDED KALMAN FILER 6.1 Introduction One of the underlying assumptions of the Kalman filter is that it is designed to estimate the states of a linear system based on measurements that are a linear

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

An artificial neural networks (ANNs) model is a functional abstraction of the

An artificial neural networks (ANNs) model is a functional abstraction of the CHAPER 3 3. Introduction An artificial neural networs (ANNs) model is a functional abstraction of the biological neural structures of the central nervous system. hey are composed of many simple and highly

More information

Multidisciplinary System Design Optimization (MSDO)

Multidisciplinary System Design Optimization (MSDO) oday s opics Multidisciplinary System Design Optimization (MSDO) Approimation Methods Lecture 9 6 April 004 Design variable lining Reduced-Basis Methods Response Surface Approimations Kriging Variable-Fidelity

More information

Improving the Convergence of Back-Propogation Learning with Second Order Methods

Improving the Convergence of Back-Propogation Learning with Second Order Methods the of Back-Propogation Learning with Second Order Methods Sue Becker and Yann le Cun, Sept 1988 Kasey Bray, October 2017 Table of Contents 1 with Back-Propagation 2 the of BP 3 A Computationally Feasible

More information

STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION

STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION TAMARA G. KOLDA, ROBERT MICHAEL LEWIS, AND VIRGINIA TORCZON Abstract. We present a new generating set search (GSS) approach

More information

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 2005, DR RAPHAEL HAUSER 1. The Quasi-Newton Idea. In this lecture we will discuss

More information

Math 411 Preliminaries

Math 411 Preliminaries Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector

More information

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23 Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is

More information

Optimization II: Unconstrained Multivariable

Optimization II: Unconstrained Multivariable Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained

More information

5 Handling Constraints

5 Handling Constraints 5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest

More information

OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review

OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization

More information

Numerical Optimization: Basic Concepts and Algorithms

Numerical Optimization: Basic Concepts and Algorithms May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some

More information

AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods

AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α

More information

Numerical Optimization of Partial Differential Equations

Numerical Optimization of Partial Differential Equations Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada

More information

Higher-Order Methods

Higher-Order Methods Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth

More information

5 Quasi-Newton Methods

5 Quasi-Newton Methods Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min

More information

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term; Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many

More information

Optimization and Root Finding. Kurt Hornik

Optimization and Root Finding. Kurt Hornik Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding

More information

Scientific Computing: Optimization

Scientific Computing: Optimization Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture

More information

NonlinearOptimization

NonlinearOptimization 1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,

More information

Numerical Methods. Root Finding

Numerical Methods. Root Finding Numerical Methods Solving Non Linear 1-Dimensional Equations Root Finding Given a real valued function f of one variable (say ), the idea is to find an such that: f() 0 1 Root Finding Eamples Find real

More information

A Primer on Multidimensional Optimization

A Primer on Multidimensional Optimization A Primer on Multidimensional Optimization Prof. Dr. Florian Rupp German University of Technology in Oman (GUtech) Introduction to Numerical Methods for ENG & CS (Mathematics IV) Spring Term 2016 Eercise

More information

Chapter 8 Gradient Methods

Chapter 8 Gradient Methods Chapter 8 Gradient Methods An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Introduction Recall that a level set of a function is the set of points satisfying for some constant. Thus, a point

More information

Search Directions for Unconstrained Optimization

Search Directions for Unconstrained Optimization 8 CHAPTER 8 Search Directions for Unconstrained Optimization In this chapter we study the choice of search directions used in our basic updating scheme x +1 = x + t d. for solving P min f(x). x R n All

More information

Geometry optimization

Geometry optimization Geometry optimization Trygve Helgaker Centre for Theoretical and Computational Chemistry Department of Chemistry, University of Oslo, Norway European Summer School in Quantum Chemistry (ESQC) 211 Torre

More information

Review of Optimization Basics

Review of Optimization Basics Review of Optimization Basics. Introduction Electricity markets throughout the US are said to have a two-settlement structure. The reason for this is that the structure includes two different markets:

More information

2. Quasi-Newton methods

2. Quasi-Newton methods L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization

More information

Numerical Optimization

Numerical Optimization Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function

More information

8 Numerical methods for unconstrained problems

8 Numerical methods for unconstrained problems 8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields

More information

Performance Surfaces and Optimum Points

Performance Surfaces and Optimum Points CSC 302 1.5 Neural Networks Performance Surfaces and Optimum Points 1 Entrance Performance learning is another important class of learning law. Network parameters are adjusted to optimize the performance

More information

Numerical Methods in Informatics

Numerical Methods in Informatics Numerical Methods in Informatics Lecture 2, 30.09.2016: Nonlinear Equations in One Variable http://www.math.uzh.ch/binf4232 Tulin Kaman Institute of Mathematics, University of Zurich E-mail: tulin.kaman@math.uzh.ch

More information

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues

More information

Optimization Tutorial 1. Basic Gradient Descent

Optimization Tutorial 1. Basic Gradient Descent E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

More information

Data Mining (Mineria de Dades)

Data Mining (Mineria de Dades) Data Mining (Mineria de Dades) Lluís A. Belanche belanche@lsi.upc.edu Soft Computing Research Group Dept. de Llenguatges i Sistemes Informàtics (Software department) Universitat Politècnica de Catalunya

More information

0.1. Linear transformations

0.1. Linear transformations Suggestions for midterm review #3 The repetitoria are usually not complete; I am merely bringing up the points that many people didn t now on the recitations Linear transformations The following mostly

More information

Exploring the energy landscape

Exploring the energy landscape Exploring the energy landscape ChE210D Today's lecture: what are general features of the potential energy surface and how can we locate and characterize minima on it Derivatives of the potential energy

More information

MORE CURVE SKETCHING

MORE CURVE SKETCHING Mathematics Revision Guides More Curve Sketching Page of 3 MK HOME TUITION Mathematics Revision Guides Level: AS / A Level MEI OCR MEI: C4 MORE CURVE SKETCHING Version : 5 Date: 05--007 Mathematics Revision

More information

g(x,y) = c. For instance (see Figure 1 on the right), consider the optimization problem maximize subject to

g(x,y) = c. For instance (see Figure 1 on the right), consider the optimization problem maximize subject to 1 of 11 11/29/2010 10:39 AM From Wikipedia, the free encyclopedia In mathematical optimization, the method of Lagrange multipliers (named after Joseph Louis Lagrange) provides a strategy for finding the

More information

Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2

Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods

More information

Iterative Methods for Solving A x = b

Iterative Methods for Solving A x = b Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http

More information

Review of Classical Optimization

Review of Classical Optimization Part II Review of Classical Optimization Multidisciplinary Design Optimization of Aircrafts 51 2 Deterministic Methods 2.1 One-Dimensional Unconstrained Minimization 2.1.1 Motivation Most practical optimization

More information

Introduction to Geometry Optimization. Computational Chemistry lab 2009

Introduction to Geometry Optimization. Computational Chemistry lab 2009 Introduction to Geometry Optimization Computational Chemistry lab 9 Determination of the molecule configuration H H Diatomic molecule determine the interatomic distance H O H Triatomic molecule determine

More information

Lecture 7: Minimization or maximization of functions (Recipes Chapter 10)

Lecture 7: Minimization or maximization of functions (Recipes Chapter 10) Lecture 7: Minimization or maximization of functions (Recipes Chapter 10) Actively studied subject for several reasons: Commonly encountered problem: e.g. Hamilton s and Lagrange s principles, economics

More information

Lecture 16: October 22

Lecture 16: October 22 0-725/36-725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:

More information

Numerical solutions of nonlinear systems of equations

Numerical solutions of nonlinear systems of equations Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points

More information

Introduction to unconstrained optimization - direct search methods

Introduction to unconstrained optimization - direct search methods Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the

More information

UNCONSTRAINED OPTIMIZATION

UNCONSTRAINED OPTIMIZATION UNCONSTRAINED OPTIMIZATION 6. MATHEMATICAL BASIS Given a function f : R n R, and x R n such that f(x ) < f(x) for all x R n then x is called a minimizer of f and f(x ) is the minimum(value) of f. We wish

More information

CLASS NOTES Computational Methods for Engineering Applications I Spring 2015

CLASS NOTES Computational Methods for Engineering Applications I Spring 2015 CLASS NOTES Computational Methods for Engineering Applications I Spring 2015 Petros Koumoutsakos Gerardo Tauriello (Last update: July 2, 2015) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material

More information

Optimization Methods: Optimization using Calculus - Equality constraints 1. Module 2 Lecture Notes 4

Optimization Methods: Optimization using Calculus - Equality constraints 1. Module 2 Lecture Notes 4 Optimization Methods: Optimization using Calculus - Equality constraints Module Lecture Notes 4 Optimization of Functions of Multiple Variables subect to Equality Constraints Introduction In the previous

More information

Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems

Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems International Journal of Scientific and Research Publications, Volume 3, Issue 10, October 013 1 ISSN 50-3153 Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming

More information

Gradient-Based Optimization

Gradient-Based Optimization Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems

More information

Introduction to gradient descent

Introduction to gradient descent 6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our

More information

Physics 403. Segev BenZvi. Numerical Methods, Maximum Likelihood, and Least Squares. Department of Physics and Astronomy University of Rochester

Physics 403. Segev BenZvi. Numerical Methods, Maximum Likelihood, and Least Squares. Department of Physics and Astronomy University of Rochester Physics 403 Numerical Methods, Maximum Likelihood, and Least Squares Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Review of Last Class Quadratic Approximation

More information

MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year

MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,

More information

UNCONSTRAINED OPTIMIZATION PAUL SCHRIMPF OCTOBER 24, 2013

UNCONSTRAINED OPTIMIZATION PAUL SCHRIMPF OCTOBER 24, 2013 PAUL SCHRIMPF OCTOBER 24, 213 UNIVERSITY OF BRITISH COLUMBIA ECONOMICS 26 Today s lecture is about unconstrained optimization. If you re following along in the syllabus, you ll notice that we ve skipped

More information

Lecture 18: November Review on Primal-dual interior-poit methods

Lecture 18: November Review on Primal-dual interior-poit methods 10-725/36-725: Convex Optimization Fall 2016 Lecturer: Lecturer: Javier Pena Lecture 18: November 2 Scribes: Scribes: Yizhu Lin, Pan Liu Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:

More information

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 3 Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 2 3.1. Gradient method Classical gradient method: to minimize a differentiable convex

More information

Vasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks

Vasil Khalidov & Miles Hansard. C.M. Bishop s PRML: Chapter 5; Neural Networks C.M. Bishop s PRML: Chapter 5; Neural Networks Introduction The aim is, as before, to find useful decompositions of the target variable; t(x) = y(x, w) + ɛ(x) (3.7) t(x n ) and x n are the observations,

More information

Convex Optimization CMU-10725

Convex Optimization CMU-10725 Convex Optimization CMU-10725 Quasi Newton Methods Barnabás Póczos & Ryan Tibshirani Quasi Newton Methods 2 Outline Modified Newton Method Rank one correction of the inverse Rank two correction of the

More information

Linear Discriminant Functions

Linear Discriminant Functions Linear Discriminant Functions Linear discriminant functions and decision surfaces Definition It is a function that is a linear combination of the components of g() = t + 0 () here is the eight vector and

More information

Statistical Geometry Processing Winter Semester 2011/2012

Statistical Geometry Processing Winter Semester 2011/2012 Statistical Geometry Processing Winter Semester 2011/2012 Linear Algebra, Function Spaces & Inverse Problems Vector and Function Spaces 3 Vectors vectors are arrows in space classically: 2 or 3 dim. Euclidian

More information

Order of convergence. MA3232 Numerical Analysis Week 3 Jack Carl Kiefer ( ) Question: How fast does x n

Order of convergence. MA3232 Numerical Analysis Week 3 Jack Carl Kiefer ( ) Question: How fast does x n Week 3 Jack Carl Kiefer (94-98) Jack Kiefer was an American statistician. Much of his research was on the optimal design of eperiments. However, he also made significant contributions to other areas of

More information

Deep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.

Deep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C. Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning

More information

Computer Problems for Taylor Series and Series Convergence

Computer Problems for Taylor Series and Series Convergence Computer Problems for Taylor Series and Series Convergence The two problems below are a set; the first should be done without a computer and the second is a computer-based follow up. 1. The drawing below

More information

AM 205: lecture 18. Last time: optimization methods Today: conditions for optimality

AM 205: lecture 18. Last time: optimization methods Today: conditions for optimality AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3

More information

Optimization Methods

Optimization Methods Optimization Methods Categorization of Optimization Problems Continuous Optimization Discrete Optimization Combinatorial Optimization Variational Optimization Common Optimization Concepts in Computer Vision

More information

Handling Nonpositive Curvature in a Limited Memory Steepest Descent Method

Handling Nonpositive Curvature in a Limited Memory Steepest Descent Method Handling Nonpositive Curvature in a Limited Memory Steepest Descent Method Fran E. Curtis and Wei Guo Department of Industrial and Systems Engineering, Lehigh University, USA COR@L Technical Report 14T-011-R1

More information