Unconstrained minimization: assumptions
|
|
- Theodora Williamson
- 5 years ago
- Views:
Transcription
1 Unconstrained minimization I terminology and assumptions I gradient descent method I steepest descent method I Newton s method I self-concordant functions I implementation IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 1 Unconstrained minimization: assumptions minimize f (x) I f convex, twice continuously di erentiable (hence dom f open) I we assume optimal value p? =inf x f (x) isattained(and finite) Unconstrained minimization methods I produce sequence of points x (k) 2 dom f, k =0, 1,... with f (x (k) )! p? (commonly, f (x (k) ) & p? ) I in practice, terminated in finite time, when f (x k ) p? apple, where > 0 is pre-specified I can be interpreted as iterative methods for solving optimality condition rf (x? )=0 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 2
2 Descent methods x (k+1) = x (k) + t (k) x (k) with f (x (k+1) ) < f (x (k) ) I other notation options: x + = x + t x, x := x + t x I x is the step, orsearch direction; t is the step size, orstep length I from convexity, f (x + ) < f (x) impliesrf (x) T x < 0 (i.e., x is a descent direction) General descent method. given a starting point x 2 dom f. repeat 1. Determine a descent direction x. 2. Line search. Choose a step size t > Update. x := x + t x. until stopping criterion is satisfied. IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 3 Further assumptions: Initial point and sublevel set Algorithms in this chapter require a starting point x (0) such that I x (0) 2 dom f I sublevel set S = {x f (x) apple f (x (0) )} is closed 2nd condition is hard to verify, except when all sublevel sets are closed: I equivalent to condition that epi f is closed I true if dom f = R n I true if f (x)!1as x! bd dom f examples of di erentiable functions with closed sublevel sets:! mx mx f (x) = log exp(ai T x + b i ), f (x) = log(b i ai T x) i=1 i=1 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 4
3 Strong convexity and implications f is strongly convex on S if there exists an m > 0 such that r 2 f (x) mi for all x 2 S Implications 1. for x, y 2 S, f (y) f (x)+rf(x) T (y x)+ m 2 kx yk Hence, S is bounded: take any y 2 S and x = x? 2 S in the above 3. p? > 1, andforx 2 S, f (x) p? apple 1 2m krf (x)k2 2 useful as stopping criterion (if you know m) 4. Also, 8x 2 S, kx x? k 2 apple 2 m krf (x)k 2, and hence opt. solution is unique. IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 5 Descent methods (reprise) x (k+1) = x (k) + t (k) x (k) with f (x (k+1) ) < f (x (k) ) I other notation options: x + = x + t x, x := x + t x I x is the step, orsearch direction; t is the step size, orstep length I from convexity, f (x + ) < f (x) impliesrf (x) T x < 0 (i.e., x is a descent direction) General descent method. given a starting point x 2 dom f. repeat 1. Determine a descent direction x. 2. Line search. Choose a step size t > Update. x := x + t x. until stopping criterion is satisfied. IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 6
4 Line search types Exact line search: t = argmin t>0 f (x + t x) Backtracking line search (with parameters 2 (0, 1/2), 2 (0, 1)) I starting at t = 1, repeat t := t until f (x + t x) < f (x)+ trf (x) T x I graphical interpretation: backtrack until t apple t 0 f(x + t x) f(x) + t f(x) T x f(x) + αt f(x) T x t t = 0 t 0 I after backtracking, stepsize t will satisfy t =1ort 2 ( t 0, t 0 ] IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 7 Gradient descent method general descent method with x = rf (x) given a starting point x 2 dom f. repeat 1. x := rf (x). 2. Line search. Choose step size t via exact or backtracking line search. 3. Update. x := x + t x. until stopping criterion is satisfied. I stopping criterion usually of the form krf (x)k 2 apple I convergence result: for strongly convex f, f (x (k) ) p? apple c k (f (x (0) ) p? ) c 2 (0, 1) depends on m, x (0),linesearchtype I will terminate in at most log((f (x 0 ) p? )/ ) iterations log(1/c) I very simple, but often very slow; rarely used in practice IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 8
5 Quadratic problem in R 2 f (x) =(1/2)(x x 2 2 ) ( > 0) with exact line search, starting at x (0) =(, 1): x (k) 1 = 1 k ( ), x (k) ( 1) 2 = k I very slow if 1or 1 I example for = 10: 4 x2 0 4 x 1 x (0) x (1) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 9 Nonquadratic example f (x 1, x 2 )=e x 1+3x e x 1 3x e x x (0) x (0) ts x (2) PSfrag replacements x (1) x (1) backtracking line search exact line search IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 10
6 A problem in R 100 f (x) =c T x X500 log(b i ai T x) i= f(x (k) ) p exact l.s. backtracking l.s k linear convergence, i.e., a straight line on a semilog plot IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 11 Complexity analysis of gradient descent Assumptions and observations I Assumptions: f 2 C 2 (D); S = {x : f (x) apple f (x 0 )} is closed; 9m > 0:r 2 f (x) mi 8x 2 S I Observations: I 8x, y 2 S, f (y) f (x)+rf(x) T (y x)+ m 2 kx yk2 2 I S is bounded I p? > 1, andforx 2 S, f (x) p? apple 1 2m krf (x)k2 2 I 9M m: r 2 f (x) MI 8x 2 S (since S is bdd). This implies: I 8x, y 2 S, f (y) apple f (x)+rf (x) T (y x)+ M 2 kx yk2 2 I Minimizing each side over y gives: 8x 2 S, p? apple f (x) 1 2M krf (x)k2 2 I Geometric implication: If S is a sub-level set of f with apple f (x 0 ), then cond(s ) apple M m, where cond( ) is the square of the ratio of maximum to minimum widths of a convex set IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 12
7 Complexity analysis of gradient descent Algorithm analysis Let x(t) =x trf (x). Then: I f (x(t)) apple f (x) tkrf (x)k Mt2 2 krf (x)k2 2 for any t I If exact line search is used: I f (x + ) apple min{f (x) tkrf (x)k Mt2 t 2 krf (x)k2 2} = 1 f (x) 2M krf 1 (x)k2 2 apple f (x) 2M 2m(f (x) p? ) I We conclude: f (x + ) p? apple (f (x) p? )(1 m/m) Linear convergence with c = 1 I If backtracking (BT) line search is used: I BT exit condition is guaranteed for t 2 [0, 1/M] (for < 1/2) I BT terminates either with t =1ort /M I Using these bounds on t and analysis similar to the above, get linear convergence with m M c =1 min{2m, 2 m/m} < 1. IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 13 Steepest descent method Normalized steepest descent direction (at x, for general norm k k): x nsd = argmin{rf (x) T v kvk =1} interpretation: for small v, f (x + v) f (x)+rf(x) T v; direction x nsd is unit-norm step with most negative directional derivative (Unnormalized) steepest descent direction x sd = krf (x)k x nsd satisfies rf (x) T x sd = krf (x)k 2 Steepest descent method I general descent method with x = x sd I convergence properties similar to gradient descent IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 14
8 Examples I Euclidean norm: x sd = rf (x) I quadratic norm kxk P =(x T Px) 1/2 (P 2 S n ++): x sd = P 1 rf (x) I `1-norm: x sd = (@f (x)/@x i )e i,wherei is chosen so (x)/@x i = krf (x)k 1 unit balls and normalized steepest descent directions for a quadratic norm and the `1-norm: f(x) x nsd x nsd f(x) s PSfrag replacements IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 15 Choice of norm for steepest descent ements x (0) x (1) x (2) PSfrag replacements x (0) x (2) x (1) I steepest descent with backtracking line search for two quadratic norms I ellipses show {x kx x (k) k P =1} I equivalent interpretation of steepest descent with quadratic norm k k P : gradient descent after change of variables x = P 1/2 x shows choice of P has strong e ect on speed of convergence IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 16
9 Newton step x nt = r 2 f (x) 1 rf (x) interpretations I x + x nt minimizes second order approximation bf (x + v) =f (x)+rf(x) T v v T r 2 f (x)v I x + x nt solves linearized optimality condition rf (x + v) r b f (x + v) =rf (x)+r 2 f (x)v =0 PSfrag replacements Sfrag replacements bf (x, f(x)) (x + x nt, f(x + x nt )) f 0 bf f (x + x nt, f (x + x nt )) (x, f (x)) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 17 Newton step steepest descent interpretation I x nt is steepest descent direction at x in local Hessian norm 1/2 kuk r 2 ( f (x) = u T r 2 ) f (x)u x s x + x nsd x + x nt dashed lines are contour lines of f ; ellipse is {x + v v T r 2 f (x)v =1} arrow shows rf (x) I Newton step is a ne invariant, i.e., independent of a ne change of coordinates IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 18
10 Newton decrement (x) = 1/2 rf (x) T r 2 f (x) 1 rf (x) a measure of the proximity of x to x? Properties I gives an estimate of f (x) p?, using quadratic approximation bf : f (x) inf bf (y) =f (x) f b (x + xnt )= 1 y 2 (x)2 I equal to the norm of the Newton step in the quadratic Hessian norm 1/2 (x) = xntr T 2 f (x) x nt I directional derivative in the Newton direction: rf (x) T x nt = (x) 2 I a ne invariant (unlike krf (x)k 2 ) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 19 Newton s method given a starting point x 2 dom f, tolerance > 0. repeat 1. Compute the Newton step and decrement. x nt := r 2 f (x) 1 rf (x); 2 := rf (x) T r 2 f (x) 1 rf (x). 2. Stopping criterion. quit if 2 /2 apple. 3. Line search. Choose step size t by backtracking line search. 4. Update. x := x + t x nt. A ne invariant, i.e., independent of linear changes of coordinates: Newton iterates for f (y) =f (By) with starting point y (0) = B 1 x (0) are y (k) = B 1 x (k) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 20
11 Classical convergence analysis assumptions I f strongly convex on S with constant m I r 2 f is Lipschitz continuous on S, with constant L > 0: kr 2 f (x) r 2 f (y)k 2 apple Lkx yk 2 (L measures how well f can be approximated by a quadratic function) outline: there exist constants 2 (0, m 2 /L), I if krf (x (k) )k 2,thenf (x (k+1) ) f (x (k) ) apple I if krf (x (k) )k 2 <, thent (k) = 1, and > 0 such that L L 2 2m 2 krf (x (k+1) )k 2 apple 2m 2 krf (x (k) )k 2 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 21 Two phases of Newton s method: apple 3(1 2 ) m2 L, = m 2 M 2 Damped Newton phase (krf (x)k 2 ) I most iterations require backtracking steps I function value decreases by at least I if p? > 1, this phase ends after at most (f (x (0) ) p? )/ iterations Quadratically convergent phase (krf (x)k 2 < ) I all iterations use step size t =1 I krf (x)k 2 converges to zero quadratically: if krf (x (k) )k 2 <, then L L 2 l 2m 2 krf (x l )k 2 apple 2m 2 krf (x k )k 2 k apple 1 2 l 2 k, l k IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 22
12 Conclusion: number of iterations until f (x) p? apple is bounded above by f (x (0) ) p? + log 2 log 2 ( 0 / ) I, 0 are constants that depend on m, L, x (0) I second term is small (of the order of 6) and almost constant for practical purposes I in practice, constants m, L (hence, 0 )areusuallyunknown I provides qualitative insight into convergence properties (i.e., explains two algorithm phases) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 23 Examples example in R ts x (0) x (1) f(x (k) ) p k I backtracking parameters =0.1, =0.7 I converges in only 5 steps I quadratic local convergence IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 24
13 Example in R f(x (k) ) p exact line search backtracking step size t (k) exact line search backtracking k k I backtracking parameters =0.01, =0.5 I backtracking line search almost as fast as exact l.s. (and much simpler) I clearly shows two phases in algorithm IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 25 Example in R (with sparse a i ) f (x) = X log(1 xi 2 ) i=1 cements 10 5 X log(b i ai T x) i=1 f(x (k) ) p k I backtracking parameters =0.01, =0.5. I performance similar to small examples IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 26
14 Self-concordance shortcomings of classical convergence analysis I depends on unknown constants (m, L,...) I bound is not a nely invariant, although Newton s method is convergence analysis via self-concordance (Nesterov and Nemirovski) I does not depend on any unknown constants I gives a ne-invariant bound I applies to special class of convex functions ( self-concordant functions) I developed to analyze polynomial-time interior-point methods for convex optimization IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 27 Self-concordant functions definition I Convex f : R! R is self-concordant if f 000 (x) apple 2f 00 (x) 3/2 for all x 2 dom f I Convex f : R n! R is self-concordant if g(t) =f (x + tv) is self-concordant for all x 2 dom f, v 2 R n examples on R I linear and quadratic functions I negative logarithm f (x) = log x I negative entropy plus negative logarithm: f (x) =x log x log x a ne invariance: if f : R! R is s.c., then f (y) =f (ay + b) is s.c.: f 000 (y) =a 3 f 000 (ay + b), f 00 (y) =a 2 f 00 (ay + b) be careful: if f is s.c., f may not be IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 28
15 Self-concordant calculus properties I preserved under positive scaling I preserved under composition with a 1, and sum ne function I if g is convex with dom g = R ++ and g 000 (x) apple 3g 00 (x)/x then f (x) = log( g(x)) log x is self-concordant examples: properties can be used to show that the following are s.c. I f (x) = P m i=1 log(b i ai T x) on {x ai T x < b i, i =1,...,m} I f (X )= log det X on S n ++ I f (x) = log(y 2 x T x) on {(x, y) kxk 2 < y} IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 29 Convergence analysis for self-concordant functions summary: there exist constants 2 (0, 1/4], > 0 such that I if (x) >, then f (x (k+1) ) f (x (k) ) apple I if (x) apple, then 2 (x (k+1) ) apple 2 2 (x (k) ) ( and only depend on backtracking parameters, ) complexity bound: number of Newton iterations bounded by f (x (0) ) p? + log 2 log 2 (1/ ) for =0.1, =0.8, = 10 10, bound evaluates to 375(f (x (0) ) p? )+6 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 30
16 Bounding second derivatives of a SC function I Let : R! R be a SCSC function I SC condition can be restated as 1 apple d 00 ( ) 1/2 apple dom (2) d I Assume [0, t] 2 dom get t apple Z t 0. Integrating (2) between 0 and t, we d d i.e., t apple 00 (t) 1/2 00 (0) 1/2 apple t I Rearranging: 00 (0) (1 + t 00 (0) 1/2 ) 2 apple 00 (t) apple 00 ( ) 1/2 d apple t, (3) 00 (0) (1 t 00 (0) 1/2 ) 2 (4) I the lower bound is valid for any 0 apple t 2 dom,andtheupper bound is valid for 0 apple t < 00 (0) 1/2 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 31 Preliminaries for analysis of the Newton s algorithm I Newton step: x nt = r 2 f (x) 1 rf (x) I Newton decrement: (x) =krf (x)k r 2 f (x) 1 I Can show: (x) =sup v6=0 v T rf (x) kvk r2 f (x) I Let v T rf (x) < 0 and define v (t) =f (x + tv) I 0 v (0) = rf (x) T v and 00 v (0) = v T r 2 f (x)v (5) I According to (5), (x) 0 v (0) 00 v (0) 1/2 (with equality when v = x nt ) I If v = x nt, the exit condition for BT line search with parameters < 1/2 and is (t) apple (0) + t 0 (0) = (0) t (x) 2 (6) I (we will drop the subscript of when v is the Newton direction) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 32
17 Termination criterion and bound on suboptimality I Integrate the lower bound in (4) as applied to v ( ) twice: v (t) v (0) + t 0 v (0) + t 00 v (0) 1/2 log(1 + t 00 v (0) 1/2 ) (7) I The right-hand side of (7) can be minimized analytically, so inf t 0 v (t) v (0) 0 v (0) 00 v (0) 1/2 +log(1+ v (0) 0 00 v (0) 1/2 ) I u + log(1 u) isdecreasinginu, so bound (8) further: (8) inf t 0 v (t) v (0) + (x) + log(1 (x)) (9) I If v is the descent direction that leads from x to x?, I Conclusion: p? v (0)+ (x)+log(1 (x)) f (x) (x) 2 if (x) apple 0.68 f (x) p? apple (x) 2 for (x) apple 0.68, (10) so termination criterion (x) 2 apple results in a guaranteed sub-optimality bound when applied to a SCSC function IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 33 One iteration of Newton s method on SC functions I From now on,v = x nt, and drop the subscript on (t) I We would like to show that there exist (problem-independent) parameters > 0and 2 (0, 1/4], such that I If (x) >, thenf (x + ) f (x) apple,and I If (x) apple, thent =1and2 (x + ) apple (2 (x)) 2 I Useful derivation step: Integrating the upper bound of (4) twice, we get: (t) apple (0) t (x) 2 t (x) log(1 t (x)) for t 2 [0, 1/ (x)) (11) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 34
18 One iteration of Newton s method on SC functions Analysis of the Damped Newton phase I Claim: ˆt = 1 1+ (x) satisfies the BT exit condition. Using (11): (ˆt) apple (0) (x) + log(1 + (x)) apple (0) (x) 2 /(2(1 + (x))) (true since (x) 0) apple (0) (x) 2 /(1 + (x)) (true since < 1/2) = (0) (x) 2ˆt I Hence, after BT, t 1+ (x) I Combining t so, set 1+ (x) with (6), we get: (t) (0) apple = 2 1+ (x) 2 1+ (x), (12) (13) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 35 One iteration of Newton s method on SC functions Analysis of the quadratically convergence phase I The following values only depend on BT settings: =(1 2 )/4 < 1/4 and = 2 1+ (14) I Returning to (11): I It implies that t =12 dom if (x) < 1 I Since in this phase (x) apple < (1 2 )/2, we have (1) apple (0) (x) 2 (x) log(1 (x)) apple (0) 1 2 (x)2 + (x) 3 (true since 0 apple (x) apple 0.81) apple (0) (x) 2, so t = 1 satisfies the BT exit condition. I Can show: (x + (x) ) apple 2 for x + = x + x (1 (x)) 2 nt and SCSC f I In particular, if (x) apple 1/4, we have (x + ) apple 2 (x) 2 IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 36
19 One iteration of Newton s method on SC functions Final complexity bound I Termination criterion: (x) 2 apple I The algorithm will perform as follows: I It will spend at most f (x (0) ) p? iterations in the damped Newton s phase. I Upon entering the quadratically convergent phase in iteration k, the Newton decrement at iteration l k will satisfy 2 (x (l) ) apple (2 (x (k) )) 2l k apple (2 ) 2l k apple 2 l 1 2 k I Thus, the termination criterion will be satisfied for any l such that 1 2 l k+1 apple, 2 from which we conclude that at most log 2 log 2 (1/ ) iterations will be needed in this second phase IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 37 Numerical example: 150 randomly generated instances of minimize f (x) = P m i=1 log(b i ai T x) : m = 100, n = 50 : m = 1000, n = 500 : m = 1000, n = 50 iterations f(x (0) ) p I number of iterations much smaller than 375(f (x (0) ) p? )+6 I bound of the form c(f (x (0) ) p? )+6withsmallerc (empirically) valid IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 38
20 Implementation main e ort in each iteration: evaluate derivatives and solve Newton system H x = g where H = r 2 f (x), g = rf (x) via Cholesky factorization H = LL T, x nt = L T L 1 g, (x) =kl 1 gk 2 I cost (1/3)n 3 flops for unstructured system I cost (1/3)n 3 if H sparse, banded IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 39 Example of dense Newton system with structure f (x) = nx i=1 i(x i )+ 0 (Ax + b), H = D + A T H 0 A I assume A 2 R p n,dense,withp n I D diagonal with diagonal elements 00 i (x i); H 0 = r 2 0 (Ax + b) method 1: formh, solve via dense Cholesky factorization: (cost (1/3)n 3 ) method 2: factorh 0 = L 0 L T 0 ; write Newton system as D x + A T L 0 w = g, L T 0 A x w =0 eliminate x from first equation; compute w and x from (I + L T 0 AD 1 A T L 0 )w = L T 0 AD 1 g, D x = g A T L 0 w cost: 2p 2 n (dominated by computation of L T 0 AD 1 AL 0 ) IOE 611: Nonlinear Programming, Fall Unconstrained minimization Page 7 40
10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationUnconstrained minimization
CSCI5254: Convex Optimization & Its Applications Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions 1 Unconstrained
More informationConvex Optimization. 9. Unconstrained minimization. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University
Convex Optimization 9. Unconstrained minimization Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2017 Autumn Semester SJTU Ying Cui 1 / 40 Outline Unconstrained minimization
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationConvex functions. Definition. f : R n! R is convex if dom f is a convex set and. f ( x +(1 )y) < f (x)+(1 )f (y) f ( x +(1 )y) apple f (x)+(1 )f (y)
Convex functions I basic properties and I operations that preserve convexity I quasiconvex functions I log-concave and log-convex functions IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationEquality constrained minimization
Chapter 10 Equality constrained minimization 10.1 Equality constrained minimization problems In this chapter we describe methods for solving a convex optimization problem with equality constraints, minimize
More information11. Equality constrained minimization
Convex Optimization Boyd & Vandenberghe 11. Equality constrained minimization equality constrained minimization eliminating equality constraints Newton s method with equality constraints infeasible start
More informationGradient Descent. Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh. Convex Optimization /36-725
Gradient Descent Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh Convex Optimization 10-725/36-725 Based on slides from Vandenberghe, Tibshirani Gradient Descent Consider unconstrained, smooth convex
More information5 Overview of algorithms for unconstrained optimization
IOE 59: NLP, Winter 22 c Marina A. Epelman 9 5 Overview of algorithms for unconstrained optimization 5. General optimization algorithm Recall: we are attempting to solve the problem (P) min f(x) s.t. x
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationLecture 14: Newton s Method
10-725/36-725: Conve Optimization Fall 2016 Lecturer: Javier Pena Lecture 14: Newton s ethod Scribes: Varun Joshi, Xuan Li Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes
More informationDescent methods. min x. f(x)
Gradient Descent Descent methods min x f(x) 5 / 34 Descent methods min x f(x) x k x k+1... x f(x ) = 0 5 / 34 Gradient methods Unconstrained optimization min f(x) x R n. 6 / 34 Gradient methods Unconstrained
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 14-1 Set
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More information6. Proximal gradient method
L. Vandenberghe EE236C (Spring 2016) 6. Proximal gradient method motivation proximal mapping proximal gradient method with fixed step size proximal gradient method with line search 6-1 Proximal mapping
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationProximal Newton Method. Ryan Tibshirani Convex Optimization /36-725
Proximal Newton Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: primal-dual interior-point method Given the problem min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h
More information1. Gradient method. gradient method, first-order methods. quadratic bounds on convex functions. analysis of gradient method
L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, first-order methods quadratic bounds on convex functions analysis of gradient method 1-1 Approximate course outline First-order
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationGradient Descent. Ryan Tibshirani Convex Optimization /36-725
Gradient Descent Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: canonical convex programs Linear program (LP): takes the form min x subject to c T x Gx h Ax = b Quadratic program (QP): like
More informationGRADIENT = STEEPEST DESCENT
GRADIENT METHODS GRADIENT = STEEPEST DESCENT Convex Function Iso-contours gradient 0.5 0.4 4 2 0 8 0.3 0.2 0. 0 0. negative gradient 6 0.2 4 0.3 2.5 0.5 0 0.5 0.5 0 0.5 0.4 0.5.5 0.5 0 0.5 GRADIENT DESCENT
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationBarrier Method. Javier Peña Convex Optimization /36-725
Barrier Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: Newton s method For root-finding F (x) = 0 x + = x F (x) 1 F (x) For optimization x f(x) x + = x 2 f(x) 1 f(x) Assume f strongly
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationConvex Optimization. Lecture 12 - Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12 - Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationProximal Newton Method. Zico Kolter (notes by Ryan Tibshirani) Convex Optimization
Proximal Newton Method Zico Kolter (notes by Ryan Tibshirani) Convex Optimization 10-725 Consider the problem Last time: quasi-newton methods min x f(x) with f convex, twice differentiable, dom(f) = R
More informationConvex Optimization. Prof. Nati Srebro. Lecture 12: Infeasible-Start Newton s Method Interior Point Methods
Convex Optimization Prof. Nati Srebro Lecture 12: Infeasible-Start Newton s Method Interior Point Methods Equality Constrained Optimization f 0 (x) s. t. A R p n, b R p Using access to: 2 nd order oracle
More informationGradient descent. Barnabas Poczos & Ryan Tibshirani Convex Optimization /36-725
Gradient descent Barnabas Poczos & Ryan Tibshirani Convex Optimization 10-725/36-725 1 Gradient descent First consider unconstrained minimization of f : R n R, convex and differentiable. We want to solve
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 3 Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 2 3.1. Gradient method Classical gradient method: to minimize a differentiable convex
More informationAnalytic Center Cutting-Plane Method
Analytic Center Cutting-Plane Method S. Boyd, L. Vandenberghe, and J. Skaf April 14, 2011 Contents 1 Analytic center cutting-plane method 2 2 Computing the analytic center 3 3 Pruning constraints 5 4 Lower
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More informationA : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution:
1-5: Least-squares I A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: f (x) =(Ax b) T (Ax b) =x T A T Ax 2b T Ax + b T b f (x) = 2A T Ax 2A T b = 0 Chih-Jen Lin (National
More informationSelf-Concordant Barrier Functions for Convex Optimization
Appendix F Self-Concordant Barrier Functions for Convex Optimization F.1 Introduction In this Appendix we present a framework for developing polynomial-time algorithms for the solution of convex optimization
More informationCSCI 1951-G Optimization Methods in Finance Part 09: Interior Point Methods
CSCI 1951-G Optimization Methods in Finance Part 09: Interior Point Methods March 23, 2018 1 / 35 This material is covered in S. Boyd, L. Vandenberge s book Convex Optimization https://web.stanford.edu/~boyd/cvxbook/.
More informationEE364a Homework 8 solutions
EE364a, Winter 2007-08 Prof. S. Boyd EE364a Homework 8 solutions 9.8 Steepest descent method in l -norm. Explain how to find a steepest descent direction in the l -norm, and give a simple interpretation.
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationLecture 5: September 12
10-725/36-725: Convex Optimization Fall 2015 Lecture 5: September 12 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Barun Patra and Tyler Vuong Note: LaTeX template courtesy of UC Berkeley EECS
More informationLecture 17: October 27
0-725/36-725: Convex Optimiation Fall 205 Lecturer: Ryan Tibshirani Lecture 7: October 27 Scribes: Brandon Amos, Gines Hidalgo Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationAn Optimal Affine Invariant Smooth Minimization Algorithm.
An Optimal Affine Invariant Smooth Minimization Algorithm. Alexandre d Aspremont, CNRS & École Polytechnique. Joint work with Martin Jaggi. Support from ERC SIPA. A. d Aspremont IWSL, Moscow, June 2013,
More informationLecture 14 Barrier method
L. Vandenberghe EE236A (Fall 2013-14) Lecture 14 Barrier method centering problem Newton decrement local convergence of Newton method short-step barrier method global convergence of Newton method predictor-corrector
More information2. Quasi-Newton methods
L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationA : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution:
1-5: Least-squares I A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: f (x) =(Ax b) T (Ax b) =x T A T Ax 2b T Ax + b T b f (x) = 2A T Ax 2A T b = 0 Chih-Jen Lin (National
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationDeterminant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University
Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationLecture 5: Gradient Descent. 5.1 Unconstrained minimization problems and Gradient descent
10-725/36-725: Convex Optimization Spring 2015 Lecturer: Ryan Tibshirani Lecture 5: Gradient Descent Scribes: Loc Do,2,3 Disclaimer: These notes have not been subjected to the usual scrutiny reserved for
More informationMore First-Order Optimization Algorithms
More First-Order Optimization Algorithms Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 3, 8, 3 The SDM
More information6. Proximal gradient method
L. Vandenberghe EE236C (Spring 2013-14) 6. Proximal gradient method motivation proximal mapping proximal gradient method with fixed step size proximal gradient method with line search 6-1 Proximal mapping
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationThis can be 2 lectures! still need: Examples: non-convex problems applications for matrix factorization
This can be 2 lectures! still need: Examples: non-convex problems applications for matrix factorization x = prox_f(x)+prox_{f^*}(x) use to get prox of norms! PROXIMAL METHODS WHY PROXIMAL METHODS Smooth
More informationOptimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationFast proximal gradient methods
L. Vandenberghe EE236C (Spring 2013-14) Fast proximal gradient methods fast proximal gradient method (FISTA) FISTA with line search FISTA as descent method Nesterov s second method 1 Fast (proximal) gradient
More information5. Subgradient method
L. Vandenberghe EE236C (Spring 2016) 5. Subgradient method subgradient method convergence analysis optimal step size when f is known alternating projections optimality 5-1 Subgradient method to minimize
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationLINEAR AND NONLINEAR PROGRAMMING
LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico
More informationProximal and First-Order Methods for Convex Optimization
Proximal and First-Order Methods for Convex Optimization John C Duchi Yoram Singer January, 03 Abstract We describe the proximal method for minimization of convex functions We review classical results,
More informationOptimization methods
Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,
More informationSupplement: Universal Self-Concordant Barrier Functions
IE 8534 1 Supplement: Universal Self-Concordant Barrier Functions IE 8534 2 Recall that a self-concordant barrier function for K is a barrier function satisfying 3 F (x)[h, h, h] 2( 2 F (x)[h, h]) 3/2,
More informationLinear Algebra, part 2 Eigenvalues, eigenvectors and least squares solutions
Linear Algebra, part 2 Eigenvalues, eigenvectors and least squares solutions Anna-Karin Tornberg Mathematical Models, Analysis and Simulation Fall semester, 2013 Main problem of linear algebra 2: Given
More informationA New Trust Region Algorithm Using Radial Basis Function Models
A New Trust Region Algorithm Using Radial Basis Function Models Seppo Pulkkinen University of Turku Department of Mathematics July 14, 2010 Outline 1 Introduction 2 Background Taylor series approximations
More informationApolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints
Apolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints Oliver Hinder, Yinyu Ye Department of Management Science and Engineering Stanford University June 28, 2018 The problem I Consider
More informationConvex Optimization Lecture 16
Convex Optimization Lecture 16 Today: Projected Gradient Descent Conditional Gradient Descent Stochastic Gradient Descent Random Coordinate Descent Recall: Gradient Descent (Steepest Descent w.r.t Euclidean
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationOptimization methods
Optimization methods Optimization-Based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_spring16 Carlos Fernandez-Granda /8/016 Introduction Aim: Overview of optimization methods that Tend to
More information10. Ellipsoid method
10. Ellipsoid method EE236C (Spring 2008-09) ellipsoid method convergence proof inequality constraints 10 1 Ellipsoid method history developed by Shor, Nemirovski, Yudin in 1970s used in 1979 by Khachian
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationHessian Riemannian Gradient Flows in Convex Programming
Hessian Riemannian Gradient Flows in Convex Programming Felipe Alvarez, Jérôme Bolte, Olivier Brahic INTERNATIONAL CONFERENCE ON MODELING AND OPTIMIZATION MODOPT 2004 Universidad de La Frontera, Temuco,
More informationInterior Point Methods for Mathematical Programming
Interior Point Methods for Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Florianópolis, Brazil EURO - 2013 Roma Our heroes Cauchy Newton Lagrange Early results Unconstrained
More informationPrimal-Dual Interior-Point Methods for Linear Programming based on Newton s Method
Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationAlgorithms for Nonsmooth Optimization
Algorithms for Nonsmooth Optimization Frank E. Curtis, Lehigh University presented at Center for Optimization and Statistical Learning, Northwestern University 2 March 2018 Algorithms for Nonsmooth Optimization
More informationFrank-Wolfe Method. Ryan Tibshirani Convex Optimization
Frank-Wolfe Method Ryan Tibshirani Convex Optimization 10-725 Last time: ADMM For the problem min x,z f(x) + g(z) subject to Ax + Bz = c we form augmented Lagrangian (scaled form): L ρ (x, z, w) = f(x)
More informationarxiv: v1 [math.oc] 1 Jul 2016
Convergence Rate of Frank-Wolfe for Non-Convex Objectives Simon Lacoste-Julien INRIA - SIERRA team ENS, Paris June 8, 016 Abstract arxiv:1607.00345v1 [math.oc] 1 Jul 016 We give a simple proof that the
More informationThe proximal mapping
The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More informationA function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3
Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all
More informationPart 2: Linesearch methods for unconstrained optimization. Nick Gould (RAL)
Part 2: Linesearch methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More information