Optimization Methods for Circuit Design
|
|
- Edgar Preston
- 6 years ago
- Views:
Transcription
1 Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Optimization Methods for Circuit Design Compendium H. Graeb
2 Version 2.8 (WS 12/13) Michael Zwerger Version (WS 08/09 - SS 12) Michael Eick Version (SS 07 - SS 08) Husni Habal Presentation follows: H. Graeb, Analog Design Centering and Sizing, Springer, R. Fletcher, Practical Methods of Optimization, John Wiley & Sons, 2nd Edition, Status: October 12, 2012 Copyright Optimization Methods for Circuit Design Compendium H. Graeb Technische Universität München Institute for Electronic Design Automation Arcisstr Munich, Germany graeb@tum.de Phone: All rights reserved.
3 Contents 1 Introduction Parameters, performance, simulation Performance specification Minimum, minimization Unconstrained optimization Constrained optimization Classification of optimization problems Classification of constrained optimization problems Structure of an iterative optimization process without constraints with constraints Trust-region approach Optimality conditions Optimality conditions unconstrained optimization Necessary first-order condition for a local minimum of an unconstrained optimization problem Necessary second-order condition for a local minimum of an unconstrained optimization problem Sufficient and necessary conditions for second-order derivative 2 f (x ) to be positive definite Optimality conditions constrained optimization Feasible descent direction r Necessary first-order conditions for a local minimum of a constrained optimization problem Necessary second-order condition for a local minimum of a constrained optimization problem Sensitivity of the optimum with regard to a change in an active constraint Optimization of Analog Circuits i
4 3 Unconstrained optimization Univariate unconstrained optimization, line search Wolfe-Powell conditions Backtracking line search Bracketing Sectioning Golden Sectioning Line search by quadratic model Unimodal function Multivariate unconstrained optimization without derivatives Coordinate search Polytope method (Nelder-Mead simplex method) Multivariate unconstrained optimization with derivatives Steepest descent Newton approach Quasi-Newton approach Levenberg-Marquardt approach (Newton direction plus trust region) Least-squares (plus trust-region) approach Conjugate-gradient (CG) approach Constrained optimization problem formulations Quadratic Programming (QP) QP linear equality constraints QP - inequality constraints Example Sequential Quadratic programming (SQP), Lagrange Newton SQP equality constraints Penalty function Optimization of Analog Circuits ii
5 5 Statistical parameter tolerances Univariate Gaussian distribution (normal distribution) Multivariate normal distribution Transformation of statistical distributions Example Expectation values and their estimators Expectation values Definitions Linear transformation of expectation value Linear transformation of variance Translation law of variances Normalizing a random variable Linear transformation of a normal distribution Estimation of expectation values Expectation value estimator Variance estimator Variance of the expectation value estimator Linear transformation of estimated expectation value Linear transformation of estimated variance Translation law of estimated variance Optimization of Analog Circuits iii
6 7 Worst-case analysis Task Typical tolerance regions Classical worst-case analysis Task Linear performance model Peformance type f good, specification type f > f L Performance type f good, specification type f < f U Realistic worst-case analysis Task Peformance type f good, specification type f f L : Performance type f good, specification type f f U General worst-case analysis Task Peformance type f good, specification type f f L : Performance type f good, specification type f f U : General worst-case analysis with tolerance box Input/output of worst-case analysis Summary of discussed worst-case analysis problems Optimization of Analog Circuits iv
7 8 Yield analysis Task Acceptance function Parametric yield Statistical yield analysis/monte-carlo analysis Variance of yield estimator Estimated variance of yield estimator Importance sampling Geometric yield analysis for linearized performance feature ( realistic geometric yield analysis ) Yield partition Defining worst-case distance β W L as difference from nominal performance to specification bound as multiple of standard deviation σ f of linearized performance feature (β W L -sigma design) Yield partition as a function of worst-case distance β W L Worst-case distance β W L defines tolerance region specification type f f U Geometric yield analysis for nonlinear performance feature ( general geometric yield analysis ) Problem formulation Advantages of geometric yield analysis Lagrange function and first-order optimality conditions of problem (336) Lagrange function of problem (337) Second-order optimality condition of problem (336) Worst-case distance Remarks Overall yield Consideration of range parameters Optimization of Analog Circuits v
8 9 Yield optimization/design centering/nominal design Optimization objectives Derivatives of optimization objectives Problem formulations of analog optimization Analysis, synthesis Sizing Nominal design, tolerance design Optimization without/with constraints Sizing rules for analog circuit optimization Single (NMOS) transistor Sizing rules for single transistor that acts as a voltage-controlled current source (VCCS) Transistor pair: current mirror (NMOS) Sizing rules for current mirror A Matrix and vector notations 113 A.1 Vector A.2 Matrix A.3 Addition A.4 Multiplication A.5 Special cases A.6 Determinant of a quadratic matrix A.7 Inverse of a quadratic non-singular matrix A.8 Some properties B Abbreviated notations of derivatives using the nabla symbol 119 C Norms 121 Optimization of Analog Circuits vi
9 D Pseudo-inverse, singular value decomposition (SVD) 123 D.1 Moore-Penrose conditions D.2 Singular value decomposition E Linear equation system, rectangular system matrix with full rank 125 E.1 underdetermined system of equations E.2 overdetermined system of equations E.3 determined system of equations F Partial derivatives of linear, quadratic terms in matrix/vector notation129 G Probability space 131 H Convexity 133 H.1 Convex set K R n H.2 Convex function Optimization of Analog Circuits vii
10 Optimization of Analog Circuits viii
11 1 Introduction 1.1 Parameters, performance, simulation design parameters x d R n xd e.g. transistor widths, capacitances statistical parameters x s R nxs e.g. oxide thickness, threshold voltage range parameters x r R nxr e.g. operational parameters: (circuit) parameters x = [ x T d xt s x T r ] T supply voltage, temperature performance feature f i e.g. gain, bandwidth, slew rate, phase margin, delay, power (circuit) performance f = [ f i ] T R n f (circuit) simulation x f(x) e.g. SPICE abstraction from physical level! A design parameter and a statistical parameter may refer to the same physical parameter. E.g., an actual CMOS transistor width is the sum of a design parameter W k and a statistical parameter W. W k is the specific width of transistor T k while W is a width reduction that varies globally and equally for all the transistors on a die. A design parameter and a statistical parameter may be identical. 1.2 Performance specification performance specification feature (upper or lower limit on a performance): f i f L,i or f i f U,i (1) number of performance specification features: n f n P SF 2n f (2) performance specification: f L,1 f 1 (x) f U,1. f L,nf f nf (x) f U,nf f L f(x) f U (3) Optimization of Analog Circuits 1
12 f (a) strong local minimum weak local minimum global minimum x f (b) x Figure 1. Smooth function (a), i.e. continuous and differentiable at least several times on a closed region of the domain. Non-smooth continuous function (b). 1.3 Minimum, minimization without loss of generality: optimum minimum because: max f min f (4) { minimum, i.e., a result min minimize, i.e., a process (5) min f(x) f (x)! min (6) min f(x) x, f(x ) = f (7) 1.4 Unconstrained optimization f = min f(x) min f min f(x) x x min{f(x)} x = argmin f(x) argmin f argmin f(x) x x argmin{f(x)} (8) Optimization of Analog Circuits 2
13 1.5 Constrained optimization E: set of equality constraints I: set of inequality constraints min f(x) s.t. c i (x) = 0, i E c i (x) 0, i I (9) Alternative formulations where, min x f s.t. x Ω (10) min f (11) x Ω min {f(x) x Ω} (12) { c i (x) = 0, i E Ω = x c i (x) 0, i I } Lagrange function combines objective function and constraints in a single expression L(x, λ) = f(x) λ i c i (x) (13) i E I λ i : Lagrange multiplier associated with constraint i Optimization of Analog Circuits 3
14 1.6 Classification of optimization problems deterministic, stochastic continuous, discrete local, global scalar, vector constrained, unconstrained with or without derivatives The iterative search process is deterministic or random. Optimization variables can take an infinite number of values, e.g., the set of real numbers, or take a finite set of values or states. The objective value at a local optimal point is better than the objective values of other points in its vicinity. The objective value at a global optimal point is better than the objective values of any other point. In a vector optimization problem, multiple objective functions shall be optimized simultaneously (multiplecriteria optimization, MCO). Usually, objectives have to be traded off with each other. A Pareto-optimal point is characterized in that one objective can only be improved at the cost of another. Pareto optimization determines the set of all Pareto-optimal points. Scalar optimization refers to a single objective. A vector optimization problem is scalarized by combining the multiple objectives into a single overall objective, e.g., by a weighted sum, least-squares, or min/max. Besides the objective function that has to be optimized, constraints on the optimization variables may be given as inequalities or equalities. The optimization process may be based on gradients (first derivative) or on gradients and Hessians (second derivative), or it may not require any derivatives of the objective/constraint functions. 1.7 Classification of constrained optimization problems objective function constraint functions linear linear linear programming quadratic linear quadratic programming nonlinear nonlinear nonlinear programming convex linear equality convex programming constraints (local global minimum) concave inequality constraints Optimization of Analog Circuits 4
15 1.8 Structure of an iterative optimization process without constraints Taylor series of a function f about iteration point x (κ) : f(x) = f(x (κ) ) + f ( x (κ)) T ( ) x x (κ) ( x x (κ) ) T 2 f ( x (κ)) (x x (κ)) +... (14) = f (κ) + g (κ)t (x x (κ)) ( x x (κ) ) T H (κ) (x x (κ)) +... (15) f (κ) : value of function f at point x (κ) g (κ) : gradient (first derivative, direction of steepest ascent) at point x (κ) H (κ) : Hessian matrix (second derivative) at point x (κ) Taylor series about search direction r starting from point x (κ) : x(r) = x (κ) + r (16) f(x (κ) + r) f(r) = f (κ) + g (κ)t r rt H (κ) r +... (17) Taylor series about step length along search direction r (κ) starting from point x (κ) : x(α) = x (κ) + α r (κ) (18) f(x (κ) + α r (κ) ) f(α) = f (κ) + g (κ)t r (κ) α r(κ)t H (κ) r (κ) α (19) = f (κ) + f (α = 0) α f (α = 0) α (20) f (α = 0) : slope of f along direction r (κ) 2 f (α = 0) : curvature of f along r (κ) repeat determine the search direction r (κ) determine the step length α (κ) (line search) x (κ+1) = x (κ) + α (κ) r (κ) κ:= κ + 1 until termination criteria are fulfilled Steepest-descent approach search direction: direction of steepest descent, i.e., r (κ) = g (κ) Optimization of Analog Circuits 5
16 x 0 x Figure 2. Visual illustration of the steepest-descent approach for Rosenbrock s function f (x 1, x 2 ) = 100(x 2 x 2 1) 2 + (1 x 1 ) 2. A backtracking line search is applied (see Sec , page 19) with an initial x (0) = [ 1.0, 0.8] T and α (0) = 1, α := c 3 α. The search terminates when the Armijo condition is satisfied with c 1 = 0.7, c 3 = 0.6. Optimization of Analog Circuits 6
17 with constraints Constraint functions and objective functions are combined in an unconstrained optimization problem in each iteration step Lagrange formulation penalty function Sequential Quadratic Programming (SQP) Projection on active constraints, i.e. into subspace of an unconstrained optimization problem in each iteration step active-set methods Trust-region approach model of the objective function: f(x) m ( x (κ) + r ) (21) min r m ( x (κ) + r ) s.t. r trust region (22) e.g., r < search direction and step length are computed simultaneously trust region to consider the model accuracy Optimization of Analog Circuits 7
18 Optimization of Analog Circuits 8
19 2 Optimality conditions 2.1 Optimality conditions unconstrained optimization Taylor series of the objective function around the optimum point x : f(x) = f(x ) + f (x ) T (x x ) }{{}}{{} f For x = x + r close to the optimum: g T (x x ) T 2 f (x ) }{{} (x x ) +... (23) H f : value of the function at the optimum x g : gradient at the optimum x H : Hessian matrix at the optimum x f(r) = f + g T r rt H r +... (24) x is optimal there is no descent direction, r, such that f (r) < f. x 2 f ( x (κ)) gradient (κ) x direction steepest descent direction f (x) > f ( x (κ)) f (x) = f ( x (κ)) f (x) < f ( x (κ)) x 1 Figure 3. Descent directions from x (κ) : shaded area. Optimization of Analog Circuits 9
20 2.1.1 Necessary first-order condition for a local minimum of an unconstrained optimization problem x : stationary point descent direction r: f ( x (κ)) T r < 0 steepest descent direction: r = f ( x (κ)) r 0 g T r 0 (25) g = f (x ) = 0 (26) Figure 4. Quadratic functions: (a) minimum at x, (b) maximum at x, (c) saddle point at x, (d) positive semidefinite with multiple minima along trench. Optimization of Analog Circuits 10
21 2.1.2 Necessary second-order condition for a local minimum of an unconstrained optimization problem r 0 rt 2 f (x ) r 0 2 f (x ) is positive semidefinite f has non-negative curvature (27) sufficient: r 0 rt 2 f (x ) r > 0 2 f (x ) is positive definite has positive curvature (28) Figure 5. Contour plots of quadratic functions that are (a),(b) positive or negative definite, (c) indefinite (saddle point), (d) positive or negative semidefinite. Optimization of Analog Circuits 11
22 2.1.3 Sufficient and necessary conditions for second-order derivative 2 f (x ) to be positive definite all eigenvalues > 0 has a Cholesky decomposition: f 2 (x ) = L L T with l ii > 0 f 2 (x ) = L D L T with l ii = 1 and d ii > 0 (29) all pivot elements during gaussian elimination without pivoting > 0 all principal minors are > 0 x 2 unconstrained descent unconstrained c i = const f ( x (κ)) c i ( x (κ) ) x (κ) descent f = const x 1 Figure 6. Dark shaded area: unconstrained directions according to (35), light shaded area: descent directions according to (34), overlap: unconstrained descent directions. When no direction satisfies both (34) and (35) then the cross section is empty and the current point is a local minimum of the function. Optimization of Analog Circuits 12
23 2.2 Optimality conditions constrained optimization Feasible descent direction r descent direction: f ( x (κ)) T r < 0 (30) feasible direction: ( c i x (κ) + r ) ( c ) ( i x (κ) + c ) i x (κ) T r 0 (31) Inactive constraint: i is inactive ( c ) i x (κ) > 0 then each r with r < ɛ satisfies (31), e.g., ( c ) i x (κ) r = c i (x (κ) ) f (x (κ) ) f ( x (κ)) (32) (32) in (31) gives: where, ( c ) [ i x (κ) 1 c ( ) i x (κ) T ( ) ] f x (κ) 0 (33) c i (x (κ) ) f (x (κ) ) 1 c ( ) i x (κ) T ( ) f x (κ) c i (x (κ) ) f (x (κ) ) 1 Active constraint (Fig. 6): i is active c i ( x (κ) ) = 0 then (30) and (31) become: f ( x (κ)) T r < 0 (34) c i ( x (κ) ) T r 0 (35) no feasible descent direction exists: no vector r satisfies both (34) and (35) at x : f (x ) = λ i c i (x ) with λ i 0 (36) no statement about sign of λ i in case of an equality constraint (c i = 0 c i 0 c i 0) Optimization of Analog Circuits 13
24 2.2.2 Necessary first-order conditions for a local minimum of a constrained optimization problem x, f = f (x ), λ, L = L (x, λ ) Karush-Kuhn-Tucker (KKT) conditions L (x ) = 0 (37) c i (x ) = 0 i E (38) c i (x ) 0 i I (39) λ i 0 i I (40) λ i c i (x ) = 0 i E I (41) (37) is analogous to (26) (13) and (37) give: f (x ) λ i c i (x ) = 0 (42) i A(x ) A (x ) is the set of active constraints at x A (x ) = E {i I c i (x ) = 0} (43) (41) is called the complementarity condition, Lagrange multiplier is 0 (inactive constraint) or constraint c i (x ) is 0 (active constraint). from (41) and (13): L = f (44) Optimization of Analog Circuits 14
25 2.2.3 Necessary second-order condition for a local minimum of a constrained optimization problem f (x + r) = L (x + r, λ ) (45) = L (x, λ ) + r T L (x ) + 1 }{{} 2 rt 2 L (x ) r + (46) }{{} 0. =. f + 1 {}}{ 2 rt [ 2 f (x ) λ i 2 c i (x )] r + (47) for each feasible stationary direction r at x, i.e., i A(x ) F r = r r 0 c i (x ) T r 0 i A (x ) \A + c i (x ) T r = 0 i A + = { j A (x ) j E λ j > 0 } (48) necessary: sufficient: r T 2 L (x ) r 0 (49) r F r r T 2 L (x ) r > 0 (50) r F r Optimization of Analog Circuits 15
26 2.2.4 Sensitivity of the optimum with regard to a change in an active constraint perturbation of an active constraint at x by i 0 c i (x) 0 c i (x) i (51) L (x, λ, ) = f (x) i λ i (c i (x) i ) (52) f ( i ) = L ( i ) ( L = x T x + L i λ T λ + L ) i T i x,λ }{{}}{{} 0 T 0 T = L i = λ i x,λ f ( i ) = λ i (53) Lagrange multiplier: sensitivity to change in an active constraint close to x Optimization of Analog Circuits 16
27 3 Unconstrained optimization 3.1 Univariate unconstrained optimization, line search single optimization parameter, e.g., step length α f(α) f ( x (κ) + α r ) = f ( x (κ)) ( + f ) x (κ) T r α }{{}}{{} f (κ) f(α=0) rt 2 f ( x (κ)) r α 2 + (54) }{{} 2 f(α=0) error vector: global convergence: ɛ (κ) = x (κ) x (55) lim κ ɛ(κ) = 0 (56) convergence rate: ɛ (κ+1) = L ɛ (κ) p (57) p = 1: linear convergence, L < 1 p = 2: quadratic convergence exact line search is expensive, therefore only find α to: obtain sufficient reduction in f obtain a big enough step length Optimization of Analog Circuits 17
28 3.1.1 Wolfe-Powell conditions f (α = 0) c 2 f (α = 0) c 1 f (α = 0) f ( x (κ)) f f (α) 0 α opt (58) satisfied α (59) satisfied (60) satisfied Figure 7. Armijo condition: sufficient objective reduction f(α) f(0) + α c 1 f ( x (κ)) T r }{{} f(α=0) (58) curvature condition: sufficient gradient increase f ( x (κ) + α r ) T r }{{} f(α) c 2 f ( x (κ)) T r }{{} f(α=0) (59) strong curvature condition: step close to the optimum f (α) c 2 f (α = 0) (60) 0 < c 1 < c 2 < 1 (61) Optimization of Analog Circuits 18
29 3.1.2 Backtracking line search 0 < c 3 < 1 determine step length α t /* big enough, not too big */ WHILE α t violates (58) [α t := c 3 α t Optimization of Analog Circuits 19
30 3.1.3 Bracketing finding an interval [α lo, α hi ] that contains minimum /* take α lo as previous α hi and find a new, larger α hi until the minimum has passed by; if necessary exchange α lo, α hi such that f (α lo ) < f (α hi ) */ α hi := 0 REPEAT α lo := α hi determine α hi [α lo, α max ] IF α hi violates (58) OR f (α hi ) f (α lo ) /* minimum has been significantly passed, because the Armijo condition is violated or because the new objective value is larger than that at the lower border of the bracketing interval (Fig. 8) */ GOTO sectioning IF α hi satisfies (60) /* strong curvature condition satisfied and objective value smaller than at α lo, i.e., step length found (Fig. 9) */ α = α hi STOP IF f (α hi ) 0 /* α hi has lower objective value than α lo, and has passed the minimum as the objective gradient has switched sign (Fig. 10) */ exchange α hi and α lo GOTO sectioning Optimization of Analog Circuits 20
31 f α lo α hi is somewhere here: α Figure 8. f α hi is somewhere here: α lo α Figure 9. f α lo α hi is somewhere here: α Figure 10. Optimization of Analog Circuits 21
32 3.1.4 Sectioning sectioning starts with interval [α lo, α hi ] with the following properties minimum is included f (α lo ) < f (α hi ) α lo > <αhi f (α lo ) (α hi α lo ) < 0 REPEAT determine α t [α lo, α hi ] IF α t violates (58) OR f (α t ) f (α lo ) [ /* new αhi found, α lo remains (Fig. 11) */ α hi := α t ELSE /* i.e., f (α t ) f (α lo ), new α lo found */ IF α t satisfies (60) /* step length found */ α = α t STOP IF f (α t ) (α hi α lo ) 0 /* α t is on the same side from the minimum as α hi, then α lo must become the new α hi (Fig. 12) */ α hi := α lo /* Fig. 13 */ α lo := α t Optimization of Analog Circuits 22
33 f f α hi α t α lo α α lo α t α hi α new interval new interval α lo α hi Figure 11. f f α t α hi α lo α α lo α t α hi α new interval new interval α lo α hi α hi α lo Figure 12. f f α hi α t α lo α α lo α t α hi α new interval new interval α hi α lo α lo α hi Figure 13. Optimization of Analog Circuits 23
34 3.1.5 Golden Sectioning golden section ratio: 1 τ τ = τ 1 τ 2 + τ 1 = 0 : τ = (62) /* left inner point */ α 1 := L + (1 τ) R L /* right inner point */ α 2 := L + τ R L REPEAT IF f (α 1 ) < f (α 2 ) /* α [L, α 2 ], α 1 is new right inner point */ R := α 2 α 2 := α 1 α 1 := L + (1 τ) R L ELSE /* α [α 1, R], α 2 is new left inner point */ L := α 1 α 1 := α 2 α 2 := L + τ R L UNTIL < min Optimization of Analog Circuits 24
35 f L α 1 α 2 R α (1 τ) τ τ (1 τ) (2) L (2) α (2) α (2) 1 2 R (2) (1 τ) (2) τ (2) τ (2) (1 τ) (2) Figure 14. Accuracy after (κ)-th step interval size: (κ) = τ κ (0) (63) required number of iterations κ to reduce interval from (0) to (κ) : κ = 1 (0) log 4.78 log (0) log τ (κ) (κ) (64) Optimization of Analog Circuits 25
36 3.1.6 Line search by quadratic model m, f f ( f ( L (κ)) R (κ)) f ( E (κ)) m(α) f (α t ) L (κ) E (κ) α t R (κ) α Figure 15. quadratic model by, e.g., 3 sampling points first and second-order derivatives univariate Newton approach first-order condition: m (α) = f 0 + g α h α2 (65) min m (α) m (α) = g + h α t! = 0 α t = g/h (66) second-order condition: h > 0 (67) Optimization of Analog Circuits 26
37 3.1.7 Unimodal function f is unimodal over interval [L, R]: there exists exactly one value α opt [L, R] for which holds: and 2 < α opt f (α 1 ) > f (α 2 ) L<α 1 <α 2 <R (Fig. 16 (a)) (68) 1 > α opt f (α 1 ) < f (α 2 ) L<α 1 <α 2 <R (Fig. 16 (b)) (69) f α opt L α 1 α 2 α 1 α 2 (a) (b) R α Figure 16. Optimization of Analog Circuits 27
38 f (a) f (b) x 0 x α opt α Figure 17. Univariate unimodal function (a). Univariate monotone function (b). Remarks minimization of univariate unimodal function: interval reduction with f (L) > f (α t ) < f (R), α t [L, R] 3 sampling points root finding of univariate monotone function: interval reduction with sign (f (L)) = sign (f (R)), [L, R] two sampling points (κ) κ = 1 (0) (70) 2 κ = 1 (0) log 3.32 log (0) (71) log 2 (κ) (κ) Optimization of Analog Circuits 28
39 3.2 Multivariate unconstrained optimization without derivatives Coordinate search optimize one parameter at a time, alternating coordinates e j = [ ] T j-th position REPEAT FOR j = 1,, n x α t := arg minf (x + α e j ) α x := x + α t e j UNTIL convergence or maximum allowed iterations reached good for losely coupled, uncorrelated parameters, e.g., 2 f (x) diagonal matrix: x 2 x x f = const 1-st run of REPEAT loop x 1 Figure 18. Coordinate search with exact line search for quadratic objective function with diagonal Hessian matrix. Optimization of Analog Circuits 29
40 (a) 1st run of REPEAT loop 2nd run of REPEAT loop x (b) x Figure 19. Coordinate search with exact line search for quadratic objective function (a). Steepest-descent method for quadratic objective function (b). Optimization of Analog Circuits 30
41 3.2.2 Polytope method (Nelder-Mead simplex method) x 2 f = const x 3 x 2 f x 1 x 1 Figure 20. n x -simplex is the convex hull of a set of parameter vector vertices x i, i = 1,, n x + 1 the vertices are ordered such that f i = f (x i ), f 1 f 2 f nx+1 Cases of an iteration step reflection f r < f 1 f 1 f r < f nx f nx f r < f nx+1 f nx+1 f r expansion outer contraction inner contraction f e < f r? insert x r f c f r? f cc < f nx+1? yes no delete x n+1 yes no no yes x n+1 = x e x n+1 = x r reorder x n+1 = x c reduction x n+1 = x cc x 1, v 2,..., v nx+1 Optimization of Analog Circuits 31
42 Reflection x 2 x 3 x 0 x r x 1 Figure 21. x 0 = 1 n x x i (72) n x x r = x 0 + ρ (x 0 x nx+1) (73) ρ: reflection coefficient ρ > 0 (default: ρ = 1) i=1 Expansion x 2 x 3 x 0 x r x e x 1 Figure 22. χ: expansion coefficient, χ > 1, χ > ρ (default: χ = 2) x e = x 0 + χ (x r x 0 ) (74) Optimization of Analog Circuits 32
43 Outer contraction x 2 x c x r x 3 x 0 x 1 Figure 23. γ: contraction coefficient, 0 < γ < 1 (default: γ = 1 2 ) x c = x 0 + γ (x r x 0 ) (75) Inner contraction x 2 x cc x 0 x 3 x 1 Figure 24. x cc = x 0 γ (x 0 x nx+1) (76) Reduction (shrink) x 2 x 3 v 3 x 1 v 2 Figure 25. v i = x 1 + σ (x i x 1 ), i = 2,, n x + 1 (77) σ: reduction coefficient, 0 < σ < 1 (default: σ = 1 2 ) Optimization of Analog Circuits 33
44 3.3 Multivariate unconstrained optimization with derivatives Steepest descent (sec , page 5) search direction: negative of the gradient at the current point in the parameter space (i.e., linear model of the objective function) Newton approach quadratic model of objective function f ( x (κ) + r ) m ( x (κ) + r ) = f (κ) + g (κ)t r rt H (κ) r (78) minimize m (r) g (κ) = f ( x (κ)) H (κ) = 2 f ( x (κ)) (79) First-order optimality condition m (r) = 0 H (κ) r = g (κ) r (κ) (80) r (κ) : search direction for line seach r (κ) is obtained through the solution of a system of linear equations Second-order optimality condition 2 m (r) positive (semi)definite if not: steepest descent: r (κ) = g (κ) switch signs of negative eigenvalues Levenberg-Marquardt approach: ( H (κ) + λ I ) r = g (κ) (λ : steepest-descent approach) Optimization of Analog Circuits 34
45 3.3.3 Quasi-Newton approach second derivative not available successive approximation B H from gradients in the course of the optimization process, e.g., B (0) = I f ( x (κ+1)) = f ( x (κ) + α (κ) r (κ)) = g (κ+1) (81) approximate g (κ+1) from quadratic model (78) Quasi-Newton condition for approximation of B: g (κ+1) g (κ) + H (κ) α (κ) r (κ) (82) g (κ+1) g (κ) }{{} y (κ) α (κ) r (κ) = x (κ+1) x (κ) (83) = B (κ+1) (x } (κ+1) {{ x (κ) } ) (84) s (κ) Optimization of Analog Circuits 35
46 Symmetric rank-1 update (SR1) approach: B (κ+1) = B (κ) + u v T (85) substituting (85) in (84) ( B (κ) + u v ) T s (κ) = y (κ) substituting (86) in (85) B (κ+1) = B (κ) + because of the symmetry in B: B (κ+1) = B (κ) + u v T s (κ) = y (κ) B (κ) s (κ) u = y(κ) B (κ) s (κ) v T s (κ) (86) ( y (κ) B (κ) s (κ)) v T v T s (κ) (87) ( y (κ) B (κ) s (κ)) (y (κ) B (κ) s (κ)) T (y (κ) B (κ) s (κ) ) T s (κ) (88) if y (κ) B (κ) s (κ) = 0, then B (κ+1) = B (κ) SR1 does not guarantee positive definiteness of B alternatives: Davidon-Fletcher-Powell (DFP), Broyden-Fletcher-Goldfarb-Shannon (BFGS) approximation of B 1 instead of B or of a decomposition of the system matrix of linear equation system (80) to compute the search direction r (κ) s (κ), κ = 1,, n x linear independent and f quadratic with Hessian H: Quasi-Newton with SR1 terminates after not more than n x + 1 steps with B (κ+1) = H Optimization of Analog Circuits 36
47 3.3.4 Levenberg-Marquardt approach (Newton direction plus trust region) min m (r) s.t. r 2 (89) }{{} r T r constraint: trust region of model m (r), e.g., concerning positive definitiveness of B (κ), H (κ) corresponding Lagrange function: L (r, λ) = f (κ) + g (κ)t r rt H (κ) r λ ( r T r ) (90) stationary point L (r) = 0 : ( H (κ) + 2 λ I ) r = g (κ) (91) refer to the Newton approach, H (κ) indefinite Optimization of Analog Circuits 37
48 3.3.5 Least-squares (plus trust-region) approach min r f (r) f target 2 s.t. r 2 (92) linear model : trust region of linear model f (r) f (κ) (r) = f ( x (κ)) + f (x ) (κ)t r (93) }{{}}{{} f (κ) 0 S (κ) least-squares difference of linearized objective value to target value (index (κ) left out): ɛ(r) 2 = f (r) f target 2 (94) = ɛ T (r) ɛ(r) = (f 0 f target }{{} ɛ 0 +S r) T (ɛ 0 + S r) (95) = ɛ T 0 ɛ r T S T ɛ 0 + r T S T S r (96) S T S: part of second derivative of ɛ(r) 2 min r ɛ(r) 2 s.t. r 2 (97) corresponding Lagrange function: L (r, λ) = ɛ T 0 ɛ r T S T ɛ 0 + r T S T S r λ ( r T r ) (98) stationary point L (r) = 0: 2 S T ɛ S T S r + 2 λ r = 0 (99) ( ST S + λ I ) r = S T ɛ 0 (100) λ = 0: Gauss-Newton method (100) yields r (κ) for a given λ determine the Pareto front ɛ (κ) ( r (κ)) vs. r (κ) for 0 λ < (Figs. 26, 27) select r (κ) with small step length and small error (iterative process): x (κ+1) = x (κ) + r (κ) Optimization of Analog Circuits 38
49 ɛ (κ) ( r (κ)) λ λ small error small step length ɛ(r (κ) ) λ = 0 r (κ) = Figure 26. Characteristic boundary curve, typical sharp bend for ill-conditioned problems. Small step length wanted, e.g., due to limited trust region of linear model. x 2 x (κ) 2 ɛ (κ) (r) = const λ = 0 r (κ) ( ) λ r = x 1 x (κ) 1 Figure 27. Characteristic boundary curve of a two-dimensional parameter space. Optimization of Analog Circuits 39
50 3.3.6 Conjugate-gradient (CG) approach Optimization problem with a quadratic objective function f q (x) = b T x xt H x (101) n x large, H sparse min f q (x) (102) first-order optimality conditions leads to a linear equation system for the stationary point x g (x) = fq (x) = 0 (101) g (x) = b + H x = 0 (103) H x = b x (104) x is computed by solving linear equation system, not by matrix inversion second-order condition: 2 f q (x ) = H positive definite (105) Optimization of Analog Circuits 40
51 Eigenvalue decomposition of a symmetric positive definite matrix H H = U D 1 2 D 1 2 U T (106) D 1 2 : diagonal matrix of square roots of eigenvalues U: columns are orthogonal eigenvectors, i.e. U 1 = U T, U U T = I (107) substituting (106) in (101) f q (x) = b T x ( T ( ) D 1 2 UT x) D 1 2 UT x (108) coordinate transformation: x = D 1 2 UT x, x = U D 1 2 x (109) substituting (109) in (108) f q (x) = b T x x T x (110) b = D 1 2 UT b (111) Hessian of transformed quadratic function (110) is unity matrix H = I gradient of f q g = f q (x ) (110) = b + x (112) transformation of gradient from (103), (106), (109), and (111) f (x) = b + H x = U D 1 2 b + U D 1 2 x (113) from (112) g = f (x) = U D 1 2 f q (x ) = U D 1 2 g (114) min f q (x) min f q (x ) (115) solution in transformed coordinates (g! = 0) x = b (116) Optimization of Analog Circuits 41
52 Conjugate directions i j r(i)t H r (j) = 0 ( H orthogonal ) (117) from (106) ( ) T ) D 1 2 UT r (i) (D 1 2 UT r (j) = 0 i j r (i) T r (j) = 0 (118) i j transformed conjugate directions are orthogonal r (i) T = D 1 2 UT r (i) (119) Step length substitute x = x (κ) + α r (κ) in (101) f q (α) = b T (x (κ) + α r (κ)) (x (κ) + α r (κ)) T ( H x (κ) + α r (κ)) (120) f q (α) = 0: f q (α) = b T r (κ) + ( x (κ) + α r (κ)) T H r (κ) (121) (103) ( = f ) q x (κ) T }{{} r + α r (κ)t H r (κ) (122) g (κ)t α (κ) = g(κ)t r (κ) r (κ)t H r (κ) (123) New iterative solution x (κ+1) = x (κ) + α (κ) }{{} (123) r (κ) }{{} (117) r (0) = g (0) }{{} (103) = x (0) + κ α (i) r (i) (124) i=1 (125) new search direction: combination of actual gradient and previous search direction r (κ+1) = g (κ+1) + β (κ+1) r (κ) (126) substituting (126) into (117) g (κ+1)t H r (κ) + β (κ+1) r (κ)t H r (κ) = 0 (127) β (κ+1) H r (κ) = g(κ+1)t r (κ)t H r (κ) (128) Optimization of Analog Circuits 42
53 x 2 r (0) = g (0) x x (1) r (1) with r (0)T H r (1) = 0 x (0) g (1) x 1 Figure 28. x 2 r (0) r(1) with r (0) T r (1) = 0 x (1) x x (0) g (0) x 1 Figure 29. Optimization of Analog Circuits 43
54 Some properties g (κ+1) = b + H x (κ+1) = b + H x (κ) + α (κ) H r (κ) = g (κ) + α (κ) H r (κ) (129) g (κ+1)t r (κ) (129) = g (κ)t r (κ) + α (κ) r (κ)t H r (κ) (123) = g (κ)t r (κ) g (κ)t r (κ) r(κ)t H r (κ) r (κ)t H r (κ) = 0 (130) i.e., the actual gradient is orthogonal to previous search direction g (κ)t r (κ) (126) = g (κ)t ( g (κ) + β (κ) r (κ 1)) = g (κ)t g (κ) β (κ) g (κ)t r }{{ (κ 1) } 0 by (130) i.e., descent direction going along conjugate direction = g (κ)t g (κ) (131) g (κ+1)t g (κ) (129) = g (κ)t g (κ) + α (κ) r (κ)t H g (κ) (131),(126) = g (κ)t r (κ) + α (κ) r (κ)t H ( r (κ) + β (κ) r (κ 1)) (117) = g (κ)t r (κ) α (κ) r (κ)t H r (κ) (123) = g (κ)t r (κ) g (κ)t r (κ) = 0 (132) i.e., actual gradient is orthogonal to previous gradient substituting (126) into (130) g (κ+1)t ( g (κ) + β (κ) r (κ 1)) = 0 (132) g (κ+1)t r (κ 1) = 0 (133) i.e., actual gradient is orthogonal to all previous search directions substituting (129) into (132) g (κ+1)t (g (κ 1) + α (κ 1) H r (κ 1)) = 0 (126) g (κ+1)t g (κ 1) + α (κ 1) ( r (κ+1) + β (κ+1) r (κ)) H r (κ 1) = 0 (117) g (κ+1)t g (κ 1) = 0 (134) i.e., actual gradient is orthogonal to all previous gradients Optimization of Analog Circuits 44
55 Simplified computation of CG step length substituting (131) into (123) α (κ) = g(κ)t g (κ) r (κ)t H r (κ) (135) Simplified computation of CG search direction substituting H r (κ) from (129) into (128) from (135) and (132) β (κ+1) = g(κ+1)t 1 α (κ) (g (κ+1) g (κ)) r (κ)t H r (κ) (136) β (κ+1) = g(κ+1)t g (κ+1) (137) g (κ)t g (κ) CG algorithm solves optimization problem (102) or linear equation system (103) in at most n x steps g (0) = b + H x (0) r (0) = g (0) κ = 0 while r (k) 0 step length: α (κ) g = g(κ)t (κ) line-search r (κ)t H r (κ) in nonlinear programming new point: x (κ+1) = x (κ) + α (κ) r (κ) new gradient: g (κ+1) = g (κ) + α (κ) H r (κ) gradient computation in nonlinear programming Fletcher-Reeves: β (κ+1) g = g(κ+1)t (κ+1) g (κ)t g (κ) new search-direction: r (κ+1) = g (κ+1) + β (κ+1) r (κ) κ := κ + 1 end while Polak-Ribière: β (κ+1) = g(κ+1)t (g (κ+1) g (κ)) (138) g (κ)t g (κ) Optimization of Analog Circuits 45
56 Optimization of Analog Circuits 46
57 4 Constrained optimization problem formulations 4.1 Quadratic Programming (QP) QP linear equality constraints min f q (x) s.t. A x = c f q (x) = b T x xt H x H: symmetric, positive definite (139) A <nc n x>, n c n x, rank (A) = n c A x = c (140) underdetermined linear equation system with full rank (Appendix E). Optimization of Analog Circuits 47
58 Transformation of (139) into an unconstrained optimization problem by coordinate transformation approach: x = [ Y <nx nc> Z] }{{ <nx nx> } non-singular c <nc> y <n x> = Y c }{{} feasible point of (140) + Z y }{{} degrees of freedom in (140) (141) A x = } A {{ Y} c + A }{{ Z} y = c (142) (143) (144) A Y = I <nc n c> (143) A Z = 0 <nc n x n c> (144) φ (y) = f q (x (y)) (145) = b T (Y c + Z y) (Y c + Z y)t H (Y c + Z y) (146) ( = b + 1 ) T 2 H Y c Y c (147) + (b + H Y c) T Z y yt Z T H Z y min f q (x) s.t. A x = c min φ (y) (148) y : solution of the unconstrained optimization problem (148), Z y lies in the kernel of A Stationary point of optimization problem (148): φ (y) = 0 ( ZT H Z ) y = Z T (b + H Y c) y x = Y c + Z y (149) }{{}}{{} reduced Hessian reduced gradient Computation of Y, Z by QR-decomposition from (143) and (483): R T Q T Y = I Y = Q R T (150) from (143) and (483): R T Q T Z = 0 Z = Q (151) Optimization of Analog Circuits 48
59 x 3 A x = c feasible space z 2 Y c Z y z 1 x 2 x 1 Figure 30. Y c: minimum length solution, i.e., orthogonal to plane A x = c. z i, i = 1,, n x n c : orthonormal basis of kernel (null-space) of A, i.e. A (Z y) = Computation of Lagrange factor λ required for QP with inequality constraints Lagrange function of problem (139) L (x, λ) = f q (x) λ T (A x c) (152) L (x) = 0 : A T λ = f q (x) (overdetermined) λ (143) = Y T f q (x ) (153) or, in case of QR-decomposition λ = Y T (b + H x ) (154) A T λ = b + H x (155) from (483) Q R λ = b + H x (156) R λ = Q T (b + H x ) backward substitution λ (157) Optimization of Analog Circuits 49
60 Computation of Y c in case of a QR-decomposition (483) R T Q T x }{{} u A x = c = c R T u = c Y c forward substitution (150) = Q R T c = Q u (insert u) u (158) Transformation of (139) into an unconstrained optimization problem by Lagrange multipliers first-order optimality conditions of (139) according to Lagrange function (152) L (x) = 0 : b + H x A T λ = 0 A x c = 0 } (159) [ ] [ ] [ ] H A T x b = x, λ (160) A 0 λ c }{{} Lagrange matrix L L: symmetric, positive definite iff Z T H Z positive definite e.g., L 1 = [ K ] T T T U (161) K = Z (Z T H Z ) 1 Z T (162) T = Y K H Y (163) U = Y T H K H Y Y T H Y (164) factorization of L 1 by factorization of Z T H Z and computation of Y Z Optimization of Analog Circuits 50
61 4.1.2 QP - inequality constraints min f q (x) s.t. a T i x = c i, i E a T j x c j, j I f q (x) = b T x xt H x H: symmetric, positive definite (165) A <nc nx>, n c n x, rank (A) = n c. a T i, i E. A =. a T j, j I. Optimization of Analog Circuits 51
62 compute initial solution x (0) that satisfies all constraints, e.g., by linear programming κ := 0 repeat /* problem formulation at current iteration point x (κ) + δ for active constraints A (κ) (active set method) */ min f q (δ) s.t. a T i δ = 0, i A (κ) δ (κ) case I: δ (κ) = 0 /* i.e., x (κ) optimal for current A (κ) */ compute λ (κ) e.g., by (154) or (157) case Ia: λ (κ) i A (κ) i 0, /* i.e., first-order optimality conditions satisfied */ [ I x = x (κ) ; stop case Ib: λ (κ) i A (κ) i < 0, I /* i.e., constraint becomes inactive, further improvement of f possible */ q = arg minλ (κ) i, i A (κ) I /* i.e., q is most sensitive constraint to become inactive */ A (κ+1) = A (κ) \ {q} x (κ+1) = x (κ) case II: δ (κ) 0, /* i.e., f can be reduced for current A (κ) ; step-length computation; if no new constraint becomes active then α (κ) = 1 (case IIa); else find inequality constraint that becomes inactive first and corresponding α (κ) < 1 (case IIb) */ safety margin bound {}}{ α (κ) c = min(1, min i a T i x (κ) i / A (κ), a T i δ (κ) < 0 a T i δ (κ) ) }{{}}{{} consumed safety margin inactive constraints that at α (κ) = 1 approach their lower bound q = argmin ( ) case IIa: α (κ) = 1 [ A (κ+1) = A (κ) case IIb: α (κ) < 1 [ A (κ+1) = A (κ) (q ) x (κ+1) = x (κ) + α (κ) δ (κ) κ := κ + 1 Optimization of Analog Circuits 52
63 4.1.3 Example (B) f q = const x 2 x (4) = x (3) x 1 δ (4) x (5) x (6) δ (5) δ (2) x (1) = x (2) (C) (A) Figure 31. (κ) A (κ) case (1) {A, B} Ib (2) {B} IIa (3) {B} Ib (4) { } IIb (5) {C} IIa (6) {C} Ia Concerning q = arg minλ (κ) i for κ = 1: i A (1) I A x (1) B f q Figure 32. f q = λ A A + λ B B λ A < 0, λ B = 0 Optimization of Analog Circuits 53
64 4.2 Sequential Quadratic programming (SQP), Lagrange Newton SQP equality constraints min f (x) s.t. c (x) = 0 c <nc>, n c n x (166) Lagrange function of (166) L (x, λ) = f (x) λ T c (x) = f (x) i λ i c i (x) (167) L (x, λ) = L(x) L (λ) = = [ f (x) i λ i c i (x) c (x) g(x) A T (x) {}}{ [ c i (x) ] λ c (x) ] (168) 2 L (x, λ) = = 2 L (x) 2 L λ x T 2 L x λ T 2 L (λ) 2 f (x) i λ i 2 c i (x). c i (x) T. }{{} A(x) A T (x) 0 (169) Optimization of Analog Circuits 54
65 Newton approach to (167) in κ-th iteration step [ ) ) δ (κ) L(x } (κ) {{ + δ (κ) }, λ } (κ) + {{ λ (κ) } ) L (x (κ), λ (κ) + 2 L (x (κ), λ (κ) x }{{}}{{} (κ+1) λ (κ+1) L (κ) 2 L (κ) with (168) and (169) W {}} (κ) { 2 f(x (κ) ) λ (κ) ( i 2 c ) [ i x (κ) A (κ)t δ (κ) i A (κ) 0 λ (κ) ] = λ (κ) ]! = 0 (170) [ g (κ) + A (κ)t λ (κ) c (κ) ] (171) [ W (κ) A (κ)t ] [ δ (κ) ] [ g (κ) ] A (κ) 0 λ (κ+1) = c (κ) (172) L (δ, λ) = g (κ)t δ δt W (κ) δ λ T (A (κ) δ + c (κ)) (173) min g (κ)t δ δt W (κ) δ s.t. A (κ) δ = c (κ) (174) QP problem, W (κ) according to (171) including quadratic parts of objective and constraint, linearized equality constraints from (166) Quasi-Newton approach possible inequality constraints, A (κ) δ c (κ) I, treated as in QP Penalty function transformation into a penalty function of an unconstrained optimization problem with penalty parameter µ > 0 quadratic: logarithmic: l 1 exact: P quad (x, µ) = f (x) + 1 2µ ct (x) c (x) (175) P log (x, µ) = f (x) µ log c i (x) (176) P l1 (x, µ) = µ f (x) i E i c i (x) + i I max ( c i (x), 0) (177) Optimization of Analog Circuits 55
66 Optimization of Analog Circuits 56
67 5 Statistical parameter tolerances modeling of manufacturing variations through a multivariate continuous distribution function of statistical parameters x s cumulative distribution function (cdf): cdf (x s ) = xs,1 xs,nxs pdf (t) dt (178) (discrete: cumulative relative frequencies) probability density function (pdf): dt = dt 1 dt 2... dt nxs pdf (x s ) = nxs xs,1 xs,nxs cdf (x s ) (179) (discrete: relative frequencies) x s,i denotes the random number value of a random variable X s,i Optimization of Analog Circuits 57
68 5.1 Univariate Gaussian distribution (normal distribution) x s N ( x s,0, σ 2) (180) x s,0 σ 2 σ : mean value : variance : standard deviation probability density function of the univariate normal distribution: pdf N ( xs, x s,0, σ 2) = 1 2π σ e 1 2 ( xs xs,0 σ ) 2 (181) pdf N 3σ 2σ σ 0 σ 2σ 3σ x s x s,0 1 cdf N 0.5 3σ 2σ σ 0 σ 2σ 3σ x s x s,0 Figure 33. Probability density function, pdf, and corresponding cdf of a univariate Gaussian distribution. The area of the shaded region under the pdf is the value of the cdf as shown. x s x s,0 3σ 2σ σ 0 σ 2σ 3σ 4σ cdf (x s x s,0 ) 0.1% 2.2% 15.8% 50% 84.1% 97.7% 99.8% 99.99% Standard normal distribution table Optimization of Analog Circuits 58
69 5.2 Multivariate normal distribution x s N (x s,0, C) (182) x s,0 : vector of mean values of statistical parameters x s C: covariance matrix of statistical parameters x s, symmetric, positive definite probability density function of the multivariate normal distribution: pdf N (x s, x s,0, C) = 1 2π n xs det (C) e 1 2 β2 (x s,x s,0,c) (183) β 2 (x s, x s,0, C) = (x x s,0 ) T C 1 (x x s,0 ) (184) C = Σ R Σ (185) 1 ρ σ 1 0 1,2 ρ 1,nxs Σ =..., R = ρ 1,2 1 ρ 2,nxs. 0 σ.... nxs ρ 1,nxs ρ 2,nxs 1 σ1 2 σ 1 ρ 1,2 σ 2 σ 1 ρ 1,nxs σ nxs σ 1 ρ 1,2 σ 2 σ2 2 σ 2 ρ 2,nxs σ nxs C = σ 1 ρ 1,nxs σ nxs σn 2 xs (186) (187) R : correlation matrix of statistical parameters σ k : standard deviation of component x s,k, σ k > 0 σ 2 k : variance of component x s,k σ k ρ k,l σ l : covariance of components x s,k, x s,l ρ k,l : correlation coefficient of components x s,k and x s,l, 1 < ρ k,l < 1 ρ k,l = 0 : uncorrelated and also independent if jointly normal ρ k,l = 1 : strongly correlated components Optimization of Analog Circuits 59
70 x s,2 x s,0,2 x s,0 β 2 (x s ) = const x s,0,1 x s,1 Figure 34. Level sets of a two-dimensional normal pdf (a) (b) (c) x s,2 β 2 (x s) = const x s,2 β 2 (x s) = const x s,2 β 2 (x s) = const x s,0,2 x s,0 x s,0,2 xs,0 x s,0,2 x s,0 x 1 x s,0,1 x s,1 x s,0,1 x s,1 x s,0,1 x s,1 Figure 35. Level sets of a two-dimensional normal pdf with general covariance matrix C, (a), with uncorrelated components, R = I, (b), and uncorrelated components of equal spread, C = σ 2 I, (c). Optimization of Analog Circuits 60
71 x 2 ρ = 0 a 2 σ2 β 2 = const = a 2 a 2 σ 1 x 1 Figure 36. Level set with β 2 = a 2 of a two-dimensional normal pdf for different values of the correlation coefficient, ρ. Optimization of Analog Circuits 61
72 5.3 Transformation of statistical distributions y R ny, z R nz, n y = n z z = z (y), y = y (z) such that the mapping from y to z is smooth and bijective (precisely z = φ (y), y = φ 1 (z)) cdf y (y) = = y z pdf y (y ) dy = z(y) ( ) pdf y (y (z )) y det dz z T pdf z (z ) dz = cdf z (z) (188) y pdf y (y ) dy = z pdf z (z ) dz (189) ( ) pdf z (z) = pdf y (y (z)) y det (190) z T univariate case: pdf z (z) = pdf y (y(z)) y z (191) In the simple univariate case the function pdf z has a domain that is a scaled version of the domain of pdf y. ( ) y determines the scaling factor. In high-order cases, the random z variable space is scaled and rotated with the Jacobian matrix y determining the scaling z T and rotation. pdf y (y) pdf z (z) y 1 y 2 y 3 y z 1 z 2 z 3 z Figure 37. Univariate pdf with random number y is transformed to a new pdf of new random number z = z (y). According to (188) the shaded areas as well as the hatched areas under the curve are equal. Optimization of Analog Circuits 62
73 5.3.1 Example Given: probability density function pdf U (z), here a uniform distribution: pdf U (z) = { 1 for 0 < z < 1 0 otherwise (192) probability density function pdf y (y), y R random number z Find: random number y z y from (188) z y 0 pdf z (z ) }{{} 1 for 0 z 1 dz = pdf y (y ) dy (193) from (192) hence z = y pdf y (y ) dy = cdf y (y) (194) y = cdf 1 y (z) (195) This example details a method to generate sample values of a random variable y with an arbitrary pdf pdf y if sample values are available from a uniform distribution pdf z : insert pdf y (y) in (194) compute cdf y by integration compute inverse cdf 1 y create uniform random number, z, insert into (195) to get sample value, y, according to pdf y (y) Optimization of Analog Circuits 63
74 Optimization of Analog Circuits 64
75 6 Expectation values and their estimators 6.1 Expectation values Definitions h (z): function of a random number z with probability density function pdf (z) Expectation value E {h (z)} = E {h (z)} = pdf(z) + h (z) pdf (z) dz (196) Moment of order κ Mean value (first-order moment) Central moment of order κ m (κ) = E {z κ } (197) m (1) = m = E {z} (198) E {z 1 } m = E {z} =. (199) E {z nz } c (κ) = E {(z m) κ } c (1) = 0 (200) Variance (second-order central moment) σ: standard deviation Covariance c (2) = E { (z m) 2} = σ 2 = V {z} (201) cov {z i, z j } = E {(z i m i ) (z j m j )} (202) Variance/covariance matrix C = V {z} = E {(z } m) (z m) T V {z 1 } cov {z 1, z 2 } cov {z 1, z nz } cov {z 2, z 1 } V {z 2 } cov {z 2, z nz } =..... cov {z nz, z 1 } cov {z nz, z 2 } V {z nz } V {h (z)} = E (203) { (h (z) E {h (z)}) (h (z) E {h (z)}) T } (204) Optimization of Analog Circuits 65
Simulation und Optimierung analoger Schaltungen Optimization Methods for Circuit Design
Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Simulation und Optimierung analoger Schaltungen Optimization Methods
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationOptimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23
Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationNumerical Optimization: Basic Concepts and Algorithms
May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some
More information1 Numerical optimization
Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationNumerical Optimization of Partial Differential Equations
Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationStatic unconstrained optimization
Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationOptimization Methods
Optimization Methods Categorization of Optimization Problems Continuous Optimization Discrete Optimization Combinatorial Optimization Variational Optimization Common Optimization Concepts in Computer Vision
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationStatistics 580 Optimization Methods
Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of
More information2.3 Linear Programming
2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationcomponent risk analysis
273: Urban Systems Modeling Lec. 3 component risk analysis instructor: Matteo Pozzi 273: Urban Systems Modeling Lec. 3 component reliability outline risk analysis for components uncertain demand and uncertain
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationMinimization of Static! Cost Functions!
Minimization of Static Cost Functions Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2017 J = Static cost function with constant control parameter vector, u Conditions for
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationOptimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng
Optimization 2 CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Optimization 2 1 / 38
More informationNumerical Analysis of Electromagnetic Fields
Pei-bai Zhou Numerical Analysis of Electromagnetic Fields With 157 Figures Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona Budapest Contents Part 1 Universal Concepts
More informationLecture Notes: Geometric Considerations in Unconstrained Optimization
Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections
More informationLINEAR AND NONLINEAR PROGRAMMING
LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationLecture 7 Unconstrained nonlinear programming
Lecture 7 Unconstrained nonlinear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University,
More informationMVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg
MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material
More informationSubject: Optimal Control Assignment-1 (Related to Lecture notes 1-10)
Subject: Optimal Control Assignment- (Related to Lecture notes -). Design a oil mug, shown in fig., to hold as much oil possible. The height and radius of the mug should not be more than 6cm. The mug must
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationContents. Preface. 1 Introduction Optimization view on mathematical models NLP models, black-box versus explicit expression 3
Contents Preface ix 1 Introduction 1 1.1 Optimization view on mathematical models 1 1.2 NLP models, black-box versus explicit expression 3 2 Mathematical modeling, cases 7 2.1 Introduction 7 2.2 Enclosing
More informationECS550NFB Introduction to Numerical Methods using Matlab Day 2
ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationDetermination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study
International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:
More informationQuasi-Newton methods for minimization
Quasi-Newton methods for minimization Lectures for PHD course on Numerical optimization Enrico Bertolazzi DIMS Universitá di Trento November 21 December 14, 2011 Quasi-Newton methods for minimization 1
More informationConstrained Nonlinear Optimization Algorithms
Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu Institute for Mathematics and its Applications University of Minnesota August 4, 2016
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationGradient-Based Optimization
Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More informationMiscellaneous Nonlinear Programming Exercises
Miscellaneous Nonlinear Programming Exercises Henry Wolkowicz 2 08 21 University of Waterloo Department of Combinatorics & Optimization Waterloo, Ontario N2L 3G1, Canada Contents 1 Numerical Analysis Background
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one
More informationMATH2070 Optimisation
MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationQuadratic Programming
Quadratic Programming Outline Linearly constrained minimization Linear equality constraints Linear inequality constraints Quadratic objective function 2 SideBar: Matrix Spaces Four fundamental subspaces
More informationOn Lagrange multipliers of trust-region subproblems
On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationReview of Classical Optimization
Part II Review of Classical Optimization Multidisciplinary Design Optimization of Aircrafts 51 2 Deterministic Methods 2.1 One-Dimensional Unconstrained Minimization 2.1.1 Motivation Most practical optimization
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationSolving linear equations with Gaussian Elimination (I)
Term Projects Solving linear equations with Gaussian Elimination The QR Algorithm for Symmetric Eigenvalue Problem The QR Algorithm for The SVD Quasi-Newton Methods Solving linear equations with Gaussian
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationNonlinearOptimization
1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,
More informationGeometry optimization
Geometry optimization Trygve Helgaker Centre for Theoretical and Computational Chemistry Department of Chemistry, University of Oslo, Norway European Summer School in Quantum Chemistry (ESQC) 211 Torre
More informationQuasi-Newton Methods
Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications
More informationMATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,
More informationCHAPTER 11. A Revision. 1. The Computers and Numbers therein
CHAPTER A Revision. The Computers and Numbers therein Traditional computer science begins with a finite alphabet. By stringing elements of the alphabet one after another, one obtains strings. A set of
More informationAppendix A Taylor Approximations and Definite Matrices
Appendix A Taylor Approximations and Definite Matrices Taylor approximations provide an easy way to approximate a function as a polynomial, using the derivatives of the function. We know, from elementary
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationArc Search Algorithms
Arc Search Algorithms Nick Henderson and Walter Murray Stanford University Institute for Computational and Mathematical Engineering November 10, 2011 Unconstrained Optimization minimize x D F (x) where
More informationPart 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)
Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x
More informationQuasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG. Limited memory BFGS (L-BFGS)
Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno (BFGS) Limited memory BFGS (L-BFGS) February 6, 2014 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationOptimization Concepts and Applications in Engineering
Optimization Concepts and Applications in Engineering Ashok D. Belegundu, Ph.D. Department of Mechanical Engineering The Pennsylvania State University University Park, Pennsylvania Tirupathi R. Chandrupatia,
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationMarch 5, 2012 MATH 408 FINAL EXAM SAMPLE
March 5, 202 MATH 408 FINAL EXAM SAMPLE Partial Solutions to Sample Questions (in progress) See the sample questions for the midterm exam, but also consider the following questions. Obviously, a final
More informationInverse Problems and Optimal Design in Electricity and Magnetism
Inverse Problems and Optimal Design in Electricity and Magnetism P. Neittaanmäki Department of Mathematics, University of Jyväskylä M. Rudnicki Institute of Electrical Engineering, Warsaw and A. Savini
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationNotes on Numerical Optimization
Notes on Numerical Optimization University of Chicago, 2014 Viva Patel October 18, 2014 1 Contents Contents 2 List of Algorithms 4 I Fundamentals of Optimization 5 1 Overview of Numerical Optimization
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More information