An (exhaustive?) overview of optimization methods

Transcription

1 An (exhaustive?) overview of optimization methods C. Dutang Summer 2010

2 2 CONTENTS Contents 1 Minimisation problems Continuous optimisation, uncountable set X Unconstrained optimisation Constrained optimisation Saddle point Root problems General equation, uncountable set X Fixed-point problems 17 4 Variational Inequality and Complementarity problems Examples and problem reformulation Examples Problem reformulation Algorithms for CPs Algorithms for VIPs A Bibliography 24 B Websites 25

3 3 The following materials come mainly from Raydan & Svaiter (2001), Fletcher (2001), Madsen et al. (2004), Bonnans et al. (2006), Boggs & Tolle (1996), Dennis & Schnabel (1996), Conte & de Boor (1980), Ye (1996), Lange (1994) for optimization and root problems, Varadhan (2004), Roland et al. (2007) for fixed-point problems, and Facchinei & Pang (2003a,b) for variational inequality problems. 1 Minimisation problems Minimisation problems consist in solving Notations: gradient vector g(x) = f(x), Hessian matrix H(x) = 2 f(x), min x X R n f(x) such that c j(x) = 0, j E and c j (x) 0, j I. Jacobian matrix of the function c J c = ( ci x j )ij, positive and negative part of z, z + = max(z, 0) and z = min(z, 0), c(x) is such that c(x) i = c i(x) if i E or c i (x) + if i I, the superscript T denotes the transpose. 1.1 Continuous optimisation, uncountable set X Unconstrained optimisation, E, I = 1. Quadratic problems f(x) = 1 2 xt Mx + b T x + c. Prop: unique solution if matrix M is symmetric positive and definite. Thus solve Mx + b = 0. (a) General descent scheme (x k+1 = x k + t k d k with stepsize t k and direction d k such that f(x k+1 ) < f(x k )) i. d k = (Mx k + b): steepest descent method, ii. d k = (Mx k + b) and t k = (x k x k 1 ) T (x k x k 1 ) : Barzilai-Borwein method, (x k x k 1 ) T M(x k x k 1 ) iii. relaxed descent scheme (x k+1 = x k + t k θ k d k with relaxation parameters 0 < θ k < 2) Assuming direction d k = g(x k ) and optimal stepsize t k = g(x k) T g(x k ), the choice of relaxation parameters are g(x k ) T Mg(x k ) θ k = 1 steepest descent method, θ k = 2 we get f(x k+1 ) = f(x k ), If the sequence (θ k ) k has an accumulation point θ then the relaxed Cauchy method converges.

4 4 1 MINIMISATION PROBLEMS (b) Conjugate gradient methods (x k+1 = x k + t k d k+1 given the full information at kth iteration): i. classic CG method: Init: x 0, d 1 = g 1 Iter: g k+1 = g k + t k Md k with t k = g k 2 g T k Md k d k+1 = g k+1 + c k d k with c k = g k 2 g T k Md k NB: all directions (d 1,..., d k ) are conjugate w.r.t. M. ii. preconditioned conjugate gradient method, (c) Gauss-Newton method for least square problems (f(x) = p j=1 f j 2(x)) Iteration is x k+1 = x k + d k with d k = G(x k ) 1 f(x), gradient f(x) = p j=1 f j(x) f j (x), approximate Hessian G(x) = p j=1 f j(x) f j (x) T. 2. Smooth non linear problems f C 1,. (a) General descent scheme (x k+1 = x k + t k d k with stepsize t k and direction d k such that f(x k+1 ) < f(x k )) Direction rule: i. d k = arg min g(x k ) T d d <δ L 1 norm, d k is the index of the largest component. Gauss Siedel? L 1 norm, d k = g(x k ): Steepest descent method. Stepsize rule: i. fixed: method of successive approximation, ii. optimal t k = arg minf(x k + td k ) (useless in practice), t>0 iii. Wolfe s rule, iv. Goldstein and Price, v. Armijo. Direction+Stepsize rule: i. d k = g(x k ) and t k = d T k 1 d k 1 d T k 1 (g(x k) g(x k 1 )) Barzilai-Borwein method, ii. d k = g(x k ) and t k = arg min t>0 f(x k + td k ) Cauchy method, (b) Conjugate gradient methods:

5 1.1 Continuous optimisation, uncountable set X 5 i. classic CG Init: d 1 = g(x 1 ) Iter: d k = g(x k ) + β k d k 1 for k > 1 β k = g(x k) T g(x k ) : Fletcher-Reeves update, g(x k 1 ) T g(x k 1 ) β k = g(x k) T (g(x k ) g(x k 1 )) : Polak-Ribière update, g(x k 1 ) T g(x k 1 ) Beale-Sorenson, Hestenses-Stiefel, Conjugate-Descent,... NB: all directions (d 1,..., d k ) are conjugate to the Hessian matrix H(x k )? ii. preconditionned CG (c) Newton method (x k+1 = x k + d k with direction d k minimizes the quadratic function q f (d) = f(x k ) + g(x k ) T d dt g(x k )d). i. exact Newton method, d k solves g(x k ) + g(x k )d = 0 i.e. minimizer of the local quadratic approximation q f. ii. Quasi-Newton methods, d k approximates the exact minimizer of q f Scheme: Init: x 0, W 0 Iter: while g(x k ) > ɛ approximate Hessian inverse W k = W k 1 + B k compute direction d k = W k g(x k ) line search for t k Choice of W : Constraints: symmetric, positive, definite and verified the quasi-newton equation W k (g k+1 g k ) = x k+1 x k, known methods for W : Davidon-Fletcher-Powell (DFP) or Broyden-Fletcher-Goldfarb-Shanno (BFGS). iii. inexact Newton or truncated Newton methods: It requires that d k decreases the linear residual, H(x k )d k + g(x k ) η c g(x k ), where 0 < η c < 1 is the forcing term. NB: univariate case (n = 1), quasi-newton has a unique equation: secant method or regula falsi method. (d) Trust region algorithms (x k+1 = x k + h k with h k a local minimizer) Scheme: h k ( k ) = arg min f(x k ) + g(x k ) T h ht Hk h with H k is positive definite matrix, h < k if f(x k + h k ( k )) f(x k ) mh k ( k ) then terminates iteration, otherwise decrease k. NB: 0 < m < 1 a fixed coefficient. Choice of matrix H k :

6 6 1 MINIMISATION PROBLEMS H k is the Hessian H(x k ): positive-definiteness is not guaranteed, H k = H(x k ) + λi n with λ > 0: Levenberg-Marquardt algorithm (derived with KKT conditions), quasi-newton approximation, Gauss-Newton approximation NB: Hk = 0 gives the steepest descent. 3. Non smooth problems (a) direct search (derivative free) methods i. Nelder-Mead algorithm ii. Hooke-Jeeves algorithm iii. multi-directional algorithm iv.... (b) metaheuristics i. evolutionary algorithms ii. stochastic algorithms simulated annealing Monte-Carlo method ant colony (c) regularizing techniques i. filter algorithms ii. noisy algorithms (d) NSO methods i. subgradient methods ii. bundle methods iii. gradient sampling methods iv. hybrid methods (e) special problems i. piecewise linear ii. minmax iii. partially separable

7 1.1 Continuous optimisation, uncountable set X 7 Function Method type Algorithms type Method R function - package quadratic smooth non smooth large scale descent scheme Gauss-Newton method conjugate gradient (CG) descent scheme conjugate gradient Newton methods trust-region direct search methods metaheuristics regularizing techniques NSO methods steepest descent relaxed descent scheme CG methods preconditionned CG steepest descent (BB, Cauchy) Gauss-Siedel CG methods preconditioned CG exact quasi-newton truncated Newton direct Hessian Levenberg-Marquardt quasi-newton Nelder-Mead algorithm multidirectional algorithm evolutionary algorithms stochastic algorithms filter algorithms noisy algorithms subgradient methods bundle methods gradient sampling methods Barzilai-Borwein Cauchy DFP BFGS genetic algorithm covariance matrix adaptation evo. strat. simulated annealing Monte-Carlo method ant colony dfsane BB optim stats dfsane BB optim stats nlm stats optim stats trust trust optim stats genoud rgenoud, mco cmaes optim stats limited-memory algorithms L-BFGS optim stats Hessian free methods conjugate gradient (CG) optim stats

8 8 1 MINIMISATION PROBLEMS Constrained optimisation (E, I ), also known as nonlinear programming The Lagrangian function is defined as L(x, λ) = f(x) + λ T c(x), with λ the Lagrange multiplier. The Karush-Kuhn-Tucker (KKT) conditions are x L(x, λ) = 0, equality constraints j E, c j (x) = 0 and λ j R, active inequality constraints j I, c j (x) = 0 and λ j > 0, inactive inequality constraints j I, c j (x) < 0 and λ j = box constraints (a) projected Newton method, (b) limited-memory box-constraint BFGS method: L-BFGS-B, 2. linear constraints (a) linear programming (f is linear) i. simplex method, ii. dual simplex method, iii. Karmarkar algorithm, (b) Interior point methods for linear constraints Consider the problem min f(x) such that Ax = b and x 0. Let a log potential be π(x) = x i log(x i). We want to minimize the penalised function f(x) + µπ(x). A central path or path following algorithm is a sequence of points (x µ n, λ µ n, s µ n ) solution of the problem for a sequence of decreasing (µ n ) n to 0. Main algorithms are i. potential reduction algorithm, ii. primal-dual symmetric algorithm,. first order optimality conditions x.s = µ n 1 f(x) + A T λ = s Ax = b such that x 0, s 0,

9 1.1 Continuous optimisation, uncountable set X 9 iii. generic predictor-corrector algorithms or adaptive path-following algorithm, Generic predictor-corrector algorithms solve a linear complementarity reformulation of this problem with an iterative method: a small-neighborhood algorithm, a predictor-corrector algorithm with modified field, a large-neighborhood algorithm. 3. general constraints (a) Sequential linear programming linearize both objective and constraint functions using Taylor series expansion. (b) Sequential quadratic programming i. Local methods for equality constraints KKT conditions reduce to f(x) + J c (x) T λ = 0 and c(x) = 0: Newton method is an iterative method { ( xk+1 = x k + d k 2 computed from xx L(x k, λ k ) J c (x k ) T ) ( ) ( ) dk f(xk ) = λ k+1 = λ k + µ k J c (x k ) 0 λ k + µ k c(x k ) full Hessian approximation: replace 2 xxl with an approximated Hessian B k (possible scheme PSB, BFGS), augmented Lagrangian (add a constraint-penalty term) to guarantee positiveness, reduced Hessian matrix: positiveness is only guaranteed on a subspace, ii. Local methods for (in)equality constraints The SQP algorithm consists in solving a sequence of quadratic (Taylor) approximations of the Lagrangian. Scheme: Init: x 0, λ 0 Iterate while a termination criterion x k+1 = x k + d k solution of min d f(x k ) T d dt 2 xxl(x k, λ k )d such that c E (x k ) + J ce (x k )d = 0 and c I (x k ) + J ci (x k )d 0 Implementations: active-set strategies, interior-point algorithms, dual approaches. iii. Globalization of SQP method A. trust region method: Iterative method of a sequence of quadratic bounded sub-problem region radius k is updated at each stage. min d < k f(x k ) T d dt B k d, where the trust

10 10 1 MINIMISATION PROBLEMS B. line search method: Iterative method x k+1 = x k + t k d k, where direction d k is approximated by a solution of quadratic sub-problem and t k chosen to ensure a reasonable decrease of a merit function (e.g. f(x) + σ c(x) p ). NB: the penalty parameter has to be updated (σ k ) k. NB: for both those techniques, one particular approximation of the Hessian has to be chosen: quasi-newton, reduced quasi-newton,... (c) Sequential unconstrained methods i. Exterior point methods The general idea is to minimize the merit function f(x) + p(x) where p is a. During the optimization process, an extertior-point method does not guarantee that all points are in the feasible region (hence the name). Possible penalty terms are: quadratic penalty: p(x) = σ/2 c(x) 2 2, Lagrangian: p(x) = µ T c(x), augmented Lagrangian: p(x) = µ T c(x) + σ/2 c(x) 2 2 with c(x) j = c j (x) if j E and max( µ j σ, c j(x)) if j I, non differentiable (exact) function: p(x) = σ c(x) p Then we can either minimize of the associated Lagrangian function is carried out by an iterative method, or solve the dual problem arg max θ(u) with θ(u) = min f(x) + up(x). u 0 x ii. Interior point methods (or adaptive barrier methods): Barrier methods operates in the interior of the feasible region: adapting iteratively the barrier permits the convergence to a boundary by lowering the strength of the corresponding barrier. Let a log potential be π(x) = i log(x i). Using the convex property on f(x) + µπ(x), two majorants of f(x) can be found log surogate function: g l (x, y, µ) = f(x) µ j I c j(y) log c j (x) + µ j I c j (y)(x y), power surogate function: g p (x, y, µ) = f(x) + µ j I c j(y) α+β log c j (x) α + µ j I c j(y) β c j (y)(x y). Then a MM algorithm can be used with an adaptive barrier constant µ k : Init: x 0, µ 0, Iterate while a termination criterion x k+1 = arg min g l (x, x k, µ k ) or arg min g p (x, x k, µ k ), x x adapt µ k+1.

11 1.1 Continuous optimisation, uncountable set X 11 Constraint type Method type Algorithm type Algorithms R function - package box constraints linear constraints general constraints projected Newton method L-BFGS-B linear program interior-point methods sequential linear program sequential quadratic program unconstrained program simplex method dual simplex method Karmarkar algorithm potential reduction algorithm primal-dual symmetric algorithm adaptive path-following algorithm trust-region SQP line-search SQP exterior-point algorithms interior-point algorithms small-neighborhood algorithm predictor-corrector algorithm large-neighborhood algorithm optim stats, Rvmmin Min-max saddle point A saddle point x satisfies the following inequality u, v, φ(u, v) φ(u, v ) φ(u, v ), with x = (u T v T ) T. Grantham (2005) provide an unified way of most algorithms solving saddle points. If we see the iterative algorithm as a function of time, then iterative ( ) algorithm can be defined as the solution of partial differential equation (PDE). Let g be the gradient of φ: g(x) = φ x (x) = gu (x). Then a general algorithm solve the PDE g v (x) x t = P (x)g(x),

12 12 1 MINIMISATION PROBLEMS where P (x) denotes a matrix. Let H(x) be the Hessian matrix ( ) Huu H H(x) = uv (x). H T uv H vv We have the following algorithms 1. steepest descent P (x) = diag(i u, I v ), 2. Newton method P (x) = H(x) 1, 3. gradient enhanced min-max ( ) Huu + α P (x) = u I u H uv (x) 1. H vv α v I v H T uv

13 13 2 Root problems Root problems consist in solving f(x) = 0, x X R n. 2.1 General equation, uncountable set X 1. f linear: Ax = b The solution is unique if A is invertible, i.e. det(a) 0. (a) direct inversion Compute A 1, then x = A 1 b. (b) Gaussian elimination If A is not upper triangular, transform A into a upper triangular matrix, and then compute ascendently the solution (if it exists). Note that it corresponds to a factorization P LU where L is a lower triangular matrix, U upper triangular and P permutation matrix. i. Gauss pivot method ii. Gauss Jordan method iii. Grassman algorithm (c) Decomposition Decompose A into three matrix D E F, with D the diagonal part, E the opposite lower part and F the opposite upper part. i. Jacobi iterations: x k+1 = D 1 (E + F )x k + D 1 b, ii. Gauss-Siedel iterations: x k+1 = (D E) 1 F x k + (D E) 1 b, iii. Successive Over Relaxation: (D ωe)x k+1 = (ωf + (1 ω)d)x k + ωb for ω [0, 1]. (d) projection methods for large scale problems 2. f univariate (a) Newton methods i. Newton-Raphson method Assuming, we have the gradient f, the algorithm is Init: x 0 Iter: x k+1 root of f(x k ) + f (x k )(x k+1 x k ) = 0.

14 14 2 ROOT PROBLEMS ii. the secant method it consists in replacing f (x k ) by a finite-difference approximation f(x k) f(x k 1 ) x k x k 1 iii. the Muller method It consists in approximating the function by the quadratic function q(x) = f(x k ) + (x x k )f[x k, x k 1 ] + (x x k )(x x k 1 )f[x k, x k 1, x k 2 ], in the Newton-Raphson method. where f[.,.] denotes divided differences. Of course, analytical solution exists to find the next iterate x k+1. (b) dichotomic search i. bisection method Assuming f is continuous, the procedure is Init: a 0 and b 0 such that f(a 0 )f(b 0 ) < 0, Iter: let c = a k+b k 2. If f(c)f(a k ) > 0 then a k+1 = c, b k+1 = b k, otherwise b k+1 = c, a k+1 = a k. ii. regula falsi or false position method a hybrid combining dichotomic search and the secant method. Replace the c of the bissection method (middle of [a k, b k ]) by c k root of f(b k ) + f(b k) f(a k ) b k a k (c k b k ) = f polynomial p (a) Bairstow method: Consider a polynom with real coefficients. It consists in dividing succesively the polynom by a quadratic polynom. Thus we get pairs of conjugate zeros. (b) Bernoulli method: Iterative method that use the characteristic polynom to compute zero one after another, and deflate the polynom at each stage. (c) Muller method: Use a quadratic approximation of the polynom to find zeros. (d) Newton method: Iterates the Newton method on the polynom. If combines with Muller method, it can be powerful to find zeros succesively. (e) Laguerre method: It use the derivatives of log p, used in an iterative method. (f) Durand-Kerner or Weierstrass method: Use a fixed-point iteration procedure to compute all roots at a time. 4. f smooth

15 2.1 General equation, uncountable set X 15 (a) Newton-Raphson method Assuming, we have the gradient f, the algorithm is Init: x 0 Iter: x k+1 root of f(x k ) + f(x k )(x k+1 x k ) = 0. (b) quasi-newton methods Approximate the gradient by G k with an update scheme: Init: x 0 Iter: x k+1 root of f(x k ) + G k (x k+1 x k ) = 0. Update schemes Broyden method : G k = G k 1 + (y k 1 G k 1 s k 1 )s T k 1 s T k 1 s k 1 DFP or BFGS direct approximation of G 1 k. with y k 1 = f(x k ) f(x k 1 ) and s k = x k x k 1.

16 16 2 ROOT PROBLEMS Function Methods Algorithms direct inversion linear univariate polynomial smooth Gaussian elimination decomposition projection methods Newton method dichotomic search Bairstow Bernoulli Muller Newton Laguerre Weierstrass Newton-Raphson method quasi-newton Gauss pivot method Gauss-Jordan method Grassman method Jacobi iterations Gauss-Siedel iterations Succesive over relaxations Newton-Raphson method secant method Muller method bissection method regula falsi Broyden BFGS

17 17 3 Fixed-point problems Root problems consist in solving F (x) = 0, x X R n. 1. Direct method: x k+1 = F (x k ), 2. Polynomials methods: x k+1 = d i=0 γ i,kf i (x k ) with F i the ith composition of F, Let u k i be F i (x k ), one iterate is x k u k 0,..., uk d x k+1. The sequence u k i s is called a cycle or a restart. (a) 1st order method: x k+1 can be rewritten as x k α k r k with r k = F (x k ) x k, 1 i. relaxation method: α k independent of x k such as k+1 or a random uniform number in ]0, 2[, ii. Lemaréchal method or RRE1: α k = <v k,r k > <v k,v k > where v k = F (F (x k )) 2F (x k ) + x k, iii. Brezinski method or MPE1: α k = <r k,r k > <r k,v k >, (b) dth order method: The coefficients γ i,k must satisfy the constraints d i=0 γ i,k = 1, d i=0 γ i,kβ i,j,k = 0. i. Reduced Rank Extrapolation (RREd): β i,j,k =< i,1 x k, j,2 x k >, ii. Minimal Polynomial Extrapolation (MPEd): β i,j,k =< i,1 x k, j,1 x k >, with i,j x k = j l=0 ( 1)l j Cj lf l+i (x k ) and Cj l the binomial coefficients. Squaring methods consist in applying twice a cycle step to get the next iterate. So we have x k+1 = d d i=0 j=0 γ i,kγ j,k F i+j (x k ). (a) squaring 1st order method: x k+1 can be rewritten as x k 2α k r k + αk 2v k, i. SqRRE1: α k = <v k,r k > <v k,v k >, ii. SqMPE1: α k = <r k,r k > <r k,v k >, (b) squaring dth order method: i. SqRREd: β i,j,k =< i,1 x k, j,2 x k >, ii. SqMPEd: β i,j,k =< i,1 x k, j,1 x k >, NB: the RRE1 method is (sometimes) called the Richardson method when F is linear and Lemaréchal method otherwise. The SqRRE1 method is called the Cauchy Barzilai Borwein method when F is linear. The MPE1 method is called the Cauchy method when F is linear and Brezinski method otherwise. 3. Epsilon algorithms: (a) Scalar ɛ algorithm SEA (b) Vector ɛ algorithm VEA (c) Topological ɛ algorithm TEA

18 18 4 VARIATIONAL INEQUALITY AND COMPLEMENTARITY PROBLEMS 4 Variational Inequality and Complementarity problems Variational inequality problems VI(K, F ) consist in finding x K such that y K, (y x) T F (x) 0, where F : K R n. We talk about quasi-variational inequality problems for the problem y K(x), (y x) T F (x) 0. Complementarity problems CP(K, F ) consist in finding x K such that x K, F (x) K, x T F (x) = 0 where K denotes the dual cone of K, i.e. K = {d R n, k K, k T d = 0}. Let us note x T y = 0 is equivalent to x is orthogonal to y, usually noted by x y. Furthermore if K is a cone, then CP(K, F ) is equivalent to VI(K, F ). 4.1 Examples and problem reformulation Examples Here are few examples of VI problems. classic complementary problems: when K = R n +, CP(K, F ) reduces to x 0 F (x) 0, i.e. i, x i 0, F (x i ) 0, x i F i (x) = 0. mixed complementary problems: if K = R m R m n + and F (u, v) = (G(u, v) T, H(u, v) T ) T, then G(u, v) = 0, v 0 H(u, v) 0. linear variational inequality problems: F (x) = q + Mx and K be a polyhedral set, a closed rectangle or the positive orthant. link with optimization problem: if we consider min x K x)t θ(x) 0, i.e. VI(K, θ). If θ(x) = q T x xt Mx, then the VIP and the optimization are equivalent if M is symmetric and K is polyhedron. extented KKT system: Let K be {x R n, h(x) = 0, g(x) 0} for h : R n R l and g : R n R m. If x solves VI(K, F ) then there exist µ R l, λ R m such that. Each composant of x and F (x) are complement.

19 4.1 Examples and problem reformulation 19 L(x, µ, λ) = F (x) + j µ j h j (x) + i λ i g i (x) = 0, h(x) = 0, λ 0 g(x) Problem reformulation A necessary tool for VIP and CP is complementarity function. We say ψ : R 2 R is a complementarity function if a, b R 2, ψ(a, b) = 0 a 0, b 0, ab = 0. Here are some examples ψ min (x, y) = min(x, y), ψ FB (x, y) = x 2 + y 2 (x + y). A class of compl. function is the Mangasarian functions: ψ Man (a, b) = ϕ( a b ) ϕ(a) ϕ(b) where ϕ is a strictly increasing function from R to R. Typically, we use ϕ(t) = t and ϕ(t) = t 3. Using this tool, we can reformulate the above extended KKT system as L(x, µ, λ) h(x) = 0. ψ min (λ, g(x)) Another useful tool is the Euclidean projector on K, which for a point x find nearest on K according to the Euclidean distance. That is to say 1 Π K : x arg min y K 2 (y x)t (y x). There are two possibilities to reformulate VI(K, F ) problems: natural mapping: x solves VI(K, F ) FK nat (x) = 0 with F nat K (x) = x Π K(x F (x)), normal mapping: x solves VI(K, F ) z R n, x = Π K (z), FK nor (z) = 0 with F nor K (z) = F (Π K(z)) + z Π K (z). Example for CP(R n +, F ), we have FK nor(z) = F (z +) z The last tool we introduce is the merit functions. A merit function for VI(K, F ) is a function θ : X K R + such that x solves VI(K, F ) x X, θ(x) = 0 x = arg min θ(y) for a closed set X and min θ(y) = 0. y X y X Examples: for VI(R +, F ) we have θ ψ (x) = i ψ(x i, F i (x)) 2 with ψ a compl. function, for VI(K, F ), we have θ gap (x) = supf (x) T (x y). If K is a cone, then θ gap (x) becomes x T F (x). y K

20 20 4 VARIATIONAL INEQUALITY AND COMPLEMENTARITY PROBLEMS 4.2 Algorithms for CPs Problem type: 1. non linear CPs: CP(R n +, F ): Complementarity functions (a) FB based methods: The equation is F FB (x) = 0 and the merit function is θ FB (x) = F FB (x) T F FB (x) where F FB (x) = ψ FB (x 1, F 1 (x)). ψ FB (x n, F n (x)) Tools: for linear Newton approximation scheme T, we choose a matrix H in JacF FB, Set the set B to {i {1,..., n}, x i = 0 = F i (x)}, Choose z R n such that z i 0 for i B, For all columns c of H T, ( ) ( ) x i 1 F e (H T x 2 i + i (x) 1 F ).c = i +F i (x) 2 x 2 i (x) if c / B i +F i (x) ( ) ( 2 ) z i 1 F e z 2 i + i (x) T z 1 F i +( F i (x) T z) 2 z 2 i (x) if c B i +( F i (x) T z) 2. Where e i is the vector with 1 at the ith position. linear CP(K, x q + Mx) Algorithms i. line-search methods: Algo (F FB, θ FB, T ) Init: x 0 R n, ρ > 0, p > 1 and γ ]0, 1[, Iter while x k is not a stationary point of θ FB : select H k in T (x k ) find d k root of F FB (x k ) + H k d = 0 if the equation is not solvable or θ T FB (x k)d k > ρ d k p then d k = θ FB (x k ) find the smallest i k N such that θ FB (x k + 2 i d k ) θ FB (x k ) + γ2 i θ T FB (x k)d k x k+1 = x k + t k d k with t k = 2 i k NB: some variants include further checks on the direction d k.

21 4.2 Algorithms for CPs 21 ii. trust region approach Algo (F FB, θ FB, T ) Init: x 0 R n, 0 < γ 1 < γ 2 < 1, 0, min > 0, Iter: select H k in T (x k ) find d k = arg min d < k q FB (d, x k ) with q FB (d, x) = θ T FB (x)d dt H T k H kd if q FB (d k, x k ) = 0 then stops if θ FB (x k + d k ) θ FB (x k ) + γ 1 q FB (d k, x k ) then { max(2 k, x k+1 = x k + d k and k = min ) if θ FB (x k + d k ) θ FB (x k ) + γ 2 q FB (d k, x k ) max( k, min ) otherwise else x k+1 = x k and k = k /2 NB: a variant includes an additional constraint on the direction d k. iii. constrained methods Algo (F FB, θ FB ) Init: x 0 R n, ρ > 0, p > 1 and γ ]0, 1[ Iter while x k is not a stationary point of θ FB : find y k+1 solution of linear CP(q k, JacF (x k )) and d k = y k+1 x k with q k = F (x k ) JacF (x k )x k, if the equation is not solvable or θ T FB (x k)d k > ρ d k p then d k = min(x k, θ FB (x k )) find the smallest i k N such that θ FB (x k + 2 i d k ) θ FB (x k ) + γ2 i θ T FB (x k)d k x k+1 = x k + t k d k with t k = 2 i k NB: a variant consists in replacing the linear CP by a convex subprogram solved by a Levenberg-Marquardt method. (b) min based methods The equation is F min (x) = 0 and the merit function is θ min (x) = F min (x) T F min (x) where i. line-search method In the following, we use φ(x, d) = i,x i >F i (x) min(x 1, F 1 (x)) F min (x) =.. min(x n, F n (x)) (F i (x) + F i (x) T d) 2 + i,x i F i (x) (x i + d i ) 2 + ρ(θ min(x)) d T d 2

22 22 4 VARIATIONAL INEQUALITY AND COMPLEMENTARITY PROBLEMS and σ(x, d) = F i (x) F i (x) T d + x i d i. i,x i >F i (x) i,x i F i (x) Algorithm 9.2.2( F min, θ min, φ, σ) Init: x 0 R n and γ ]0, 1[ Iter: find d k solution of arg min φ(x k, d) x k +d 0 if d k = 0 then stops find the smallest i k N such that θ min (x k + 2 i d k ) θ min (x k ) γ2 i σ(x k, d k ) x k+1 = x k + t k d k with t k = 2 i k ii. trust region approach not possible since min is not everywhere differentiable iii. mixed compl. func. method Algorithm 9.2.3( F FB, θ FB, F min ) Init: x 0 R n, ɛ > 0, p > 1, ρ > 0 and γ ]0, 1[ Iter while x k is not a stationary point of θ FB : select H k in T min (x k ) d k solves F min (x k ) + H k d = 0 if the system is solvable and F min (x k + d k ) F min (x k ) then x k+1 = x k + t k d k with t k = 1 else if θ T FB (x k)d k > ρ d k p then d k = θ FB (x k ) find the smallest i k N such that θ FB (x k + 2 i d k ) θ FB (x k ) + γ2 i θ T FB (x k)d k x k+1 = x k + t k d k with t k = 2 i k (c) extension to other compl. functions FB based methods can use with other complementarity functions. Here are some examples. ψ LT (a, b) = (a, b) q (a + b) with q > 1 (a b) ψ KK (a, b) = 2 +2qab (a+b) 2 q with 0 q < 2 ψ CCK (a, b) = ψ FB (a, b) qa + b + with q finite lower VIs: CP(R n +, F ): 3. finite upper VIs: CP(R n +, F ): 4. mixed CPs 5. box constrained VIs

23 4.3 Algorithms for VIPs Algorithms for VIPs

24 24 REFERENCES A Bibliography References Boggs, P. T. & Tolle, J. W. (1996), Sequential quadratic programming, Acta Numerica. 3 Bonnans, J. F., Gilbert, J. C., Lemaréchal, C. & Sagastizábal, C. A. (2006), Numerical Optimization: Theoretical and Practical Aspects, Second edition, Springer-Verlag. 3 Conte, S. D. & de Boor, C. (1980), Elementary numerical analysis: an algorithmic approach, Springer International series in pure and applied mathematics. 3 Dennis, J. E. & Schnabel, R. B. (1996), Numerical methods for unconstrained optimization and nonlinear equations, SIAM. 3 Facchinei, F. & Pang, J.-S. (2003a), Finite-Dimensional Variational Inequalities and Complementary Problems. Volume I, Springer- Verlag New York, Inc. 3 Facchinei, F. & Pang, J.-S. (2003b), Finite-Dimensional Variational Inequalities and Complementary Problems. Volume II, Springer- Verlag New York, Inc. 3 Fletcher, R. (2001), On the barzilai-borwein method, Numerical Analysis Report Grantham, W. J. (2005), Gradient transformation trajectory following algorithms for determining stationary min-max saddle points. working paper. 11 Lange, K. (1994), Optimization, Springer-Verlag. 3 Madsen, K., Nielsen, H. B. & Tingleff, O. (2004), Optimization with constraints, Internet. 3 Raydan, M. & Svaiter, B. F. (2001), Relaxed steepest descent and cauchy-barzilai-borwein method. preprint. 3 Roland, C., Varadhan, R. & Frangakis, C. E. (2007), Squared polynomial extrapolation methods with cycling: an application to the positron emission tomography problem, Numerical Algorithms 44(2), Varadhan, R. (2004), Squared extrapolation methods (squarem): a new class of simple and efficient numerical schemes for accelerating the convergence of the em algorithm. working paper. 3 Ye, Y. (1996), Interior-point algorithm: Theory and practice, online. 3

25 25 B Websites Applied mathematics: Decision tree for optimization software: Optimization online: Interior-point algorithms: helmberg/sdp ip.html,