11. Equality constrained minimization

Transcription

1 Convex Optimization Boyd & Vandenberghe 11. Equality constrained minimization equality constrained minimization eliminating equality constraints Newton s method with equality constraints infeasible start Newton method implementation 11 1

2 Equality constrained minimization minimize f(x) subject to Ax = b f convex, twice continuously differentiable A R p n with ranka = p we assume p is finite and attained optimality conditions: x is optimal iff there exists a ν such that f(x )+A T ν = 0, Ax = b Equality constrained minimization 11 2

3 equality constrained quadratic minimization (with P S n +) minimize (1/2)x T Px+q T x+r subject to Ax = b optimality condition: P A T A 0 x ν = q b coefficient matrix is called KKT matrix KKT matrix is nonsingular if and only if Ax = 0, x 0 = x T Px > 0 equivalent condition for nonsingularity: P +A T A 0 Equality constrained minimization 11 3

4 Eliminating equality constraints represent solution of {x Ax = b} as {x Ax = b} = {Fz + ˆx z R n p } ˆx is (any) particular solution range of F R n (n p) is nullspace of A (rankf = n p and AF = 0) reduced or eliminated problem minimize f(fz + ˆx) an unconstrained problem with variable z R n p from solution z, obtain x and ν as x = Fz + ˆx, ν = (AA T ) 1 A f(x ) Equality constrained minimization 11 4

5 example: optimal allocation with resource constraint minimize f 1 (x 1 )+f 2 (x 2 )+ +f n (x n ) subject to x 1 +x 2 + +x n = b eliminate x n = b x 1 x n 1, i.e., choose ˆx = be n, F = I 1 T R n (n 1) reduced problem: minimize f 1 (x 1 )+ +f n 1 (x n 1 )+f n (b x 1 x n 1 ) (variables x 1,..., x n 1 ) Equality constrained minimization 11 5

6 Newton step Newton step x nt of f at feasible x is given by solution v of 2 f(x) A T A 0 v w = f(x) 0 interpretations x nt solves second order approximation (with variable v) minimize f(x+v) = f(x)+ f(x) T v +(1/2)v T 2 f(x)v subject to A(x+v) = b x nt equations follow from linearizing optimality conditions f(x+v)+a T w f(x)+ 2 f(x)v +A T w = 0, A(x+v) = b Equality constrained minimization 11 6

7 Newton decrement properties λ(x) = ( x T nt 2 f(x) x nt ) 1/2 = ( f(x) T x nt ) 1/2 gives an estimate of f(x) p using quadratic approximation f: f(x) inf Ay=b f(y) = 1 2 λ(x)2 directional derivative in Newton direction: d dt f(x+t x nt) = λ(x) 2 t=0 in general, λ(x) ( f(x) T 2 f(x) 1 f(x) ) 1/2 Equality constrained minimization 11 7

8 Newton s method with equality constraints given starting point x domf with Ax = b, tolerance ǫ > 0. repeat 1. Compute the Newton step and decrement x nt, λ(x). 2. Stopping criterion. quit if λ 2 /2 ǫ. 3. Line search. Choose step size t by backtracking line search. 4. Update. x := x + t x nt. a feasible descent method: x (k) feasible and f(x (k+1) ) < f(x (k) ) affine invariant Equality constrained minimization 11 8

9 Newton s method and elimination Newton s method for reduced problem minimize f(z) = f(fz + ˆx) variables z R n p ˆx satisfies Aˆx = b; rankf = n p and AF = 0 Newton s method for f, started at z (0), generates iterates z (k) Newton s method with equality constraints when started at x (0) = Fz (0) + ˆx, iterates are x (k+1) = Fz (k) + ˆx hence, don t need separate convergence analysis Equality constrained minimization 11 9

10 Newton step at infeasible points 2nd interpretation of page 11 6 extends to infeasible x (i.e., Ax b) linearizing optimality conditions at infeasible x (with x dom f) gives 2 f(x) A T xnt f(x) = A 0 w Ax b (1) primal-dual interpretation write optimality condition as r(y) = 0, where y = (x,ν), r(y) = ( f(x)+a T ν,ax b) linearizing r(y) = 0 gives r(y + y) r(y)+dr(y) y = 0: 2 f(x) A T xnt f(x)+a = T ν A 0 Ax b same as (1) with w = ν + ν nt ν nt Equality constrained minimization 11 10

11 Infeasible start Newton method given starting point x domf, ν, tolerance ǫ > 0, α (0,1/2), β (0,1). repeat 1. Compute primal and dual Newton steps x nt, ν nt. 2. Backtracking line search on r 2. t := 1. while r(x + t x nt,ν + t ν nt ) 2 > (1 αt) r(x,ν) 2, t := βt. 3. Update. x := x + t x nt, ν := ν + t ν nt. until Ax = b and r(x,ν) 2 ǫ. not a descent method: f(x (k+1) ) > f(x (k) ) is possible directional derivative of r(y) 2 in direction y = ( x nt, ν nt ) is d dt r(y +t y) 2 = r(y) 2 t=0 Equality constrained minimization 11 11

12 Solving KKT systems H A T A 0 v w g = h solution methods LDL T factorization elimination (if H nonsingular) AH 1 A T w = h AH 1 g, Hv = (g +A T w) elimination with singular H: write as H +A T QA A T A 0 v w g +A = T Qh h with Q 0 for which H +A T QA 0, and apply elimination Equality constrained minimization 11 12

13 Equality constrained analytic centering primal problem: minimize n i=1 logx i subject to Ax = b dual problem: maximize b T ν + n i=1 log(at ν) i +n three methods for an example with A R , different starting points 1. Newton method with equality constraints (requires x (0) 0, Ax (0) = b) 10 5 f(x (k) ) p k Equality constrained minimization 11 13

14 2. Newton method applied to dual problem (requires A T ν (0) 0) 10 5 p g(ν (k) ) infeasible start Newton method (requires x (0) 0) k r(x (k),ν (k) ) Equality constrained minimization k

15 complexity per iteration of three methods is identical 1. use block elimination to solve KKT system diag(x) 2 A T A 0 x w reduces to solving Adiag(x) 2 A T w = b = diag(x) solve Newton system Adiag(A T ν) 2 A T ν = b+adiag(a T ν) use block elimination to solve KKT system diag(x) 2 A T A 0 x ν = diag(x) 1 1 Ax b reduces to solving Adiag(x) 2 A T w = 2Ax b conclusion: in each case, solve ADA T w = h with D positive diagonal Equality constrained minimization 11 15

16 Network flow optimization n minimize i=1 φ i(x i ) subject to Ax = b directed graph with n arcs, p+1 nodes x i : flow through arc i; φ i : cost flow function for arc i (with φ i (x) > 0) node-incidence matrix à R(p+1) n defined as 1 arc j leaves node i à ij = 1 arc j enters node i 0 otherwise reduced node-incidence matrix A R p n is b R p is (reduced) source vector ranka = p if graph is connected à with last row removed Equality constrained minimization 11 16

17 KKT system H A T A 0 v w g = h H = diag(φ 1(x 1 ),...,φ n(x n )), positive diagonal solve via elimination: AH 1 A T w = h AH 1 g, Hv = (g +A T w) sparsity pattern of coefficient matrix is given by graph connectivity (AH 1 A T ) ij 0 (AA T ) ij 0 nodes i and j are connected by an arc Equality constrained minimization 11 17

18 Analytic center of linear matrix inequality variable X S n minimize log det X subject to tr(a i X) = b i, i = 1,...,p optimality conditions X 0, (X ) 1 + p νja i = 0, j=1 tr(a i X ) = b i, i = 1,...,p Newton equation at feasible X: X 1 XX 1 + p w j A i = X 1, j=1 tr(a i X) = 0, i = 1,...,p follows from linear approximation (X + X) 1 X 1 X 1 XX 1 n(n+1)/2+p variables X, w Equality constrained minimization 11 18

19 solution by block elimination eliminate X from first equation: X = X p j=1 w jxa j X substitute X in second equation p tr(a i XA j X)w j = b i, i = 1,...,p (2) j=1 a dense positive definite set of linear equations with variable w R p flop count (dominant terms) using Cholesky factorization X = LL T : form p products L T A j L: (3/2)pn 3 form p(p+1)/2 inner products tr((l T A i L)(L T A j L)): (1/2)p 2 n 2 solve (2) via Cholesky factorization: (1/3)p 3 Equality constrained minimization 11 19