Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Lagrangian Consider the optimization problem in standard form min f 0 (x) s.t. f i (x) 0, i = 1,,m h i (x) = 0, i = 1,,p variable x R n, domain D = m i=0 domf i p i=1 domh i, optimal value p Lagrangian: L : R n R m R p R, with dom L = D R m R p m p L(x,λ,ν) = f 0 (x)+ λ i f i (x)+ ν i h i (x) i=1 i=1 λ i : Lagrange multiplier associated with f i (x) 0 ν i : Lagrange multiplier associated with h i (x) = 0
Lagrangian dual function Lagrangian dual function: g : R m R p R, g(λ,ν) = inf x D L(x,λ,ν) = inf x D f 0(x)+ m λ i f i (x)+ i=1 g is concave, can be for some λ,ν p ν i h i (x) i=1
Lagrangian dual function: lower bound property Theorem (lower bounds on optimal value) For any λ ર 0 and any ν, we have g(λ,ν) p Proof. Suppose x is a feasible point. Since λ ર 0, m λ i f i ( x)+ i=1 Therefore L( x,λ,ν) f 0 ( x). Hence p ν i h i ( x) 0 i=1 g(λ,ν) = inf x D L(x,λ,ν) L( x,λ,ν) f 0( x) which holds for every feasible x. Thus the lower bound follows.
The dual problem Lagrange dual problem max g(λ,ν) s.t. λ ર 0 find the best lower bound on p a convex optimization problem; optimal value denoted d λ,ν are dual feasible if λ ર 0,(λ,ν) domg (means g(λ,ν) > ) (λ,ν ): dual optimal multipliers p d is called the optimal duality gap
Weak and strong duality Weak duality: d p always true (for both convex and nonconvex problems) Strong duality: d = p does not hold in general (usually) holds for convex problems conditions that guarantee strong duality in convex problems are called constraint qualifications
Slater s constraint qualification Consider the standard convex optimization problem min f 0 (x) s.t. f i (x) 0, i = 1,,m Ax = b with variable x R n, domain D = m i=0 domf i. Slater s condition: exists a point that is strictly feasible, i.e., x relintd such that f i (x) < 0, i = 1,,m, Ax = b (interior relative to affine hull) can be relaxed: affine inequalities do not need to hold with strict inequalities Slater s theorem: The strong duality holds if the Slater s condition holds and the problem is convex.
Complementary slackness Suppose strong duality holds; x is primal optimal; (λ,ν ) is dual optimal f 0 (x ) = g(λ,ν ) = inf x D f 0(x)+ f 0 (x )+ f 0 (x ) m λ i f i(x)+ i=1 m λ i f i (x )+ i=1 Hence the inequalities must hold with equality x minimizes L(x,λ,ν ) λ i f i(x ) = 0 for all i = 1,,m: p νi h i(x) i=1 p νi h i (x ) i=1 λ i = 0 = f i (x ) = 0, f i (x ) < 0 = λ i = 0 known as complementary slackness
Karush-Kuhn-Tucker (KKT) conditions If strong duality holds, x is primal optimal, (λ,ν) is dual optimal, and f i,h i are differentiable, then the following four conditions (called KKT conditions) must hold 1. primal constraints: f i (x) 0,i = 1,,m,h i (x) = 0,i = 1,,p 2. dual constraints: λ ર 0 3. complementary slackness: λ i f i(x ) = 0 4. gradient of Lagrangian w.r.t. x vanishes: f 0 (x)+ m λ i f i (x)+ i=1 p ν i h i (x) = 0 i=1
KKT conditions for convex problem If x, λ, ν satisfy KKT for a convex problem, then they are optimal: from complementary slackness: f 0 ( x) = L( x, λ, ν) from the 4-th condition and convexity: g( λ, ν) = L( x, λ, ν) So f 0 ( x) = g( λ, ν) If the Slater s condition is satisfied and f i is differentiable, then x is optimal iff λ,ν that satisfy KKT
Minimax interpretation Given Lagrangian The primal problem: L(x,λ,ν) = f 0 (x)+λ T f(x)+ν T h(x) (P) inf sup x P λર0,ν L(x,λ,ν) The dual problem: (D) sup λર0,ν inf L(x,λ,ν) x P Weak duality: sup λર0,ν inf x P L(x,λ,ν) inf x P sup λર0,ν L(x,λ,ν)
Saddle point implies strong duality Strong duality: sup λર0,ν inf x P L(x,λ,ν) = inf x P sup λર0,ν L(x,λ,ν) Saddle-point interpretation: (x,λ,ν ) is a saddle point of L if for all λ ર 0,ν,x P. L(x,λ,ν) L(x,λ,ν ) L(x,λ,ν ) The strong duality holds if (x,λ,ν ) a saddle point of L
Examples
Standard form LP (P) min c T x s.t. Ax = b, x ર 0 (D) max b T ν A T ν +c ર 0
Quadratic program (P) min x T Px s.t. Ax b (assume P S n ++ ) (D) max 1 4 λt AP 1 A T λ b T λ s.t. λ ર 0
Equality constrained norm minimization (P) min x s.t. Ax = b (D) max b T ν A T ν 1 Note: y = sup{x T y x 1} is the dual norm of. inf x ( x y T x) = 0 if y 1 and otherwise.
Two-way partitioning (P) min x T Wx s.t. x 2 i = 1, i = 1,,n nonconvex; feasible set: 2 n discrete points partition {1,,n} into two sets; W ij cost of associating i,j to the same set; W ij cost of assigning to different sets (D) max 1 T ν s.t. W +diag(ν) ર 0 lower bound example: ν = λ min (W)1 gives p nλ min (W)
Perturbation and sensitivity anaysis perturbed optimization problem min f 0 (x) s.t. f i (x) u i, i = 1,,m h i (x) = v i, i = 1,,p its dual min g(λ,ν) u T λ v T ν s.t. λ ર 0 x primal variable; u,v are parameters p (u,v) is optimal value as a function of u,v
Global sensitivity anaysis assume strong duality holds for unperturbed problem (u = 0,v = 0), and λ,ν are dual optimal for unperturbed problem By weak duality on the perturbed problem: sensitivity interpretation: p (u,v) g(λ,ν ) u T λ v T ν p (0,0) u T λ v T ν λ i large: small u i = large change in p ν i large and positive: v i < 0 = large increase in p
Local sensitivity analysis: LP Consider LP (P) min c T x s.t. Ax y (D) max y T λ A T λ+c = 0, λ ર 0 The optimal value: p = y T λ. Thus p y i yi=0= λ i
Local sensitivity analysis Suppose strong duality holds for unperturbed problem (u = 0,v = 0), and λ,ν are dual optimal for unperturbed problem. If p (u,v) is differentiable at (0, 0), then λ i = p (0,0) u i ν i = p (0,0) v i If p (u,v) is not differentiable at (0,0), then ( λ, ν ) p (0,0). Proof. By the weak duality on the perturbed problem: p (0,0) u i p (0,0) u i Thus equality holds. Similar proof for ν i p (te i,0) p (0,0) = lim λ i t 0 t p (te i,0) p (0,0) = lim λ i t 0 t
Problems with generalized inequalities Lagrangian min f 0 (x) s.t. f i (x) ર Ki 0, i = 1,,m h i (x) = v i, i = 1,,p λ i : Lagrange multiplier for f i (x) Ki 0 Lagrangian L: L(x,λ 1,,λ m,ν) = f 0 (x)+ Dual function g: m λ T i f i (x)+ i=1 p ν i h i (x) i=1 g(λ 1,,λ m,ν) = inf x D L(x,λ 1,,λ m,ν,x)
Problems with generalized inequalities: dual problem Theorem (lower bound property) λ i ર K i 0, i = 1,,m = g(λ 1,,λ m,ν) p Lagrange dual problem max g(λ 1,,λ m,ν) s.t. λ i ર K i 0, i = 1,,m weak duality: p d p d : optimal duality gap strong duality: p = d for convex problem with constraint qualification. Slater s: primal problem is strictly feasible
Semidefinite program (SDP) Primal SDP where F i,g S k Dual SDP where Z S k. min c T x s.t. x 1 F 1 + +x n F n G max tr(gz) s.t. tr(f i Z)+c i = 0, i = 1,,n Z ર 0 Strong duality if primal SDP is strictly feasible, i.e. x with x 1 F 1 + +x n F n G