Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14
24 Dual Problems Consider a primal convex optimization problem minimize f (x) subject to g(x) 0 h(x) = 0 where g(x) = (g 1 (x),, g L (x)) and h(x) = (h 1 (x),, h M (x)) Following the Lagrange multiplier method, we form a modified objective function called the Lagrangian Λ(x, λ, µ) = f (x) + L λ l g l (x) + l=1 M µ m h m (x) m=1 = f (x) + λ T g(x) + µ T h(x) where λ = (λ 1,, λ L ) and µ = (µ 1,, µ M ) are called dual variables P Saengudomlert (2015) Optimization Session 4 2 / 14
Define the dual function 1 q(λ, µ) = inf Λ(x, λ, µ) x X where X is the domain set for all functions For λ 0, the dual function provides a lower bound on f To see why, let x F, ie, any primal feasible solution Then, q(λ, µ) = inf x X Λ(x, λ, µ) Λ(x, λ, µ) = f (x ) + λ T g(x ) }{{} + µ T h(x ) f (x ) }{{} 0 for λ 0 =0 1 The infimum (inf) is the largest lower bound The supremum (sup) is the smallest upper bound To see why we need inf in addition to min, consider interval (1, 2) It has no minimum but has the infimum equal to 1 P Saengudomlert (2015) Optimization Session 4 3 / 14
Denote the supremum of the dual function by q = sup q(λ, µ) λ 0,µ R M For λ 0, since q(λ, µ) f (x ) for any feasible x and any optimal solution must be feasible, then q(λ, µ) f Since q is the supremum of q(λ, µ) for all λ 0 and µ R M, q f Theorem 29 (Weak duality theorem): q f P Saengudomlert (2015) Optimization Session 4 4 / 14
Example 23: Minimize f (x) = (x + 1) 2 subject to x 0 The Lagrangian is Λ(x, λ) = (x + 1) 2 λx The dual function is q(λ) = inf x R (x + 1)2 λx = 1 (2 λ)2, 4 The dual function is concave with the maximum equal to 1 Since the primal optimal cost is f = 1, it is clear that q(λ) is a lower bound of f Note that q = 1 dual optimal cost (maximum) 1 2 4 dual feasible set P Saengudomlert (2015) Optimization Session 4 5 / 14
The dual problem is maximize q(λ, µ) subject to λ 0 A dual feasible solution is a point (λ, µ) such that λ 0 and (λ, µ) is in the domain of q D q = {(λ, µ) q(λ, µ) > } Let F d denote the feasible set of the dual problem NOTE: If the dual problem has an optimal solution, q is the optimal cost f q is called the duality gap From weak duality, the duality gap is always nonnegative P Saengudomlert (2015) Optimization Session 4 6 / 14
Problem 26: Let N 2 be a positive integer Let c 1,, c N > 0 Consider the following convex optimization problem minimize subject to N c i x i i=1 N e x i 1 i=1 Write down the dual function in terms of N, c 1,, c N and the dual variables Write down the dual problem P Saengudomlert (2015) Optimization Session 4 7 / 14
25 Lagrange Multipliers Definition (Lagrange multipliers): Dual variables (λ, µ ) F d are called the Lagrange multipliers for the primal problem if λ 0 and f = inf x X Λ(x, λ, µ ) = q(λ, µ ) NOTE: Lagrange multipliers may or may not exist When they exist, strong duality holds, ie, f = q Proof: By definitions of (λ, µ ) and q, f = q(λ, µ ) q From weak duality, q f The two inequalities yield f = q P Saengudomlert (2015) Optimization Session 4 8 / 14
NOTE (cont): Strong duality does not imply existence of Lagrange multipliers When the Lagrange multipliers exist, they may not be unique The above two properties will be demonstrated by upcoming examples If strong duality holds and there is a dual optimal solution, then a dual optimal solution is a set of Lagrange multipliers The above property means that we can first try solving the dual problem If we find a dual optimal solution with zero duality gap, then we have found a set of Lagrange multipliers P Saengudomlert (2015) Optimization Session 4 9 / 14
Example 23: Minimize f (x) = (x + 1) 2 subject to x 0 The Lagrangian is Λ(x, λ) = (x + 1) 2 λx The dual function is q(λ) = 1 (2 λ)2 4 The dual problem is to maximize q(λ) = 1 (2 λ) 2 /4 subject to λ 0 The dual optimal solution is λ = 2, with the dual optimal cost q = 1 Since q = f, there is no duality gap, and λ = 2 is a unique Lagrange multiplier dual optimal cost (maximum) 1 2 4 dual feasible set P Saengudomlert (2015) Optimization Session 4 10 / 14
Example 24: Consider minimizing f (x) = x 1 + x 2 subject to x 1 0 The Lagrangian is The dual function is Λ(x, λ) = x 1 + x 2 λx 1 = (1 λ)x 1 + x 2 q(λ) = inf x R 2(1 λ)x 1 + x 2 = Since D q =, the dual problem is infeasible It follows that there is no Lagrange multiplier P Saengudomlert (2015) Optimization Session 4 11 / 14
Example 25: Consider minimizing f (x) = x subject to x 2 0 Since x = 0 is the only feasible solution, x = 0 is optimal with f = 0 The Lagrangian is Λ(x, λ) = x + λx 2 The dual function is q(λ) = inf x R x + λx 2 = 1 4λ Note that D q = (0, ) The dual problem is then to maximize 1/4λ subject to λ > 0 Since q = sup λ>0 1/4λ = 0, there is no duality gap However, there is no dual optimal solution It follows there there is no Lagrange multiplier P Saengudomlert (2015) Optimization Session 4 12 / 14
Example 26: Consider minimizing f (x) = x subject to x 0 By inspection, x = 0 is optimal with f = 0 The Lagrangian is Λ(x, λ) = x λx The dual function is q(λ) = inf x λx, x R which is 0 for λ [ 1, 1] and is undefined for λ / [ 1, 1] It follows that any λ [0, 1] is dual optimal with q = 0 Since q = f, any λ [0, 1] is a Lagrange multiplier P Saengudomlert (2015) Optimization Session 4 13 / 14
Slator Conditions Several sets of conditions can guarantee the existence of Lagrange multipliers One such set are the Slator conditions Slator conditions: 1 X is a convex set 2 f, g 1,, g L are convex functions 3 h 1,, h M are such that h(x) can be expressed as Ax + b for some matrix A and vector b In addition, A has full rank, ie, rank(a) = M 4 The primal optimal cost f is finite 5 There exists a feasible solution x X such that g(x ) < 0 NOTE: The last conditions says that there is at least one feasible point in the strict interior of the feasible set P Saengudomlert (2015) Optimization Session 4 14 / 14