Lecture 7: Lagrangian Relaxation and Duality Theory (3 units) Outline Lagrangian dual for linear IP Lagrangian dual for general IP Dual Search Lagrangian decomposition 1 / 23
Joseph Louis Lagrange Joseph Louis Lagrange (1736-1813) Lagrange was one of the creators of the calculus of variations, deriving the Euler-Lagrange equations for extrema of functionals. He also extended the method to take into account possible constraints, arriving at the method of Lagrange multipliers. 2 / 23
Lagrangian Dual for Linear Integer Programming IP with nice and complicated constraints: max c T x s.t. Ax b (nice constraints) Dx d (complicated constraints x Z n +. Observation: Dropping Dx d results in an easy problem. 3 / 23
More general form: (IP) max c T x where Dx d is m hard constraints. Largrangian relaxation: (LP(u)) s.t. Dx d x X, z(u) = max c T x + u T (d Dx) s.t. x X. Property: Problem (LP(u)) is a relaxation of (IP) for all u 0. Lagrnagian dual of (IP): (LD) min z(u). u 0 For equality constraints: Dx = d. The Lagrangain dual is (LD) min u R m z(u). 4 / 23
Example 1 Uncapacitated facility location problem:the uncapacitated facility location problem (UFLP) involves locating an undetermined number of facilities to minimize the sum of the (annualized) fixed setup costs and the variable costs of serving the market demand from these facilities. Formulation: (IP) max c ij x ij f j y j i M j N j N s.t. x ij = 1, i M j N x ij y j 0, i M, j N x R M N, y {0, 1} N. Locations: N = {1,..., n}; Clients: M = {1,..., m}, c ij : profit 5 / 23
Lagrangian relaxation (IP(u)): z(u) = max ij u i )x ij i M j N(c f j y j + u i j N i M s.t. x ij y j 0, i M, j N x R M N, y {0, 1} N. Decomposition: z(u) = z j (u) + i M u i, where z j (u) = max i M(c ij u i )x ij f j y j s.t. x ij y j 0, i M x ij 0, i M, y j {0, 1}. It is a subproblem for location j. We have z j (u) = max{0, max(c ij u i, 0) f j }. i M 6 / 23
Example 2 Generalized assignment problem (GAP). The generalized assignment problem is an excellent example for illustrating Lagrangian relaxation because it is rich with readily apparent structure. Formulation: Z = m n min c ij x ij (1) s.t. i=1 j=1 m x ij = 1, j = 1,..., n (2) i=1 n a ij x ij b i, i = 1,..., m (3) j=1 x ij {0, 1}, for all i, j. (4) 7 / 23
Relaxations 1: Z D1 (u) = min m i=1 j=1 n c ij x ij + n m u j ( x ij 1) j=1 s.t. (3), (4) m n = min (c ij + u j )x ij i=1 j=1 s.t. (3), (4). i=1 n j=1 This problem reduces to m 0-1 knapsack problems and can thus be solved in time proportional to n m i=1 b i. u j 8 / 23
Relaxation 2 Z D2 (v) = min m i=1 j=1 n c ij x ij + m n v i ( a ij x ij b i ) i=1 s.t. (2), (4) n m = min (c ij + v i a ij )x ij j=1 i=1 s.t. (2), (4). j=1 m v i b i Since constraints (2) are generalized upper bound (GUB) constraints, it is 0-1 GUB problem. Such a problem is easily solved in time proportional to nm by determining min i (c ij + v i a ij ) for each j and setting the associated x ij = 1. Remaining x ij are set to zero. Classical paper: Marshall L. Fisher, The Lagrangian Relaxation Method for Solving Integer Programming Problems, Manage. Sci., Vol. 50, No. 12 Supplement, December 2004, pp. 1861 1871. (Original, MS, Vol. 27(1), 1-18, 1981). i=1 9 / 23
How Good is Lagrangian Dual? Consider linear program: min{c T x Ax b, Cx d, x X }, where X is a finite integer set, e.g., X = {0, 1} n. Let {x 1,..., x N } be the finite integer points in {x X Cx d}. Dual problem: v(d) = max λ 0 d(λ) = max λ 0 min x X [ct x + λ T (Ax d)] = max min λ 0 i=1,...,n [ct x i + λ T (Ax i b)] = max η s.t. c T x i + λ T (Ax i b) η, i = 1,..., N, λ 0, η R. 10 / 23
The latter problem is a linear problem. Taking its dual gives v(d) = min s.t. N µ i (c T x i ) i=1 N µ i (Ax i d) 0, i=1 N µ i = 1, µ i 0. i=1 Setting x = N i=1 µ ix i, we imply that (D) is equivalent to the following LP: (PR) min c T x s.t. Ax b, x Co({x X Cx d}). 11 / 23
Theorem: v(d) = v(pr) = min{c T x Ax b, x Co({x X Cx d})}. v(d) v(lp) = min{c T x Ax b, Cx d}). If Co({x X Cx b}) = {x X Cx b}, then v(d) = v(lp). Lagrangian bound is equal to the LP relaxation. Example: {x Z n + l x u}. 12 / 23
Figure: Lagrangian bound for linear integer programming. 13 / 23
Key issues in Lagrangian dual How to solve the Lagrangian relaxation? How to solve the dual problem? When Lagrangian relaxation useful? 14 / 23
Solving Relaxation Problem Linear integer program; Many combinatorial optimization problems separable integer programming (knapsack problem) 0-1 unconstrained quadratic problem 0-1 quadratic knapsack program 15 / 23
Dual Search Methods Problem: How to solve the dual problem (D)? (D) max d(λ). λ 0 Three Basic Methods: Subgradient method Outer approximation method Bundle method 16 / 23
Subgradient method Subgradient method λ k+1 = P + (λ k + s k ξ k / ξ k ), where s k > 0 is the stepsize. Stepsize rules for s k : Rule 1 for stepsize (constant): s k = ɛ, where ɛ > 0 is a constant. Rule 2 for stepsize: + s 2 k < + and + k=1 k=1 Rule 3 for stepsize: s k 0, k +, and s k = +. + k=1 s k = +. 17 / 23
A sophisticated stepsize rule: s k = ρ w k d(λ k ) ξ k, 0 < ρ < 2, where w k is an approximation of the optimal value v(d), w k d(λ k ) and ξ k 0. 18 / 23
Outer approximation method Rewrite dual problem (D) as max µ s.t. µ f (x) + λ T (g(x) b), x X, λ 0. Approximate the LP by: max µ s.t. µ f (x j ) + λ T (g(x j ) b), x j T k, λ 0. Finite convergence Singly constrained case of (P): (P s ) min f (x) s.t. g(x) b x X. 19 / 23
8 6 4 y 2 d(λ) 0 l 3 2 l 1 λ 1 =5/3 λ 2 =2 l2 4 0 0.5 1 1.5 2 2.5 3 3.5 4 λ Figure: Dual function. 20 / 23
Lagrangian decomposition Purpose: Induce decomposition of the problem into independent subproblems; Capture different structural characteristics of the problem; Obtain stronger bounds than by standard Lagrangean relaxation schemes How? Identify the parts of the problem that should be split; Replace variables in each part by copies or substitute new expressions; Dualize the copy or substitution expression. 21 / 23
Convex program with nice and hard constraints: (CP) min f (x) + g(x) s.t. x X (nice constraints) x Y (hard constraints) Lagrangian decomposition via copying constraints (CP) min f (x) + g(z) s.t. x X (nice constraints) x = z, (link constraints) z Y (hard constraints) 22 / 23
Associating a multiplier to each link constraint: (CP) min f (x) + g(z) s.t. x X λ x = z, z Y Decomposition: d(λ) = min f (x) + g(z) + λ T (x z) s.t. x X z Y = min f (x) + λ T x + min g(z) λ T z s.t. x X, s.t. z Y = d 2 (λ) + d 2 (λ) two easier subproblems! 23 / 23