Applied Lagrange Duality for Constrained Optimization


 Rose Arnold
 1 years ago
 Views:
Transcription
1 Applied Lagrange Duality for Constrained Optimization February 12, 2002
2 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual Problem ffl Steps in the Construction of the Dual Problem ffl Examples of Dual Constructions ffl cfl2002 Massachusetts Institute of Technology. All rights reserved
3 Overview The Dual is a Concave Maximization Problem ffl Weak Duality ffl The Column Geometry of the Primal and Dual Problems ffl Strong Duality ffl ffl Duality Strategies ffl Illustration of Lagrange Duality in Discrete Optimization cfl2002 Massachusetts Institute of Technology. All rights reserved
4 Context of Duality Previous Examples of Duality Models of electrical networks ffl Models of economic markets ffl Structural design ffl Duality in Linear Optimization ffl cfl2002 Massachusetts Institute of Technology. All rights reserved
5 Context of Duality Importance of Duality Identifying nearoptimal solutions ffl Proving optimality ffl Sensitivity analysis of the primal problem ffl KarushKuhnTucker (KKT) conditions ffl Convergence of improvement algorithms ffl Good Structure ffl Other uses, too... ffl cfl2002 Massachusetts Institute of Technology. All rights reserved
6 Review of Convexity Local and Global Optima Problem SetUp The ball centered at μx with radius ffl is the set: B(μx; ffl) := fxjkx μxk» fflg: Consider the following optimization problem over the set F: P : min x or max x f(x) s.t. x 2 F cfl2002 Massachusetts Institute of Technology. All rights reserved
7 Review of Convexity Local and Global Optima Definition x 2 F 1 is a local minimum P of if there ffl > 0 exists such f(x)» f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 2 is a global minimum P of if» f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved
8 Review of Convexity Strict Local & Global Minima Definition x 2 F 3 is a strict local minimum P of if there ffl > 0 exists such f(x) < f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 4 is a strict global minimum P of if < f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved
9 Review of Convexity Local & Global Maxima Definition x 2 F 5 is a local maximum P of if there ffl > 0 exists such f(x) f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 6 is a global maximum P of if f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved
10 Review of Convexity Strict Local & Global Maxima Definition x 2 F 7 is a strict local maximum P of if there ffl > 0 exists such f(x) > f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 8 is a strict global maximum P of if > f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved
11 Review of Convexity Local versus Global Minima F Illustration of local versus global optima. cfl2002 Massachusetts Institute of Technology. All rights reserved
12 Review of Convexity Definition 9 Convex Sets and Functions A subset S ρ < n is a convex set if y 2 S ) x + (1 )y 2 S x; for 2 [0; any 1]. Illustration of convex and nonconvex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved
13 Review of Convexity Intersections of Convex Sets Proposition 1 S; T If are convex sets, S T then is a convex set. Proposition 2 The intersection of any collection of convex sets is a convex set. Illustration of the intersection of convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved
14 Review of Convexity Convex Functions Definition 10 A function f(x) is a convex function if + (1 )y)» f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 11 function if A function f(x) is a strictly convex + (1 )y) < f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved
15 Review of Convexity Convex Functions Illustration of convex and strictly convex functions. cfl2002 Massachusetts Institute of Technology. All rights reserved
16 Review of Convexity Convex Optimization P : minimize x f(x) x 2 F s:t: Proposition 3 Suppose F that is a convex set, : F! < is a convex function, and μx is a local f minimum P of. μx Then is a global minimum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved
17 Review of Convexity Proof of Proposition Proof of the Proposition: Suppose μx is not a global minimum, i.e., there exists y 2 F for which f(y) < f(μx). Let = μx + (1 )y, which is a convex combination of μx and y y( ) 2 [0; 1] for (and y( ) 2 F therefore, 2 [0; for 1]). Note that! μx as! 1. y( ) From the convexity of f( ), = f( μx + (1 )y)» f(μx) + (1 )f(y) < f(μx) + (1 )f(μx) = f(μx) f(y( )) for 2 (0; all 1). f(y( )) < f(μx) Therefore, for 2 (0; all 1), and μx so is not a local minimum, resulting in a contradiction. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved
18 Review of Convexity Examples of Convex Functions Functions of One Variable Examples of convex functions of one variable: ffl f(x) = ax + b ffl f(x) = x 2 + bx + c ffl f(x) = jxj ffl f(x) = ln(x) for x > 0 for = 1 x x > 0 ffl f(x) = e x f(x) ffl cfl2002 Massachusetts Institute of Technology. All rights reserved
19 Review of Convexity Concave Functions & Maximization The opposite of a convex function is a concave function: Definition 12 A function f(x) is a concave function if + (1 )y) f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 13 A function f(x) is a strictly concave function if + (1 )y) > f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved
20 Review of Convexity Concave Functions & Maximization Illustration of concave and strictly concave functions. cfl2002 Massachusetts Institute of Technology. All rights reserved
21 Review of Convexity Concave Maximization Now consider the maximization problem P : P : maximize x f(x) s:t: x 2 F Proposition 4 Suppose F that is a convex set, : F! < is a concave function, and μx is a local f maximum P of. μx Then is a global maximum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved
22 Review of Convexity Convex Program : min x or max x f(x) P x 2 F s.t. We call P a convex program if ffl F is a convex set, and we are minimizing and is a convex function, or ffl we are maximizing and f(x)is a concave function. Then every local optima of the objective function of a convex program is a global optima of the objective function. cfl2002 Massachusetts Institute of Technology. All rights reserved
23 Review of Convexity Convex Program The class of convex programs is the class of wellbehaved optimization problems. cfl2002 Massachusetts Institute of Technology. All rights reserved
24 Review of Convexity Linear Functions Proposition 5 A linear f(x) = a function x + b is both convex and concave. T Proposition 6 f(x) If is both convex and concave, f(x) then is a linear function. A linear function is convex and concave. cfl2002 Massachusetts Institute of Technology. All rights reserved
25 Separating Hyperplane Theory Strong Separation Theorem 1 Strong Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S. Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x > ff for all x 2 S u T μx < ff cfl2002 Massachusetts Institute of Technology. All rights reserved
26 Separating Hyperplane Theory Strong Separation S x utx = α Illustration of strong separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved
27 Separating Hyperplane Theory Weak Separation Theorem 2 Weak Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S or μx Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x ff for all x 2 S u T μx» ff cfl2002 Massachusetts Institute of Technology. All rights reserved
28 Separating Hyperplane Theory Weak Separation S x utx = α Illustration of weak separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved
29 Separating Hyperplane Theory Intersecting Line Dual Primal: minimum x height of x s:t: 2 S; x 2 L x Dual: maximum ß intercept of line ß with L s:t: S ß line lies below cfl2002 Massachusetts Institute of Technology. All rights reserved
30 Separating Hyperplane Theory Intersecting Line Dual z L S π Illustration intersecting line dual. r cfl2002 Massachusetts Institute of Technology. All rights reserved
31 OP : minimum x Definition of the Dual Problem General Constrained Problem Most General Format f(x) g 1 (x)» 0; s:t: =. m (x)» 0; g 2 P; x : < n 7! < f(x) i (x) : < n 7! <; i = 1; : : : ; m. g cfl2002 Massachusetts Institute of Technology. All rights reserved
32 ffl P = < n OP : z Λ = minimum x ffl P = Φ x j x 2 Z n + Ψ m (x)» 0; g 2 P; x Definition of the Dual Problem General Constrained Problem Standard Format... f(x) g 1 (x)» 0; s:t:. ffl P = fx j x 0g ffl P = fx j g i (x)» 0; i = m + 1; : : : ; m + kg P = fx j Ax» bg ffl cfl2002 Massachusetts Institute of Technology. All rights reserved
33 OP : z Λ = minimum x L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format... f(x) Form the Lagrangian function: g i (x)» 0; i = 1; : : : ; m; s:t: 2 P; x Solve the presumably easier problem: i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The L function (u) is called the dual function. Λ We presume that computing L Λ (u) is an easy task. cfl2002 Massachusetts Institute of Technology. All rights reserved
34 L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format Form the Lagrangian function: Form the dual function L Λ (u): i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The dual problem is then defined to be: D : v Λ = maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved
35 OP : z Λ = minimum x Steps in the Construction of the Dual Problem f(x) Step 1. Create the Lagrangian g i (x)» 0; i = 1;:::;m; s:t: 2 P; x u) := f(x) + u T g(x) : L(x; Step 2. Create the dual function: L Λ (u) := minimum x f(x) + u T g(x) Step 3. Create the dual problem: s:t: x 2 P D : v Λ = maximum u L Λ (u) s:t: u 0 cfl2002 Massachusetts Institute of Technology. All rights reserved
36 LP : minimum x Examples of Dual Constructions of OPs The Dual of a Linear Problem Consider the linear optimization problem: c T x s:t: Ax b What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved
37 IP : minimum x The Dual of a Binary Integer Problem Examples of Dual Constructions of OPs Consider the binary integer optimization problem: c T x s:t: b Ax j 2 f0; 1g; j = 1; : : : ; n : x What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved
38 BP : minimum x1 ;x 2 ;x 3 5x 1 + 7x 2 4x 3 3 P Examples of Dual Constructions of OPs The Dual of a LogBarrier Problem Consider the following logarithmic barrier problem: ln(x j ) j=1 s.t. x 1 + 3x x 3 = 37 x 1 > 0; x 2 > 0; x 3 > 0 : What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved
39 OP : minimum x f(x) + P i2l u i g i (x) + P u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints Remarks... f(x) s:t: i (x)» 0; i 2 L g i (x) 0; i 2 G g g i (x) = 0; i 2 E The Lagrangian is: x 2 P; L(x; u) := f(x) + u T g(x) = u i g i (x) cfl2002 Massachusetts Institute of Technology. All rights reserved i2g i2e
40 u) := f(x) + u T g(x) = f(x) + P L(x; u i g i (x) + P i2l D : v Λ = maximum u u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints...Remarks u i g i (x) The dual function L Λ (u): i2g i2e L Λ (u) := minimum x f(x) + u T g(x) The dual problem is: s:t: x 2 P L Λ (u) s:t: 0; i 2 L u» 0; i 2 G u > < 0; i 2 E u cfl2002 Massachusetts Institute of Technology. All rights reserved
41 OP : z Λ = minimum x The Dual is a Concave Maximization Problem f(x) s:t: g i (x)» 0; i = 1; : : : ; m x 2 P; L(x; u) := f(x) + u T g(x) L Λ (u) := minimum x f(x) + u T g(x) s:t: x 2 P Theorem 3 The dual function L Λ (u) is a concave function. cfl2002 Massachusetts Institute of Technology. All rights reserved
42 = min x2p f(x) + u T 1 g(x) Λ + (1 ) f(x) + u T 2 g(x) Λ min x2p f(x) + u T 1 g(x) Λ + (1 ) min x2p (f(x) + u T 2 g(x) Λ The Dual is a Concave Maximization Problem Proof of Concavity of L Λ (u) Proof: Let u 1 0 and u 2 0 be two values of the dual variables, and let u = u 1 + (1 )u 2, where 2 [0; 1]. Then L Λ (u) = min x2p f(x) + u T g(x) = L Λ (u 1 ) + (1 )L Λ (u 2 ) : Therefore L Λ (u) is a concave function. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved
43 OP : z Λ = minimum x g i (x)» 0; i = 1;:::;m s:t: 2 P; x Weak Duality f(x) D : v Λ = maximum u L Λ (u) s:t: u 0 Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) Λ v Λ : z cfl2002 Massachusetts Institute of Technology. All rights reserved
44 Weak Duality Proof of Weak Duality Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) z Λ v Λ : Proof: If μx is feasible for OP and μu is feasible for D, then + T g(μx) min x2p f(x) + μu T g(x) = L Λ (μu) : f(μx) f(μx) μu v Therefore. q.e.d. Λ z Λ cfl2002 Massachusetts Institute of Technology. All rights reserved
45 OP : z Λ = minimum x A = 1 (x) g 2 (x) g m (x) g f(x) A = Column Geometry of Primal and Dual Problems Resources and Costs For this section only, we assume equality format: f(x) g i (x) = 0; i = 1;:::;m s:t: 2 P; x D : v Λ = maximum u L Λ (u) s:t: u > < 0 For each x 2 P, we have an array of resources and costs associated with x: r z 1 r 2 r B = B. C C B B. C C g(x) r m z cfl2002 Massachusetts Institute of Technology. All rights reserved
46 Column Geometry of Primal and Dual Problems Resources and Costs S := Φ (r; z) 2 < m+1 j (r; z) = (g(x); f(x)) for some x 2 P Ψ : z L = {( r, z ) r = 0 } S H r 1 r 2 r = 0 The column geometry of the primal and dual problem. cfl2002 Massachusetts Institute of Technology. All rights reserved
47 Column Geometry of Primal and Dual Problems Column Geometry Interpretation of OP Every point in the intersection of S and L corresponds to a feasible solution of OP, and the feasible solution with the lowest cost corresponds to the lowest point in the intersection of S and L. This lowest value is exactly the value of the primal problem, namely z Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved
48 Column Geometry of Primal and Dual Problems Supporting Hyperplanes of S H = H u;ff = Φ (r; z) 2 < m+1 j z + u T r = ff Ψ : L = Φ (r; z) 2 < m+1 j r = 0 Ψ : H L = f(r; z) = (0; ff)g cfl2002 Massachusetts Institute of Technology. All rights reserved
49 maximum u;ff Column Geometry of Primal and Dual Problems MaximumHeight Supporting Hyperplane of S ff s:t: H u;ff lies below S = maximum u;ff ff s:t: z + u T r ff; for all (r; z) 2 S = maximum u;ff ff s:t: f(x) + u T g(x) ff; for all x 2 P cfl2002 Massachusetts Institute of Technology. All rights reserved
50 maximum u;ff > 0 : < Column Geometry of Primal and Dual Problems MaximumHeight Supporting Hyperplane of S ff s:t: f(x) + u T g(x) ff; for all x 2 P = maximum u;ff ff s:t: L Λ (u) ff; = maximum u L Λ (u) s:t: u cfl2002 Massachusetts Institute of Technology. All rights reserved
51 Column Geometry of Primal and Dual Problems MaximumHeight Supporting Hyperplane of S The dual problem corresponds to finding that hyperplane H u;ff lying below S whose intersection with L is the highest. This highest value is exactly the value of the dual problem, namely v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved
52 Strong Duality Convexity and Strong Duality z L S r = 0 H r The column geometry of the dual problem when S is convex. cfl2002 Massachusetts Institute of Technology. All rights reserved
53 Strong Duality Absence of Convexity z L z* v* S H r The column geometry of the dual problem when S is not convex. cfl2002 Massachusetts Institute of Technology. All rights reserved
54 Strong Duality Strong Duality Theorem Theorem 5 Strong Duality Theorem: Suppose that OP is a convex optimization problem, that is, all constraints of OP are of the form g i (x)» 0; i = 1; : : : ; m, f(x) as well as g i (x) are convex functions, i = 1; : : : ; m, and P is a convex set. Then under very mild additional conditions, z Λ = v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved
55 Duality Strategies Dualizing Bad Constraints OP : minimum x c T x s:t:» b Ax» g : Nx Suppose that optimization over the constraints Nx» g is easy. The addition of the constraints Ax» b makes the problem much more difficult. Let = fx j Nx» gg P cfl2002 Massachusetts Institute of Technology. All rights reserved
56 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: Nx» g : OP : minimum x c T x Ax» b s:t: 2 P : x cfl2002 Massachusetts Institute of Technology. All rights reserved
57 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: where x 2 P : The Lagrangian is: L(x; u) = c T x + u T (Ax b) = u T b + (c T + u T A)x L Λ (u) := minimum x u T b + (c T + u T A)x s:t: x 2 P : cfl2002 Massachusetts Institute of Technology. All rights reserved
58 Λ (u) := minimum x u T b + (c T + u T A)x L x 2 P : s:t: Duality Strategies Dualizing Bad Constraints D : maximum u L Λ (u) s:t: u 0 Notice that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem D. cfl2002 Massachusetts Institute of Technology. All rights reserved
59 OP : minimum x1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A 2 x 2» b 2 A Notice here that if it were not for the constraints 1 x 1 + B 2 x 2» d, that we would be able to separate B the problem into two separate problems. Let us dualize on these constraints. Let: P = Φ (x 1 ; x 2 ) j A 1 x 1» b 1 ; A 2 x 2» b 2Ψ cfl2002 Massachusetts Institute of Technology. All rights reserved
60 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 A 2 x 2» b 2 P = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A B 1 x 1 +B 2 x 2» d s:t: 1 ; x 2 ) 2 P (x cfl2002 Massachusetts Institute of Technology. All rights reserved
61 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 L Λ (u) = minimum x1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 ;x 2 ) 2 P (x = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ P The Lagrangian is: L(x; u) = (c 1 ) T x 1 + (c 2 ) T x 2 + u T (B 1 x 1 + B 2 x 2 d) = u T d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 ; (x 1 ; x 2 ) 2 P s:t: cfl2002 Massachusetts Institute of Technology. All rights reserved
62 L Λ (u) = minimum x 1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems s:t: (x 1 ;x 2 ) 2 P L Λ (u) = T d u minimum A1 x 1»b 1 ((c1 ) T + u T B 1 )x minimum A 2 x 2»b 2 ((c2 ) T + u T B 2 )x 2 Notice once again that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem: D : maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved
63 Lagrange Duality Illustrated Constrained Shortest Path Problem Definition Constrained Shortest Path Problem: find the shortest path from node to node 6, among those paths using 4 exactly arcs cfl2002 Massachusetts Institute of Technology. All rights reserved
64 Lagrange Duality Illustrated Constrained Shortest Path Problem Context In practice: Arcs might have two types of costs (time and money) There may be millions of nodes and billions of paths cfl2002 Massachusetts Institute of Technology. All rights reserved
65 Lagrange Duality Illustrated Constrained Shortest Path Enumeration of Paths Path Number of Arcs Cost Path is the optimal solution. cfl2002 Massachusetts Institute of Technology. All rights reserved
66 Lagrange Duality Illustrated Constrained Shortest Path Motivation for Dual Suppose that we have available an efficient algorithm for solving (unconstrained) shortest path problems. We will use this algorithm to try to solve our problem through the dual cfl2002 Massachusetts Institute of Technology. All rights reserved
67 z Λ = min xij x 12 +x 13 +x 23 +x 24 +3x 34 +2x 35 +x 54 +x 46 +3x 56 Lagrange Duality Illustrated Constrained Shortest Path IP Formulation... CSPP : s.t. x 12 +x 13 +x 23 +x 24 +x 34 +x 35 +x 54 +x 46 +x 56 = 4 x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved
68 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation... P i;j P x ij = 4 s:t: i;j Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved
69 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation P i;j P x ij = 4 s:t: i;j where: x 2 P P = fx j Nx = b ; x ij 2 f0; 1g for all arcs i jg cfl2002 Massachusetts Institute of Technology. All rights reserved
70 L(x; u) := P i;j 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path The Lagrangian ψ x ij! c ij x ij + u cfl2002 Massachusetts Institute of Technology. All rights reserved
71 L Λ (u) := minimum xij 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path L Λ (u)... ψ P c ij x ij + u x ij! i;j s:t: Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved
72 Lagrange Duality Illustrated Constrained Shortest Path...L Λ (u) Λ (u) = min xij (1 u)x 12 +(1 u)x 13 +(1 u)x 23 L u)x 24 +(3 u)x 34 +(2 u)x 35 +(1 +(1 u)x 54 +(1 u)x 46 +(3 u)x 56 +4u s.t. x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved
73 Lagrange Duality Illustrated Constrained Shortest Path Comments on L Λ (u) For a fixed value of u, 4u is a constant. For a fixed value of u, this is a shortest path problem with modified arc costs given by: c 0 ij = c ij u : cfl2002 Massachusetts Institute of Technology. All rights reserved
74 Lagrange Duality Illustrated Constrained Shortest Path The Dual Problem D : v Λ = maximum u L Λ (u) s.t. u < > 0 cfl2002 Massachusetts Institute of Technology. All rights reserved
75 Lagrange Duality Illustrated Constrained Shortest Path The set P Recall that P is the set of solutions of: x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved
76 Lagrange Duality Illustrated Constrained Shortest Path Resources and Costs z (cost) conv(s) r (number of arcs) z* = 5 v* = 4.5 cfl2002 Massachusetts Institute of Technology. All rights reserved
77 Lagrange Duality Illustrated Constrained Shortest Path Duality Gap ffl The optimal u is u = 1:5 ffl The value of the dual is v Λ = 4:5 ffl The value of the primal is z Λ = 5:0 ffl There is a duality gap: z Λ v Λ = 5:0 4:5 = 0:5. cfl2002 Massachusetts Institute of Technology. All rights reserved
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 00 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More informationOptimization Theory. Lectures 46
Optimization Theory Lectures 46 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the nonnegative orthant {x0ú n x$0} Existence of a maximum:
More informationThe KuhnTucker Problem
Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming  The KuhnTucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The KuhnTucker
More informationLagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark
Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationNonlinear Programming (NLP)
Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming  Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume
More informationPrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method
PrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationLagrangian Duality for Dummies
Lagrangian Duality for Dummies David Knowles November 13, 2010 We want to solve the following optimisation problem: f 0 () (1) such that f i () 0 i 1,..., m (2) For now we do not need to assume conveity.
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a realvalued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationDuality of LPs and Applications
Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationQuiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006
Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in
More informationSECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING
Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More informationMathematical Foundations 1 Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7
Mathematical Foundations  Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum
More informationLecture 9: DantzigWolfe Decomposition
Lecture 9: DantzigWolfe Decomposition (3 units) Outline DantzigWolfe decomposition Column generation algorithm Relation to Lagrangian dual Branchandprice method Generated assignment problem and multicommodity
More informationLecture #21. c T x Ax b. maximize subject to
COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationThe KarushKuhnTucker (KKT) conditions
The KarushKuhnTucker (KKT) conditions In this section, we will give a set of sufficient (and at most times necessary) conditions for a x to be the solution of a given convex optimization problem. These
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of WisconsinMadison May 2014 Wright (UWMadison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More information8 Barrier Methods for Constrained Optimization
IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality
More informationLP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra
LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality
More informationPrimalDual InteriorPoint Methods
PrimalDual InteriorPoint Methods Lecturer: Aarti Singh Coinstructor: Pradeep Ravikumar Convex Optimization 10725/36725 Outline Today: Primaldual interiorpoint method Special case: linear programming
More informationSolution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark
Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT
More informationLecture 15 Newton Method and SelfConcordance. October 23, 2008
Newton Method and SelfConcordance October 23, 2008 Outline Lecture 15 Selfconcordance Notion Selfconcordant Functions Operations Preserving Selfconcordance Properties of Selfconcordant Functions Implications
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions HyangWon Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationIE 5531 Midterm #2 Solutions
IE 5531 Midterm #2 s Prof. John Gunnar Carlsson November 9, 2011 Before you begin: This exam has 9 pages and a total of 5 problems. Make sure that all pages are present. To obtain credit for a problem,
More informationSymmetric and Asymmetric Duality
journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,
More informationNonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6
Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms  A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationSTATIC LECTURE 4: CONSTRAINED OPTIMIZATION II  KUHN TUCKER THEORY
STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II  KUHN TUCKER THEORY UNIVERSITY OF MARYLAND: ECON 600 1. Some Eamples 1 A general problem that arises countless times in economics takes the form: (Verbally):
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework  MC/MC Constrained optimization
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to CyberPhysical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationz = f (x; y) f (x ; y ) f (x; y) f (x; y )
BEEM0 Optimization Techiniques for Economists Lecture Week 4 Dieter Balkenborg Departments of Economics University of Exeter Since the fabric of the universe is most perfect, and is the work of a most
More informationDate: July 5, Contents
2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........
More informationLecture 1: Background on Convex Analysis
Lecture 1: Background on Convex Analysis John Duchi PCMI 2016 Outline I Convex sets 1.1 Definitions and examples 2.2 Basic properties 3.3 Projections onto convex sets 4.4 Separating and supporting hyperplanes
More informationWeek 4: Calculus and Optimization (Jehle and Reny, Chapter A2)
Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) TsunFeng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization
More informationTutorial on Convex Optimization for Engineers Part II
Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D98684 Ilmenau, Germany jens.steinwandt@tuilmenau.de
More informationEC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline
LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for
More informationOn the Method of Lagrange Multipliers
On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should
More informationELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications
ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February
More informationSupport Vector Machines
Wien, June, 2010 Paul Hofmarcher, Stefan Theussl, WU Wien Hofmarcher/Theussl SVM 1/21 Linear Separable Separating Hyperplanes NonLinear Separable SoftMargin Hyperplanes Hofmarcher/Theussl SVM 2/21 (SVM)
More informationDeterminant maximization with linear. S. Boyd, L. Vandenberghe, S.P. Wu. Information Systems Laboratory. Stanford University
Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition
More informationCS , Fall 2011 Assignment 2 Solutions
CS 940, Fall 20 Assignment 2 Solutions (8 pts) In this question we briefly review the expressiveness of kernels (a) Construct a support vector machine that computes the XOR function Use values of + and
More informationMath 273a: Optimization Convex Conjugacy
Math 273a: Optimization Convex Conjugacy Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Convex conjugate (the Legendre transform) Let f be a closed proper
More informationInteger Programming ISE 418. Lecture 12. Dr. Ted Ralphs
Integer Programming ISE 418 Lecture 12 Dr. Ted Ralphs ISE 418 Lecture 12 1 Reading for This Lecture Nemhauser and Wolsey Sections II.2.1 Wolsey Chapter 9 ISE 418 Lecture 12 2 Generating Stronger Valid
More informationLinear Programming: Simplex
Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of WisconsinMadison. IMA, August 2016 Stephen Wright (UWMadison) Linear Programming: Simplex IMA, August 2016
More informationComputational Integer Programming. Lecture 2: Modeling and Formulation. Dr. Ted Ralphs
Computational Integer Programming Lecture 2: Modeling and Formulation Dr. Ted Ralphs Computational MILP Lecture 2 1 Reading for This Lecture N&W Sections I.1.1I.1.6 Wolsey Chapter 1 CCZ Chapter 2 Computational
More informationSupport Vector Machine (SVM) and Kernel Methods
Support Vector Machine (SVM) and Kernel Methods CE717: Machine Learning Sharif University of Technology Fall 2015 Soleymani Outline Margin concept HardMargin SVM SoftMargin SVM Dual Problems of HardMargin
More informationCONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES. Contents 1. Introduction 1 2. Convex Sets
CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES DANIEL HENDRYCKS Abstract. This paper develops the fundamentals of convex optimization and applies them to Support Vector
More informationLecture 7: Weak Duality
EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More informationSection 6.1: Composite Functions
Section 6.1: Composite Functions Def: Given two function f and g, the composite function, which we denote by f g and read as f composed with g, is defined by (f g)(x) = f(g(x)). In other words, the function
More informationInterior Point Methods for Linear Programming: Motivation & Theory
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Linear Programming: Motivation & Theory Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationCS 6820 Fall 2014 Lectures, October 320, 2014
Analysis of Algorithms Linear Programming Notes CS 6820 Fall 2014 Lectures, October 320, 2014 1 Linear programming The linear programming (LP) problem is the following optimization problem. We are given
More informationInteger Programming ISE 418. Lecture 2. Dr. Ted Ralphs
Integer Programming ISE 418 Lecture 2 Dr. Ted Ralphs ISE 418 Lecture 2 1 Reading for This Lecture N&W Sections I.1.1I.1.6 Wolsey Chapter 1 CCZ Chapter 2 ISE 418 Lecture 2 2 Formulations and Models Our
More informationChapter 1: Linear Programming
Chapter 1: Linear Programming Math 368 c Copyright 2013 R Clark Robinson May 22, 2013 Chapter 1: Linear Programming 1 Max and Min For f : D R n R, f (D) = {f (x) : x D } is set of attainable values of
More information Wellcharacterized problems, minmax relations, approximate certificates.  LP problems in the standard form, primal and dual linear programs
LPDuality ( Approximation Algorithms by V. Vazirani, Chapter 12)  Wellcharacterized problems, minmax relations, approximate certificates  LP problems in the standard form, primal and dual linear programs
More informationEC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes.
London School of Economics Department of Economics Dr Francesco Nava Offi ce: 32L.3.20 EC400 Math for Microeconomics Syllabus 2016 The course is based on 6 sixty minutes lectures and on 6 ninety minutes
More informationMATINF4110/MATINF9110 Mathematical optimization
MATINF4110/MATINF9110 Mathematical optimization Geir Dahl August 20, 2013 Convexity Part IV Chapter 4 Representation of convex sets different representations of convex sets, boundary polyhedra and polytopes:
More informationRankone LMIs and Lyapunov's Inequality. Gjerrit Meinsma 4. Abstract. We describe a new proof of the wellknown Lyapunov's matrix inequality about
Rankone LMIs and Lyapunov's Inequality Didier Henrion 1;; Gjerrit Meinsma Abstract We describe a new proof of the wellknown Lyapunov's matrix inequality about the location of the eigenvalues of a matrix
More informationPrimaldual relationship between LevenbergMarquardt and central trajectories for linearly constrained convex optimization
Primaldual relationship between LevenbergMarquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department
More informationTangent spaces, normals and extrema
Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3space, with a point a S where S looks smooth, i.e., without any fold or cusp or selfcrossing, we can intuitively define the tangent
More informationInterior Point Methods for Convex Quadratic and Convex Nonlinear Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationOptimality Conditions
Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.
More informationConvex Optimization Overview (cnt d)
Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize
More informationGeorgia Department of Education Common Core Georgia Performance Standards Framework CCGPS Advanced Algebra Unit 2
Polynomials Patterns Task 1. To get an idea of what polynomial functions look like, we can graph the first through fifth degree polynomials with leading coefficients of 1. For each polynomial function,
More informationNewtonType Algorithms for Nonlinear Constrained Optimization
NewtonType Algorithms for Nonlinear Constrained Optimization Angelika AltmannDieses Faculty of Management Science and Engineering Karlsruhe University of Applied Sciences Moritz Diehl Department of Microsystems
More informationMIT Algebraic techniques and semidefinite optimization February 14, Lecture 3
MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications
More informationSupport Vector Machines
Support Vector Machines Bingyu Wang, Virgil Pavlu March 30, 2015 based on notes by Andrew Ng. 1 What s SVM The original SVM algorithm was invented by Vladimir N. Vapnik 1 and the current standard incarnation
More informationSupport Vector Machines
EE 17/7AT: Optimization Models in Engineering Section 11/1  April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable
More informationOn the Value Function of a Mixed Integer Linear Optimization Problem and an Algorithm for its Construction
On the Value Function of a Mixed Integer Linear Optimization Problem and an Algorithm for its Construction Ted K. Ralphs and Anahita Hassanzadeh Department of Industrial and Systems Engineering, Lehigh
More informationInput: System of inequalities or equalities over the reals R. Output: Value for variables that minimizes cost function
Linear programming Input: System of inequalities or equalities over the reals R A linear cost function Output: Value for variables that minimizes cost function Example: Minimize 6x+4y Subject to 3x + 2y
More informationInterior Point Methods for LP
11.1 Interior Point Methods for LP Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor, Winter 1997. Simplex Method  A Boundary Method: Starting at an extreme point of the feasible set, the simplex
More informationDecomposition Methods for Large Scale LP Decoding
Decomposition Methods for Large Scale LP Decoding Siddharth Barman Joint work with Xishuo Liu, Stark Draper, and Ben Recht Outline Background and Problem Setup LP Decoding Formulation Optimization Framework
More informationOptimization Theory. A Concise Introduction. Jiongmin Yong
October 11, 017 16:5 wsbook9x6 Book Title Optimization Theory 01708Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 01708Lecture Notes page Optimization
More informationCLASSICAL FORMS OF LINEAR PROGRAMS, CONVERSION TECHNIQUES, AND SOME NOTATION
(Revised) October 12, 2004 CLASSICAL FORMS OF LINEAR PROGRAMS, CONVERSION TECHNIQUES, AND SOME NOTATION Linear programming is the minimization (maximization) of a linear objective, say c1x 1 + c2x 2 +
More informationDual and Optimality Criteria Methods 9
Dual and Optimality Criteria Methods 9 In most of the analytically solved examples in Chapter 2, the key to the solution is the use of an algebraic or a differential equation which forms the optimality
More informationConstrained Optimization
Constrained Optimization Joshua Wilde, revised by Isabel Tecu, Takeshi Suzuki and María José Boccardi August 13, 2013 1 General Problem Consider the following general constrained optimization problem:
More informationREGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROLSTATE CONSTRAINTS
REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROLSTATE CONSTRAINTS fredi tröltzsch 1 Abstract. A class of quadratic optimization problems in Hilbert spaces is considered,
More informationStructured Problems and Algorithms
Integer and quadratic optimization problems Dept. of Engg. and Comp. Sci., Univ. of Cal., Davis Aug. 13, 2010 Table of contents Outline 1 2 3 Benefits of Structured Problems Optimization problems may become
More informationContinuity of convex functions in normed spaces
Continuity of convex functions in normed spaces In this chapter, we consider continuity properties of realvalued convex functions defined on open convex sets in normed spaces. Recall that every infinitedimensional
More informationMathematical Economics: Lecture 16
Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave
More informationChapter 2. Binary and Mary Hypothesis Testing 2.1 Introduction (Levy 2.1)
Chapter 2. Binary and Mary Hypothesis Testing 2.1 Introduction (Levy 2.1) Detection problems can usually be casted as binary or Mary hypothesis testing problems. Applications: This chapter: Simple hypothesis
More informationLecture 1 Introduction
L. Vandenberghe EE236A (Fall 201314) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation
More informationLinear Matrix Inequalities in Control
Linear Matrix nequalities in Control Carsten Scherer and Siep Weiland Department of Mathematics University of Stuttgart Germany Department of Electrical Engineering Eindhoven University of Technology The
More information4. The Dual Simplex Method
4. The Dual Simplex Method Javier Larrosa Albert Oliveras Enric RodríguezCarbonell Problem Solving and Constraint Programming (RPAR) Session 4 p.1/34 Basic Idea (1) Algorithm as explained so far known
More informationLECTURE SLIDES ON BASED ON CLASS LECTURES AT THE CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS.
LECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON 6.253 CLASS LECTURES AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS http://web.mit.edu/dimitrib/www/home.html
More informationarxiv: v1 [stat.ml] 3 Apr 2017
Geometric Insights into SVM Tuning Geometric Insights into Support Vector Machine Behavior using the KKT Conditions arxiv:1704.00767v1 [stat.ml] 3 Apr 2017 Iain Carmichael Department of Statistics and
More informationPrimalDual InteriorPoint Methods. Ryan Tibshirani Convex Optimization /36725
PrimalDual InteriorPoint Methods Ryan Tibshirani Convex Optimization 10725/36725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,...
More informationThe Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation
The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation Yixin Zhao, Torbjörn Larsson and Department of Mathematics, Linköping University, Sweden
More informationDual methods and ADMM. Barnabas Poczos & Ryan Tibshirani Convex Optimization /36725
Dual methods and ADMM Barnabas Poczos & Ryan Tibshirani Convex Optimization 10725/36725 1 Given f : R n R, the function is called its conjugate Recall conjugate functions f (y) = max x R n yt x f(x)
More informationLagrangean relaxation
Lagrangean relaxation Giovanni Righini Corso di Complementi di Ricerca Operativa Joseph Louis de la Grange (Torino 1736  Paris 1813) Relaxations Given a problem P, such as: minimize z P (x) s.t. x X P
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primaldual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primaldual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationCHAPTER 12: SHADOW PRICES
Essential Microeconomics  CHAPTER : SHADOW PRICES An intuitive approach: profit maimizing firm with a fied supply of an input Shadow prices 5 Concave maimization problem 7 Constraint qualifications
More informationConcave programming. Concave programming is another special case of the general constrained optimization. subject to g(x) 0
1 Introduction Concave programming Concave programming is another special case of the general constrained optimization problem max f(x) subject to g(x) 0 in which the objective function f is concave and
More information