Applied Lagrange Duality for Constrained Optimization

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Applied Lagrange Duality for Constrained Optimization"

Transcription

1 Applied Lagrange Duality for Constrained Optimization February 12, 2002

2 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual Problem ffl Steps in the Construction of the Dual Problem ffl Examples of Dual Constructions ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

3 Overview The Dual is a Concave Maximization Problem ffl Weak Duality ffl The Column Geometry of the Primal and Dual Problems ffl Strong Duality ffl ffl Duality Strategies ffl Illustration of Lagrange Duality in Discrete Optimization cfl2002 Massachusetts Institute of Technology. All rights reserved

4 Context of Duality Previous Examples of Duality Models of electrical networks ffl Models of economic markets ffl Structural design ffl Duality in Linear Optimization ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

5 Context of Duality Importance of Duality Identifying near-optimal solutions ffl Proving optimality ffl Sensitivity analysis of the primal problem ffl Karush-Kuhn-Tucker (KKT) conditions ffl Convergence of improvement algorithms ffl Good Structure ffl Other uses, too... ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

6 Review of Convexity Local and Global Optima Problem Set-Up The ball centered at μx with radius ffl is the set: B(μx; ffl) := fxjkx μxk» fflg: Consider the following optimization problem over the set F: P : min x or max x f(x) s.t. x 2 F cfl2002 Massachusetts Institute of Technology. All rights reserved

7 Review of Convexity Local and Global Optima Definition x 2 F 1 is a local minimum P of if there ffl > 0 exists such f(x)» f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 2 is a global minimum P of if» f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

8 Review of Convexity Strict Local & Global Minima Definition x 2 F 3 is a strict local minimum P of if there ffl > 0 exists such f(x) < f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 4 is a strict global minimum P of if < f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

9 Review of Convexity Local & Global Maxima Definition x 2 F 5 is a local maximum P of if there ffl > 0 exists such f(x) f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 6 is a global maximum P of if f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

10 Review of Convexity Strict Local & Global Maxima Definition x 2 F 7 is a strict local maximum P of if there ffl > 0 exists such f(x) > f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 8 is a strict global maximum P of if > f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

11 Review of Convexity Local versus Global Minima F Illustration of local versus global optima. cfl2002 Massachusetts Institute of Technology. All rights reserved

12 Review of Convexity Definition 9 Convex Sets and Functions A subset S ρ < n is a convex set if y 2 S ) x + (1 )y 2 S x; for 2 [0; any 1]. Illustration of convex and non-convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

13 Review of Convexity Intersections of Convex Sets Proposition 1 S; T If are convex sets, S T then is a convex set. Proposition 2 The intersection of any collection of convex sets is a convex set. Illustration of the intersection of convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

14 Review of Convexity Convex Functions Definition 10 A function f(x) is a convex function if + (1 )y)» f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 11 function if A function f(x) is a strictly convex + (1 )y) < f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

15 Review of Convexity Convex Functions Illustration of convex and strictly convex functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

16 Review of Convexity Convex Optimization P : minimize x f(x) x 2 F s:t: Proposition 3 Suppose F that is a convex set, : F! < is a convex function, and μx is a local f minimum P of. μx Then is a global minimum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

17 Review of Convexity Proof of Proposition Proof of the Proposition: Suppose μx is not a global minimum, i.e., there exists y 2 F for which f(y) < f(μx). Let = μx + (1 )y, which is a convex combination of μx and y y( ) 2 [0; 1] for (and y( ) 2 F therefore, 2 [0; for 1]). Note that! μx as! 1. y( ) From the convexity of f( ), = f( μx + (1 )y)» f(μx) + (1 )f(y) < f(μx) + (1 )f(μx) = f(μx) f(y( )) for 2 (0; all 1). f(y( )) < f(μx) Therefore, for 2 (0; all 1), and μx so is not a local minimum, resulting in a contradiction. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

18 Review of Convexity Examples of Convex Functions Functions of One Variable Examples of convex functions of one variable: ffl f(x) = ax + b ffl f(x) = x 2 + bx + c ffl f(x) = jxj ffl f(x) = ln(x) for x > 0 for = 1 x x > 0 ffl f(x) = e x f(x) ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

19 Review of Convexity Concave Functions & Maximization The opposite of a convex function is a concave function: Definition 12 A function f(x) is a concave function if + (1 )y) f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 13 A function f(x) is a strictly concave function if + (1 )y) > f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

20 Review of Convexity Concave Functions & Maximization Illustration of concave and strictly concave functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

21 Review of Convexity Concave Maximization Now consider the maximization problem P : P : maximize x f(x) s:t: x 2 F Proposition 4 Suppose F that is a convex set, : F! < is a concave function, and μx is a local f maximum P of. μx Then is a global maximum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

22 Review of Convexity Convex Program : min x or max x f(x) P x 2 F s.t. We call P a convex program if ffl F is a convex set, and we are minimizing and is a convex function, or ffl we are maximizing and f(x)is a concave function. Then every local optima of the objective function of a convex program is a global optima of the objective function. cfl2002 Massachusetts Institute of Technology. All rights reserved

23 Review of Convexity Convex Program The class of convex programs is the class of well-behaved optimization problems. cfl2002 Massachusetts Institute of Technology. All rights reserved

24 Review of Convexity Linear Functions Proposition 5 A linear f(x) = a function x + b is both convex and concave. T Proposition 6 f(x) If is both convex and concave, f(x) then is a linear function. A linear function is convex and concave. cfl2002 Massachusetts Institute of Technology. All rights reserved

25 Separating Hyperplane Theory Strong Separation Theorem 1 Strong Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S. Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x > ff for all x 2 S u T μx < ff cfl2002 Massachusetts Institute of Technology. All rights reserved

26 Separating Hyperplane Theory Strong Separation S x utx = α Illustration of strong separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

27 Separating Hyperplane Theory Weak Separation Theorem 2 Weak Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S or μx Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x ff for all x 2 S u T μx» ff cfl2002 Massachusetts Institute of Technology. All rights reserved

28 Separating Hyperplane Theory Weak Separation S x utx = α Illustration of weak separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

29 Separating Hyperplane Theory Intersecting Line Dual Primal: minimum x height of x s:t: 2 S; x 2 L x Dual: maximum ß intercept of line ß with L s:t: S ß line lies below cfl2002 Massachusetts Institute of Technology. All rights reserved

30 Separating Hyperplane Theory Intersecting Line Dual z L S π Illustration intersecting line dual. r cfl2002 Massachusetts Institute of Technology. All rights reserved

31 OP : minimum x Definition of the Dual Problem General Constrained Problem Most General Format f(x) g 1 (x)» 0; s:t: =. m (x)» 0; g 2 P; x : < n 7! < f(x) i (x) : < n 7! <; i = 1; : : : ; m. g cfl2002 Massachusetts Institute of Technology. All rights reserved

32 ffl P = < n OP : z Λ = minimum x ffl P = Φ x j x 2 Z n + Ψ m (x)» 0; g 2 P; x Definition of the Dual Problem General Constrained Problem Standard Format... f(x) g 1 (x)» 0; s:t:. ffl P = fx j x 0g ffl P = fx j g i (x)» 0; i = m + 1; : : : ; m + kg P = fx j Ax» bg ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

33 OP : z Λ = minimum x L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format... f(x) Form the Lagrangian function: g i (x)» 0; i = 1; : : : ; m; s:t: 2 P; x Solve the presumably easier problem: i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The L function (u) is called the dual function. Λ We presume that computing L Λ (u) is an easy task. cfl2002 Massachusetts Institute of Technology. All rights reserved

34 L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format Form the Lagrangian function: Form the dual function L Λ (u): i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The dual problem is then defined to be: D : v Λ = maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

35 OP : z Λ = minimum x Steps in the Construction of the Dual Problem f(x) Step 1. Create the Lagrangian g i (x)» 0; i = 1;:::;m; s:t: 2 P; x u) := f(x) + u T g(x) : L(x; Step 2. Create the dual function: L Λ (u) := minimum x f(x) + u T g(x) Step 3. Create the dual problem: s:t: x 2 P D : v Λ = maximum u L Λ (u) s:t: u 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

36 LP : minimum x Examples of Dual Constructions of OPs The Dual of a Linear Problem Consider the linear optimization problem: c T x s:t: Ax b What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

37 IP : minimum x The Dual of a Binary Integer Problem Examples of Dual Constructions of OPs Consider the binary integer optimization problem: c T x s:t: b Ax j 2 f0; 1g; j = 1; : : : ; n : x What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

38 BP : minimum x1 ;x 2 ;x 3 5x 1 + 7x 2 4x 3 3 P Examples of Dual Constructions of OPs The Dual of a Log-Barrier Problem Consider the following logarithmic barrier problem: ln(x j ) j=1 s.t. x 1 + 3x x 3 = 37 x 1 > 0; x 2 > 0; x 3 > 0 : What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

39 OP : minimum x f(x) + P i2l u i g i (x) + P u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints Remarks... f(x) s:t: i (x)» 0; i 2 L g i (x) 0; i 2 G g g i (x) = 0; i 2 E The Lagrangian is: x 2 P; L(x; u) := f(x) + u T g(x) = u i g i (x) cfl2002 Massachusetts Institute of Technology. All rights reserved i2g i2e

40 u) := f(x) + u T g(x) = f(x) + P L(x; u i g i (x) + P i2l D : v Λ = maximum u u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints...Remarks u i g i (x) The dual function L Λ (u): i2g i2e L Λ (u) := minimum x f(x) + u T g(x) The dual problem is: s:t: x 2 P L Λ (u) s:t: 0; i 2 L u» 0; i 2 G u > < 0; i 2 E u cfl2002 Massachusetts Institute of Technology. All rights reserved

41 OP : z Λ = minimum x The Dual is a Concave Maximization Problem f(x) s:t: g i (x)» 0; i = 1; : : : ; m x 2 P; L(x; u) := f(x) + u T g(x) L Λ (u) := minimum x f(x) + u T g(x) s:t: x 2 P Theorem 3 The dual function L Λ (u) is a concave function. cfl2002 Massachusetts Institute of Technology. All rights reserved

42 = min x2p f(x) + u T 1 g(x) Λ + (1 ) f(x) + u T 2 g(x) Λ min x2p f(x) + u T 1 g(x) Λ + (1 ) min x2p (f(x) + u T 2 g(x) Λ The Dual is a Concave Maximization Problem Proof of Concavity of L Λ (u) Proof: Let u 1 0 and u 2 0 be two values of the dual variables, and let u = u 1 + (1 )u 2, where 2 [0; 1]. Then L Λ (u) = min x2p f(x) + u T g(x) = L Λ (u 1 ) + (1 )L Λ (u 2 ) : Therefore L Λ (u) is a concave function. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

43 OP : z Λ = minimum x g i (x)» 0; i = 1;:::;m s:t: 2 P; x Weak Duality f(x) D : v Λ = maximum u L Λ (u) s:t: u 0 Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) Λ v Λ : z cfl2002 Massachusetts Institute of Technology. All rights reserved

44 Weak Duality Proof of Weak Duality Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) z Λ v Λ : Proof: If μx is feasible for OP and μu is feasible for D, then + T g(μx) min x2p f(x) + μu T g(x) = L Λ (μu) : f(μx) f(μx) μu v Therefore. q.e.d. Λ z Λ cfl2002 Massachusetts Institute of Technology. All rights reserved

45 OP : z Λ = minimum x A = 1 (x) g 2 (x) g m (x) g f(x) A = Column Geometry of Primal and Dual Problems Resources and Costs For this section only, we assume equality format: f(x) g i (x) = 0; i = 1;:::;m s:t: 2 P; x D : v Λ = maximum u L Λ (u) s:t: u > < 0 For each x 2 P, we have an array of resources and costs associated with x: r z 1 r 2 r B = B. C C B B. C C g(x) r m z cfl2002 Massachusetts Institute of Technology. All rights reserved

46 Column Geometry of Primal and Dual Problems Resources and Costs S := Φ (r; z) 2 < m+1 j (r; z) = (g(x); f(x)) for some x 2 P Ψ : z L = {( r, z ) r = 0 } S H r 1 r 2 r = 0 The column geometry of the primal and dual problem. cfl2002 Massachusetts Institute of Technology. All rights reserved

47 Column Geometry of Primal and Dual Problems Column Geometry Interpretation of OP Every point in the intersection of S and L corresponds to a feasible solution of OP, and the feasible solution with the lowest cost corresponds to the lowest point in the intersection of S and L. This lowest value is exactly the value of the primal problem, namely z Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

48 Column Geometry of Primal and Dual Problems Supporting Hyperplanes of S H = H u;ff = Φ (r; z) 2 < m+1 j z + u T r = ff Ψ : L = Φ (r; z) 2 < m+1 j r = 0 Ψ : H L = f(r; z) = (0; ff)g cfl2002 Massachusetts Institute of Technology. All rights reserved

49 maximum u;ff Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: H u;ff lies below S = maximum u;ff ff s:t: z + u T r ff; for all (r; z) 2 S = maximum u;ff ff s:t: f(x) + u T g(x) ff; for all x 2 P cfl2002 Massachusetts Institute of Technology. All rights reserved

50 maximum u;ff > 0 : < Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: f(x) + u T g(x) ff; for all x 2 P = maximum u;ff ff s:t: L Λ (u) ff; = maximum u L Λ (u) s:t: u cfl2002 Massachusetts Institute of Technology. All rights reserved

51 Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S The dual problem corresponds to finding that hyperplane H u;ff lying below S whose intersection with L is the highest. This highest value is exactly the value of the dual problem, namely v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

52 Strong Duality Convexity and Strong Duality z L S r = 0 H r The column geometry of the dual problem when S is convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

53 Strong Duality Absence of Convexity z L z* v* S H r The column geometry of the dual problem when S is not convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

54 Strong Duality Strong Duality Theorem Theorem 5 Strong Duality Theorem: Suppose that OP is a convex optimization problem, that is, all constraints of OP are of the form g i (x)» 0; i = 1; : : : ; m, f(x) as well as g i (x) are convex functions, i = 1; : : : ; m, and P is a convex set. Then under very mild additional conditions, z Λ = v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

55 Duality Strategies Dualizing Bad Constraints OP : minimum x c T x s:t:» b Ax» g : Nx Suppose that optimization over the constraints Nx» g is easy. The addition of the constraints Ax» b makes the problem much more difficult. Let = fx j Nx» gg P cfl2002 Massachusetts Institute of Technology. All rights reserved

56 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: Nx» g : OP : minimum x c T x Ax» b s:t: 2 P : x cfl2002 Massachusetts Institute of Technology. All rights reserved

57 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: where x 2 P : The Lagrangian is: L(x; u) = c T x + u T (Ax b) = u T b + (c T + u T A)x L Λ (u) := minimum x u T b + (c T + u T A)x s:t: x 2 P : cfl2002 Massachusetts Institute of Technology. All rights reserved

58 Λ (u) := minimum x u T b + (c T + u T A)x L x 2 P : s:t: Duality Strategies Dualizing Bad Constraints D : maximum u L Λ (u) s:t: u 0 Notice that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem D. cfl2002 Massachusetts Institute of Technology. All rights reserved

59 OP : minimum x1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A 2 x 2» b 2 A Notice here that if it were not for the constraints 1 x 1 + B 2 x 2» d, that we would be able to separate B the problem into two separate problems. Let us dualize on these constraints. Let: P = Φ (x 1 ; x 2 ) j A 1 x 1» b 1 ; A 2 x 2» b 2Ψ cfl2002 Massachusetts Institute of Technology. All rights reserved

60 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 A 2 x 2» b 2 P = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A B 1 x 1 +B 2 x 2» d s:t: 1 ; x 2 ) 2 P (x cfl2002 Massachusetts Institute of Technology. All rights reserved

61 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 L Λ (u) = minimum x1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 ;x 2 ) 2 P (x = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ P The Lagrangian is: L(x; u) = (c 1 ) T x 1 + (c 2 ) T x 2 + u T (B 1 x 1 + B 2 x 2 d) = u T d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 ; (x 1 ; x 2 ) 2 P s:t: cfl2002 Massachusetts Institute of Technology. All rights reserved

62 L Λ (u) = minimum x 1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems s:t: (x 1 ;x 2 ) 2 P L Λ (u) = T d u minimum A1 x 1»b 1 ((c1 ) T + u T B 1 )x minimum A 2 x 2»b 2 ((c2 ) T + u T B 2 )x 2 Notice once again that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem: D : maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

63 Lagrange Duality Illustrated Constrained Shortest Path Problem Definition Constrained Shortest Path Problem: find the shortest path from node to node 6, among those paths using 4 exactly arcs cfl2002 Massachusetts Institute of Technology. All rights reserved

64 Lagrange Duality Illustrated Constrained Shortest Path Problem Context In practice: Arcs might have two types of costs (time and money) There may be millions of nodes and billions of paths cfl2002 Massachusetts Institute of Technology. All rights reserved

65 Lagrange Duality Illustrated Constrained Shortest Path Enumeration of Paths Path Number of Arcs Cost Path is the optimal solution. cfl2002 Massachusetts Institute of Technology. All rights reserved

66 Lagrange Duality Illustrated Constrained Shortest Path Motivation for Dual Suppose that we have available an efficient algorithm for solving (unconstrained) shortest path problems. We will use this algorithm to try to solve our problem through the dual cfl2002 Massachusetts Institute of Technology. All rights reserved

67 z Λ = min xij x 12 +x 13 +x 23 +x 24 +3x 34 +2x 35 +x 54 +x 46 +3x 56 Lagrange Duality Illustrated Constrained Shortest Path IP Formulation... CSPP : s.t. x 12 +x 13 +x 23 +x 24 +x 34 +x 35 +x 54 +x 46 +x 56 = 4 x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

68 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation... P i;j P x ij = 4 s:t: i;j Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

69 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation P i;j P x ij = 4 s:t: i;j where: x 2 P P = fx j Nx = b ; x ij 2 f0; 1g for all arcs i jg cfl2002 Massachusetts Institute of Technology. All rights reserved

70 L(x; u) := P i;j 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path The Lagrangian ψ x ij! c ij x ij + u cfl2002 Massachusetts Institute of Technology. All rights reserved

71 L Λ (u) := minimum xij 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path L Λ (u)... ψ P c ij x ij + u x ij! i;j s:t: Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

72 Lagrange Duality Illustrated Constrained Shortest Path...L Λ (u) Λ (u) = min xij (1 u)x 12 +(1 u)x 13 +(1 u)x 23 L u)x 24 +(3 u)x 34 +(2 u)x 35 +(1 +(1 u)x 54 +(1 u)x 46 +(3 u)x 56 +4u s.t. x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

73 Lagrange Duality Illustrated Constrained Shortest Path Comments on L Λ (u) For a fixed value of u, 4u is a constant. For a fixed value of u, this is a shortest path problem with modified arc costs given by: c 0 ij = c ij u : cfl2002 Massachusetts Institute of Technology. All rights reserved

74 Lagrange Duality Illustrated Constrained Shortest Path The Dual Problem D : v Λ = maximum u L Λ (u) s.t. u < > 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

75 Lagrange Duality Illustrated Constrained Shortest Path The set P Recall that P is the set of solutions of: x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

76 Lagrange Duality Illustrated Constrained Shortest Path Resources and Costs z (cost) conv(s) r (number of arcs) z* = 5 v* = 4.5 cfl2002 Massachusetts Institute of Technology. All rights reserved

77 Lagrange Duality Illustrated Constrained Shortest Path Duality Gap ffl The optimal u is u = 1:5 ffl The value of the dual is v Λ = 4:5 ffl The value of the primal is z Λ = 5:0 ffl There is a duality gap: z Λ v Λ = 5:0 4:5 = 0:5. cfl2002 Massachusetts Institute of Technology. All rights reserved

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

Optimization Theory. Lectures 4-6

Optimization Theory. Lectures 4-6 Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:

More information

The Kuhn-Tucker Problem

The Kuhn-Tucker Problem Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

More information

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

More information

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

More information

Nonlinear Programming (NLP)

Nonlinear Programming (NLP) Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume

More information

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach

More information

Lagrangian Duality for Dummies

Lagrangian Duality for Dummies Lagrangian Duality for Dummies David Knowles November 13, 2010 We want to solve the following optimisation problem: f 0 () (1) such that f i () 0 i 1,..., m (2) For now we do not need to assume conveity.

More information

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Duality of LPs and Applications

Duality of LPs and Applications Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

More information

SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING

SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL

More information

Solving Dual Problems

Solving Dual Problems Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem

More information

Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7

Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7 Mathematical Foundations -- Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum

More information

Lecture 9: Dantzig-Wolfe Decomposition

Lecture 9: Dantzig-Wolfe Decomposition Lecture 9: Dantzig-Wolfe Decomposition (3 units) Outline Dantzig-Wolfe decomposition Column generation algorithm Relation to Lagrangian dual Branch-and-price method Generated assignment problem and multi-commodity

More information

Lecture #21. c T x Ax b. maximize subject to

Lecture #21. c T x Ax b. maximize subject to COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the

More information

Lecture 6: Conic Optimization September 8

Lecture 6: Conic Optimization September 8 IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

More information

The Karush-Kuhn-Tucker (KKT) conditions

The Karush-Kuhn-Tucker (KKT) conditions The Karush-Kuhn-Tucker (KKT) conditions In this section, we will give a set of sufficient (and at most times necessary) conditions for a x to be the solution of a given convex optimization problem. These

More information

10 Numerical methods for constrained problems

10 Numerical methods for constrained problems 10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

More information

Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

More information

8 Barrier Methods for Constrained Optimization

8 Barrier Methods for Constrained Optimization IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality

More information

LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

More information

Primal-Dual Interior-Point Methods

Primal-Dual Interior-Point Methods Primal-Dual Interior-Point Methods Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 Outline Today: Primal-dual interior-point method Special case: linear programming

More information

Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark

Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT

More information

Lecture 15 Newton Method and Self-Concordance. October 23, 2008

Lecture 15 Newton Method and Self-Concordance. October 23, 2008 Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications

More information

Lecture 2: Convex Sets and Functions

Lecture 2: Convex Sets and Functions Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are

More information

Optimization Tutorial 1. Basic Gradient Descent

Optimization Tutorial 1. Basic Gradient Descent E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

More information

IE 5531 Midterm #2 Solutions

IE 5531 Midterm #2 Solutions IE 5531 Midterm #2 s Prof. John Gunnar Carlsson November 9, 2011 Before you begin: This exam has 9 pages and a total of 5 problems. Make sure that all pages are present. To obtain credit for a problem,

More information

Symmetric and Asymmetric Duality

Symmetric and Asymmetric Duality journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,

More information

Nonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6

Nonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6 Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains

More information

Optimization for Machine Learning

Optimization for Machine Learning Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

More information

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY UNIVERSITY OF MARYLAND: ECON 600 1. Some Eamples 1 A general problem that arises countless times in economics takes the form: (Verbally):

More information

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

More information

Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

More information

z = f (x; y) f (x ; y ) f (x; y) f (x; y )

z = f (x; y) f (x ; y ) f (x; y) f (x; y ) BEEM0 Optimization Techiniques for Economists Lecture Week 4 Dieter Balkenborg Departments of Economics University of Exeter Since the fabric of the universe is most perfect, and is the work of a most

More information

Date: July 5, Contents

Date: July 5, Contents 2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........

More information

Lecture 1: Background on Convex Analysis

Lecture 1: Background on Convex Analysis Lecture 1: Background on Convex Analysis John Duchi PCMI 2016 Outline I Convex sets 1.1 Definitions and examples 2.2 Basic properties 3.3 Projections onto convex sets 4.4 Separating and supporting hyperplanes

More information

Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2)

Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Tsun-Feng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization

More information

Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

More information

EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline

EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for

More information

On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

More information

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

More information

Support Vector Machines

Support Vector Machines Wien, June, 2010 Paul Hofmarcher, Stefan Theussl, WU Wien Hofmarcher/Theussl SVM 1/21 Linear Separable Separating Hyperplanes Non-Linear Separable Soft-Margin Hyperplanes Hofmarcher/Theussl SVM 2/21 (SVM)

More information

Determinant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University

Determinant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition

More information

CS , Fall 2011 Assignment 2 Solutions

CS , Fall 2011 Assignment 2 Solutions CS 94-0, Fall 20 Assignment 2 Solutions (8 pts) In this question we briefly review the expressiveness of kernels (a) Construct a support vector machine that computes the XOR function Use values of + and

More information

Math 273a: Optimization Convex Conjugacy

Math 273a: Optimization Convex Conjugacy Math 273a: Optimization Convex Conjugacy Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Convex conjugate (the Legendre transform) Let f be a closed proper

More information

Integer Programming ISE 418. Lecture 12. Dr. Ted Ralphs

Integer Programming ISE 418. Lecture 12. Dr. Ted Ralphs Integer Programming ISE 418 Lecture 12 Dr. Ted Ralphs ISE 418 Lecture 12 1 Reading for This Lecture Nemhauser and Wolsey Sections II.2.1 Wolsey Chapter 9 ISE 418 Lecture 12 2 Generating Stronger Valid

More information

Linear Programming: Simplex

Linear Programming: Simplex Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Linear Programming: Simplex IMA, August 2016

More information

Computational Integer Programming. Lecture 2: Modeling and Formulation. Dr. Ted Ralphs

Computational Integer Programming. Lecture 2: Modeling and Formulation. Dr. Ted Ralphs Computational Integer Programming Lecture 2: Modeling and Formulation Dr. Ted Ralphs Computational MILP Lecture 2 1 Reading for This Lecture N&W Sections I.1.1-I.1.6 Wolsey Chapter 1 CCZ Chapter 2 Computational

More information

Support Vector Machine (SVM) and Kernel Methods

Support Vector Machine (SVM) and Kernel Methods Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2015 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

More information

CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES. Contents 1. Introduction 1 2. Convex Sets

CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES. Contents 1. Introduction 1 2. Convex Sets CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES DANIEL HENDRYCKS Abstract. This paper develops the fundamentals of convex optimization and applies them to Support Vector

More information

Lecture 7: Weak Duality

Lecture 7: Weak Duality EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly

More information

You should be able to...

You should be able to... Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

More information

Section 6.1: Composite Functions

Section 6.1: Composite Functions Section 6.1: Composite Functions Def: Given two function f and g, the composite function, which we denote by f g and read as f composed with g, is defined by (f g)(x) = f(g(x)). In other words, the function

More information

Interior Point Methods for Linear Programming: Motivation & Theory

Interior Point Methods for Linear Programming: Motivation & Theory School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Linear Programming: Motivation & Theory Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

CS 6820 Fall 2014 Lectures, October 3-20, 2014

CS 6820 Fall 2014 Lectures, October 3-20, 2014 Analysis of Algorithms Linear Programming Notes CS 6820 Fall 2014 Lectures, October 3-20, 2014 1 Linear programming The linear programming (LP) problem is the following optimization problem. We are given

More information

Integer Programming ISE 418. Lecture 2. Dr. Ted Ralphs

Integer Programming ISE 418. Lecture 2. Dr. Ted Ralphs Integer Programming ISE 418 Lecture 2 Dr. Ted Ralphs ISE 418 Lecture 2 1 Reading for This Lecture N&W Sections I.1.1-I.1.6 Wolsey Chapter 1 CCZ Chapter 2 ISE 418 Lecture 2 2 Formulations and Models Our

More information

Chapter 1: Linear Programming

Chapter 1: Linear Programming Chapter 1: Linear Programming Math 368 c Copyright 2013 R Clark Robinson May 22, 2013 Chapter 1: Linear Programming 1 Max and Min For f : D R n R, f (D) = {f (x) : x D } is set of attainable values of

More information

- Well-characterized problems, min-max relations, approximate certificates. - LP problems in the standard form, primal and dual linear programs

- Well-characterized problems, min-max relations, approximate certificates. - LP problems in the standard form, primal and dual linear programs LP-Duality ( Approximation Algorithms by V. Vazirani, Chapter 12) - Well-characterized problems, min-max relations, approximate certificates - LP problems in the standard form, primal and dual linear programs

More information

EC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes.

EC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes. London School of Economics Department of Economics Dr Francesco Nava Offi ce: 32L.3.20 EC400 Math for Microeconomics Syllabus 2016 The course is based on 6 sixty minutes lectures and on 6 ninety minutes

More information

MAT-INF4110/MAT-INF9110 Mathematical optimization

MAT-INF4110/MAT-INF9110 Mathematical optimization MAT-INF4110/MAT-INF9110 Mathematical optimization Geir Dahl August 20, 2013 Convexity Part IV Chapter 4 Representation of convex sets different representations of convex sets, boundary polyhedra and polytopes:

More information

Rank-one LMIs and Lyapunov's Inequality. Gjerrit Meinsma 4. Abstract. We describe a new proof of the well-known Lyapunov's matrix inequality about

Rank-one LMIs and Lyapunov's Inequality. Gjerrit Meinsma 4. Abstract. We describe a new proof of the well-known Lyapunov's matrix inequality about Rank-one LMIs and Lyapunov's Inequality Didier Henrion 1;; Gjerrit Meinsma Abstract We describe a new proof of the well-known Lyapunov's matrix inequality about the location of the eigenvalues of a matrix

More information

Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization

Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department

More information

Tangent spaces, normals and extrema

Tangent spaces, normals and extrema Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent

More information

Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming

Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

Optimality Conditions

Optimality Conditions Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.

More information

Convex Optimization Overview (cnt d)

Convex Optimization Overview (cnt d) Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize

More information

Georgia Department of Education Common Core Georgia Performance Standards Framework CCGPS Advanced Algebra Unit 2

Georgia Department of Education Common Core Georgia Performance Standards Framework CCGPS Advanced Algebra Unit 2 Polynomials Patterns Task 1. To get an idea of what polynomial functions look like, we can graph the first through fifth degree polynomials with leading coefficients of 1. For each polynomial function,

More information

Newton-Type Algorithms for Nonlinear Constrained Optimization

Newton-Type Algorithms for Nonlinear Constrained Optimization Newton-Type Algorithms for Nonlinear Constrained Optimization Angelika Altmann-Dieses Faculty of Management Science and Engineering Karlsruhe University of Applied Sciences Moritz Diehl Department of Microsystems

More information

MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3

MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3 MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications

More information

Support Vector Machines

Support Vector Machines Support Vector Machines Bingyu Wang, Virgil Pavlu March 30, 2015 based on notes by Andrew Ng. 1 What s SVM The original SVM algorithm was invented by Vladimir N. Vapnik 1 and the current standard incarnation

More information

Support Vector Machines

Support Vector Machines EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable

More information

On the Value Function of a Mixed Integer Linear Optimization Problem and an Algorithm for its Construction

On the Value Function of a Mixed Integer Linear Optimization Problem and an Algorithm for its Construction On the Value Function of a Mixed Integer Linear Optimization Problem and an Algorithm for its Construction Ted K. Ralphs and Anahita Hassanzadeh Department of Industrial and Systems Engineering, Lehigh

More information

Input: System of inequalities or equalities over the reals R. Output: Value for variables that minimizes cost function

Input: System of inequalities or equalities over the reals R. Output: Value for variables that minimizes cost function Linear programming Input: System of inequalities or equalities over the reals R A linear cost function Output: Value for variables that minimizes cost function Example: Minimize 6x+4y Subject to 3x + 2y

More information

Interior Point Methods for LP

Interior Point Methods for LP 11.1 Interior Point Methods for LP Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor, Winter 1997. Simplex Method - A Boundary Method: Starting at an extreme point of the feasible set, the simplex

More information

Decomposition Methods for Large Scale LP Decoding

Decomposition Methods for Large Scale LP Decoding Decomposition Methods for Large Scale LP Decoding Siddharth Barman Joint work with Xishuo Liu, Stark Draper, and Ben Recht Outline Background and Problem Setup LP Decoding Formulation Optimization Framework

More information

Optimization Theory. A Concise Introduction. Jiongmin Yong

Optimization Theory. A Concise Introduction. Jiongmin Yong October 11, 017 16:5 ws-book9x6 Book Title Optimization Theory 017-08-Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 017-08-Lecture Notes page Optimization

More information

CLASSICAL FORMS OF LINEAR PROGRAMS, CONVERSION TECHNIQUES, AND SOME NOTATION

CLASSICAL FORMS OF LINEAR PROGRAMS, CONVERSION TECHNIQUES, AND SOME NOTATION (Revised) October 12, 2004 CLASSICAL FORMS OF LINEAR PROGRAMS, CONVERSION TECHNIQUES, AND SOME NOTATION Linear programming is the minimization (maximization) of a linear objective, say c1x 1 + c2x 2 +

More information

Dual and Optimality Criteria Methods 9

Dual and Optimality Criteria Methods 9 Dual and Optimality Criteria Methods 9 In most of the analytically solved examples in Chapter 2, the key to the solution is the use of an algebraic or a differential equation which forms the optimality

More information

Constrained Optimization

Constrained Optimization Constrained Optimization Joshua Wilde, revised by Isabel Tecu, Takeshi Suzuki and María José Boccardi August 13, 2013 1 General Problem Consider the following general constrained optimization problem:

More information

REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROL-STATE CONSTRAINTS

REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROL-STATE CONSTRAINTS REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROL-STATE CONSTRAINTS fredi tröltzsch 1 Abstract. A class of quadratic optimization problems in Hilbert spaces is considered,

More information

Structured Problems and Algorithms

Structured Problems and Algorithms Integer and quadratic optimization problems Dept. of Engg. and Comp. Sci., Univ. of Cal., Davis Aug. 13, 2010 Table of contents Outline 1 2 3 Benefits of Structured Problems Optimization problems may become

More information

Continuity of convex functions in normed spaces

Continuity of convex functions in normed spaces Continuity of convex functions in normed spaces In this chapter, we consider continuity properties of real-valued convex functions defined on open convex sets in normed spaces. Recall that every infinitedimensional

More information

Mathematical Economics: Lecture 16

Mathematical Economics: Lecture 16 Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave

More information

Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1)

Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1) Chapter 2. Binary and M-ary Hypothesis Testing 2.1 Introduction (Levy 2.1) Detection problems can usually be casted as binary or M-ary hypothesis testing problems. Applications: This chapter: Simple hypothesis

More information

Lecture 1 Introduction

Lecture 1 Introduction L. Vandenberghe EE236A (Fall 2013-14) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation

More information

Linear Matrix Inequalities in Control

Linear Matrix Inequalities in Control Linear Matrix nequalities in Control Carsten Scherer and Siep Weiland Department of Mathematics University of Stuttgart Germany Department of Electrical Engineering Eindhoven University of Technology The

More information

4. The Dual Simplex Method

4. The Dual Simplex Method 4. The Dual Simplex Method Javier Larrosa Albert Oliveras Enric Rodríguez-Carbonell Problem Solving and Constraint Programming (RPAR) Session 4 p.1/34 Basic Idea (1) Algorithm as explained so far known

More information

LECTURE SLIDES ON BASED ON CLASS LECTURES AT THE CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS.

LECTURE SLIDES ON BASED ON CLASS LECTURES AT THE CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS. LECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON 6.253 CLASS LECTURES AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS http://web.mit.edu/dimitrib/www/home.html

More information

arxiv: v1 [stat.ml] 3 Apr 2017

arxiv: v1 [stat.ml] 3 Apr 2017 Geometric Insights into SVM Tuning Geometric Insights into Support Vector Machine Behavior using the KKT Conditions arxiv:1704.00767v1 [stat.ml] 3 Apr 2017 Iain Carmichael Department of Statistics and

More information

Primal-Dual Interior-Point Methods. Ryan Tibshirani Convex Optimization /36-725

Primal-Dual Interior-Point Methods. Ryan Tibshirani Convex Optimization /36-725 Primal-Dual Interior-Point Methods Ryan Tibshirani Convex Optimization 10-725/36-725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,...

More information

The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation

The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation Yixin Zhao, Torbjörn Larsson and Department of Mathematics, Linköping University, Sweden

More information

Dual methods and ADMM. Barnabas Poczos & Ryan Tibshirani Convex Optimization /36-725

Dual methods and ADMM. Barnabas Poczos & Ryan Tibshirani Convex Optimization /36-725 Dual methods and ADMM Barnabas Poczos & Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given f : R n R, the function is called its conjugate Recall conjugate functions f (y) = max x R n yt x f(x)

More information

Lagrangean relaxation

Lagrangean relaxation Lagrangean relaxation Giovanni Righini Corso di Complementi di Ricerca Operativa Joseph Louis de la Grange (Torino 1736 - Paris 1813) Relaxations Given a problem P, such as: minimize z P (x) s.t. x X P

More information

Agenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms

Agenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point

More information

CHAPTER 1-2: SHADOW PRICES

CHAPTER 1-2: SHADOW PRICES Essential Microeconomics -- CHAPTER -: SHADOW PRICES An intuitive approach: profit maimizing firm with a fied supply of an input Shadow prices 5 Concave maimization problem 7 Constraint qualifications

More information

Concave programming. Concave programming is another special case of the general constrained optimization. subject to g(x) 0

Concave programming. Concave programming is another special case of the general constrained optimization. subject to g(x) 0 1 Introduction Concave programming Concave programming is another special case of the general constrained optimization problem max f(x) subject to g(x) 0 in which the objective function f is concave and

More information