Applied Lagrange Duality for Constrained Optimization

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Applied Lagrange Duality for Constrained Optimization"

Transcription

1 Applied Lagrange Duality for Constrained Optimization February 12, 2002

2 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual Problem ffl Steps in the Construction of the Dual Problem ffl Examples of Dual Constructions ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

3 Overview The Dual is a Concave Maximization Problem ffl Weak Duality ffl The Column Geometry of the Primal and Dual Problems ffl Strong Duality ffl ffl Duality Strategies ffl Illustration of Lagrange Duality in Discrete Optimization cfl2002 Massachusetts Institute of Technology. All rights reserved

4 Context of Duality Previous Examples of Duality Models of electrical networks ffl Models of economic markets ffl Structural design ffl Duality in Linear Optimization ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

5 Context of Duality Importance of Duality Identifying near-optimal solutions ffl Proving optimality ffl Sensitivity analysis of the primal problem ffl Karush-Kuhn-Tucker (KKT) conditions ffl Convergence of improvement algorithms ffl Good Structure ffl Other uses, too... ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

6 Review of Convexity Local and Global Optima Problem Set-Up The ball centered at μx with radius ffl is the set: B(μx; ffl) := fxjkx μxk» fflg: Consider the following optimization problem over the set F: P : min x or max x f(x) s.t. x 2 F cfl2002 Massachusetts Institute of Technology. All rights reserved

7 Review of Convexity Local and Global Optima Definition x 2 F 1 is a local minimum P of if there ffl > 0 exists such f(x)» f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 2 is a global minimum P of if» f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

8 Review of Convexity Strict Local & Global Minima Definition x 2 F 3 is a strict local minimum P of if there ffl > 0 exists such f(x) < f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 4 is a strict global minimum P of if < f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

9 Review of Convexity Local & Global Maxima Definition x 2 F 5 is a local maximum P of if there ffl > 0 exists such f(x) f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 6 is a global maximum P of if f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

10 Review of Convexity Strict Local & Global Maxima Definition x 2 F 7 is a strict local maximum P of if there ffl > 0 exists such f(x) > f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 8 is a strict global maximum P of if > f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

11 Review of Convexity Local versus Global Minima F Illustration of local versus global optima. cfl2002 Massachusetts Institute of Technology. All rights reserved

12 Review of Convexity Definition 9 Convex Sets and Functions A subset S ρ < n is a convex set if y 2 S ) x + (1 )y 2 S x; for 2 [0; any 1]. Illustration of convex and non-convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

13 Review of Convexity Intersections of Convex Sets Proposition 1 S; T If are convex sets, S T then is a convex set. Proposition 2 The intersection of any collection of convex sets is a convex set. Illustration of the intersection of convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

14 Review of Convexity Convex Functions Definition 10 A function f(x) is a convex function if + (1 )y)» f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 11 function if A function f(x) is a strictly convex + (1 )y) < f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

15 Review of Convexity Convex Functions Illustration of convex and strictly convex functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

16 Review of Convexity Convex Optimization P : minimize x f(x) x 2 F s:t: Proposition 3 Suppose F that is a convex set, : F! < is a convex function, and μx is a local f minimum P of. μx Then is a global minimum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

17 Review of Convexity Proof of Proposition Proof of the Proposition: Suppose μx is not a global minimum, i.e., there exists y 2 F for which f(y) < f(μx). Let = μx + (1 )y, which is a convex combination of μx and y y( ) 2 [0; 1] for (and y( ) 2 F therefore, 2 [0; for 1]). Note that! μx as! 1. y( ) From the convexity of f( ), = f( μx + (1 )y)» f(μx) + (1 )f(y) < f(μx) + (1 )f(μx) = f(μx) f(y( )) for 2 (0; all 1). f(y( )) < f(μx) Therefore, for 2 (0; all 1), and μx so is not a local minimum, resulting in a contradiction. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

18 Review of Convexity Examples of Convex Functions Functions of One Variable Examples of convex functions of one variable: ffl f(x) = ax + b ffl f(x) = x 2 + bx + c ffl f(x) = jxj ffl f(x) = ln(x) for x > 0 for = 1 x x > 0 ffl f(x) = e x f(x) ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

19 Review of Convexity Concave Functions & Maximization The opposite of a convex function is a concave function: Definition 12 A function f(x) is a concave function if + (1 )y) f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 13 A function f(x) is a strictly concave function if + (1 )y) > f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

20 Review of Convexity Concave Functions & Maximization Illustration of concave and strictly concave functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

21 Review of Convexity Concave Maximization Now consider the maximization problem P : P : maximize x f(x) s:t: x 2 F Proposition 4 Suppose F that is a convex set, : F! < is a concave function, and μx is a local f maximum P of. μx Then is a global maximum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

22 Review of Convexity Convex Program : min x or max x f(x) P x 2 F s.t. We call P a convex program if ffl F is a convex set, and we are minimizing and is a convex function, or ffl we are maximizing and f(x)is a concave function. Then every local optima of the objective function of a convex program is a global optima of the objective function. cfl2002 Massachusetts Institute of Technology. All rights reserved

23 Review of Convexity Convex Program The class of convex programs is the class of well-behaved optimization problems. cfl2002 Massachusetts Institute of Technology. All rights reserved

24 Review of Convexity Linear Functions Proposition 5 A linear f(x) = a function x + b is both convex and concave. T Proposition 6 f(x) If is both convex and concave, f(x) then is a linear function. A linear function is convex and concave. cfl2002 Massachusetts Institute of Technology. All rights reserved

25 Separating Hyperplane Theory Strong Separation Theorem 1 Strong Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S. Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x > ff for all x 2 S u T μx < ff cfl2002 Massachusetts Institute of Technology. All rights reserved

26 Separating Hyperplane Theory Strong Separation S x utx = α Illustration of strong separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

27 Separating Hyperplane Theory Weak Separation Theorem 2 Weak Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S or μx Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x ff for all x 2 S u T μx» ff cfl2002 Massachusetts Institute of Technology. All rights reserved

28 Separating Hyperplane Theory Weak Separation S x utx = α Illustration of weak separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

29 Separating Hyperplane Theory Intersecting Line Dual Primal: minimum x height of x s:t: 2 S; x 2 L x Dual: maximum ß intercept of line ß with L s:t: S ß line lies below cfl2002 Massachusetts Institute of Technology. All rights reserved

30 Separating Hyperplane Theory Intersecting Line Dual z L S π Illustration intersecting line dual. r cfl2002 Massachusetts Institute of Technology. All rights reserved

31 OP : minimum x Definition of the Dual Problem General Constrained Problem Most General Format f(x) g 1 (x)» 0; s:t: =. m (x)» 0; g 2 P; x : < n 7! < f(x) i (x) : < n 7! <; i = 1; : : : ; m. g cfl2002 Massachusetts Institute of Technology. All rights reserved

32 ffl P = < n OP : z Λ = minimum x ffl P = Φ x j x 2 Z n + Ψ m (x)» 0; g 2 P; x Definition of the Dual Problem General Constrained Problem Standard Format... f(x) g 1 (x)» 0; s:t:. ffl P = fx j x 0g ffl P = fx j g i (x)» 0; i = m + 1; : : : ; m + kg P = fx j Ax» bg ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

33 OP : z Λ = minimum x L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format... f(x) Form the Lagrangian function: g i (x)» 0; i = 1; : : : ; m; s:t: 2 P; x Solve the presumably easier problem: i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The L function (u) is called the dual function. Λ We presume that computing L Λ (u) is an easy task. cfl2002 Massachusetts Institute of Technology. All rights reserved

34 L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format Form the Lagrangian function: Form the dual function L Λ (u): i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The dual problem is then defined to be: D : v Λ = maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

35 OP : z Λ = minimum x Steps in the Construction of the Dual Problem f(x) Step 1. Create the Lagrangian g i (x)» 0; i = 1;:::;m; s:t: 2 P; x u) := f(x) + u T g(x) : L(x; Step 2. Create the dual function: L Λ (u) := minimum x f(x) + u T g(x) Step 3. Create the dual problem: s:t: x 2 P D : v Λ = maximum u L Λ (u) s:t: u 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

36 LP : minimum x Examples of Dual Constructions of OPs The Dual of a Linear Problem Consider the linear optimization problem: c T x s:t: Ax b What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

37 IP : minimum x The Dual of a Binary Integer Problem Examples of Dual Constructions of OPs Consider the binary integer optimization problem: c T x s:t: b Ax j 2 f0; 1g; j = 1; : : : ; n : x What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

38 BP : minimum x1 ;x 2 ;x 3 5x 1 + 7x 2 4x 3 3 P Examples of Dual Constructions of OPs The Dual of a Log-Barrier Problem Consider the following logarithmic barrier problem: ln(x j ) j=1 s.t. x 1 + 3x x 3 = 37 x 1 > 0; x 2 > 0; x 3 > 0 : What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

39 OP : minimum x f(x) + P i2l u i g i (x) + P u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints Remarks... f(x) s:t: i (x)» 0; i 2 L g i (x) 0; i 2 G g g i (x) = 0; i 2 E The Lagrangian is: x 2 P; L(x; u) := f(x) + u T g(x) = u i g i (x) cfl2002 Massachusetts Institute of Technology. All rights reserved i2g i2e

40 u) := f(x) + u T g(x) = f(x) + P L(x; u i g i (x) + P i2l D : v Λ = maximum u u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints...Remarks u i g i (x) The dual function L Λ (u): i2g i2e L Λ (u) := minimum x f(x) + u T g(x) The dual problem is: s:t: x 2 P L Λ (u) s:t: 0; i 2 L u» 0; i 2 G u > < 0; i 2 E u cfl2002 Massachusetts Institute of Technology. All rights reserved

41 OP : z Λ = minimum x The Dual is a Concave Maximization Problem f(x) s:t: g i (x)» 0; i = 1; : : : ; m x 2 P; L(x; u) := f(x) + u T g(x) L Λ (u) := minimum x f(x) + u T g(x) s:t: x 2 P Theorem 3 The dual function L Λ (u) is a concave function. cfl2002 Massachusetts Institute of Technology. All rights reserved

42 = min x2p f(x) + u T 1 g(x) Λ + (1 ) f(x) + u T 2 g(x) Λ min x2p f(x) + u T 1 g(x) Λ + (1 ) min x2p (f(x) + u T 2 g(x) Λ The Dual is a Concave Maximization Problem Proof of Concavity of L Λ (u) Proof: Let u 1 0 and u 2 0 be two values of the dual variables, and let u = u 1 + (1 )u 2, where 2 [0; 1]. Then L Λ (u) = min x2p f(x) + u T g(x) = L Λ (u 1 ) + (1 )L Λ (u 2 ) : Therefore L Λ (u) is a concave function. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

43 OP : z Λ = minimum x g i (x)» 0; i = 1;:::;m s:t: 2 P; x Weak Duality f(x) D : v Λ = maximum u L Λ (u) s:t: u 0 Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) Λ v Λ : z cfl2002 Massachusetts Institute of Technology. All rights reserved

44 Weak Duality Proof of Weak Duality Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) z Λ v Λ : Proof: If μx is feasible for OP and μu is feasible for D, then + T g(μx) min x2p f(x) + μu T g(x) = L Λ (μu) : f(μx) f(μx) μu v Therefore. q.e.d. Λ z Λ cfl2002 Massachusetts Institute of Technology. All rights reserved

45 OP : z Λ = minimum x A = 1 (x) g 2 (x) g m (x) g f(x) A = Column Geometry of Primal and Dual Problems Resources and Costs For this section only, we assume equality format: f(x) g i (x) = 0; i = 1;:::;m s:t: 2 P; x D : v Λ = maximum u L Λ (u) s:t: u > < 0 For each x 2 P, we have an array of resources and costs associated with x: r z 1 r 2 r B = B. C C B B. C C g(x) r m z cfl2002 Massachusetts Institute of Technology. All rights reserved

46 Column Geometry of Primal and Dual Problems Resources and Costs S := Φ (r; z) 2 < m+1 j (r; z) = (g(x); f(x)) for some x 2 P Ψ : z L = {( r, z ) r = 0 } S H r 1 r 2 r = 0 The column geometry of the primal and dual problem. cfl2002 Massachusetts Institute of Technology. All rights reserved

47 Column Geometry of Primal and Dual Problems Column Geometry Interpretation of OP Every point in the intersection of S and L corresponds to a feasible solution of OP, and the feasible solution with the lowest cost corresponds to the lowest point in the intersection of S and L. This lowest value is exactly the value of the primal problem, namely z Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

48 Column Geometry of Primal and Dual Problems Supporting Hyperplanes of S H = H u;ff = Φ (r; z) 2 < m+1 j z + u T r = ff Ψ : L = Φ (r; z) 2 < m+1 j r = 0 Ψ : H L = f(r; z) = (0; ff)g cfl2002 Massachusetts Institute of Technology. All rights reserved

49 maximum u;ff Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: H u;ff lies below S = maximum u;ff ff s:t: z + u T r ff; for all (r; z) 2 S = maximum u;ff ff s:t: f(x) + u T g(x) ff; for all x 2 P cfl2002 Massachusetts Institute of Technology. All rights reserved

50 maximum u;ff > 0 : < Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: f(x) + u T g(x) ff; for all x 2 P = maximum u;ff ff s:t: L Λ (u) ff; = maximum u L Λ (u) s:t: u cfl2002 Massachusetts Institute of Technology. All rights reserved

51 Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S The dual problem corresponds to finding that hyperplane H u;ff lying below S whose intersection with L is the highest. This highest value is exactly the value of the dual problem, namely v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

52 Strong Duality Convexity and Strong Duality z L S r = 0 H r The column geometry of the dual problem when S is convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

53 Strong Duality Absence of Convexity z L z* v* S H r The column geometry of the dual problem when S is not convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

54 Strong Duality Strong Duality Theorem Theorem 5 Strong Duality Theorem: Suppose that OP is a convex optimization problem, that is, all constraints of OP are of the form g i (x)» 0; i = 1; : : : ; m, f(x) as well as g i (x) are convex functions, i = 1; : : : ; m, and P is a convex set. Then under very mild additional conditions, z Λ = v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

55 Duality Strategies Dualizing Bad Constraints OP : minimum x c T x s:t:» b Ax» g : Nx Suppose that optimization over the constraints Nx» g is easy. The addition of the constraints Ax» b makes the problem much more difficult. Let = fx j Nx» gg P cfl2002 Massachusetts Institute of Technology. All rights reserved

56 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: Nx» g : OP : minimum x c T x Ax» b s:t: 2 P : x cfl2002 Massachusetts Institute of Technology. All rights reserved

57 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: where x 2 P : The Lagrangian is: L(x; u) = c T x + u T (Ax b) = u T b + (c T + u T A)x L Λ (u) := minimum x u T b + (c T + u T A)x s:t: x 2 P : cfl2002 Massachusetts Institute of Technology. All rights reserved

58 Λ (u) := minimum x u T b + (c T + u T A)x L x 2 P : s:t: Duality Strategies Dualizing Bad Constraints D : maximum u L Λ (u) s:t: u 0 Notice that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem D. cfl2002 Massachusetts Institute of Technology. All rights reserved

59 OP : minimum x1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A 2 x 2» b 2 A Notice here that if it were not for the constraints 1 x 1 + B 2 x 2» d, that we would be able to separate B the problem into two separate problems. Let us dualize on these constraints. Let: P = Φ (x 1 ; x 2 ) j A 1 x 1» b 1 ; A 2 x 2» b 2Ψ cfl2002 Massachusetts Institute of Technology. All rights reserved

60 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 A 2 x 2» b 2 P = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A B 1 x 1 +B 2 x 2» d s:t: 1 ; x 2 ) 2 P (x cfl2002 Massachusetts Institute of Technology. All rights reserved

61 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 L Λ (u) = minimum x1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 ;x 2 ) 2 P (x = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ P The Lagrangian is: L(x; u) = (c 1 ) T x 1 + (c 2 ) T x 2 + u T (B 1 x 1 + B 2 x 2 d) = u T d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 ; (x 1 ; x 2 ) 2 P s:t: cfl2002 Massachusetts Institute of Technology. All rights reserved

62 L Λ (u) = minimum x 1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems s:t: (x 1 ;x 2 ) 2 P L Λ (u) = T d u minimum A1 x 1»b 1 ((c1 ) T + u T B 1 )x minimum A 2 x 2»b 2 ((c2 ) T + u T B 2 )x 2 Notice once again that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem: D : maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

63 Lagrange Duality Illustrated Constrained Shortest Path Problem Definition Constrained Shortest Path Problem: find the shortest path from node to node 6, among those paths using 4 exactly arcs cfl2002 Massachusetts Institute of Technology. All rights reserved

64 Lagrange Duality Illustrated Constrained Shortest Path Problem Context In practice: Arcs might have two types of costs (time and money) There may be millions of nodes and billions of paths cfl2002 Massachusetts Institute of Technology. All rights reserved

65 Lagrange Duality Illustrated Constrained Shortest Path Enumeration of Paths Path Number of Arcs Cost Path is the optimal solution. cfl2002 Massachusetts Institute of Technology. All rights reserved

66 Lagrange Duality Illustrated Constrained Shortest Path Motivation for Dual Suppose that we have available an efficient algorithm for solving (unconstrained) shortest path problems. We will use this algorithm to try to solve our problem through the dual cfl2002 Massachusetts Institute of Technology. All rights reserved

67 z Λ = min xij x 12 +x 13 +x 23 +x 24 +3x 34 +2x 35 +x 54 +x 46 +3x 56 Lagrange Duality Illustrated Constrained Shortest Path IP Formulation... CSPP : s.t. x 12 +x 13 +x 23 +x 24 +x 34 +x 35 +x 54 +x 46 +x 56 = 4 x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

68 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation... P i;j P x ij = 4 s:t: i;j Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

69 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation P i;j P x ij = 4 s:t: i;j where: x 2 P P = fx j Nx = b ; x ij 2 f0; 1g for all arcs i jg cfl2002 Massachusetts Institute of Technology. All rights reserved

70 L(x; u) := P i;j 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path The Lagrangian ψ x ij! c ij x ij + u cfl2002 Massachusetts Institute of Technology. All rights reserved

71 L Λ (u) := minimum xij 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path L Λ (u)... ψ P c ij x ij + u x ij! i;j s:t: Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

72 Lagrange Duality Illustrated Constrained Shortest Path...L Λ (u) Λ (u) = min xij (1 u)x 12 +(1 u)x 13 +(1 u)x 23 L u)x 24 +(3 u)x 34 +(2 u)x 35 +(1 +(1 u)x 54 +(1 u)x 46 +(3 u)x 56 +4u s.t. x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

73 Lagrange Duality Illustrated Constrained Shortest Path Comments on L Λ (u) For a fixed value of u, 4u is a constant. For a fixed value of u, this is a shortest path problem with modified arc costs given by: c 0 ij = c ij u : cfl2002 Massachusetts Institute of Technology. All rights reserved

74 Lagrange Duality Illustrated Constrained Shortest Path The Dual Problem D : v Λ = maximum u L Λ (u) s.t. u < > 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

75 Lagrange Duality Illustrated Constrained Shortest Path The set P Recall that P is the set of solutions of: x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

76 Lagrange Duality Illustrated Constrained Shortest Path Resources and Costs z (cost) conv(s) r (number of arcs) z* = 5 v* = 4.5 cfl2002 Massachusetts Institute of Technology. All rights reserved

77 Lagrange Duality Illustrated Constrained Shortest Path Duality Gap ffl The optimal u is u = 1:5 ffl The value of the dual is v Λ = 4:5 ffl The value of the primal is z Λ = 5:0 ffl There is a duality gap: z Λ v Λ = 5:0 4:5 = 0:5. cfl2002 Massachusetts Institute of Technology. All rights reserved

Duality Theory of Constrained Optimization

Duality Theory of Constrained Optimization Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

Lecture 4: Optimization. Maximizing a function of a single variable

Lecture 4: Optimization. Maximizing a function of a single variable Lecture 4: Optimization Maximizing or Minimizing a Function of a Single Variable Maximizing or Minimizing a Function of Many Variables Constrained Optimization Maximizing a function of a single variable

More information

Convex Optimization Boyd & Vandenberghe. 5. Duality

Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

5. Duality. Lagrangian

5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

More information

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

More information

ICS-E4030 Kernel Methods in Machine Learning

ICS-E4030 Kernel Methods in Machine Learning ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

More information

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

More information

Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

More information

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

More information

minimize x subject to (x 2)(x 4) u,

minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

More information

Lecture: Duality.

Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

More information

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

More information

Pattern Classification, and Quadratic Problems

Pattern Classification, and Quadratic Problems Pattern Classification, and Quadratic Problems (Robert M. Freund) March 3, 24 c 24 Massachusetts Institute of Technology. 1 1 Overview Pattern Classification, Linear Classifiers, and Quadratic Optimization

More information

1. f(β) 0 (that is, β is a feasible point for the constraints)

1. f(β) 0 (that is, β is a feasible point for the constraints) xvi 2. The lasso for linear models 2.10 Bibliographic notes Appendix Convex optimization with constraints In this Appendix we present an overview of convex optimization concepts that are particularly useful

More information

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction

More information

CSCI : Optimization and Control of Networks. Review on Convex Optimization

CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

More information

Lecture: Duality of LP, SOCP and SDP

Lecture: Duality of LP, SOCP and SDP 1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

More information

EE/AA 578, Univ of Washington, Fall Duality

EE/AA 578, Univ of Washington, Fall Duality 7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Additional Homework Problems

Additional Homework Problems Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.

More information

Optimization Theory. Lectures 4-6

Optimization Theory. Lectures 4-6 Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:

More information

A Brief Review on Convex Optimization

A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

More information

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian

More information

Applications of Linear Programming

Applications of Linear Programming Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

More information

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

More information

Lakehead University ECON 4117/5111 Mathematical Economics Fall 2002

Lakehead University ECON 4117/5111 Mathematical Economics Fall 2002 Test 1 September 20, 2002 1. Determine whether each of the following is a statement or not (answer yes or no): (a) Some sentences can be labelled true and false. (b) All students should study mathematics.

More information

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

More information

Lagrangian Duality Theory

Lagrangian Duality Theory Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

More information

Linear and non-linear programming

Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

More information

Introduction to Nonlinear Stochastic Programming

Introduction to Nonlinear Stochastic Programming School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS

More information

The Kuhn-Tucker Problem

The Kuhn-Tucker Problem Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

More information

MVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg

MVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material

More information

Chap 2. Optimality conditions

Chap 2. Optimality conditions Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

More information

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

More information

Lecture 18: Optimization Programming

Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

More information

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

More information

Algorithms for constrained local optimization

Algorithms for constrained local optimization Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

More information

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

More information

Nonlinear Programming (NLP)

Nonlinear Programming (NLP) Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume

More information

Convex Optimization. Ofer Meshi. Lecture 6: Lower Bounds Constrained Optimization

Convex Optimization. Ofer Meshi. Lecture 6: Lower Bounds Constrained Optimization Convex Optimization Ofer Meshi Lecture 6: Lower Bounds Constrained Optimization Lower Bounds Some upper bounds: #iter μ 2 M #iter 2 M #iter L L μ 2 Oracle/ops GD κ log 1/ε M x # ε L # x # L # ε # με f

More information

Generalization to inequality constrained problem. Maximize

Generalization to inequality constrained problem. Maximize Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

More information

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

More information

Convex Optimization and Support Vector Machine

Convex Optimization and Support Vector Machine Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

More information

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

More information

Lagrangian Methods for Constrained Optimization

Lagrangian Methods for Constrained Optimization Pricing Communication Networks: Economics, Technology and Modelling. Costas Courcoubetis and Richard Weber Copyright 2003 John Wiley & Sons, Ltd. ISBN: 0-470-85130-9 Appendix A Lagrangian Methods for Constrained

More information

CE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review

CE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley

More information

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach

More information

Lagrangian Duality for Dummies

Lagrangian Duality for Dummies Lagrangian Duality for Dummies David Knowles November 13, 2010 We want to solve the following optimisation problem: f 0 () (1) such that f i () 0 i 1,..., m (2) For now we do not need to assume conveity.

More information

Computational Finance

Computational Finance Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples

More information

Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization

Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization Duality Uses and Correspondences Ryan Tibshirani Conve Optimization 10-725 Recall that for the problem Last time: KKT conditions subject to f() h i () 0, i = 1,... m l j () = 0, j = 1,... r the KKT conditions

More information

Lagrange Relaxation and Duality

Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

More information

Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering

Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization

More information

Duality. Geoff Gordon & Ryan Tibshirani Optimization /

Duality. Geoff Gordon & Ryan Tibshirani Optimization / Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

More information

CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS

CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics

More information

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko

More information

Massachusetts Institute of Technology 6.854J/18.415J: Advanced Algorithms Friday, March 18, 2016 Ankur Moitra. Problem Set 6

Massachusetts Institute of Technology 6.854J/18.415J: Advanced Algorithms Friday, March 18, 2016 Ankur Moitra. Problem Set 6 Massachusetts Institute of Technology 6.854J/18.415J: Advanced Algorithms Friday, March 18, 2016 Ankur Moitra Problem Set 6 Due: Wednesday, April 6, 2016 7 pm Dropbox Outside Stata G5 Collaboration policy:

More information

MATHEMATICAL ECONOMICS: OPTIMIZATION. Contents

MATHEMATICAL ECONOMICS: OPTIMIZATION. Contents MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4

More information

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

More information

Lecture 8. Strong Duality Results. September 22, 2008

Lecture 8. Strong Duality Results. September 22, 2008 Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

More information

4. Algebra and Duality

4. Algebra and Duality 4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone

More information

Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1

Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with

More information

2 ffl Construct a ve r s i o n o f t h e F ran-wolfe method and solve this problem starting from the point ( x1; x 2 ) = (0 :5; 3:0). ffl Plot the ite

2 ffl Construct a ve r s i o n o f t h e F ran-wolfe method and solve this problem starting from the point ( x1; x 2 ) = (0 :5; 3:0). ffl Plot the ite 15.094/SMA5223 Systems Optimization: Models and Computation Assignment 2 (85 p o i n ts) Due March 2, 2004 1 Lagrangian Dual (10 points) Consider the problem: 1 x T x + c T x P : min x f (x) = 2 s.t. Ax

More information

Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

More information

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

More information

Calculus and optimization

Calculus and optimization Calculus an optimization These notes essentially correspon to mathematical appenix 2 in the text. 1 Functions of a single variable Now that we have e ne functions we turn our attention to calculus. A function

More information

Nonlinear Optimization

Nonlinear Optimization Nonlinear Optimization Etienne de Klerk (UvT)/Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos Course WI3031 (Week 4) February-March, A.D. 2005 Optimization Group 1 Outline

More information

Lecture 3. Optimization Problems and Iterative Algorithms

Lecture 3. Optimization Problems and Iterative Algorithms Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

More information

Introduction to Support Vector Machines

Introduction to Support Vector Machines Introduction to Support Vector Machines Shivani Agarwal Support Vector Machines (SVMs) Algorithm for learning linear classifiers Motivated by idea of maximizing margin Efficient extension to non-linear

More information

Lecture 3: Semidefinite Programming

Lecture 3: Semidefinite Programming Lecture 3: Semidefinite Programming Lecture Outline Part I: Semidefinite programming, examples, canonical form, and duality Part II: Strong Duality Failure Examples Part III: Conditions for strong duality

More information

CO 250 Final Exam Guide

CO 250 Final Exam Guide Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,

More information

Interior-Point Methods for Linear Optimization

Interior-Point Methods for Linear Optimization Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function

More information

Lecture: Algorithms for LP, SOCP and SDP

Lecture: Algorithms for LP, SOCP and SDP 1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:

More information

Benders Decomposition Methods for Structured Optimization, including Stochastic Optimization

Benders Decomposition Methods for Structured Optimization, including Stochastic Optimization Benders Decomposition Methods for Structured Optimization, including Stochastic Optimization Robert M. Freund May 2, 2001 Block Ladder Structure Basic Model minimize x;y c T x + f T y s:t: Ax = b Bx +

More information

A ten page introduction to conic optimization

A ten page introduction to conic optimization CHAPTER 1 A ten page introduction to conic optimization This background chapter gives an introduction to conic optimization. We do not give proofs, but focus on important (for this thesis) tools and concepts.

More information

MATH2070 Optimisation

MATH2070 Optimisation MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints

More information

Homework Set #6 - Solutions

Homework Set #6 - Solutions EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique

More information

Support Vector Machines: Maximum Margin Classifiers

Support Vector Machines: Maximum Margin Classifiers Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind

More information

Tutorial on Convex Optimization: Part II

Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

More information

Duality of LPs and Applications

Duality of LPs and Applications Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will

More information

Linear and Combinatorial Optimization

Linear and Combinatorial Optimization Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

More information

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints

ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained

More information

Support Vector Machines for Regression

Support Vector Machines for Regression COMP-566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

More information

Support Vector Machines

Support Vector Machines Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal

More information

Maximum likelihood estimation

Maximum likelihood estimation Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization

More information

Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

More information

Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur

Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module - 5 Lecture - 22 SVM: The Dual Formulation Good morning.

More information

A Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions

A Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions A Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions Angelia Nedić and Asuman Ozdaglar April 15, 2006 Abstract We provide a unifying geometric framework for the

More information