# Applied Lagrange Duality for Constrained Optimization

Save this PDF as:

Size: px
Start display at page:

Download "Applied Lagrange Duality for Constrained Optimization" ## Transcription

1 Applied Lagrange Duality for Constrained Optimization February 12, 2002

2 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual Problem ffl Steps in the Construction of the Dual Problem ffl Examples of Dual Constructions ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

3 Overview The Dual is a Concave Maximization Problem ffl Weak Duality ffl The Column Geometry of the Primal and Dual Problems ffl Strong Duality ffl ffl Duality Strategies ffl Illustration of Lagrange Duality in Discrete Optimization cfl2002 Massachusetts Institute of Technology. All rights reserved

4 Context of Duality Previous Examples of Duality Models of electrical networks ffl Models of economic markets ffl Structural design ffl Duality in Linear Optimization ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

5 Context of Duality Importance of Duality Identifying near-optimal solutions ffl Proving optimality ffl Sensitivity analysis of the primal problem ffl Karush-Kuhn-Tucker (KKT) conditions ffl Convergence of improvement algorithms ffl Good Structure ffl Other uses, too... ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

6 Review of Convexity Local and Global Optima Problem Set-Up The ball centered at μx with radius ffl is the set: B(μx; ffl) := fxjkx μxk» fflg: Consider the following optimization problem over the set F: P : min x or max x f(x) s.t. x 2 F cfl2002 Massachusetts Institute of Technology. All rights reserved

7 Review of Convexity Local and Global Optima Definition x 2 F 1 is a local minimum P of if there ffl > 0 exists such f(x)» f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 2 is a global minimum P of if» f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

8 Review of Convexity Strict Local & Global Minima Definition x 2 F 3 is a strict local minimum P of if there ffl > 0 exists such f(x) < f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 4 is a strict global minimum P of if < f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

9 Review of Convexity Local & Global Maxima Definition x 2 F 5 is a local maximum P of if there ffl > 0 exists such f(x) f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 6 is a global maximum P of if f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

10 Review of Convexity Strict Local & Global Maxima Definition x 2 F 7 is a strict local maximum P of if there ffl > 0 exists such f(x) > f(y) that for all 2 B(x; ffl) F. y Definition x 2 F 8 is a strict global maximum P of if > f(y) for all y 2 F. f(x) cfl2002 Massachusetts Institute of Technology. All rights reserved

11 Review of Convexity Local versus Global Minima F Illustration of local versus global optima. cfl2002 Massachusetts Institute of Technology. All rights reserved

12 Review of Convexity Definition 9 Convex Sets and Functions A subset S ρ < n is a convex set if y 2 S ) x + (1 )y 2 S x; for 2 [0; any 1]. Illustration of convex and non-convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

13 Review of Convexity Intersections of Convex Sets Proposition 1 S; T If are convex sets, S T then is a convex set. Proposition 2 The intersection of any collection of convex sets is a convex set. Illustration of the intersection of convex sets. cfl2002 Massachusetts Institute of Technology. All rights reserved

14 Review of Convexity Convex Functions Definition 10 A function f(x) is a convex function if + (1 )y)» f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 11 function if A function f(x) is a strictly convex + (1 )y) < f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

15 Review of Convexity Convex Functions Illustration of convex and strictly convex functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

16 Review of Convexity Convex Optimization P : minimize x f(x) x 2 F s:t: Proposition 3 Suppose F that is a convex set, : F! < is a convex function, and μx is a local f minimum P of. μx Then is a global minimum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

17 Review of Convexity Proof of Proposition Proof of the Proposition: Suppose μx is not a global minimum, i.e., there exists y 2 F for which f(y) < f(μx). Let = μx + (1 )y, which is a convex combination of μx and y y( ) 2 [0; 1] for (and y( ) 2 F therefore, 2 [0; for 1]). Note that! μx as! 1. y( ) From the convexity of f( ), = f( μx + (1 )y)» f(μx) + (1 )f(y) < f(μx) + (1 )f(μx) = f(μx) f(y( )) for 2 (0; all 1). f(y( )) < f(μx) Therefore, for 2 (0; all 1), and μx so is not a local minimum, resulting in a contradiction. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

18 Review of Convexity Examples of Convex Functions Functions of One Variable Examples of convex functions of one variable: ffl f(x) = ax + b ffl f(x) = x 2 + bx + c ffl f(x) = jxj ffl f(x) = ln(x) for x > 0 for = 1 x x > 0 ffl f(x) = e x f(x) ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

19 Review of Convexity Concave Functions & Maximization The opposite of a convex function is a concave function: Definition 12 A function f(x) is a concave function if + (1 )y) f(x) + (1 )f(y) f( x for x all y and and for 2 [0; all 1]. Definition 13 A function f(x) is a strictly concave function if + (1 )y) > f(x) + (1 )f(y) f( x for x all y and and for 2 (0; all 1). cfl2002 Massachusetts Institute of Technology. All rights reserved

20 Review of Convexity Concave Functions & Maximization Illustration of concave and strictly concave functions. cfl2002 Massachusetts Institute of Technology. All rights reserved

21 Review of Convexity Concave Maximization Now consider the maximization problem P : P : maximize x f(x) s:t: x 2 F Proposition 4 Suppose F that is a convex set, : F! < is a concave function, and μx is a local f maximum P of. μx Then is a global maximum f of over F. cfl2002 Massachusetts Institute of Technology. All rights reserved

22 Review of Convexity Convex Program : min x or max x f(x) P x 2 F s.t. We call P a convex program if ffl F is a convex set, and we are minimizing and is a convex function, or ffl we are maximizing and f(x)is a concave function. Then every local optima of the objective function of a convex program is a global optima of the objective function. cfl2002 Massachusetts Institute of Technology. All rights reserved

23 Review of Convexity Convex Program The class of convex programs is the class of well-behaved optimization problems. cfl2002 Massachusetts Institute of Technology. All rights reserved

24 Review of Convexity Linear Functions Proposition 5 A linear f(x) = a function x + b is both convex and concave. T Proposition 6 f(x) If is both convex and concave, f(x) then is a linear function. A linear function is convex and concave. cfl2002 Massachusetts Institute of Technology. All rights reserved

25 Separating Hyperplane Theory Strong Separation Theorem 1 Strong Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S. Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x > ff for all x 2 S u T μx < ff cfl2002 Massachusetts Institute of Technology. All rights reserved

26 Separating Hyperplane Theory Strong Separation S x utx = α Illustration of strong separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

27 Separating Hyperplane Theory Weak Separation Theorem 2 Weak Separating Hyperplane Theorem: Suppose that S is a convex set in < n, and that we are given a point μx =2 S or μx Then there exists a vector u 6= 0 and a scalar ff for which the following hold: u T x ff for all x 2 S u T μx» ff cfl2002 Massachusetts Institute of Technology. All rights reserved

28 Separating Hyperplane Theory Weak Separation S x utx = α Illustration of weak separation of a point from a convex set. cfl2002 Massachusetts Institute of Technology. All rights reserved

29 Separating Hyperplane Theory Intersecting Line Dual Primal: minimum x height of x s:t: 2 S; x 2 L x Dual: maximum ß intercept of line ß with L s:t: S ß line lies below cfl2002 Massachusetts Institute of Technology. All rights reserved

30 Separating Hyperplane Theory Intersecting Line Dual z L S π Illustration intersecting line dual. r cfl2002 Massachusetts Institute of Technology. All rights reserved

31 OP : minimum x Definition of the Dual Problem General Constrained Problem Most General Format f(x) g 1 (x)» 0; s:t: =. m (x)» 0; g 2 P; x : < n 7! < f(x) i (x) : < n 7! <; i = 1; : : : ; m. g cfl2002 Massachusetts Institute of Technology. All rights reserved

32 ffl P = < n OP : z Λ = minimum x ffl P = Φ x j x 2 Z n + Ψ m (x)» 0; g 2 P; x Definition of the Dual Problem General Constrained Problem Standard Format... f(x) g 1 (x)» 0; s:t:. ffl P = fx j x 0g ffl P = fx j g i (x)» 0; i = m + 1; : : : ; m + kg P = fx j Ax» bg ffl cfl2002 Massachusetts Institute of Technology. All rights reserved

33 OP : z Λ = minimum x L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format... f(x) Form the Lagrangian function: g i (x)» 0; i = 1; : : : ; m; s:t: 2 P; x Solve the presumably easier problem: i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The L function (u) is called the dual function. Λ We presume that computing L Λ (u) is an easy task. cfl2002 Massachusetts Institute of Technology. All rights reserved

34 L(x; u) := f(x) + u T g(x) = f(x) + m P Definition of the Dual Problem General Constrained Problem...Standard Format Form the Lagrangian function: Form the dual function L Λ (u): i=1 u i g i (x) L Λ (u) := minimum x f(x) + u T g(x) x 2 P s:t: The dual problem is then defined to be: D : v Λ = maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

35 OP : z Λ = minimum x Steps in the Construction of the Dual Problem f(x) Step 1. Create the Lagrangian g i (x)» 0; i = 1;:::;m; s:t: 2 P; x u) := f(x) + u T g(x) : L(x; Step 2. Create the dual function: L Λ (u) := minimum x f(x) + u T g(x) Step 3. Create the dual problem: s:t: x 2 P D : v Λ = maximum u L Λ (u) s:t: u 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

36 LP : minimum x Examples of Dual Constructions of OPs The Dual of a Linear Problem Consider the linear optimization problem: c T x s:t: Ax b What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

37 IP : minimum x The Dual of a Binary Integer Problem Examples of Dual Constructions of OPs Consider the binary integer optimization problem: c T x s:t: b Ax j 2 f0; 1g; j = 1; : : : ; n : x What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

38 BP : minimum x1 ;x 2 ;x 3 5x 1 + 7x 2 4x 3 3 P Examples of Dual Constructions of OPs The Dual of a Log-Barrier Problem Consider the following logarithmic barrier problem: ln(x j ) j=1 s.t. x 1 + 3x x 3 = 37 x 1 > 0; x 2 > 0; x 3 > 0 : What is the dual of this problem? cfl2002 Massachusetts Institute of Technology. All rights reserved

39 OP : minimum x f(x) + P i2l u i g i (x) + P u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints Remarks... f(x) s:t: i (x)» 0; i 2 L g i (x) 0; i 2 G g g i (x) = 0; i 2 E The Lagrangian is: x 2 P; L(x; u) := f(x) + u T g(x) = u i g i (x) cfl2002 Massachusetts Institute of Technology. All rights reserved i2g i2e

40 u) := f(x) + u T g(x) = f(x) + P L(x; u i g i (x) + P i2l D : v Λ = maximum u u i g i (x) + P Examples of Dual Constructions of OPs Problems w/ Different Formats of Constraints...Remarks u i g i (x) The dual function L Λ (u): i2g i2e L Λ (u) := minimum x f(x) + u T g(x) The dual problem is: s:t: x 2 P L Λ (u) s:t: 0; i 2 L u» 0; i 2 G u > < 0; i 2 E u cfl2002 Massachusetts Institute of Technology. All rights reserved

41 OP : z Λ = minimum x The Dual is a Concave Maximization Problem f(x) s:t: g i (x)» 0; i = 1; : : : ; m x 2 P; L(x; u) := f(x) + u T g(x) L Λ (u) := minimum x f(x) + u T g(x) s:t: x 2 P Theorem 3 The dual function L Λ (u) is a concave function. cfl2002 Massachusetts Institute of Technology. All rights reserved

42 = min x2p f(x) + u T 1 g(x) Λ + (1 ) f(x) + u T 2 g(x) Λ min x2p f(x) + u T 1 g(x) Λ + (1 ) min x2p (f(x) + u T 2 g(x) Λ The Dual is a Concave Maximization Problem Proof of Concavity of L Λ (u) Proof: Let u 1 0 and u 2 0 be two values of the dual variables, and let u = u 1 + (1 )u 2, where 2 [0; 1]. Then L Λ (u) = min x2p f(x) + u T g(x) = L Λ (u 1 ) + (1 )L Λ (u 2 ) : Therefore L Λ (u) is a concave function. q.e.d. cfl2002 Massachusetts Institute of Technology. All rights reserved

43 OP : z Λ = minimum x g i (x)» 0; i = 1;:::;m s:t: 2 P; x Weak Duality f(x) D : v Λ = maximum u L Λ (u) s:t: u 0 Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) Λ v Λ : z cfl2002 Massachusetts Institute of Technology. All rights reserved

44 Weak Duality Proof of Weak Duality Theorem 4 Weak Duality Theorem: If μx is feasible for OP and μu is feasible for D, then In particular, f(μx) L Λ (μu) z Λ v Λ : Proof: If μx is feasible for OP and μu is feasible for D, then + T g(μx) min x2p f(x) + μu T g(x) = L Λ (μu) : f(μx) f(μx) μu v Therefore. q.e.d. Λ z Λ cfl2002 Massachusetts Institute of Technology. All rights reserved

45 OP : z Λ = minimum x A = 1 (x) g 2 (x) g m (x) g f(x) A = Column Geometry of Primal and Dual Problems Resources and Costs For this section only, we assume equality format: f(x) g i (x) = 0; i = 1;:::;m s:t: 2 P; x D : v Λ = maximum u L Λ (u) s:t: u > < 0 For each x 2 P, we have an array of resources and costs associated with x: r z 1 r 2 r B = B. C C B B. C C g(x) r m z cfl2002 Massachusetts Institute of Technology. All rights reserved

46 Column Geometry of Primal and Dual Problems Resources and Costs S := Φ (r; z) 2 < m+1 j (r; z) = (g(x); f(x)) for some x 2 P Ψ : z L = {( r, z ) r = 0 } S H r 1 r 2 r = 0 The column geometry of the primal and dual problem. cfl2002 Massachusetts Institute of Technology. All rights reserved

47 Column Geometry of Primal and Dual Problems Column Geometry Interpretation of OP Every point in the intersection of S and L corresponds to a feasible solution of OP, and the feasible solution with the lowest cost corresponds to the lowest point in the intersection of S and L. This lowest value is exactly the value of the primal problem, namely z Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

48 Column Geometry of Primal and Dual Problems Supporting Hyperplanes of S H = H u;ff = Φ (r; z) 2 < m+1 j z + u T r = ff Ψ : L = Φ (r; z) 2 < m+1 j r = 0 Ψ : H L = f(r; z) = (0; ff)g cfl2002 Massachusetts Institute of Technology. All rights reserved

49 maximum u;ff Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: H u;ff lies below S = maximum u;ff ff s:t: z + u T r ff; for all (r; z) 2 S = maximum u;ff ff s:t: f(x) + u T g(x) ff; for all x 2 P cfl2002 Massachusetts Institute of Technology. All rights reserved

50 maximum u;ff > 0 : < Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S ff s:t: f(x) + u T g(x) ff; for all x 2 P = maximum u;ff ff s:t: L Λ (u) ff; = maximum u L Λ (u) s:t: u cfl2002 Massachusetts Institute of Technology. All rights reserved

51 Column Geometry of Primal and Dual Problems Maximum-Height Supporting Hyperplane of S The dual problem corresponds to finding that hyperplane H u;ff lying below S whose intersection with L is the highest. This highest value is exactly the value of the dual problem, namely v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

52 Strong Duality Convexity and Strong Duality z L S r = 0 H r The column geometry of the dual problem when S is convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

53 Strong Duality Absence of Convexity z L z* v* S H r The column geometry of the dual problem when S is not convex. cfl2002 Massachusetts Institute of Technology. All rights reserved

54 Strong Duality Strong Duality Theorem Theorem 5 Strong Duality Theorem: Suppose that OP is a convex optimization problem, that is, all constraints of OP are of the form g i (x)» 0; i = 1; : : : ; m, f(x) as well as g i (x) are convex functions, i = 1; : : : ; m, and P is a convex set. Then under very mild additional conditions, z Λ = v Λ. cfl2002 Massachusetts Institute of Technology. All rights reserved

55 Duality Strategies Dualizing Bad Constraints OP : minimum x c T x s:t:» b Ax» g : Nx Suppose that optimization over the constraints Nx» g is easy. The addition of the constraints Ax» b makes the problem much more difficult. Let = fx j Nx» gg P cfl2002 Massachusetts Institute of Technology. All rights reserved

56 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: Nx» g : OP : minimum x c T x Ax» b s:t: 2 P : x cfl2002 Massachusetts Institute of Technology. All rights reserved

57 P = fx j Nx» gg Duality Strategies Dualizing Bad Constraints : minimum x c T x OP Ax» b s:t: where x 2 P : The Lagrangian is: L(x; u) = c T x + u T (Ax b) = u T b + (c T + u T A)x L Λ (u) := minimum x u T b + (c T + u T A)x s:t: x 2 P : cfl2002 Massachusetts Institute of Technology. All rights reserved

58 Λ (u) := minimum x u T b + (c T + u T A)x L x 2 P : s:t: Duality Strategies Dualizing Bad Constraints D : maximum u L Λ (u) s:t: u 0 Notice that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem D. cfl2002 Massachusetts Institute of Technology. All rights reserved

59 OP : minimum x1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A 2 x 2» b 2 A Notice here that if it were not for the constraints 1 x 1 + B 2 x 2» d, that we would be able to separate B the problem into two separate problems. Let us dualize on these constraints. Let: P = Φ (x 1 ; x 2 ) j A 1 x 1» b 1 ; A 2 x 2» b 2Ψ cfl2002 Massachusetts Institute of Technology. All rights reserved

60 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 A 2 x 2» b 2 P = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 x 1» b 1 A B 1 x 1 +B 2 x 2» d s:t: 1 ; x 2 ) 2 P (x cfl2002 Massachusetts Institute of Technology. All rights reserved

61 OP : minimum x 1 ;x 2 (c1 ) T x 1 +(c 2 ) T x 2 L Λ (u) = minimum x1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems B 1 x 1 +B 2 x 2» d s:t: 1 ;x 2 ) 2 P (x = Φ (x 1 ;x 2 ) j A 1 x 1» b 1 ;A 2 x 2» b 2Ψ P The Lagrangian is: L(x; u) = (c 1 ) T x 1 + (c 2 ) T x 2 + u T (B 1 x 1 + B 2 x 2 d) = u T d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 ; (x 1 ; x 2 ) 2 P s:t: cfl2002 Massachusetts Institute of Technology. All rights reserved

62 L Λ (u) = minimum x 1 ;x 2 ut d + ((c 1 ) T + u T B 1 )x 1 + ((c 2 ) T + u T B 2 )x 2 Duality Strategies Dualizing A Large Problem into Many Small Problems s:t: (x 1 ;x 2 ) 2 P L Λ (u) = T d u minimum A1 x 1»b 1 ((c1 ) T + u T B 1 )x minimum A 2 x 2»b 2 ((c2 ) T + u T B 2 )x 2 Notice once again that L Λ (u) is easy to evaluate for any value of u, and so we can attempt to solve OP by designing an algorithm to solve the dual problem: D : maximum u L Λ (u) cfl2002 Massachusetts Institute of Technology. All rights reserved

63 Lagrange Duality Illustrated Constrained Shortest Path Problem Definition Constrained Shortest Path Problem: find the shortest path from node to node 6, among those paths using 4 exactly arcs cfl2002 Massachusetts Institute of Technology. All rights reserved

64 Lagrange Duality Illustrated Constrained Shortest Path Problem Context In practice: Arcs might have two types of costs (time and money) There may be millions of nodes and billions of paths cfl2002 Massachusetts Institute of Technology. All rights reserved

65 Lagrange Duality Illustrated Constrained Shortest Path Enumeration of Paths Path Number of Arcs Cost Path is the optimal solution. cfl2002 Massachusetts Institute of Technology. All rights reserved

66 Lagrange Duality Illustrated Constrained Shortest Path Motivation for Dual Suppose that we have available an efficient algorithm for solving (unconstrained) shortest path problems. We will use this algorithm to try to solve our problem through the dual cfl2002 Massachusetts Institute of Technology. All rights reserved

67 z Λ = min xij x 12 +x 13 +x 23 +x 24 +3x 34 +2x 35 +x 54 +x 46 +3x 56 Lagrange Duality Illustrated Constrained Shortest Path IP Formulation... CSPP : s.t. x 12 +x 13 +x 23 +x 24 +x 34 +x 35 +x 54 +x 46 +x 56 = 4 x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

68 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation... P i;j P x ij = 4 s:t: i;j Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

69 CSPP : minimum xij c ij x ij Lagrange Duality Illustrated Constrained Shortest Path...IP Formulation P i;j P x ij = 4 s:t: i;j where: x 2 P P = fx j Nx = b ; x ij 2 f0; 1g for all arcs i jg cfl2002 Massachusetts Institute of Technology. All rights reserved

70 L(x; u) := P i;j 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path The Lagrangian ψ x ij! c ij x ij + u cfl2002 Massachusetts Institute of Technology. All rights reserved

71 L Λ (u) := minimum xij 4 P i;j Lagrange Duality Illustrated Constrained Shortest Path L Λ (u)... ψ P c ij x ij + u x ij! i;j s:t: Nx = b x ij 2 f0; 1g cfl2002 Massachusetts Institute of Technology. All rights reserved

72 Lagrange Duality Illustrated Constrained Shortest Path...L Λ (u) Λ (u) = min xij (1 u)x 12 +(1 u)x 13 +(1 u)x 23 L u)x 24 +(3 u)x 34 +(2 u)x 35 +(1 +(1 u)x 54 +(1 u)x 46 +(3 u)x 56 +4u s.t. x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

73 Lagrange Duality Illustrated Constrained Shortest Path Comments on L Λ (u) For a fixed value of u, 4u is a constant. For a fixed value of u, this is a shortest path problem with modified arc costs given by: c 0 ij = c ij u : cfl2002 Massachusetts Institute of Technology. All rights reserved

74 Lagrange Duality Illustrated Constrained Shortest Path The Dual Problem D : v Λ = maximum u L Λ (u) s.t. u < > 0 cfl2002 Massachusetts Institute of Technology. All rights reserved

75 Lagrange Duality Illustrated Constrained Shortest Path The set P Recall that P is the set of solutions of: x 12 +x 13 = 1 x 12 x 23 x 24 = 0 x 13 +x 23 x 34 x 35 = 0 x 24 +x 34 +x 54 x 46 = 0 x 35 x 54 x 56 = 0 x 46 +x 56 = 1 x ij 2 f0; 1g for all x ij. cfl2002 Massachusetts Institute of Technology. All rights reserved

76 Lagrange Duality Illustrated Constrained Shortest Path Resources and Costs z (cost) conv(s) r (number of arcs) z* = 5 v* = 4.5 cfl2002 Massachusetts Institute of Technology. All rights reserved

77 Lagrange Duality Illustrated Constrained Shortest Path Duality Gap ffl The optimal u is u = 1:5 ffl The value of the dual is v Λ = 4:5 ffl The value of the primal is z Λ = 5:0 ffl There is a duality gap: z Λ v Λ = 5:0 4:5 = 0:5. cfl2002 Massachusetts Institute of Technology. All rights reserved

### Duality Theory of Constrained Optimization Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive

### Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

### Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### Lecture 4: Optimization. Maximizing a function of a single variable Lecture 4: Optimization Maximizing or Minimizing a Function of a Single Variable Maximizing or Minimizing a Function of Many Variables Constrained Optimization Maximizing a function of a single variable

### Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### 5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

### I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

### ICS-E4030 Kernel Methods in Machine Learning ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

### Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

### Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

### Convex Optimization & Lagrange Duality Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

### Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

### minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

### Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

### Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

### Pattern Classification, and Quadratic Problems Pattern Classification, and Quadratic Problems (Robert M. Freund) March 3, 24 c 24 Massachusetts Institute of Technology. 1 1 Overview Pattern Classification, Linear Classifiers, and Quadratic Optimization

### 1. f(β) 0 (that is, β is a feasible point for the constraints) xvi 2. The lasso for linear models 2.10 Bibliographic notes Appendix Convex optimization with constraints In this Appendix we present an overview of convex optimization concepts that are particularly useful

### UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction

### CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

### Lecture: Duality of LP, SOCP and SDP 1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

### EE/AA 578, Univ of Washington, Fall Duality 7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Additional Homework Problems Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.

### Optimization Theory. Lectures 4-6 Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x\$0} Existence of a maximum:

### A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

### Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian

### Applications of Linear Programming Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

### Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

### Lakehead University ECON 4117/5111 Mathematical Economics Fall 2002 Test 1 September 20, 2002 1. Determine whether each of the following is a statement or not (answer yes or no): (a) Some sentences can be labelled true and false. (b) All students should study mathematics.

### Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

### Lagrangian Duality Theory Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

### Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

### Introduction to Nonlinear Stochastic Programming School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS

### The Kuhn-Tucker Problem Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

### MVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material

### Chap 2. Optimality conditions Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

### Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

### Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

### Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

### Algorithms for constrained local optimization Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

### Nonlinear Programming (NLP) Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume

### Convex Optimization. Ofer Meshi. Lecture 6: Lower Bounds Constrained Optimization Convex Optimization Ofer Meshi Lecture 6: Lower Bounds Constrained Optimization Lower Bounds Some upper bounds: #iter μ 2 M #iter 2 M #iter L L μ 2 Oracle/ops GD κ log 1/ε M x # ε L # x # L # ε # με f

### Generalization to inequality constrained problem. Maximize Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

### Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

### Convex Optimization and Support Vector Machine Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

### Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

### Lagrangian Methods for Constrained Optimization Pricing Communication Networks: Economics, Technology and Modelling. Costas Courcoubetis and Richard Weber Copyright 2003 John Wiley & Sons, Ltd. ISBN: 0-470-85130-9 Appendix A Lagrangian Methods for Constrained

### CE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley

### Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach

### Lagrangian Duality for Dummies Lagrangian Duality for Dummies David Knowles November 13, 2010 We want to solve the following optimisation problem: f 0 () (1) such that f i () 0 i 1,..., m (2) For now we do not need to assume conveity.

### Computational Finance Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples

### Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization Duality Uses and Correspondences Ryan Tibshirani Conve Optimization 10-725 Recall that for the problem Last time: KKT conditions subject to f() h i () 0, i = 1,... m l j () = 0, j = 1,... r the KKT conditions

### Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

### Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization

### Duality. Geoff Gordon & Ryan Tibshirani Optimization / Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

### CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko

### Massachusetts Institute of Technology 6.854J/18.415J: Advanced Algorithms Friday, March 18, 2016 Ankur Moitra. Problem Set 6 Massachusetts Institute of Technology 6.854J/18.415J: Advanced Algorithms Friday, March 18, 2016 Ankur Moitra Problem Set 6 Due: Wednesday, April 6, 2016 7 pm Dropbox Outside Stata G5 Collaboration policy:

### MATHEMATICAL ECONOMICS: OPTIMIZATION. Contents MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4

### Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

### Lecture 8. Strong Duality Results. September 22, 2008 Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

### 4. Algebra and Duality 4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone

### Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with

### 2 ffl Construct a ve r s i o n o f t h e F ran-wolfe method and solve this problem starting from the point ( x1; x 2 ) = (0 :5; 3:0). ffl Plot the ite 15.094/SMA5223 Systems Optimization: Models and Computation Assignment 2 (85 p o i n ts) Due March 2, 2004 1 Lagrangian Dual (10 points) Consider the problem: 1 x T x + c T x P : min x f (x) = 2 s.t. Ax

### Nonlinear Optimization: What s important? Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### Calculus and optimization Calculus an optimization These notes essentially correspon to mathematical appenix 2 in the text. 1 Functions of a single variable Now that we have e ne functions we turn our attention to calculus. A function

### Nonlinear Optimization Nonlinear Optimization Etienne de Klerk (UvT)/Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos Course WI3031 (Week 4) February-March, A.D. 2005 Optimization Group 1 Outline

### Lecture 3. Optimization Problems and Iterative Algorithms Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

### Introduction to Support Vector Machines Introduction to Support Vector Machines Shivani Agarwal Support Vector Machines (SVMs) Algorithm for learning linear classifiers Motivated by idea of maximizing margin Efficient extension to non-linear

### Lecture 3: Semidefinite Programming Lecture 3: Semidefinite Programming Lecture Outline Part I: Semidefinite programming, examples, canonical form, and duality Part II: Strong Duality Failure Examples Part III: Conditions for strong duality

### CO 250 Final Exam Guide Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,

### Interior-Point Methods for Linear Optimization Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function

### Lecture: Algorithms for LP, SOCP and SDP 1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:

### Benders Decomposition Methods for Structured Optimization, including Stochastic Optimization Benders Decomposition Methods for Structured Optimization, including Stochastic Optimization Robert M. Freund May 2, 2001 Block Ladder Structure Basic Model minimize x;y c T x + f T y s:t: Ax = b Bx +

### A ten page introduction to conic optimization CHAPTER 1 A ten page introduction to conic optimization This background chapter gives an introduction to conic optimization. We do not give proofs, but focus on important (for this thesis) tools and concepts.

### MATH2070 Optimisation MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints

### Homework Set #6 - Solutions EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique

### Support Vector Machines: Maximum Margin Classifiers Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind

### Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

### Duality of LPs and Applications Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will

### Linear and Combinatorial Optimization Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained

### Support Vector Machines for Regression COMP-566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### Support Vector Machines Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal

### Maximum likelihood estimation Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization

### Lectures 9 and 10: Constrained optimization problems and their optimality conditions Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module - 5 Lecture - 22 SVM: The Dual Formulation Good morning. A Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions Angelia Nedić and Asuman Ozdaglar April 15, 2006 Abstract We provide a unifying geometric framework for the