Agenda 1 Duality for LP 2 Theorem of alternatives 3 Conic Duality 4 Dual cones 5 Geometric view of cone programs 6 Conic duality theorem 7 Examples
Lower bounds on LPs By eliminating variables (if needed) any LP can be put in inequality form minimize c T x subject to Ax b We use this form for simplicity Purpose of Lagrange duality: get a lower bound on optimal value p?
Lagrange duality Lagrangian: L : R n R m! R L(x, )=c T x T (Ax b) : Lagrange multipliers (Lagrange) dual function g( )=inf x2r n L(x, )= Suppose 0 & x is feasible ( b T A T = c 1 otherwise L(x, )=c T x T (Ax b) apple c T x This implies that g( ) apple c T x for all feasible x ) g( ) apple inf x:ax b ct x = p?! Dual function always gives a lower bound on optimal value p?
Dual problem Dual problem: best lower bound on p? maximize g( ) subject to 0 () maximize subject to A T = c 0 b T (Lagrange) dual problem This is an LP Optimal value d? obeys d? apple p? :weakduality
LP duality Theorem For LP, strong duality always holds (remarkable and very useful fact) d? = p? p? = 1 (primal infeasible) =) d? = p? p? = 1 =) d? = 1 (dual infeasible)
Theorem of alternatives and strong LP duality min {c T x : Ax b} How do we decide if LP is feasible? How can we certify feasible set is empty? One possible approach: combine inequalities and get a contradiction 0 and Ax b =) T Ax T b Get such a contradiction if A T =0and b T >0 (only way to be sure) Remarkably, this is su cient but also necessary Theorem (Theorem of alternatives) Exactly one of (1) and (2) is true (1) 9x s.t. Ax b (2) 9 0 s.t. A T =0and b T >0
Lower bounds for LPs min {c T x : Ax b} Theorem of alternatives: a apple p? i a apple p?, {x : Ax b and c T x<a} = ; 9( 0, ) 0 s.t. 0c + A T =0 0 + b T 0 If LP is feasible, can take 0 > 0, hencea apple p? i system below has a solution (D) A T b T 0 = c a Proposition (D) has a sol (, a) =) a apple p? LP is feasible and a apple p? =) existence of a sol to (D)
Strong duality Proposition (D) has a sol (, a) =) a apple p? LP is feasible and a apple p? =) existence of a sol to (D) (, a) sol to (D) =) (, b T ) sol to (D) Since b T a, best lower bound max {b T : A T = c, 0} Conclusion This proves strong duality feasible for dual problem =) b T apple p? (d? apple p? ) If LP feasible, then for every a apple p?, 9 dual feasible with b T a ) d? = p?
Proof of theorem of alternatives
Similar notion for cone programs? Conic program minimize subject to c T x Ax K b Interested in something similar to LP s 0 and Ax b 0 =) h, Ax bi 0 For cones, we would like a set C such that 2 C and a K 0 =) h, ai 0 If 2 C and x is feasible, we get a lower bound hc, xi h, Ax bi applehc, xi
Or similarly For LP s: 0 gives min {c T x : Ax K 0} Ax b =) T Ax T b If A T = c, we get a lower bound For cones, we would like a set C such that 2 C and a K 0 =) h, ai 0 Best lower bound max {b T : 2 C and A T = c}
Dual cones Definition Dual cone K K = { 2 R n : T a 0 8a 2 K}
Properties of dual cone K 6= ; (nonempty set) 1 K is a closed convex cone 2 int(k) 6= ; =) K is pointed 3 K closed, convex and pointed =) int(k ) 6= ; 4 K closed, convex and pointed =) K closed and convex (K ) = K Corollary: K proper () K proper
Dual of a conic problem Primal (P) minimize subject to c T x Ax K b Lagrangian L(x, )=c T x T (Ax b) 2 K Dual (D) maximize subject to A T = c 2 K b T Weak duality: d apple p x feasible, feasible =) b T apple c T x
Dual of conic problem, continued (P) minimize c T x subject to A i x Ki b i, i =1,...,m (D) maximize subject to P b T P i i A T i i = c i K i 0, i =1,...,m
Geometrical view of primal/dual pair (P) minimize subject to c T x Ax K b (D) maximize subject to A T = c K 0 b T Primal: minimize linear function over an a ne slice of cone K Dual: maximize linear function over an a ne slice of cone K Same problem: only di erence due to how we represent the problem
Assume 9d : c = A T d () c 2 row(a) otherwise dual is infeasible: d? = 1 and p? 2 {±1} Assume for simplicity A has full column rank (so that c = A T d as a solution) Primal feasible set: v = Ax b (i) v 2 V b, V = range(a) (ii) v K 0 (iii) Gives c T x = d T Ax = d T b + d T v Conic program can be formulated as min {d T b + d T v : v 2 V b, v K 0} Dual feasible set: A T = c, 2 K () 2 V? + d, 2 K So that dual problem is max {b T : 2 V? + d, K 0}
Geometric view of primal/dual pair, continued dual feasible K * primal feasible R - b K
Symmetry of duality (P) minimize subject to c T x Ax K b (D) If K is proper, then (P) is dual to (D) because K = K maximize subject to A T = c K 0 b T dual of (D) minimize T c subject to v K 0 v = A b () minimize ct x subject to Ax b K 0
Examples of dual cones The following are all self-dual cones: (a) K = R n + K = { : X ia i 0 8a 0} = R n + (b) K = L R n+1, L = {(x, t) :kxk applet} (y, s) 2 K () (y, x)+st 0 8(x, t) 2 L t x
L = {(x, t) :kxk applet} (y, s) 2 L () hy, xi + st 0 8(x, t) 2 L Claim: L = L (i) L L Take (y, s) 2 L, hy, xi + st kykkxk + st 0 (ii) L L Take (y, s) 2 L and set (x, t) =( y, kyk) 2 L hy, yi + skyk 0 =) kyk apples () (y, s) 2 L
(c) K = S n + Claim: K = K K = { 2 S n : Tr( X) 0 8X 0} (i) K S n + Take 2 K. Then for all x 2 R n and so 0 Tr( xx T )=Tr(x T x) =x T x 0 (ii) S n + K Take 0 and decompose X 0 as X = P kx k x T k,with k 0 Tr(X ) = X k ktr(x k x T k ) = X k kx T k x k 0
Implication of conic self duality Primal LP SOCP SDP Dual LP SOCP SDP
Example: minimum total-variation denoising Constraints minimize kxk TV subject to kx bk apple kx bk apple () (D ij x, t ij ) 2 L ij () () apple x b 2 L0 () apple D 0 0 I P tij minimize subject to kx bk apple kd ij xkapplet ij apple x t apple I 0 0 0 L 0 apple x t L0 apple b L = Y L ij
Dual maximize T 0 apple b subject to 0 L0 0 ij Lij 0 Dual variables apple I 0 0 0 apple u0 apple D T 0 s 0 0 I apple u s + apple 0 1 =0 0 = apple apple u0 uij : ku s 0 kapples 0 ij = 0 s ij : ku ij kapples ij Dual problem (SOCP) max hu 0,bi s 0 s. t. ku 0 kapples 0, ku ij kapples ij u 0 + D T u =0 s =1 () max hb, DT ui kd T uk s. t. ku ij kapple1
Example: SDP maximize hc, xi subject to x 1 F 1 +...+ x m F m F 0 (F i 2 S p ) Dual cone problem maximize hf 0, i subject to 0 hf i, i = c i, i =1,...,m
SDP formulation minimize ka 0 (x 1 A 1 +...+ x m A m )k minimize subject to t apple ti A(x) A T (x) ti with A(x) =A 0 (x 1 A 1 +...+ x m A m ) apple I 0 X apple 0 LMI () t x 0 I i A T i 0 i Ai 0 apple 0 A0 + A T 0 0 0
Dual problem With = apple 11 12 T 12 22, this gives apple 0 A0 maximize h A T, i 0 0 subject to apple 0 0 Ai h A T, i =0 apple i 0 I 0 h, i =1 0 I maximize 2Tr(A T 0 12 ) subject to Tr(A T i 12) =0 Tr( ) = 1 0 () maximize 2Tr(A T 0 12 ) subject to Tr(A T i 12) =0 Tr( 11 )+Tr( 22 )=1 apple 11 12 T 12 22 0
Conic duality theorem Can get results paralleling LP duality provided primal is strictly feasible Strict feasibility means that V b \ int(k) 6= ; Theorem (1) Assume (P) is strictly feasible (9x : Ax K b)andp? 6= 1. Then(D)is solvable and d? = p?. (2) Assume (D) is strictly feasible (9 : K 0,AT = c) andd? 6= 1. Then (P) is solvable and d? = p?. (3) Assume one of the problems is bounded and strictly feasible. Then (x, ) primal/dual optimal i b T = c T x () T (Ax b) =0
Proof (1) Claim: If (P) is strictly feasible and 9x : Ax K b then (D) is solvable and d? = p? Enough to show that 9 dual feasible s.t. b T p? If c =0,thenp? =0and? =0does the job. So we assume c 6= 0. M = {Ax b : x 2 R n,c T x apple p? } R - b K M 6= ; (image of halfspace) M \ int(k) =; Why? Assume v 2 M \ int(k) () v = Ax 0 b with Ax 0 b K 0 and c T x 0 apple p? and hence x 0 is primal optimal 9 0 s.t. Ax b 0 8x : kx x 0kapple c 6= 0 =) 9x s.t. kx x 0kapple and c T x<c T x 0 apple p? =)(= M d
M is convex, nonempty, and M \ int(k) =; Separation theorem for convex sets: 9 6= 0 s.t sup h, vi apple inf h, vi v2m v2int(k) hence T v bounded below on int(k) =) T y 0 8y 2 int(k) =) T y 0 8y 2 K () 2 K In summary sup h, vi apple0 & 2 K () 2 K v2m T (Ax b) apple 0 8x : c T x apple p?! Linear form bounded above on a half-plane. This is possible only if A T = µc µ 0
We claim µ>0 Why? µ =0 =) A T =0, T b 0 Take x 0 s.t. Ax 0 b K 0 This is a contradiction K 0 and 6= 0 =) T (Ax 0 b) > 0 =) T b>0? = µ 1 obeys? 2 K?, A T? = c, hc, xi appleh?,bi 8x : c T x apple p? Taking an x such that c T x = p? =) value of dual objective is at least p?
End of proof (2) Same by symmetry (3) x primal feasible, dual feasible =) c T x b T =(Ax b) T! Complementary slackness i zero duality gap Enough to show (x, ) optimal () DG =0 DG is zero gives c T x b T = c T x p? + d? b T + p? d? =0 since c T x p? =0and d? b T =0must hold In conclusion, x primal optimal dual optimal nota bene: no strict feasibility required Conversely, p? = d? =) c T x = b T
Example Primal / dual strict feasibility (P) minimize c T x subject to x 1 F 1 +...+ x n F n F 0 (D) maximize hf 0, i subject to 0 hf i, i = c i (x, ) optimal () c T x hf 0, i = h X x i F i F 0, i =0
Example for which p? >d? Feasible set: apple 0 x1 x 1 x 1 + x 2 0 x 1 +1 0 minimize x2 1 0 x 1 0 3 subject to 4x 1 x 1 + x 2 0 5 0 0 0 x 1 +1 () x 1 + x 2 0 x 2 1 0 x 1 1 () x 1 =0 x 2 0 ) p? =0
Dual 2 4 0 0 0 3 2 0 0 05 + x 1 4 0 1 0 3 2 1 1 05 + x 2 4 0 0 0 3 0 1 05 0 0 0 1 0 0 1 0 0 0 which gives maximize 2 33 subject to dual feasible =) 33 =1 4 1 11 33 1 33 2 2 0 33 ) d? = 1 3 5 0 SDP violates conditions of conic duality theorem Both (P) and (D) are feasible, but not strictly 6= LP where strong duality always holds
Nuclear norm Theorem Operator norm and nuclear norm are dual Proof: want to show kxk = max ky kapple1 hy,xi (a) Take SVD of X = U V T and set Y = UV T.Then (b) In the other direction (D) maximize subject to Tr(Y T X)=Tr( ) = kxk =) max ky kapple1 hy,xi hy,xi apple I Y Y T I 0 duality $ (P) kxk 1 minimize apple2 Tr( 1)+ 1 2 Tr( 2) 1 X subject to X T 0 2 (D) has variable Y and (P) has variables 1, 2 1 = U U T 2 = V V T =) apple 1 X X T 2 = apple U V apple 0 0 apple U V T val(d) apple val(p) apple Tr( ) = kxk
Consequence minimize subject to kxk X 2C If C is SDP-representable, this is an SDP; e.g. A(X) =b ka(x) bk 2 apple ka (A(X) b)k apple... CVX knows about this