Solutons HW #2 Dual of general LP. Fnd the dual functon of the LP mnmze subject to c T x Gx h Ax = b. Gve the dual problem, and make the mplct equalty constrants explct. Soluton. 1. The Lagrangan s L(x, λ, ν) = c T x + λ T (Gx h) + ν T (Ax b) = (c T + λ T G + ν T A)x hλ T ν T b, whch s an affne functon of x. It follows that the dual functon s gven by { λ g(λ, ν) = nf L(x, λ, ν) = T h ν T b c + G T λ + A T ν = 0 x otherwse. 2. The dual problem s maxmze g(λ, ν) subject to λ 0. After makng the mplct constrants explct, we obtan maxmze λ T h ν T b subject to c + G T λ + A T ν = 0 λ 0. Pecewse-lnear mnmzaton. We consder the convex pecewse-lnear mnmzaton problem wth varable x R n. mnmze max,...,m (a T x + b ) (1) 1. Derve a dual problem, based on the Lagrange dual of the equvalent problem wth varables x R n, y R m. mnmze max,...,m y subject to a T x + b = y, = 1,..., m, 2. Formulate the pecewse-lnear mnmzaton problem (1) as an LP, and form the dual of the LP. Relate the LP dual to the dual obtaned n part (a). 1
3. Suppose we approxmate the objectve functon n (1) by the smooth functon ) f 0 (x) = log exp(a T x + b ), and solve the unconstraned geometrc program mnmze log exp(at x + b ) ). (2) A dual of ths problem s gven by (5.62). Let p pwl and p gp be the optmal values of (1) and (2), respectvely. Show that 0 p gp p pwl log m. 4. Derve smlar bounds for the dfference between p pwl and the optmal value of mnmze (1/γ) log exp(γ(at x + b )) ), where γ > 0 s a parameter. What happens as we ncrease γ? Soluton. 1. The dual functon s g(λ) = nf x,y ( max y +,...,m ) λ (a T x + b y ). The nfmum over x s fnte only f λ a = 0. To mnmze over y we note that { nf (max y λ T 0 λ 0, 1 y) = T λ = 1 y otherwse. To prove ths, we frst note that f λ 0, 1 T λ = 1, then λ T y = j λ j y j j λ j max y = max y, wth equalty f y = 0, so n that case nf (max y λ T y) = 0. y If λ 0, say λ j < 0, then choosng y = 0, j, and y j = t, wth t 0, and lettng t go to nfnty, gves max y λ T y = 0 + tλ k. Fnally, f 1 T λ 1, choosng y = t1, gves max y λ T y = t(1 1 T λ), f t and 1 < 1 T λ, or f t and 1 > 1 T λ. 2
Summng up, we have { b g(λ) = λ λ a = 0, otherwse. λ 0, 1 T λ = 1 The resultng dual problem s 2. The problem s equvalent to the LP maxmze b T λ subject to A T λ = 0 1 T λ = 1 λ 0. The dual problem s mnmze t subject to Ax + b t1. maxmze b T z subject to A T z = 0, 1 T z = 1, z 0, whch s dentcal to the dual derved n (a). 3. Frst both prmal problems PWL and GP are convex so strong dualty holds. Suppose z s dual optmal for the dual GP (5.62) (wth the conventon 0 log 0 = 0) maxmze b T z m z log z subject to 1 T z = 1 A T z = 0 z 0 Then z s also feasble for the dual of the pecewse-lnear formulaton, wth objectve value Ths provdes a lower bound on p pwl : p pwl p gp + b T z = p gp + z log z. z log z p gp log m. The bound follows from concavty of log usng Jensens s nequalty On the other hand, we also have z log(1/z ) log 1 = log m. max(a T x + b ) log exp(a T x + b ) for all x, and therefore p pwl p gp. In concluson, p gp log m p pwl p gp. 3
4. We frst reformulate the problem as The Lagrangan s mnmze (1/γ) log m exp(γy ) subject to Ax + b = y. L(x, y, z) = 1 m γ log exp(γy ) + z T (Ax + b y). L s bounded below as a functon of x only f A T z = 0. To fnd the optmum over y, we compute the conjugate functon for z R m, f (z) = sup γz T y log y exp(γy ) = sup y If there exsts z < 0, then, takng y = te, we get z T y log exp(y ). z T y log exp(y ) = tz log((m 1) + e t ), whch goes to nfnty for t. If z 0 and 1 T z 1, then, takng y = t1, we get z T y log exp(y ) = t1 T z log(m) t, whch goes to nfnty for t + or dependng on the sgn of 1 T z 1. If z 0 and 1 T z = z 0 z = 1, we have usng concavty of log, z T y z log z = z (y log z ) log z exp(y log(z )) log exp(y ), z 0 z 0 wth equalty by takng y = log z f z > 0 and y for z = 0. Therefore we have f (z) = z log z. The Lagrange dual functon s then gven for z 0, 1 T z = 1, A T z = 0 by and the dual problem s g(z) = b T z 1 γ z log z, maxmze b T z (1/γ) m z log z subject to A T z = 0 1 T z = 1. 4
Let p gp(γ) be the optmal value of the GP. Followng the same argument as above, we can conclude that p gp(γ) 1 γ log m p pwl p gp(γ). In other words, p gp(γ) approaches p pwl as γ ncreases. Suboptmalty of a smple coverng ellpsod. Recall the problem of determnng the mnmum volume ellpsod, centered at the orgn, that contans the ponts a 1,..., a m R n (problem (5.14), page 222): mnmze f 0 (X) = log det(x 1 ) subject to a T Xa 1, = 1,..., m, wth dom f 0 = S n ++. We assume that the vectors a 1,..., a m span R n (whch mples that the problem s bounded below). 1. Show that the matrx ) 1 X sm = a k a T k, k=1 s feasble. Hnt. Show that [ m k=1 a ka T k a a T 1 ] 0, and use Schur complements ( A.5.5) to prove that a T Xa 1 for = 1,..., m. Soluton. [ m k=1 a ka T k a k a T 1 ] = [ k a ka T k 0 0 0 ] [ a + 1 ] [ a 1 s the sum of two postve semdefnte matrces, hence postve semdefnte. The Schur complement of the 1, 1 block of ths matrx, m k=1 a ka T k whch s nvertble by hypothess, s therefore also postve semdefnte: ] T 1 a T ( m ) 1 a k a T k a 0, k=1 whch s the desred concluson. 2. Now we establsh a bound on how suboptmal the feasble pont X sm s, va the dual problem, maxmze log det λ a a T ) 1 T λ + n subject to λ 0, wth the mplct constrant m λ a a T 0. (Ths dual s derved on page 222.) To derve a bound, we restrct our attenton to dual varables of the form λ = t1, where t > 0. Fnd (analytcally) the optmal value of t, and evaluate the dual objectve at ths λ. Use ths to prove that the volume of the ellpsod {u u T X sm u 1} s no more than a factor (m/n) n/2 more than the volume of the mnmum volume ellpsod. 5
Soluton. The dual functon evaluated at λ = t1 s ) g(λ) = log det a a T + n log t mt + n. Now we ll maxmze over t > 0 to get the best lower bound. Settng the dervatve wth respect to t equal to zero yelds the optmal value t = n/m. Usng ths λ we get the dual objectve value ) g(λ) = log det a a T + n log(n/m). The prmal objectve value for X = X sm s gven by ) 1 log det a a T, so the dualty gap assocated wth X sm and λ s n log(m/n). (Recall that m n, by our assumpton that a 1,..., a m span R n.) It follows that, n terms of the objectve functon, X sm s no more than n log(m/n) suboptmal. The volume V of the ellpsod E assocated wth the matrx X s gven by V = exp( O/2), where O s the assocated objectve functon, O = log det X. The bound follows. Dual problem. Derve a dual problem for mnmze N A x + b 2 + (1/2) x x 0 2 2. The problem data are A R m n, b R m, and x 0 R n. Frst ntroduce new varables y R m and equalty constrants y = A x + b. Soluton. The Lagrangan s L(x, z 1,..., z N ) = We frst mnmze over y. We have N y 2 + 1 2 x x 0 2 2 + nf y ( y 2 + z T y ) = N z T (y A x b ). { 0 z 2 1 otherwse. (If z 2 > 1, choose y = tz and let t, to show that the functon s unbounded below. If z 2 1, t follows from the Cauchy-Schwarz nequalty that y 2 + z T y 0, so the mnmum s reached when y = 0.) We can mnmze over x by settng the gradent wth respect to x equal to zero. Ths yelds x = x 0 + N A T z. Substtutng n the Lagrangan gves the dual functon { N g(z 1,..., z N ) = (A x 0 + b ) T z 1 2 N AT z 2 2 z 2 1, = 1,..., N otherwse. 6
The dual problem s maxmze N (A x 0 + b ) T z 1 2 N AT z 2 subject to z 2 1, = 1,..., N. Lagrangan relaxaton of Boolean LP. A Boolean lnear program s an optmzaton problem of the form mnmze subject to c T x Ax b x {0, 1}, = 1,..., n, and s, n general, very dffcult to solve. In exercse (4.15) we studed the LP relaxaton of ths problem, mnmze c T x subject to Ax b (3) 0 x 1, = 1,..., n, whch s far easer to solve, and gves a lower bound on the optmal value of the Boolean LP. In ths problem we derve another lower bound for the Boolean LP, and work out the relaton between the two lower bounds. 1. Lagrangan relaxaton. The Boolean LP can be reformulated as the problem mnmze subject to c T x Ax b x (1 x ) = 0, = 1,..., n, whch has quadratc equalty constrants. Fnd the Lagrange dual of ths problem. The optmal value of the dual problem (whch s convex) gves a lower bound on the optmal value of the Boolean LP. Ths method of fndng a lower bound on the optmal value s called Lagrangan relaxaton. 2. Show that the lower bound obtaned va Lagrangan relaxaton, and va the LP relaxaton (3), are the same. Hnt. Derve the dual of the LP relaxaton (3). Soluton. 1. The Lagrangan s L(x, µ, ν) = c T x + µ T (Ax b) ν T x + x T dag(ν)x = x T dag(ν)x + (c + A T µ ν) T x b T µ. Mnmzng over x gves the dual functon { b g(µ, ν) = T µ (1/4) n (c + a T µ ν ) 2 /ν ν 0 otherwse where a s the th column of A, and we adopt the conventon that a 2 /0 = f a 0, and a 2 /0 = 0 f a = 0. 7
The resultng dual problem s maxmze b T µ (1/4) n (c + a T µ ν ) 2 /ν subject to ν 0. In order to smplfy ths dual, we optmze analytcally over ν, by notng that sup ( (c + a T µ ν ) 2 ) { 4(c + a = T µ) c + a T µ 0 ν 0 ν 0 c + a T µ 0 = 4 mn{0, (c + a T µ)} Ths allows us to elmnate ν from the dual problem, and smplfy t as maxmze b T µ + n mn{0, c + a T µ} subject to µ 0. 2. We follow the hnt. The Lagrangan and dual functon of the LP relaxaton are The dual problem s L(x, u, v, w) = c T x + u T (Ax b) v T x + w T (x 1) = (c + A T u v + w) T x b T u 1 T w { b g(u, v, w) = T u 1 T w A T u v + w + c = 0 otherwse. maxmze b T u 1 T w subject to A T u v + w + c = 0 u 0, v 0, w 0, whch s equvalent to the Lagrange relaxaton problem derved above. We conclude that the two relaxatons gve the same value. 8