Homework Set #6 - Solutions

EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique optimal point or solution is x = and the optimal value is p = 5 b Here we have f x = x 1 and f 1 x = x x 4 = x 6x 8 Thus the Lagrangian Lx λ is given by Lx λ = f x λf 1 x = λ 1 x 6λx 8λ 1 A plot of the objective function the constraint function the feasible set the optimal point and value and the Lagrangian Lx λ for several positive values of λ is shown in Figure 1 From the plot it is clear that minimum value of Lx λ over x ie gλ is always less than p It increases as λ varies from to reaches its maximum at λ = and then decreases again as λ increases above We have equality namely p = gλ for λ = 3 5 15 Lx 3 Lx Lx 1 f 1 5 x p f 1 5 1 1 3 4 5 Figure 1: Plot of the objective function f x the constraint function f 1 x the feasible set [ 4] the optimal point and value x p = 5 along with Lagrangian Lx λ for several positive values of λ x For λ > 1 the Lagrangian is a convex parabola and reaches its minimum at x = 3λ/ 1 λ On the other hand for λ 1 the Lagrangian is a concave parabola and as such is unbounded below Thus the dual function gλ is given by gλ = 9λ λ 1 8λ 1 λ > 1 λ 1

A plot of the dual function gλ is shown in Figure 6 4 gλ 4 6 8 1 1 1 3 4 λ Figure : Plot of the dual function gλ versus λ We can verify that the dual function is concave that its value is equal to p = 5 for λ = and less than p for other values of λ Specifically by taking the derivative of gλ with respect to λ and setting the result to zero leads to the condition λ λ 4 = which for λ > 1 can only be satisfied for λ = For λ = we have g = 5 = p indeed c The Lagrange dual problem is maximize 9λ λ 1 8λ 1 λ A second derivative test on the objective gλ yields g λ = 18 λ 1 3 < λ > 1 Thus the objective is a concave function of λ over the domain of the problem and so the dual problem is indeed a concave maximization problem Setting the derivative of the objective with respect to λ equal to zero yields the condition λ λ 4 = which for λ can only be satisfied for λ = We have g = d = 5 here Hence the dual optimal solution is λ = and the dual optimal value is d = 5 As p = 5 we can verify that strong duality indeed holds for this example as it must since Slater s constraint qualification is satisfied

d The perturbed problem is infeasible for u < 1 since the infimum of f 1 x about all x R is equal to 1 For u 1 the feasible set is the interval [ 3 u 1 3 u 1 ] given by the two roots of x 6x 8 = u For 1 u 8 the optimum solution is x u = 3 1 u On the other hand for u > 8 the optimum is the unconstrained minimum of f ie x u = In summary we have the following undefined u < 1 x u = 3 u 1 1 u 8 u > 8 u < 1 = p u = 11 u 6 u 1 1 u 8 1 u > 8 A plot of the optimal value function p u and its epigraph is shown in Figure 3 In addition a supporting hyperplane or more appropriately a line in this case at u = is shown 1 8 6 epi p u p u 4 p λ u 4 6 8 1 Figure 3: Plot of the optimal value p u as a function of the perturbation u Included with this is a plot of the epigraph of the optimal value along with a supporting hyperplane at u = corresponding to the nominal optimal value u From our expression for p u above it can be seen that it is a differentiable function of u at u = Note that we have Thus we have dp u du = 1 dp du 3 u 1 for 1 u 8 = = λ a Consider the objective function of the LP namely x T y Note that we have the following x T y = x k y k = x [k] y ik

where i k for k = 1 n is a permutation of the index set {1 n} corresponding to the elements of x arranged in descending order In other words we have x ik = x [k] for all k Under the constraints that y k 1 and n y k = r it is clear that we have r r x T y = x [k] y ik x [k] y ik x [k] = fx k=r1 In other words the upper bound on x T y the constraints on y is obtained by extracting the r largest components of x and assigning the maximum possible weight of one to each of these components But this bound can be achieved by setting y i1 = = y ir = 1 and y ir1 = = y in = Hence fx is equal to the optimal value of the LP maximize x T y y 1 1 T y = r with y R n as the variable This can be more compactly written as fx = sup { x T y : y 1 1 T y = r } b First change the objective from maximization to minimization as follows: minimize x T y y 1 1 T y = r Let us introduce a Lagrange multiplier λ for the lower bound on y ie y u for the upper bound on y ie y 1 and t for the equality constraint namely 1 T y = r This yields the Lagrangian Ly λ u t = x T y λ T y u T y 1 t 1 T y r Minimizing over y yields the dual function = rt 1 T u x λ u t1 T y gλ u t = { rt 1 T u x λ u t1 = otherwise The dual problem is to maximize g λ and u This yields the problem maximize rt 1 T u λ = t1 u x λ u By changing the objective to minimization by minimizing the negative of the objective above and eliminating the slack variable λ in the equality constraint we obtain the problem minimize rt 1 T u as desired t1 u x u

c The constraint that no more than half of the total power is in any m lamps is equivalent to the condition m p [k] 1 p k = 1 1T p In other words if n m and r = m then the constraint is equivalent to fp α where α 1 1T p But from the results of part b we know that this condition is true if and only if there exist s R and q R m such that m s 1 T q α s1 q p q Thus with this constraint in effect the patch illumination problem becomes minimize t a T k p I dest k = 1 n [ ] Ides t a T k p t a T k p k = 1 n p p max 1 α = 1 1T p m s 1 T q α s1 q p q with variables t R p R m α R s R and q R m Eliminating the trivial equality constraint α = 1 1T p we obtain minimize t a T k p I dest k = 1 n [ ] Ides t a T k p t a T k p k = 1 n p p max 1 m s 1 T q 1 1T p s1 q p q with variables t R p R m s R and q R m as desired 3 Introducing λ for the inequality constraint x ν for the scalar equality constraint 1 T x = 1 and z for the vector equality constraint Px = y the Lagrangian becomes Lx y λ ν z = c T x y i log y i λ T x ν 1 T x 1 z T Px y = c λ ν1 P T z T m x y i log y i z T y ν The minimum over x is bounded below if and only if c λ ν1 P T z =

To minimize over y we set the derivative with respect to y i equal to zero which gives 1 log y i z i = y i = e z i 1 Thus the dual function is e zi 1 z i 1 z i e zi 1 ν c λ ν1 P T z = gλ ν z = otherwise e zi 1 ν λ = P T z ν1 c = otherwise Hence the dual problem is maximize e zi 1 ν P T z ν1 c This can be simplified by introducing a new variable w = z ν1 and exploiting the fact that 1 = P T 1 Specifically the dual problem becomes maximize e wi ν 1 ν P T w c Finally we can maximize the objective function with respect to ν by setting the derivative equal to zero This yields m ν = log e w i 1 From this the objective function becomes m e ν e w i 1 ν = 1 log As such the dual problem can be simplified to which can be written as maximize minimize log m P T w c log m e w i 1 e w i e w i P T w c = log m with variable w R m Note that this is a GP in convex form with linear inequality constraints ie monomial inequality constraints in the associated standard form GP e w i

4 a Introducing λ for the inequality constraint x and ν for the equality constraint 1 T x = 1 it follows that the KKT conditions are as follows Primal feasibility: Dual feasibility: Complementary slackness: Stationarity: x 1 T x = 1 λ λ k x k = k = 1 n f x λ ν 1 = Here f x = log a T x log b T x and so we have f x k = 1 a T x a k 1 b T x b k f x = 1 a T x a 1 b T x b Thus the stationarity condition becomes λ = ν 1 1 a T x a 1 b T x b Combining the results so far we have the following simplifed KKT conditions x 1 T x 1 = 1 a T x 1 b T x b ν 1 x k ν 1 a T x a k 1 b T x b k = k = 1 n 1 We now show that x = 1/ 1/ and ν = satisfy all of the conditions in 1 and are hence primal and dual optimal respectively The feasibility conditions x 1 T x = 1 obviously hold and the complementary slackness conditions are trivially satisfied for k = n Thus from 1 it remains to verify the inequalities a k a T x b k b T x ν k = 1 n and the complementary slackness conditions x k ν 1 a T x a k 1 b T x b k = k = 1 n 3 For x = 1/ 1/ and ν = the inequality holds with equality for k = 1 and k = n since we have for k = 1 and a 1 a T x b 1 b T x = a 1 a n a T x b n b T x = a n /a 1 1/a 1 1/a n = a 1 a n a n a 1 = /a n 1/a 1 1/a n = a n a 1 a n a 1 =

for k = n Therefore also 3 is satisfied for k = 1 n as desired The remaining inequalities in reduce to a k a T x b k b T x = which is equivalent to a k /a k 1/a 1 1/a n = a k a 1 a n /a k k = n 1 a k a 1 a n /a k 1 k = n 1 4 To show that 4 is valid note that the function gt ta 1a n/t a 1 a n is convex for t R Since the inequality in 4 holds with equality for k = 1 and k = n it follows that t a 1 a n /t 1 t [a n a 1 ] Hence x = 1/ 1/ and ν satisfy all of the KKT conditions of 1 and are thus primal and dual optimal respectively b Express A in terms of its eigenvalue decomposition as A = QΛQ T and define a k λ k b k 1/λ k and x k = [ Q T u ] k Note that we clearly have x k for all k Also as u = 1 we have 1 T x = [ Q T u ] k = Q T u = ut QQ T u = u T u = 1 Thus the choice of x k = [ Q T u ] is indeed feasible Continuing further note that k a T [ x = λ k Q T u ] k = ut Au and that b T x = 1/λ k [ Q T u ] k = ut A 1 u Hence from the result proven in part a we have log u T Au log u T A 1 u logλ 1 / λ n / log1/ λ 1 1/ λ n After some algebraic manipulation this becomes u T Au u T A 1 u λ1 λ n 1 1 λ 1 λ n This can be further simplified to u T Au u T A 1 u 1 4 λ 1 /λ n λ n = 1 λ1 λ 1 4 λ n Taking square roots of both sides of the above inequality yields u T Au 1/ u T A 1 u 1/ λ1 λ n for all u with u = 1 which is Kantorovich s inequality = 1 4 λ 1 λ n λ 1 1 λ 1 n λn λ 1 λn λ 1

*5 Define the following quantities y T 1 1 d 1 y 1 [ A b C In n 1 1 n ym T 1 d m y m ] f [ n 1 1/ ] Also define z x t Then with this notation the problem becomes minimize Ax b z T Cz f T z = Introducing ν for the equality constraint we obtain the following for the Lagrangian Lz ν = Az b ν z T Cz f T z = z T A T Az b T Az b T b z T νc z νf T z = z T A T A νc z A T b νf T z b Note that this is bounded below as a function of z if and only if A T A νc A T b νf R A T A νc Therefore the KKT conditions are as follows Primal feasibility: z T Cz f T z = Dual feasibility: A T A νc A T b νf R A T A νc Stationarity: A T A νc z = A T b νf Note that this implies the range condition for dual feasibility Method 1: We derive the dual problem If ν is feasible then the dual function is given by gν = A T b νf T A T A νc # A T b νf b So the dual problem can be expressed as the SDP maximize s b [ A T A νc A T b νf A T b νf T s ] which is equivalent to the following SDP minimize s b [ A T A νc A T b νf A T b νf T s ]

Solving this in cvx gives ν = 5898 Using ν in the stationarity condition we get Hence x = 1369 6446 z = A T A ν C 1 A T b ν f = 1369 6446 1765 Method : Alternatively we can solve the KKT conditions directly To simplify the equations we make a change of variables w = Q T L T z where L is the lower triangular matrix obtained from the Cholesky decomposition of A T A ie A T A = LL T and Q is the matrix of eigenvectors of L 1 CL T ie L 1 CL T = QΛQ T This transforms the KKT conditions to w T Λw g T w = I νλ I νλ w = h νg where we have g = Q T L 1 f h = Q T L 1 A T b From the last equation of the KKT conditions we find w k = h k νg k 1 νλ k k = 1 n 1 Substituting this into the first equation of the KKT conditions we get the following nonlinear equation in ν n1 λ k h k νg k rν = 1 νλ k g k h k νg k = 1 νλ k In our example the eigenvalues are λ 1 = 514 λ = 735 λ 3 = Plots of the function rν for this example are shown in Figures 4a and b In Figure 4a we have a zoomed out plot showing all three solutions to rν = whereas in Figure 4b we have a zoomed out plot showing the correct solution The correct solution of rν = is the one that satisfies 1 νλ k for k = 1 n 1 ie the solution to the right of the two singularities in this case This solution can be determined by using Newton s method by repeating the iteration ν := ν rν r ν a few times starting at a value close to the desired solution This gives ν = 5896 which is very close to the value obtained from cvx From ν we determine x as in the first method A contour plot of the objective f for the given problem data along with the sensor position vectors y k and optimal source position vector x is shown in Figure 5

1 1 9 9 8 8 7 7 6 6 rν 5 4 rν 5 4 3 3 1 1 1 5 4 3 1 1 3 4 5 a ν 1 6 5 4 3 1 1 b ν Figure 4: Plots of the nonlinear function rν used to determine the optimal ν which satisfies the KKT conditions: a zoomed out plot showing all three solutions to rν = and b zoomed in plot showing a close-up of the correct solution to rν = 35 3 5 y 1 y 4 x 15 y 3 y y 5 1 5 x 5 1 15 5 3 35 Figure 5: Contour plot of the objective f x 1 x for the given problem data with the sensor position vectors y k and optimal source position vector x indicated by circles x 1