Midterm 1 Solutions. 1. (2 points) Show that the Frobenius norm of a matrix A depends only on its singular values. Precisely, show that

Transcription

1 EE127A L. El Ghaoui YOUR NAME HERE: SOLUTIONS YOUR SID HERE: 42 3/27/9 Midterm 1 Solutions The eam is open notes, but access to the Internet is not allowed. The maimum grade is 2. When asked to prove something, do not merely take an eample; provide a rigorous proof with clear steps. Also, some parts are more difficult than others and you are not epected to finish the eam. 1. (2 points) Show that the Frobenius norm of a matri A depends only on its singular values. Precisely, show that A F = σ 2, where σ := (σ 1,..., σ r ) is the vector formed with the singular values of A, and r is the rank of A. Solution: We present two solutions. Assume A R m n. a) By re-writing this in terms of the squared Frobenius norm, A 2 F = tr(at A) = n λ i (A T A) = r λ i (A T A) = r σi 2 = σ 2 2, the sum going to r because only the first r eigenvalues of A are non-zero. b) We can use the SVD of A = U ΣV T, with Σ = diag(σ 1,..., σ r,,..., ), directly. We have A 2 F = U ΣV T 2 F = tr(v ΣU T U ΣV T ) = tr(v Σ 2 V T ) = tr( Σ 2 V T V ) = tr Σ 2, which proves the result. 2. (8 points) We are given a matri A R m n. We consider a matri least-squares problem AX I m X F, (1) where the variable is X R n m, and I m is the m m identity matri. (a) Show that the problem can be reduced to a number of ordinary least-squares problems, each involving one column of the matri X. Make sure you define precisely the form of these least-squares problems. Solution: Let j R n denote the j-th column of A, so that X = [ 1,..., m ]. We have AX I m = [A 1 e 1,..., A m e m ], where e i is the i-th basis vector in 1

2 R m. Since the squared Frobenius norm of a matri is the sum of squares of the Euclidean norms of its columns, the objective function of our problem is AX I m 2 F = [A 1 e 1,..., A m e m ] 2 F = m A i e i 2 2. Note that the vector i appears only once, in the i-th term of the sum. Hence, to imize the above objective function, we can simply imize the i-th term with respect to i, independently of the other terms. That is: X=[ 1,..., m] AX I m 2 F = 1,..., m m A e i 2 2 = m A e i 2 2. Hence, to solve our problem we can simply solve m ordinary least-squares problems, of the form A e i 2, i = 1,..., m. If i is an optimal solution for the above, then X = [ 1,..., m] is optimal for our original problem. (b) Show that, when A is full column rank, then the optimal solution is unique, and given by X = (A T A) 1 A T. Solution: When A is full rank, the solution to the least-squares problem A y 2 is unique, and given by = (A T A) 1 A T y. Applying this to y = e i, i = 1,..., m, we obtain that the optimal X is unique, and given by X = [ 1,..., m], with i = (A T A) 1 A T e i, i = 1,..., m. That is: X = [ (A T A) 1 A T e 1 (A T A) 1 A T e m ] = (A T A) 1 A T [e 1 e m ] = (A T A) 1 A T I m = (A T A) 1 A T. Another method is to simply take derivatives in the matri problem. Indeed, the objective Eq. (1) is conve in X, and the problem is unconstrained. Taking the matri derivative of AX I m 2 F, and setting it equal to zero, we have 2A T AX 2A T I m =. When A is full column rank, A T A is full rank and invertible, so we have X = (A T A) 1 A T. (c) Show that in general, a solution is X = A, the pseudo-inverse of A. Hint: use the SVD of A, and eploit the fact that the Frobenius norm of a matri is the Euclidean norm of the vector formed with its singular values. 2

3 Solution: We know that if i is optimal for the ordinary least-squares problem A e i 2, then X := [ 1,..., m] is optimal for the original problem. Since i = A e i is optimal for the above problem, we obtain that X = [ 1,..., m] = A is optimal for the matri problem. (d) Assume that we would like to form an estimate X of the pseudo-inverse of a matri A that is as sparse as possible, while maintaining a good accuracy, epressed in terms of the objective function of problem (1). How would you formulate the corresponding trade-off? How would you use CVX to plot a trade-off curve? Solution: Ideally, to enforce sparsity, we sould like to solve a problem of the form X AX I m F + λ card(x) (2) where card(x) denotes the cardinality (number of non-zeros) of X. This is, however, wildly non-conve, so we rela Eq. (2) to the l 1 -regularized problem AX I m X F + λ i,j X ij. (3) We can then increase or decrease λ to look at the trade-off in the error AX I m F in terms of sparsity in X. 1 CVX code is below, and in Fig. 1 we plot the error in AX I m F versus the sparsity in X. m = 2; n = 1; A = randn(m, n); lambdas = [ ]; cv_quiet(true); Xs = cell(numel(lambdas), 1); for i = 1:numel(lambdas); lambda = lambdas(i); cv_begin variable X(n, m); imize(norm(a*x - eye(m), fro ) + lambda * sum(sum(abs(x)))); cv_end Xs{i} = X.*(abs(X) > 1e-8); end splevel = zeros(numel(lambdas), 1); errlevel = zeros(numel(lambdas), 1); for i = 1:numel(lambdas) 1 Note that if A is full row-rank (so that AX = I m is solvable), it may make sense to imize i,j X ij subject to AX = I m. 3

4 4.5 4 AX Im Proportion zero entries in X Figure 1: Tradeoffs for sparsity in terms of reconstruction error. splevel(i) = (n * m - nnz(xs{i})) / (m * n); errlevel(i) = norm(a*xs{i} - eye(m), fro ); end figure; plot(splevel, errlevel); label( Proportion zero entries in $X$,... Interpreter, late, fontsize, 2); h = ylabel( $\ AX - I_m\ $ ); set(h, Interpreter, late, fontsize, 2); 3. (3 points) Let p, p 1,..., p m be a collection of (m + 1) points in R n. Consider the set of points closer (in Euclidean norm) to p than to the remaining points p 1,..., p m : P = { R n : i = 1,..., m, p 2 p i 2 }. Show that the set P is a polyhedron, and provide a representation of it in terms of the problem s data, as P = { : A b}, where A is matri and b a vector, which you will detere. Solution: There are (at least) two equally reasonable ways to solve this problem. The first is to simply re-write the constraints and re-arrange symbols to give ourselves polyhedral inequalities. That is, we note that p 2 p i 2 iff p 2 2 p i 2 2, 4

5 and the second inequality can be re-written as T 2p T + p T p T 2p T i + p T i p i 2p T i 2p T p T i p i p T p (p i p ) T 1 2 (p i p ) T (p i + p ) Evidently, then, if we set A R m n and b as (p 1 p ) T (p 1 p ) T (p 1 + p ) (p 2 p ) T A =. and b = 1 (p 2 p ) T (p 2 + p ) 2., (4) (p m p ) T (p m p ) T (p m + p ) we can write the set as P = { : A b}. A more geometric way of deriving the above is as follows. First, we recognize that we will be intersecting a series of conve sets, so if we can describe p 2 p i 2 as a linear inequality, we can simply stack the inequalities. Now, the set of points that is closer to the point p than it is to the point p i is a half-space, and the normal to the half-space is p i p. Now all that remains is to find the point on the line halfway between p and p i, because this will be the point that gives the inequality defining our half-space. Clearly, this point is (p i + p )/2. See Fig. 2 for a picture. Thus, since our normal is given by (p i p ) and we know that all points in the half space must be closer to p than the point (p i + p )/2, we have all in the half space satisfy (p i p ) T (p i p ) T (p i + p )/2, which gives the same linear inequalities as the matri A and vector b in Eq. (4). p 1 (p 1 p ) 1 2 (p + p 1 ) p Figure 2: Halfspace separation 4. (3 points) We consider a production planning problem. The variables j R, j = 1,..., n, in the problem are activity levels (for eample, production levels for different products manufactured by a company). These activities consume m resources, which 5

6 are limited. Activity j consumes A ij j of resource i. (Ordinarily we have A ij, that is, activity j consumes resource i, but we allow the possibility that A ij <, which means that activity j actually generates resource i as a by-product.) The total resource consumption is additive, so the total of resource i consumed is c i = j=1 A ij j. Each resource consumption is limited:for every activity i, we must have c i c ma i are given. Activity j generates revenue r j ( j ), where r j is a function of the form { pj u if u q r j (u) = j, p j q j + p disc j (u q j ) otherwise, where c ma i where q j >, p disc j > are given parameters. The above revenue model says that above a certain level of activity, the revenue is discounted, and not as high as for lower activity level. Show how to formulate the problem of maimizing revenue under the resource constraints as an LP. Solution: We let our optimization variables be r and, where r j corresponds to the r j above. We want to maimize r j (our revenue) subject to the constraint that r j {p j j, p j q j + p disc j ( j q j )}, which is to say, r j is less than both. If we define the matrices and vectors q 1 c ma P = diag(p 1,..., p n ), P disc = diag(p disc 1,..., p disc n ), q =., c ma = 1. we can re-write the problem as ma r, 1 T r subject to A c ma, 5. (4 points) Consider the unconstrained QP q n r P, r P q + P disc ( q). p 1 = 2 T Q c T where Q = Q T R n n, Q, and c R n are given. The goal of this problem is to detere the optimal value p and the optimal set X opt, in terms of c and the eigenvalues and eigenvectors of the (symmetric) matri Q. (a) Assume that Q. Show that the optimal set is a singleton, and that p is finite. Detere both in terms of Q, c. Solution: The problem is conve and unconstrained. Thus, the optimal set is that of points such that the gradient of the objective is zero. Denoting the objective functionby f, the optimality condition is f() = Q c. 6 c ma n

7 Since Q, Q is invertible, and there is a unique solution to Q = c. Thus = Q 1 c is the unique optimal point, and p = 1 2 ( ) T Q c T = 1 2 ct Q 1 c c T Q 1 c = 1 2 ct Q 1 c. Assume from now on that Q is not invertible. (b) Assume further that Q is diagonal: Q = diag(λ 1,..., λ n ), with λ 1... λ r > λ r+1 =... = λ n =, where r is the rank of Q (1 r < n). Solve the problem in that case. (You will distinguish between two cases.) Solution: Partition the variable in two parts: = (y, z) with y R r and z R n r. Likewise partition c in two parts: c = (a, b) with a R r and b R n r. Since Q = diag(λ, ), with Λ = diag(λ 1,..., λ r ), we have f() = 1 2 T Q c T = 1 2 yt Λy a T y b T z = f 1 (y) + f 2 (z), where f 1 (y) = 1 2 yt Λy a T y and f 2 (z) = b T z. The above is a sum of two functions f 1, f 2 of independent variables y and z. Thus our problem can be solved by imizing f 1 over y and f 2 over z independently: f 1(y) + f 2 (z) = =(y,z) y f 1 (y) + z f 2 (z). The imization over y is eactly the same as in the previous part, since the matri Λ is positive definite. That is, the optimal y is unique, and given by y = Λ 1 a. For the imization with respect to z, we distinguish two cases. Either the vector b =, in which case the optimal value of z f 2 (z) is zero, and any z is optimal. Or, b, in which case the corresponding optimal value is and the problem is not attained. To summarize: If b =, then the optimal set is the subspace X opt = { (Λ 1 a, z) : z R n r}, and the optimal value is 1 2 at Λ 1 a. If b, the optimal value is, and the optimal set is empty (the optimal value is not attained). Note that the condition b = can simply be epressed as c R(Q), where R(Q) is the range of Q. (c) Now we do not assume that Q is diagonal anymore. Under what conditions (on Q, c) is the optimal value finite? 7

8 Solution: Let Q = U ΛU T be the eigenvalue decomposition of the symmetric matri Q, with U orthogonal and Λ = diag(λ, ), and Λ diagonal. Our problem reads 1 2 T U T ΛU c T 1 = 2 T Λ ct, where := U, and c = Uc. This change of variable allows us to go back to the diagonal case, which we solved in the previous part. Partition c = (ã, b) with ã R r the component of c along the range of Λ, and b R n r the component of c orthogonal to the range of Λ. We obtain that if b, then the optimal value is and the optimal set is empty. This corresponds to the case when c is not in the range of Q. We conclude that the optimal value is finite if and only if c is in the range of Q. (d) Detere the optimal value and optimal set. Be as specific as you can. Solution: We present two solutions. a) Assume that c is in the range of Q, so that the optimal value is finite. If that is the case, then c is also in the range of Q 1/2, because the ranges of Q and its symmetric square root are identical (both matrices share the same system of eigenvectors, and the range depends only on the eigenvectors). Let d be such that c = Q 1/2 d. We can write our objective function, f, as f() = 1 2 T Q c T = 1 2 T Q d T Q 1/2 = 1 2 Q1/2 d 2 2 d 2 2. Minimizing f amounts to solve the ordinary least-squares problem The solution set is Q 1/2 d 2. X opt = (Q 1/2 ) d + N(Q 1/2 ), where N(Q 1/2 ) is the range of Q 1/2 is the range of Q 1/2, which, as noted before is the same as the range of Q itself. We can write the vector (Q 1/2 ) d in terms of Q, c as follows: (Q 1/2 ) d = Q c. Indeed, if Q = U T ΛU is an eigenvalue decomposition of Q, with Λ = diag(λ, ), and Λ diagonal, then ( ) ( ) ( ) Λ (Q 1/2 ) d = U T 1/2 Λ Ud = U T 1 Λ U U T 1/2 d = Q c. Hence, if c Range(Q), we have } {{ } =Q X opt = Q c + N(Q), 8 } {{ } =c

9 where N(Q) is the nullspace of Q. The optimal value is the value of the objective evaluated at any of the points in the optimal set. For eample, p = 1 2 (Q c) T Q(Q c) c T Q c = 1 2 ct Q c, where we have eploited the fact that Q QQ = Q. b) Here is an alternative solution. We take the same notation as in the previous part. Assume that b =, that is, if c is in the range of Q, so that the optimal value is finite. Then the optimal set (in the space of -variables) is X = { (Λ 1 ã, z) : z R n r}. For every X, we can write = + ( z ), where With c = Uc: = ( Λ 1 ) ( ã b ) = := U T = U T ( Λ 1 ( Λ 1 ) c ) Uc = Q c. When z spans R n r, the vector (, z) spans the nullspace of diag(λ, ), so that the vector U T (, z) spans the nullspace of Q. Hence, if c Range(Q), the optimal set is where N(Q) is the nullspace of Q. X opt = Q c + N(Q), 9