164 Final Solutions 1

Size: px

Start display at page:

Download "164 Final Solutions 1"

Melvin Chapman
6 years ago
Views:

1 164 Final Solutions 1 1. Question 1 True/False a) Let f : R R be a C 3 function such that fx) for all x R. Then the graient escent algorithm starte at the point will fin the global minimum of f. FALSE. The graient escent algorithm may only terminate at a local minimum of a function. Let x R an consier fx) = x 1) 2 x + 1) 2. Note that fx) for all x R. Then f x) = x 1) 2 2x+1)+x+1) 2 2x 1) = 2x+1)x 1)x+1+x 1) = 4xx+1)x 1). So, f ) =, an the graient escent algorithm cannot move anywhere, since x =, an x 1 = x + εf ) = x, for any ε >, an more generally x n = x for any n. So, fx n ) = 1 for all n, but f1) =. That is, the algorithm has not foun the global minimum of f. b) Let A be a 5 5 real symmetric matrix. Let x R 5. As usual, efine x = x T x) 1/2. Assume that, for any x R 5, we have lim n A n x =. Then any eigenvalue λ of A must satisfy λ < 1. TRUE. If A has an eigenvalue λ with λ 1 with eigenvector x R 5, x, then λ R by the Spectral Theorem Theorem 2.22 in the notes), an A n x = λ n x, so A n x = λ n x = λ n x. c) Let n 2 be an integer. Let f : R n R be a function. Assume that f C 3. Fix x R n. As usual, efine the matrix of secon erivatives D 2 fx) so that D 2 fx)) ij = 2 fx) x i x j for any 1 i, j n. Assume that fx) = an all eigenvalues of D 2 fx) are nonnegative. Then x is a local minimum of f. FALSE. Let fx, y) = x 4 y 4. Then f, ) =, D 2 f) =, so all eigenvalues of D 2 f) are nonnegative, but f, ) = while f, t) < for all t. Therefore,, ) is not a local minimum of f. ) Let A be a 2 2 real matrix. For any x, y R 2, efine x, y A := x T Ay. Then for any x R 2, x, x A. ) 1 FALSE. Let A =. Let x = 1 1 ). Then x T Ax = 1 <. e) Suppose we have a primal linear program which is infeasible. Then the ual linear program is unboune. FALSE; it is possible for both the primal an ual to be infeasible, from the first item in the Strong Duality Theorem for Linear Programming. For example, if c = 1, b = 1, A =, the primal problem with x R has feasible set x = 1, x, an the ual problem has feasible set y 1. Both feasible regions are therefore empty. f) Suppose both the primal an ual linear program are feasible, where both the primal an ual are efine in the Reference Sheet. Then there exist x, y which are feasible for the primal an ual problems respectively such that c T x = b T y. TRUE; it is the last part of the strong uality theorem. g) Let m, n be positive integers with m n. Let A be a real m n matrix with full row rank, an let b R m. Let K = {x R n : x, Ax = b}. Then x is a vertex of K if an only if x is a basic feasible solution. 1 December 9, 216, c 216 Steven Heilman, All Rights Reserve. 1

2 TRUE; Lemma 4.21 from the notes. h) Suppose we have a boune, feasible linear program in stanar form given by an m n matrix A, c R n, b R m. Then the Simplex algorithm will fin the minimum of this linear program in a number of steps which is a polynomial in n an in m. FALSE; we mentione this in class. If the feasible region is the boune, nonempty) hypercube P = {x R n : x i 1, 1 i n}, then P has 2 n vertices, an the Simplex algorithm may nee to visit all of these vertices. That is, for any n 1, the algorithm might nee 2 n steps to terminate, an 2 n excees any polynomial in n. 2. Question 2 If the statement below is true, prove it. If the statement below is false, o your best to explain why it is false a counterexample woul be best, but a counterexample woul not be require.) Let f : R 2 R be a C 3 function with exactly one critical point x R 2. Assume that x is a local minimum of f. Then x is a global minimum of f. Solution. This statement is false. Let fx, y) = x 2 + y 2 x) 3 for any x, y R. Then fx, y) = 2x+31+x) 2 y 2, 2y1+x) 3 ) = only when 2y1+x) 3 = an 2x = 31+x) 2 y 2. From the first equation, either y = or x = 1. If y =, then 2x =, so x =. If x = 1, then 2x = so x =. So, the only critical point of f is, ). We have 2 f/ x 2 = x)y 2, ) 2 f/ y 2 = 2 x) 3, an 2 f/ x y = 6y x) 2. So, at, ), 2 we have D 2 f, ) =. So,, ) is a local minimum of f. However,, ) is not a 2 global minimum of f since f, ) =, an f 2, 3) <. 3. Question 3 Let f : R R be a function with three continuous erivatives. Assume that f x) for all x R. Show that f is convex. You can freely use Taylor s Theorem with integral remainer.) Solution. From Taylor s Theorem, for any x, y R, we have fx) = fy) + x y)f y) x y f t)x t)t fy) + x y)f y). Let a, b R an let t, 1). Set y := ta + 1 t)b. From ), we euce tfa) tfy) + ta y)f y), ) 1 t)fb) 1 t)fy) + 1 t)b y)f y) Note that ta y) + 1 t)b y) = ta + 1 t)b y = by efinition of y. So, aing the inequalities, tfa) + 1 t)fb) fy) + f y)[ta y) + 1 t)b y)] = fy) = fta + 1 t)b). 4. Question 4 Let m n be positive integers. Let A be a real m n matrix with rank n. Let b R m. Show that the global minimum of Ax b 2 among all x R n occurs when x = A T A) 1 A T b. You may assume without proof that A T A is invertible. As usual x = xx T ) 1/2. ) 2

3 Solution. Let x R n an efine fx) := Ax b 2 = Ax b) T Ax b) = x T A T Ax b T Ax x T Ab + b T b n n n = x i A T A) ij x j b T A) i x i Ab) i x i + b T b. i,j=1 For any 1 i, j n, f = 2 x i j : j i i=1 i=1 A T A) ij x j + 2x i A T A) ii b T A) i Ab) i. 2 f x i x j = A T A) ij. That is, fx) = 2A T Ax 2A T b an D 2 fx) = 2A T A. In particular, if x = A T A) 1 A T b, then fx) =. We now show that f is a convex function. Let t, 1) an let x, y R n. Then tfx) + 1 t)fy) ftx + 1 t)y) = t Ax b t) Ay b 2 Atx + 1 t)y) b 2 = t Ax b t) Ay b 2 tax b) + 1 t)ay b) 2 = t Ax b t) Ay b 2 t 2 Ax b 2 1 t) 2 Ay b 2 2t1 t)ax b) T Ay b) = t1 t) Ax b 2 + t1 t) Ay b 2 2t1 t)ax b) T Ay b) ) = t1 t) Ax b 2 + Ay b 2 2Ax b) T Ay b) = t1 t) Ax b) Ay b) 2, since < t < 1. That is, f is convex. So, f is convex, an the point x = A T A) 1 A T b satisfies fx) =. We conclue that this point x is the global minimum of f, since if y R n, an if we efine gt) := fx + ty), t R, then Taylor s Theorem with integral remainer implies that gt) = g) + y x)g ) x y g t)x t)t. Then g ) = y, fx) = since fx) =, an g t) = y T [D 2 fx + ty)]y = 2y A A T Ay = 2Ay) T Ay. Therefore, That is, fz) fx) for all z R n. gt) g) = fx). 5. Question 5 For a linear program in stanar form, show that the ual problem of the ual problem is the primal problem. 3

4 The ual problem is maximize b T y subject to A T y c. Equivalently, the ual problem is minimize b T y subject to A T y c. We rewrite the constraint as minimize b T b y+ y subject to the constraints z A T A T I ) y+ y z = c, y+ y z. By the efinition of the ual, the ual of this problem is maximize c T x subject to the constraints A T A T I ) T x b b. The constraint now says Ax b an Ax b, i.e. Ax b. That is, Ax = b an x. Equivalently, this linear program is to minimize c T x) subject to the constraints A x) = b an x. Relabeling x as x, this linear program can be written as: minimize c T x subject to the constraints Ax = b an x, as esire. 6. Question 6 Give an example of a linear program in stanar form where the primal problem is unboune, an the ual problem is infeasible. Or, prove that no such linear program exists. Solution. We use the following linear program where c = 1, A =, b = : minimize x 1 subject to the constraint x 1 =, x 1. This problem is unboune since every positive integer n is feasible, an n as n The ual of this problem is: maximize subject to the constraint y 1 1. Since no real number y 1 satisfies y 1 1, the ual problem is infeasible. 7. Question 7 Let K be the following set of positive semiefinite 2 2 matrices { ) } a b K = : a, b, c R, a + c = 1. b c Show that K has infinitely many extreme points. Hint: which matrices in K have eterminant zero?) 4

5 Solution. Show that K has) infinitely many extreme points. Let a, b R an consier a a b matrix of the form For a matrix in K, note that b 1 a ) a b et = a1 a) b b 1 a 2. So, this matrix has eterminant zero when a1 a) = b 2. So, let a 1, an consier a a1 a) matrices of the form ). This matrix has eterminant zero an trace a1 a) 1 a 1, so its eigenvalues x, y satisfy xy = an x + y = 1. That is, one of the eigenvalues is zero, a a1 a) an the other is one. We claim that every matrix of the form ), a1 a) 1 a a 1 is an extreme point of K. This claim completes the problem. We have alreay verifie that each of these matrices is in K. So, consier any such matrix A. Let x R 2 be the zero eigenvector of C, x. We argue by contraiction. Suppose there exist B, C K, < t < 1 such that A = tb + 1 t)c an B C, then = x T Ax = x T tb + 1 t)c)x = tx T Bx + 1 t)x T Cx. Since B, C, we have x T Bx an x T Cx. Since t > an 1 t) >, we conclue that x T Bx = x T Cx =. That is, x is also a zero eigenvector for B an C. That is, both B, C have a zero eigenvalue. So, both B, C have zero eterminant. In summary, there exist a, b, c 1 such that ) a a1 a) A = a1 a) 1 a ) ) b b1 b) B =, C = c c1 c). b1 b) 1 b c1 c) 1 c Define a function fa) := a1 a) = a a 2, a 1. Then f a) = 1/2)[a1 a)] 1/2 [ 2a + 1], an f a) = [a1 a)] 1/2 + 2a + 1)1/2) 1/2)[a1 a)] 3/2 2a + 1) = [a1 a)] 3/2 [ a1 a) 1/4)1 2a) 2a + 1)] = [a1 a)] 3/2 [ 1/4] = [a1 a)] 3/2 /4. So, f a) < for all < a < 1, f) = f1) =. So, f is strictly concave on [, 1]. That is, if < t < 1, an if b c, which is true since B C), then tfb) + 1 t)fc) < ftb + 1 t)c). By assumption A = tb + 1 t)c, so a = tb + 1 t)b, a1 a) = t b1 b) + 1 t) c1 c). But tfb) + 1 t)fc) < ftb + 1 t)c) says a1 a) > t b1 b) + 1 t) c1 c). We have foun a contraiction. The proof is complete. 8. Question 8 Let g : [, 1] R be a continuous function. Assume that, for any C 1 function h: [, 1] R with h) = h1) =, we have Conclue that gx) = for all x [, 1]. gx)hx)x =. 5

6 Solution. We argue by contraiction. Assume gx) for some x [, 1]. Without loss of generality, gx) >. By the efinition of continuity, ε >, there exists δ > such that, for all y [, 1] with < y x < δ, we have gx) gy) < ε. So, choosing ε := gx)/2, we get some δ > such that, for all y [, 1] [x δ, x+δ], we have gy) > gx) ε > gx)/2 >. Choosing some smaller interval insie [, 1] [x δ, x+δ] if necessary, there is some z, 1) such that z, 4z) [, 1] an for all y z, 4z) we have gy) > gx)/2 >. Let h: [, 1] R so that, if t z ht) = t z) 2 t 4z) 2, if z t 4z, if t > 4z. Then h) = h1) =, h C 1, hy)gy) for all y [, 1], an hy)gy) > gx)z 4 /2 for all y [2z, 3z]. Therefore, hy)gy)y 3z 2z hy)gy)y > z 5 gx)/2 >, a contraiction. We conclue that gx) = for all x [, 1]. 9. Question 9 Let f : [, 1] R be a C 1 function. Assume f) = an f1) = 2. Show that ) 2 5 x fx) x. You may assume that there exists a function g : [, 1] R such that g is C 1, g) =, g1) = 2, an such that ) 2 1 ) 2 x gx) x = min f : [,1] R, x fx) x. ) f)=, f1)=2, f C 1 Solution. Let t R, an let h: [, 1] R such that h) = h1) = an such that h is a C 1 function. Let g t := g + th. Then g t ) =, g t 1) = 1. So, since g minimizes the length, the following erivative is zero: = t t= ) 2 1 x g tx) x = = ) ) 2 1/2 x gx) x x gx) x hx)x ) ) 2 1/2 x gx) In the last line, we integrate by parts. Then, by Question 8, ) ) 2 1/2 x x gx) x gx) =. x gx) hx)x. 6

7 That is, there exists a constant c R such that ) ) 2 1/2 x gx) gx) = c. x Rearranging a bit, ) 2 x gx) 1 c 2 ) = c 2. If c 2 = 1 we get a contraiction = 1. So, we can ivie both sies by 1 c 2 to conclue that /x)gx) is constant. Since /x)gx) is continuous, we conclue that gx) is x constant. Since g) = an g1) = 2, we must therefore have gx) = 2x for all x [, 1]. Equation ) therefore conclues the proposition. 7

Linear and non-linear programming

Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)