FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution manual). Copying others solutions or programs is strictly prohibited and will result in grade of 0 to all involved students. Please type your answers in Latex and submit a single PDF file on icollege before due time. Please do not include your name anywhere in the submitted PDF file. Instead, name your PDF file as hw4 123456789.pdf (replace 123456789 by your own Panther ID number). It is recommended to use notations that are consistent to lectures. By default all vectors are treated as column vectors. 1
Problem 1. (1 point) Let f(x), x [x 1, x 2 ] T R 2, be given by f(x) = 5 2 x2 1 + 1 2 x2 2 + 2x 1 x 2 3x 1 x 2. (a) Express f(x) in the form of f(x) = 1 2 xt Qx x T b. (b) Find the minimizer of f using the conjugate gradient algorithm. x (0) = [0, 0] T. Use a starting point of (c) Calculate the minimizer of f analytically form Q and b, and check it with your answer in part (b). 2
Problem 2. (1 point) Given f : R n R, f C 1, consider the algorithm x (k+1) = x (k) + α k d (k), where d (1), d (2),... are vectors in R n, and α k 0 is chosen to minimize f(x (k) + αd (k) ); that is α k = arg min f(x (k) + αd (k) ). α 0 Note that the above general algorithm encompasses almost all algorithms that we discussed in this part, including the steepest descent, Newton, conjugate gradient, and quasi-newton algorithms. Let g (k) = f(x (k) ), and assume d (k)t g (k) < 0 (a) Show that d (k) is a descent direction for f, in the sense that there exists ᾱ > 0 such that forall α (0, ᾱ], f(x (k) + αd (k) ) < f(x (k) ). (b) Show that α k > 0. (c) Show that d (k)t g (k+1) = 0. (d) Show that the following algorithms all satisfy the condition d (k)t g (k) < 0, if g (k) 0: 1. Steepest descent algorithm; 2. Newton s method, assuming the Hessian is positive definite; 3. Conjugate gradient algorithm; 4. Quasi-Newton algorithm, assuming H (k) 0. (e) For the case where f(x) = 1 2 xt Qx x T b, with Q symmetric and positive definite, derive an expression for α k in terms of Q, d (k), and g (k). 3
Problem 3. (1 point) Consider the optimization problem: minimize c 1 x 1 + c 2 x 2 + + c n x n subject to Ax = b where c i 0, i = 1,..., n. Convert the above problem into an equivalent standard form linear programing problem. Hint: Given any x R, we can find unique numbers x +, x R, x +, x 0, such taht x = x + + x and x = x + x. 4
Problem 4. (1 point) Consider the linear program: maximize 2x 1 + x 2 subject to 0 x 1 5 0 x 2 7 x 1 + x 2 9 Convert the problem to standard form and solve it using the simplex method. 5
Problem 5. (1 point) Write a MATLAB (or Python) routine for implementing the conjugate gradient algorithm for general functions. Use the secant method for the line search. Test the three different formulas, Hestenes-Stiefel, Polak-Ribière, and Fletcher-Reeves, for β k on Rosenbrock s function f(x) = 100(x 2 x 2 1) 2 + (1 x 1 ) 2 with an initial condition x (0) = [ 2, 2] T. For this exercise, reinitialize the update direction to the negative gradient every 6 iterations. Set the stopping criterion to g (k) < ε for ε = 10 7. For each of the three formulas, show the number of iterations needed to terminate the algorithm and create a table of x (k), x (k) x, g (k), and f(x (k) ) for k = 0, 1, 2, 3, 4 using the following template (keep 4 digits in all numbers): k x (k) x (k) x g (k) f(x (k) ) 0 1 2 3 4 6
Problem 6. (1 point) Write a MATLAB (or Python) routine for implementing the quasi-newton algorithm for general functions. Use the secant method for the line search. Test the three different formulas H (k) on the Rosenbrock s function f(x) = 100(x 2 x 2 1) 2 + (1 x 1 ) 2 with an initial condition x (0) = [ 2, 2] T. For this exercise, reinitialize the update direction to the negative gradient every 6 iterations. Set the stopping criterion to g (k) < ε for ε = 10 7. For each of the three formulas, show the number of iterations needed to terminate the algorithm and create a table of x (k), x (k) x, g (k), and f(x (k) ) for k = 0, 1, 2, 3, 4 using the following template (keep 4 digits in all numbers): k x (k) x (k) x g (k) f(x (k) ) 0 1 2 3 4 7