MS&E 318 (CME 338) Large-Scale Numerical Optimization

Transcription

1 Stanford University, Management Science & Engineering (and ICME MS&E 38 (CME 338 Large-Scale Numerical Optimization Course description Instructor: Michael Saunders Spring 28 Notes : Review The course teaches reliable numerical methods for solving linear equations (x = b and linear and nonlinear optimization problems (LO and NLO, with a focus on sparse matrices as in the conjugate-gradient method, the simplex method, and large-scale constrained optimization. 3 units, 5 homeworks (6%, project (4%, no mid-term or final. Prerequisites: Basic numerical linear algebra, including LU and QR factorizations, and an interest in Matlab, sparse-matrix methods, and gradient-based algorithms for constrained optimization. Syllabus. Review of dense factorizations (LU, QR, EVD, SVD, U T V = T 2. Overview of optimization software (problem types, NEOS, Matlab, TOMLB 3. Suggested projects 4. Iterative methods for symmetric x = b (symmetric Lanczos process, CG, SYMMLQ, MINRES, MINRES-QLP 5. Iterative methods for unsymmetric x = b and least squares (Golub-Kahan process, CGLS, LSQR, LSMR, Craig, rnoldi process, GMRES 6. The primal simplex method (phase in practice, basis factorization, updating, crash, scaling, degeneracy 7. LUSOL: Basis Factorization Package (the engine for MINOS, SQOPT, SNOPT, MILES, PTH, lp solve 8. Basis LU updates (Product-Form, Bartels-Golub, Forrest-Tomlin, Block-LU 9. Primal-dual interior methods for LO (CPLEX, HOPDM, IPOPT, KNITRO, LOQO, MOSEK and convex nonlinear objectives (PDCO, Basis Pursuit, BP Denoising (Lasso, LRS, Homotopy, SP. The reduced-gradient method (MINOS part. BCL methods (ugmented Lagrangians, LNCELOT 2. LCL methods (MINOS part 2, Knossos 3. SQP methods (NPSOL, SQOPT, SNOPT 4. SNOPT input/output

2 2 MS&E 38 (CME 338 Large-Scale Numerical Optimization im and notation We first review matrix factorizations and the elementary triangular and orthogonal matrices that are used to compute them. We also show the existence of orthogonal transformations of symmetric or rectangular matrices to tridiagonal or bidiagonal form. Systems of equations are denoted x = b, where and b are always real. Depending on the topic, may be symmetric, positive definite, indefinite, or rectangular. If is square and nonsingular, its condition number is defined to be cond(. Wherever possible, we use Householder notation, whereby {, a, α} refer to {matrix, vector, scalar}. Symmetric posdef Symmetric indefinite Square or tall Symmetric Square or tall Factors LL T or LDL T LDL T, D block-diag with and 2 2 blocks LU or QR, triangular L, U, R, orthogonal Q V T V T, tridiagonal T UBV T or UT V T, bidiagonal B, tridiagonal T Exercise If x solves x = b and x+ x solves a perturbed system (+ (x+ x = b, x show that x+ x cond(. Thus, the condition number measures the sensitivity of the solution of x = b. This is a property of the problem (not of a method for solving it. stable method for solving x = b is one that will always give a computed solution x+ x for which the size of the error x is comparable to the sensitivity of x (not worse.. Elementary triangular matrices Sometimes we make use of n n triangular matrices of the form L(µ, L µ k (µ µ 4, k = 3, n = 6, µ 5 µ 6 where µ τ and µ i τ for some τ [, ] say. In using such matrices to form an LU factorization of, it s easier to think of multiplying by matrices M(µ or M k (µ that are their inverse: M(µ = L(µ, M µ k (µ µ 4 µ 5 µ 6 = L(µ. We can think of multiplying by M (µ in order to change the first column of to zero except for the first entry. Ultimately, = LU is equivalent to M = U, where L and M are the product of elementary triangular matrices. Example Beware of the effect of large numbers: 6 =, 2 6 = 6 6. Our aim here (and throughout numerical linear algebra is to avoid creating large numbers, because floating-point numbers can represent both large and small numbers within a fixed number of bits (64 bits for IEEE double-precision floating-point arithmetic, but the absolute error is larger for larger numbers.

3 Spring 28, Notes Review 3 Cholesky example If is symmetric positive definite (spd, the Cholesky factorization = LL T shows that the ith row of L satisfies i j= L2 ij = ii, so that each L ij is strongly bounded. Cholesky does not generate large numbers. It is extremely stable. It s the best way to test if is spd. + µ if µ, Exercise Show that cond(l(µ = cond(m(µ 2.68 if µ =, µ 2 if µ..2 LU factorization square unsymmetric system of equations x = b is normally solved by LU factorization of. This is commonly called Gaussian elimination with X pivoting, where X refers to a pivot strategy that makes the method stable in the context of numerical computation. Pivot strategies permute the rows and/or columns of to ensure that one factor (L or U is friendly or well-conditioned. This is how we preserve numerical stability. Thus, P P 2 = LU for some permutations P and P 2, where L and U are lower and upper triangular and either L or U is well-conditioned. Typically, L has unit diagonals and bounded off-diagonals like L k (µ above: L ij τ, where we can enforce τ = for maximum stability, but τ = 2 or or allows attention to sparsity. Such an L is likely to be well-conditioned. The other factor U is likely to reflect the condition of. More generally, P P 2 = LDU, where L and U both have unit diagonals, at least one of them has bounded off-diagonals, and the condition of is reflected in D. Partial pivoting means that the off-diagonals of only L (or only U are bounded. Rook pivoting bounds the off-diagonals of both L and U. Complete pivoting has the same effect..3 Elementary orthogonal matrices Plane rotations and Householder transformations take the form γ σ P R 2 Ik, H σ γ k I β k v k vk T R m m, k < m, where γ 2 + σ 2 = and β k = 2/ v k 2 (with β k = if v k =. There are no large numbers, and for vectors v and w of suitable dimension, P v and H k w have the same 2-norm as v and w because P T P = I and H 2 k = I. For example, if H reduces the first column of an m n matrix (m > n to a multiple of e, we have the beginning of the QR factorization = QR, where Q = H H 2... H n and R is upper triangular. The construction of β k and the first element of v k is intricate to ensure stability, but the remainder of v k is proportional to the values that H k reduces to zero..4 QR factorization For an m n matrix with m n, we can multiply on the left by a product of Householder matrices to reduce to upper triangular form (where H n = I if m = n: ( Q T Rn = R, Q = H H 2... H n, R =..5 Two least-squares problems For an m n matrix with m > n and m < n respectively, two types of least-squares problem are solved by normal equations of the first and second kind: Overdetermined system: min x b 2 2 T x = T b Underdetermined system: min x 2 2 s.t. x = b T y = b, x = T y

4 4 MS&E 38 (CME 338 Large-Scale Numerical Optimization ssuming has full rank, we can use the QR factors of and T to obtain x: ( ( R c Overdetermined system: b = Q Rx = c c ( 2 ( R z Underdetermined system: T = Q R T z = b, x = Q.6 Orthogonal tridiagonalization of symmetric symmetric n n can be reduced to tridiagonal form by multiplying on the left and right by H 2, then H 3,..., then H n. Note that is not altered, and no large numbers are produced. Thus, V T V = T, where V = H 2 H 3... H n and T is tridiagonal. This is the first part of computing the eigenvalue decomposition of symmetric. Note that the first row and column of V = H 2 H 3... H n must be the first row and column of I n. The existence of V T V = T for symmetric means that there exists a transformation ( V T ( ( b T b V ( β e = T β e T, ( where b is a nonzero n-vector and β = b. We generate this transformation when we apply the Lanczos process to symmetric n n with starting vector b..7 Orthogonal bidiagonalization of general For an m n matrix with m n, we can multiply on the left and right by different Householder transformations to reduce to upper-bidiagonal form B (with H n = I if m = n: U T V = B, U = H H 2... H n, V = H 2 H3... H n. This is the first part of computing the singular value decomposition of unsymmetric. Note that H depends on the first column of, and then H 2 depends on the effect of H. gain, the first row and column of V must be the first row and column of I n. The existence of U T V = B (upper bidiagonal means there exists a transformation U T ( b ( = ( β V e B, (2 where b is a nonzero m-vector, β = b, and B is lower bidiagonal. We generate this transformation when we apply the Golub-Kahan process to m n with starting vector b..8 Orthogonal tridiagonalization of general gain for an m n matrix with m n, we can multiply on the left and right by different Householder transformations to reduce to tridiagonal form T : U T V = T, U = H 2 H 3... H n, V = H 2 H3... H n, where is not altered. Note that H 2 depends on the first column of and H 2 depends on the first row of, independent of H 2. The existence of U T V = T for general means that there exists a transformation ( U T ( ( c T b V ( γ e = T β e T, (3 where b and c are nonzero m- and n-vectors, β = b, and γ = c. We generate this transformation when we apply the orthogonal tridiagonalization process to m n with starting vectors b and c. The orthogonal reductions in sections.6.8 are existence proofs for three iterative processes that we use later for solving equations or least-squares problems.

5 Spring 28, Notes Review 5.9 Further reading Golub and Van Loan [2] discuss elementary triangular matrices (Gaussian transformations in chapter 3.2, and elementary orthogonal transformations (Householder reflections and Givens rotations in chapter 5.. The Lanczos process was originally used for computing eigenvalues of symmetric [3]. Much later it was used by Paige and Saunders [4] for solving symmetric indefinite x = b. The Golub-Kahan process was introduced almost incidentally by Golub and Kahan in their paper about computing the singular value decomposition (SVD of dense matrices []. It was used by Paige and Saunders [5, 6] for solving least-squares problems min x b 2 (and also square x = b and underdetermined systems min x 2 st x = b. Orthogonal tridiagonalization was introduced by Saunders, Simon, and Yip [8] for solving square x = b. Reichel and Ye [7] used the same iterative process later for solving square x = b and rectangular min x b 2. References [] G. H. Golub and W. Kahan. Calculating the singular values and pseudoinverse of a matrix. SIM J. Numer. nal., 2:25 224, 965. [2] G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. The Johns Hopkins University Press, Baltimore, fourth edition, 23. [3] C. Lanczos. n iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Res. Natl. Bur. Stand., 45: , 95. [4] C. C. Paige and M.. Saunders. Solution of sparse indefinite systems of linear equations. SIM J. Numer. nal., 2:67 629, 975. [5] C. C. Paige and M.. Saunders. LSQR: n algorithm for sparse linear equations and sparse least squares. CM Trans. Math. Software, 8(:43 7, 982. [6] C. C. Paige and M.. Saunders. lgorithm 583; LSQR: Sparse linear equations and least-squares problems. CM Trans. Math. Software, 8(2:95 29, 982. [7] L. Reichel and Q. Ye. generalized LSQR algorithm. Numer. Linear lgebra ppl., 5:643 66, 28. [8] M.. Saunders, H. D. Simon, and E. L. Yip. Two conjugate-gradient-type methods for unsymmetric linear equations. SIM J. Numer. nal., 25(4:927 94, 988.