MS&E 318 (CME 338) Large-Scale Numerical Optimization. Generalized Least Squares

Transcription

1 Stanford University, Dept of Management Science and Engineering MS&E 318 (CME 338) Large-Scale Numerical Optimization Instructor: Michael Saunders Spring 2010 Project Due Wednesday June 9 Generalized Least Squares Suppose A R m n and b R m. In [1], Chris Paige gives a numerically stable algorithm for solving the generalized least squares problem, in which the linear model b = Ax+w has an unknown noise vector w with mean zero and covariance W. If W is positive definite, an estimate of x may be thought of as solving the optimization problem min (Ax b) T W 1 (Ax b). (1) x Clearly we should expect numerical difficulty if W is ill-conditioned. In order to derive a well-posed version of the problem, consider the linear system ( W A A T ) ( y x ) = ( b 0 ). (2) Furthermore, suppose we obtain the Cholesky factorization W = LL T, where L is lower triangular and nonsingular (since W is positive definite), and consider the optimization problem min x, v v T v subject to Ax + Lv = b, (3) where v = L T y, whose solution is given by A T x 0 I L T v = 0. (4) A L y b This can be rearranged to look more like (2): L A y b L T I v = 0. (5) A T x 0 The GLS problem (1) is solved by all of (2) (5). One of our aims is to find if system (2) is adequate, or if system (5) has an accuracy advantage. Paige s algorithm [1] applies to dense data and provides an efficient and stable method for solving (5). Two sequences of plane rotations Q and P are applied from the left and right respectively to reduce ( b A ) to lower-triangular form while maintaining the triangularity of L. They eliminate one superdiagonal at a time, starting at top-right of A. (The rotations are interleaved. The first Q rotation operates on the top two rows to reduce A 1n to zero. The next two Q rotations operate on rows 2 and 3 to reduce A 2n to zero, and then on rows 1 and 2 to eliminate A 1,n 1, working backwards up the superdiagonal. Similarly for all superdiagonals. Each Q rotation generates one superdiagonal element in L, and the next P rotation promptly restores L to lower-triangular form.) 1

2 2 MS&E 318 (CME 338) Large-Scale Numerical Optimization Files The following files are in the class directory msande318/homework/projectgls: WLSprob.m WLS.m GLS.m compare.m paige-sinum-1979-gls.pdf [A,W,b,x,y] = WLSprob( m,n,conda,condw ) generates sparse data A and W of specified dimensions and condition, along with rhs vector b, such that problem (2) is solved by (x, y). [x,y] = WLS( A,W,b ) solves (2) directly and gives an indication of accuracy by printing t and s for the residuals t = b Ax W y and s = A T y. [x,y] = GLS( A,W,b ) solves (5) directly and again prints t and s. compare( m,n,conda,condw ) runs several solvers on the same test problem and prints the errors in x and y. Questions 1. If W is not too ill-conditioned, problem (1) can be solved by ordinary leastsquares (with the help of W = LL T ). Write function [x,y] = OLS( A,W,b ) for this purpose. Test it on a sequence of increasingly ill-conditioned examples. You may do this by calling compare( 50,30,1e+1,1e+1 ) compare( 50,30,1e+2,1e+2 ) compare( 50,30,1e+3,1e+3 )... until you think it makes no sense to have more ill-conditioned data. Summarize (in words) the results that you see. 2. Systems (2) and (5) may be well defined even if W is singular. A special case of singularity would be when the last q rows and columns of W are zero: ( ) ( ) ( ) W1 A1 b1 W =, A =, b =. (6) 0 What kind of problem does the GLS problem (1) or (2) become? 3. Luckily we can persuade Matlab s sparse Cholesky routine to proceed successfully on a singular W of this kind. Function GLS already allows for this. Write new functions WLSprobq and compareq that generate problems with q zero rows and columns in W as in (6) and compare methods WLS and GLS. A 2 b 2

3 Spring 2010 Project 3 WLS.m function [x,y] = WLS( A,W,b ) % [x,y] = WLS( A,W,b ); % solves the weighted least-squares problem whose solution x % is defined by % [W A](y) = (b), % where W is symmetric and positive definite (or semidefinite?). % 25 May 2010: First version, for MS&E318 (CME338) class project. % Michael Saunders, MS&E, Stanford. % Just use backslash and a step of refinement. [m,n] = size(a); m1 = m+1; mn = m+n; K2 = [W A A sparse(n,n)]; c1 = [b; zeros(n,1)]; z1 = K2\c1; y1 = z1(1:m); x1 = z1(m1:mn); r1 = b - A*x1; t1 = r1 - W*y1; s1 = -A *y1; fprintf( \nm =%5i n =%5i\n b-ax-wy A y\n, m,n) fprintf( WLS %8.1e %8.1e\n, norm(t1,inf), norm(s1,inf)) % Refinement. c2 = [t1; s1]; z2 = K2\c2; dy = z2(1:m); dx = z2(m1:mn); x2 = x1 + dx; y2 = y1 + dy; r2 = b - A*x2; t2 = r2 - W*y2; s2 = -A *y2; fprintf( Refine %8.1e %8.1e\n, norm(t2,inf), norm(s2,inf)) x = x2; y = y2;

4 4 MS&E 318 (CME 338) Large-Scale Numerical Optimization GLS.m function [x,y] = GLS( A,W,b ) % [x,y] = GLS( A,W,b ); % solves the weighted least-squares problem defined by % [W A](y) = (b), % where W is symmetric and positive definite (or semidefinite?). % It follows the Generalized Least Squares approach of Paige (1979) % in working with W = L*L and solving the linear system % [ L A](y) (b) % [L -I ](v) = (0). % 25 May 2010: First version, for MS&E318 (CME338) class project. % Michael Saunders, MS&E, Stanford. % Use backslash on the bigger system and a step of refinement. % % Main reference: % C. C. Paige (1979). % Fast numerically stable computations for % generalized linear least squares problems, % SIAM J. Numer. Anal. 16(1), [m,n] = size(a); m1 = m+1; mm = m+m; mm1 = mm+1; mmn = m+m+n; [L,p] = chol(w, lower ); if p==0 rankl = m; else rankl = p-1; Lsave = L; L = sparse(m,m); L(1:rankL,1:rankL) = Lsave; end % L is m x m and nonsingular. % L is m x rankl. % L is m x m again (but singular). K3 = [sparse(m,m) L A L -speye(m) sparse(m,n) A sparse(n,m) sparse(n,n)]; c1 = [b; zeros(m,1); zeros(n,1)]; z1 = K3\c1; y1 = z1(1:m); v1 = z1(m1:mm); x1 = z1(mm1:mmn); r1 = b - A*x1; t1 = r1 - W*y1; s1 = -A *y1;

5 Spring 2010 Project 5 fprintf( \nm =%5i n =%5i\n b-ax-wy A y\n, m,n) fprintf( GLS %8.1e %8.1e\n, norm(t1,inf), norm(s1,inf)) % Refinement. c2 = [r1-l*v1; v1-l *y1; s1]; z2 = K3\c2; dy = z2(1:m); dv = z2(m1:mm); % Not used dx = z2(mm1:mmn); x2 = x1 + dx; y2 = y1 + dy; r2 = b - A*x2; t2 = r2 - W*y2; s2 = -A *y2; fprintf( Refine %8.1e %8.1e\n, norm(t2,inf), norm(s2,inf)) x = x2; y = y2; compare.m function compare( m,n,conda,condw ) % compare( m,n,conda,condw ) % generates m x n data A and m x m positive definite W % and compares 3 methods for solving the Generalized Least Squares problem % [W A](y) = (b), % where W is symmetric and positive definite. % 25 May 2010: First version, for MS&E318 (CME338) class project. % Michael Saunders, MS&E, Stanford. [A,W,b,x,y] = WLSprob( m,n,conda,condw ); [xo,yo] = OLS( A,W,b ); xerro = norm(x-xo,inf); yerro = norm(y-yo,inf); [xw,yw] = WLS( A,W,b ); xerrw = norm(x-xw,inf); yerrw = norm(y-yw,inf); [xg,yg] = GLS( A,W,b ); xerrg = norm(x-xg,inf); yerrg = norm(y-yg,inf); fprintf( \n xerr yerr\n ) fprintf( OLS %8.1e %8.1e\n, xerro,yerro) fprintf( WLS %8.1e %8.1e\n, xerrw,yerrw) fprintf( GLS %8.1e %8.1e\n, xerrg,yerrg)

6 6 MS&E 318 (CME 338) Large-Scale Numerical Optimization Application In [2], Paige applied his generalized least-squares approach to the block-structured problems arising in Kalman filtering and information filtering. The aim is to estimate the state vector in dynamic linear models when new measurements are gathered at each time step. Just as the GLS approach allows W above to be singular, the resulting filtering implementation allows various covariance or inverse covariance matrices to be singular or ill-conditioned. It is the only approach that does so. References [1] C. C. Paige. Fast numerically stable computations for generalized linear least squares problems. SIAM J. Numer. Anal., 16(1): , [2] C. C. Paige. Covariance matrix representation in linear filtering. Contemporary Mathematics, 47: , 1985.