Updating and downdating an upper trapezoidal sparse orthogonal factorization

Transcription

1 IMA Journal of Numerical Analysis (2006) 26, 1 10 doi: /imanum/dri027 Advance Access publication on July 20, 2005 Updating and downdating an upper trapezoidal sparse orthogonal factorization ÁNGEL SANTOS-PALOMO AND PABLO GUERRERO-GARCÍA Departamento de Matemática Aplicada, University of Málaga, Complejo Tecnológico, Campus Teatinos, Málaga, Spain [Received on 20 August 2001; revised on 22 February 2005] We describe how to update and downdate an upper trapezoidal sparse orthogonal factorization, namely the sparse QR factorization of A k, where A k is a tall and thin full column rank matrix formed with a subset of the columns of a fixed matrix A. In order to do this, we have adapted Saunders techniques of the early 1970s for square matrices, to rectangular matrices (with fewer columns than rows) by using the static data structure of George and Heath of the early 1980s but allowing row downdating on it. An implicitly determined column permutation allows us to dispense with computing a new ordering after each update/downdate; it fits well into the LINPACK downdating algorithm and ensures that the updated trapezoidal factor will remain sparse. We give all the necessary formulae even if the orthogonal factor is not available, and we comment on our implementation using the sparse toolbox of MATLAB 5. Keywords: sparse linear algebra; orthogonalization; static structure; factorization update and downdate. 1. Motivation and related work In certain non-simplex active-set methods (also currently known as basis-deficiency-allowing simplex variations) for linear programming we need to solve a sequence of sparse compatible systems of the form: A A k x = b and A k y = c or k O c d =, 1 where b and c are iteration-dependent vectors, and x, y and d are unknown vectors. Furthermore, the matrix A k R n m k is a tall and thin iteration-dependent full column rank matrix with m k n and rank(a k ) = m k. Such a matrix A k is a subset of the columns of a fixed matrix A R n m with m n and rank(a) = n, and A k+1 is obtained by appending/deleting columns to/from A k with rank(a k+1 ) = rank(a k ) ± 1; when exchanging, deletion is done before appending and rank(a k+1 ) = rank(a k ). In this paper we describe how to update and downdate the sparse orthogonal QR factorization of A k. Saunders (1972) used it in his implementation of the simplex method for linear programming (hence with m k = n for all k), and what we do here is to adapt Saunders methodology to the case m k < n (in fact, his proposal to do an exchange was to delete before adding, and he proclaimed the numerical stability of this exchange in spite of the rank reduction after the deletion). We use the data structure of George and Heath (George & Heath, 1980; Heath, 1982), but we allow row downdating on such a static structure. As we work on top of a static structure, we do not have to deal with intricate data structures Abridged version of a talk presented at the 19th Biennial Conference on Numerical Analysis, Dundee, Scotland, June 2001, and at the 5th FoCM Workshop on Numerical Linear Algebra, Santander, Spain, July santos@ctima.uma.es Corresponding Author. pablito@ctima.uma.es c The author Published by Oxford University Press on behalf of the Institute of Mathematics and its Applications. All rights reserved.

2 2 Á. SANTOS-PALOMO AND P. GUERRERO-GARCÍA as in dynamic approaches like Davis & Hager (1999) and Edlund (2002). Furthermore, small non-zeros which actually would be structural zeros (Barlow, 1990) can be avoided. As pointed out by a referee, the algorithms proposed depend on knowing the structure of all possible updates (i.e. the structure of the larger fixed matrix A) in advance, and thus the algorithms are not applicable to situations where potential updates are not known in advance. The interested reader can find a more detailed survey of related work in Santos & Guerrero (2003, Section 1). As we shall see in the following sections, the key point is the management of an implicitly defined column permutation in order to dispense with determining a new ordering in each step; we shall also show that it fits well into the LINPACK downdating algorithm and this ensures that the updated trapezoidal factor will remain sparse. We also provide mathematically rigorous descriptions of both the updating and downdating processes, which are usually avoided when dealing with static structures in sparse settings, as well as illustrative examples. 2. Initial factorization and sparse issues Let us consider the sparse QR factorization of a short and fat matrix A k, e.g. with m k = 3rowsand n = 7 columns. When we compute this factorization with the MATLAB 5 qr black-box we would obtain, say, the Wilkinson diagram qr = O O. O O O O O A k Q k R k Π k Note that R k Π k is a staircase-shaped, permuted upper trapezoidal matrix, where the permutation Π k is implicitly defined by the staircase shape of the structure; in this case Π k = [e 1, e 3, e 6, e 2, e 4, e 5, e 7 ] or Π k = [e 1, e 4, e 2, e 5, e 6, e 3, e 7 ], where e i R 7 is the i-th column of the identity matrix I 7. It could even be the case that O O qr = O O O, O O O O O O A k Q k R k Π k Πk = [e 3, e 4, e 7, e 1, e 2, e 5, e 6 ] or Π k = [e 4, e 5, e 1, e 2, e 6, e 7, e 3 ]. In a more general framework, let A k R n m k be sparse and rank(a k ) = m k n. We know that there exists an implicitly defined permutation Πk of the columns of A k such that A k Π k = Q k R k. = Qk [R i R d ], Q k, R i R m k m k and R d R m k (n m k ), with R k upper trapezoidal, R i upper triangular and non-singular and Q k orthogonal. (We have used R i and R d rather than a more standard notation R 1 and R 2 to avoid subindex clash problems with R k.) Nevertheless, the permutation Πk is not explicitly performed, thus R k Π k is what we are going to

3 DOWNDATING A SPARSE ORTHOGONAL FACTORIZATION 3 maintain. Note also that a reordering of the columns of A k has no impact on R k, for S k A k Π k = (S k Q k)r k, thus only a reordering of the rows of Q k would take place. This factorization is computable with the black-box qr in MATLAB 5 (but not every A k can be factorized in this way with the sparse sqr package of Matstoms (1994), because it is intended for matrices with more rows than columns). However, this factorization is not updatable with qrupdate in MATLAB 5 because it only works with dense matrices. Also, there are no efficient ways yet to update this QR factorization using multifrontal techniques. Enlarging R k below with n m k zero rows we obtain the sparse Cholesky factor of Π k A k A k Π k 0 (see Higham, 2002, Theorem 10.9, p. 201). Computing this factor from the sparse QR given above makes it unnecessary to form the product A k A k and the complete pivoting that we would have to perform if a dense semidefinite Cholesky method on A k A k were used; in other words, our Π k is defined only for sparsity purposes rather than for numerical purposes. This is not expected to cause numerical difficulties. It is well known that when A k is a subset of the columns of a fixed matrix A R n m with m n and rank(a) = n, a static structure (i.e. a priori set up) can be used to try to minimize fill in the triangular factor. The sparse structure of A k A k is a subset of that of AA (George & Heath, 1980), hence an a priori permutation P (not related to Π k ) of the rows of A can be chosen to sparsify the Cholesky factor R of PAA P. Moreover, when a row is deleted, there must be space in the static structure, as hyperbolic rotations modify the structure in the same way as Givens rotations. Additional remarks on the suitability of the different row orderings that can be obtained with MATLAB 5 can be found in Santos & Guerrero (2003, Section 2). The column order of A does not affect the density of the Cholesky factor R of PAA P, but does affect the amount of intermediate work to compute it. When we have to refactorize A k (or to calculate an initial factorization of A 0 ), an a priori ordering of the columns of A can be beneficial (Björck, 1996, pp ). Since access to A and R is done by columns, they are stored by a simple compressed column scheme that allows us to take advantage of the locality of reference (Stewart, 1998, p. 110). As pointed out by Davis & Hager (1999, Section 7.1), in MATLAB every change in the structure of a sparse matrix would entail a new copy of the entire matrix. Hence, when MATLAB annihilates an element of a sparse matrix, the operating system does not deallocate the memory locations previously allocated for that element. Thus, rather than overlaying a data structure on the triangular factor (as Vempati et al., 1991, did), we mark with an special value (e.g. NaN in MATLAB) the elements not used in the structure previously set up, and we convert them into zeros just before selecting the submatrix to operate with. Thus, sparse numerical linear algebra primitives of the sparse toolbox (Gilbert et al., 1992) can be used, but we have to recover the structure after the algebraic operation involved, reinserting NaNs in those elements where fill-in did not occur yet; this overhead could be avoided with low-level programming. 3. Updating the factorization When we add the p-th constraint to the active set, we have to add a new column to the right of A k, i.e. we have to add a new row a p to the bottom of A k : A A k+1 = k a. p

4 4 Á. SANTOS-PALOMO AND P. GUERRERO-GARCÍA Then, given the QR factorization of A k Π k, we want to determine the new QR factorization after the addition of a p ; this problem is known as updating the QR factorization. Since A k = Q k R k Π k, we can write A A k+1 = k Qk O Rk Π k a = p O 1 a, p so we have Q k O A k Rk Π k O O O 1 a = p a = p O O O O O. (3.1) Let f j be the index of the column of Πk where e j appears; e.g. regarding the first Wilkinson diagram of Section 2, f 1 = 1, f 3 = 2, f 6 = 3 and f 2 = 4 > 3 =. m k ; or, with respect to the second Wilkinson diagram of Section 2, f 3 = 1, f 4 = 2, f 7 = 3 and f 1 = 4 > 3. = m k. For each j 1: n, we proceed to annihilate the j-th element (if non-zero) of the updated version of a p ; such an annihilation is done by a Givens rotation G( f j, m k +1) R (m k+1) (m k +1) (Golub & Van Loan, 1996, Section 5.1.8) only if f j m k, since otherwise the factorization is complete. Hence, after m k rotations (in the worst case) we have G. = G( fn, m k + 1) G( f 2, m k + 1) G( f 1, m k + 1), F G Rk Π k. a = R k+1 Π k+1, (3.2) p where Π k+1 is a permutation matrix such that R k+1 is trapezoidal with non-zero diagonal elements, and F is a suitable row permutation for R k+1 Π k+1 to be staircase shaped. Then, from (3.1) and (3.2), A k+1 Π k+1 = Qk O O GFF G Rk Π k 1 a p Π k+1. = Q k+1 R k+1. It is well known that in practical algorithms we can dispense with the orthogonal factor, since R k can be updated even if Q k is not available. Furthermore, as we shall illustrate in the following example, the static sparsity structure itself implicitly defines both the updated column permutation Π k+1 and the suitable row permutation F ; in fact, the actual Givens rotation used to annihilate the j-th element is G( j, n + 1) R (n+1) (n+1). This ensures that the updated trapezoidal factor will remain sparse. 3.1 Example of updating Let us consider the matrix whose structure is given in George et al. (1988, p. 104). In MATLAB, symbfact(a, col ) accomplishes the symbolic analysis of AA directly on A and consequently

5 DOWNDATING A SPARSE ORTHOGONAL FACTORIZATION 5 returns an upper bound structure for R : 1 1 A =, R = Note that we have considered R instead of R because of locality of reference issues (MATLAB stores the sparse matrices by columns). We have chosen to proceed row-wise in the mathematical description of the updating due to familiarity reasons; however, for the example to be as real as possible, we shall work column-wise, postmultiplying by Givens matrices rather than premultiplying by their transposes. We have rotated rows a 6 and a 7 into R and now we are going to rotate a 3 (i.e. k = 2, A k = [a 6 a 7 ] and A k+1 = [a 6 a 7 a 3 ], where a i is column i of A): 1 1 [R 1 1 a 3 ] = and [Π2 R 2 a ] =, [ ] where the working vector is located to the right of the vertical bar. Postmultiplying by 1/ 2 1/ 2 [ 1/ 2 1/ ] 2 and by 6/3 1/ 3 1/ 3, first columns 1 and 3 and then columns 2 and 3, the elements of the working 6/3 vector are annihilated downhill using Givens rotations: / 2 1 1/ and Π3 R 3 = 1/ 2 6/ /3 1/ /3 2/ 3 Note that the fill is restricted to the working vector and the column with respect to which we are rotating. The column just created has appeared in the last position (hence F = I ), but it does not happen in general. Although we have shown here only the columns of R that occur in Πk R k, we actually work within the whole structure to avoid the explicit computation of the permutations Π k and F, sothe

6 6 Á. SANTOS-PALOMO AND P. GUERRERO-GARCÍA mathematical application of F amounts to a simple copy of the working vector to the right position within the sparsity structure. Finally, the addition of the 5th constraint entails no rotations: Π4 R 4 = 1/ 2 6/ /3 1/ 3 0, /3 2/ 3 1 thus obtaining the permuted trapezoidal matrix for A 4 = [a 6 a 7 a 3 a 5 ] or any column permutation A 4 S Downdating the factorization When we delete the q-th constraint from the active set, we can assume that it is the first row of A k : a A k = q, A k+1 because if it were not the first row we only have to exchange beforehand the corresponding row and the first row in A k. Then, given the QR factorization of A k Π k, we want to determine the new QR factorization after aq is deleted; this problem is known as downdating the QR factorization. It can be shown (see Guerrero, 2002, Section 3.2.4) that the presence of the permutation matrix does not modify in essence the classical downdating algorithm, although it turns out not to be suitable to be combined with the static structure unless a separate working vector is introduced. As was pointed out by one of the referees, the computations of the classical downdating method could be reorganized to rotate into the bottom row (more precisely, into the row corresponding to the bottom-most non-zero element of the vector q R m k defined below), with that row being the working vector. What we are (equivalently) going to do here is to reorganize the computation to rotate into a new top row, with that row being the working vector, to illustrate the suitability of the proposal to be used in a LINPACK setting. In a sparse setting it is usual to dispense with Q k. Let us show how the implicitly determined column permutation fits well into the LINPACK algorithm (Saunders, 1972) to downdate the Cholesky factor, because the system Πk R k q = a q is also compatible with only one solution, and solving it we obtain q; furthermore, it does not destroy the predicted static structure. First we enlarge the matrix R k Π k with a suitable number δ n to be determined below: O O O O O O O. δn O R k = = q R k Π k O O. O O O O O Now if we apply to R k several Givens rotations G(1, j) R (m k+1) (m k +1) with j m k + 1: 1: 2 to annihilate the non-zeros of q, we obtain F G. R k = F G(1, 2) G(1, m k ) G(1, m k + 1). R k+1 R k = O,

7 DOWNDATING A SPARSE ORTHOGONAL FACTORIZATION 7 where R k+1 is the enlarged matrix corresponding to the new factor with R k+1 trapezoidal with non-zero diagonal elements:. α αv R Π k+1 k+1 = (α =±1), O R k+1 Π k+1 and F is the permutation needed to move the zero row just created to the m k + 1 position. G and F being orthogonal matrices implies that R k+1 R k+1 = R k R k : [ α O ] α αv Π k+1 δn q δn O απk+1 v =. Π k+1 R k+1 O R k+1 Π k+1 O q R k Π k Π k R k Multiplying out and comparing blocks we have that Πk+1 v = Π k R k q = a q and δn 2 = 1 q 2 2 = 0, because [ 1 v ] Π k+1 δ 2 n + q 2 2 q R k Π k Πk+1 v Π k+1 (R k+1 R k+1 + vv = )Π k+1 Πk R k q Π k R k R. kπ k Thus, Πk+1 R k+1 R k+1π k+1 = Πk R k R kπ k a q aq, which is what we wanted, for A k+1 A k+1 = Π k+1 R k+1 R k+1π k+1 = Π k R k R kπ k a q a q = A k A k a qa q. As in the algorithm given above, the Givens matrices are (m k + 1) (m k + 1). Using the additional working row we can get useful information to trigger the refactorization when Πk+1 v and a q differ substantially; furthermore, the data structure always has enough space allocated to work, and the fill-in is confined to the working row, as in the updating algorithm but not in the classical downdating algorithm (Guerrero, 2002, Section 3.2.4). There is a subtle difference with the m k > n LINPACK case, in which q is the first row of an orthonormal matrix (a matrix whose columns are orthonormal vectors, cf. Stewart, 1998, p. 56) and hence q 2 1 in general. On the other hand, we can ensure q 2 = 1 in exact arithmetic when m k n, since q is the first row of an orthogonal matrix. This case was excluded from LINPACK because it leads to a reduction in rank after the downdate but, as was pointed out in 1972 by Saunders (1972), numerical difficulties are not expected. 4.1 Example of downdating Let us go on with the example of Section 3.1; now we want to delete a7. First we solve the compatible system Π4 R 4 q = a 7 to obtain q = [0; 6/3; 1/ 3; 0]; as was pointed out by Heath (1982, p. 229) (with his R 1 being the same as our R i ), It is unnecessary to extract R 1 from the data structure or write a special back substitution routine to skip over the zero rows, since the same effect may be obtained by simply setting the zero diagonal elements equal to 1 and noting that the corresponding components of the right-hand sides will already be zero. and hence solving this system does not need any special back substitution routine, since we can proceed by overlaying Π4 R 4 on I 6 to obtain [0; 0; 6/3; 1/ 3; 0; 0] and then select the first, third, fourth

8 8 Á. SANTOS-PALOMO AND P. GUERRERO-GARCÍA and fifth components. Now we enlarge Π4 R 4 and rotate with respect to the first column to annihilate the non-zero elements of q westbound: 0 0 6/3 1/ R 4. δn q = O Π4 = R 0 1/ 2 6/2 0 0, /3 1/ /3 2/ 3 1 where the working vector is now located to the left of the vertical bar; postmultiplying by [ 0 1] 1 0 and [ 1/ 3 6/3 ] by, first columns 1 and 4 and then columns 1 and 3, we obtain 6/3 1/ 3 1/ 3 0 6/ / / / 3 0, [ R 5. O]F = 1 1/ 2 1/ / / / Note that the result is the same (barring signs and the last column) as just after having annihilated the first element when we were rotating a 3, that the vector a 7 has been recovered in the additional column, and that in this case α = 1. We have to reset to NaN the zeros shown in bold in order to conserve the predicted structure for R in the columns involved. The zero column just created has not appeared in the last position, hence F I in general. Once again, the work with the static structure avoids the explicit computation of the permutation F. We can check the result using MATLAB with [Q,R]= qr(sparse([ ; ; ])). 5. Conclusions and future work We have described how to update and downdate an upper trapezoidal sparse orthogonal factorization. We have adapted the techniques of Saunders (1972) for square matrices to rectangular matrices (with fewer columns than rows), by using the static data structure of George and Heath (George & Heath, 1980; Heath, 1982) but allowing sparse row downdating on it. An implicitly determined column permutation allows us to dispense with its recomputation; we have shown how it fits well into the LINPACK downdating algorithm and ensures that the updated trapezoidal factor will remain sparse. A FORTRAN implementation of the updating process (without the downdating) can be found in the SPARSPAK package, which is currently being translated into C++ as described by George & Liu

9 DOWNDATING A SPARSE ORTHOGONAL FACTORIZATION 9 (1999). The availability of this SPARSPAK++ package implies a rapid prototyping of a low-level implementation of the sparse proposal given here. Nowadays, the updatable sparse factorization described in this paper has been implemented in a MATLAB toolbox, which has been used to solve the sequence of sparse compatible systems needed to implement certain non-simplex active-set methods for linear programming. The formulae needed and the computational results obtained when solving the first 15 smallest NETLIB problems with a highly degenerate Phase I are the subject of a forthcoming article (Guerrero & Santos, 2003), in which we also comment on several advantages of using QR rather than LU, its suitability to a combined interior-point simplex methodology, and several parallelization issues. The numerical analysis of the behaviour in floating-point arithmetic when solving dense quadratic programs with an orthogonal factorization of A k similar to the one given in Section 2 was developed by Powell (1981, and references therein). When numerical difficulties with the rank arise, the matrix R k can be postprocessed as indicated by Heath (1982) to obtain a rank-revealing factorization. At present we deal only with problems in which we can choose an a priori permutation P of the rows of A such that the Cholesky factor of PAA P is sparse. When this cannot be done (e.g. when A has dense columns), it would be better to use dynamic techniques as in Davis & Hager (1999) and Edlund (2002), adapting them to the semidefinite case to work with A k A k or else regularizing the problem to work with σ I n + A k A k. Acknowledgements The authors thank the referees for having put a lot of work into producing the reports and marking up our previous manuscript, and for providing us with some references that we were not aware of. We also want to thank John Gilbert for confirming that the sparse orthogonal factorization in MATLAB 5isroworiented and Givens-based, and providing us technical details about the interface, as well as Christof Vömel at CERFACS for several remarks on the first sections. REFERENCES BARLOW, J. L. (1990) On the use of structural zeros in orthogonal factorization. SIAM J. Sci. Stat. Comput., 11, BJÖRCK, Å. (1996) Numerical Methods for Least Squares Problems. Philadelphia, PA: SIAM. DAVIS, T.& HAGER, W. (1999) Modifying a sparse Cholesky factorization. SIAM J. Matrix Anal. Appl., 20, EDLUND, O. (2002) A software package for sparse orthogonal factorization and updating. ACM Trans. Math. Soft., 28, GEORGE, A.& HEATH, M. T. (1980) Solution of sparse linear least squares problems using Givens rotations. Linear Algebra Appl., 34, GEORGE, A.& LIU, J. (1999) An object-oriented approach to the design of a user interface for a sparse matrix package. SIAM J. Matrix Anal. Appl., 20, GEORGE, A., LIU, J.&NG, E. (1988) A data structure for sparse QR and LU factorizations. SIAM J. Sci. Stat. Comput., 9, GILBERT, J.R.,MOLER, C.B.&SCHREIBER, R. S. (1992) Sparse matrices in MATLAB: design and implementation. SIAM J. Matrix Anal. Appl., 13, GOLUB, G. H.& VAN LOAN, C. L. (1996) Matrix Computations. Baltimore, MD: North Oxford Academic Publishing. GUERRERO-GARCÍA, P. (2002) Range space methods for sparse linear programs. Ph.D. Thesis, Department of Applied Mathematics, University of Málaga. (In Spanish.)

10 10 Á. SANTOS-PALOMO AND P. GUERRERO-GARCÍA GUERRERO-GARCÍA, P.& SANTOS-PALOMO, Á. (2003) Solving a sequence of sparse compatible systems. Technical Report MA-02-03, Department of Applied Mathematics, University of Málaga. HEATH, M. T. (1982) Some extensions of an algorithm for sparse linear least squares problems. SIAM J. Sci. Stat. Comput., 3, HIGHAM, N. J. (2002) Accuracy and Stability of Numerical Algorithms, 2nd edn. Philadelphia, PA: SIAM. MATSTOMS, P. (1994) Sparse QR factorization with applications to linear least squares problems. Ph.D. Thesis, Department of Mathematics, Linköping University. POWELL, M. J. D. (1981) An upper triangular matrix method for quadratic programming 4. Nonlinear Programming, (O. L. Mangasarian, R. R. Meyer & S. M. Robinson eds). New York: Academic Press, pp SANTOS-PALOMO, Á. & GUERRERO-GARCÍA, P. (2003) Updating and downdating an upper trapezoidal sparse orthogonal factorization. Technical report MA-02-02, Department of Applied Mathematics, University of Málaga. SAUNDERS, M. A. (1972) Large-scale linear programming using the Cholesky factorization. Technical Report CS-252, Computer Science Department, Stanford University. STEWART, G. W. (1998) Matrix Algorithms I: Basic Decompositions. Philadelphia, PA: SIAM. VEMPATI, N., SLUTSKER, I. W.& TINNEY, W. F. (1991) Enhancements to Givens rotations for power system state estimation. IEEE Trans. Power Syst., 6,