ON THE EFFICIENT UPDATE OF RECTANGULAR LU FACTORIZATIONS SUBJECT TO LOW RANK MODIFICATIONS

Transcription

1 ON THE EFFICIENT UPDATE OF RECTANGULAR LU FACTORIZATIONS SUBJECT TO LOW RANK MODIFICATIONS PETER STANGE, ANDREAS GRIEWANK, AND MATTHIAS BOLLHÖFER Abstract. In ths paper we ntroduce a new method for the computaton of KKT matrces that arse from solvng constraned, nonlnear optmzaton problems. Ths method requres updatng of null-space factorzatons after a low rank modfcaton. The update procedure has the advantage that t s sgnfcantly cheaper than a re-factorzaton of the system at each new terate. Ths paper focuses on the cheap update of a rectangular LU decomposton after a rank-1 modfcaton. Two dfferent procedures for updatng the LU factorzaton are presented n detal and compared regardng ther costs of computaton and ther stablty. Moreover we wll ntroduce an extenson of these algorthms whch further mproves the computaton tme. Ths turns out to be an excellent alternatve to algorthms based on orthogonal transformatons. Key words. KKT-System, LU factorzaton, low-rank modfcaton 1. Introducton. Ths work s motvated by the soluton of the followng constraned optmzaton problem: { c (x) = 0 I mn f(x) subject to where I E =, (I E) = {1,..., k}. (1.1) x n c j (x) 0 j E If m k,.e. c j (x) = 0 for j E, actve constrants are known, optma of ths problem are locally characterzed as saddle ponts of the Lagrange functon L(x, λ) = f(x) + λ T c(x) = f(x) + m λ c (x), where λ = (λ 1,..., λ m ) T. (1.2) These saddle ponts can be computed by solvng the statonary condton =1 0 = x,λ L(x, λ) [g(x, λ), c(x) m [ f + λ c (x), c(x). =1 Ths can be done by the quas-newton method ntroduced n [7. Thereby a sequence of lnearzed KKT systems of the form [ [ [ Bk A T k sk gk = (1.3) A k 0 σ k c k has to be solved. Here the matrces A k m n and B k n n (wth n m) approxmate the Jacoban c(x) T of the actve constrants and the Hessan of the Lagrange functon 2 x L = 2 f(x) + m λ 2 c (x). The vectors s k and σ k are the current optmzaton steps from whch the =1 new pont and the new Lagrange multplers can be obtaned by x k+1 = x k + s k and λ k+1 = λ k + σ k. (1.4) Ths s a full step. The convergence can be globalzed by reduced steps x k+1 = x k + α k s k and λ k+1 = λ k + α k σ k. (1.5) ths work was supported by the DFG-research center Matheon Mathematcs for key technologes n Berln Insttute for Mathematcs, Technsche Unverstät Berln, Germany Insttute for Mathematcs, Humboldt-Unverstät zu Berln, Germany Insttute for Mathematcs, Technsche Unverstät Berln, Germany 1

2 wth a step length α k, e.g. obtaned by a lne search. Throughout the paper we make no assumptons on the sparsty pattern of c and 2 L,.e., A k, B k mght be dense matrces. The approxmate Jacoban A k and the approxmate Hessan B k wll be updated n every step of the optmzaton procedure. Here we consder rank-one updates that are lnearly nvarant and can be effcently computed by Automatc Dfferentaton [5. Wth ths technque t s feasble to compute matrx-vector and vector-matrx products wth dervatve matrces cheaply. In the sequel we wll drop the ndex k to smplfy the notaton. The updated matrces A k+1, B k+1 wll be denoted by A +, B Updatng the Hessan. The Hessan wll be modfed as shown n [7 by the symmetrcrank-one update formula (SR1): where B + = B + (w Bs)(w Bs)T (w Bs) T s B + ɛhh T, (1.6) w = B + s g(x +, λ) g(x, λ) (1.7) and s s k as n (1.3). It can be seen from (1.7) that ths update satsfes the drect secant condton. By Automatc Dfferentaton the vectors g can be evaluated n the adjont mode wthout the knowledge of the full Jacoban c(x) Updatng the Jacoban. Ths matrx can be updated smlarly to the Hessan wth the two-sded-rank-one update formula (TR1) [7: It satsfes the drect secant condton A + = A + (y As)(µT σ T A) µ T s σ T As and the adjont secant condton up to O( σ s 2 ) A + δrρ T. (1.8) A + s = y c(x + ) c(x) (1.9) σ T A + = µ T σ T c(x + ) (1.10) where σ σ k and s s k are as n (1.3). Equaton (1.10) can be computed n the reverse or adjont mode of automatc dfferentaton. Unless the constrant functon c(x) s affne the two condtons wll not be exactly consstent. But the devaton wll only be of order O( σ s 2 ) whch s wthn the scope of quas Newton methods. More detals about these two updates n ths optmzaton context are gven n [ Null-space Representaton. For the soluton of KKT systems t s necessary to solve the lnear system of equatons (1.3). One way to do ths conssts of decomposng the matrces A and B. Here and throughout the paper we wll assume that A has full rank. In contrast to [8 a complete LU factorzaton nstead of a QR decomposton of the approxmate Jacoban wll be performed. P z AP s = LU (1.11) where U = [ U 1 U 2 m m m d wth d = n m and U1 nonsngular. As n [8 the approxmate Hessan wll be projected to the null- and range-space of A T. 2 The

3 columns of [ U 1 Z = P s Z wth Z = 1 U 2 I d (1.12) form a bass of the null-space of A. As range-space bass we wll use the columns of the permuted dentty augmented by a block of zeros [ Im Y = P s Ỹ = P s 0 n m. (1.13) Combnng these spaces to Q = [ Y Z, (1.3) can be transformed by multplyng from left and rght by Q, respectvely ts transposed to E C U 1 T LT P z C T M 0 s y s z = Y T g Z T g, (1.14) Pz T LU σ c wth the notaton E = Y T BY m m, C = Y T BZ m d, M = Z T BZ d d. (1.15) Intally we assume that M s postve defnte. Ths can be acheved by startng wth dentty matrces for A and B. Factorzng the matrx M s necessary to solve system (1.14). To do so we wll use a transposed Cholesky factorzaton: M = RR T where R s an upper trangular matrx. Ths s only possble f n every step M s postve defnte. We wll ensure ths e.g. by dampng the (SR1) or (TR1) update. Further note that because of ther structures t s not necessary to store Y and Z explctly. Instead we use the mplct representaton va the LU decomposton of A. Consequently our total storage requrement s the same as that for A and B, and far less than that requred by the QR factorzaton Updatng Decompostons. Decomposng the KKT systems (1.3) n the prevous descrbed way costs O(n 3 ) operatons n every Newton step whch s qute expensve n the case of large problems. The computatonal effort can be reduced by one order to O(n 2 ) f A and B need not to be refactorzed n every step. Because of the low-rank correctons to A and B by the (SR1) and (TR1) update formulas ths s possble by updatng the factors drectly. Two dfferent procedures for updatng the LU factorzaton wll be presented n detal and compared regardng ther costs of computaton and ther stablty. Moreover we wll ntroduce an extenson of these algorthms whch wll further mprove the computaton tme. Numercal examples confrm that ths approach s an excellent alternatve to algorthms based on orthogonal transformatons. 2. LU Updatng. The LU factorzaton P z AP s = LU of a dense matrx A m n has an algebrac complexty of O(nm 2 ) operatons n general. In partcular n our applcaton where we have to solve a sequence of KKT systems the assocated sub-matrces undergo a sequence of low-rank modfcatons. Ths requres the factorzaton of A n every step. Thus t s better to factorze A only once at the begnnng of the computaton. Then the factors P z, P s, L and U can be updated drectly wth an effort of O(mn) operatons. Next, two dfferent algorthms of Bennett [1 and Schwetlck/Kelbasnsk [10 respectvely Fletcher/Matthews [2 wll be llustrated. Also a new method combnng these two algorthms s shown. In addton a new approach for the case of column permutatons wll be presented. Further we wll show how the updatng procedure can be faster by an effcent row-wse mplementaton and regardng the structure of the low rank term. Moreover t wll be shown that the algorthm by Bennett s advantageous n the symmetrc postve defnte case. The basc problem can be descrbed as follows. Let the rank-one modfcaton be A + = A + uv T, (2.1) 3

4 where u m and v n and let the decomposton P z AP s = LU (2.2) be gven and assume that A + also has full rank. We want to compute new updated factors L +, U +, P z +, P s + such that A + = (P + z ) T L + U + (P + s ) T = P T z LUP T s + uv T. (2.3) 2.1. Algorthm I - Schwetlck/Kelbasnsk. At frst we ntroduce the updatng algorthm by Schwetlck/Kelbasnsk [10. It can be used for updatng an LU factorzaton by a rank-one term only usng row pvotng. In ths method L s a lower trangular matrx wth unt dagonal. Durng the procedure the column permutaton P s s kept unchanged. In the case of rectangular matrces A ths may cause problems. After updatng t can occur that the leadng part U 1 + of U + does not have full rank. Then t s necessary to permute a column from the rear part to the front n order to restore the regularty of U 1. A new method how to fnd and permute approprate columns wll be descrbed n addton to the man algorthm. The updatng procedure wthout column pvotng (P s I) works as follows: From (2.3) one obtans A + = Pz T L(U + ũvt ) wth Pz T Lũ = u. (2.4) Usng a sequence of elementary transformatons, the vector ũ wll be reduced to a multple of the frst vector of unty. Ths wll be done by elmnatng the components of ũ step by step, startng from the last one and gong upwards. Ths procedure requres pvotng snce the entres of ũ may be small. The elmnaton process looks lke where à = (Pz T P u T ) P T 1 (P u LT u ) } {{ } L (T u U + T u ũv T ) Ũ (2.5) T u = T m 1 T m 2 T 1, P u = P m 1 P m 2 P 1. (2.6) In the smplest case we can fnd a lower trangular matrx T such that A () = P z (LT 1 )(T U + T ũv T ) (2.7) and T elmnates ũ +1. If pvotng s requred then T = T,U T,L P s used, where a) P s chosen to nterchange ũ and ũ +1, b) T,L s a lower unt trangular matrx that elmnates the + 1-th component of P ũ and fnally c) T,U turns (P LP T trangular form. T 1 1,L )T,U back to lower P T P T P T P T P T (P LP T ) (P U L a [ ( [ U a (L a T 1,L ) (T,LU a L b [ ( [ U b (L b T 1,U ) (T,U U b L 0 U 0 [ ( [ P1 T + P ũ + v T ) u a ( ) ) ( ) + T,Lu a v T ) u b + ( 0 ) ( ) + T,Uu b v T ) u 0 + ( 0 ) ( ) ) ) 4

5 Repeatng the elmnaton process fnally leads to where à = (Pz T Pu T ) P T 1 (P u LT u ) } {{ } L (T u U + T u ũ v T ) (2.8) û Ũ T u = T m 1 T m 2 T 1, P u = P m 1 P m 2 P 1, (2.9) and Ũ s upper Hessenberg, P s a permutaton matrx and L s lower unt trangular. Snce û s a multple of the frst unt vector the rank-one term ûv T can be added to Ũ wthout destroyng the Hessenberg form. Fnally, to elmnate the extra lower sub-dagonal n Ũ a second sequence of transformatons has to be done accordng to the same prncple: where A + = ( P T Pd T ) 1 (P d LT d ) (T d Ũ) (2.10) L + U + P + z T d = T 1 T 2 T m 1, P d = P 1 P 2 P m 1. (2.11) The number of operatons amounts knm for updatng an (m n) -matrx by ths algorthm. Here s 5 k 9 where the best case arses f no permutatons are requred and the worst bound s attaned n the opposte case.e. f one has to permute every row. To acheve numercal stablty pvotng s done n [10 f u < l +1, u + u +1. (2.12) Ths condton ensures that T 1,L and T 1,U always elmnate a small element by a larger one n modulus. Further ths guarantees the property that all entres n the frst lower sub-dagonal of L reman smaller than one n modulus. Hence all other elements n ths matrx can only grow by a factor of three. Certanly, ths strategy causes a large number of permutatons whereas the runtme of the algorthm s growng. For ths reason t s advsable to nclude a dampng factor τ n (2.12): u < τ l +1, u + u +1 (2.13) where 0 < τ 1. As a compromse between numercal stablty and algebrac effcency we suggest to chose τ = Column Permutatons. In the case of updatng a rectangular LU factorzaton, column permutatons n U are requred to keep the leadng part of U nonsngular. They are necessary f an element on the man dagonal n U 1 becomes very small or equals zero. Then ths column u U 1 must be nterchanged wth a sutable column u j U 2. Ths permutaton corresponds wth U + = UP j (2.14) where P j only nterchanges the columns, j and 0 < m < j n. Assume that after a rank-one modfcaton the matrx U has the followng form: U (1) = u 0 ɛ 0 0 [ U 1 U (2.15)

6 where 0 ɛ max[ dag(u 1 ). Then column u can be seen as an approxmate lnear combnaton of the columns n front of t. In ths case U 1 s almost rank defcent. To restore nonsngularty, u s moved to the rear part U 2. The frst task s to determne a sutable column u j U 2 whch can be nterchanged wth u to restore the regularty of the frst block. Therefore at frst the element ɛ must be transformed to the last dagonal poston u mm. So we have to remove column u and re-nsert t n the m -th poston. Ths corresponds to a column permutaton n U 1 : U (2) = U (1) P (1) = u 0 ɛ 0 0 (2.16) Now U (2) has become upper Hessenberg and the new lower sub-dagonal entres have to be elmnated wth the elementary transformatons ntroduced n Secton 2.1. U (3) = T U (2) = ũ u j ɛ u mj (2.17) Now the small element ɛ has been moved down n U and we can compare t wth the other elements n the last row of U 2. The column u j of U 2 assocated wth largest entry u mj n modulus wll be nterchanged wth ũ. U + = U (3) P (2) = u j (2.18) u mj ɛ ũ Now the matrx U 1 has been transformed to the desred nonsngular form. The steps (2.16) and (2.17) are only necessary to determne whch column of U 2 must be permuted. The permutaton tself can be done as the followng rank-one update drectly nterchangng u and u j U + = U + ( u u j ) ( ej e ) T. (2.19) So t s more advantageous to perform (2.16) and (2.17) on a temporary vector nstead explctly on U Algorthm II - Bennett. The algorthm by Bennett s used to update a trangular factorzaton by drectly changng the factors L and U step by step. All matrces except the permutatons are as n (2.1)-(2.3). To our knowledge ths procedure can not be combned wth pvotng. Of course, ths can cause numercal nstabltes but offers some great mprovements n runtme. Bennetts approach s qute dfferent to Algorthm I. Here the update s done recursvely based on: [ [ [ [ [ A11 A A = U11 U = L A 21 A 1 U 22 0 à 1 = 12, (2.20) I 0 à 0 I where à = A 22 L 21 U 12 represents the Schur complement. L 1 s the dentty matrx where only the elements on the frst column are replaced by correspondng elements of the frst column of L, 6 L 21

7 U 1 s defned analogously. Usng ths notaton we can update the matrx factors row by row and column by column begnnng at the top. Gven [ ( ) ( ) T 1 0 A + = A + γuv T u1 v1 = L 1 U 0 à 1 + γ (2.21) u 2 v 2 we obtan [ [ [ A + U + = 11 U + L + 21 I 0 à I (2.22) L + 1 U + 1 where U + 11 = U 11 + u 1 v 1, L + 21 = L 21U 11 + u 2 v 1 U + 11, U + 12 = U 12 + u 1 v T 2. (2.23) In addton the new Schur complement Ã+ can be represented as a remanng rank-one modfcaton of the former Schur complement n the form à + = à + γũṽt (2.24) where γ = γ U + 11, ũ = u 2 L 21 u 1, ṽ T = U 11 v T 2 v 1 U 12. (2.25) After calculatng these terms the procedure can be restarted wth the new reduced rank-one update (2.24) for Ã. The new scalar factor γ can be ncluded n one of the vectors ũ, ṽt. That way the complete updatng process can be done step by step obtanng the new factors L + and U +. The number of operatons amounts 4nm for updatng an (m n)-matrx by ths algorthm, cp. fgure (2.1). Ths s always better than Algorthm I even f there are no permutatons necessary. Furthermore ths method can be easly extended for hgher rank modfcatons. In addton t s applcable to the symmetrc postve defnte case. Moreover, Bennett s algorthm can be mplemented n a way whch allows row-wse computaton of the new elements of L and U. Ths s a sgnfcant pont n modern computatons where memory accesses needs more and more tme compared to floatng pont operatons whch become faster. Smlar to Algorthm I where the row-wse memory access can only be done f no pvotng s requred t reduces sgnfcantly the computatonal tme for updatng the matrx L. Therefore the modfcaton of L has to be delayed. Ths corresponds to the kj-varant of the Gaussan elmnaton whch s used e.g. for ncomplete LU factorzatons [11. Fgure (2.1) dsplays the new row-wse Algorthm n contrast to the standard verson of Bennett. In fgure (2.2) and fgure (2.3) the dfferences n matrx access between these two algorthms s shown. The hatchng shows whch matrx areas have been changed untl step durng the updatng procedure. Nevertheless the man problem n ths algorthm s that there are no known possbltes to combne pvotng wth the low rank update. Ths can cause numercal stablty problems durng the updatng procedure. Remark In the case of updatng the LDL T factorzaton of a symmetrc postve defnte matrx, ths method corresponds to the algorthm ntroduced n [3. It turns out that ths updatng technque s an excellent alteratve to methods usng plane rotatons. 7

8 Standard recursve LU updatng 1: for = 1 to m do 2: //dagonal update 3: U = U + u v 4: v = v /U 5: for j = + 1 to m do 6: //L update 7: u j = u j u L j 8: L j = L j + v u j 9: end for 10: for j = + 1 to n do 11: //U update 12: U j = U j + u v j 13: v j = v j v U j 14: end for 15: end for Row-wse recursve LU updatng 1: for = 1 to m do 2: for j = 1 to 1 do 3: //delayed L update 4: u = u u j L j 5: L j = L j + v j u 6: end for 7: //dagonal update 8: U = U + u v 9: v = v /U 10: for j = + 1 to n do 11: //U update 12: U j = U j + u v j 13: v j = v j v U j 14: end for 15: end for Fg Standard and Row-wse Algorthms for the Rank-One Modfcaton of the LU Factorzaton PSfrag replacements Fg Standard Bennett Algorthm PSfrag replacements Fg Row-wse Bennett Algorthm PSfrag replacements 2.4. Algorthm III - Extenson. Here a new combnaton of the prevous descrbed algorthms wll be presented. Thereby the good features from both wll be combned. Furthermore the method wll be mproved for update vectors begnnng wth a leadng part consstng of zero elements. Ths s useful n the KKT applcaton ntroduced n ths paper, where lnear constrants causes such zero entres. Ths algorthm wll be done sequentally by the followng three stages: explotng leadng zeros n u resp. v to the update startng wth Algorthm II as long as t s stable swtchng to Algorthm I n the case that pvotng s requred In each step the whole updatng problem wll be reduced as dsplayed n Fgure (2.4). U 11 U 21 L 11 Ũ 11 Ũ12 L 11 L 21 L 22 U 22 L 21 L 22 Ũ 22 ˆL Û Fg Alg.III 8

9 Step 1: If one of the update vectors has a leadng block of zero entres, only a sub-matrx of A has to be changed. We wll llustrate the nfluence on the factors n the case u T = ( ) 0 u T 2. Here only the lower part of A wll be modfed: [ [ [ A + A + = 1 A1 + 0 ρ = T L A 2 + u 2 v T = 11 U 11 L 11 U 12 L 21 U 11 + u 2 v1 T L 21 U 12 + L 22 U 22 + u 2 v2 T. (2.26) A + 2 That means that the matrx parts L 11, U 11 and U 12 reman unchanged. The other parts can be computed as follows. Gven we obtan the formula to calculate L + 21 L + 21 U + 12 = L 21U 12 + ũ 2 v T 1 (L + 21 L 21)U 12 = ũ p v T 1 L + 21 = L 21 + ũ p v T 1 U Now the remanng new sub-matrces L + 22 and U 22 + updatng problem. Ths s represented by can be computed n form of a new dense LU L + 21 U 12 + L + 22 U 22 + = L 21U 12 + L 22 U 22 + ũ 2 v2 T (L + 21 L 21)U 12 + L + 22 U 22 + = L 22U 22 + ũ 2 v2 T L + 22 U 22 + = L 22U 22 + ũ 2 (v2 T vt 1 U ). (2.27) ṽ T From (2.27) we conclude that t s suffcent to consder the reduced updatng problem for L 22 and U 22. Ths computaton can be done n an analogous way f the vector v has leadng zero entres. Step 2: Gven the remanng submatrces L 22, U 22 and the update vectors ũ 2, ṽ T, we ntally start usng Algorthm II where v 2 s replaced by ṽ. As we wll see n Secton (4), ths algorthm wll be sgnfcantly faster than Algorthm I. For reasons of effcency we wll chose the row-wse verson. Algorthm II wll be used as long as t s stable (see step 3). If we have to stop t for stablty, then L + 21 s not yet computed. Thereby L 21 wll be modfed usng L 11, Ũ11. Ths wll be done by the standard verson of Algorthm II. Step 3: In step we swtch to Algorthm I f U + τ max( U + 2 ) where (0 τ 1). (2.28) Ths condton s used to safeguard the updatng process and t avods tny pvots. Up from ths pont the remanng matrx parts ˆL and Û wll be updated by Algorthm I. Notce that row and column permutatons appled to ˆL and Û wll also be performed on L+ 21, L + 21, U 12 +, Ũ KKT Updatng. At frst we have to modfy the approxmate Jacoban A + = A + δrρ T, (3.1) where δ, r and ρ are as n Secton 1.2. These terms are computed as part of the optmzaton process usng s and σ accordng to (1.8) after solvng (1.3). Here we assume that A and A + have full rank. Ths can be guaranteed, e.g. by dampng the update (3.1). The modfcaton (3.1) can be done by one of the algorthms descrbed n Secton 2. Whenever A wll be modfed we have to adjust ts correspondng null-space Z. Due to the regularty of the approxmate Jacoban, Z s regular n every step. 9

10 3.1. Updatng the Null-Space. If the approxmate Jacoban s updated by a rank-one term, the modfcaton of the correspondng null-space to A s of rank-one, too. Ths rank correcton can be computed n a way that Z mantans ts trapezod structure. So we obtan Z + = Z + zρ T z, where z n, ρ z d. (3.2) 1 Modfcatons only occur n the matrx product Ẑ = U1 U 2 of Z. Hence only the upper part of z denoted by z m s non-zero. These vectors z and ρ z can be calculated usng the Sherman-Morrson-formula [4. The new sgnfcant null-space s gven by Ẑ + = (U 1 + ) 1 U 2 + = [ U 1 + r ρ T 1 [ 1 U2 + r ρ T 2 where ρ T = [ ρ T 1 ρ T 2 = ρ T P s and r = δl 1 Pz T r. Straghtforward computaton yelds [ [ (U2 ) + r( ρ T 2 ) Ẑ + = Ẑ U 1 1 r z ( ρ T 2 ) ( ρt 1 )U 1 1 } 1 + ( ρ T 1 1 r {{ } ρ T z )U 1 (3.3), (3.4) whch represents the null-space modfcaton as rank-one correcton correspondng to the update of A. Snce a column nterchange n A can also be read as a rank-one update, the null-space modfcaton of Ẑ can be computed analogously. In the partcular case of an addtonal column nterchange n A, t s necessary to collect the two separate rank-one updates nto a sngle rank-two update to avod numercal problems. Wth the knowledge of z and ρ z the projected Hessan can be adjusted wth respect to the null-space of A. In the followng ths wll be descrbed n detal Updatng the Projected Hessan. The projected Hessan has to be modfed whenever one of the followng three cases occurs: (a) the SR1 update (1.6): B + = B + ɛhh T (b) modfcaton of the null-space (3.2): Z+ = Z + zρ T z (c) column permutaton n A: P s + = P j P s (a) The matrces E and C are updated wth respect to the rank-one correcton of B. We obtan and E + = Y T B + Y = Y T (B + ɛhh T )Y = Y } T {{ BY } +ɛy T hh T Y (3.5) E C + = Y T B + Z = Y T (B + ɛhh T )Z = Y } T {{ BZ } +ɛy T hh T Z. (3.6) C The update of the projected null-space M = Z T BZ T can be represented by Z T B + Z = R + R T + = ZT ( B + ɛhh T ) Z = RR T + ɛh z h T z, (3.7) where h z = Z T h. As long as the rank-one update s constructed to preserve postve defnteness, t can be computed wth several algorthms for updatng the Cholesky factorzaton [3, 4. Here we have to pay attenton to choose ɛ such that M remans postve defnte. Ths can be acheved e.g. by dampng the rank-one term or by pre-adjustng the Hessan [8. (b) The changes n E and C caused by the null-space modfcaton of Z can be done n a very smlar way to case (a). In contrast to the frst case, M underles a rank-two modfcaton R + R+ T = Z+BZ T + = ( Z T + ρ z z T ) B ( Z + zρ T ) z = Z } T {{ BZ } +ρ z b T z + b z ρ T z + µρ z ρ T z, (3.8) RR T 10

11 where b z = Z T B z and µ = z T B z. Once more we have to avod to loose postve defnteness of M. Ths can be acheved usng the strateges descrbed n [8. (c) In the case of column permutatons appled to A at frst we adjust the projected Hessan smlarly to case (b) but use a rank-two term. Ths rank-two correcton conssts of the modfcatons n Ẑ caused by the rank-one update of A (1.8) and by the column permutaton n U (2.19), compare Secton 3.1. Furthermore, the two rows n P s whch were nterchanged, cause addtonal changes n E, C and M. As before the row nterchange n the null-space P s + Z + = P j P s Z+ can be read as a rank-one update whch leads us to case (b) agan. At last the permutaton P s occurrng n the range-space bass Y (1.13) has to be regarded, too. Ths requres to remove and re-compute some sngle rows, resp. sngle columns n E and C. We wll show ths for the matrx C. Startng wth C = Ỹ T Ps T BP s Z we obtan C (b) = Ỹ T Ps T BP s + Z + after a low rank correcton from the rght. Our objectve s to compute C + = Ỹ (P T s )+ BP + s Z + = Ỹ P T s P T j BP + s Z +. (3.9) Suppose that C (b) s gven, then C + can be obtaned from C (b) by replacng the -th row c T c T j,.e. c T 1. C (b) = c T. c T m c T 1. = C + = c T j. c T m by, (3.10) where 1 m < j n and c T j can be computed as c T j = e T j P s T BZ. To do so, we need the sngle row [e T j P s T B of B. Snce B s not explctly stored we have to use ts representaton [ Y T Z T B [ Y Z [ E C = C T RR T [ [ Im 0 E C = B = P s ẐT I d C T RR T [ Im Ẑ 0 I d P T s. (3.11) 4. Numercal Results. In ths secton we wll compare the algorthms descrbed prevously regardng ther runtme. The tests were done on an Athlon-XP machne wth 256kB CPU cache and 512 MB man memory. We mplemented the algorthms n C usng the operatng system Lnux and the gcc compler wth the opton -o3. At frst we wll compare the computaton tme of several low-rank update algorthms for rectangular matrces. After that we wll solve a KKT problem arsng from constraned optmzaton by the quas Newton approach of Secton Rectangular LU Updatng. The updatng algorthms Alg.I, Alg.II and the QRupdatng algorthm of [10 wll be compared n runtme. In our computaton we started wth the dentty and appled our algorthms to 50 randomly generated rank-one modfcatons. The computaton tmes n Table 4.1 are gven n seconds. Alg.I represents the updatng method of Schwetlck. It s used n four dfferent varatons. The frst two cases use τ = 0 whch means that no row permutaton wll be done. These two varants dffer n the way L s updated. The thrd verson (τ = 1) represents the orgnal algorthm descrbed n [10. Verson 4 of Schwetlck s method uses the relaxed parameter τ = 0.1 whch typcally reduces the number of row nterchanges. Alg.II shows the method of Bennett whch s appled n ts orgnal form [1 and n the new row-wse mplementaton (Fgure 2.1). The performance of the QR-updatng algorthm s shown n the column QR. 11

12 Table 4.1 Comparng LU Updatng dmenson Alg.I Alg.I Alg.I Alg.I Alg.II Alg.II QR m n τ = 0 τ = 0, row-wse τ = 1 τ = 0.1 row-wse As we can see, the row-wse verson of Alg.II s always the fastest. Especally n the cases of large matrx dmensons t s sgnfcantly faster than all other methods. Of course t offers no possbltes to prevent numercal nstabltes by pvotng. Among all algorthms that address stablty t has turned out that Alg.I wth τ = 0.1 performs best. It s approx. 30% faster than n the case where τ = 1 and t s approx. 40% faster than the QR update. For these experments based on random updates we dd not use column pvotng snce we observed that column nterchanges were hardly necessary for ths class of problems. For ths reason Alg.III whch combnes Alg.I and Alg.II wll be dscussed n the next example. In a further example we wll update a rectangular matrx 50 tmes by structured rank-one modfcatons. That means we want to use a sequence of rank-one correctons n the way as we expect n the optmzaton procedure. Startng wth vectors whch have a large norm leads to a lot of pvotng at the begnnng. Step by step we reduce ths vector norm so that no permutatons wll be requred fnally. We wll compare Alg.III to the QR-based method, to Alg.I usng full pvotng and to the fast Bennett method. In Table 4.2 the runtme for the total updatng process s pre- Table 4.2 Comparng Structured Updatng QR-based LU-based n m Alg.I Alg.II Alg.III sented. We can see that all LU-based algorthms, Alg.I, Alg.II (row-wse) and Alg.III are faster than the QR-verson. Furthermore t turns out that only Alg.III can take an advantage of the specal structure of the updatng sequence KKT solvng. In ths secton we wll compare methods for solvng a whole KKT problem. On one hand QR-based algorthms for factorzng and updatng A k together wth Gvenstechnques for modfyng B k are consdered. On the other hand the new LU-methods from Alg.I and Alg.III appled to A k are used n combnaton wth the algorthm of [3 for B k. We use an optmzaton envronment whch s provded by Andrea Walther 1. Ths code uses a globalzng approach based on lne search. For computng the requred dervatves the AD-tool ADOL-C [6 s used. We wll solve the followng optmzaton problem [9: 1 Insttute for Scentfc Computng, D Dresden, Germany 12

13 Mnmze f n 1 ( ) 2 f(x) = x + x +1 (4.1) =1 subject to c x + 2x x +2 1 = 0 (1 n 2). (4.2) We ntalze x as a random vector where ( 0.5 x 0.5). The dervatve matrces n the frst step are chosen as A 0 = [ I 0 and B 0 = I. In fact of the quadratc-lnear structure of ths problem, the optmal soluton should be acheved after n teratons. As we can see n Table 4.3 the LU-based method s sgnfcantly faster than the Table 4.3 KKT Example n LU-based steps QR-based steps QR-based verson. Further, the number of teratons are close to the expected theoretcal value. 5. Conclusons. We have ntroduced several new effcent algorthms for updatng LUfactorzatons after low-rank modfcatons. The algorthm of Kelbasnsk/Schwetlck was extended for rectangular matrces usng addtonal column pvotng wthout loosng ts quadratc complexty. Ths s a central pont for applyng LU decompostons to non-square systems. Moreover we mproved the algorthm of Bennett regardng to effcent matrx access n memory. In addton a new method for specal structured updates was developed. These new algorthms can sgnfcantly mprove dfferent applcatons. Ths s shown n the case of solvng nonlnear constrant optmzaton problems. Numercal results prove that our method s much faster than, e.g. usng a QR-based verson. So far we have not dscussed the case when B k turns to be ndefnte. Currently the updates are suffcently damped. The generalzaton to the ndefnte case wll be dscussed n an upcomng paper. Our promsng numercal results ndcate that our low-rank-modfcaton algorthms can also be used effcently n a wde feld of applcatons. 13

14 REFERENCES [1 J. Bennett, Trangular factors of modfed matrces, Numersche Mathematk, 7 (1965), pp [2 R. Fletcher and S. Matthews, Stable modfcaton of explct lu factors for smplex updates, Mathematcal Programmng, 30 (1984), pp [3 R. Fletcher and M. Powell, On the modfcaton of ldl t factorzatons, Mathematcs of Computaton, 28 (1974), pp [4 G. Golub and C. van Loan, Matrx Computatons, The Johns Hopkns Unversty Press, second ed., [5 A. Grewank, Evaluatng Dervatves, Prncples and Technques of Algorthmc Dfferentaton, Number 19 n Fronters n Appl. Math. SIAM, [6 A. Grewank, D. Juedes, and J. Utke, Adol-c, a package for the automatc dfferentaton of algorthms wrtten n c/c++, ACM Trans. Math. Software 22, (1996), pp [7 A. Grewank and A. Walther, On constraned optmzaton by adjont based quas-newton methods, Optmzaton Methods and Software, 17 (2002), pp [8 A. Grewank, A. Walther, and M. Korzec, Mantanng factorzed kkt systems subject to rank-one updates of hessans and jacobans, (2005). [9 W. Hock and K. Schttkowsk, Test examples for nonlnear programmng codes, Lectures Notes n Economcs and Mathematcal Systems, (1987). [10 A. Kelbasnsk and H. Schwetlck, Numersche lneare Algebra, Verlag Harr Deutsch, [11 Y. Saad, Iteratve Methods for Sparse Lnear Systems, PWS Publshng, Boston,