A Ferris-Mangasarian Technique. Applied to Linear Least Squares. Problems. J. E. Dennis, Trond Steihaug. May Rice University

Transcription

1 A Ferrs-Mangasaran Technque Appled to Lnear Least Squares Problems J. E. Denns, Trond Stehaug CRPC-TR98740 May 1998 Center for Research on Parallel Computaton Rce Unversty 6100 South Man Street CRPC - MS 41 Houston, TX Submtted May 1998

2 A Ferrs-Mangasaran Technque Appled to Lnear Least Squares Problems J. E. Denns Computatonal and Appled Mathematcs Rce Unversty Houston TX Trond Stehaug y Department of Informatcs Unversty of Bergen Hyteknologsenteret N-5020 Bergen Norway May 1, 1998 Abstract Ths note specalzes to lnear least squares problems an approach suggested by Ferrs and Mangasaran [4] for solvng constraned optmzaton problems on parallel computers. It wll be shown here that ths specalzaton leads to an algorthm whch s mathematcally equvalent to an acceleraton and convergence forcng modcaton of the block Jacob teraton appled to the normal equatons. The resultng algorthm s a promsng way to speed up a parallel multsplttng algorthm of Renaut [9] for lnear least squares. Renaut's algorthm s related to a specalzaton of part of the Ferrs and Mangasaran approach. Research supported by DOE FG03-93ER25178, CRPC CCR , AFOSR- F , The Boeng Company, and the REDI Foundaton. y Research supported by The Research Councl of Norway and VISTA { a research cooperaton between the Norwegan Academy of Scence and Den norske stats oljeselskap a.s (Statol). 1

3 1 Introducton Ths note specalzes to lnear least squares problems an approach suggested by Ferrs and Mangasaran [4] for solvng constraned optmzaton problems on parallel computers. It wll be shown here that ths specalzaton leads to an acceleraton and convergence forcng mechansm for the block Jacob teraton appled to the normal equatons. We do not form the full normal equatons, but our numercal results hnt that the condton number of the normal equatons aects the number of teratons requred for a gven problem. The target problems are assumed to be large. The technque suggested has for each teraton two basc stages both of whch nvolve the soluton of smaller lnear least squares problems. The rst stage s to partton the optmzaton varables for the problem and then on separate processors to solve the smaller least squares problem n whch only those varables n a sngle partton are allowed to move. The parttonng of the varables s at the dscreton of the user, and hence t can be used to select the sze of the problem to be solved on each processor. The user may make other consderatons n parttonng the varables such as consstent scalng of the parttoned problems. It s well known that the teraton dened by ncrementng each varable by the amount ndcated n ths rst stage and then updatng the resdual for the next teraton s the block Jacob teraton appled to the normal equatons for the orgnal lnear least squares problem. See Bjorck [1] and the references theren. The classcal Jacob teraton alone may not converge [5, 8]. The second stage of each teraton s to compute a new terate by a synchronzaton step that nvolves solvng a least squares problem n a smaller space of surrogate varables dented by the rst stage. If the surrogate varables are taken only to be the ncrements produced by solvng the subspace least squares problems, then we wll call ths the Jacob method wth subspace correcton, and t always converges for full rank problems. It turns out that ths method has already been consdered by Renaut [9]. She calls t Optmal Recombnaton Least Squares Multsplttng (ORLSMS), and she gves further developments based on the multsplttng method of O'Leary and Whte [7]. For smplcty, we wll restrct ourselves here to the case that at each of the two stages we accurately solve the smaller mnmzaton problems, but ths 2

4 s nether necessary to the theory nor even possble n the general settng consdered n [4]. Mangasaran [6] and Ferrs and Mangasaran [4] ntroduced n stage one auxlary varables, whch they called "forget-me-not" varables. The contrbuton of ths paper s to show how to use the \forget-me-not" varables to make some very promsng reductons n the number of teratons needed f only the ORLSMS or the Jacob method wth subspace correcton s used. Our results hnt that perhaps the choce we advocate for least squares may be useful back n the general mnmzaton settng consdered n [4]. If the sze of the problem and the number of processors ndcate, the problem n the surrogate varables can n turn be attacked by the parttonng technque of the rst stage, and ths can be contnued to reduce the dmenson of the problem to be solved n the synchronzaton step untl the number of surrogate varables s manageable. In the next secton, we present some prelmnares to set the stage for the new Jacob-Ferrs-Mangasaran algorthm n Secton 3. Secton 4 presents some promsng numercal results, and Secton 5 s devoted to a dscusson and conclusons concernng ths approach. 2 Prelmnares 2.1 The lnear least squares problem Let A be an m n real matrx, m n; b 2 R m. Let M be an m m postve dente weghtng matrx. The weghted lnear least-squares problem s: mn x2r kax? b k M; where kyk 2 n M = y T My: (1) Let the columns of A be parttoned nto g blocks A = [A 1 A 2 : : : A g ]; where A s m n. Further let x be parttoned consstently nto blocks x 1 ; x 2 ; : : : ; x g. The least squares problem (1) s equvalent to gx mn x2r nfk A x? bk M : x 2 R n ; = 1; : : : ; gg: (2) =1 3

5 We want to dstrbute the varables to the avalable processors and solve a smaller subproblem on each processor n parallel. 2.2 Varable dstrbuton To solve the weghted lnear least squares problem (2) we dstrbute the varables among the avalable processors. In ths secton we wll assume that each group s assgned to ts own processor. Let x k be an approxmaton to the soluton x to (1), and partton x k nto x k 1; x k 2; : : : ; x k g. Parallelzaton: = 1; 2; : : : ; g Solve for x k+1 2 R n : mn x2r n ka x? (b? gx 1=j6= A j x k j )k M : (3) Followng the notaton and dervaton n [3], we ntroduce the drecton d k =? x k, and note that successve resduals satsfy x k+1 r k+1 = gx j=1 A j x k+1 j? b = r k + Then the th least squares subproblem (3) s: Solve for d k gx j=1 A j d k j : 2 R n : mn d2r n fka d + r k k M g; = 1; 2; : : : ; g; (4) and the th block of the new approxmate soluton s x k+1 = x k + d k ; = 1; 2; : : : ; g: (5) For d 2 R n ntroduce the vector d 2 R n whch s obtaned by startng wth a zero vector and placng the nonzero entres of d n the postons correspondng to the column ndces n A of A. Dene the drecton d k : d k = gx j=1 d k j : 4

6 Then (5) can be wrtten as x k+1 = x k + d k. Ths s [1] the classcal block Jacob method on the normal equatons A T MAx = A T Mb: (6) Assume that A has full rank. Then the followng result says that the block Jacob method converges f A T MA s sucently `block dagonally domnant'. Theorem 1 Let A have full rank, and let C be a block dagonal matrx wth th block A T MA. The correspondng block Jacob method wll converge to x, a soluton of (1) f 2 C? A T MA s postve dente. Proof: Corollary 2.1 of [5]. Even when 2 C? A T MA s not postve dente, we can force convergence by the followng two small modcatons of the block Jacob method. Let f : R n! R be dened by f(x) = kax? bk 2 M = x T A T MAx? 2(A T Mb) T x? b T Mb : (7) We can force convergence by ntroducng a smple lnesearch:. k = argmn f(x k + d k ) : (8) Theorem 2 Let A have full rank. Gven x k, choose x k+1 = x k + k d k ; where k s dened by (8). Then lm k!1 x k = x. Proof: Chapter 6 of [2]. Of course, ths lnesearch functons as a synchronzaton step, and so the attractve parallelsm n the Jacob teraton s compromsed. Note that k s the easy soluton of the 1-dmensonal least squares problem to solve for k n k(ad k ) k + r k k M. We wll ntroduce a more general lnesearch n the next secton. Fnally, we end ths secton wth a smple convergence result that follows from the applcaton of a degenerate form of the Ferrs-Mangasaran 2nd stage. 5

7 Theorem 3 Let A have full rank. Gven x k, choose x k+1 = argmnff(x k + d k ); f(x k + d k ); = 1; : : : ; gg ; (9) then lm k!1 x k = x, where x s the unque soluton of (1). Proof: Snce A s full rank, f s strongly convex and the result follows from Theorem 2.3 of [4] or Theorem 5 of the next secton. 3 The Ferrs-Mangasaran Correcton Step In the last secton, we saw that the Jacob teraton converges when the block cross terms n the coecent matrx A T MA are weak enough to be neglected. Ths secton wll ntroduce smple technques for ncorporatng the nuence of these cross terms nto the teraton. Unfortunately, these all wll take the form of a synchronzaton step and so parallelsm wll be compromsed. Mangasaran [6] and Ferrs and Mangasaran [4] ntroduced a synchronzaton step n whch the step x k = x k+1?x k s chosen by approxmately mnmzng f(x k + x) n the subspace spanned by d = 1; : : : ; g. In the followng, we call ths a subspace-correcton step. Thus, the block Jacob teraton can be seen as choosng adaptvely a sngle surrogate varable d k to represent the subspace spanned by the th block of varables x k n the correcton step. The subspace-correcton teraton step s then chosen to be the step that provdes approxmately the most decrease from x k for f n the space of Jacob-surrogate varables. Ths sort of dmensonal reducton s common n engneerng desgn through so-called surrogate varable or reduced bass technques. The derence here s that the surrogate varables are beng chosen adaptvely by the Jacob teraton rather than to be chosen a pror by engneerng judgement. 3.1 Supplementary varables The subspaces spanned by the column blocks A can be supplemented by what Ferrs and Mangasaran call \forget-me-not" varables. For our settng, a more approprate name would be \look-ahead" varables. Thus, we wll use the more neutral desgnaton \supplementary varables". 6

8 Begnnng wth a sngle full space vector, the procedures of the prevous secton are used to obtan supplementary varables to expand each subspace. Unfortunately, ths requres us to ntroduce stll more complcated notaton, whch we wll gve now. Then we wll dscuss strateges for choosng supplementary varables that we have found to be so advantageous as to justfy the added fuss. Let I be the n n dentty matrx and let I be the nn matrx formed from columns of the nn dentty matrx so that A I = A. Let the supplementary vector p 2 R n be parttoned accordngly and dene the n(g?1+n ) matrx P P = p 1 p?1 I p +1 p g : (10) For ~n = n + g? 1, dene the m ~n matrx ea = AP = [Ap 1 Ap?1 A Ap +1 Ap g ] : (11) For a gven supplementary vector p k 2 R n the g subproblems (3) are replaced by Solve for e d k 2 R ~n : mnfk A ek d e + r k k M g (12) d e where e A k s dened n (11) for the gven vector p k. The step d k 2 R n s d k = gx =1 P k e d k (13) Of course, the ncluson of p n the algorthmc mx rases the queston of how to choose an deal p for the teraton. That queston turns out to have a smple answer, whch we gve n the followng theorem and then follow wth some algorthmc modcatons amed at approxmatng the deal p. Theorem 4 Let x k 2 R n be arbtrary, and set p k = e k = x? x k. Then, each P e d k = e k, and x = x k + 1 g dk. Proof: To smplfy notaton, we wll consder the case = 1. Let v (e kt 1 ; 1; 1; ; 1) T 2 R ~n 1. Frst we wll show that d ek 1 = v solves (12). 7

9 Notce that ea k 1 = AP 1 = [A 1 A 2 e k 2 A g e k g] ; and P 1 v = e k. Thus, ea k 1v + r k = AP 1 v + r k = Ae k + r k = Ax? b ; and f d ek 1 s any other soluton, then t must gve the same resdual. Thus, by the uncty of x, s unque. P 1 e d k 1 = P 1 v = e k So, f we could choose p k = x? x k e k, then each P d ek would be e k. Of course, f we knew e k, we would be nshed, but ths ponts to takng p k to be our best estmate of e k. The best way we have thought to do ths at a partcular teraton s by takng p k = x k? x k?1, and even ths crude approxmaton to e k leads to a sgncant reducton n teratons. However, a more elaborate scheme s reasonable because f p k does not depend on k, and f the subproblems are solved usng a Cholesky factorzaton of the n n matrx A et M A e, then the Cholesky factors are saved and solvng the subproblems requre only a back substtuton (forward and backward substtuton). Ths suggests that we mght protably explot the lnear algebra savngs to try a predctor/corrector scheme dened by keepng p = p k?1 xed for several predctor teratons to obtan say x pred wthout havng to redo any factorzatons. The sole purpose of these predctor teratons s to obtan a better approxmaton x pred? x k e k to use as p k n a corrector teraton to obtan x k+1. We wll gve numercal results supportng ths procedure. 3.2 The complete algorthm At ths pont, we have obtaned the full set of supplementary varables from the block Jacob subproblems supplemented by the projectons of p. To nsh specalzng the Ferrs-Mangasaran technque to lnear least squares, we wll 8

10 explan the subspace-correcton step, and then we wll gve the complete algorthm. For a gven d 2 R n, dene the n g matrx Consder the m g matrx D = d1 d g : (14) ba = AD = A d 1 A d g : Then the columns of b A are the full set of surrogate varables. We solve the least squares problem n ths set of varables to get the subspace corrected step, whch s gven by Solve for s k 2 R g : mn k b Ask + r k k M : (15) We use the vector d k dened by (13), and the new terate s x k+1 = x k +D k s k where D k s dened n (14). Before we gve the Ferrs-Mangasaran convergence theorem for the more general nonlnear optmzaton algorthm, we pause to sum up all the specalzatons we have suggested n the followng Algorthm: Jacob-Ferrs-Mangasaran Subdvde A nto g blocks. Choose x 0. Compute r 0 = Ax 0? b: for k = 0 step 1 untl convergence do Choose vector p k. Ths may nvolve several predctor teratons for = 1; : : : ; g n parallel Compute P k n (10). Let e A k = AP k Solve for e d k Compute d k = P g =1 P k : mnfk A e e k d k + r k e k M g. d k. Compute D k n (14) and b Ak = AD k. Solve for s k : mnk b Ak s k + r k k M. x k+1 = x k + D k s k : r k+1 = r k + b Ak s k : Check for convergence. 9

11 Algorthm: Predctor teratons Let A e = AP k?1 and P = P k?1 for = 1; 2; : : : ; g. Let z 0 = x k? x k?1 ; v 0 = r k + Az 0. for j = 0; 1; : : : ; l? 1 do for = 1; : : : ; g n parallel Solve for d ej : mnfk e P Ad ej + vj k M g. Compute d j g = =1 P e d j. Compute D j and b Aj = AD j. Solve for s j : mnk b Aj s j + v j k M. z j+1 = z j + D j s j : v j+1 = v j + b Aj s j : Let p k = z l The followng result follows from Ferrs and Mangasaran [4] Theorem 2.3 by notcng that f A has full rank then the functon f n (7) s strongly convex. Theorem 5 Assume that fp k g s bounded ndependent of k. If A has full rank, then lm k!1 x k = x, where x s the unque soluton of (1). 4 Numercal results The convergence of the methods wll be llustrated on a class of randomly generated least squares problems (1). The m n coecent matrx A = QD + "R where Q s m n wth orthonormal columns, D s a n n dagonal matrx and R s a m n matrx. The elements n R and on the dagonal of D are randomly dstrbuted. For small values of " the matrx A T A s dagonally domnated. The elements n the m vector b n the least squares problem (1) are ether random (for 'non-zero' resdual case problems) or the vector s chosen to be b = Ac where c s a random n vector for 'zero resdual' problems. The weght matrx M s the dentty matrx. All tests are run on a SPARC wth Sun-4 oatng-pont usng Matlab. In Table 1, varatons of the Ferrs and Mangasaran technque are compared to the Gauss-Sedel teraton and the Jacob teraton wth the subspacecorrected step on the normal equatons (6). For ths problem D 0, " = 1, 10

12 and the elements n R are unformly dstrbuted n [-1,1]. Note that ths problem gets more dagonally domnant as m ncreases. For the partcular case reported here m = 280, n = 256. The condton number (the square of the rato of largest and smallest sngular values of A) s The block Jacob method does not converge wthout a lnesearch or subspace correcton for the reported values of g. The stoppng crteron s that the `2 derence between the exact and approxmate solutons s not more than 10?6. In all numercal experments the columns of the matrx are partoned nto g groups based on the natural order: the 1st n=g columns form the rst group, the 2nd n=g columns form the next group, etc. The startng pont s x 0 = 0. g Gauss- Jacob wth sub- Jacob-Ferrs-Mangasaran Sedel space correcton p=1 F&M p=ds Pred Zero resdual: >20000 > Non zero resdual: >20000 > Table 1: Number of teratons The columns n Table 1 gve the number of teratons to acheve the desred accuracy. The Guass-Sedel method s appled to the normal equatons wthout formng the normal equatons (see for example [1, 3]). For the Jacob method we use the subspace corrected step (15) to guarantee convergence. The column marked "p=1" are the teratons usng the supplementary varables dened by p k = (1; : : : ; 1) T 2 R n. Ferrs and Mangasaran [4] suggest usng the vector p k to be [p k] 1 j = [(rf(x k + e )? rf(x k )) ] j 1 = [A TMA for j = 1; 2; : : : ; n ; = 1; 2; : : : ; g (16) e ] j 11

13 where e 2 R n s a vector wth all ones and [ ] j denotes the jth component of a n vector. Note that ths p k does not depend on the teraton ndex k. The column marked "F&M" are the teratons usng the supplementary varables dened by (16). All tests ndcate very lttle derence between the Ferrs and Mangasaran choce of p k and p k = (1; : : : ; 1) T. The number of teratons for the algorthm that uses supplementary varables dened by choosng p k = x k?x k?1 = D k s k s n the column marked "p=ds". In the column marked "Pred" the supplementary varables are determned usng one predctor step to compute the new p k. All tests ndcate very small varatons between zero and non-zero resdual cases. Ths small derence ndcates that the governng condton number for the methods s the condton number of the normal equaton (6). For p k = x k? x k?1 the number of teratons does not ncrease wth the number of groups g on most problems. However, for the method that uses one predctor step, we see that the number of teratons n some cases ncreases wth the number of groups. Ths s nvestgated further n Table 2. Here we have chosen " = 1, the dagonal elements n D are unformly dstrbuted n [1,2] and the elements of R are n the nterval [0,1]. Further m = 26 and n = 24. g p=ds Predctor teratons l l = 1 l = 2 l = 3 l = Table 2: Number of teratons and predctor teratons A predctor teraton has the same cost n terms of arthmetc operatons as one teraton of the algorthm wth p k ndependent of the teraton ndex. If the cost of computng the QR factorzatons of the smaller systems s neglected, then usng l predctor teratons has the same cost as l+1 teratons usng "p=ds". If we consder g = 8 n Table 2 we see that t s more ecent to use 2 or 3 predctor teratons than use p k = x k? x k?1. If we use l = 2 12

14 predctor teratons and compare wth the results n Table 1 for "Pred" (l = 1) the number of teratons decreases from 2955 to 306 for the zero resdual case and g = 32. For the non zero resdual case the number of teratons decreased from 3000 to 444 when l = 2. 5 Conclusons Ths paper rases more questons than t answers, and we hope to pursue some of these questons soon. We beleve that we have found a valuable choce of the \forget-me-not" varables of Ferrs and Mangasaran, and the behavor of the predctor/corrector farly cres out for an adaptve way to decde when how many predctor teratons to do. At ths pont, we can only say that more groups means more predctor teratons. It would be also nterestng to know whether our choce would be useful n the general nonlnear optmzaton case. We suspect t would. These questons wll have to wat n order that we can make the deadlne to have our paper consdered for the ssue to honor Olv Mangasaran on hs 65th brthday. We jon all of Olv's frends, not just contrbuters to ths volume, n wshng Olv and Clare many more happy and healthy years. References [1] A. Bjorck, Numercal methods for least squares problems, SIAM, [2] J. E. Denns, Jr., Robert B. Schnabel, Numercal methods for unconstraned optmzaton and nonlnear equatons, Prentce-Hall, [3] J. E. Denns, Jr., and T. Stehaug, On the successve projectons approach to least squares problems, SIAM J. Numer. Anal. 23 (1986) [4] M. C. Ferrs and O. L. Mangasaran, Parallel varable dstrbuton, SIAM J. Optmzaton 4(1994) [5] H. B. Keller, On the soluton of sngular and semdente lnear systems by teraton, SIAM J. Numer.Anal. 2(1965)

15 [6] O. L. Mangasaran, Parallel gradent dstrbuton n unconstraned optmzaton, SIAM J. Control and Optmzaton 33(1995) [7] D. P. O'Leary and R. E. Whte, Multsplttng of matrces and parallel solutons of lnear systems, SIAM J. Algebrac Dscrete Methods, 6(1985) [8] J. M. Ortega, Introducton to parallel and vector soluton of lnear systems, Plenum Press, [9] R. A. Renaut, A parallel multsplttng soluton of the least squares problem, Numercal Lnear Algebra wth Applcatons, 4(1997)