ISSN 2066-6594 Ann. Acad. Rom. Sc. Ser. Math. Appl. Vol. 8, No. 2/2016 On a drect solver for lnear least squares problems Constantn Popa Abstract The Null Space (NS) algorthm s a drect solver for lnear systems of equatons. It was ntally desgned and theoretcally analyzed by M. Benz n 1993 for square nonsngular systems, and ts man dea conssts on projectons of an ntal set of vectors onto the hyperplanes assocated to the system equatons, by usng projectons parallel wth some specfc drectons whch are constructed durng the development of the algorthm. In ths paper we ext and theoretcally analyze the NS algorthm to lnear least squares problems. MSC: 65F05, 65F20 keywords: consstent lnear systems; Null Space algorthm; lnear least squares problems; normal equatons; augmented system 1 Introducton For A : m n and b IR m we wll consder the system of lnear equatons Ax = b. (1) If b s n the range of A,.e. at least one z IR n exsts such that Az = b, we say that (1) s consstent and denote by S(A; b) the set of all ts solutons Accepted for publcaton n revsed form on November 1-st 2016 cpopa1956@gmal.com; cpopa@unv-ovdus.ro; Ovdus Unversty of Constanta, Romana 145
146 C. Popa and by x LS the mnmal norm one. If Az b, z IR n we say that (1) s nconsstent and reformulate t as a lnear least-squares problem: fnd x IR n such that Ax b = nf{ Ax b, x IR n }. (2) Let LSS(A; b) be the set of all ts solutons and x LS the (unque) soluton of mnmal norm. A frst equvalent consstent formulaton of the problem (2) s gven by the assocated normal equaton (see e.g. [8]). A T Ax = A T b. (3) Unfortunately, n order to use ths equaton, the computaton of the product A T A s requred, and ths s a very expensve procedure for both computatonal tme and memory aspects. Fortunately, t exsts a more convenent consstent equvalent formulaton of (3) through the augmented system,.e. [ I A A T 0 ] [ r x ] = [ b 0 ]. (4) The equvalence of the problems (2) and (4) s shown n Proposton 1.2 from [7]. In [2] (see also [3]) the authors ntroduce and theoretcally analyze the Null Space (NS) algorthm, as a drect method for numercal soluton of square nonsngular systems of lnear equatons. Accordng to these aspects, the paper s organzed as follows: n secton 2 we brefly descrbe the orgnal NS algorthm from [2] - [3]. In secton 3 we ext and theoretcally analyze the algorthm NS to general rectangular, but consstent systems of lnear equatons and we pont out how ths extenson can solve also nconsstent lnear systems (lnear least squares problems), through the equvalent formulatons (3) - (4). 2 The Null Space algorthm In ths secton we wll brefly descrbe the orgnal Null Space (NS) algorthm, ntroduced n [2] (see also [3]) for square nonsngular systems of lnear equatons of the form Ax = b, (5) wth x = A 1 b ts unque soluton, where A : n n and b IR n. We shall denote by,, the Eucldean scalar product and norm, and by S = {x IR n, A, x = 0}, H = {x IR n, A, x = b } = S + b A 2 A, (6)
A drect solver for LLS problems 147 the vector subspace and hyperplane, respectvely assocated to the -th equaton of the system (5), where A, = 1,..., n are the rows of A. For d IR n such that d, A 0 and any x IR n there exsts the projectons of x parallel wth d onto S and H (also called drectonal projectons ), defned by P d S (x) = x x, A d, A d, P H d (x) = x x, A b d. (7) d, A For any 1 k n, let A (k) : k n be the submatrx of A formed wth ts frst k rows,.e. A T 1 A (k) = A T 2..., (8) A T k and N k the null space of A (k). The followng relatons are obvous and N k = S 1 S k, {0} = N n N n 1 N 2 N 1 (9) dm(n k ) = n k, k = 1,..., n. (10) In [2] (see also [3]) the authors consder a set of null vectors such that.e. N = {z 2, z 3,..., z n } IR n (11) z k 0, z k N k 1, z k / N k, k = 2,..., n, (12) z k, A j = 0, j = 1,..., k 1 and z k, A k 0. (13) If such a set of null vectors (11)-(13) s avalable, we can compute x from (5) by the followng drect solver (DS). Algorthm DS. Step 0 (Intalzaton): any vector x H 1 Step 1 (Successve projectons): for = 2 : n x = PH z (x) = x x, A b z, A z
148 C. Popa Remark 1. We shall denote by e 1, e 2,..., e n the canoncal bass n IR n. Then, a possble choce for x n the above Step 0 can be (see [3]) We reobtan the followng result from [2]. x = b 1 A 1k0 e k 0, where A 1k0 0. (14) Proposton 1. The algorthm DS gves us the unque soluton x from (5). Proof. We frst observe that from (12)-(13) t results that z k S 1 S 2 S k 1, (15) k = 2,..., n. The statement of the proposton s equvalent wth the followng property x k H 1 H 2 H k, (16) 1 k n, whch we wll prove by a recursve argument followng k. For k = 1 we get that x 1 H 1 accordng to the constructon n Step 0 of the algorthm DS. Then, let n 1 k 2 be such that (16) holds for t. By usng Step 1 of DS we get x k+1 = PH zk+1 k+1 (x k ) = x k xk, A k+1 b k+1 z k+1 z k+1 H k+1. (17), A k+1 From (16) (.e x k H j, j = 1,..., k), (15) (.e. z k+1 S j, j = 1,..., k), the last equalty n (6) and (17) (.e. x k+1 = x k + α k z k+1 (α k IR s the scalar from (17)) we get x k+1 H j, j = 1,..., k, whch together wth (17) completes the recurson argument and the proof. In order to construct the above set of null vectors, the authors proposed n [3] the followng Null Vectors (NV) algorthm. Algorthm NV. Step 1 (Intalzaton): any bass Z 1 = {z (1) 2,..., z(1) n } n N 1 Step 2 Because dm(n 2 ) = n 2 we must have A 2, z (1) 0, for some {2, 3,..., n 1, n}, so we can permute the elements of Z 1 such that A 2, z (1) 2 0. Moreover, for the numercal stablty of the algorthm we may choose 0 < A 2, z (1) 2 = max A 2, z (1) 2 n. (18)
A drect solver for LLS problems 149 Then, we produce the new set Z 2 = {z (2) 3,..., z(2) n } by usng drectonal projectons z (2) j = P z(1) 2 H 2 (z (1) j ) = z (1) j z(1) j, A 2 z (1) 2, A 2 z(1) 2, j = 3,..., n. (19) Step n Because dm(n n 1 ) = 1 we must have A n 1, z (n 2) 0, for some {n 1, n}, so we can permute the elements of Z n 2 = {z (n 2) n 1, z(n 2) n } such that A n 1, z (n 2) n 1 = 0. Moreover, for the numercal stablty of the algorthm we may choose. 0 < A n 1, z (n 2) n 1 = max A n 1, z (n 2) n 1 n. (20) Then, we produce the last set Z n 1 = {z n (n 1) } by z (n 1) n = P z(n 2) n 1 H n 1 (z n (n 2) ) = z n (n 2) (n 2) n, A n 1 z (n 2) n 1, A n 1 z(n 2) n 1. (21) Step n+1 The set Z of null vectors s gven by a last drectonal projecton Z = {z (1) 2, z(2) 3,..., z(n 1) n }. (22) Remark 2. Because we start wth a bass Z 1 n N 1, t can be proved that the vectors from Z 2,..., Z n 1 are also bass n N 2,..., N n 1, respectvely (for the proof see [3]). Remark 3. Followng the deas from [3], a bass Z 1 = {z (1) 2,..., z(1) n } n N 1 can be constructed as ndcated n the followng Matlab code. Gven A: n x n; I=eye(n);z=zeros(n); [A1k,k]=max(abs(A(1,:)));z(:,1)=I(:,k); f (k==1) for =1:n-1 z(:,+1)=i(:,+1)-(a(1,+1)/a(1,k))*i(:,k); else
150 C. Popa for =1:k-1 z(:,+1)=i(:,)-(a(1,)/a(1,k))*i(:,k); for =k:n-1 z(:,+1)=i(:,+1)-(a(1,+1)/a(1,k))*i(:,k); We shall call the above NV algorthm together wth the soluton part DS the Null Space algorthm (NS). It can be wrtten as follows. Algorthm NS. Step 1 (Intalzaton): any bass Z 1 = {z 2,..., z n } n N 1 Step 2 for =2:n-1 maxm = max j n A, z j = A, z j. p() = A, z j f (j > ) nterchange the vectors z and z j for j=+1:n z j = PH z (z j ) = z j zj,a p() Step 3 (Intalzaton): any vector x H 1 Step 4 (Successve projectons): for = 2 : n z x = PH z (x) = x x, A b p() z 3 The generalzed Null Space algorthm For A a general m n matrx and b R(A) IR m we wll consder the consstent lnear system of equatons Ax = b. (23)
A drect solver for LLS problems 151 Wthout restrctng the generalty of (23) we wll suppose that the rows A and columns A j of A satsfy the assumptons A 0, = 1,..., m, A j 0, j = 1,..., n. (24) We propose for the numercal soluton of the system (23) the followng extenson of the algorthm Null Space from the prevous secton. Algorthm General Null Space (GNS) Step 1 (Intalzaton): any bass Z 1 = {z 2,..., z n } n N 1 ; τ = 2 Step 2 for =2:m f (maxm 0) maxm = max τ j n A, z j = A, z j. p() = A, z j f (j > τ) nterchange the vectors z τ and z j for j=τ+1:n z j = PH zτ (z j ) = z j zj,a p() z τ τ = τ + 1 else p() = 0 Step 3 (Intalzaton): any vector x H 1 ; set τ = 2 Step 4 for =2:m f (p() 0) x = PH zτ (x) = x x,a b p() τ = τ + 1 z τ We shall analyse n what follows the propertes of the algorthm GDPM. Let r = rank(a) 1 and 1 = 1 < 2 <... r m (25) ndces of a set of lnearly ndepent rows A 1,..., A r,.e.
152 C. Popa for [ 1 + 1, 2 1], A deps lnearly on A 1 for [ 2 + 1, 3 1], A deps lnearly on A 1, A 2........................... for [ r 1 + 1, r 1], A deps lnearly on A 1,... A r 1 for [ r + 1, m], A deps lnearly on A 1,... A r We shall use n what follows the notaton z q (k) for the vectors generated durng the algorthm GNS (as n NS, secton 2). The evoluton of these vectors durng the algorthm s as follows. for [ 1 + 1, 2 1], {z (1) 2,..., z(1) n } A 1 = A 1 for [ 2, 3 1], {z (2) 3,..., z(2) n } A 1, A 2........................... for [ r 1, r 1], {z (r 1) r,..., z n (r 1) } A 1, A 2,..., A r 1 for [ r, m], {z (r) r+1,..., z(r) n } A 1, A 2,..., A r {z (r) r+1,..., z(r) n } A 1, A 2,..., A m {z (r) r+1,..., z(r) n } N (A) (26) Moreover, t s clear from the algorthm GNS that and p() 0 { 1, 2,..., r } (27) z (k) k+1 A 1,..., A k, k = 1,..., r 1. (28) From all the above consderatons we derve the followng result for GNS. Proposton 2. () The vectors {z (r) r+1,..., z(r) n } form a bass n N (A). () In the steps 3 and 4 of GNS we obtan a soluton of (23) (whch generally dffers from x LS.) Proof. () From (25) we get that dm(n (A)) = n r. As n [3] we obtan that, durng the constructon from the algorthm GNS, the vectors zk+1 k,..., zk n are lnearly ndepent, k = 1,..., r. In partcular, the n r vectors n the last obtaned set {z (r) r+1,..., z(r) n } wll form a bass N (A). () Accordng to (25) the consstent problem (23) s equvalent wth A k, x = b k, k = 1,..., r. (29)
A drect solver for LLS problems 153 Moreover, from all the above notatons and consderatons, t results that Steps 3 and 4 n the algorthm GNS can be (formally) wrtten as Step 3-1 (Intalzaton): any vector x H 1 Step 4-1 for k=2:r = k x = P zk 1 k H (x) = x x,a b p() z k 1 k By usng (29), the defnton of H n (6), and the arguments n [3] page 1162, we obtan that the resultng vector x n the Step 4-1 wll satsfy.e., accordng to (29) x S(A; b). x H k, k = 1,..., r, (30) Remark 4. Accordng to the equvalent consstent formulatons (3)-(4), we can drectly apply the GNS algorthm to them. But, an unpleasant aspect related to the applcaton of the GNS algorthm to (3) conssts on the presence of the product A T A as the system matrx. A row n ths matrx s of the form A T A whch determnes a (too) bg computatonal effort n the steps 2-4 of the algorthm GNS. Accordng to the second equvalent formulaton, f we denote by B the system matrx from (4), ts row B s gven by [ e T, A T ], 1 m B = [ (A m ) T, 0 ] (31), m + 1 m + n Ths tell us that the computatonal effort n steps 2-4 of GNS are comparable to NS. A detaled analyss of these aspects, together wth an effcent mplementaton of GNS, and systematc numercal experments and comparsons wth other algorthms, wll be the subject of a future paper. Fnal comments. Other consderatons essentally related to the computaton of a bass for the null space of a gven matrx A can be found n papers [1, 5, 6, 9, 10]. They use n ths respect QR or SVD decompostons of
154 C. Popa the matrx A, but they are not concerned wth a soluton of a gven system or least squares problem. Acknowledgements. The author thanks to the anonymous referees for ther valuable comments. References [1] M. Arol, L. Baldn: A backward error analyss of a null space algorthm n sparse quadratc programmng, Sam J. Matrx Anal. Appl., 23(2)(2001), 425 442. [2] M. Benz: A drect row-projecton method for sparse lnear systems. Ph.D. Thess, Department of Mathematcs, North Carolna State Unversty, Ralegh, NC, 1993. [3] M. Benz M., C. Mayer: A drect projecton method for sparse lnear systems, SIAM J. Sc. Comput., 16(5)(1995), 1159 1176. [4] M. Benz, C. Mayer, M. Tuma: A sparse approxmate nverse precondtoner for the conjugate gradent method, SIAM J. Sc. Comput., 17(5)(1996), 1135 1149. [5] Anthony J. Cox, Ncholas J. Hgham: Accuracy and stablty of the null space method for solvng the equalty constraned least squares problem, BIT, vol. 439(1)(1999)4, 34 50. [6] X. Lu, Y. Yuan: A null-space prmal-dual nteror-pont algorthm for nonlnear optmzaton wth nce convergence propertes, Math. Program., Ser. A 125(2010), 163 193. [7] A. Ncola, C. Popa: Kaczmarz Exted versus augmented system soluton n mage reconstructon, ROMAI Journal, 6(1)(2010), 145 152. [8] C. Popa: Projecton algorthms - classcal results and developments. Applcatons to mage reconstructon, Lambert Academc Publshng - AV Akademkerverlag GmbH & Co. KG, Saarbrücken, Germany, 2012. [9] B. Recht, W. Xu, B. Hassb: Null space condtons and thresholds for rank mnmzaton, Math. Program, Ser. B 127(2011), 175202. [10] A. Sharma et al: Null space based feature selecton method for gene expresson data, Int. J. Mach. Learn. & Cyber., 3(2012), 269276.