THE CORE PROBLEM WITHIN A LINEAR APPROXIMATION PROBLEM AX B WITH MULTIPLE RIGHT-HAND SIDES. 1. Introduction. Consider a linear approximation problem

Transcription

1 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B WIH MULIPLE RIGH-HAND SIDES IVEA HNĚYNKOVÁ, MARIN PLEŠINGER, AND ZDENĚK SRAKOŠ Abstract his paper focuses on total least squares (LS) problems AX B with multiple right-hand sides Existence and uniqueness of a LS solution for such problems was analyzed in the paper I Hnětynková, M Plešinger, D M Sima, Z Strakoš, and S Van Huffel (211) For LS problems with single right-hand sides the paper C C Paige and Z Strakoš (26) showed how necessary and sufficient information for solving Ax b can be revealed from the original data through the so-called core problem concept In this paper we present a theoretical study extending this concept to problems with multiple right-hand sides he data reduction we present here is based on the singular value decomposition of the system matrix A We show minimality of the reduced problem; in this sense the situation is analogous to the single right-hand side case Some other properties of the core problem, however, can not be extended to the case of multiple right-hand sides Key words total least squares problem, multiple right-hand sides, core problem, linear approximation problem, error-in-variables modeling, orthogonal regression, singular value decomposition AMS subject classifications 15A6, 15A18, 15A21, 15A24, 65F2, 65F25 1 Introduction Consider a linear approximation problem Id AX B, or, equivalently, B A, (11) X where A R m n, X R n d, B R m d, without any further assumption on the positive integers m, n, d, anda B (this eliminates the trivial case where it does not make sense to approximate B by a linear combination of the columns of A; see also14) he equivalent B A form in(11) has been chosen so that our transformations will take it to block form (as for example in (21) belowforthed =1 case) revealing the core problem most simply We will focus on incompatible problems, ie R(B) R(A) If R(B) R(A), then the system AX = B can be solved using standard methods Consider changes of the coordinate systems in R m, R n,andr d represented by orthogonal transformations  X (P AQ)(Q XR) (P BR) B, (12) where P 1 = P, Q 1 = Q, R 1 = R ; or, equivalently, ( )( B  Id R R P X Id B A Q Q X ) R (13) his work has been supported by the GAČR grant No P21/ S he research of M Plešinger has been partly supported by the ESF grants No CZ17/23/9155 and No CZ17/23/365 he research of Z Strakoš has been partly supported by the GAČR grant 21/9/917 Charles University in Prague, Faculty of Mathematics and Physics, Czech Republic, and Institute of Computer Science, AS CR ( hnetynkova@cscascz) echnical University of Liberec, Department of Mathematics, Czech Republic, and Institute of Computer Science, AS CR ( martinplesinger@tulcz) Charles University in Prague, Faculty of Mathematics and Physics, Czech Republic ( strakos@karlinmffcunicz) 1

2 2 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ We require that X solves (11) if and only if X = Q XR solves (12) andcallsuch problems orthogonally invariant he total least squares problem (LS) min X,E,G G E F subject to (A + E)X = B + G (14) serves as an important example; see 12, 13, 7, 5, section 6, 18 Mathematically equivalent problems have been independently investigated under the names orthogonal regression and errors-in-variables modeling; see2, 21 In 8 it is shown that even with d = 1 (which gives Ax b, where b is an m-vector) the LS problem may not have a solution and, when the solution exists, it may not be unique; see also 6, pp In order to resolve this difficulty, the classical book 19 introduces the so-called nongeneric solution his book also extends the LS theory to problems with multiple right-hand sides, ie for d>1 he existence and uniquenessofalssolutionwithd>1isthendiscussedinfull generality in the recent paper 1, giving a new classification of all possible cases he sequence of papers 12, 13, and 14 by C C Paige and Z Strakoš investigates, using a unified framework, different least squares formulations for problems with d = 1 he last paper 14 introduced the so-called core problem that separates the necessary and sufficient information for solving the problem from the rest It gives the necessary and sufficient condition for existence of the LS solution, explains when the LS solution exists and when it is unique, and clarifies the meaning of the nongeneric solution For a brief summary see also the recent paper 1, pp , which also shows that there is a class of problems for which the classical LS approach described in 19, 22, 23, 9, Chapter 63, pp (in particular the so-called classical LS algorithm; see19, Chapter 361, pp 87 9, 1, Algorithm 1, p 767) is unable to find the existing LS solutions he first steps in generalizing the core problem theory for d>1weredoneby Å Björck in the series of talks 1, 2, 3, and also in the unpublished manuscript 4, and by D M Sima and S Van Huffel in 16, 17 In a theoretical study presented in this paper we further develop the data reduction suggested in 15 which gives the core problem, we investigate its properties and prove its minimality We do not advocate a computational technique for solving the problem his is a matter of further work and results will be published elsewhere he organization of this paper is as follows Section 2 recalls the core problem concept for a single right-hand side Section 3 describes the data reduction for multiple right-hand sides, shows how to assemble the transformation matrices, and discusses basic properties of the reduced problem Section 4 proves the minimality of the reduced problem, and thus justifies the definition of the core problem Section 5 concludes the paper hroughout the paper, R(M) andn (M) denote the range and null space of a matrix M, respectively; I l (or just I) denotes an l l identity matrix; and l,ξ (or just ) denotes an l ξ zero matrix he matrices A, B, B A, and X from (11) are called the system matrix, theright-hand sides (or the observation) matrix, the extended (or data) matrix, andthematrix of unknowns, respectively 2 Data reduction in the single right-hand side case Consider the linear approximation problem (11) withd = 1 In 14 itwasshownthatthereexist

3 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 3 orthogonal matrices P, Q, that transform the original problem into the block form P b1 A b A Q Q = 11 1 x x A 1, (21) 22 x 2 where b 1 and A 11 are of minimal dimensions Such transformation can be obtained using the singular value decomposition (SVD) of the system matrix A, A = UΣV, U R m m, Σ R m n, V R n n, (22) where U 1 = U, V 1 = V LetA have k distinct nonzero singular values σ 1 >σ 2 >>σ k >, (23) and let their multiplicities be m j, j =1,,k; k j=1 m j = r rank(a) hen Σ = diag(σ 1 I m1,,σ k I mk, m r,n r ) (24) Consider the partitioning U =U 1,,U k,u k+1, U j R m mj, j =1,,k,and U k+1 R m m k+1,wherem k+1 m r is the dimension of the null space N (A ) Columns of U j represent an orthonormal basis of the jth left singular vector subspaces of A hen U 1 b A = U b AV f Σ, (25) V where f U b =f 1,,f k,f k+1, f j U j b, j =1,,k+1, and ϕ j f j (26) Note that ϕ j =ifandonlyifb is orthogonal to the jth left singular vector subspace In order to be conformal with the multiple right-hand sides case, we keep (unlike in 14) the zero and nonzero components f j together until the last permutation Let S j R mj mj, S 1 j = Sj, be a Householder reflection matrix such that and let S j f j = e 1 ϕ j, e 1 =1,,, R mj, j =1,,k+1, (27) S =diag(s 1,,S k ), S L = diag(s,s k+1 ), S R =diag(s,i n r ) Note that S L ΣS R = Σ he orthogonal transformation (US L ) b A(VS R ) = S L f ΣS R =S L f Σ maximizes the number of zero entries in the right-hand side vector S L f =ϕ 1e 1,,ϕ ke 1,ϕ k+1e 1

4 4 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ (as mentioned above, we may have ϕ j =forsomej) Let b have nonzero components f jl in n left singular vector subspaces corresponding to nonzero singular values σ j with indices j 1,,j n,1 n k he component f k+1 (the component of b in the null space of A ) is nonzero due to the fact that the problem is incompatible Consider the row permutation Π L of the matrix SL f Σ such that Π LSL f =b 1, ϕ j1,,ϕ jn,ϕ k+1,,,, ie, all entries of b 1 =ϕ j1,,ϕ jn,ϕ k+1 = f j1,, f jn, f k+1 R n+1 (28) are positive hen there exists a column permutation Π R of the matrix Π L Σsuch that Π A11 LΣΠ R =, A 22 where the block A 11 R (n+1) n is (with b R(A)) rectangular, containing at most one copy of each nonzero singular value σ j on its diagonal and having the zero last row All the other singular values are moved to the diagonal of the second block A 22, which can be of any shape, or nonexistent Summarizing, we obtain P b1 A b AQ = 11, P US A L Π L, Q VS R Π R, 22 where b 1 A 11 anda 11 are of minimal dimensions; see 14 he corresponding transformation and conformal splitting of vector of unknowns is Q x =(VS R Π R ) x1 x = x 2 he subproblem b 1 A 11, or A 11 x 1 b 1, contains the necessary and sufficient information for solving the original problem Ax b, and it is called the core problem he solution of the second subproblem A 22 x 2 with the maximally dimensioned block A 22 R (m n 1) (n n) canbeconsideredtobex 2 = (see the discussion in 14) giving x1 x1 x = Q = VS x R Π R 2 (29) he core problem (in a different form) can also be revealed by the Golub-Kahan bidiagonalization; see 14, Section3and11 For d = 1 the core problem has a unique LS solution, see 14 Using (29), the core problem defines the minimum norm LS solution if it exists, or the minimum norm nongeneric solution of (11); see 14 and1, section 31, pp Computationally (assuming exact arithmetic) the solution (29) is for d = 1 therefore identical to the output of the classical LS algorithm given in 19, Chapter 361, pp 87 9; see also 1, Algorithm 1, p 767 Unlike in the classical LS algorithm, the solution (29) is constructed in a straightforward way by separating the LSmeaningful part A 11 x 1 b 1 of the problem Ax b from the redundant and irrelevant part represented by A 22 x 2, x 2 = In the rest of this paper it will be shown how to generalize the SVD-based data reduction and obtain the core problem for d>1, ie, for the problem AX B

5 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 5 3 Data reduction in the multiple right-hand side case Consider the problem (11) withd>1 In this section, we construct orthogonal matrices P, Q, R, that transform the original data matrix B A into the block form P R B A =P Q BR P B1 A AQ = 11, (31) A 22 where B 1 and A 11 are of minimal dimensions (the proof of minimality will be given in section 4) he orthogonal transformation (31) is done in four successive steps: preprocessing of the right-hand side B (section 31), transformation of the system matrix A (SVD of A) (section 32), transformation of the right-hand side B (section 33), and final permutation (section 34) 31 Preprocessing of the right-hand side Let d rank(b) min{m, d} Consider the SVD of B in the form B = SΘR, S R m d, Θ R d d, R R d d, (32) m d d d d = d where S has mutually orthonormal columns, ie S S = I d,θisoffull row rank, and R is square, ie R 1 = R (Note that the schema illustrates the case m>d>d; other cases can be illustrated analogously) We will see later that this R plays the role of the transformation matrix R in (31) If d<d,thenb contains linearly dependent columns representing redundant information that can be removed from the original problem (11) Multiplication of (11) from the right by R gives A(XR) BR, (33) where BR = SΘ C, R m d, C R m d, and (34) XR Y,Y R n d, Y R n d ; if d = d, thenbr = C, XR = Y With this notation I Id d BR A =C, A I XR d d =AY C AY (35) Y Y he original problem (11) is in this way split into two subproblems AY C and AY, (36) where the second problem is homogeneous Following the arguments in 14, we consider the meaningful solution Y In this way, the approximation problem (11) reduces to Id AY C, or, equivalently, C A, (37) Y where A R m n, Y R n d,andc R m d is of full column rank From(32) (34) it follows that the right-hand side matrix C has mutually orthogonal columns

6 6 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ 32 ransformation of the system matrix Consider the SVD of A given by (22) with(23) and(24) he problem (37) can then be transformed analogously to (25), ( U AV )( V Y ) =ΣZ F, (38) where Z = V Y, F = U CEquivalently, Id F Σ (39) Z he approximation problem ΣZ F has the full column rank right-hand side matrix F with mutually orthogonal columns and the system matrix Σ in a diagonal form 33 ransformation of the right-hand side Similarly to the single righthand side case, we now transform the right-hand side matrix of (38) inordertoget as many zero rows as possible Consider a partitioning of F into the block-rows with respect to the multiplicities of the singular values of the system matrix A, ie F =F 1,,F k,f k+1, where F j R mj d, j =1,,k,k+1 Let r j rank(f j ) min{m j, d} Consider the SVD of F j in the form F j = S j Θ j W j, S j R mj mj, Θ j R mj rj, W j R d rj, (31) m j = d m j r j d r j where S j is square, ie Sj 1 = Sj,Θ j is of full column rank, andw j has mutually orthonormal columns, ie Wj W j = I rj, j =1,,k,k + 1 (Note that, as above, the schema illustrates the case r j <m j < d) he matrix S j generalizes the role of the identically denoted matrix in section 2; see(27) Consider the block diagonal orthogonal matrices S =diag(s 1,,S k ), S L = diag(s,s k+1 ), S R =diag(s,i n r ), (311) where as in (24) r =rank(a), and recall that Σ=diag(σ 1 I m1,,σ k I mk, m r,n r ) R m n, σ 1 >σ 2 >>σ k >, see (23), (24) hen ΣZ F from (38) can be transformed to ( S L ΣS R )( S R Z ) S L F and S L ΣS R =Σ (312) Equivalently, (39) becomes S L F Σ I d SR Z, and the extended (data) matrix has the form Θ 1 W1 σ 1 I m1 S L F Σ = Θ k Wk σ k I mk Rm (n+d) (313) Θ k+1 Wk+1

7 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 7 If m j >r j, then the block S j F j =Θ j W j contains zero rows at the bottom, see (31) Denote Θ j W j Φj, Φ j R rj d, (314) where Φ j is the block of nonzero rows (if r j =, then the block Φ j has no rows) For r j = m j we simply have Θ j Wj Φ j It follows from (31) thatφ j has mutually orthogonal rows hematrixφ j generalizes the role of the number ϕ j ;see(26), (28) 34 Final permutation Now the aim is to find a permutation of (313) that reveals the block diagonal structure (31) his can be done analogously to the single right-hand side case (see also 14, Section 2), by moving the rows of (313) with zero blocks in Θ j Wj (see (314)) to the bottom submatrix of the whole matrix, see the matrix in the middle row of (315), with subsequent moving of the corresponding columns with the diagonal blocks σ j I mj r j in the bottom to the right, Π L Θ 1 W1 σ 1 I m1 Θ k Wk σ k I mk Θ k+1 Wk+1 Id Π R Φ 1 σ 1 I r1 Φ k σ k I rk = Φ k+1 σ 1 I m1 r 1 σ k I mk r k B1 A 11 (315) A 22 Here Π L R m m is given by Ir 1 I m1 r 1 Π L Ir k I mk r k Ir k+1 I mk+1 r k+1 (316) and it permutes the block-rows starting with Φ j up, while moving the block-rows starting with zero blocks down Analogously, Π R R n n given by Π R I m1 r 1 Ir k I mk r k I n r Ir 1 (317)

8 8 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ rearranges the block-columns of the system matrix Note that if for some j we have r j = m j, then the block I mj r j and the corresponding block-rows and block-columns vanish; if r j =, then the block I rj and the corresponding block-rows and blockcolumns vanish Let us briefly summarize the whole transformation 35 Summary of the transformation Using the SVD B = SΘR, defined in (32), the original approximation problem (11) is transformed to AX B, A R m n, X R n d, B R m d, AY C, A R m n, Y R n d, C R m d, with the full column rank right-hand side hen using the SVD A = UΣV in (22) (24), the problem is further transformed to defined ΣZ F, Σ R m n, Z R n d, F R m d, with the diagonal system matrix Σ Using singular value decompositions F j = S j Θ j W j of block-rows of F,see(31), and the orthogonal matrix S given by (311), the righthand side matrix gets the structure with the full row rank block-rows Φ j and the zero block-rows, while the diagonal system matrix Σ stays unchanged Finally, the permutation matrices Π L,Π R given by (316) and(317) are used to collect the full row rank blocks, and to transform the system matrix to the block diagonal form with two diagonal (in general rectangular) blocks, see (315) We give a quick summary of the mathematical transformations, each preceded by the relevant equation numbers: (11), (32): AX B = SΘR =C, R, { (U AV )(V XR) U BR, (22), (34) (38): Σ(V XR) U C, = F,, { (S L U AV S R )(SR (31) (314): V XR) SL U BR, Σ(SR V XR) SL F,, (315) (317): (Π L S L U AV S R Π R )(Π R S R V XR) Π L S L U BR, (Π L ΣΠ R)(Π R S R V XR) Π L S L F,, diag(a 11,A 22 )(Π R S R V XR) diag(b 1, ), so that the full transformations of A, X, andb can be summarized as (P AQ)(Q XR) P BR, P US L Π L, Q VS R Π R (318) Clearly P 1 = P, Q 1 = Q, R 1 = R,and R P B A Q B1 A = 11 A 22 }{{} d }{{} d d }{{} n } m } m m }{{} n n (319)

9 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 9 is of the form (31), where from (315) Φ 1 σ 1 I r1 B 1 A 11 Φ k σ k I rk Φ k+1 Rm (n+d), (32) and m k+1 j=1 r j, n k j=1 r j, d rank(b) he block A 22 has the form A 22 diag(σ 1 I m1 r 1,,σ k I mk r k, m r rk+1,n r) (321) hus the original problem AX B is transformed into the block form A11 (Q XR ) B1, A 22 compare with (12) Using a conformal partitioning of the matrix of unknowns Q X1 X 1 XR = } n X 2 X 2 } n n }{{}}{{} d d d, (322) where Q X1 X 2 = Y, Q X 1 X 2 = Y, and Y,Y = XR, see (34), the block diagonal structure of the system matrix and the right-hand sides matrix allows us to split the original problem AX B into (generally four) subproblems A 11 X 1 B 1, and A 22 X 2, A 11 X 1, A 22X 2 he last three subproblems are homogenous and we consider, following the arguments in 14, X 2, X 1, X 2 Only the subproblem Id A 11 X 1 B 1, or, equivalently, B 1 A 11, (323) X 1 where A 11 R m n, X 1 R n d, B 1 R m d, has to be solved If its solution is X 1, the solution X of the original problem AX B is then X1 X Q R (324) he dimensions of the reduced problem satisfy max{n, d} m n + r k+1 n + d Note that d can be smaller than, equal to, or even largerthan n From the construction we immediately have the following properties: (CP1) he matrix A 11 R m n is of full column rank equal to n m (CP2) he matrix B 1 R m d is of full column rank equal to d m (CP3) he matrices Φ j R rj d are of full row rank equal to r j d, j =1,,k+1

10 1 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ Remark 31 Note that instead of the SVD preprocessing (32) of the right-hand side B (section 31), one can use an LQ decomposition (producing a matrix with m d lower triangular block in the column echelon form and an orthogonal matrix), or another decomposition giving a full column rank matrix multiplied by an orthogonal matrix from the right Similarly, instead of the singular value decompositions (31) of F j (section 33) one can use QR decomposition(s) (producing a matrix with an r j d upper triangular block in the row echelon form and an orthogonal matrix), or another decomposition(s) giving a full row rank matrix multiplied by an orthogonal matrix from the left Such modifications lead, in general, to a different subproblem B 1 Ã11 with the same dimension as (32) and satisfying (CP1) (CP3) Moreover, there exist orthogonal matrices P 1 = P, Q 1 = Q, R 1 = R such that P B 1 A 11 R Q = B 1 Ã11 (325) Properties (CP1) (CP3) are invariant with respect to any orthogonal transformation of the form (325) 4 he Core Problem Let B 1 A 11 be the subproblem of the given problem B A obtained by the transformation (318) (319) he subproblem B 1 A 11, B 1 R m d, A 11 R m n, has the properties (CP1) (CP3) Consider now arbitrary orthogonal transformations of the form in (12) of the original problem analogous to (318) (319) such that P B A R Q B1 Â = 11, (41) Â 22 where P 1 = P, Q 1 = Q, R 1 = R,and B 1 R bm bd, Â11 R bm bn Substituting for B A from(319) gives (P P ) B1 A 11 R R B1 Â A 22 Q = 11 (42) Q Â 22 We will show that the subproblem B 1 A 11 on the left side of (42) has minimal dimensions over all possible subproblems B 1 Â11 on the right-hand side; ie d d, m m, andn n In analogy with the single right-hand side case, this justifies calling B 1 A 11 acore problem within B A 41 Proof of minimality he proof consists of five successive steps presented, for an easy orientation, in separate subsections he first gives d d he second step describes the structure of the orthogonal matrix R R he proof of the inequality m m is then based on the projections of B to the left singular subspaces of A he fourth step describes the structure of the orthogonal matrix P P which is needed for finalizing the proof by showing n n 411 Number of columns of the reduced observation matrix In (41) B 1 R bm bd,and P B R = B1 has rank( B 1 )=rank(b) =d, so that d d

11 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B Structure of the R R matrix Consider the partitioning R R R = 11 R 12 R 21 R 22, R 11 R d bd, R 22 R (d d) (d d) b hen (42) gives (P P ) B1 (R R) =(P P ) B1 R 11 B 1 R 12 } {{ } d } {{ } d d Because B 1 is of full column rank, then R 12 = Consequently R R = R 11 R 21 R 22 B1 = }{{} d }{{} d d, (43) and therefore R 11 has d orthonormal rows 413 Number of rows of the reduced data matrix he number m = k+1 j=1 r j of rows of B 1 A 11 is the sum of dimensions of intersections of the range of B with the individual left singular subspaces of A, and these dimensions are invariant with respect to the orthogonal transformation of the form (318) (319); see also (32) he desired inequality m m will be shown by comparing three different forms of the SVD of the system matrix A Recall that A = UΣV (see (22) (24)) is the standard SVD of A Now consider the singular value decompositions  11 = Û1 Σ 1 V 1,  22 = Û2 Σ 2 V 2, with square orthogonal matrices Û1, Û2, V 1,and V 2, and diagonal matrices Σ 1, Σ 2 hen, using (41), ( ) ( Û1 Σ1 A = P Û 2 Σ2 Q V1 V2 ) (44) represents another SVD of the system matrix A Furthermore, A 11, A 22 in (32), (321) are diagonal matrices he transformation (318) (319) gives A11 A = P Q, where P = US A L Π L, (45) 22 which also represents the SVD of A Because the singular values (and their multiplicities) of A are unique, for some permutation matrices Π L, Π R (analogous to Π L,Π R in (316), (317)) Σ= Π L Σ1 Σ2 Π A11 R =Π L A 22 Π R he left singular vector subspaces of A can be expressed using (44), (45) andsome orthogonal matrices Ŝ L = diag(ŝ1,,ŝk, Ŝk+1), Ŝ j R mj mj, j =1,,k,k+1, (46)

12 12 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ as U = P Û1 Û 2 Π LŜ L = P Π L S L (47) Using (41) and(319), the projections of B in the left singular subspaces of A are U B = ŜL Π L Û 1 B1 R = S L Π L B1 R, and, with (43), Û 1 B 1 = Π LŜ L S LΠ L B1 R 11 (48) he matrix R 11 R d bd, d d, has linearly independent rows; see (43) and section 411 Now every row of B 1 is nonzero (see (32) and (CP3)), so any row of B 1 multiplied by R 11 on the right must also be nonzero, and the matrix B 1R 11 has m nonzero rows he matrix Π L permutes rows (see (316)), and matrices S L, ŜL (see (311), (46)) are orthogonal block diagonal matrices such that (see (315), (314)), } Ŝ1 S Φ1 R 11 1 m 1 Ŝ L S LΠ L B1 R 11 = } Ŝk S Φk R 11 k } Ŝk+1 S Φk+1 R 11 k+1 m k m k+1, (49) where each block of m j rows corresponds to one singular value σ j (of the matrix Σ) with the multiplicity m j Because Φ j R 11 has all r j rows nonzero (since the rows of Φ j are linearly independent, the rows of Φ j R 11 are, in fact, also linearly independent and therefore each Φ j R 11 is of full row rank) and Ŝ j S j are orthogonal matrices, the block matrix on the right of (49) has at least m nonzero rows, see (32) he permutation matrix Π L then moves all the nonzero rows (and possibly also some zero rows) of (49) tothetopblockû 1 B 1, R bm d of the leftmost matrix in (48) hus Û 1 B 1 has at least m nonzero rows (and possibly some zero rows) Because Û1 is square, B 1 R bm bd has at least m rows, so that m m 414 Structure of the P P matrix Consider the partitioning of the permutation matrices Π L in (316) withm = k+1 j=1 r j as in (32), and Π L in (48) where Û 1 is m m, Π L =Π 1, Π 2, Π 1 R m m, ΠL = Π 1, Π 2, Π1 R m bm hen (48) can be rewritten as Û 1 B1 Π = 1 ŜL S LΠ 1 Π 1 ŜL S LΠ 2 Π 2 Ŝ L S LΠ 1 Π 2 ŜL S LΠ 2 B1 R 11,

13 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 13 yielding the condition ( Π 2 Ŝ L S L Π 1 )(B 1 R 11) = (41) Using the structure of the matrices Π L, S (see (311), (316)) and analogously for Π L and Ŝ (see section 413) Π 2 Ŝ L S L Π 1 =,I m1 br1 Ŝ 1 S 1 Ir 1,I mk brk Ŝ k S k Ir k,i mk+1 brk+1 Ŝ k+1 S k+1 Ir k+1 and thus with (32) the condition (41) is split into k +1blockparts ( ),I mj br Ŝ j j S Irj j (Φ j R 11 )=, j =1,,k,k+1 Because Φ j R 11 are of full row rank, this gives for all indices j,i mj br Ŝ j j S Irj j =, and thus also Π 2 ŜL S LΠ 1 = Using (47) we finally obtain, P P =Π L SL ŜL Π Û1 L Û 2 Π = 1 SL ŜL Π 1 Û1 Π 2 SL ŜL Π 1 Û1 Π 2 SL ŜL Π 2 Û2 P 11 } m P 21 P 22 } m m (411) }{{}}{{} m m m and therefore P 11 has m orthonormal rows 415 Number of columns of the reduced system matrix he relation (42) gives, using the structure of P P in (411), P 11 A11 P 21 P 22 (Q Â11 Q) = A 22 Â 22 hus (P 11 ) A 11, (P 21 ) A 22 (Q Q) = Â 11, Because (P 11 ) has orthonormal columns and Q Q is a square orthogonal matrix, A 11 R m n in (32) hasrankn, andâ11 R bm bn (see (41)), n =rank(a 11 )=rank((p 11 ) A 11 ) rank(â11, ) = rank(â11) n completing our proof

14 14 I HNĚYNKOVÁ, M PLEŠINGER, Z SRAKOŠ 5 Summary and concluding remarks We formulate the minimality property as a theorem heorem 51 (Minimality) Consider the problem AX B (11) anditssub- problem A 11 X 1 B 1 (323), where A 11 R m n and B 1 R m d are obtained by the transformation (319) described in section 3 he subproblem A 11 X 1 B 1 has minimal dimensions over all subproblems Â11 X 1 B 1, Â 11 R bm bn, B1 R bm d b obtained by an orthogonal transformation of the form (41), ie d d, m m, and n n his motivates the following definition of the core problem within (11) Definition 52 (Core problem) he subproblem A 11 X 1 B 1 is a core problem within the approximation problem AX B if B 1 A 11 is minimally dimensioned and A 22 maximally dimensioned subject to (31), ie subject to the orthogonal transformations of the form P B A R Q =P BR P B1 A AQ 11 A 22 he core problem obtained by the data reduction based on the SVD described in section 3 iscalledthecoreprobleminthesvdform his extends the core problem definition within the single right-hand side problems formulated by C C Paige and Z Strakoš in14 he data reduction presented here is based on the singular value decomposition of the system matrix A he original paper 14 presents two ways of determining the core problem he second is based on Golub-Kahan iterative bidiagonalization his can also be generalized to problems with multiple right-hand sides It leads to a band algorithm which was for this purpose proposed by Å Björck; see 1, 2, 3, and also the unpublished manuscript 4 he core problem concept is a useful tool in understanding the LS problems, which was the original motivation in 14 he data reduction and core problem based on the generalization of the Golub-Kahan iterative bidiagonalization is under investigation he results will be presented in the near future Acknowledgments he authors thank Diana M Sima for valuable discussions about the LS problems he numerous comments and suggestions of two anonymous referees led to significant improvements of the text REFERENCES 1 Å Björck, Bidiagonal decomposition and least squares, Presentation, Canberra (25) 2, A Band-Lanczos generalization of bidiagonal decomposition, Presentation, Conference in Honor of G Dahlquist, Stockholm (26) 3, A band-lanczos algorithm for least squares and total least squares problems, In: book of abstracts of 4th otal Least Squares and Errors-in-Variables Modeling Workshop, Leuven, Katholieke Universiteit Leuven (26), pp , Block bidiagonal decomposition and least squares problems with multiple right-hand sides, unpublished manuscript 5 G H Golub, Some modified matrix eigenvalue problems, SIAM Review, 15 (1973), pp G H Golub, A Hoffman, and G W Stewart, A generalization of the Eckart-Young- Mirsky matrix approximation theorem, Linear Algebra Appl, 88/89 (1987), pp G H Golub and C Reinsch, Singular value decomposition and least squares solutions, Numerische Mathematik, 14 (197), pp 43 42

15 HE CORE PROBLEM WIHIN A LINEAR APPROXIMAION PROBLEM AX B 15 8 G H Golub and C F Van Loan, An analysis of the total least squares problem, SIAMJ Num Anal, 17 (198), pp , Matrix Computations, fourth ed, he Johns Hopkins University Press, Baltimore and London, I Hnětynková, M Plešinger,DMSima,ZStrakoš, and S Van Huffel, he total least squares problem in AX B: A new classification with the relationship to the classical works, SIAM J on Matrix Anal and Appl, 32 (211), pp I Hnětynková and Z Strakoš, Lanczos tridiagonalization and core problems, Linear Algebra Appl, Vol 421 (27), pp C C Paige and Z Strakoš, Scaled total least squares fundamentals, Numerische Mathematik, 91 (22), pp , Unifying least squares, total least squares and data least squares, In: otal Least Squares and Errors-in-Variables Modeling, S Van Huffel and P Lemmerling, eds, Dordrecht, Kluwer Academic Publishers (22), pp , Core problem in linear algebraic systems, SIAM J Matrix Anal Appl, 27 (26), pp M Plešinger, he otal Least Squares Problem and Reduction of Data in AX B, PhD thesis, echnical University of Liberec, Liberec, Czech Rebuplic, D M Sima, Regularization techniques in model fitting and parameter estimation, PhDthesis, Dept of Electrical Engineering, Katholieke Universiteit Leuven, D M Sima and S Van Huffel, Core problems in AX B, echnical Report, Dept of Electrical Engineering, Katholieke Universiteit Leuven (26) 18 A van der Sluis, Stability of the solutions of linear least squares problems, Numerische Mathematik, 23 (1975), pp S Van Huffel and J Vandewalle, he otal Least Squares Problem: Computational Aspects and Analysis, SIAM Publications, Philadelphia, PA, S Van Huffel, ed, Recent Advances in otal Least Squares echniques and Errors-in- Variables Modeling, Proceedings of the Second Int Workshop on LS and EIV, Philadelphia, SIAM Publications, S Van Huffel and P Lemmerling, eds, otal Least Squares and Errors-in-Variables Modeling: Analysis, Algorithms and Applications, Dordrecht, Kluwer Academic Publishers, M Wei, he analysis for the total least squares problem with more than one solution, SIAM J Matrix Anal Appl, 13 (1992), pp , Algebraic relations between the total least squares and least squares problems with more than one solution, Numer Math, 62 (1992), pp