MV-PURE Estimator. Minimum-Variance Pseudo-Unbiased Reduced-Rank Estimator for Linearly Constrained Ill-Conditioned Inverse Problems

Transcription

1 MV-PURE Estimato Minimum-Vaiance Pseudo-Unbiased Reduced-Rank Estimato fo Linealy Constained Ill-Conditioned Invese Poblems Tomasz Piotowski, Student Membe, IEEE, Isao Yamada, Senio Membe, IEEE Depatment of Communications and Integated Systems (S3-60), Tokyo Institute of Technology, Tokyo , Japan Phone: , Fax: addesses: {tpiotowski, Abstact This pape poposes a novel estimato named Minimum-Vaiance Pseudo-Unbiased Reduced-Rank Estimato (MV-PURE) fo linea egession model, designed specially fo the case whee the model matix is ill-conditioned and the unknown deteministic paamete vecto to be estimated is subjected to known linea constaints. As a natual genealization of the Gauss-Makov (BLUE) estimato, the MV-PURE estimato is a solution of the following hieachical nonconvex constained optimization poblem diectly elated to the mean squae eo expession. In the fist stage optimization, unde a ank constaint, we minimize simultaneously all unitaily invaiant noms of an opeato applied to the unknown paamete vecto in view of suppessing bias of the poposed estimato. Then, in the second stage optimization, among all pseudo-unbiased educed-ank estimatos defined as the solutions of the fist stage optimization, we find the one achieving minimum vaiance. We deive a closed algebaic fom of the MV-PURE estimato and show that well-known estimatos: the Gauss-Makov (BLUE) estimato, the genealized Maquadt s educed-ank estimato and the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions ae all special cases of the MV-PURE estimato. We demonstate the effectiveness of the poposed estimato in a numeical example, whee we employ the MV- PURE estimato to the ill-conditioned poblem of econstucting a 2-D image subjected to linea constaints fom blued, noisy obsevation. This example demonstates that the MV-PURE estimato outpefoms all afoementioned estimatos, as it achieves smalle MSE fo all values of SNR. Index Tems MV-PURE estimato, linea egession, educed-ank estimation, ill-conditioned invese poblem Febuay 18, 2008

2 1 MV-PURE Estimato I. INTRODUCTION The poblem of linea estimation of an unknown deteministic paamete vecto in the linea egession model has a vey long and ich histoy. Its oigins can be taced back to Gauss [1], who poposed the classical least squaes estimato. Since then, owing to its univesality and mathematical simplicity, the linea egession model has been applied to a huge vaiety of sciences and engineeing, fo example communications, signal pocessing, economics and medicine, to name just a few. Howeve, as it has been shown in [2] [4], the least squaes estimato is inheently inadequate fo ill-conditioned poblems (the concept of ill-conditioned poblem can be taced back to Hadamad [5]), and this inadequacy cannot be emedied by any unbiased estimato even in the simplest case whee the noise is assumed to be white (see Section II-B). A vaiety of biased estimatos has been developed, which ae capable of achieving lowe mean squae eo to the unknown deteministic paamete vecto than the oiginal Gauss invention, the least squaes estimato. These include, fo example, the idge egession estimato [2], [3], essentially based on common idea of the so-called Tikhonov s egulaization [6] [9], the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions [10], the ank-shaping estimato [11] and the ank-eduction estimatos [4], [12]. Howeve, in spite of the diffeences between them, all these estimatos shae the common popety: as the mean squae eo depends explicitly on the unknown deteministic paamete vecto to be estimated (and hence cannot be minimized diectly), they aim at poviding optimal solutions fo some altenative citeia. This appoach casts inheent difficulties in suppessing the mean squae eo, fo example the idge egession estimato equies a cetain delicate choice of the egulaization paamete, which has motivated many studies on efficient appoximate selections of this paamete (see fo example [13] [15] and efeences theein). Theefoe, ecent contibutions such as [16], [17] focus on deivation of estimatos which ae solutions of poblems closely elated to the mean squae eo expession, and this pinciple is implemented in this pape as follows. We popose a novel educed-ank estimato named the minimum-vaiance pseudo-unbiased educed-ank estimato (MV-PURE) as a natual extension of the Gauss-Makov (BLUE) estimato [18] [20] to the case of educed-ank estimato, with the unknown vecto of paametes possibly subjected to linea constaints. The MV-PURE is the solution of an optimization poblem diectly elated to the mean squae eo expession: unde a ank constaint, minimize simultaneously all unitaily invaiant noms of an opeato applied to the unknown paamete vecto in view of suppessing bias of the poposed estimato, and among all solutions find the one achieving minimum vaiance. Moeove, this optimization poblem diectly incopoates auxiliay knowledge in the fom of linea estictions on the unknown paamete vecto to be estimated [9], [21] 1, which in well-defined settings educes futhe the mean squae eo of the poposed estimato even when no such constaints ae explicitly available [25]. Additionally, 1 Fo example, image econstuction is an ill-conditioned poblem in which it is common to have an auxiliay knowledge on the image being econstucted in the fom of linea constaints, see e.g. [22] [24] fo discussion and numeical examples. Febuay 18, 2008

3 2 the MV-PURE offes an unified view fo existing extensions of the Gauss-Makov (BLUE) estimato. Indeed, the ecently intoduced genealized Maquadt s educed-ank estimato [12] and the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions [10] ae paticula cases of the MV-PURE estimato, if the linea constaints on the paamete vecto to be estimated o the ank constaint on estimato ae not imposed. Peliminay shot vesions of this pape have been pesented at confeences [26] [28]. In paticula, all subsequent esults ae based on the oiginal pape [26], whee the hieachical nonconvex constained optimization poblem leading to the MV-PURE estimato was motivated, posed and solved fo the fist time, without consideing the linea constaints on the unknown vecto of paametes. Theefoe, based on the esults of [26], we achieved a chaacteization of the genealized Maquadt s educed-ank estimato [12] as a solution of the optimization poblem diectly elated to the mean-squae eo expession, which futhe motivated us to achieve the esults pesented in this pape. To the best of authos knowledge, the MV-PURE is the fist estimato with the ability to combine the educedank appoach with incopoating the auxiliay linea constaints on the unknown deteministic vecto of paametes. We veify the powe of this novel appoach in a numeical example, whee we demonstate the gain in pefomance of MV-PURE ove estimatos which ae obtained as its special cases in a simple image estoation poblem of econstucting a 2-D image fom blued, noisy obsevation. The pape is oganized as follows: in Section II we intoduce notation and definitions used thoughout the pape and pesent in detail the motivation behind the intoduction of ou novel MV-PURE estimato. In Section III we define the poposed estimato as the solution of a specially constucted optimization poblem and deive a closed algebaic fom of ou estimato. We close the main pat of the pape with a numeical example illustating the pefomance of ou estimato, desciption of which is povided in Section IV. Fo eades convenience, all necessay mathematical facts ae collected in Appendix I. The emaining appendices contain all poofs of esults stated in this pape. A. Notation and Definitions II. PRELIMINARIES In this pape we use as an essential tool the singula value decomposition (SVD), which is a ank-evealing decomposition of any ectangula matix. 2 Let us theefoe ecall that an SVD of a matix X R m n is given by: X = P ΓQ t = min(m,n) i=1 γ i p i q t i, (1) whee P = (p 1, p 2,..., p m ) R m m, Q = (q 1, q 2,..., q n ) R n n ae othogonal matices and Γ R m n contains on its main diagonal the singula values γ 1,..., γ min(m,n) of X and 0 s elsewhee. 3 Without losing geneality we assume that all SVDs consideed have singula values oganized in noninceasing ode, that is 2 Fo a complete discussion of the singula value decomposition, see e.g. [9], [29]. 3 In geneal, in this pape the singula values ae always denoted by the same lette (in lowecase) as the matix containing them. Febuay 18, 2008

4 3 γ 1 γ 2 γ k(x) > 0 and γ s = 0 fo s > k(x), whee k(x) stands fo the ank of X. Futhemoe, by X sub(i j), whee i m, j n, we will mean the i j pincipal submatix of X, and fo ϱ k(x) we define: tun ϱ {Γ} = Γ sub(ϱ ϱ) 0 R m n. We denote the Mooe-Penose pseudoinvese (see e.g. [9], [30]) of any matix X R m n by X R n m, which can be expessed in tems of a given SVD of X (1) as X = QΓ P t, whee Γ is of the fom: ( 1 Γ ( )) 0 sub k(x) k(x) Γ = X is the unique matix satisfying the following fou conditions: R n m. (i) XX = (XX ) t, (ii) X X = (X X) t, (iii) XX X = X, (iv) X XX = X. (2) Futhemoe, we have X = (X t X) X t, thus if k(x) = n, then X = (X t X) 1 X t and X X = I n, whee I n stands fo the identity matix in R n n. We denote by I the index set of all unitaily invaiant noms on R m n (i.e. noms satisfying UXV = X fo all othogonal U R m m, V R n n and all X R m n, see e.g. [29]). In paticula, note that the widely used noms: the Fobenius nom X F = min(m,n) t[x t X] = i=1 γi 2, the spectal nom X 2= max{ λ : λ an eigenvalue of X t X} = γ 1, and the tace (nuclea) nom X t = min(m,n) i=1 γ i, ae all elements of I. 4 Fo ϱ m we define U m ϱ -the set of all semiothogonal matices: U m ϱ = {Λ R m ϱ : Λ t Λ = I ϱ }. Note that Λ U m ϱ if and only if Λ = W sub(m ϱ) fo some othogonal matix W U m m. We say that a squae matix D R n n is a diect sum of squae matices D j R nj nj, j = 1,..., k, such that k n and k j=1 n j = n, if D is a block diagonal matix of the fom: D D 2 D =... In this case, we will denote D shotly as D = k j=1 D j. We denote the set of ank constained matices in R m n as X m n 0 D k. = {X R m n : k(x ) }, min(m, n), and if m = n we denote this set shotly as X m. In geneal, by X we will always mean matix of k(x ). Fo a given matix X R m n, we denote by N (X) R n the nullspace of X, by R (X) R m the ange of X, and by P N (X) and P R(X) the othogonal pojectos onto N (X) and R (X), espectively. Fo a given positive semidefinite matix S R n n we denote by S 1/2 R n n the unique positive semidefinite matix such that S 1/2 S 1/2 = S [29]. 4 Note that the spectal and tace noms ae dual noms, hence in paticula both ae elements of I [31]. Febuay 18, 2008

5 4 Finally, let us give the following notational emak: by X we will be denoting matix, which is optimal in the sense of a given poblem unde consideation. B. Ill-Conditioned Linea Regession Model and Linea o Affine Estimatos 1) Unbiased Estimatos and Reduced-Rank Estimatos: In the linea egession model it is assumed that we can obseve data vecto y R n of the fom: y = Lβ + ɛ, (3) whee L R n m is a known model matix (of full column ank 5 m) with an SVD: m L = UΣV t = σ i u i vi t, (4) β R m is an unknown deteministic paamete vecto to be estimated and ɛ R n is a andom vecto with zeo mean and positive definite covaiance matix E(ɛɛ t ) = σ 2 Q R n n, whee σ > 0. The emaining discussion assumes knowledge of Q, wheeas σ > 0 may not be explicitly known. The goal of linea estimation is to estimate the unknown vecto β by: i=1 β := Φy, (5) whee Φ R m n is a constant matix called hee an estimato. Moe pecisely, the majo goal is to find Φ suppessing the mean squae eo of Φy: ( J(Φ) := E β β 2) = E ( Φy E(Φy) 2) + E(Φy) β 2 = σ 2 t(φqφ t ) + (ΦL I m )β 2, (6) }{{}}{{} vaiance bias 2 whee E denotes expectation and the Euclidean nom. The citical poblem is that β in (6) is unknown, hence it is impossible to minimize J(Φ) globally ove Φ. An obvious choice is to estict attention to the class of unbiased linea estimatos, i.e. those satisfying ΦL = I m (which implies (ΦL I m )β 2 = 0 fo all β R m ). A natual selection among all unbiased estimatos is the estimato with the smallest vaiance, that is, the estimato which is the solution to the following constained optimization poblem: Indeed, upon setting: the Gauss-Makov (BLUE) estimato [18] [20] given by: minimize t [ΦQΦ t ] subject to ΦL = I m. L := Q 1/2 L, (8) Φ BLUE := (L t Q 1 L) 1 L t Q 1 = ( L t L) 1 Lt Q 1/2 = L Q 1/2, (9) (7) 5 Note that the case of possibly ank-deficient model matix L has also been consideed in the liteatue, see e.g. [9, p ]. This moe geneal case was also consideed in the oiginal pape [26], which in paticula motivated us to obtain an efficient epesentation of MV-PURE fo the case of ank-deficient L. Results in this diection will be epoted elsewhee. Febuay 18, 2008

6 5 is the unique solution to poblem (7), whee L stands fo the Mooe-Penose pseudoinvese of L. Howeve, note that fo Q = I n the BLUE estimato educes to the least squaes estimato [1], [18], which is simply the Mooe-Penose pseudoinvese of L: Φ ls := (L t L) 1 L t = L = V Σ U t = Moeove, in this case fom (6) and (10) we obtain that: J(Φ ls ) = J(L ) = σ 2 m 1 σ 2 i=1 i } {{ } vaiance m i=1. 1 σ i v i u t i. (10) Hence, we obseve that the least squaes estimato yields inheently a dastic inadequacy when the linea egession model (3) is ill-conditioned, i.e. when L possesses some vey small singula values σ +1,..., σ m, and futhemoe, that any unbiased estimato cannot emedy it, even in the simplest case Q = I n. Theefoe, Maquadt poposed in [4] fo the case Q = I n, to use the following educed-ank estimato (see also [12]): Φ M := V tun {Σ }U t = i=1 1 σ i v i u t i = V V t Φ ls, (11) whee we denote V = (v 1,..., v ) fo V = (v 1,..., v m ) and < m. Obviously, Φ M satisfies k(φ M ) =. Moeove, fom (6) we obtain: J(Φ M ) = σ 2 1 σ 2 i=1 i + m i=+1 v t iβ 2. Hence, unde the following easonable assumption fo ill-conditioned cases [4]: m 1 > β 2 σ 2, we have that: which is equivalent to J(Φ M ) < J(Φ ls ). 1 σ 2 σ 2 i=+1 i m i=+1 v t i β 2 < m 1, σ 2 i=+1 i In this pape an estimato Φ R m n is called a educed-ank estimato if Φ satisfies k(φ) < m. Note that fom (6) any educed-ank estimato Φ cannot be unbiased fo all β R m because the equality ΦL = I m neve holds in this case. Thus, the above Maquadt s estimato Φ M, which is a simple example of the educed-ank estimato, demonstates the following fact: if we allow estimato to be slightly biased, unde a mild condition we may expect much bigge savings in the vaiance of the estimato, thus achieving smalle mean squae eo than any unbiased estimato (see also [32] fo a thoough discussion of the educed-ank estimation appoach). The Maquadt s estimato was ecently extended to the geneal case of coelated noise (when Q is allowed to be any positive definite matix) by Chipman in [12] as follows: if we set an SVD of L in (8) to be of the fom: L = Ũ ΣṼ t, (12) then Chipman s estimato: the genealized Maquadt s educed-ank estimato is defined as follows, whee we Febuay 18, 2008

7 6 denote Ṽ = (ṽ 1,..., ṽ ): Φ C := ṼṼ t L Q 1/2 = ṼṼ t Φ BLUE. (13) Clealy, fo Q = I n we obtain Φ C = Φ M, and fo = m (i.e., when no ank constaint is imposed on the estimato) we have Φ C = Φ BLUE, hence Φ C may be viewed not only as a genealization of the Maquadt s estimato fo the case of coelated noise, but also as a educed-ank extension of the Gauss-Makov (BLUE) estimato, just like Maquadt s estimato was intoduced above as a educed-ank extension of the least squaes estimato. Howeve, the fundamental diffeence between the BLUE estimato (9) and the educed-ank Maquadt s (11) and Chipman s (13) estimatos is that Φ BLUE is the solution of the well-defined mathematical optimization poblem (7) diectly elated to the mean-squae eo expession (6), wheeas Φ M and Φ C ae not. This key diffeence, which guaantees optimality of the BLUE estimato (9) among all unbiased estimatos, leaves the question of optimality of Φ M and Φ C in the class of educed-ank estimatos open. 2) Linea Constaints on the Unknown Paamete Vecto in Linea Regession Model: Assume that we have auxiliay knowledge: the unknown deteministic paamete vecto β R m in (3) is an element of the set: } V := { β R m : A β = b, (14) whee A R s m and b R s ae given and satisfy: k(a) = a s m. 6 Thus, the linea egession model (3) is now of the following fom: y = Lβ + ɛ, β V. (15) Since the constaint set V is a linea vaiety (a tanslation of a subspace of R m by a constant vecto), we elax the peviously employed linea estimato Φ to an affine estimato Ψ : R n R m, which is a constant shift of a linea estimato. Now, ou goal is to find an affine estimato Ψ such that: β β = Ψ(y) V. (16) Indeed, by genealizing the Gauss-Makov (BLUE) estimato Φ BLUE (9), an efficient affine estimato: the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions has been poposed in [10] (see also [12]) fo the case whee a = s < m : Ψ CR (y) := (I m A A)Φ BLUE y + A b, (17) whee: A = (L t Q 1 L) 1 A t [ A(L t Q 1 L) 1 A t] 1 = ( Lt L) 1 A t [ A( L t L) 1 A t] 1. (18) Clealy, fo unconstained β (setting A = 0 R s m, b = 0 R s in (14)) we obtain Ψ CR = Φ BLUE and futhemoe, paticula conditions unde which the affine estimato Ψ CR achieves lowe mean squae eo than 6 We exclude the tivial case of A being an invetible matix, that is a = s = m, as in such case V is simply a singleton. Febuay 18, 2008

8 7 the Gauss-Makov (BLUE) estimato Φ BLUE have been given in [12]. Thus, the estimato Ψ CR is an impotant genealization of the BLUE estimato onto the case of linealy constained β V. Howeve, we may ask the following question: since, as we obseved above, the educed-ank appoach is of geat use fo the case of unconstained β R m, would it be possible to incopoate it in the case of affine estimation when β V? III. MV-PURE ESTIMATOR A. Poblem Fomulation and Intepetation We popose a novel MV-PURE estimato, which combines the educed-ank appoach with the ability to use auxiliay linea constaints on the unknown deteministic vecto of paametes. Moeove, it will be demonstated that MV-PURE encompasses the estimatos intoduced in the pevious Sections as special cases as shown in Fig. 1. Fig. 1. Relations between MV-PURE and Φ BLUE (9), Φ ls (10), Φ M (11), Φ C (13) and Ψ CR (17). In paticula, it will be shown that Ψ CR can be expessed in much simple fom than (17). We popose an affine estimato fo the linealy constained model (15) as a constant shift of the linea educed-ank estimato as follows. 1) Homogeneous linea constaints: Suppose fistly that the unknown deteministic vecto of paametes β R m satisfies linea homogeneous constaints, which yields the linea egession model: y = Lβ + ɛ, β N (A), (19) whee the constaint set is the nullspace of A: } N (A) = { β R m : A β = 0. Febuay 18, 2008

9 8 Since β = P N (A) z fo all z β + N (A), we find that the mean squae eo of a linea estimato Φ R m n can be expessed as: J(Φ) = σ 2 t(φqφ t )+ (ΦL I m )β 2 = σ 2 t(φqφ t ) + (ΦLP }{{} N (A) P N (A) )z 2, z β + N (A). (20) }{{} vaiance bias 2 As was emphasized above, the mean squae eo J(Φ) cannot be minimized diectly due to explicit dependence of the bias tem on unknown paamete β, which can be an abitay element of N (A). Moeove, if we impose the ank constaint k(φ) < k(p N (A) ) on Φ, the equality ΦLP N (A) = P N (A) (which guaantees unifom unbiasedness of Φ) cannot be achieved. Now, by noting that: ( β + N (A) ) = N (A) + N (A) = R m, β N (A) and by extending the idea behind the BLUE estimato (9), we find that it is vey natual to select Φ fo which (ΦLP N (A) P N (A) ) is as close as possible to 0 R m n. The closeness to 0 R m n inheently elies on the matix nom employed as a citeion. Howeve, if thee exists Φ minimizing simultaneously a vaiety of matix noms of (ΦLP N (A) P N (A) ), then such a matix must be a natual educed-ank genealization of the unbiased estimato. Theefoe, in this pape we popose the novel educed-ank estimato as the solution to the following poblem, fo given ank constaint k(p N (A) ). Define: P ι := ag min Φ X m n Φ LP N (A) P N (A) 2 ι, ι I, (21) whee I is the index set of all unitaily invaiant noms (see Section II-A). Then, we call Φ X m n a pseudounbiased educed-ank estimato of β N (A) fo the linea egession model (19) if Φ ι I P ι. In paticula, Φ mvp X m n is called the minimum-vaiance pseudo-unbiased educed-ank estimato (MV-PURE) of β N (A) fo the linea egession model (19) if Φ mvp is a solution of: minimize t [Φ Q(Φ ) t ] subject to Φ ι I P ι. (22) It is seen theefoe that the idea of the MV-PURE poblem (22) is to find a educed-ank, minimum-vaiance estimato Φ mvp among estimatos minimizing simultaneously all unitaily invaiant noms of an opeato applied to the unknown paamete z R m. Note that the MV-PURE poblem (22) may be viewed as a diect extension of the BLUE poblem (7), as fo no ank constaint imposed on the estimato, = k(p N (A) ), and fo unconstained β R m (which coesponds to A = 0, thus P N (A) = I m ), MV-PURE poblem (22) educes to BLUE poblem (7). Moeove, using Lemma 3 in the latte pat of this section, it will be shown that: thus Φ mvp P ι = ag ι I min Φ X m n is the solution of the poblem (22) if and only if Φ mvp Φ LP N (A) P N (A) 2 F, (23) is a solution of: Febuay 18, 2008

10 9 minimize t [Φ Q(Φ ) t ] subject to Φ ag min Φ LP N (A) P N (A) 2 Φ X m n F. (24) Let us note that since the set of ank-constained matices X m n is clealy nonconvex, the hieachical constained optimization poblem (24) is nonconvex, and theefoe we could not use the poweful methods of hieachical convex optimization (see e.g. [33] [35]). Howeve, we will show that fo all ank constaints k(p N (A) ) solution to poblem (24) exists, as we will give an explicit algebaic fom of such solution. Futhemoe, it will be also demonstated that Φ mvp satisfies: Φ mvp y = β N (A) y R n, (25) that is, the estimate β of the unknown paamete vecto β is guaanteed to be in the constaint set N (A) accoding to the linea egession model (19) unde consideation. See Appendix II fo futhe emaks on poblem (22)-(24), whee we discuss an altenative appoach (posed also in [26]) of employing the spectal nom in the fist stage of the MV-PURE poblem (22). 2) Extension to geneal linea constaints: Let us note that the constaint set V (14) can be equivalently ecast as (see e.g. [9, p.53]): V = N (A) + A b, (26) thus β V β = β + A b, β N (A). By inseting this expession of β V into (15) we obtain, upon setting y = y LA b: y = Lβ + ɛ, β N (A), (27) which is equivalent to (19). Theefoe, we intoduce the affine MV-PURE estimato Ψ mvp fo the linea egession model (15) as: Note that Ψ mvp (y) V in view of (25) and (26). Ψ mvp (y) := Φ mvp y + A b = Φ mvp (y LA b) + A b. (28) Finally, let us note that the phase MV-PURE estimato was used both to denote Φ mvp -the educed-ank estimato fo the homogeneously constained model (19), which is a solution to poblem (22)-(24), and its constant shift Ψ mvp -the affine estimato fo model (15). We did it to highlight the cental ole of the MV-PURE poblem (22)-(24) in ou method. Note that this does not lead to any ambiguity, since fo homogeneous constaints V = N (A) (b = 0) we obviously have y = y and Ψ mvp = Φ mvp. B. Deivation of MV-PURE Estimato Let us solve fistly poblem (24) and show theeafte that this is equivalent to solving ou initial poblem (22). Let us define: Π := ΦQ 1/2 and L := Q 1/2 LP N (A), (29) Febuay 18, 2008

11 10 then it can be immediately veified that (24) is equivalent to finding Φ mvp of the following poblem: minimize subject to Π Π 2 F = Π mvp Q 1/2, whee Π mvp is a solution ag min Π L P N (A) 2 Π X m n F. (30) Let us moeove set an SVD of A = MΥN t, so that P N (A) = I m A A can be expessed as: P N (A) = I m NΥ M t MΥN t = NI m N t N I a 0 N t = N N t. 0 I m a Hence, with ou usual notation N = (n 1,..., n m ) and upon setting N R m m to be the othogonal matix of the fom: ( N = N m a N a ), whee N m a = (n a+1,..., n m ) and N a = (n 1,..., n a ), (31) we obtain: P N (A) = N I m a 0 N t = N m a N t m a. (32) Remak 1: Note that fom (32) we obtain in paticula that k(p N (A) ) = dim(n (A)) = m a, hence we will be consideing ank constaints on Π in poblem (30) within the set {1,..., m a}. Befoe poceeding to the main pat of this section, let us note fistly the following Lemma which will be useful in achieving compact deivation of ou estimato. Lemma 1: Let Γ R m m be a diagonal matix with diagonal elements (γ 1,..., γ m ), and let us assume that they ae aanged in noninceasing ode, i.e. γ 1 γ m. Let us futhemoe define: Ω = {1,..., m}, and its subsets: and, fo i = 2,..., m: Moeove, let us set: Ω i = Then, an othogonal matix Z R m m satisfy: Ω 1 = { ω 1 Ω : γ ω1 = max k Ω γ } k, { ω i Ω i 1 j=1 Ω j : γ ωi = max l = max i: Ω i i. k Ω S γ i 1 k j=1 Ωj }. ZΓ = ΓZ, (33) if and only if Z is a diect sum of cetain othogonal matices Z j R Ωj Ωj fo j = 1,..., l, i.e. Z is of the following fom: Poof: See Appendix III. Z = l Z j R m m. (34) j=1 Remak 2: The above Lemma can be equivalently stated fo the case whee the diagonal elements (γ 1,..., γ m ) ae oganized in nondeceasing ode, i.e. γ 1 γ m, by eplacing max by min in the definitions of sets Ω i. Febuay 18, 2008

12 11 As will be shown below, an essential pat of obtaining a closed algebaic fom of the MV-PURE estimato is the knowledge of complete paametization of the set of best ank m a appoximations (in the sense of Fobenius nom) of P N (A). Such paametization is given in the following Lemma: Lemma 2: Let us set ank constaint m a and let us denote: C := ag min P N (A) X m P N (A) P N (A) 2 F. Then PN (A) C if and only if P N (A) is of the following fom: PN (A) = N ΛΛt R (m a) (m a) 0 N t R m m, (35) whee N is defined as in (31) and Λ U (m a). In paticula, P N (A) 2 F =. Poof: See Appendix IV. The following Lemma, inteesting in its own ight, will be used to pove the main esult of this pape. Lemma 3: With notation as in Lemma 2, we have, fo given ank constaint m a: C = { } ag min P N (A) P N (A) 2 P N (A) X m ι. (36) ι I Poof: See Appendix V. 7 Remak 3: Note that fom Fact I-1 8 and Step 1. in Appendix IV we obtain immediately that: P N (A) C if and only if k(p N (A) ) = and P N (A) P N (A) 2 F = m a. (37) We can show now how esults obtained in Lemma 2 can be used to find a complete solution set of the poblem (30). We fistly note the following useful emaks. Remak 4: Let us ecall that in this pape we assume that the design matix L R n m is of full column ank m and that we denote L = Q 1/2 LP N (A) (29). We have k(l ) = k(p N (A) ) = m a, since by definition of L we have k(l ) m a, and the equality follows immediately fom the fact that P N (A) = L Q 1/2 L and the equality k(l Q 1/2 ) = m. Remak 5: Let P N (A) C. Fom (32) and (35) we obseve that the equation ΠL = P N (A) is solvable (with espect to Π), since fo example Π = P N (A) L Q 1/2 is a paticula solution. Thus, fom Fact I-5 (see also Remak 6 made below the statement of Fact I-5) we obtain that fo given P N (A) C, the unique solution of equation ΠL = P N (A) having minimum Fobenius nom is of the fom: Π F := P N (A) (L ). (38) Lemma 4: Let us set ank constaint m a and let us define f : C R m n as follows: f(p N (A) ) = Π F, (39) 7 We note that by using completely simila aguments to the ones used in poofs of Lemmas 2 and 3, one may show that fo any given matix Z R m n, the set of best ank k(z) appoximations of Z in the sense of Fobenius nom is the set of best ank k(z) appoximations of Z in the sense of all unitaily invaiant noms. 8 Fo eade s convenience, we collected all known esults in Appendix I, thus in paticula I-1 efes to Fact 1 in Appendix I. Febuay 18, 2008

13 12 whee Π F is of the fom (38). Then Π mvp Π mvp R m n is a solution of poblem (30) if and only if: { ag min Π F 2 F, Π F R (f) }, (40) } whee R (f) = {X R m n : X = f(p N ) fo some P N C (A) (A). Moeove, k(π mvp ) = and: Poof: See Appendix VI. Π mvp L P N (A) 2 F = P N (A) P N (A) 2 F = m a. Thus, by Lemmas 2 and 4 we conclude that in ode to fully solve poblem (30), we have to find Λ min U (m a) appeaing in the expession of P N (A) (35), fo which the Fobenius nom of Π F = P N (A) (L ) is minimized. Theoem 1: 1. Let us set ank constaint < m a, and let L = Q 1/2 LP N (A), k(l ) = m a (see Remak 4). Let futhemoe N R m m be an othogonal matix such that P N (A) = N I m a 0 N t in (32). Moeove, let us set: Then K is positive definite and Π mvp K := [ N t (L ) (L t ) N ] sub((m a) (m a)). is a solution to poblem (30) if and only if Π mvp is of the following fom: = N E E t 0 N t (L ), (41) Π mvp whee K = E E t is any eigenvalue decomposition of K with eigenvalues oganized in nondeceasing ode: 0 < δ 1 δ 2 δ m a, and whee we denoted E = E sub((m a) ). The nom of Π mvp is given by: Π mvp 2 F = δ i. (42) Moeove, if δ δ +1, the solution Π mvp is unique. 2. Fo no ank constaint imposed, i.e. when = m a, the solution to poblem (30) is uniquely given by: with the nom of Π mvp Π mvp m a 2 F = m a Poof: See Appendix VII. m a = N δ i. i=1 I m a 0 i=1 N t (L ) = P N (A) (L ), (43) We obtain the following solution to ou initial poblem (22) which is the Φ mvp -MV-PURE estimato fo the linea egession model (19). The affine MV-PURE estimato fo the linea egession model (15) is obtained fom Φ mvp as shown in (28). Theoem 2: Let us set ank constaint m a. Then, with notation as in Theoem 1, Φ mvp solution to poblem (22) if and only if Φ mvp R m n is a = Π mvp Q 1/2, whee Π mvp is a solution to poblem (30). We have: k(φ mvp ) =, (44) Febuay 18, 2008

14 13 and the vaiance of Φ mvp is given by: Φ mvp LP N (A) P N (A) 2 F = m a, (45) σ 2 t [ Φ mvp Moeove, Φ mvp satisfies the popety (25). Poof: See Appendix VIII. Q(Φ mvp ) t] = σ 2 Π mvp 2 F = σ 2 δ i. (46) Let us emphasize the impotance of (44)-(46). Not only do they shed light onto theoetical popeties of Φ mvp (in paticula, (46) gives the exact value of vaiance of Φ mvp ), but they may also be used to detemine the citeia fo choosing the ank of the estimato fo a given pactical situation. An example of such citeion will be given in Section IV with application to two diffeent pactical scenaios. We close this section with the following Theoem elating the MV-PURE estimato to the genealized Maquadt s educed-ank estimato Φ C (13) and the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions Ψ CR (17). Theoem 3: 1. Let us assume that no linea constaints have been imposed on the unknown vecto of paametes β R m in the linea egession model (15), which is thus equivalent to the basic model (3). Then, fo all ank constaints < m, we have Φ mvp = Φ C and such estimato is unique if the th and the ( + 1) st singula values of L (8) ae distinct. Moeove, fo = m we have Φ mvp m = Φ BLUE. 2. Let us assume that no ank constaint has been imposed on the estimato, = m a, and that the necessay conditions of existence of Ψ CR ae satisfied: a = s < m (see [10] and [12]). Then, fo all linea constaints V (14) in the linea egession model (15) we have Ψ mvp = Ψ CR. Poof: See Appendix IX. i=1 IV. NUMERICAL EXAMPLE The poblem we conside is that of estoation of a 2-D image fom an obsevation degaded by atmospheic tubulence blu and contaminated with Gaussian noise. 9 The spase bluing matix is a ealization of the MATLAB algoithm povided in [36] (see also [23]), and the small 8 8 pixels image to be estoed is shown in Fig.2. This image is oiginally stoed in a matix fom as P R 8 8, whee gayscale pixels wee epesented as numbes in [0, 1], with 0 epesenting black and 1 epesenting white pixels. To accomodate linea constaints in ou example, we assume that P is an element of linea vaiety V of all images having the fist two and the last two ows of white pixels. These linea constaints can be easily expessed as V = { X R 8 8 : AX = B }, whee: A = I 2 0 R 4 8, B = R I By denoting vecx R 64 to be the columnwise stacked vesion of a matix X R 8 8, and using the fact that the matix equation AX = B can be equivalently expessed as (I 8 A)vecX = vecb, whee by we denote 9 We used MATLAB fo implementation of this poblem. Febuay 18, 2008

15 14 Fig. 2. A simple image to be estoed. the Konecke poduct of matices (see e.g. [31]), we see that ou estoation poblem can be cast into the linea egession model (15) as follows: let L R be the bluing matix taken fom [36] 10, and let us moeove ewite the definition of linea constaints V as: V = { vecx R 64 : (I 8 A)vecX = vecb }. Thus, if we denote by Y R 8 8 the blued, noisy obsevation, we ecognize that the estoation poblem is that of estimating vecp fom vecy in the linea egession model: vecy = L vecp + ɛ, vecp V, (47) which is of the fom (15). Let us note that in ou example we have a = s = 32 < m = 64, thus in paticula we could compae the pefomance of ou estimato with the pefomance of the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions Ψ CR (17). We conside the following estimatos: the Gauss-Makov (BLUE) Φ BLUE (9), the genealized Maquadt s educed-ank Φ C (13), the minimum-vaiance conditionally unbiased affine estimato subject to linea estictions Ψ CR (17) and the MV-PURE estimato. Futhemoe, we conside the case of white Gaussian noise which may coespond to impecision in measuing tools at ou disposal. We have thus E(ɛɛ t ) = σ 2 I n, and we define the SNR as: SNR = LvecP 2 T (Q) = LvecP 2 σ 2. n 10 This matix is seveely ill-conditioned, as its singula values ae σ , σ ,..., σ , σ , σ , σ Febuay 18, 2008

16 15 We examine the pefomance of ou estimatos at the levels {20, 22,..., 40} of SNR[dB]. With notation as in Theoem 1, the ank-selection citeion is the following [37]: { } δ ank = ag max < δ, (48) δ1 which is elated to the vaiance of Φ mvp (46). Note that the citeion (48) may be used in moe demanding scenaios than ou simple numeical example, since fo given δ 1, upon vaying noise statistics, design matix o linea constaints, this citeion will automatically eflect the changing conditions (see also below). Moeove, let us note that Theoem 2 (45), (46) povides quantities upon which intepetations an altenative ank-selection citeion might be chosen fo a paticula application at hand Least Squaes (BLUE) Chipman s LC (Genealized) Maquadt MV PURE MSE [db] SNR [db] Fig. 3. MV-PURE achieves impovement in pefomance by combining the educed-ank appoach with incopoation of known linea constaints. The MSE obtained is aveaged ove 1000 ealizations of noise. Moeove, in ou example we have δ δ +1 fo all = 1,..., m a 1, hence the MV-PURE estimato is always uniquely defined fo all ank constaints. Note that since Q = I n in ou example, we have Φ C = Φ M (see Section II-B), and in view of Theoem 3 this estimato coesponds to the MV-PURE with no linea constaints and white noise. Thus, we use the same ank-selection citeion (48) fo this estimato, which in this case is simply expessed as (see Theoem 3): { } σ1 ank = ag max < δ, σ i.e., in tems of singula values of design matix L (4). We set the value of δ = 10 fo both Ψ mvp and Φ C = Φ M estimatos, and obseve significant gain in pefomance of the MV-PURE estimato which comes fom taking into Febuay 18, 2008

17 16 acount the available linea constaints. Note that it may be agued that the use of MSE as a citeion fo the image econstuction pefomance may be questionable. Moeove, we did not implement the post-pocessing step of constaining the values of the obtained image estimate to be within [0, 1]. Howeve, we would like to emphasize that the single eason to choose this example was to show the powe of the MV-PURE estimato in possibly simplest and shotest way. Fo futhe applications we efe the eade to [37], whee the MV-PURE is applied as the linea eceive in the multiuse MIMO wieless systems. In such settings, the intepetation of MV-PURE poblem (22)-(24) is the minimization of the eceive s output powe subject to maintaining unitay gain and cancelling self-intefeence of the desied use. In this pape we also use the ank-selection ule (48) fo the poposed eceive, and note significant gain in pefomance ove peviously known eceives in the case of coelated wieless channels. In paticula, we obseve that the ank-selection citeion (48) woks coectly, fo the same value of δ 1, unde vaying the channel and obseved signal statistics 11. APPENDIX I KNOWN RESULTS USED Fo convenience, let us ecall that in this pape we assume that all singula value decompositions consideed have singula values oganized in noninceasing ode. Fact 1 (Misky-Schmidt Appoximation Theoem [38]): Let C R m n be a given matix of k(c) = c and let us set ank constaint < c. Then: X S { ag ι I min X X m n X C ι }, (49) if X S is of the following fom: X S = Gtun {Υ}H t, (50) whee C = GΥH t is an abitay SVD of C. In paticula, the appoximation eo of X S fo the Fobenius nom is: c X S C F = υi 2, (51) and fo the spectal nom: i=+1 X S C 2= υ +1. (52) Moeove, fo the Fobenius nom, such best appoximation is unique if and only if the th and the ( + 1) st singula values of C ae distinct (see [9, p.212]), i.e. υ > υ +1. Fact 2 ( [39]): Let Γ R m m be a diagonal matix with diagonal elements (γ 1,..., γ m ), and let Z R m m be any given matix. Then, matices Z and Γ satisfy: ZΓ = ΓZ, (53) 11 Moe pecisely, the matix Q epesents in [37] a sample covaiance matix of the signal obseved at the eceive. Febuay 18, 2008

18 17 if and only if fo each pai i, j such that γ i γ j we have Z i,j = 0. Futhemoe, if Z commutes with Γ, then also Z t commutes with Γ. Fact 3 ( [29, p.448]): Let matices C 1, C 2 R m n be given with SVDs C 1 = G 1 Υ 1 H1 t, C 2 = G 2 Υ 2 H2 t. Then: C 1 C 2 F Υ 1 Υ 2 F. (54) Fact 4 ( [39]): Let U 1, U 2 R m m be any othogonal matices and let Γ 1, Γ 2 R m m be diagonal matices with nonnegative diagonal enties oganized in noninceasing ode such that = min(k(γ 1 ), k(γ 2 )). Then: t [U 1 Γ 1 U 2 Γ 2 ] t [Γ 1 Γ 2 ], (55) with equality if and only if: U 1 = G 2 H t 1 and U 2 = G 1 H t 2 (56) fo cetain othogonal matices G j, H j R m m satisfying: (G j ) sub(m ) = (H j ) sub(m ) = (K j ) sub(m ) fo j = 1, 2, whee K 1, K 2 R m m ae othogonal matices such that: K j Γ j = Γ j K j fo j = 1, 2. Fact 5 ( [9, p.109]): Let C R m n, b R m. If Cx = b is solvable fo x, then the unique solution fo which x is smallest is given by: whee C min Rn m is any matix such that: x = C minb, (57) CC min C = C and (C min C)t = C min C. Remak 6: The solution of minimum nom x = C min b is invaiant unde the choice of matix C min Rn m satisfying the above conditions, theefoe in applications of Fact I-5 we will be always setting C min = C, the Mooe-Penose pseudoinvese of C. Moeove, it is clea that if we conside the solvable matix equation CX = B fo some unknown matix X, then the unique solution of minimum Fobenius nom (taking into account the fist pat of this emak) is given by X F = C B. Finally, since we have (Z ) t = (Z t ) fo any matix Z, we note that the unique solution of minimum Fobenius nom of the equation Y C = B (which can be equivalently witten as C t Y t = B t ) is given by Y F = BC. Fact 6 ( [29, p.399]): If A R n n is a positive definite matix, then fo any C R m n the matix CAC t is positive semidefinite such that k(cac t ) = k(c). Fact 7 ( [40]): Let C, D R m m be symmetic matices with eigenvalue decompositions C = P ΣP t and D = QΛQ t, whee in both cases the eigenvalues ae oganized in noninceasing ode. Then: t [CD] t [ΣΛ], (58) with equality if and only if P t Q = G t H, whee G, H R m m ae any othogonal matices commuting with Σ and Λ, espectively. Febuay 18, 2008

19 18 Remak 7: With notation as in Fact I-7, we have equivalently: t [C( D)] t [Σ( Λ)], (59) with the same condition of equality as above. The following Fact is easily deduced fom [31, p.147]: Fact 8: Let C R m m be a symmetic matix with eigenvalue decomposition C = P ΣP t. Then all eigenvalue decompositions of C of the fom C = XΣX t ae such that X = P Z, whee Z R m m is an othogonal matix commuting with Σ, i.e. ZΣ = ΣZ. Poof: We have C = P ΣP t = XΣX t, hence P t XΣ = ΣP t X and theefoe P t X = Z fo some othogonal matix Z R m m commuting with Σ, thus X = P Z, as claimed. Fact 9 ( [41]): If A R m n, B R n p, then (AB) = ( A AB ) ( ) ABB. APPENDIX II REMARKS ON MV-PURE PROBLEM (22) At fist glance, the choice of spectal nom in the MV-PURE poblem (22)-(24) seems natual, since it is the nom induced by the Euclidean nom on vectos (see e.g. [29, p ]). Howeve, we will show that posing the MV-PURE poblem in the sense of the spectal nom leads to the tivial solution, wheeas solving (22)-(24) intoduced a new efficient estimato encompassing peviously known estimatos: Φ BLUE (9), Φ ls (10), Φ M (11), Φ C (13), Ψ CR (17) as special cases. Let us conside the following poblem in place of (22): minimize t [Φ Q(Φ ) t ] subject to Φ ag min Φ LP N (A) P N (A) 2 Φ X m n 2. Then, it may be eadily veified that fo all ank constaints < m a, the unique solution of the poblem is the zeo matix 0 R m n, since: 0LP N (A) P N (A) 2 2 = P N (A) 2 2 = 1, which in view of Misky-Schmidt Appoximation Theoem (Fact I-1) implies that: 0 ag min Φ LP N (A) P N (A) 2 2 Φ X m n fo any < m a. Clealy, t[0q0] = 0 which is the unique global minimum, hence 0 is the unique solution of (60) fo all ank constaints < m a. Similaly, if, fo given ank constaint < m a, we insisted on solutions having ank exactly, then it may be easily veified that the following matices: X (k) = 1 k N I 0 N t L, k = 1, 2,..., (60) Febuay 18, 2008

20 19 whee P N (A) = N thus X (k) ag I m a 0 min X X m n N t (32), satisfies k(x (k) ) = and: X (k) LP N (A) P N (A) 2 2= 1, X LP N (A) P N (A) 2 2. Moeove, we have: [ ( t X (k) Q X (k)) ] t = 1 k 2 t N I 0 N t L Q ( L ) t. Assume now existence of X R m n achieving minimum of t [X QX] t in ag min X LP N (A) P N (A) 2 2. [ k(x )= We obseve that by choosing suitably lage k, we obtain t X (k) Q ( X (k)) ] [ t < t X Q (X ) t]. Theefoe, such optimal X Poof: does not exist, as lim k X(k) = 0, which has zeo ank. APPENDIX III PROOF OF LEMMA 1 (= ) Assume fistly that Z commutes with Γ. By using the equivalent condition of commutativity stated in Fact I-2, we obtain that Z must be of the fom: Z = l Z j R m m, (61) j=1 fo some matices Z j R Ωj Ωj, j = 1,..., l. If thee exists some Z j which is not othogonal, this would violate ou assumption of othogonality of Z, which shows that indeed Z must be of the fom (34). ( =) Convesely, if Z is of the fom (34), then fom Fact I-2 we obtain immediately that it satisfies the sufficient condition of commutativity. APPENDIX IV PROOF OF LEMMA 2 Poof: Let P N (A) X m and let us set an SVD of P N (A) = E F t. Moeove, to simplify subsequent notation let us denote Ĩm a = I m a 0, Ĩ = I 0, whee Ĩm a, Ĩ R m m. In paticula, in view of (32) we have P N (A) = N Ĩ m a N t. Step 1. If P N (A) C then = Ĩ. Suppose not: Ĩ. Then, fom Facts I-1 and I-3 we obtain: m a = P N (A) P N (A) 2 F Ĩm a 2 F > m a, which contadicts ou assumption, hence it must be = Ĩ. Thus, if P N (A) C then P N (A) = EĨF t fo some othogonal matices E, F R m m to be detemined, theefoe we obtain in paticula that P N (A) 2 F =. Febuay 18, 2008

21 20 Step 2. Let us expess E and F by E = N R, F = N S, whee R = N t E R m m and S = N t F R m m ae othogonal matices, so that: P N (A) = N RĨS t N t. (62) Then: P N (A) C if and only if t [ P N (A) P N (A) ] = t [ RĨS t Ĩ m a ] =. (63) In ode to show the above equivalence, let us note fistly that: P N (A) P N (A) 2 F = P N (A) 2 F + P N (A) 2 F 2t [ P N (A) P N (A) ] = m a+ 2t [ RĨS t Ĩ m a ]. (64) Assume now that P N (A) C. Since by ou assumption P N (A) P N (A) 2 F = m a, fom (64) we obtain ] ] immediately that it must be t [RĨS t Ĩ m a =. Convesely, assume that t [RĨS t Ĩ m a =. Fom (64) we have that P N (A) P N (A) 2 F = m a, hence P N (A) C. Step 3. Thus, it suffices now to find the full paametization of the pais of othogonal matices (R, S) achieving ] equality t [RĨS t Ĩ m a =. This can be immediately obtained fom Fact I-4, whee upon substituting U 1 = R, U 2 = S t, Γ 1 = Ĩ and Γ 2 = Ĩ m a we obtain that all othogonal matices (R, S) achieving the desied equality must be of the fom (56). Theefoe, by inseting the obtained expessions of optimal (R, S) into (62) we obtain that P N (A) C if and only if: The above expession of P N (A) P N (A) = N G 2 H t 1ĨG 1 H t 2N t. (65) can be simplified significantly in the following way: we have fom Fact I-4 that H 1 and G 1 shae the fist columns with an othogonal matix K 1 commuting with Ĩ, and fom Lemma 1 we obtain that K 1 must be of the following fom: K 1 = V R 0 0 X 0 R (m ) (m ) whee V, X 0 ae cetain othogonal matices. Theefoe, we conclude that H 1, G 1 ae of the fom: H 1 = V R 0, G 1 = V R 0 0 X 1 R (m ) (m ) 0 X 2 R (m ) (m ),, (66) fo some othogonal matices X 1, X 2. Similaly, we obseve fom Lemma 1 that the othogonal matix K 2 commuting with Ĩm a must be of the fom: K 2 = W R(m a) (m a) 0 0 Y 0 R a a whee W, Y 0 ae cetain othogonal matices. Upon setting Λ = W sub((m a) ) U (m a) we obtain thus:, H 2 = Λ 0 Y 1 R m (m ), G 2 = Λ 0 Y 2 R m (m ), (67) Febuay 18, 2008

22 21 whee Y 1, Y 2 ae any given matices, choice of which is ielevant up to the othogonality of G 2 and H 2. Hence, by inseting (66) and (67) into (65) we obtain that P N (A) C if and only if P N (A) is of the fom: P N (A) = N Λ 0 Y 2 V t 0 0 X1 t N Λ 0 Ĩ V 0 Y 2 Λt 0 N t = 0 X 2 Y1 t I 0 Λt 0 N t = N APPENDIX V PROOF OF LEMMA 3 Y t 1 ΛΛt 0 N t. By definition, C is the set of all best appoximations of P N (A) of ank at most in the sense of the Fobenius nom, theefoe it suffices to show that C { } ag min P N (A) P N (A) 2 P N (A) X m ι. To this end, let us ι I fistly ecall (see Section II-A) that Λ U (m a) if and only if Λ = W sub((m a) ) fo some othogonal matix W U (m a) (m a). Let us theefoe set: T = W R(m a) (m a) 0 R m m, 0 X R a a whee W, X ae any othogonal matices. Then T is an othogonal matix satisfying: T I m a 0 T t = I m a 0. Moeove, let Z = N T whee N is of the fom (31). Then, fom (32) it is clea that: P N (A) = Z I m a 0 Z t, (68) and fom Lemma 2 we obtain that all PN (A) C can be expessed as: PN (A) = Z I 0 Z t. (69) We obseve that (68) povides an SVD of P N (A) fo any choice of T. Theefoe, fom (69) and Fact I-1 (50), we obtain that C { } ag min P N (A) P N (A) 2 P N (A) X m ι, as claimed. ι I APPENDIX VI Poof: Step 1. We will show fistly that: Π ag min Π X m n PROOF OF LEMMA 4 Π L P N (A) 2 F if and only if k(π ) = and Π L P N (A) 2 F = m a. (70) Fom Remak 3 we have that m a is a minimum distance to P N (A) among all X X m, hence it is enough to pove the necessity in the above equivalence, which can be obtained fom Remak 5 as follows: since equation Febuay 18, 2008

23 22 ΠL = P N (A) has a paticula solution Π = P N (A) L Q 1/2 (fo any P N (A) C ) such that k(π ) =, 12 we see that if Π ag min Π X m n immediately fom Remak 3. Π L P N (A) 2 F then Π L P N (A) 2 F m a, and the equality follows To show the second pat, suppose that thee exists Π < ag min Π X m n Π L P N (A) 2 F such that k(π <) <, and let similaly as above set Y = Π < L. We have k(y ) < = k(p N (A) ), but Y P N (A) 2 F = m a, hence again we would violate Remak 3. Let us note that (70) can be equivalently fomulated as: Π ag min Π X m n Step 2. By definition of Π mvp Π L P N (A) 2 F if and only if k(π ) = and P N (A) C such that P N (A) = Π L. as a solution to poblem (30) and in view of Step 1, Π mvp chaacteized as a solution of the following poblem: minimize Π 2 F subject to k(π ) = and Π L = PN (A) fo some P N (A) C. (71) can be equivalently Since the function f defined in (39) assigns to each P N (A) C the unique solution of equation P N (A) = Π L of minimum Fobenius nom given by Π F (72) = P N (A) (L ) (see (38)) such that k(π F ) =, we conclude that poblem (72) is equivalent to that of finding Π mvp being a solution of: minimize Π 2 F (73) subject to Π R (f), which is equivalent to (40). APPENDIX VII PROOF OF THEOREM 1 Poof: 1. Step 1. With notation as in Lemma 4 and Theoem 1, let Π F = PN (A) (L ) R (f), and let us note that (P N (A) ) t P N (A) = P N (A) fo all P N (A) C. We have: Π F 2 F [P = t N (A) (L ) [(L ) ] t (P N (A) ) t] [ = t t N ΛΛt 0 N t (L ) [(L ) ] t = t ΛΛt 0 P N (A) (L ) [(L ) ] t] = N t (L ) [(L ) ] t N = t [ ΛΛ t K ]. (74) Moeove, if we set an SVD of L = P ΓR t and denote S = N t R, Σ = Γ (Γ ) t, then it is staightfowad to veify that N t (L ) [(L ) ] t N = SΣS t, and hence, by k(l ) = m a (see Remak 4): K = (SΣS t ) sub(m a) (m a) = S sub(m a) (m a) Σ sub(m a) (m a) S t sub(m a) (m a). Clealy, Σ sub(m a) (m a) is positive definite because it is a diagonal matix with positive diagonal enties, and theefoe by Fact I-6 K is positive semidefinite with k = k(k) = k(s sub(m a) (m a) ) = k[(n t R) sub(m a) (m a) ]. 12 Note that is a minimum ank among anks of solutions of equation ΠL = P N (A). Febuay 18, 2008

24 23 Thus, to complete the poof of positive definiteness of K, it suffices to show that k = m a, which will be demonstated in Step 4. ( ) Step 2. Let us now set W R (m a) (m a) to be any othogonal matix of the fom W = Λ?, whee? is any matix peseving othogonality of W. We have: ΛΛ t = W I 0 W t, and upon setting P P t to be an eigenvalue decomposition of K (with eigenvalues oganized in nondeceasing ode), we obtain: δ 1 0 t [ ΛΛ t K ] = t W I 0 W t P P t = t W I 0 0 δ W t 2 P... P t. (75) 0 0 δ m a We can now use Fact I-7 by substituting C = ΛΛ t = W I 0 W t and D = K = P ( )P t, which yields: Π F 2 F = t [ ΛΛ t K ] δ i, (76) with equality if and only if W t P = G t H fo evey othogonal matices G, H R (m a) (m a) commuting with I 0 ( ) and, espectively. 13 Since W = Λ?, we have thus that Λ achieving equality in (76) must be of the fom: i=1 Λ min = P H t G I, 0 and hence: Λ min Λ t min = P H t G I 0 G t HP t = P H t I 0 HP t, (77) whee the last equality is implied fom the fact that G commutes with I 0. Moeove, since H is any othogonal matix with the popety H = H, fom Fact I-8 we obseve that E = P H t is a geneal fom of othogonal matix E such that E E t nondeceasing ode, and hence: is an eigenvalue decomposition of K with eigenvalues oganized in Λ min Λ t min = E I 0 E t = E E t, and by inseting this expession of Λ min Λ t min into Π F equality in (76) we obtain (42). (see (35) and (38)) we obtain (41), and since Λ min achieves 13 Clealy, H commutes with if and only if H commutes with, which can be immediately veified fom Lemma 1. Febuay 18, 2008