Kaczmarz algorithm in Hilbert space

Size: px

Start display at page:

Download "Kaczmarz algorithm in Hilbert space"

Conrad Price
5 years ago
Views:

1 STUDIA MATHEMATICA 169 (2) (2005) Kaczmarz algorithm in Hilbert space by Rainis Haller (Tartu) and Ryszard Szwarc (Wrocław) Abstract The aim of the Kaczmarz algorithm is to reconstruct an element in a Hilbert space from data given by inner products of this element with a given sequence of vectors The main result characterizes sequences of vectors leading to reconstruction of any element in the space This generalizes some results of Kwapień and Mycielski 1 Introduction Let {e n } be a sequence of unit vectors in a Hilbert space H For a given x H we have the numbers { x, e n } We want to reconstruct x from these numbers The sequence {e n } should be linearly dense Define x 0 = x, e 0 e 0, x n = x + x x, e n e n We are interested in when x n x for any x H The sequences {e n } for which this holds will be called effective The formula is called the Kaczmarz algorithm In 1937 Kaczmarz (see [1]) considered this problem in the finite-dimensional case He proved that if dim H < and the sequence {e n } is linearly dense and periodic then it is effective Let P n be the orthogonal projection onto e n Then we have (1) x n = x + (I P n )(x x ), x x n = P n (x x ), x x n = P n P P 1 P 0 x Therefore the sequence {e n } is effective if and only if the operators P n P P 1 P 0 tend to zero strongly Since the norms of these operators are bounded it suffices to get pointwise convergence on a linearly dense subset of vectors, eg on members of the sequence {e n } 2000 Mathematics Subject Classification: Primary 41A65 Key words and phrases: Kaczmarz algorithm, Hilbert space Supported by Research Training Network Harmonic Analysis and Related Problems Contract HPRN-CT The authors were also supported by the Estonian Science Foundation, Grant no 4400 and KBN (Poland), Grant 2 P03A , respectively [123]

2 124 R Haller and R Szwarc The Kaczmarz theorem can now be proved as follows Let dim H < and {e n } be N-periodic For A = P N 1 P 1 P 0 it suffices to show that A n tends to zero We claim that A < 1 If not, there is a vector x such that Ax = x = 1 Then P 0 x Ax = x, hence P 0 x = x Similarly P 1 x = x,, P N 1 x = x, which implies that x e 0, e 1,, e N 1 Since the vectors {e n } N 1 are linearly dense we get x = 0 We now turn to the infinite-dimensional case We recall some basic properties of the algorithm which can be found in [2] By construction the vector x n is a linear combination of e 0, e 1,, e n It can be shown that (2) x n = x, g i e i, where the vectors g n are given by the recurrence relation (3) or (4) By (2) we have g 0 = e 0, g n = e n e n, e i g i m ni g i = e n, m ni = e n, e i x x = x x n + x, g n e n Since by (1) the vectors x x n and e n are orthogonal we get x 2 = x x x, g 0 2, x x 2 = x x n 2 + x, g n 2, n 1 Summing up these equalities gives x 2 lim n x x n 2 = x, g n 2 Therefore the sequence {e n } is effective if and only if (5) x 2 = x, g n 2 for any x H This equation means that {g n } is a tight frame with constant 1 as was already mentioned in [2] We have noticed before that it suffices to check formula (5) on vectors {e n }, provided they form a linearly dense subset in H

3 Kaczmarz algorithm in Hilbert space Characterization of effective sequences The formula (3) implies that g n is a linear combination of the vectors e 0, e 1,, e n, ie (6) c ni e i + e n = g n for some coefficients c nj By (4) these coefficients can be obtained by taking the algebraic inverse matrix to the lower triangular matrix I + M where m M = m 20 m m 30 m 31 m , m ij = e i, e j Namely (I + M) 1 = I + U, where c U = c 20 c c 30 c 31 c The matrix U and the coefficients c ni will play a crucial role in all what follows Since the matrices I + M and I + U are inverse to each other we get (7) (8) UM = MU = M U, U M = M U = M U Our first simple result shows that the matrix U is a contraction in the Hilbert space l 2 (N 0 ) Proposition 1 Let U and M be strictly lower triangular matrices such that MU = UM = U M Then U is a contraction if and only if the matrix M + M + I is positive definite In that case there is a Hilbert space H and vectors {e n } in H such that M + M + I is the Gram matrix of these vectors Proof Let M n and U n denote the truncated matrices given by 0 0 m 10 0 c M n = m n0 m n, 0, U n = c n0 c n,

4 126 R Haller and R Szwarc Then M n and U n are bounded on l 2 (N 0 ) and by assumption M n U n = U n M n = U n M n Assume the matrix M + M + I is positive definite Then the matrix M n + M n + I corresponds to a positive bounded operator on l 2 (N 0 ) Thus 0 (U n + I)(M n + M n + I)(U n + I) = I U nu n Hence U n 1, where denotes the operator norm Consequently, we obtain U 1 The converse implication follows from (M n + I)(I U nu n )(M n + I) = M n + M n + I Indeed, if U 1 then U n 1 Therefore the matrix M n + Mn + I is positive definite, which implies that so also is M + M + I It is then well known that there exist a Hilbert space H and vectors {e n } such that m ij = e i, e j, i > j Now we can state the main result of our paper Theorem 1 The sequence {e n } is effective if and only if it is linearly dense and U is a partial isometry, ie U U is an orthogonal projection Proof Assume {e n } is effective By (5) and by the polar identity we get (9) x, y = x, g n g n, y for any x, y H In particular (10) m ij = e i, e j = e i, g n g n, e j We want to state the formula (10) in terms of matrices on l 2 (N 0 ) Let δ i denote the sequence in l 2 (N 0 ) whose ith entry is 1 and all other entries are 0 We have the following Lemma 1 g n, e j = (UM + M + I)δ j, δ n l 2 (N 0 ) Proof of Lemma 1 Set c nn = 1 Then by (6) we have c ni m ij for j > n, g n, e j = j 1 c ni m ij + c ni m ij for j n i=j

5 Kaczmarz algorithm in Hilbert space 127 Since (I + U)(I + M) = I we get c ni m ij = δ jn, for j n Therefore g n, e j = j 1 m ji δ i, i=j c ni δ i l + δ j, δ n l 2 2 = M δ j, (U + I)δ n l 2 + δ j, δ n l 2 = (UM + M + I)δ j, δ n l 2 Let A = UM + M + I Applying Lemma 1 to (10) and using the Parseval identity gives (11) m ij = Aδ j, δ n δ n, Aδ i l 2 = Aδ j, Aδ i l 2 Let A n = U n Mn +Mn +I Unlike A, the matrices A n correspond to bounded operators on l 2 (N 0 ) Since Mnδ k = M n δ k for n k and U n U strongly we have (12) lim n A na n δ j, δ i l 2 = lim A nδ j, A n δ i l 2 = Aδ j, Aδ i l 2 n On the other hand, the relation M n U n = U n M n implies Hence A na n = M n U nu n M n M n (U n + M n) + M n U n (U n + M n )M n + M n M n + M n + U n M n + M n + I = M n + M n + I + M n U nu n M n M n M n (13) lim n A na n δ j, δ i l 2 = m ij + (UM δ j, UM δ i ) l 2 (M δ j, M δ i ) l 2 Combining (11) (13) yields (14) (UM δ j, UM δ i ) l 2 = (M δ j, M δ i ) l 2 Let F(N 0 ) = span {δ 0, δ 1, } Formula (14) states that the operator U is isometric on H 0 = M (F(N 0 )) It suffices to show that U vanishes on H0 To this end observe that the matrices U and M leave the subspace F(N 0 ) invariant The formula M (U +I) = U implies that U (F(N 0 )) H 0 Taking orthogonal complements of both sides results in H 0 ker U, which completes the proof that U is a partial isometry

6 128 R Haller and R Szwarc Conversely, let U be a partial isometry Hence U is isometric on H 0 = U (F(N 0 )) The formula U (M + I) = M implies that U is isometric on M (F(N 0 )), which is equivalent to (14) Now tracking backwards the proof of the first part implies the formula (10) In particular for i = j we obtain e i 2 = e i, g n 2 for any i 0 This implies effectivity (see comments at the end of the Introduction) Remark 1 Theorem 1 can be used to construct examples of effective sequences of vectors It suffices to come up with a strictly lower triangular partial isometry U Then one has to compute the algebraic inverse I + M of I + U As a result a positive definite matrix M + M + I is constructed The corresponding vectors form a sequence effective in the closed linear span of these vectors Remark 2 Proposition 1 and Theorem 1 can be interpreted as follows There are as many effective sequences among sequences of unit vectors as there are partial isometries among strictly lower triangular contractions on l 2 (N 0 ) This can be compared with a result of Kwapień and Mycielski who showed that if we choose the sequence of unit vectors at random then almost surely we end up with an effective sequence More precisely, fix a probability Borel measure µ on the unit sphere of H, such that the support of µ is linearly dense Then drawing consecutive vectors independently with respect to that measure almost surely yields an effective sequence of vectors 3 Strongly effective sequences For a partial isometry U the carrier space, ie Im U, may vary from the trivial space to the whole space H For instance when the system {e n } is orthonormal we get M = U = 0 The other extreme case is when Im U = H, which is equivalent to U U = I While proving Theorem 1 we showed that for an effective sequence {e n } the carrier space for U coincides with M (F(N 0 )) Hence U U = I if and only if M (F(N 0 )) = l 2 (N 0 ) This occurs exactly when the rows of the matrix M form a linearly dense subset of l 2 (N 0 ) For example this is the case when m n+1,n 0 in the matrix 0 m 10 0 m 21 0 m 32 0

7 Kaczmarz algorithm in Hilbert space 129 It turns out that sequences {e n } with U U = I have many interesting properties Let (15) v in = c nj e j, n > i j=i The next lemma is related to a formula obtained in the proof of Theorem 2 of [2] Lemma 2 For 0 i, j < n the following equalities hold: v in, v jn = (I U nu n )δ j, δ i l 2 Proof Without loss of generality we may assume that j i Then using the fact that I + M and I + U are inverse to each other we get v in, v jn = c ki e k, c lj e l = m kl c lj = m kl c lj k=i = k c ki k=i l=j = c ki δ jk + k=i l=j m kl c lj + l=i+1 = (I U nu n )δ j, δ i l 2 c ki k=i l=j c ki k=i l=k+1 c lj l 1 m kl c lj m lk c ki = δ ij k=i l=i+1 c ki k=i l=j c lj c li The next corollary should be compared with Remark 2 of [2] Corollary 1 Assume U U = I Then for any j 0, (16) e j = c ij e i i=j+1 Proof By Lemma 2 we get v jn 0 as n Hence c ij e i = 0 i=j Since c jj = 1 we get the conclusion Definition 1 The sequence {e n } will be called strongly effective if {e n } n=k is effective for each k 0 In particular dropping finitely many vectors from {e n } does not spoil linear density Hence this is a highly nonorthogonal case Theorem 2 Assume the sequence {e n } is linearly dense in a Hilbert space H Then {e n } is strongly effective if and only if U U = I

8 130 R Haller and R Szwarc Proof By Corollary 1 for any k the sequence {e n } n=k is linearly dense Let M (k) and U (k) denote truncated matrices obtained by removing the first k rows and the first k columns from M and U, respectively These matrices correspond to the sequence {e n } n=k Also U U = I implies (U (k) ) U (k) = I for any k Hence U (k) is a partial isometry Now we can use Theorem 1 to get the conclusion Conversely, suppose that {e n } is strongly effective Let Q k denote the orthogonal projection from l 2 (N) onto the orthogonal complement of {δ 0, δ 1,, δ k 1 } Let U (k) = UQ k Then U (k) = 0 k U (k), where 0 k denotes the k k zero matrix Hence U (k) are partial isometries just as U is But this is possible only if U U and Q k commute On the other hand, if U U commutes with Q k for any k then U U must be diagonal Assume that U U I Then Uδ j = 0 for some j This implies that Mδ j = 0 and consequently e j is orthogonal to all the vectors e i, i > j Hence {e n } cannot be strongly effective (17) Lemma 3 Assume U U = I Then U M = M I, U (M + M + I) = (M + M + I) Proof By taking the inner product with e k in (16) we obtain m jk = c ij m ik for any k 0 Let k j Then (18) For k > j we get (19) 0 = m jk = i=j+1 c ij m ik = i=j i=j+1 c ij m ik = (U M) jk k c ij m ki + i=j i=k+1 c ij m ik = ((M + I)U) jk + (U M) jk = (U M) jk Combining (18) and (19) gives the first equality The second equality now follows by applying U M = U M Theorem 3 Assume the matrix U associated with the sequence {e n } satisfies U U = I Then e i, e j 2 =

9 Kaczmarz algorithm in Hilbert space 131 for any j In particular the conclusion holds if the sequence {e n } is effective and the rows of M are linearly dense in l 2 (N 0 ) Proof Let G = M + M + I Observe that G is nothing other than the Gram matrix of the vectors {e n } Assume that e i, e j 2 < for some j Then v = Gδ j l 2 (N 0 ) By (17) we get U v = v and (U ) n v = ( 1) n v But U 1 and U is strictly upper triangular Hence (U ) n tends to zero strongly, which implies v = 0 This gives a contradiction because v(j) = 1 4 Stationary case Assume e i+1, e j+1 = e i, e j Then the matrix M is constant on diagonals: 0 a 1 0 M = a 2 a 1 0 a 3 a 2 a 1 0 By the Herglotz theorem there is a measure µ on the unit circle such that e i+n, e i = a n = T z n dµ(z) Kwapień and Mycielski showed that the sequence {e n } is effective if and only if either µ is the Lebesgue measure (orthogonal case) or it is singular with respect to the Lebesgue measure We now reprove this result by applying our Theorem 1 Also U is constant on diagonals, ie it is a Toeplitz operator, 0 u 1 0 U = u 2 u 1 0 u 3 u 2 u 1 0 It is then unitarily equivalent to the multiplication operator on H 2 (T) with the function u(z) = u n z n n=1

10 132 R Haller and R Szwarc Moreover U = u(z) H (T) 1 Now multiplication with u(z) is a partial isometry if and only if the boundary values of u(z) are 0 or 1 By the F Riesz and M Riesz theorem, u(e iθ ) 0 or u(e iθ ) 1 The first case corresponds to the orthogonal case, because M = 0 The second case is equivalent to the singularity of the spectral measure µ Indeed, for a(z) = n=1 a nz n we have (a(z) + 1)(u(z) + 1) = 1 Therefore 1 + a(z) + a(z) = 1 u(z) 2 u(z) By the Fatou theorem lim r 1 [1 + a(reiθ ) + a(re iθ )] = dµ dθ Therefore µ is singular if and only if u(e iθ ) 1 almost everywhere Acknowledgements The first named author wishes to acknowledge that this work was done during his fellowship stay at the University of Wrocław in He would like to thank warmly Prof E Damek and Prof R Szwarc for their hospitality References [1] S Kaczmarz, Approximate solution of systems of linear equations, Bull Acad Polon Sci Lett A 35 (1937), (in German); English transl: Internat J Control 57 (1993), [2] S Kwapień and J Mycielski, On the Kaczmarz algorithm of approximation in infinitedimensional spaces, Studia Math 148 (2001), Institute of Pure Mathematics Faculty of Mathematics and Computer Science University of Tartu Ülikooli Tartu, Estonia RainisHaller@utee Institute of Mathematics University of Wrocław Pl Grunwaldzki 2/ Wrocław, Poland szwarc@mathuniwrocpl Received July 21, 2004 Revised version February 24, 2005 (5460)

Recall that any inner product space V has an associated norm defined by

Hilbert Spaces Recall that any inner product space V has an associated norm defined by v = v v. Thus an inner product space can be viewed as a special kind of normed vector space. In particular every inner