Dynamical Systems Method for Solving Ill-conditioned Linear Algebraic Systems

Transcription

1 Dynamical Systems Method for Solving Ill-conditioned Linear Algebraic Systems Sapto W. Indratno Department of Mathematics Kansas State University, Manhattan, KS , USA A G Ramm Department of Mathematics Kansas State University, Manhattan, KS , USA ramm@math.ksu.edu Abstract A new method, the Dynamical Systems Method (DSM), justified recently, is applied to solving ill-conditioned linear algebraic system (ICLAS). The DSM gives a new approach to solving a wide class of ill-posed problems. In this paper a new iterative scheme for solving ICLAS is proposed. This iterative scheme is based on the DSM solution. An a posteriori stopping rules for the proposed method is justified. This paper also gives an a posteriori stopping rule for a modified iterative scheme developed in A.G.Ramm, JMAA,33 (7), , and proves convergence of the solution obtained by the iterative scheme. MSC: 15A1; 47A5; 65F5; 65F Keywords: Hilbert matrix, Fredholm integral euations of the first kind, iterative regularization, variational regularization, discrepancy principle, Dynamical Systems Method Biographical notes: Professor Alexander G. Ramm is an author of more than 58 papers, patents, 1 monographs, an editor of 3 books, and an associate editor of several mathematics and computational mathematics Journals. He gave more than 135 addresses at various Conferences, visited many Universities in Europe, Africa, America, Asia, and Australia. He won Khwarizmi Award in Mathematics, was Mercator Professor, Distinguished Visiting Professor supported by the Royal Academy of Engineering, invited plenary speaker at the Seventh PanAfrican Congress of Mathematicians, a London Mathematical Society speaker, distinguished HKSTAM speaker, CNRS research professor, Fulbright professor in Israel, distinguished Foreign professor in Mexico and 1

2 Egypt. His research interests include inverse and ill-posed problems, scattering theory, wave propagation, mathematical physics, differential and integral euations, functional analysis, nonlinear analysis, theoretical numerical analysis, signal processing, applied mathematics and operator theory. Sapto W. Indratno is currently a PhD student at Kansas State University under the supervision of Prof. Alexander G. Ramm. He is a coauthor of three accepted papers. His fields of interest are numerical analysis, optimization, stochastic processes, inverse and ill-posed problems, scattering theory, differential euations and applied mathematics. 1 Introduction We consider a linear euation Au = f, (1) where A : R m R m, and assume that euation (1) has a solution, possibly nonuniue. According to Hadamard [9, p.9], problem (1) is called well-posed if the operator A is injective, surjective, and A 1 is continuous. Problem (1) is called ill-posed if it is not well-posed. Ill-conditioned linear algebraic systems arise as discretizations of ill-posed problems, such as Fredholm integral euations of the first kind, b a k(x, t)u(t)dt = f(x), c x d, () where k(x, t) is a smooth kernel. Therefore, it is of interest to develop a method for solving ill-conditioned linear algebraic systems stably. In this paper we give a method for solving linear algebraic systems (1) with an ill-conditioned-matrix A. The matrix A is called ill-conditioned if κ(a) >> 1, where κ(a) := A A 1 is the condition number of A. If the null-space of A, N (A) := {u : Au = }, is nontrivial, then κ(a) =. Let A = UΣV be the singular value decomposition (SVD) of A, UU = U U = I, V V = V V = I, and Σ = diag(σ 1, σ,..., σ m ), where σ 1 σ σ m are the singular values of A. Applying this SVD to the matrix A in (1), one gets f = i β i u i and y = i,σ i> β i σ i v i, (3) where β i = u i, f. Here, denotes the inner product of two vectors. The terms with small singular values σ i in (3) cause instability of the solution, because the coefficients β i are known with errors. This difficulty is essential when one deals with an ill-conditioned matrix A. Therefore a regularization is needed for solving ill-conditioned linear algebraic system (1). There are many methods to solve (1) stably: variational regularization, uasisolutions, iterative regularization (see e.g, [], [4], [6], [9]). The method proposed in this paper is based on the Dynamical Systems Method (DSM) developed in [9, p.76]. The DSM for

3 solving euation (1) with, possibly, nonlinear operator A consists of solving the Cauchy problem u(t) = Φ(t, u(t)), u() = u ; u(t) := du dt, (4) where u H is an arbitrary element of a Hilbert space H, and Φ is some nonlinearity, chosen so that the following three conditions hold: a) there exists a uniue solution u(t) t, b) there exists u( ), and c) Au( ) = f. In this paper we choose Φ(t, u(t)) = (A A + a(t)i) 1 f u(t) and consider the following Cauchy problem: u a (t) = u a (t) + [A A + a(t)i m ] 1 A f, u a () = u, (5) where a(t) >, and a(t) as t, (6) A is the adjoint matrix and I n is an m m identity matrix. The initial element u in (5) can be chosen arbitrarily in N(A), where N (A) := {u Au = }. (7) For example, one may take u = in (5) and then the uniue solution to (5) with u() = has the form u(t) = t e (t s) T 1 a(s) A fds, (8) where T := A A, T a := T + ai, I is the identity operator. In the case of noisy data we replace the exact data f with the noisy data f δ in (8), i.e., u δ (t) = tδ e (t δ s) T 1 a(s) A f δ ds, (9) where t δ is the stopping time which will be discussed later. There are many ways to solve the Cauchy problem (5). For example, one may apply a family of Runge-Kutta methods for solving (5). Numerically, the Runge-Kutta methods reuire an appropriate stepsize to get an accurate and stable solution. Usually the stepsizes have to be chosen sufficiently small to get such a solution. The number of steps will increase when t δ, the stopping time, increases, see []. Therefore the computation time will increase significantly. Since lim δ t δ =, as was proved in [9], the family of the Runge-Kutta method may be less efficient for solving the Cauchy problem (5) than the method, proposed in this paper. We give a simple iterative scheme, based on the DSM, which produces stable solution to euation (1). The novel points in our paper are iterative schemes (1) and (13) (see below), which are constructed on the basis of formulas (8) and (9), and a modification of the iterative scheme given in [8]. Our stopping rule for the iterative scheme (13) is given in (85) (see below). In [9, p.76] the function a(t) is assumed to be a slowly decaying monotone function. In this 3

4 paper instead of using the slowly decaying continuous function a(t) we use the following piecewise-constant function: a (n) (t) = α j+1 χ (tj,t j+1](t), (, 1), t j = j ln(), n N, (1) j= where N is the set of positive integer, t =, α >, and χ (tj,t j+1](t) = { 1, t (tj, t j+1 ];, otherwise. (11) The parameter α in (1) is chosen so that assumption (17) (see below) holds. This assumption plays an important role in the proposed iterative scheme. Definition (1) allows one to write (8) in the form u n+1 = u n + (1 )T 1 α n+1 A f, u =. (1) A detailed derivation of the iterative scheme (1) is given in Section. When the data f are contaminated by some noise, we use f δ in place of f in (8), and get the iterative scheme We always assume that u δ n+1 = u δ n + (1 )T 1 α n+1 A f δ, u δ =. (13) f δ f δ, (14) where f δ are the noisy data, which are known, while f is unknown, and δ is the level of noise. Here and throughout this paper the notation z denotes the l -norm of the vector z R m. In this paper a discrepancy type principle (DP) is proposed to choose the stopping index of iteration (13). This DP is based on discrepancy principle for the DSM developed in [11], where the stopping time t δ is obtained by solving the following nonlinear euation tδ e (t δ s) a(s) Q 1 a(s) f δ ds = Cδ, C (1, ]. (15) It is a non-trivial task to obtain the stopping time t δ satisfying (15). In this paper we propose a discrepancy type principle based on (15) which can be easily implemented numerically: iterative scheme (13) is stopped at the first integer n δ satisfying the ineualities: n δ 1 j= ( n δ j 1 n j )α j+1 Q 1 α j+1 f δ ds Cδ ε < ( n j 1 n j )α j+1 Q 1 α f j+1 δ, 1 n < n δ, j= (16) 4

5 and it is assumed that (1 )α Q 1 α f δ Cδ ε, C > 1, ε (, 1), α >. (17) We prove in Section that using discrepancy-type principle (16), one gets the convergence: lim δ uδ n δ y =, (18) where u δ n is defined in (13). About other versions of discrepancy principles for DSM we refer the reader to [6],[1]. In this paper we assume that A is bounded. If the operator A is unbounded then f δ may not belong to the domain of A. In this case the expression A f δ is not defined. In [7], [8] and [1] solving (1) with unbounded operators is discussed. In these papers the unbounded operator A is assumed to be linear, closed, densely defined operator in a Hilbert space. Under these assumptions one may use the operator A (AA +ai) 1 in place of Ta 1 A. This operator is defined for any f in the Hilbert space. In [8] an iterative scheme with a constant regularization parameter is given: u δ n+1 = ata 1 u δ n + Ta 1 A f δ, (19) but the stopping rule, which produces a stable solution of euation (1) by this iterative scheme, has not been discussed in [8]. In this paper the constant regularization parameter a in iterative scheme (19) is replaced with the geometric series {α n } n=1, α >, (, 1), i.e. u δ n+1 = α n T 1 α nuδ n + T 1 α na f δ. () Stopping rule (85) (see below) is used for this iterative scheme. Without loss of generality we use α = 1 in (). The convergence analysis of this iterative scheme is presented in Section 3. In Section 4 some numerical experiments are given to illustrate the efficiency of the proposed methods. Derivation of the proposed method In this section we give a detailed derivation of iterative schemes (1) and (13). Let us denote by y R m the uniue minimal-norm solution of euation (1). Throughout this paper we denote T a(t) := A A+a(t)I m, where I m is the identity operator in R m, and a(t) is given in (1). Lemma.1. Let g(x) be a continuous function on (, ), c > and (, 1) be constants. If lim g(x) = g() := g, (1) x + then lim n j=1 ( n j 1 n j) g(c j+1 ) = g. () 5

6 Proof. Let and Then ω j (n) := n j n+1 j, ω j (n) >, (3) l 1 F l (n) := ω j (n)g(c j ). (4) j=1 n F n+1 (n) g F l (n) + ω j (n)g(c j ) g. Take ɛ > arbitrary small. For sufficiently large l(ɛ) one can choose n(ɛ), such that F l(ɛ) (n) ɛ, n > n(ɛ), j=l because lim n n =. Fix l = l(ɛ) such that g(c j ) g ɛ This is possible because of (1). One has for j > l(ɛ). and n j=l(ɛ) ω j (n)g(c j ) g F l(ɛ) (n) ɛ, n > n(ɛ) ɛ n n ω j (n) g(c j ) g + ω j (n) 1 g j=l(ɛ) n j=l(ɛ) ω j (n) + n l(ɛ) g ɛ + g n l(ɛ) ɛ, if n is sufficiently large. Here we have used the relation n ω j (n) = 1 n+1 l. j=l Since ɛ > is arbitrarily small, Lemma.1 is proved. j=l(ɛ) Let us define u n := tn e (tn s) T 1 a (n) (s) A fds, t n = n ln(), (, 1). (5) 6

7 Note that u n = t tn e (tn s) T 1 a (n) (s) A fds + e (tn s) T 1 a (n) (s) A fds t t = e (tn t) e (t s) T 1 a(s) A fds + = e (tn t) u + tn Using definition (1), one gets tn t e (tn s) T 1 a (n) (s) A fds. t e (tn s) T 1 a (n) (s) A fds u n = e (tn t) u + [1 e (tn t) ]T 1 α na f = n u + (1 n 1 )T α na f. Therefore, (5) can be rewritten as iterative scheme (1). Lemma.. Let u n be defined in (1) and Ay = f. Then u n y n ( y + n j 1 n j) α j+1 T 1 α y, n 1, (6) j+1 and j= Proof. By definitions (5) and (1) we obtain tn ( u n = e (tn s) T 1 n a(s) A fds = u n y as n. (7) j= From (8) and the euation Ay = f, one gets: ( ) n n u n = j+1 j T 1 α A f j+1 j= ( n = j= j= n j+1 j n j+1 j ) T 1 α j+1 (T α j+1 α j+1 I m )y j= ) T 1 α j+1 A f. (8) ( = n j 1 n j) ( y n j 1 n j) α j+1 T 1 α y j+1 = y n ( y n j 1 n j) α j+1 T 1 α y. j+1 j= Thus, estimate (6) follows. To prove (7), we apply Lemma.1 with g(a) := y. Since y N (A), it follows from the spectral theorem that a T 1 a a lim a g (a) = lim a (a + s) d E sy, y = P N (A) y =, 7

8 where E s is the resolution of the identity corresponding to A A, and P is the orthogonal projector onto N (A). Thus, by Lemma.1, (7) follows. Let us discuss iterative scheme (13). The following lemma gives the estimate of the difference of the solutions u δ n and u n. Lemma.3. Let u n and u δ n be defined in (1) and (13), respectively. Then u δ n u n 1 w n, n, (9) 3/ where w n := (1 ) δ α. n Proof. Let H n := u δ n u n. Then from the definitions of u δ n and u n we get the estimate H n+1 u δ n u n + (1 ) T 1 α n+1 A (f δ f) H n + w n. (3) Let us prove ineuality (9) by induction. For n = one has u = u δ =, so (9) holds. For n = 1 one has u δ δ 1 u 1 (1 ), so (9) holds. If (9) α holds for n k, then for n = k + 1 one has ( ) 3/ H k+1 H k + w k w 3/ k = 1 w 3/ k (31) 1 w k 1 = w 1 3/ k+1 wk+1. w k+1 1 3/ Hence (9) is proved for n..1 Stopping criterion In this section we give a stopping rule for iterative scheme given in (13). Let Q := AA, Q a := Q + ai m, and G n := tn e (tn s) a(s) Q 1 a(s) f δ ds = ( n j 1 n j )α j+1 Q 1 α f j+1 δ, n 1, j= (3) where t n = n ln, (, 1) and α >. Then stopping rule (16) can be rewritten as G nδ Cδ ε < G n, 1 n < n δ, ε (, 1), C > 1, G 1 > Cδ ε. (33) 8

9 Note that n G n+1 = ( n j n+1 j )α j+1 Q 1 α f j+1 δ so j= = ( n j n+1 j )α j+1 Q 1 α f j+1 δ + (1 )α n+1 Q 1 α f n+1 δ j= = G n + (1 )α n+1 Q 1 α n+1 f δ, G n = G + (1 )α n Q 1 α nf δ, n 1, G =. (34) Lemma.4. Let G n be defined in (34). Then G n Proof. Using the identity 1 1 α n y + δ, n 1, (, 1). (35) aq 1 a the estimates = AT 1 a A I m, a >, T := A A, T a := T + ai m, a Q 1 a 1, f δ f δ, and a a ATa 1, where Q := AA, Q a := Q + ai m, we get G n = G + (1 ) AT 1 α na f δ f δ = G + (1 ) AA Q 1 α nf δ f δ = G + (1 ) (AA + α n I α n I)Q 1 n f δ f δ = G + (1 )α n Q 1 α nf δ = G + (1 )α n Q 1 α n(f δ f + f) G + (1 )α n Q 1 α n(f δ f) + (1 )α n Q 1 α nf G + (1 )δ + (1 ) AT 1 α na f f = G + (1 )δ + (1 ) A(T 1 α na Ay y) = G + (1 )δ + (1 ) A( α n T 1 α ny) = G + (1 )δ + (1 )α n ATα 1 ny G + (1 )δ + (1 )α n y α n = G + (1 )δ + (1 ) α n y = G + (1 )δ + α y. (36) 9

10 Therefore, G n δ (G δ) + α y, n 1, G =. (37) Let us prove relation (35) by induction. From relation (37) we get α G 1 δ δ + y δ α 1 y 1 α y. (38) Thus, for n = 1 relation (35) holds. Suppose that 1 G n δ 1 α n y, 1 n k. (39) Then by ineualities (37) and (39) we obtain G k+1 δ (G k δ) + α k y 1 1 α k y + α k y = 1 α k y = 1 α k α k+1 y α k α k+1 y. (4) Thus, relation (35) is proved. Lemma.5. Let G n be defined in (34), (, 1), and α > be chosen such that G 1 > Cδ ε, ε (, 1), C > 1. Then there exists a uniue integer n c such that G nc 1 < G nc and G nc > G nc+1, n c 1. (41) Moreover, Proof. From Lemma.4 we have G n G n+1 < G n, n n c. (4) 1 1 α n y + δ, n 1, (, 1). Since G 1 > Cδ ε and lim sup n G n δ < Cδ ε, it follows that there exists an integer n c 1 such that G nc 1 < G nc and G nc > G nc+1. Let us prove the monotonicity of G n, for n n c. We have G nc+1 G nc <. Using definition (34), we get G nc+1 G nc = G nc + (1 )α nc+1 Q 1 α f nc+1 δ G nc ( ) = (1 ) α nc+1 Q 1 α f nc+1 δ G nc <. (43) 1

11 This implies Note that α nc+1 α Q 1 nc+1 f δ G nc <. (44) G n+1 G n = (1 )(α n+1 Q 1 α n+1 f δ G n ). Therefore, to prove the monotonicity of G n for n n c, one needs to prove the ineuality α n+1 Q 1 α n+1 f δ G n <, n n c. This ineuality is a conseuence of the following lemma: Lemma.6. Let G n be defined in (34), and (44) holds. Then α n+1 Q 1 α n+1 f δ G n <, n n c. (45) Proof. Let us prove Lemma.6 by induction. Let and D n := α n+1 Q 1 α n+1 f δ G n h(a) := a Q 1 a f δ. The function h(a) is a monotonically growing function of a, a >. Indeed, by the spectral theorem, we get h(a 1 ) = a 1 Q 1 a 1 f δ = a 1 (a 1 + s) d F sf δ, f δ a (a + s) d F sf δ, f δ = a Q 1 a f δ = h(a ), (46) where F s is the resolution of the identity corresponding to Q := AA, because a 1 a (a 1+s) (a +s) if < a 1 < a and s. By the assumption we have D nc = α nc+1 Q 1 α f nc+1 δ G nc <. Thus, relation (45) holds for n = n c. For n = n c + 1 we get D nc+1 = α nc+ Q 1 α nc+ f δ (1 )α nc+1 Q 1 α nc+1 f δ G nc = h(α nc+ ) h(α nc+1 ) + h(α nc+1 ) G nc = h(α nc+ ) h(α nc+1 ) + ( h(α nc+1 ) G nc ) (47) = h(α nc+ ) h(α nc+1 ) + D nc = h(α nc+ ) h(α nc+1 ) + D nc <. Here we have used the monotonicity of the function h(a). Thus, relation (45) holds for n = n c + 1. Suppose D n <, n c n n c + k 1. 11

12 This, together with the monotonically growth of the function h(a) := a Q 1 f δ, yields D nc+k = α nc+k+1 Q 1 α f nc+k+1 δ G nc+k = h(α nc+k+1 ) (1 ) h(α nc+k ) G nc+k 1 = h(α nc+k+1 ) h(α nc+k ) + ( h(α nc+k ) G nc+k 1) = h(α nc+k+1 ) h(α nc+k ) + D nc+k 1 = h(α nc+k+1 ) h(α nc+k ) + D nc+k 1 <. (48) Thus, D n <, n 1. Lemma.6 is proved. Let us continue with the proof of Lemma.5. From relation (34) we have G n+1 G n = ( 1)G n + (1 )α n+1 Q 1 α f n+1 δ ( ) = (1 ) α n+1 Q 1 α f n+1 δ G n. Using assumption (44) and applying Lemma.6, one gets G n+1 G n <, n n c. Let us prove that the integer n c is uniue. Suppose there exists another integer n d such that G nd 1 < G nd and G nd > G nd +1. One may assume without loss of generality that n c < n d. Since G n > G n+1, n n c, and n c < n d, it follows that G nd 1 > G nd. This contradicts the assumption G nd 1 < G nd. Thus, the integer n c is uniue. Lemma.5 is proved. Lemma.7. Let G n be defined in (34). If α is chosen such that relations G 1 > Cδ ε, C > 1, ε (, 1), holds then there exists a uniue n δ satisfying ineuality (33). Proof. Let us show that there exists an integer n δ so that ineuality (33) holds. Applying Lemma.4, one gets lim sup G n δ. (49) n Since G 1 > Cδ ε and lim sup n G n δ < Cδ ε, it follows that there exists an index n δ satisfying stopping rule (33). The uniueness of the index n δ follows from the monotonicity of G n, see Lemma.5. Thus, Lemma.7 is proved. Lemma.8. Let Ay = f, y N (A), and n δ be chosen by rule (33). Then so lim δ n δ =, (, 1), (5) lim n δ =. (51) δ 1

13 Proof. From rule (33) and relation (34) we have Cδ ε + (1 )α n δ Q 1 α n δ f δ < G nδ 1 + (1 )α n δ Q 1 α n δ f δ = G nδ Cδ ε, (5) so (1 )α n δ Q 1 α n δ f δ (1 )Cδ ε. (53) Thus, α n δ Q 1 α n δ f δ < Cδ ε. (54) Note that if f then there exists a λ > such that F λ f, F λ f, f := ξ >, (55) where ξ is a constant which does not depend on δ, and F s is the resolution of the identity corresponding to the operator Q := AA. Let For a fixed number c 1 > we obtain h(δ, c 1 ) = c 1 Q c1 f δ = h(δ, α) := α Q 1 α f δ. c 1 (c 1 + s) d F sf δ, f δ λ c 1 (c 1 + s) d F sf δ, f δ c λ 1 (c 1 + λ ) d F s f δ, f δ = c 1 F λ f δ (c 1 + λ ), δ >. (56) Since F λ is a continuous operator, and f f δ < δ, it follows from (55) that Therefore, for the fixed number c 1 > we get lim F λ f δ, f δ = F λ f, f >. (57) δ h(δ, c 1 ) c > (58) for all sufficiently small δ >, where c is a constant which does not depend on δ. For example one may take c = ξ provided that (55) holds. Let us derive from estimate (54) and the relation (58) that n δ as δ. From (54) we have h(δ, α n δ ) (Cδ ε ). Therefore, lim h(δ, α n δ ) =. (59) δ Suppose lim δ nδ. Then there exists a subseuence δ j such that where c 1 is a constant. By (58) we get α n δ j c1 >, (6) h(δ j, α n δ j ) > c >, δ j as j. (61) This contradicts relation (59). Thus, lim δ n δ =. Lemma.8 is proved. 13

14 Lemma.9. Let n δ be chosen by rule (33). Then δ α n δ as δ. (6) Proof. Relation (35), together with stopping rule (33), implies Cδ ε 1 < G nδ 1 1 α n δ 1 y + δ. (63) Then This yields 1 α y n δ 1 (1 )δ ε, ε (, 1). (64) (C 1) lim δ δ α n δ = lim δ Lemma.9 is proved. δ α n δ 1 lim δ δ 1 ε (1 y =. (65) )(C 1) Theorem.1. Let y N (A), f δ f δ, f δ > Cδ ε, C > 1, ε (, 1). Suppose n δ is chosen by rule (33). Then where u δ n is given in (13). lim δ uδ n δ y =, (66) Proof. Using Lemma. and Lemma.3, we get the estimate u δ n δ y u δ n δ u nδ + u nδ y 1 (1 ) δ 3/ α n + u nδ y δ := I 1 + I, δ (67) where I 1 := (1 ) 1 3/ α and I n δ := u nδ y. Applying Lemma.9, one gets lim δ I 1 =. Since n δ as δ, it follows from Lemma. that lim δ I =. Thus, lim δ u δ n δ y =. Theorem.1 is proved. The algorithm based on the proposed method can be stated as follows: Step 1. Assume that (14) holds. Choose C (1, ) and ε (.9, 1). Fix (, 1), and choose α > so that (17) holds. Set n = 1, and u =. Step. Use iterative scheme (13) to calculate u n. Step 3. Calculate G n, where G n is defined in (34). Step 4. If G n Cδ ε then stop the iteration, set n δ = n, and take u δ n δ as the approximate solution. Otherwise set n = n + 1, and go to Step 1. 14

15 3 Iterative scheme In [8] the following iterative scheme for the exact data f is given: u n+1 = ata 1 u n + Ta 1 A f, u 1 = u 1 N (A), (68) where a is a fixed positive constant. It is proved in [8] that iterative scheme (68) gives the relation lim u n y =, n y N (A). In the case of noisy data the exact data f in (68) is replaced with the noisy data f δ, i.e. u δ n+1 = ata 1 u δ n + Ta 1 A f δ, u 1 = u 1 N (A), (69) where f δ f δ for sufficiently small δ >. It is proved in [8] that there exist an integer n δ such that lim δ uδ n δ y =, (7) where u δ n is the approximate solution corresponds to the noisy data. But a method of choosing the integer n δ has not been discussed. In this section we modify iterative scheme (68) by replacing the constant parameter a in (68) with a geometric seuence { } n=1, i.e. u n+1 = n T 1 u n n + T 1 n A f, u 1 =, (71) where (, 1). The initial approximation u 1 is chosen to be. In general one may choose an arbitrary initial approximation u 1 in the set N (A). If the data are noisy then the exact data f in (71) is replaced with the noisy data f δ, and iterative scheme (69) is replaced with: u δ n+1 = n T 1 n uδ n + T 1 n A f δ, u δ 1 =. (7) We prove convergence of the solution obtained by iterative scheme (71) in Theorem 3.1 for arbitrary (, 1), i.e. lim u n y =, (, 1). n In the case of noisy data we use discrepancy-type principle (85) to obtain the integer n δ such that lim δ uδ n δ y =. (73) We prove relation (73), for arbitrary (, 1), in Theorem 3.6. Let us prove that the seuence u n, defined by iterative scheme (71), converges to the minimal norm solution y of euation (1). Theorem 3.1. Consider iterative scheme (71). Let y N (A). Then lim u n y =. (74) n 15

16 Proof. Consider the identity y = ata 1 y + Ta 1 A f, Ay = f. (75) Let w n := u n y and B n := n T 1 n. Then w n+1 = B n w n, w 1 = y u 1 = y. One uses (75) and gets y u n = B B n... B 1 w 1 = B B n... B 1 y ( n ) = + s n + s... d E s y, y + s ( ) ( ) n ( ) = + s n... d E s y, y + s + s n ( + s) n d E sy, y, (76) where E s is the resolution of the identity corresponding to the operator T := A A. Here we have used the identity (75) and the monotonicity of the function φ(x) := x (x+s), s. From estimate (76) we derive relation (74). Indeed, write n b ( + s) n d E sy, y = n ( + s) n d E sy, y + b n ( + s) n d E sy, y, (77) where b is a sufficiently small number which will be chosen later. For any fixed b > one has < 1 if s b. Therefore, it follows that +s +b On the other hand one has b b n ( + s) n d E sy, y as n. (78) n ( + s) n d E sy, y b d E s y, y. (79) Since y N (A), one has lim b b d E sy, y =. Therefore, given an arbitrary number ɛ > one can choose b(ɛ) such that b(ɛ) n ( + s) n d E sy, y < ɛ. (8) Using this b(ɛ), one chooses sufficiently large n(ɛ) such that b(ɛ) n ( + s) n d E sy, y < ɛ, n > n(ɛ). (81) Since ɛ > is arbitrary, Theorem 3.1 is proved. 16

17 As we mentioned before if the exact data f are contaminated by some noise then iterative scheme (7) is used, where f δ f δ. Note that u δ n+1 u n+1 n T 1 n (uδ n u n ) + δ n uδ n u n + δ n. (8) To prove the convergence of the solution obtained by iterative scheme (7), we need the following lemmas: Lemma 3.. Let u n and u δ n be defined in (71) and (7), respectively. Then u δ n u n 1 δ, n 1. (83) n Proof. Let us prove relation (83) by induction. For n = 1 one has u δ 1 u 1 =. Thus, for n = 1 the relation holds. Suppose u δ l u l 1 δ, 1 l k. (84) l Then from (8) and (84) we have Thus, u δ k+1 u k+1 u δ k u k + Lemma 3. is proved. δ k δ (1 ). k+1 1 u δ n u n δ (1 ) n, n 1. δ + δ k k Let us formulate our stopping rule: the iteration in iterative scheme (7) is stopped at the first integer n δ satisfying AT 1 n δ A f δ f δ Cδ ε < AT 1 n A f δ f δ, 1 n < n δ, C > 1, ε (, 1), (85) and it is assumed that f δ > Cδ ε. Lemma 3.3. Let u δ n be defined in (7), and W n := AT 1 n A f δ f δ. Then Proof. Note that W n+1 W n, n 1. (86) W n = AA Q 1 n f δ f δ = (Q n n I m )Q 1 n f δ f δ = n Q 1 n f δ, (87) where Q := AA, and Q a := Q + ai m. Using the spectral theorem, one gets W n+1 = (n+1) ( n+1 + s) d F sf δ, f δ 17 n ( n + s) d F sf δ, f δ = W n, (88)

18 where F s is the resolution of the identity corresponding to the operator Q := AA. Here we have used the monotonicity of the function g(x) = x (x+s), s. Thus, W n+1 W n, n 1. (89) Lemma 3.3 is proved. Lemma 3.4. Let u δ n be defined in (7), and f δ > Cδ ε, ε (, 1), C > 1. Then there exists a uniue index n δ such that ineuality (85) holds. Proof. Let e n := AT 1 n A f δ f δ. Then where Q a := AA + ai. Therefore, e n = n Q 1 n f δ, (9) e n n Q 1 n (f δ f) + n Q 1 n f f δ f + n Q 1 n Ay δ + n y, (91) where the estimate Q 1 a A = ATa 1 1 a lim sup e n δ. n was used. Thus, This shows that the integer n δ, satisfying (85), exists. The uniueness of n δ follows from its definition. Lemma 3.5. Let u δ n be defined in (7). If n δ is chosen by rule (85) then Proof. From (91) we have lim δ AT 1 A f δ f δ δ + δ n δ =. (9) e 1, (93) where e 1 := u 1 y = y. It follows from stopping rule (85) and estimate (93) that Therefore, and so Cδ ε AT 1 n δ 1 A f δ f δ n δ 1 e 1 + δ. (94) (C 1) δ ε n δ 1 e 1, (95) 1 e 1, ε (, 1). (96) n δ 1 (C 1)δε 18

19 This implies Thus, δ n δ δ n δ = δ n δ 1 e 1 δ 1/ (C 1)δ ε = as δ. Lemma 3.5 is proved. e 1 1/ (C 1) δ1 ε. (97) The proof of convergence of the solution obtained by iterative scheme (7) is given in the following theorem: Theorem 3.6. Let u δ n be defined in (7), y N (A), f δ > Cδ ε, ε (, 1), C > 1, (, 1). If n δ is chosen by rule (85), then u δ n y as δ. (98) Proof. From Lemma 3. we get the following estimate: u δ n δ y u δ n δ u nδ + u nδ y 1 δ n + u nδ y := I 1 +I, (99) δ 1 δ where I 1 := and I n δ := u nδ y. By Lemma 3.5 one gets I 1 as δ. To prove lim δ I = one needs the relation lim δ n δ =. This relation is a conseuence of the following lemma: Lemma 3.7. If n δ is chosen by rule (85), then n δ as δ, (1) so Proof. Note that lim n δ =. (11) δ AT 1 a A f δ f δ = AA Q 1 f δ f δ = (AA + ai m ai m )Q 1 f δ f δ a = f δ aq 1 a f δ f δ = aq 1 a f δ, where a >, Q := AA and Q a := Q + ai. From stopping rule (85) we have AT 1 n δ A f δ f δ Cδ ε. Thus, 1 lim AT δ n δ A f δ f δ = lim n δ Q 1 n δ f δ =. (1) δ Using an argument given in the proof of Lemma.8, (see formulas (54)-(61) in which α = 1), one gets lim δ n δ =, so lim δ n δ =. Lemma 3.7 is proved. Lemma 3.7 and Theorem 3.1 imply I as δ. Thus, Theorem 3.6 is proved. a 19

20 4 Numerical experiments In all the experiments we measure the accuracy of the approximate solutions using the relative error: Rel.Err = uδ n δ y, y where. is the Euclidean norm in R n. The exact data are perturbed by some noises so that f δ f δ, where f δ = f + δ e e, δ is the noise level, and e R n is the noise taken from the Gaussian distribution with mean and standard deviation 1. The MATLAB routine called randn with seed 15 is used to generate the vector e. The iterative schemes (13) and (7) will be denoted by IS 1 and IS, respectively. In the iterative scheme IS 1, for fixed (, 1), one needs to choose a sufficiently large α > so that ineuality (17) hold, for example one may choose α 1. The number of iterations of IS 1 and IS are denoted by Iter 1 and Iter, respectively. We compare the results obtained by the proposed methods with the results obtained by using the variational regularization method (VR). In VR we use the Newton method for solving the euation for regularization parameter. In [4] the nonlinear euation Au V R (a) f δ = (Cδ), C = 1.1, (13) where u V R (a) := Ta 1 A f δ, is solved by the Newton s method. In this paper the initial value of the regularization parameter is taken to be α = α, where k k δ δ is the first integer such that the Newton s method for solving (13) converges. We stop the iteration of the Newton s method at the first integer n δ satisfying the ineuality ATa 1 n A f δ f δ (Cδ) 1 3 (Cδ), a := α. The number of iterations needed to complete a convergent Newton s method is denoted by Iter V R. 4.1 Ill-conditioned linear algebraic systems n Table 1: Condition number of some Hilbert matrices κ(a) = A A

21 where Consider the following system: H (m) u = f, (14) H (m) 1 ij =, i, j = 1,,..., m, i + j + 1 is a Hilbert matrix of dimension m. The system (14) is an example of a severely ill-posed problem if m > 1, because the condition number of the Hilbert matrix is increasing exponentially as m grows, see Table (1). The minimal eigenvalues of Hilbert matrix of dimension m can be obtained using the following formula λ min (H (m) ) = 15/4 π 3/ m( + 1) (4m+4) (1 + o(1)). (15) This formula is proved in [3]. Since κ(h (m) ) = λmax(h(m) ) λ min(h, it follows from (15) (m) ) that the condition number grows as O( e3.555m m ). The following exact solution is used to test the proposed methods: y R m, where y k =.5k, k = 1,,..., m. The Hilbert matrix of dimension m = is used in the experiments. This matrix has condition number of order 1 33, so it is a severely ill-conditioned matrix. In Table one can see that the number of iterations of the iterative Table : Hilbert matrix problem: the number of iterations and the relative errors with respect to the parameter (α = 1, δ = 1 ). IS 1 IS REl.Err Iter 1 REl.Err Iter scheme IS 1 and IS increases as the value of increases. The relative errors start to increase at =.15. By these observations, we suggest to choose the parameter in the interval (.15,.5). In Table 3 the results of the experiments with various values of δ are presented. Here the parameter ε was.99. The geometric seuence {.5 } n=1 was used in the iterative schemes IS 1 and IS. The parameter C in (16) and (85) were 1.1. The parameter k δ in the variational regularization method was 1. One can see that the relative errors of IS 1 and IS are smaller than these for the VR. The relative error decreases as the noise level decreases which can be seen on the same table. This shows that the proposed method produces stable solutions. 1

22 Table 3: ICLAS with Hilbert matrix: the relative errors and the number of iterations δ IS 1 IS VR REl.Err Iter 1 REl.Err Iter REl.Err Iter V R. 5% % % Fredholm integral euations of the first kind (FIEFK) Here we consider two Fredholm integral euations : a) f(s) = 3 3 k(t s)u(t)dt, (16) where { 1 + cos( π k(z) = 3 z), z < 3; (17), z 3, and { [ (6 + z) 1 1 f(z) = cos( π 3 z)] 9 πz π sin( 3 ), z 6; (18), z > 6. b) where and f(s) = 1 k(s, t) = k(s, t)u(t)dt, s (, 1), (19) { s(t 1), s < t; t(s 1), s t, (11) f(s) = (s 3 s)/6. (111) The problem a) is discussed in [5] where the solution to this problem is u(x) = k(x). The second problem is taken from [1] where the solution is u(x) = x. The Galerkin s method is used to discretized the integrals (16) and (19). For the basis functions we use the following orthonormal box functions and { m c1 φ i (s) :=, [s i 1, s i ] ;, otherwise, { m c ψ i (t) :=, [t i 1, t i ] ;, otherwise, (11) (113)

23 where s i = d 1 + i d m, t i = d 3 + i d4 m, i =, 1,,..., m. In the problem a) the parameters c 1, c, d 1, d, d 3 and d 4 are set to 1, 6, 6, 1, 3 and 6, respectively. In the second problem we use d 1 = d 3 = and c 1 = c = d = d 4 = 1. Here we approximate the solution u(t) by ũ = m j=1 c jψ j (t). Therefore solving problem (16) is reduced to solving the linear algebraic system where in problem a) A ij = A c = f, c, f R m, (114) k(t s)φ i (s)ψ j (t)dsdt and f i = 6 6 f(s)φ i(s)ds, i, j = 1,,..., m, and in problem b) A ij = 1 1 k(s, t)φ i (s)ψ j (t)dsdt and f i = 1 f(s)φ i(s)ds, and c j = c j i, j = 1,,..., m. Table 4: Problem a): the number of iterations and the relative errors with respect to the parameter (α =, δ = 1 ). IS 1 IS REl.Err Iter 1 REl.Err Iter Table 5: Problem a): the relative errors and the number of iterations δ IS 1 IS VR REl.Err Iter 1 REl.Err Iter REl.Err Iter V R. 5% % % The parameter m = 6 is used in problem a). In this case the condition number of the matrix A with m = 6 is , so it is an ill-conditioned matrix. Here the parameter C in IS 1 and IS are and 1.1, respectively. For problem b) the parameter m is. In this case the condition number of the 3

24 Table 6: Problem b): the number of iterations and the relative errors with respect to the parameter (α = 4, δ = 1 ). IS 1 IS REl.Err Iter 1 REl.Err Iter Table 7: Problem b): the relative errors and the number of iterations δ IS 1 IS VR REl.Err Iter 1 REl.Err Iter REl.Err Iter V R. 5% % % matrix A is The parameter C is 1.1 in the both iterative schemes IS 1 and IS. In Tables 4 and 6 we give the relation between the parameter and the number of iterations and the relative errors of the iterative schemes IS 1 and IS. The closer the parameter to 1, the larger number of iterations we get, and the closer the parameter to, the smaller the number of iterations we get. But the relative error starts to increase if the parameter is chosen too small. Based on the numerical results given in Tables 4 and 5, we suggest to choose the parameter in the interval (.15,.5). In the iterative schemes IS 1 and IS we use the geometric seuence {.5 } n=1 for problem a). The geometric series {4.5 } n=1 is used in problem b). In the variational regularization method we use α = and α = 4 as the initial regularization parameter of the Newton s method in problem a) and b), respectively. Since the Newton s method for solving (13) is locally convergent, in problem b) we need to choose a smaller regularization parameter α than for IS 1 and IS methods. Here k δ = 8 was used. The numerical results on Table 5 show that the solutions produced by the proposed iterative schemes are stable. In problem a) the relative errors of the iterative scheme IS, are smaller than these for the iterative scheme IS 1 and than these for the variational regularization, VR. In Table 7 the relative errors produced by the three methods for solving problem b) are presented. The relative error of IS 1 is smaller than the one for the other two methods. 4

25 5 Conclusion We have demonstrated that the proposed iterative schemes can be used for solving ill-conditioned linear algebraic systems stably. The advantage of the iterative scheme (13) compared with iterative scheme (7) is the following: one applies the operator Ta 1 only once at each iteration. Note that the difficulty of using the Newton s method is in choosing the initial value for the regularization parameter, since the Newton s method for solving euation (13) converges only locally. In solving (13) by the Newton s method one often has to choose an initial regularization parameter a sufficiently close to the root of euation (13) as shown in problem b) in Section 4.. In our iterative schemes the initial regularization parameter can be chosen in the interval [1, 4] which is larger than the initial regularization parameter used in the variational regularization method. In the iterative scheme IS 1 we modified the discrepancy-type principle t e (t s) a(s) Q 1 a(s) ds = Cδ, C (1, ), given in [11], by using (1) to get discrepancy-type principle (33), which can be easily implemented numerically. In Section 3 we used the geometric series {α n } n=1 in place of the constant regularization parameter a in the iterative scheme u n+1 = ata 1 u n + T 1 A f δ developed in [8]. This geometric series of the regularization parameter allows one to use the a posteriori stopping rule given in (85). We proved that this stopping rule produces stable approximation of the minimal norm solution of euation (1). In all the experiments stopping rules (33) and (85) produce stable approximations to the minimal norm solution of euation (1). It is of interest to develop a method for choosing the parameter in the proposed methods which gives sufficiently small relative error and small number of iterations. a References [1] Delves, L. M. and Mohamed, J. L., Computational Methods for Integral Euations, Cambridge University Press, 1985; p. 31. [] Hoang, N.S. and Ramm, A. G. Solving ill-conditioned linear algebraic systems by the dynamical systems method (DSM), Inverse Problems in Sci. and Engineering, 16, N5, (8), [3] Kalyabin, G. A., An asymptotic formula for the minimal eigenvalues of Hilbert type matrices, Functional analysis and its applications, Vol. 35, No. 1, pp 67-7, 1. [4] Morozov, V., Methods of solving incorrectly posed problems, Springer Verlag, New York,

26 [5] Phillips, D. L., A techniue for the numerical solution of certain integral euations of the first kind, J Assoc Comput Mach, 9, 84-97, 196. [6] Ramm, A. G., Inverse problems, Springer, New York, 5. [7] Ramm, A. G., Ill-posed problems with unbounded operators, Journ. Math. Anal. Appl., 35,, ,7. [8] Ramm, A. G., Iterative solution of linear euations with unbounded operators, Jour. Math. Anal. Appl., 33, N, , 7. [9] Ramm, A. G., Dynamical systems method for solving operator euations, Elsevier, Amsterdam, 7. [1] Ramm, A. G., On unbounded operators and applications, Appl. Math. Lett., 1, , 8. [11] Ramm, A. G., Discrepancy principle for DSM, I, II, Comm. Nonlin. Sci. and Numer. Simulation, 1, N1, (5), 95-11; 13, (8), [1] Ramm, A. G., A new discrepancy principle, Journ. Math.Anal. Appl., 31, (5),