Accelerating Convergence

Accelerating Convergence MATH 375 Numerical Analysis J. Robert Buchanan Department of Mathematics Fall 2013

Motivation We have seen that most fixed-point methods for root finding converge only linearly to a solution. Today we describe a technique which can be used accelerate the convergence of any linearly convergent sequence.

Linear Convergence Suppose sequence {p n } n=0 converges linearly to p.

Linear Convergence Suppose sequence {p n } n=0 converges linearly to p. p n+1 p 0 < lim = λ < 1 n p n p

Linear Convergence Suppose sequence {p n } n=0 converges linearly to p. p n+1 p 0 < lim = λ < 1 n p n p Assume the signs of p n+2 p, p n+1 p, and p n p are all the same.

Linear Convergence Suppose sequence {p n } n=0 converges linearly to p. p n+1 p 0 < lim = λ < 1 n p n p Assume the signs of p n+2 p, p n+1 p, and p n p are all the same. For large n then p n+2 p p n+1 p p n+1 p p n p.

Re-writing the Sequence (1 of 2) Solve for p. p n+2 p p n+1 p p n+1 p p n p (p n+1 p) 2 (p n+2 p)(p n p)

Re-writing the Sequence (1 of 2) Solve for p. p n+2 p p n+1 p p n+1 p p n p (p n+1 p) 2 (p n+2 p)(p n p) p p n+2p n p 2 n+1 p n+2 2p n+1 + p n

Re-writing the Sequence (2 of 2) p p n+2p n p 2 n+1 p n+2 2p n+1 + p n Add and subtract the terms p 2 n and 2p n p n+1 in the numerator.

Re-writing the Sequence (2 of 2) p p n+2p n p 2 n+1 p n+2 2p n+1 + p n Add and subtract the terms p 2 n and 2p n p n+1 in the numerator. p p2 n+1 + 2p np n+1 p 2 n + p n+2 p n + p 2 n 2p n p n+1 p n+2 2p n+1 + p n Now factor the numerator.

Re-writing the Sequence (2 of 2) p p n+2p n p 2 n+1 p n+2 2p n+1 + p n Add and subtract the terms p 2 n and 2p n p n+1 in the numerator. p p2 n+1 + 2p np n+1 p 2 n + p n+2 p n + p 2 n 2p n p n+1 p n+2 2p n+1 + p n Now factor the numerator. p (p n+1 p n ) 2 + p n (p n+2 + p n 2p n+1 ) p n+2 2p n+1 + p n (p n+1 p n ) 2 = p n p n+2 2p n+1 + p n

Aitken s 2 Method Define a new sequence {ˆp n } n=0 as ˆp n = p n We would like to show that (p n+1 p n ) 2 p n+2 2p n+1 + p n. lim ˆp n = p = lim p n, and n n sequence {ˆp n } n=0 converges to p faster than sequence {p n } n=0.

Aitken s 2 Method Define a new sequence {ˆp n } n=0 as ˆp n = p n We would like to show that (p n+1 p n ) 2 p n+2 2p n+1 + p n. lim ˆp n = p = lim p n, and n n sequence {ˆp n } n=0 converges to p faster than sequence {p n } n=0. We will come to call this iterative method as Aitken s 2 Method.

Example ( ) 1 Define p n = sin for n 1. Find the limit of this sequence n and compare its convergence to that of the sequence ˆp n.

Example ( ) 1 Define p n = sin for n 1. Find the limit of this sequence n and compare its convergence to that of the sequence ˆp n. n p n ˆp n 1 0.841471 0.216744 2 0.479426 0.159517 3 0.327195 0.122193 4 0.247404 0.098604 5 0.198669 0.082537 6 0.165896 0.070932 7 0.142372 0.062169 8 0.124675 0.055324 9 0.110883 10 0.099833

Forward Difference Definition Given sequence {p n } n=0, the forward difference denoted p n is defined as p n = p n+1 p n, for n 0. Higher powers of the operator are defined recursively as ) k p n = ( k 1 p n, for n 2.

Forward Difference (2 of 2) p n = p n+1 p n 2 p n = (p n+1 p n ) = p n+2 p n+1 (p n+1 p n ) = p n+2 2p n+1 + p n 3 p n = (p n+2 2p n+1 + p n ) = p n+3 3p n+2 + 3p n+1 p n.

Forward Difference (2 of 2) p n = p n+1 p n 2 p n = (p n+1 p n ) = p n+2 p n+1 (p n+1 p n ) = p n+2 2p n+1 + p n 3 p n = (p n+2 2p n+1 + p n ) = p n+3 3p n+2 + 3p n+1 p n. We can write Aitken s 2 Method as ˆp n = p n ( p n) 2 2 p n.

Convergence of Aitken s 2 Method Theorem Let {p n } n=0 be a sequence converging linearly to p with asymptotic error constant λ < 1. The sequence {ˆp n } n=0 converges to p faster than {p n } n=0 in the sense that ˆp n p lim n p n p = 0.

Proof (1 of 5) By assumption {p n } n=0 converges linearly to p, i.e. p n+1 p 0 < lim = λ < 1. n p n p Assume the signs of p k p are all the same for sufficiently large k.

Proof (1 of 5) By assumption {p n } n=0 converges linearly to p, i.e. p n+1 p 0 < lim = λ < 1. n p n p Assume the signs of p k p are all the same for sufficiently large k. Define a new sequence δ n = p n+1 p λ for n 0. Note that lim n δ n = 0. p n p

Proof (2 of 5) Consider ˆp n p p n p = 1 p n p ( p n ( p n) 2 2 p n p )

Proof (2 of 5) Consider ˆp n p p n p = 1 p n p = 1 p n p ( ) p n ( p n) 2 2 p p n ( ) p n p ( p n) 2 2 p n ( p n ) 2 = 1 (p n p) 2 p n

Proof (2 of 5) Consider ˆp n p p n p = 1 p n p = 1 p n p ( ) p n ( p n) 2 2 p p n ( ) p n p ( p n) 2 2 p n ( p n ) 2 = 1 (p n p) 2 p n (p n+1 p n ) 2 = 1 (p n p)(p n+2 2p n+1 + p n ) = 1 ((p n+1 p) (p n p)) 2 (p n p)((p n+2 p) 2(p n+1 p) + (p n p))

Proof (3 of 5) Recall p n+1 p p n p = δ n + λ p n+1 p = (δ n + λ)(p n p)

Proof (3 of 5) Recall p n+1 p p n p = δ n + λ p n+1 p = (δ n + λ)(p n p) This implies p 2 p = (δ 1 + λ)(p 1 p) p 3 p = (δ 2 + λ)(p 2 p) = (δ 1 + λ)(δ 2 + λ)(p 1 p). n p n+1 p = (p 1 p) (δ i + λ). i=1

Proof (4 of 5) Consequently ˆp n p p n p = 1 ((p n+1 p) (p n p)) 2 (p n p)((p n+2 p) 2(p n+1 p) + (p n p)) = 2 n n 1 (p 1 p) (δ i + λ) (p 1 p) (δ i + λ) i=1 i=1 1 ( n 1 n+1 n n 1 ) (p 1 p) (δ i + λ) (p 1 p) (δ i + λ) 2(p 1 p) (δ i + λ) + (p 1 p) (δ i + λ) i=1 = 1 n 1 (δ i + λ) = 1 i=1 n 1 i=1 i=1 i=1 ( n+1 i=1 n 1 i=1 n n 1 (δ i + λ) (δ i + λ) i=1 2 ) n n 1 (δ i + λ) 2 (δ i + λ) + (δ i + λ) i=1 2 (δ i + λ) (δ i + λ) 2 i=1 (δ n+λ 1) 2 ((δ n+1 +λ)(δ n+λ) 2(δ n+λ)+1) i=1 i=1

Proof (5 of 5) ˆp n p p n p = 1 = 1 n 1 i=1 n 1 i=1 2 (δ i + λ) (δ i + λ) 2 (δ n+λ 1) 2 ((δ n+1 +λ)(δ n+λ) 2(δ n+λ)+1) (δ n + λ 1) 2 (δ n+1 + λ)(δ n + λ) 2(δ n + λ) + 1

Proof (5 of 5) ˆp n p p n p = 1 = 1 n 1 i=1 Taking the limit as n : ˆp n p lim n p n p = lim n = 1 = 0 n 1 i=1 2 (δ i + λ) (δ i + λ) 2 (δ n+λ 1) 2 ((δ n+1 +λ)(δ n+λ) 2(δ n+λ)+1) (δ n + λ 1) 2 (δ n+1 + λ)(δ n + λ) 2(δ n + λ) + 1 [ 1 (δ n + λ 1) 2 ] (δ n+1 + λ)(δ n + λ) 2(δ n + λ) + 1 (λ 1)2 λ 2 2λ + 1

Steffensen s Method We can use Aitken s 2 Method to accelerate the convergence of any linearly convergent sequence generated by fixed-point iteration.

Steffensen s Method We can use Aitken s 2 Method to accelerate the convergence of any linearly convergent sequence generated by fixed-point iteration. Consider the fixed point problem g(x) = x and an initial approximation p 0. Calculate p 1 = g(p 0 ) p 2 = g(p 1 ) { ˆp 0 = 2} (p 0 ) p 4 = g(ˆp 0 ) p 5 = g(p 4 ) { ˆp 1 = 2} (ˆp 0 ) p 7 = g(ˆp 1 ) p 8 = g(p 7 ) { ˆp 2 = 2} (ˆp 1 )

Steffensen s Method We can use Aitken s 2 Method to accelerate the convergence of any linearly convergent sequence generated by fixed-point iteration. Consider the fixed point problem g(x) = x and an initial approximation p 0. Calculate p 1 = g(p 0 ) p 2 = g(p 1 ) { ˆp 0 = 2} (p 0 ) p 4 = g(ˆp 0 ) p 5 = g(p 4 ) { ˆp 1 = 2} (ˆp 0 ) p 7 = g(ˆp 1 ) p 8 = g(p 7 ) { ˆp 2 = 2} (ˆp 1 ) Every 3rd term is calculated using the 2 method, all other terms use fixed-point iteration.

Algorithm INPUT initial approximation p 0, tolerance ɛ, maximum iterations N. STEP 1 Set i = 1. STEP 2 While i N do STEPS 3 6. STEP 3 Set p 1 = g(p 0 ); p 2 = g(p 1 ); p = { 2} (p 0 ). STEP 4 If p p 0 < ɛ then OUTPUT p; STOP. STEP 5 Set i = i + 1. STEP 6 Set p 0 = p. STEP 7 OUTPUT Method failed after N iterations. ; STOP.

Example Use Steffensen s Method to approximate the solution to x 3 x = 0 for x [0, 1].

Example Use Steffensen s Method to approximate the solution to x 3 x = 0 for x [0, 1]. n Fixed-point Steffensen s 0 0.100000 0.100000 1 0.895958 0.580610 2 0.373697 0.547940 3 0.663287 0.547809 4 0.482538 0.547809

Final Result Remark: Steffensen s Method appears to generate a quadratically convergent sequence for root finding without requiring computation of a derivative.

Final Result Remark: Steffensen s Method appears to generate a quadratically convergent sequence for root finding without requiring computation of a derivative. Theorem Suppose g(x) = x has a solution p with g (p) 1. If there exists δ > 0 such that g C 3 [p δ, p + δ], the Steffensen s Method gives quadratic convergence for any p 0 [p δ, p + δ].

Homework Read Section 2.5. Exercises: 1, 2, 7, 9, 11