Order of convergence

Transcription

1 Order of convergence

2 Linear and Quadratic Order of convergence

3 Computing square root with Newton s Method Given a > 0, p def = a is positive root of equation Newton s Method p k+1 = p k p2 k a 2p k = 1 2 f (x) def = x 2 a = 0. (p k + apk ), k = 0, 1, 2,, Newton s Method is well defined for any p 0 > 0.

4 Newton s Method for a converges quadratically if p 0 > 0 {p k } k=1 bounded below by a: p k a = 1 ( p k 1 + a ) ( pk 1 a ) 2 a = 0, k = 1, 2,, 2 2p k 1 p k 1 {p k } k=1 monotonically decreasing: p k+1 p k = 1 ) (p k + apk p k = a p2 k 0, k = 1, 2,, 2 2p k def lim k p k = p exists and satisfies Newton Iteration: p = 1 ( p + a ) a, therefore p = a. 2 p Newton s Method quadratically convergent lim k p k a p k 1 a 2 = lim k 1 2p k 1 = 1 2 a.

5 Linear Convergence Theorem of Fixed Point Iteration Let g C[a, b] satisfy g(x) [a, b] for all x [a, b]. Assume g (x) continuous with g (x) κ for all x (a, b) and constant κ < 1. Then Fixed Point Iteration p k = g(p k 1 ), k = 1, 2,, for any p 0 [a, b] converges to fixed point p [a, b] with p k+1 p lim = g (p) κ, if p k p for all k. k p k p (not linear convergence if g (p) = 0)

6 Proof of Linear Convergence By Fixed Point Theorem, Fixed Point p [a, b] uniquely exists, and FPI converges to p. By Mean Value Theorem, there exists a ξ k in between p k and p such that Therefore lim k g(p k ) g(p) = g (ξ k )(p k p), for all k. lim (ξ k ) k = g (p) p k+1 p p k p = lim k g(p k ) g(p) p k p = lim k g (ξ k )(p k p) p k p = g (p)

7 Quadratic Convergence of Fixed Point Iteration Assume FPI {p k } k=1 converges to p, with g (p) = 0, then p k+1 p lim k p k p 2 = 1 2 g (p), if p k p for all k. (not quadratic convergence if g (p) = 0)

8 Proof of Quadratic Convergence Taylor expansion at p: g(p k ) ξ k between p, p k ========== g(p) + g (p)(p k p) g (ξ k )(p k p) 2 Therefore ========== g(p) g (ξ k )(p k p) 2 p k+1 p g(p k ) p lim k p k p 2 = lim k p k p 2 = 1 2 lim g (ξ k )(p k p) 2 k p k p 2 = 1 2 g (p)

9 Quadratic Convergence of Newton Iteration Newton Iteration is FPI with and with f (p) = 0. derivatives therefore g (x) = f (x)f (x) (f (x)) 2 g(x) = x f (x) f (x), g (x) = f (x) f (x) + f (x) f (x) 2 (f (x)) 2 (f (x)) 3 f (x) g (p) = 0, and g (p) = f (p) f (p), p k+1 p lim k p k p 2 = 1 f (p) 2 f (p), if f (p) 0.

10 Multiple roots: f (p) = 0, f (p) = 0 Definition: A solution p of f (x) = 0 is a root of multiplicity m of f if for x p, we can write f (x) = (x p) m q(x), where lim x p q(x) 0. Theorem: The function f C m [a, b] has a root of multiplicity m at p in (a, b) if and only if 0 = f (p) = f (p) = f (p) = = f (m 1) (p), but f (m) (p) 0. simple root: f satisfies f (p) = 0, but f (p) 0.

11 Example: f (x) = e x x 1, f (0) = f (0) = 0, f (0) = 1 Function Newton s Method

12 Modified Newton Method p k+1 = p k m f (p k) f, for k = 1, 2,, (m = multiplicity of root p.) (p k ) m = 2 for function f (x) = e x x 1,

13 Modified Newton Method is Quadratically Convergent p k+1 = g(p k ), g(x) = x m f (x) f, for k = 1, 2,, (x) Let f (x) = (x p) m q(x), q(p) 0. g(x) = x m f (x) f (x) = x g (p) = 0, hence quadratic convergence. m (x p)q(x) mq(x) + (x p)q (x).

14 Accelerating Convergence: Aitken s 2 Method Suppose {p k } k=1 linearly converges to limit p, Define p k+1 p lim k p k p p k+1 p p k p so that {λ k } k=1 converges to λ. It follows that Solve for p: = λ, λ < 1. def = λ k, 0 λ k+1 λ k = p k+2 p p k+1 p p k+1 p p k p. p = p2 k+1 p kp k+2 2p k+1 p k p k+2 + (λ k+1 λ k )(p k+1 p)(p k p) 2(p k+1 p) (p k p) (p k+2 p).

15 Accelerating Convergence: Aitken s 2 Method p k def = p 2 k+1 p kp k+2 2p k+1 p k p k+2 (p k+1 p k ) 2 def = p k = { 2 }(p k ). p k+2 2p k+1 + p k Approximation Error p k p = (λ k+1 λ k )(p k+1 p)(p k p) 2(p k+1 p) (p k p) (p k+2 p) (λ k+1 λ k )(p k p) = ( ) 2 pk+1 p 1 p k p p k+2 p p k+1 p (λ k+1 λ k )(p k p) 2 λ 1 λ O ( p k p ).

16 Accelerating Convergence: Aitken s 2 Method p n = cos(1/n) Error Plots

17 Accelerating Convergence: Steffensen s Method Aitken s Acceleration for a given {p k } k=1 : { 2 }(p k ) = p k (p k+1 p k ) 2 p k+2 2p k+1 + p k. Steffensen s Method: use Aitken s Acceleration as soon as iterations are generated: Given p (0) 0 for k = 0, 1, 2,, p (k) 1 = g(p (k) 0 ), p(k) 2 = g(p (k) 1 ), p(k+1) 0 = { 2 }(p (k) 0 ).

18

19 Fixed point for g(x) = log(2 + 2x 2 ) 0 FixedPoint Method with 66 Points

20 Fixed Point Iteration: Linear Convergence 10 0 Convergence Rate, Fixed Point Method

21 Aitken s Method: Faster, but Linear Convergence 10 0 Convergence Rate, Aitken Method

22 Steffenson s Method: Quadratic Convergence 10 0 Convergence Rate, Steffen Method

23 Newton s Method on Polynomial roots = Horner s Method Let P(x) = a n x n + a n 1 x n a 1 x + a 0. Horner s Method computes P(x 0 ) and P (x 0 ) define b n = a n for k = n 1, n 2,, 1, 0 then it follows b 0 = P(x 0 ) b k = a k + b k+1 x 0, P(x) = (x x 0 ) Q(x) + b 0 Q(x) = b n x n 1 + b n 1 x n b 2 x + b 1. P (x) = Q(x) + (x x 0 ) Q (x) P (x 0 ) = Q(x 0 )

24 Horner s Method

25 Muller s Method: finding complex roots Given three points (p 0, f (p 0 )), (p 1, f (p 1 )), and (p 2, f (p 2 )). Construct a parabola through them, p 3 is the intersection of x-axis with parabola closer to p 2.

26 Secant Method vs. Muller s Method

27 Muller s Method: derivation Choose parabola a, b, c satisfy P(x) = a(x p 2 ) 2 + b(x p 2 ) + c.

28 Muller s Method: finding complex roots p 3 is the intersection of x-axis with parabola closer to p 2.

29 Muller s Method 10 1 Muler Method function f(p) = p 4 3*p 3 +p 2 +p

30 Choices, and more Choices Bisection Method: slow (linearly convergent); reliable (always finds a root given interval [a, b] with f (a) f (b) < 0.) Fixed Point Iteration: slow (linearly convergent); no need for interval [a, b] with f (a) f (b) < 0. Newton s Method: fast (quadratically convergent); could get burnt (need not converge.) Secant Method: between Bisection and Newton in speed; need not converge; no derivative needed. Steffensen s Method: fast (quadratically convergent); no derivative needed. not a method of choice in higher dimensions. Muller s Method: can handle complex roots; need not converge.

31 Brent s Method: Motivation (Mostly) speed of Steffensen s Method Reliability of Bisection Method

32 Brent s Method = Zeroin

33 Brent s Method: Zeroin (Wikipedia)

34 Brent s Method: Zeroin

35 Brent s Method Worst case number of iterations = (n 2 ), where n is number of iterations of Bisection Method for a given tolerance.

36 Modified Brent s Method

37 Modified Brent s Method

38 Modified Brent s Method

39 Modified Brent s Method

40 Interpolation and Approximation (connecting the dots) US population census data available every ten years. What was US population in year 1996? What will US population be in year 2017?

41 Connecting two dots Given two distinct dots, there is a unique line connecting them.

42 Connecting two dots Given two distinct points (p 0, f (p 0 )) and (p 1, f (p 1 )), there is a unique line connecting them P(x) = f (p 0 ) x p 1 + f (p 1 ) x p 0 p 0 p 1 p 1 p 0 def = f (p 0 ) L 0 (x) + f (p 1 ) L 1 (x), with L 0 (x), L 1 (x) unrelated to f (p 0 ), f (p 1 ). L 0 (p 0 ) = 1, L 0 (p 1 ) = 0, L 1 (p 0 ) = 0, L 1 (p 1 ) = 1.

43 Connecting two dots Given two distinct points (p 0, f (p 0 )) and (p 1, f (p 1 )), there is a unique line connecting them P(x) = f (p 0 ) x p 1 p 0 p 1 + f (p 1 ) x p 0 p 1 p 0

44 Connecting n + 1 dots Given n + 1 distinct points (x 0, f (x 0 )), (x 1, f (x 1 )),, (x n, f (x n )), Find a degree n polynomial P(x) = a n x n + a n 1 x n a 1 x + a 0, so that P(x 0 ) = f (x 0 ), P(x 1 ) = f (x 1 ),, P(x n ) = f (x n ).

45 Connecting n + 1 dots Would like to write P(x) as P(x) = f (x 0 )L 0 (x) + f (x 1 )L 1 (x) + + f (x n )L n (x), where L 0 (x), L 1 (x),, L n (x) are unrelated to f (x 0 ), f (x 1 ),, f (x n ). at each node x j, j = 0, 1,, n, f (x j ) = P(x j ) = f (x 0 )L 0 (x j ) + f (x 1 )L 1 (x j ) + + f (x n )L n (x j ) This suggests for all i, j, L i (x j ) = { 1, if i = j, 0, if i j,

46 Largrangian Polynomials P(x) = f (x 0 )L 0 (x) + f (x 1 )L 1 (x) + + f (x n )L n (x), { 1, if i = j, L i (x j ) = 0, if i j, x j is a root of L i (x) for every j i, so L i (x) = j i x x j x i x j.

47 Quadratic Interpolation

48 Quadratic Interpolation: Solution

49 Quadratic Interpolation: Solution

50 Quadratic Interpolation: Solution

51 Quadratic Interpolation: Solution