Bisection Method. and compute f (p 1 ). repeat with p 2 = a 2+b 2

Transcription

1 Bisection Method Given continuous function f (x) on the interval [a, b] with f (a) f (b) < 0, there must be a root in (a, b). To find a root: set [a 1, b 1 ] = [a, b]. set p 1 = a 1+b 1 2 and compute f (p 1 ). if f (p 1 ) = 0, then quit with root p 1 (must be very lucky.) if f (a1 ) f (p 1 ) < 0, then set [a 2, b 2 ] = [a 1, p 1 ], otherwise (f (p1 ) f (b 1 ) < 0) set [a 2, b 2 ] = [p 1, b 1 ], repeat with p 2 = a 2+b 2 2.

2 Bisection Method

3 Naive Bisection Method

4

5

6 Proof of Thm 2.1 Assume that f (p n ) 0 for all n. By construction a 1 a 2 a n b n b 2 b 1. Thus sequences {a n } and {b n } monotonically converge to limits a b, respectively. Since f (a n ) f (b n ) < 0 for all n, it follows that f (a ) f (b ) 0, and thus a root p [a, b ] [a n, b n ] exists. Since p n = an+bn 2, it follows that p n p bn an By construction 2. b n a n = b n 1 a n 1 = b n 2 a n = = b 1 a 1 2 n 1 = b a 2 n 1. Put together, p n p b a 2 n. In fact, a = b = p.

7 Example Function with Root 7 function = $e x (2+2x 2 )$

8 7 Bisection Method with 52 Points

9 Fixed Point Iteration The number p is a fixed point for a given function g if g(p) = p.

10 Fixed Point Example

11 Fixed Point Theorem (I)

12 Proof of Thm 2.3 If g(a) = a or g(b) = b,then g has a fixed point at an endpoint. Otherwise, g(a) > a and g(b) < b. The function h(x) = g(x) x is continuous on [a, b], with h(a) = g(a) a > 0 and h(b) = g(b) b < 0. This implies that there exists p (a, b), h(p) = 0. g(p) p = 0, or p = g(p). If g (x) k < 1 for all x in (a, b), and p and q are two distinct fixed points in [a, b]. Then a number ξ exists So 1 = p q p q g(p) g(q) p q = g(p) g(q) p q = g (ξ) < 1. = g (ξ) < 1. This contradiction implies uniqueness of fixed point.

13 Fixed Point Iteration Given initial approximation p 0, define Fixed Point Iteration If iteration converges to p, then p n = g(p n 1 ), n = 1, 2,, p = lim n p n = lim n g(p n 1) = g(p).

14 Fixed Point Theorem (II)

15 Fixed Point Example x x 2 = 0: no convergence g(x) = x 2 [1, ] for x [1, ], g (x) unbounded in[1, ].

16 Fixed Point Example x sin(x) = 0: slow convergence g(x) = sin(x) [ 1, 1] for x [ 1, 1], g (x) 1 [0, 1].

17 Fixed Point Example x sin(x) = 0: slow convergence g(x) = sin(x) [ 1, 1] for x [ 1, 1], g (x) 1 [0, 1].

18 Fixed Point Example x log(2 + 2x 2 ) = 0: normal convergence g(x) = log(2 + 2x 2 ) [2, 3] for x [2, 3], g (x) 4 5 [2, 3]. 0 FixedPoint Method with 66 Points

19 Fixed Point: x (1 cos(x)) = 0: fast convergence g(x) = 1 cos(x) [ 1, 1] for x [ 1, 1], g (x) x

20 Fixed Point: x (1 cos(x)) = 0: fast convergence g(x) = 1 cos(x) [ 1, 1] for x [ 1, 1], g (x) x [0, 1]. 2

21 Fixed Point Theorem (II)

22 Proof of Thm 2.4 A unique fixed point p [a, b] exists. p n p = g(p n 1 ) g(p) = g (ξ n )(p n 1 p) k p n 1 p Since p n p k p n 1 p k 2 p n 2 p k n p 0 p. {p n } n=0 converges to p. lim n kn = 0,

23 Newton s Method for solving f (p) = 0 Suppose that f C 2 [a, b]. Let p 0 [a, b] be an approximation to p with f (p 0 ) 0, and p p 0 small. Taylor expand f (x) at x = p: Solve for p: 0 = f (p) = f (p 0 ) + (p p 0 )f (p 0 ) + (p p 0) 2 f (ξ(p)). 2 p = p 0 f (p 0) f (p 0 ) (p p 0) 2 2f (p 0 ) p 0 f (p 0) def f = p 1. (p 0 ) f (ξ(p)) Newton s Method: p k+1 = p k f (p k) f (p k ), k = 0, 1,

24 Newton s Method for solving f (p) = 0 p = p 0 f (p 0) f (p 0 ) (p p 0) 2 2f (p 0 ) p 0 f (p 0) def f = p 1. (p 0 ) f (ξ(p)) If p 0 close to p, we can expect fast convergence. Best hope in practice: p 0 not too far from p. Newton s method may or may not converge. If Newton s method converges, it converges quickly.

25 Geometry of Newton s Method Taylor expand f (x) at x = p: 0 = f (p) = f (p 0 ) + (p p 0 )f (p 0 ) + (p p 0) 2 f (ξ(p)). 2 Replace f (x) by a straight line: f (p 0 ) + (p p 0 )f (p 0 ) 0. p p 0 f (p 0) f (p 0 ) is the horizontal intercept of straight line y = f (p 0 ) + (x p 0 )f (p 0 )

26 Newton Method

27

28

29 Newton Method for f (x) = e x (2 + 2x 2 ) 8 Newton Method with 8 Points

30 Computing square root with Newton s Method Given a > 0, p def = a is positive root of equation Newton s Method p k+1 = p k p2 k a 2p k = 1 2 f (x) def = x 2 a = 0. (p k + apk ), k = 0, 1, 2,, Newton s Method is well defined for any p 0 > 0.

31 Newton Method for square root

32

33 Proof of Theorem 2.6 Newton s method is fixed point iteration p n = g(p n 1 ), g(x) = x f (x) f (x). Since f (p) 0, there exists an interval [p δ 1, p + δ 1 ] [a, b] on which f (x) 0. Thus, g(x) is defined on [p δ 1, p + δ 1 ]. g (x) = 1 f (x) f (x) f (x) f (x) (f (x)) 2 = f (x) f (x) (f (x)) 2 C[p δ 1, p+δ 1 ]. Since g (p) = 0, there exists 0 < δ < δ 1 so that g (x) κ (= 1 ), for all x [p δ, p + δ]. 2 If x [p δ, p + δ], then g(x) p = g(x) g(p) = g (ξ)(x p) κ x p x p. Therefore g(x) [p δ, p + δ]. {p n } converges to p by Fixed Point Theorem.

34 Newton Method Divergence Example: f (x) = x 1/3

35 Secant Method: Poor man s Newton Method Motivation Newton method style of fast convergence Avoid need for derivative calculations. Approach Newton method: p n+1 = p n f (pn) f (p n). Replace f (p n ) by its cheap approximation Secant method f f (p n ) f (x) (p n ) = lim f (p n) f (p n 1 ). x p n x p n p n 1 p n+1 = p n f (p n)(p n p n 1 ), n = 1, 2,. f (p n ) f (p n 1 )

36 Secant Method: Geometry Approximate f (x) by a straight line f (x) (x p 0)f (p 1 ) (x p 1 )f (p 0 ) p 1 p 0. Both f (x) and straight line go through points (p 0, f (p 0 )), (p 1, f (p 1 )). Let approximate root p 2 be the x-intercept of the straight line p 2 = p 0f (p 1 ) p 1 f (p 0 ) f (p 1 ) f (p 0 ) = p 1 f (p 1)(p 1 p 0 ) f (p 1 ) f (p 0 ).

37 Secant Method: Illustration

38 Fixed point for g(x) = log(2 + 2x 2 ) 25 SecantMethod with 11 Points

39 Performance: number of iterations vs. error in the solution Function to be considered g(x) = log(2 + 2x 2 ), f (x) = x g(x) = x log(2 + 2x 2 ). Root p of f (i.e., f (p) = 0) p = e Bisection Method Fixed Point Iteration Newton s Method Secant Method

40 Bisection Method Order of Convergence 10 0 Convergence Rate, Bisection Method

41 Fixed Point Iteration Order of Convergence 10 0 Convergence Rate, Fixed Point Method

42 Secant Method Order of Convergence 10 2 Convergence Rate, Secant Method

43 Newton Method Order of Convergence 10 0 Convergence Rate, Newton Method

44 Order of convergence

45 Linear and Quadratic Order of convergence

46 Recall rate of convergence: the Big O the Big O() = rate of convergence

47 Recall rate of convergence: the Big O

48 Linear and Quadratic Order of convergence Suppose that {p n } n=1 is linearly convergent to 0, p n+1 p n+1 lim = 0.5, or roughly 0.5, n p n p n hence p n (0.5) n p 0. Suppose that { p n } n=1 is quadratically convergent to 0, p n+1 lim n p n 2 = 0.5, or roughly p n+1 p n But now

49 Linear and Quadratic Order of convergence

50 Order of convergence

51 Linear and Quadratic Order of convergence

52 Computing square root with Newton s Method Given a > 0, p def = a is positive root of equation Newton s Method p k+1 = p k p2 k a 2p k = 1 2 f (x) def = x 2 a = 0. (p k + apk ), k = 0, 1, 2,, Newton s Method is well defined for any p 0 > 0.

53 Newton s Method for a converges quadratically if p 0 > 0 {p k } k=1 bounded below by a: p k a = 1 ( p k 1 + a ) ( pk 1 a ) 2 a = 0, k = 1, 2,, 2 2p k 1 p k 1 {p k } k=1 monotonically decreasing: p k+1 p k = 1 ) (p k + apk p k = a p2 k 0, k = 1, 2,, 2 2p k def lim k p k = p exists and satisfies Newton Iteration: p = 1 ( p + a ) a, therefore p = a. 2 p Newton s Method quadratically convergent lim k p k a p k 1 a 2 = lim k 1 2p k 1 = 1 2 a.

54 Linear Convergence Theorem of Fixed Point Iteration Let g C[a, b] satisfy g(x) [a, b] for all x [a, b]. Assume g (x) continuous with g (x) κ for all x (a, b) and constant κ < 1. Then Fixed Point Iteration p k = g(p k 1 ), k = 1, 2,, for any p 0 [a, b] converges to fixed point p [a, b] with p k+1 p lim = g (p) κ, if p k p for all k. k p k p (not linear convergence if g (p) = 0)

55 Proof of Linear Convergence By Fixed Point Theorem, Fixed Point p [a, b] uniquely exists, and FPI converges to p. By Mean Value Theorem, there exists a ξ k in between p k and p such that Therefore lim k g(p k ) g(p) = g (ξ k )(p k p), for all k. lim (ξ k ) k = g (p) p k+1 p p k p = lim k g(p k ) g(p) p k p = lim k g (ξ k )(p k p) p k p = g (p)

56 Quadratic Convergence of Fixed Point Iteration Assume FPI {p k } k=1 converges to p, with g (p) = 0, then p k+1 p lim k p k p 2 = 1 2 g (p), if p k p for all k. (not quadratic convergence if g (p) = 0)

57 Proof of Quadratic Convergence Taylor expansion at p: g(p k ) ξ k between p, p k ========== g(p) + g (p)(p k p) g (ξ k )(p k p) 2 Therefore ========== g(p) g (ξ k )(p k p) 2 p k+1 p g(p k ) p lim k p k p 2 = lim k p k p 2 = 1 2 lim g (ξ k )(p k p) 2 k p k p 2 = 1 2 g (p)

58 Quadratic Convergence of Newton Iteration Newton Iteration is FPI with and with f (p) = 0. derivatives therefore g (x) = f (x)f (x) (f (x)) 2 g(x) = x f (x) f (x), g (x) = f (x) f (x) + f (x) f (x) 2 (f (x)) 2 (f (x)) 3 f (x) g (p) = 0, and g (p) = f (p) f (p), p k+1 p lim k p k p 2 = 1 f (p) 2 f (p), if f (p) 0.

59 Multiple roots: f (p) = 0, f (p) = 0 Definition: A solution p of f (x) = 0 is a root of multiplicity m of f if for x p, we can write f (x) = (x p) m q(x), where lim x p q(x) 0. Theorem: The function f C m [a, b] has a root of multiplicity m at p in (a, b) if and only if 0 = f (p) = f (p) = f (p) = = f (m 1) (p), but f (m) (p) 0. simple root: f satisfies f (p) = 0, but f (p) 0.

60 Example: f (x) = e x x 1, f (0) = f (0) = 0, f (0) = 1 Function Newton s Method

61 Modified Newton Method p k+1 = p k m f (p k) f, for k = 1, 2,, (m = multiplicity of root p.) (p k ) m = 2 for function f (x) = e x x 1,

62 Modified Newton Method is Quadratically Convergent p k+1 = g(p k ), g(x) = x m f (x) f, for k = 1, 2,, (x) Let f (x) = (x p) m q(x), q(p) 0. g(x) = x m f (x) f (x) = x g (p) = 0, hence quadratic convergence. m (x p)q(x) mq(x) + (x p)q (x).

63 Accelerating Convergence: Aitken s 2 Method Suppose {p k } k=1 linearly converges to limit p, Define p k+1 p lim k p k p p k+1 p p k p so that {λ k } k=1 converges to λ. It follows that Solve for p: = λ, λ < 1. def = λ k, 0 λ k+1 λ k = p k+2 p p k+1 p p k+1 p p k p. p = p2 k+1 p kp k+2 2p k+1 p k p k+2 + (λ k+1 λ k )(p k+1 p)(p k p) 2(p k+1 p) (p k p) (p k+2 p).

64 Accelerating Convergence: Aitken s 2 Method p k def = p 2 k+1 p kp k+2 2p k+1 p k p k+2 (p k+1 p k ) 2 def = p k = { 2 }(p k ). p k+2 2p k+1 + p k Approximation Error p k p = (λ k+1 λ k )(p k+1 p)(p k p) 2(p k+1 p) (p k p) (p k+2 p) (λ k+1 λ k )(p k p) = ( ) 2 pk+1 p 1 p k p p k+2 p p k+1 p (λ k+1 λ k )(p k p) 2 λ 1 λ O ( p k p ).

65 Accelerating Convergence: Aitken s 2 Method p n = cos(1/n) Error Plots

66 Accelerating Convergence: Steffensen s Method Aitken s Acceleration for a given {p k } k=1 : { 2 }(p k ) = p k (p k+1 p k ) 2 p k+2 2p k+1 + p k. Steffensen s Method: use Aitken s Acceleration as soon as iterations are generated: Given p (0) 0 for k = 0, 1, 2,, p (k) 1 = g(p (k) 0 ), p(k) 2 = g(p (k) 1 ), p(k+1) 0 = { 2 }(p (k) 0 ).

67

68 Fixed point for g(x) = log(2 + 2x 2 ) 0 FixedPoint Method with 66 Points

69 Fixed Point Iteration: Linear Convergence 10 0 Convergence Rate, Fixed Point Method

70 Aitken s Method: Faster, but Linear Convergence 10 0 Convergence Rate, Aitken Method

71 Steffenson s Method: Quadratic Convergence 10 0 Convergence Rate, Steffen Method

72 Newton s Method on Polynomial roots = Horner s Method Let P(x) = a n x n + a n 1 x n a 1 x + a 0. Horner s Method computes P(x 0 ) and P (x 0 ) define b n = a n for k = n 1, n 2,, 1, 0 then it follows b 0 = P(x 0 ) b k = a k + b k+1 x 0, P(x) = (x x 0 ) Q(x) + b 0 Q(x) = b n x n 1 + b n 1 x n b 2 x + b 1. P (x) = Q(x) + (x x 0 ) Q (x) P (x 0 ) = Q(x 0 )

73 Horner s Method

74 Muller s Method: finding complex roots Given three points (p 0, f (p 0 )), (p 1, f (p 1 )), and (p 2, f (p 2 )). Construct a parabola through them, p 3 is the intersection of x-axis with parabola closer to p 2.

75 Secant Method vs. Muller s Method

76 Muller s Method: derivation Choose parabola a, b, c satisfy P(x) = a(x p 2 ) 2 + b(x p 2 ) + c.

77 Muller s Method: finding complex roots p 3 is the intersection of x-axis with parabola closer to p 2.

78 Muller s Method 10 1 Muler Method function f(p) = p 4 3*p 3 +p 2 +p

79 Choices, and more Choices Bisection Method: slow (linearly convergent); reliable (always finds a root given interval [a, b] with f (a) f (b) < 0.) Fixed Point Iteration: slow (linearly convergent); no need for interval [a, b] with f (a) f (b) < 0. Newton s Method: fast (quadratically convergent); could get burnt (need not converge.) Secant Method: between Bisection and Newton in speed; need not converge; no derivative needed. Steffensen s Method: fast (quadratically convergent); no derivative needed. not a method of choice in higher dimensions. Muller s Method: can handle complex roots; need not converge.

80 Brent s Method: Motivation (Mostly) speed of Steffensen s Method Reliability of Bisection Method

81 Brent s Method = Zeroin

82 Brent s Method: Zeroin (Wikipedia)

83 Brent s Method: Zeroin

84 Brent s Method Worst case number of iterations = (n 2 ), where n is number of iterations of Bisection Method for a given tolerance.

85 Modified Brent s Method

86 Modified Brent s Method

87 Modified Brent s Method

88 Modified Brent s Method

89 Interpolation and Approximation (connecting the dots) US population census data available every ten years. What was US population in year 1996? What will US population be in year 2017?

90 Connecting two dots Given two distinct dots, there is a unique line connecting them.

91 Connecting two dots Given two distinct points (p 0, f (p 0 )) and (p 1, f (p 1 )), there is a unique line connecting them P(x) = f (p 0 ) x p 1 + f (p 1 ) x p 0 p 0 p 1 p 1 p 0 def = f (p 0 ) L 0 (x) + f (p 1 ) L 1 (x), with L 0 (x), L 1 (x) unrelated to f (p 0 ), f (p 1 ). L 0 (p 0 ) = 1, L 0 (p 1 ) = 0, L 1 (p 0 ) = 0, L 1 (p 1 ) = 1.

92 Connecting two dots Given two distinct points (p 0, f (p 0 )) and (p 1, f (p 1 )), there is a unique line connecting them P(x) = f (p 0 ) x p 1 p 0 p 1 + f (p 1 ) x p 0 p 1 p 0

93 Connecting n + 1 dots Given n + 1 distinct points (x 0, f (x 0 )), (x 1, f (x 1 )),, (x n, f (x n )), Find a degree n polynomial P(x) = a n x n + a n 1 x n a 1 x + a 0, so that P(x 0 ) = f (x 0 ), P(x 1 ) = f (x 1 ),, P(x n ) = f (x n ).

94 Connecting n + 1 dots Would like to write P(x) as P(x) = f (x 0 )L 0 (x) + f (x 1 )L 1 (x) + + f (x n )L n (x), where L 0 (x), L 1 (x),, L n (x) are unrelated to f (x 0 ), f (x 1 ),, f (x n ). at each node x j, j = 0, 1,, n, f (x j ) = P(x j ) = f (x 0 )L 0 (x j ) + f (x 1 )L 1 (x j ) + + f (x n )L n (x j ) This suggests for all i, j, L i (x j ) = { 1, if i = j, 0, if i j,

95 Largrangian Polynomials P(x) = f (x 0 )L 0 (x) + f (x 1 )L 1 (x) + + f (x n )L n (x), { 1, if i = j, L i (x j ) = 0, if i j, x j is a root of L i (x) for every j i, so L i (x) = j i x x j x i x j.

96 Quadratic Interpolation

97 Quadratic Interpolation: Solution

98 Quadratic Interpolation: Solution

99 Quadratic Interpolation: Solution

100 Quadratic Interpolation: Solution