Interpolation, Extrapolation & Polynomial Approximation Lectures based on course notes by Pablo Laguna and Kostas Kokkotas revamped by Deirdre Shoemaker Spring 2014
Introduction In many cases, a function f (x) is only known at a set of points {x 1, x 2,..., x N }, and we are interested estimating its value at an arbitrary point. Estimating f (x) with x [x 1, x N ] is called interpolation. Estimating f (x) with x / [x 1, x N ] is called extrapolation.
Polynomial Approximations Polynomial functions are the most popular. Rational and trigonometric functions are also used quite frequently. We will study the following methods for polynomial approximations: Lagrange s Polynomial Hermite Polynomial Taylor Polynomial Cubic Splines
Lagrange Polynomial Consider the following data: x 0 x 1 x 2 x 3 x 3.2 2.7 1.0 4.8 f (x) 22.0 17.8 14.2 38.3 f 0 f 1 f 2 f 3 A possible interpolating polynomial is : P 3 (x) = ax 3 + bx 2 + cx + d (i.e. a 3th order polynomial). This leads to 4 equations for the 4 unknown coefficients. The solutions are a = 0.5275, b = 6.4952, c = 16.117, d = 24.3499 Thus P 3 (x) = 0.5275x 3 + 6.4952x 2 16.117x + 24.3499
Lagrange Polynomial For a large number of points, this procedure could be quite laborious. Given a set of n + 1 points {x i, f i } i=0,...,n, Lagrange developed a direct way to find the polynomial P n (x) = f 0 L 0 (x) + f 1 L 1 (x) +... + f n L n (x) = n f i L i (x) i=0 where L i (x) are the Lagrange coefficient polynomials The coefficients are given by L j (x) = (x x 0)(x x 1 )...(x x j 1 )(x x j+1 )...(x x n ) (x j x 0 )(x j x 1 )...(x j x j 1 )(x j x j+1 )...(x j x n ) n x x k = with k j x j x k k=0
Notice L j (x k ) = δ jk = where δ jk is Kronecker s symbol. Therefore, { 0 if j k 1 if j = k P n (x j ) = n f i L i (x j ) = i=0 n f i δ ij = f (x j ) i=0 The error when using Lagrange interpolation is: E n (x) = f (x) P n (x) = (x x 0 )(x x 1 )...(x x n ) f (n+1) (ξ) (n + 1)! where ξ [x 0, x N ] Notice that Lagrange polynomial applies to both evenly and unevenly spaced points.
Lagrange Polynomial Formula Derivation Consider the case of three points. f (x 1 ) = f (x) + (x 1 x) f (x) + 1 2 (x 1 x) 2 f (x) +... f (x 2 ) = f (x) + (x 2 x) f (x) + 1 2 (x 2 x) 2 f (x) +... f (x 3 ) = f (x) + (x 3 x) f (x) + 1 2 (x 3 x) 2 f (x) +... Is is reasonable to think that f (x) p(x) and f (x) p (x). Thus
Lagrange Polynomial Formula Derivation f (x 1 ) = p(x) + (x 1 x) p (x) + 1 2 (x 1 x) 2 p (x) f (x 2 ) = p(x) + (x 2 x) p (x) + 1 2 (x 2 x) 2 p (x) f (x 3 ) = p(x) + (x 3 x) p (x) + 1 2 (x 3 x) 2 p (x) We have a system of three equations for three unknowns (p(x), p (x), p (x)). Solving for p(x) one gets the desired answer. Notice that one can also get the expression for the derivatives.
Lagrange Polynomial : Example Find the Lagrange polynomial that approximates the function y = cos(πx) using the following data. x i 0 0.5 1 f i 1 0.0-1 The Lagrange coeffiecient polynomials are: L 1 (x) = (x x 2)(x x 3 ) (x 0.5)(x 1) = (x 1 x 2 )(x 1 x 3 ) (0 0.5)(0 1) = 2x 2 3x + 1, L 2 (x) = (x x 1) (x x 3 ) (x 0)(x 1) = (x 2 x 1 )(x 2 x 3 ) (0.5 0)(0.5 1) = 4x 2 + 4x L 3 (x) = (x x 1)(x x 2 ) (x 0)(x 0.5) = (x 3 x 1 )(x 3 x 2 ) (1 0)(1 0.5) = 2x 2 x thus P(x) = (1)(2x 2 3x +1)+(0.0)( 4x 2 +4x)+( 1)(2x 2 x) = 2x + 1
The error from using P(x) = 2x + 1 will be: E(x) = (x x 0 )(x x 1 )... (x x n ) f (n+1) (ξ) (n + 1)! for example E(x = 0.25) 0.24. = (x x 0 )(x x 1 )(x x 2 ) f (3) (ξ) (3)! = x (x 0.5) (x 1) π3 sin(πξ) 3!
Hermite Polynomial Interpolation This type of interpolation is very useful when in addition to the values of f (x) one also has its derivative f (x) P n (x) = n A i (x)f i + i=1 n i=1 B i (x)f i where A i (x) = [1 2(x x i )L i (x i)] [L i (x)] 2 B i (x) = (x x i ) [L i (x)] 2 and L i (x) are the Lagrange coefficients.
Hermite Polynomial Interpolation Example The Lagrange coefficients are: k x k y k y k 0 0 0 0 1 4 2 0 L 0 (x) = x x 1 = x 4 x 0 x 1 0 4 = x 4 4 L 0(x) = 1 x 0 x 1 = 1 4 L 1(x) = L 1 (x) = x x 0 x 1 x 0 = x 4 1 x 1 x 0 = 1 4 Thus A 0 (x) = [ 1 2 L 0(x x 0 ) ] [ ( L 2 0 = 1 2 1 ) ] ( ) 2 x 4 (x 0) 4 4 A 1 (x) = [ 1 2 L 0(x x 1 ) ] [ L 2 1 = 1 2 1 ] ( x ) 2 ( (x 4) = 3 x ) ( x ) 2 4 4 2 4 ( ) 2 ( ) 2 x 4 x 4 ( x ) 2 B 0 (x) = (x 0) = x B 1 (x) = (x 4) 4 4 4 And P(x) = (6 x) x 2 16.
Taylor Polynomial Interpolation Instead of finding a polynomial P(x) such that P(x) = f (x) at N points (Lagrange) or that both P(x) = f (x) and P (x) = f (x) at N points (Hermite), Taylor polynomial interpolation consists of, for a given value x 0, finding a function P(x) that agrees up to the Nth derivative at x 0. That is: P (i) (x 0 ) = f (i) (x 0 ) for i = 0, 1,..., n and the Taylor polynomial is given by P N (x) = N i=0 f (i) (x 0 ) (x x 0 ) i i! The error is given by E N (x) = (x x 0 ) N+1 f (N+1) (x 0 ) (N + 1)!
Taylor Polynomial Interpolation Example Find out how many Taylor expansion terms are required for 13-digit approximation of e = 2.718281828459... Let y(x) = e x. All the derivatives are y (i) (x) = e x. Thus at x = 0, one has y (i) (x = 0) = 1. Therefore Evaluate at x = 1 P n (x) = P n (1) = n i=0 n i=0 x i i! 1 i! and E n (x) = x n+1 (n + 1)! and E n (1) = 1 (n + 1)! and you will find that you need n = 15 and E 15 = 1.433 10 13
Interpolation with Cubic Splines In some cases the typical polynomial approximation cannot smoothly fit certain sets of data. For instance, consider the function f (x) = 0 1 x 0.2 1 5 x 0.2 < x < 0.2 0 0.2 x 1.0 We can easily verify that we cannot fit the above data with any polynomial degree! P(x) = 1 26x 2 + 25x 4
Spline Fitting General Idea: Consider the tabulated function y i = y(x i ) with i = 0,..., N. Split the domain [x 0, x N ] into N intervals [x i, x i+1 ] with i = 0,..., N 1. For each interval construct a cubic polynomial or spline such that neighboring splines have the same slope and curvature at their joining point. That is, the essential idea is to fit a piecewise function of the form s 0 (x) if x 0 x x 1 s 1 (x) if x 1 x x 2 S(x) =. s N 1 (x) if x N 1 x x N where s i (x) = a i (x x i ) 3 + b i (x x i ) 2 + c i (x x i ) + d i s i (x) = 3 a i(x x i ) 2 + 2 b i (x x i ) + c i s i (x) = 6 a i (x x i ) + 2 b i
Cubic Spline Interpolation Properties S(x) interpolates all data points. That is, S(x i ) = y i. Since x i [x i, x i+1 ], one has that y i = s i (x i ) = a i (x i x i ) 3 + b i (x i x i ) 2 + c i (x i x i ) + d i = d i S(x) is continuous on the interval [x 0, x N ], thus s i (x i ) = s i 1 (x i ) with s i (x i ) = d i s i 1 (x i ) = a i 1 (x i x i 1 ) 3 + b i 1 (x i x i 1 ) 2 + c i 1 (x i x i 1 ) + d i 1 = a i 1 h 3 i 1 + b i 1h 2 i 1 + c i 1h i 1 + d i 1 where h i 1 = x i x i 1. Therefore d i = a i 1 h 3 i 1 + b i 1h 2 i 1 + c i 1h i 1 + d i 1
Cubic Spline Interpolation Properties S (x) is continuous on the interval [x 0, x N ], thus s i (x i) = s i 1 (x i) with s i (x i) = c i s i 1 (x i) = 3 a i 1 (x i x i 1 ) 2 + 2 b i 1 (x i x i 1 ) + c i 1 = 3 a i 1 h 2 i 1 + 2 b i 1h i 1 + c i 1 Therefore c i = 3 a i 1 h 2 i 1 + 2 b i 1h i 1 + c i 1
Cubic Spline Interpolation Properties S (x) is continuous on the interval [x 0, x N ], thus s i (x i ) = s i 1 (x i) with s i (x i ) = 2 b i s i 1 i) = 6, a i 1 (x i x i 1 ) + 2 b i 1 = 6 a i 1 h i 1 + 2 b i 1 Therefore 2 b i = 6 a i 1 h i 1 + 2 b i 1
In Summary s i (x i ) = s i 1 (x i) b i = 3 a i 1 h i 1 + b i 1 s i (x i) = s i 1 (x i) c i = 3 a i 1 h 2 i 1 + 2 b i 1h i 1 + c i 1 s i (x i ) = s i 1 (x i ) d i = a i 1 h 3 i 1 + b i 1h 2 i 1 + c i 1h i 1 + d i 1
Let s define M i s i (x i ). Then from s i (x i ) = 2 b i and 2 b i+1 = 6 a i h i + 2 b i, we have that Also, from b i = M i 2, a i = M i+1 M i 6h i s i (x) = a i (x x i ) 3 + b i (x x i ) 2 + c i (x x i ) + d i and s i (x i ) = y i, s i (x i+1 ) = y i+1 and, we have that Therefore y i+1 = M i+1 M i hi 3 + M i 6h i 2 h2 i + c i h i + y i c i = y i+1 y i h i h i 6 (2M i + M i+1 ) and d i = y i
Recall the condition that the slopes of the two cubics joining at (x i, y i ) are the same. That is s i (x i) = s i 1 (x i), which yielded c i = 3 a i 1 h 2 i 1 + 2 b i 1h i 1 + c i 1 Substitution of a, b, c and d yields: y i+1 y i 2h im i + h i M i+1 h i 6 ( ) Mi M i 1 = 3 hi 1 2 6h i 1 +2 M i 1 2 h i 1 + y i y i 1 2h i 1M i 1 + h i 1 M i h i 1 6 and by simplifying we get: ( yi+1 y i h i 1 M i 1 + 2 (h i 1 + h i ) M i + h i M i+1 = 6 h i y ) i y i 1 h i 1
If we have n + 1 points, the relationship ( yi+1 y i h i 1 M i 1 + 2 (h i 1 + h i ) M i + h i M i+1 = 6 h i y ) i y i 1 h i 1 can be applied to the n 1 internal points. Thus we create a system of n 1 equations for the n + 1 unknown M i. This system can be solved if we specify the values of M 0 and M n.
The system of n 1 equations with n + 1 unknown will be written as: = 6... h 0 2(h 0 + h 1 ) h 1 h 1 2(h 1 + h 2 ) h 2 h 2 2(h 2 + h 3 ) h 3...... h n 2 2(h n 2 + h n 1 ) h n 1... y 2 y 1 h 1 y 1 y 0 h 0 y 3 y 2 h 2 y 2 y 1 h 1... y n y n 1 h n 1... y n 1 y n 2 h n 2 M 0 M 1 M 2 : M n 2 M n 1 Mn Recall ( yi+1 y i h i 1 M i 1 + 2 (h i 1 + h i ) M i + h i M i+1 = 6 h i y ) i y i 1 h i 1
From the solution of this linear systems we get the coefficients a i, b i, c i and d i via the relations: a i = M i+1 M i 6h i b i = M i 2 c i = y i+1 y i 2h im i + h i M i+1 h i 6 d i = y i
Let s define and Y 6 y 2 y 1 h 1 y 1 y 0 h 0 y 3 y 2 h 2 y 2 y 1 h 1... y n y n 1 h n 1 y n 1 y n 2 h n 2 M M 1 M 2 : M n 2 M n 1
Choice I Take, M 0 = 0 and M n = 0. This will lead to the solution of the (n 1) (n 1) linear system: where H H M = Y 2(h 0 + h 1 ) h 1 h 1 2(h 1 + h 2 ) h 2 h 2 2(h 2 + h 3 ) h 3... h n 2 2(h n 2 + h n 1 )
Choice II Take, M 0 = M 1 and M n = M n 1. This will lead to the solution of the (n 1) (n 1) linear system: H M = Y where H 3h 0 + 2h 1 h 1 h 1 2(h 1 + h 2 ) h 2 h 2 2(h 2 + h 3 ) h 3...... h n 2 2h n 2 + 3h n 1
Choice III Use linear extrapolation M 1 M 0 h 0 = M 2 M 1 h 1 M 0 = (h 0 + h 1 )M 1 h 0 M 2 h 1 M n M n 1 h n 1 Then = M n 1 M n 2 h n 2 M n = (h n 2 + h n 1 )M n 1 h n 1 M n 2 h n 2 M (h 0 +h 1 )(h 0 +2h 1 ) h 1 h 2 1 h2 0 h 1 h 1 2(h 1 + h 2 ) h 2 h 2 2(h 2 + h 3 ) h 3...... h 2 n 2 h2 n 1 h n 2 (h n 1 +h n 2 )(h n 1 +2h n 2 ) h n 2
Choice IV Force the slopes at the end points to assume certain values. If f (x 0 ) = A and f (x n ) = B then ( ) y1 y 0 2h 0 M 0 + h 1 M 1 = 6 A h 0 ( h n 1 M n 1 + 2h n M n = 6 B y ) n y n 1 h n 1 Then H 2h 0 h 1 h 0 2(h 0 + h 1 ) h 1 h 1 2(h 1 + h 2 ) h 2... h n 2 2h n 1
Interpolation with Cubic Splines : Example Fit a cubic spline in the data (y = x 3 8): x 0 1 2 3 4 y -8-7 0 19 56 Condition I : M 0 = 0, M 4 = 0 4 1 0 M 1 1 4 1 M 2 = 0 1 4 M 3 36 72 108 Condition II : M 0 = M 1, M 4 = M 3 s 1 0 1 4 1 M 1 M 2 = 36 72 0 1 s M 3 108 M 1 = 6.4285 M 2 = 10.2857 M 3 = 24.4285 M 1 = M 0 = 4.8 M 2 = 1.2 M 3 = 19.2 = M 4
Condition III : 6 0 0 1 4 1 0 0 6 M 1 M 2 M 3 = 36 72 108 M 0 = 0 M 1 = 6 M 2 = 12 M 3 = 18 M 4 = 24 Condition IV : 2 1 0 0 0 1 4 1 0 0 0 1 4 1 0 0 0 1 4 1 0 0 0 1 2 M 0 M 1 M 2 M 3 M 4 = 6 36 72 108 66 M 0 = 0 M 1 = 6 M 2 = 12 M 3 = 18 M 4 = 24
Tri-diagonal Matrix The system of equations in the cubic spline fitting method has the following form: b 1 c 1 0... a 2 b 2 c 2... 0 a 3 b 3......... b N 2 c N 2 0... a N 1 b N 1 c N 1... 0 a N b N x 1 x 2 x 3... x N 2 x N 1 x N = d 1 d 2 d 3... d N 2 d N 1 d N The matrix has the so-called tri-diagonal form. That is, each equations has the form a i x i 1 + b i x i + c i x i+1 = d i
Method to Solve a Tri-diagonal System Redefine and c i = d i = c i b i i = 1 c i b i c i 1 a i i = 2, 3,..., n 1 d i b i i = 1 d i d i 1 a i b i c i 1 a i i = 2, 3,..., n
With these new coefficients the systems takes the form 1 c 1 0... 0 1 c 2... 0 0 1......... 1 c N 2 0... 0 1 c N 1... 0 0 1 x 1 x 2 x 3... x N 2 x N 1 x N The systems can be solved using back substitution = d 1 d 2 d 3... d N 2 d N 1 d N x n = d n i = n x i = d i c i x i+1 i = n 1, n 2, n 3,..., 1
Chi-Square Fitting Consider a set of N data points {x i, y i } i=0,...,n 1 with standard deviations σ i. The objective is to find a model function f (x; β) with β = {β 0,..., β M } a set of M adjustable parameters. Such that ( N 1 χ 2 y i f (x i ; β) ) 2 σ i i=0 is a minumum. Notice that r i y i f (x i ; β) are the residuals.
Fitting Data to a Straight Line Consider the case f (x; β) = a + b x. Therefore N 1 ( χ 2 yi a b x i i=0 σ i ) 2 To minimize χ 2 with respect to a and b, we need to solve 0 = χ2 N 1 a = 2 i=0 0 = χ2 N 1 b = 2 i=0 y i a b x i σ 2 i x i (y i a b x i ) σ 2 i
Define Then S N 1 i=0 1 σ 2 i S xx N 1 i=0 S x x 2 i σ 2 i N 1 i=0 x i σ 2 i S xy N 1 i=0 S y x i y i σ 2 i N 1 i=0 y i σ 2 i a S + b S x = S y a S x = b S xx = S xy Finally S S xx S 2 x a = S xx S y S x S xy b = S S xy S x S y
Propagation of errors with Thus N 1 σf 2 =a,b = i=0 σ 2 i ( f y i ) 2 a y i = S xx S x x i σ 2 i b y i = S x i S x σ 2 i σ 2 a = S xx / σ 2 b = S/
Variance, Covariance and Correlation Consider the case in which σ i = 1. Then S N 1 i=0 1 σ 2 i = N S x N 1 i=0 x i σ 2 i = N x S y N 1 i=0 y i σ 2 i = N y S xx N 1 i=0 x 2 i σ 2 i = N x 2 S xy where the over line denotes average. Thus N 1 i=0 x i y i σ 2 i = N x y S S xx S 2 x = N 2 x 2 N 2 x 2 = N 2 (x 2 x 2 ) a = S xx S y S x S xy b = S S xy S x S y = x 2 y x xy x 2 x 2 = xy x y x 2 x 2
Notice where a = x 2 y x xy x 2 x 2 = y x b = xy x y x 2 x 2 ( xy x y x 2 x 2 = x 2 y x 2 y + x 2 y x xy x 2 x ) 2 = y x b Var[x] = x 2 x 2 Variance Cov[x,y] = xy x y Covariance
Finally, from y = a + b x with a = y x b b = xy x y x 2 x 2 we have that y y = (x x) y y y 2 y 2 = ( ) xy x y x 2 x 2 x x x r 2 x 2 xy where r xy = xy x y x 2 x 2 y 2 y 2 Correlation