Interpolation and Polynomial Approximation I

Interpolation and Polynomial Approximation I If f (n) (x), n are available, Taylor polynomial is an approximation: f (x) = f (x 0 )+f (x 0 )(x x 0 )+ 1 2! f (x 0 )(x x 0 ) 2 + Example: e x = 1 + x 1! + x 2 2! + x 3 3! + Chih-Jen Lin (National Taiwan Univ.) 1 / 51

Interpolation and Polynomial Approximation II Need only information of one point If finite terms selected, only approximate points near x 0 Primal use of Taylor polynomials for numerical methods: derivation of numerical techniques but not approximation Chih-Jen Lin (National Taiwan Univ.) 2 / 51

Lagrange Polynomials I Given (x 0, f (x 0 )), (x 1, f (x 1 )),..., (x n, f (x n )), find a function passing all of them Why consider polynomials: the simplest form of functions Degree 1 polynomial passing two points: A line Chih-Jen Lin (National Taiwan Univ.) 3 / 51

Lagrange Polynomials II Given (x 0, f (x 0 )), (x 1, f (x 1 )). Define Then L 0 (x) x x 1, L 1 (x) x x 0, x 0 x 1 x 1 x 0 P(x) L 0 (x)f (x 0 ) + L 1 (x)f (x 1 ) P(x 0 ) = 1f (x 0 ) + 0f (x 1 ) = f (x 0 ) P(x 1 ) = 0f (x 1 ) + 1f (x 1 ) = f (x 1 ) Generalization: higher-degree polynomials Chih-Jen Lin (National Taiwan Univ.) 4 / 51

Lagrange Polynomials III (n + 1) points a polynomial with degree at most n (x 0, f (x 0 )), (x 1, f (x 1 )),..., (x n, f (x n )) Construct L n,k (x). We hope that { 0 i k L n,k (x i ) = 1 i = k n: degree, k: index. Then P(x) = L n,k (x)f (x k ) k=0 Chih-Jen Lin (National Taiwan Univ.) 5 / 51

Lagrange Polynomials IV Thus we have Define P(x i ) = f (x i ) L n,k (x) (x x 0) (x x k 1 )(x x k+1 ) (x x n ) (x k x 0 ) (x k x k 1 )(x k x k+1 ) (x k x n ) Assume x i x j Chih-Jen Lin (National Taiwan Univ.) 6 / 51

Spline Interpolation I Disadvantages of using polynomials Very high degrees Large fluctuation We will try piecewise polynomial approximation Different polynomials on each interval Linear: a series of straight lines joining (x 0, f (x 0 )),..., (x n, f (x n )) Example x = [ 0 1 2 3 ], y = [ 0 1 4 3 ] Chih-Jen Lin (National Taiwan Univ.) 7 / 51

Spline Interpolation II 4 3.5 3 2.5 2 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 Function forms: s 0 (x),..., s n 1 (x) s j (x) = x x j+1 x j x j+1 y j + x x j x j+1 x j y j+1, x j x x j+1 Chih-Jen Lin (National Taiwan Univ.) 8 / 51

Spline Interpolation III Disadvantage: not differentiable at end points Let f (x) be continuous We require s j (x j ) = f (x j ), j = 0,..., n 1, s n 1 (x n ) = f (x n ) s j+1 (x j+1 ) = s j (x j+1 ), j = 0,..., n 2 s j+1(x j+1 ) = s j(x j+1 ), j = 0,..., n 2 How many conditions: 2n + (n 1) = 3n 1 Chih-Jen Lin (National Taiwan Univ.) 9 / 51

Spline Interpolation IV Quadratic piecewise interpolation: 3n variables Each interval: a quadratic polynomial, 3 variables Most common piecewise approximation: cubic polynomials (Spline) 4n variables Chih-Jen Lin (National Taiwan Univ.) 10 / 51

Spline Interpolation V Let s j (x j ) = f (x j ), j = 0,..., n 1, (1) s n 1 (x n ) = f (x n ) s j+1 (x j+1 ) = s j (x j+1 ), j = 0,..., n 2 (2) s j+1(x j+1 ) = s j(x j+1 ), j = 0,..., n 2 s j+1(x j+1 ) = s j (x j+1 ), j = 0,..., n 2 Number of conditions n + 1 + 3(n 1) = 4n 2 Chih-Jen Lin (National Taiwan Univ.) 11 / 51

s j (x j ) = a j = f (x j ), j = 0,..., n 1 Chih-Jen Lin (National Taiwan Univ.) 12 / 51 Spline Interpolation VI Boundary conditions s 0 (x 0 ) = s n 1(x n ) = 0 or s 0(x 0 ) = f (x 0 ) and s n 1(x n ) = f (x n ) Total 4n conditions n cubic polynomials Construct cubic polynomials s j (x) a j + b j (x x j ) + c j (x x j ) 2 + d j (x x j ) 3, j = 0,..., n 1 Immediately

Spline Interpolation VII Now (2) becomes Define a j+1 = s j (x j+1 ) = a j + b j (x j+1 x j ) + c j (x j+1 x j ) 2 +d j (x j+1 x j ) 3, j = 0,..., n 2 h j x j+1 x j a n f (x n ) Earlier a j is only from 0 to n 1 Chih-Jen Lin (National Taiwan Univ.) 13 / 51

Spline Interpolation VIII We have Then a n = s n 1 (x n ) = a n 1 + b n 1 (x n x n 1 ) + c j (x n x n 1 ) 2 + d j (x n x n 1 ) 3 a j+1 = a j + b j h j + c j h 2 j + d j h 3 j, j = 0,..., n 1 (3) Chih-Jen Lin (National Taiwan Univ.) 14 / 51

Spline Interpolation IX Define Earlier b j is only 0 to n 1 Using b n s n 1(x n ) (4) s j(x) = b j + 2c j (x x j ) + 3d j (x x j ) 2 s j+1(x j+1 ) = s j(x j+1 ), j = 0,..., n 2 b j+1 = b j + 2c j h j + 3d j h 2 j, j = 0,..., n 1 (5) Chih-Jen Lin (National Taiwan Univ.) 15 / 51

c j+1 = c j + 3d j (h j ), j = 0,..., n 1 Chih-Jen Lin (National Taiwan Univ.) 16 / 51 Spline Interpolation X Define Since c n s n 1(x n )/2 we have s j+1(x j+1 ) = s j (x j+1 ), j = 0,..., n 2 s j (x) = 2c j + 6d j (x x j ) s j (x j+1 ) = 2c j + 6d j h j s j+1(x j+1 ) = 2c j+1

Spline Interpolation XI Thus so (3) becomes d j = c j+1 c j 3h j a j+1 = a j + b j h j + c j h 2 j + c j+1 c j 3h j = a j + b j h j + h2 j 3 (2c j + c j+1 ), j = 0,..., n (6) 1 h 3 j Chih-Jen Lin (National Taiwan Univ.) 17 / 51

Spline Interpolation XII (5) becomes b j+1 = b j + 2c j h j + 3d j h 2 j = b j + 2c j h j + 3 c j+1 c j hj 2 3h j = b j + 2c j h j + h j (c j+1 c j ) = b j + h j (c j + c j+1 ), j = 0,..., n 1 (7) We have known a j, unknowns: b j, c j Chih-Jen Lin (National Taiwan Univ.) 18 / 51

Spline Interpolation XIII From (6) b j = 1 h j (a j+1 a j ) h j 3 (2c j + c j+1 ), j = 0,..., n 1 That is b j 1 = 1 h j 1 (a j a j 1 ) h j 1 3 (2c j 1 + c j ), j = 1,..., n Chih-Jen Lin (National Taiwan Univ.) 19 / 51

Spline Interpolation XIV Substituting to (7) 1 h j (a j+1 a j ) h j 3 (2c j + c j+1 ) = 1 h j 1 (a j a j 1 ) h j 1 3 (2c j 1 + c j ) + h j 1 (c j 1 + c j ), j = 1,..., n 1 j cannot be zero now because b j = b j 1 +, j = 1,..., n Chih-Jen Lin (National Taiwan Univ.) 20 / 51

Spline Interpolation XV Finally so h j 1 3 c j 1 + 2(h j 1 + h j ) c j + h j 3 3 c j+1 = 1 (a j+1 a j ) 1 (a j a j 1 ) h j h j 1 h j 1 c j 1 + 2(h j 1 + h j )c j + h j c j+1 = 3 (a j+1 a j ) 3 (a j a j 1 ), j = 1,..., n (8) 1 h j h j 1 Chih-Jen Lin (National Taiwan Univ.) 21 / 51

Spline Interpolation XVI Only unknowns: c j, j = 0,..., n Once we know c j, then we can find b j and d j Boundary conditions If s 0 (x 0) = s n 1 (x n) = 0 then s 0 (x) = 2c 0 + 6d 0 (x x 0 ) s 0 (x 0 ) = 2c 0 = 0, s n 1(x n ) = 2c n = 0 (9) Solve c j, j = 0,..., n using (n + 1) equalities: (8) and (9) Chih-Jen Lin (National Taiwan Univ.) 22 / 51

Spline Interpolation XVII We can use other boundary conditions: Example: s 0(x 0 ) = f (x 0 ) and s n 1(x n ) = f (x n ) >> x = [ -2-1 0 1 2] ; >> y = [4-1 2 1 8] ; >> yy=interp1(x,y,-2:0.1:2, spline ) ; >> plot(-2:0.1:2,yy); Chih-Jen Lin (National Taiwan Univ.) 23 / 51

Spline Interpolation XVIII 8 7 6 5 4 3 2 1 0 1 2 2 1.5 1 0.5 0 0.5 1 1.5 2 Chih-Jen Lin (National Taiwan Univ.) 24 / 51

Spline Interpolation XIX >> [sp] = spline(x,y) sp = form: pp breaks: [-2-1 0 1 2] coefs: [4x4 double] pieces: 4 order: 4 dim: 1 spline: returns the spline structure interp1: 1-D interpolation and returns the approximate function value Chih-Jen Lin (National Taiwan Univ.) 25 / 51

Spline Interpolation XX Homework 8-1: Write a program which can solve general spline problems. Test your program by using some data and compare with the result of Matlab Chih-Jen Lin (National Taiwan Univ.) 26 / 51

Function Approximation I Given data points (x i, y i ), i = 1,..., n Find a function to approximate these points Why not just fitting them? That is, y i = f (x i ), i = 1,..., n. Data have noise. There are outliers Chih-Jen Lin (National Taiwan Univ.) 27 / 51

Function Approximation II Chih-Jen Lin (National Taiwan Univ.) 28 / 51

Function Approximation III If fitting all data, functions are not generalized enough for future use Generalization: Data under some distribution Given some samples, your approximate f has the trend Minimizing the error: min f (y i f (x i )) 2 (10) or min f y i f (x i ) Chih-Jen Lin (National Taiwan Univ.) 29 / 51

Function Approximation IV with f (x) = ax + b Assume x R 1, y R 1 (10): least square approximation why (10): easier calculation Chih-Jen Lin (National Taiwan Univ.) 30 / 51

Least Square Approximation I Finding a and b. Let E = (y i (ax i + b)) 2 Chih-Jen Lin (National Taiwan Univ.) 31 / 51

Least Square Approximation II Then min a,b min a,b min a,b (y i (ax i + b)) 2 y 2 i 2y i (ax i + b) + (ax i + b) 2 2y i (ax i + b) + a 2 x 2 i + 2abx i + b 2 Chih-Jen Lin (National Taiwan Univ.) 32 / 51

Least Square Approximation III ( min ( a,b +( 2y i x i )a + ( 2y i )b + ( ) 2x i )ab + nb 2 x 2 i )a 2 Chih-Jen Lin (National Taiwan Univ.) 33 / 51

Least Square Approximation IV First derivative E a = 0 ( 2y i x i ) + 2( E b = 0 ( 2y i ) + ( x 2 i )a + ( 2x i )b = 0 2x i )a + 2nb = 0 Chih-Jen Lin (National Taiwan Univ.) 34 / 51

Least Square Approximation V Rearrange terms: Define S xx = S xy = x 2 i, S x = x i y i, S y = x i y i Chih-Jen Lin (National Taiwan Univ.) 35 / 51

Least Square Approximation VI Now Then Issues: S xx a + S x b = S xy S x a + nb = S y a = ns xy S x S y ns xx S 2 x b = S xxs y S x S xy ns xx S 2 x Chih-Jen Lin (National Taiwan Univ.) 36 / 51

Least Square Approximation VII Does E = 0 imply optimality? We need secondary derivative (i.e. 2 E) to be positive semi-definite [ E ] 2 E E = a a a b E E [ b a b b] 2Sxx 2S = x 2S x 2n Chih-Jen Lin (National Taiwan Univ.) 37 / 51

Least Square Approximation VIII S xx > 0, n > 0, and determinant: = S xx n Sx 2 1 2 ( x 2 i x i ) 2 0 Remember (a 1 b 1 + + a n b n ) 2 (a 2 1 + + a 2 n)(b 2 1 + + b 2 n) Chih-Jen Lin (National Taiwan Univ.) 38 / 51

Least Square Approximation IX Nonlinear least-square min f E = m (y i f (x i )) 2 with f (x) = ax 2 + bx + c Quadratic least square Now you have three variables and three equations Homework 8-2 1 Write down the three equations of E = 0 Chih-Jen Lin (National Taiwan Univ.) 39 / 51

Least Square Approximation X 2 Write a program doing quadratic least square 3 Randomly generate some (x i, y i ) and draw a figure showing your approximation Chih-Jen Lin (National Taiwan Univ.) 40 / 51

Data in Higher Dimensional Space I If x R m, m > 1 f (x) = a T x + b, a R m Let E = (y i (a T x i + b)) 2 Chih-Jen Lin (National Taiwan Univ.) 41 / 51

Data in Higher Dimensional Space II Then n min (y i (a T x i + b)) 2 a,b min a,b min a,b min n n ( n a,b n y 2 i 2y i (a T x i + b) + (a T x i + b) 2 2y i(a T x i + b) + (a T x i ) 2 + 2ba T x i + b 2y ix i ) T a + ( n (at x i ) 2 + ( n 2y i)b + 2x i) T ab + nb 2 Chih-Jen Lin (National Taiwan Univ.) 42 / 51

Data in Higher Dimensional Space III First derivative E a = 0 ( 2y i x i ) + 2 (a T x i )x i + ( 2x i )b = 0 E b = 0 ( 2y i ) + ( 2x i ) T a + 2nb = 0 Chih-Jen Lin (National Taiwan Univ.) 43 / 51

Data in Higher Dimensional Space IV Note (a T x i )x i = x i xi T : an m by m matrix Define S xx = S xy = x i x T i, S x = y i x i, S y = x i xi T a x i y i Chih-Jen Lin (National Taiwan Univ.) 44 / 51

Data in Higher Dimensional Space V Solve a linear system of a and b [ ] [ ] Sxx S x a Sx T = n b m + 1 variables [ Sxy S y ] Chih-Jen Lin (National Taiwan Univ.) 45 / 51

Application: Time Series Prediction I Given (y 1,, y n ) as known series Predict y n+1,..., Error propagation is the usual difficulty Chih-Jen Lin (National Taiwan Univ.) 46 / 51

A Practical Example I Electricity load forecasting. Given Electricity load per half hour from 1997 to 1998 Average daily temperature from 1995 to 1998 List of holidays Goal: Predict daily maximal load of January 1999 i.e., predict 31 values Chih-Jen Lin (National Taiwan Univ.) 47 / 51

Regression for Time Series Prediction I Finding a function f (x) = a T x + b Consider (y t,, y t 1 ) as attributes (features) of x i y t as the target value y i : usually called embedding dimension Solving (a, b) using (x 1, y 1 ),..., (x n, y n ) One-step ahead prediction Chih-Jen Lin (National Taiwan Univ.) 48 / 51

Regression for Time Series Prediction II Prediction: Starting from the last segment (y n +1,..., y n ) ŷ n+1 Repeat by using newly predicted values Chih-Jen Lin (National Taiwan Univ.) 49 / 51

Evaluation of Time Series Prediction I MSE (Mean Square Error): (y i ŷ i ) 2 MAPE (Mean absolute percentage error) 1 n y i ŷ i y i Error propagation: larger error later Unfair to earlier prediction if MSE is used Chih-Jen Lin (National Taiwan Univ.) 50 / 51

Evaluation of Time Series Prediction II There are other criteria Chih-Jen Lin (National Taiwan Univ.) 51 / 51