CHAPTER 8. Approximation Theory

CHAPTER 8 Approximation Theory 1. Introduction We consider two types of problems in this chapter: (1) Representing a function by a simpler function that is easier to evaluate. () Fitting functions to given data and finding the best function in a certain class that can be used to represernt the data. Given a set of data points class of functions such that. Discrete Least Squares Approximation (x i, y i ) m, we look for the best function f of some m yi f(x i ) is minimized. 174

. DISCRETE LEAST SQUARES APPROIMATION 175 In the figure on the previous page, we see the quadratic polynomial that best approximates our data points in the sense of least squares. Notice, as opposed to interpolation, there is no requirement that any of the data points actually lie on the graph of the approximating function. Instead, we choose the quadratic the sum of the squares of the vertical distances from the points to the curve is minimized. This choice is made using the theory of max-min problems from calculus. Linear Least Squares This is illustrative and the easiest to work with. Suppose we have m data points (x i, y i ) m that appear to be somewhat linear, so we want to find the line y = a 1 x + a 0 that best approximates the data points in the sense of least squares, i.e., we want to minimize the function m E(a 0, a 1 ) = yi (a 1 x 1 + a 0 ). Set @E @a 0 = @E @a 1 = m (y i a 1 x i a 0 )( 1) = 0 m (y i a 1 x i a 0 )( x i ) = 0 We get the system of linear equations in a 0 and a 1 : m m a 0 m + a 1 x i = a 0 m We solve for a 0 and a 1 to get x i + a 1 m x i = y i m x i y i

176 8. APPROIMATION THEORY P m P m a 0 = x i y P m i x P m iy i x i m P m P m x i x i a 1 = m P m x P m iy i x P m i y i m P m P m x i x i Maple. See regression.mw or regression.pdf. 6. Trigonometric Polynomial Approximation Trigonometric functions are used to approximate functions that have periodic behavior, functions with the property that for some constant T, f(x + T ) = f(x) for all x. We can generally also transform the problem so that T = and restrict the approximation to the interval [, ]. Definition (Page 333). An integrable function w is a weight function on the interval I if w(x) 0 for all x I, but w 6 0 (not identically 0) on any subinterval of I. The purpose of a weight function is to assign varying degrees of importance on certain portions of the interval. For example, if w(x) = 1 x on I = [0, 1], more importance is given to the points near 0 than the points near 1. Definition (Page 334). The set of functions { 0, 1,..., n} is said to be orthogonal for the interval [a, b] with respect to the weight function w if for some positive numbers 0, 1,..., n, Z ( b 0, when j 6= k w(x) j (x) k (x) dx = k > 0, when j = k. a If, in addition, k = 1 for each k = 0, 1,..., n, the set is said to be orthonormal.

6. TRIGONOMETRIC POLYNOMIAL APPROIMATION 177 Continuous Least Squares for Orthogonal Functions Theorem (Page 335). If { 0, 1,..., n} is an orthogonal set of functions on an interval [a, b] with respect to the weight function w, then the least squares approximation to f on [a, b] with respect to w is n P (x) = a k k (x), a k = k=0 R b a w(x) k(x)f(x) dx R b a w(x)( k(x)) dx = 1 k Z b a w(x) k (x)f(x) dx. For each positive integer n, the set T n of trigonometric polynomials of degree less than or equal to n is the set of all linear combinations of { 0, 1,..., n 1}, 0(x) = 1, k(x) = cos kx, for each k = 1,,..., n, and n+k(x) = sin kx, for each k = 1,,..., n. The set { 0, 1,..., n 1} is orthogonal on [, ] with respect to the weight function w(x) 1. For f C[ T n is, ], the continuous least squares approximation by functions in a k = 1 Z S n (x) = 1 a 0 + n k=1 a k k (x) f(x) k (x) dx for each k = 0, 1,..., n 1.

178 8. APPROIMATION THEORY Another way to express this is S n (x) = a 0 + a n cos nx + a k = 1 b k = 1 Z Z f(x) cos kx dx n 1 k=1 a k cos kx + b n sin kx for each k = 0, 1,..., n, and f(x) sin kx dx for each k = 1,,..., n 1. lim S n(x) is called the Fourier series of f. Fourier series are used to describe n!1 the solution of various ordinary and partial di erential equations that occur in physical situations. Discrete Least Squares Approximation and Interpolation We now look at a discrete analog to Fourier series that is useful for large amounts of equally spaced points. Suppose we have data points (x j, y j ) 1 with the x j equally partitioning [, ], i.e., j x j = + for each j = 0, 1,..., n 1. m Although not in our set of points, x =. Also, note that x 0 =. Our goal: to determine the trigonometric polynomial in T n that minimizes E(S n ) = n yj S n (x j ),

6. TRIGONOMETRIC POLYNOMIAL APPROIMATION 179 i.e., we need to choose constants a 0, a 1,..., a n, b 1, b,..., b n 1 such that ( n h a0 E(S n ) = y j + a n 1 i ) n cos nx + a k cos kn + b n sin kx is minimized. Finding the constants is simplified by the fact that { 0, 1,..., n 1} is orthogonal with respect to summation over equally spaced points (x j, y j ) 1 in [, ], i.e., Why? for k 6= l, k=1 k(x j ) l (x j ) = 0. Theorem. If r is not a multiple of, r an integer, cos rx j = 0 and if r is not a multiple of m, Proof. (cos rx j ) = m 1 and and 1 sin rx j = 0, (sin rx j ) = m. Recall. : n (1) r i = 1 + r + r + + r n = 1 rn+1 1 r. () i = 1. (3) e iz = cos z + i sin z.

180 8. APPROIMATION THEORY Note. : (1) e irx j = e ir( +j m ) = e ir e irj m. () e ir = cos r + i sin r. (3) sin t 1 cos t = 1 sin(t1 t ) + sin(t 1 + t ). Then cos rx j + i cos rxj + i sin rx j = e ir e ir 1 =) e irj m = {z} {r not a multiple of } cos r + i sin r r not a multiple of m =) Similarly, 1 e ir m cos rx j = 0 (cos rx j ) = 1 " 1 1 + (sin rx j ) = m. 1 and cos rx j # sin rx j = e irx j = e ir 1 = e ir e ir( +j m ) = eir 1 e ir m 1 [1 + 0] 1 e ir m sin rx j = 0. 1 1 + cos rxj = = 1 h i + 0 = m. = = 0

Sample case: " 1 Overall, we have 6. TRIGONOMETRIC POLYNOMIAL APPROIMATION 181 k(x j ) n+l (x j ) = sin(l k)x j + (cos kx j )(sin lx j ) = 1 # sin(l + k)x j = 1 [0 + 0] = 0. Theorem. The constants in the summation S n (x) = a 0 + a n 1 n cos nx + a k cos kx + b k sin kx that minimize the least squares sum are and k=1 E(a 0,..., a n, b 1,..., b n 1 ) = a k = 1 m b k = 1 m Maple. See trigpoly.mw or trigpoly.pdf. yj S n (x j ) y j cos kx j, for each k = 0, 1,..., n, y j sin kx j, for each k = 1,,..., n 1.

18 8. APPROIMATION THEORY 7. Fast Fourier Transforms The interpolatory trigonometric polynomial on the data points (x j, y j ) 1 j with x j = + is the least squares polynomial from T n for this collec- m tion of points. However, if we want the coe cients to have the same form with n = m as the discrete least squares approximation, a modification is needed. Interpolation requires computing (cos mx j ) = instead of for r not a multiple of m. (cos rx j ) = m 1 We then replace a m with a m, giving an interpolating polynomial of the form and S m (x) = a 0 + a n cos mx a k = 1 m b k = 1 m + m 1 k=1 a k cos kx + b k sin kx y j cos kx j, for each k = 0, 1,..., m, y j sin kx j, for each k = 1,,..., m 1. This procedure rquires approximately () multiplications and () additions. In problems with many thousands of data points, round-o error can dominate the approximation.

7. FAST FOURIER TRANSFORMS 183 The fast Fourier transform (FFT) algorithm requires O(m log m) multiplications and O(m log m) additions. Suppose the number of data points is a power of, i.e., m = p. Then, for j x j = + with j = 0, 1,..., 1, we have = m p+1 points (x j, z j ) 1 the z j may be complex (we usually use y j if they are real). The FFT procedure computes the complex coe cients c k in the formula c k = F (x) = 1 m k=0 c k e ikx, y j e ijk/m, for each k = 0, 1,..., 1. Then 1 m c ke i k = a k + ib k. Maple. In Maple, FourierTransform in the Discrete Transforms Package uses c k = p 1 y j e ijk/m, for each k = 0, 1,..., 1, m so See t.mw or t.pdf. a k = ( 1) k r m <(c k) and b k = ( 1) k r m =(c k).