Jim Lmbers MAT 77 Fll Semester 1-11 Lecture 1 Notes These notes correspond to Sections 9. nd 9.3 in the text. Best Approximtion in the -norm Suppose tht we wish to obtin function f n (x) tht is liner combintion of given functions {φ j (x)} n, nd best fits function f(x) t discrete set of dt points {(x i, f(x i ))} m in lest-squres sense. Tht is, we wish to find constnts {c j } n such tht m m [f n (x i ) f(x i )] = c j φ j (x i ) f(x i ) is minimized. This cn be ccomplished by solving system of n + 1 liner equtions for the {c j }, known s the norml equtions. Now, suppose we hve continuous set of dt. Tht is, we hve function f(x) defined on n intervl [, b], nd we wish to pproximte it s closely s possible, in some sense, by function f n (x) tht is liner combintion of given functions {φ j (x)} n. If we choose m eqully spced points {x i } m in [, b], nd let m, we obtin the continuous lest-squres problem of finding the function f n (x) = c j φ j (x) tht minimizes E(c, c 1,..., c n ) = [f n (x) f(x)] dx = c j φ j (x) f(x) To obtin the coefficients {c j } n, we cn proceed s in the discrete cse. We compute the prtil derivtives of E(c, c 1,..., c n ) with respect to ech c k nd obtin E b = φ k (x) c j φ j (x) f(x) dx, c k nd requiring tht ech prtil derivtive be equl to zero yields the norml equtions [ ] φ k (x)φ j (x) dx c j = φ k (x)f(x) dx, k =, 1,..., n. 1 dx.
We cn then solve this system of equtions to obtin the coefficients {c j } n. This system cn be solved s long s the functions {φ j (x)} n re linerly independent. Tht is, the condition c j φ j (x), x [, b], is only true if c = c 1 = = c n =. In prticulr, this is the cse if, for j =, 1,..., n, φ j (x) is polynomil of degree j. This cn be proved using simple inductive rgument. Exmple We pproximte f(x) = e x on the intervl [, 5] by fourth-degree polynomil The norml equtions hve the form f 4 (x) = c + c 1 x + c x + c 3 x 3 + c 4 x 4. ij c j = b i, i =, 1,..., 4, or, in mtrix-vector form, Ac = b, where ij = x i x j dx = b i = Integrtion by prts yields the reltion x i+j dx = 5i+j+1, i, j =, 1,..., 4, i + j + 1 x i e x dx, i =, 1,..., 4. b i = 5 i e 5 ib i 1, b = e 5 1. Solving this system of equtions yields the polynomil f 4 (x) =.3 6.6x + 9.5487x 3.86x 3 +.674x 4. As Figure 1 shows, this polynomil is brely distinguishble from e x on [, 5]. However, it should be noted tht the mtrix A is closely relted to the n n Hilbert mtrix H n, which hs entries [H n ] ij = 1 i + j 1, 1 i, j n. This mtrix is fmous for being highly ill-conditioned, mening tht solutions to systems of liner equtions involving this mtrix tht re computed using floting-point rithmetic re highly sensitive to roundoff error. In fct, the mtrix A in this exmple hs condition number of 1.56 1 7, which mens tht chnge of size ε in the right-hnd side vector b, with entries b i, cn cuse chnge of size 1.56ε 1 7 in the solution c.
Figure 1: Grphs of f(x) = e x (red dshed curve) nd 4th-degree continuous lest-squres polynomil pproximtion f 4 (x) on [, 5] (blue solid curve) Inner Product Spces As the preceding exmple shows, it is importnt to choose the functions {φ j (x)} n wisely, so tht the resulting system of norml equtions is not unduly sensitive to round-off errors. An even better choice is one for which this system cn be solved nlyticlly, with reltively few computtions. An idel choice of functions is one for which the tsk of computing f n+1 (x) cn reuse the computtions needed to compute f n (x). To tht end, recll tht two m-vectors u = u 1, u,..., u m nd v = v 1, v,..., v m re orthogonl if m u v = u i v i =, where u v is the dot product, or inner product, of u nd v. By viewing functions defined on n intervl [, b] s infinitely long vectors, we cn generlize the inner product, nd the concept of orthogonlity, to functions. To tht end, we define the inner product of two rel-vlued functions f(x) nd g(x) defined on the intervl [, b] by f, g = f(x)g(x) dx. 3
Then, we sy f nd g re orthogonl with respect to this inner product if f, g =. In generl, n inner product on vector spce V over R, be it continuous or discrete, hs the following properties: 1. f + g, h = f, h + g, h for ll f, g, h V. cf, g = c f, g for ll c R nd ll f V 3. f, g = g, f for ll f, g V 4. f, f for ll f V, nd f, f = if nd only if f =. This inner product cn be used to define the norm of function, which generlizes the concept of the mgnitude of vector to functions, nd therefore provides mesure of the mgnitude of function. Recll tht the mgnitude of vector v, denoted by v, cn be defined by v = (v v) 1/. Along similr lines, we define the -norm of function f(x) defined on [, b] by ( 1/ f = ( f, f ) 1/ = [f(x)] dx). As we will see, it cn be verified tht this function does in fct stisfy the properties required of norm. The continuous lest-squres problem cn then be described s the problem of finding such tht f n (x) = ( f n f = c j φ j (x) ) 1/ [f n (x) f(x)] dx is minimized. This minimiztion cn be performed over C[, b], the spce of functions tht re continuous on [, b], but it is not necessry for function f(x) to be continuous for f to be defined. Rther, we consider the spce L (, b), the spce of rel-vlued functions such tht f(x) is integrble over (, b). One very importnt property tht hs is tht it stisfies the Cuchy-Schwrz inequlity f, g f g, f, g V. This cn be proven by noting tht for ny sclr c R, c f + c f, g + g = cf + g. 4
The left side is qudrtic polynomil in c. In order for this polynomil to not hve ny negtive vlues, it must either hve complex roots or double rel root. This is the cse if the discrimnt stisfies 4 f, g 4 f g, from which the Cuchy-Schwrz inequlity immeditely follows. By setting c = 1 nd pplying this inequlity, we immeditely obtin the tringle-inequlity property of norms. Suppose tht we cn construct set of functions {φ j (x)} n tht is orthogonl with respect to the inner product of functions on [, b]. Tht is, φ k, φ j = φ k (x)φ j (x) dx = { k = j α k > k = j. Then, the norml equtions simplify to trivil system [ ] [φ k (x)] dx c k = φ k (x)f(x) dx, k =, 1,..., n, or, in terms of norms nd inner products, φ k c k = φ k, f, k =, 1,..., n. It follows tht the coefficients {c j } n of the lest-squres pproximtion f n(x) re simply c k = φ k, f φ k, k =, 1,..., n. If the constnts {α k } n k= bove stisfy α k = 1 for k =, 1,..., n, then we sy tht the orthogonl set of functions {φ j (x)} n is orthonorml. In tht cse, the solution to the continuous lest-squres problem is simply given by c k = φ k, f, k =, 1,..., n. Next, we will lern how sets of orthogonl polynomils cn be computed. 5