Lecture 19: Continuous Least Squares Approximation

Transcription

1 Lecture 19: Continuous Lest Squres Approximtion 33 Continuous lest squres pproximtion We begn 31 with the problem of pproximting some f C[, b] with polynomil p P n t the discrete points x, x 1,, x m for some m n This exmple motivted our study of discrete lest squres problems ( subject with mny other diverse pplictions), but the choice of the m points is somewht rbitrry Suppose we simply wish for the pproximting polynomil to represent f throughout ll of [, b] Wht vlue should m tke? How should one pick the points {x k }? Suppose we uniformly distribute these pproximtion points over [, b]: set h m := (b )/m nd let x k = + kh m The lest squres error formul, when scled by h m, tkes the form of Riemnn sum tht, in the m limit, pproximtes n integrl: lim m h m m (f(x k ) p(x k )) 2 = (f(x) p(x)) 2 dx Tht is, in the limit of infinitely mny uniformly spced pproximtion points, we re ctully minimizing n integrl, rther thn sum In this lecture, we will see how to pose such problems s mtrix problem of dimension (n + 1)-by-(n + 1), insted of discrete lest squres problem with mtrix of dimension -by-(n + 1) 331 Inner products for function spces To fcilitte the development of continuous lest squres pproximtion theory, we introduce forml structure for C[, b] First, recognize tht C[, b] is liner spce: ny liner combintion of continuous functions on [, b] must itself be continuous on [, b] Definition The inner product of the functions f, g C[, b] is given by f, g = f(x)g(x) dx This inner product stisfies the following bsic xioms: αf + g, h = α f, h + g, h for ll f, g, h C[, b] nd ll α R; f, g = g, f for ll f, g C[, b]; f, f for ll f C[, b] Just s the vector 2-norm nturlly follows from the vector inner product ( x 2 = x x), so we hve ( f L 2 := f, f 1/2 b 1/2 = f(x) dx) 2 Here the superscript 2 in L 2 refers to the fct tht the integrnd involves the squre of the function f; the L stnds for Lebesgue, coming from the fct tht this inner product cn be generlized from R b If we wnted to consider complex-vlued functions f nd g, the inner product would be generlized to f, g = f(x)g(x) dx, giving f, g = g, f 18 Jnury M Embree, Rice University

2 C[, b] to the set of ll functions tht re squre-integrble in the sense of Lebesgue integrtion By restricting our ttention to continuous functions, we dodge the mesure-theoretic complexities (The Lebesgue theory gives more robust definition of the integrl thn the conventionl Riemnn pproch; for detils, consult MATH 425) 332 Lest squres minimiztion vi clculus Given some f C[, b], the bsic L 2 pproximtion problem seeks the polynomil p P n tht minimizes the error f p in the L 2 norm In symbols: min p P n f p L 2 We shll denote the polynomil tht ttins this minimum by p We cn solve this minimiztion problem using bsic clculus Consider this exmple for n = 1, where we optimize the error over polynomils of the form p(x) = c + c 1 x Note tht f p L 2 will be minimized by the sme polynomil s f p 2 L 2 Thus for ny given p P 1, the error function is given by E(c, c 1 ) := f(x) (c + c 1 x) 2 L 2 = = = (f(x) c c 1 x) 2 dx ( ) f(x) 2 2f(x)(c + c 1 x) + (c 2 + 2c c 1 x + c 2 1x 2 ) dx f(x) 2 dx 2c f(x) dx 2c 1 xf(x) dx + c 2 (b ) + c c 1 (b 2 2 ) c2 1(b 3 3 ) To find the optiml polynomil, p, we need to optimize E over c nd c 1, ie, we must find the vlues of c nd c 1 for which = = c c 1 First, compute c c 1 = 2 = 2 Setting these prtil derivtives equl to zero yields f(x) dx + 2c (b ) + c 1 (b 2 2 ) xf(x) dx + c (b 2 2 ) c 1(b 3 3 ) 2c (b ) + c 1 (b 2 2 ) = 2 c (b 2 2 ) c 1(b 3 3 ) = 2 f(x) dx xf(x) dx These equtions, liner in the unknowns c nd c 1, cn be written in the mtrix form [ 2(b ) b 2 2 ] [ ] [ ] c 2 b (b3 3 = f(x) dx ) c 1 2 xf(x) dx 18 Jnury M Embree, Rice University

3 When b this system lwys hs unique solution The resulting c nd c 1 re the coefficients for the monomil-bsis expnsion of the lest squres pproximtion p P 1 to f on [, b] Exmple: f(x) = e x We pply this result to the function f(x) = e x for x [, 1] Since 1 e x dx = e 1, 1 xe x dx = [e x (x 1)] 1 x= = 1, we must solve the system [ ] [ ] [ ] 2 1 c 2e 2 2 = 1 3 c 1 2 The desired solution is c = 4e 1, c 1 = 18 6e Below we show plot of this pproximtion (left), nd the error f(x) p (x) 3 25 f(x) p(x) f(x) p * (x) x x We cn see from these pictures tht the pproximtion looks decent to the eye, but the error is not terribly smll (In fct, f p L 2 = 6277 ) We cn decrese tht error by incresing the degree of the pproximting polynomil Just s we used 2-by-2 liner system to find the best liner pproximtion, generl (n + 1)-by-(n + 1) liner system cn be constructed to yield the L 2 -optiml degree-n pproximtion 333 Generl polynomil bses Note tht we performed the bove minimiztion in the monomil bsis: p(x) = c + c 1 x is liner combintion of 1 nd x Our experience with interpoltion suggests tht different choices for the bsis my yield pproximtion lgorithms with superior numericl properties Thus, we develop the form of the pproximting polynomil in n rbitrry bsis Suppose {φ k } n is bsis for P n Then ny p P n cn be written s p(x) = c k φ k (x) 18 Jnury M Embree, Rice University

4 The error expression tkes the form E(c,, c n ) := f(x) p(x) 2 L 2 = ( f(x) 2 c k φ k (x)) dx As before, compute / c j for j =,, n: = f, f 2 c j = 2 f, φ j + Setting / c j = gives the n + 1 equtions f, φ j = c k f, φ k + 2c k φ k, φ j c k φ k, φ j l= c k c l φ k, φ l This is simply system of liner lgebric equtions, which cn be written in the mtrix form φ, φ φ, φ 1 φ, φ n c f, φ φ 1, φ φ 1, φ 1 c 1 = f, φ 1, φ n, φ φ n, φ 1 φ n, φ n c n f, φ n which we shll denote s Hc = b Suppose we pply this method on the intervl [, b] = [, 1] with the monomil bsis, φ k (x) = x k In tht cse, 1 φ k, φ j = x k, x j = x j+k 1 dx = j + k + 1, nd the coefficient mtrix hs n elementry structure In fct, this is form of the notorious Hilbert mtrix It is exceptionlly difficult to obtin ccurte solutions with this mtrix in floting point rithmetic, reflecting the fct tht the monomils re poor bsis for P n on [, 1] Let H denote the n + 1-dimensionl Hilbert mtrix, nd suppose b is constructed so tht the exct solution to the system Hc = b is c = (1, 1,, 1) T Let ĉ denote computed solution to the system in MATLAB Idelly the forwrd error c ĉ 2 will be nerly zero (if the rounding errors incurred while constructing b nd solving the system re smll) Unfortuntely, this is not the cse entirely consistent with our nlysis of the sensitivity of liner systems, studied in Section 142 n κ(h) c ĉ See M-D Choi, Tricks or trets with the Hilbert mtrix, Americn Mth Monthly 9 (1983) Jnury M Embree, Rice University

5 Clerly these errors re not cceptble! The lst few 2-norm condition numbers re in fct smller thn they ought to be, consequence of the fct tht MATLAB is not computing the singulr vlue decomposition of the Hilbert mtrix exctly (MATLAB computes the condition number s the rtio of the mximum nd minimum singulr vlues) The stndrd lgorithm for computing singulr vlues obtins nswers with smll bsolute ccurcy, but not smll reltive ccurcy Thus we expect tht singulr vlues smller thn bout 1 16 H 2 my not even be computed to the correct order of mgnitude In the next lecture, we will see how better-conditioned bses for P n yield mtrices H for which we cn solve Hx = b much more ccurtely 334 Connection to discrete lest squres Why did the continuous lest squres pproximtion problem studied bove directly led to squre (n + 1) (n + 1) liner system, while the discrete lest squres problem introduced in Lecture 16 led to n (m + 1) (n + 1) lest squres problem? In the discrete cse, we seek to minimize c Af 2, where (using the monomil bsis) 1 x x 2 x n c f(x ) 1 x 1 x 2 1 x n 1 c 1 f(x 1 ) A = 1 x 2 x 2 2 x n 2, c = c 2, f = f(x 2 ) 1 x m x 2 m x n m c n f(x m ) We hve seen tht this discrete problem cn be solved vi the norml equtions A Ac = A f Now compute n f(x k) n x kf(x k ) A f = n x2 k f(x k) C n+1 n xn k f(x k) Notice tht if m + 1 pproximtion points re uniformly spced over [, b], x k h m = (b )/m, we hve f(x) dx f, 1 xf(x) dx f, x lim h ma f = m x2 f(x) dx = f, x 2, xn f(x) dx f, x n = + kh m for 18 Jnury M Embree, Rice University

6 which is precisely the right hnd side vector b C n+1 obtined for the continuous lest squres problem Similrly, the (j + 1, k + 1) entry of the mtrix A A C (n+1) (n+1) for the discrete problem cn be formed s m m (A A) j+1,k+1 = x j l xk l = x j+k l, l= nd thus for uniform grids, we hve in the limit tht Thus in ggregte we hve lim h m(a A) j+1,k+1 = m lim h ma A = H, m l= x j+k dx = x j, x k where H is the mtrix tht rose in the continuous lest squres problem We rrive t the following beutiful conclusion: The norml equtions A Ac = A f formed for polynomil pproximtion by discrete lest squres converges to exctly the sme (n + 1) (n + 1) system Hc = b s we independently derived for polynomil pproximtion by continuous lest squres In the ltter cse, clculus led us directly to the norml eqution form of the solution 18 Jnury M Embree, Rice University