97 lecture 16: Itroductio to Lest Squres Approximtio.4 Lest squres pproximtio The miimx criterio is ituitive objective for pproximtig fuctio. However, i my cses it is more ppelig (for both computtio d for the give pplictio) to fid pproximtio to f tht miimizes the itegrl of the squre of the error. Give f C[, b], fid P P such tht 1/ 1/. ( f (x) P (x)) dx mi ( f (x) p(x)) dx pp This is exmple of lest squres problem..4.1 Ier products for fuctio spces To fcilitte the developmet of lest squres pproximtio theory, we itroduce forml structure for C[, b]. First, recogize tht C[, b] is lier spce: y lier combitio of cotiuous fuctios o [, b] must itself be cotiuous o [, b]. Defiitio.. The ier product of the fuctios f, g C[, b] is h f, gi f (x)g(x) dx. The ier product stisfies the followig bsic xioms: h f + g, hi h f, hi + hg, hi for ll f, g, h C[, b] d ll IR; h f, gi hg, f i for ll f, g C[, b]; h f, f i for ll f C[, b]. With this ier product we ssocite the orm For simplicity we re ssumig tht f d g re rel-vlued. To hdle complex-vlued fuctios, oe geerlizes the ier product to h f, gi f (x)g(x) dx, which the gives h f, gi hg, f i. 1/. k f k : h f, f i 1/ f (x) dx This is ofte clled the L orm, where the superscript i L refers to the fct tht the itegrd ivolves the squre of the fuctio f ; the L stds for Lebesgue, comig from the fct tht this ier product c be geerlized from C[, b] to the set of ll fuctios tht re squre-itegrble, i the sese of Lebesgue itegrtio. By restrictig our ttetio to cotiuous fuctios, we dodge the mesuretheoretic complexities. The Lebesgue theory gives more robust defiitio of the itegrl th the covetiol Riem pproch. With such otios oe c exted lest squres pproximtio beyod C[, b], to more exotic fuctio spces.
98.4. Lest squres miimiztio vi clculus We re ow redy to solve the lest squres problem. We shll cll the optiml polyomil P P, i.e., k f P k mi pp k f pk. We c solve this miimiztio problem usig bsic clculus. Cosider this exmple for 1, where we optimize the error over polyomils of the form p(x) c + c 1 x. The polyomil tht miimizes k f pk will lso miimize its squre, k f pk. For y give p P 1, defie the error fuctio E(c, c 1 ) : k f (x) (c + c 1 x)k L ( f (x) c c 1 x) dx f (x) f (x)(c + c 1 x)+(c + c c 1 x + c 1 x ) dx f (x) dx c f (x) dx To fid the optiml polyomil, P, optimize E over c d c 1, i.e., fid the vlues of c d c 1 for which c 1 xf(x) dx + c (b )+c c 1 (b )+ 1 c 1 (b ). First, compute. c c 1 Z b f (x) dx + c (b )+c c 1 (b ) Z b xf(x) dx + c (b )+c c 1 (b ). 1 Settig these prtil derivtives equl to zero yields c (b )+c 1 (b ) f (x) dx c (b )+c 1 (b ) xf(x) dx. These equtios, lier i the ukows c d c 1, c be writte i the mtrix form " (b ) b # b (b ) #" c c 1 4 R b R b f (x) dx xf(x) dx Whe b 6 this system lwys hs uique solutio. The resultig c d c 1 re the coefficiets for the moomil-bsis expsio of the lest squres pproximtio P P 1 to f o [, b]. 5.
99 Exmple.4 ( f (x) e x ). Apply this result to f (x) e x for x [, 1]. Sice Z 1 e x dx e 1, Z 1 xe x dx e x (x 1) 1 x 1, we must solve the system " 1 #" c # " e # 1 c 1. The desired solutio is c 4e 1, c 1 18 6e. Figure.7 compres f to this lest squres pproximtio P d the miimx pproximtio p computed erlier..5 1.5 1.5.1...4.5.6.7.8.9 1 x Figure.7: Top: Approximtio of f (x) e x (blue) over x [, 1] vi lest squres (P, show i red) d miimx (p, show s gry lie). Bottom: Error curves for lest squres, f P (red), d miimx, f p (gry) pproximtio. While the curves hve similr shpe, ote tht the red curve does ot tti its mximum devitio from f t + poits, while the gry oe does.. error, f (x) p(x).15.1.5 -.5 -.1 lest squres miimx -.15.1...4.5.6.7.8.9 1 x We c see from the plots i Figure.7 tht the pproximtio looks decet to the eye, but the error is ot terribly smll. We c I fct, k f P k.677.... This is decrese tht error by icresig the degree of the pproximtig polyomil. Just s we used -by- lier system to fid the best lier pproximtio, geerl ( + 1)-by-( + 1) lier system c be costructed to yield the degree- lest squres pproximtio. ideed smller th the -orm error of the miimx pproximtio p : k f p k.78....
1.4. Geerl polyomil bses Note tht we performed the bove miimiztio i the moomil bsis: p(x) c + c 1 x is lier combitio of 1 d x. Our experiece with iterpoltio suggests tht differet choices for the bsis my yield pproximtio lgorithms with superior umericl properties. Thus, we develop the form of the pproximtig polyomil i rbitrry bsis. Suppose {f k } is bsis for P. Ay p P c be writte s p(x) The error expressio tkes the form c k f k (x). E(c,...,c ) : k f (x) p(x)k L f (x) c k f k (x) dx h f, f i c k h f, f k i + ` To miimize E, we seek criticl vlues of c [c,...,c +1 ] T IR +1, i.e., we wt coefficiets where the grdiet of E with respect to c is zero: r c E. To compute this grdiet, evlute / for j,..., : h f, f i c k h f, f k i + h f, f j i + c j hf j, f j i + k6j ` c k c j hf k, f j i + ` `6j c j c`hf j, f`i + I this lst lie, we hve broke the double sum o the previous lie ito four prts: oe tht cotis c j, two tht coti c j (c k c j for k 6 j; c j c` for ` 6 j), d oe (the double sum) tht does ot ivolve c j t ll. This decompositio mkes it esier to compute the derivtive: c j c hf j, f j i + j k6j c k c j hf k, f j i + c j hf j, f j i + c j hf j, f j i + k6j k6j ` `6j c j c`hf j, f`i + c k hf k, f j i + c k hf k, f j i. ` `6j ` k6j `6j c`hf j, f`i +. ` k6j `6j
11 These terms cotribute to / to give (.1) h f, f i + j c k hf k, f j i. To miimize E, set / for j,...,, which gives the + 1 equtios (.1) c k hf k, f j i h f, f j i, j,...,, i the + 1 ukows c,...,c. Sice these equtios re lier i the ukows, write them i mtrix form: hf, f i hf, f 1 i hf, f i c h f, f i hf 1, f i hf 1, f 1 i. c 1. h f, f 6. 1 i, 4.... 7 6 7 6 5 4 5 4. 7 5 hf, f i hf, f 1 i hf, f i h f, f i which we deote Gc b. The mtrix G is clled the Grm mtrix. Usig this mtrix-vector ottio, we c ccumulte the prtil derivtives formuls (.1) for E ito the grdiet r c E Gc b. Sice c is criticl poit if d oly if r c E(c), we must sk: How my criticl poits re there? Equivletly, how my c solve Gc b? If c is criticl poit, is it (locl or eve globl) miimum? We will swer the first questio by showig tht G is ivertible, d hece E hs uique criticl poit. To swer the secod questio, we must ispect the Hessi c r c E r c (r c E)G. The criticl poit c is locl miimum if d oly if the Hessi is symmetric positive defiite. The symmetry of the ier product implies hf j, f k i hf k, f j i, d hece G is symmetric. (I this cse, symmetry lso follows from the equivlece of mixed prtil deritivtes.) The followig theorem cofirms tht G is ideed positive defiite. A mtrix G is positive defiite provided z Gz > for ll z 6.