Discrete Lest-squres Approximtions Given set of dt points (x, y ), (x, y ),, (x m, y m ), norml nd useful prctice in mny pplictions in sttistics, engineering nd other pplied sciences is to construct curve tht is considered to be the fit best for the dt, in some sense Severl types of fits cn be considered But the one tht is used most in pplictions is the lest-squres fit Mthemticlly, the problem is the following: Discrete Lest-Squres Approximtion Problem Given set of dt points (x k, y i ), i =,, m, find n lgebric polynomil P n (x) = + x + + n x n (n < m) such tht the error in the lest-squres m sense in minimized; tht is, E = (y i x i n x n i ) is minimum For E to be minimum, we must hve i= E j =, j =,, n Now, E = E = m (y i x i n x n i ) i= m x i (y i x i n x n i ) i= E n = m x n i (y i x i n x n i ) i= Setting these equtions to be zero we hve + m i= m i= x i + x i + m i= m i= x i + + n x i + + n m i= m i= x n i = x n+ i = m i= y i m x i y i i= m i= x n i + m i= x n+ i + + n m i= x n i = m x n i y i i=
Set now m x k i = s k, k =,,, n, nd denoting the right hnd side entries s b,, b n, i= the bove eqution cn be written s: s + s + + s n n = b (Note tht m i= x i = s = m) s + s + + s n+ n = b s n + s n+ + s n n = b n This is system of (n + ) equtions in (n + ) unknowns,,, n These equtions re clled Norml Equtions This system now cn be solved to obtin these (n + ) unknowns, provided solution to the system exists We will not show tht this system hs unique solution if x i s re distinct The system cn be written in the following mtrix form: or where Define s = s s s n s s s n+ s n s n+ s n s s s n s s s n+ s n s n+ s n Then the bove system hs the form: s = b n, = = n x x x n x x x n V = x 3 x 3 x n 3 x m x m x n m V T V = b b b b n, b = The mtrix V is known s the Vndermonde mtrix, nd it cn be shown [Exercise] tht it hs full rnk if x i s re distinct In this cse the mtrix S = V T V is symmetric nd positive definite [Exercise] nd is therefore nonsingulr Thus, if x i s re distinct, the eqution S = b hs unique solution b b b n
Theorem (Existence nd uniqueness of Discrete Lest-Squres Solutions) Let (x, y ), (x, y ),, (x n, y n ) be n distinct points Then the discrete lest-squre pproximtion problem hs unique solution Lest-Squres Approximtion of Function We hve described lest-squres pproximtion to fit set of discrete dt Here we describe continuous lest-squre pproximtions of function f(x) by using polynomils The problem cn be stted s follows: Lest-Squre Approximtions of Function Using Stndrd Polynomils Given function f(x), continuous on [, b], find polynomil P n (x) of degree t most n: P n (x) = + x + x + + n x n such tht the integrl of the squre of the error is minimized Tht is, is minimized E = [f(x) P n (x)] dx The polynomil P n (x) is clled the Lest-Squres Polynomil,,, n, we denote this by E(,,, n ) For minimiztion, we must hve E i =, i =,,, n Since E is function of As before, these conditions will give rise to norml system of (n + ) equtions in (n + ) unknowns:,,, n Solution of these equtions will yield the unknowns Setting up the Norml Equtions Since We hve E = [f(x) ( + x + x + + n x n )] dx 3
E = E = E n = so, E = [f(x) x x n x n ]dx x[f(x) x x n x n ]dx x n [f(x) x x n x n ]dx Similrly, E = i,, 3,, n dx + x dx + x dx + n x n dx = So, (n + ) norml equtions in this cse re: x i dx+ x i+ dx+ x i+ dx + n x i+n dx = f(x)dx x i dx, i = i = : i = : dx + x dx + x dx + + n x n dx = x dx + x dx + 3 x 3 dx + + n x n dx = f(x) xf(x)dx i = n : Denote x n dx + x n+ dx + x n+ dx + + n x n dx = x n f(x)dx x i dx = s i, i =,,, 3,, n, nd b i = Then the bove (n + ) equtions cn be written s x i f(x)dx, i =,,, n s + s + s + + n s n = b s + s + s 3 + + n+ s n+ = b s n + s n+ + s n+ + + n s n = b n 4
or in mtrix nottion s s s s n s s s 3 s n+ s n s n+ s n n = b b b n Denote S = (s ii ), = n, b = b b b n Then we hve the liner system: S = b The solution of these equtions will yield the coefficients,,, n of the lest-squres polynomil P n (x) A Specil Cse: Let the intervl be [, ] Then s i = x i dx = i +, Thus, in this cse the mtrix of the norml equtions i =,,, n n S = 3 n + n n+ n + n + n which is Hilbert Mtrix It is well-known to be ill-conditioned Algorithm: Lest-Squres Approximtion using Polynomils Inputs: (i) f(x) - A continuous function on [, b] (ii) n - The degree of the desired lest-squre polynomil 5
Output: The coefficients,,, n of the desired lest-squres polynomil: P n (x) = + x + + n x n Step Compute s, s,, s n For i =,, n do End s i = x i dx Step Compute b, b,, b n : For i =,,, n do b i = End x i f(x)dx Step 3 Form the mtrix S nd the vector b s s s n s s s n+ S = s n s n+ s n b = b b b n Step 4 Solve the (n + ) (n + ) system of equtions: S = b, where = n Exmple Find Liner nd Qudrtic lest-squres pproximtions to f(x) = e x on [, ] 6
Liner Approximtion: n = ; P (x) = + x s = dx = [ ] x s = xdx = = ( ) = [ ] x s = x 3 dx = = ( ) 3 3 = 3 3 ( ) ( ) s s Thus, S = = s s 3 b = b = e x dx = e e 354 e x xdx = e 7358 The norml system of equtions is: ( 3 ) ( ) = ( b b ) This gives = 75, = 37 The liner lest-squres polynomil P (x) = 75 + 37x Check Accurcy: P (5) = 77 e 5 = 6487 Reltive Error 6487 77 77 = 453 7
Qudrtic Fitting: n = P (x) = + x + x s =, s =, s = 3 s 3 = [ ] x 4 x3 dx = = 4 s 4 = [ ] x 5 x4 dx = = 5 5 b = b = b = The system of norml equtions is: 3 3 3 5 e x dx = e e 354 xe x dx = e 7358 x e x dx = e 5 e 8789 = The solution is: = 9963, = 37, = 5368 354 7358 8789 The qudrtic lest-squres polynomil P (x) = 9963 + 37x + 5368x 8
Check the ccurcy: P (5) = 6889 e 5 = 6487 Reltive error P (5) e 5 e 5 = 684 6487 6487 = 4 Exmple Find liner nd Qudrtic lest-squres polynomil pproximtion to f(x) = x + 5x + 6 in [, ] Liner Fit: b = b = The norml equtions re: 3 P (x) = + x s = s = s = dx = xdx = x dx = 3 (x + 5x + 6)dx = 3 + 5 + 6 = 53 6 x(x + 5x + 6)dx = ( = 4 + 5 3 + 6 = 59 ) = 53 6 59 (x 3 + 5x + 6x)dx ] = 58333 = 6 The liner lest squres polynomil P (x) = 58333 + 6x Check Accurcy: f(5) = 875; P (5) = 8833 9
Reltive error: 8833 875 875 = 95 Qudrtic Lest-Squre Approximtion: P (x) = + x + x b = 53 6, b = 59 b = x (x + 5x + 6)dx = S = 3 3 4 3 4 5 (x 4 + 5x 3 + 6x )dx = 5 + 5 4 + 6 3 = 69 The solution of the liner system is: = 6, = 5, = P (x) = 6 + 5x + x (Exct) Use of Orthogonl Polynomils in Lest-squres Approximtions The lest-squres pproximtion using polynomils, s described bove, is not numericlly effective; since the system mtrix S of norml equtions is very often ill-conditioned For exmple, when the intervl is [,], we hve seen tht S is Hilbert mtrix, which is notoriously ill-conditioned for even modest vlues of n When n = 5, the condition number of this mtrix = cond(s) = O( 5 ) Such computtions cn, however, be mde computtionlly effective by using specil type of polynomils, clled orthogonl polynomils Definition The set of functions φ, φ,, φ n is clled set of orthogonl functions, with respect to weight function w(x), if { if i j w(x)φ j (x)φ i (x)dx = if i = j where C j is rel positive number Furthermore, if C j =, j =,,, n, then the orthogonl set is clled n orthonorml set Using this interesting property, lest-squres computtions cn be more numericlly effective, s shown below Without ny loss of generlity, let s ssume tht w(x) = Ide: The ide is to find n pproximtion of f(x) on [, b] by mens of polynomil of the form P n (x) = φ (x) + φ (x) + + n φ n (x), where {φ n } n k= is set of orthogonl polynomils Tht is, the bsis for generting P n(x) in this cse is set of orthogonl polynomils C j
Lest-squres Approximtion of Function Using Orthogonl Polynomils Given f(x), continuous on [, b], find,,, n using polynomil of the form: P n (x) = φ (x) + φ (x) + + n φ n (x), where {φ k (x)} n k= is given set of orthogonl polynomils on [, b], such tht the error function: is minimized As before, we set E(,,, n ) = [f(x) ( φ (x) + n φ n (x))] dx Now E i =, i =,,, n E = Setting this equl to zero, we get φ (x)[f(x) φ (x) φ (x) n φ n (x)]dx φ (x)f(x)dx = Since, {φ k (x)} n k= is n orthogonl set, we hve, nd ( φ (x) + + n φ n (x))φ (x)dx φ (x) dx = C, φ (x)φ i (x) dx =, i Applying the bove orthogonl property, we see from bove tht Tht is, φ (x)f(x)dx = C
= φ (x)f(x)dx C Similrly, E = φ (x)[f(x) φ (x) φ (x) n φ n (x)]dx The orthogonl property of {φ j (x)} n j= implies tht so, setting E =, we get φ (x) = C nd φ (x)φ i (x) =, i, = φ (x)f(x)dx C In generl, k = φ k (x)f(x)dx, k =,,, n, C k where C k = φ k(x)dx Expresions for k with Weight Function w(x) If the weight function w(x) is included, we obtin k = w(x)f(x)φ k (x)dx, k =,, n C k
Algorithm: Lest-Squres Approximtion Using Orthogonl Polynomils Inputs: f(x) - A continuous function on [, b] w(x) - A weight function (n integrble function on [, b]) {φ k (x)} n k= - A set of n orthogonl functions on [, b] Output: The coefficients,,, n such tht is minimized w(x)[f(x) φ (x) φ (x) n φ n (x)] dx Step Compute C k, k =,,, n s follows: For k =,,,, n do End C k = w(x)φ k(x)dx Step Compute k, k =,, n s follows: For k =,,,, n do k = w(x)f(x)φ k (x)dx C k End Lest-Squres Approximtion Using Legendre s Polynomils Recll tht the Legendre Polynomils {φ k (x)} re given by φ (x) = φ (x) = x φ (x) = x 3 φ 3 (x) = x 3 3 5 x etc re orthogonl polynomils on [, ], with respect to the weight function w(x) = If these polynomils re used for lest-squres pproximtion, then it is esy to see tht 3
C = C = C = φ (x)dx = φ (x)dx = φ (x)dx = dx = x dx = ( 3 x ) dx = 8 3 45 nd so on Exmple: Find liner nd qudrtic lest-squres pproximtion to f(x) = e x using Legendre polynomils Liner Approximtion: P (x) = φ (x) + φ (x) φ (x) =, φ (x) = x C = φ (x)dx = = φ (x)e x dx C The liner lest-squres polynomil dx = [x] = So, = e x dx = [ex ] = [ x C = φ (x)dx = x 3 dx = 3 = 3 xe x dx = 3 [ ] = 3 e e ( e ) e ] P (x) = φ (x) + φ (x) = [ e ] + 3 e e x = 3 + 3 = 3 Accurcy Check: P (5) = e 5 = 6487 [ e ] + 3 e e 5 = 77 Reltive error: 77 6487 6487 = 475 4
Qudrtic Approximtion: P (x) = φ (x) + φ (x) + φ (x) = ( e ), = 3 e e C = φ (x)dx = = ( x 5 5 3 x3 3 + x = e x φ (x)dx C = 45 8 (x 3 ) dx ) ( e x x ) dx 3 = 8 45 = e 7 e Qudrtic lest-squres polynomil: P (x) = ( e ) + 3e ( e x + e 7 ) ( x ) e 3 Accurcy check: Reltive error P n (5) = 5868 e 5 = 6487 5868 6487 6487 = 375 Compre this reltive error with tht obtined erlier with n non-orthogonl polynomil of degree 5
Chebyshev polynomils: Another wonderful fmily of orthogonl polynomils Definition: The set of polynomils defined by T n (x) = cos[n rccos x], n on [, ] re clled the Chebyshev polynomils To see tht T n (x) is polynomil of degree n in our fmilir form, we derive recursive reltion by noting tht T (x) = (A polynomil of degree zero) T (x) = x (A polynomil of degree ) A Recursive Reltion for Generting Chebyshev Polynomils: Substitute θ = rc cos x Then, T n (x) = cos(nθ), θ π T n+ (x) = cos(n + )θ = cos nθ cos θ sin nθ sin θ T n (x) = cos(n )θ = cos nθ cos θ + sin nθ sin θ Adding the lst two equtions, we obtin T n+ (x) + T n (x) = cos nθ cos θ The right hnd side still does not look like polynomil in x But note tht cos θ = x So, or T n+ (x) = cos nθ cos θ T n (x) = x cos(n cos rc x) T n (x) = xt n (x) T n (x) T n+ (x) = xt n (x) T n (x), n Using this recursive reltion, the Chebyshev polynomils of the succesive degrees cn be generted n = : T (x) = xt (x) T (x) = x n = : T 3 (x) = xt (x) T (x) = x(x ) x = 4x 3 3x nd so on 6
The orthogonl property of the Chebyshev polynomils We now show tht Chebyshev polynomils re orthogonl with respect to the weight function w(x) =, in the intervl [, ] x To demonstrte the orthogonl property of these polynomils, consider = = T m (x)t n (x)dx, m n x π = = = cos(rccos x) cos(n rccos x) x dx cos mθ cos nθdθ ( By chnging the vrible from x to θ with substitution of rccosx = θ) π [ cos(m + n)θdθ + sin(m + n)θ (m + n) ] π π + cos(m n)θdθ [ sin(m n)θ (m n) ] π Similrly, it cn be shown [Exercise] tht Summrizing: T n(x)dx x = π for n Orthogonl Property of the Chebyshev Polynomils T m (x)t n (x) dx = x if m n π if m = n The Lest-Squre Approximtion using Chebyshev Polynomils 7
As before, the Chebyshev polynomils cn be used to find lest-squres pproximtions to function f(x) s stted below The lest-squres pproximting polynomil P n (x) of f(x) using Chebyshev polynomils is given by: P n (x) = C T (x) + C T (x) + + C n T n where nd C i = π f(x)t i (x)dx, i =,, n x C = π f(x)dx x Find liner lest-squres pproximtion of f(x) = e x using Chebyshev poly- Exmple: nomils Here P (x) = φ (x) + φ i (x) = T (x) + T (x) = + x, where Thus, P (x) = 66 + 33x Check the ccurcy: = π = π e x dx x 66 xe x dx 33 x P (5) = 975; e 5 = 6487 Reltive error 6487 975 975 = 4 Monic Chebyshev Polynomils Note tht T k (x) is Chebyshev polynomil of degree k with the leding coefficient k, k Thus we cn generte set of monic Chebyshev polynomils from the polynomils T k (x) s follows: 8
The Monic Chebyshev Polynomils, T k (x), re then given by T (x) =, T k (x) = k T k(x), k The k zeros of T k (x) re esily clculted [Exercise]: ( ) j x j = cos k π, j =,,, k The mximum or minimum vlues of T k (x) occur t x j = cos T k ( x j ) = ()j, j =,,, k k ( ) jπ, nd k Polynomil Approximtions with Chebyshev the polynomils: As seen bove the Chebyshev polynomils cn, of course, be used to find lest-squres polynomil pproximtions However, these polynomils hve severl other wonderful polynomil pproximtion properties Some of them re stted below The mximum bsolute vlue of ny monic polynomil of degree n over [, ] is lwys greter thn or equl to tht of T n (x) over the sme intervl; which is, by the lst property, n Minimx Property of the Chebyshev Polynomils If P n (x) is ny monic polynomil of degree n, then = mx T n n (x) mx P n (x) x [,] x [,] Moreover, this hppens when P n (x) = T n (x) Proof: By contrdiction [Exercise] 9
Choosing the interpolting nodes with the Chebyshev Zeros Recll tht error in polynomil interpoltion by polynomil P n (x) of degree t most n is given by where Ψ(x) = (x x )(x x ) (x x n ) E = f(x) P (x) = f n+ (ξ) (n + )! Ψ(x), The question is: How to choose these (n + ) nodes x, x,, x n so tht Ψ(x) is minimized in [-, ]? The nswer cn be given from the lst-mentioned property of the monic Chebyshev polynomils Note tht Ψ(x) is monic polynomil of degree (n + ) So, by the minimx property mx T n+ (x) mx Ψ(x) x [,] x [,] Tht is, the mximum vlue of ψ(x) is smllest when x, x,, x n re chosen s the (n + ) zeros of T n+ (x) nd this mximum vlue is n Choosing the Nodes for Minimizing Polynomil Interpoltion error To minimize the polynomil interpoltion error, choose the nodes x, x,, x n s the (n+) zeros of the (n + )th degree monic Chebyshev polynomil Note (Working with n rbitrry intervl) If the intervl is [, b], different from [, ], then, the zeros of T n+ (x) need to be shifted by using the trnsformtion: x = [(b )x + ( + b)] Exmple Let the interpolting polynomil be of degree t most nd the intervl be [5, ] The three zeros of T 3 (x) in [, ] re given by x = cos π 6, x = cos π, nd x 3 = cos 5 6 π These zeros re to be shifted using trnsformtion: x new = [( 5) x i + ( + 5)]
Use of Chebyshev Polynomils to Economize Power Series Power Series Economiztion Let P n (x) = + x+ + n x n be polynomil of degree n obtined by truncting power series expnsion of continuous function on [, b] The problem is to find polynomil P r (x) of degree r (< n) such tht P n (x) P r (x) < ɛ, where ɛ is tolernce supplied by users The problem is esily solved by using the Minimx Property of the Chebyshev polynomils First note tht n P n (x) P n (x) is monic polynomil So, by the minimx property, we hve Thus, if we choose mx P n (x) P n (x) n mx T n (x) = n n P n (x) = P n (x) n Tn (x), then the minimum vlue of mx P n (x) P n (x) = n n n If this quntity,, plus error due to the trunction of the power series is within the n permissible tolernce ɛ, we cn then repet the process by constructing P n (x) from P n (x) s bove The process cn be continued until the ccumulted error exceeds the tolernce ɛ So, the process cn be summrized s follows: Power Series Economiztion Process by Chebyshev Polynomils Obtin P n (x) = + x n + + n x n by truncting the power series expnsion of f(x) Find the error of trunction E P T Compute P n (x): P n (x) = P n (x) n Tn (x) Check if the totl error ( E P T + n n ) is less thn ɛ If so, continue the process by decresing the degree of the polynomils successively until the ccumulted error becomes greter thn ɛ