Least Squares Curve Fitting A.F. Emery 1

Transcription

1 Least Squares Curve Fitting AF Emery 1 Introduction Suppose that a system has a constant response, α, which we attempt to determine by measuring the response n times Let these responses be x i and let them be chosen from a population with a mean of α and a standard deviation of σ Then let us estimate α by requiring that S = (x i α) 2 = minimum (1) Since α is unknown, we require that the variation of S with respect to α be zero, or which leads to the result δs = 2 (x i α)δα = 0 (2) α = x = and we see that the arithmetic mean is the Least Squares estimator of the true system response α Weights Now suppose that instead of using Eq 1 as it stands, we group the readings, x i into m sets, each of n j readings That is m n j m xi nj x j α = = (4) n n where x j represents the average of each set of n j readings Suppose that we consider the values of n j as weights which act upon the readings x j and we revise our approach to consider a weighted sum in Equation 1 That is we write S = xi n (3) m w j ( x j α) 2 = minimum (5) Following the same derivation as before we would obtain the result for α of α = x = m wj x j wj (6) By comparing Equations 4 and 6, we see that the appropriate definition of the weights is w j = n j The fundamental interpretation of a weight is that it represents a number of repeated readings

2 2 Least Squares Curve Fitting AF Emery To clarify the role of the weights, let us define w j = k/σ 2 ( x j ), that is the weight is inversely proportional to the variance of the set of readings which make up the j th average, x j What is the appropriate value of the constant k? We write w j = n j = k σ 2 ( x j ) = kn j σ 2 (7) Where we have used the fact that the variance of a mean is related to the variance of the population by σ 2 ( x j ) = σ 2 /n j From Eq 7, we find that k = σ 2 Thus we finally obtain that the appropriate weight for any quantity f is given by w f = σ2 σ 2 f (8) Sometimes the weight functions in the example are expressed as w j = n j /n, ie, normalized However this obscures the fundamental interpretation of the weight and also destroys some of the symmetry in the development of weights for generalized quantities, f Clearly, w f is a measure of the accuracy by which a set of readings of f represents the true function For example, if x i is known exactly, w xi is infinite, ie, σ x = 0 Fitting Data Consider a set of n values f i which we wish to approximate by a set of functions φ j in the form f(x) = a j φ j (x) (9) j where φ j are user define functions These functions are chosen to fit the data f i in the best way possible The coefficients a j are determined by requiring that the errror, E, is minimized [f k a j φ j (x k )] 2 (10) If the φ(x) do not adequately represent the data, the approximation will be poor If φ j (x) = x j, then when M = n, the method reduces to the polynomial approximation If φ j (x) = cos(jx) and M = n, the result is the cosine Fourier series We minimize E by differentiating it with respect to the unknown coefficients a j and setting the derivatives to zero E/ a l = 2 [f k a j φ j (x k )]φ l (x k ) = 0 (11a)

3 Least Squares Curve Fitting AF Emery 3 Ignoring the 2, and dropping the summation over k sign for convenience, this equation reduces to a j φ j (x k )φ l (x k ) = fφ l (x k ) Expressing this in matrix form we have, φ 1 φ 1 φ 2 φ 1 φ 3 φ 1 φ M φ 1 φ 1 φ 2 φ 2 φ 2 φ 3 φ 2 φ M φ 2 φ 1 φ M φ 2 φ M φ 3 φ M φ M φ M a 1 a 2 a M = f 1 φ 1 f 2 φ 2 f M φ M (11b) (12) Although the set of equations appears to be very simple to solve (particularly since it is symmetric), if the data points are equally spaced and the φ j = x j, the determinant is extremely small, ie, it is very ill-conditioned, for values of M as small as 7 CONSTRAINTS It is common to require that the fitting function pass through specific points, or that it satisfy some constraints expressed in terms of integrals or derivatives The usual way to force the least squares fit to pass through a point, is to include a weighting function of the form w k [f k a j φ j (x k )] 2 (13) where the weights w k are adjusted to have a large value where you wish the error to be the smallest This method, although widely used, and reasonably appropriate when emphasizing the importance of some points with respect to other points, is totally incorrect when forcing the curve pass through specific points Let us consider the augmented error form [f k a j φ j (x k )] 2 + 2λG(x p ) (14) where G(x p ) represents a constraint upon the fitting function For example, if we required that the fitting function pass through the point x p, then G(x p ) would be given by G(x p ) = f p a j φ j (x p ) = 0 (15) Note that adding G(x p ) does not affect the value of E since G = 0 Now differentiating E with respect to a l and to λ will give

4 4 Least Squares Curve Fitting AF Emery E/ x l = 2 E/ λ = f p The matrix form then becomes [f k a j φ j (x k )]φ l (x k ) 2λφ l (x p ) = 0 a j φ j (x p )] = 0 φ 1 φ 1 φ 2 φ 1 φ 3 φ 1 φ M φ 1 φ 1 (x p ) φ 1 φ 2 φ 2 φ 2 φ 3 φ 2 φ M φ 2 φ 2 (x p ) φ 1 φ M φ 2 φ M φ 3 φ M φ M φ M φ M (x p ) φ 1 (x p ) φ 2 (x p ) φ 3 (x p ) φ M (x p ) 0 a 1 a 2 a M λ = f 1 φ 1 f 2 φ 2 f M φ M f p (16a) (16b) where terms of the form φ i φ j and f k φ j represent summations over all of the data point, while terms of the form f p or φ j (x p ) are to be evaluated at the point p If there are several points the curve must pass through, then there must be a λ for each point If the constraint is of the form [ a j φ j (x)] = V then the constraint would be given by G = [ a j φ j (x)] V = 0 (17a) (17b) Note that the matrix involving λ, while symmetric, has a zero diagonal element This suggests that one must be careful in solving using Gaussian elminination to be sure that the constraints are not the first equations in the set unless pivoting is implemented in your solver Statistics of the Fit One of the most common questions in curve fitting data is whether the fit is good and if the parameters are different from 0 For example, if we fit using a polynomial, P n (x), what is the best order to use It might be thought that S where S = m w j (y j P j ) 2 = minimum (18a) is a good measure since the closer the fit is to the points, the smaller S is However, if there are n points and we fit with a curve which has n free parameters, we are not fitting

5 Least Squares Curve Fitting AF Emery 5 but interpolating and the curve will pass exactly through each point, yielding S = 0 A better estimate of the goodness of the fit is the variance of the fit, σ 2 As the fit improves, σ 2 will diminish Since we do not know this variance, we must estimate it by σ 2 (est) = S/(n p) (18b) where n is the number of points and p is the number of parameters plus additional constraints, that is n-p = number of degrees of freedom The approach is to fit with an ever increasing number of parameters (ie, increasing the complexity of the fitting function) and to check the variance of the fit As the complexity of the fit is increased, σ(est) should decrease Whenever it begins to increase, we have exceeded the best fit Orthogonal Functions If the functions φ j (x) in Eq 9 are orthogonal, then the off diagonal terms in the matrix, Eq (12) would be zero and we wouldn t have to worry about the ill conditioning of the matrix A suitable set of orthogonal polynomials is given by with where p j+1 (x) = (x α j+1 )p j (x) β j p j 1 (x), j = 0,1,2,,n p 1 (x) = 0,p 0 (x) = 1 (19a) (19b) α k+1 = w i x i p 2 k (x i) w i p 2 k (x i) β k = w i x i p 2 k (x i) w i p 2 k 1 (x i) (19c) (19d) If w i = 1 and the data are equally spaced, these polynomials are known as the Gram polynomials