Least Squares Regression Chemical Engineering 2450 - Numerical Methods Given N data points x i, y i, i 1 N, and a function that we wish to fit to these data points, fx, we define S as the sum of the squared errors as S [y i fx i ] 2 1 In words, we take the observed value at each data point eg, y 1 and subtract the function value at that point eg, fx 1 and we square that number We do this for each data point and add them all up Our goal is to minimize S by choosing the best values for the parameters in the function fx 1 Polynomials Let s consider the situation where fx in equation 1 is a polynomial, ie, n p fx a k x k, 2 where n p is the order of the polynomial Substituting 2 into 1, we obtain k1 S n p 2 y i a k x k 3 k0 Equation 3 has n p + 1 coefficients, a k, k 0 n p, which we want to change to minimize S To do this, we take partial derivatives of S with respect to each a k Given a polynomial form for fx, the general formula for this is a k n p 2x k i y i a j x j i k 0 n p 4 j0 To find the minimum of S with respect to the parameters a k, we solve a k 0 for all a k Note that the a k coefficients enter linearly in equation 4 That means that 4 represents a linear system of equations for the n p + 1 unknowns a k This regression technique is thus called linear least squares, and applies to any polynomial function We will consider other forms for fx in 2 1
11 Matrix Form 1 POLYNOMIALS 11 Matrix Form Equations 4 may also be written as A T Aφ A T b, 5 where A, φ, and b are given as 1 x 1 x 2 1 x np 1 1 x A 2 x 2 2 x np 2, φ 1 x N x 2 N xnp N a np, b y 1 y N The math to show this is a bit tedious, so we won t show it here However, in 12 we will go through this for linear polynomials 12 Linear Polynomials For a linear polynomial, 2 becomes fx + x and 3 becomes S y i x 2 6 We now need to obtain expressions for a 9 and From 4, we set k 0 and n p 1 to find Setting k 1 in 4 gives 2 y i x i 7 2x i y i x i 8 We would solve equations 7 and 8 for what values of and make the equations equal to zero In other words, we solve y i x i 0 9 x i y i x i 0 10 for and At that point we have found the values of and that minimize S 2
12 Linear Polynomials 1 POLYNOMIALS 121 Matrix Form We can write 9 and 10 in matrix form by rearranging them Recall that and are the unknowns Therefore, we rewrite the equations to define coefficients multiplying and as y i a N 0 1 + a 1 x i, x iy i a N 0 x i + a N 1 x2 i In matrix form this becomes [ N x i x i x2 i ] N y i x 11 iy i 122 What s with the? It scares me! Some of you may still be scared of using the notation, so let s derive this another way Starting from 6, let s expand the summation to find S y 1 x 1 2 + y 2 x 2 2 + + y N x N 2 Now we can express using the chain rule as 2 y 1 x 1 1 + 2 y 2 x 2 1 + + 2 y N x N 1, 2 y 1 x 1 2 y 2 x 2 2 y N x N 12 and we can find the same way as 2 y 1 x 1 x 1 + 2 y 2 x 2 x 2 + + 2 y N x N x N, 2x 1 y 1 x 1 2x 2 y 2 x 2 2x N y N x N 13 Comparing equations 7 and 8 with 12 and 13, we see that they are equivalent We solve 0 y 1 x 1 + y 2 x 2 + + y N x N, 0 x 1 y 1 x 1 + x 2 y 2 x 2 + + x N y N x N, for and This is equivalent to 9 and 10 Matrix Form We can rewrite these equations in matrix form as well We first collect terms on and to find y 1 + y 2 + + y N N + x 1 + x 2 + + x N, x 1 y 1 + x 2 y 2 + + x N y N x 1 + x 2 + + x N + x 2 1 + x 2 2 + + x 2 N 3
2 NONLINEAR LEAST SQUARES In matrix form we have [ N x 1 + x 2 + x N x 1 + x 2 + x N x 2 1 + x2 2 + + x2 N ] y 1 + y 2 + + y N x 1 y 1 + x 2 y 2 + + x N y N, which is equivalent to 11 2 Nonlinear Least Squares For situations where the form of fx is such that its parameters enter nonlinearly, we have nonlinear least squares regression We again start with equation 1, and differentiate it with respect to the parameters in fx Let s consider a simple example to illustrate this point 21 Example: Power-Law Functions Assume we have a function in the form fx ax b Here, the parameters are a and b and they enter nonlinearly into the equation they are multiplied and b is an exponent In this case 1 becomes S y i ax b i 2 Taking partial derivative of S with respect to the parameters a and b gives a b 2 y i ax b i x b i 2 2 y i ax b i ax b i ln x i x b i 2ab y i ax b i, x b 1 i y i ax b i We now have two nonlinear equations to solve simultaneously for a and b Specifically, we solve the equations x b i y i ax b i x b i ln x i y i ax b i 0, 0, for a and b These equations are nonlinear since we have b as an exponent and we have products of the unknowns a and b 4
21 Example: Power-Law Functions 2 NONLINEAR LEAST SQUARES 211 Rewriting as a Linear Least Squares Problem Nonlinear equations are typically much more difficult to solve than linear equations In some cases, functions that appear to require nonlinear regression can be rewritten to use linear regression Let s look at the function fx ax b again For convenience, write this as y ax b Now we take the natural log of each side to find ln y ln ax b, ln a + ln x b, ln a + b ln x Now let s define α ln a, x ln x, and ỹ ln y We now have ỹ α + b x, which is linear with respect to α and b This looks identical to the situation we discussed in 12 Now we can perform regression by solving 9 and 10 to obtain α and b Once we have the solution for α and b, we recover a by a exp α Now instead of solving nonlinear equations we can solve linear equations! This is not always the case, but in many situations it is 5