Curve Fitting. Least Squares Regression. Linear Regression

Transcription

1 Curve Fitting Curve fitting is expressing or modelling a discrete set of data points as a continuous function. There are two main branches of curve fitting which are regression and interpolation. Regression is typically used for experimental data which may have some significant amount of error (e.g. noise in accelerometer data). Therefore, in regression, the function we fit does not have to connect (i.e. pass through) all of the data points. In regression, the fitted curve represents a model that describes the general trend or pattern of the data points as depicted in Figures 1 and 2. It is common for the experimental data to be affected by uncontrolled factors. For example, Figure-1 may describe the performance of an employee where the y-axis corresponds to the number of completed papers and the x-axis corresponds to time. It is logical that as time passes, more papers are completed but keep in mind that there can be some papers which may take more time to complete and the employee mood can change from day to day. Fig. 1 Linear regression Fig. 2 Polynomial regression Interpolation is applied when the data is known to be very precise and we fit a function or a series of functions that connect all of the data points as depicted in Figures 3 and 4. Example: robot motion planning. Fig. 3 Quadratic polynomial interpolation Fig. 4 Spline interpolation. f(x), g(x) and h(x) are three separate functions. Least Squares Regression Linear Regression The aim is to fit a straight line y = a + a x to a data set composed of the data points (x, y ) where i = 1,2,, n. Here, a is the y-intercept, a is the slope and e is the error (or deviation, or residual) of the i th data point. e is simply the vertical distance between the i th data point and the straight line. The fitted straight line is called the regression line. Then, e = y a a x as shown in the below figure. 1

2 We would like to draw a best-fit line that minimises the error (or deviation). When such a best-fit line is determined, the parameters a and a are also found. One can think of several possibilities to minimise the error; these are: a) Minimising the sum of individual errors b) Minimising the sum of absolute values of individual errors c) Minimising the maximum error d) Minimising the sum of squares of individual errors Option (d) is the preferred method and it provides a unique (i.e. single) best-fit line (or regression line). Please read the textbook to see why the other options fail. Minimising the sum of squares of individual errors S = e = (y a a x ) S is the sum of squares the residuals (i.e. errors). We need to find the parameters a and a by minimising S as follows: S = 2 (y a a a x ) = 0 na + x a = y Eqn. 1 S = 2 (y a a a x )x = 0 x a + x a = x y Eqn. 2 Writing the above two equations in matrix form, we get the normal equations as given below. n x x a y a = x x y Solving the above equations for a and a, we get a = n x y x y n x ( x ), a = y a x where y and x are the arithmetic means of y-data and x-data, respectively (see Eqn. 1). 2

3 Dependent and independent variables in Regression In the regression formulas developed in this course, we typically have two variables x and y which are the independent and dependent variables, respectively. Thus, the function or curve that we want to fit is typically expressed as y = f(x). The independent variable x is the input variable which we can control and adjust precisely. The variable x is not random and should not involve any significant error. The variable x can also be a variable which does not depend on anything such as time. The dependent variable y is the output variable which we measure as a result of the experiment. The variable y can be affected by unknown or uncontrollable factors. The variable y can involve measurement errors. Quantification of Error of Linear Regression Definitions: Arithmetic mean of the dependent variable y is given by y as expressed below where n is the number of data points. y = y /n Standard deviation s is a measure of spread about the mean of the data. s = S n 1 where S = (y y) If the individual measurements are distributed widely around the mean, s and S are large. If they are grouped closely, s and S are small. The distribution of the data can also represented by the square of standard deviation, which is called the variance s. s = S n 1 For n = 1, s. So, the distribution of a single data is meaningless, we need at least 2 data points. Coefficient of variation c. v. provides a normalised measure of the distribution relative to the mean y. c. v. = s 100 y Remember that the sum of squares the residuals is given by S. S = (y a a x ) Standard deviation for the regression line is called the "standard error of the estimate" and it is given by s /. s / = S n 2 s / quantifies the spread aroung the regression line. The subscript y/x indicates that the error is for a predicted value of y corresponding to a particular value of x. If the number of data points n is equal to 2, the regression line connects the two data points, then s / becomes indeterminate since S and (n 2) 3

4 become both zero; hence for n = 2 the distribution of data around the straight-line connecting the two datapoints is meaningless. Let's analyse the distribution of the data points (x, y ) where i = 1,2,, n. In Figure 5, the distribution (or spread) of the data points around y (i.e the mean of the dependent variable y) is shown by utilising a bellshaped curve. In Figure 6, the distribution of the data points around the regression line is shown. In comparison to Figure 5, the bell-shaped curve in Figure 6 is thinner indicating that the points are grouped more tightly around the regression line. Then we can define r = S S S where r is called the coefficient of determination and r is called the correlation coefficient. Fig. 5 Spread of data points around the mean of y Fig. 6 Spread of data points around the regression line S S quantifies the improvement or error reduction in representing the data in terms of a straight-line (i.e. regression line) rather than as an average value given by y. Hence, our quality of straight-line fit is with respect to the horizontal line which has a y value of y as shown in Figures 5 and 6. For a perfect fit, S = 0 which means all data points are on the straight-line (i.e. regression line) thus r = r = 1 and there is finite S (see Figure 7). If S = S, then r = r = 0 which means we haven't applied regression at all and we are using the horizontal line (i.e. the mean value of y) to represent the trend of the data; this is a poor approximation as shown in Figure 8. See that S > 0 and S > 0, then since r > 0, we have S > S. It may be said that when r or r is closer to 1, the fit is better but this is not always true. Therefore, we should always plot the data and the regression line to check visually the goodness of fit. Fig. 7 Perfect fit: S = 0 Fig. 8 Poor fit: S = S Linearisation of Nonlinear Behaviour The relationship between the dependent and independent variables in a data-set may not be linear; therefore in such cases it is more suitable to fit a nonlinear curve (or function) to the data. However, upto now we have formulas for linear regression only. Nevertheless, it is sometimes possible to transform the nonlinear 4

5 function into a linear form so that we can use the formulas developed for linear regression. Here are some examples in which Y and X are the dependent and independent variables, respectively: 1) Exponential equation: Y = A e This equation can be linearised by taking the natural logarithm of both sides of the equation as given below. Then, y = ln(y), a = ln (A ), a = B, x = X ln(y) = ln (A ) + B X y = a + a x Linearisation 2) Power equation: Y = A X This equation can be linearised by taking the base-10 logarithm of both sides of the equation as given below. log (Y) = log (A ) + B log (X) y = a + a x Then, y = log (Y), a = log (A ), a = B, x = log (X) Linearisation 3) Saturation-growth rate equation: Y = A X/(B + X) This equation can be linearised by inverting it as follows. 1 Y = 1 A + B A 1 X Then, y = 1/Y, a = 1/A, a = B /A, x = 1/X y = a + a x Linearisation 5

6 Once we obtain the linearised equation by using any linearisation procedure, we will apply least-squares regression to the linearised equation. However, we should keep in mind that linearisation as such, in some cases, does not really mean that we are applying least-squares regression to the original nonlinear equation. Polynomial Regression We can also fit polynomials to a given data if the distribution of the data resembles the shape of a polynomial. An m th degree polynomial can be written as y = a + a x + a x + + a x Considering that the number of data points is n, we can write S = e = (y a a x a x a x ) See that we have (m + 1) unknown coefficients a, a, a,, a and we need to write (m + 1) equations to find them. These (m + 1) equations are written by setting S /a = 0 for each j = 0,1,2,, m thus S a = 0, S a = 0, S a = 0,, S a = 0 The standard error of the estimate s / and the coefficient of determination r are given as S s / = n (m + 1), r = S S S, S = (y y) Looking at s /, we can identify three cases as given below: i) s / is defined if n > (m + 1) i.e. n (m + 2). In this case, we have a typical and well-defined leastsquares regression problem hence it is possible to find a unique solution for the unknown coefficients of the polynomial. In other words, it is possible to determine uniquely the polynomial to be fitted to the given data points. ii) If n = (m + 1), then it can be proven that the polynomial that we fit, connects all the data points. In other words, the data points are exactly on the polynomial that we fit. Thus, e = 0 for i = 0,1,2,, n and S = 0. Consequently, s / becomes indeterminate since we don't see or get any distribution of the data points around the polynomial. This case is called interpolation which we will study in the next chapter. iii) If n < (m + 1), then it can be shown that there is no unique solution for the unknown coefficients of the polynomial. In other words if m n, we cannot find a unique polynomial. In this case, if you use the Matlab function polyfit to perform polynomial regression, Matlab gives a warning and returns a particular solution to this problem by setting a number of coefficients to zero without changing the degree of the polynomial (See Example 3). Example1: Consider the following data where y is the measured variable which depends on x. Fit a 2 nd order (or degree) polynomial y = a + a x + a x to the data using least-squares regression. x=0:5; y=[ ]; Solution1: You can use Matlab s function polyfit. See that n = 6 and m = 2 thus n > (m + 1). >> p=polyfit(x,y,2) p = % p gives the coefficients of the polynomial ordered in descending powers. 6

7 Therefore, a = p(3), a = p(2), a = p(1) % Read the help files of the functions polyfit and polyval. % To plot the data together with the fitted polynomial, define >> xx=0:0.01:5; >> plot(x,y,'o',xx,polyval(p,xx)) % Alternatively, you can solve this problem using the Basic Fitting tool of Matlab. You just need to plot the data first by typing plot(x,y,'o'). A figure window then opens and in this window, select the Tools menu to access the Basic Fitting tool. Example2: Consider the data set given in Example-1 but this time select 3 data-points out of this set as given below. Again fit a 2 nd order (or degree) polynomial to the selected data-points given below using leastsquares regression. See that n = 3 and m = 2 thus n = (m + 1). Solution2: x=[1 2 4]'; y=[ ]'; >> p=polyfit(x,y,2); xx=0:0.01:5; plot(x,y,'o',xx,polyval(p,xx)) >> xlabel('x'), ylabel('y'), legend('data points','fitted polynomial','location','best') Let's evaluate the residuals (or errors) of the data-points. Remember that the errors correspond to the vertical distances between the data points and the fitted curve. >> y-polyval(p,x) 7

8 ans = 1.0e-14 * See that the errors are almost zero for all the data-points. Theoretically, all of the errors must be exactly zero. Since there is round-off error, we didn't get exactly zero errors. Example3: Consider the data set given in Example-2 but this time fit a 3 rd order (or degree) polynomial to the data-points using least-squares regression. See that n = 3 and m = 3 thus n < (m + 1). Solution3: >> x=[1 2 4]'; y=[ ]'; >> p=polyfit(x,y,3), xx=0:0.01:5; plot(x,y,'o',xx,polyval(p,xx)) Warning: Polynomial is not unique; degree >= number of data points. p = >> xlabel('x'), ylabel('y'), legend('data points','fitted polynomial','location','best') Since the solution is not unique, the polyfit function finds a particular solution to this problem by setting one of the coefficients to zero. The residuals (or errors) given below indicate that Matlab has chosen a 3 rd order polynomial which passes through (or connects) all the data points. >> y-polyval(p,x) ans = 1.0e-14 * Fitting a regression line with zero intercept The function to be fitted is y = a x. S = (y a x ) S = 2 [(y a a x )x ] = 0 x y = a x ) Finally, a = ( x y )/( x 8

9 Remember that the functions y = a x and y = a x/(b + x) also have zero intercept. Functions with zero intercepts cross the origin of the coordinate system. 2nd order polynomial regression with zero intercept The function to be fitted is y = a x+a x. S = (y a x a x ) S = 2 (y a a x a x ) x = 0 a x + a x = x y S = 2 (y a a x a x ) x = 0 a x + a x = x y Solve for a and a. Multiple Linear Regression The function to be fitted has a dependent variable y which is a linear function of two or more independent variables as given below. y = a + a x + a x + + a x In the above equation (or function), there are m independent variables x, x,..., x and the number of data is n. Then we can write S s / = n (m + 1), r = S S S, S = (y y) S = (y a a x a x a x ) Show that when y = a + a x + a x, the coefficients a, a, a are found by solving the following system of linear equations. n x x x x x x x x x x a a = a y x y x y Remember from Calculus that the equation y = a + a x + a x represents a plane therefore we have a regression plane here which follow the general trend of the data points as shown in Figure 9. 9

10 Fig. 9 Regression plane and the data-points If the function to be fitted is of the form y = a x x x, then multiple linear regression can still be applied by performing linearisation as follows: log(y) = log(a ) + a log(x ) + a log(x ) + + a log(x ) 10