General Linear Least-Squares and Nonlinear Regression

Transcription

1 Part 4 Chapter 14 General Linear Least-Squares and Nonlinear Regression PowerPoints organized by Dr. Michael R. Gustafson II, Duke University Revised by Prof. Jang, CAU All images copyright The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

2 Chapter Objectives Knowing how to implement polynomial regression. Knowing how to implement multiple linear regression. Understanding the formulation of the general linear least-squares model. Understanding how the general linear leastsquares model can be solved with MATLAB using either the normal equations or left division. Understanding how to implement nonlinear regression with optimization techniques.

3 Polynomial Regression The least-squares procedure from Chapter 13 can be readily extended to fit data to a higher-order polynomial. Again, the idea is to minimize the sum of the squares of the estimate residuals. The figure shows the same data fit with: a) A first order polynomial b) A second order polynomial

4 Process and Measures of Fit For a second order polynomial, the best fit would mean minimizing: n n S r e i y i a 0 a 1 x i a x i i1 i1 In general, this would mean minimizing: n S r e i n i1 i1 y i a 0 a 1 x i a x m i a m x i The standard error for fitting an m th order polynomial to n data points is: s y/ x S r n m 1 because the m th order polynomial has (m+1) coefficients. The coefficient of determination r is still found using: r S t S r S t

5 Example 14.1 (1/4) Q. Fit a second-order polynomial to the data in the first two columns below. xi y y a a x a x i ( y y) i i 0 1 i i

6 Example (/4) Sr ( yi a0 a1xi axi ) a 0 S a S r 1 xi( yi a0 a1xi axi ) r xi ( yi a0 a1 xi a xi ) a ( na ) 0 xia1xi a yi x a x a x a x y i i i i i x a x a x a x y 3 4 i i i i i a0 i i i a1 i a i i 3 m x x 4 i 15 i 979 n6 y 15.6 x y x.5 x 55 x y y 5.43 x i 5 The simultaneous linear equations are

7 Using MATLAB Example (3/4) >> N=[ ; ; ]; >> N=[ ]; >> a = N\r a = The least-squares quadratic equations for this case is y x1.8607x

8 Example (4/4) The standard error of the estimate based on the regression polynomial is sy/ x The coefficient of determination : r r The correlation coefficient % of the original uncertainty has been explained by the fit (model). -> the equation represents an excellent fit.

9 General Linear Least Squares Linear, polynomial, l and multiple l linear regression all belong to the general linear least-squares squares model: y a 0 z 0 a 1 z 1 a z a m z m e where z 0, z 1,, z m are a set of m+1 basis functions and e is the error of the fit. The basis functions can be any function data but cannot contain any of the coefficients a 0, a 1, etc.

10 Solving General Linear Least Squares Coefficients The equation: y a 0 z 0 a 1 z 1 a z a m z m e can be re-written for each data point as a matrix equation: y Z e a where {y} contains the dependent data, {a} contains the coefficients of the equation, {e} contains the error at each point, and [Z] is: z 01 z 11 z m1 Z z 0n z 1n z mn z 0 z 1 z m with z ji representing the the value of the j th basis function calculated at the i th point.

11 Solving General Linear Least Squares Coefficients Generally, [Z] is not a square matrix, so simple inversion cannot be used to solve for {a}. Instead the sum of the squares of the estimate residuals is n n m minimized: S r e i y i a j z ji i1 1 i1 1 j0 0 This quantity can be minimized by taking its partial derivative with respect to each of the coefficients and setting the resulting equations to zero. The outcome of this minimization yields in the following matrix form: T T Z Z a Z y r y i y i where y 1 1 of S S y y ˆ y y St Sr Sr t t i i y i ˆ yˆ represent the prediction the least - squares fit and. y Z a

12 Example 14.3 (1/3) Q. Use matrix operations o to repeat Ex >> x= [ ]' ; % enter the data to fit >> y= [ ]' ; >> z = [ones(size(x)) x x.^] %[z] matrix z =

13 Example 14.3 (/3) >> z'*z %[ [z] T [z] ans = >> a = (z'*z)\(z *y) *y) % Solve for the coefficients of eq. ans = T Z Z a Z T y

14 Example 14.3 (3/3) >> Sr = sum((y-z*a).^) % Sr Sr = >> r = 1-Sr/sum((y-mean(y)).^) % r r = >> syx = sqrt(sr/(length(x)-length(a))) % s y/x syx = r ˆ i i St S y y r S r 1 1 s y/ x S S y y t t i i S r n m 1

15 Nonlinear Regression As seen in the previous chapter, not all fits are linear equations of coefficients and basis functions. One method is to perform optimization techniques to directly determine the leastsquares fit. First write a function that returns the sum of the squares of fthe estimate t residuals for a fit and dthen use MATLAB s fminsearch function to find the values of the coefficients where a minimum occurs. (1 ax 1 y a ) ax 1 0 e e f( a0, a1) yi a0(1 e i ) [x, fval] =fminsearch(fun, x0, options, p1, p, ) n i1

16 Nonlinear Regression in MATLAB Example (option) Given dependent d force data F for independent d velocity data v, determine the coefficients for the fit: a 1 F a 0 v First - write a function called fssr.m containing the following: function f = fssr(a, xm, ym) yp = a(1)*xm.^a(); f = sum((ym-yp).^); yp) ); Then, use fminsearch in the command window to obtain the values of a that minimize fssr: a = fminsearch(@fssr, [1, 1], [], v, F) where [1, 1] is an initial guess for the [a0, a1] vector, [] is a placeholder for the options

17 Nonlinear Regression vs. Linear Regression Empolying Transformations Although the model coefficients are different, it is difficult to judge which fit is superior. Nonlinear regression: original i data Linear Regression Empolying Transformations: the transformed data