SECTION 7: CURVE FITTING. MAE 4020/5020 Numerical Methods with MATLAB

Transcription

1 SECTION 7: CURVE FITTING MAE 4020/5020 Numerical Methods with MATLAB

2 2 Introduction

3 Curve Fitting 3 Often have data,, that is a function of some independent variable,, but the underlying relationship is unknown Know s and s (perhaps only approximately), but don t know Measured data Tabulated data Determine a function (i.e., a curve) that best describes relationship between An approximation to (the unknown) This is curve fitting and

4 Regression vs. Interpolation 4 We ll look at two categories of curve fitting: Least squares regression Noisy data uncertainty in value for a given value Want good agreement between and data points Curve (i.e., ) may not pass through any data points Polynomial interpolation Data points are known exactly noiseless data Resulting curve passes through all data points

5 5 Review of Basic Statistics Before moving on to discuss least squares regression, we ll first review a few basic concepts from statistics.

6 Basic Statistical Quantities 6 Arithmetic mean the average or expected value Standard deviation (unbiased) a measure of the spread of the data about the mean 1 where is the total sum of the squares of the residuals

7 Basic Statistical Quantities 7 Variance another measure of spread The square of the standard deviation Useful measure due to relationship with power and power spectral density of a signal or data set or

8 Normal (Gaussian) Distribution 8 Many naturally occurring random process are normally distributed Measurement noise Very often assume noise in our data is Gaussian Probability density function (pdf): where is the variance, and random variable, is the mean of the

9 Statistics in MATLAB 9 Generation of random values Very useful for simulations including noise Uniformly distributed: rand(m,n) E.g., generate an mn matrix of uniformly distributed points between xmin and xmax: x = xmin + (xmax xmin)*rand(m,n) Normally distributed: randn(m,n) E.g., generate an m n matrix of normally distributed points with mean of mu and standard deviation of sigma: x = mu + sigma*randn(m,n)

10 Statistics in MATLAB 10 Histogram plots: hist(x,binx,nbins) Graphical depiction of the variation of random quantities Plots the frequency of occurrence of ranges (bins) of values Provides insight into the nature of the distribution E.g., generate a histogram of the values in the vector x with 20 bins: hist(x,20) Optional input argument, binx, specifies x values at the centers of the bins Built in statistical functions: min.m, max.m, mean.m, var.m, std.m, median.m, mode.m

11 Statistics in MATLAB 11

12 12 Linear Least Squares Regression

13 Linear Regression 13 Noisy data,,values at known values Suspect relationship between and is linear i.e., assume Determine and that define the bestfit line for the data How do we define the best fit?

14 Measured Data 14 Assumed a linear relationship between and : Due to noise, can t measure Can only approximate values exactly at each Measured values are approximations True value of plus some random error or residual

15 Best Fit Criteria 15 Noisy data do not all line on a single line discrepancy between each point and the line fit to the data The error, or residual: Minimize some measure of this residual: Minimize the sum of the residuals Positive and negative errors can cancel Non unique fit Minimize the sum of the absolute values of the residuals Effect of sign of error eliminated, but still not a unique fit Minimize the maximum error minimax criterion Excessive influence given to single outlying points

16 Least Squares Criterion 16 Better fitting criterion is to minimize the sum of the squares of the residuals Yields a unique best fit line for a given set of data The sum of the squares of the residuals is a function of the two fitting parameters, and, Minimize by setting its partial derivatives to zero and solving for and

17 Least Squares Criterion 17 At its minimum point, partial derivatives of with respect to and will be zero Breaking up the summation: 0 0

18 Normal Equations 18 and form a system of two equations with two unknowns, and 1 2 In matrix form: 3 These are the normal equations

19 Normal Equations 19 Normal equations can be solved for and : Or solve the matrix form of the normal equations, (3), in MATLAB using mldivide.m (\) \

20 Linear Least Squares Example 20 Noisy data with suspected linear relationship Calculate summation terms in the normal equations:,,,

21 Linear Least Squares Example 21 Assemble normal equation matrices Solve normal equations for vector of coefficients,, using mldivide.m

22 Goodness of Fit 22 How well does a function fit the data? Is a linear fit best? A quadratic, higher order polynomial, or other non linear function? Want a way to be able to quantify goodness of fit Quantify spread of data about the mean prior to regression: Following regression, quantify spread of data about the regression line (or curve):

23 Goodness of Fit 23 quantifies the spread of the data about the mean quantifies spread about the best fit line (curve) The spread that remains after the trend is explained The unexplained sum of the squares represents the reduction in data spread after regression explains the underlying trend Normalize to the coefficient of determination

24 Coefficient of Determination 24 For a perfect fit: No variation in data about the regression line 0 1 If the fit provides no improvement over simply characterizing data by its mean value: 0 If the fit is worse at explaining the data than their mean value: 0

25 Coefficient of Determination 25 Calculate for previous example:

26 Coefficient of Determination 26 Don t rely too heavily on the value of Anscombe s famous data sets: Chapra Same line fit to all four data sets in each case

27 27 Linearization of Nonlinear Relationships

28 Nonlinear functions 28 Not all data can be explained by a linear relationship to an independent variable, e.g. Exponential model Power equation Saturation growth rate equation

29 Nonlinear functions 29 Methods for nonlinear curve fitting: Linearization of the nonlinear relationship Transform the dependent and/or independent data values Apply linear least squares regression Inverse transform the determined coefficients back to those that define the nonlinear functional relationship Nonlinear regression Treat as an optimization problem more later

30 Linearizing an Exponential Relationship 30 Have noisy data that is believed to be best described by an exponential relationship Linearize the fitting equation: or where,

31 Linearizing an Exponential Relationship 31 Fit a line to the transformed data using linear leastsquares regression Determine and : Can calculate for the line fit to the transformed data Note that original data must be positive

32 Linearizing an Exponential Relationship 32 Transform the linear fitting parameters, and, back to the parameters defining the exponential relationship Exponential fit: where, Note that is different than that for the line fit to the transformed data

33 Linearizing an Exponential Relationship 33

34 Linearizing a Power Equation 34 Have noisy data that is believed to be best described by an power equation Linearize the fitting equation: log log log or log log where log,

35 Linearizing a Power Equation 35 Fit a line to the transformed data using linear leastsquares regression Determine and : Can calculate for the line fit to the transformed data Note that original data both and must be positive

36 Linearizing a Power Equation 36 Transform the linear fitting parameters, and, back to the parameters defining the power equation Power equation: where, Note that is different than that for the line fit to the transformed data

37 Linearizing a Power Equation 37

38 Linearizing a Saturation Growth Rate Equation 38 Have noisy data that is believed to be best described by a saturation growth rate equation Linearize the fitting equation: or where 1 1,

39 Linearizing a Saturation Growth Rate Equation 39 Fit a line to the transformed data using linear leastsquares regression Determine and : Can calculate for the line fit to the transformed data

40 Linearizing a Saturation Growth Rate Equation 40 Transform the linear fitting parameters, and, back to the parameters defining the saturation growth rate equation Saturation growth rate equation: where, Note that is different than that for the line fit to the transformed data

41 Linearizing a Saturation Growth Rate Equation 41

42 42 Polynomial Regression

43 Polynomial Regression 43 So far we ve looked at fitting straight lines to linear and linearized data sets Can also fit m th order polynomials directly to data using polynomial regression Same fitting criterion as linear regression: Minimize the sum of the squares of the residuals m+1 fitting parameters for an m th order polynomial m+1 normal equations

44 Polynomial Regression 44 Assume, for example, that we have data we believe to be quadratic in nature 2 nd order polynomial regression Fitting equation: Best fit will minimize the sum of the squares of the residuals:

45 Polynomial Regression Normal Equations 45 Best fit polynomial coefficients will minimize Differentiate w.r.t. each coefficient and set to zero

46 Polynomial Regression Normal Equations 46 Rearranging the normal equations yields Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Which can be put into matrix form: Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ This system of equations can be solved for the vector of unknown coefficients using MATLAB s mldivide.m (\)

47 Polynomial Regression Normal Equations 47 For m th order polynomial regression the normal equations are: Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Σ Again, this system of 1equations can be solved for the vector of 1unknown polynomial coefficients using MATLAB s mldivide.m (\)

48 Polynomial Regression Example 48

49 Polynomial Regression polyfit.m 49 p = polyfit(x,y,m) x: vector of independent variable data values y: vector of dependent variable data values m: order of the polynomial to be fit to the data p: 1 vector of best fit polynomial coefficients Least squares polynomial regression if: 1 i.e., for over determined systems Polynomial interpolation if: 1 Resulting fit passes through all (x,y) points more later

50 Polynomial Regression polyfit.m 50 Note that the result matches that obtained by solving normal equations

51 Polynomial Regression Using polyfit.m 51 Exercise Determine the 4 th order polynomial with roots at Generate noiseless data points by evaluating this polynomial at integer values of from to Add Gaussian white noise with a standard deviation of to your data points Use polyfit.m to fit a 4 th order polynomial to the noisy data Calculate the coefficient of determination, Plot the noisy data points, along with the best fit polynomial

52 52 Multiple Linear Regression

53 Multiple Linear Regression 53 We have so far fit lines or curves to data described by functions of a single variable For functions of multiple variables, fit planes or surfaces to data Linear function of two independent variables: multiple linear regression Sum of the squares of the residuals is now,,

54 Multiple Linear Regression Normal Equations 54 Differentiate w.r.t. fitting coefficients and equate to zero The normal equations:,,,,,,,,,,,, Solve as before now fitting coefficients,, define a plane

55 55 General Linear Least Squares Regression

56 General Linear Least Squares 56 We ve seen three types of least squares regression Linear regression Polynomial regression Multiple linear regression All are special cases of general linear least squares regression The s are basis functions Basis functions may be nonlinear This is linear regression, because dependence on fitting coefficients,, is linear

57 General Linear Least Squares 57 For linear regression simple or multiple:,,, For polynomial regression:,,, In all cases, this is a linear combination of basis function, which may, themselves, be nonlinear

58 General Linear Least Squares 58 The general linear least squares model: Can be expressed in matrix form: where is an 1 matrix, the design matrix, whose entries are the 1 basis functions evaluated at the independent variable values corresponding to the measurements: where is the basis function evaluated at the independent variable value. (Note: is not the row index and is not the column index, here.)

59 General Linear Least Squares 59 The least squares model is: More measurements than coefficients is not square tall and narrow Over determined system does not exist

60 General Linear Least Squares Design Matrix Example 60 For example, consider fitting a quadratic to five measured values,, at Model is: Basis functions are,, and Least squares equation is

61 General Linear Least Squares Residuals 61 Linear least squares model is: Residual: (1) (2) Sum of the squares or the residuals: Expanding, (3) (4)

62 Deriving the Normal Equations 62 Best fit will minimize the sum of the squares of the residuals Differentiate with respect to the coefficient vector,, and set to zero We ll need to use some matrix calculus identities: (5) (6)

63 Deriving the Normal Equations 63 Using the matrix derivative relationships, (6), Equation (7) is the matrix form of the normal equations: Solution to (8) is the vector of least squares fitting coefficients: (7) (8) (9)

64 Solving the Normal Equations 64 (9) Remember, our starting point was the linear leastsquares model: Couldn t we have solved (10) for fitting coefficients as No, must solve using (9), because: Don t have, only noisy approximations, We have an over determined system is not square does not exist (10) (11)

65 Solving the Normal Equations 65 Solution to the linear least squares problem is: (12) where (13) is the Moore Penrose pseudo inverse of Use the pseudo inverse to find the least squares solutions to an over determined system

66 Coefficient of Determination 66 Goodness of fit characterized by the coefficient of determination: where is given by (3) and (14) (15)

67 General Least Squares in MATLAB 67 Have measurements at known independent variable values and a model, defined by 1basis functions Generate design matrix by evaluating 1basis functions at all values of

68 General Least Squares in MATLAB 68 Solve for vector of fitting coefficients as the solution to the normal equations or by using mldivide.m (\) Result is the same, though the methods are different mldivide.m uses QR factorization to find the solution to over determined systems

69 69 Nonlinear Regression

70 Nonlinear Regression fminsearch.m 70 Nonlinear models: Have nonlinear dependence on fitting parameters E.g., Two options for fitting nonlinear models to data Linearize the model first, then use linear regression Fit a nonlinear model directly by treating as an optimization problem Want to minimize a cost function Cost function is the sum of the squares of the residuals Find the minimum of a multi dimensional optimization Use MATLAB s fminsearch.m

71 Nonlinear Regression fminsearch.m 71 Have noisy data that is believed to be best described by an exponential relationship Cost function: Find and to minimize Use fminsearch.m

72 Nonlinear Regression fminsearch.m 72

73 Nonlinear Regression lsqcurvefit.m 73 An alternative to minimizing a cost function using fminsearch.m, if the optimization toolbox is available: a = lsqcurvefit(f,a0,x,y) f: handle to the fitting function e.g., f a(1)*exp(a(2)*x); a0: initial guess for fitting parameters x: independent variable data y: dependent variable data a: best fit parameters

74 Nonlinear Regression lsqcurvefit.m 74

75 75 Polynomial Interpolation

76 Polynomial Interpolation 76 Sometimes we know both and values exactly Want a function that describes Allows for interpolation between know data points Fit an order polynomial to data points Polynomial will pass through all points We ll look at polynomial interpolation using Newton s polynomial The Lagrange polynomial

77 Polynomial Interpolation 77 Can approach similar to linear least squares regression where,, For an order polynomial, we have with unknowns In matrix form equations

78 Polynomial Interpolation 78 Now, unlike for linear regression All 1values in are known exactly 1equations with 1unknown coefficients is square Could solve in MATLAB using mldivide.m \ is a Vandermonde matrix Tend to be ill conditioned The techniques that follow are more numerically robust

79 79 Newton Interpolating Polynomial

80 Linear Interpolation 80 Fit a line (1 st order polynomial) to two data points using a truncated Taylor series (or simple trigonometry): where is the function for the line fit to the data, and are the known data values This is the Newton linear interpolation formula

81 Quadratic Interpolation 81 To fit a 2 nd order polynomial to three data points, consider the following form Evaluate at to find Back substitution and evaluation at will yield the other coefficients and at and

82 Quadratic Interpolation 82 Can still view this as a Taylor series approximation represents an offset is slope is curvature Choice of initial quadratic form (Newton interpolating polynomial) was made to facilitate the development Resulting polynomial would be the same for any initial form of an order polynomial Solution is unique

83 Order Newton Interpolating Polynomial 83 Extending the quadratic example to order Solve for coefficients as before with back substitution and evaluation of denotes a finite divided difference

84 Finite Divided Differences 84 First finite divided difference, Second finite divided difference,,,, finite divided difference,,,,,,,, Calculate recursively

85 85 Order Newton Interpolating Polynomial order Newton interpolating polynomial in terms of divided differences:,,,,, Divided difference table for calculation of coefficients: Chapra

86 Newton Interpolating Polynomial Example 86

87 87 Lagrange Interpolating Polynomial

88 Linear Lagrange Interpolation 88 Fit a first order polynomial (a line) to two known data points: and and are weighting functions, where 1, 0, 1, 0, The interpolating polynomial is a weighted sum of the individual data point values

89 Linear Lagrange Interpolation 89 For linear (1 st order) interpolation, the weighting functions are: The linear Lagrange interpolating polynomial is:

90 Order Lagrange Interpolation 90 Lagrange interpolation technique can be extended to order polynomials where

91 Lagrange Interpolating Polynomial Example 91

92 92 MATLAB s Curve Fitting Tool

93 MATLAB s Curve Fitting Tool GUI 93 Type cftool at the command line Launches curve fitting tool GUI in a new window Import data from the workspace Quickly try fitting different types of functions to the data Polynomial Exponential Power Fourier Custom equations, and more Interpolation for Least squares regression for

94 MATLAB s Curve Fitting Tool GUI 94

95 Curve Fitting with the cftool GUI 95 Go to the course website and download the following data file: cftool_ex_data.mat Exercise Load data file into the workspace At the command line, type: load( cftool_ex_data ) Launch the Curve Fitting Tool Type cftool at the command line Try fitting different functions to each of the three data sets Can open a new tab in the GUI for each fit