Non-Linear Models. Estimating Parameters from a Non-linear Regression Model

Transcription

1 Non-Linear Models Mohammad Ehsanul Karim Institute of Statistical Research and training; University of Dhaka, Dhaka, Bangladesh Estimating Parameters from a Non-linear Regression Model Illustrative Example: Let the data I file is in c:\ draper.txt, which is as follows: t Y.8.. Let that, we have to estimate the parameter θ in the non-linear model from the above observations. t Y e θ + ε Now for starting analysis, we need a initial value. But we have no prior information about what values θ might take. There are a few suggestions for such situations: one of which is to make a grid search using Statistical Software. Let us demonstrate how we can do this. See updates on this at Using Statistical Software: SAS: Making a Grid Search We write the following program in SAS (I used SAS. to get an estimate (note that we told SAS to do the search in Linearization technique by setting METHODNEWTON: DATA DS; INPUT T Y; DATALINES;.8.. RUN; I We are using this simple and small data set just for easy illustration purpose. This data set is taken from Draper, Smith (998 Applied Regression Analysis, Third ed., Page, Exercise.A.

2 PROC NLIN DATADS METHODNEWTON; PARAMETERS G- TO BY.; MODEL Y EXP(-(G*T; OUTPUT OUTNLINOUT PREDICTEDPRED L9ML9MEAN U9MU9MEAN L9L9IND U9U9IND; RUN; PROC PRINT DATANLINOUT; RUN; There will be an enormous search to find this estimate since we assumed our potential parameter (in the program, we wrote θ G space to be to and told SAS to look within each. interval (so that we can get minimum residual SS or minima. However, after some iteration (when a convergent value will be found SAS will bring the following output (edited to shorten the length but the key parts of the output is unedited: The SAS System : Sunday, (Detailed Results on Non-Linear Least Squares Grid Search is omitted from here. Non-Linear Least Squares Iterative Phase Dependent Variable Y Method: Newton Iter G Sum of Squares NOTE: Convergence criterion met. The SAS System : Sunday, April, 99 Non-Linear Least Squares Summary Statistics Dependent Variable Y Source DF Sum of Squares Mean Square Regression Residual.8.98 Uncorrected Total.8 (Corrected Total.89 Parameter Estimate Asymptotic Asymptotic 9 % Std. Error Confidence Interval Lower Upper G Asymptotic Correlation Matrix Corr G ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ G The SAS System : Sunday, April, 99 OBS T Y PRED L9MEAN U9MEAN L9IND U9IND Therefore, from SAS, we get estimate of θ.888. So we may suppose that the actual value may be around zero for now and check it in other ways.

3 Calculation By Hand: Non-linear Regression: We will now go through some iterations just to see how the process works. Readers may think it as a slow motion of what happens to any Statistical Software when we run a non-linear regression. Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with approximated initial value θ (as our grid search indicates note that, since we have to estimate only one value, this is just a scaler, but for multi-parameter case, we would have a vertor. t f f( ξθ, e since t is a vector. Y f.8 -. Now, y Y f Y f. -. Y f. -.9 f( ξθ, t Z ( t e θ θ + + ( ( -. ( f Z Y (-. (-+(-. (-+(-.9 (-. b ( Z ( Y f...9 θ ˆ θ ˆ + b Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with.9..9 ˆ t e θ.88 since t is a vector.. Y f -. Now, y Y f Y f -.88 Y f f( ξθ, ˆ t Z ( t e θ -.89 θ θ -.. ( (. -

4 Z Y.889 ( f b ( Z ( Y f.89 θ ˆ θ ˆ + b.9 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with.9..8 ˆ t e θ.8 since t is a vector..88 Y f -.9 Now, y Y f Y f -. Y f f( ξθ, ˆ t Z ( t e θ -.98 θ θ ( (9.8 - ( f Z Y.9 b ( Z ( Y f. θ ˆ θ ˆ + b.8 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with ˆ t e θ.8 since t is a vector..98 Y f Now, y Y f Y f -.8 Y f f( ξθ, ˆ t Z ( t e θ θ θ ( ( ( f Z Y.8 b ( Z ( Y f.99 θ ˆ θ ˆ + b.

5 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with..8 ˆ t e θ. since t is a vector..9 Y f -. Now, y Y f Y f.9 Y f f( ξθ, ˆ t Z ( t e θ -.88 θ θ ( (.98 - ( f Z Y.98 b ( Z ( Y f.8 θ ˆ θ ˆ + b.8 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with.8.89 ˆ t e θ.989 since t is a vector..88 Y f Now, y Y f Y f.88 Y f f( ξθ, ˆ t Z ( t e θ -. θ θ ( (.89 - ( f Z Y.8 - b ( Z ( Y f.8 - θ ˆ θ ˆ + b.8 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with.8

6 .89 ˆ t e θ. since t is a vector..888 Y f -.9 Now, y Y f Y f.89 Y f f( ξθ, ˆ t Z ( t e θ -.9 θ θ ( (.89 - ( f Z Y. - b ( Z ( Y f.98 - θ ˆ θ ˆ + b.89 Iteration : We have the function Y f( ξθ, + ε e θt + ε in our hand with ˆ t e θ.8 since t is a vector..88 Y f Now, y Y f Y f.888 Y f f( ξθ, ˆ t Z ( t e θ -.9 θ θ ( ( ( f Z Y. -9 b ( Z ( Y f θ ˆ θ ˆ + b.89 8 Since θ ˆ ˆ 8 θ we terminate the process and declare θ ˆ 8.89 as our desired estimate.

7 8 Using Statistical Software: R: Non-linear Regression Now, using Statistical Software like R (I used R.. we can do the whole procedure just in a blink (and of course, without pain! by writing the following commands in the R console: > options(prompt" R> " R> draper<-read.table("c:\\draper.txt",headert R> draper.data<-data.frame(draper R> attach(draper.data R> nls(y~exp(-(g*t, datadraper.data, start c(g, trace T. :.999 :.9. :.9.9 :.8. :..8 :.8.8 :.8.8 :.89 Nonlinear regression model model: Y ~ exp(-(g * t data: draper.data g.89 residual sum-of-squares:.8 R> coef(nls(y~exp(-(g*t, datadraper.data, start c(g, trace T (--- g.89 Note that, the estimated value of θ.89 approximately in all our (slow motion hand calculation and R results within a blink cases which matches our SAS result too. Thus, we declare.89 as our estimate of the parameter θ based on the given data (See for the details discussions on it and more.