Non-Linear Models Topic Outlines

Transcription

1 Non-Linear Models Mohammad Ehsanl Karim Institte of Statistical Research and training; University of Dhaka, Dhaka, Bangladesh Topic Otlines Intrinsically Linear Regression Models Intrinsically Non-linear Regression Models Least Sqares in the Non-linear Case Linearization Method to obtain Estimates Using Statistical Packages (SAS, R, NLREG)

2 Linear in parameters models (inclding only first order models in p- independent variables with p parameters) can be generalized as Y = β + βz+ βz+ β3z β p Zp + ε where the Z i can represent any fnctions of the basic predictor variables X, X, X 3,, X k. While the above eqation can represent a wide variety of relationships, there are many sitations in which a model of this form is not appropriate; for example, when definite information is available abot the form of the relationship between the response and the predictor variables. Sch information might occasionally involve direct knowledge of the actal form of the tre model or might be represented by a set of differential eqations that the model mst satisfy. Sometimes the information leads to several alternative models, may be of a non-linear form and we wold sally prefer sch a model whenever possible, rather than to fit an alternative, perhaps less realistic linear model I. A regression model is called nonlinear, if the derivatives of the model with respect to the model parameters depends on one or more parameters. This definition is essential to distingish nonlinear from crvilinear regression. A regression model is not necessarily nonlinear if the graphed regression trend is crved. A polynomial model appears crved when y is plotted against x. It is, however, not a nonlinear model. Fitting a nonlinear regression model to data is slightly more involved than fitting a linear model, bt they have specific advantages: - Nonlinear models are often derived on the basis of physical and/or biological considerations, e.g., from differential eqations, and have jstification within a qantitative conceptalization of the process of interest. - The parameters of a nonlinear model sally have direct interpretation in terms of the process nder stdy. - Constraints can be bilt into a nonlinear model easily and are harder to enforce for linear models. Intrinsically Linear and Intrinsically Non-linear Regression Models Any model not of the above given form will be called a non-linear model, that is non-linear in parameters. Non-linear regression models can be classified into two grops according to whether they can or cannot be made linear with respect to the parameters to be estimated.. Intrinsically Linear Models: A non-linear model with respect to the variables bt linear with respect to the parameters to be estimated. Sitable transformations of data can freqently (not always) be fond that will redce a theoretical nonlinear model to a linear form. These transformations are said to be Linearizable II and comprise a class of fnctions that may exist either occr in practice or may themselves provide reasonable approximations to fnctions that occr in practice. Linearizing may reqire transforming both the independent and dependent ( t ) variable. For instance, Y = e θ + θ + ε which can be transformed in a model which is I Althogh this is tre that Linear regression models can provide a large and rich framework that sits the needs of many analysis, and the seflness of linear models is more general than apparent, bt linear regression cannot be adeqate for all problems theoretically or empirically, since sometimes the response and the predictors are related throgh a known non-linear fnction. II This term is sed by Chatterjee, Price (99) Regression analysis by Example, nd ed., ch and Weisberg (98) Applied linear regression, ch 6.

3 3 linear in parameter jst by taking logarithms to the base e and we shall call the model Intrinsically Linear that is, an intrinsically linear model is one that can be made linear by a transformation of parameters. The basic common characteristic of sch models is that they can be converted into ordinary linear models by sitable transformation of variables and sch transformation amonts to nothing more than re-labeling one or more of the variables. While it may be sefl, at times, to transform a model of this type so that it can be easily fitted, it will remain a non-linear model, whatever the transformation applied. However, the least sqares estimates of the parameters will not in general be eqivalent to the non-linear parameter estimates in the original model in this case. The reason is that in the original non-linear model least sqares implies minimization of the sm of sqared residals on y, whereas in the transformed model, we are minimizing the sm of sqared residal on transformed response (that is, ln y in or present case). Also, the isse often resolves arond the error strctre III (sch as so the standard assmptions on the errors apply to the original non-linear model or to the linearized one?) is sometimes not an easy qestion to answer.. Intrinsically Non-linear Models: Not all fnctions are linearizable, nor in some cases it is desirable to transform to linearity. For example, θ θt θt Y = [ e e ] + ε however, is impossible to convert into a form linear θ θ in parameters, which we will call Intrinsically Nonlinear IV. III When an intrinsically linear model has been transformed into the linear model form, it is important to stdy the linearized model for aptness since the normally distribted error term may not be normally distribted anymore when transformed. IV Nonlinear least sqares and maximm log-likelihood are the most common methods for parameter estimation of nonlinear models in econometrics. For some more examples related to Economics, readers may conslt Gjarati (3) Basic Econometrics 4 th ed., page 564-5, Kotsoyiannis (977) Theory of Economertics An introdction exposition of Economic Methods, nd ed., page 34-8, Jan Kmenta (986) Elements of Econometrics, nd ed., page 53-6 for these two classifications.

4 4 Least Sqares in the Non-linear Case Notations: The standard notation for non-linear Least Sqares V sitations is a bit different from that of linear Least Sqares cases. Sppose the postlated model is of the form Y = f( ξ, ξ,..., ξ ; θ, θ,..., θ ) + ε k p where the θ s are the parameters, and ξ s are the predictor variables and the ε term follows sal assmptions regarding error terms as we know from linear regression sch as E(ε ) = and V(ε ) = σ, where ε N(, σ ) i.i.d and are ncorrelated.. If we write ξ θ ξ θ ξ = and θ = ξ k θ k p p We denote the nmber of independent ξ variables by k, since the nmber of ξ variables in non-linear regression is not directly related to the nmber of parameters, nlike linear regression VI. Ths, the above model can be shorten to Y = f( ξθ, ) + ε EY ( ) = f( ξθ, ) Now, let s extend the model for n observations as Y = f( ξ, ξ,..., ξ ; θ, θ,..., θ ) + ε, for =(,,,n) k p Y = f( ξ, θ) + ε ε ε where, ε = N(, Iσ )... εn Procedre: We define the Error Sm of Sqares VII for the non-linear model as n S( θ) = [ Y f( ξ, θ)] = We shall denote by θ ˆVIII, a least sqares estimate of θ, that is a vale of θ which minimizes S(θ ). To find the least sqares estimate θ ˆ, we need to differentiate the Error Sm of Sqares with respect to θ V Ordinary Least sqares applied to a non-linear regression model is called non-linear Least Sqares. VI Readers may conslt Neter, Wasserman, Ktner (983) Applied Linear Regression Models, page 47- for this matter. VII Since Y, ξ are fixed observations, the sm of sqares is a fnction of θ only. VIII It trns ot that θ ˆ is also a Maximm Likelihood estimate of θ in this case.

5 5 n S( θ) f( ξ, θ) = ( ){ Y f( ξ, θ)} θi = θi When the p partial derivatives are set eqal to zero, and the parameters θ i are replaced by θ ˆ, and after simplification, we obtain the p normal eqations. The normal eqations take the form n f( ξ, θ) { Y f( ξ, θ)} =, for i = (,,,p) = θi θ= θˆ where the qantity in the brackets is the derivative of f( ξ, θ ) with respect to θ i with all θ s replaced by the corresponding θ ˆ s, which have the same sbscript. When the fnction f( ξ, θ ) was linear, this qantity was a fnction of ξ only and did not involve the θ ˆ s at all. When the model is non-linear in the θ s, so will be the normal eqations. Problem: However, in non-linear case, the solving the normal eqations is not easy (the development of the least sqares estimators for a non-linear model brings abot complications not encontered in the case of the linear model) and in most cases, it is extremely difficlt to obtain and iterative methods mst be employed in nearly all cases. To compond the difficlties, it may happen that mltiple soltions exist, corresponding to mltiple stationary vales of the fnction S( θ ˆ). The statistical literatre is qite rich in algorithms for minimization of the residal sm of sqares in non-linear sitations. Next, we will, therefore, discss methods that have been sed to estimate the parameters in non-linear systems. Approaches to Estimation Non-linear Regression Models: In some non-linear problem, it is most convenient to write down the normal eqations and develop a direct (method below) iterative techniqe for solving them. Whether this works satisfactorily or not depends on the form of the eqations and the iterative method sed. There are some of the alternative approaches. Direct search (Trial-and-Error or Derivative-free techniqe IX ). Linearization (iterative method or Gass-Newton Method) 3. Steepest descent X (Direct Optimization) 4. Marqardt s compromise XI IX This method is not generally sed since it takes hge time and effort and; at the end of the day we may find some estimated parameter vales with poor properties! X This method proceeds very systematically (and ths sometimes very slowly) starting with some initial vales. This method is particlarly effective when the starting vales are not good and far from the final vales. While, theoretically, the steepest descent method will converge, it may do so in practice with agonizing slowness after some rapid initial progress. A frther disadvantage of the steepest descent method is that it is not scale invariant. The indicated direction of movement changes if the scales of the variables are changed, nless all are changed by the same factor. The steepest descent method is, on the whole, slightly less favored than the linearization method (described later) bt will work satisfactorily for many nonlinear problems, especially if modifications are made to the basic techniqe. XI The method developed by D. W. Marqardt ( An algorithm for least sqares estimation of nonlinear parameters, Jornal of the Society for Indstrial and Applied Mathematics,, 963, 43-44) appears to enlarge considerably the nmber of practical problems that can be tackled by nonlinear estimation. This method is a compromise between the nd and 3 rd method mentioned here, and occpies a middle grond between these two methods.

6 6 5. In addition to these approach, there are several crrently employed methods available for obtaining the parameter estimates by Statistical Compter Packages However, we will mention only the nd one in or present docmentation. Linearization Method: The Linearization XII (followed by Gass-Newton iteration based on Taylor Series XIII ) method ses the reslt of linear least sqares in a sccession of stages. Sppose the postlated model is of form: Y = f( ξ, θ) + ε Let θ, θ,..., θ p be initial vales XIV for the parameters θ, θ,..., θ p. These initial vales XV may be gesses or preliminary estimates based on whatever information is available. These initial vales will, hopeflly, be improved pon in the sccessive iterations to be described below. If we carry ot a Taylor series expansion of f( ξ, θ ) abot the point θ, where θ = ( θ, θ,..., θ p) and crtail the expansion at the first derivatives, we can say that, approximately, when θ is close to θ, XII Linearization is accomplished by a Taylor series expansion of f( ξ, θ ) abot the point (or initial starting vector vales) with only the linear terms retained (ignoring all the higher order terms other than the first order term). XIII Any fnction (continos and has a continos p-th derivative) can be written as f(x)= 3 p ( x a) ( x a) ( x a) ( p) f( a) + ( x a) f ( a) + f ( a) + f ( a) f ( a) + Rp+! 3! p! expanded by Taylor series where f ( a) = df( x)/ dx arond x = a, and R p + is the remainder term which is reasonably small if p is sfficiently large. Also, the fnction f(x) mst converge at the point a. If we omit the nd and higher order terms, then the eqation is linear in a and we can apply a natral mechanism to estimate the coefficients in the linearized version of f(x). XIV We also call them Starting Vales. Sometimes, these are pre gesses, sometimes based on prior experience or prior empirical work or obtained by jst fitting a linear regression model even thogh it may not be appropriate. All available prior information shold be sed to make these starting vales as reliable as they possibly can be. However, the choice of initial starting vales is very important in this method becase a poor choice may reslt in slow convergence. Good starting vales will generally reslt in faster convergence, and if mltiple maxima exist, will lead to a soltion that is the global minimm rather than a local minimm. If there are several local minima in addition to an absolte minimm, poor starting vales may reslt in convergence to an nwanted stationary point of the sm of sqares srface. Also, it is often desirable to try other set of starting vales after a soltion has been obtained to make sre that the same soltion will be fond. Readers may conslt Myers (986) Classical and Modern Regression with Applications, page 36-7, Draper, Smith (998) Applied Regression Analysis, Third ed., Page 57-8; Montgomery, Peck (99) Introdction to Linear Regression Analysis, nd ed., Page 43- for leaning how they sggest to get sch initial vales. Grid points in the parameter space is often sefl. Some Statistical packages for non-linear reqires that the sers specify the starting vales for the regression parameter, and others do a grid search to obtain starting vales. XV For example, they may be vales sggested by the information gained in fitting a similar eqation in a different laboratory or sggested as abot right by the experimenter based on his/her experience and knowledge.)

7 7 f( ξ, θ) f( ξ, θ) ( ξ, θ ) ( θ θ ) p = f + i i i= θi θ= θ f( ξ, θ) f( ξ, θ) + ( θ θ) θ θ= θ f( ξ, θ ) = f( ξ, θ) + ( θ θ) θ θ= θ f( ξ, θ) ( θp θp) θp θ = θ This above eqation represents what is essentially a linearization of the non-linear form of the fnction f( ξ, θ ) - and the eqation may be viewed as a linear approximation in a neighborhood of the starting vales.. Now, we set f = f( ξ, θ ), β = ( θ θ ), i i i f( ξ, θ) Zi = and get the new form θ i θ= θ p = + β + β β p p + ε = + βi i + ε i= Y f Z Z Z f Z p βi i ε i= Y f = Z + which is very similar to the linear regression model and the procedre essentially involves a linear regression analysis on the above model. If we write Z βˆ Z Z... Z p Z Z... Zp = Z i =, for (i,) = [(,,,p), (,,,n)] Z n Zn... Zpn ( n p) b βˆ Y f ˆ b β = b = Y f = and y = Y f =... b ˆ p β Y p n f n n p then the reformed model is y = Z βˆ + ε = Zb + ε

8 8 and the estimate XVI of β, b is given by b = ( ZZ ) Z ( Y f ) = a p vector Ths, the vector b will be minimizing the Sm of Sqares n p SS( θ) = Y f( ξ, θ) βi Zi = i= with respect to the β i, where β ˆ i = θi θi. Note that the above eqation is the approximating linear expansion of or model. At this point, we can examine whether the revised regression coefficients represent adjstments in the proper direction by checking come criterion measre. If the method is working effectively, then SSE () shold be smaller than SSE () since the revised regression coefficients shold be better estimates XVII. Of corse, the analyst cannot work with the approximated linear model directly in a one step operation (need to be iterated). Let s write b ˆ i = θi θi, then the θ i; i =,,,p can be thoght as the revised best estimate of θ. Now, we can replace the vales θ i, the revised estimates, in the same roles as were played above by the vales θ i and go exactly the same procedre described above, by replacing all zero sbscripts by ones. This will lead to another set of revised estimates θ i, and so on. In vector form, we can write abot the things of j-th iteration as θˆ = θˆ + b where, j+ j j ˆ ˆ j θ j θ j ( ZZ j j) Z + j( Y f ) = + Z = Z, j j i n p f j j f j f =, and θ... j f n j θj θ j =... θ pj XVI Any analyst who embarks on a non-linear estimation exercise shold be made aware of what is known (or nknown) abot the properties of the estimators. In the non-linear case, we cannot make any general statements abot the properties of the estimators except for large sample since only approximate procedres for statistical tests and confidence intervals are available. The estimators are not nbiased in general, bt they are nbiased and minimm variance estimators in the limit, that is, as the sample grows large. Therefore, non-linear estimates do not possess optimal properties in finite samples ths, the reslts fond from small samples mst be interpreted careflly. There are asymptotic variance-covariance reslts that we can se to obtain approximate confidence intervals and to constrct t-statistics on the parameters. For the asymptotic covariance martix formla, readers may conslt Montgomery, Peck (99) Introdction to Linear Regression Analysis, nd ed., Page XVII For details, readers may conslt Neter, Wasserman, Ktner (983) Applied Linear Regression Models, page 474-9, and Myers (986) Classical and Modern Regression with Applications, page 35-.

9 9 In general, the exact iteration process is as follows:. Estimate β s in the model by linear least sqares. We shall denote these first iteration estimates by b s.. Compte θˆ ˆ i = θi + bi (i =,,,p) bt this is not or final estimates. 3. The θ ˆi vale is treated as the initial vale in or first approximated linear model. 4. We retrn to the first step and again compte b s (at each iteration, new b s represent increments that are added to the estimates from the previos iteration according to nd step) and eventally find θ ˆi. 5. We contine the process ntil convergence XVIII is reached. Stopping Rle: This iteration process is contined ntil the soltion converges, that is, ntil in sccessive iterations j,(j+), { θi( j+ ) θij} < δ, i = (,,,p) θ ij where δ is some predetermined small amont (e.g.,. or. 6 ). At each stage of the iteration procedre, S( θ i ) can be evalated to see if a redction in its vale has actally been achieved. Limitations: The ser may experience some difficlties with the Linearization procedre, sch as. It may converge very slowly, that is, a very large nmber of iteration may be reqired before the soltion stabilizes even thogh the sm of sqares S( θ i ) may decrease consistently as j increases. This sort of behavior is not common bt occrs.. It may oscillate widely, continally reversing direction, and often increasing, as well as decreasing the sm of sqares. Nevertheless, the soltion may stabilize eventally. 3. It may not converge at all, and even diverge (move at the wrong direction), so that the sm of sqares increases iteration after iteration withot bond. For these drawbacks, several modifications of this process has been sggested XIX to improve its performance. XVIII Convergence implies that after, say r iterations, the residal sm of sqares and the parameter estimates are no longer (significantly) changing. XIX Readers may conslt Montgomery, Peck (99) Introdction to Linear Regression Analysis, nd ed., Page 48; Draper, Smith (998) Applied Regression Analysis, Third ed., Page 5- for these. Readers may like to conslt Bates, D. M. and Watts, D. G. (988) Nonlinear Regression Analysis and Its Applications, New York: John Wiley and Sons (specially ch-,3; page-3-33) for more general discssion (however, not best for the starters / beginners of regression) abot this Non-linear Regression topic which is very mch appreciated by so many other athors of this topic. In fact, the poplar S software s (origin of S-pls and R) methods are based on those described in Bates and Watts. Also, Seber, G. A. F. and Wild, C. J. (989) Nonlinear Regression, New York: John Wiley and Sons is another encyclopaedic reference.

10 Nonlinear Regression sing Statistical Software: SAS The SAS System offers a powerfl procedre to fit nonlinear regression models, PROC NLIN XX. The NLIN procedre implements iterative methods that attempt to find least-sqares estimates for nonlinear models. This procedre performs nivariate nonlinear regression sing the least sqares method. It was improved dramatically in release 6. of The SAS System with the addition of a differentiator. Prior to release 6., if we wanted to fit a nonlinear model we had to spply the model specification as well as the formlas for the derivatives of the model. The latter was a real hazzle, especially if the model is complicated. A method to circmvent the specification of derivatives was to choose a fitting algorithm that approximates the derivatives by differences. This algorithm, known as DUD (Does not Use Derivatives) was hence very poplar. However, the algorithm is also known to be qite poor in compting good estimates. A method sing derivatives is to be preferred. With release 6. SAS will calclate derivatives for s if we wish. The ser still has the option to spply derivatives. It is, however, recommended to let The SAS System calclate them for s. The minimm specification to fit a nonlinear regression with PROC NLIN demands that the ser specify the model and the parameters in the model. All terms in the model not defined as parameters are looked for in the data set that PROC NLIN processes. Since nonlinear models are often difficlt to estimate, PROC NLIN may not always find the globally optimal least-sqares estimates. Using a Starting Vale Grid: A grid search is also available to select starting vales for the parameters. If we are not sre abot the starting vales, we can se a grid by offering SAS more than one starting vale. It will calclate the initial residal sm of sqares for all combinations of starting vales and start the iterations with the best set. Choosing the Fitting Algorithm: If or data and model are well behaved, it shold not make a difference how we fit the nonlinear model to data. Unfortnately, this can not be said for all nonlinear regression models. We may have to choose careflly, which algorithm to se. In PROC NLIN different fitting algorithms are invoked with the METHOD= option of the PROC NLIN statement. Here are a few gidelines: - If possible, choose a method that ses derivatives, avoid DUD. Unfortnately, if we do not specify derivatives and a METHOD= option, SAS will defalt to the DUD method. - If the parameters are highly correlated, choose the Levenberg-Marqardt method (keyword METHOD=MARQUARDT) - Among the derivative dependent methods, prefer the Newton-Raphson (METHOD=NEWTON) over the Gass (METHOD=GAUSS) method. Calclating predicted vales and their confidence intervals: Predicted vales are not displayed on screen in PROC NLIN. However, we can reqest to save them to a data set for later se. Along with the predicted vales, we can calclate confidence bonds for the mean predictions, prediction intervals for an individal predictions and so forth. XX I am indebted to Professor Oliver Schabenberger (998) of Virginia Tech for prodcing this SAS part of this docmentation. For more abot this SAS procedre, readers may conslt via internet at

11 Nonlinear Regression sing Statistical Software: R R ses nls() fnction to perform estimation in non-linear Regression models. Also, the same fnction exists in S-pls 4 and above. The argments XXI to nls are the following. formla A non-linear model formla. The form is response ~ mean, where the right-hand side can have either of two forms. The standard form is an ordinary algebraic expression containing both parameters and determin-ing variables. Note that the operators now have their sal arithmetical meaning. data An optional data frame for the variables (and sometimes parameters). start A list or nmeric vector specifying the starting vales for the parameters in the model. The names of the components of start are also sed to specify which of the variables occrring on the right-hand side of the model formla are parameters. All other variables are then assmed to be determining vari-ables. control An optional argment allowing some featres of the defalt iterative procedre to be changed. algorithm An optional character string argment allowing a particlar fitting algorithm to be specified. The defalt procedre is simply "defalt". trace An argment allowing tracing information from the iterative procedre to be printed. By defalt none is printed. XXI I am indebted to W. N. Venables and B. D. Ripley () Modern Applied Statistics with S, 4 th ed., for prodcing this part of the docmentation.

12 Illstrative Example: Let the data XXII file is in c:\ draper.txt, which is as follows: t Y Let that, we have to estimate the parameter θ in the non-linear model from the above observations. t Y = e θ + ε Now for starting analysis, we need a initial vale. Bt we have no prior information abot what vales θ might take. There are a few sggestions for sch sitations: one of which is to make a grid search sing Statistical Software. Let s demonstrate how we can do this. Using Statistical Software: SAS: Making a Grid Search We write the following program in SAS (I sed SAS 6.) to get an estimate (note that we told SAS to do the search in Linearization techniqe by setting METHOD=NEWTON): DATA DS; INPUT T Y; DATALINES; RUN; PROC NLIN DATA=DS METHOD=NEWTON; PARAMETERS G=- TO BY.; MODEL Y = EXP(-(G*T)); OUTPUT OUT=NLINOUT PREDICTED=PRED L95M=L95MEAN U95M=U95MEAN L95=L95IND U95=U95IND; RUN; PROC PRINT DATA=NLINOUT; RUN; There will be an enormos search to find this estimate since we assmed or potential parameter (in the program, we wrote θ = G) space to be to and told SAS to look within each. interval (so that we can get minimm residal SS or minima). However, after some iteration (when a convergent vale will be fond) SAS will bring the following otpt (edited to shorten the length bt the key parts of the otpt is nedited): XXII We are sing this simple and small data set jst for easy illstration prpose. This data set is taken from Draper, Smith (998) Applied Regression Analysis, Third ed., Page 553, Exercise 4.A.

13 3 The SAS System :5 Snday, (Detailed Reslts on Non-Linear Least Sqares Grid Search is omitted from here.) Non-Linear Least Sqares Iterative Phase Dependent Variable Y Method: Newton Iter G Sm of Sqares NOTE: Convergence criterion met. The SAS System :5 Snday, April 6, Non-Linear Least Sqares Smmary Statistics Dependent Variable Y Sorce DF Sm of Sqares Mean Sqare Regression Residal Uncorrected Total (Corrected Total).894 Parameter Estimate Asymptotic Asymptotic 95 % Std. Error Confidence Interval Lower Upper G Asymptotic Correlation Matrix Corr G ƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒƒ G The SAS System :5 Snday, April 6, OBS T Y PRED L95MEAN U95MEAN L95IND U95IND Therefore, from SAS, we get estimate of θ = So we may sppose that the actal vale may be arond zero for now and check it in other ways. Calclation By Hand: Non-linear Regression: We will now go throgh some iterations jst to see how the process works. Readers may think it as a slow motion of what happens to any Statistical Software when we rn a non-linear regression. Iteration : We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated initial vale θ = (as or grid search indicates note that, since we have to estimate only one vale, this is jst a scaler, bt for mlti-parameter case, we wold have a vertor).

14 4 t f = f( ξθ, ) = e = since t is a 3 vector. Y f.8 -. Now, y = Y f = Y f = = Y3 f f( ξθ, ) t Z = ( t) e 4 = = θ θ= 6 ZZ = = 73 ( ZZ ) = (73) - = ( f ) Z Y = (-.) (-)+(-.55) (-4)+(-.96) (-6) = 7.76 b = ( ZZ ) Z ( Y f ) = = θ ˆ = θ ˆ + b = = Iteration : We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ = ˆ t f f( ξθ, ) e θ = = =.7788 since t is a 3 vector Y f Now, y = Y f = Y f = Y3 f f( ξθ, ) ˆ t Z ( t) e θ = = = θ θ= θˆ ZZ = 4.3 ( ZZ ) = (4.3) - ( f ) Z Y = b = ( ZZ ) Z ( Y f ) = θ ˆ = θ ˆ + b =.3396 Iteration : We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ = = (, ) = = ˆ t f f ξθ e θ since t is a 3 vector.

15 5 Y f Now, y = Y f = Y f = Y3 f f( ξθ, ) ˆ t Z ( t) e θ = θ = = θ= θˆ ZZ = ( ZZ ) = ( ) - ( f ) Z Y = b = ( ZZ ) Z ( Y f ) =.547 θ ˆ = θ ˆ + b = Iteration 3: We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ 3= ˆ t f f( ξθ, 3) e θ = = =.4758 since t is a 3 vector Y f Now, y3 = Y f = Y f -.58 = 3 Y3 f f( ξθ, ) ˆ t Z3 ( t) e θ = = = θ θ= θˆ ZZ 3 3 = ( ZZ ) = ( ) ( f 3 ) 3 Z Y = b3 = ( ZZ 3 3) Z 3( Y f ) =.5969 θ ˆ = θ ˆ + b = Iteration 4: We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ 4= ˆ t f f( ξθ, 4) e θ = = = since t is a 3 vector Y f Now, y4 = Y f = Y f.5439 = 4 Y3 f

16 f( ξθ, ) ˆ t Z4 ( t) e θ = θ = = θ= θˆ ZZ 4 4 = ( ZZ ) = (4.9836) ( f 4 ) 4 Z Y = b4 = ( ZZ 4 4) Z 4( Y f ) =.7855 θ ˆ = θ ˆ + b = Iteration 5: We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ 5= ˆ t f f( ξθ, 5) e θ = = = since t is a 3 vector Y f Now, y5 = Y f = Y f.688 = 5 Y3 f f( ξθ, ) ˆ t Z5 ( t) e θ = = = θ θ= θˆ ZZ 5 5 = ( ZZ ) = ( ) ( f 5 ) 5 Z Y = b5 = ( ZZ 5 5) Z 5( Y f ) = θ ˆ = θ ˆ + b = Iteration 6: We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ 6= ˆ t f f( ξθ, 6) e θ = = = since t is a 3 vector Y f Now, y6 = Y f = Y f = 6 Y3 f 3.46

17 f( ξθ, ) ˆ t Z6 ( t) e θ = θ = = θ= θˆ ZZ 6 6 = ( ZZ ) = (4.8964) ( f 6 ) 6 Z Y = b6 = ( ZZ 6 6) Z 6( Y f ) = θ ˆ = θ ˆ + b = Iteration 7: We have the fnction Y = f( ξθ, ) + ε = e θt + ε in or hand with approximated vale θ ˆ 7= ˆ t f f( ξθ, 7) e θ = = = since t is a 3 vector Y f Now, y7 = Y f = Y f = 7 Y3 f f( ξθ, ) ˆ t Z7 ( t) e θ = = = θ θ= θˆ ZZ 7 7 = ( ZZ ) = (4.8959) ( f 7 ) 7 Z Y = b7 = ( ZZ 7 7) Z 7( Y f ) = θ ˆ = θ ˆ + b = Since θ ˆ ˆ 8 = θ 7 we terminate the process and declare θ ˆ 8 = as or desired estimate.

18 8 Using Statistical Software: R: Non-linear Regression Now, sing Statistical Software like R (I sed R.7.) we can do the whole procedre jst in a blink (and of corse, withot pain!) by writing the following commands in the R console: > options(prompt=" R> " ) R> draper<-read.table("c:\\draper.txt",header=t) R> draper.data<-data.frame(draper) R> attach(draper.data) R> nls(y~exp(-(g*t)), data=draper.data, start = c(g = ), trace = T).64 : : : : : : : : Nonlinear regression model model: Y ~ exp(-(g * t)) data: draper.data g residal sm-of-sqares: R> coef(nls(y~exp(-(g*t)), data=draper.data, start = c(g = ), trace = T)) (---) g Note that, the estimated vale of θ = approximately in all or (slow motion hand calclation and R reslts within a blink) cases which matches or SAS reslt too. Ths, we declare as or estimate of the parameter θ based on the given data.

19 9 By now, readers may think that R is not capable of doing a grid search to find the initial vale. Bt this is not tre: let s see a few R commands XXIII to clear it p (that is, grid search is possible in R bt we will go throgh another way): R> attach(draper.data) R> fit <- lm(log(y)~t-, draper) R> fit Call: lm(formla = log(y) ~ t -, data = draper) Coefficients: t -. XXIV R> fit <- nls(y~exp(-theta*t), data=draper, start=c(theta=-.)) R> fit Nonlinear regression model model: Y ~ exp(-theta * t) data: draper THETA residal sm-of-sqares: R> smmary(fit) Formla: Y ~ exp(-theta * t) Parameters: Estimate Std. Error t vale Pr(> t ) THETA *** --- Signif. codes: `***'. `**'. `*'.5 `.'. ` ' Residal standard error:.8 on degrees of freedom Alternatively, we cold compte the sm of sqares for all vales of THETA = seq(,., ) in a loop, then find the minimm by eye. Also, for a problem that has several parameters, "expand.grid" will prodce a grid, and we can compte the vale of fnction and the sm of sqares of residals at every point in the grid in a single loop, etc. XXIII Thanks to Spencer Graves and Doglas Bates for the kind direction abot this grid search sing R. XXIV We wold expect that (-.) shold provide a reasonable starting vale for nls.

20 Using Statistical Software: NLREG nonlinear regression program NLREG nonlinear regression program XXV has a "Sweep" statement that can be sed to try mltiple starting points for a parameter or do a grid search if there are mltiple parameters. However, it is rarely needed. Here is an NLREG program to solve or example problem. The Sweep statement tries starting vales for Theta from to in steps of.5. It trns ot that the program doesn't need the Sweep statement becase it qickly achieves convergence with the defalt starting vale of.. The optimm vale of Theta to fit yor fnction is ; it was fond in iterations. Variables t, Y; Parameter Theta; Fnction Y = EXP(-(THETA * t)); Sweep Theta =,,.5; Plot; Data; And the ot pt is as follows : NLREG version 6. Copyright (c) 99-4 Phillip H. Sherrod XXVI. Nmber of observations = 3 Maximm allowed nmber of iterations = 5 Convergence tolerance factor =.E- Stopped de to: Both parameter and relative fnction convergence. Nmber of iterations performed = 8 Final sm of sqared deviations = E-4 Final sm of deviations = E-3 Standard error of estimate =.848 Average deviation =.855 Maximm deviation for any observation =.599 Proportion of variance explained (R^) =.999 (99.9%) Adjsted coefficient of mltiple determination (Ra^) =.999 (99.9%) Drbin-Watson test for atocorrelation =.8 Analysis completed 5-Apr-4 5:5. Rntime =.44 seconds Descriptive Statistics for Variables ---- Variable Minimm vale Maximm vale Mean vale Standard dev t Y Calclated Parameter Vales ---- Parameter Initial gess Final estimate Standard error t Prob(t) Theta XXV Demonstration version (free for a month) is available at XXVI I am indebted to Phil Sherrod for writing the above example program for me personally.