Nonlinear regression

Size: px

Start display at page:

Download "Nonlinear regression"

Beatrice Armstrong
5 years ago
Views:

1 7//6 Nonlinear regression What is nonlinear model? In general nonlinearit in regression model can be seen as:. Nonlinearit of dependent variable ( ) for eample is binar (logit model), nominal (multivariate nominal model), counts (Poisson model), interval-valued (smbolic data regression), etc.. Nonlinearit within independent variables ( s) until now we have considered a linear relations (dependence, connection) between independent variables. In this case we can have: a) functions (relations) that can be transformed to linear functions, b) Functions (relations) that can not be transformed to linear form

2 7//6 How to detect nonlinearit?. Theor in man sciences we have theories about nonlinear relations within som phenomenon. For eample in economics Laffer curve shows a nonlinear relations between taation and the hpothetical resulting levels of government revenue. Scatterplot when looking at the plot ou can see that data points are not linear or even not nearl linear 3. Seasonalit in data i.e. in agriculture, building industr we often have seasonalit within the data 4. Estimated model does not fit the data well or does not fit it at all; the estimated s are not significant this might suggest nonlinearit 5. Can often do incremental F tests or Wald tests Linear or notlinear? That is the question

3 7//6 Linear or notlinear? That is the question Linear or notlinear? That is the question 3

4 7//6 Linear or notlinear? That is the question Linear or notlinear? That is the question 4

5 7//6 Transformation of nonlinear models to linear models Eponential function: we have to use logarithms: ln ln ( ) ln ln ln we have to substitute: ln, ln, ln and we get linear model: ln ln ln Transformation of nonlinear models to linear models Logarithms: ln we have to substitute: ln and we get linear function: 5

6 7//6 Transformation of nonlinear models to linear models Power function: we have to use logarithms: ln ln ln ( ) ln ln ln we have to substitute: ln, ln, ln and we get linear function: Transformation of nonlinear models to linear models Logistic function: e where: >, > we use substitutions:,,, e 6

7 7//6 7 Transformation of nonlinear models to linear models Hperbolic function: we use substitutions for equation (): and we get: we use substitutions for equation (): to get: () (),,, Nonlinear models in R To estimate nonlinear models (that can be transformed to linear models) we use lm function, but we have to define formula. Some eamples will be presented: (8) (7) (6) (5) (4) log (3) ln ln () () t t t we have to use logarithms for equations 6 and 7

8 7//6 Nonlinear models in R How to appl these functions in formula: Function no. ~ ~- or ~ 3 ~log()log() 4 log()~ Formula 5 ~I(/)sqrt(3)I(4^) 6 log()~ 7 log()~log()log() 8 z.~vv where: z. t, v t-, v t- The nonlinear regression model is the generalization of the linear regression model in which the conditional mean of the response variable is not a linear function of parameters. For eample data about decennial U. S. Census population for the United States (in millions), from 79 through 8

9 7//6 A common simple model for population growth is the logistic growth model: m (,θ) θ ε ε ep [ ( θ θ ) ] where: response, predictor, θ (θ, θ, θ 3 ) parameters Changing θ (theta s) stretches or shrinks the aes, and changes the rate at which the curve varies from its lower value at to its maimum value 3 Let s assume (θ, θ,θ 3 ) 9

10 7//6 In general nonlinear regression model is: ( ) ε m(, ) ε E θ This model posits that the mean E( ) depends on through the kernel mean function m(, θ), where the predictor has one or more components and the parameter vector θ also has one or more components In the logistic growth model consists of the single predictor ear and the parameter vector θ (θ, θ, θ 3 ) has three components The model further assumes that the errors ε are independent with variance σ /w, where the w are known nonnegative weights, and σ is a generall unknown variance to be estimated, in most applications w for all observations. The nls function can be used to estimate θ as the values that minimize the residua sum of squares: S From now on we will use sum of squares θ ( θ) w[ m( θ, ) ] for the minimizer of the residual

11 7//6 Unlike the linear least-squares problem, there is usuall no formula that provides the minimizer of the equation S( θ) w[ m( Rather θ, ) ] than an iterative procedure is used, which in broad outline is as follows: ) The user supplies an initial guess, sa t of starting values for the parameters. Whether or not the algorithm can successfull find a minimizer will depend on getting starting values that are reasonabl close to the solution. We discuss how this might be done for the logistic growth function below. For some special mean functions, including logistic growth, R has self-starting functions that can avoid this step. ) At iteration j, the current guess tj is obtained b updating t j-. If S(t j ) is smaller than S(t j - ) b at least a predetermined amount, then the counter j is increased b and this step is repeated. If no improvement is possible then t j- is taken as the estimator This simple algorithm hides at least three important considerations. First, we want a method that will guarantee that at each step we either get a smaller value of S or at least S will not increase. There are man nonlinear least-squares algorithms; see, for eample, Bates and Watts (988). Man algorithms make use of the derivatives of the mean function with respect to the parameters.

12 7//6 The default algorithm in nls uses a form of Gauss-Newton iteration that emplos derivatives approimated numericall unless we provide functions to compute the derivatives. Second, the sum of squares function S ma be a perverse function with multiple minima. As a consequence, the purported least-squares estimates could be a local rather than global minimizer of S. Third, as given the algorithm can go on forever if improvements to S are small at each step. As a practical matter, therefore, there is an iteration limit that gives the maimum number of iterations permitted, and a tolerance that denes the minimum improvement that will be considered to be greater than. Function nls uses following arguments: formula The formula argument is used to tell nls about the mean function. start The argument start is a list that tells nls which of the named quantities on the right side of the formula are parameters, and thus implicitl which are predictors. It also provides starting values for the parameter estimates. algorithm "default" The "default" algorithm used in nls is a Gauss-Newton algorithm. Other possible values are "plinear" for the Golub-Perera algorithm for partiall linear models and "port" for a algorithm that should be selected if there are constraints on the parameters

13 7//6 lower -Inf, upper Inf One of the characteristics of nonlinear models is that the parameters of the model might be constrained to lie in a certain region. In the logistic populationgrowth model, for eample, we must have θ 3 >, as population size is increasing, and we must also have θ >. trace FALSE If TRUE, print the value of the residual sum of squares and the parameter estimates at each iteration. Unlike in linear least squares, most nonlinear least-squares algorithms require specication of starting values for the parameters A starting value of the asmtote θ is some value larger than an value in the data, and so value around t 4 is a resonable start). The estimated population in before official Census count was released was 37 milion. 3

7//6 Unlike in linear least squares, most nonlinear least-squares algorithms require specication of starting values for the parameters Linear model can be used to get some information (guess) what

14 7//6 Unlike in linear least squares, most nonlinear least-squares algorithms require specication of starting values for the parameters Linear model can be used to get some information (guess) what initial values should be m<-lm(logit(population/4) ~ ear, USPop) print(m) Coefficients: (Intercept) ear pop.mod <- nls(population ~ theta/( ep(-(theta theta3*ear))), startlist(theta 4, theta -49, theta3.5), datauspop, tracetrue) B setting tracetrue, we can see that S evaluated at the starting values is 36. The first iteration reduces this to 558.5, the net iteration to 458, and the remaining iterations result in onl ver small changes. We get convergence in 6 iterations. 4

15 7//6 summar(pop.mod) The column marked Estimates displas the least squares estimates. The estimated upper bound for the U. S. population is 44.8, or about 44 million. The column marked Std. Error displas the estimated standard errors of these estimates. The ver large standard error for the asmptote reflects the uncertaint in the estimated asmptote when all the observed data is much smaller than the asmptote. The standard error of this estimate can be computed with the deltamethod function in the car package: se<-deltamethod(pop.mod, "-theta/theta3") print(se) The estimated ear in which the population is half the asmptote is: θ θ and 976,6 so the standard error is about 7.6 ears 3 5

16 7//6 The column t value in the summar output shows the ratio of each parameter estimate to its standard error. In sufficientl large samples, this ratio will generall have a normal distribution, but interpreting sufficientl large" is difficult with nonlinear models. Even if the errors ε are normall distributed, the estimates ma be far from normall distributed in small samples. The p-values shown are based on asmptotic normalit. The residual standard deviation is the estimate of σ : σ S θ ( ) ( n k) U. S. population, with logistic growth t etrapolated to. The circles represent observed Census population counts, while represents the estimated population. The broken horizontal lines are drawn at the asmptotes and midwa between the asmptotes; the broken vertical line is drawn at the ear corresponding to the mid-wa point. 6

17 7//6 This figure suggests there are sstematic features that are missed, reflecting differences in growth rates, perhaps due to factors such as changes in immigration Bates and Watts (988, Sec. 3.) describe man techniques for finding starting values for fitting nonlinear models. For the logistic growth model described we stud, for eample, finding starting values amounts to: () guessing the parameter θ as a value larger than an observed in the data; and () substituting this value into the mean function, rearranging terms, and then getting other starting values b OLS simple linear regression. 7

18 7//6 The self-starting logistic growth model in R is based on a different, but equivalent, parametrization of the logistic function. We will start again with the logistic growth model, with mean function: m (, θ) Fitting a nonlinear model with the self-starting logistic growth function in R is quite eas: pop.ss <- nls(population ~ SSlogis(ear, phi, phi, phi3), datauspop) summar(pop.ss) ep θ [ ( θ θ ) ] 3 8

19 7//6 The right side of the formula is now the name of a function that has the responsibilit for computing the mean function and for finding starting values. The estimate of the asmptote parameter φ and its standard error are identical to the estimate and standard error for θ in the θ-parametrization. 9

Exponential, Logistic, and Logarithmic Functions

Exponential, Logistic, and Logarithmic Functions CHAPTER 3 Eponential, Logistic, and Logarithmic Functions 3.1 Eponential and Logistic Functions 3.2 Eponential and Logistic Modeling 3.3 Logarithmic Functions and Their Graphs 3.4 Properties of Logarithmic