Nonlinear Regression. Chapter 2 of Bates and Watts. Dave Campbell 2009

Transcription

1 Nonlinear Regression Chapter 2 of Bates and Watts Dave Campbell 2009

2 So far we ve considered linear models Here the expectation surface is a plane spanning a subspace of the observation space. Our expectation surface has a flat shape. Our model has a linear shape When things are flat projections are easy to do and understand.

3 nonlinear least squares Gauss-Newton Geometry Matlab Functions

4 The N observations are modeled by Y n = f (X n,θ) + Z n Where f (X n,θ) might be: Y 0 e α X + β / α θ 1 X θ 2 + X e β 0 β 1 X

5 We will define the expectation function η(θ) = f (X n,θ) And the observation process Y n = η(θ) + Z n And again assume E(Z) = 0 var(z) = E(Z 'Z) = σ 2 I

6 Lipoprotein problem from Bates and Watts % of the original tracer observations by time (days)

7 Consider the 1 compartment model: η(θ) = f (X n,θ) η(θ) = θ 1 e θ 2 X Although we could transform the data into the model: log(y n ) = log(θ 1 ) θ 2 X n + error n We will ignore this linearization and use the nonlinear model as a simple example

8 The response surface is defined by changing values of the single parameter in η(θ) = 100e θ 2 X Our expectation surface is then a one dimensional manifold

9 Lipoprotein observation and response surface

10 Again we wish to minimize the residual vector But there may be more than 1 location where the angle between the residual vector and the tangent to the expectation surface are orthogonal

11 The problem is even worse in the 2 dimensional model where initial conditions are unknown η(θ) = f (X n,θ) η(θ) = θ 1 e θ 2 X

12 Lipoprotein observation and 2 parameter response surface lines show the response surface where theta 1 is fixed but altering theta2 dot connecting spreading lines making the kinks show changes in theta1 with fixed theta 2

13 steps: 1. Find the point on the expectation surface closest to ˆη = ˆ Y Y ˆθ 2. Find corresponding to this point

14 Gauss-Newton Method Use a linear approximation to the expectation surface to iteratively improve an initial guess for θ We will use a linear Taylor approximation to the expectation surface, and then use linear regression methods. We will need to keep updating the Taylor expansion and keep updating our estimate. θ (0)

15 The expectation function η(θ) = θ 1 e θ 2 X For a single observation the Taylor expansion for the p dimensional parameter vector: η(θ) η(θ (0) ) + p k =1 η(θ) θ θ (0) θ k k θ =θ (0 ) Including all observations we get η(θ) η(θ (0) ) + V (0) (θ θ (0) ) where V (0) = η(x 1,θ) θ 1... η(x 1,θ) θ p η(x n,θ) η(x... n,θ) θ 1 θ p θ =θ (0 )

16 The model η(θ) = θ 1 e θ 2 X V (0) = e θ 2 X θ 1 Xe θ 2 X θ =θ (0 )

17 The fit to the observations on the response surface and the Gauss Newton path

18 The first step jump really far away from the region where the linear Taylor approximation is valid. We can improve the Gauss Newton algorithm by enforcing the condition SSE(θ (i+1) ) < SSE(θ (i) ) This means that at each step we have to adjust the step size so that it doesn t take us to a worse location in the response surface

19 We adjust the algorithm by halving if the SSE condition is not met and trying again.

20

21 Geometry 1. Approximate the expectation surface η(θ (0) ) by an expectation plane at the current value ˆη(θ (0) ) 2. generate a residual vector z = y η(θ (0) ) 3. Project the residual onto the tangent plane to get new value of expectation surface ˆη(θ (1) ) 4. Map the move to ˆη(θ (1) ) through the linear approximation to get a step δ 5. move to the point on the actual expectation surface η(θ (0) + δ )

22 Gauss Newton Convergence

23 Gauss Newton Convergence In Matlab code I told it when to stop

24 Gauss Newton Convergence In Matlab code I told it when to stop Ideal convergence is based on the angle of the residual vector

25 Inference Inference is based on a linear approximation to the expectation surface then we project a disk onto the approximated expectation plane Then transform the values back to the parameter space to get an inference region

26 Other Intervals Marginal intervals: Use the same linear approximation and apply the linear marginal inference regions Inference bands for expected response x 0 ˆβ f (x 0, ˆθ) Replace in linear case with and replace the matrix X by ˆV and the derivative vector x 0 with the corresponding derivative matrix entry v 0

27 1. Why is it so wide near time 0? 2. Why is it so narrow near time 10? 3. Why is it so wide generally everywhere? 4. why is it wavy?

28 Using 3 or 12 observations

29 Using 3 or 12 observations

30 Matlab NLS function [beta,r,j,covb,mse] = nlinfit(x,y,fun,beta0) X is the same as our X y is a vector of observations fun is a function handle for the nonlinear function, the function takes inputs (beta,x) and gives Yhat as an output beta0 is the starting point for the iterative estimates

31 Matlab NLS function [beta,r,j,covb,mse] = nlinfit(x,y,fun,beta0) beta is NLS point estimate r is a residual vector J is the Jacobian of the function fun evaluated at each observation: this is our V COVB is the estimated covariance matrix for parameters mse is an estimate of the error variance term: our s 2

32 Marginal Confidence intervals for parameters CI_NLS = nlparci(beta,resids,'jacobian',j) beta is the parameter estimate output from nlinfit resids is the residual vector output from nlinfit J is the Jacobian output from nlinfit

33 Confidence interval for a new value or x [ypred,delta] = nlpredci(fun,x,beta,resid,'jacobian',j) beta is the parameter estimate output from nlinfit resids is the residual vector output from nlinfit J is the Jacobian output from nlinfit ypred ± delta is the CI

34 Or do it all using a GUI nlintool(x,y,@lipomodel_xlast,theta) Give it your data, the model and a starting point and it will give you point estimates for parameters confidence bounds for single points, the response function, and future observations bounds - simultaneous are for all points simultaneously, non-simultaneous are for individual points

35 The downside to the Matlab built-in functions: They do not produce joint interval estimate ellipses for parameters. We must use the QR decomposition for this.

36 Nonlinear Least Squares for ODEs Simplest method: Combine NLS and ODE solvers, Just use a solver built into fun [beta,r,j,covb,mse] = nlinfit(x,y,fun,beta0)

37 dv ( ) dt = γ V V3 / 3 + R dr ( ) dt = 1 γ βr + α V

38 dv ( ) dt = γ V V3 / 3 + R dr ( ) dt = 1 γ βr + α V

39 Inputs to fun must be ß and X but in our case X is time. X must a vector with the same length as data Y The output of fun must be the fit to the data, (the expectation surface at the current parameter point), we have to make it a vector. function [yfit] = NLS_FhN(pars,time) odefn odeopts = odeset('reltol',1e-13); [junk,path] = ode45(odefn,time(1:401),pars(4:5),odeopts,pars(1:3)); yfit=[path(:,1);path(:,2)];

40 To run this program: clear load(strcat('/volumes/iamdavecampbell/mcmc many times/',... '1000 random data sets/fhn MCMC data set_38.mat')) time=[time;time]; % stack time Y=[Ydata(:,1);Ydata(:,2)]; % stack data beta0=[.5,.5,1,-.5,.5]; fun=@nls_fhn [beta,r,j,covb,mse] = nlinfit(time,y,fun,beta0);

41 To run this program: [Ypred,delta] = nlpredci(fun,time,beta,r,'jacobian',j) n=length(ydata(:,1)); CI_Y=repmat(Ypred,1,3)+[-delta,zeros(2*n,1),delta]; plot(time(1:n),ci_y(1:n,:),'b',time(1:n),ydata(:,1),'.b',... time(1:n),ci_y(n+1:2*n,:),'k',time(1:n),ydata(:,2),'.k')

42 CI for the expectation function

43 To run this program: CI_NLS = nlparci(beta,r,'jacobian',j) [CI_NLS(:,1),beta',CI_NLS(:,2)] >> [CI_NLS(:,1),beta',CI_NLS(:,2)] ans =

44 nlinfit uses a numerical Jacobian to produce the Taylor approximation to the response surface. There are other functions that let you include the formula for the Jacobian: lsqnonlin