Sensitivity and Asymptotic Error Theory

Transcription

1 Sensitivity and Asymptotic Error Theory H.T. Banks and Marie Davidian MA-ST 810 Fall, 2009 North Carolina State University Raleigh, NC Center for Quantitative Sciences in Biomedicine North Carolina State University 1

2 Introduction Goal: To illustrate use and possible pitfalls in statistical inverse problem analysis of sensitivity equations and related standard errors. When data is sampled from a region near a steady state of a curve, the optimized parameters will not necessarily improve. The standard errors for estimated parameters may increase with more data points, contrary to intuition. To explain this, we will look at a specific example. 2

3 Statistical Review We assume that n scalar longitudinal observations are represented by the statistical model Y j f j (θ 0 ) + E j, j = 1, 2,..., n. Thus we seek to use data {y j } for the observation process {Y j } with the model to seek a value ˆθ n that minimizes n J n (θ) = f j (θ) y j 2. j=1 3

4 Since Y j is a random variable, we have that the estimator ˆθ n OLS is also a random variable with a distribution called the sampling distribution. For large n, the sampling distribution approximately satisfies ˆθ n OLS(Y ) N p (θ 0, σ 2 0[χ T (θ 0 )χ(θ 0 )] 1 ) := N p (θ 0, Σ 0 ), where χ(θ) = F θ (θ) is the n p sensitivity matrix with elements χ jk (θ) = f j(θ) θ k and F θ (θ) (f 1θ (θ),..., f nθ (θ)) T. F θ (θ) can be found using difference quotients, sensitivity equations, or direct analytic calculations (in some cases). 4

5 Typically several ways to compute the matrix F θ (which are actually the well known sensitivity functions widely used in applied mathematics and engineering see the discussions in [BDSS]). First, the elements of the matrix χ = (χ jk ) can always be estimated using the forward difference χ jk (θ) = f(t j, θ) θ k f(t j, θ + h k ) f(t j, θ), h k where h k is a p-vector with a nonzero entry in only the k th component. But, of course, the choice of h k can be problematic in practice. 5

6 Alternatively, if the f(t j, θ) correspond to longitudinal observations y(t j ) = Cx(t j ; θ) of solutions x R N to a parameterized N-vector differential equation system x = g(t, x(t), θ), then one can use the N p matrix sensitivity equations (see [BEG, BN] and the references therein) to obtain d dt ( ) x θ f(t j, θ) θ k = g x x θ + g θ = C x(t j, θ) θ k. In some cases the function f(t j, θ) may be sufficiently simple so as to allow one to derive analytical expressions for the components of F θ. But for more complex models often encountered, can use automatic differentiation software packages. (1) 6

7 Since θ 0, σ 0 are not known, we follow standard practice and use the approximation Σ 0 Σ(ˆθ n ) = ˆσ 2 [χ T (ˆθ n )χ(ˆθ n )] 1 (2) where ˆθ n is the parameter estimate obtained, and the approximation ˆσ 2 to σ0 2 is given by σ0 2 ˆσ 2 = 1 n f j (ˆθ n ) y j 2. (3) n p Standard errors to be used in confidence interval calculations are thus given by SE k (ˆθ n ) = Σ kk (ˆθ n ), k = 1, 2,..., p and consequently we define the confidence level parameters associated with the estimated parameters so that j=1 P {ˆθ n k t 1 α/2 SE k (ˆθ n ) < θ n k < ˆθ n k + t 1 α/2 SE k (ˆθ n )} = 1 α. 7

8 Under reasonable assumptions on smoothness and regularity (the smoothness requirements for model solutions are readily verified using continuous dependence results for differential equations in most examples; the regularity requirements include, among others, conditions on how the observations are taken as sample size increases, i.e., as n ), the standard nonlinear regression approximation theory ([DG, G, J], and Chapter 12 of [SeWi]) for asymptotic (as n ) distributions can be invoked. When one is taking longitudinal samples corresponding to solutions of a dynamical system, the n p sensitivity matrix depends explicitly on where in time the observations are taken when f(t j, θ) = Cx(t j, θ) as mentioned above. That is, the sensitivity matrix ( ) f(tj, θ) χ(θ) = F θ (θ) = θ k depends on the number n and the nature (for example, how 8

9 taken) of the sampling times {t j }. Moreover, it is the matrix [χ T χ] 1 in (2) and the parameter ˆσ 2 in (3) that ultimately determine the standard errors and confidence intervals. At first investigation of (3), it appears that an increased number n of samples might drive ˆσ 2 (and hence the SE) to zero as long as this is done in a way to maintain a bound on the residual sum of squares in (3). However, we observe that the condition number of the Fisher matrix F = χ T χ is also very important in these considerations and increasing the sampling could potentially adversely affect the inversion of χ T χ. In this regard, we note that among the important hypotheses in the asymptotic statistical theory (see p. 571 of [SeWi]) is the existence of a matrix function Ω(θ) such that 1 n χnt (θ)χ n (θ) Ω(θ) uniformly in θ as n, with Ω 0 = Ω(θ 0 ) a nonsingular matrix. 9

10 Moreover, there are conditions on the observation times {t j } such as existence of a finite measure μ on [0, T ] satisfying 1 n Σn j=1h(t j ) for each continuous function h. T 0 h(t)dμ(t) (4) It is these conditions that are rather easily violated in practice when one is dealing with data from differential equation systems, especially near an equilibrium or steady state (see the examples of [BEG]). 10

11 Example Consider the logistic growth population model dx (1 dt = rx x ) K where K is the carrying capacity and r is the intrinsic growth rate. The solution is given by x K x(t) = 1 + K ( ). K x 0 1 e rt K/2 11

12 Analytical Considerations We will examine the problem x = ax bx 2, where as k = a x 0 x(t) = = a/b 1 + ( a/b x 0 1)e at a b + ke at b. Here x(t) has an asymptote at a b = K. 12

13 We begin with an ordinary least squares problem for θ = (a, b, x 0 ) where X = x θ = with the covariance matrix x(t 1 ) a. x(t n ) a Σ = ˆσ 2 (X T X) 1 = 1 n 3 x(t 1 ) b. x(t n ) b x(t 1 ) x 0. x(t n ) x 0 n y j f(t j, ˆθ n ) 2 (X T X) 1 j=1 and the estimate of the standard error SE k = ˆσ 2 (X T X) 1 kk, k = 1, 2, 3. 13

14 Let R 0 be the region where t [0, τ 1 ], R 1 be the region corresponding to t [τ 1, τ 2 ], and R 2 where t [τ 2, ). a/b x 0 R 1 R 2 R 0 τ 1 τ 2 Figure 1: Partition of solution curve into distinct regions. 14

15 In order to divide the problem x = ax bx 2 into separate regions, we consider the solution x = a b + ke at and then examine the partial derivatives x a, x b, and x x 0. As t, which corresponds to region R 2, we see that x(t) a b. Also, x(t) We will denote x(0) = x 0 = a b+k as t 0, which corresponds to region R 0. a b+k. 15

16 We have the following partial derivatives x a x b = x x 0 = b + (atk b)e at (b + ke at ) 2, = a(1 e at ) (b + ke at ) 2, a 2 e at x 2 0 (b + ke at ) 2, which are n 1 vectors for t = t 1,..., t n. The matrix X is composed of these vectors and we obtain the following matrix, X T X = n j=1 ( x(tj ) a x(t j ) b ) 2 x(t j ) x(t j ) x(t j ) x(t j ) a b a x ( ) 0 2 x(tj ) x(t j ) x(t j ) b b x 0 x(t j ) a x(t j ) x(t j ) x 0 a x(t j ) x(t j ) x 0 b ( x(tj ) x 0 ) 2. 16

17 If we sample data from R 0, where t j < τ 1 for j = 1... n, we have x(t j ) a Also notice that 0, n j=1 x(t j ) b 0, x(t j ) x 0 x(t j ) x 0 = n 1 = n j=1 x(t j ) x 0 1. for t j < τ 1 in R 0. Hence, in this region, X T X n 17

18 Next, consider region R 2, where t j > τ 2 for j = 1... n. Notice that in R 2, and hence, X T X x(t j ) a n j=1 x(t j ) a x(t j ) b 1 b, x(t j ) b x(t j ) a x(t j ) a a b 2, x(t j ) x 0 0, x(t j ) b 0 x(t j ) x(t j ) x(t j ) a b b 0 = n b a 2 b 0 3 b a b 3 a 2 Notice that the second column of X T Xis a scalar multiple of the first column, which means that we should be able to estimate the ratio a b but not the individual parameters. 18

19 Implementation We create a simulated data set, y j, j = 1,..., n, using the analytical solution with a specific θ 0. We want to solve an inverse problem The cost function uses ode15s to approximate the solution and returns J(θ) = n y j f(t j ; θ) 2 j=1 where f(t; θ) = x(t; a, b, x 0 ) is the approximation to the solution. We use the MATLAB function fminsearch to optimize θ in order to obtain the minimized cost J(ˆθ n ). Here, ˆθ n represents the optimized value of θ over n data points. 19

20 80 70 Steep Moderate Flat Exact solutions on the whole region Solution Time Figure 2: Three sets of simulated data: a relatively flat curve with θ 0 = (0.5, 0.1, 0.1), a moderately sloped curve with θ 0 = (0.7, 0.04, 0.1), and a steep curve with θ 0 = (0.8, 0.01, 0.1). Define R 0 [0, 2], R 1 [2, 12], R 2 [12, 16]. 20

21 Results Regional Partitions Regions R 0, R 1, R 2 Standard Errors Regions R 0 R 1 and R 1 R 2 Simulated Noise 21

22 Region R 0 The flat curve with β 0 =( 0.6, 0.2, 0.5). 13 The moderate curve with β 0 =( 0.8, 0.1, 0.3). 13 The steep curve with β 0 =( 0.7, 0.2, 0.5) Model Model Model Time Time Time Figure 3: Simulated data plotted with the solution obtained from the estimated parameters in region R 0 for the a) flat curve with θ 0 = (0.6, 0.2, 0.5) b) moderate curve with θ 0 = (0.8, 0.1, 0.3) c) steep curve with θ 0 = (0.7, 0.2, 0.5). 22

23 Region R 1 The flat curve with β 0 =( 0.6, 0.2, 0.5). 61 The moderate curve with β 0 =( 0.8, 0.1, 0.3). 61 The steep curve with β 0 =( 0.7, 0.2, 0.5) Model Model Model Time Time Time Figure 4: Simulated data plotted with the solution obtained from the estimated parameters in region R 1 for the a) flat curve with θ 0 = (0.6, 0.2, 0.5) b) moderate curve with θ 0 = (0.8, 0.1, 0.3) c) steep curve with θ 0 = (0.7, 0.2, 0.5). 23

24 Region R 2 The flat curve with β 0 =( 0.6, 0.2, 0.5). 25 The moderate curve with β 0 =( 0.8, 0.1, 0.3). 25 The steep curve with β 0 =( 0.7, 0.2, 0.5) Model Model Model Time Time Time Figure 5: Simulated data plotted with the solution obtained from the estimated parameters in region R 2 for the a) flat curve with θ 0 = (0.6, 0.2, 0.5) b) moderate curve with θ 0 = (0.8, 0.1, 0.3) c) steep curve with θ 0 = (0.7, 0.2, 0.5). 24

25 Regions R 0 and R 1 The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 0 n 1 ˆθ n Standard Errors 25 0 (0.7244,0.1414,0.0992) (0.0049, , ) 49 0 (0.7241,0.1401,0.0992) (0.0094, , ) (0.6998,0.0397,0.1) (2.2059e 4, e 5, e 4 ) Table 1: The standard errors for a, b, and x 0 for the optimized ˆθ n. 25

26 Regions R 1 and R 2 The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 1 n 2 ˆθ n Standard Errors 0 25 (1.0057,0.0582,0.3249) (1.0794, , ) 0 49 (1.0058,0.0582,0.3249) (2.1739, , ) (0.6996,0.04,0.1003) (5.0686e 3, e 4, e 3 ) Table 2: The standard errors for a, b, and x 0 for the optimized ˆθ n. 26

27 Simulated Noise We consider two sets of simulated noisy data for the moderate curve. One set corresponds to R 0 and is perturbed using σ 0 = and the other corresponds to R 2 and has σ 0 = 0.5. These values represent an added noise level of approximately ten percent at the lowest bound in each region. 20 σ 0 =0 σ 0 = Time Figure 6: Noisy data using σ 0 = 0.5 where θ 0 = (0.8, 0.1, 0.3). 27

28 Regions R 0 and R 1 with Noise The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 0 n 1 ˆθn Standard Errors 25 0 (0.7244,0.1455,0.0988) (0.052, , ) 49 0 (0.7245,0.1452,0.0988) (0.1060, , ) (0.7086,0.0517,0.0988) (0.0497, , 0.033) (0.703,0.0435,0.0994) (0.091, , ) 0 25 (0.7058,0.0491,0.0994) (0.0133, , 0.009) 0 49 (0.6958,0.0362,0.1008) (0.0252, , ) Table 3: The standard errors for the optimized ˆθ n with σ = using data sampled from the interval [0, 4]. 28

29 Regions R 1 and R 2 with Noise The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 1 n 2 ˆθn Standard Errors 0 25 (1.0058,0.0582,0.3249) (2.7953, , ) 0 49 (1.0057,0.0582,0.3249) (6.3606, , ) (0.5286,0.0297,0.4397) (3.7015, , ) (0.673,0.0384,0.123) ( , 0.679, ) 25 0 (0.5175,0.0281,0.4307) (1.0073, , ) 49 0 (0.4713,0.0249,0.5921) (2.1167, , ) Table 4: The standard errors for the optimized ˆθ n with σ = 0.5 using data sampled from the interval [8, 16]. 29

30 Conclusions Sampling data from regions near steady states alone may not provide sufficient information to obtain all parameters with any degree of accuracy (lack of sensitivity). Including additional data from the same regions near steady states will only increase the standard errors without significant improvement to the optimized parameters (no new information from new sample points). When extra data points are sampled in a way that provides new information from a region where the solution is sensitive to the parameters, then the parameter estimation will be improved (i.e., region R 1 from this problem). 30

31 References [BDSS] H.T. Banks, M. Davidian, J.R. Samuels, Jr., and K.L. Sutton, An Inverse Problem Statistical Methodology Summary, CRSC-TR08-01, January, 2008; Chapter 11 in Statistical Estimation Approaches in Epidemiology, (edited by Gerardo Chowell, Mac Hyman, Nick Hengartner, Luis M.A Bettencourt and Carlos Castillo-Chavez), Springer, Berlin Heidelberg New York, 2009, pp [BN] H. T. Banks and H. K. Nguyen, Sensitivity of dynamical system to Banach space parameters, CRSC Tech Rep., CRSC-TR05-13, N.C. State University, February, 2005; J. Math Anal. Appl., 323 (2006), [BEG] H.T. Banks, Stacey L. Ernstberger and Sarah L. Grove, Standard errors and confidence intervals in inverse problems: Sensitivity and associated pitfalls, CRSC-TR06-10, March, 2006; J. Inverse and Ill-posed Problems, 15 (2007), [CB] G. Casella and R. L. Berger, Statistical Inference, Duxbury, California, [DG] M. Davidian and D. Giltinan, Nonlinear Models for Repeated Measurement, Chapman & Hall, London, [G] A. R. Gallant, Nonlinear Statistical Models, John Wiley & Sons, Inc., New York, [J] R. I. Jennrich, Asymptotic properties of non-linear least squares estimators., Ann. Math. Statist., 40: ,

32 [K] M. Kot, Elements of Mathematical Ecology, Cambridge University Press, 2001, p [SeWi] G. A. F. Seber and C. J. Wild, Nonlinear Regression, John Wiley & Sons, Inc., New York,

33 Region R 1 with σ 0 = The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 1 Interval ˆθn Standard Errors 25 [2,4] (0.7058,0.0491,0.0994) (0.0133, , 0.009) 49 [2,4] (0.6958,0.0362,0.1008) (0.0252, , ) 49 [2,6] (0.6981,0.0394,0.1006) (0.0355, , ) 73 [2,8] (0.6999,0.04,0.1001) (0.0794, , ) Table 5: The standard errors for the optimized ˆθ n with σ = using data sampled from various intervals within R 1. 33

34 Region R 1 with σ 0 = 0.5 The moderate curve with θ 0 = (0.7, 0.04, 0.1) using θ 0 = (0.8, 0.1, 0.3). n 1 Interval ˆθn Standard Errors 25 [8,12] (0.5175,0.0281,0.4307) (1.0073, , ) 49 [8,12] (0.4713,0.0249,0.5921) (2.1167, , ) 49 [4,12] (0.6720,0.0383,0.1254) (5.1571, , ) Table 6: The standard errors for the optimized ˆθ n with σ = 0.5 using data sampled from various intervals within R 1. 34