Interval Estimation III: Fisher's Information & Bootstrapping

Transcription

1 Interval Estimation III: Fisher's Information & Bootstrapping

2 Frequentist Confidence Interval Will consider four approaches to estimating confidence interval Standard Error (+/ se) Likelihood Profile (+3.84 Deviance) Fisher Information Bootstrap All require additional assumptions

3 CI constructed based on difference in Deviance from MLE 2 (0.95,1 d.f.) --> D ~ 3.84, lnl ~ 1.92 qchisq(0.95,1) p-value = 1-pchisq(D,1)

4 Fisher's Information Uses the curvature of the ln likelihood to estimate variance of parameter error dist'n I = d 2 ln L d 2 ML se = 1 I Quadratic approximation of lnl (exact for N) C.I. is based on standard error

5 Example: Normal mean Assume L = N(x µ,σ 2 ) 1 L= 2 2 exp 2 x i n/2 2 2 lnl= n 2 ln 2 2 x i 2 lnl = 1 2 x i = x i 2 2 lnl n 2 = n 2 I = n 2 se= n

6 Example: Exponential L= exp x i lnl=nln x i lnl = n x i 2 lnl 2 = n 2 I = n 2 se= ML n

7 Fisher's Pro/Con Analytical solution Generalization Requires Math Approximation Can be biased, especially at small sample size Asymptotic CLT assumption that parameter dist'n is asymptotically Normal

8 Bootstrap Monte Carlo method (numerical) Based on idea of generating parameter distribution based on large number of replicate data sets that are the same size as original (data random) Two variants Parametric: pseudodata Nonparametric: resample data

9 Non-parametric bootstrap Draw a replicate data set by resampling from the original data Fit parameters to resample Repeat procedure n times Estimate parameter CI based on sample quantiles Estimate parameter std error as sample s.d.

10 Resampling Original Sample Sample sample N For CI, resample all covariates simultaneously For null model, resample response variable independent of covariates Difficult for highly structured data R: sample(x,length(x),replace=true) Hint: For simult. sample the indices not the data

11 R example: Quadratic lnl <- function(beta,x,y){ ### - ln likelihood -sum(dnorm(y,beta[1] + beta[2]*x + beta[3]*x^2,beta[4],log=true)) } ic <- c(mean(y),0,0,sd(y)) ### initial condition outmle <- optim(ic,lnl,x=x,y=y) ### MLE fit ### general code for non-parameteric bootstrap nboot < Bboot <- matrix(na,nboot,4) ### storage for(i in 1:nboot){ samp <- sample(1:length(y),length(y),replace=true) ### sample out <- optim(ic,lnl,x=x[samp],y=y[samp]) ### fit sample Bboot[i,] <- out$par }

12

13

14 Parametric bootstrap Based on parameters fit to original data set generate pseudodata with same dist'n Fit parameters to resample Repeat procedure n times Estimate parameter CI based on sample quantiles Estimate parameter std error as sample s.d.

15 R example: Quadratic lnl <- function(beta,x,y){ ### - ln likelihood -sum(dnorm(y,beta[1] + beta[2]*x + beta[3]*x^2,beta[4],log=true)) } ic <- c(mean(y),0,0,sd(y)) ### initial condition outmle <- optim(ic,lnl,x=x,y=y) beta <- outmle$par ### MLE fit ### general code for non-parameteric bootstrap nboot < Bboot <- matrix(na,nboot,4) ### storage for(i in 1:nboot){ yboot <- rnorm(n,beta[1] + beta[2]*x + beta[3]*x^2,beta[4]) ##pseudo out <- optim(ic,lnl,x=x,y=yboot) ### fit pseudo Bboot[i,] <- out$par }

16

17 Bootstrap Pro/Con No fancy math Code/computation (though less than MCMC) Easy to extend to Multiple parameter models Estimation of covariance Prediction Nonparameteric: Inference limited to sample Not for small sample size (var sample < var pop'n) Parameteric: assumes model is true

18

19 Frequentist Model Intervals Recall that in practice, Bayesian model CI and PI were generated from quantiles of model predictions from MCMC CI: parameter uncertainty PI: parameter + data uncertainty (pseudodata) The simplest Frequentist CI and PI is based on the bootstrap Implementation is identical except use Bootstrap parameter sample rather than MCMC sample

20 R example ##storage yconf <- matrix(na,nboot,length(xseq)) ypred <- matrix(na,nboot,length(xseq)) #loop over bootstrap samples for(i in 1:nboot){ ey <- Bboot[i,1] + Bboot[i,2]*xseq + Bboot[i,3]*xseq^2 yconf[i,] <- ey ypred[i,] <- rnorm(length(xseq),ey,bboot[i,4]) } Note: need to generate new pseudodata even if we did a parametric bootstrap to account for parameter uncertainty ## summary quantiles ci <- apply(yconf,2,quantile,c(0.025,0.5,0.975)) pi <- apply(ypred,2,quantile,c(0.025,0.975))

21

22 Classic Error Propagation Def'n of Var: Var[f(x )] = E[(f(x ) E[f(x )])2 ] Taylor series approximation: var [f x ] f i i j f i Individual parameter variances generated from methods discussed earlier (Fisher's I, bootstrap, MCMC) 2 var[ i ] f j cov[ i, j ]

23 Example: Regression CI f x = 0 1 x f 0 =1 f 1 =x Var [f x ] 1 Var [ 0 ] x 2 Var [ 1 ] x Cov [ 0, 1 ]

24 Example: Regression PI f x = 0 1 x f 0 =1 f 1 =x f =1 Var [f x ] Var[ 0 ] x 2 Var [ 1 ] 2 x Cov[ 0, 1 ]

25

26 Frequentist Confidence Interval Will consider four approaches to estimating confidence interval Standard Error (+/ se) Likelihood Profile (+3.84 Deviance) Fisher Information Bootstrap (simulation) Model CI and PI Bootstrap se = 1 I Taylor Series approximation