Preliminaries The bootstrap Bias reduction Hypothesis tests Regression Confidence intervals Time series Final remark. Bootstrap inference

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Preliminaries The bootstrap Bias reduction Hypothesis tests Regression Confidence intervals Time series Final remark. Bootstrap inference"

Transcription

1 1 / 171 Bootstrap inference Francisco Cribari-Neto Departamento de Estatística Universidade Federal de Pernambuco Recife / PE, Brazil October 2013

2 2 / 171 Unpaid advertisement Graduate program in Statistics: Masters and PhD Graduate Program in Statistics at Federal University of Pernambuco: CAPES: 5 Research areas: asymptotic theory, econometrics, game theory, multivariate analysis, probability theory, regression analysis, signal processing, time series.

3 3 / 171 Figure 1 : Boa Viagem beach.

4 4 / 171 Figure 2 : Boa Viagem beach (at night).

5 5 / 171 Figure 3 : Recife ( the Brazilian Venice ).

6 6 / 171 Figure 4 : Porto de Galinhas beach (near Recife).

7 7 / 171 In a world in which the price of calculation continues to decrease rapidly, but the price of theorem proving continues to hold steady or increase, elementary economics indicates that we ought to spend a larger fraction of our time on calculation. John W. Tukey, 1986

8 8 / 171 Figure 5 : This is the man.

9 9 / 171 Some references General 1 Chernick, M.R. (1999). Bootstrap Methods: A Practitioner s Guide. New York: Wiley. 2 Davison, A.C. & Hinkley, D.V. (1997). Bootstrap Methods and their Application. Cambridge: Cambridge University Press. 3 Efon, B. (1982) The Jackknife, the Bootstrap and Other Resampling Plans. Philadelphia: Society for Industrial and Applied Mathematics. 4 Efron, B. & Tibshirani, R.J. (1993). An Introduction to the Bootstrap. New York: Chapman & Hall.

10 10 / 171 Some references General 5 Godfrey, L. (2009). Bootstrap Tests for Regression Models. New York: Palgrave MacMillan. 6 Hall, P. (1992). The Bootstrap and Edgeworth Expansion. New York: Springer Verlag. 7 Shao, J. & Tu, D. (1995). The Jackknife and Bootstrap. New York: Springer.

11 11 / 171 Some references Specific 1 Booth, J.G. & Hall, P. (1994). Monte Carlo approximation and the iterated bootstrap. Biometrika, 81, Cribari-Neto, F. & Zarkos, S.G. (1999). Bootstrap methods for heteroskedastic regression models: evidence on estimation and testing. Econometric Reviews, 18, Cribari-Neto, F. & Zarkos, S.G. (2001). Heteroskedasticity-consistent covariance matrix estimation: White s estimator and the bootstrap. Journal of Statistical Computation and Simulation, 68,

12 12 / 171 Some references Specific (cont.) 4 Cribari-Neto, F. (2004). Asymptotic inference under heteroskedasticity of unknown form. Computational Statistics and Data Analysis, 45, Cribari-Neto, F.; Frery, A.C. & Silva, M.F. (2002). Improved estimation of clutter properties in speckled imagery. Computational Statistics and Data Analysis, 40, Davidson, R. & MacKinnon, J.G. (2000). Bootstrap tests: how many bootstraps? Econometric Reviews, 19,

13 13 / 171 Some references Specific (cont.) 7 Ferrari, S.L.P. & Cribari-Neto, F. (1997). On bootstrap and analytical bias corrections. Economics Letters, 58, Ferrari, S.L.P. & Cribari-Neto, F. (1999). On the robustness of analytical and bootstrap corrections to score tests in regression models. Journal of Statistical Computation and Simulation, 64,

14 14 / 171 Some references Specific (cont.) 9 Lemonte, A.J.; Simas, A.B.; Cribari-Neto, F. (2008). Bootstrap-based improved estimators for the two-parameter Birnbaum-Saunders distribution. Journal of Statistical Computation and Simulation, 78, MacKinnon, J.G. & Smith, Jr., A.A. (1998). Approximate bias correction in econometrics. Journal of Econometrics, 85, Wu, C.F.J. (1996). Jackknife, bootstrap and other resampling methods in regression analysis (with discussion). Annals of Statistics, 14,

15 15 / 171 A fundamental equation Figure 6 : C.R. Rao C.R. Rao: uncertain knowledge + knowledge about the uncertainty = useful knowledge

16 15 / 171 A fundamental equation Figure 6 : C.R. Rao C.R. Rao: uncertain knowledge + knowledge about the uncertainty = useful knowledge

17 15 / 171 A fundamental equation Figure 6 : C.R. Rao C.R. Rao: uncertain knowledge + knowledge about the uncertainty = useful knowledge

18 15 / 171 A fundamental equation Figure 6 : C.R. Rao C.R. Rao: uncertain knowledge + knowledge about the uncertainty = useful knowledge

19 16 / 171 As Anthony Davison and David Hinkley remind us... The explicity recognition of uncertainty is central to statistical sciences. Notions such as prior information, probability models, likelihood, standard errors and confidence limits are all intended to formalize uncertainty and thereby make allowance for it. Davison & Hinkley

20 16 / 171 As Anthony Davison and David Hinkley remind us... The explicity recognition of uncertainty is central to statistical sciences. Notions such as prior information, probability models, likelihood, standard errors and confidence limits are all intended to formalize uncertainty and thereby make allowance for it. Davison & Hinkley

21 17 / 171 The big picture (the grand scheme of things) Sampling POPULATION DATA model = f(parameters) The great scheme of things.

22 18 / 171 What is the bootstrap? The bootstrap is a computer-based method for assessing the accuracy of statistical estimates and tests. It was first proposed by Bradley Efron in a 1979 Annals of Statistics paper. Main idea: Treat the data as if they were the (true, unknown) population, and draw samples (with replacement) from the data as if you were sampling from the population. Repeat the procedure a large number of times (say, B), each time computing the quantity of interest. Then, use the B values of the quantity of interest to estimate its unknown distribution.

23 18 / 171 What is the bootstrap? The bootstrap is a computer-based method for assessing the accuracy of statistical estimates and tests. It was first proposed by Bradley Efron in a 1979 Annals of Statistics paper. Main idea: Treat the data as if they were the (true, unknown) population, and draw samples (with replacement) from the data as if you were sampling from the population. Repeat the procedure a large number of times (say, B), each time computing the quantity of interest. Then, use the B values of the quantity of interest to estimate its unknown distribution.

24 18 / 171 What is the bootstrap? The bootstrap is a computer-based method for assessing the accuracy of statistical estimates and tests. It was first proposed by Bradley Efron in a 1979 Annals of Statistics paper. Main idea: Treat the data as if they were the (true, unknown) population, and draw samples (with replacement) from the data as if you were sampling from the population. Repeat the procedure a large number of times (say, B), each time computing the quantity of interest. Then, use the B values of the quantity of interest to estimate its unknown distribution.

25 19 / 171 In a nutshell... population sample }{{} real world bootstrap samples }{{} virtual world

26 20 / 171 Does it work? Question: Does it work well? Answer: Yes (most of the time). In the simplest nonparametric problems we do literally sample from the data, and a common initial reaction is that this is a fraud. In fact it is not. Davison and Hinkley, 1997

27 20 / 171 Does it work? Question: Does it work well? Answer: Yes (most of the time). In the simplest nonparametric problems we do literally sample from the data, and a common initial reaction is that this is a fraud. In fact it is not. Davison and Hinkley, 1997

28 21 / 171 Asymptotic refinement Question: When does the bootstrap provide an asymptotic refinement? The quantity being bootstrapped must be asymptotically pivotal. That is: It must have a limiting distribution free of unknown parameters.

29 21 / 171 Asymptotic refinement Question: When does the bootstrap provide an asymptotic refinement? The quantity being bootstrapped must be asymptotically pivotal. That is: It must have a limiting distribution free of unknown parameters.

30 21 / 171 Asymptotic refinement Question: When does the bootstrap provide an asymptotic refinement? The quantity being bootstrapped must be asymptotically pivotal. That is: It must have a limiting distribution free of unknown parameters.

31 22 / 171 Point estimation in a nutshell Suppose that model that represents the population is indexed by the parameter θ = (θ 1,..., θ p ) Θ, where Θ is the parameter space. Estimator: A statistic used to estimate θ. The estimator, say θ, is typically obtained from the minimization of some undesirable quantity (e.g., sum of squared errors) or from the maximization of some desirable quantity.

32 22 / 171 Point estimation in a nutshell Suppose that model that represents the population is indexed by the parameter θ = (θ 1,..., θ p ) Θ, where Θ is the parameter space. Estimator: A statistic used to estimate θ. The estimator, say θ, is typically obtained from the minimization of some undesirable quantity (e.g., sum of squared errors) or from the maximization of some desirable quantity.

33 22 / 171 Point estimation in a nutshell Suppose that model that represents the population is indexed by the parameter θ = (θ 1,..., θ p ) Θ, where Θ is the parameter space. Estimator: A statistic used to estimate θ. The estimator, say θ, is typically obtained from the minimization of some undesirable quantity (e.g., sum of squared errors) or from the maximization of some desirable quantity.

34 23 / 171 Point estimation in a nutshell (cont.) Some of the most important properties an estimator can enjoy are: Unbiasedness: IE( θ) = θ θ Θ; Consistency ( ): θ p θ; Asymptotic normality: When n is large, θ approx normally distributed; Efficiency: (more generally, optimality in some class; e.g., Gauss-Markov Theorem).

35 23 / 171 Point estimation in a nutshell (cont.) Some of the most important properties an estimator can enjoy are: Unbiasedness: IE( θ) = θ θ Θ; Consistency ( ): θ p θ; Asymptotic normality: When n is large, θ approx normally distributed; Efficiency: (more generally, optimality in some class; e.g., Gauss-Markov Theorem).

36 23 / 171 Point estimation in a nutshell (cont.) Some of the most important properties an estimator can enjoy are: Unbiasedness: IE( θ) = θ θ Θ; Consistency ( ): θ p θ; Asymptotic normality: When n is large, θ approx normally distributed; Efficiency: (more generally, optimality in some class; e.g., Gauss-Markov Theorem).

37 23 / 171 Point estimation in a nutshell (cont.) Some of the most important properties an estimator can enjoy are: Unbiasedness: IE( θ) = θ θ Θ; Consistency ( ): θ p θ; Asymptotic normality: When n is large, θ approx normally distributed; Efficiency: (more generally, optimality in some class; e.g., Gauss-Markov Theorem).

38 23 / 171 Point estimation in a nutshell (cont.) Some of the most important properties an estimator can enjoy are: Unbiasedness: IE( θ) = θ θ Θ; Consistency ( ): θ p θ; Asymptotic normality: When n is large, θ approx normally distributed; Efficiency: (more generally, optimality in some class; e.g., Gauss-Markov Theorem).

39 24 / 171 Setup Y 1,..., Y n i.i.d. F 0 (θ), where θ Θ IR p. We can write the unknown parameter θ as a functional of F 0 : θ = θ(f 0 ). We can denote an estimator of θ (say, the MLE) as θ, which can be written as the functional θ = θ( F), where F is the empirical c.d.f. of Y 1,..., Y n.

40 24 / 171 Setup Y 1,..., Y n i.i.d. F 0 (θ), where θ Θ IR p. We can write the unknown parameter θ as a functional of F 0 : θ = θ(f 0 ). We can denote an estimator of θ (say, the MLE) as θ, which can be written as the functional θ = θ( F), where F is the empirical c.d.f. of Y 1,..., Y n.

41 24 / 171 Setup Y 1,..., Y n i.i.d. F 0 (θ), where θ Θ IR p. We can write the unknown parameter θ as a functional of F 0 : θ = θ(f 0 ). We can denote an estimator of θ (say, the MLE) as θ, which can be written as the functional θ = θ( F), where F is the empirical c.d.f. of Y 1,..., Y n.

42 25 / 171 Plug-in principle Y 1,..., Y n iid F0. Plug-in Write the parameter as θ = θ(f 0 ). Estimator: θ = θ( F). Example: mean Parameter: θ(f 0 ) = y df 0 = IE(Y). Estimator: θ = y d F = n 1 n i=1 y i = y.

43 25 / 171 Plug-in principle Y 1,..., Y n iid F0. Plug-in Write the parameter as θ = θ(f 0 ). Estimator: θ = θ( F). Example: mean Parameter: θ(f 0 ) = y df 0 = IE(Y). Estimator: θ = y d F = n 1 n i=1 y i = y.

44 25 / 171 Plug-in principle Y 1,..., Y n iid F0. Plug-in Write the parameter as θ = θ(f 0 ). Estimator: θ = θ( F). Example: mean Parameter: θ(f 0 ) = y df 0 = IE(Y). Estimator: θ = y d F = n 1 n i=1 y i = y.

45 25 / 171 Plug-in principle Y 1,..., Y n iid F0. Plug-in Write the parameter as θ = θ(f 0 ). Estimator: θ = θ( F). Example: mean Parameter: θ(f 0 ) = y df 0 = IE(Y). Estimator: θ = y d F = n 1 n i=1 y i = y.

46 25 / 171 Plug-in principle Y 1,..., Y n iid F0. Plug-in Write the parameter as θ = θ(f 0 ). Estimator: θ = θ( F). Example: mean Parameter: θ(f 0 ) = y df 0 = IE(Y). Estimator: θ = y d F = n 1 n i=1 y i = y.

47 Main idea: Plug-in principle. Example: Let Y = n 1 n i=1 Y i. We know that if Y i (µ, σ 2 ), then Y (µ, n 1 σ 2 ), so that n 1 σ 2 gives us an indication of the accuracy of the estimate Y. In particular, the standard error of the estimate can be obtained as s.e.(y) = σ 2 n, σ2 = 1 n 1 n (y i ȳ) 2, ( ) for a given observed sample Y 1 = y 1,..., Y n = y n. Bootstrap approach: Write σ 2 = σ 2 (F 0 ), and replace F 0 by F to obtain: σ 2 b.s.e.(y) = n, σ2 σ 2 ( F) = 1 n (y i ȳ) 2. n This is the bootstrap estimate. i=1 i=1 26 / 171

48 Main idea: Plug-in principle. Example: Let Y = n 1 n i=1 Y i. We know that if Y i (µ, σ 2 ), then Y (µ, n 1 σ 2 ), so that n 1 σ 2 gives us an indication of the accuracy of the estimate Y. In particular, the standard error of the estimate can be obtained as s.e.(y) = σ 2 n, σ2 = 1 n 1 n (y i ȳ) 2, ( ) for a given observed sample Y 1 = y 1,..., Y n = y n. Bootstrap approach: Write σ 2 = σ 2 (F 0 ), and replace F 0 by F to obtain: σ 2 b.s.e.(y) = n, σ2 σ 2 ( F) = 1 n (y i ȳ) 2. n This is the bootstrap estimate. i=1 i=1 26 / 171

49 Main idea: Plug-in principle. Example: Let Y = n 1 n i=1 Y i. We know that if Y i (µ, σ 2 ), then Y (µ, n 1 σ 2 ), so that n 1 σ 2 gives us an indication of the accuracy of the estimate Y. In particular, the standard error of the estimate can be obtained as s.e.(y) = σ 2 n, σ2 = 1 n 1 n (y i ȳ) 2, ( ) for a given observed sample Y 1 = y 1,..., Y n = y n. Bootstrap approach: Write σ 2 = σ 2 (F 0 ), and replace F 0 by F to obtain: σ 2 b.s.e.(y) = n, σ2 σ 2 ( F) = 1 n (y i ȳ) 2. n This is the bootstrap estimate. i=1 i=1 26 / 171

50 27 / 171 Noteworthy Note: (i) the difference between the two estimates is minor and vanishes as n ; (ii) F places probability mass 1/n on y 1,..., y n.

51 27 / 171 Noteworthy Note: (i) the difference between the two estimates is minor and vanishes as n ; (ii) F places probability mass 1/n on y 1,..., y n.

52 28 / 171 Problem: We are usually interested in estimates more complicated than the sample mean, and for such statistics we may not have a directly available formula like ( ). [E.g., we may be interested in the correlation coefficient, the median, a given quantile, the coefficients of a quantile regression, etc.] Solution: The bootstrap approach allows us to numerically evaluate σ 2 = σ 2 ( F).

53 28 / 171 Problem: We are usually interested in estimates more complicated than the sample mean, and for such statistics we may not have a directly available formula like ( ). [E.g., we may be interested in the correlation coefficient, the median, a given quantile, the coefficients of a quantile regression, etc.] Solution: The bootstrap approach allows us to numerically evaluate σ 2 = σ 2 ( F).

54 29 / 171 Bootstrap standard error Question: How can we use bootstrap resampling to obtain a standard error for a given estimate?

55 30 / 171 What s in a number?

56 31 / 171 The basic bootstrap algorithm Suppose we wish to obtain an standard error for θ = θ( F), an estimate of θ = θ(f 0 ), from an i.i.d. sample of size n. Here is how we proceed: 1 Compute θ for our sample. 2 Sample from the data with replacement and construct a new sample of size n, say y = (y 1,..., y n). 3 Compute θ for the bootstrap sample obtained in (ii). 4 Repeat steps (ii) and (iii) B times. 5 Use the B realizations of θ to obtain an estimate for the standard error of θ.

57 32 / 171 The basic bootstrap algorithm That is, where 1 b.s.e.( θ ) = B B 1 b=1 { θ b θ ( )} 2, θ ( ) = 1 B B θ b. b=1

58 33 / 171 It is important to notice that... Note that the bootstrap generalizes the jackknife in the sense that resampling is carried out in a random fashion, and not in a deterministic and systematic way ( leave one out ).

59 34 / 171 Parametric versus nonparametric bootstrap The bootstrap may be performed parametrically or nonparametrically. Nonpametric bootstrap: Resampling from F. That is, sample from the data (with replacement). Parametric bootstrap: Sample from F( θ ). The nonparametric bootstrap is more robust against distributional assumptions whereas the parametric bootstrap is expected to be more efficient when the parametric assumptions are true.

60 34 / 171 Parametric versus nonparametric bootstrap The bootstrap may be performed parametrically or nonparametrically. Nonpametric bootstrap: Resampling from F. That is, sample from the data (with replacement). Parametric bootstrap: Sample from F( θ ). The nonparametric bootstrap is more robust against distributional assumptions whereas the parametric bootstrap is expected to be more efficient when the parametric assumptions are true.

61 34 / 171 Parametric versus nonparametric bootstrap The bootstrap may be performed parametrically or nonparametrically. Nonpametric bootstrap: Resampling from F. That is, sample from the data (with replacement). Parametric bootstrap: Sample from F( θ ). The nonparametric bootstrap is more robust against distributional assumptions whereas the parametric bootstrap is expected to be more efficient when the parametric assumptions are true.

62 34 / 171 Parametric versus nonparametric bootstrap The bootstrap may be performed parametrically or nonparametrically. Nonpametric bootstrap: Resampling from F. That is, sample from the data (with replacement). Parametric bootstrap: Sample from F( θ ). The nonparametric bootstrap is more robust against distributional assumptions whereas the parametric bootstrap is expected to be more efficient when the parametric assumptions are true.

63 35 / 171 Noparametric bootstrap sampling EMPIRICAL DISTRIBUTION: Puts equal weight probabilities n 1 at each sample value y i. EMPIRICAL DISTRIBUTION FUNCTION (EDF): F(y) = #{y i y}. n Notice that the values of the EDF are fixed: (0, 1/n, 2/n,..., n/n). Hence, the EDF is equivalent to its points of increase: y (1) y (n) (ordered sample values).

64 35 / 171 Noparametric bootstrap sampling EMPIRICAL DISTRIBUTION: Puts equal weight probabilities n 1 at each sample value y i. EMPIRICAL DISTRIBUTION FUNCTION (EDF): F(y) = #{y i y}. n Notice that the values of the EDF are fixed: (0, 1/n, 2/n,..., n/n). Hence, the EDF is equivalent to its points of increase: y (1) y (n) (ordered sample values).

65 35 / 171 Noparametric bootstrap sampling EMPIRICAL DISTRIBUTION: Puts equal weight probabilities n 1 at each sample value y i. EMPIRICAL DISTRIBUTION FUNCTION (EDF): F(y) = #{y i y}. n Notice that the values of the EDF are fixed: (0, 1/n, 2/n,..., n/n). Hence, the EDF is equivalent to its points of increase: y (1) y (n) (ordered sample values).

66 36 / 171 Noparametric bootstrap sampling (cont.) Since the EDF puts equal probabilities on the data values y 1,..., y n, each Y is independently sampled at random from those data values. Hence, the bootstrap sample is a random sample taken with replacement from the original data.

67 36 / 171 Noparametric bootstrap sampling (cont.) Since the EDF puts equal probabilities on the data values y 1,..., y n, each Y is independently sampled at random from those data values. Hence, the bootstrap sample is a random sample taken with replacement from the original data.

68 37 / 171 Smoothed bootstrap The empirical distribution function ( F) is discrete. Sampling from it boils down to sampling from data with replacement. An interesting idea: To sample from a smoothed distribution function. We replace F by a smooth distribution based on, e.g., a kernel density estimate of F (the derivative of F with respect to y). An example using the correlation coefficient can be found in Efron s 1982 monograph: Efron, B. (1982). The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: SIAM.

69 37 / 171 Smoothed bootstrap The empirical distribution function ( F) is discrete. Sampling from it boils down to sampling from data with replacement. An interesting idea: To sample from a smoothed distribution function. We replace F by a smooth distribution based on, e.g., a kernel density estimate of F (the derivative of F with respect to y). An example using the correlation coefficient can be found in Efron s 1982 monograph: Efron, B. (1982). The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: SIAM.

70 37 / 171 Smoothed bootstrap The empirical distribution function ( F) is discrete. Sampling from it boils down to sampling from data with replacement. An interesting idea: To sample from a smoothed distribution function. We replace F by a smooth distribution based on, e.g., a kernel density estimate of F (the derivative of F with respect to y). An example using the correlation coefficient can be found in Efron s 1982 monograph: Efron, B. (1982). The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: SIAM.

71 37 / 171 Smoothed bootstrap The empirical distribution function ( F) is discrete. Sampling from it boils down to sampling from data with replacement. An interesting idea: To sample from a smoothed distribution function. We replace F by a smooth distribution based on, e.g., a kernel density estimate of F (the derivative of F with respect to y). An example using the correlation coefficient can be found in Efron s 1982 monograph: Efron, B. (1982). The Jackknife, the Bootstrap, and Other Resampling Plans. Philadelphia: SIAM.

72 38 / 171 Bayesian bootstrap Suppose y 1,..., y n are i.i.d. realizations of Y which has distribution function F(θ), where θ is scalar. Let θ be an estimator of θ. We know that the bootstrap can be used to construct an estimate of the distribution of such an estimator. Instead of sampling from the data with replacement (i.e., sampling each y i with probability 1/n), the Bayesian bootstrap uses a posterior probability distribution for y i. The posterior probability distribution is centered at 1/n, but varies for each y i. How is that done?

73 38 / 171 Bayesian bootstrap Suppose y 1,..., y n are i.i.d. realizations of Y which has distribution function F(θ), where θ is scalar. Let θ be an estimator of θ. We know that the bootstrap can be used to construct an estimate of the distribution of such an estimator. Instead of sampling from the data with replacement (i.e., sampling each y i with probability 1/n), the Bayesian bootstrap uses a posterior probability distribution for y i. The posterior probability distribution is centered at 1/n, but varies for each y i. How is that done?

74 38 / 171 Bayesian bootstrap Suppose y 1,..., y n are i.i.d. realizations of Y which has distribution function F(θ), where θ is scalar. Let θ be an estimator of θ. We know that the bootstrap can be used to construct an estimate of the distribution of such an estimator. Instead of sampling from the data with replacement (i.e., sampling each y i with probability 1/n), the Bayesian bootstrap uses a posterior probability distribution for y i. The posterior probability distribution is centered at 1/n, but varies for each y i. How is that done?

75 38 / 171 Bayesian bootstrap Suppose y 1,..., y n are i.i.d. realizations of Y which has distribution function F(θ), where θ is scalar. Let θ be an estimator of θ. We know that the bootstrap can be used to construct an estimate of the distribution of such an estimator. Instead of sampling from the data with replacement (i.e., sampling each y i with probability 1/n), the Bayesian bootstrap uses a posterior probability distribution for y i. The posterior probability distribution is centered at 1/n, but varies for each y i. How is that done?

76 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

77 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

78 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

79 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

80 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

81 39 / 171 Bayesian bootstrap Draw a random sample of size n 1 from the standard uniform distribution. Order the sampled values: u (1),..., u (n 1). Let u (0) = 0 and u (n) = 1. Compute g i = u (i) u (i 1), i = 1,..., n. The g i s are called the gaps between the uniform order statistics. The vector g = (g 1,..., g n ) is used to assign probabilities to the Bayesian bootstrap. That is, sample y i with probability g i (not 1/n). Note that we obtain a different g in each bootstrap replication.

82 40 / 171 Bayesian bootstrap ADVANTAGE: It can used to make Bayesian inference on θ based on the estimated posterior distribution for θ. The bootstrap distribution of θ and the Bayesian bootstrap posterior distribution for θ will be similar in many applications. Reference: Rubin, D.B. (1981). The Bayesian bootstrap. Annals of Statistics, 9,

83 40 / 171 Bayesian bootstrap ADVANTAGE: It can used to make Bayesian inference on θ based on the estimated posterior distribution for θ. The bootstrap distribution of θ and the Bayesian bootstrap posterior distribution for θ will be similar in many applications. Reference: Rubin, D.B. (1981). The Bayesian bootstrap. Annals of Statistics, 9,

84 40 / 171 Bayesian bootstrap ADVANTAGE: It can used to make Bayesian inference on θ based on the estimated posterior distribution for θ. The bootstrap distribution of θ and the Bayesian bootstrap posterior distribution for θ will be similar in many applications. Reference: Rubin, D.B. (1981). The Bayesian bootstrap. Annals of Statistics, 9,

85 41 / 171 Revisiting the big picture Sampling POPULATION DATA model = f(parameters) The great scheme of things.

86 42 / 171 Software Programming Programming bootstrap resampling is easy: (i) PARAMETRIC: Sample from F( θ), (ii) NONPARAMETRIC: Sample from F (empirical distribution function). Sampling from F: 1) Obtain a standard uniform draw, i.e., obtain u from U(0, 1). 2) Generate a random integer (sau, i ) from {1,..., n} as u n + 1. The ith observation in the bootstrap sample is i th observation in the original sample.

87 42 / 171 Software Programming Programming bootstrap resampling is easy: (i) PARAMETRIC: Sample from F( θ), (ii) NONPARAMETRIC: Sample from F (empirical distribution function). Sampling from F: 1) Obtain a standard uniform draw, i.e., obtain u from U(0, 1). 2) Generate a random integer (sau, i ) from {1,..., n} as u n + 1. The ith observation in the bootstrap sample is i th observation in the original sample.

88 43 / 171 Software R package boot: functions and datasets for bootstrapping from the book Bootstrap Methods and Their Applications by A.C. Davison and D.V. Hinkley (1997, Cambridge University Press).

89 44 / 171 Bootstrap bias correction Suppose that θ is biased (although consistent) for θ, and that we would like to obtain a new estimate which is bias-corrected up to some order of accuracy. The bias of θ is: IE( θ) θ (systematic error). Ideally, we would like to have: θ bias, but this is not feasible (since the bias depends on θ). Define the bias-corrected estimate as: θ = θ bias. We then take bias to be bias B = θ ( ) θ, which implies that θ = θ { θ ( ) θ} = 2 θ θ ( ).

90 44 / 171 Bootstrap bias correction Suppose that θ is biased (although consistent) for θ, and that we would like to obtain a new estimate which is bias-corrected up to some order of accuracy. The bias of θ is: IE( θ) θ (systematic error). Ideally, we would like to have: θ bias, but this is not feasible (since the bias depends on θ). Define the bias-corrected estimate as: θ = θ bias. We then take bias to be bias B = θ ( ) θ, which implies that θ = θ { θ ( ) θ} = 2 θ θ ( ).

91 44 / 171 Bootstrap bias correction Suppose that θ is biased (although consistent) for θ, and that we would like to obtain a new estimate which is bias-corrected up to some order of accuracy. The bias of θ is: IE( θ) θ (systematic error). Ideally, we would like to have: θ bias, but this is not feasible (since the bias depends on θ). Define the bias-corrected estimate as: θ = θ bias. We then take bias to be bias B = θ ( ) θ, which implies that θ = θ { θ ( ) θ} = 2 θ θ ( ).

92 44 / 171 Bootstrap bias correction Suppose that θ is biased (although consistent) for θ, and that we would like to obtain a new estimate which is bias-corrected up to some order of accuracy. The bias of θ is: IE( θ) θ (systematic error). Ideally, we would like to have: θ bias, but this is not feasible (since the bias depends on θ). Define the bias-corrected estimate as: θ = θ bias. We then take bias to be bias B = θ ( ) θ, which implies that θ = θ { θ ( ) θ} = 2 θ θ ( ).

93 44 / 171 Bootstrap bias correction Suppose that θ is biased (although consistent) for θ, and that we would like to obtain a new estimate which is bias-corrected up to some order of accuracy. The bias of θ is: IE( θ) θ (systematic error). Ideally, we would like to have: θ bias, but this is not feasible (since the bias depends on θ). Define the bias-corrected estimate as: θ = θ bias. We then take bias to be bias B = θ ( ) θ, which implies that θ = θ { θ ( ) θ} = 2 θ θ ( ).

94 45 / 171 We shall call the above bias correction BC1. NOTE: θ ( ) is not itself the bootstrap bias-corrected estimate. Let s look into that. (For futher details, see, e.g., MacKinnon & Smith, Journal of Econometrics, 1998). Assuming that IE( θ) exists, write θ = θ 0 + B(θ 0, n) + R(θ 0, n), where B(θ 0, n) = IE( θ) θ 0 (i.e., B(, ) is the bias function) and R(θ 0, n) is defined so that the above equation holds. Assume we know the distribution of Y i up to the unknown parameter θ (so that we can use the parametric bootstrap).

95 45 / 171 We shall call the above bias correction BC1. NOTE: θ ( ) is not itself the bootstrap bias-corrected estimate. Let s look into that. (For futher details, see, e.g., MacKinnon & Smith, Journal of Econometrics, 1998). Assuming that IE( θ) exists, write θ = θ 0 + B(θ 0, n) + R(θ 0, n), where B(θ 0, n) = IE( θ) θ 0 (i.e., B(, ) is the bias function) and R(θ 0, n) is defined so that the above equation holds. Assume we know the distribution of Y i up to the unknown parameter θ (so that we can use the parametric bootstrap).

96 45 / 171 We shall call the above bias correction BC1. NOTE: θ ( ) is not itself the bootstrap bias-corrected estimate. Let s look into that. (For futher details, see, e.g., MacKinnon & Smith, Journal of Econometrics, 1998). Assuming that IE( θ) exists, write θ = θ 0 + B(θ 0, n) + R(θ 0, n), where B(θ 0, n) = IE( θ) θ 0 (i.e., B(, ) is the bias function) and R(θ 0, n) is defined so that the above equation holds. Assume we know the distribution of Y i up to the unknown parameter θ (so that we can use the parametric bootstrap).

97 45 / 171 We shall call the above bias correction BC1. NOTE: θ ( ) is not itself the bootstrap bias-corrected estimate. Let s look into that. (For futher details, see, e.g., MacKinnon & Smith, Journal of Econometrics, 1998). Assuming that IE( θ) exists, write θ = θ 0 + B(θ 0, n) + R(θ 0, n), where B(θ 0, n) = IE( θ) θ 0 (i.e., B(, ) is the bias function) and R(θ 0, n) is defined so that the above equation holds. Assume we know the distribution of Y i up to the unknown parameter θ (so that we can use the parametric bootstrap).

98 46 / 171 Noteworthy If θ is n consistent and asymptotically normal, the bias will be typically be O(n 1 ). (Otherwise, n( θ θ 0 ) would not have mean zero asymptotically.)

99 47 / 171 Suppose that B(θ, n) = B(n) for all θ. That is, suppose the bias function if flat. Here, it does not matter at which value of θ we evaluate the bias function since it is flat. An obvious candidate, however, is θ, the MLE. And what we get here is exactly our bias correction BC1: θ = θ B( θ, n) = θ { θ ( ) θ} = 2 θ θ ( ). NOTE: In many (most?) cases, however, the bias function is not flat.

100 47 / 171 Suppose that B(θ, n) = B(n) for all θ. That is, suppose the bias function if flat. Here, it does not matter at which value of θ we evaluate the bias function since it is flat. An obvious candidate, however, is θ, the MLE. And what we get here is exactly our bias correction BC1: θ = θ B( θ, n) = θ { θ ( ) θ} = 2 θ θ ( ). NOTE: In many (most?) cases, however, the bias function is not flat.

101 47 / 171 Suppose that B(θ, n) = B(n) for all θ. That is, suppose the bias function if flat. Here, it does not matter at which value of θ we evaluate the bias function since it is flat. An obvious candidate, however, is θ, the MLE. And what we get here is exactly our bias correction BC1: θ = θ B( θ, n) = θ { θ ( ) θ} = 2 θ θ ( ). NOTE: In many (most?) cases, however, the bias function is not flat.

102 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

103 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

104 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

105 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

106 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

107 48 / 171 Suppose now that the bias function is linear in θ, that is, B(θ, n) = α 0 + α 1 θ. ( ) The main idea is to evaluate ( ) at two points and then solve for α 0 and α 1. (Note that this will require two sets of simulations!) Obvious choices for the two points at which we evaluate the DGP are θ and θ. The solution is α 0 = B B B θ θ θ, α 1 = B B θ θ. (NOTE: Here we are using a shorthand notation.) The estimated α s will converge to the true ones as B increases.

108 49 / 171 The bias-corrected estimator can then be defined as θ = θ α 0 α 1 θ. (Here we are evaluating the bias function at θ itself.) The solution, therefore, is θ = α 1 ( θ α 0 ). We can call the above bias correction BC2.

109 49 / 171 The bias-corrected estimator can then be defined as θ = θ α 0 α 1 θ. (Here we are evaluating the bias function at θ itself.) The solution, therefore, is θ = α 1 ( θ α 0 ). We can call the above bias correction BC2.

110 49 / 171 The bias-corrected estimator can then be defined as θ = θ α 0 α 1 θ. (Here we are evaluating the bias function at θ itself.) The solution, therefore, is θ = α 1 ( θ α 0 ). We can call the above bias correction BC2.

111 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

112 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

113 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

114 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

115 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

116 What if the bias function is nonlinear? In that case, we define a bias-corrected estimator as θ = θ B( θ, n). One way of implement this is as follows. Start w/ B obtained as in the BC1, i.e., B = θ ( ) θ. Now, compute (sequentially) θ (j) = (1 λ) θ (j 1) + λ( θ B( θ (j 1), n)), where θ (0) = θ and 0 < λ 1. Stop when θ (j) θ (j 1) < ɛ for a sufficiently small ɛ. Suggestion: Start with λ = 1. If the procedure does not converge, try smaller values of λ. 50 / 171

117 51 / 171 An alternative bootstrap bias estimate was introduced by Efron (1990). It is carried out nonparametrically and uses an auxiliary (n 1) resampling vector, whose elements are the proportions of observations in the original sample y = (y 1,..., y n ) that were included in the bootstrap sample. Let P = (P1, P 2,..., P n) be the resampling vector. Its jth element (j = 1, 2,..., n), Pj, is defined with respect to a given bootstrap sample y = (y 1,..., y n) as P j = n 1( #{y k = y j} ). It is important to note that the vector P 0 = (1/n, 1/n,..., 1/n) corresponds to the original sample.

118 51 / 171 An alternative bootstrap bias estimate was introduced by Efron (1990). It is carried out nonparametrically and uses an auxiliary (n 1) resampling vector, whose elements are the proportions of observations in the original sample y = (y 1,..., y n ) that were included in the bootstrap sample. Let P = (P1, P 2,..., P n) be the resampling vector. Its jth element (j = 1, 2,..., n), Pj, is defined with respect to a given bootstrap sample y = (y 1,..., y n) as P j = n 1( #{y k = y j} ). It is important to note that the vector P 0 = (1/n, 1/n,..., 1/n) corresponds to the original sample.

119 51 / 171 An alternative bootstrap bias estimate was introduced by Efron (1990). It is carried out nonparametrically and uses an auxiliary (n 1) resampling vector, whose elements are the proportions of observations in the original sample y = (y 1,..., y n ) that were included in the bootstrap sample. Let P = (P1, P 2,..., P n) be the resampling vector. Its jth element (j = 1, 2,..., n), Pj, is defined with respect to a given bootstrap sample y = (y 1,..., y n) as P j = n 1( #{y k = y j} ). It is important to note that the vector P 0 = (1/n, 1/n,..., 1/n) corresponds to the original sample.

120 51 / 171 An alternative bootstrap bias estimate was introduced by Efron (1990). It is carried out nonparametrically and uses an auxiliary (n 1) resampling vector, whose elements are the proportions of observations in the original sample y = (y 1,..., y n ) that were included in the bootstrap sample. Let P = (P1, P 2,..., P n) be the resampling vector. Its jth element (j = 1, 2,..., n), Pj, is defined with respect to a given bootstrap sample y = (y 1,..., y n) as P j = n 1( #{y k = y j} ). It is important to note that the vector P 0 = (1/n, 1/n,..., 1/n) corresponds to the original sample.

121 51 / 171 An alternative bootstrap bias estimate was introduced by Efron (1990). It is carried out nonparametrically and uses an auxiliary (n 1) resampling vector, whose elements are the proportions of observations in the original sample y = (y 1,..., y n ) that were included in the bootstrap sample. Let P = (P1, P 2,..., P n) be the resampling vector. Its jth element (j = 1, 2,..., n), Pj, is defined with respect to a given bootstrap sample y = (y 1,..., y n) as P j = n 1( #{y k = y j} ). It is important to note that the vector P 0 = (1/n, 1/n,..., 1/n) corresponds to the original sample.

122 52 / 171 Also, any bootstrap replicate θ can be defined as a function of the resampling vector. For example, if θ = s(y) = y = n 1 n i=1 y i, then θ = y 1 + y y n n = (np 1 )y (np n)y n n = #{y k = y 1}y #{y k = y n}y n n = P y.

123 52 / 171 Also, any bootstrap replicate θ can be defined as a function of the resampling vector. For example, if θ = s(y) = y = n 1 n i=1 y i, then θ = y 1 + y y n n = (np 1 )y (np n)y n n = #{y k = y 1}y #{y k = y n}y n n = P y.

124 53 / 171 Suppose we can write the estimate of interest, obtained from the original sample y, as G(P 0 ). It is now possible to obtain bootstrap estimates θ b using the resampling vectors P b, b = 1, 2,..., R, as G(P b ). Efron s (1990) bootstrap bias estimate, BˆF(ˆθ, θ), is defined as BˆF(ˆθ, θ) = ˆθ ( ) G(P ( ) ), where P ( ) = 1 R R P b, b=1 which differs from ˆBˆF(ˆθ, θ), since ˆBˆF(ˆθ, θ) = ˆθ ( ) G(P 0 ). Notice that this bias estimate uses an additional information, namely: the proportions of the n observations that were selected each nonparametric resampling.

125 53 / 171 Suppose we can write the estimate of interest, obtained from the original sample y, as G(P 0 ). It is now possible to obtain bootstrap estimates θ b using the resampling vectors P b, b = 1, 2,..., R, as G(P b ). Efron s (1990) bootstrap bias estimate, BˆF(ˆθ, θ), is defined as BˆF(ˆθ, θ) = ˆθ ( ) G(P ( ) ), where P ( ) = 1 R R P b, b=1 which differs from ˆBˆF(ˆθ, θ), since ˆBˆF(ˆθ, θ) = ˆθ ( ) G(P 0 ). Notice that this bias estimate uses an additional information, namely: the proportions of the n observations that were selected each nonparametric resampling.

126 53 / 171 Suppose we can write the estimate of interest, obtained from the original sample y, as G(P 0 ). It is now possible to obtain bootstrap estimates θ b using the resampling vectors P b, b = 1, 2,..., R, as G(P b ). Efron s (1990) bootstrap bias estimate, BˆF(ˆθ, θ), is defined as BˆF(ˆθ, θ) = ˆθ ( ) G(P ( ) ), where P ( ) = 1 R R P b, b=1 which differs from ˆBˆF(ˆθ, θ), since ˆBˆF(ˆθ, θ) = ˆθ ( ) G(P 0 ). Notice that this bias estimate uses an additional information, namely: the proportions of the n observations that were selected each nonparametric resampling.

127 53 / 171 Suppose we can write the estimate of interest, obtained from the original sample y, as G(P 0 ). It is now possible to obtain bootstrap estimates θ b using the resampling vectors P b, b = 1, 2,..., R, as G(P b ). Efron s (1990) bootstrap bias estimate, BˆF(ˆθ, θ), is defined as BˆF(ˆθ, θ) = ˆθ ( ) G(P ( ) ), where P ( ) = 1 R R P b, b=1 which differs from ˆBˆF(ˆθ, θ), since ˆBˆF(ˆθ, θ) = ˆθ ( ) G(P 0 ). Notice that this bias estimate uses an additional information, namely: the proportions of the n observations that were selected each nonparametric resampling.

128 54 / 171 After obtaining an estimate for the bias, it is easy to obtain a bias-adjusted estimator: θ = s(y) BˆF(ˆθ, θ) = ˆθ ˆθ ( ) + G(P ( ) ).

129 55 / 171 It is important to note that the bias estimation procedure proposed by Efron (1990) requires the estimator ˆθ to have closed form. However, oftentimes the maximum likelihood estimator of θ, the parameter that indexes the model used to represent the population, does not have a closed form. Rather, it needs to be obtained by numerically maximizing the log likelihood function using a nonlinear optimization algorithm, such as a Newton or quasi-newton algorithm. Cribari-Neto, Frery and Silva (2002) proposed an adaptation of Efron s method that can be used with estimators that cannot be written in closed form.

130 55 / 171 It is important to note that the bias estimation procedure proposed by Efron (1990) requires the estimator ˆθ to have closed form. However, oftentimes the maximum likelihood estimator of θ, the parameter that indexes the model used to represent the population, does not have a closed form. Rather, it needs to be obtained by numerically maximizing the log likelihood function using a nonlinear optimization algorithm, such as a Newton or quasi-newton algorithm. Cribari-Neto, Frery and Silva (2002) proposed an adaptation of Efron s method that can be used with estimators that cannot be written in closed form.

131 55 / 171 It is important to note that the bias estimation procedure proposed by Efron (1990) requires the estimator ˆθ to have closed form. However, oftentimes the maximum likelihood estimator of θ, the parameter that indexes the model used to represent the population, does not have a closed form. Rather, it needs to be obtained by numerically maximizing the log likelihood function using a nonlinear optimization algorithm, such as a Newton or quasi-newton algorithm. Cribari-Neto, Frery and Silva (2002) proposed an adaptation of Efron s method that can be used with estimators that cannot be written in closed form.

132 56 / 171 They use the resampling vector to modify the log likelihood function, and then maximize the modified log likelihood. The main idea is to write the log likelihood function in terms of P 0, replace this vector by P ( ), and then maximize the resulting (modified) log likelihood function. The maximizer of such a function is a bias-corrected maximum likelihood estimator.

133 56 / 171 They use the resampling vector to modify the log likelihood function, and then maximize the modified log likelihood. The main idea is to write the log likelihood function in terms of P 0, replace this vector by P ( ), and then maximize the resulting (modified) log likelihood function. The maximizer of such a function is a bias-corrected maximum likelihood estimator.

134 56 / 171 They use the resampling vector to modify the log likelihood function, and then maximize the modified log likelihood. The main idea is to write the log likelihood function in terms of P 0, replace this vector by P ( ), and then maximize the resulting (modified) log likelihood function. The maximizer of such a function is a bias-corrected maximum likelihood estimator.

Preliminaries The bootstrap Bias reduction Hypothesis tests Regression Confidence intervals Time series Final remark. Bootstrap inference

Preliminaries The bootstrap Bias reduction Hypothesis tests Regression Confidence intervals Time series Final remark. Bootstrap inference 1 / 172 Bootstrap inference Francisco Cribari-Neto Departamento de Estatística Universidade Federal de Pernambuco Recife / PE, Brazil email: cribari@gmail.com October 2014 2 / 172 Unpaid advertisement

More information

Heteroskedasticity-Robust Inference in Finite Samples

Heteroskedasticity-Robust Inference in Finite Samples Heteroskedasticity-Robust Inference in Finite Samples Jerry Hausman and Christopher Palmer Massachusetts Institute of Technology December 011 Abstract Since the advent of heteroskedasticity-robust standard

More information

Bayesian Interpretations of Heteroskedastic Consistent Covariance Estimators Using the Informed Bayesian Bootstrap

Bayesian Interpretations of Heteroskedastic Consistent Covariance Estimators Using the Informed Bayesian Bootstrap Bayesian Interpretations of Heteroskedastic Consistent Covariance Estimators Using the Informed Bayesian Bootstrap Dale J. Poirier University of California, Irvine September 1, 2008 Abstract This paper

More information

Political Science 236 Hypothesis Testing: Review and Bootstrapping

Political Science 236 Hypothesis Testing: Review and Bootstrapping Political Science 236 Hypothesis Testing: Review and Bootstrapping Rocío Titiunik Fall 2007 1 Hypothesis Testing Definition 1.1 Hypothesis. A hypothesis is a statement about a population parameter The

More information

Bootstrap-Based Improvements for Inference with Clustered Errors

Bootstrap-Based Improvements for Inference with Clustered Errors Bootstrap-Based Improvements for Inference with Clustered Errors Colin Cameron, Jonah Gelbach, Doug Miller U.C. - Davis, U. Maryland, U.C. - Davis May, 2008 May, 2008 1 / 41 1. Introduction OLS regression

More information

New heteroskedasticity-robust standard errors for the linear regression model

New heteroskedasticity-robust standard errors for the linear regression model Brazilian Journal of Probability and Statistics 2014, Vol. 28, No. 1, 83 95 DOI: 10.1214/12-BJPS196 Brazilian Statistical Association, 2014 New heteroskedasticity-robust standard errors for the linear

More information

UNIVERSITÄT POTSDAM Institut für Mathematik

UNIVERSITÄT POTSDAM Institut für Mathematik UNIVERSITÄT POTSDAM Institut für Mathematik Testing the Acceleration Function in Life Time Models Hannelore Liero Matthias Liero Mathematische Statistik und Wahrscheinlichkeitstheorie Universität Potsdam

More information

Stat 5101 Lecture Notes

Stat 5101 Lecture Notes Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random

More information

Inference via Kernel Smoothing of Bootstrap P Values

Inference via Kernel Smoothing of Bootstrap P Values Queen s Economics Department Working Paper No. 1054 Inference via Kernel Smoothing of Bootstrap P Values Jeff Racine McMaster University James G. MacKinnon Queen s University Department of Economics Queen

More information

LESLIE GODFREY LIST OF PUBLICATIONS

LESLIE GODFREY LIST OF PUBLICATIONS LESLIE GODFREY LIST OF PUBLICATIONS This list is in two parts. First, there is a set of selected publications for the period 1971-1996. Second, there are details of more recent outputs. SELECTED PUBLICATIONS,

More information

Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk

Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk Axel Gandy Department of Mathematics Imperial College London a.gandy@imperial.ac.uk user! 2009, Rennes July 8-10, 2009

More information

σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) =

σ(a) = a N (x; 0, 1 2 ) dx. σ(a) = Φ(a) = Until now we have always worked with likelihoods and prior distributions that were conjugate to each other, allowing the computation of the posterior distribution to be done in closed form. Unfortunately,

More information

Bias-corrected Estimators of Scalar Skew Normal

Bias-corrected Estimators of Scalar Skew Normal Bias-corrected Estimators of Scalar Skew Normal Guoyi Zhang and Rong Liu Abstract One problem of skew normal model is the difficulty in estimating the shape parameter, for which the maximum likelihood

More information

Online Appendix to Correcting Estimation Bias in Dynamic Term Structure Models

Online Appendix to Correcting Estimation Bias in Dynamic Term Structure Models Online Appendix to Correcting Estimation Bias in Dynamic Term Structure Models Michael D. Bauer, Glenn D. Rudebusch, Jing Cynthia Wu May 4, 2012 A Bootstrap bias correction The bootstrap has become a common

More information

Quantile regression and heteroskedasticity

Quantile regression and heteroskedasticity Quantile regression and heteroskedasticity José A. F. Machado J.M.C. Santos Silva June 18, 2013 Abstract This note introduces a wrapper for qreg which reports standard errors and t statistics that are

More information

On the Accuracy of Bootstrap Confidence Intervals for Efficiency Levels in Stochastic Frontier Models with Panel Data

On the Accuracy of Bootstrap Confidence Intervals for Efficiency Levels in Stochastic Frontier Models with Panel Data On the Accuracy of Bootstrap Confidence Intervals for Efficiency Levels in Stochastic Frontier Models with Panel Data Myungsup Kim University of North Texas Yangseon Kim East-West Center Peter Schmidt

More information

Theory and Methods of Statistical Inference. PART I Frequentist likelihood methods

Theory and Methods of Statistical Inference. PART I Frequentist likelihood methods PhD School in Statistics XXV cycle, 2010 Theory and Methods of Statistical Inference PART I Frequentist likelihood methods (A. Salvan, N. Sartori, L. Pace) Syllabus Some prerequisites: Empirical distribution

More information

Theory and Methods of Statistical Inference. PART I Frequentist theory and methods

Theory and Methods of Statistical Inference. PART I Frequentist theory and methods PhD School in Statistics cycle XXVI, 2011 Theory and Methods of Statistical Inference PART I Frequentist theory and methods (A. Salvan, N. Sartori, L. Pace) Syllabus Some prerequisites: Empirical distribution

More information

A Simple, Graphical Procedure for Comparing Multiple Treatment Effects

A Simple, Graphical Procedure for Comparing Multiple Treatment Effects A Simple, Graphical Procedure for Comparing Multiple Treatment Effects Brennan S. Thompson and Matthew D. Webb May 15, 2015 > Abstract In this paper, we utilize a new graphical

More information

Statistics - Lecture One. Outline. Charlotte Wickham 1. Basic ideas about estimation

Statistics - Lecture One. Outline. Charlotte Wickham  1. Basic ideas about estimation Statistics - Lecture One Charlotte Wickham wickham@stat.berkeley.edu http://www.stat.berkeley.edu/~wickham/ Outline 1. Basic ideas about estimation 2. Method of Moments 3. Maximum Likelihood 4. Confidence

More information

ON THE NUMBER OF BOOTSTRAP REPETITIONS FOR BC a CONFIDENCE INTERVALS. DONALD W. K. ANDREWS and MOSHE BUCHINSKY COWLES FOUNDATION PAPER NO.

ON THE NUMBER OF BOOTSTRAP REPETITIONS FOR BC a CONFIDENCE INTERVALS. DONALD W. K. ANDREWS and MOSHE BUCHINSKY COWLES FOUNDATION PAPER NO. ON THE NUMBER OF BOOTSTRAP REPETITIONS FOR BC a CONFIDENCE INTERVALS BY DONALD W. K. ANDREWS and MOSHE BUCHINSKY COWLES FOUNDATION PAPER NO. 1069 COWLES FOUNDATION FOR RESEARCH IN ECONOMICS YALE UNIVERSITY

More information

Day 3B Nonparametrics and Bootstrap

Day 3B Nonparametrics and Bootstrap Day 3B Nonparametrics and Bootstrap c A. Colin Cameron Univ. of Calif.- Davis Frontiers in Econometrics Bavarian Graduate Program in Economics. Based on A. Colin Cameron and Pravin K. Trivedi (2009,2010),

More information

Theory and Methods of Statistical Inference

Theory and Methods of Statistical Inference PhD School in Statistics cycle XXIX, 2014 Theory and Methods of Statistical Inference Instructors: B. Liseo, L. Pace, A. Salvan (course coordinator), N. Sartori, A. Tancredi, L. Ventura Syllabus Some prerequisites:

More information

Testing Statistical Hypotheses

Testing Statistical Hypotheses E.L. Lehmann Joseph P. Romano Testing Statistical Hypotheses Third Edition 4y Springer Preface vii I Small-Sample Theory 1 1 The General Decision Problem 3 1.1 Statistical Inference and Statistical Decisions

More information

AN EMPIRICAL LIKELIHOOD RATIO TEST FOR NORMALITY

AN EMPIRICAL LIKELIHOOD RATIO TEST FOR NORMALITY Econometrics Working Paper EWP0401 ISSN 1485-6441 Department of Economics AN EMPIRICAL LIKELIHOOD RATIO TEST FOR NORMALITY Lauren Bin Dong & David E. A. Giles Department of Economics, University of Victoria

More information

Double Bootstrap Confidence Interval Estimates with Censored and Truncated Data

Double Bootstrap Confidence Interval Estimates with Censored and Truncated Data Journal of Modern Applied Statistical Methods Volume 13 Issue 2 Article 22 11-2014 Double Bootstrap Confidence Interval Estimates with Censored and Truncated Data Jayanthi Arasan University Putra Malaysia,

More information

Bootstrapping heteroskedastic regression models: wild bootstrap vs. pairs bootstrap

Bootstrapping heteroskedastic regression models: wild bootstrap vs. pairs bootstrap Bootstrapping heteroskedastic regression models: wild bootstrap vs. pairs bootstrap Emmanuel Flachaire To cite this version: Emmanuel Flachaire. Bootstrapping heteroskedastic regression models: wild bootstrap

More information

Bias evaluation and reduction for sample-path optimization

Bias evaluation and reduction for sample-path optimization Bias evaluation and reduction for sample-path optimization 1 1 Department of Computing Science and Operational Research Université de Montréal; CIRRELT Québec, Canada Eigth IMACS Seminar on Monte Carlo

More information

Advanced Econometrics

Advanced Econometrics Advanced Econometrics Dr. Andrea Beccarini Center for Quantitative Economics Winter 2013/2014 Andrea Beccarini (CQE) Econometrics Winter 2013/2014 1 / 156 General information Aims and prerequisites Objective:

More information

Bagging During Markov Chain Monte Carlo for Smoother Predictions

Bagging During Markov Chain Monte Carlo for Smoother Predictions Bagging During Markov Chain Monte Carlo for Smoother Predictions Herbert K. H. Lee University of California, Santa Cruz Abstract: Making good predictions from noisy data is a challenging problem. Methods

More information

The assumptions are needed to give us... valid standard errors valid confidence intervals valid hypothesis tests and p-values

The assumptions are needed to give us... valid standard errors valid confidence intervals valid hypothesis tests and p-values Statistical Consulting Topics The Bootstrap... The bootstrap is a computer-based method for assigning measures of accuracy to statistical estimates. (Efron and Tibshrani, 1998.) What do we do when our

More information

Bootstrap Methods and their Application

Bootstrap Methods and their Application Bootstrap Methods and their Application Anthony Davison c 2006 A short course based on the book Bootstrap Methods and their Application, by A C Davison and D V Hinkley c Cambridge University Press, 1997

More information

FULL LIKELIHOOD INFERENCES IN THE COX MODEL

FULL LIKELIHOOD INFERENCES IN THE COX MODEL October 20, 2007 FULL LIKELIHOOD INFERENCES IN THE COX MODEL BY JIAN-JIAN REN 1 AND MAI ZHOU 2 University of Central Florida and University of Kentucky Abstract We use the empirical likelihood approach

More information

A Course in Applied Econometrics Lecture 18: Missing Data. Jeff Wooldridge IRP Lectures, UW Madison, August Linear model with IVs: y i x i u i,

A Course in Applied Econometrics Lecture 18: Missing Data. Jeff Wooldridge IRP Lectures, UW Madison, August Linear model with IVs: y i x i u i, A Course in Applied Econometrics Lecture 18: Missing Data Jeff Wooldridge IRP Lectures, UW Madison, August 2008 1. When Can Missing Data be Ignored? 2. Inverse Probability Weighting 3. Imputation 4. Heckman-Type

More information

Density Estimation. Seungjin Choi

Density Estimation. Seungjin Choi Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/

More information

Discussion of the paper Inference for Semiparametric Models: Some Questions and an Answer by Bickel and Kwon

Discussion of the paper Inference for Semiparametric Models: Some Questions and an Answer by Bickel and Kwon Discussion of the paper Inference for Semiparametric Models: Some Questions and an Answer by Bickel and Kwon Jianqing Fan Department of Statistics Chinese University of Hong Kong AND Department of Statistics

More information

Resampling and the Bootstrap

Resampling and the Bootstrap Resampling and the Bootstrap Axel Benner Biostatistics, German Cancer Research Center INF 280, D-69120 Heidelberg benner@dkfz.de Resampling and the Bootstrap 2 Topics Estimation and Statistical Testing

More information

A COMPARISON OF HETEROSCEDASTICITY ROBUST STANDARD ERRORS AND NONPARAMETRIC GENERALIZED LEAST SQUARES

A COMPARISON OF HETEROSCEDASTICITY ROBUST STANDARD ERRORS AND NONPARAMETRIC GENERALIZED LEAST SQUARES A COMPARISON OF HETEROSCEDASTICITY ROBUST STANDARD ERRORS AND NONPARAMETRIC GENERALIZED LEAST SQUARES MICHAEL O HARA AND CHRISTOPHER F. PARMETER Abstract. This paper presents a Monte Carlo comparison of

More information

One-Sample Numerical Data

One-Sample Numerical Data One-Sample Numerical Data quantiles, boxplot, histogram, bootstrap confidence intervals, goodness-of-fit tests University of California, San Diego Instructor: Ery Arias-Castro http://math.ucsd.edu/~eariasca/teaching.html

More information

The Bayesian Approach to Multi-equation Econometric Model Estimation

The Bayesian Approach to Multi-equation Econometric Model Estimation Journal of Statistical and Econometric Methods, vol.3, no.1, 2014, 85-96 ISSN: 2241-0384 (print), 2241-0376 (online) Scienpress Ltd, 2014 The Bayesian Approach to Multi-equation Econometric Model Estimation

More information

Bootstrap Confidence Intervals for the Correlation Coefficient

Bootstrap Confidence Intervals for the Correlation Coefficient Bootstrap Confidence Intervals for the Correlation Coefficient Homer S. White 12 November, 1993 1 Introduction We will discuss the general principles of the bootstrap, and describe two applications of

More information

A Simulation Comparison Study for Estimating the Process Capability Index C pm with Asymmetric Tolerances

A Simulation Comparison Study for Estimating the Process Capability Index C pm with Asymmetric Tolerances Available online at ijims.ms.tku.edu.tw/list.asp International Journal of Information and Management Sciences 20 (2009), 243-253 A Simulation Comparison Study for Estimating the Process Capability Index

More information

Bayesian Quadrature: Model-based Approximate Integration. David Duvenaud University of Cambridge

Bayesian Quadrature: Model-based Approximate Integration. David Duvenaud University of Cambridge Bayesian Quadrature: Model-based Approimate Integration David Duvenaud University of Cambridge The Quadrature Problem ˆ We want to estimate an integral Z = f ()p()d ˆ Most computational problems in inference

More information

Department of Criminology

Department of Criminology Department of Criminology Working Paper No. 2015-13.01 Calibrated Percentile Double Bootstrap for Robust Linear Regression Inference Daniel McCarthy University of Pennsylvania Kai Zhang University of North

More information

large number of i.i.d. observations from P. For concreteness, suppose

large number of i.i.d. observations from P. For concreteness, suppose 1 Subsampling Suppose X i, i = 1,..., n is an i.i.d. sequence of random variables with distribution P. Let θ(p ) be some real-valued parameter of interest, and let ˆθ n = ˆθ n (X 1,..., X n ) be some estimate

More information

Estimation of Stress-Strength Reliability Using Record Ranked Set Sampling Scheme from the Exponential Distribution

Estimation of Stress-Strength Reliability Using Record Ranked Set Sampling Scheme from the Exponential Distribution Filomat 9:5 015, 1149 116 DOI 10.98/FIL1505149S Published by Faculty of Sciences and Mathematics, University of Niš, Serbia Available at: http://www.pmf.ni.ac.rs/filomat Estimation of Stress-Strength eliability

More information

Numerical Analysis for Statisticians

Numerical Analysis for Statisticians Kenneth Lange Numerical Analysis for Statisticians Springer Contents Preface v 1 Recurrence Relations 1 1.1 Introduction 1 1.2 Binomial CoefRcients 1 1.3 Number of Partitions of a Set 2 1.4 Horner's Method

More information

Inversion Base Height. Daggot Pressure Gradient Visibility (miles)

Inversion Base Height. Daggot Pressure Gradient Visibility (miles) Stanford University June 2, 1998 Bayesian Backtting: 1 Bayesian Backtting Trevor Hastie Stanford University Rob Tibshirani University of Toronto Email: trevor@stat.stanford.edu Ftp: stat.stanford.edu:

More information

University of California San Diego and Stanford University and

University of California San Diego and Stanford University and First International Workshop on Functional and Operatorial Statistics. Toulouse, June 19-21, 2008 K-sample Subsampling Dimitris N. olitis andjoseph.romano University of California San Diego and Stanford

More information

Inference Based on the Wild Bootstrap

Inference Based on the Wild Bootstrap Inference Based on the Wild Bootstrap James G. MacKinnon Department of Economics Queen s University Kingston, Ontario, Canada K7L 3N6 email: jgm@econ.queensu.ca Ottawa September 14, 2012 The Wild Bootstrap

More information

NONINFORMATIVE NONPARAMETRIC BAYESIAN ESTIMATION OF QUANTILES

NONINFORMATIVE NONPARAMETRIC BAYESIAN ESTIMATION OF QUANTILES NONINFORMATIVE NONPARAMETRIC BAYESIAN ESTIMATION OF QUANTILES Glen Meeden School of Statistics University of Minnesota Minneapolis, MN 55455 Appeared in Statistics & Probability Letters Volume 16 (1993)

More information

Beyond stochastic gradient descent for large-scale machine learning

Beyond stochastic gradient descent for large-scale machine learning Beyond stochastic gradient descent for large-scale machine learning Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with Eric Moulines - October 2014 Big data revolution? A new

More information

Supplement to Quantile-Based Nonparametric Inference for First-Price Auctions

Supplement to Quantile-Based Nonparametric Inference for First-Price Auctions Supplement to Quantile-Based Nonparametric Inference for First-Price Auctions Vadim Marmer University of British Columbia Artyom Shneyerov CIRANO, CIREQ, and Concordia University August 30, 2010 Abstract

More information

Estimation, Inference, and Hypothesis Testing

Estimation, Inference, and Hypothesis Testing Chapter 2 Estimation, Inference, and Hypothesis Testing Note: The primary reference for these notes is Ch. 7 and 8 of Casella & Berger 2. This text may be challenging if new to this topic and Ch. 7 of

More information

Testing Statistical Hypotheses

Testing Statistical Hypotheses E.L. Lehmann Joseph P. Romano, 02LEu1 ttd ~Lt~S Testing Statistical Hypotheses Third Edition With 6 Illustrations ~Springer 2 The Probability Background 28 2.1 Probability and Measure 28 2.2 Integration.........

More information

Tolerance limits for a ratio of normal random variables

Tolerance limits for a ratio of normal random variables Tolerance limits for a ratio of normal random variables Lanju Zhang 1, Thomas Mathew 2, Harry Yang 1, K. Krishnamoorthy 3 and Iksung Cho 1 1 Department of Biostatistics MedImmune, Inc. One MedImmune Way,

More information

Estimation of the Conditional Variance in Paired Experiments

Estimation of the Conditional Variance in Paired Experiments Estimation of the Conditional Variance in Paired Experiments Alberto Abadie & Guido W. Imbens Harvard University and BER June 008 Abstract In paired randomized experiments units are grouped in pairs, often

More information

Robust Performance Hypothesis Testing with the Variance. Institute for Empirical Research in Economics University of Zurich

Robust Performance Hypothesis Testing with the Variance. Institute for Empirical Research in Economics University of Zurich Institute for Empirical Research in Economics University of Zurich Working Paper Series ISSN 1424-0459 Working Paper No. 516 Robust Performance Hypothesis Testing with the Variance Olivier Ledoit and Michael

More information

Interval Estimation III: Fisher's Information & Bootstrapping

Interval Estimation III: Fisher's Information & Bootstrapping Interval Estimation III: Fisher's Information & Bootstrapping Frequentist Confidence Interval Will consider four approaches to estimating confidence interval Standard Error (+/- 1.96 se) Likelihood Profile

More information

A Bootstrap Test for Causality with Endogenous Lag Length Choice. - theory and application in finance

A Bootstrap Test for Causality with Endogenous Lag Length Choice. - theory and application in finance CESIS Electronic Working Paper Series Paper No. 223 A Bootstrap Test for Causality with Endogenous Lag Length Choice - theory and application in finance R. Scott Hacker and Abdulnasser Hatemi-J April 200

More information

A Bootstrap Test for Conditional Symmetry

A Bootstrap Test for Conditional Symmetry ANNALS OF ECONOMICS AND FINANCE 6, 51 61 005) A Bootstrap Test for Conditional Symmetry Liangjun Su Guanghua School of Management, Peking University E-mail: lsu@gsm.pku.edu.cn and Sainan Jin Guanghua School

More information

A multiple testing procedure for input variable selection in neural networks

A multiple testing procedure for input variable selection in neural networks A multiple testing procedure for input variable selection in neural networks MicheleLaRoccaandCiraPerna Department of Economics and Statistics - University of Salerno Via Ponte Don Melillo, 84084, Fisciano

More information

The Exact Distribution of the t-ratio with Robust and Clustered Standard Errors

The Exact Distribution of the t-ratio with Robust and Clustered Standard Errors The Exact Distribution of the t-ratio with Robust and Clustered Standard Errors by Bruce E. Hansen Department of Economics University of Wisconsin June 2017 Bruce Hansen (University of Wisconsin) Exact

More information

MCMC algorithms for fitting Bayesian models

MCMC algorithms for fitting Bayesian models MCMC algorithms for fitting Bayesian models p. 1/1 MCMC algorithms for fitting Bayesian models Sudipto Banerjee sudiptob@biostat.umn.edu University of Minnesota MCMC algorithms for fitting Bayesian models

More information

Bayesian Inference and the Parametric Bootstrap. Bradley Efron Stanford University

Bayesian Inference and the Parametric Bootstrap. Bradley Efron Stanford University Bayesian Inference and the Parametric Bootstrap Bradley Efron Stanford University Importance Sampling for Bayes Posterior Distribution Newton and Raftery (1994 JRSS-B) Nonparametric Bootstrap: good choice

More information

arxiv: v2 [stat.co] 23 Nov 2017

arxiv: v2 [stat.co] 23 Nov 2017 Inference in a bimodal Birnbaum-Saunders model Rodney V. Fonseca a,, Francisco Cribari-Neto a arxiv:1609.00339v2 [stat.co] 23 Nov 2017 a Departamento de Estatística, Universidade Federal de Pernambuco,

More information

INFERENCE FOR MULTIPLE LINEAR REGRESSION MODEL WITH EXTENDED SKEW NORMAL ERRORS

INFERENCE FOR MULTIPLE LINEAR REGRESSION MODEL WITH EXTENDED SKEW NORMAL ERRORS Pak. J. Statist. 2016 Vol. 32(2), 81-96 INFERENCE FOR MULTIPLE LINEAR REGRESSION MODEL WITH EXTENDED SKEW NORMAL ERRORS A.A. Alhamide 1, K. Ibrahim 1 M.T. Alodat 2 1 Statistics Program, School of Mathematical

More information

g-priors for Linear Regression

g-priors for Linear Regression Stat60: Bayesian Modeling and Inference Lecture Date: March 15, 010 g-priors for Linear Regression Lecturer: Michael I. Jordan Scribe: Andrew H. Chan 1 Linear regression and g-priors In the last lecture,

More information

CPSC 540: Machine Learning

CPSC 540: Machine Learning CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is

More information

Long-Run Covariability

Long-Run Covariability Long-Run Covariability Ulrich K. Müller and Mark W. Watson Princeton University October 2016 Motivation Study the long-run covariability/relationship between economic variables great ratios, long-run Phillips

More information

Bootstrap Testing in Nonlinear Models

Bootstrap Testing in Nonlinear Models !" # $%% %%% Bootstrap Testing in Nonlinear Models GREQAM Centre de la Vieille Charité 2 rue de la Charité 13002 Marseille, France by Russell Davidson russell@ehesscnrs-mrsfr and James G MacKinnon Department

More information

Some Monte Carlo Evidence for Adaptive Estimation of Unit-Time Varying Heteroscedastic Panel Data Models

Some Monte Carlo Evidence for Adaptive Estimation of Unit-Time Varying Heteroscedastic Panel Data Models Some Monte Carlo Evidence for Adaptive Estimation of Unit-Time Varying Heteroscedastic Panel Data Models G. R. Pasha Department of Statistics, Bahauddin Zakariya University Multan, Pakistan E-mail: drpasha@bzu.edu.pk

More information

Modified Simes Critical Values Under Positive Dependence

Modified Simes Critical Values Under Positive Dependence Modified Simes Critical Values Under Positive Dependence Gengqian Cai, Sanat K. Sarkar Clinical Pharmacology Statistics & Programming, BDS, GlaxoSmithKline Statistics Department, Temple University, Philadelphia

More information

Testing Goodness Of Fit Of The Geometric Distribution: An Application To Human Fecundability Data

Testing Goodness Of Fit Of The Geometric Distribution: An Application To Human Fecundability Data Journal of Modern Applied Statistical Methods Volume 4 Issue Article 8 --5 Testing Goodness Of Fit Of The Geometric Distribution: An Application To Human Fecundability Data Sudhir R. Paul University of

More information

Rejection sampling - Acceptance probability. Review: How to sample from a multivariate normal in R. Review: Rejection sampling. Weighted resampling

Rejection sampling - Acceptance probability. Review: How to sample from a multivariate normal in R. Review: Rejection sampling. Weighted resampling Rejection sampling - Acceptance probability Review: How to sample from a multivariate normal in R Goal: Simulate from N d (µ,σ)? Note: For c to be small, g() must be similar to f(). The art of rejection

More information

Post-exam 2 practice questions 18.05, Spring 2014

Post-exam 2 practice questions 18.05, Spring 2014 Post-exam 2 practice questions 18.05, Spring 2014 Note: This is a set of practice problems for the material that came after exam 2. In preparing for the final you should use the previous review materials,

More information

arxiv: v4 [stat.ml] 26 Jun 2017

arxiv: v4 [stat.ml] 26 Jun 2017 A nonparametric sequential test for online randomized experiments arxiv:1612490v4 [stat.ml] 26 Jun 2017 Vineet Abhishek Walmart Labs CA, USA vabhishek@walmartlabs.com Abstract Shie Mannor Technion Haifa,

More information

A Note on Bayesian Inference After Multiple Imputation

A Note on Bayesian Inference After Multiple Imputation A Note on Bayesian Inference After Multiple Imputation Xiang Zhou and Jerome P. Reiter Abstract This article is aimed at practitioners who plan to use Bayesian inference on multiplyimputed datasets in

More information

Econometrics. Week 4. Fall Institute of Economic Studies Faculty of Social Sciences Charles University in Prague

Econometrics. Week 4. Fall Institute of Economic Studies Faculty of Social Sciences Charles University in Prague Econometrics Week 4 Institute of Economic Studies Faculty of Social Sciences Charles University in Prague Fall 2012 1 / 23 Recommended Reading For the today Serial correlation and heteroskedasticity in

More information

Bayesian Heteroskedasticity-Robust Regression. Richard Startz * revised February Abstract

Bayesian Heteroskedasticity-Robust Regression. Richard Startz * revised February Abstract Bayesian Heteroskedasticity-Robust Regression Richard Startz * revised February 2015 Abstract I offer here a method for Bayesian heteroskedasticity-robust regression. The Bayesian version is derived by

More information

Markov-Switching Models with Endogenous Explanatory Variables. Chang-Jin Kim 1

Markov-Switching Models with Endogenous Explanatory Variables. Chang-Jin Kim 1 Markov-Switching Models with Endogenous Explanatory Variables by Chang-Jin Kim 1 Dept. of Economics, Korea University and Dept. of Economics, University of Washington First draft: August, 2002 This version:

More information

AFT Models and Empirical Likelihood

AFT Models and Empirical Likelihood AFT Models and Empirical Likelihood Mai Zhou Department of Statistics, University of Kentucky Collaborators: Gang Li (UCLA); A. Bathke; M. Kim (Kentucky) Accelerated Failure Time (AFT) models: Y = log(t

More information

Lecture 2: From Linear Regression to Kalman Filter and Beyond

Lecture 2: From Linear Regression to Kalman Filter and Beyond Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing

More information

Bayesian Inference. Chapter 1. Introduction and basic concepts

Bayesian Inference. Chapter 1. Introduction and basic concepts Bayesian Inference Chapter 1. Introduction and basic concepts M. Concepción Ausín Department of Statistics Universidad Carlos III de Madrid Master in Business Administration and Quantitative Methods Master

More information

On robust and efficient estimation of the center of. Symmetry.

On robust and efficient estimation of the center of. Symmetry. On robust and efficient estimation of the center of symmetry Howard D. Bondell Department of Statistics, North Carolina State University Raleigh, NC 27695-8203, U.S.A (email: bondell@stat.ncsu.edu) Abstract

More information

Confidence Distribution

Confidence Distribution Confidence Distribution Xie and Singh (2013): Confidence distribution, the frequentist distribution estimator of a parameter: A Review Céline Cunen, 15/09/2014 Outline of Article Introduction The concept

More information

Outline of GLMs. Definitions

Outline of GLMs. Definitions Outline of GLMs Definitions This is a short outline of GLM details, adapted from the book Nonparametric Regression and Generalized Linear Models, by Green and Silverman. The responses Y i have density

More information

Lecture 1: Random number generation, permutation test, and the bootstrap. August 25, 2016

Lecture 1: Random number generation, permutation test, and the bootstrap. August 25, 2016 Lecture 1: Random number generation, permutation test, and the bootstrap August 25, 2016 Statistical simulation 1/21 Statistical simulation (Monte Carlo) is an important part of statistical method research.

More information

Lecture 1: Supervised Learning

Lecture 1: Supervised Learning Lecture 1: Supervised Learning Tuo Zhao Schools of ISYE and CSE, Georgia Tech ISYE6740/CSE6740/CS7641: Computational Data Analysis/Machine from Portland, Learning Oregon: pervised learning (Supervised)

More information

A COMPARISON OF POISSON AND BINOMIAL EMPIRICAL LIKELIHOOD Mai Zhou and Hui Fang University of Kentucky

A COMPARISON OF POISSON AND BINOMIAL EMPIRICAL LIKELIHOOD Mai Zhou and Hui Fang University of Kentucky A COMPARISON OF POISSON AND BINOMIAL EMPIRICAL LIKELIHOOD Mai Zhou and Hui Fang University of Kentucky Empirical likelihood with right censored data were studied by Thomas and Grunkmier (1975), Li (1995),

More information

WEIGHTED LIKELIHOOD NEGATIVE BINOMIAL REGRESSION

WEIGHTED LIKELIHOOD NEGATIVE BINOMIAL REGRESSION WEIGHTED LIKELIHOOD NEGATIVE BINOMIAL REGRESSION Michael Amiguet 1, Alfio Marazzi 1, Victor Yohai 2 1 - University of Lausanne, Institute for Social and Preventive Medicine, Lausanne, Switzerland 2 - University

More information

where x i and u i are iid N (0; 1) random variates and are mutually independent, ff =0; and fi =1. ff(x i )=fl0 + fl1x i with fl0 =1. We examine the e

where x i and u i are iid N (0; 1) random variates and are mutually independent, ff =0; and fi =1. ff(x i )=fl0 + fl1x i with fl0 =1. We examine the e Inference on the Quantile Regression Process Electronic Appendix Roger Koenker and Zhijie Xiao 1 Asymptotic Critical Values Like many other Kolmogorov-Smirnov type tests (see, e.g. Andrews (1993)), the

More information

Sequential Bayesian Updating

Sequential Bayesian Updating BS2 Statistical Inference, Lectures 14 and 15, Hilary Term 2009 May 28, 2009 We consider data arriving sequentially X 1,..., X n,... and wish to update inference on an unknown parameter θ online. In a

More information

Nonlinear Regression

Nonlinear Regression Nonlinear Regression James H. Steiger Department of Psychology and Human Development Vanderbilt University James H. Steiger (Vanderbilt University) Nonlinear Regression 1 / 36 Nonlinear Regression 1 Introduction

More information

The Bootstrap in Econometrics

The Bootstrap in Econometrics The Bootstrap in Econometrics by Russell Davidson Department of Economics and CIREQ McGill University Montréal, Québec, Canada H3A 2T7 AMSE and GREQAM Centre de la Vieille Charité 2 rue de la Charité 13236

More information

Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process

Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process Applied Mathematical Sciences, Vol. 4, 2010, no. 62, 3083-3093 Sequential Procedure for Testing Hypothesis about Mean of Latent Gaussian Process Julia Bondarenko Helmut-Schmidt University Hamburg University

More information

Economics 536 Lecture 7. Introduction to Specification Testing in Dynamic Econometric Models

Economics 536 Lecture 7. Introduction to Specification Testing in Dynamic Econometric Models University of Illinois Fall 2016 Department of Economics Roger Koenker Economics 536 Lecture 7 Introduction to Specification Testing in Dynamic Econometric Models In this lecture I want to briefly describe

More information

Smooth nonparametric estimation of a quantile function under right censoring using beta kernels

Smooth nonparametric estimation of a quantile function under right censoring using beta kernels Smooth nonparametric estimation of a quantile function under right censoring using beta kernels Chanseok Park 1 Department of Mathematical Sciences, Clemson University, Clemson, SC 29634 Short Title: Smooth

More information

Smooth simultaneous confidence bands for cumulative distribution functions

Smooth simultaneous confidence bands for cumulative distribution functions Journal of Nonparametric Statistics, 2013 Vol. 25, No. 2, 395 407, http://dx.doi.org/10.1080/10485252.2012.759219 Smooth simultaneous confidence bands for cumulative distribution functions Jiangyan Wang

More information

COS513: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 9: LINEAR REGRESSION

COS513: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 9: LINEAR REGRESSION COS513: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 9: LINEAR REGRESSION SEAN GERRISH AND CHONG WANG 1. WAYS OF ORGANIZING MODELS In probabilistic modeling, there are several ways of organizing models:

More information