Simulation Li Zhao, SJTU Spring, 2017 Li Zhao Simulation 1 / 19
Introduction Simulation consists of drawing from a density, calculating a statistic for each draw, and averaging the results. Simulation methods play a very important role in Approximating integration (such as simulated maximum likelihood, simulated method of moments); Maximization; Forecasting; Counterfactual analysis; Bayesian analysis. Li Zhao Simulation 2 / 19
Outline 1 Simulate Random Variables 2 Approximate Expectation Simple Case Importance Sampling 3 Simulated-Assisted Estimation Maximum Simulated Likelihood Method of Simulated Moments 4 GHK Simulator Li Zhao Simulation 3 / 19
Simulate One Random Variable Statistical software can draw many types of random numbers. Uniform, binomial, normal, etc. Drawing a random variable from corresponding cumulative distribution F (ε). Step 1: Draw u (r) U[0,1]. A draw from the standard uniform provides a number between zero and one. Step 2: Solve ε (r) = F 1 (u (r) ). The cumulative distribution of ε (r) equals F ( ). Note that this procedure works only for univariate distributions. Li Zhao Simulation 4 / 19
Example: Simulate Type 1 Extreme Value Distribution The Type 1 extreme value distribution, which is the basis for logit model, has cumulative distribution F (ε) = exp( exp( ε)). A draw from this density if obtained as ε = ln( ln µ)). Suppose that (X 1,X 2,...) is a sequence of independent random variables, each with the standard exponential distribution G(x) = 1 e x. The distribution of Y = max{x 1,X 2,...X n ) ln(n) converges to the (type 1) standard extreme value distribution as n. We can verify this using simulation. Li Zhao Simulation 5 / 19
Simulate Multivariate Normals Suppose we want to draw ε N(b,Σ). ε is a p 1 vector. A Choleski factor of is dened as a lower-triangular matrix L such that L L = Σ. L is called the Choleski factor. Then we can generate η (r) N(0 3,I 3 ). Then b + Lη (r) N(b,Σ). For example, when p = 3, we generate ε 1 l 11 0 0 ε 2 = b + l 21 l 22 0 ε 3 l 31 l 32 l 33 η 1 η 2 η 3 Li Zhao Simulation 6 / 19
Outline 1 Simulate Random Variables 2 Approximate Expectation Simple Case Importance Sampling 3 Simulated-Assisted Estimation Maximum Simulated Likelihood Method of Simulated Moments 4 GHK Simulator Li Zhao Simulation 7 / 19
Approximate Expectation Suppose x F ( ). Principle of integration via simulation (Monte Carlo integration"): approximate an expectation as a sample average. If we are interested in E(x), we can draw ε (1),ε (2),...ε (R) independently from f (x). By law of large number, E(x) = ε f (ε) dε 1 R F R ε (r). r=1 If we are interested in E(t(x)), we can draw ε (1),ε (2),...ε (R) independently from f (x). By LLN, E(t(x)) = t(ε) f (ε) dε 1 R F R t(ε (r) ). r=1 Li Zhao Simulation 8 / 19
Importance Sampling Suppose we want to approximate. E(t(x)) = t(ε) f (ε) dε 1 R F R t(ε (r) ) r=1 What is is hard to fraw ε f ( ) but it is easy to draw ε g( )? We can draw ε (1),ε (2),...ε (R) independently from g( ) E(t(x)) = t(ε)f (ε)dε = t(ε) f (ε) F G t g(ε) g(ε)ds 1 R R t(ε (r) ) f (ε(r) ) r=1 g(ε (r) ) Sampling t(ε) from f ( ) distribution is equivalent to sampling t(ε) w(ε) from g( ) distribution, with importance sampling weight w(ε) = f (ε) g(ε). f and g should have the same support. Li Zhao Simulation 9 / 19
Importance Sampling - Algorithm Suppose ε has a density f (ε) that cannot be easily drawn from by the other procedures. Suppose further that there is another density, g(ε), that can easily be drawn from. Draws from f ( ) can be obtained as follows. 1) Take a draw from g(ε) and label it ε 1. 2) Weight the draw by f (ε 1 )/g(ε 2 ). 3) Repeat this process many times. The set of weighted draws is equivalent to a set of draws from f. Li Zhao Simulation 10 / 19
Importance Sampling - Example Suppose we want to simulate the mean of a standard normal distribution, truncated to the unit interval [0, 1]. The desired sampling density is: f (x) = φ(x) 1 0 φ(x)dx. We may draw ε N(0,1) and throw away observations outside [0,1]. Or, we can use importance sampling, draw ε U[0,1] so g(ε) = 1. For each draw, we multiple its weight w(ε) = f (x) g(x) = φ(x) 10 φ(x)dx. The simulated mean 1 R R r=1 ε (r) w(ε (r) ) is the mean of truncated normal. We will use importance sampling in discrete-outcome game-theoretic model. In that application, we are asked to draw truncated multivariate normal distribution. Li Zhao Simulation 11 / 19
Outline 1 Simulate Random Variables 2 Approximate Expectation Simple Case Importance Sampling 3 Simulated-Assisted Estimation Maximum Simulated Likelihood Method of Simulated Moments 4 GHK Simulator Li Zhao Simulation 12 / 19
Maximum Simulated Likelihood The log-likelihood function is: LL(θ) = lnp n (θ). n P n (θ) is the (exact) probability of the observed choice of observation n, and the summation is over a sample. Let P S n (θ) be a simulated approximation to P n (θ). The simulated log-likelihood function is SLL(θ) = lnpn S (θ). n Li Zhao Simulation 13 / 19
Method of Simulated Moments Method of moments: The moments, like the expectation of any variables, are functions of the parameters: E(g(x,y,θ)) = 0. For example. E(x(y x β)) = 0. g(x,yβ) = x(y x β). We take sample moments ĝ. The GMM estimator minimizes The MSM ĝ(θ) W ĝ(θ). If g(β) is dicult to calculate, we can simulate it, denote as g S (β). MSM estimator minimizes gˆ S (θ) W g ˆS (θ). Li Zhao Simulation 14 / 19
Outline 1 Simulate Random Variables 2 Approximate Expectation Simple Case Importance Sampling 3 Simulated-Assisted Estimation Maximum Simulated Likelihood Method of Simulated Moments 4 GHK Simulator Li Zhao Simulation 15 / 19
GHK simulator GewekeHajivassiliouKeane (GHK) multivariate normal simulator. Consider the Probit Model where ε 1 ε 2 ε 3 What is P 0? How to get ˆP 0? P 0 = N(0,Σ). u 0 = 0 u 1 = βx 1 + ε 1 u 2 = βx 2 + ε 2 u 3 = βx 3 + ε 3 P 0 = Pr(u 1 < 0,u 2 < 0,u 3 < 0). βx1 βx2 βx3 φ 123 (ε 1,ε 2,ε 3 ;Σ)dε 3 dε 2 dε 1. numerical approximations perform poorly in computing high order integrals. Li Zhao Simulation 16 / 19
GHK Simulator - Trivariate Case (1) Let us illustrate the GHK simulator in the trivariate case (generalization to higher orders is straightforward). We wish to evaluate P 0 = Pr(ε 1 < βx 1,ε 2 < βx 2,ε 3 < βx 3 ). The joint distribution can be rewritten as a product of conditional probabilities: Pr(ε 1 < βx 1 ) Pr(ε 2 < βx 2 ε 1 < βx 1 ) Pr(ε 3 < βx 2 ε 1 < βx 1,ε 2 < βx 2 ) Let L be the lower triangular Cholesky decomposition of Σ, such that: L L = Σ. ε 1 l 11 0 0 η 1 ε 2 = l 21 l 22 0 η 2 ε 3 l 31 l 32 l 33 η 3 Li Zhao Simulation 17 / 19
GHK simulator - Trivariate Case (2) Pr(ε 1 < βx 1 ) Pr(ε 2 < βx 2 ε 1 < βx 1 ) Pr(ε 3 < βx 2 ε 1 < βx 1,ε 2 < βx 2 ). Pr(ε 1 < βx 1 ) = Pr(η 1 < βx 1 l 11 ) Pr(ε 2 < βx 2 ε 1 < βx 1 ) = Pr(η 2 < βx 2 l 21 η 1 l 22 η 1 < βx 1 l 11 ) Pr(ε 3 < βx 2 ε 1 < βx 1,ε 2 < βx 2 ) =Pr(η 3 < βx 3 l 31 η 1 l 32 η 2 l 33 η 1 < βx 1 l 11,η 2 < βx 2 l 21 η 1 l 22 ) Because (η 1,η 2,η 3 ) are independent, we can draw η (s) 1, η (s) 2 from truncated normal, and approximate P 0 as a product of univariate CDF. P 0,GHK = 1 S S s=1 Φ( βx 1 l 11 )Φ( βx2 l η (s) 21 1 l 22 )Φ( βx 3 l 31 η (s) 1 l 32 η (s) 2 ). l 33 Li Zhao Simulation 18 / 19
Summarize Like other mathematically intensive sciences, economics is becoming increasingly computerized. Common software Stata / EViews / GAUSS/ R / Matlab / C / Fortran / Python / Julia. Econometrics: Approximate expectation, SML, MSM. Approximate standard error by bootstrap and subsampling. Optimization. Forecast and counterfactual analysis. Li Zhao Simulation 19 / 19