Economics 241B Review of Limit Theorems for Sequences of Random Variables

Transcription

1 Economics 241B Review of Limit Theorems for Sequences of Random Variables Convergence in Distribution The previous de nitions of convergence focus on the outcome sequences of a random variable. Convergence in distribution refers to the probability distribution of each element of the sequence directly. De nition. A sequence of random variables Y1 ; Y 2 ; : : : in distribution to a random variable Y if at all c such that F Y lim P Y n < c = P (Y < c) is continuous. We express this as Y n D! Y. An equivalent de nition is De nition. A sequence of random variables Y1 ; Y 2 ; : : : in distribution to F Y if lim F Y n (c) = F Y (c) at all c such that F Y is continuous. is said to converge is said to converge We express this as Y D n! F Y. The distribution F Y is the asymptotic (or limiting) distribution of Y n. Convergence in distribution is simply the pointwise convergence of F Yn to F Y. ( The requirement that F Y be continuous at all c will hold for all applications in this course, with the exception of binary dependent variables.) In most cases F Y is a Gaussian distribution, for which we write Y n D! N ; 2 : Convergence in distribution is closely related to convergence in probability, if the limit quantity is changed. In the discussion above, a sequence of random variables was shown to converge (in probability) to a constant. One can also establish that a sequence of random variables converges in probability to a random variable. Recall, for convergence (i.p.) to a constant, the probability that Y n is in an neighborhood of must be high, P ( Yn < ) > 1. That is P! : Yn (!) < > 1 : For convergence to a random variable, Y, we need P! : Yn (!) Y (!) < > 1 ; (1)

2 that is, for large n we want the probability that the histogram for Y n is close to the histogram for Y. There is no natural measure of distance here, although we think of as de ning the histogram bin width. If the two histograms are arbitrarily close as n increases, then we write Y n P! Y: Because the two histograms are arbitrarily close, it should not be surprising that Y n P! Y ) Y n D! F Y (sometimes written Y n D! Y ). Why doesn t the reverse hold? Because of the basic workings of probability spaces, about which we rarely concern ourselves. By the fact that both Y n and Y are indexed by the same! in (1), both quantities are de ned on the same probability space. No such requirement is made in establishing convergence of distribution, we simply discuss a sequence of distribution functions. Not only could Y n and Y be de ned on di erent probability spaces, for each n, Y n could be de ned on a di erent probability space. Yet the two de nitions are almost i. The Skorhorod construction is a proof that establishes that if Y D n! Y, then it is always possible to construct sequences Y n 0 and Y 0 that are de ned on the same probability space and are similar to Y n and Y so that Y n 0 P! Y 0. The extension to a sequence of random vectors is immediate: Yn! D Y if the joint c.d.f. F n converges to the joint c.d.f. F of Y at every continuity point of F. Note, however, that unlike other concepts of convergence, for convergence in distribution, element by element convergence does not necessarily mean convergence for the vector sequence. That is "each element Y D n!corresponding element of Y " does not imply " Y D n! Y ". A common way to connect scalar convergence and vector convergence in distribution is Multivariate Convergence in Distribution Theorem: Let Yn be a sequence of K-dimensional random vectors. Then Y n D! Y, 0 Yn D! 0 Y for any K-dimensional vector of real numbers. Convergence in Distribution vs. Convergence in Moments The moments of the limit distribution of Y n are not necessarily equal to the limit of the moments of Y n. Thus Y n D! Y does not imply that lim E Y n = E (Y ). However

3 Lemma (convergence in distribution and moments): Let sn be the s-th moment of Yn and lim sn = s where s is nite (i.e. a real number). Then Y n D! Y ) s is the s-th moment of Y: Thus, for example, if the variance of a sequence of random variables (converging in distribution) converges to some nite number, then that number is the variance of the limiting distribution. Relation among Modes of Convergence As noted above, some modes of convergence are weaker than others. Lemma (relation among the four modes of convergence): (a) Yn m:s:! ) Y p n!. So Yn m:s:! Y ) Y p n! Y. (b) Yn a:s:! ) Y p n!. So Yn a:s:! Y ) Y p n! Y. (c) Yn p!, Yn d!. Part (c) states that if the limiting random variable is a constant (a trivial random variable) then convergence in distribution is the same as convergence in probability. Three Useful Results The great use of asymptotic analysis is that we can easily de ne the convergence properties of ratios, while we cannot easily de ne the expectation of ratios (and so cannot easily do nite sample analysis). These results, which highlight such features, are essential for large-sample theory. Lemma (continuous mapping theorem): Suppose that g () is a continuous function that does not depend on n. Then: Y n P! ) g Y n P! g () Y n D! Y ) g Y n D! g (Y ) : The basic idea is that because g () is continuous, g Y n will be close to g () provided that Y n is close to. Note g Y 1 n = n 2 Yn depends on n and is not covered by this proposition. An immediate implication is that the usual arithmetic

4 operations preserve convergence in probability. For example Y n P! ; Xn P! ) Y n + X n P! + Y n P! ; Xn P! ) Y n Xn P! Y n P! ; Xn P! ) Y n = X n P! = provided that 6= 0 Y n P! ) Y 1 n P! 1 provided that is invertible. The next result combines convergence in probability and distribution, and is used repeatedly to derive the limit theory for our estimators. Lemma (parts a and c are called Slutzky s Theorem): (a) Y n P! and X n D! X ) Y n + X n D! X + (b) Y n P! 0 and X n D! X ) Y 0 nx n P! 0 (c) A n P! A and X n D! X ) A n X n D! AX (d) A n P! A and X n D! X ) X 0 na 1 n X n D! X 0 A 1 X Parts (c)-(d) require that X and A be conformable and (for part (d)) that A be invertible. In particular, if X N (0; V ), then Y n +X n D! N (; V ) and A n X n D! N (0; AV A 0 ). Also, if we set = 0, then part (a) implies Y n P! 0 and X n D! X ) Y n + X n D! X: Let Z n = Y n + X n, so Y P p n! 0 (i.e. Z n X n! 0) implies that the asymptotic p distribution of Z n is the same as that of X n. When Z n X n! 0 we sometimes say that the two sequences are asymptotically equivalent (Z n X n ) and write a Z n = X n + o P where o P is some suitable random variable ( Y n here) that converges to zero in probability. We employ the lemma in a standard trick to derive the limit distribution of a sequence of random variables. Suppose we wish to nd the limit distribution of Y nx 0 n, where we know X D n! X and Y P n!. Then we replace Y n with, and determine the limit distribution of 0 X n. Why does this work? Because Y nx 0 n a 0 X n, which is more easily seen in the equivalent statement Y 0 nx n = 0 X n + o P

5 where o P here is Yn 0 Xn. To test nonlinear hypotheses, given the asymptotic distribution of the estimator, we need Lemma (delta method): Let fb n g n1 be a sequence of K-dimensional random variables such that B n P! n 1 2 (Bn ) D! N (0; V ) : Let g () : R K! R Q have continuous rst derivatives, with G () denoting the Q K matrix of rst derivatives evaluated at : G () 0 : Then n 1 2 (g (Bn ) g ()) D! N 0; G () V G () 0 : Proof: A mean-value expansion yields n 1 2 g (B n ) = n 1 2 g ()+n 1 2 G (B ) (B n ), where Bn is the mean value between B n and. Because B P n!, Bn P! and, by continuity, G (Bn)! P G (). Thus n 1 2 [g (Bn ) g ()] = n 1 2 G (B n ) (B n ) D! G () N (0; V ) : Viewing Estimators as Sequences of Random Variables Let B n be an estimator of. If B P n!, then B n is consistent for. The asymptotic bias of an estimator is not an agreed upon concept. common de nition is lim E (B n) ; The most so consistency and asymptotic unbiasedness are distinct concepts. Hayashi uses p lim B n, so consistency and asymptotic unbiasedness are identical for him. If, in addition, n 1 2 (B n )! D N (0; V ), then B n is asymptotically Gaussian (and n 1 2 consistent, hence CAN). The asymptotic variance is V.

6 Laws of Large Numbers and Central Limit Theorems To verify convergence in probability from basic principles, we would need to know the joint distribution of the rst n elements in the sequence. To verify convergence almost surely, we would need to know the joint distribution of the entire (in nite dimensional) sequence. To verify convergence in quadratic mean, we would need to know the rst two moments of the nite sample distribution of the estimator. In general, we do not know any of these distributions. Instead, we turn to laws of large numbers. Laws of large numbers are one method of establishing convergence in probability (or almost surely) to a constant. They are applied to sequences of random variables, for which each member of the sequence is an (increasing) sum of random variables, such as Yn. An LLN is strong if the convergence is almost sure and weak if the convergence is in probability. Chebychev s Weak LLN: " lim E Y n = ; lim V ar Y n = 0" ) Yn p! : The following strong LLN assumes that fy t g is i.i.d. but the variance does not need to be nite Kolmogorov s Second Strong LLN: Let fy t g be i.i.d. with EY t =. a:s: Y n!. Then Central limit theorems are a principal method of establishing convergence in distribution, as they govern the limit behavior of the di erence between Y n and EY n (which equals EY t if fy t g is i.i.d.) blown up by p n. For i.i.d. sequences, the only CLT we need is Lindberg-Levy CLT: Let fy t g be i.i.d. with EY t = and V ar (Y t ) =. Then p n Yn = p 1 nx (Y t )! d N (0; ) : n t=1 We read the above CLT as: a sequence of random vectors p n Y n converges in distribution to a random vector whose distribution is N (0; ). To understand how to relate the above CLT for random vectors, with the CLT for random scalars, let be any vector of real numbers with the same dimension as Y t. Now f 0 Y t g is a sequence of random variables with E ( 0 Y t ) = 0 and V ar ( 0 Y t ) = 0. The scalar version of Lindberg-Levy implies p n 0 Yn 0 = 0p n Y n d! N (0; 0 ) :

7 Yet this limit distribution is the distribution of 0 Z, where Z N (0; ), hence the multivariate convergence in distribution theorem establishes the multivariate Lindberg-Levy CLT.