Elements of Probability Theory

Transcription

1 Chapter 5 Elements of Probability Theory The purpose of this chapter is to summarize some important concepts and results in probability theory. Of particular interest to us are the limit theorems which are powerful tools to analyze the convergence behaviors of econometric estimators and test statistics. These properties are the core of the asymptotic analysis in subsequent chapters. For a more complete and thorough treatment of probability theory see Davidson (1994) and other probability textbooks, such as Ash (1972) and Billingsley (1979). Bierens (1994), Gallant (1997) and White (2001) also provide concise coverages of the topics in this chapter. Many results here are taken freely from the references cited above; we will not refer to them again in the text unless it is necessary. 5.1 Probability Space and Random Variables Probability Space The probability space associated with a random experiment is determined by three components: the outcome space Ω whose element ω is an outcome of the experiment, a collection of events F whose elements are subsets of Ω, and a probability measure IP assigned to the elements in F. Given the subset A of Ω, its complement is defined as A c = {ω Ω: ω A}. Inthe probability space (Ω, F, IP), F is a σ-algebra (σ-field) in the sense that it satisfies the following requirements. 1. Ω F. 2. If A F,thenA c F. 3. If A 1,A 2,... are in F, then n=1 A n F. 111

2 112 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY The first and second properties together imply that Ω c = is also in F. Combining the second and third properties we have from de Morgan s law that ( ) c A n = A c n F. n=1 n=1 A σ-algebra is thus closed under complementation, countable union and countable intersection. The probability measure IP : F [0, 1] is a real-valued set function satisfying the following axioms. 1. IP(Ω) = IP(A) 0 for all A F. 3. if A 1,A 2,... F are disjoint, then IP( n=1 A n )= n=1 IP(A n ). From these axioms we easily deduce that IP( ) =0,IP(A c )=1 IP(A), IP(A) IP(B) if A B, and IP(A B) =IP(A)+IP(B) IP(A B). Moreover, if {A n } is an increasing (decreasing) sequence in F with the limiting set A, then lim n IP(A n )=IP(A). Let C be a collection of subsets of Ω. The intersection of all the σ-algebras that contain C is the smallest σ-algebra containing C; see Exercise 5.1. This σ-algebra is referred to as the σ-algebra generated by C, denoted as σ(c). When Ω = R, theborel field is the σ-algebra generated by all open intervals (a, b) inr, usually denoted as B d. Note that open intervals, closed intervals [a, b], half-open intervals (a, b] orhalf lines (,b] can be obtained from each other by taking complement, union and/or intersection. For example, (a, b] = n=1 ( a, b + 1 ), (a, b) = n n=1 ( a, b 1 ]. n Thus, the collection of all closed intervals (half-open intervals, half lines) generates the same Borel field. This is why open intervals, closed intervals, half-open intervals and half lines are also known as Borel sets. TheBorelfieldonR d, denoted as B d, is generated by all open hypercubes: (a 1,b 1 ) (a 2,b 2 ) (a d,b d ).

3 5.1. PROBABILITY SPACE AND RANDOM VARIABLES 113 Equivalently, B d can be generated by all closed hypercubes: [a 1,b 1 ] [a 2,b 2 ] [a d,b d ], or by (,b 1 ] (,b 2 ] (,b d ]. The sets that generate the Borel field B d are all Borel sets Random Variables A random variable z defined on (Ω, F, IP) is a function z :Ω R such that for every B in the Borel field B, itsinverse image of B is in F, i.e., z 1 (B) ={ω : z(ω) B} F. We also say that z is a F/B-measurable (or simply F-measurable) function. Nonmeasurable functions are very exceptional in practice and hence are not of general interest. Given the random outcome ω, the resulting value z(ω) isknownasarealization of z. The realization of z varies with ω and hence is governed by the random mechanism of the underlying experiment. A R d -valued random variable (random vector) z defined on (Ω, F, IP) is a function z :Ω R d such that for every B B d, z 1 (B) ={ω : z(ω) B} F; that is, z is a F/B d -measurable function. Given the random vector z, its inverse images z 1 (B) formaσ-algebra, denoted as σ(z). This σ-algebra must be in F, and it is the smallest σ-algebra contained in F such that z is measurable. This is known as the σ-algebra generated by z or, more intuitively, the information set associated with z. A function g : R R is said to be B-measurable or Borel measurable if {ζ R: g(ζ) b} B. If z is a random variable defined on (Ω, F, IP), then g(z) is also a random variable defined on the same probability space provided that g is Borel measurable. Note that the functions we usually encounter (e.g., continuous functions and integrable functions) are Borel measurable. Similarly, for the d-dimensional random vector z, g(z) isa random variable provided that g is B d -measurable.

4 114 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Recall from Section 2.1 that the joint distribution function ofz is the non-decreasing, right-continuous function F z such that for ζ =(ζ 1... ζ d ) R d, F z (ζ) =IP{ω Ω: z 1 (ω) ζ 1,...,z d (ω) ζ d }, with lim F z (ζ) =0, lim ζ 1,..., ζ d F z (ζ) =1. ζ 1,..., ζ d The marginal distribution function of the i th component of z is such that F zi (ζ i )=IP{ω Ω: z i (ω) ζ i } = F z (,...,,ζ i,,..., ). Note that while IP is a set function defined on F, the distribution function of z is a point function defined on R d. Two random variables y and z are said to be (pairwise) independent if, and only if, for any Borel sets B 1 and B 2, IP(y B 1 and z B 2 )=IP(y B 1 )IP(z B 2 ). This immediately leads to the standard definition of independence: y and z are independent if, and only if, their joint distribution is the product of their marginal distributions, as in Section 2.1. A sequence of random variables {z i } is said to be totally independent if ( ) IP {z i B i } = IP(z i B i ), all i all i for any Borel sets B i. In what follows, a totally independent sequence will be referred to an independent sequence or a sequence of independent variables for convenience. For an independent sequence, we have the following generalization of Lemma 2.1. Lemma 5.1 Let {z i } be a sequence of independent random variables and h i, i = 1, 2,..., be Borel-measurable functions. Then {h i (z i )} is also a sequence of independent random variables Moments and Norms The expectation of the i th element of z is IE(z i )= z i (ω) dip(ω), Ω

5 5.1. PROBABILITY SPACE AND RANDOM VARIABLES 115 where the right-hand side is a Lebesgue integral. In view of the distribution function defined above, a change of ω causes the realization of z to change so that IE(z i )= ζ i df z (ζ) = ζ i df zi (ζ i ), R d R where F zi is the marginal distribution function of the i th component of z, as defined in Section 2.2. For the Borel measurable function g of z, IE[g(z)] = g(z(ω)) d IP(ω) = g(ζ) df z (ζ). Ω R d Other moments, such as variance and covariance, can also be defined as Lebesgue integrals with respect to the probability measure; see Section 2.2. A function g is said to be convex on a set S if for any a [0, 1] and any x, y in S, g ( ax +(1 a)y ) ag(x)+(1 a)g(y); g is concave on S if the inequality above is reversed. For example, g(x) =x 2 is convex, and g(x) = logx for x > 0 is concave. The result below is concerned with convex (concave) transformations of random variables. Lemma 5.2 (Jensen) For the Borel measurable function g that is convex on the support of the integrable random variable z, suppose that g(z) is also integrable. Then, g(ie(z)) IE[g(z)]; the inequality reverses if g is concave. For the random variable z with finite p th moment, let z p =[IE(z p )] 1/p denote its L p -norm. Also define the inner product of two square integrable random variables z i and z j as their cross moment: z i,z j =IE(z i z j ). Then, L 2 -norm can be obtained from the inner product as z i 2 = z i,z i 1/2. It is easily seen that for any c>0andp>0, c p IP( z c) =c p 1 {ζ: ζ c} df z (ζ) ζ p df z (ζ) IE z p, {ζ: ζ c} where 1 {ζ: ζ c} is the indicator function which equals one if ζ c and equals zero otherwise. This establishes the following result.

6 116 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Lemma 5.3 (Markov) Let z be a random variable with finite p th moment. Then, IE z p IP( z c) c p, where c is a positive real number. For p =2,Lemma5.3isalsoknownastheChebyshev inequality. Ifc is small such that IE z p /c p > 1, Markov s inequality is trivial. When c becomes large, the probability that z assumes very extreme values will be vanishing at the rate c p. Another useful result in probability theory is stated below without proof. Lemma 5.4 (Hölder) Let y be a random variable with finite p th moment (p >1) and z a random variable with finite q th moment (q = p/(p 1)). Then, IE yz y p z q. For p =2,wehaveIE yz y 2 z 2. By noting that IE(yz) < IE yz, we immediately have the next result; cf. Lemma 2.3. Lemma 5.5 (Cauchy-Schwartz) Let y and z be two square integrable random variables. Then, IE(yz) y 2 z 2. Let y =1andx = z p. Then for q>pand r = q/p, Hölder s inequality also ensures that IE z p x r y r/(r 1) =[IE(z pr )] 1/r =[IE(z q )] p/q. This shows that when a random variable has finite q th moment, it must also have finite p th moment for any p<q, as stated below. Lemma 5.6 (Liapunov) Let z be a random variable with finite q th moment. Then for p<q, z p z q. The inequality below states that the L p -norm of a finite sum is less than the sum of individual L p -norms.

7 5.2. CONDITIONAL DISTRIBUTIONS AND MOMENTS 117 Lemma 5.7 (Minkowski) Let z i, i =1,...,n, be random variables with finite p th moment (p 1). Then, n n z i z i p. i=1 p i=1 When there are only two random variables in the sum, this is just the triangle inequality for L p -norms; see also Exercise Conditional Distributions and Moments Given two events A and B in F, ifitisknownthatb has occurred, the outcome space is restricted to B, so that the outcomes of A must be in A B. The likelihood of A is thus characterized by the conditional probability IP(A B) =IP(A B)/ IP(B), for IP(B) 0. ItcanbeshownthatIP( B) satisfies the axioms for probability measures; see Exerise 5.4. This concept is readily extended to construct conditional density function and conditional distribution function Conditional Distributions Let y and z denote two integrable random vectors such that z has the density function f z.forf y (η) 0, define the conditional density function of z given y = η as f z y (ζ y = η) = f z,y(ζ, η), f y (η) which is clearly non-negative whenever it is defined. This function also integrates to one on R d because f z y (ζ y = η) dζ = 1 R d f y (η) f z,y (ζ, η) dζ = 1 R d f y (η) f y (η) =1. Thus, f z y is a legitimate density function. For example, the bivariate density function of two random variables z and y forms a surface on the zy-plane. By fixing y = η, we obtain a cross section (slice) under this surface. Dividing the joint density by the marginal density f y (η) amounts to adjusting the height of this slice so that the resulting area integrates to one.

8 118 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Given the conditional density function f z y,wehavefora B d, IP(z A y = η) = f z y (ζ y = η) dζ. A Note that this conditional probability is defined even when IP(y = η) maybezero. In particular, when A =(,ζ 1 ] (,ζ d ], we obtain the conditional distribution function: F z y (ζ y = η) =IP(z 1 ζ 1,...,z d ζ d y = η). When z and y are independent, the conditional density (distribution) simply reduces to the unconditional density (distribution) Conditional Moments Analogous to unconditional expectation, the conditional expectation of the integrable random variable z i given the information y = η is IE(z i y = η) = ζ i df z y (ζ i y = η); R the conditional expectation of the random vector z is IE(z y = η) which is defined elementwise. By allowing y to vary across all possible values η, we obtain the conditional expectation function IE(z y) whose value depends on η, the realization of y. Thus, IE(z y) is a function of y and hence also a random vector. More generally, the conditional expectation can be defined by taking a suitable σ-algebra as the conditioning set. Let G be a sub-σ-algebra of F. The conditional expectation IE(z G) is the integrable and G-measurable random variable satisfying IE(z G)dIP= z dip, G G. G G This definition basically says that the conditional expectation with respect to G is such that its weighted sum is the same as that of z over any G in G. Suppose that G is the trivial σ-algebra {Ω, }, i.e., the smallest σ-algebra that contains no extra information from any random vectors. For the conditional expectation with respect to the trivial σ-algebra, it is readily seen that it must be a constant c with probability one so as to be measurable with respect to {Ω, }. Then, IE(z) = z dip= c dip=c. Ω Ω

9 5.2. CONDITIONAL DISTRIBUTIONS AND MOMENTS 119 That is, the conditional expectation with respect to the trivial σ-algebra is the unconditional expectation IE(z). Consider now G = σ(y), the σ-algebra generated by y. We also write IE(z y) =IE[z σ(y)], which is interpreted as the prediction of z given all the information associated with y. Similar to unconditional expectations, conditional expectations are monotonic: if z x with probability one, then IE(z G) IE(x G) with probability one; in particular, if z is non-negative with probability one, then IE(z G) 0 with probability one. Moreover, If z is independent of y, thenie(z y) =IE(z). For example, when z is a constant vector c which is independent of any random variable, IE(z y) =c. The linearity result below is analogous to Lemma 2.2 for unconditional expectations. Lemma 5.8 Let z (d 1) andy (c 1) be integrable random vectors and A (n d) and B (n c) be non-stochastic matrices. Then with probability one, IE(Az + By G)=AIE(z G)+BIE(y G). If b (n 1) isanon-stochasticvector,ie(az + b G)=AIE(z G)+b with probability one. From the definition of conditional expectation, we immediately have IE[IE(z G)] = IE(z G)dIP= z dip=ie(z); Ω Ω this is known as the law of iterated expectations. This result also suggests that if conditional expectations are taken sequentially with respect to a collection of nested σ-algebras, only the smallest σ-algebra matters. For example, for k random vectors y 1,...,y k, IE[IE(z y 1,...,y k ) y 1,...,y k 1 ]=IE(z y 1,...,y k 1 ). A formal result is stated below; see Exercise 5.5. Lemma 5.9 (Law of Iterated Expectations) Let G and H be two sub-σ-algebras of F such that G H. Then for the integrable random vector z, IE[IE(z H) G]=IE[IE(z G) H]=IE(z G); in particular, IE[IE(z G)] = IE(z).

10 120 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY If z is G-measurable, all the information resulted from z is already contained in G so that z can be treated as known in IE(z G) and taken out from the conditional expectation. That is, IE(z G) = z with probability one. Hence, IE(zx G)=zIE(x G). In particular, z can be taken out from the conditional expectation when z itself is a conditioning variable. This result is generalized as follows. Lemma 5.10 Let z be a G-measurable random vector. Then for any Borel-measurable function g, IE[g(z)x G]=g(z)IE(x G), with probability one. Two square integrable random variables z and y are said to be orthogonal if their inner product IE(zy) = 0. This definition allows us to discuss orthogonal projection in the space of square integrable random vectors. Let z be a square integrable random variable and z be a G-measurable random variable. Then, by Lemma 5.9 (law of iterated expectations) and Lemma 5.10, IE [( z IE(z G) ) z ] =IE [IE [( z IE(z G) ) z G ]] =IE [ IE(z G) z IE(z G) z ] =0. That is, the difference between z and its conditional expectation IE(z G)mustbe orthogonal to any G-measurable random variable. It can then be seen that for any square integrable, G-measurable random variable z, IE(z z) 2 =IE[z IE(z G)+IE(z G) z] 2 =IE[z IE(z G)] 2 +IE[IE(z G) z] 2 IE[z IE(z G)] 2, where in the second equality the cross-product term vanishes because both IE(z G) and z are G-measurable and hence orthogonal to z IE(z G). That is, among all G-measurable random variables that are also square integrable, IE(z G) is the closest to z in terms of the L 2 -norm. This shows that IE(z G) is the orthogonal projection of z onto the space of all G-measurable, square integrable random variables.

11 5.2. CONDITIONAL DISTRIBUTIONS AND MOMENTS 121 Lemma 5.11 Let z be a square integrable random variable. Then IE[z IE(z G)] 2 IE(z z) 2, for any G-measurable random variable z. In particular, let G = σ(y), where y is a square integrable random vector. Lemma 5.11 implies that IE [ z IE ( z σ(y) )] 2 IE ( z h(y) ) 2, for any Borel-measurable function h such that h(y) is also square integrable. Thus, IE[z σ(y)] minimizes the L 2 -norm z h(y) 2, and its difference from z is orthogonal to any function of y that is also square integrable. We may then say that, given all the information generated from y, IE[z σ(y)] is the best approximation of z in terms of the L 2 -norm (or simply the best L 2 predictor). The conditional variance-covariance matrix of z given y is var(z y) =IE ( [z IE(z y)][z IE(z y)] y ) =IE(zz y) IE(z y)ie(z y). Similar to unconditional variance-covariance matrix, we have for non-stochastic matrices A and b, var(az + b y) =A var(z y) A, which is nonsingular provided that A has full row rank and var(z y) is positive definite. It can also be shown that var(z) =IE[var(z y)] + var ( IE(z y) ) ; see Exercise 5.6. That is, the variance of z can be expressed as the sum of two components: the mean of its conditional variance and the variance of its conditional mean. This is also known as the decomposition of analysis of variance. Example 5.12 Suppose that (y x ) is distributed as a multivariate normal random vector: [ ] ([ ] [ ]) y μ N y Σ, y Σ xy. x μ x Σ xy Σ x

12 122 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY It is easy to see that the conditional density function of y given x, obtained from dividing the multivariate normal density function of y and x by the normal density of x, is also normal with the conditional mean IE(y x) =μ y Σ xyσ 1 x (x μ x ), and the conditional variance-covariance matrix var(y x) =var(y) var ( IE(y x) ) = Σ y Σ xyσ 1 x Σ xy. Note that IE(y x) is a linear function of x and that var(y x) doesnotvarywithx. 5.3 Modes of Convergence Consider now a sequence of random variables {z n (ω)} n=1,2,... defined on the probability space (Ω, F, IP). For a given ω, {z n } is a realization (a sequence of sample values) of the random element ω with the index n, and that for a given n, z n is a random variable which assumes different values depending on ω. In this section we will discuss various modes of convergence for sequences of random variables Almost Sure Convergence We first introduce the concept of almost sure convergence (convergence with probability one). Suppose that {z n } is a sequence of random variables and z is a random variable, all defined on the probability space (Ω, F, IP). The sequence {z n } is said to converge to z almost surely if, and only if, IP(ω : z n (ω) z(ω) asn )=1, a.s. denoted as z n z or z n z a.s. Note that for a given ω, the realization z n (ω) may or may not converge to z(ω). Almost sure convergence requires that z n (ω) z(ω) for almost all ω Ω, except for those ω in a set with probability zero. That is, almost all the realizations z n (ω) will be eventually close to z(ω) for all n sufficiently large; the event that z n will not approach z is improbable. When z n and z are both R d -valued, almost sure convergence is defined elementwise. That is, z n z a.s. if every element of z n converges almost surely to the corresponding element of z. The following result shows that continuous transformation preserves almost sure convergence. Lemma 5.13 Let g : R R be a function continuous on S g R.

13 5.3. MODES OF CONVERGENCE 123 a.s. [a] If z n z, wherez is a random variable such that IP(z S g )=1,theng(z n ) a.s. g(z). [b] If z n a.s. c, wherec is a real number at which g is continuous, then g(z n ) a.s. g(c). Proof: Let Ω 0 = {ω : z n (ω) z(ω)} and Ω 1 = {ω : z(ω) S g }.Thus,forω (Ω 0 Ω 1 ), continuity of g ensures that g(z n (ω)) g(z(ω)). Note that (Ω 0 Ω 1 ) c =Ω c 0 Ωc 1, which has probability zero because IP(Ω c 0 )=IP(Ωc 1 ) = 0. It follows that Ω 0 Ω 1 has probability one. This proves that g(z n ) g(z) with probability one. The second assertion is just a special case of the first result. z n Lemma 5.13 is easily generalized to R d -valued random variables. a.s. z implies For example, z 1,n + z 2,n z 1,n z 2,n z 2 1,n + z 2 2,n a.s. z 1 + z 2, a.s. z 1 z 2, a.s. z z 2 2, where z 1,n,z 2,n are two elements of z n and z 1,z 2 are the corresponding elements of z. Also, provided that z 2 0 with probability one, z 1,n /z 2,n z 1 /z 2 a.s Convergence in Probability A convergence concept that is weaker than almost sure convergence is convergence in probability. A sequence of random variables {z n } is said to converge to z in probability if for every ɛ>0, lim IP(ω : z n n (ω) z(ω) >ɛ)=0, or equivalently, lim IP(ω : z n n (ω) z(ω) ɛ) =1, IP denoted as z n z or z n z in probability. We also say that z is the probability limit of z n, denoted as plim z n = z. In particular, if the probability limit of z n is a constant c, all the probability mass of z n will concentrate around c when n becomes

14 124 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY large. For R d -valued random variables z n and z, convergence in probability is also defined elementwise. In the definition of convergence in probability, the events Ω n (ɛ) ={ω : z n (ω) z(ω) ɛ} vary with n, and convergence is referred to the probabilities of such events: p n =IP(Ω n (ɛ)), rather than the random variables z n. By contrast, almost sure convergence is related directly to the behaviors of random variables. For convergence in probability, the event Ω n that z n will be close to z becomes highly likely when n tends to infinity, or its complement (z n will deviate from z by a certain distance) becomes highly unlikely when n tends to infinity. Whether z n will converge to z is not of any concern in convergence in probability. More specifically, let Ω 0 denote the set of ω such that z n (ω) convergestoz(ω). For ω Ω 0,thereissomem such that ω is in Ω n (ɛ) for all n>m.thatis, Ω 0 Ω n (ɛ) F. m=1 n=m As n=m Ω n (ɛ) isalsoinf and non-decreasing in m, it follows that ( ) ( ) IP(Ω 0 ) IP Ω n (ɛ) = lim IP Ω m n (ɛ) m=1 n=m n=m lim m IP( Ω m (ɛ) ). This inequality proves that almost sure convergence implies convergence in probability, but the converse is not true in general. We state this result below. Lemma 5.14 If z n a.s. IP z, thenz n z. The following well-known example shows that when there is convergence in probability, the random variables themselves may not even converge for any ω. Example 5.15 Let Ω = [0, 1] and IP be the Lebesgue measure (i.e., IP{(a, b]} = b a for (a, b] [0, 1]). Consider the sequence {I n } of intervals [0, 1], [0, 1/2), [1/2, 1], [0, 1/3), [1/3, 2/3), [2/3, 1],..., and let z n = 1 In be the indicator function of I n : z n (ω) =1if ω I n and z n = 0 otherwise. When n tends to infinity, I n shrinks toward a singleton which has the Lebesgue measure zero. For 0 <ɛ<1, we then have IP( z n >ɛ)=ip(i n ) 0, IP which shows z n 0. On the other hand, it is easy to see that each ω [0, 1] must be covered by infinitely many intervals. Thus, given any ω [0, 1], z n (ω) = 1 for

15 5.3. MODES OF CONVERGENCE 125 infinitely many n, and hence z n (ω) does not converge to zero. Note that convergence in probability permits z n to deviate from the probability limit infinitely often, but almost sure convergence does not, except for those ω in the set of probability zero. Intuitively, when z n has finite variance such that var(z n ) vanishes asymptotically, the distribution of z n would shrink toward its mean IE(z n ). If, in addition, IE(z n ) tends to a constant c (or IE(z n ) = c), then z n ought to be degenerate at c in the limit. These observations suggest the following sufficient conditions for convergence in probability; see Exercises 5.7 and 5.8. In many cases, it is easier to establish convergence in probability by verifying these conditions. Lemma 5.16 Let {z n } be a sequence of square integrable random variables. If IE(z n ) IP c and var(z n ) 0, thenz n c. Analogous to Lemma 5.13, continuous functions also preserve convergence in probability. Lemma 5.17 Let g : R R be a function continuous on S g R. IP [a] If z n z, wherez is a random variable such that IP(z S g )=1,theng(z n ) IP g(z). [b] (Slutsky) If z n g(z n ) IP g(c). IP c, wherec is a real number at which g is continuous, then Proof: By the continuity of g, foreachɛ>0, we can find a δ>0 such that {ω : z n (ω) z(ω) δ} {ω : z(ω) S g } {ω : g(z n (ω)) g(z(ω)) ɛ}. Taking complementation of both sides and noting that the complement of {ω : z(ω) S g } has probability zero, we have IP( g(z n ) g(z) >ɛ) IP( z n z >δ). As z n converges to z in probability, the right-hand side converges to zero and so does the left-hand side.

16 126 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY z n Lemma 5.17 is readily generalized to R d -valued random variables. IP z implies For instance, z 1,n + z 2,n z 1,n z 2,n z 2 1,n + z 2 2,n IP z 1 + z 2, IP z 1 z 2, IP z z 2 2, where z 1,n,z 2,n are two elements of z n and z 1,z 2 are the corresponding elements of z. IP Also, provided that z 2 0 with probability one, z 1,n /z 2,n z 1 /z Convergence in Distribution Another convergence mode, known as convergence in distribution or convergence in law, concerns the behavior of the distribution functions of random variables. Let F zn and F z be the distribution functions of z n and z, respectively. A sequence of random variables D {z n } is said to converge to z in distribution, denoted as z n z, if lim F n z n (ζ) =F z (ζ), for every continuity point ζ of F z. That is, regardless the distributions of z n,convergence in distribution ensures that F zn will be arbitrarily close to F z for all n sufficiently large. The distribution F z is thus known as the limiting distribution of z n. We also say that A z n is asymptotically distributed as F z, denoted as z n Fz. D For random vectors {z n } and z, z n z if the joint distributions Fzn converge to F z for every continuity point ζ of F z. It is, however, more cumbersome to show convergence in distribution for a sequence of random vectors. The so-called Cramér- Wold device allows us to transform this multivariate convergence problem to a univariate one. This result is stated below without proof. Lemma 5.18 (Cramér-Wold Device) Let {z n } be a sequence of random vectors in R d D.Thenz n z if and only if α D z n α z for every α R d such that α α =1. There is also a uni-directional relationship between convergence in probability and convergence in distribution. To see this, note that for some arbitrary ɛ>0anda continuity point ζ of F z,wehave IP(z n ζ) =IP({z n ζ} { z n z ɛ})+ip({z n ζ} { z n z >ɛ}) IP(z ζ + ɛ)+ip( z n z >ɛ).

17 5.3. MODES OF CONVERGENCE 127 Similarly, IP(z ζ ɛ) IP(z n ζ)+ip( z n z >ɛ). IP If z n z, then by passing to the limit and noting that ɛ is arbitrary, the inequalities above imply lim IP(z n n ζ) =IP(z ζ). That is, F zn (ζ) F z (ζ). The converse is not true in general, however. When z n converges in distribution to a real number c, it is not difficult to show that z n also converges to c in probability. In this case, these two convergence modes are equivalent. To be sure, note that a real number c canbeviewedasadegenerate random variable with the distribution function: { 0, ζ < c, F (ζ) = 1, ζ c, which is a step function with a jump point at c. Whenz n c, all the probability mass IP of z n will concentrate at c as n becomes large; this is precisely what z n c means. More formally, for any ɛ>0, IP( z n c >ɛ)=1 [F zn (c + ɛ) F zn ((c ɛ) )], where (c ɛ) D denotes the point adjacent to and less than c ɛ. Now,z n c implies that F zn (c + ɛ) F zn ((c ɛ) ) converges to one, so that IP( z n c >ɛ)convergesto zero. We summarizes these results below. Lemma 5.19 If z n D c. z n IP z, thenz n D z. For a constant c, zn D IP c is equivalent to The continuous mapping theorem below asserts that continuous functions preserve convergence in distribution; cf. Lemmas 5.13 and Lemma 5.20 (Continuous Mapping Theorem) Let g : R R be a function continuous almost everywhere on R, except for at most countably many points. If z n z, D then g(z n ) D g(z). For example, if z n converges in distribution to the standard normal random variable, the limiting distribution of zn 2 is χ2 (1). Generalizing this result to R d -valued random variables, we can see that when z n converges in distribution to the d-dimensional standard normal random variable, the limiting distribution of z nz n is χ 2 (d).

18 128 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Two sequences of random variables {y n } and {z n } are said to be asymptotically equivalent if their differences y n z n converge to zero in probability. Intuitively, the limiting distributions of two asymptotically equivalent sequences, if exist, ought to be the same. This is stated in the next result without proof. Lemma 5.21 Let {y n } and {z n } be two sequences of random vectors such that y n IP D D z n 0. Ifz n z, thenyn z. The next result is concerned with two sequences of random variables such that one converges in distribution and the other converges in probability. Lemma 5.22 If y n converges in probability to a constant c and z n converges in distribution to z, then y n + z n c + z, yn z n cz, and zn /y n z/c if c D D D Stochastic Order Notations It is typical to use order notations to describe the behavior of a sequence of numbers, whether it converges or not. Let {c n } denote a sequence of positive real numbers. 1. Given a sequence {b n },wesaythatb n is (at most) of order c n, denoted as b n = O(c n ), if there exists a Δ < such that b n /c n Δ for all sufficiently large n. When c n diverges, b n cannot diverge faster than c n ;whenc n converges to zero, the rate of convergence of b n is no slower than that of c n. For example, the polynomial a + bn is O(n), and the partial sum of a bounded sequence n i=1 b i is O(n). Note that an O(1) sequence is a bounded sequence. 2. Given a sequence {b n },wesaythatb n is of smaller order than c n, denoted as b n = o(c n ), if b n /c n 0. When c n diverges, b n must diverge slower than c n ;when c n converges to zero, the rate of convergence of b n should be faster than that of c n. For example, the polynomial a + bn is o(n 1+δ ) for any δ>0, and the partial sum n i=1 αi, α < 1, is o(n). Note that an o(1) sequence is a sequence that converges to zero. If b n is a vector (matrix), b n is said to be O(c n )(o(c n )) if every element of b n is O(c n ) (o(c n )). It is also easy to verify the following results; see Exercise Lemma 5.23 Let {a n } and {b n } be two non-stochastic sequences.

19 5.5. LAW OF LARGE NUMBERS 129 (a) If a n = O(n r ) and b n = O(n s ),thena n b n = O(n r+s ) and a n + b n = O(n max(r,s) ). (b) If a n = o(n r ) and b n = o(n s ),thena n b n = o(n r+s ) and a n + b n = o(n max(r,s) ). (c) If a n = O(n r ) and b n = o(n s ),thena n b n = o(n r+s ) and a n + b n = O(n max(r,s) ). The order notations can be easily extended to describe the behavior of sequences of random variables. A sequence of random variables {z n } is said to be O a.s. (c n )(oro(c n ) almost surely) if z n /c n is O(1) a.s., and it is said to be O IP (c n )(oro(c n ) in probability) if for every ɛ>0, there is some Δ such that IP( z n /c n Δ) ɛ, for all n sufficiently large. Similarly, {z n } is o a.s. (c n )(oro(c n ) almost surely) if z n /c n IP 0, and it is o IP (c n )(oro(c n ) in probability) if z n /c n 0. a.s. If {z n } is O a.s. (1) (o a.s (1)), we say that z n is bounded (vanishing) almost surely; if {z n } is O IP (1) (o IP (1)), z n is bounded (vanishing) in probability. Note that Lemma 5.23 also holds for stochastic order notations. In particular, if a sequence of random variables is bounded almost surely (in probability) and another sequence of random variables is vanishing almost surely (in probability), the products of their corresponding elements are vanishing almost surely (in probability). That is, y n = O a.s. (1) and z n = o a.s (1), then y n z n is o a.s (1). D When z n z, we know that zn does not converge in probability to z in general, but more can be said about the behavior of z n.letζbeacontinuity point of F z.then for any ɛ>0, we can choose a sufficiently large ζ such that IP( z >ζ) <ɛ/2. As D z, we can also choose n large enough such that z n IP( z n >ζ) IP( z >ζ) <ɛ/2, which implies IP( z n >ζ) <ɛ. This leads to the following conclusion. Lemma 5.24 Let {z n } be a sequence of random vectors such that z n z n = O IP (1). D z. Then 5.5 Law of Large Numbers We first discuss the law of large numbers which is concerned with the averaging behavior of random variables. Intuitively, a sequence of random variables obeys a law of

20 130 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY large numbers when its sample average essentially follows its mean behavior; random irregularities (deviations from the mean) are wiped out in the limit by averaging. When a law of large numbers holds almost surely, it is a strong law of large numbers (SLLN); when a law of large numbers holds in probability, it is a weak law of large numbers (WLLN). For a sequence of random vectors (matrices), a SLLN (WLLN) is defined elementwise. There are different versions of the SLLN (WLLN) for various types of random variables. Below is a well known SLLN for i.i.d. random variables. Lemma 5.25 (Kolmogorov) Let {z t } be a sequence of i.i.d. random variables with mean μ o.then, 1 a.s. z T t μ o. This result asserts that, when z t have a finite, common mean μ o, the sample average of z t is essentially close to μ o, a non-stochastic number. Note, however, that i.i.d. random variables need not obey Kolmogorov s SLLN if they do not have a finite mean; for instance, Lemma 5.25 does not apply to i.i.d. Cauchy random variables. As almost sure convergence implies convergence in probability, the same condition in Lemma 5.25 ensures that {z t } also obeys a WLLN. When {z t } is a sequence of independent random variables with possibly heterogeneous distributions, it may still obey a SLLN (WLLN) under a stronger condition. Lemma 5.26 (Markov) Let {z t } be a sequence of independent random variables with non-degenerate distributions such that for some δ>0, IE z t 1+δ is bounded for all t. Then, 1 [z T t IE(z t )] a.s. 0, Comparing to Kolmogorov s SLLN, Lemma 5.26 requires a stronger moment condition: bounded (1+δ) th moment, yet z t need not have a common mean. This SLLN indicates that the sample average of z t eventually behaves like the average of IE(z t ). Note that the average of IE(z t ) may or may not converge; if it does converge to, say, μ, 1 a.s. 1 z T t lim IE(z T T t )=:μ.

21 5.5. LAW OF LARGE NUMBERS 131 Finally, as non-stochastic numbers can be viewed as independent random variables with degenerate distributions, it is understood that a non-stochastic sequence obeys a SLLN if its sample average converges. The following example shows that a sequence of correlated random variables may also obey a WLLN. Example 5.27 Suppose that y t is generated as a weakly stationary AR(1) process: y t = α o y t 1 + u t, α o < 1, where u t are i.i.d. random variables with mean zero and variance σu 2. In view of Section 4.3, we have IE(y t )=0,var(y t )=σu 2/(1 α2 o ), and σ 2 u cov(y t,y t j )=α j o 1 α 2. o These results imply that IE(T 1 T y t)=0and ( T ) 1 var y t = var(y t )+2 (T τ)cov(y t,y t τ ) var(y t )+2T τ=1 1 τ=1 cov(y t,y t τ ) = O(T ). ( The latter result shows that var T 1 ) T y t = O(T 1 ) which converges to zero as T approaches infinity. It follows from Lemma 5.16 that 1 IP y T t 0; that is, {y t } obeys a WLLN. It can be seen that a key condition in the proof above is that the variance of T y t does not grow too rapidly (it is O(T )). The facts that y t has a constant variance and that cov(y t,y t j ) goes to zero exponentially fast as j tends to infinity are sufficient for this condition. This WLLN result is readily generalized to weakly stationary AR(p) processes. The example above shows that it may be quite cumbersome to establish a WLLN for weakly stationary processes. The lemma below gives a strong law for correlated random variables and is convenient in practice; see Davidson (1994, p. 326) for a more general result.

22 132 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Lemma 5.28 Let y t = i=0 π i u t i,whereu t are i.i.d. random variables with mean zero and variance σu 2. If π i are absolutely summable, i.e., i= π i <, then T 1 T y a.s. t 0. In Example 5.27, y t = i=0 αi ou t i with α o < 1, so that i=0 αi o <. Hence, Lemma 5.28 ensures that the average of y t also converges to its mean (zero) almost surely. If y t = z t μ o, then the average of z t converges to IE(z t )=μ o almost surely. Comparing to Example 5.27, Lemma 5.28 is quite general and applicable to any process that can be expressed as an MA process with absolutely summable weights. From Lemmas 5.25, 5.26 and 5.28 we can see that a SLLN (WLLN) does not always hold. The random variables in a sequence must be well behaved (i.e., satisfying certain regularity conditions) to ensure a SLLN (WLLN). In particular, the sufficient conditions for a SLLN (WLLN) usually regulate the moments and dependence structure of random variables. Intuitively, random variables without certain bounded moment may exhibit aberrant behavior so that their random irregularities cannot be completely averaged out. For random variables with strong correlations over time, the variation of their partial sums may grow too rapidly and cannot be eliminated by simple averaging. More generally, it is also possible for a sequence of weakly dependent and heterogeneously distributed random variables to obey a SLLN (WLLN). This usually requires even stronger conditions on their moments and dependence structure. To avoid technicality, we will not give a SLLN (WLLN) for such general sequences but refer to White (2001) and Davidson (1994) for details. The following examples illustrate why a SLLN (WLLN) may fail to hold. Example 5.29 Consider the sequences {t} and {t 2 }, t =1, 2,...Itiswellknownthat t = T (T +1)/2, t 2 = T (T + 1)(2T +1)/6. Hence, T 1 T t and T 1 T t2 both diverge. Thus, these sequences do not obey a SLLN. Example 5.30 Suppose that u t are i.i.d. random variables with mean zero and variance σu.thus,t 2 1 T u a.s. t 0 by Kolmogorv s SLLN (Lemma 5.25). Consider now {tu t }.

23 5.5. LAW OF LARGE NUMBERS 133 This sequence does not have bounded (1 + δ) th moment because IE tu t 1+δ grows with t and therefore does not obey Markov s SLLN (Lemma 5.26). Moreover, note that ( T ) var tu t = t 2 var(u t )=σu 2 T (T + 1)(2T +1). 6 By Exercise 5.11, T tu t = O IP (T 3/2 ). It follows that T 1 T tu t = O IP (T 1/2 ) which diverges in probability. This shows that the sequence {tu t } does not obey a WLLN either. Example 5.31 Suppose that y t is generated as a random walk: y t = y t 1 + u t, t =1, 2,..., with y 0 =0,whereu t are i.i.d. random variables with mean zero and variance σu 2. Clearly, y t = t u i, i=1 which has mean zero and unbounded variance tσ 2 u.fors<t,write y t = y s + t i=s+1 u i = y s + v t s, where v t s = t i=s+1 u i is independent of y s.wethenhave cov(y t,y s )=IE(y 2 s )=sσ2 u, for t>s.consequently, ( T ) 1 var y t = var(y t )+2 τ=1 t=τ+1 cov(y t,y t τ ). It can be verified that the first term on the right-hand side is var(y t )= tσu 2 = O(T 2 ), and that the second term is 1 2 τ=1 t=τ+1 1 cov(y t,y t τ )=2 τ=1 t=τ+1 (t τ)σ 2 u = O(T 3 ).

24 134 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Thus, var( T y t )=O(T 3 ), so that T y t = O IP (T 3/2 ) by Exercise This shows that 1 T y t = O IP (T 1/2 ), which diverges in probability. This shows that when {y t } is a random walk, it does not obey a WLLN. In this case, y t have unbounded variances and strong correlations over time. Due to these correlations, the variation of the partial sum of y t grows much too fast. (Recall that the variance of T y t is only O(T ) in Example 5.27.) The conclusions above will not be altered when {u t } is a white noise or a weakly stationary process. Example 5.32 Suppose that y t is generated as a random walk: y t = y t 1 + u t, t =1, 2,..., with y 0 = 0, as in Example Then, the sequence {y t 1 u t } has mean zero and var(y t 1 u t )=IE(y 2 t 1) IE(u 2 t )=(t 1)σ 4 u. More interestingly, it can be seen that for s<t, cov(y t 1 u t,y s 1 u s )=IE(y t 1 y s 1 u s )IE(u t )=0. We then have ( T ) var y t 1 u t = var(y t 1 u t )= (t 1)σu 4 = O(T 2 ), and T y t 1 u t = O IP (T ). Note, however, that var(t 1 T y t 1 u t )convergesto σu 4/2, rather than 0. Thus, T 1 T y t 1 u t cannot behave like a non-stochastic number in the limit. This shows that {y t 1 u t } does not obey a WLLN, even though its partial sums are O IP (T ). In the asymptotic analysis of ecnometric estimators and test statistics, we usually encounter functions of several random variables, e.g., the product of two random variables. In some cases, it is easy to find sufficient conditions ensuring a SLLN (WLLN) for these functions. For example, suppose that z t = x t y t,where{x t } and {y t } are two mutually independent sequences of independent random variables, each with bounded (2 + δ) th moment. Then, z t are also independent random variables and have bounded

25 5.6. UNIFORM LAW OF LARGE NUMBERS 135 (1 + δ) th moment by the Cauchy-Schwartz inequality. Lemma 5.26 then provides the SLLN for {z t }.When{x t } and {y t } are two sequences of correlated (or weakly dependent) random variables, it is more cumbersome to find suitable conditions on x t and y t that ensure a SLLN (WLLN). if In what follows, a sequence of integrable random variables z t is said to obey a SLLN 1 T [z t IE(z t )] a.s. 0; (5.1) it is said to obey a WLLN if the almost sure convergence above is replaced by convergence in probability. When IE(z t )isaconstantμ o, (5.1) simplifies to 1 a.s. z T t μ o. In our analysis, we may only invoke this generic SLLN (WLLN). 5.6 Uniform Law of Large Numbers It is also common to deal with functions of random variables and model parameters. For example, q(z t (ω); θ) is a random variable for a given parameter θ, and it is a function of θ for a given ω. When θ is fixed, we may impose conditions on q and z t such that {q(z t (ω); θ)} obeys a SLLN (WLLN), as discussed in Section 5.5. When θ assumes values in the parameter space Θ, a SLLN (WLLN) that does not depend on θ is then needed. More specifically, suppose that {q(z t ; θ)} obeys a SLLN for each θ Θ: Q T (ω; θ) = 1 T q(z t (ω); θ) a.s. Q(θ), where Q(θ) is a non-stochastic function of θ. As this convergent behavior may depend on θ, Ω c 0 (θ) ={ω : Q T (ω; θ) Q(θ)} varies with θ. When Θ is an interval of R, θ Θ Ω c 0 (θ) is an uncountable union of non-convergence sets and hence may not have probability zero, even though each Ω c 0 (θ) does. Thus, the event that Q T (ω; θ) Q(θ) for all θ, i.e., θ Θ Ω 0 (θ), may occur with probability less than one. In fact, the union of all Ω c 0 (θ) maynotevenbeinf (only countable unions of the elements in F are guaranteed to be in F). If so, we cannot conclude anything regarding the convergence of Q T (ω; θ). Worse still is when θ also depends on T,asinthecasewhereθ is replaced

26 136 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY by an estimator θ T. There may not exist a finite T such that Q T (ω; θ T ) are arbitrarily close to Q(ω; θ T ) for all T>T. These observations suggest that we should study convergence that is uniform on the parameter space Θ. In particular, Q T (ω; θ) convergestoq(θ) uniformly in θ almost surely (in probability) if the largest possible difference: sup Q T (θ) Q(θ) 0, a.s. (in probability). θ Θ In what follows we always assume that this supremum is a random variables for all T. The example below, similar to Example 2.14 of Davidson (1994), illustrates the difference between uniform and pointwise convergence. Example 5.33 Let z t be i.i.d. random variables with zero mean and Tθ, 0 θ 1 2T, q T (z t (ω); θ) =z t (ω)+ 1 1 Tθ, 2T <θ 1 T, 1 0, T <θ<. Observe that for θ 1/T and θ =0, Q T (ω; θ) = 1 T q T (z t ; θ) = 1 T z t, which converges to zero almost surely by Kolmogorov s SLLN. Thus, for a given θ, we can always choose T large enough such that Q T (ω; θ) a.s. 0, where 0 is the pointwise limit. On the other hand, it can be seen that Θ = [0, ) and sup Q T (ω; θ) = z T +1/2 a.s. 1/2, θ Θ so that the uniform limit is different from the pointwise limit. Let z Tt denote the t th random variable in a sample of T variables. These random variables are indexed by both T and t and form a triangular array. In this array, there is only one random variable z 11 when T = 1, there are two random variables z 21 and z 22 when T = 2, there are three random variables z 31, z 32 and z 33 when T =3,andso on. If this array does not depend on T, it is simply a sequence of random variables. We now consider a triangular array of functions q Tt (z t ; θ), t =1, 2,...,T,wherez t are integrable random vectors and θ is the parameter vector taking values in the parameter

27 5.6. UNIFORM LAW OF LARGE NUMBERS 137 space Θ R m. For notation simplicity, we will not explicitly write ω in the functions. We say that {q Tt (z t ; θ)} obeys a strong uniform law of large numbers (SULLN) if sup θ Θ 1 T [q Tt (z t ; θ) IE(q Tt (z t ; θ))] a.s. 0, (5.2) cf. (5.1). Similarly, {q Tt (z t ; θ)} is said to obey a weak uniform law of large numbers (WULLN) if the convergence condition above holds in probability. If q Tt is R m -valued functions, the SULLN (WULLN) is defined elementwise. We have seen that pointwise convergence alone does not imply uniform convergence. An interesting question one would ask is: What are the additional conditions required to guarantee uniform convergence? Let Q T (θ) = 1 T Suppose that Q T and θ in Θ, [q Tt (z t ; θ) IE(q Tt (z t ; θ))]. satisfies the following Lipschitz-type continuity requirement: for θ Q T (θ) Q T (θ ) C T θ θ a.s., where denotes the Euclidean norm, and C T is a random variable bounded almost surely and does not depend on θ. Under this condition, Q T (θ ) can be made arbitrarily close to Q T (θ), provided that θ is sufficiently close to θ. Using the triangle inequality and taking supremum over θ we have sup Q T (θ) sup Q T (θ) Q T (θ ) + Q T (θ ). θ Θ θ Θ Let Δ denote an almost sure bound of C T.Thengivenanyɛ>0, choosing θ such that θ θ <ɛ/(2δ) implies sup Q T (θ) Q T (θ ɛ ) C T θ Θ 2Δ ɛ 2, uniformly in T.Moreover,becauseQ T (θ) converges to 0 almost surely for each θ in Θ, Q T (θ ) is also less than ɛ/2 for sufficiently large T.Consequently, sup Q T (θ) ɛ, θ Θ for all T sufficiently large. As these results hold almost surely, we have a SULLN for Q T (θ); the conditions ensuring a WULLN are analogous.

28 138 CHAPTER 5. ELEMENTS OF PROBABILITY THEORY Lemma 5.34 Suppose that for each θ Θ, {q Tt (z t ; θ)} obeys a SLLN (WLLN) and that for θ, θ Θ, Q T (θ) Q T (θ ) C T θ θ a.s., where C T is a random variable bounded almost surely (in probability) and does not depend on θ. Then,{q Tt (z t ; θ)} obeys a SULLN (WULLN). Lemma 5.34 is quite convenient for establishing a SULLN (WULLN) because it requires only two conditions. First, the random functions must obey a standard SLLN (WLLN) for each θ in the parameter space. Second, the function q Tt must satisfy a Lipschitz-type continuity condition which amounts to requiring q Tt to be sufficiently smooth in the second argument. Note, however, that C T being bounded almost surely may imply that the random variables in q Tt are also bounded almost surely. This requirement is much too restrictive in applications. Hence, a SULLN may not be readily obtained from Lemma On the other hand, a WULLN is practically more plausible because the requirement that C T is O IP (1) is much weaker. For example, the boundedness of IE C T is sufficient for C T being O IP (1) by Markov s inequality. For more specific conditions ensuring these requirements we refer to Gallant and White (1988) and Bierens (1994). 5.7 Central Limit Theorem Thecentral limit theorem (CLT) is another important result in probability theory. When a CLT holds, the distributions of suitably normalized averages of random variables are close to the standard normal distribution in the limit, regardless of the original distributions of these random variables. This is a very powerful result in applications because, as far as the approximation of normalized sample averages is concerned, only the standard normal distribution matters. There are also different versions of CLT for various types of random variables. The following CLT applies to i.i.d. random variables. Lemma 5.35 (Lindeberg-Lévy) Let {z t } be a sequence of i.i.d. random variables with mean μ o and variance σo 2 > 0. Then, T ( zt μ o ) D N (0, 1). σ o

29 5.7. CENTRAL LIMIT THEOREM 139 A sequence of i.i.d. random variables need not obey this CLT if they do not have a finite variance, e.g., random variables with t(2) distribution. Comparing to Lemma 5.25, one can immediately see that the Lindeberg-Lévy CLT requires a stronger condition (i.e., finite variance) than does Kolmogorov s SLLN. Remark: In this example, z T converges to μ o in probability, and its variance σo 2/T vanishes when T tends to infinity. To prevent a degenerate distribution in the limit, it is natural to consider the normalized average T 1/2 ( z T μ o ), which has a constant variance σo 2 for all T. This explains why the normalizing factor T 1/2 is needed. For a normalizing factor T a with a<1/2, the normalized average still converges to zero because its variance vanishes in the limit. For a normalizing factor T a with a>1/2, the normalized average diverges. In both cases, the resulting normalized averages cannot have a well-behaved, non-degenerate distribution in the limit. Thus, when {z t } obeys a CLT, z T is said to converge to μ o at the rate T 1/2. Independent random variables may also have the effect of a CLT. Below is a a version of Liapunov s CLT for independent (but not necessarily identically distributed) random variables. Lemma 5.36 Let {z Tt } be a triangular array of independent random variables with mean μ Tt and variance σtt 2 > 0 such that σ 2 T = 1 T σtt 2 σo 2 > 0. If for some δ>0, IE z Tt 2+δ are bounded for all t, then T ( zt μ T ) D N (0, 1). σ o Note that this result requires a stronger condition (bounded (2 + δ) th moment) than does Markov s SLLN, Lemma Comparing to Lindeberg-Lévy s CLT, Lemma 5.36 allows mean and variance to vary with t at the expense of a stronger moment condition. The sufficient conditions for a CLT are similar to but usually stronger than those for a WLLN. In particular, the random variables that obey a CLT have bounded moment up to some higher order and are asymptotically independent with dependence vanishing sufficiently fast. Moreover, every random variable must also be asymptotically negligible, in the sense that no random variable is influential in affecting the partial sums.