6.1 Moment Generating and Characteristic Functions

Transcription

1 Chapter 6 Limit Theorems The power statistics can mostly be seen when there is a large collection of data points and we are interested in understanding the macro state of the system, e.g., the average, or the sum of the random variables. To enable such abstraction, we need certain mathematical tools known as the limit theorems, in particular the law of large numbers and central limit theorem. 6. Moment Generating and Characteristic Functions Consider two independent random variables X and Y with PDFs f X (x) and f Y (y). If we are interested in finding the PDF of the sum, i.e., Z = X +Y, we know from the previous chapter that the PDF is the convolution of f X and f Y. However, convolution could be challenging to compute, especially when we have a large number of random variables to convolve. In this case, we can resort to some kind of frequency domain approach, which transforms the PDFs to another domain, and perform multiplication instead convolution to make the calculation easier. Moment Generating Function Definition. For any random variable X, the moment generating function (MGF) M X (s) is M X (s) = E [ e sx]. (6.) In the discrete case, MGF is equivalent to M X (s) = e sx p X (x), (6.2) x whereas in the continuous case, the MGF is equivalent to M X (s) = e sx f X (x)dx. (6.3)

2 From the continuous version, we can interpret MGF as the Laplace Transform of the PDF. Therefore, s is the variable in the transformed domain. If s = jω, then M X (jω) becomes the Fourier Transform of the PDF. Example. Find the MGF for a Poisson random variable. Solution. The MGF of Poisson random variable can be found as M X (s) = x=0 e sx λx e λ x! = (λe s ) x e λ = e λes e λ. x! x=0 Example 2. Find the MGF for an exponential random variable. Solution. The MGF of an exponential random variable can be found as M X (s) = Properties of MGF 0 e sx λe λx dx = 0 λe (s λ)x dx = λ, if λ > s. λ s Theorem. Let X and Y be two independent random variables. Let Z = X + Y. The MGF of Z is M Z (s) = M X (s)m Y (s). (6.4) Proof. By definition of MGF, we have that M Z (s) = E [ e s(x+y )] = E [ e sx] E [ e sy ] = M X (s)m Y (s), where the second equality holds because X and Y are independent. Implication. The implication of this proposition is that if we have a sequence of independent random variables X,..., X N, the MGF of Z = N n= X n is M Z (s) = N M Xn (s). n= If these random variables are further assumed to be identically distributed, the MGF can be simplified as M Z (s) = (M Xn (s)) N. This would significantly simplify the analysis of Z. c 207 Stanley Chan. All Rights Reserved. 2

3 Theorem 2. The MGF has the property that M X (0) = dk ds k M X (s) s=0 = E[X k ]. Proof. The first property can be proved by noting that The second property holds because Setting s = 0 yields d k ds k M X(s) = M X (0) = E[e 0X ] = E[] =. d k ds k esx f X (x)dx = x k e sx f X (x)dx. d k ds M X(s) k s=0 = x k f X (x)dx = E[X k ]. Example. Prove that a sum of i.i.d. Bernoulli is a Binomial random variable. Solution. Let us consider a sequence of i.i.d. Bernoulli R.V. X n Bernoulli(p) for n =,..., N. Let Z = X X N. The moment generating function of Z is M Z (s) = E[e sz ] = E[e s(x +...+X N ) ] = = N E[e sxn ] n= N ( pe s + ( p)e s0) = (pe s + ( p)) N. n= Now, let us check the moment generating function of a binomial R.V.: If Z Binomial(N, p), then N ( ) N M Z (s) = E[e sz ] = e sk p k ( p) N k k n=0 N ( ) N = (pe s ) k ( p) N k = (pe s + ( p)) N, k n=0 where the last equality holds because N n=0 ( N k ) a k b N k = (a + b) N. Therefore, we observe that the two moment generating functions are identical. c 207 Stanley Chan. All Rights Reserved. 3

4 Characteristic Function When we restrict s to the imaginary axis, i.e., s = jω, the moment generating function becomes the characteristic function Definition 2. The characteristic function of a random variable X is Φ X (jω) = E[e jωx ]. (6.5) However, we note that since ω can take any value in (, ), it does not matter if we consider E[e jωx ]. This leads to our ECE 302 definition of the characteristic function: Definition 3. The characteristic function of a random variable X (valid in ECE 302 only) is Φ X (jω) = E[e jωx ]. (6.6) The reason of introducing our own characteristic function is that E[e jωx ] is the Fourier transform of f X (x) but E[e jωx ] is the inverse Fourier transform of f X (x). The former is more convenient in notation for students who have taken ECE 30. However, historically the characteristic function is E[e jωx ]. Example. Let X and Y be independent, and let { λe λx, x 0 f X (x) = f Y (y) = 0, x < 0 Find the PDF of Z = X + Y. { λe λy, y 0 0, y < 0. Solution. The characteristic function of X and Y can be found from the Fourier table that Φ X (jω) = Φ Y (jω) = Therefore, the characteristic function of Z is λ λ + jω, λ λ + jω. Φ Z (jω) = Φ X (jω)φ Y (jω) = λ 2 (λ + jω) 2. By inverse Fourier transform, we have that { } f Z (z) = F λ 2 = λ 2 ze λz, z 0. (λ + jω) 2 c 207 Stanley Chan. All Rights Reserved. 4

5 Why Φ X (jω) but not M X (s)? The answer is that the moment generating function is not always defined. Recall that the expectation E[X] exists only when f X (x) is absolutely integrable, or E[ X ] <. For characteristics function, this is always valid because E[ e jωx ] = E[] =. However, for moment generating function, E[ e sx ] could possibly be unbounded. To see a counter example, we consider the Cauchy distribution. Example. Consider the Cauchy distribution with PDF Show that the MGF of X is undefined. f X (x) = Solution. It can be shown that the MGF is M X (s) = π(x 2 + ). e sx π(x 2 + ) dx e sx π(x 2 + ) dx (sx) 3 6π(x 2 + ) dx, because esx (sx)3 6 (sx) 3 s3 dx = xdx =. 6π(2x 2 ) 2π Therefore, the MGF is undefined. On the other hand, by Fourier table we know that { } Φ X (jω) = F = e ω. π(x 2 + ) 6.2 Probability Inequalities The second set of tools we need in studying limit theorems is the probability inequalities. We will introduce a few basic ones in this section. Theorem 3 (Union Bound). Let A,..., A n be a collection of sets. Then, [ n ] P A i i= n P[A i ]. (6.7) i= Proof. Without loss of generality let us assume that A A 2... A n. Construct a collection of disjoint sets i B i = A i A j. c 207 Stanley Chan. All Rights Reserved. 5 j=

6 Then, B i A i for all i =,..., n. Thus, we have that P[B i ] P[A i ] for all i. Now, because of the construction, B i s are all disjoint. By Axiom III, we can then have [ n ] [ n ] P A i = P B i = i= i= n P [B i ] i= n P [A i ]. i= Remark: The union bound is a very coarse upper bound but is easy to use. In a nutshell, union bound can be considered as a divide and conquer. For a system of n variables, we can decompose the system into small events, and use the union bound to upper limit the overall probability. If we can further ensure that each small event has a very tiny probability of happening, then we can have a good bound on the overall probability. Theorem 4 (Cauchy Schwarz Inequality). Let X and Y be two random variables. Then, E[XY ] 2 E[X 2 ]E[Y 2 ]. (6.8) Proof. Let f(s) = E[(sX + Y ) 2 ] for any real s. Then we can show that f(s) = E[X 2 ]s 2 + 2E[XY ]s + E[Y 2 ]. This is a quadratic equation, and it holds that f(s) 0 for all s because E[(sX + Y ) 2 ] 0. Therefore, the discriminant must be negative: (2E[XY ]) 2 4E[X 2 ]E[Y 2 ] 0. This implies that 4E[XY ] 2 4E[X 2 ]E[Y 2 ] 0, which completes the proof. Remark: The Cauchy-Schwarz inequality is useful in handling E[XY ], i.e., covariance if we assume X and Y are zero mean. In fact, we used Cauchy-Schwarz inequality to prove that the correlation coefficient ρ is bounded between - and. Theorem 5 (Markov Inequality). Let X 0 be a non-negative random variable. Then, for any ε > 0 we have P[X ε] E[X]. (6.9) ε Proof. Consider εp[x ε]. It holds that εp[x ε] = ε ε f X (x)dx ε xf X (x)dx. c 207 Stanley Chan. All Rights Reserved. 6

7 where the inequality holds because for any x ε, the integrand which is non-negative will always increase. It then follows that xf X (x)dx ε 0 xf X (x)dx = E[X]. Remark: The Markov inequality is useful in the sense that it relates the probability P[X ε] with the expectation. Typically, the probability P[X ε] could be difficult to evaluate if the PDF is complicated. The expectation, on the other hand, could be easier to evaluate. Theorem 6 (Chebyshev Inequality). Let X be a random variable with mean µ. Then, for any ε > 0 we have P[ X µ ε] Var[X] ε 2. (6.0) Proof. We apply Markov inequality to show that P[ X µ ε] = P[(X µ) 2 ε 2 ] E[(X µ)2 ] ε 2 = Var[X] ε 2. Remark: The Chebyshev inequality is a basic tool to prove the law of large number. It says that the probability of having X deviated significantly from µ, i.e., making ε big, is very small. Theorem 7 (Chernoff Inequality (Optional)). Let X be a random variable. Then, for any ε 0, we have that P[X ε] e f(ε), (6.) where and M X (s) is the moment generating function. Proof. Since e x is an increasing function, we have that f(ε) = min s {sε log M X (s)}, (6.2) P[X ε] = P[e sx e sε ] E[esX ] e sε = e sε M X (s) = e sε+log M X(s). c 207 Stanley Chan. All Rights Reserved. 7

8 The inequality holds due to Markov inequality, and we used the definition of MGF that E[e sx ] = M X (s). Now, note that this result holds for all s. That means it must also hold for the s that minimizes e sε+log MX(s). This implies that { P[X ε] min } e sε+log M X (s) s Again, since e x is increasing, the minimizer of the above probability is also the minimizer of the following function f(ε) = min {sε log M X (s)} s Thus, we conclude that. P[X ε] e f(ε). Example. Let X N (0, σ 2 /N). Find the Chernoff bound of P[X ε]. Solution. The MGF of a zero-mean Gaussian is Therefore, the function f can be shown as f(ε) = min s M X (s) = e σ2 s 2 2N. {sε log M X (s)} = min s {sε σ2 s 2 To minimize the function, we take derivative and set to zero. This yields } d {sε σ2 s 2 = 0 s = Nε ds 2N σ. 2 Substituting into f(ε) we can show that f(ε) = ε2 N, and hence 2σ { 2 } P[X ε] exp ε2 N. (6.3) 2σ 2 Therefore, as N, the probability P[X ε] drops to zero at an exponential decay speed. 6.3 Law of Large Number We are ready to discuss the statistical behavior of a sum of N random variables. Formally, let us define the sample mean as follows. Definition 4. The sample mean of a sequence of random variables X,..., X N is 2N }. M N = N N X n. n= c 207 Stanley Chan. All Rights Reserved. 8

9 Sample mean is an estimate of the true population mean. For example, by surveying 0,000 Americans we can find out the empirical mean of the age. As the survey size grows we will have greater confidence that the empirical mean will approach the true mean. Example. Consider an experiment of drawing random variables X,..., X N. We construct a sample average M N = N N n= X n, and we compute two quantities: E[M N ] and Var[M N ]. As N, we like to see if M N is converging to µ, and if Var[M N ] is converging to 0. Figure 6. below shows an empirical result probability number of trials Figure 6.: Illustration of law of large numbers. We now state and prove the weak law of large number. Theorem 8 (Weak Law of Large Number). Let X,..., X N be a sequence of i.i.d. random variables with common mean µ. Then, for any ε > 0, lim P [ M N µ > ε] = 0. (6.4) Proof. To prove the weak law of large number, we apply Chebyshev Inequality to show that Therefore, setting N we have P [ M N µ > ε] Var[M N] ε 2 = Var[X ] Nε 2. lim P [ M Var[X ] N µ > ε] = lim = 0. (6.5) Nε 2 c 207 Stanley Chan. All Rights Reserved. 9

10 Remark: The decay rate of the weak law can be improved by using Chernoff bound. For example, if X n N (0, σ 2 ), then we know that M N N (0, σ 2 /N). Therefore, Chernoff bound shows that { } P [ M N µ > ε] 2 exp ε2 N, 2σ 2 which is exponentially faster than Chebyshev. Convergence in Probability (Optional) The convergence type in WLLN is known as convergence in probability, formally defined as Definition 5. A sequence of random variables M,..., M N converges in probability to µ if lim P [ M N µ > ε] = 0. (6.6) We write M N p µ to denote convergence in probability. p So what is the difference between M N µ versus MN µ? We should remember that M N is a random variable. Therefore, it does not have a fixed specific value but a PDF which tells us the probability where a particular value appears. Since M N has multiple states and each state appears randomly, the classical notion of convergence M N µ is undefined. The p probabilistic convergence M N µ states that as N, the probability of having MN µ deviated significantly away from ε is arbitrarily small. Thus, where there is randomness in M N, the probability measure P( ) takes care of it. We have to emphasize that WLLN is weak because convergence in probability only specifies the low likelihood of having big deviation. Having a low chance does not mean that it will not happen. It is possible that from time to time, M N will still deviate significantly away from µ. The chance of this happening is just getting smaller and smaller as N grows. Almost Sure Convergence (Optional) If one really wants to seek a stronger type of convergence, we have to use the notion of almost sure convergence, defined as follows. Definition 6. A sequence of random variables M,..., M N converge almost surely to µ if [ ] P lim M N µ > ε = 0. (6.7) We write M N a.s. µ to denote almost sure convergence. Almost sure convergence provides a direct response to the fact that M N µ is undefined using the classical convergence. Since M N is a random variable, one has to seek a limiting c 207 Stanley Chan. All Rights Reserved. 0

11 object to which M N is converging. This limiting object is usually another random variable. After seeking the limiting object, we take the probability of the limiting object. Putting the limit inside and outside the probability has a drastic different meaning. In almost sure convergence, the probability is taken on the limiting object. There is no intermediate evaluation of the probability as N grows. If that limiting object has 0 probability of having big deviation, then there is no chance that it will enter a bad-luck experiment as in the convergence in probability case. Interestingly, the sample mean of a sequence of i.i.d. random variables X,..., X N can be shown to converge almost surely to µ. Theorem 9 (Strong Law of Large Number). Let X,..., X N be a sequence of i.i.d. random variables with common mean µ. Then, for any ε > 0, [ ] P lim M N µ > ε = 0. (6.8) The proof of the strong law of large number is beyond the scope of this course. Generally, we need some assumptions of the high order moments of X n to prove the strong law. 6.4 Central Limit Theorem The Law of Large Number provides a probabilistic way of measuring the mean of the sample mean M N. What about the PDF of M N? Can we say anything about the PDF as N? Convergence in Distribution (Optional) Before we provide an answer to this question, let us first discuss the concept of convergence in distribution. Definition 7. Let Z,..., Z N be a sequence of random variables with CDFs F Z,..., F ZN respectively. We say that Z,..., Z N converge in distribution to a random variable Z with CDF F Z if lim F Z N (z) = F Z (z), (6.9) for every continuous point z of F Z. We write Z N d Z to denote convergence in distribution. Convergence in distribution concerns about the CDFs of a sequence of random variables Z,..., Z N. If this sequence converges to a random variable Z in distribution, then the CDFs of Z,..., Z N evaluated at z should converge to the CDF of Z evaluated at z. There are a few points we should pay attention to: c 207 Stanley Chan. All Rights Reserved.

12 Why CDF but not PDF? As we discuss in Chapter 4, PDF is not defined for discrete random variables because delta functions are indeed not functions. However, the CDF is always defined because the CDF of discrete random variables are step functions. In order to include both discrete and continuous random variables, convergence in distribution is defined using the CDF. Why every continuous point of F Z? A discontinuous point of F Z only has its limit defined on one side. There are cases where we want to allow convergence without worry about the one-side limit. For example, let Z N be a random variable that P[Z N = /N] =. Then, as N, approaches a step function at 0. However, if we look at the CDF s value at z = 0, we can show that lim F ZN (0) = 0 but F Z (0) =. Therefore, if consider the discontinuous point at z = 0, we will not be able to allow convergence of Z N. For discrete random variables, continuous points of F Z are on the piecewise constant regions of the CDF. Is convergence in distribution stronger than convergence in probability? Convergence in distribution is actually weaker. Consider a continuous random variable X with a symmetric PDF f X (x) such that f X (x) = f X ( x). It holds that the PDF of X has the same PDF. If we define the sequence Z N = X if N is odd and Z N = X if N is even, and let Z = X, then F ZN (z) = F Z (z) for every z because the PDF of X and X are identical. Therefore, Z d p N Z. However, Z N Z because Z N oscillates between the random variables X and X. These two random variables are different (although they have the same CDF), because P[X = X] = P[{ω : X(ω) = X(ω)}] = P[{ω : X(ω) = 0}] = 0. Central Limit Theorem With the notion of convergence in distribution, we can now state the Central Limit Theorem. Theorem 0 (Central Limit Theorem). Let X,..., X N be a sequence of i.i.d. random variables of mean E[X n ] = µ and variance Var[X n ] = σ 2. Also, assume E[ Xn ] 3 <. Let def Z N = N ( M N ) µ. Then, σ where Z N (0, ). In other words, lim P[Z N z] = Φ(z) = Z N d Z, (6.20) z 2π e y2 /2 dy. (6.2) The powerfulness of the Central Limit Theorem is that the result holds for any distribution of X,..., X N. That is, regardless of the distribution of X,..., X N, the CDF of M N is approaching Gaussian. However, it should be very careful when we say CDF of M N c 207 Stanley Chan. All Rights Reserved. 2

13 approaches Gaussian. It does not mean that the random variable M N approaches to a Gaussian random variable. It only means that if we compute a probabilistic event of M N, the probability can be approximated by using a Gaussian. Remark: As a short hand notation, we write N ( M N ) µ d N (0, ) unless confusion σ arises. It also holds that the statement in the Central Limit Theorem is equivalent to d N (µ, σ 2 /N). M N Proof. Let Z N = N ( M N ) µ σ. Then, [ ( E[e sz N ] = E e s MN )] µ N σ = N n= N [ = E + s n= σ N (X n µ) + N [ = + s n= σ N E[X n µ] + ) N = ( + s2. 2N E [e s σ (Xn µ)] N s2 2σ 2 N (X n µ) 2 + O s2 2σ 2 N E [ (X n µ) 2] ] ( )] (Xn µ) 3 σ 3 N N ( ) N It remains to show that + s2 2N e s 2 /2. If we can show that, then we have shown that the MGF of Z N is also the MGF of N (0, ). To this end, we consider log( + x). By Taylor approximation, we have that Therefore, we have log( + x) log() + As N, the limit becomes lim N log ( ) ( ) d d 2 x 2 dx log x x= x + dx log x 2 x= 2 + O(x3 ). ) log ( + s2 s2 2N 2N s4 4N. 2 ) ( + s2 s2 2N 2 lim and so taking exponential on both sides yields ) N lim ( + s2 = e s2 2. 2N s 4 4N = s2 2, c 207 Stanley Chan. All Rights Reserved. 3

14 Example. Suppose X n Poisson(λ). Use Central Limit Theorem to approximate P[a M N b]. Solution. We first show that E[M N ] = E[X n ] = λ Var[M N ] = N Var[X n] = λ N Therefore, Central Limit Theorem implies that M N d N ( λ, λ N The probability can be found as ( ) ( ) b λ a λ P[a M N b] = Φ Φ. λ/n λ/n ). Theorem (Delta Method (Optional)). Let M N be the sample mean and suppose that Central Limit Theorem gives N(M N µ) d N (0, σ 2 ), where N is the sample size. Then, for any continuously differentiable function f, it holds that N(f(MN ) f(µ)) d N ( 0, σ 2 f (µ) 2). (6.22) Proof. By Taylor approximation, we can that that f(m N ) = f(µ) + (M N µ)f (µ) + O((M N µ) 2 ). This implies that N (f(mn ) f(µ)) = N (M N µ) f (µ). By Central Limit Theorem, we know that N (M N µ) f (µ) completes the proof. d N (0, σ 2 f (µ) 2 ). This Example. A Poisson random variable X n Poisson(λ) has a variance λ. Let f(λ) = λ. Show that N (f(m N ) f(λ)) d N (0, /4). That is, by applying the transformation f, we make the variance to a constant. Solution. Since f(λ) = λ, we have that (f (λ)) 2 = ( 2 λ /2) 2 = 4 λ. Thus, the Delta Method gives us N (f(m N ) f(λ)) d N (0, /4). c 207 Stanley Chan. All Rights Reserved. 4

15 Limitation of Central Limit Theorem (Optional) If we recall the statement of the Central Limit Theorem, we notice that the statement requires [ ( ) ] N lim P MN µ ε = Φ(ε). (6.23) σ Rearranging the terms we can show that [ lim P M N µ + σε ] = Φ(ε). N This implies that as N, the deviation which CLT can handle, i.e., σε N, goes to 0. In other words, CLT can only handle small deviations. If we want to conduct analysis of large deviations, we need to resort to tools such as Chernoff bound. Intuitive, we can understand the limitation of CLT by realizing the fact that the Gaussian (i.e., quadratic) approximation holds only for small deviations. The tail probabilities are typically not approximated well by CLT. c 207 Stanley Chan. All Rights Reserved. 5