Topic 7: Convergence of Random Variables

Transcription

1 Topic 7: Convergence of Ranom Variables Course 003, 2016 Page 0

2 The Inference Problem So far, our starting point has been a given probability space (S, F, P). We now look at how to generate information about the probability space by analyzing a sample of outcomes. This process is referre to as statistical inference. Inference proceures are parametric when we make assumptions about the probability space from which our sample is rawn (for example, each sample observation represents an outcome of a normally istribute ranom variable with unknown mean an unit variance). If we make no such assumptions our proceures are nonparametric. Our focus in this course is on parametric inference. This involves both the estimation of population parameters an testing hypotheses about them. Page 1

3 Defining a Statistic Definition: Any real-value function T = r(x 1,..., X n ) is calle a statistic. Notice that: a statistic is itself a ranom variable we ve consiere several functions of ranom variables, whose istributions are well efine such as : Y = X µ σ where X N(µ, σ 2 ). We showe that Y N(0, 1). Y = n X i, where each X i has a bernoulli istribution with parameter p was shown to have a binomial istribution with parameters n an p. Y = n X 2 i where each X i has a stanar normal istribution was shown to have a χ 2 n istribution etc... Only some of these functions of ranom variables are statistics. This istinction is important because statistics have sample counterparts. In a problem of estimating an unknown parameter, θ, our estimator will be a statistic whose value can be regare as an estimate of θ. It turns out that for large samples, the istributions of some statistics, such as the sample mean, are well-known. Page 2

4 Markov s Inequality We begin with some useful inequalities which provie us with istribution-free bouns on the probability of certain events an are useful in proving the law of large numbers, one of our two main large sample theorems. Markov s Inequality: Let X be a ranom variable with ensity function f(x) such that P(X 0) = 1. Then for any given number t > 0, P(X t) E(X) t Proof. (for iscrete istributions) E(X) = x xf(x) = x<t xf(x) + x t xf(x) All terms in these summations are non-negative by assumption, so we have E(X) xf(x) tf(x) = tp(x t) x t x t This inequality obviously hols for t E(X) (why?). Its main interest is in bouning the probability in the tails. For example, if the mean of X is 1, the probability of X taking values bigger than 100 is less than.01. This is true irrespective of the istribution of X- this is what makes the result powerful. Page 3

5 Chebyshev s Inequality This is a special case of Markov s inequality an relates the variance of a istribution to the probability associate with eviations from the mean. Chebyshev s Inequality: Let X be a ranom variable with finite variance σ 2 an mean µ. Then, for every t > 0, or equivalently, P( X µ t) σ2 t 2 P( X µ < t) 1 σ2 t 2 Proof. Use Markov s inequality with Y = (X µ) 2 an use t 2 in place of the constant t. Then Y takes only non-negative values an E(Y) = Var(X) = σ 2. In particular, this tells us that for any ranom variable, the probability that values taken by the variable will be more than 2 stanar eviations away from the mean cannot excee 1 4 an 3 stanar eviations away cannot excee 1 9 P( X µ 3σ) 1 9 For most istributions, this upper boun is consierably higher than the actual probability of this event. What are these probabilities for a Normal istribution? Page 4

6 Probability bouns..an example Chebyshev s Inequality can, in principle be use for computing bouns for the probabilities of certain events. In practice this is not often use because the bouns it provies are much larger than the actual probabilities of tail events as seen in the following example: 1 Let the ensity function of X be given by f(x) = (2 3) I ( 3, 3) (x). In this case µ = 0 an σ 2 = (b a)2 12 = 1. If t = 3 2, then Pr( X µ 3 2 ) = Pr( X 3 2 ) = x = =.13 Chebyshev s inequality gives us 1 t 2 = 4 9 while our boun is 1 4. which is much higher. If t = 2, the exact probability is 0, Page 5

7 The sample mean Recall the mean an variance of the sample mean: E(X n ) = 1 n n E(X i ) = n 1.nµ = µ Var(X n ) = 1 n 2 Var( n X i ) = 1 n 2 n X n = 1 n (X X n ) Var(X i ) = 1 n 2 nσ 2 = σ2 n So we know the following about the sample mean Its expectation is equal to that of the population. It is more concentrate aroun its mean value µ than was the original istribution. The larger the sample, the lower the variance of X n. Page 6

8 Sample size an precision of the sample mean We can use Chebyshev s Inequality to ask how big a sample we shoul take, if we want to ensure a certain level of precision in our estimate of the sample mean. Suppose the ranom sample is picke from a istribution which unknown mean an variance equal to 4 an we want to ensure an estimate which is within 1 unit of the real mean with probability.99. So we want Pr( X µ 1).01. Applying Chebyshev s Inequality we get Pr( X µ 1) n 4. Since we want n 4 n = 400. =.01 we take This calculation oes not use any information on the istribution of X an therefore often gives us a much larger number than we woul get if this information was available. That s the trae-off between the more efficient parametric proceures an more robust non-parametric ones. Page 7

9 Sample size an precision: Bernoulli example We want to pick a ranom sample size of size n from a Bernoulli istribution an use the sample mean to estimate the parameter p. How big shoul n be for the sample mean of (X 1,... X n ) to lie within.1 of the population mean with probability equal to k? If we on t use the istribution of X i an apply Chebyshev s Inequality, we get: P( X n µ.1) 1 σ2 Xn 1 or P( X n µ.1) 1 p(1 p) n Setting the RHS equal to k, we get: n = 100p(1 p) 1 k With σ 2 Xn = 4 1, (p = 1 2 ) an k =.7, we choose n = 84 so that our sample mean to lies in the interval [.4,.6], with probability equal to.7. What if p 1 2? Now let s use the fact that X i is Bernoulli. We want P(p.1 Xi n p +.1) k P(np.1n X i np +.1n) k When n = 15 an p =.5, np = 7.5 an for the Binomial istribution F(9) F(6) =.7 so we can use this sample size for the esire level of precision in our estimate of X n. Page 8

10 Convergence of Real Sequences We woul like our estimators to be well-behave. What oes this mean? One esirable property is that our estimates get closer to the parameter that we are trying to estimate as our sample gets larger. We re going to make precise this notion of getting closer. Recall that a sequence is just a function from the set of natural numbers N to any set A (Examples: y n = 2 n, y n = 1 n ) A real number sequence {y n } converges to y if for every ɛ > 0, there exists N(ɛ) for which n N(ɛ) = y n y < ɛ. In such as case, we say that {y n } y Which of the above sequences converge? If we have a sequence of functions {f n }, the sequence is sai to converge to a function f if f n (x) f(x) for all x in the omain of f. In the case of matrices, a sequence of matrices converges if each the sequences forme by (i, j) th elements converge, i.e. Y n [i, j] Y[i, j]. Page 9

11 Sequences of Ranom Variables A sequence of ranom variables is a sequence for which the set A is a collection of ranom variables an the function efining the sequence puts these ranom variables in a specific orer. n Examples: Y n = n 1 X i, where X i N(µ, σ 2 ), or Y n = n X i, where X i Bernoulli(p) We now nee to moify our notion of convergence, since the sequence {Y n } no longer efines a given sequence of real numbers, but rather, many ifferent real number sequences, epening on the realizations of X 1,..., X n. Convergence questions can no longer be verifie unequivocally since we are not referring to a given real sequence, but they can be assigne a probability of occurrence base on the probability space for ranom variables involve. There are several types of ranom variable convergence iscusse in the literature. We ll focus on two of these: Convergence in Distribution Convergence in Probability Page 10

12 Convergence in Distribution Definition: Let {Y n } be a sequence of ranom variables, an let {F n } be the associate sequence of cumulative istribution functions. If there exists a cumulative istribution function F such that F n (y) F(y) then F is calle the limiting CDF of {Y n }. Denoting by Y the r.v. with the istribution function F, we say that Y n converges in istribution to the ranom variable Y an enote this by Y n Y. The notation Y n F is also use to enote Y n Y F Convergence in istribution hols if there is convergence in the sequence of ensities (f n (y) f(y)) or in the sequence of MGFs (M Yn (t) M Y (t)). In some cases, it may be easier to use these to show convergence in istribution. Result:Let X n X, an let the ranom variable g(x) be efine by a function continous function g(.) Then g(x n ) g(x) Example: Suppose Z n Z N(0, 1), then 2Z n + 5 2Z + 5 N(5, 4) (why?) Page 11

13 Convergence in Probability This concept formalizes the iea that we can bring the outcomes of the ranom variable Y n arbitrarily close to the outcomes of the ranom variable Y for large enough n. Definition: The sequence of ranom variables {Y n } converges in probability to the ranom variable Y iff lim P( y n y < ɛ) = 1 ɛ > 0 n p We enote this by Y n Y or plim Y n = Y. This justifies using outcomes of Y as an approximation for outcomes of Y n since the two are very close for large n. Notice that while convergence in istribution is a statement about the istribution functions of Y n an Y whereas convergence in probability is a statement about the joint ensity of outcomes, y n an y. Distribution functions of very ifferent experiments may be the same: an even number on a fair ie an a hea on a fair coin have the same istribution function, but the outcomes of these ranom variables are unrelate. p Therefore Y n Y implies Y n p c an the two are equivalent. Y n Y. In the special case where Y n c, we also have Page 12

14 The Weak Law of Large Numbers Consier now the convergence of the ranom variable sequence whose n th term is given by: X n = 1 n n X i WLLN: Let {X n } be a sequence of i.i.. ranom variables with finite mean µ an variance σ 2. Then p µ. X n Proof. Using Chebyshev s Inequality, P( X µ < ɛ) 1 σ2 nɛ 2 Hence lim P( X µ < ɛ) = 1 or plim X = µ n The WLLN will allow us to use the sample mean as an estimate of the population mean, uner very general conitions. Page 13

15 Central Limit Theorems Central limit theorems specify conitions uner which sequences of ranom variables converge in istribution to known families of istributions. These are very useful in eriving asymptotic istributions of test statistics whose exact istributions are either cumbersome or ifficult to erive. There are a large number of theorems which vary by the assumptions they place on the ranom variables ( epenent or inepenent, ientically or non-inentically istribute). The Linberg-Levy CLT: Let {X n } be a sequence of i.i.. ranom variables with EX i = µ an var(x i ) = σ 2 (0, ) i. Then X n µ σ = n n(xn µ) σ N(0, 1) In wors: Whenever a ranom sample of size n (large) is taken from any istribution with mean µ an variance σ 2, the sample mean X n will have a istribution which is approximately normal with mean µ an variance σ2 n. Page 14

16 Linberg-Levy CLT..applications Approximating Binomial Probabilities via the Normal Distribution: Let {X n } be a sequence of i.i.. Bernoulli ranom variables. Then, by the LLCLT: Xi n p p(1 p) n = n X i np np(1 p) N(0, 1) an n a X i N(np, np(1 p)) In this case, n X i is of the form az n + b with a = np(1 p) an b = np an since Z n converges to Z in istribution, the asymptotic istribution n X i is normal with mean an variance given above (base on our results on normally istribute variables). Approximating χ 2 Probabilities via the Normal Distribution: Let {X n } be a sequence of i.i.. chi-square ranom variables with 1 egree of freeom. Using the aitivity property of variables with gamma istributions, we have n X i χ 2 n Recall that the mean of gamma istribution is α β an its variance is α β 2. For a χ 2 n ranom variable, α = v 2 an β = 1 2. Then, by the LLCLT: Xi n 1 2 n = n X i n 2n N(0, 1) an n a X i N(n, 2n) Page 15