1.6 Families of Distributions

Transcription

1 Your text 1.6. FAMILIES OF DISTRIBUTIONS 15 F(x) Density 0.10 cdf a b c 0.0 x Figure 1.1: N(4.5, 2) Distribution Function and Cumulative Distribution Function for 1.6 Families of Distributions Example Normal distributionn(µ,σ 2 ) The density function is given by f X (x) = 1 σ e (x µ) 2 2σ 2. (1.8) 2π There are two parameters which tell us about the position and the shape of the density curve. There is an extensive theory of statistical analysis for data which are realizations of normally distributed random variables. This distribution is most common in applications, but sometimes it is not feasible to assume that what we observe can indeed be a sample from such a population. In Example 1.6 we observe time to a specific response to a drug candidate. Such a variable can only take nonnegative values, while the normal rv s domain isr. A lognormal distribution is often used in such cases. X has a lognormal distribution if log X is normally distributed. Exercise 1.6. Recall the following discrete distributions: 1. UniformU(n); 2. Bernoulli(p); 3. Geom(p); 4. Hypogeometric(n, M, N);

2 16 CHAPTER 1. ELEMENTS OF PROBABILITY DISTRIBUTION THEORY 5. Poisson(λ). Exercise 1.7. Recall the following continuous distributions: 1. Uniform U(a, b); 2. Exp(λ); 3. Gamma(α, λ); 4. Chi-squared with ν degrees of freedom, χ 2 ν; Note that all the distributions depend on some parameters, likep,λ,µ,σ or other. These values are usually unknown, so their estimation is one of the important problems in statistical analyzes. These parameters determine some characteristics of the shape of the pdf/pmf of a random variable. It can be location, spread, skewness etc. We denote by P θ the distribution function of a rv X depending on the parameter θ (a scalar or an s-dimensional vector). Definition The distribution family is a set P = {P θ : θ Θ R s } defined on a common measurable space(ω,a). Example Assume that in the efficacy Example 1.9 we have 3 mice. Then the sample space Ω consists of triples of efficacious or non-efficacious responses, Ω = {(ω 1,ω 1,ω 1 ),(ω 1,ω 1,ω 2 ),...,(ω 2,ω 2,ω 2 )}. Take theσ-algebra A as the power set onω. The random variable defined as the number of successes inn = 3 trials has Binomial distribution with probability of success p. X : Ω {0,1,2,3}. Here p is the parameter which together with the distribution function defines the family on the common measurable space(ω,a). Forp= 1 we have the symmetric mass function 2

3 1.7. EXPECTED VALUES 17 X = x P(X = x) , while forp = 3 4 the asymmetric mass function X = x P(X = x) Both functions belong to the same family of binomial distributions withn = Expected Values Definition The expected value of a functiong(x) is defined by g(x)f(x)dx, for a continuous r.v., E(g(X)) = g(x j )p(x j ) for a discrete r.v., j=0 andg is any function such thate g(x) <. Two important special cases ofg are: Mean E(X), also denoted byex, when g(x) = X, Variance E(X EX) 2, when g(x) = (X EX) 2. The following relation is very useful while calculating the variance E(X EX) 2 = EX 2 (EX) 2. (1.9) Example Let X be a random variable such that 1 sinx, for x [0,π], f(x) = 2 0 otherwise. Then the expectation and variance are following

4 18 CHAPTER 1. ELEMENTS OF PROBABILITY DISTRIBUTION THEORY Expectation Variance EX = 1 2 π 0 0 xsinxdx = π 2. var(x) = EX 2 (EX) 2 = 1 π ( π 2 x 2 sinxdx 2 2) = π2 4. Example Geom(p) (the number of independent Bernoulli trials until first success ): The support set and the pmf are, respectively, X = {1, 2,...} and P(X = x) = p(1 p) x 1 = pq x 1, x X, wherep [0,1] is the probability of success, q = 1 p. E(X) = 1 1 p, var(x) =. p p 2 Here we will prove the formula for the expectation. By the definition of the expectation we have E(X) = xpq x 1 = p xq x 1. x=1 Note that forx 1 the following equality holds Hence, d dq (qx ) = xq x 1. x=1 E(X) =p xq x 1 d = p dq (qx ) x=1 x=1 ( = p d ) q x. dq The latter is the sum of a geometric sequence and is equal toq/(1 q). Therefore, E(X) = p d ( ) ( ) q 1 = p = 1 dq 1 q (1 q) 2 p. Similar method can be used to show that the var(x) = q/p 2 (second derivative with respect toq ofq x can be applied for this). x=1

5 1.8. MOMENTS AND MOMENT GENERATING FUNCTIONS 19 The following useful properties of the expectation follow from properties of integration (summation). Theorem 1.5. Let X be a random variable and let a, b and c be constants. Then for any functionsg(x) andh(x) whose expectations exist we have: (a) E[ag(X)+bh(X)+c] = ae[g(x)]+be[h(x)]+c; (b) Ifg(x) h(x) for allx, thene(g(x)) E(h(X)); (c) If g(x) 0 for all x, then E(g(X)) 0; (d) Ifa g(x) b for allx, thena E(g(X)) b. Exercise 1.8. Show thate(x b) 2 is minimized by b = E(X). Variance of a random variable together with the mean are the most important parameters used in the theory of statistics. The following theorem is a result of the properties of the expectation function. Theorem 1.6. If X is a random variable with a finite variance, then for any constants a and b, var(ax +b) = a 2 varx. Exercise 1.9. Prove Theorem Moments and Moment Generating Functions Definition The nth moment (n N) of a random variablex is defined as µ n = EXn Thenth central moment ofx is defined as µ n = E(X µ) n, where µ = µ 1 = EX.

6 20 CHAPTER 1. ELEMENTS OF PROBABILITY DISTRIBUTION THEORY Note, that the second central moment is the variance of a random variablex, usually denoted by σ 2. Moments give an indication of the shape of the distribution of a random variable. Skewness and kurtosis are measured by the following functions of the third and fourth central moment respectively: the coefficient of skewness is given by γ 1 = E(X µ)3 σ 3 = µ 3 µ 3 2 ; the coefficient of kurtosis is given by γ 2 = E(X µ)4 3 = µ 4 3. σ 4 µ 2 2 Moments can be calculated from the definition or by using so called moment generating function. Definition The moment generating function (mgf) of a random variable X is a functionm X : R [0, ) given by M X (t) = Ee tx, provided that the expectation exists for t in some neighborhood of zero. More explicitly, the mgf ofx can be written as M X (t) = M X (t) = x X e tx f X (x)dx, if X is continuous, e tx P(X = x)dx, if X is discrete. The method to generate moments is given in the following theorem. Theorem 1.7. If X has mgfm X (t), then E(X n ) = dn dt nm X(t) t=0, That is, the n-th moment is equal to the n-th derivative of the mgf evaluated at t = 0.

7 1.8. MOMENTS AND MOMENT GENERATING FUNCTIONS 21 Proof. Assuming that we can differentiate under the integral sign we may write d dt M X(t) = d dt = = ( d e tx f X (x)dx dt etx = E(Xe tx ). ) f X (x)dx ( xe tx ) f X (x)dx Hence, evaluating the last expression at zero we obtain Forn = 2 we will get d dt M X(t) 0 = E(Xe tx ) 0 = E(X). d 2 dt 2M X(t) 0 = E(X 2 e tx ) 0 = E(X 2 ). Analogously, it can be shown that for anyn N we can write d n dt nm X(t) 0 = E(X n e tx ) 0 = E(X n ). Example Find the mgf of X Exp(λ) and use results of Theorem 1.7 to obtain the mean and variance ofx. By definition the mgf can be written as M X (t) = E(e t X) = For the exponential distribution we have e tx f X (x)dx. f X (x) = λe λx I (0, ) (x), whereλ R +. Hence, integrating by the method of substitution, we get M X (t) = 0 e tx λe λx dx = λ 0 e (t λ)x dx = λ λ t provided that t < λ.

8 22 CHAPTER 1. ELEMENTS OF PROBABILITY DISTRIBUTION THEORY Now, using Theorem 1.7 we obtain the first and the second moments, respectively: Hence, the variance ofx is E(X) = M X (0) = λ (λ t) 2 t=0 = 1 λ, E(X 2 ) = M (2) 2λ X (0) = t=0 = 2 (λ t) 3 λ 2. var(x) = E(X 2 ) [E(X)] 2 = 2 λ 2 1 λ 2 = 1 λ 2. Exercise Calculate mgf for Binomial and Poisson distributions. Moment generating functions provide methods for comparing distributions or finding their limiting forms. The following two theorems give us the tools. Theorem 1.8. Let F X (x) and F Y (y) be two cdfs whose all moments exist. Then, if the mgfs ofx and Y exist and are equal, i.e.,m X (t) = M Y (t) for alltin some neighborhood of zero, thenf X (u) = F Y (u) for allu. Theorem 1.9. Suppose that X 1,X 2,... is a sequence of random variables, each with mgf M Xi (t). Furthermore, suppose that lim M X i (t) = M X (t), for alltin a neighborhood of zero, i and M X (t) is an mgf. Then, there is a cdf F X whose moments are determined by M X (t) and, for allxwhere F X (x) is continuous, we have lim F X i (x) = F X (x). i This theorem means that the convergence of mgfs implies convergence of cdfs.

9 1.8. MOMENTS AND MOMENT GENERATING FUNCTIONS 23 Example We know that the Binomial distribution can be approximated by a Poisson distribution when p is small and n is large. Using the above theorem we can confirm this fact. The mgf ofx n Bin(n,p) and ofy Poisson(λ) are, respectively: M Xn (t) = [pe t +(1 p)] n, M Y (t) = e λ(et 1). We will show that the mgf ofx tends to the mgf ofy, whereλ np. We will need the following useful result given in the lemma: Lemma 1.1. Let a 1,a 2,... be a sequence of numbers converging to a, that is, lim n a n = a. Then ( lim 1+ a ) n n = e a. n n Now, we can write M Xn (t) = ( pe t +(1 p) ) n ( = 1+ 1 n np(et 1) ( = 1+ np(et 1) n ) n ) n 1) = M n eλ(et Y (t). Hence, by Theorem 1.9 the Binomial distribution converges to a Poisson distribution.