Sampling Distributions

Transcription

1 In statistics, a random sample is a collection of independent and identically distributed (iid) random variables, and a sampling distribution is the distribution of a function of random sample. For example, the average and the variance formula are functions of random sample. The term statistic is referred to an individual function of a random sample, and the understanding of sampling distributions is the major undertaking of statistics. Moment generating function. Let X be a random variable. The moment generating function (mgf) of X is given by M(t) E [ e tx]. The mgf M(t) is a function of t defined on some open interval (c, c 1 ) around with c < < c 1. In particular when X is a continuous random variable having the pdf f(x), the mgf M(t) can be expressed as M(t) e tx f(x) dx. The most significant property of moment generating function is that the moment generating function uniquely determines the distribution. MGF of normal distribution. Suppose that X is a standard normal random variable. Then we can compute the mgf of X as follows. M X (t) 1 π e t 1 π e tx e x dx 1 π e (x t) dx exp e 1 (x t) + t ( ) t Since X (Y µ)/σ becomes a standard normal random variable and Y σx + µ, the mgf of Y can be given by ( ) σ M Y (t) e µt t M X (σt) exp + µt for < t <. MGF of gamma distribution. Suppose now that X is a gamma random variable with parameter (α, λ). Then the mgf M X (t) of X can be computed as follows. M X (t) e tx λ α λα Γ(α) xα 1 e λx dx (λ ( t) α Γ(α) ) α λ 1 1 λ t Γ(α) λα Γ(α) dx x α 1 e (λ t)x dx ((λ t)x) α 1 e (λ t)x (λ t)dx ( ) α λ u α 1 e u du λ t where (λ t)x is substituted by u, and (λ t)dx by du. It should be noted that this mgf M X (t) is defined on the open interval (, λ). Joint moment generating function. Let X and Y be random variables having a joint density function f(x, y). Then we can define the joint moment generating function M(s, t) of X and Y Page 1 Probability and Statistics II/February 6, 18

2 by M(s, t) E [ e sx+ty ] e sx+ty f(x, y) dx dy. If X and Y are independent, then the joint mgf M(s, t) becomes M(s, t) E [ e sx+ty ] E [ e sx] E [ e ty ] M X (s) M Y (t). Moreover, it has been known that if the joint mgf M(s, t) of X and Y is of the form M 1 (s) M (t), then X and Y are independent and have the respective mgf M 1 (s) and M (t). Sum of independent random variables. Let X and Y be independent random variables having the respective probability density functions f X (x) and f Y (y). Then the cumulative distribution function F Z (z) of the random variable Z X + Y can be given as follows. [ z x ] F Z (z) P (X + Y z) f X (x)f Y (y) dy dx where F Y f X (x)f Y (z x) dx, is the cdf of Y. By differentiating F Z (z), we can obtain the pdf f Z (z) of Z as f Z (z) d ( ) d dz F Z(z) f X (x) dz F Y (z x) dx f X (x)f Y (z x) dx. Convolution. The function of z (f g)(z) f(x)g(z x) dx is called the convolution of f and g. Thus, the pdf f Z (z) for Z X + Y is given by the convolution of the pdf s f X (x) and f Y (y). Then we can easily verify [ ] M Z (t) e tz f X (x)f Y (z x) dx dz M X (t)m Y (t) Therefore, as we will see in the examples below, the construction of moment generating function is much easier in order to find the distribution of the sum of independent random variables. Example 1. Let X and Y be independent normal random variables with the respective parameters (µ x, σ x) and (µ y, σ y). Find the distribution for Z X + Y. Solution. We can calculate ( ) σ M Z (t) M X (t) M Y (t) exp x t + µ xt ( (σ x + σy)t ) exp + (µ x + µ y )t ( σ y t ) exp + µ yt which is the mgf of normal distribution with parameter (µ x + µ y, σ x + σ y). Thus, we find that Z is a normal random variable with parameter (µ x + µ y, σ x + σ y). Page Probability and Statistics II/February 6, 18

3 Example. Let X and Y be independent gamma random variables with the respective parameters (α 1, λ) and (α, λ). Find the distribution for Z X + Y. Solution. We can calculate M Z (t) M X (t) M Y (t) ( ) α1 ( ) α ( ) α1 +α λ λ λ λ t λ t λ t which is the mgf of gamma distribution with parameter (α 1 + α, λ). random variable with parameter (α 1 + α, λ). Thus, Z is a gamma Chi-square distribution. The gamma distribution f(x) 1 n/ Γ(n/) xn/ 1 e x/, x with parameter (n/, 1/) is called the chi-square distribution with n degrees of freedom (df). Since it has only one parameter n, the mgf M(t) of chi-square distribution can be simply expressed as ( ) n/ 1/ M(t) (1 t) n/ 1/ t Chi-square distribution of one degree of freedom. Let X be a standard normal random variable, and let Y X be the square of X. Then we can express the cdf F Y of Y as F Y (t) P (X t) P ( t X t) Φ( t) Φ( t) in terms of the standard normal cdf Φ. By differentiating the cdf, we can obtain the pdf f Y as follows. f Y (t) d dt F Y (t) 1 π t 1 e t 1 1/ Γ(1/) t 1/ e t/, where we use Γ(1/) π. Thus, the square X of X is the chi-square random variable with 1 degree of freedom. Chi-square distribution with n df. Let X 1,..., X n be iid standard normal random variables. Since X i s have the gamma distribution with parameter (1/, 1/), the sum Y n i1 X i has the gamma distribution with parameter (n/, 1/). That is, the sum Y has the chi-square distribution with n degree of freedom. Student s t-distribution. Let X be a chi-square random variable with n degrees of freedom, and let Y be a standard normal random variable. Suppose that X and Y are independent. Then the distribution of the quotient Z Y X/n Page 3 Probability and Statistics II/February 6, 18

4 is called the Student s t-distribution with n degrees of freedom. The pdf f Z (z) is given by f Z (z) Γ((n + 1)/) nπγ(n/) (1 + z /n) (n+1)/, < z <. F -distribution. Let U and V be independent chi-square random variables with r 1 and r degrees of freedom, respectively. Then the distribution of the quotient Z U/r 1 V/r is called the F-distribution with (r 1, r ) degree of freedom. The pdf f Z (z) is given by f Z (z) Γ((r ( 1 + r )/) Γ(r 1 /)Γ(r /) (r 1/r ) r1/ z r 1/ r ) (r1 +r )/ 1 z, < z <. r Quotient of two random variables. Let X and Y be independent random variables having the respective pdf s f X (x) and f Y (y). Then the cdf F Z (z) of the quotient can be computed as follows. Z Y/X F Z (z) P (Y/X z) P (Y zx, X < ) + P (Y zx, X > ) [ ] [ xz ] f Y (y) dy f X (x) dx + f Y (y) dy f X (x) dx By differentiating, we obtain xz f Z (z) d dz F Z(z) x f Y (xz)f X (x) dx. [ xf Y (xz)]f X (x) dx + [xf Y (xz)]f X (x) dx Derivation of t-distribution. Let X be a chi-square random variable with n degrees of freedom. Then the pdf of the random variable U X/n is given by for t ; otherwise, f U (t). f U (x) d dx P ( X/n x) d dx F X(nx ) nxf X (nx ) n n/ n/ 1 Γ(n/) xn 1 e nx / Derivation of t-distribution, continued. Suppose that Y is a standard normal random variable and independent of X. Then the quotient Z Y X/n Page 4 Probability and Statistics II/February 6, 18

5 has a t-distribution with n degrees of freedom. The pdf f Z (z) can be computed as follows. n n/ x f Y (xz)f U (x) dx (n 1)/ x n e (z +n)x / dx πγ(n/) 1 (1 + z /n) (n+1)/ t (n 1)/ e t dt nπγ(n/) Γ((n + 1)/) nπγ(n/) (1 + z /n) (n+1)/ for < z <. Derivation of F -distribution. Let U and V be independent chi-square random variables with r 1 and r degrees of freedom, respectively. Then the quotient Z U/r 1 V/r has F -distribution with (r 1, r ) degree of freedom. By putting Y U/r 1 and X V/r, we can compute the pdf f Z (z) as follows. x f Y (xz)f X (x) dx r r 1/ 1 r r / (r 1+r )/ Γ(r 1 /)Γ(r /) zr 1/ 1 r r 1/ 1 r r / Γ(r 1 /)Γ(r /) (r 1z + r ) (r 1+r )/ z r 1/ 1 Γ((r 1 + r )/) Γ(r 1 /)Γ(r /) (r 1/r ) r 1/ z r 1/ 1 x (r 1+r )/ 1 e (r 1z+r )x/ dx ( 1 + r 1 r z t (r 1+r )/ 1 e t dt ) (r1 +r )/, < z <. Population and random sample. The data values recorded x 1,..., x n are typically considered as the observed values of iid random variables X 1,..., X n having a common probability distribution f(x). Specifically we may assume that f(x) is a normal distribution with parameter (µ, σ ). In a typical statistical problem it is useful to envisage a population from which the sample should be drawn. A random sample is chosen at random from the population, ensuring that the sample is representative of the population. Sample mean. Let X 1,..., X n be iid normal random variables with parameter (µ, σ ). The function of random sample X 1 n X i n is called the sample mean. Then we obtain i1 E[ X] µ and Var( X) σ n. Furthermore, X has the normal distribution with parameter (µ, σ /n). Page 5 Probability and Statistics II/February 6, 18

6 Sample variance. The function of random sample S 1 n (X i n 1 X) is called the sample variance. The sample variance S can be rewritten as ( n ) ( n ) S 1 Xi n n 1 X 1 (X i µ) n( n 1 X µ). In particular, we can obtain i1 E[S ] 1 n 1 1 n 1 i1 i1 ( n ) E[(X i µ) ] ne[( X µ) ] i1 ( n ) Var(X i ) nvar( X) σ. i1 Theorem 3. X has the normal distribution with parameter (µ, σ /n), and X and S are independent. Furthermore, (n 1)S W n 1 1 n (X σ σ i X) has the chi-square distribution with (n 1) degrees of freedom. Rough explanation of Theorem 3. Assuming that X and S are independent (which is not so easy to prove), we can show that the random variable W n 1 has the chi-square distribution with (n 1) degrees of freedom. By introducing V 1 n σ i1 (X i µ) and W 1 n ( X µ), σ we can write V W n 1 + W 1. Since W n 1 and W 1 are independent, the mgf M V (t) of V can be expressed as M V (t) M Wn 1 (t) M W1 (t), where M Wn 1 (t) and M W1 (t) are the mgf s of W n 1 and W 1, respectively. Observe that W 1 has the chi-square distribution with 1 degrees of freedom. Thus, we can obtain M Wn 1 (t) M V (t) M W1 i1 (1 t) n/ (1 t) (n 1)/ (1 t) 1/ which is the mgf of the chi-square distribution with (n 1) degrees of freedom. t-statistic. Assuming iid normal random variables X 1,..., X n with parameter (µ, σ ), we can construct a standard normal random variable Z 1 X µ σ/ and a chi-square random variable W n n 1 of Theorem 3. Furthermore, the function of random sample ) T X µ S/ n ( X µ σ/ n (n 1)S /σ n 1 Z 1 W n 1 n 1 has the t-distribution with (n 1) degrees of freedom, and it is called t-statistic. The statistic S S is called the sample standard deviation. Sampling distributions under normal assumption. Page 6 Probability and Statistics II/February 6, 18

7 (a) The sample mean X is normally distributed with parameter (µ, σ /n), and Z 1 n( X µ) σ N(, 1) (b) The statistic W n 1 1 σ n (X i X) i1 (n 1)S σ χ n 1 has a chi-square distribution with (n 1) df, and independent of X (Theorem 3). (c) The t-statistic has t-distribution with (n 1) df X µ S/ n n( X µ) σ (n 1)S /σ n 1 Z 1 W n 1 n 1 t n 1 Critical points. Given a random variable X, the critical point for level α is defined as the value c satisfying P (X > c) α. (a) Suppose that a random variable X has the chi-square distribution with n degrees of freedom. Then the critical point of chi-square distribution, denoted by χ α,n, is defined by P (X > χ α,n ) α. (b) Suppose that a random variable Z has the t-distribution with m degrees of freedom. Then the critical point is defined as the value t α,m satisfying P (Z > t α,m ) α. Critical points, continued. Suppose that a random variable Z has the t-distribution with m degrees of freedom. Since the t-distribution is symmetric, the critical point t α,m can be equivalently given by P (Z < t α,m ) α. Together we can obtain the expression P ( t α/,m Z t α/,m ) 1 P (Z < t α/,m ) P (Z > t α/,m ) 1 α. Extrema. Let X 1,..., X n be independent and identically distributed random variables (iid random variables) with the common cdf F (x) and the pdf f(x). Then we can define the random variable V min(x 1,..., X n ) of the minimum of X 1,..., X n. To find the distribution of V, consider the survival function P (V > x) of V and calculate as follows. P (V > x) P (X 1 > x,..., X n > x) P (X 1 > x) P (X n > x) [1 F (x)] n. Page 7 Probability and Statistics II/February 6, 18

8 Extrema, continued. Since F V (x) 1 P (V > x), we obtain the cdf F V (x) and the pdf f V (x) of V F V (x) 1 [1 F (x)] n and f V (x) d dx F V (x) nf(x)[1 F (x)] n 1. In particular, if X 1,..., X n be independent exponential random variables with the common parameter λ, then the minimum min(x 1,..., X n ) has the density (nλ)e (nλ)x which is again the exponential density function with parameter (nλ). Order statistics. Let X 1,..., X n be iid random variables with the common cdf F (x) and the pdf f(x). When we sort X 1,..., X n as X (1) < X () < < X (n), the random variable X (k) is called the k-th order statistic. Theorem 4. The pdf of the k-th order statistic X (k) is given by f k (x) n! (k 1)!(n k)! f(x)f k 1 (x)[1 F (x)] n k. (1.1) Proof of theorem: Infinitesimal probability. If f is continuous at x and δ > is small, then the probability that a random variable X falls into the small interval [x, x + δ] is approximated by δf(x). In differential notation, we can write P (x X x + dx) f(x)dx. Now consider the k-th order statistic X (k) and the event that x X (k) x + dx. This event occurs when (k 1) of X i s are less than x and (n k) of X i s are greater than x x + dx. Since there are ( n k 1,1,n k) arrangements for Xi s, we can obtain the probability of this event as which implies (1.1). f k (x)dx n! (k 1)!(n k)! (F (x))k 1 (f(x)dx)[1 F (x)] n k, Beta distribution. When X 1,..., X n be iid uniform random variables on [, 1], the pdf of the k-th order statistic X (k) becomes f k (x) n! (k 1)!(n k)! xk 1 (1 x) n k, x 1, which is known as the beta distribution with parameters α k and β n k + 1. Page 8 Probability and Statistics II/February 6, 18

9 Exercises Exercises Problem 1. Consider a sample X 1,..., X n of normally distributed random variables with variance σ 5. Suppose that n 31. (a) What is the value of c for which P (S c).9? (b) What is the value of c for which P (S c).95? Problem. Consider a sample X 1,..., X n of normally distributed random variables with mean µ. Suppose that n 16. (a) What is the value of c for which P ( 4( X µ)/s c).95? (b) What is the value of c for which P ( 4( X µ)/s c).99? Problem 3. Suppose that X 1,..., X 1 are iid normal random variables with parameters µ and σ >. (a) Let Z X 1 + X + X 3 + X 4. Then what is the distribution for Z? (b) Let W X 5 + X 6 + X 7 + X 8 + X 9 + X 1. Then what is the distribution for W σ? (c) Describe the distribution for the random variable (d) Find the value c for which P ( ) Z W c.95. Z/ W/6. Problem 4. Suppose that system s components are connected in series and have lifetimes that are independent exponential random variables X 1,..., X n with respective parameters λ 1,..., λ n. (a) Let V be the lifetime of the system. Show that V min(x 1,..., X n ). (b) Show that the random variable V is exponentially distributed with parameter n i1 λ i. Problem 5. Each component A i of the following system has lifetime that is iid exponential random variable X i with parameter λ. A 1 A 3 A 5 A A 4 A 6 (a) Let V be the lifetime of the system. Show that V max(min(x 1, X ), min(x 3, X 4 ), min(x 5, X 6 )). Page 9 Probability and Statistics II/February 6, 18

10 Exercises (b) Find the cdf and the pdf of the random variable V. Problem 6. A card contains n chips and has an error-correcting mechanism such that the card still functions if a single chip fails but does not function if two or more chips fail. If each chip has an independent and exponentially distributed lifetime with parameter λ, find the pdf of the card s lifetime. Problem 7. Suppose that a queue has n servers and that the length of time to complete a job is an exponential random variable. If a job is at the top of the queue and will be handled by the next available server, what is the distribution of the waiting time until service? What is the distribution of the waiting time until service of the next job in the queue? Page 1 Probability and Statistics II/February 6, 18

11 Optional problems Optional problems Joint distribution of order statistics. Let U and V be iid uniform random variable on [, 1], and let X min(u, V ) and Y max(u, V ). We have discussed that the probability P (x X x + dx) is approximated by f(x) dx. Similarly, we can find the following formula in the case of joint distribution. In particular, we can obtain P (x X x + dx, y Y y + dy) f(x, y) dx dy. P (x U x + dx, y V y + dy) dx dy. Problem 8. By showing that P (x X x + dx, y Y y + dy) P (x U x + dx, y V y + dy) + P (x V x + dx, y U y + dy), justify that the joint density function f(x, y) of X and Y is given by { if x y 1; f(x, y) otherwise. Problem 9. Continue Problem 8, and answer the following questions. (a) Find the conditional density f X Y (x y) and f Y X (y x). (b) Find the conditional expectations E(X Y y) and E(Y X x). Problem 1. Let f(x, y) {( n x) y x (1 y) n x if x, 1,..., n and y 1; otherwise. (a) Find the marginal density f Y (y) n x f(x, y) for y 1, and the conditional frequency function f X Y (x θ) given Y θ. Then identify the distribution for f X Y (x θ). (b) Find the marginal freqency function f X (x) 1 f(x, y) dy. (c) Given x k, find the conditional density and f Y X (y k), and identify the distribution for f Y X (y k). In Bayesian statistics (later in this semester) f Y (y) is called a prior distribution of parameter θ of binomial distribution (n, θ), and f Y X (y k) is a posterior distribution of parameter θ provided that we have observed X k from the binomial distribution. Page 11 Probability and Statistics II/February 6, 18

12 Answers to exercises Answers to exercises Problem 1. Observe that 3S /5 6S has a χ -distribution with 3 degrees of freedom. (a) P (S c) P (6S 6c).9 implies 6c χ.1, Thus, c 6.7. (b) P (S c) P (6S 6c).95 implies 6c χ.5, Thus, c 7.3. Problem. Observe that 4( X µ) S (a) c t.5, (b) c t.5, X µ S/ 16 has a t-distribution with 15 degrees of freedom. Problem 3. (a) N(, 4σ ). (b) Since W/σ (X 5 /σ) + +(X 1 /σ) and each (X i /σ) has the standard normal distribution, W/σ has the chi-square distribution with 6 degrees of freedom. (c) We can write Z/ W/6 standard normal distribution, (d) Since P Thus, c ( ) ( Z Z/ W c P W/6 Z/(σ) (W/σ )/6. Since Z/(σ) is independent of W/σ, and has the Z/ W/6 has the t-distribution with 6 degree of freedom. ) 6 c, the result of (c) implies that 6 c t.5,6 Problem 4. (b) P (V > t) P (X 1 > t) P (X 1 > t) e λ 1t e λnt exp ( n i1 λ i) is the survival function of V. Therefore, V is an exponential random variable with parameter n i1 λ i. Problem 5. (b) Let Y 1 min(x 1, X ), Y min(x 3, X 4 ), and Y 3 min(x 5, X 6 ). Then Y 1, Y and Y 3 are iid exponential random variables with parameter λ. Since V Y (3), we obtain f V (t) 6λe λt (1 e λt ), t >. By integrating it, we can find the cdf F V (t) (1 e λt ) 3. Problem 6. Let X i denote the lifetime of the i-th chip, and let V denote the lifetime of the card. Then X i s are iid exponential random variables with λ, and V X (). Thus, we obtain f V (t) n(n 1)λ[e λ(n 1)t e λnt ], t >. Problem 7. Each of n servers completes a job independently in an exponentially distributed time with λ. Then it is known that the sojourn time between the jobs is exponential with nλ, and that the sojourn times are independent each other. Therefore, (i) the distribution of the waiting time at the top of the queue is exponential with nλ, and (ii) the distribution of the waiting time for the second in the queue is given by the sum of two iid exponential random variables, that is, the gamma distribution with (, nλ). Page 1 Probability and Statistics II/February 6, 18