MATH2715: Statistical Methods

Transcription

1 MATH2715: Statistical Methods Exercises V (based on lectures 9-10, work week 6, hand in lecture Mon 7 Nov) ALL questions count towards the continuous assessment for this module. Q1. If X gamma(α,λ), write down the moment generating function of X. Using moment generating functions, deduce the distribution of Y = bx where b is a constant. Q2. If X 1,X 2,...,X n are independent random variables satisfying X i gamma(α i,λ), use moment generating functions to obtain the distribution of S n = X i. Use moment generating functions to obtain the distribution of Xn where X n = S n n. Q3. Suppose U is a non-negative random variable with probability density function f U (u) and b is a positive constant. Prove the Markov inequality, E[U] bpr{u b}. Hint: Write E[U] and pr{u b} as integrals in terms of u and f U (u). Look at the proof of Chebychev s inequality. Where would you split the region of integration? Q4. Suppose X has a uniform distribution over the interval (0,1) with mean µ and variance σ 2. Calculate pr{ X µ kσ} for a number of different values of k. Compare the exact numerical results with the bound given by Chebyshev s inequality. Hint: Recall that pr{ X µ c} = pr{x (µ c)} + pr{x (µ + c)}. You could use R to evaluate the probabilities. For example k=c(0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0) # Assign some k values. chebyshev=1/k^2 # Chebyshev s bound. prob=punif(k,0,1) # If X is uniform(0,1), prob = pr(x < k). # prob gives the cdf F(x) evaluated at x=k. # prob does NOT give pr( X-mu > k*sigma). # What SHOULD you type here? Q5. For the discrete random variable X satisfying pr{x = 0} = k2, pr{x = k} = 2k2, pr{x = +k} = 2k2, k 1, show that Chebyshev s inequality becomes an equality for this particular value of k. Hint: First obtain the mean µ and the variance σ 2 of X. 56

2 Background Notes: Lecture 10. Weak law of large numbers Example Let Z n = X n µ σ/ n. Figure 26 shows the cumulative distribution function of Z 1 and Z 4 together with the standard ind ind normal cumulative distribution function in the two cases X i exponential(λ = 1) and X i uniform(0, 1). As n it can be seen that in both cases the cumulative distribution function of Z n tends to that of the standard normal cumulative distribution function. Cdf n=1 n=4 N(0,1) cdf Cdf n=1 n=4 N(0,1) cdf z z ind Figure 26: cumulative distribution function of Z 1 and Z 4 for (left) X i exponential(λ = 1), ind (right) X i uniform(0,1). Chebyshev s Inequality For a random variable X with mean µ and finite variance σ 2 <, Chebyshev s 58 inequality states pr{ X µ kσ} 1 k 2. Further reading for lectures 9 and 10 Rice, J.A. (1995) Mathematical Statistics and Data Analysis (2nd edition), sections 4.2, 4.5, 5.2. Hogg, R.V., McKean, J.W. and Craig, A.T. (2005) Introduction to Mathematical Statistics (6th 58 Pafnuti Chebyshev ( ) was a Russian mathematician. Many variants of the spelling of his name exist, Pafnut-i/y (T/Ts)-cheb-i/y-sh/ch/sch-e-v/ff/w. Uses of Chebyshev s (1867) inequality include constructing confidence intervals, so that pr{ X µ < 4.472σ X} > If X is known to be a normal random variable, we have the tighter bound pr{ X µ < 1.96σ X} = The general application of Chebyshev s inequality has lead it to be used in many fields including digital communication, gene matching, and detecting computer security attacks. 57

3 edition), sections 1.9, 1.10, 4.2. Larsen, R.J. and Marx, M.L. (2010) An Introduction to Mathematical Statistics and its Applications (5th edition), section 3.12, 5.7. Miller, I. and Miller, M. (2004) John E. Freund s Mathematical Statistics with Applications, sections 4.4, 4.5, 5.4,

4 MATH2715: Statistical Methods Worked Examples V Worked Example: If X N(µ,σ 2 ), use the moment generating function to obtain the distribution of Y = θx where θ is a constant. Answer: If X N(µ,σ 2 ), then m X (t) = e µt+1 2 σ2 t 2. Mgf of Y is m Y (t) = E[e ty ] = E[e t(θx) ] = E[e (θt)x ] = m X (θt). Thus m Y (t) = e µ(θt)+1 2 σ2 (θt) 2 = e (µθ)t+1 2 (σ2 θ 2 )t 2 = e (µθ)t+1 2 (σθ)2 t 2. By inspection of the mgf it can be seen that Y N(µθ,σ 2 θ 2 ). Worked Example: If X has an exponential(λ) distribution, use moment generating functions to obtain the distribution of U = ax where a (a > 0) is a known constant. Answer: X exponential(λ) has mgf m X (t) = λ λ t for t < λ. Thus m U (t) = E[e tu ] = E[e t(ax) ] = E[e (at)x ] = m X (at) = By inspection of the mgf it can be seen that U exponential(λ/a). λ λ at = (λ/a) (λ/a) t. Worked Example: If X and Y are independent exponential(λ) random variables, use moment generating functions to obtain the distribution of U = X + Y. Answer: X has mgf m X (t) = λ λ t for t < λ. Similarly m Y (t) = λ λ t for t < λ. Hence m U (t) = E[e tu ] = E[e t(x+y ) ] = E[e tx e ty ] = E[e tx ]E[e ty ] by independence of X and Y. Thus m U (t) = λ λ t λ ( ) λ 2 λ t =. λ t By inspection of the mgf we deduce that U gamma(2,λ). Worked Example: If X 1, X 2 and X 3 are independent gamma(α,λ) random variables, use moment generating functions to obtain the distribution of U = X 1 + X 2 + X 3. ( ) λ α Answer: The X i have mgf m Xi (t) =. Hence λ t m U (t) = E[e tu ] = E[e t(x 1+X 2 +X 3 ) ] = E[e tx 1+tX 2 +tx 3 ] = E[e tx 1 e tx 2 e tx 3 ] = E[e tx 1 ]E[e tx 2 ]E[e tx 3 ] by independence of the X i. Thus m U (t) = By inspection of the mgf, U gamma(3α,λ). ( ) λ α ( ) λ α ( ) λ α ( ) λ 3α =. λ t λ t λ t λ t 59

5 Worked Example: Suppose X N(µ,σ 2 ). Calculate pr{ X µ kσ} for a number of different values of k and compare the exact results with the bound given by Chebyshev s inequality. Answer: pr{ X µ kσ} = pr{x µ kσ} + pr{x µ kσ} = pr{x µ + kσ} + pr{x µ kσ}. But for the normal distribution we evaluate cumulative probabilities by standardising. Thus { } { } X µ X µ pr{ X µ kσ} = pr k + pr k = pr{z k} + pr{z k} σ σ where Z = X µ σ N(0,1) with Φ(z) = pr{z z}. Thus pr{ X µ kσ} = (1 pr{z k}) + pr{z k} = (1 Φ(k)) + Φ( k) = (1 Φ(k)) + (1 Φ(k)) = 2(1 Φ(k)). The probabilities can be generated for different k using the R code below. The probabilities and Chebyshev bound are shown in table 1. k=c(0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0) prob=2*(1-pnorm(k)) chebyshev=1/k^2 # Assign some k values. # Exact probabilities. # Chebyshev s bound. Figure 27(left) shows pr{ X µ 1.5σ} for X N(µ = 10,σ 2 = 4). Figure 27(right) gives the exact probability and Chebyshev s bound for a range of values k. Pdf Probability Chebyshev s bound Exact probability pr( X µ kσ) k Figure 27: (left) pr{ X µ 1.5σ} for X N(µ = 10,σ 2 = 4); (right) Chebyshev s bound and the exact probability pr{ X µ kσ} for X N(µ,σ 2 ). Worked Example: Suppose X exponential(λ) with mean µ and variance σ 2. Obtain pr{ X µ kσ}. Calculate pr{ X µ kσ} for a number of different values of k and compare the exact result with the bound given by Chebyshev s inequality. 60

6 k pr{ X µ kσ} Chebyshev bound ( ) Table 1: Chebyshev s inequality and the exact probability for X N(µ,σ 2 ). ( ) For k 1 the bound is replaced by unity. Answer: f X (x) = λe λx for x > 0 and F X (x) = pr{x x} = 1 e λx for x > 0. Also µ = E[X] = 1/λ and σ 2 = Var[X] = 1/λ 2. Hence pr{ X µ kσ} = pr{x µ kσ} + pr{x µ + kσ} = pr { X 1 k } { } λ + pr X 1+k λ = F X ( 1 k λ ) + (1 F X( 1+k λ )) = This can be evaluated for different values of k. { 1 e (1 k) + e (1+k) if k < 1, e (1+k) if k 1. Figure 28 shows pr{ X µ > 0.6σ} and pr{ X µ > 1.5σ} for X exponential(λ = 2). 59 Exponential(2) pdf Exponential(2) pdf x x Figure 28: for X exponential(λ = 2), (left) pr{ X µ 0.6σ} and (right) pr{ X µ 1.5σ}. Suitable R commands for plotting the exact probability pr{ X µ kσ} and the Chebyshev bound 1/k 2 are given below for the case X exponential(λ = 1). The result is plotted in figure k=c(1:30)*0.1 # Set k=0.1, 0.2,..., Thus here µ = 1, σ = 1, and pr{ X µ 0.6σ} = pr{x 0.8} + pr{x 0.2} while pr{ X µ 1.5σ} = 2 2 pr{x 1.25} + pr{x 0.25} where here pr{x 0.25} = Chebyshev s inequality says that pr{ X µ kσ} 1/k 2 for k > 1. If k < 1 the right-hand side of Chebyshev s inequality should be set equal to one! 61

7 prob=pexp(1-k)+(1-pexp(k+1)) # Exact probability. cheb=1/k^2 # Chebyshev upper bound. cheb[cheb>1]=1 # Make all values of cheb>1 equal to one. plot(k,prob,type="l",ylim=c(0,1)) points(k,cheb,type="l",lty=2) Probability Chebyshev s bound Exact probability pr( X µ kσ) k Figure 29: Chebyshev s bound and the exact probability pr{ X µ kσ} in the case where X exponential(λ). Worked Example: In the Markov 61 inequality, a non-negative random variable U satisfies E[U] bpr{u b}. Thus if a random variable X has mean µ and variance σ 2 and U = g(x) is a non-negative function of X, then it follows that E[g(X)] bpr{g(x) b}. By considering the special case g(x) = (X µ) 2, derive Chebyshev s inequality pr{ X µ kσ} 1 k 2. Answer: The Markov inequality 62 can be written as pr{g(x) b} E[g(X)]/b. Now write g(x) = (X µ) 2. Thus pr { (X µ) 2 b } E[(X µ)2 ] = Var[X] b b Since (X µ) 2 = X µ 2, then pr { (X µ) 2 b } { = pr X µ } b. = σ2 b. Write b = kσ, b = k 2 σ 2 so pr { (X µ) 2 b } σ2 b gives pr{ X µ kσ} 1 as required. k2 61 Andrei Andreyevich Markov ( ) was a Russian mathematician and a student of Chebyshev. 62 The Markov inequality is still useful for leading to new results. For example, consider Herman Chernoff s (1952) inequality pr{x c} e tc m X(t), where X has moment generating function m X(t). This result follows because pr{x c} = pr e tx e tc and putting g(x) = e tx and b = e tc in the Markov inequality gives n pr e tx e tco E[etX ]. e tc 62

8 Worked Example: Does there exist a random variable X for which Chebyshev s inequality becomes an equality for all k 1? Answer: There is no distribution which attains the Chebyshev bound for all k 1. We use a proof by contradiction and consider the continuous case. Suppose there exists a random variable X with finite variance σ 2 and satisfying pr{ X µ kσ} = 1 k 2 k 1. ( ) Let Z = X µ, with E[Z] = 0 and Var[Z] = 1 ( ) so that pr{ Z k} = 1 σ k 2 k 1. Hence pr{ Z z} = pr{z z} + pr{z z} = 1 for z 1. z2 If Z has cumulative distribution function F Z (z) = pr{z z} and probability density function f Z (z), then F Z ( z) + {1 F Z (z)} = 1 z 2. Differentiating this with respect to z gives f Z ( z) f Z (z) = 2 z 3. Clearly f Z( z) + f Z (z) = 2 z 3 for z 1. Since the area under the probability density function integrates to unity it follows that so that 1 = 1 = 1 1 f Z (z)dz = f Z (z)dz f Z (z)dz {f Z ( z) + f Z (z)}dz = f Z (z)dz f Z (z)dz 2 f Z (z)dz + 1 z 3 dz and then 1 [ ] 1 f Z (z)dz = 1 1 z 2 = 0. 1 As probability density functions are non-negative this can only hold if f Z (z) = 0 for 1 < z < 1. Since by assumption E[Z] = 0, the variance of Z satisfies Var[Z] = E[Z 2 ] so that Var[Z] = z 2 f Z (z)dz = 1 z 2 {f Z ( z) + f Z (z)}dz = 1 2 dz =. z Thus Var[Z] 1 as assumed at ( ). Clearly X cannot have finite variance σ 2. There is thus no random variable X which satisfies ( ). Worked Example: Let X 1,X 2,X 3,... be a sequence of random variables with a common mean µ, common variance σ 2, and with covariances, for i j, given by { ρσ 2 if j = i ± 1, cov(x i,x j ) = 0 otherwise. Consider now the sequence of means { X n }, n = 1,2,3,..., where X n = 1 X i. Show that n ( ) Var[ X n ] = σ2 1) + 2(n n n 2 ρσ 2. Deduce that Var[ X 3n 2 n ] n 2 σ 2 and so use Chebyshev s inequality to show that a weak law of large numbers applies to X n. 63

9 Answer: Let S n = [ ] X i. Then Var[S n ] = Var X i = The double sum has n 2 terms as shown in the table. j=1 cov(x i,x j ). j = 1 j = 2 j = 3 j = n i = 1 cov(x 1,X 1 ) = σ 2 cov(x 1,X 2 ) = ρσ 2 cov(x 1,X 3 ) = 0 cov(x 1,X n ) = 0 i = 2 cov(x 2,X 1 ) = ρσ 2 cov(x 2,X 2 ) = σ 2 cov(x 2,X 3 ) = ρσ 2 cov(x 2,X n ) = i = n cov(x n,x 1 ) = 0 cov(x n,x 2 ) = 0 cov(x 1,X 3 ) = 0 cov(x n,x n ) = σ 2 Hence Var[S n ] = cov(x i,x i ) + cov(x i,x j ) = Var[X i ] + 2 cov(x i,x j ) j=1 j=1 i j i<j = nσ {cov(x 1,X 2 ) + cov(x 2,X 3 ) + + cov(x n 1,X n )} = nσ 2 + 2(n 1)ρσ 2. This gives Var[ X n ] = Var [ ] Sn = 1n n 2Var[S n] = σ2 1) + 2(n n n 2 ρσ 2. Since 1 ρ 1 and cov(x i,x j ) = ρσ 2, then σ 2 cov(x i,x j ) σ 2. Hence Var[ X n ] σ2 1) + 2(n n n 2 σ 2 = Var[ X n ] ( 3n 2 n 2 ) σ 2. In Chebyshev s inequality, pr{ Y µ Y kσ Y } 1/k 2. Put Y = X n with µ Y = µ and put ǫ = kσ Y so that pr { X n µ ǫ } Var[ X n ] (3n 2)σ2 ǫ 2 n 2 ǫ 2. For given η, choose n 0 such that (3n 0 2)σ 2 n 2 < η. Then, for all n > n 0, 0 ǫ2 as required, pr { X n µ ǫ } < η if n > n 0. Thus X p n µ as n. (3n 2)σ2 n 2 ǫ 2 < η and so, 64

10 MATH2715: Solutions to exercises II Q1. E[(X µ) 4 ] = E[X 4 4µX 3 + 6µ 2 X 2 4µ 3 X + µ 4 ] = E[X 4 ] 4µE[X 3 ] + 6µ 2 E[X 2 ] 4µ 3 E[X] + µ 4 as µ is a constant. Thus E[(X µ) 4 ] = µ 4 4µµ 3 + 6µ2 µ 2 4µ4 + µ 4 = µ 4 4µµ 3 + 6µ2 µ 2 3µ4, where µ r = E[X r ] and µ 1 = E[X] = µ. Q2. E[X 3 ] = x x 3 f X (x)dx = = x=0 x=0 x 3λα x α 1 e λx Γ(α) λ α x α+2 e λx Γ(α) dx dx = λα x α+2 e λx dx. Γ(α) x=0 Since f X (x) is a probability density function it integrates to one, so that x f X (x)dx = 1 x=0 This must hold for all values of α so that Putting A = α + 3 we have Thus as required since λ α x α 1 e λx dx = 1 Γ(α) x=0 x=0 x A 1 e λx dx = Γ(A) λ A. x α+2 e λx dx = Γ(α + 3) λ α+3. x=0 x α 1 e λx dx = Γ(α) λ α. E[X 3 ] = λα x α+2 e λx dx = λα Γ(α + 3) α(α + 1)(α + 2) Γ(α) x=0 Γ(α) λ α+3 = λ 3. Γ(α + 3) = (α + 2)Γ(α + 2) = (α + 2)(α + 1)Γ(α + 1) = (α + 2)(α + 1)αΓ(α). Q3. Since X Bin(n,θ), E[X] = nθ and Var[X] = nθ(1 θ). For method of moments estimators equate 63 : x = E[X] and m 2 = Var[X]. Thus x = nθ and m 2 = nθ(1 θ) gives 1 θ = m 2 x θ = 1 m 2 x 63 You could alternatively equate s 2 = Var[X], where s 2 = 1 m 1 give a slightly different estiate for θ and n, namely θ e = 1 and en = s2 x x2 mx (x i x) 2 is the sample variance. This would x s 2. 65

11 and so estimate n by ñ = x θ = x2 x m 2. Problem: Do we expect our estimate for n to be an integer? A bigger problem: Is ñ always non-negative? 64 Q4. E[X] = x xf X (x)dx = x=1 x θ ] dx = θx θ dx = [θ x θ+1 xθ+1 x=1 ( θ + 1) x=1 since lim x x θ+1 = 0 because θ > 1. For method of moments estimates equate x = E[X] so that θ = ( θ + 1) = θ θ 1 x = θ θ 1 θ = x x 1. Q5. For these data x = x i = 4.54, m 2 = x 2 i = , m 2 = m 2 x2 = For a gamma(α,λ) distribution E[X] = α λ, Var[X] = α λ 2. For method moments estimates equate x = E[X] and m 2 = Var[X] so that α = x2 m 2, λ = x m 2. These give α = and λ = The best fitting gamma distribution is thus a gamma(α = 4.038,λ = ) distribution. Notice that α 4. Using R to do the calculations: x=c(8.7,3.3,5.5,5.8,6.1,3.9,1.2,0.7,4.7,5.5) # Set values into x. xbar=mean(x) # Mean is mean(x^2) # m2 is m2=mean(x^2)-xbar^2 # m2 is alpha=xbar^2/m2 # alpha is lambda=xbar/m2 # lambda is A simple R simulation shows that en can be negative: nn=numeric(10000) # Initialise nn to be a vector long. m=20; n=10; theta=0.3 # Initialse required constants m, n and theta. for (k in 1:10000){ # Do simulations. x=rbinom(m,n,theta) # Simulate m values from Bin(n,theta) and store in x. m2=mean((x-mean(x)) 2) # Obtain m2 values. nn[k]=mean(x) 2/(mean(x)-m2) # Obtain n estimate. } # End k loop. hist(nn,100) # Histogram of n values with about 100 breaks. 66

12 MATH2715: Statistical Methods Examples Class V, week 6 The questions below will be looked at in Examples Class V held in week 6. Q1. Suppose that g(x) is a non-negative function 65 which is symmetric about zero and which is a non-decreasing function of x for positive x. If X is a continuous random variable prove that, for any ǫ > 0, pr{ X ǫ} E[g(X)]. g(ǫ) (Hint: Split the region of integration (,+ ) for E[g(X)] into suitable parts.) By writing X = Y µ, deduce Chebyshev s inequality as a special case. Q2. Suppose that Y 1,Y 2,... is a sequence of random variables which satisfies 2 n with probability n Y n = 0 with probability 1 3 n 2 n with probability n for n = 1,2,... Prove that p lim n Y n = 0. What is the mean and variance of Y n? What happens as n? 65 Examples include g(x) = x and g(x) = 1/(1 + e x2 ). 67