STAT215: Solutions for Homework 1

Transcription

1 STAT25: Solutions for Homework Due: Wednesday, Jan 30. (0 pt) For X Be(α, β), Evaluate E[X a ( X) b ] for all real numbers a and b. For which a, b is it finite? (b) What is the MGF M log X (t) for the random variable Y : [log X]? [HINT: Integration by parts is not needed for, and no new calculations at all are needed for (b)!] E[X a ( X) b ] 0 x a ( x) b f(x)dx Γ(α + β) x (a+α) ( x) b+β dx Γ(α)Γ(β) 0 Γ(α + β) Γ(α + a)γ(β + b) Γ(α)Γ(β) Γ(α + β + a + b) (b) When a > α, b > β, it is finite. M Y (t) E(e ty ) e ty f(y)dy x t f(x)dx E(X t ) This is just a special case with a t, b 0. So M log X (t) Γ(α+β) Γ(α) t > α 2. (0 pt) Let X Ge(p). Find the moment generating function M X (t) : E [ e tx]. (b) Use this MGF to find the mean and variance of X. Γ(α+t) Γ(α+β+t),

2 M X (t) E[e tx ] e tx p( p) x x0 p, for t < log( p) ( p)et (b) The cumulant generating function c X (t) log(m X (t)) log p log( qe t ),so EX c X(0) p p V arx c X (0) (EX)2 Note: If you use f(y) pq y, then M Y (t) qe t ( qe t ) 2 t0 p p 2 pet, EY /p, V ary q/p 2. qe t 3. (0 pt) The double-exponential distribution has pdf f(x) λ 2 e λ x, for fixed λ > 0. Find the MGF M X (t) of a double exponential. For which t is it finite? (b) Let U, V iid Ex(), and find the MGF M Y (t) of Y : U V. distribution of Y? What is the (c) Find the mean and variance of a double exponential. If X,, X n are i.i.d. double exponentials, find the MGF M n (t) of the standardized mean, W n : V(X)/n X E[X]. (d) What is the limit of M n (t) as n? What distribution has this function for its MGF? M X (t) e tx λ 2 e λ x dx 0 λ e (t λ)x dx + λ e (t+λ)x dx λ 2 λ + t e(t λ)x 0 + λ + t e(t+λ)x 0 λ 2 if t < λ λ 2 t2 2

3 (b) M Y (t) E[e t(u V ) ] E[e tu ] E[e tv ] t + t t 2 which is the MGF of a double exponential so Y is a double exponential with λ. (c) For a double-exponential, EX M X (0) 0, V arx M X (0) 2/λ2. So M n (t) E(e twn ) M X (t/ nv (x)) M Xn (t/ nv (x)) M X (t/ nv (x)) n ( t 2 /2n )n (d) As n, M n (t) e t2 /2 et2 /2,which is the MGF of a No(0, ) 4. (0 pt) Bickel & Doksum pg 7, problem.2. π(θ θ x 0) Similarly, π(θ θ 2 x 0) π(θ θ x ) π(θ θ )π(x 0 θ θ ) π(θ θ )π(x 0 θ θ ) + π(θ θ 2 )π(x 0 θ θ 2 ) π(θ θ 2 x ) We might summarize this posterior frequency function in the following way: θ, x 0 2 θ 3 θ

4 (b) Note that π(x, x 2,..., x n θ θ ) n i (0.2)x i (0.8) x i (0.2) Σx i (0.8) n Σx i. π(θ θ x, x 2,..., x n ) where S n Σx i. Similarly, π(θ θ x, x 2,..., x n ) (c) Similar to (b). We have: π(θ θ )π(x, x 2,..., x n θ θ ) π(θ θ )π(x, x 2,..., x n θ θ ) + π(θ θ 2 )π(x, x 2,..., x n θ 2 (0.2)Σx i (0.8) n Σx i 2 (0.2)Σx i (0.8) n Σx i + 2 (0.6)Σx i (0.4) n Σx i 3 Sn 2 n Sn 4 n Sn + 3 Sn 2 n Sn 2 n 2 n + 6 Sn 6Sn. 2 n +6 Sn π(θ θ x, x 2,..., x n ) and π(θ θ 2 x, x 2,..., x n ) π(θ θ )π(x, x 2,..., x n θ θ ) π(θ θ )π(x, x 2,..., x n θ θ ) + π(θ θ 2 )π(x, x 2,..., x n θ (0.25) (0.2) Σx i (0.8) n Σx i (0.25) (0.2) Σx i (0.8) n Σx i + (0.75) (0.6) Σx i(0.4) n Σx i 2 n 2 n + 6 Sn (0.75) (0.6) Σx i (0.4) n Σx i (0.25) (0.2) Σx i (0.8) n Σx i + (0.75) (0.6) Σx i(0.4) n Σx i 6 Sn 2 n Sn (d) When n 2, then the observation gives Σx i. The posterior distribution of θ in that case is π(θ θ Σx i ) , and π(θ θ Σx i ) 0.6 for the original prior. For the prior π given in part (c), the posterior distribution of θ is π(θ θ Σx i ) , and π(θ θ Σx i ) When n 00, then the observation gives Σx i 50. In that case, the original prior gives the following posterior distribution of θ: π(θ θ Σx i 50) , and π(θ θ Σx i ) for the original prior. For the prior π given in part (c), the posterior distribution of θ is π(θ θ Σx i ) , and π(θ θ 2 Σx i ). 4

5 (e) Given that Σx i k, we have, under the initial prior π, π(θ θ Σx i k) and π(θ θ 2 Σx i k) 2 (0.2)k (0.8) n k, 2 (0.2)k (0.8) n k + 2 (0.4)k (0.6) n k 2 (0.4)k (0.6) n k. 2 (0.2)k (0.8) n k + 2 (0.4)k (0.6) n k Since these denominators are the same, we may find the value of θ having the higher posterior probability by comparing numerators only: π(θ θ Σx i k) > π(θ θ 2 Σx i k) 2 (0.2)k (0.8) n k > 2 (0.6)k (0.4) n k 0.2 k 0.8 n k > k log 3 + (n k) log 2 > 0 k/n < log (after a little algebra). log 6 This shows that under the original prior π, θ θ has a higher posterior probability given Σx i than does θ θ 2, so long as k < 0.39n. For the prior π, similar algebra shows that θ θ has a higher posterior probability given Σx i than does θ θ 2, so long as k < 0.39n 0.6, approximately. Under the original prior, for n 2, ˆθ θ if and only if k 0. For n 00, ˆθ θ if and only if k {0,,..., 38}. Under the prior π, for n 2, ˆθ θ if and only if k 0, as before. For n 00, ˆθ θ if and only if k {0,,..., 38}, exactly the same as under the other prior. (f) The only time the two ˆθ s will disagree is if k (0.39n 0.6, 0.39n) k 0.39n (approximately). For example, if n 000, then the two different priors would yield different estimates of ˆθ only if k happened to be 390. As n goes to infinity, the probability of x taking any one particular value k goes to zero, so the region on which the two ˆθ s disagree has a probability going to zero. 5. (0 pt) Bickel & Doksum pg 74, problem.2.4 The solution to this problem is worked out in Gelman, Carlin, Stern, and Rubin ( Bayesian Data Analysis, aka The Red Book ), pp It involves more than one rather unpretty completing of a square. The result is as follows: Posterior predictive distribution of x n+ : x n+ x, x 2,..., x n No(µ n, σ0 2 + τ n 2 ), where µ n τ 2 n ( ««τ 0 2 n θ 0 + σ 0 2 x «τ 0 2 n + σ 0 2 τ0 2 + n σ 2 0 ). «, and Here, x n Σn ix i. The predictive distribution (marginal) of x n+ is just a special case of the predictive distribution when n 0, so we have x n+ No(θ 0, σ τ 2 0 ). 5

6 (b) As n goes to infinity we note that the predictive distribution of x n+ never changes. This is to be expected, since by not taking into account the data we re gaining no information. Note that the mean of x n+ is always the prior mean on θ, θ 0, and the variance of x n+ comes from two sources: the uncertainty in the observation (σ0 2) and the uncertainty in the prior estimate of θ (τ 0 2). However, as n goes to infinity the posterior predictive distribution of x n+ given x, x 2,..., x n behaves differently. The posterior mean, which is a weighted average of the prior mean and the sample mean, gets a heavier and heavier weight on the sample mean. And the variance has less and less of its uncertainty coming from the uncertainty in θ, so that nearly all the uncertainty comes from the uncertainty in the observation. In the limit, x n+ x, x 2,..., x n No( x, σ0) (0 pt) Bickel & Doksum pg 84, problem.5.3 The factorization theorem will be useful to determine a sufficient statistic for each example. p(x, x 2,..., x n θ) ( θx θ ) θ n x θ i θ n exp ((θ )Σ log x i ) (b) By the factorization theorem, Σ log x i is a sufficient statistic for θ. Note that taking the log was not necessary. Another sufficient statistic would be x i. The latter is more cumbersome computationally, so it is common in a situation like this to take the logs. p(x, x 2,..., x n θ) ( θax a exp( θx a ) ) exp (n log θ + n log a + (a )Σ log x i θσ (x a i )) (c) A sufficient statistic for θ is Σx a i. p(x, x 2,..., x n θ) θa θ A sufficient statistic for θ is Σ log x i. x θ+ i exp (Σ (log θ + θ log a (θ + ) log x i )) exp (n log θ + nθ log a (θ + )Σ log x i ) 6

7 7. (0 pt) Bickel & Doksum pg 85, problem.5.4 Suppose a bijection f exists between T and T 2. Then T (x) T (y) f(t (x)) f(t (y)) T 2 (x) T 2 (y). And the reverse is also true because f, being a bijection, has an inverse. (b) Let f(x) log(x). Then f( x i ) log( x i ) Σ log x i. f is a bijection, so these are equivalent statistics. (c) These are not equivalent statistics. Here s a counterexample: If x and x 2 4 then Σx i 5. And if y 2 and y 2 3, then we also have Σy i 5. But log + log 4 log 2 + log 3. So the information contained in these two statistics is not identical. (d) Σ(x i x) 2 Σ(x 2 i 2x i x + x 2 ) Σ(x 2 i ) 2 xσx i + n x 2 ( Σ(x 2 i ) 2 n Σx i ) (Σx i ) + n Σ(x 2 i ) 2 n (Σx i) 2 + n (Σx i) 2 Σ(x 2 i ) n (Σx i) 2 ( ) 2 n Σx i We can uniquely express the pair [Σx i, Σ(x i x) 2 ] in terms of the pair [Σx i, Σx 2 i ]. A little algebra shows that the reverse is also uniquely possible, with Σx 2 i Σ(x i x) 2 + n (Σx i) 2. Thus, a bijection between these two statistics exist, and they are therefore equivalent. (e) These are not equivalent statistics. Here s an example, let s begin with x 3, x 2 0, x 3 0, then we have T (3, 27)andT 2 (3, 6); now another set 3,,-, then again T (3, 27) but T 2 (3, 0). More generally, to obtain a counterexample, let s begin with x, x 2 2, and x 3 3. We have Σx i 6 and Σx 3 i 36. That system has two equations but three variables, and therefore more than one distinct solution to construct a counter example. 8. (0 pt) Bickel & Doksum pg 86, problem.5.4 π(θ)π(x θ) By definition, the posterior distribution of θ given X is π(θ X) R θ π(θ)π(x θ)dθ If the posterior distribution depends upon X only through T (X), then the likelihood function π(x θ) must also depend upon X only through T (X), for it is only in the likelihood function that X appears in the expression for the posterior distribution of θ given X. And if the likelihood function π(x θ) depends upon X only through T (X), then all X having a common T (X) belong to the same information class, and their distribution given T (X) will not depend upon θ. Thus, in that case, T (X) is a sufficient statistic. 7

8 (b) If T (X) is a sufficient statistic for θ, then by the factorization theorem, the distribution of X given θ can be factored thus: π(x θ) g(t (X), θ)h(x). But then the posterior distribution of θ given X can be expressed in a similar form, π(θ)g(t (X),θ)h(X) by substitution: π(θ X). Thus, the posterior distribution Rθ π(θ)g(t (X),θ)h(X)dθ of θ given X depends upon X only through T (X). 9. (0 pt) Bickel & Doksum pg 87, problem.6.2 (b) (c) η(θ) θ B(θ) n log θ T (x) n i log x i h(x) I (0,) (x :n ) f(x θ) Π n i θxθ i I (0,) (x :n ) exp(n log θ + (θ ) n log x i )I (0,) (x :n ) f(x θ) Π n iθaxi a exp( θx a i )I (0, ) (x :n ), (a is a constant ) n exp(n log θ θ x a i )an Π n i xa i I (0, ) (x :n ) η(θ) θ B(θ) n log θ T (x) n i xa i h(x) a n Π n ix a i I (0, ) (x :n ) i i f(x theta) Π n θa θ i I (a, ) (x :n ) x θ+ i exp(n(log θ + θ log a) (θ + ) n log x i )I (a, ) (x :n ) i η(θ) (θ + ) B(θ) n log θ nθ log a T (x) n i log x i h(x) I (a, ) (x :n ) 8

9 0. (0 pt) Bickel & Doksum pg 87, problem.6.4 The family of uniform distributions is not an exponential family. If X has pdf π(x θ) θ [0,θ](x), then π(x θ) takes the value 0 over a part of its domain that depends upon θ. But if a function f is in an exponential family expressed as f(x θ) h(x)e T (x)η(θ) B(θ), then the only way the function can take the value 0 is if h(x) 0, which cannot depend upon θ. (b) Rewriting this in a different form shows that it is a triangular function: π(x θ) 2x over x (0, θ) and π(x θ) 0 elsewhere. This is not an exponential family, θ 2 for the same reason as the uniform distribution in. The locations where π(x θ) 0 depends upon θ, but in an exponential family, such a set of locations could only depend upon x. (c) Once again, this function has probability mass equal to zero at lots of locations, but they depend upon θ. So it cannot be in any exponential family. (d) WHOOPS! Sorry we didn t catch this earlier this is kind of a trick question. In.6.3 Bickel & Doksum define curved exponential families, where (as in this example) the parameter space Θ has lower dimension (l ) than the natural parameter η(θ) or natural sufficient statistic T (X) (both k 2). The authors would say this is not an exponential family, since l < k. I feel the distinction is subtle and misleading so we will treat either yes (with k 2 and η, T as below) or no (with the explanation that l < k) as correct answers. If you answered no and did not receive credit for this problem, please bring your homework to either Zhenglei or to me for us to correct our grading error. Meanwhile, the yes answer we ll accept is: Yes, this is an exponential family. ( π(x θ) exp ) (x θ)2 2πθ 2θ2 ( exp ) log(2π) log θ (x θ)2 2 2θ2 ( exp ) log(2π) log θ 2 2θ 2 (x2 2xθ + θ 2 ) 2 x2 exp ( log(2π) log θ + ( 2θ + x 2 θ ) 2 ) ( exp 2 log(2π) ) exp ( x2 2 2θ + x ) 2 θ log θ ( exp 2 log(2π) ) ( exp [x, x 2 ] [ 2 θ, ) log(θ) 2θ 2 ] Here we have η(θ) [ θ, 2θ 2 ], T (x) [x, x 2 ], B(θ) log θ, and h(x) exp ( 2 log(2π) 2). 9

10 (e) π(x θ) 2(x+θ) +2θ exp (log 2 + log(x + θ) log( + 2θ)) The log(x + θ) part of this expression cannot be factored into a T (x) part and an η(θ) part. So this family is not an exponential family. (f) π(x θ) ( n ) x θ x ( θ) n x ( θ) ( ) n n ( θ) n ) x ( θ x ( θ) n θ ) ( ( ) ( θ) n exp log ( θ) n ( n x ) + x(log θ log( θ)) This is an exponential family. η(θ) log θ log( θ), T (x) x, B(θ) log ( ( θ) n ( θ) n ), and h(x) ( n x). 0