STAT:5100 (22S:193) Statistical Inference I

Transcription

1 STAT:5100 (22S:193) Statistical Inference I Week 10 Luke Tierney University of Iowa Fall 2015 Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

2 Monday, October 26, 2015 Recap Multivariate normal distribution Linear combinations of random variables Copula models Conditional distributions Conditional PMFs and PMFs Conditional expectation Law of total expectation Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

3 Monday, October 26, 2015 Conditional Expectations Some useful properties of conditional expectation: E[g(Y ) + X Y ] = g(y ) + E[X Y ] E[g(Y ) Y ] = g(y ) E[g(Y )X Y ] = g(y )E[X Y ] This shows that E[g(Y )X ] = E[g(Y )E[X Y ]] since E[g(Y )X ] = E[E[g(Y )X Y ]] = E[g(Y )E[X Y ]] This property can be taken as the definition of E[X Y ]. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

4 Monday, October 26, 2015 Conditional Expectations A formal definition of conditional expectation: Definition Let X, Y be random variables, and assume that E[ X ] <. A conditional expectation E[X Y ] of X given Y is any random variable Z such that for some measurable function h, and Z = h(y ) E[Xg(Y )] = E[Zg(Y )] for all bounded measurable functions g. Theorem A version of the conditional expectation E[X Y ] exists, and any two versions are almost surely equal. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

5 Monday, October 26, 2015 Conditional Expectations A useful consequence of the properties of conditional expectation: Theorem Suppose X and Y are random variables and g is a function such that E[X 2 ] < and E[g(Y ) 2 ] <. Then E[(X g(y )) 2 ] = E[(X E[X Y ]) 2 ] + E[(E[X Y ] g(y )) 2 ] Notes E[(X g(y )) 2 ] is the mean squared error for using g(y ) to predict X. The function of Y with the smallest mean square prediction error is E[X Y ]. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

6 Monday, October 26, 2015 Conditional Expectations Proof. E[(X g(y )) 2 ] =E[(X E[X Y ] + E[X Y ] g(y )) 2 ] =E[(X E[X Y ]) 2 ] + E[(E[X Y ] g(y )) 2 ] + 2E[(X E[X Y ])(E[X Y ] g(y ))] To complete the proof we need to show that the cross product term is zero. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

7 Monday, October 26, 2015 Conditional Expectations Proof (continued). For any function h with E[h(Y ) 2 ] < E[(X E[X Y ])h(y )] = E[E[X E[X Y ] Y ]h(y )] and So E[X E[X Y ] Y ] = E[X Y ] E[X Y ] = 0. E[(X E[X Y ])h(y )] = 0. The cross product expectation corresponds to and is therefore zero. h(y ) = E[X Y ] g(y ) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

8 Monday, October 26, 2015 Conditional Expectations Corollary If X, Y are random variables with E[X 2 ] < then Var(X ) = E[Var(X Y )] + Var(E[X Y ]) Proof. Take g(y ) = E[X ]: Var(X ) = E[(X E[X ]) 2 ] = E[(X E[X Y ]) 2 ] + E[(E[X Y ] E[X ]) 2 ] = E[E[(X E[X Y ]) 2 Y ]] + E[(E[X Y ] E[E[X Y ]]) 2 ] = E[Var(X Y )] + Var(E[X Y ]) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

9 Monday, October 26, 2015 Conditional Expectations Example Consider again the example of N customers making total purchases S = N X i. i=1 Suppose the independent purchase amounts have finite variance σ 2. The conditional variance of S given N = n is ( N ) Var(S N = n) = Var X i N = n i=1 ( n ) = Var X i N = n i=1 ( n ) = Var X i i=1 fix upper limit independence of X i and N = nσ 2 mutual independence of X i. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

10 Monday, October 26, 2015 Conditional Expectations Example (continued) The conditional variance of S given N is the random variable Var(S N) = Nσ 2. The variance of S is therefore Var(S) = E[Var(S N)] + Var(E[S N]) = E[Nσ 2 ] + Var(Nµ) = E[N]σ 2 + Var(N)µ 2 = λσ 2 + λµ 2 Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

11 Monday, October 26, 2015 Conditional Expectations Corollary Let X, Y be random variables E[X 2 ] <. Then for any g such that E[g(Y ) 2 ] < E[(X g(y )) 2 ] E[(X E[X Y ]) 2 ] Proof. From the theorem, E[(X g(y )) 2 ] = E[(X E[X Y ]) 2 ] + E[(E[X Y ] g(y )) 2 ] E[(X E[X Y ]) 2 ]. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

12 Monday, October 26, 2015 Conditional Expectations Conditional Expectation and Orthogonality A distance among random variables can be defined as X Y = E(X Y ) 2. In terms of this distance E[X Y ] is the closest function of Y to X. Put another way: Among all functions of Y the function with the lowest mean squared error for predicting X is E[X Y ]. The result that E[(X E[X Y ])h(y )] = 0 for all h can be interpreted as: The prediction error X E[X Y ] is orthogonal to the set of all functions of Y. E[X Y ] can be viewed as the orthogonal projection of X onto the set of all random variables that are functions of Y. There are strong parallels to least squares fitting in regression analysis. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

13 Monday, October 26, 2015 Hierarchical Models Hierarchical Models Often it is useful to build up models from conditional distributions. Example Each customer visiting to a store buys something with probability p. Customers make their purchasing decisions independently. Let X = number who buy something N = number who come to store Then X N Binomial(N, p). Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

14 Monday, October 26, 2015 Hierarchical Models Example (continued) Suppose the number of customers who arrive in a given period has a Poisson(λ) distribution. This is a two-stage hierarchical model: X N Binomial(N, p) N Poisson(λ) If the store is chosen at random, then λ would vary from store to store. This produces a three-stage hierarchical model: X N, λ Binomial(N, p) N λ Poisson(λ) λ f p might also vary from store to store. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

15 Monday, October 26, 2015 Hierarchical Models Example (continued) The marginal distribution of X is a mixture distribution. In the two-stage model, X is a Poisson mixture of binomials: f X (x) = P(X = x) = n P(X = x, N = n) = n P(X = x N = n)p(n = n) = So X Poisson(pλ). n=x ( n )p x n x λn (1 p) x n! e λ = (pλ)x e λ 1 [(1 p)λ]n x x! (n x)! n=x = (pλ)x x! e λ+(1 p)λ = (pλ)x e pλ x! Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

16 Monday, October 26, 2015 Hierarchical Models Example (continued) With a third stage, P(X = x) = = 0 0 P(X = x λ)f (λ)dλ (pλ) x e pλ f (λ)dλ x! What forms of f are both flexible and convenient? Gamma is a natural choice: f (λ) = 1 Γ(α)β α λα 1 e λ/β 1 (0, ) (λ) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

17 Monday, October 26, 2015 Hierarchical Models Example (continued) Then the marginal PMF of X is (pλ) x P(X = x) = e pλ 1 0 x! Γ(α)β α λα 1 e λ/β dλ p x = x!γ(α)β α λ x+α 1 e λ(p+1/β) dλ 0 p x Γ(x + α) = x!γ(α)β α (p + 1/β) ( x+α ) Γ(x + α) p x ( ) 1/β α = Γ(α)x! p + 1/β p + 1/β For α a nonnegative integer this is a negative binomial distribution. For non-integer α this also a negative binomial distribution this is a definition. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

18 Wednesday, October 28, 2015 Recap Properties of conditional expectation Variance decomposition Geometry of conditional expectation Hierarchical models Poisson mixture of binomial distributions Gamma mixture of Poisson distributions Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

19 Wednesday, October 28, 2015 Hierarchical Models Example Suppose we observe N = n customers visiting the store. What is the conditional distribution of the store s λ value? The joint density/pmf of λ and N is f (λ, n) = f N λ (n λ)f (λ) = λn 1 n! e λ Γ(α)β α λα 1 e λ/β = λn+α 1 n!γ(α)β α e λ(1+1/β). The conditional density of λ given N = n is therefore f λ N (λ n) = f (λ, n) f N (n) f (λ, n) λn+α 1 e λ(1+1/β) = λ n+α 1 e λ(β+1)/β. This corresponds to a Γ(n + α, β/(1 + β)) distribution. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

20 Wednesday, October 28, 2015 Hierarchical Models Example A poll of size n is to be take to see whether voters prefer Candidate A or Candidate B. Assuming sampling with replacement, the number X in the sample who favor A is Binomial(n, p) with p the population proportion who favor A. The race is close and we are fairly confident that p is between 0.4 and 0.6. We can capture this by thinking of p as a random variable with a distribution that puts about 90% probability between 0.4 and 0.6. This is our prior probability distribution on p. Once we have collected our data we can compute a posterior probability like P(p > 0.5 data). This is an example of Bayesian inference. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

21 Wednesday, October 28, 2015 Hierarchical Models Example (continued) A convenient form for the prior distribution is a Beta(α, β) distribution. The joint density/pmf of X, p is then of the form f (x, p) = f X p (x p)f (p) ( ) n = p x (1 p) n x 1 x B(α, β) pα 1 (1 p) β 1. The posterior density of p given X = x is f (p x) = f (x, p) f X (x) f (x, p) p x+α 1 (1 p) n x+β 1. This is the density of a Beta(x + α, n x + β) distribution. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

22 Wednesday, October 28, 2015 Hierarchical Models Example (continued) A Beta distribution with α = β = 33 is symmetric about 0.5 and assigns approximately 0.9 probability to the interval between 0.4 and 0.6. Results for some possible sample sizes and observed counts: x n p P(p > 0.5 data) ,046 2, All three scenarios produce approximately the same p-value for testing H 0 : p 0.5 against H 1 : p > 0.5. Some plots: p <- seq(0, 1, len = 101) plot(p, dbeta(p, 33, 33), type = "l", ylim = c(0, 40)) abline(v = 0.5, lty = 2) lines(p, dbeta(p, , ), col = "red") lines(p, dbeta(p, , ), col = "forestgreen") lines(p, dbeta(p, , ), col = "blue") Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

23 Wednesday, October 28, 2015 Hierarchical Models Example Suppose X, Y are bivariate normal with standard normal marginals and correlation ρ. The joint density is f (x, y) = { 1 2π 1 ρ exp x 2 + y 2 } 2ρxy 2 2(1 ρ 2. ) The conditional density of Y given X = x is { f Y X (y x) exp y 2 } 2ρxy 2(1 ρ 2. ) This is a N(ρx, 1 ρ 2 ) density. For a general bivariate normal distribution ( ) x µ X Y X = x N µ Y + ρσ Y, σy 2 (1 ρ 2 ). σ X Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

24 Wednesday, October 28, 2015 Hierarchical Models Example Suppose X 1, X 2, given M = m, are independent N(m, 1). The marginal distribution of M is N(0, δ 2 ). Then X 1, X 2 are bivariate normal with E[X 1 ] = E[X 2 ] = E[E[X 1 M]] = E[M] = 0 Var(X 1 ) = Var(X 2 ) = E[Var(X 1 M)] + Var(E[X 1 M]) The covariance of X 1, X 2 is So the correlation is = 1 + Var(M) = 1 + δ 2. Cov(X 1, X 2 ) = E[X 1 X 2 ] = E[E[X 1 X 2 M]] = E[E[X 1 M]E[X 2 M]] = E[M 2 ] = δ 2. ρ = δ2 1 + δ 2. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

25 Wednesday, October 28, 2015 Change of Variables Change of Variables Suppose X, Y are jointly continuous on a set A. Let U, V be defined as U = g 1 (X, Y ) V = g 2 (X, Y ) The image of A under the transformation g is B = {(u, v) : u = g 1 (x, y) and v = g 2 (x, y) for some (x, y) A} Assume g = (g 1, g 2 ) is one-to-one on A. Then g has an inverse h = (h 1, h 2 ) defined on B, and X = h 1 (U, V ) Y = h 2 (U, V ) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

26 Wednesday, October 28, 2015 Change of Variables A small rectangle [u, u + du] [v, v + dv] has area dudv. The image of this rectangle under h is (approximately) a parallelogram. (u, v) (x, y) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

27 Wednesday, October 28, 2015 Change of Variables The area of this parallelogram is approximately J(u, v) dudv where ( x J(u, v) = dxdy dudv = det u y u = x y u v x y v u x v y v ) J is called the Jacobian determinant of the transformation. This generalizes to three or more variables in the obvious way. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

28 Wednesday, October 28, 2015 Change of Variables The density of U, V for (u, v) B can be derived as f U,V (u, v)dudv = f X,Y (x, y)dxdy. Then f U,V (u, v) = f X,Y (x, y) dxdy dudv = f X,Y (x, y) J(u, v) Alternatively, f U,V (u, v) = f X,Y (h 1 (u, v), h 2 (u, v)) J(u, v) Here J(u, v) = det ( h1 (u,v) u h 2 (u,v) u h 1 (u,v) v h 2 (u,v) v ) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

29 Friday, October 30, 2015 Recap Simple Bayesian inference examples Gaussian hierarchical models Change of variables for jointly continuous random variables Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

30 Friday, October 30, 2015 Change of Variables Example Let X, Y be independent with X Gamma(α, 1) Y Gamma(β, 1) Define U = X /(X + Y ) V = X + Y Then A = (0, ) (0, ) B = (0, 1) (0, ). Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

31 Friday, October 30, 2015 Change of Variables Example (continued) The inverse transformation is The Jacobian is J(u, v) = det x = h 1 (u, v) = uv y = h 2 (u, v) = v uv = (1 u)v ( ) v u = v(1 u) + vu = v. v (1 u) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

32 Friday, October 30, 2015 Change of Variables Example (continued) The joint density of U and V for 0 < u < 1 and v > 0 is therefore f U,V (u, v) = f X (uv)f Y ((1 u)v)v = 1 Γ(α) (uv)α 1 e uv 1 Γ(β) ((1 u)v)β 1 e (1 u)v v 1 = Γ(α)Γ(β) uα 1 (1 u) β 1 v α+β 1 e v Incorporating indicator functions for B gives f U,V (u, v) = 1 [ u α 1 (1 u) β 1 1 (0,1) (u) ] [ v α+β 1 e v 1 (0, ) (v) ]. Γ(α)Γ(β) So U, V are independent with U Beta(α, β) V Gamma(α + β, 1) Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

33 Friday, October 30, 2015 Change of Variables Example Suppose X, Y are uniformly distributed on the region A = {(x, y) : 1 x 1, 1 y 1}. The joint density of X, Y is { 1 f X,Y (x, y) = 4 if 1 x 1 and 1 y 1 0 otherwise. Let U = X + Y V = X Y. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

34 Friday, October 30, 2015 Change of Variables Example (continued) The inverse transformation is The range of the transformation is x = u + v 2 y = u v. 2 B = {(u, v) : 2 u + v 2, 2 u v 2}, The Jacobian determinant of the inverse transformation is ( 1 ) 1 J(u, v) = det = = Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

35 Friday, October 30, 2015 v Change of Variables Example (continued) The joint density of U, V is therefore ( u + v f U,V (u, v) = f X,Y, u v ) J(u, v) 2 2 { 1 = 8 if 2 u + v 2 and 2 u v 2 0 otherwise. This is a uniform distribution on the square B: u v = 2 u + v = 2 u + v = 2 u v = Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall u

36 Friday, October 30, 2015 Change of Variables Example (continued) We can compute the marginal density of U by integrating out v from the joint density: f U (u) = f U,V (u, v)dv 2+u 1 2 u 8 dv = 1 8 2(2 + u) = u 4 if 2 u 0 = 2 u 1 u 2 8 dv = 1 8 2(2 u) = 1 2 u 4 if 0 u 2 0 otherwise { 1 = 2 u 4 if u 2 0 otherwise. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

37 Friday, October 30, 2015 Change of Variables Example Suppose X, Y are uniformly distributed on the unit disk The joint density of X and Y is A = {(x, y) : x 2 + y 2 1} f X,Y (x, y) = 1 π 1 A(x, y) Let U = X /Y V = Y Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

38 Friday, October 30, 2015 Change of Variables Example (continued) The inverse transformation is x = uv y = v. The range of the transformation is B = {(u, v) : u 2 v 2 + v 2 1} = {(u, v) : v 1/ 1 + u 2 }. The Jacobian determinant of the inverse transformation is ( ) v u J(u, v) = det = v 0 1 Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

39 Friday, October 30, 2015 Change of Variables Example (continued) The joint density of U, V is therefore The marginal density of U is f U (u) = f U,V (u, v) = f X,Y (uv, v) J(u, v) { v = π if v 1/ 1 + u 2 0 otherwise. f U,V (u, v)dv = 1/ 1+u 2 v = 2 0 This is a Cauchy density. π dv = v 2 1/ 1+u 2 v 1/ 1+u 2 π dv 1/ 1+u 2 π = 1 1 π 1 + u 2 0 Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

40 Friday, October 30, 2015 Change of Variables Two problems: 1. X, Y are continuous, independent. Find the density of U = X + Y. 2. X N(0, 1), Y χ 2 p, independent. Find the density of U = X / Y /p t p. Two possible approaches: a. identify the region U u in the x, y plane integrate the joint density over this region differentiate the result with respect to u b. Add another variable V so that (X, Y ) (U, V ) is one-to-one find the joint density of U, V integrate out V The second approach is often easier since it only requires a one-dimensional integral V can be chosen to make this integral easier Often taking V = X or V = Y is sufficient. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

41 Friday, October 30, 2015 Change of Variables Example For the first problem with U = X + Y take V = X. The inverse transformation is X = V Y = U V The Jacobian determinant of the inverse is ( ) 0 1 J(u, v) = det = So f U,V (u, v) = f X (v)f Y (u v) The marginal distribution of U is therefore f U (u) = f U,V (u, v)dv = This is called the convolution of f X and f Y. f X (v)f Y (u v)dv Transforms (e.g. moment generating functions) turn convolutions into products. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

42 Friday, October 30, 2015 Change of Variables Example For the second problem with U = X / Y /p set V = Y : Then the inverse transformation is U = X / Y /p V = Y X = U V /p Y = V The Jacobian determinant of the inverse transformation is ( v/p u J(u, v) = det ) 2p v/p = v/p. 0 1 The range of the transformation is B = {(u, v) : < u <, 0 < v < }. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

43 Friday, October 30, 2015 Change of Variables Example (continued) So the joint density of U and V is f U,V (u, v) = f X (u v/p)f Y (v) v/p { 1 = 2π exp { 1 2 u2 v/p } 1 v p/2 1 e v/2 v/p v > 0 Γ(p/2)2 p/2 0 v 0 Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall

44 Friday, October 30, 2015 Change of Variables Example (continued) The marginal density of U is therefore { 1 f U (u) = exp 1 } 1 2π 2 u2 v/p Γ(p/2)2 v p/2 1 e v/2 v/pdv p/2 = 0 1 2π pγ(p/2)2 p/2 0 v p e 1 2 v(1+u2 /p) dv = Γ ( ) p (p+1)/2 1 2πΓ(p/2) p2 p/2 (1 + u 2 /p) (p+1)/2 Γ ( ) p = Γ(p/2)(pπ) 1/2 (1 + u 2 /p) (p+1)/2 This is the density of Student s t distribution with p degrees of freedom. For p = 1 this is a Cauchy distribution. Luke Tierney (U Iowa) STAT:5100 (22S:193) Statistical Inference I Fall