Foundations of Statistical Inference

Transcription

1 Foundations of Statistical Inference Julien Berestycki Department of Statistics University of Oxford MT 2015 Julien Berestycki (University of Oxford) SB2a MT / 16

2 Lecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Julien Berestycki (University of Oxford) SB2a MT / 16

3 Example 2 2 From Wikipedia article on contingency tables: Left-handed Right-handed Total Male 9 (y 1 ) Female 4 (y 2 ) Total Hypothesis: θ 1 = Proportion of left-handed men > θ 2 = proportion of left handed women. Julien Berestycki (University of Oxford) SB2a MT / 16

4 Example 2 2 From Wikipedia article on contingency tables: Left-handed Right-handed Total Male 9 (y 1 ) Female 4 (y 2 ) Total Hypothesis: θ 1 = Proportion of left-handed men > θ 2 = proportion of left handed women. Model: y 1 Binom(n 1, θ 1 ), y 2 Binom(n 2, θ 2 ). Julien Berestycki (University of Oxford) SB2a MT / 16

5 Example 2 2 From Wikipedia article on contingency tables: Left-handed Right-handed Total Male 9 (y 1 ) Female 4 (y 2 ) Total Hypothesis: θ 1 = Proportion of left-handed men > θ 2 = proportion of left handed women. Model: y 1 Binom(n 1, θ 1 ), y 2 Binom(n 2, θ 2 ). Use uniform priors θ i U [0,1] =Beta(1, 1). Julien Berestycki (University of Oxford) SB2a MT / 16

6 Example 2 2 From Wikipedia article on contingency tables: Left-handed Right-handed Total Male 9 (y 1 ) Female 4 (y 2 ) Total Hypothesis: θ 1 = Proportion of left-handed men > θ 2 = proportion of left handed women. Model: y 1 Binom(n 1, θ 1 ), y 2 Binom(n 2, θ 2 ). Use uniform priors θ i U [0,1] =Beta(1, 1). Posteriors p(θ 1 y 1, n 1 ) = Beta(y 1 +1, n 1 y 1 +1), p(θ 2 y 2, n 2 ) = Beta(y 2 +1, n 2 y 2 +1) Then compute posterior P(Z 1 > Z 2 ) Julien Berestycki (University of Oxford) SB2a MT / 16

7 Example 2 2 From Wikipedia article on contingency tables: Left-handed Right-handed Total Male 9 (y 1 ) Female 4 (y 2 ) Total Hypothesis: θ 1 = Proportion of left-handed men > θ 2 = proportion of left handed women. Model: y 1 Binom(n 1, θ 1 ), y 2 Binom(n 2, θ 2 ). Use uniform priors θ i U [0,1] =Beta(1, 1). Posteriors p(θ 1 y 1, n 1 ) = Beta(y 1 +1, n 1 y 1 +1), p(θ 2 y 2, n 2 ) = Beta(y 2 +1, n 2 y 2 +1) Then compute posterior P(Z 1 > Z 2 ) Either compute an integral or by simulation. Julien Berestycki (University of Oxford) SB2a MT / 16

8 Example 2 2: simulations See R code. Generate M sample from joint posterior p(θ 1, θ 2 y 1, n 1, y 2, n 2 ) = p(θ 1 y 1, n 1 ) p(θ 2 y 2, n 2 ) and then use Monte-Carlo approximation P[θ 1 > θ 2 ] 1 (i) I(θ M 1 > θ(i) 2 ) Julien Berestycki (University of Oxford) SB2a MT / 16

9 Example 2 2: simulations See R code. Generate M sample from joint posterior p(θ 1, θ 2 y 1, n 1, y 2, n 2 ) = p(θ 1 y 1, n 1 ) p(θ 2 y 2, n 2 ) and then use Monte-Carlo approximation P[θ 1 > θ 2 ] 1 (i) I(θ M Outputs M=10000 : 1 > θ(i) 2 ) Posterior Simulation of Male - Female Lefties 2.5% 50% 97.5% print(mean(theta1>theta2)) [1] p(theta1 - theta2 y, n) theta1 - theta2 Julien Berestycki (University of Oxford) SB2a MT / 16

10 Contingency table analysis North Carolina State University data. EC : Extra Curricular activities in hours per week. EC < 2 2 to 12 > 12 C or better D or F Let y = (y ij ) be the matrix of counts. Julien Berestycki (University of Oxford) SB2a MT / 16

11 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test y i,j is cardinal of cell i, j < 2 2 to 12 > 12 C or better D or F Julien Berestycki (University of Oxford) SB2a MT / 16

12 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test Sum rows and columns < 2 2 to 12 > 12 total C or better D or F total Julien Berestycki (University of Oxford) SB2a MT / 16

13 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test E i,j = r i c j /N < 2 2 to 12 > 12 total C or better D or F total Julien Berestycki (University of Oxford) SB2a MT / 16

14 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test χ 2 = i,j (y i,j E i,j ) 2 /E i,j < 2 2 to 12 > 12 total C or better D or F total 6.92 Julien Berestycki (University of Oxford) SB2a MT / 16

15 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test χ 2 = i,j (y i,j E i,j ) 2 /E i,j < 2 2 to 12 > 12 total C or better D or F total 6.92 X-squared = , df = (3-1)(2-1) = 2, p-value = The p-value is , evidence that grades are related to time spent on extra curricular activities. Julien Berestycki (University of Oxford) SB2a MT / 16

16 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Julien Berestycki (University of Oxford) SB2a MT / 16

17 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Julien Berestycki (University of Oxford) SB2a MT / 16

18 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Julien Berestycki (University of Oxford) SB2a MT / 16

19 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Julien Berestycki (University of Oxford) SB2a MT / 16

20 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Consider two models M I the two categorical variables are independent Julien Berestycki (University of Oxford) SB2a MT / 16

21 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Consider two models M I the two categorical variables are independent the two categorical variables are dependent. M D Julien Berestycki (University of Oxford) SB2a MT / 16

22 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Consider two models M I (p 11, p 12, p 13 ) (p 21, p 22, p 23 ) M D the two categorical variables are dependent. The Bayes factor is BF = P(y M D) P(y M I ). Julien Berestycki (University of Oxford) SB2a MT / 16

23 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Let p = {p 11,..., p 23 }. The model is that Y = (y 1,1,..., y 2,3 ) is a multinomial (N, p) (i.e. N trials with P(X k = (i, j)) = p i,j and y i,j = #{X k = (i, j)}.) Bayesian method: make p a variable. Consider two models M I (p 11, p 12, p 13 ) (p 21, p 22, p 23 ) M D (p 11, p 12, p 13 ) (p 21, p 22, p 23 ) The Bayes factor is BF = P(y M D) P(y M I ). Julien Berestycki (University of Oxford) SB2a MT / 16

24 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Julien Berestycki (University of Oxford) SB2a MT / 16

25 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Dirichlet distribution Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. Julien Berestycki (University of Oxford) SB2a MT / 16

26 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Dirichlet distribution Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. A representation that makes the Dirichlet easy to simulate from is the following. Julien Berestycki (University of Oxford) SB2a MT / 16

27 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Dirichlet distribution Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. A representation that makes the Dirichlet easy to simulate from is the following. Let W 1,..., W k be independent Gamma (ν 1, θ),... Gamma (ν k, θ) random variables, W = W i and set Z i = W i /W, i = 1,..., k. (Does not depend on θ). Julien Berestycki (University of Oxford) SB2a MT / 16

28 Examples of 3D Dirichlet distributions Julien Berestycki (University of Oxford) SB2a MT / 16

29 Calculating marginal likelihoods The model is that f (y, p) is a P(y M D ) = P(y p)π(p)dp p Julien Berestycki (University of Oxford) SB2a MT / 16

30 Calculating marginal likelihoods The model is that f (y, p) is a P(y M D ) = P(y p)π(p)dp = p p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ij Julien Berestycki (University of Oxford) SB2a MT / 16

31 Calculating marginal likelihoods The model is that f (y, p) is a P(y M D ) = P(y p)π(p)dp = p p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ij where ( ) y = ( y ij )!/ y ij! y Julien Berestycki (University of Oxford) SB2a MT / 16

32 Calculating marginal likelihoods The model is that f (y, p) is a P(y M D ) = P(y p)π(p)dp where = p p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ( ) y = ( y ij )!/ y ij! y Under M D (p 1,1, p 1,2, p 1,3 ) (p 2,1, p 2,2, p 2,3 ) so choose a uniform distribution for p i.e. Dirichlet(1,..., 1) ij Julien Berestycki (University of Oxford) SB2a MT / 16

33 Calculating marginal likelihoods The model is that f (y, p) is a P(y M D ) = P(y p)π(p)dp where = p p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ( ) y = ( y ij )!/ y ij! y Under M D (p 1,1, p 1,2, p 1,3 ) (p 2,1, p 2,2, p 2,3 ) so choose a uniform distribution for p i.e. Dirichlet(1,..., 1) π(p) = Γ(RC), p p 23 = 1 ij Julien Berestycki (University of Oxford) SB2a MT / 16

34 Calculating marginal likelihoods P(y M D ) = ( ) y Γ(RC) y p p 23 =1 ( ) yij pij dp11 dp 23 ij Julien Berestycki (University of Oxford) SB2a MT / 16

35 Calculating marginal likelihoods P(y M D ) = = ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) p p 23 =1 ( ) yij pij dp11 dp 23 ij Julien Berestycki (University of Oxford) SB2a MT / 16

36 Calculating marginal likelihoods P(y M D ) = = = ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) ( ) y D(y + 1) y D(1 RC ) p p 23 =1 ( ) yij pij dp11 dp 23 ij Julien Berestycki (University of Oxford) SB2a MT / 16

37 Calculating marginal likelihoods P(y M D ) = = = where ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) ( ) y D(y + 1) y D(1 RC ) p p 23 =1 D(ν) = Γ(ν i )/Γ( ν i ) ( ) yij pij dp11 dp 23 ij and y + 1 denotes the matrix of counts with 1 added to all entries and 1 RC denotes a vector of length RC with all entries equal to 1. Julien Berestycki (University of Oxford) SB2a MT / 16

38 Calculating marginal likelihoods Under M I the probabilities are determined by the marginal probabilities p r = {p 1, p 2, } and p c = {p 1, p 2, p 3 } < 2 2 to 12 > 12 C or better p 11 p 12 p 13 p 1 D or F p 21 p 22 p 23 p 2 p 1 p 2 p 3 Julien Berestycki (University of Oxford) SB2a MT / 16

39 Calculating marginal likelihoods Under M I the probabilities are determined by the marginal probabilities p r = {p 1, p 2, } and p c = {p 1, p 2, p 3 } < 2 2 to 12 > 12 C or better p 11 p 12 p 13 p 1 D or F p 21 p 22 p 23 p 2 p 1 p 2 p 3 Under M I we have a table where p ij = p i p j. Under independence M I the prior for the row sums and column sums are independent uniform priors: Dirichlet distribution (with R=2 and C = 3 respectively) π(p r ) = Γ(R) Γ(1) p p 1 1 R = Γ(R), π(p c ) = Γ(C) Γ(1) p p 1 1 C = Γ(C) Julien Berestycki (University of Oxford) SB2a MT / 16

40 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij Julien Berestycki (University of Oxford) SB2a MT / 16

41 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j Julien Berestycki (University of Oxford) SB2a MT / 16

42 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j ( ) y Γ(yi + 1) Γ(y j + 1) = Γ(R)Γ(C) y Γ( y + R) Γ( y + C) Julien Berestycki (University of Oxford) SB2a MT / 16

43 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j ( ) y Γ(yi + 1) Γ(y j + 1) = Γ(R)Γ(C) y Γ( y + R) Γ( y + C) ( ) y D(yR + 1)D(y = C + 1) y D(1 R )D(1 C ) Julien Berestycki (University of Oxford) SB2a MT / 16

44 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) Julien Berestycki (University of Oxford) SB2a MT / 16

45 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) Julien Berestycki (University of Oxford) SB2a MT / 16

46 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is The Bayes factor is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) 11!68!3!9!23!5!1!2! 124! !121! 5!20!91!8!82!37! = 1.66 Julien Berestycki (University of Oxford) SB2a MT / 16

47 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is The Bayes factor is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) 11!68!3!9!23!5!1!2! 124! !121! 5!20!91!8!82!37! = 1.66 which gives modest support against independence. Julien Berestycki (University of Oxford) SB2a MT / 16

48 Normal Linear regression model Julien Berestycki (University of Oxford) SB2a MT / 16

49 Normal Linear regression model Model: Response variable n 1 vector Y = (y 1,..., y n ), predictor variables n p matrix X = (x 1,..., x p ). Julien Berestycki (University of Oxford) SB2a MT / 16

50 Normal Linear regression model Model: Response variable n 1 vector Y = (y 1,..., y n ), predictor variables n p matrix X = (x 1,..., x p ). Y = Xβ + ɛ, ɛ N(0, σ 2 I) Julien Berestycki (University of Oxford) SB2a MT / 16

51 Normal Linear regression model Model: Response variable n 1 vector Y = (y 1,..., y n ), predictor variables n p matrix X = (x 1,..., x p ). Y = Xβ + ɛ, ɛ N(0, σ 2 I) Recall that classical unbiased estimates are ˆβ = (X t X) 1 X T Y, ˆσ 2 = (Y X ˆβ) T (Y X ˆβ) Julien Berestycki (University of Oxford) SB2a MT / 16

52 Normal Linear regression model Model: Response variable n 1 vector Y = (y 1,..., y n ), predictor variables n p matrix X = (x 1,..., x p ). Y = Xβ + ɛ, ɛ N(0, σ 2 I) Recall that classical unbiased estimates are ˆβ = (X t X) 1 X T Y, ˆσ 2 = (Y X ˆβ) T (Y X ˆβ) and predicted Y is Ŷ = X ˆβ = P X Y, P X = X(X T X) 1 X T. Julien Berestycki (University of Oxford) SB2a MT / 16

53 Normal Linear regression model To sum up: Y β, σ 2, X N n (Xβ, σ 2 I) Julien Berestycki (University of Oxford) SB2a MT / 16

54 Normal Linear regression model To sum up: Y β, σ 2, X N n (Xβ, σ 2 I) Bayesian formulation: Assume that (β, σ 2 ) have a non-informative prior g(β, σ 2 ) 1 σ 2 Julien Berestycki (University of Oxford) SB2a MT / 16

55 Posterior distribution q(β, σ 2 Y ) = q(β Y, σ 2 )q(σ 2 Y ) Julien Berestycki (University of Oxford) SB2a MT / 16

56 Posterior distribution q(σ 2 Y ) q(β, σ 2 Y ) = q(β Y, σ 2 )q(σ 2 Y ) 1 (σ 2 ) ( n p IG 2 (n p)/2+1 exp, (n p)ˆσ2 2 Recall Inverse Gamma(a, b) is y a 1 exp{ b/y} } (n p)s2 { 2σ 2 ) Julien Berestycki (University of Oxford) SB2a MT / 16

57 Posterior distribution q(σ 2 Y ) q(β, σ 2 Y ) = q(β Y, σ 2 )q(σ 2 Y ) 1 (σ 2 ) ( n p IG 2 (n p)/2+1 exp, (n p)ˆσ2 2 Recall Inverse Gamma(a, b) is y a 1 exp{ b/y} } (n p)s2 { 2σ 2 ) where q(β Y, σ 2 ) = N( ˆβ, V β σ 2 ) β = (X T X) 1 X T Y, V β = (X T X) 1 Julien Berestycki (University of Oxford) SB2a MT / 16

58 Posterior distribution q(σ 2 Y ) q(β, σ 2 Y ) = q(β Y, σ 2 )q(σ 2 Y ) 1 (σ 2 ) ( n p IG 2 (n p)/2+1 exp, (n p)ˆσ2 2 Recall Inverse Gamma(a, b) is y a 1 exp{ b/y} } (n p)s2 { 2σ 2 ) where q(β Y, σ 2 ) = N( ˆβ, V β σ 2 ) β = (X T X) 1 X T Y, V β = (X T X) 1 Julien Berestycki (University of Oxford) SB2a MT / 16

59 Posterior The posterior density comes from a classical factorization of the likelihood 1 { (2πσ 2 ) n/2 exp 1 } 2σ 2 (y Xβ)T (y Xβ) knowing that (y Xβ) T (y Xβ) = (y X β) (y X β) + ( β β) T X T X( β β) Julien Berestycki (University of Oxford) SB2a MT / 16

60 Posterior The posterior density comes from a classical factorization of the likelihood 1 { (2πσ 2 ) n/2 exp 1 } 2σ 2 (y Xβ)T (y Xβ) knowing that (y Xβ) T (y Xβ) = (y X β) (y X β) + ( β β) T X T X( β β) P(β Y ) is a non-cenral multivariate t n p distribution. Julien Berestycki (University of Oxford) SB2a MT / 16

61 Posterior The posterior density comes from a classical factorization of the likelihood 1 { (2πσ 2 ) n/2 exp 1 } 2σ 2 (y Xβ)T (y Xβ) knowing that (y Xβ) T (y Xβ) = (y X β) (y X β) + ( β β) T X T X( β β) P(β Y ) is a non-cenral multivariate t n p distribution. For each j β j ˆβ j t n p s (X T X) 1 jj Julien Berestycki (University of Oxford) SB2a MT / 16

62 Prediction New covariate matrix X, predict Ỹ. Julien Berestycki (University of Oxford) SB2a MT / 16

63 Prediction New covariate matrix X, predict Ỹ. p(ỹ Y ) = p(ỹ β, σ2 )p(β, σ 2 Y )dβdσ 2 Julien Berestycki (University of Oxford) SB2a MT / 16

64 Prediction New covariate matrix X, predict Ỹ. p(ỹ Y ) = p(ỹ β, σ2 )p(β, σ 2 Y )dβdσ 2 Simulate or Julien Berestycki (University of Oxford) SB2a MT / 16

65 Prediction New covariate matrix X, predict Ỹ. p(ỹ Y ) = p(ỹ β, σ2 )p(β, σ 2 Y )dβdσ 2 Simulate or p(ỹ Y ) is a multivariate t distribution t n p ( X ˆβ, ˆσ 2 (I + X(X T X) 1 X T )) Julien Berestycki (University of Oxford) SB2a MT / 16