Lecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Jonathan Marchini (University of Oxford) BS2a MT / 15

Size: px

Start display at page:

Download "Lecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Jonathan Marchini (University of Oxford) BS2a MT / 15"

Linda Boyd
5 years ago
Views:

1 Lecture 16 : Bayesian analysis of contingency tables. Bayesian linear regression. Jonathan Marchini (University of Oxford) BS2a MT / 15

2 Contingency table analysis North Carolina State University data. EC : Extra Curricular activities in hours per week. EC < 2 2 to 12 > 12 C or better D or F Let y = (y ij ) be the matrix of counts. Jonathan Marchini (University of Oxford) BS2a MT / 15

3 Frequentist analysis Usual χ 2 test from R. Pearson s Chi-squared test X-squared = , df = (3-1)(2-1) = 2, p-value = The p-value is , evidence that grades are related to time spent on extra curricular activities. Jonathan Marchini (University of Oxford) BS2a MT / 15

4 Bayesian analysis EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Jonathan Marchini (University of Oxford) BS2a MT / 15

5 Bayesian analysis Let p = {p 11,..., p 23 }. EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 Jonathan Marchini (University of Oxford) BS2a MT / 15

6 Bayesian analysis Let p = {p 11,..., p 23 }. Consider two models M I EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 the two categorical variables are independent Jonathan Marchini (University of Oxford) BS2a MT / 15

7 Bayesian analysis Let p = {p 11,..., p 23 }. Consider two models M I M D EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 the two categorical variables are independent the two categorical variables are dependent. Jonathan Marchini (University of Oxford) BS2a MT / 15

8 Bayesian analysis Let p = {p 11,..., p 23 }. Consider two models M I M D EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 D or F p 21 p 22 p 23 the two categorical variables are independent the two categorical variables are dependent. The Bayes factor is BF = P(y M D) P(y M I ). So we need to calculate the marginal likelihoods. Jonathan Marchini (University of Oxford) BS2a MT / 15

9 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Jonathan Marchini (University of Oxford) BS2a MT / 15

10 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 Dirichlet distribution z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. Jonathan Marchini (University of Oxford) BS2a MT / 15

11 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 Dirichlet distribution z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. A representation that makes the Dirichlet easy to simulate from is the following. Jonathan Marchini (University of Oxford) BS2a MT / 15

12 The Dirichlet distribution Dirichlet integral z 1 + +z k =1 Dirichlet distribution z ν z ν k 1 k dz 1 dz k = Γ(ν 1) Γ(ν k ) Γ( ν i ) Γ( ν i ) Γ(ν 1 ) Γ(ν k ) zν z ν k 1 k, z z k = 1 The means are E[Z i ] = ν i / ν i, i = 1,..., k. A representation that makes the Dirichlet easy to simulate from is the following. Let W 1,..., W k be independent Gamma (ν 1 ),... Gamma (ν k ) random variables, W = W i and set Z i = W i /W, i = 1,..., k. Jonathan Marchini (University of Oxford) BS2a MT / 15

13 Examples of 3D Dirichlet distributions Jonathan Marchini (University of Oxford) BS2a MT / 15

14 Calculating marginal likelihoods P(y M D ) = p P(y p)π(p)dp Jonathan Marchini (University of Oxford) BS2a MT / 15

15 Calculating marginal likelihoods P(y M D ) = = p P(y p)π(p)dp p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ij Jonathan Marchini (University of Oxford) BS2a MT / 15

16 Calculating marginal likelihoods where P(y M D ) = = p P(y p)π(p)dp p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ( ) y = ( y ij )!/ y ij! y ij Jonathan Marchini (University of Oxford) BS2a MT / 15

17 Calculating marginal likelihoods P(y M D ) = = p P(y p)π(p)dp p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ij where ( ) y = ( y ij )!/ y ij! y Under M D choose a uniform distribution for p i.e. Dirichlet(1,..., 1) Jonathan Marchini (University of Oxford) BS2a MT / 15

18 Calculating marginal likelihoods P(y M D ) = = p P(y p)π(p)dp p p 23 =1 ( ) y ( ) yij pij π(p)dp11 dp 23 y ij where ( ) y = ( y ij )!/ y ij! y Under M D choose a uniform distribution for p i.e. Dirichlet(1,..., 1) π(p) = Γ(RC), p p 23 = 1 Jonathan Marchini (University of Oxford) BS2a MT / 15

19 Calculating marginal likelihoods P(y M D ) = ( ) y Γ(RC) y p p 23 =1 ( ) yij pij dp11 dp 23 ij Jonathan Marchini (University of Oxford) BS2a MT / 15

20 Calculating marginal likelihoods P(y M D ) = = ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) p p 23 =1 ( ) yij pij dp11 dp 23 ij Jonathan Marchini (University of Oxford) BS2a MT / 15

21 Calculating marginal likelihoods P(y M D ) = = = ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) ( ) y D(y + 1) y D(1 RC ) p p 23 =1 ( ) yij pij dp11 dp 23 ij Jonathan Marchini (University of Oxford) BS2a MT / 15

22 Calculating marginal likelihoods P(y M D ) = = = where ( ) y Γ(RC) y ( ) y Γ(yij + 1) Γ(RC) y Γ( y + RC) ( ) y D(y + 1) y D(1 RC ) p p 23 =1 D(ν) = Γ(ν i )/Γ( ν i ) ( ) yij pij dp11 dp 23 ij and y + 1 denotes the matrix of counts with 1 added to all entries and 1 RC denotes a vector of length RC with all entries equal to 1. Jonathan Marchini (University of Oxford) BS2a MT / 15

23 Calculating marginal likelihoods Under M I the probabilities are determined by the marginal probabilities p r = {p 1, p 2, } and p c = {p 1, p 2, p 3 } EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 p 1 D or F p 21 p 22 p 23 p 2 p 1 p 2 p 3 Jonathan Marchini (University of Oxford) BS2a MT / 15

24 Calculating marginal likelihoods Under M I the probabilities are determined by the marginal probabilities p r = {p 1, p 2, } and p c = {p 1, p 2, p 3 } EC < 2 2 to 12 > 12 C or better p 11 p 12 p 13 p 1 D or F p 21 p 22 p 23 p 2 p 1 p 2 p 3 Under M I we have a table where p ij = p i p j. Under independence M I the row sums and column sums have a Dirichlet distribution (with R=2 and C = 3 respectively) π(p r ) = Γ(R) Γ(1) p p 1 1 R = Γ(R), π(p c ) = Γ(C) Γ(1) p p 1 1 C = Γ(C) Jonathan Marchini (University of Oxford) BS2a MT / 15

25 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij Jonathan Marchini (University of Oxford) BS2a MT / 15

26 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j Jonathan Marchini (University of Oxford) BS2a MT / 15

27 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j ( ) y Γ(yi + 1) Γ(y j + 1) = Γ(R)Γ(C) y Γ( y + R) Γ( y + C) Jonathan Marchini (University of Oxford) BS2a MT / 15

28 The marginal likelihood under M I is therefore ( ) y ( ) yij P(y M I ) = pi p j π(pr )π(p c )dp r dp c y p r p c ij ( ) y = Γ(R)Γ(C) (p i ) y ( ) i dp y j r p j dpc y p r i p c j ( ) y Γ(yi + 1) Γ(y j + 1) = Γ(R)Γ(C) y Γ( y + R) Γ( y + C) ( ) y D(yR + 1)D(y = C + 1) y D(1 R )D(1 C ) Jonathan Marchini (University of Oxford) BS2a MT / 15

29 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) Jonathan Marchini (University of Oxford) BS2a MT / 15

30 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) Jonathan Marchini (University of Oxford) BS2a MT / 15

31 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is The Bayes factor is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) 11!68!3!9!23!5!1!2! 124! !121! 5!20!91!8!82!37! = 1.66 Jonathan Marchini (University of Oxford) BS2a MT / 15

32 Bayes Factor Combining the two marginal likelihoods we get the Bayes Factor Our data is The Bayes factor is BF = P(y M D) P(y M I ) = D(y + 1)D(1 R )D(1 C ) D(1 RC )D(y R + 1)D(y C + 1) 11!68!3!9!23!5!1!2! 124! !121! 5!20!91!8!82!37! = 1.66 which gives modest support against independence. Jonathan Marchini (University of Oxford) BS2a MT / 15

33 Normal Linear regression model Model: Response variable y, predictor variables x 1,..., x k. Jonathan Marchini (University of Oxford) BS2a MT / 15

34 Normal Linear regression model Model: Response variable y, predictor variables x 1,..., x k. ( E y i β, X) = β 1 x i1 + β k x ik, i = 1,..., n Jonathan Marchini (University of Oxford) BS2a MT / 15

35 Normal Linear regression model Model: Response variable y, predictor variables x 1,..., x k. ( E y i β, X) = β 1 x i1 + β k x ik, i = 1,..., n In terms of vectors x i = (x i1,..., x ik ) and β = (β 1,..., β k ), ( ) E y i β, X = x i β Jonathan Marchini (University of Oxford) BS2a MT / 15

36 Normal Linear regression model Model: Response variable y, predictor variables x 1,..., x k. ( E y i β, X) = β 1 x i1 + β k x ik, i = 1,..., n In terms of vectors x i = (x i1,..., x ik ) and β = (β 1,..., β k ), ( ) E y i β, X = x i β {y i } are conditionally independent given values of the parameters and the predictor variables. Jonathan Marchini (University of Oxford) BS2a MT / 15

37 Normal Linear regression model Assume that var(y i θ, X) = σ 2 where unknown parameters are θ = (β, σ 2 ) Jonathan Marchini (University of Oxford) BS2a MT / 15

38 Normal Linear regression model Assume that var(y i θ, X) = σ 2 where unknown parameters are θ = (β, σ 2 ) Assume that the errors ( ) ɛ i = y i E y i β, X are independent with mean 0 and variance σ 2. Jonathan Marchini (University of Oxford) BS2a MT / 15

39 Normal Linear regression model Assume that var(y i θ, X) = σ 2 where unknown parameters are θ = (β, σ 2 ) Assume that the errors ( ) ɛ i = y i E y i β, X are independent with mean 0 and variance σ 2.That is y β, σ 2, X N n (Xβ, σ 2 I) Jonathan Marchini (University of Oxford) BS2a MT / 15

40 Normal Linear regression model Assume that var(y i θ, X) = σ 2 where unknown parameters are θ = (β, σ 2 ) Assume that the errors ( ) ɛ i = y i E y i β, X are independent with mean 0 and variance σ 2.That is y β, σ 2, X N n (Xβ, σ 2 I) Bayesian formulation: Assume that (β, σ 2 ) have a non-informative prior g(β, σ 2 ) 1 σ 2 Jonathan Marchini (University of Oxford) BS2a MT / 15

41 Posterior distribution q(β, σ 2 y) = q(β y, σ 2 )q(σ 2 y) Jonathan Marchini (University of Oxford) BS2a MT / 15

42 Posterior distribution q(β, σ 2 y) = q(β y, σ 2 )q(σ 2 y) The first density on the right is multivariate normal with mean β and covariance matrix V β σ 2, where β = (X X) 1 X y, V β = (X X) 1 Jonathan Marchini (University of Oxford) BS2a MT / 15

43 Posterior distribution q(β, σ 2 y) = q(β y, σ 2 )q(σ 2 y) The first density on the right is multivariate normal with mean β and covariance matrix V β σ 2, where β = (X X) 1 X y, V β = (X X) 1 Define the inverse gamma (a, b) density as proportional to y a 1 exp{ b/y} then the marginal posterior of σ 2 is inverse gamma ((n k)/2, S/2), where S = (y X β) (y X β) Jonathan Marchini (University of Oxford) BS2a MT / 15

44 The posterior density comes from a classical factorization of the likelihood 1 { (2πσ 2 ) n/2 exp 1 } 2σ 2 (y Xβ) (y Xβ) knowing that (y Xβ) (y Xβ) = (y X β) (y X β) + ( β β) X X( β β) Jonathan Marchini (University of Oxford) BS2a MT / 15

Foundations of Statistical Inference

Foundations of Statistical Inference Julien Berestycki Department of Statistics University of Oxford MT 2015 Julien Berestycki (University of Oxford) SB2a MT 2015 1 / 16 Lecture 16 : Bayesian analysis