36-720: Log-Linear Models: Three-Way Tables

Transcription

1 36-720: Log-Linear Models: Three-Way Tables Brian Junker September 10, 2007 Hierarhial Speifiation of Log-Linear Models Higher-Dimensional Tables Digression: Why Poisson If We Believe Multinomial? Produt Multinomial Log-linear Models September 10, 2007

2 Hierarhial Speifiation of Log-Linear Models A full set of log-linear models for 3-way tables that we have onsidered so far inlude: Log-Linear Model Generator M ( 1) log m i jk = u [0] M (0) log m i jk = u+u 1(i) + u 2( j) + u 3(k) [1][2][3] M (1) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 23( jk) [1][23] M (2) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 13(ik) [2][13] M (3) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 12(i j) [3][12] M (4) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 13(ik) + u 23( jk) [13][23] M (5) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 12(i j) + u 23( jk) [12][23] M (6) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 12(i j) + u 13(ik) [12][13] M (7) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 12(i j) + u 13(ik) + u 23( jk) [12][13][23] M (8) log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 12(i j) + u 13(ik) + u 23( jk) + u 123(i jk) [123] September 10, 2007

3 These model speifiations have several noteworthy features: They obey the hierarhy priniple: If the k-way interation is in the model then every lower order interation and main effet is also in the model. For example, sine M (7) ontains u 12(ik) terms then we also know it ontains u 1(i) and u 2( j) terms). Under the hierarhy priniple, nested models are obtained by adding or dropping higher-order interations in the model. Following the hierarhy priniple, a model an be ompletely speified by speifying the indies of the highest-order interations (the generators). E.g.: [1][23] u 1(i) and u 23( jk) are in the model u+u 1(i) and u+u 2( j) + u 3(k) + u 23( jk) are in the model log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 23( jk) is the model This is how R s linear modeling notation works also:shl + Risk*Beh expands to1 + Shl + Risk + Beh + Risk:Beh. The generators are useful mnemonis for What the (onditional) independene model is, e.g. [12][13] 2 3 1; What the suffiient statistis are, e.g. for [12][13] the suffiient statistis are n 12(i j) and n 13(ik) September 10, 2007

4 These ten hierarhial models an be organized by nesting as follows: [123] [12][13][23] [13][23] [12][23] [13][12] [1][23] [2][13] [3][12] [1][2][3] [0] September 10, 2007

5 Example: Residual Devianes (LR vs [123]) in boys deviane data Take 1=Beh, 2=Risk, 3=Shl: [123] (0/0) [12][13][23] (0.9/2) [13][23] (1.9/3) [12][23] (4.1/4) [13][12] (11.3/4) [1][23] (5.6/5) [2][13] (12.8/5) [3][12] (14.9/6) [1][2][3] (16.4/7) [0] (100.7/11) September 10, 2007

6 Higher-Dimensional Tables Christensen provides data on the relationship between two drugs (k=1, 2) and musle tensions (l=1, 2) for two weights (i=1, 2) and types ( j=1, 2) of musles in mie. Drug (k) Tension (l) Weight (i) Musle ( j) Drug 1 Drug 2 High Type Type High Low Type Type Low High Type Type Low Type Type Whih log-linear models best desribe this data? September 10, 2007

7 It is already painful to write down the saturated model [1234] in hierarhial log-linear form: log m i jk = u+u 1(i) + u 2( j) + u 3(k) + u 4(l) + u 12(i j) + u 13(ik) + u 14(il) + u 23( jk) + u 24( jl) + u 34( jl) + u 123(i jk) + u 124(i jl) + u 134(ikl) + u 234( jkl) + u 1234(i jkl) However, the generator notation makes model speifiation easier: The independene model [1][2][3][4] onsists of the first line above. The model of no three-way interation [12][13][14][23][24][34] onsists of the first two lines above. The model of no four-way interation [123][124][134][234] onsists of the first three lines above. The model [123][4] (or [WMD][T]) spefiies that musle weight, type and drug are independent of tension. The model [TMD][WMD] speifies that (tension) (weight) (type,drug) September 10, 2007

8 As an illustration we fit three models n <- san()... Wt <- rep(rep(("h","l"),(4,4)),2) # [W]eight Mt <- rep(rep(("1","2"),(2,2)),4) # [M]usle type Dt <- rep(("1","2"),8) # [D]rug type Tn <- rep(("h","l"),(8,8)) # [T]ension mus <- data.frame(n,wt,mt,dt,tn) Independene: [T][W][M][D] glm(n Tn + Wt + Mt + Dt, data = mus, family = poisson) No three-way interation: [TW][TM][TD][WM][WD][MD] glm(n Tn*Wt + Tn*Mt + Tn*Dt + Wt*Mt + Wt*Dt + Mt*Dt,...) No four-way interation: [TWM][TWD][TMD][WMD] glm(n Tn*Wt*Mt*Dt - Tn:Wt:Mt:Dt,...) Model Resid. Df Resid. Dev P[χ 2 d f> Dev] Independene No three-way e-09 No four-way September 10, 2007

9 Digression: Why Poisson If We Believe Multinomial? Maximum likelihood for the Poisson model We re-index a table suh as n i jk to be just n, and we suppose that n, =1,..., C are independent Poisson ounts with means m. Then f (n m ) = e m m n n! = exp[n log m m log n!] = exp[n θ expθ log n!] whih is an exponential family model with natural parameterθ = log m. If we modelθ = log m linearly as θ C 1 = [log m ] C 1 = X C D β D 1 then the above is a generalized linear model for Poisson ounts with the log link funtionθ = logµ September 10, 2007

10 The log-likelihood may be written L(n β) = f (n β) = exp (n log m m ) log n! = exp nt Xβ exp([xβ] )+g(data) and, either expliitly setting derivatives equal to zero, or by using the general theory of glm s, we see that the likelihood equations redue to (n m )x d = 0, d= 1,..., D This is the usual result that the MLE in an exponential family model equates observed and expeted suffiient statistis n T X= m T X In log-linear models for tables, these are invariably appropriate marginal totals for the table (see example, next slide) September 10, 2007

11 Sub-digression: Where does X ome from? Consider the table n i jk with i=1, 2, j=1, 2, and k=1, 2. If we lay out the log expeted ell ounts in a olumn, the model of independene log m i jk = u+u 1(i) + u 2( j) + u 3(k) looks like this: θ= log m 111 log m 112 log m 121 log m 122 log m 211 log m 212 log m 221 log m 222 = u u 1(2) u 2(2) u 3(2) = Xβ Clearly the observed suffiient statistis for the u parameters are n T X= (n +++, n 2++, n +2+, n ++2 ) whih are equated by ML to the expeted suffiient statistis m T X= (m +++, m 2++, m +2+, m ++2 ) September 10, 2007

12 Bak to the Poisson Likelihood Further differentiation (or appliation of general results from glm s) shows that the information matrix forβis [ 2 ] = X T ŴX β d1 β d2 where W= diag( ˆm) and hene D D Var (β)=[x T ŴX] 1 Sine the MLE s under the saturated model (no relations among the m i s) are ˆm i = n i, the log-lr statisti for testing against the saturated model is [ ] 2[log L(n β) log L(n saturated)] = 2 n log(n / ˆm ) n + ˆm = 2 n log(n / ˆm ) sine n + = ˆm + as long as the log-linear model has the interept u in it September 10, 2007

13 Sine the individual ells ounts n i Poiss(m i ), with E[n i ]=Var (n i )=m i, it follows that the Pearson residuals r i = (n i ˆm i )/ ˆm i are approximately mean 0, variane 1 (and should be approximately normal for large n i ). This is why they are sensible residuals to use. Reall that the model deviane (LR statisti for testing against the saturated model) is G 2 = 2 n i log(n i / ˆm i )= The deviane residuals are i r D i = sgn (n i ˆm i ) d i and these are approximately normally distributed (sine they are signed square roots of approximately 1-df ontributions to G 2 ). i d i September 10, 2007

14 Maximum likelihood for the Multinomial model We again take a table like n i jk and re-index the ells as n 1,...,n C. The key observation here is that if we ondition on n + = n, C independent Poisson s beome a C-ell multinomial. Begin with a Poisson log-linear model with an intereptα, log m =α+ x T β (=u+[other u-terms] ) where (1, x T ) is the th row of X. Up to a funtion of the data g(data)= log n!, the Poisson log-likelihood is log L(n α, β) = n log m = m n (α+ xβ) T exp(α+ xβ) T = n + α+ n xβ τ T whereτ= m = exp(α+ x T β)=e α exp(x T β) September 10, 2007

15 Now sine logτ=α+log exp(x T β), it follows that log L(n α, β) = log L(n τ, β) = n xβ n T log exp(xβ) T + [n + logτ τ] = n xβ T n log exp(x T β) + [n + logτ τ] = n log p + [n + logτ τ] where p = m /m + = exp(α+ x T β)/ exp(α+ xt β)= exp(x T β)/ exp(xt β). The term in brakets is the multinomial log-likelihood, up to a funtion of the data h(data)=log n +! log n!. Thus, as log-likelihoods: Poisson α,β (n)={multinomial β (n) n + }+[Poisson τ (n + ) ] September 10, 2007

16 That is worth repeating: For the log-linear model log m =α+ x T β, Poisson α,β (n)={multinomial β (n) n + }+[Poisson τ (n + ) ]. as log-likelihoods. Therefore: There is a 1-1 orrespondene between Multinomial log-linear models and Poisson log-linear models with the same log-linear form. The Poisson model requires one more parameter (τ), orresponding to the grand total n +. The (non-interept) parametersβin the two models have the same MLE s and the same variane-ovariane matrix (sine theβ s only enter in the braketed expressions on the previous slide). The interept parameterαwill be different in the two models. The Pearson residuals are still variane-stabilized, and as long as the grand total n + is large relative to the ell ount n i, eah r i will be roughly N(0, 1). You an also show that the omputation and interpretation of G 2 and X 2 also do not hange between the models September 10, 2007

17 Produt Multinomial Log-linear Models A similar argument an be used to derive produt-multinomial MLE s from Poisson MLE s. The key to the equivalene between Poisson MLE s ˆβ and single-multinomial MLE s ˆβ was to inlude the intereptαorresponding to the fixed grand total n + in the log-linear model: log m =α+ x T β. Then as log-likelihoods Poisson α,β (n)={multinomial β (n) n + }+[Poisson τ (n + ) ]. The key to equivalene between Poisson MLE s and Produt Multinomial MLE s will be to inlude log-linear termsα (1),...,α (H) orresponding to the fixed margins n (1) +,..., n (H) +. Then as log-likelihoods Poisson α,β (n)= H h=1 { Multinomial β (n (h) ) n (h) + }+[Poisson τ (n (h) + ) ]. These results have been known for some time (e.g. Birh, 1963, JRSSB); a reent update/generalization is Lang (1996, JRSSB) September 10, 2007

18 Example Consider again the Aspirin and Heart Attak data: Myoardial Infartion Fatal Nonfatal Attak Attak Total Plaebo n 11 = 18 n 12 = 171 n 1+ = 189 Aspirin n 21 = 5 n 22 = 99 n 2+ = 104 Total n +1 = 23 n +2 = 270 n ++ = 293 If we onsider this to be observational data with only n ++ fixed and a multinomial model for n i j n ++, we an useglm(..., family=poisson) to fit log-linear models to this data as long as we inlude the interept u in eah log-linear model. If we onsider this to be a designed experiment so that the ells are produt multinomial with fixed row totals n 1+ and n 2+, then we an useglm(..., family=poisson) to fit log-linear models to this data as long as we inlude the u 1(i) terms in eah log-linear model September 10, 2007

19 Testing independene with the Single Multinomial Model: n <- san() Tx <- rep(("plaebo","aspirin"),(3,3)) Obs <- rep(("fatal","nonfatal","noattak"),2) aha.data <- data.frame(n,tx,obs) print(fit <- glm(n Tx + Obs,data=aha.data,family=poisson)) G 2 = on 2 d.f.; p , so again we rejet independene. Testing independene with the Produt Multinomial Model: Sine the model of independene already has log-linear terms (Tx, i.e. u 1(i) ) orresponding to the fixed row totals n i+, the same Poisson fit above also gives the results for the Produt Multinomial. In general, when H 0 already ontains terms orresponding to the fixed margins in the table, testing H 0 is idential under the Poisson, Multinomial, and Produt Multinomial sampling models September 10, 2007

20 Example Bak to the musle study in mie: Drug (k) Tension (l) Weight (i) Musle ( j) Drug 1 Drug 2 High Type Type High Low Type Type Low High Type Type Low Type Type We treated this data before as Poisson or single multinomial. In fat, it was a designed study with the total number of musles of eah type fixed in advane. So every log-linear model should have themterms (u 2( j) ) in it September 10, 2007

21 An advantage of hierarhial priniple is that the interesting models automatially ontain the main effets, so if the totals in one dimension are fixed, the sampling sheme doesn t matter for model fit/omparison: Model Resid. Df Resid. Dev P[χ 2 d f> Dev] [T][W][M][D] [TW][TM][TD][WM][WD][MD] e-09 [TWM][TWD][TMD][WMD] In the study, [T]ension and [W]eight of musle are measured on eah ombination of [M]usle type and [D]rug. Two more models of interest might be 1. [TMD][WMD]: (tension) (weight) (musle type,drug) 2. [TWM][DM]: (tension,weight) (drug) (musle type) In the real study, only the totals for musle [T]ype were fixed, so either of these models ould be fitted as well September 10, 2007

22 In another study of this type, perhaps musle [W]eight and [D]rug type would be fixed in advane. In that ase all models would need to inlude the [WD] interation: we ould fit and interpret [TMD][WMD] but not [TWM][DM] (without the [WD] interation, the totals for (weight) (drug) ombinations would not be fixed by the log-linear model; the produt multinomial model with [WD] margins fixed ould not be represented) September 10, 2007