ESTIMATING VARIANCE CO~WONENTS FOR THE NESTED MODEL, MINQUE OR RESTRICTED MAXIMUM LIKELIHOOD

Transcription

1 ,. ESTIMATING VARIANCE CO~WONENTS FOR THE NESTED MODEL, MINQUE OR RESTRICTED MAXIMUM LIKELIHOOD by Francis Giesbrecht and Peter Burrows Institute of Statistics Mimeograph Series No Raleigh - June 1976

2 ' Estimating Variance Components for the Nested Model, MINQUE or Restricted Maximum Likelihood by Francis Giesbrecht and Peter Burrows ABSTRACT This paper outlines ~n efficient method for computing MINQUE or restricted maximum likelihood estimates of variance components for the nested or hierarchical classification model. The method does. not require storage of any large matrices and consequently does not impose limits on the size of the data set. Numerical examples indicate that the computations converge very rapidly.

3 . ' Estimating Variance Components for the Nested Model, MINQUE or Restricted Maximum Likelihood Francis Giesbrecht and Peter Burrows I. Introduction In a series of papers Rao (1970, 1971a, 1971b, 1972) has developed the MINQUE (MInimum Norm Quadratic Unbiased Estimation) theory of estimating variance components. These papers provide a complete solution to the problem in the sense that if normality holds then the method will give locally best (minimum variance) ~uadratic unbiased estimates of the variance components. Unfortunately, the form of the estimation e~ua,ti.ons given is such as to tax even a very large computer if one has a reasonably large data set since they involve inversion of an n x n matrix where n is the number of data points. Also the procedure requires prior. information about the relative magnitudes. of the variance components. The possibility of an iterative scheme seems reasonable, but places an even larger premium on efficient computations. Hemmerle and Hartley (1973) develop a method for computing maximum likelihood estimates of the variance components. Corbeil and Searle (1976) modify the Hemmerle and Hartley algorithm to give restricted maximum likelihood estimates by partitioning the likelihood into two parts and then maximizing the part which is free of the fixed effects. Howeve;r, both algorithms require the repeated inversion of a ~ x ~ matrix, where ~ is the total number of random levels in the model. Typically ~ will be much smaller than U, though for large data sets even that rapidly becomes unreasonable. Both algorithms are iterative and as such re~uire only reasonable initial starting values for the unknown variance components rather than prior estimates. The object of this paper is to show that for the hierarchal model the

4 2 MINQUE formulas can be re-written in a form that requires only that a P x P matrix be inverted, where P is the number of variance components or levels of nesting in the model. It is shown that all necessary totals can be obtained by oo.e pass through the data set. It is also shown that if the MINQUE process is iterated (with some provision to replace negative components with small positive values) and converges, then the final answers will also satisfy. the equations obtained by applying the restricted maximum likelihood method. II. MlllQUE With Invariance and Normality Following Rao (1971a, 1971b) let Y be an n-vector of observations with structure (1) where X is a known nx s matrix of rank s, 13 Parameters, U. is an s-vector of unknown is a known n x c. matrix and s. is a c.-vector of normal random ~ J. J. ~. variables with zero means and dispersion matrix I a~ of appropriate dimension. ~ ~ It is also assumed that s. and e. are uncorrelatedfor all i.;. j. ~ J Clearly F.[y] = Xfi Jiy] = criv1 + + (J2y pp and where for i = 1,., p. The quadrati~unbiased and invariant (invariant in the sense that the estimates are unchanged if we replace the vector of observationsy by Y + Xb for any s-vector b) estimate of + given by theminque principle is

5 3 where Q i I = Y RViRY, R = V*-l(I_P), P = X(X/V*-~f~/V*-l, and the {A.) are suitable (derivable) constants for arbitrary {P.). ~ ~ Replacing V* by + a V pp amounts to selecting a different norm and disturbs neither the unbiasedness nor invariance. but unknown [cr~} 1. Further, if the {a.} happen tobe proportional to the true ~ then the estimates obtained will also be the minim.um variance quadratic unb.iased invariant estimates at that particular point in the parameter space. If all variance components are to be estimated simultaneously, rather than only a particular linear function,then the [Ai} need never be computed. The procedure is to compute the [Q.}, equate to their expectations and solve the 1. resulting system of equations. This can be written as = (2) where Sij = tr(rvirv j ). It seems reasonable that if prior knowledge of the relative magnitudes of

6 4 the unknown variance components is available then this information should be used to define a set of {O'i) and the matrix V'*- replaced by V in (2). Since (2) is a linear equation and the right hand sides are quadratic forms, the variancecovariance matrix of the estimated variance components follows immediately (in terms of the true variance components and {O'.). J. Formally this provides a complete solution for the variance component estimation problem. However the computational aspects of obtaining the elements of (2) directly are formidable. They involve inverting an n x n matrix. Consequently the examination of special cases is in order. Of particular interest here is the nested analysis of variance model, since La Motte (1972) -l has obtained V in a manageable form. In. Connection with Restricted Maximum Likelihood In order to obtain the maximum likelihood estimates of the v~riance components we proceed in two steps. The first is to assume a~ = O'i for i = 1,, p e and estimate~. This yields ~ = (X'V-~)-~'v-ly. The second step is to compute residuals,. Z = Y - X~ = Q Y v and apply maximum likelihood to obtain estimates of {a~} Define Z * = T Y v where Tv consists of n - s linearly independent rows of Q v The likelihood can now be written as K s., = ---"-1 IT VT'\2 v v exp r_~ Y'T v ' (T VT')-~ Y}. " v v v 2 Treating T as a constant (the la.} have been replaced by to'j.'}) gives the v J.

7 .. 5 likelihood equations, ~..en 1 2 ~ a. ~ for i = 1, 2,, p. = -itr[(t VT,)-IT V.T '] v v v ~ v +.l.y't' (T VT' )-IT V. T' (T VT,)-lT Y 2 V V V V ~ V V V V Equating the partial derivatives to zero and rearranging terms leads to fori = 1, 2,.., p. tr[t'(t VT,)-IT V.] v v v v ~ = 'lit' (T VT I rlt V. T I (T VT,)-IT Y I vvv v~vvv v Khatri (1966) has shown that for any symmetric positive definite matrix C, Direct substitution into (3) gives for i = 1, 2,.., p. Also we can replace V by V to obtain '"-1 I "'-L. "'-1 trey v~ v ~ "V Q,.V.) = Y Q",V v.v G",Y (4) for i =1, 2,, p. The quadratic forms in (4) are similar to those required by MINQUE, with the only difference being the nature of V or V. In MINQuE theory V is a function of f~.} while in maximum likelihood estimation theory Vis a function of (&~}.. ~.. ~ Also if the [0'.1 are chosen correctly i.e. agree with fa~1. ~ ~ then the expected values of the quadratic forms obtained under the MINQUE principle can be written as

8 6 It follows immediately that if one uses an iterative scheme for solving the maximum. likelihood equations in which the {a.} are updated to agree with the ~ (ail (with some provision to replace negative values with some small positive constants) the solution will agree exactly with that obtained from the MI:NQUE equations using the same rules for iteration. IV. Estimation in the Nested Model The development in sections II and In has been in terms of a relatively general linear model. For the remainder of this paper we will specialize to the 3:-level nested model with only the one fixed parameter ~. The 3 -level \,..., ne6ted model was selected because it is sufficiently complex to display the general pa~tern of terms that arise when more levels are introduced without the notation becomi~ too complex. The extension to the case where the rendom effects are nested within levels of a fixed effect is obvious and will not be discussed. The 3 -level nested model is given by Y hijk = J.L + ~ +b. + c hij + e hijk (5) h~ for k= 1,2,.., n hij. j = 1, 2,.., ~i i= 1, 2,.., ~ h= 1, 2,.., m and where (Y } are the observations, hijk ~ is an unknown constant, (a ) are independent N(O, h ai), (bhil are independent N(O, a~), {c } are independent N(O, hij a~), (ehijkl are independent N(O, a~) and

9 ' 7 the latter 4 sets are also mutually independent. Since we will have occasion to re~er to the value obtained by summation over certain subscripts, we will use the convention o~ indicating such a sum by replacing the subscript with the + symbol. For example ~.+ = E~.. 1: j J.J and etc. To establish the connection between (1) and (5), note that Y corresponds to the n+++ elements {Y hijk } in d~ctionary order. X is' a column of lis and ~ the one element ~. U l corresponds to an n+++ x m matrix of 0'sand l's, U 2 to an n+++ x m+ matrix o~ 0 t sand l's, U 3 to an n+++ x m++ matrix and U 4 the n+++ x n+++ identity matrix. The covariance between Y hijk andyh'i'j'k' can be written as where i~ L = m elm = i~ t :/ m The corresponding element o~ V can be written as where a. represents the prior estimate ~or a~. Since only the relative J. J. magnitudes of the ta.} are important they can be scaled so that one o~ J. them is 1. Also if one wishes the more primitive form of the MI~UE, Rao (1972), all can be set all equal to 1. The extension to more levels of nesting is immediate. The corresponding element of V- l can be written as

10 8 where,w hij = 0'3(0'3 n hij + 0(4)-1, w hi = 0'2(0'2 n + )",1 hi 0'3 w h = 0'1(0'1 n h + 0'2)-1, W = (t w h nh)-l h. n hi = ; ~ijwhij and n h = ~ nhiw hi J. Summa~ion over hi, ii, j' and k ' yields as ~he elemen~ of ~he vec~or v-lx corresponding ~o Yhijk in Y. Also no~e that the scalar X'v-lx is O'~lw-1 The general element of R now follows as --;,. - w h ',w h ' whw;w'h J.J J.. IW h I, J. IW h J. J 0'1 I, I, I -1 The pattern for the case with more levels is obvious. The typical element of Z = RY is

11 ,. 9 - whi,w h ' (t w h ' "Y h, "+. )0'3-1 J ~" ~J ~J. J -W h,,whowh(t I: w h ' IW h I "Y h, I, '+)C{2-1 ~J ~,I" ~ ~ J ~ J ~ J - Whi,Wh,whw(t I: E Wh/Wh"/Wh/ "/Yh"/j '+)C{1-1, J ~ h/i/j' ~ ~ J ~ where The quadratic forms in (2) are now obtained as 2 E E t t zh' 'k h i j k ~J, t tt Z~,,+', h i j ~J 2 E E zh'++ h i J; and 2 E zlrt++, h where the + symbol indicates summation over the subscript before squarihg. We note at this point that these quadratic forms are obtained by two passes through the data file and are not affected by size of data set. In order to obtain expressions for the elements in RV 1 ' RV 2 ' RV 3 ' RV 4 and the various products in a systematic pattern it is convenient to define -1-1 t h = C{2 + whw 0'1 ' -J. + t hi = C{3 whiwht h and -J. t hij = C{4 + WhijWhithi The elements in RV 1 ' RV 2 ' RV 3 and RV 4 are given in Table 1 in a convenient form. Expressions for tr(rv,rv,) for 4 ~ i ~ j ~ 1 are given in AppendixA. ~ J These expressions can (and should) be condensed before being used for computations. However when this is done the pattern is not as clear.. We also

12 .10 'notice that if one arrives at a point where the estimates obtained agree with the prior values then the variance-covariance matrix of the estimates can be e approximated by twice the inverse of the left side of 2.

13 e e e Table I. Elements in the RV l, RV 2, RV 3, and RV 4 Matrices Coefficient RV 4 RV 3 RV 2 RV 1 &hh,iiu'&jj,li kk, -1 0'4 -w hij t hij WhijWhiWh 0'1 -I\w h wo'l Whij(~31_~ijWhijWhithi} WhijW~i(O';l_~iWhiwhth} ( -1-1) 6 hh,6 U,6jj I (l-li kk,) -Whijthij Whij(n;l_I\ijWhijWhithi} WhijWhi(O';l_I\iWhiWh~) ( -1-1) WhijWhiWh 0'1 -I\w h WO'l li hh,l'u,(1-6 jj,} -WhijWhiWhij,thi -WhijWhiWhij'~ij,thi WhijWhi(O';l_~iWhiwhth} ( -1-1) WhijWhiWh 0'1 -I\WhWO'l 6 hh, (1-6 U ') -WhijWhiWhWhi,Whi'jth -WhijWhiWhWhi'Whi/j'~i'j,tn -w hij w hi w h w bi '~i' ~ ( -1-1) WhijWhiWh 0'1 -~whwo'l. (1-6 hh,) WhijWhiWhWWh,wh'i,Wh'i'j'O'l -WhijWhiWhWWh,wh'i,Wh'i'j'~'i'j'O'l -WhijWhiWhWWh,wh'i'~'i'O'l -w hij w hi w hwwh'~ '0'1...

14 12 V. Computational Aspects An examination of the formulas developed in the previous section shows that a very efficient computing routine is possible. If the data are sorted or arranged according to the nesting structure then all totals required in equation (2) can be obtained in one pass. Alternatively, if one wishes to minimize the programming effort, all the accumulations can be obtained rather easily by using any one of a number of standard programs for computing means and totals of subsets of observations in a data set. \ Summaries need to be obtained at each level of nesting. These summaries are then merged and a final set of accumulations obtained. In order to examine the behavior of the suggested iterative scheme several analyses were performed on synthetic, unbalanced data sets. The results of three typical cases are given in Table II. of these computations is the very rapid convergence. The striking feature In fact it appears that 2 or at most 3 cycles are sufficient. The extent of the unbalance in the data can be judged from Table III. VI. Acknowledgement The authors wish to thank T. M. Gerig for his valuable comments and suggestions in the preparation of this paper.

15 ,. 13 TABLE II Summary ofth~ee Analyses (,2 0'1 2 0'3 4 A. True parameter va1ues* Starting values Estimates first cycle Estimates second cycle Estimates third cycle Oll Estimates fourth cycle Estimates fifth cyc1e EstimatesA.o.V B. True parameter values* Starting values Estimates first cycle Estimates second cycle Estimates third cycle l.l4~ Estimates fourth cycle c. True parameter values Starting values Estimates first cycle Estimates second cycle Estimates third cycle Estimates fourth cycle Estimates fifth cycle Estimates A.o.V

16 TABLE III Pattern of ~ij's in Simulated Data Sets h i ,1 2 1,1,2 3,1,1 2,1,3 3,3 2 3,1 3 3,2,1 2 11,2 2,1 2 2,1,1 2,3,1 2,3 2 2,2,1 1,3,3 Sets 3 I 2,1 2,3,1 1,3 1,1 2,3 1 1,1 A & B 4 3 2,2 3 3, ,1 2, l,2,2 1, ,g,1 1,1 3, ,1 1 1,1,3 2,2 1,2,3 2,3,1 1,1,1 1 Set 3 I 2 2 2,1,3 3 1,3,2 2,3,1 2,3,3 2 c 4 I 1,3,1 3,3,3 2,2 1 3, ,3, ~. e, e e

17 ,. 15 REFERENCES Corbeil, R. R. and Searle, S. R. (1976). Restricted maximum likelihood (REML) estimation of variance components in the mixed model. Technometrics 18, Hemmerle, W. J. and Hartley, H. a. (.1973). Computing maximum likelihood estimates for the mixed A.a.V. model using the W-transformation. Technometrics 15, Kh~tri, C. G. (1966). A note on a manova model applied to problems in growth curve. Ann. Inst. Stat. Math. 18, L&~tte, L. R. (1972). Notes on the covariance matrix of a random nested anova model. Ann. Math. Statist. 43, Rao, C. R. (1970). Estimation of heteroscedastic variances in linear models. J. Amer. Statist. Assoc. 65, Rae, C. R. (197la). MmQUE theory. Estimation of variance and covariance components- J. Mult. Analysis 1, Rae, C. R. (1971b). Minimum variance quadratic unbiased estimation of variance components. J. Mult. Analysis 1, Rae,C. R. (1972). Estimation of variance and covaria.l'lce components in linear models. J. Amer. Statist. Assoc. 67,

18 A-l, APPENDIX I: «L L L n. 0 0 w 0 0w 0 w ) h h i j n ~J h ~J h ~ h (L L OWhOjWho)wh)(L Ln.. owho owho)whw cx o 0 n~j ~ ~ 0 j n~j ~J ~ I ~ J ~ -I -1. tr(rv 4 RV 3 ) = L L L owho 0(cx 4 - W oot 00)(cx3-11. OOWhOjWhotho) h i j n~j ~J n~j n~j n~j ~ ' + I: L L «L. n. 00 who 0 ) - 11.iowho 0)11. h i j j owho owhotho n~j ~J n J ~J h1j ~J L «L L L n. i ow ho ow h ow. h ) h h i j h J 1J (I: L ~rh 00W 0 W h h» (1.: L n 0 ow 00W 0)w.hW CX 0 h1j 1J n~j h 1J h 1 I i J ~ J

19 , < A :: 1:: 1:: n...(n... - l)w: h.wh t. (0'2 - n..wh W.ht h ).'. m.j h~j ~J]. h~j h~ ~ h ~ J. (( :: 1:: 1:: 1:: n...w.,w.) - h (1:: n.. w.wh' h h ))n..w. wht hi i j hlj lj 1 j. hlj lj 1. hll h h, L: «1:: 1:: 1:: n... W"h' w h w h ) h.h i j n~j ~J ~ (.J ~ ~»( 2 ~ )-rl -2 - _.;.y~.,-l:j,..--~ij hij hi h ~ ~i hi h 0'1 ~ ~ :: 1:: 1:: n...(n.. - 1)'W h "1 h wht.., (0'1 - n. w h w0'1 ) h i j n~j n~j ~J ~ hj.j h L: «1:: 1:: 1:: n...w h.w h. 1ITḣ) h h i j. n1.j J.J 1.

20 A tr(rv 3 RV 3 )... I: I: I: 11.,.W h, (0'3.- n.,.whi wh,t h,) h i j hj.j J.J hj.j J J. J. «2 2 2) + I: I: 'E 'E n., w h i i j,,wḣ, hj.j ḣ J.J J. 2" !: «'E 'E 'E n.,.w ḣ '. wḣ,w h ). h h i j hj.j J.J J." ('E 1:: n., w. h,.wh,w.h»(:e :E n. "w h ' W.h )Whw 0'1,, hj.j J.J J. 'j hj.j J.J J. J. J J tr(rv 3 RV 2 )... :E :E:E n.,.w. h '.w h ' (0'3 - n...w.,.w h ' t h ) (0'2 - n.,w h ' wht h ) h i j hj.j J.J J. hj.j hj.j J. J. hj. J I: 'E 'E «:E n. i w'h'.) - n.,.w h '.)n.,.w h '.w ḣ t h, (0'2-11.,w h ' wht.. ) h i jj h J J.J. hj.j J.J hj.j J.J J. J. hj.. J. h I: «'E :E :E n. i w h '.w h ' w h ) h h i j h J J.J! J (1:::En.,.w h '.wh,)wh)(:e n.,wh )whw 0'1 i j hj.j J.J. J. i hj. J.

21 A I: I: I: _(n )w. h - W ḣ.wḣ (O'3 - n. -.wḣ ".w h -t h -)(O'1 -.n. w h wo'1 ) h i jhj.j. hj.j J.J J.... hj.j.. J.J.. J.1 h I: I: I: «I: w h -) W: h - -) w h -.wḣ.w.ht h. (0' w h wo'1 ) h i j j h1j 1J h1j 1J h1j 1J 1 1 h + ( 1).; w 2 (-1. W w ) 2 I: ~ ~ ~ij ~ij" hij hi 0'2.. ~i hi h~h h J. J 2.? 2': 2..}J2 + 1: I: «I: n. - -W -) ) 11. h - hth h i i h1 h1 h1 1. h1 1

22 A-5 ~.,. + ~ ~ ( 1)W 2 w 2 w ( -1 w W t )(. -1 W w. -1) ~ i j ~ij ~lij - hij hi h 0'2 - ~i hi h h 0'1 - ~ h 0' 'Wḣ. wḣ) h hi m. J. + t «t tn.. + ( W) W ~ if-( -1 W w_.- 1 )2 I:. ~ ~ ~i - ~ij hij ~ij hij hi h 0'1 - ~ h 'u1 h J. J