Regression Analysis when there is Prior Information about Supplementary Variables Author(s): D. R. Cox Source: Journal of the Royal Statistical Society. Series B (Methodological), Vol. 22, No. 1 (1960), pp. 172-176 Published by: Wiley for the Royal Statistical Society Stable URL: http://www.jstor.org/stable/2983886 Accessed: 03-03-2017 13:46 UTC JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://about.jstor.org/terms Royal Statistical Society, Wiley are collaborating with JSTOR to digitize, preserve and extend access to Journal of the Royal Statistical Society. Series B (Methodological)
172 [No. 1, Regression Analysis when there is Prior Information about Supplementary Variables By D. R. Cox Birkbeck College, University of London [Received September, 1959] SUMMARY In estimating the regression coefficient of y on x, we can sometimes increase precision by using supplementary observations, u. We need to have prior information about the form of the relations among y, x, and u. An implication for experimental design is noted. CONSIDER a linear regression of the form 1. INTRODUCTION Yj= a+xi+ei (i= 1,...,n), (1) where the E's are uncorrelated random quantities with zero mean and constant variance cu2, and suppose that we wish to estimate /. More generally we may have a multiple regression of the form Yi=?+glxli+... (i=l,..., n) (2) and may wish to estimate 91, g..., p Suppose further that we have observations not only on y and x, but also on a supplementary variable u, or more generally on a set of supplementary variables ul,...,ur. If we can make prior assumptions about the form of the relationship between y, x and u, we may be able to use the observations u to obtain an estimate of / more precise than the straightforward sample regression coefficie y on x. A very familiar example is analysis of covariance, using a concomitant variable u to increase the precision of treatment comparisons. In the simplest case of a completely randomized design comparing p treatments, Xki = 1 if the ith observation received treatment k and Xki = 0 otherwise. The parameters 9..., / p are treatment effects, and the straightforward regression coefficients of y on x are the unadjusted sample means. The assumptions necessary to make use of u can be put in various ways. The simplest mathematically is to require that the term 0c+ Ei in (2) is of the form x' + yui + ei, where ei is a random quantity of zero mean and constant variance a2, different e's being uncorrelated with one another and with the values of x. Thus (2) becomes Yi = a-' + 9, xii + *** + 9p xpi + )'ui + ei (3) and the least-squares estimates of l,..., /3 p are the usual treatment estimates adjuste for regression on u. The prior assumption that ui is independent of the x's is of cours
1960] Cox - Regression Analysis with Prior Information of Variables 173 crucial; without it, even if the regression (3) were linear, the /'s in (3) would not be the same as those in (2). In this paper we consider situations where the prior information about y, x and u takes a rather different form. Some of what follows is similar to work on linear relations in econometrics, in particular to the study of instrumental variables. The object in this paper, however, is to obtain increased precision, not to secure identifiability. 2. A SIMPLE SPECIAL CASE Consider an industrial process with two stages. Suppose that in the ith run, xi i a measurement on the raw material before the first stage of processing, or the level of a factor governing the first stage of processing. For the same run, let yi measure a property of the final product, and let ui measure a property of the output from the first stage. Suppose that we are interested in the total regression coefficient of y on x, i.e. in the regression ignoring u. Sometimes it may be reasonable to postulate from general knowledge of the process that xi may affect u*, but that xi can only affect y, through the value of u*, i.e. that given ui, yi is independent of xi. For linear relationships, this can be expressed in the equations ui = A+ 8Uxi +,qi, (4) Yi = 0 + Oui + Ci, (5) where A. ju, 0, Q are unknown parameters and of zero mean and variances Ca"Gr,. Equations (4) and (5) lead to Yi = 0+ OA + Otxi + -i + (i = cx+3xi +,i, (6) say. Here / = OA2 and Ei cr2 = o 2 + a2? In equations (4) and (5) yi and ui are observed values of random vari the xi also are random we argue conditionally on the observed values. Th of / from (6) is the sample regression coefficient P* of y on x and o2 V(/*) = (7) css (x) or 2 u + 02 71bu a2(8 (8) css (x) where css (x) is the corrected sum of squares of the x*, Z(x* -)2. If we assume that i and G in (4) and (5) are independently normally distributed, the log-likelihood is - n log (2us - A -u X)2 (y2-o-u)2 (9) It follows that the maximum likelihood estimate of /3 = u is 0-2A, where $ respectively the simple regression coefficients of y on u and of u on x. Further it
174 Cox - Regression Analysis with Prior Information of Variables [No. 1, follows, for example from the information matrix, that $ and p: are asymptotically independent with U2 V($)= ( _ (10) css (u) at;, ~~~~~~~~(11) yu2css (x) +nu2 V(H)=. (12) css (x) Thus V(P3), 2 V( P)+ 2 V(H) (13) 2 CSS (X) (U2+ 02 r2q) + n02 ag (14) 9 1 1 (14) CSS (X) [[k2 C The estimate 3 is exactly unbiased; a more detailed study of its properties is facilitated by assuming that the xi are independently normally distributed. In the analysis of data we estimate V($), V(,a) by the usual formulae for the sampling variance of regression coefficients, and then substitute directly in (13). To compare theoretically the precision of the two estimates * and /, we have from (8) and (14) that ( V(/*) c k2css (x)+ + nn 2? (15) V() p2 CSS (X +n2[02 C2/(2 + 02 S2)]. Therefore the asymptotic variance of 3 is always less than the variance of /3*. An alternative form of (15) is obtained by defining Py,u and p,u by p2y [CS()71a7 (16) p _ 02 [112 css x)+ nur2] 2 nag 12 2 2css _P2U (x) /I l (17) (16) and (17) are, from (4) and (5), ratios of and hence the p's can be considered as c V(f*) P - PYu -Pu P18 V( 2 u+ p2 X-2p2 up2 u18 This ratio greatly exceeds one when both p2u and P2u are small, and approaches one when either or both of p 2u and pu are near one. This expresses the qualitatively obvious fact that appreciable additional precision can be obtained by introducing u only when u contains much information not already available in y and x. The distinction between /* and 3 can be seen by considering, in the usual notation for total and partial regression coefficients, the formula gvx= 9Y.u+Fgu.X9uX; (19) under our special assumption 9YX = 0, 9U z = /,g vz = ]ugux. (20) The estimate 3 corresponds to formula (20), the estimate ignoring u to the general formula (19).
1960] Cox - Regression Analysis with Prior Information of Variables 175 3. A NUMERICAL EXAMPLE To illustrate these formulae the artificial data in Table 1 were constructed using tables of random normal deviates. The estimate /* ignoring u, i.e. the sample regression coefficient of y on x, is 0 74 with an estimated standard error of 0-233. The estimate / is obtained as the product of the regression coefficient of y on u times that of u on x and is 0-973 x 0-610 = 0-593. The variances of the factors in this product are as estimated by the usual formulae and are combined by (13) to give an approximate standard error for of 0 197. In this example / is appreciably nearer than 6* to the true value: -=; the approximate efficiency of A* relative to 1 is (0-197/0-233)2 = 0-71. The prior assumption underlying the use of /B is that the partial regression coefficient of y on x given u is zero. It will, in practice, always be advisable to check that the data are consistent with the assumption. In the present example the sample partial regression coefficient is 0-215 with an estimated standard error of 0-217. TABLE 1. Artificial data for dependent variable y, independent variable x and supplementary variable u. y x U y x U y x 1 1 4 46 22 32 58 25 27 60 33 28 47 24 09 46 28 38 47 22 3 5 62 32 2-1 47 23 2.1 40 1 3 50 59 42 29 52 25 24 4 1 2-1 42 40 38 1.1 51 12 37 35 2-0 37 53 34 46 60 4-1 30 49 20 36 58 32 1 6 46 1.9 28 4-8 34 39 49 38 21 36 22 14 39 2-2 30 60 26 02 34 05 1 8 36 23 48 56 34 20 5-1 23 22 47 08 In equations (4), (5) the following values have been taken: A = 0 = 0, y = i, q = 1, with xi, -i, i normally distributed with means 5, 0 and 0 and unit variance. 4. GENERALIZATIONS The situation of section 2 can be generalized in several ways of which the following are examples. We may have the multiple regression (2) on several x variables; this includes as a special case the specification in a standard lay-out of observations in terms of treatment effects and row, column, etc., effects. With a single supplementary variab u, the equations analogous to (4) and (5) would be u* = A + tt xli + * * + Mp xpi + qt(21) y*= 0 +O?ui+ si, (22) leading to / t = b1i. The estimatio multiple regression of u on the x's, a likelihood estimate of gi is then $,&. enter into (22), for example the x's representing row and column effects in a Latin square; the estimate A is then a residual regression coefficient.
176 Cox - Regression Analysis with Prior Information of Variables [No. 1, If there are several supplementary variables u1,...,ut, the aim should be to set up, if appropriate, linear relations, with independent errors, for the random variables y,u1,..., ur. The equations should involve the x's linearly and should express the special prior knowledge about the system that is assumed to be available. Separate analyses of these equations yield maximum likelihood estimates of the parameters in the equations. The true regression coefficients 91,,fP of y on xi,..., xp can obtained as functions of the parameters in the starting relations, and hence maximum likelihood estimates of the P's are obtained. The whole procedure is an immediat generalization of that of section 2 and will not be discussed in detail here. It is possible to express in matrix form the conditions that have to be satisfied for the resulting estimate to be different from, and to have smaller variance than, the sample regression coefficients of y on xl,... xp ignoring ul,..., u,. An example would be the analysis of a randomized block experiment with a pair of supplementary variables of the type considered above and in addition a concomitant variable. 5. APPLICATION TO EXPERIMENTAL DESIGN Suppose that we are designing an experiment to compare a number of alternative treatments, the final observation in terms of which the treatments are to be compared being y. The results above show that the precision of the treatment comparisons can be improved by recording a supplementary variable u, possibly of no practical importance in itself, measuring the state of the experimental unit after the treatments have exerted their full effect, but before y can be measured. That is, there is to be a single regression relation between y and u, the same for all treatments. The purpose of such a supplementary variable is to remove the effect of random variation entering the system after the treatments have exerted their effect. As such it is to be contrasted with analysis of covariance using a concomitant variable, which aims at removing variation associated with the experimental units before the allocation of treatments. The correctness of the prior assumption underlying the covariance analysis is ensured by first measuring the concomitant variable and then applying the treatments in a random way, independently of the concomitant variable. The correctness of the assumption underlying the method proposed here cannot be ensured in a similar way, and remains a prior assumption requiring critical thought. Cases where the assumption can properly be made probably do not arise very frequently. If the assumption is satisfied, there should not be an appreciable variation between treatments in y after regression on u has been eliminated; this should always be checked, but it would, of course, be quite wrong to use u in the way described simply because the mean square of y between treatments adjusting for regression on u is insignificant. A more general use of "intermediate" variables is not for increasing precision but for "explaining" treatment effects on the principal variable y. I am grateful to members of the Operational Research Department, Steel Company of Wales, for asking some questions that led to this investigation.