DERIVING PROPER UNIFORM PRIORS FOR REGRESSION COEFFICIENTS

Transcription

1 DERIVING PROPER UNIFORM PRIORS FOR REGRESSION COEFFICIENTS N. van Erp and P. van Gelder Structural Hydraulic and Probabilistic Design, TU Delft Delft, The Netherlands Abstract. In probles of odel coparison between copeting regression odels, one ust take care not to use iproper priors. Iproper priors introduce inverse infinities in the evidence factors, which do not cancel if one proceeds to copute the posterior probabilities of odels which have different nubers of regression coefficients. We therefore derive a siple and parsionious proper unifor prior for ultiple regression odels. We then look at the evidence values that result fro using this prior. Keywords: Bayesian Model Selection, Proper Unifor Priors, Regression Coefficients, Evidence Values, Invariance PACS: Cw INTRODUCTION There is a long tradition of the use of iproper unifor priors for regression coefficients, that is, location paraeters, in probles of paraeter estiation [1]. However, in probles of odel coparison between copeting regression odels one ust generally avoid iproper priors, whether unifor or not. This is because iproper priors will introduce inverse infinities in the evidence values which ay not cancel out if one coputes the corresponding posterior probabilities [2]. We shall therefore derive a parsionious proper unifor prior for univariate regression odels. This univariate prior is then generalized to its ultivariate equivalent. The resulting proper unifor prior has the nice property that it causes the scaled evidence values, e.g. the posterior probabilities of the copeting regression odels, to be invariant for changes in scale in both the dependent and independent variables. DERIVING PROPER UNIFORM PRIORS FOR THE UNIVARIATE CASE In what follows we derive the, trivial, liits of the univariate unifor prior for a single regression coefficient. The extension to the ultivariate case (in the next paragraph) is based upon the basic idea introduced here. Suppose we wish to regress an dependent vector y upon an independent vector x. Then, using atrix algebra, the regression coefficient β ay be coputed as β = x,y x,x = x y cosθ x 2 = y cosθ. (1) x

2 By exaing (1) we see that β ust lie in the interval y ax (cosθ) β y ax (cosθ) ax. (2) Since (cosθ) = 1 and (cosθ) ax = 1, interval (2) reduces to: y ax β y ax. (3) So, having prior knowledge of the ial length of the predictor,, and the axial length of the outcoe variable, y ax, we ay set liits to the possible values of β. It follows that the proper unifor prior of β ust be the univariate unifor distribution with liits as given in (3): p(β I) = 2 y ax, y ax β y ax, (4) where I stands for the prior background inforation that allows us to assign values to and y ax. DERIVING PROPER UNIFORM PRIORS FOR THE MULTIVARIATE CASE We now derive the liits of the ultivariate unifor prior for regression coefficients. The basic idea is a generalization of the very siple idea used to derive the liits for the univariate case. This generalization will involve a transition fro univariate line pieces to ultivariate ellipsoids. Say we have independent vectors x i that span soe -diensional subspace S. Let ŷ be the part of y that lies in S and let e be the part that is orthogonal to S, then y = ŷ + e. (5) Now, the projection ŷ is apped on the orthogonal base spanned by the vectors x i through the regression coefficients β i, that is, where ŷ = x i β i, (6) β i = x i,y x i,x i = x i,ŷ + e = x i,ŷ x i,x i x i,x i = ŷ x i cosθ i. (7) Because of the independence of the x i we have that x i,x j = 0 for i j. So, if we take the nor of (6) we find ŷ 2 2 = x i β i = ŷ 2 cos 2 θ i. (8)

3 It follows fro identity (8) that the angles θ i in (7) ust obey the constraint cos 2 θ i = 1. (9) Cobining (7) and (9), we see that all possible values of the coordinates β i ust lie on the surface of an -variate ellipsoid centered at the origin and with respective axes r i = ŷ x i. (10) Since ŷ y, the axes (10) ay be axiized through our prior knowledge of the ial lengths of the predictors and the axial length of the outcoe variable, that is, ax(r i ) = y ax x i. (11) It follows that all possible values of the regression coefficients β i ust lie in the -variate ellipsoid centered at the origin and with respective axes (11). If we substitute (11) into the identity for the volue of an -variate ellipsoid V = r i, (12) we find the volue of the paraeter space of all possible values of the β i, as V = x. (13) i Let X [x 1 x ]. Then for independent variables x i the product of the nors is equivalent to the square root of the deterant of X T X, that is, x i = X T X 1/2, (14) which is also the volue of the parallelepiped defined by the vectors x i. If the predictors x i are not independent we can transfor the to an orthogonal basis x i using the Gra-Schidt orthogonalization process. Let X [ x 1 x ]. Then, since the volue of the parallelepiped is invariant under orthogonalization, we have X T X 1/2 = X T X 1/2. (15) Upon aking the appropriate substitutions, we find the axiu volue of the prior accessible paraeter space of the β i, for both independent and dependent predictors, to be V ax = X T X 1/2. (16)

4 That is, the paraeter point (β 1,...,β ) ust lie in soe ellipsoid with axiu volue (16). To assign the proper unifor prior that spans the largest possible paraeter space, we let this proper prior be equal in value to the inverse of (16). It then follows that the ultivariate equivalent of (4) ay be written as p(β 1,...,β I) = where (β 1,...,β ) ellipsoid and X T X 1/2 = ( x i ) X T X 1/2, (17) 1 cosφ 12 ax cosφ 1 ax cosφ 12 ax 1 cosφ 2 ax cosφ 1 ax cosφ 2 ax 1 where cosφi j is the absolute value of the correlation between the predictors xi and x j and I stands for the prior background inforation that allows us to assign values to y ax and X T X 1/2. Fro (17) and (18), we see that axiizing the volue of the prior paraeter space is accoplished by axiizing the length of the dependent variable y and iizing the deterant of the inner product of the atrix X. The latter is accoplished by iizing the lengths of the independent variables x 1,...,x and axiizing the absolute values of their correlations, cosφ 12,...,cosφ 1,. 1/2 (18) DECONSTRUCTING THE EVIDENCE VALUES Having derived a suitable parsionious proper unifor prior for the ultivariate case, we now look ore closely at the evidence values which result fro using this prior. Suppose we wish the copute the evidence of a specific odel M M : y = Xβ + e, (19) where β = (β 1,...,β ), e = (e 1,...,e N ) and e i N (0,σ) for i = 1,...,N and soe value of σ. Then the corresponding likelihood is [ ] 1 p(y X,β,σ,M) = exp (y Xβ)T (y Xβ) (2πσ 2 N/2 ) 2σ 2. (20) For the unknown beta coefficients β of odel (19) we will use the derived proper unifor prior (17), and for the unknown error σ we use a proper Jeffreys prior with noralizing constant A. The proper prior for the unknown paraeters then becoes p(β,σ I) = X T X 1/2 A σ, (21)

5 where (β 1,...,β ) ellipsoid and a σ b. If we cobine the prior (21) with the likelihood (20), we get the following ultivariate distribution X T X 1/2 [ ] A 1 p(y,β,σ X,M,I) = y exp (y Xβ)T (y Xβ) ax σ (2πσ 2 N/2 ) 2σ 2. (22) Upon integrating out the + 1 unknown paraeters β and σ we are left with the following evidence value, [1]: X T X 1/2 A (2π)( N)/2 Γ[(N )/2] X T X 1/2 2 2 (N )/2 p(y X,M,I) = y ŷ N (23) where ŷ X ( X T X ) 1 X T y is the axiu likelihood estiate of y. We ay rearrange these ters to get p(y X,M,I) = y ŷ X T X 1/2 In the evidence (24), the factor X T X 1/2 shrinkage = Γ[(N )/2] y ŷ N y ŷ X T X 1/2 X T X 1/2 A 2π N/2. (24) is the shrinkage of the posterior accessible paraeter space of β relative to the prior accessible space. The greater this shrinkage, the saller the evidence. Shrinkage ay occur if ŷ y. If this is the case, then the factor that rewards goodness of fit goodness of fit = (25) 1 y ŷ N (26) will copensate for the shrinkage penalty. So there is a trade-off between the shrinkage (25) and the goodness of fit (26). Furtherore, there is a factor penalty = Γ[(N )/2] (27) that penalizes( regression odels with a greater nuber of regression coefficients. The last factor A/ 2π N/2) is shared by the evidence values of all the copeting regression odels and eventually cancels out if one coputes the corresponding posterior probabilities. INVARIANCE FOR CHANGE OF SCALE OF THE EVIDENCE The evidence value (24) is invariant under a change in scale of the dependent variables, that is, for the transforation X cx, where c is soe constant. The noralized

6 evidence value, e.g. posterior probability of the odel, is also invariant for a change in scale in the dependent variable, that is, under the transforation ỹ cy. If one uses instead of prior (21) the ore generic prior then the corresponding evidence ter becoes p(β,σ I) = AB σ, (28) B Γ[(N )/2] A p(y X,M,I) = X T X 1/2 y ŷ N 2π (N )/2 (29) and there sees to be a lack of invariance of the evidence values of regression odels under changes of scale. But if we replace the generic ter B with the proper unifor prior (17), we see that invariance was never lacking. DISCUSSION Using inforational consistency requireents, Jaynes [3] derived the for of axial non-inforative priors for location paraeters, that is, regression coefficients, to be unifor. However, this does not tell us what the liits of this this unifor distribution should be, that is, what particular unifor distribution to use. If we are faced with a paraeter estiation proble these liits of the unifor prior are irrelevant, since we ay scale the product of the iproper unifor prior and the likelihood to one, thus obtaining a properly noralized posterior. However, if we are faced with a proble of odel selection then the value of the unifor prior is an integral part of the evidence, which is used to rank the various copeting odels. We have given here soe guidelines for choosing a parsionious proper unifor prior. To construct such a parsionious prior one needs to assign a axial length to the dependent variable y, ial lengths to the independent variables x 1,...,x and axial absolute values to the correlations of the independent variables, cosφ 12,...,cosφ 1,. REFERENCES 1. Arnold Zellner, An Introduction to Bayesian Inference in Econoetrics, J. Wiley & Sons, Inc., New York (1971); Wiley Classics Library Edition (1996). 2. Larry G. Bretthorst, Bayesian Spectru Analysis and Paraeter Estiation, Springer-Verlag, New- York, Edwin T. Jaynes, Prior probabilities, IEEE Trans. Systes Sci. Cybernetics SSC-4 (3), (1968).