Dipartimento di Matematica

Transcription

1 Dipartimento di Matematica S. RUFFA ON THE APPLICABILITY OF ALGEBRAIC STATISTICS TO REGRESSION MODELS WITH ERRORS IN VARIABLES Rapporto interno N. 8, maggio 006 Politecnico di Torino Corso Duca degli Abruzzi, Torino-Italia

2 On the applicability of algebraic statistics to regression models with errors in variables Suela Ruffa Politecnico di Torino Abstract In this paper we study the applicability of algebraic statistics methods to regression linear models with errors in variables (EIV models). In fact the advantage in the algebraic statistics methods is that we can find saturated linear models identifiable with designs which are not regular fractions of full factorial designs. First we introduce the main concepts of algebraic statistics applied to the design of experiments (see [1]) and next we formulate the model of interest: the Berkson s model (see []) in the case of a polynomial linear regression problem. By mean of simple designs we illustrate the main issues met in the generalization of algebraic statistic theory to EIV models. Mainly these are due to the fact that the design points are unknown realizations of random variables. For the application we have in mind, the assumption of normality is natural. The generalization from these simple designs to higher dimensional and non regular fractional designs is straightforward. The paper concludes with a discussion on the applicability of currently available software for algebraic statistics analysis to designs with errors in variables. Keywords: EIV models, design of experiments, random design. 1 Algebraic statistics in design of experiments Algebraic statistics is a recent trend in mathematical and applied statistics for which concepts of algebraic commutative algebra and algebraic geometry are applied to the study of statistical models. The result of algebraic statistics of most practical relevance is in design of experiments. For any design D, that is a finite set of distinct points in IR m, it returns an IR-vector space basis of the set of real functions defined over D in a time polynomial in m and in n (see [1]), where n is the number of design points. That is it returns a saturated polynomial model which is identifiable by D. Computationally harder but still possible is to obtain the family of all saturated linear polynomial identifiable models with a hierarchical structure (see below). The bases considered have two main properties: 1. they consist of n monomials, where n is the number of points in D;. they are hierarchical, that is if the monomial x α = x α 1 1 x αm m is included in the basis and x β divides x α, then also x β = x β 1 1 x βm m is included in the basis. 1

3 The necessary computations can be performed with softwares for computational commutative algebra, like CoCoA with the function IdealOfPoints (available on the site dima. unige.it). This function works only for no replication designs even if the theory can been extended also for design with replicated points (see [4]). The theory on which this computation is based uses concepts of algebraic commutative algebra and in particular the concept of Gröbner basis (see any computational commutative algebra book as [5]). The following is technical but is the key point of all algebraic statistics theory. Over a field k (like R or C) we consider the polynomial ring k[x 1,..., x m ], the ring of m-dimensional polynomials with coefficients in k. Given a design D we define the design ideal, denoted by I =Ideal(D) as the set of all polynomials in k[x 1,..., x m ] whose zeros are design points. We say that a subset G = {g 1,..., g s } of the ideal I k[x 1,..., x m ] is a Gröbner basis if 1. g i I for all i = 1,..., s and. LT (g 1 ),..., LT (g s ) = LT (I) where LT (g i ) is the leading term (with respect to the term ordering that we fixed) of the polynomial g i and LT (I) is the ideal generated by the leading terms of the ideal I. For the notion of term ordering we refer to e.g. [5]. Here we simply note that a term ordering is a way to order monomials x α = x α 1 1 x αm m with α i Z 0, i = 1,..., n by ordering their exponents. A term ordering then corresponds to an order on the m-dimensional vectors with non-negative entries Z m 0. Given a design D we can prove that a monomial basis for the polynomials whose zeros include the design points is Est τ (D) = {x α : x α is not divisible by any LT of the elements of Gröbner basis of Ideal(D)} = {x α : x α LT (g) : g Ideal(D) } where τ is the chosen term ordering and Est τ (D) has n elements. It has been shown [1] that, given the design D the model β α x α (1) x α Est τ (D) is unambiguously identifiable with the design D. As consequence with a n point design we can identify n distinct monomials. Moreover Est τ (D) satisfies the two properties of monomial basis on the previous page. We can define also the fan of the design D as the set of all possible Est τ (D) when the term ordering τ varies over all term orderings. This is a finite set (see [10]). With an example we will clarify the steps of the process leading from a design to a saturated identifiable hierarchical model. Consider the full factorial design. Its design ideal is Ideal(D) = x 1 1, x 1 = {f 1 (x 1 1) + f (x 1) : f 1, f R[x 1, x ]}. The monomial basis with respect to any term ordering is Est(D) = {1, x 1, x, x 1 x } and the linear hierarchical saturated model identified with the design is then f(x 1, x ) = β 0 + β 1 x 1 + β x + β 3 x 1 x. Algebraic statistics methods work well if the points are fixed, but can we say something if the points are realizations of random variables? In this paper we answer this question.

4 EIV Berkson s model A regression model describes the relationship between a response variable and a set of variables called predictors or factors. Usually the response is modeled as a random variable while the predictors are considered as fixed constants. Very often the errors associated to factor settings are ignored. However there are situations where we cannot ignore that the levels of the factors are set with some error. In this case to describe the relationship between the response variables and the predictors we can use an errors in variables model (EIV model). In the case of simple linear regression (only one predictor, m = 1) the EIV model is: Y j = β 1 + β X j + ε j X j = t j + e j () where j runs over the n observations, e j and ε j are the errors associated to the variables X and Y respectively in the j-th observation and t j = E(X j ). If we assume that we can observe the value x j from the random variable X j whose mean is unknown, we can formulate a structural or a functional relationship. The models that arise from these two relationships are the most used in the literature (see e.g. [3, 8]). But in the application to design of experiments it seems to us more reasonable to assume that the t j with j = 1,..., n are known and that the x j are unknown. In fact they are the values in which we would like to observe our response. This situation is known as Berkson s model (see [, 7]). We cannot observe the true values that generate the data but in fact we suppose that the means of the random variables are known. If the planned value of the ith predictor in the jth observation is t j a value X j = t j + e j rather than the value t j generates the data point. The general form of a polynomial linear regression EIV Berkson s model is given in Equation (3). Y = β 1 f 1 (X) + + β p f p (X) + ε (3) X = t + e where X = (X 1,..., X m ) T is the column vector whose entries are predictors, f 1,..., f p are known polynomial functions of the predictors X 1,..., X m, ε is the error associated to the response variable Y, e = (e 1,..., e m ) T is the error associated to the factors (X 1,..., X m ) T, t = E(X) and β i are the regression coefficients. In the Equation (3) we omitted the subindex indicating the observation j to simplify the notations. In many cases (e.g. [7]) the assumptions on the errors are the following: e i N(0, σ i ); e i are stochastically independent, e.g. E(e i e j ) = 0 i j,..., E( m i=1 e i) = 0. Thus X i is a random variable with distribution Norm(t i, σ i ) for each i = 1,..., m and the vector X has a normal multivariate distribution Norm((t 1,..., t m ) T, Σ) with diagonal variance covariance matrix Σ =diag(σ 1,..., σ m). The previous independence assumptions on the e j mean that, from a practical viewpoint, the settings on one factor are independent of the rest of the factors. In the following sections we also assume that the random vector e i is independent of the random vector e j for any observation i j, i, j (1,..., n). We next use EIV Berkson s model to define a random design. 3

5 3 Random designs Under Berkson s model the factor settings are unknown realizations of random variables. Thus we define a random design as the set of these random variables. In the application of algebraic statistics to random designs a problem is that the environment where we have to work changes. Now we have random polynomials, that is polynomials whose coefficients are random variables thus we cannot work in a field as R. To apply algebraic statistics methods it is necessary that the set of polynomial coefficients is a field. So we have to prove that the normal independent random variables are in a field. Theorem 1 shows that this holds and thus we can perform the usual computations with polynomials but now their coefficients include also random variables. In fact we prove that the following theorem holds true. Theorem 1. Given the probability space (Ω, F, P), the set of random variables A = {z : Ω R such that P(z(ω) = 0) = 0 or P(z(ω) = 0) = 1} (4) with usual sum and product operations between functions in R is a field. Proof. The proof follows from arguments in undergraduate probability and algebra (see [13]). We want only to note that without the condition P(z(ω) = 0) = 0 or P(z(ω) = 0) = 1 the set A = {z : Ω R measurable} is not a field. In fact consider the two random variables { 1 ω B z 1 (ω) = 0 ω B { 0 ω B z (ω) = 1 ω B where B is a measurable set and B is its complementary set. These random variables are in A but z 1 (ω) z (ω) = 0 A with z 1 (ω) 0 A and z (ω) 0 A, this means that we obtain zero divisors and that A is not a field. We want to work with no replication designs (even if the theory can been extended also for design with replicated points (see [4])) because efficient software is available only in this case. If the design points are realizations of normal random variables we cannot exclude the possibility of replications but we can study the conditions under which the probability of such an event is small enough. In the next two subsections we find these conditions for random designs in one dimension and for a full factorial random design. Next we shall see how that can be extended to general designs. 3.1 One dimensional case µ 1 µ µ3 Figure 1: Means of one dimensional random design Consider the application of Berkson s model to a design with one factor and three levels. We have three independent random variables X i Norm(µ i, σ i ) for i = 1,, 3 4

6 and, without lack of generality, we can consider that the means are ordered as in Figure 1. Given ε > 0, 0 < α < 1, three random variables X i with i = 1,, 3 we say that the random variables composing a random design are sufficiently distant if P( X j X i < ε) < α i j. (5) If α is small then with high probability the realizations of these random variables are different from each others. In this case we can consider that with high probability the design has no replication. Once the distributions of X i are known one between α and ε determines the other one. With some standard calculation we can obtain conditions involving means and variances of random variables that assure the Equation (5) is satisfied. Under the normality assumption for X i Norm(µ i, σ i ) for i = 1,, 3, the conditions are the following: σ j + σ i < ε + µ j µ i for all j = i + 1, i = 1, (6) where is the αth percentile of a standard normal distribution. Thus ε reflects the variance of the factor settings and α is used to make sure that the design has no replication with high probability. We can extend the arguments to one dimensional designs with n points. In fact for such a design it is sufficient to verify that Equation (6) is satisfied for each set of tree adjacent points. Thus for a design with n points we obtain (n ) conditions of the type (6). 3. The full factorial random design We consider the design in Figure where the dots are the means of the random variables (µ 31, µ 3 ) (µ 41, µ 4 ) x (µ 1, µ ) x 1 (µ 11, µ 1 ) Figure : Means of a full factorial random design X i Norm((µ i1, µ i ), Σ i ) where Σ i is diag(σ i1, σ i), for i = 1,, 3, 4. As in the case of one dimensional designs we have to verify that the points of the random design are sufficiently distant. In this case we have to find conditions for which: P( X i X j < ε) < α i j where for X = (X 1, X ) T then X = X 1 + X. The following proposition gives sufficient conditions for that to happen. 5

7 Proposition. Let X = (X 1, X ) T Norm((µ X1,µ X ), Σ X ) and Y = (Y 1, Y ) T Norm((µ Y 1,µ Y ), Σ Y ) be two independent random vectors whose variance covariance matrices Σ X and Σ Y are diagonal. For ε > 0 if then P( X 1 Y 1 < ε ) < α or P( X Y < P( X Y < ε) < α. ε ) < α (7) To prove this proposition we use the independence assumption within and between random vectors. The last hypothesis in the case of independent random variables follows from the diagonal variance covariance matrix (see [13]). Now we go back to our example and the application of Proposition 9 implies the following conditions involving means and variances: σ 11 + σ 1 < ε + µ 11 µ 1 σ 1 + σ 31 < ε + µ 1 µ 31 σ 3 + σ 4 < ε + µ 3 µ 4 (8) σ 11 + σ 41 < ε + µ 11 µ 1 σ 1 + σ 41 < ε + µ 1 µ 41 σ 31 + σ 11 < ε + µ 31 µ 11 where σ i1 and σ i are the first and the second element of the diagonal matrix Σ i, is the α-th percentile of a standard normal distribution as in (6) and ε is a small positive number (see Proposition ). 3.3 Generalization to higher dimensional designs In general if we work in an m-dimensional space we can generalize the Proposition as follow. Proposition 3. Let X= (X 1,..., X m ) T Norm((µ X1,..., µ Xm ), Σ X ) and Y = (Y 1,..., Y m ) T Norm((µ Y 1,..., µ Y m ), Σ Y ) be two random independent random variables whose variance covariance matrices Σ X and Σ Y are diagonal. For ε > 0 if then P( X i Y i < ε m ) < α for at least one i (1,..., m) (9) P( X Y < ε) < α. Thus in the case of higher dimension designs we can find conditions similar to those in Equation (8). For any l m full factorial design with m factors and l levels each, the total number of conditions is ( l m ) if we impose that each point has to be sufficiently 6

8 distant from any other point (i.e. one condition for each pair of points). This number can be further reduced by considering only conditions relative to adjacent points, as in the one dimensional case. For example for m = we need only (l 1)(l 1) conditions, reducing the order of magnitude of the number of conditions in Proposition 3 from l m to l. In general the number of necessary conditions it easily seem to be smaller than m 1 ( m 1)(l 1) m. The conditions that guarantee the distance between the points for an l m full factorial design also guarantee this for any of its fractions. In fact any design is a fraction of a full factorial design with level sets of minimal size. 4 Applicability of algebraic statistics methods Now we return to the simple case of a random design. If this design satisfies the conditions given in Equation (8) then Theorem 4 allows us to apply algebraic statistics methods to the random design. Theoretically we can construct the design ideal as the set of polynomials whose zeros are random variables X 11,..., X 4 and with the method illustrated in [1] we can obtain the Gröbner bases of the design. In fact when we work with a random design we do not know precisely which are the factor settings. All we can say is that the fan of the random design is maximal with probability 1 (see [1]). For the full factorial random design, the fan is composed of five bases with four monomials. These are obtained varying the monomial ordering over all possible monomial orderings and are the following ones: {1, x 1, x, x 1 x } {1, x 1, x 1, x 3 1} {1, x 1, x, x 1} (10) {1, x 1, x, x } {1, x, x, x 3 }. Now we can define when with high probability the random design can be associated to a non random design whose points are the means of random variables and thus reduce our random design to a determinist design and make direct use of e.g. IdealOfPoints. For 0 < α < 1 we define ε α of a random design as a value for which P( X i µ i < ε α ) = 1 α. (11) Recall t i = µ i for i = 1,..., n. The value ε α can be seen as the precision of the instrument used in the measurement process. So Equation (11) can be interpreted in the following two ways: if we fix the probability level 1 α with α small, we can compute the value ε α such that Equation (11) is satisfied. The computed value can be compared with the precision of the measurement instrument η: if η < ε α then we can say with high (1 α) probability that the monomial basis for the random design is {1, x 1, x, x 1 x }. This is the monomial basis of the full factorial non-random design whose points are the means of the random variables. If η > ε α then we can use some classical statistical methods to evaluate which of the identifiable models in (10) is best associated with the design; 7

9 if we fix the value of ε α = η, then we compute the probability P( X i µ i < η). If this probability is sufficiently high we can affirm that with probability 1 α the monomial basis for the random design is {1, x 1, x, x 1 x }, and the other four models in (10) are not identifiable, that is there is collinearity among their elements. If the probability is low, we have again to use some other classical statistical methods to find which of the linear model in (10) better suits our design. Similar arguments to those in Section 3.3 generalize this to high dimensional design. Acknowledgements The author thanks Prof. Eva Riccomagno for her help during this work and Hugo Maruri- Aguilar for his useful suggestions. References [1] J. Abbott, A. Bigatti, M. Kreuzer, and L. Robbiano. Computing ideals of points. J. Symbolic Comput., 30: , 000. [] J. Berkson. Are there two regression? J. Am. Statist. Ass., 45: , [3] G. Casella and R. L. Berger. Statistical Inference. Duxbury, 00. [4] A. M. Cohen, A. Di Bucchianico, and E. Riccomagno. Replications with Gröbner bases. In Proceedings of the 6th International Workshop on Model-Oriented Data Analysis (Puchberg/Schneeberg, 001), pages Physica, Heidelberg, 001. [5] D. Cox, J. Little, and D. O Shea. Using Algebraic Geometry. Springer Verlag, [6] A. N. Donev. Dealing with errors in variables in response surface exploration. Commun. Statist.-Theory methods, 9: , 000. [7] A. N. Donev. Design of experiment in presence of errors in factor levels. Jour. Statist. Plan. Inf., 16: , 004. [8] W. A. Fuller. Measurement Error Models. Wiley, New York, [9] M. Kreuzer and L. Robbiano. Computational Commutative Algebra 1. Springer Verlag, 000. [10] T. Mora and L. Robbiano. The Gröbner fan of an ideal. J. Symbolic Comput., 6(-3):183 08, Computational aspects of commutative algebra. [11] T. Nummi and J. Möttönen. Estimation and prediction for low degree polynomial models under measurement errors with an application to forest harversters. Appl. Statist., 53:Part , 004. [1] G. Pistone, E. Riccomagno, and H. P. Wynn. Algebraic Statistics. Chapmann & hall/crc, 001. [13] S. Ruffa. La Statistica algebrica dei modelli di regressione con errori nelle variabili. Politecnico di Torino,