Variable selection in multiple regression with random design

Transcription

1 Variable selection in multiple regression with random design arxiv: v1 [math.st] 26 Jun 2015 Alban Mbina Mbina 1, Guy Martial Nkiet 2, Assi Nguessan 3 1 URMI, Université des Sciences et Techniques de Masuku, Franceville, Gabon albanmbinambina@yahoo.fr 2 URMI, Université des Sciences et Techniques de Masuku, Franceville, Gabon gnkiet@hotmail.com 3 Université des Sciences et Technologies de Lille, Lille, France assi.nguessan@polytech-lille.fr Abstract We propose a method for variable selection in multiple regression with random predictors. This method is based on a criterion that permits to reduce the variable selection problem to a problem of estimating suitable permutation and dimensionality. Then, estimators for these parameters are proposed and the resulting method for selecting variables is shown to be consistent. A simulation study that permits to gain understanding of the performances of the proposed approach and to compare it with an existing method is given. Keywords Variable selection; Multiple linear regression; Random design; Selection criterion; Consistency 1. Introduction The selection of variables and models is an old and important problem in statistics, and several approaches have been proposed to deal with it for various methods of multivariate statistical analysis. For linear regression, many model selection criteria have been proposed in the literature. Surveys on earlier work in this field may be found in [6, 13, 14], whereas some monographs on this topic are avalaible e.g.,[7, 8]. Most of the methods that have been proposed for variable selection in linear regression deal with the case where the covariates are assumed to be nonrandom; for this case, many selection criteria have been introduced in the literature. These include the FPE criterion [13, 14, 12, 15], cross-validation [16, 11], AIC and C p type criteria e.g., [4], the prediction error criterion [5], and so on. There is just a few works dealing with the case where the covariates are random, although its importance that have been recognized in [2] who argued that that this case typically gives higher prediction errors than the fixed design counterparts and hence more is gained by variable selection. Linear regression with random design were considered in [17, 9] for variable selection, but these works only deal with univariate models, that is models for which the response is a real random variable. A recent work that considered multiple regression model is [1] in which a method based on 1

2 applying an adaptative LASSO type penalty and a novel BIC-type selection criterion have been proposed in order to select both predictors and responses. In this paper we extend the approach introduced in [9] to the case of multiple regression. In Section 2, the multiple regression model that is used is presented as well as a statement of the variable selection problem. Then, the used criterion is introduced and we give a characterization result that permits to reduce the variable selection problem to an estimation problem for two parameters. In Section 3, we propose our method for selecting variables by estimating the two previous parameters, and we prove its consistency. Section 4 is devoted to a simulation study which permits to evaluate finite sample performances of the proposal and to compare it with the method given in [1]. Proofs of lemmas and theorems are given in Section Model and criterion for selection In this section, the multiple regresion model in which we are interested is introduced and a statement of the corresponding variable selection problem is given. It is described as a problem of estimation of a suitable set. A criterion permitting to characterize this set is propsed as well as an estimator of this criterion. Finally, we give a result that permits to obtain asymtotic properties of this estimator Model and statement of the problem We consider the multiple regression model given by: Y = BX +ε 1 where X and Y are random vectors valued into R p and R q respectively with p 2 and q 2, B is a q p matrix of real coefficients, and ε is a random vector valued into R q and which is independent of X. Writing and X = X 1. X p B =, Y = Y 1. Y q, ε = b 11 b 12 b 1p b 21 b 22 b 2p. b q1.. b q2 b qp it is easily seen that Model 1 is equivalent to having a set of p univariate regression models given by: and can also be writen as where Y i = ε 1. ε q b ij X j +ε i, i = 1,,q, 2 Y = X j b j +ε 3 b j = We are interested with the variable selection problem, that is identifying the X j s which are not relevant in the previous set of models, on the basis of an i.i.d. sample X k,y k of X,Y. We say that a 1 k n b 1j b 2j. b qj. 2

3 variable X j is not relevant if the corresponding coefficients vector b j is null. So, putting I = {1,,p} we consider the subset I 0 = {j I/ b j R q = 0} which is assumed to be non-empty, and we tackle the variable selection problem in Model 1 as a problem of estimating the set I 0 or, equivalently, the set I 1 = I I 0. In order to simplify the estimation of I 1 we will first characterize it by means of a criterion which introduced below Characterization of I 1 Without loss of generality, we assume that X and Y are centered; thus, that is also the case for ε. Furthermore, denoting by R k the usual Euclidean norm of R k, we assume that E X 4 R p < + and E Y 4 R q < +. Then, it is possible to define the covariance operators V 1 = EX X and V 12 = EY X, 4 where denotes the tensor product of vectors defined as follows: when E and F are euclidean spaces and u,v is a pair belonging to E F, the tensor product u v is the linear map from E to F such that where, E denotes the inner product in E. h E, u vh = u,h E v, Remark 1. In all of the paper, we essentially use covariance operators, but the translation into matrix terms is obvious and more details can be found in [3]. Particularly, when u and v are vectors in R p and R q respectively, the matrix related to the operator u v, relative to canonical bases, is vu T where u T is the transpose of u. So, if matricial expressions are prefered to operators, one can identify the operators given in 4 with the matrices V 1 = E XX T and V 12 = E XY T. In all of the paper, the operator V 1 is assumed to be invertible. For any subset K of I, let A K be the projector x = x i i I R p x K = x i i K R cardk and put Π K := A K A KV 1 A K 1 A K, where A denotes the adjoint operator of A. Then, we introduce the criterion ξ K = V 12 V 1 Π K V 12 5 where denotes the usual operator norm given by A = tra A. This criterion permits to give a more explicit expression of I 1 as stated in the following lemma. Lemma 1. We have I 1 K if, and only if, ξ K = 0. This lemma permits to characterize the fact that an interger i belongs to I 0. Indeed, since having i I 0 is equivalent to having I 1 I {i}, we deduce from this lemma that one has i I 0 if, and only if, ξ Ki = 0 where K i = I {i}. Then I 1 consists of the elements of I for which ξ Ki does not vanish. Now, let us consider the unique permutation σ of I satisfying: i ξ Kσ1 ξ Kσ2 ξ Kσp ; ii ξ Kσi = ξ Kσj and i < j imply σi < σj. Since I 0 is a not empty, there exists an integer s I, that we call the dimensionality, satisfying ξ Kσ1 ξ Kσ2 ξ Kσs > 0 = ξ Kσs+1 = = ξ Kσs+1. Therefore, we obviously have the following characterization of I 1 : Lemma 2. I 1 = {σk/1 k s}. 3

4 This result shows that estimation of I 1 reduces to that of the two parameters σ and s. So, our method for selecting variables will be based on estimating these parameters; in the next subsection, an estimator of the used criterion will be introduced. That will be the basis of the proposed procedure for variable selection Estimation of the criterion Recalling that we have an i.i.d. sample X k,y k of X,Y, we consider the sample means 1 k n and the empirical covariance operators and X n = n 1 X k, Y n = n 1 Y k, V 1 = n 1 V 12 = n 1 n X k X X k X, n Y k Y X k X. Then, for any K I, an estimator of ξ K is given by Π ξ K = V 12 V 1 Π K V 12 where K = A K A KV 1 A K 1 A K. The result given below permits to obtain asymptotic properties of this estimator. As usual, when E and F are Euclidean vector spaces, we denote by LE,F the vector space of operators from E to F. When E = F, we simply write LE instead of LE,E. Each element A of LR p+q can be writen as A = A 11 A 12 A 21 A 22 where A 11 LR p, A 12 LR q,r p, A 21 LR p,r q and A 22 LR q. Then we consider the projectors and we have: Proposition 1. We have P 1 : A LR p+q A 11 LR p and P 2 : A LR p+q A 12 LR q,r p, n ξ K = Ψ K Ĥ + nδ K, where δ K = V 12 V 1 Π K V 12, Ψ K n N is a sequence of random operators which converges almost surely, as n +, to the operator Ψ K of LLR p+q,lr q,r p given by: Ψ K A = P 2 A P 1 AΠ K V 12 +V 1 Π K P 1 AΠ K V 12 V 1 Π K P 2 A, and Ĥ n N is a sequence of random variables valued into LR p+q which converges in distribution to random variable H having a normal distributon with mean 0 and covariance operator given by: Γ = E Z Z V Z Z V, Z being the R p+q -valued random variable given by X Z = Y and is the tensor product between elements of LR p+q related to the inner product < A,B >= tra B. 4

5 3. Selection of variables Lemma 2 shows that estimation of I 1 reduces to that of σ and s. In this section, estimators for these two parameters are proposed and consistency properties are established for them Estimation of σ and s Let us consider a sequence f n n N of functions from I to R + such that there exists a real α ]0,1/2[ and a strictly decreasing function f : I R + satisfying: Then, recalling that K i = I {i}, we put i I, lim n + nα f n i = f i. φ i = ξ K i +f n i i I and we take as estimator of σ the random permutation σ of I such that φ σ i φ φ φ σ 1 σ 2 σ p and if = φ σ j with i < j, then σ i < σ j. Furthermore, we consider the random set = { σ j; 1 j i } and the random variable Ĵ i ψ i = ξ +g Ĵ n σ i i i I where g n n N is a sequence of functions from I to R + such that there exist a real β ]0,1[ and a strictly increasing function g : I R + satisfying: i I, lim n β g n i = gi. n + Then, we take as estimator of s the random variable { ŝ = min i I / ψ i = min j I ψ } j. The variable selection is achieved by taking the random set Î 1 = { σ i ; 1 i ŝ } as estimator of I Consistency The following theorem establishes consistency for the preceding estimators : Theorem 2. We have: i lim n + P σ = σ = 1; ii ŝ converges in probability to s, as n +. As a consequence of this theorem, we easily obtain: lim n + P of our method for selecting variables in the model Simulations Î 1 = I 1 = 1. This shows the consistency 5

6 Sample size Proposed method ASCCA e e e e e e e e e-8 Table 1: Average of prediction errors over 2000 replications In this section, we report results of a simulation study which was made in order to check the efficacy of the proposed approach and to compare it with an existing method: the ASCCA method introduced by An et al This latter method is based on re-casting the multivariate regression problem as a classical CCA problem for which a least quares type formulation is constructed, and applying an adaptative LASSO type penalty together with a BIC-type selection criterion see [1] for more details. Our simulated data is based on two independent data sets: training data and test data, each with sample size n = 50, 100, 500, 800, 1000, Thetrainingdataisusedforselectingvariablesbyusingboth ourmethod, with penalty terms f n i = n 1/4 i 1 and g n i = n 3/4 i, and the ASCCA method. The test data is used for computing prediction error given by e = 1 n n Y k Ŷk 2 R q, where Y k is an observed response and Ŷ k is the usual linear predictor of Y k computed by using the variables selected at the previous step, that is Ŷ k = X T X 1 Y k where X is a matrix with n rows and columns containing the observations of the X j s that have been selected in the previous step. Each data set was generated as follows: X k is generated from a multivariate normal distribution in R 7 with mean 0 and covariance covx k i,x k j = 0.5 i j for any 1 i,j 7, and the corresponding response Y k is generated according to 1 with B = and the related error term ε k having a multivariate normal distribution in R 5 with mean 0 and covariance matrix 0.5I 5, where I 5 denotes the 5-dimensional identity matrix. The outputs of the numerical experiment are the averages of the aforementioned prediction errors over 2000 independent replications. The results are reported in Table 1. Our method gives the better results for n 100 but was outperformed by the ASCCA method for n = Proofs 5.1. Proof of Lemma 1 6

7 Denoting by Ω, A, P the considered probability space, we consider the operators: x 1 y L 1 : x =. 1. R p x j X j L 2 Ω,A,P and L 2 : y =.. R q x p with adjoints are respectively given by: L 1 : Z L2 Ω,A,P EZX R p, and L 2 : Z L2 Ω,A,P EZY R q. y q q y i Y i L 2 Ω,A,P It is easy to verify that L 1L 1 = V 1 and L 1L 2 = V 12. Denoting by RA the range of the operator A, and from the fact that the orthogonal projector Π RA onto RA is given by Π RA = AA A 1 A, we clearly have ξ K = L 1L 2 L 1L 1 A KA K L 1L 1 A K 1 A K L 1L 2 = L 1L 2 L 1Π RL1A K L 2 = L 1Π RL1A K L 2, 6 where E denotes the orthogonal space of the vector space E. For any vector α = α 1,,α q T in R q, one has q q q q L 2 α = α i Y i = b ij X j +ε i = α i b ij X j + α i ε i. i=1 i=1 Since for any u = u 1,,u p T R p, we have < L 1 u,α i ε i >= α i u j < X j,α i ε i >= i=1 u j α i EX j ε i = it follows that α i ε i RL 1 and, from RL 1 RL 1 A K, we obtain Thus, where i=1 i=1 u j α i EX j Eε i = 0, L 1Π RL1A K α iε i = L 1α i ε i = Eα i ε i X = α i Eε i EX = 0. L 1 Π RL 1A K L 2α = L 1 Π RL 1A K q i=1 b i = α i b ij X j = b i1 b i2. b ip. q α i L 1 Π RL 1A L 1b K i, 7 If ξ K = 0, then considering, for i = 1,,q, the vector α = 0,,0,1,0,,0 of R q whose coordinates are null except the i-th one which equals 1, we deduce from 7 that L 1 Π RL 1A K L 1b i = 0. Since, for any operator A, kera A =kera, it follows that we have Π RL1A K L 1b i = 0, that is i=1 L 1 b i RL 1 A K. 8 Denoting by K the cardinality of K and putting K = {k 1,k 2,,k K }, we deduce from 8 that there exists a vector β = T β 1,,β K R K such that L 1 b i = L 1 A Kβ, that is K b ij X j = β l X kl l=1 7

8 and, equivalently, K b ikl β l X kl + l=1 l I K b ij X j = 0. 9 Since V 1 is invertible we have kerl 1 = kerl 1L 1 = kerv 1 = {0}. Then, X 1,,X p are linearly independent and, therefore, 9 implies that, for all j I K, bij = 0. This property holds for any i {1,,q}, then we deduce that I K I 0 and, equivalently, that I 1 K. Reciprocally, we first have L 1Π RL1A K L 1b i = L 1Π RL1A K b ij X j = L 1Π RL1A b K ij X j + b ij X j j K j I K K = L 1Π RL1A b K ikl X kl + b ij X j. l=1 j I K If I 1 K, then I K I 0 and, consequently, for all j I K, b ij = 0. Thus K L 1Π RL1A L 1b K i = L 1Π RL1A b K ikl X kl = L 1Π RL1A L 1A K Kb i = 0 because L 1 A K b i RL 1 A K. Then, from 7 and 6, we deduce that ξ K = Proof of Proposition 1 We have: l=1 n ξ K = n V 12 V 12 n V 1 V 1 Π K 12 V n Π 1 K Π K V 12 + nδ K, V V 1 Π K n V 12 V 12 and since it follows: Π K Π K = A K A K V 1 A K 1 A K V 1 A K 1 A K = A K A K V 1 A K 1 A K V 1 A K A K V 1 A K A K V 1 A K 1 A K = Π K V 1 V 1 Π K, n ξ K = n V 12 V 12 n V 1 V 1 Π K n + V 1 Π K V 1 V 1 Π K V 12 V 1 Π K n V 12 V 12 + nδ K. V Let us consider the R p+q -valued random vectors X Z =, Z Y k X k = Y k, k = 1,,n; 8

9 the covariance operator of Z is given by V = EZ Z and can be writen as V = V 1 V V 21 V 2 where V 2 = EY Y and V 21 = V12. Further, putting we can write where Z n = n 1 Z k, and V n = n 1 Z k Z Z k Z, V = V 1 V 21 V 12 V 2 V 2 = n 1 n Y k Y Y k Y and 11 and 12 that n ξ K = Ψ operator from LR p+q to LR p defined by 12 V 21 = K Ĥ + nδ K, where Ĥ = n V V. V 12 Then we deduce from 10, A LR p+q, Ψ K A = P 2A P 1 A Π K V 12 +V 1 Π K P 1AΠ A V 12 V 1 Π K P 2 A. and Ψ K is the random Considering the usual operators norm defined in LE,F by A = sup x E {0} Ax F / x E and recalling that, for two operators A and B, one has AB A B, we obtain Ψ K A Ψ Π KA = P 1 A K Π K V 12 P 1 AΠ K V 12 V 12 +V 1 Π K Π K P 1 AΠ K V 12 + V 1 Π K P 1 AΠ K V 12 V 12 P 1 A [ Π K Π K V 12 + Π K V 12 V 12 + V 1 Π K Π K Π K V 12 ] + V 1 Π K 2 V 12 V 12 [ Π K Π K V 12 + Π K V 12 V 12 + V 1 Π K Π K Π K V 12 + V 1 Π K 2 V 12 V 12 ] P 1, A, where T, := sup A LR p+q {0} TA / A. Hence Ψ K Ψ K, [ 1+ V 1 Π K ] V 12 Π K Π K P 1, 13 From the strong law of large numbers it is easily seen that n + to V 1 resp. V 12. Therefore, we deduce that distribution of Ĥ. We have Ĥ = Ĥ 1 Ĥ 2 where Ĥ 1 = 1 n n Z k Z k V n +[1+ V 1 Π K ] Π K V 12 V 12 P 1,. V 1 resp. V 12 converges almost surely, as Π K converges almost surely, as n + to Π K, and from 13 Ψ K converges almost surely, as n + to Ψ K. It remains to obtain the asymptotic and Ĥ 2 = 1 nz nz. n 9

10 The central limit theorem ensures that Ĥ 1 resp. nz converges in distribution, as n +, to a random variable H resp. U having a centered normal distribution with covariance operator Γ resp. Γ given by Γ = E Z Z V Z Z V resp. Γ = EZ Z. Hence, Ĥ 2 converges in probability, as n +, to 0 and Slustky theorem permits to conclude that Ĥ converges in distribution, as n +, to H Proof of Theorem 2 We just need to prove the lemma which is given below. Then the proof of Theorem 1 is similar than that of Theorem 3.1 in [10]. Let r N and m 1,,m r N r such that r l=1 m l = p and ξ Kσ1 = = ξ Kσm1 > ξ K σm1 +1 = = ξ K σm1 +m 2 > > ξ K σm1 +m 2 + +m r 1 +1 = = ξ K σm1 +m 2 + +mr. Then, putting E = {l N /1 l r, m l 2} and F l := m 0 = 0, we have: ξ Lemma 3. If E, then for all l E and all i F l, the sequence n α probability to 0 as n +. Proof. Let us put γ l = ξ Kσi = ξ Kσi+1 ; if γ l = 0, then n α ξ K σi ξ K σi+1 = n α 1 2 Ψ K Ĥ σi Ψ { l 1 k=0 m l } k +1,, k=0 m k 1 with K σi K σi+1 Ĥ Ψ n α 1 2 K σi Ψ Ĥ K σi+1 n α 1 2 Ψ K σi Ψ K σi+1 Ĥ, ξ K σi+1 converges in Since Ψ K σi and Ψ K σi+1 converge almost surely, as n +, to Ψ Kσi and Ψ Kσi+1 respectively, and since Ĥ converges ξ in distribution, as n +, to H, it follows from the preceding inequality and from α < 1/2 that n α K σi ξ K σi+1 converges in probability to 0 as n +. If γ l 0, we have n α ξ K σi ξ K σi+1 = n α 1 2 = + = + Ψ Ψ K Ĥ σi + nδ Kσi Ψ K Ĥ σi+1 + nδ Kσi+1 n α 1 2 Ψ K Ĥ σi 2 Ψ K Ĥ σi+1 2 K Ĥ σi + nδ Kσi + Ψ K Ĥ σi+1 + nδ Kσi+1 2n α δ Kσi, Ψ K Ĥ σi δ Kσi+1, Ψ K Ĥ σi+1 Ψ K Ĥ σi + nδ Kσi + Ψ K Ĥ σi+1 + nδ Kσi+1 n α 1 Ψ K Ĥ σi 2 Ψ K Ĥ σi+1 2 n 1 2 Ψ K Ĥ σi +δ Kσi + n 1 2 Ψ K Ĥ σi+1 +δ Kσi+1 2n α 1 2 δ Kσi, Ψ K Ĥ σi δ Kσi+1, Ψ K Ĥ σi+1 n 1 2 Ψ K Ĥ σi +δ Kσj + n 1 2 Ψ K Ĥ σi+1 +δ Kσi+1, 10

11 where <, > is the inner prodcut defined by < A,B >= tra B. First, n α 1 Ψ σj Ĥ 2 Ψ σj+1 Ĥ 2 n α 1 Ψ K Ĥ σi 2 + Ψ K Ĥ σi+1 2 n α 1 Ψ K σi 2 + Ψ K σi+1 2 Ĥ 2 14 and, further, 2n α 1 2 δ Kσi, 2n α 1 2 δ Kσi, 2n α 1 2 Ψ K Ĥ σi δ Kσi+1, + Ψ K σi+1 Ĥ Ψ K Ĥ σi δ Kσi+1, Ψ K Ĥ σi+1 δ Kσi Ψ K Ĥ σi + δ Kσi+1 Ψ K Ĥ σi+1 2n α 1 2 γl Ψ + Ψ K σi+1 Ĥ. 15 Equations ξ 14 and 15, and the above recalled convergence properties permit to conclude that the sequence n α K σi ξ K σi+1 converges in probability to 0, as n +. References [1] B. An, J. Guo, H. Wang. Multivariate regression shrinkage and selection by canonical correlation analysis. Comput. Statist. Data Anal., 62:93 107, [2] L. Breiman, P. Spector. Submodel selection and evaluation in regression. The X-random case. Internat. Statist. Rev., 60: , [3] J. Dauxois, Y. Romain, S. Viguier. Tensor products and statistics. Linear Algebra Appl., 210:59 88, [4] Y. Fujikoshi, K. Sato. Modified AIC and C p in multivariate linear regression. Biometrika, 84: , [5] Y. Fujikoshi, T. Kan, S. Takahashi, T. Sakurai. Prediction error criterion for selecting variables in a linear regression model. Ann. Inst. Stat. Math., 63: , [6] R. R. Hocking. The analysis and selection in linear regression. Biometrics, 32:1 49, [7] H. Linhart, W. Zucchini. Model selection. Wiley, New York, [8] A. J. Miller. Subset selection in regression. Chapman and Hall, London, [9] G. M. Nkiet. Sélection des variables dans un modèle structurel de régression linéaire. C. R. Acad. sci. paris I, 333: , [10] G. M. Nkiet. Direct variable selection for discrimination among several groups. J. Multivariate Anal., 105: , [11] J. Shao. Linear model selection by cross-validation. J. Amer. Statist. Assoc., 88: , [12] R. Shibata. Approximate efficiency of a selection procedure for the number of regression variables. Biometrika, 71:43 49,

12 [13] M. L. Thomson. Selection of variables in multiple regression. Part I. A review and evaluation. Internat. Statist. Rev., 46:1 19, [14] M. L. Thomson. Selection of variables in multiple regression. Part II. Chosen procedures, computations and examples. Internat. Statist. Rev., 46: , [15] P. Zhang. On the distributional properties of model selection criteria. J. Amer. Statist. Assoc., 87: , [16] P. Zhang. Model selection via multifold cross validation. Ann. Statist., 21: , [17] X. Zheng, W. Y. Loh. A consistent variable selection criterion for linear models with high-dimensional covariates. Statistica Sinica, 7: ,