EFFICIENCY of the PRINCIPAL COMPONENT LIU- TYPE ESTIMATOR in LOGISTIC REGRESSION

Transcription

1 EFFICIENCY of the PRINCIPAL COMPONEN LIU- YPE ESIMAOR in LOGISIC REGRESSION Authors: Jibo Wu School of Mathematics and Finance, Chongqing University of Arts and Sciences, Chongqing, China, Yasin Asar Department of Mathematics Computer Sciences, Necmettin Erbakan University, Konya, 42090, urkey, Abstract: In this paper we propose a principal component Liu-type logistic estimator by combining the principal component logistic regression estimator and Liu-type logistic estimator to overcome the multicollinearity problem. he superiority of the new estimator over some related estimators are studied under the asymptotic mean squared error matrix. A Monte Carlo simulation experiment is designed to compare the performances of the estimators using mean squared error criterion. Finally, a conclusion section is presented. Key-Words: Liu-type estimator; Logistic regression; Mean squared error matrix; Maximum likelihood estimator; Multicollinearity AMS Subject Classification: 62J07; 62J12.

2 2 Jibo Wu and Yasin Asar

3 Principal Component Liu-type Logistic Estimator 3 1. INRODUCION Consider the following binary logistic regression model 1.1 π i = exp x i β 1 + exp x i = 1,..., n iβ, where x i = 1 x i1 x iq denotes the ith row of X which is an n p p = q + 1 data matrix with q known covariate vectors, y i shows the response variable which takes on the value either 0 or 1 with y i Bernoulliπ i, y i s are supposed to be independent of one another and β = β 0 β 1 β q stands for a p 1 vector of parameters. Usually the maximum likelihood ML method is used to estimate β. he corresponding log-likelihood equation of model 1.1 is given by 1.2 L = n y i log π i + 1 y i log 1 π i i=1 where π i is the i th element of the vector π, i = 1, 2,..., n. ML estimator can be obtained by maximizing the log-likelihood equation given in 1.2. Since Equation 1.2 is non-linear in β, one should use an iterative algorithm called iteratively re-weighted least squares algorithm IRLS as follows Saleh and Kibria, 2013: 1.3 ˆβt+1 = ˆβ t + X V t X 1 X V t y ˆπ t where π t is the estimated values of π using ˆβ t and V t = diag ˆπ i t 1 ˆπ t i such that ˆπ i t is the ith element of ˆπ t. After some algebra, Equation 1.3 can be written as follows: 1.4 ˆβML = X V X 1 X V z where z = z 1 z n with η i = x i β and z i = η i + y i π i η i / π i. In linear regression analysis, multicollinearity has been regarded as a problem in the estimation. In dealing with this problem, many ways have been introduced to deal with this problem. One approach is to study the biased estimators such as ridge estimator Hoerl and Kennard, 1970, Liu estimator Liu, 1993, Liu-type estimator Huang et al., 2009, modified Liu-type estimator Alheety and Kibria, 2013 and improved ridge estimators Yüzbaşı et al., Alternatively, many authors such as Xu and Yang 2011 and Li and Yang 2011, have studied the estimation of linear models with additional restrictions. As in linear regression, estimation in logistic regression is also sensitive to multicollinearity. When there is multicollinearity, columns of the matrix X V X become close to be dependent. It implies that some of the eigenvalues of X V X

4 4 Jibo Wu and Yasin Asar become close to zero. hus, mean squared error value of MLE is inflated so that one cannot obtain stable estimations. hus many authors have studied how to reduce the multicollinearity, such as Lesaffre and Max 1993 discussed the multicollinearity in logistic regression, Schaefer et al proposed the ridge logistic RL estimator, Aguilera et al proposed the principal component logistic regression PCLR estimator, Månsson et al introduced the Liu logistic LL estimator, by combining the principal component logistic regression estimator and ridge logistic estimator to deal with multicollinearity. Moreover, Inan and Erdoğan 2013 proposed Liu-type logistic estimator LL and Asar 2017 studied some properties of LL. In this study, by combining the principal component logistic regression estimator and the Liu-type logistic estimator, the principal component Liu-type logistic estimator is introduced as an alternative to the PCLR, ML and LL to deal with the multicollinearity. he rest of the paper is organized as follows. In Section 2, the new estimator is proposed. Some properties of the new estimator are presented in Section 3. A Monte Carlo simulation is given in Section 4 and some concluding remarks are given in Section HE NEW ESIMAOR he logistic regression model is expressed by Aguilera et al in matrix form in terms of the logit transformation as L = Xβ = X β = Zα where = [t 1,..., t p ] shows an orthogonal matrix with Z V Z = X V X = Λ and Λ = diag λ 1,..., λ p, λ 1... λ p is the ordered [ eigenvalues ] of X V X. hen Λr O and Λ may be written as = r p r and where Z O Λ rv Z r = p r rx V X r = Λ r and Z p rv Z p r = p rx V X p r = Λ p r. he Z matrix and the α vector can be partitioned as Z = Z r Z p r and α = α r α p r. he handling of multicollinearity by means of PCLR corresponds to the transition from the model L = Xβ = X r rβ + X p r p rβ = Z r α r + Z p r α p r to the reduced model L = Z r α r. hen by Equation 1.1 and PCLR method we get the PCLR estimator. Inan and Erdoğan 2013 proposed Liu-type logistic estimator LL as 2.1 ˆβ k, d = X V X + ki 1 X V z d ˆβ ML where < d < and k > 0 are biasing parameters. he principal component logistic regression estimator Aguilera et al., 2006 is defined as 2.2 ˆβr = r r X V X r 1 r X V z

5 Principal Component Liu-type Logistic Estimator 5 We can write 2.2 as follows: 2.3 ˆβr = r r X V X r 1 r X V z = r r ˆβ ML hen we can introduce a new estimator by replacing ˆβ k, d with ˆβ ML in 2.3, and we get ˆβ r k, d = r r ˆβ k, d 2.4 = r r X V X r + ki r 1 r X V X r di r r X V X r 1 r X V z where < d < and k > 0 are biasing parameters. We call this estimator as the principal component Liu-type logistic regression PCLL estimator. Remark 2.1. It is obvious that ˆβ r k, d = r r X V X r + ki r 1 r X V X r di r r ˆβr. hus we can see the PCLL estimator as a linear combination of the PCLR estimator. Remark 2.2. It is easy to obtain the followings: a b c ˆβr 0, 0 = ˆβ r = r rx V X r 1 rx V z, PCLR estimator ˆβp 0, 0 = ˆβ ML = X V X 1 X V z, ML estimator ˆβp k, d = ˆβ k, d = X V X + ki 1 X V z d ˆβ ML, LL estimator. hus, the new estimator in 2.4 includes the PCLR, ML and LL estimators as its special cases. In the next section, we will study the properties of the new estimator. 3. HE PROPERIES OF NEW ESIMAOR For the sake of convenience, we present some lemmas which are needed in the following discussions. Lemma 3.1. Farebrother, 1976; Rao and ountenburg, 1999 Suppose that M is a positive definite matrix, namely M > 0, α is some vector, then M αα 0 if and only if α M 1 α 1.

6 6 Jibo Wu and Yasin Asar Lemma 3.2. Baksalary and renkler,1991 Let C n p be the set of complex matrices and H n n be the Hermitian matrices. Further, given L C n p, L, R L and κ L denote the conjugate transpose, the range and the set of all generalized inverses, respectively of L. Let A H n n, a 1 C n 1 and a 2 C n 1 be linearly independent, f ij = a i A a j, i, j = 1, 2 and A κ L, a 1 / R A. Let [ s = a 1 I AA I AA ] / [ a 2 a 1 I AA I AA ] a 1 hen A + a 1 a 1 a 2a 2 holds: 0 if and only if one of the following sets of conditions a A 0, a i R A, i = 1, 2, f f 22 1 f 12 2 b A 0, a 1 / R A, a 2 R A : a 1, a 2 sa 1 A a 2 sa 1 1 s 2 c A = U U λvv, a i R A, i = 1, 2, v a 1 0, f , f , f f 22 1 f 12 2, where U : v shows a sub-unitary matrix, λ is a positive scalar and is a positive definite diagonal matrix. Further, the condition a, b and c denote all independent of the choice of A, A stands for the generalized inverse of A. o compare the estimators, we use the mean squared error matrix MSEM criterion which is defined for an estimator ˇβ as follows: MSEM ˇβ = Cov ˇβ + Bias ˇβBias ˇβ where Cov ˇβ is the covariance matrix of ˇβ, and Bias ˇβ is the bias vector of ˇβ. Moreover, scalar mean squared error SMSEM of an estimator ˇβ is also given as SMSE ˇβ = tr { MSEM ˇβ } Comparison of the new estimator PCLL to the ML estimator From 2.4, we can compute the asymptotic variance of the new estimator as follows: 3.1 Cov ˆβr k, d = r S r k 1 Λ 1 r S r dλ r S r dλ 1 r S r k 1 r where S r k = Λ r + ki r, S r d = Λ r di r. Using 2.4, we get: 3.2 E ˆβr k, d = r S r k 1 Λ 1 r S r dλ r rβ

7 Principal Component Liu-type Logistic Estimator 7 By 3.3 r S r k 1 Λ r r I p = p r p r + k r S r k 1 r, then we get the asymptotic bias of the new estimator as follows: Bias ˆβr k, d = p r p r d + k r S r k 1 r β. Now, we can get the asymptotic mean squared error matrix of the new estimator as follows MSEM ˆβr k, d = r S r k 1 Λ 1 r S r dλ r S r dλ 1 r S r k 1 r + p r p r d + k r S r k 1 r β 3.4 β p r p r d + k r S r k 1 r. heorem 3.1. Assume that d < k and d+k > 0 then the new estimator is superior to the ML estimator under the asymptotic mean squared error matrix criterion if and only if β r k + d 2 [ 2k + di r + k 2 d 2 Λ 1 ] 1 r r β + β p r Λ p r p rβ 1. Proof: he asymptotic mean squared error matrix of MLE is given by 3.5 MSEM ˆβ = X V X 1. Λr O By Λ = O Λ p r and = r, p r, we may obtain X V X 1 = Λ 1 = r Λ 1 r Let us consider the difference 1 = MSEM ˆβ r + p r Λ 1 p r p r. MSEM ˆβr k, d 1 = r S r k 1 [ 2k + di r + k 2 d 2 Λ 1 ] r Sr k 1 r [ + p r Λp r p rββ ] p r p r k + d 2 r S r k 1 such that 3.6 rββ S r k 1 r + k + d r S r k 1 rββ p r p r +k + d p r p rββ r S r k 1 r. Let and 3.7 Λ 1 = S = Srk k+d 0 0 Λ p r 2k+dIr+k 2 d 2 Λ 1 r 0 k+d 2. 0 Λ p r

8 8 Jibo Wu and Yasin Asar Now we can write 3.6 as = S 1 [ Λ 1 ββ ] S 1. hus 1 is a nonnegative definite matrix if and only if Λ 1 ββ is a nonnegative definite matrix. Using Lemma 3.1, Λ 1 ββ is a nonnegative definite matrix if and only if β Λ β 1. Invoking the notation of Λ in 3.7, we can prove heorem Comparison of the new estimator PCLL to the PCLR estimator heorem 3.2. Suppose that d < k and d+k > 0 then the new estimator is better than the PCLR estimator under the asymptotic mean squared error matrix criterion if and only if rβ = 0. Proof: Suppose that k = d in Equation 3.4, then we get 3.9 MSEM ˆβr = r Λ 1 r r + r r I p ββ r r I p. Now let us consider the difference 2 = MSEM that ˆβr MSEM ˆβr k, d such = r S r k 1 [ 2k + di r + k 2 d 2 Λ 1 ] r Sr k 1 r + r r I p ββ r r I p + p r p r d + k r S r k 1 r β β p r p r d + k r S r k 1 r. o apply Lemma 3.2, let A = r B r, where B = S r k 1 [ 2k + di r + k 2 d 2 Λ 1 ] r Sr k 1 and a 1 = r r I p β, a 2 = p r p r d + k r S r k 1 r β. When d < k and d + k > 0, B is a positive definite matrix. hen we get the Moore-Penrose inverses of A which is A + = r B 1 r, and AA + = r r. hus a 1 R A if and only if a 1 = 0. Since a 1 0, we cannot use part a and c of Lemma 3.2, we can only apply part b of Lemma 3.2. Using the definition of s, we may obtain that s = 1. On the other hand, a 2 a 1 = Aη, where η = d + k r S r k [ 2k + di r + k 2 d 2 Λ 1 ] 1 r rβ.

9 Principal Component Liu-type Logistic Estimator 9 hus, we can easily obtain a 2 R A : a 1. hen Using Lemma 3.2, we can get that the new estimator is superior to the PCLR estimator under the asymptotic mean squared error matrix criterion if and only if a 2 a 1 A a 2 a 1 0 or η Aη 0. In fact, a 2 a 1 A a 2 a 1 0, so the new estimator is better than the PCLR estimator under the asymptotic mean squared error matrix criterion if and only if η Aη = 0, that is β r [ 2k + dir + k 2 d 2 Λ 1 r ] 1 rβ = 0 and β [ r 2k + dir + k 2 d 2 Λ 1 ] 1 r rβ = 0 if and only if rβ = 0. hus, the proof is finished Comparison of the new estimator PCLL to the Liu-type logistic estimator heorem 3.3. he new estimator is superior to the Liu-type logistic estimator under the asymptotic mean squared error matrix criterion if and only if p rβ = Proof: Putting r = p into 3.4, we get MSEM ˆβ k, d = Sk 1 SdΛ 1 SdSk 1 +k + d 2 Sk 1 ββ Sk 1 where Sk = Λ+kI p and S d = Λ di p. Now we study the following difference 3 = MSEM ˆβ k, d MSEM ˆβr k, d where 3 = Sk 1 SdΛ 1 SdSk 1 Suppose that C = p r D p r, where r S r k 1 Λ 1 r S r dλ r S r dλ 1 r S r k 1 r +k + d 2 Sk 1 ββ Sk p r p r d + k r S r k 1 r β β p r p r d + k r S r k 1 r. D = S p r k 1 S p r dλ 1 p r S p rds p r k 1 and a 3 = d + k Sk 1 β, a 2 = p r p r d + k r S r k 1 r β. We can apply part b of Lemma 3.2. he Moore-Penrose inverse of C is C + = p r D 1 p r, and CC + = p r p r. So a 3 / R C, a 2 R C : a 3, s = 1 and a 2 a 3 = Cη 1, where η 1 = p r S p r k 1 S p r dλ 1 p r p rβ.

10 10 Jibo Wu and Yasin Asar hen by Lemma 3.2, we obtain that the new estimator is superior to the Liu-type logistic estimator under the asymptotic mean squared error matrix criterion if and only if a 2 a 3 C a 2 a 3 0 or η 1 Cη 1 0. In fact, a 2 a 3 C a 2 a 3 0, so the new estimator is better than the Liu-type logistic estimator under the asymptotic mean squared error matrix criterion if and only if η 1 Cη 1 = 0, that is β p r Λ p r p rβ = A MONE CARLO SIMULAION SUDY In this simulation study, we study the logistic regression model. In this section, we present the details and the results of the Monte Carlo simulation which is conducted to evaluate the performances of the estimators MLE, PCLR, and LL estimators and PCLL. here are several papers studying the performance of different estimators in the binary logistic regression. herefore, we follow the idea of Lee and Silvapulle 1988, Månsson et al. 2012, Asar 2017 and Asar and Genç 2016 generating explanatory variables as follows 4.1 x i j = 1 ρ 2 1/2 zi j + ρz i q where i = 1, 2,..., n, j = 1, 2,..., q and z i j s are random numbers generated from standard normal distribution. Effective factors in designing the experiment are the number of explanatory variables q, the degree of the correlation among the independent variables ρ 2 and the sample size n. Four different values of the correlation ρ corresponding to 0.8, 0.9, 0.99 and are considered. Moreover, four different values of the number of explanatory variables consisting of q = 6, 8 and 12 are considered in the design of the experiment. he sample size varies as 50, 100, 200, 500 and Moreover, we choose the number of principal components using the method of percentage of the total variability which is defined as P V = r j=1 λ j p j=1 λ j 100. In the simulation, PV is chosen as 0.75 for q = 8 and 12 and 0.83 for q = 6 see Aguilera et al he coefficient vector is chosen due to Newhouse and Oman 1971 such that β β = 1 which is a commonly used restriction, for example see Kibria We generate the n observations of the dependent variable using the Bernoulli distribution Be π i where π i = ex i β such that x 1+e x i β i is the i th row of the data matrix X. he simulation is repeated for times. o compute the simulated MSEs of the estimators, the following equation is used respectively: 4.2 MSE β = c=1 βc β βc β

11 Principal Component Liu-type Logistic Estimator 11 able 1: Simulated MSE values of the estimators when q = 6 n ρ MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL where β c is MLE, PCLR, LL, and PCLL in the c th replication. he convergence tolerance is taken to be 10 6 in the IRLS algorithm. We choose the biasing parameter as follows: { } 1. LL: We refer to Asar 2017 and choose d L L = 1 2 min λj p λ j +1 where j=1 min is the minimum function and k AM = 1 p λ j d1+λ j ˆα 2 j p j=1. λ j ˆα 2 j 2. PCLL: We propose to use the modifications of the methods given above as follows: d P CL L = 1 { } r 2 min λj λ j + 1 and k P CL L = 1 r j=1 r λ j d P CL L 1 + λ j ˆα j 2 λ j ˆα j 2. j=1 According to ables 1-3, the following results are obtained: 1. MSE of the MLE is inflated when the degree of correlation is increased

12 12 Jibo Wu and Yasin Asar able 2: Simulated MSE values of the estimators when q = 8 n ρ MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL and the sample size is low. On the other hand, the performance of MLE becomes quite well when the sample size is high enough. 2. Similarly, if we consider PCLR and LL, the MSE values are also inflated for increasing values of the degree of correlation especially when n = MLE, PCLR and LL produce high MSE values when the sample size is low and the degree of correlation is high. However, PCLL seems to be robust to this situation in most of the cases. 4. Increasing the sample size makes a positive effect on the estimators in most of the situations. However, there is a degeneracy in this property. 5. When the degree of correlation is low, there is no estimator beating all others. 6. Overall, the new estimator PCLL has the lowest MSE value in most of the situations considered in the simulation.

13 Principal Component Liu-type Logistic Estimator 13 able 3: Simulated MSE values of the estimators when q = 12 n ρ MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL MLE LL PCLR PCLL CONCLUSION In this paper, we develop a new principal component Liu-type logistic estimator as a combination of the principal component logistic regression estimator and Liu-type logistic estimator to overcome the multicollinearity problem. We have proved some theorems showing the superiority of the new estimator over the other estimators by studying their asymptotic mean squared error matrix criterion. Finally, a Monte Carlo simulation study is presented in order to show the performance of the new estimator. According to the results, it seems that PCLL is a better alternative in multicollinear situations in the binary logistic regression model. REFERENCES [1] Aguilera, A. M., Escabias, M. and Valderrama, M. J Using principal components for estimating logistic regression with high-dimensional multicollinear data, Computational Statistics and Data Analysis, 50, 8, [2] Alheety. M.I. and Kibria, B. M.G Modified Liu-ype Estimator

14 14 Jibo Wu and Yasin Asar Based on r - k Class Estimator, Communications in Statistics-heory and Methods, 2, [3] Asar, Y Some new methods to solve multicollinearity in logistic regression, Communications in Statistics-Simulation and Computation, 46, 4, [4] Asar, Y. and Genç, A New Shrinkage Parameters for the Liu-ype Logistic Estimators, Communications in Statistics-Simulation and Computation, 45, 3, [5] Baksalary, J. K and renkler, G Nonnegative and positive definiteness of matrices modified by two matrices of rank one, Linear Algebra and its Application, 151, [6] Farebrother, R.W Further Results on the Mean Square Error of Ridge Regression, Journal of the Royal Statistical Society B, 38, [7] Hoerl, A.E. and Kennard, R. W Ridge regression: biased estimation for nonorthogonal problems, echnometrics, 12, [8] Huang, W.H., Qi, J.J. and Huang, N Liu-type estimator for linear model with linear restrictions, Journal of System Sciences and Mathematical Sciences, 29, [9] Inan, D. and Erdogan, B. E Liu-type logistic estimator, Communications in Statistics-Simulation and Computation, 42, 7, [10] Kibria, B. M. G Performance of some new ridge regression estimators, Communications in Statistics-Simulation and Computation, 32, 2, [11] Lee, A. H. and Silvapulle, M. J Ridge estimation in logistic regression, Communications in Statistics-Simulation and Computation, 17, 4, [12] Lesaffre, E. and Max, B.D Collinearity in Generalized Linear Regression, Communications in Statistics-heory and Methods, 22, 7, [13] Liu, K A new class of biased estimate in linear regression, Communications in Statistics-heory and Methods, 22, [14] Li, Y.L. and Yang, H wo kinds of restricted modified estimators in linear regression model, Journal of Applied Statistics, 38, [15] Månsson, K., Kibria, B. G. and Shukur, G On Liu estimators for the logit regression model,journal of Applied Statistics, 29, 4, [16] Newhouse, J. P. and Oman, S. D An evaluation of ridge estimators, Rand Corporation, P-716-PR, [17] Rao, C.R. and outenburg, H Linear models: Least squares and Alternatives, Springer-Verlag, New York.. [18] Saleh, A. M. E. and Kibria, B. M. G Improved ridge regression estimators for the logistic regression model, Computational Statistics, 28, 6, [19] Schaefer, R. L., Roi, L. D. and Wolfe, R. A A ridge logistic estimator, Communications in Statistics-heory and Methods, 13, 1, [20] Xu, J. W. and Yang, H On the restricted almost unbiased estimators in linear regression, Communications in Statistics-heory and Methods, 38, [21] Yüzbaşı, B., Ahmed, S. E. and Güngör, M Improved penalty strategies in linear regression models, REVSA Statistical Journal, 15, 2,