Alternative Biased Estimator Based on Least. Trimmed Squares for Handling Collinear. Leverage Data Points

Transcription

1 International Journal of Contemporary Mathematical Sciences Vol. 13, 018, no. 4, HIKARI Ltd, Alternative Biased Estimator Based on Least Trimmed Squares for Handling Collinear Leverage Data Points Moawad El-Fallah Abd El-Salam Department of Statistics & Mathematics and Insurance Faculty of Commerce, Zagazig University, Egypt Copyright 018 Moawad El-Fallah Abd El-Salam. This article is distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract The multicollinearity in multiple linear regression models and the existence of leverage data points are common problems. These problems exert undesirable effects on the least squares estimators. So, it would seem important to combine methods of estimation designed to deal with these problems simultaneously. In this paper, alternative biased robust regression estimator is defined by mixing the ridge estimation technique into the robust least trimmed squares estimation to obtain the Ridge Least Trimmed Squares (RLTS). The efficiency of the combined estimator(rlts) is compared with some existing regression estimators, which namely, the Ordinary Least Squares (); Ridge Regression (RR) and Ridge Least Absolute Deviation(RLAD). The numerical results of this study show that, the RLTS regression estimator is more efficient than other estimators, based on, Bias and mean squared error criteria for many combinations of leverage data points and degree of multicollinearity. Keywords: Leverage Data Points; Multicollinearity; Ridge regression; Ridge Least Absolute Deviation; Ridge Least Trimmed Squares estimation; Bias and Mean Squared Error criteria 1. Introduction Two important problems are considered in regression analysis; multicollinearity

2 178 Moawad El-Fallah Abd El-Salam and the existence of leverage data points. The ordinary least squares estimators () of coefficients are known to possess certain optimal properties when explanatory variables are not correlated among themselves, and the disturbances of the regression equation are independent, identically distributed normal random variables. The presence of correlation among the explanatory variables may result in imprecise information being available about the regression coefficients. In addition, the least squares estimator may produce extremely poor estimates in the presence of leverage data points. Thus, various remedial techniques have been suggested for these problems separately. One such remedial technique is ridge regression to deal with multicollinearity, and the robust estimation techniques are not as strongly affected by the presence of leverage data points. However, although, we usually think of these two problems separately, but in practical situations, these problems occur simultaneously. Several robust ridge regression estimators have been suggested for handling these two problems simultaneously, see (Lukman et al. (014) [10] and Nkiruka and Uchenna (016)) [13]. In this paper, we take the initiative to develop a more robust ridge estimators to remedy these two problems. We proposed combining the ridge regression with the highly efficient and high breakdown point estimator, namely the Ridge Least Trimmed Squares(RLTS)estimator. We call this modified method, the robust ridge regression based on Least Trimmed Squares estimation (RLTS). We expect that, the modified method would be less sensitive to the presence of leverage data points and multicollinearity. So, the aim of this paper is devoted to examine some estimators which are resistant to the combined problems of multicollinearity and leverage data points. Exactly, can the ridge estimators and some robust estimation techniques be combined to produce a robust ridge regression estimator?. The remainder of the paper is organized as follows. In section (), the ridge regression estimator will be reviewed. The robust regression estimation will be discussed in section (3). In section (4), we discuss the augmented ridge robust estimators as a way of combining biased and robust regression techniques, while, Section (5) introduces the proposed combined ridge robust estimator (RLTS). Section (6) presents the results of a Monte Carlo simulation study to investigate how such estimators perform well, and some concluding remarks are presented in section (7).. Ridge Regression Estimators Consider the following linear regression model: Y X, (1) where : y is an ( n 1) vector of observations on the dependent variable, X is an ( n p) matrix of observations on the explanatory variables, is a ( p 1) vector of regression coefficients to be estimated, and is an ( n 1) vector of disturbances. The least squares estimator of can be written as:

3 Alternative biased estimator based on least trimmed squares 179 ˆ ( X' X )- 1 X 'Y () This method gives unbiased and minimum variance among all unbiased linear estimators provided that the errors are independent and identically, normally distributed. However, in the presence of multicollinearity, the singularities present in ( X ' X ) matrix and this ill-conditioned X matrix can result in very poor estimates. The degree of multicollinearity is often indicated by conditioned number (CN) of the matrix X ( or X ' X ). CN is defined as the ratio of the largest singular values of X to the smallest, λ CN ( X ) max 1, (3) λ min where: are the eigenvalues of the matrix ( X ' X ). Belsley et al. (1980) [] have empirically shown that weak dependencies are linked to CN around 5 to 10, whereas moderate to strong relations are linked to CN of 30 to 100. Hoerl and Kennard (1970) [6] pointed out that adding a small constant to the diagonal of a matrix, will improve the conditioning of a matrix as this would dramatically reduced its CN. The ridge regression is defined as follows: ˆ RR ( X'X KI ) -1 X'Y, (4) where: I is the ( p p) identity matrix and K is the biasing constant. Various methods for determining K value been introduced in the literature such as described by Hoerl and Kennard (1970) [6] and Gibbons (1981) [5] as: PS Kˆ H βˆ, (5) where, ' Y - Xβˆ Y - Xβˆ S ( )'( ) (6) n - p When k 0, ˆ ˆ RR, when K 0, ˆRR is biased but more stable and precise than estimator and when K, ˆ RR 0. Hoerl and Kennard (1970) [6] have shown that, there always exist a value K 0 shuch that MSE ( ˆ RR ) is Less than MSE( ˆ ). 3. Robust Regression Estimators Robust regression estimators have been proven to be more reliable and

4 180 Moawad El-Fallah Abd El-Salam efficient than least squares estimator especially when the data are contaminated with leverage observations. Since the leverage data points greatly influence the estimated coefficients, standard errors and test statistics, the usual statistical procedure may be most inefficient as the precision of the estimator has been affected. Several different robust regression estimators exist. Two of the most commonly considered are: LAD-estimators and LTS-estimators. 3.1 The Least Absolute Deviation Estimator (LAD): The LAD estimator, ˆ, can be defined as the solution to the following LAD minimization problem: n mim Y - X' β (7) 1 i LAD i 1 Rather than minimizing the sum of squared residuals as in least squares estimation, the sum of the absolute values of the residuals is minimized. Thus, the effect of leverage data points on the LAD estimates will be less than that on estimates. 3. The Least Trimmed Squares Estimator (LTS): The Least Trimmed Squares (LTS) estimator was proposed by Rousseeuw (1984) [16] and has the property of being highly resistant to a relatively large proportion of leverage data points. Thus LTS has a high breakdown value. For the details of this technique and its properties, see Rousseeuw and Leroy (1987) [17], and also chapter 5 of Zaman (1996) [0]. The estimated ˆLTS can be defined by: h Min e (i), (8) i 1 where, e (1) e (),, e (n) are the ordered squared residuals, e i = (yi xi ), i=1,,n, and the value of h must be determined based on trimming the data values. For example, the 0% trimmed LTS estimator is defined to be the value of (4n/ 5) ˆ minimizing 4. Chen (00) [3] defined h in the range [(n/) e ( i) ( ) i 1 +1] h [(3n+k+1)/4], and recommended that, the breakdown value for the LTS estimator is( n-h/n ), when h=[(3n+k+1)/4]. Although, there exist other high breakdown estimators, Rousseeuw and Van Driessen, (1999) [18] pointed out that, the LTS has many advantages to recommend itself, and they developed a fast algorithm for its computation. The main advantages of the LTS method is as follows. Firstly, LTS is simple to understand and easy to motivate. Also, it is more efficient than the LMS (Least Median of Squares) introduced by Rousseeuw (1984) [16] with which it shares these advantages.

5 Alternative biased estimator based on least trimmed squares 181 Many estimators commonly regarded as robust in the econometrics literature have low breakdown values and cannot deal with any significant number of leverage data. For example, the bounded influence estimator of Krasker and Welsch (198) [9], and the least absolute deviation method, both suffer heavily from the presence of a small subgroup of these data influential; see Yohai (1987) [19]. 4. Robust Ridge Regression Estimators There are many studies that have been related using the robust ridge regression estimators in literature such as: Pfaffenberger and Dieman (1984) [14]; Moawad El-Fallah (013) [1] and Mal and Dul (014) [11]. In this section, we present some combinations of ridge and robust regression estimation discussed in sections () and (3) respectively. In this respect, the RLAD estimator, which is based on the LAD and ridge estimators denoted by, ˆRLAD, can be computed using the following: ˆ (X' X K* I) -1 X' Y RLAD, (9) where the value of K * is determined from data using : PS K* LAD βˆ βˆ LAD LAD (10) and, S (Y - Xβ ˆ )'(Y - Xβ ˆ ) LAD LAD n p, (11) where, ˆLAD is the LAD estimator defined as the solution to equation (7). It be noted that the value of K * is the estimator of K presented by equation (5) with two changes. First, the LAD estimator of is used rather than estimator. Second, the estimator of used in equation (11) is modified by the LAD coefficient estimates rather than the least squares estimates. These changes are aimed to reduce the effect of extreme points on the value chosen for the biasing parameter. 5. Alternative Combined Regression Estimator Instead of ˆ estimator which was aimed to reduce the effect of RLAD leverage data points on the value chosen for the biasing parameter k. Another alternative combined estimator between robust and ridge regression estimation is ˆRLTS. In this respect, it is hoped that, the problems of multicollinearity and leve-

6 18 Moawad El-Fallah Abd El-Salam rage data points can be solved simultaneously. The estimator can be ˆRLTS calculated by the following. ˆ * -1 (X'X K I) X'Y RLTS LTS, (1) where the value of K * is given by: K LTS PS LTS βˆ βˆ LTS LTS (13) and S LTS (Y - Xβ ˆ )'(Y - Xβ ˆ ) LTS LTS n p (14) 6. Simulation Study (6.1) Design of the Experiment: We carry out a Monte Carlo simulation study to compare the performance of some alternative combined estimators under concern. The simulation is designed to allow both multicollinearity and leverage data points simultaneously. Varying degrees of multicollinearity are allowed. Also, the non-normal distributions are used to generate leverage data points. The study contains four estimators which are: 1-The least squares estimator ( ˆ ). -The ridge regression estimator ( ˆ R ). 3-The ridge least absolute deviation estimator ( ˆ ). RLAD 4-The ridge least absolute deviation estimator ( ˆ ). RLTS The Least Squares estimator was defined in equation (). The Ridge estimator was defined in equation (4) using the K value in equation (5). The Ridge Least Absolute Deviation estimator was defined in equation (9) using the K value in equation (10). The Ridge Least Trimmed Squares estimator was defined in equation (1) with K of equation (13). Suppose, we have the following linear regression model: y x x e, where i 1,,..., n (15) i o 1 i1 i i Dempster et al. (1977) [4], pointed out that, the parameter values of o, 1 and are set equal to one. The explanatory variables x i1 and x i are generated as: x ( 1 ) z z, i 1,,..., n, j 1, (16) ij ij ij

7 Alternative biased estimator based on least trimmed squares 183 Where, z are independent standard normal random numbers generated by the ij normal distribution. The value of representing the correlation between the two explanatory variables and its values were chosen as: 0.0, 0.5, and Once, for a given sample size n, the explanatory variables values were generated. The sample sizes which will be examined in this study are: 0, 40 and 60. One important factor in this study is the disturbance distribution. The following three disturbance distributions are used: Standard normal distribution. distribution with mean zero and variance two. distribution with median zero and scale parameter one. In general, all the obtained random numbers are generated using the IMSL subroutines as: Standard normal random numbers are generated using the GGNPM subroutine. random numbers are generated using the GGUBFS subroutine. random numbers are generated using the GGCAY subroutine. The simulations were performed on an IBM 4341 Model 1. Programs were written in double-precision FORTRAN. For each of the treatments in the three factor experiment, (, sample size n, number of distributions), 500 Mote Carlo trials are used, and the following statistics are computed. (1) The average of the estimates. () The mean squared error (MSE) and the 6 pairwise MSE ratios where: MSE ( βˆ i - βi ) (17) 500 i 1 (3) The 6 pairwise comparisons of " closeness " to the actual parameter values. The pairwise comparisons are : ( ˆ with ˆ R ),( ˆ with ˆ RLAD ), ( ˆ with ˆ ) RLTS, ( ˆ with ˆ R ) RLAD, ( ˆ R with ˆ ) RLTS, ( ˆ with ˆ ) RLAD RLTS. (6.) The Results of comparisons We consider the comparison of the two ridge robust estimators RLAD and RLTS. Table (1) presents the number of times that the RLTS estimates are closer than the RLAD estimates to the true value of the parameter only. While, 1 Table () presents the results of the mean squared estimation error ratios. These ratios represent the efficiency of RLTS relative to RLAD. It be noted that, values less than one indicated that RLTS is more efficient, while values greater than one indicated that RLAD is more efficient.

8 184 Moawad El-Fallah Abd El-Salam Table (1): Number of times RLTS is closer than RLAD to the true parameter value. 1 Error Distribution Values of n n n Table (): MSE Ratios of RLTS to RLAD for Estimation of * 1. Error Distribution Values of n n n It be noted that, values less than one indicate RLTS is more efficient than RLAD; while, values greater than one indicate RLAD is more efficient than RLTS.

9 Alternative biased estimator based on least trimmed squares 185 From the results of Table (1), we see that the RLTS estimator performs better than RLAD estimator over a wide range of combinations between and the error distribution. These results are supported by the mean squared estimation error ratios presented in Table (). Therefore, as the RLTS estimator clearly is superior to the RLAD estimator, the remaining comparisons will be restricted to RLTS to conserve space. Tables (3) and (5) show the number of times that the RLTS estimates are closer than the, RR and estimates, respectively, to the true value of the parameter. Also, the MSE ratios of RT to each of these estimators RR and 1 are given in Tables (4) and (6) respectively. From Tables (3) and (4), we see that the RR estimator marginally is superior than RLTS when disturbances are normal and the correlation is high. Otherwise RLTS is superior. From Tables (5) and (6), is superior when there is no correlation except for disturbances. Otherwise, RLTS is superior. To conclude, the results from comparisons of RLTS estimator to the RR, RLAD and estimators are not entirely unexpected, given the properties of the various estimators. Therefore, the most important result from these comparisons is, the RLAD estimator is superior to any other estimator over a wide range values of for the given disturbance distributions as the ridge regression, in some cases, is expected to perform well. Table (3): Number of times RLTS is closer than RR to the true parameter value. 1 Values of Error Distribution n n n

10 186 Moawad El-Fallah Abd El-Salam Table (4): MSE Ratios of RLTS to RR for Estimation of * 1 Error Distribution Values of n n n * Value less than one indicate RLTS is more efficient than RR; while, values greater than one indicate RR is more efficient than RLTS. Table (5): Number of times RLTS is closer than to the true parameter value. 1 Values of Error Distribution n n n

11 Alternative biased estimator based on least trimmed squares 187 Table (6): MSE Ratios of RLTS to for Estimation of 1 Values of Error Distribution n n n * Value less than one indicate RLTS is more efficient than ; while, values greater than one indicate is more efficient than RLTS. 7. Concluding Remarks The presence of leverage data points and multicollinearity are considered two of the more frequent problems in regression analysis. Although, we usually think of these two problems separately, however, these problems occur simultaneously in applied situations. A Monte Carlo simulation was designed to compare the performance of some combining ridge and robust regression estimators for dealing with these two problems. The results of comparisons indicated that, the ridge least trimmed squares (RLTS) estimator is better than other estimators (, RR and RLAD) for many combinations of non-normal error distribution type (which reflected the presence of leverage data points) and degree of multicollinearity (Tables (), (4) and (6)). Only, this estimator is less efficient than the others when disturbances are normal. In addition, RLAD outperforms both LAD and estimators when the degree of multicollinearity is high. Therefore, the RLTS estimator appears to be a suitable alternative to other estimators for the different combinations of multicollinearity and leverage data points which are indicated by the non-normal error distributions. There are limitations to the present study, however. First, since this is a simulation study, its limitations must be recognized. Data have been generated to try and allow generalization to practical situations, however. Second, other possible members of the robust regression approach may be used to construct the title of combined biased robust estimators.

12 188 Moawad El-Fallah Abd El-Salam References [1] K. Ayinde, Combined Estimators as Alternative to Ordinary least Squares Estimator, International Journal of Sciences: Basic and applied Research, 8 (013), no. 1, [] D. Belsley, E. Kuh and R.E. Welsh, Regression Diagnostics: Identifying Influential Data and Sources of Collinearity, Wiley & Sons, New York, [3] C. Chen, Robust regression and outlier detection with the ROBUSTRER procedure, SUGI paper, SAS Institute, (00), [4] A.P. Dempster, M. Schatzoff and N. Wermuth, A simulation study of alternatives to Ordinary Least Squares, Journal of the American Statistical Association, 7 (1977), no., [5] D. Gibbons, A simulation study of some ridge estimators, Journal of the American Statistical Association, 76 (1981), no. 3, [6] A.E. Hoerl and R. W. Kennard, Ridge Regression: Iterative Estimation of the Biasing parameter, Communications in Statistics: A Theory Methods, 5 (1970), no. 1, [7] B. Kan, O. Alpu and B. Yazici, Robust Ridge and Robust Liu Estimator for Regression Based on the LTS estimator, Journal of Applied Statistics, 40 (013), no. 3, [8] B. Kan and O. Alpu, Combining Some Biased Estimation Methods with Least Trimmed Squares Regression and its application, Rev. Colomb. Estatist., 38 (015), no., [9] W.S. Krasker and R.E. Welsch, Efficient bounded influence regression estimation, Journal of the American Statistical Association, 77 (198), no. 379, [10] A. Lukman, O. Arowolo and K. Ayinde, Some Robust Ridge Regression for Handling Multicollinearity and Outlier, International Journal of Sciences: Basic and Applied Research, 16 (014), no., [11] C.Z. Mal and Y.L. Dul, Generalized shrunken type-gm estimator and its application, International Conference on Applied Sciences (ICAS013), IOP Publishing Ltd., Vol. 57, 014,

13 Alternative biased estimator based on least trimmed squares [1] A. Moawad Abd El-Falah, The Efficiency of Some Robust Ridge Regression For Handling Multicollinearity and Nonnormal errors problems, Applied Mathematical Science, 7 (013), no. 77, [13] O.E. Nkiruka and O.J. Unhenna, A Comparative Study of Some Estimation Methods for Multicollinear Data, International Journal of Engineering and Applied Sciences, 3 (016), no. 1, [14] R.C. Pfaffenberger and T.E. Dielman, A Modified Ridge Regression Estimator Using the Least Absolute Value Criterion in the Multiple Linear Regression Model, Proceedings of the American Institute for Decision Sciences, Toronto, (1984), [15] M. Z. Siti, S.Z. Mohammad and Al. bin I. Muhammad, Weighted Ridge MM- Estimator in Robust Ridge Regression with Multicollinearity, Mathematical Models and Methods in Modern Science. Symp. Computational Statistics, 1 (01), no. 3, [16] P.J. Rousseeuw, Least median of squares regression, Journal of the American Statistical Association, 79 (1984), no. 388, [17] P.J. Rousseeuw and A.M. Leroy, Robust Regression and Outliers Detection, Wiley & Sons, New York, [18] P.J. Rousseeuw and K. Van Driessen, A fast algorithm for the minimum covariance determinant estimator, Technometrics, 41 (1999), no. 3, [19] V.J. Yohai, High breakdown- point and high efficiency robust estimates for regression, Annals of Statistics, 15 (1987), no., [0] A. Zaman, Statistical Foundations for Econometric Techniques, Academic Press, New York, Received: June 15, 018; Published: July 18, 018