Application of Independent Variables Transformations for Polynomial Regression Model Estimations

Transcription

1 3rd International Conference on Applied Maematics and Pharmaceutical Sciences (ICAMPS'03) April 9-30, 03 Singapore Application of Independent Variables Transformations for Polynomial Regression Model Estimations W. Wongrin, M. Rodchuen and P. Booamana Abstract The obective of is research was to find out e solution to e polynomial regression model which has a problem wi multicollinearity. The study was done by transforming of independent variables. The first proposed solution was centering. The second was Z[-,] transformation. The next proposed solution was Z[-,] transformation wi Legendre polynomials. The last was e transformation matrix to be orogonal matrix. Least squares meod was used to e parameters estimation wi stepwise regression. We applied e transformations to fit a 4 degrees polynomial regression for e heat capacity of solid hydrogen bromide (HBr) and temperature. It was found at orogonal matrix had e lowest VIFs and e small mean square error (MSE). Thus, e model from orogonal matrix is e best wi e smallest effect size on multicollinearity. Keywords Centering, Multicollinearity, Orogonal Matrix, Polynomial Regression. I. INTRODUCTION INEAR regression is a statistical technique at is widely L used to model e relationship between dependent variable (Y) and independent variable (), in which e relationship between e variables is linear. However, e practical non linear function or curvilinear. So, polynomial regression is appropriate for e data. The polynomial regression is a special case of multiple linear regressions. Since, we considered each term of e polynomial as an independent variable [],[]. The parameters in e polynomial regression were estimate by least squares meod. However, e least squares meod has a problem about e correlation among e term of e polynomial called multicollinearity or ill conditioned matrix. It shows e matrix ' becomes a singular matrix ( ' 0) [],[]. Multicollinearity usually leads to unreliable estimated parameters in e polynomial regression, which en has large variances. Thus, ere are many ways to reduce e effects on multicollinearity such as augmentation of e data, selection of W. Wongrin is wi a master student of Department of Statistics, Faculty of Scince, Chiang Mai University, Thailand (phone: ; weerinrada@gmail.com). M. Rodchuen, is wi e Department of Statistics, Chiang Mai University, Thailand.( r.manachai@gmail.com). P. Booamana is wi e Department of Statistics, Chiang Mai University, Thailand.( putipongb@gmail.com). variables and alternative procedure: Ridge regression, Principle component analysis, Shrun estimates, etc. Multicollinearity can be addressed by centering, but may remain larger correlation between terms of e polynomial. [3] So e independent variables transformations are proposed to minimize e effects of e collinearity by Z[-,] transformation, V[0,] transformation, W[0,] transformation and ree above transformations wi Legendre polynomials. Hence, e Condition number was used to determine e effects on multicollinearity. The Z[-,] transformation and e Z[-,] transformation wi Legendre polynomials are e best models. [4] Alough oer meods such as Weighted least squares meod, Ridge absolute value meod and Liu-Type meod are proposed to estimate e parameters. The one important ing can done to remove e multicollinearity is orogonal polynomials. [5] Therefore, e researchers were interested in fitting e data of e heat capacity of solid hydrogen bromide (HBr) and temperature [4] by e polynomial regression model wi 4 degrees. We proposed to use centering, e Z[-,] transformation, e Z[-,] transformation wi Legendre polynomials and e matrix transformation to obtain orogonal columns. This research would lie to estimate e parameters using e meod of least squares wi stepwise regression. We checed on multicollinearity wi condition number and VIFs (Variance inflation factors). In addition, we used e Mean Square Errors (MSE) to determine e efficiency of e models. II. POLYNOMIAL REGRESSION Polynomial regression is a special case of multiple linear regression which sets each term of e polynomial regression model as an independent variable. The model of e form is i = β0 + β i + β i + + β i + + β i + εi Y () When i =,,.., n, = 0,,,.., and Y i = e dependent variable of i unit, β = e parameter or regression coefficient, i = e independent variable unit i in e order of e polynomial, and ε i = e random error. 330

2 3rd International Conference on Applied Maematics and Pharmaceutical Sciences (ICAMPS'03) April 9-30, 03 Singapore Hence, e model in matrix form is Y = β + ε The least squares meod is used to estimate e parameters in e fixedx model. We find e estimators at minimize e sum of squares error of e n observed ( y) from eir predicted values ( y ˆ). III. INDEPENDENT VARIABLE TRANSFORMATIONS A. Centering Data Centering [], [3] is used to invert n observations on independent variables using eir center or eir means. It is widely used to reduce e collinearity between independent variables to zero, but still has highly collinearity among term x and x, < when + is even, [3]. xnewi xi x Where x newi = e observation unit variable already centered, = e observation unit x i = () variable, and x = e mean of independent variable We have matrix of x at centered new xnew xnew x new xnew xnew xnew = x x x newn newn newn n i on independent i on independent Hence e solution of normal equation wi centering e ' new new β ' =new Y B. Z[-,] Transformation The Z[-,] Transformation is a transformation of e data which can reduce e effects of e collinearity in regression at are widely used and recommended by statisticians. The Z[-,] Transformation used to normalized x run to - to + is given by [4] zi = ( xi xmax xmin ) / ( xmax xmin ) (3) Where z i = e observation unit i on independent variable already transformed, x max = e maximum of observed value on independent variable, and x min = e minimum of observed value on independent variable We have matrix of at transformed z z z 0 z z z Z = 0 z z z n n n n Hence e solution of normal equation wi transforming e ' ' Z Z β =ZY C. Z[-,] Transformation wi Legendre Polynomials One of approach often suggested to reduce collinearity in polynomial regression is orogonal polynomials. Because is property, e orogonal polynomials yield diagonal normal matrix. There are several ways to generate orogonal polynomials. For a large number of data points which are evenly spaced in e [-,] interval, Legendre polynomials provide an orogonal set. Hence e first six Legendre polynomials for e Z[-,] transformation are given by [4] 0( z) = ( z) = ( + 3 z ) 3 3 3( z) = ( 3z+ 5 z ) 5 4 4( z) = (3 30z + 35 z ) ( z) = (5z 70z + 63 z ) ( z) = ( z 35z + 3 z ) 3 D. Matrix Transformations by Orogonal If e columns in a matrix are highly correlated, we can transform e columns using orogonal. Therefore, we have orogonal columns which can be an orogonal matrix. The columns transformation taes e following form [], [], [5] Z = { I Z ( Z Z ) Z } = Z ( Z Z ) Z Where Z = e matrix of column vector already transformed, = e next column vector of matrix at to be transformed, and Z = transformed vector and orogonal to oer vectors in e previous Z matrix So, we have e orogonal matrix at is a diagonal matrix which it s easy to calculate e inverse of e matrix (4) 33

3 3rd International Conference on Applied Maematics and Pharmaceutical Sciences (ICAMPS'03) April 9-30, 03 Singapore z z z 0 z z z Z = 0 z z n n zn n Hence e solution of normal equation wi transforming e ' ' Z Z β =Z Y IV. MEASURING MULTICOLLINEARITY A. VIFs (Variance Inflation Factors) The VIFs show e effects of e independent variables in a multiple linear regression model. They provide an index at measures how much e variance of e estimators change because of multicollinearity. The VIFs can be obtained from When when e VIFs = ( R ) (5) R is e multiple correlation coefficient obtained predictor variable is regressed against all e remaining predictor variable i (when i) Note at if VIFs are greater an 5 or 0, it indicates e correlations among independent variables are occurred.[6] V. CONDITION NUMBER To measure e multicollinearity using Eigen system analysis from eigenvalues at is determined by condition number (K( )) from e below λ K( ) = max (6) λmin Where λ max = e maximum of e eigenvalues of e λ min = e minimum of e eigenvalues of e Note at if K( ) is greater an,000 indicates e wrong sign of multicollinearity. [6] Bo K( ) and VIFs can diagnose e collinearity on independent variables but ese indicators is at ere are no well established reshold values for em to indicate harmful level of collinerity. Therefore, ey cannot be considered as quantitative measures for collinerity [7]. VI. THE MODEL EVALUATION The performance of e models was compared using mean square error of e predicted value of dependent variables. ˆ ˆ MSEY ( ) EY ( Y) = (7) And it could be estimated by n n i i i i= i= MSE( Yˆ ) ( y yˆ ) / n = ( eˆ ) / n (8) VII. A POLYNOMIAL REGRESSION MODEL FOR THE HEAT CAPACITY OF SOLID HYDROGEN BROMIDE (HBR) In most cases, e relationship between e heat capacity of a substance and temperature is a curvilinear. The data shown in Table is e relationship between e heat capacity of solid hydrogen bromide on temperature at was fitted e polynomial regression model up to 4 order TABLE I THE DATA OF SOLID HYDROGEN BROMIDE AND TEMPERATURE Cp(Capacity) Temperature Cp(Capacity) Temperature *Data from e wor of Giauque and Wiebe in 98 VIII. RESULTS From fitted curve between e heat capacity and temperature, e relationship between e data was curvilinear. So, we should to model e highest degrees of e polynomial up to 4 order. (See Fig.). heat capacity Scatterplot of Heat capacity vs Temperature x Fig. Scatter plot of Heat Capacity vs. Temperature From analyze e data; we applied to use 4 transformations of e independent variables. The results were at e Z[-,] transformation had e smallest condition number followed by e Z[-,] transformation wi Legendre polynomial when e independent variables was not transformed have e largest condition number. The difference of condition number caused e highly observed values and e eigenvalue of e matrix. If e matrix had e smallest eigenvalue, e condition number was small as well. The

4 3rd International Conference on Applied Maematics and Pharmaceutical Sciences (ICAMPS'03) April 9-30, 03 Singapore higher of condition number indicates e effects on of multicollineariry, shown in Table II. Transformations TABLE II THE CONDITION NUMBER OF THE MATRI Condition Number Independent Variables 6.5 x 0 8 Centering x 0 0 Z[-,] Transformation Z [-,] transformation wi Legendre polynomials Orogonal matrix.858 x 0 7 The models, 3 4 Model Yˆ i = 0.47xi xi xi (9) 4 Model Yˆ i = xnewi xnewi (0) 3 Model 3 Yˆ i = zi zi zi () 3 Model 4 Yˆ i = zi zi z i () 3 Model 5 ˆ Yi = z i z i z i (3) Considering of e VIFs values, we ordered each terms among e models. It showed at e model was e highest, because of each term of e polynomial was not transformed and had e problem wi e highly collinearity among e terms. The model 5 was e lowest, since each term was orogonal, at each term of e polynomial is exactly linear independent, shown in Table III. It is noted at e model 4, e model from Z[-,] transformation wi Legendre polynomials are e orogonal polynomials. The VIFs values were quite good, but not equal to. It is suggested ere is a little correlation among terms in e polynomial model. VIFs TABLE III THE VIFS VALUES OF THE MODELS Models The results of e modeling show at all 5 models, e multiple correlation coefficient and e effects of e selection degree of e polynomial wi stepwise regression as shown in Table IV. TABLE IV THE COEFFICIENT OF DETERMINATION AND MEAN SQUARE ERRORS OF THE MODEL Models R MSE( Y ˆ) Model 99.99% Model 99.00% Model % Model % Model % Consequently, e mean square errors of e models have shown in Table IV. It is shown at e model 3, 4 and 5 had e small while e highest was e model. Alough, e mean square errors of predictor of e model 3, 4 and 5 were equally and ey had e same polynomial order and e same predicted values, but e variances of e estimate parameters were differences. The estimators variance of e model 5 was minimizing as shown in e Table V. Thus e model 5 can predict e values of e dependent variable wi highest accuracy and precision from all 5 models. TABLE V THE STANDARD ERRORS OF THE COEFFICIENT REGRESSION Models 0) ) ) 3) Model 3.08 x x x x 0-3 Model x x x x 0-3 Model x x x x 0 - I. CONCLUSION It was found at, e polynomial regression model order 4 for e relationship of e heat capacity of solid hydrogen bromide and temperature based on e independent variables transformations wi 4 transformations should use e transforming matrix into orogonal matrix, which is linearly independent among e terms of e polynomial. It also provides e error of an estimate for e dependent variable at least as well. The high accuracy of predicted values are obtained, while e model still has highly collinearity among terms of e polynomial and e highest of mean squares error. ACKNOWLEDGMENTS Thans to e professors of department of statistics for e guidance of is paper. We are grateful to e Graduate School Chiang Mai University, Faculty of Science, Department of Statistics and Science Achievement Scholarship of Thailand to support is research. REFERENCES [] T. Veraaworn, Linear Model (Theory and Application). Bango : Vittayawat, 998. [] G. A. Seber, F. and A. J. Lee, Linear Regression Analysis. nd ed. New Yor : John wiley & Sons,

5 3rd International Conference on Applied Maematics and Pharmaceutical Sciences (ICAMPS'03) April 9-30, 03 Singapore [3] R. A. Bradley, and S. S. Srivastava, Correlation in polynomial regression, Journal of e American Statistical Association, vol. 33, pp. -4, 979. [4] M. Shacham, and N. Brauner, Minimizing e effects of collinearity in polynomial regression, Journal of e American Chemical Society, vol. 36, pp , 997. [5] G-L. Tian, The comparison between polynomial regression and orogonal polynomial regression, Statistics & Probability Letters. vol. 38, pp , 998. [6] D. C. Montgomery, E. A. Pec, and G. G. Vining, Introduction to Linear Regression Analysis. 4 ed. New Yor : John wiley & Sons, 006. [7] N. Brauner, and M. Shacam, Indentifying and removing sources of imprecision in polynomial regression, Maematics and Computers in Simulation, vol. 48, pp. 75-9, 998. [8] A. C. Rencher, and G. B. Schaalie, Linear Models in Statistics. nd ed. New Yor : John wiley & Sons, 008. [9] N. R. Draper, and H. Smi, Applied Regression Analysis. 3rd ed. New Yor : John wiley & Sons, 998. [0] S. Weisberg, Applied Linear Regression. 3rd ed. New Yor : John wiley & Sons,