Least Squares and Discounted Least Squares in Autoregressive Process

Transcription

1 122 Least Squares and Discounted Least Squares in Autoregressive Process Least Squares and Discounted Least Squares in Autoregressive Process Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya Abstract his paper reports on a comparative study between the least squares (LS) method using rank-one update QR factorization and the discounted least squares with direct smoothing Both approaches are applied to solve the problem of updating estimated parameters in long rolling periods of time-series forecasting using the autoregressive (AR) process he corresponding model used in the experiment is undamped sinusoidal data with various autoregressive orders he results of the study indicate that both methods improved effectively as the model order grow However, the discounted least squares with direct smoothing had a more increasing rate of improvement Keywords : Autoregressive process; Estimating parameter; Least squares; Discounted Least squares Introduction In many industries, forecasting has become an important tool to reduce risk from uncertainty echniques can be applied to forecast demand in order to plan the production schedule or plan for reserving inventory or ordering material from suppliers Such data are time-related and highly correlated so ime Series Analysis is a particularly significant tool his study focused on one specific type of ime Series Model: the Autoregressive (AR) model he AR model was developed by Box and Jenkins in 1970 (Box, 1994) to analyze historical data that had relations within itself pmd 122

2 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 123 In this study, the parameters were obtained from univariate stationary autoregressive data (Kendall, 1976) when the population mean of the process was not zero he AR process utilizes the least squares (LS) method, and it is an analogous way to fit a model by minimizing the sum of square errors for estimating parameters he LS method uses the normal equations to implement the linear system he parameters can be solved by QR factorization he earlier observations in LS method receive the same weight as recent observations However, the recent observations may be more important for the true behavior of the process so that, in discounted least squares method, the older observations receive proportionally less weight than the recent ones Another aspect is that when periods roll for a long time, the parameters of the time series model must be changed for updating the model he disadvantage in the use of LS as a forecasting method is that the estimated parameters need to be updated at the end of each period his study attempted to reduce the computation time by using the discounted least squares method with direct smoothing to fit an AR model Brown (1962) developed the discounted least squares method with direct smoothing for estimating regression model parameters he purpose of this study was to implement the discounted least squares method with direct smoothing for estimating autoregressive model parameters which were not found in previous research As the observation period shifts, the direct smoothing can update the old estimates of the model parameters by smoothing them with the forecast error for the current period to obtain the revised estimates In addition, a comparison between the LS method using rank-one update QR factorization and the discounted least squares with direct smoothing was performed Cost and accuracy of forecasting were two aspects to be considered in comparing the methods pmd 123

3 124 Least Squares and Discounted Least Squares in Autoregressive Process Methods In this study, the linear stochastic stationary time series data were fitted to an AR model by using two methods : the LS method and the discounted least squares method In addition, we assumed that the population mean was known and not to be zero We also considered the process by using two systems of equation : x t = φ 0 +Σ φ i x t-i + ε t, t=1 i = 1,2,,p, (1) y t = Σ φ i y t-i + ε t, t=1 i = 1,2,,p, (2) where x t is an observation at time t, φ 0 is the scalar, and φ i are the unknown parameters When xô t is a forecasting value at time t, it has a random error ε t that Ε(ε t ) = 0 and V(ε t ) = σ 2 ε Equation (2) was obtained by defining y = x - µ x An AR model is simply a linear regression of the current value of the series against one or more prior values of the series he value of p is called order of AR model he least squares method : In LS method, the AR model parameters in equation (1) are estimated by minimizing the error sum of squares : Min 2 l(φ) = Σε t t= pmd 124

4 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 125 It gives the linear systems equation Ax = b from least squares normal equation as follows :, (3) where A is a square matrix formed by using the sum of backward elements at a time, while x is a parameter vector b is a right-hand side vector, and is the end of observation time When y = x-µ x, the system equation gives the linear systems equation Ax = b from least squares normal equation as follows : (4) he first and second linear systems equations are solved using the QR factorization (Golub and Van Loan, 1996) It is defined as follows : A x = b Since A = Q R then Q R x = b, Q Q R x = Q b, -1 and Q = Q R x = Q b pmd 125

5 126 Least Squares and Discounted Least Squares in Autoregressive Process he results from this method were used as the model parameters he AR models were obtained and the forecast errors computed to validate the result In time series analysis, data are always shifted to the next period because the parameters must have new estimated values to update the model he parameters are updated by calculating the next estimated values that employ rank one update QR factorization as defined by Householder (Cavallo et al, 1996) Suppose that the Q R = A r m n and A +1 = A +u v = Q +1 R +1 where u, v r n are given, such that A +1 = A + u v = Q (R + w v ), where w u = Q Compute the rotation J J w = w 2 e 1 Each l n-1 J k is the rotation in plane k and k+1 he given rotations applied to R are shown as (Golub and Van Loan, 1996) : Consequently, H = J J R (5) 1 n-1 J J (R + w v ) = H ± w 2 e 1 v = H 1 (6) 1 n-1 Given rotations, G k, k = 1 : n - 1 such that G G H = R (7) n hen obtained is the QR factorization A +1 = A + u v = Q +1 R +1, where Q +1 = Q J n-1 J 1 G 1 G n-1 (8) Now, Q +1 and R +1 are obtained from updated Q and R pmd 126

6 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 127 he discounted least squares: he recent observations may be more important than the older observations his procedure is the weighted least squares that gives a lower weight of older observations than of the recent ones From equation (2) y t = Σ φ i y t-i + ε t where W tt is the square root of the weight given t=1 to the t th error (Montgomery, 1990) o minimize discounted sum of square of errors and obtain the estimated parameters, the objective function can be stated as Min 2 2 l (φ) = Σ W tt ε t, t=1 W is the diagonal matrix of square roots of the weights W = W W W he weighted sum of square error is l (φ) = (W ε) (W ε) l (φ) = [ y - Y () φ] W 2 [y - Y () φ] hen, the least squares normal equations can be written as : Y () W 2 Y ( ) Âφ = Y ( ) W 2 y Since G ( ) = [W Y( )] [W Y( )], and g i ( ) = Y ( ) W 2 y, then G ( ) Âφ = g ( ), (9) and the solution is Âφ i ( ) = G -1 ( ) g ( ) pmd 127

7 128 Least Squares and Discounted Least Squares in Autoregressive Process L is the transition matrix (Montgomery, 1990) of the system equation (2), that is L = φ 1 φ 2 φ 3 φ p-1 φ p he transition property for system equation (2) can be described as y (t + 1) = L y(t) Let the weights W 2 be defined as W 2 = β j when j = 0,1,, tt -j,-j - 1 and 0 < β < 1 then G () = Σ β j y(- j)y (- j) j= 0 he matrix G() approaches a limit G, where -1 G lim G() = Σβ j y(- j)y (- j) j= 0 herefore G -1 needs to be computed only once he right-hand side of the normal equation may be written as : or -1 g() = Σβ j y -j y( j), j= 0 g() = y y(0) + βl -1 g( -1) Substituting for g() in equation (9), then we obtain ^ φ i () = y G -1 y(0) + βg -1 L -1 Gφ(-1) (10) pmd 128

8 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 129 Since h = G -1 y(0) and H = βg -1 L -1 G, then equation (10) may be written as ^ o simplify φ i () ^ φ i () = hy + Hφ( - 1) L -1 G = L -1 G(L ) -1 G, L -1 G = Σ β j L -1 y(-j)y (-j)(l ) -1 (L ), = β -1 (G- y(0)y (0))L Substituting L -1 G in H, we now have H = (I - G -1 y(0)y (0))L Substituting h = G -1 y(0) in H, we obtain j= 0 H = L - hy (0)L, and the single period error e 1 () is given by ^ so φ i () now becomes ^ e 1 () = y - y^ ( - 1), ^ φ i () = L φ i ( - 1) + he 1 () (11) he discounted least squares estimate the model parameters shown in equation (11) his method used to estimate the parameters at the end of th period is a linear combination of estimates made at the end of the previous period and the singleperiod forecast error pmd 129

9 130 Least Squares and Discounted Least Squares in Autoregressive Process he transition matrix (L) of equation (1) is of the form L = φ 1 φ 2 φ 3 φ p-1 φ p φ he steps of discounted least squares method to obtain the model parameters can be described as follows : Step 1: Calculate G() when =15,000 (assume period 15,000 th ) and the optimal weight is defined Step 2: Calculate G -1 () Step 3: Obtain φ Step 4: Forecast one step ahead Step 5: Calculate e +1 = x +1 - x^ +1 Step 6: Obtain h Step 7: Obtain L Step 8: Obtain φ +1 In this study, the undamped sinusoids autoregressive data (Martinez, 2002) were generated with four orders of 5, 12, 20 and 30 herefore, AR(5), AR(12), AR(20), and AR(30) were investigated and the forecasts of those models were started at 15,000 th observation by forecasting one step ahead hen, repeated the step of forecasting one step ahead for 1000, 5000, 10000, 15000, and times MALAB 70 was used to generate the data while Minitab used to check the AR property In addition, the QR factorization and the QR rank one update were applied by the algorithm in equations (6) - (9) using MALAB function he LS method and the discounted least squares method with direct smoothing were written as MALAB M-file he results were evaluated by measuring the computation time and prediction mean square error to compare computing cost and accuracy of the two forecast methods, respectively pmd 130

10 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 131 Results and discussion he results for long rolling period forecasting of the least squares method and the discounted least squares method indicate the preference to the least squares able 1 illustrates the comparison via the operating time, while able 2 illustrates the comparison via the prediction mean square error he least squares method with QR factorization had both less operation time and less prediction mean square errors However, as the order of the model grows, the variances of data decrease hus, for both methods, the prediction mean square errors were smaller he discounted least squares with direct smoothing method gave higher decreasing rate in prediction mean square errors as the order of the model grew he update QR function in MALAB has an accelerated tool but the direct smoothing uses only MALAB language in M-file, and the discounted least squares with direct smoothing adjusts the transition matrix in each period For these reasons, the discounted least squares method consumed more time than the other method Besides, the selected weight may not be appropriated for the discounted least squares method and results in the higher prediction mean square error pmd 131

11 132 Least Squares and Discounted Least Squares in Autoregressive Process able 1 Comparison of time of calculations to estimate parameters obtained for long rolling periods when using two methods ime (sec) AR(p) No of Least squares with Discounted least squares forecasts QR factorization with Direct smoothing Eq (1) Eq (2) Eq (1) Eq (2) AR(5) AR(12) AR(20) AR(30) pmd 132

12 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 133 able 2 Comparison of prediction mean square error of calculations to estimate parameters obtained for long rolling periods when using two methods Prediction Mean Square Error AR(p) No of Least squares with Discounted least squares forecasts QR factorization with Direct smoothing Eq (1) Eq (2) Eq (1) Eq (2) AR(5) AR(12) AR(20) AR(30) pmd 133

13 134 Least Squares and Discounted Least Squares in Autoregressive Process Conclusion When forecasting a stationary autoregressive process for long rolling periods with µ 0, the least square with QR factorization is appropriated, leading to higher efficiency than the discounted least squares with direct smoothing he autoregressive model as in equation (1) had more efficiency for both methods than the model in equation (2) When the order of the model grew, both methods had more accuracy in forecasting since the prediction mean square errors were decreased he discounted least squares with direct smoothing used more operation time since the transition matrix was adjusted in every period For further study, the transition matrix shoud be tried without adjustment by time, and another weight approach should be tested pmd 134

14 Phongsaphen Jantana and Prapaisri Sudasna-na-Ayudthya 135 References Box, GP et al (1994) ime Series Analysis Forecasting and Control, Prentice Hall, New Jersey Brown, RG (1962) Smoothing, Forecasting and Prediction of Discrete ime Series, Prentice-Hall Inc, New Jersey Cavallo, A et al (1996) Using MALAB, SIMULINK and Control System oolbox, Prentice-Hall Inc, London Golub, GH and Van Loan, CF (1996) Matrix Computations, he Johns Hopkins Press, London Hannan, EJ et al (1995) ime Series in ime Domain, Elsevier Science BV, Netherlands Kendall, M (1976) ime Series, Charles Griffin and Company Ltd, London Martinez,WL and Martinez, AR (2002) Computational Statistics Handbook with MALAB, Chapman & Hall/CRC, New York Montgomery, DC et al (1990) Forecasting and ime Series Analysis, McGraw-Hill Inc, Singapore pmd 135