Bayesian Analysis of the Autoregressive-Moving Average Model with Exogenous Inputs Using Gibbs Sampling

Transcription

1 Journal o Applied Sciences Research, 4(): , 8 8, INSInet Publication Bayesian Analysis o the Autoregressive-Moving Average Model with Exogenous Inputs Using Gibbs Sampling Hassan M.A. Hussein Department o Statistics, Faculty o Commerce, Zagazig University, Egypt Abstract: The problem o estimating a set o parameters in the autoregressive moving average model with exogenous inputs (ARMAX) is considered and a numerical Bayesian method proposed. This paper, develops a Bayesian analysis or the ARMAX model by implementing a ast, easy and accurate Gibbs sampling algorithm. The procedure is easy to implement and can be computed also when some priors in the ARMAX are diuse. The empirical results o the simulated examples and electricity consumption data in the industrial sector in Egypt showed the accuracy o the proposed methodology and has good statistical properties. Key words: Autoregressive-Moving Average Model With Exogenous Inputs, Gibbs Sampling INTRODUCTION Autoregressive Moving Average (ARMA) models can be generalized in other ways, such as, Autoregressive Conditional Heteroskedasticity (ARCH) models and Autoregressive Integrated Moving Average (ARIMA) models. The ARIMA models are introduced to the purposes o orecast or time series data by [] Box-Jenkins. The models are widely used as orecast techniques and developed to various econometric methodologies such as the Autoregressive Moving Average model with Exogenous inputs [7] (ARMAX),. The ARMAX model just allows explanatory variables in ARIMA system. Because o energy crisis in the 97s and the price hikes, especially in oil prices, economic growth o developing countries has been negatively aected. The association between energy consumption and economic growth have been extensively investigated since the late 97s. These studies show that the relationship between energy consumption and economic growth is still a controversial one. Dierent results have been ound not only or dierent countries but also or dierent time within the same country. The empirical results suggest that, the electricity consumption can be considered as a good approximation or the consumed quantities o energy or the industrial sector. Recently, Bayesian time series analysis has been advanced by the emergence o Markov Chain Monte Carlo (MCMC) methods, especially the Gibbs sampling method. Assuming a prior distribution on the initial observations and initial errors. Chib and Greenberg [3] [] and Marriott et al. developed a Bayesian analysis or autoregressive moving average (ARMA) models using MCMC technique. [9] Ismail employed Gibbs sampling to develop Bayesian analysis o the multiplicative, seasonal moving average model. Her approach was based on approximating the likelihood unction via estimating the unobserved errors. The approximate likelihood is then used to derive the conditional distributions required or implement the Gibbs sampler. [9] A similar approach to that o Ismail is used in this study but or developing Bayesian analysis o the Autoregressive Moving Average model with exogenous inputs (ARMAX). The procedure has the advantage that can be used under inormative and non-inormative priors on the parameters o interest. Moreover,this procedure does not require the estimation o two models, one with and the other without the restriction to be tested i the problem o testing a set o restrictions in the underlying model is considered. In this paper, we present a simple and ast algorithm to estimate the parameters and orecast uture values o Bayesian analysis or the ARMAX model by implementing a ast, easy and accurate Gibbs sampling algorithm models. The procedure is easy to implement and can be computed also when some priors in the ARMAX model are diuse. The Bayesian approach allows or incorporating uncertainty about the parameters and initial errors. The empirical results o the simulated examples and the electricity consumption data in the industrial sector in Egypt showed the accuracy o the proposed methodology and has a good statistical properties. Corresponding Author: Hassan M.A. Hussein, Department o Statistics, Faculty o Commerce, Zagazig University, Egypt 885

2 The paper is organized as ollows. Section briely describes the autoregressive moving average model with exogenous inputs (ARMAX). Section 3 is devoted to summarize posterior analysis, the ull conditional posterior distributions and the implementation details o the proposed algorithm. In Section 4, A Monte Carlo study is discussed and moreover, the results o the estimation using a simulated example and the monthly electricity consumption data in Egypt are given. Finally, conclusions are given in Section 5. Autoregressive moving-average model with exogenous inputs (ARMAX): The notation ARMAX (p, q, b) reers to the model with p autoregressive terms, q moving average terms and b exogenous inputs terms. This model contains the AR(p) and MA(q) models and a linear combination o the last b terms o a known and external time series d t. The model can be expressed as: and, the time series is as to start at time t = with unknown starting errors. Since the model is linear in γ, ζ and η which complicates the Bayesian analysis. However, the ollowing sections explain how the Gibbs sampling technique can acilitate the analysis. The ARMAX (p,q,b) model (3) is invertible i the roots o the polynomials γ(l), ξ(l) and η(l) lie outside the unit circle. For more details about the [7] properties o ARMAX (p,q b) models. Posterior Analysis Likelihood Function: Suppose y = (y,y,,y T) is a realization o the autoregressive moving average model with exogenous inputs model (). Assuming that (ε t) =N(,σ ) and employing a straightorward random variable transormation rom ε t to y t, the likelihood unction L (γ, ζ, η, σ y) is given by: or () (4) The likelihood unction (4) is a complicated unction in γ, ζ,η () and σ. Suppose the errors are estimated recursively as: where M?(t-i)= (y(t-i ): ε(t-i): d(t-i)) T k, (k=(p+q+d)), a T row vector o unknown coeicients, β' = (γ i: ζ i: η i) and the error terms ε t are generally assumed to be independent random variables (i.i.d.) sampled rom a normal distribution with zero mean ε t ~ N(,σ ) where σ is the variance. These assumptions may be weakened but doing so will change the properties o the model and η,.., η b are the parameters o the exogenous input d t. The model order (p,q,b) are assumed known and ater observing the data y t, t=,,..,t, the parameters β and the variance σ are to be estimated. Moreover, ARMAX (p, q, b) model can be written as: where, and sensible estimates and e -p-q-b = = e - = e. Several estimation methods, such as the Innovations Substitution (IS) method proposed by Koreisha and [] Pukkila, give consistent estimates or γ, ζ, η and σ.. The idea o the IS method is to it a long autoregressive model to the series and obtain the residuals. Then appropriate lagged residuals are substituted into the ARMAX model (). Lastly, the parameters are estimated using ordinary least squares. Substituting the residuals (5) in (4) results in an approximate likelihood (3) unction: where L denotes the lag-operator, i.e. or all, (6) 886

3 Setting the initial errors zeros, i.e. = - = = -p-q =, model (3) can be expressed in the ollowing matrix orm: the other parameters have already been deined, a ull implementation o the Bayesian approach requires the speciication o a prior or β ; Σ and Σ such that: P(ψ) = (7) P ( β, β, Σ, Σ ) = P (β,σ,σ ) P(β β,σ ), with Assuming independence, M is a T k, as it is customary and matrix with ti-th elements and β is a k vector o unknown parameters. Deine considering the inverse o the two matrices, just or convenience, we may take the joint prior distribution: ψ = (β, β, ), whereβ is any element in vector β and β is the remain elements o β,under the normality assumptions: P (β, = P (β ) P ( ) P( ), to have, or example, a normal-wishart Wishart orm; N (, Σ ), (8) P(β ) =N( µ,c ), () where Σ is the error term variance covariance - matrix o dimensions T T.Let is a known matrix P( )=W((σ ε S) ε,σ), ε (3) o dimensions k a; relating the parameter β to a - parameter vector β o dimensions a ; possibly P( )=W ((σ β S) β,σ), β (4) with a k; and Σ is the k k variance covariance matrix o β.and model the prior distribution on β as: where is a known matrix o dimensions a r; relating the regression vector β to a parameter vector µ β β, Σ ~ N ( β, Σ ), (9) o dimension r ; possibly with r a; and the hyper parameters µ; C; σ,s, σ and S are assumed to be This model is o the kind o hierarchical models [] introduced by Lindley and Smith, whose applications [3] abound in ields as dierent as educational testing, [5] [8] medicine and economics. Bayesian estimation o ψ is simple, in principle and works as ollows. Given the inormation contained in the data in the orm o a approximate likelihood unction: ε ε β β known. The notation W (Ω, ω) identiies a Wishart distribution with ω degrees o reedom and scale matrix Ω. With this structure, the joint posterior density o all the parameters can be written as: () and a joint prior distribution on the parameters, P(ψ y) = P (β, β, Σ ); the joint posterior distribution o the parameters conditional on the data is obtained, through the Bayes rule: P(ψ y) = () where denotes proportional to. Given P(ψ y) = P (β, β, Σ, Σ ), the marginal posterior distributions conditional on the data (P(β y),p(β y), P(Σ y) and P(Σ y)) can then be obtained by integrating out β ; β ; Σ and Σ rom the posterior distributions. Given that a prior or β conditional on where the irst line o the ormula corresponds to the approximate likelihood unction (eq.() and the 887

4 others represent the prior inormation. As said above, the marginal posterior densities o the parameters o interest can be obtained by integrating out the hyper parameters rom the joint posterior density. The required integration does not provide closed orm analytic solution in our case. However, a ull Bayesian implementation o the model is still easible, or instance with the help o the Gibbs sampler, an algorithm that, among other things, allows to derive marginal distributions rom the ull conditional densities [6] o the parameter vector. Predictive Analysis: In order to obtain the Bayesian orecasts or h periods, Let y = (y t+,,y t+h ) be the orecast vector. The predictive density is (6) Notice that, the predictive distribution does not have a closed analytic orm and it is also hard to calculate the integration numerically. However, the Gibbs sampling technique can be employed to compute the predictive distribution. Now, the equations or the h uture observations being expressed as: y = Л e where. L +, Gibbs sampler. Since ~ N(, Σ ) with and being independent, the conditional predictive density p is p = N (Л el, Σ ')) The Bayesian orecasts o the uture values can be obtained via extending the posterior analysis directly by stretching the parameter vector to include y. That is, the above posterior analysis will apply conditionally on the uture observations y where the elements o y are sampled rom the conditional predictive density (7). Estimation with Gibbs Sampling: Conventional Gibbs, sampling can be achieved with the help o Bayes rule. let ψ= (β, β, Σ, Σ ) be our parameter vector. I the prior distribution P(ψ) is chosen to be conjugate to the likelihood then the posterior will be a known distribution. I the parameter vector ψ = (β, β, Σ, Σ ) then the Gibbs sampler updates a component by 3 component. The proposed Gibbs sampling algorithm or the ARMAX (p, q, b) model can be implemented as ollows:.speciy starting values and and set j =. A set o initial estimates o the model parameters can be obtained using the IS technique o [] Koreisha and Pukkila..Calculate the residuals recursively using (5) and the IS parameter estimates. 3.Draw the parameter vector ψ as ollows: JI= And F = with e = (e, e,, e ) and =(,, ). ' ' L t t- t-k+ t+h t+h It should be noted that writing the uture values in this way acilities writing the predictive distribution or any number o uture values conational on the model parameters and initial errors. This enables the generation o any number o uture realizations the autoregressive moving average model with exogenous inputs model that is needed or the 4. Set j= j+ and repeat step 3 several time. This algorithm gives the next value o the Markov chain simulating each o the ull conditionals where the conditioning elements are revised during the cycle. This iterative process is repeated or a large number o iterations and convergence is monitored. Ater the chain has converged. say at T iterations, the simulated values 888

5 as a sample rom the joint posterior. Posterior estimates o the parameters are computed direct via sample averages o the simulation outputs. The Gibbs sampler sector in Egypt.. In the empirical study, to deal with stationery series, all o the individual variables(in irst dierence)are transormed to the nature logarithms. or and is easily seen to be Table speciied presents summary statistics or the three by the ollowing ull conditional distributions, obtained rom the joint posterior density above: (8) individual series. As Table shows, the summary statistics suggest that the three individual series that has, a small mean values which is dominated by a larger standard deviation, evidence o non-normality through (positive) skewness and excess kurtosis. Moreover, the results o the ADF test show that, the three individual (9) series in irst dierences o logarithms are trend stationary at 5% signiicance level. In conclusion, all the three individual series have a single unit root or are integrated o degree one, I(). () Monte Carlo Experiments: In this subsection, we present a simulated example to evaluate the eiciency o the proposed methodology. The example deals with generating 3 observations rom the ollowing ARMAX (,, ) model by Monte Carlo experiments. By using simulated data that are obtained rom the ollowing data generating process: () Where and where, the error variance Σ was chosen to be to model the data (and some and a non inormative prior was assumed or γ,ξ,η and parameters) by normal distributions, the ramework is Σ via setting lexible enough to accommodate other distributional with zero mean and variance Σ was used or the orms. Markov Chains Monte Carlo methods can then initial error. The starting values or the parameters be easily used to check the sensitivity o inerences to (γ,ξ,η and Σ ) were obtained using IS method. The the speciic assumptions made. starting values or and y was assumed to be zero. The Gibbs sampler was run with iterations and every tenth value in the sequences o the draws is recorded, to obtain an approximately independent samples. All posterior and predictive estimates were computed as sample averages o the simulated outputs. Table presents the true values and Data and Empirical Evidences: In order to conduct the analysis, the proposed Bayesian methodology o the autoregressive moving average model with exogenous inputs (ARMAX) can be evaluated based on a simulated example, moreover, we demonstrate how the proposed Bayesian analysis can be used to analyze the electricity consumption data (y t) in the industrial sector in Egypt. The empirical results suggest that, the electricity consumption variable is related with the consumed quantities o electricity in a previous period (y t-i) and the number o consumers (dt-i). The data set used in this paper consists o monthly time series observations 4 covers the period (99: 4:9) or estimation purpose. The proposed Bayesian analysis can be used to analyze the electricity consumption data (y t), the consumed quantities o electricity in a previous period (y ) and the number o consumers(d ) in the industrial t-i t-i Bayesian estimates o the parameters (γ,ξ,η and Σ ) and the next ive uture values (y 3, y 3, y 33, y 34, y 35). Moreover, a 95% conidence interval using.5 and o.975 percentiles o the simulated draws is constructed or every estimates and uture values. Table shows that, Bayesian estimates are closed to the true values o the parameters except the last two uture values (y 34, y 35) which are dominated by a larger standard deviations. In other words, these indings concluded that the proposed Bayesian methodology o the autoregressive moving average model with exogenous inputs (ARMAX) is stable and luctuates in the neighborhood o the true values o the parameters.. A normal p 889

6 Table : Descriptive statistics Measures / variables The electricity consumption (y t) The electricity in a previous period (y t-i) The number o consumers (d t-i) Mean -.(.45) -.(.54) -.67(.35) Median -.7(.53) -.7(.365) -.9(.34) skewness.(.474).54(.5).46(.4) kurtosis -.34 (.63) -. 35(.4) -.44 (.9) ADF Notes: The null hypothesis in the augmented Dickey Fuller (ADF) test is that there exists a unit root in the time series, that is, the time series is non-stationary. The null hypothesis is rejected i the ADF statistic is greater than the Mackinnon critical values. The ADF test lag length was determined by AIC and BIC criteria. Test statistic signiicant in rejecting the null hypothesis at the 5% level. The numbers in parentheses are the standard deviations. Table : Bayesian results o the simulated examples Parameters True values Mean Std Dev. Lower (95%) Limit Median Upper (95%)Limit γ E ξ η E Σ y y y y y Table 3: Autocorrelations results o the simulated examples Parameters Lag Lag 5 Lag Lag Lag 3 Lag 5 γ ξ η Σ y y y y y The convergence o the proposed Bayesian methodology is illustrated based on the autocorrelations or the simulated example are displayed in Table 3. Table 3 shows that the draws or each parameter (γ,ξ,η and Σ ) or ive uture values ( y 3, y 3, y 33, y 34, y 35) had small autocorrelations at lags, 5,, and 3, which means that, the convergence o the proposed Bayesian methodology was achieved. This conclusion was conirmed at lag 5. The Electricity Consumption Data: The electricity consumption data (y t) in the industrial sector in Egypt consist o 65 monthly observations. As noted above, all o the individual variables (in irst dierence) are transormed to the nature logarithms. In this subsection, we demonstrate how the proposed Bayesian analysis can be used to analyze the electricity consumption data in the industrial sector in Egypt. Using the model ARMAX (,,) in eq.(), 89

7 Table 4: Bayesian results or the dierenced electricity consumption data using ARMAX (,,) Parameters True values Mean Std Dev. Lower (95%)Limit Median Upper (95%)Limit γ ξ η Σ Z Z Z Z Z the proposed Bayesian methodology is applied to the irst 6 values o the dierenced electricity consumption series Zt and the last ive values are set aside or orecasting evaluation. Then all initial values o the parameters (γ,ξ,η and Σ ) and the next ive uture values (Z 6, Z 6, Z 63, Z 64, Z 65) are chosen as in the simulated examples. Table 4 summarizes the posterior and predictive results or the dierenced electricity consumption data in the industrial sector in Egypt series. Table 4 shows that,all o the true values o the parameters or the uture values are within the 95% conidence interval ormed by the.5 and.975 percentiles. However, the inal conclusion conirmed that the proposed algorithm has good statistical properties. It should be noted that uncertainty about the parameters and initial errors were incorporated. However, there exist some non Bayesian papers that are concerned with the prediction error arising rom uncertainly about parameter estimation, such as [5] Yamamoto. Conclusion: In this paper, we developed a simple and ast way o estimating the parameters o the autoregressive moving average with exogenous inputs (ARMAX) model using the output o the Gibbs sampling. The procedure has the advantage that can be used under inormative and non-inormative priors on the parameters o interest. Moreover, this procedure does not require the estimation o two models, one with and the other without the restriction to be tested i the problem o testing a set o restrictions in the autoregressive moving average model with exogenous inputs (ARMAX) model is considered. The empirical results o the simulated examples and electricity consumption data in the industrial sector in Egypt showed the accuracy o the proposed methodology and has a good has good statistical properties. An extensive check o convergence o the parameters and the uture values using autocorrelations at lags, 5,, and 3, which means that, the convergence o the proposed Bayesian methodology was achieved.this conclusion was conirmed at lag 5. Future work may investigate the problem o testing a set o linear restrictions in the autoregressive moving average model with exogenous inputs (ARMAX). REFERENCES. Altinay, G. and E. Karagol, 4. Structural break, unit root and the causality between energy consumption and GDP in Turkey, Energy Economics, 6: Box, G.E.P. and G.M. Jenkins, 976. Time series analysis. Forecasting and Control. Holden-Day, San Francisco. 3. Chib, S. and E. Greenberg, 994. Bayes inerence in regression models with ARMA (p, q). J. Econom., 64: Ciccarelli, M., 4. Testing restrictions in hierarchical normal data models using Gibbs sampling. Res. Econom., 58: DuMouchel, W.H. and J.E. Harris, 983. Bayes methods or combining the results o cancer studies in humans and other species: with discussion, J. Am. Statistical Assoc., 78: Geland, A.E. and A.F.M. Smith, 99. Samplingbased approaches to calculating marginal densities. J. Am. Statistical Assoc., 85: nd 7. Harvey, A., 994. Time Series Models, Ed. Cambridge, MA: MIT Press. 8. Hsiao, C., M.H. Pesaran and A.K. Tahmiscioglu, 999. Bayes estimation o short run coeicients in dynamic panel data models. In: Hsiao, C., et al. 89

8 (Eds.), Analysis o Panels and Limited Dependent Variable Models : In Honor o G.S. Maddala. Cambridge University Press, Cambridge. 9. Ismail, M.A., 3. Bayesian Analysis o the Seasonal Moving Average Model: A Gibbs Sampling Approach. Japanese J. Appl. Statistics, 3: Koreisha and Pukkila, 99. Linear methods or estimating ARMA and regression models with serial correlation, Communications in Statistics Simulation, 9: 7-.. Lindley, D.V. and A.F.M. Smith, 97. Bayes estimates or the linear model: with discussion, J. Royal Statistical Soc., Series B34: -4.. Marriott, J., N. Ravishanker, A. Geland and J. Pai, 996. Bayesian analysis o ARMA process: Complete sampling based inerence under ull likelihoods. In Bayesian Statistics and Econometrics: Essays in honor o Arnold Zellner, D., Berry, K., Chaloner, and J., Geweke, (eds). New York, Wiley. 3. Rubin, D.B., 98. Estimation in parallel randomized experiments, J. Educational Statistics, 6: Walker, D.M., F.J.P. Brarberia and G. Marion, 6. Stochastic modeling o ecological processes using hybrid Gibbs samplers. Ecological Modeling, 98: Yamamoto, T., 98. Prediction o multivariate autoregressive moving average models. Biometrika, 68: Altinay et al. (4). Ciccarelli, (4). 3 Walker et al.,(6). 4 Source: Egyptian Ministry o Electricity & Energy, Various Issues. 89