Stylianos Sp. Pappas* Vassilios C. Moussas. Sokratis K. Katsikas

Transcription

1 242 Int. J. Modelling, Identification and Control, Vol. 4, No. 3, 2008 Alication of the multi-model artitioning theory for simultaneous order and arameter estimation of multivariate ARMA models Stylianos S. Paas* Deartment of Information and Communication Systems Engineering, University of the Aegean, 83200, Karlovassi, Samos, Greece *Corresonding author Vassilios C. Moussas School of Technological Alications (STEF), Technological Educational Institution (TEI) of Athens, Egaleo, Greece Sokratis K. Katsikas Det. of Technology Education & Digital Systems, University of Piraeus, 1 Androutsou St., Piraeus GR-18532, Greece ska@unii.gr Abstract: In this aer, a study on how to erform simultaneous order and arameter estimation of multivariate (MV) ARMA (autoregressive moving average) models under the resence of noise is addressed. The roosed method, which is comutationally efficient, is an extension of a reviously resented method for MV AR models and is based on the well established and widely alied multi-model artitioning theory. A series of comuter simulations indicate that the method is infallible in selecting the correct model order in very few stes. The simultaneous estimation of the multivariate ARMA arameters is also another benefit of the roosed method. The results are comared with two other established order selection criteria namely Akaike s Information Crieterion (AIC) and Schwarz s Bayesian Information Criterion (BIC). Finally, it is shown that the method is also successful in tracking model order changes, in real time. Keywords: order determination; arameter estimation; multivariate; autoregressive moving average; ARMA; multi-model; artitioning; Kalman. Reference to this aer should be made as follows: Paas, S.S., Moussas, V.C. and Katsikas, S.K. (2008) Alication of the multi-model artitioning theory for simultaneous order and arameter estimation of multivariate ARMA models, Int. J. Modelling, Identification and Control, Vol. 4, No. 3, Biograhical notes: S.S. Paas received BEng (Hons) in Electrical and Electronic Engineering (1997) and MSc in Advanced Control (1998) from University of Manchester Institute of Science and Technology (UMIST). He is currently a PhD Student at the Deartment of Information and Communication Systems Engineering, University of the Aegean, Karlovassi, Samos, Greece. His research interest is in the area of artitioning theory and their alications. He is an IEEE Member. V.C. Moussas received his PhD in Comuter Engineering and Informatics from University of Patras, Greece in He is currently Assistant Professor at the Deartment of Civil Works Technology School of Technological Alications, Technological Educational Institution of Athens, Greece. His current research interests include material testing, fatigue and lifetime rediction, sensor arrays, multi-model algorithms and arallel networked alications. S.K. Katsikas received his Diloma in Electrical Engineering and his PhD in Comuter Engineering and Informatics from the University of Patras, Greece in 1982 and 1987 resectively and the Master of Science in Electrical and Comuter Engineering degree from the University of Massachusetts at Amherst, USA, in His research interests lie in the areas of information and communication systems security and of estimation theory and its alications where he has authored or co-authored more than 1 ournal ublications, book chaters and conference roceedings ublications. He is serving on the editorial board of several scientific ournals. He has authored/edited 20 books and has served on/chaired the technical rogramme committee of numerous international conferences. Coyright 2008 Inderscience Enterrises Ltd.

2 Alication of the multi-model artitioning theory Introduction The roblem of fitting a multivariate (MV) ARMA model to a given time series arises in a large variety of alications, such as seech analysis (Chen and Bilmes, 2007), biomedical alications (Sheng et al., 2001), hydrology (Kourosh et al., 2006), electric ower systems (Derk et al., 2007; Swider and Weber, 2007), simulating earthuake ground motions (Mobarakeh et al., 2002), effective multi-channel identification of structures under unobservable excitation (Paakos and Fassois, 2003) and many more. The aim of this aer is not to add yet another ARMA model selection criterion to the rich literature in this area. Rather, we focus on an extension to the model order selection criterion roosed for MV AR models by Paas et al. (2006). The method is based on the well known adative multi-model artitioning theory (Lainiotis, 1976a, 1976b, 1971). It is not restricted to the Gaussian case, it is alicable to on line/adative oeration and it is comutationally efficient. Furthermore, it identifies the correct model order and arameters very fast. An m-variate ARMA model of order (, ) [ARMA (, )] for a stationary time series of vectors y observed at eually saced instants k = 1, 2,, n is defined as: T k = i k i + k + k, k k i= 1 = 1 R y Α y B v v E[v v ] = (1.1) where the m-dimensional vector v k is uncorrelated random noise, not necessarily Gaussian, with zero mean and covariance matrix R, θ = (, ) is the order of the redictor and A 1,,A, B 1,,B are the m m coefficient matrices of the MV ARMA model. It is obvious that the roblem is two fold. The first task is the successful determination of the redictor s order θ = (, ). Once the model order selection task is comleted, one roceeds with the second task, i.e., the comutation of the redictor s matrix coefficients {A i, B }. Determining the order of the ARMA rocess is usually the most delicate and crucial art of the roblem. Over the ast years, substantial literature has been roduced for this roblem and various different information theoretic criteria, such as Akaike s (1969, 1973, 1974), Rissanen s (1978, 1986), Schwarz s (1978), Wax s (1988) have been roosed to imlement the order selection rocess. Akaike s and Schwarz s classical order selection rocedures are based on the minimisation of an obective function of the form 2 Akaike (AIC): AIC (m) = n ( ˆem, ) 2 ( ˆ, ) log σ + 2m (2.12) Schwarz (BIC): BIC (m) = n log σ em + mln( n) (2.13) where n is the samle size, m is the number of arameters and σ is the estimated residual variance. 2 ˆ e,m The criteria may be minimised over choices of m to form a trade off between the fit of the model (which lowers the sum of suared residuals) and the model s comlexity, which is measured by m. Increasing the number of free arameters to be estimated imroves the goodness of fit, regardless of the number of free arameters in the data generating rocess. Hence, AIC and BIC not only reward goodness of fit, but also include a enalty that is an increasing function of the number of estimated arameters. This enalty discourages over fitting. The referred model is the one with the lowest AIC or BIC value. The AIC and BIC methodology attemts to find the model that best exlains the data with a minimum of free arameters. The above mentioned criteria are not otimal and are also known to suffer from deficiencies; for examle, Akaike s (1969) information criterion suffers from over fit (Lutkeohl, 1985). Also, their erformance deends on the assumtion that the data are Gaussian and uon asymtotic results. In addition to this, their alicability is ustified only for large samles, furthermore, they are two ass methods, so they cannot be used in an on line or adative fashion. In addition to the revious criteria, one can mention grahic methods for ARMA order identification based on the autocorrelation and artial autocorrelation functions such as ACF and PACF (Box et al., 1994). The aer is organised as follows. In Section 2, the MV ARMA model order selection roblem is reformulated so that it can be fitted into the state sace under uncertainty estimation roblem framework. In the same section, the multi-model artitioning filter (MMPF) is briefly described and its alication to the secific roblem is discussed. In Section 3, simulation examles are resented which demonstrate the erformance of our method in comarison to reviously reorted ones. Finally, Section 4 summarises the conclusions. 2 Problem reformulation In Katsikas et al. (1990), the roblem of simultaneously identifying the order and estimating the arameters of a univariate AR model was reformulated in a state-sace form. This work was then extended to univariate ARMA models in Likothanassis et al. (1997), multivariate ARX models in Demiris et al. (1998) and to multivariate AR models in Paas et al. (2006). Herein, the aroach is extended to cover multivariate ARMA models. If we assume that the model order fitting the data is known and is eual to θ = (, ), we can rewrite euation (1.1) in standard state-sace form as: x( k+ 1) = x( k) (2.1) y( k) = H( k)x( k) + v( k) (2.2) where x(k) is an m 2 ( + ) 1 vector made u from the coefficients of the matrices {A 1,..., Α, B 1,..., B }, and H(k) is an m m 2 ( + ) observation history matrix of the rocess {y(k)} u to time k ( + ). Assuming that the general form of the matrix A is:

3 244 S.S. Paas et al. a11... a 1m b11... b 1m and B is: am1 a mm bm1 b mm then x( k) [ α α α α α α α α m m2 mm mm b b b b b b b b Τ m m2 mm mm H( k) [ y ( k -1) I y ( k -1) I y ( k ) I y ( k - ) I 1 m 1 v ( k -1) I v ( k -1) I v ( k ) I v ( k - ) I ] 1 m 1 where I is the m m identity matrix and θ = (, ), is the model order. If the system model and its statistics were comletely known, the Kalman filter (KF) in its various forms would be the otimal estimator in the minimum variance sense. Remarks 1 In the case where the rediction coefficients are subect to random erturbations, (2.1) becomes x( k + 1) = x( k) + w( k) w( k) [ w11 w21 wm1 w12 w22 wm2 wmm wmm w w w w w w w w ] Τ m m2 mm mm m m ] (2.3) (2.4) v(k), w(k) are indeendent, zero-mean, white rocesses, not necessarily Gaussian. 2 A comlete system descrition reuires the value assignments of the variances of the random rocesses w(k) and v(k). We adot the usual assumtion that w(k) and v(k) are at least wide sense stationary rocesses, hence their variances, Q and R resectively are time invariant. Obtaining these values is not always trivial. If Q and R are not known they can be estimated by using a method such as the one described by Sage and Husa (1969). In the case of coefficients constant in time, or slowly varying, Q is assumed to be zero (ust like in euation (2.1)). 3 It is also necessary to assume an a riori mean and variance for each {A i, B i }. The a riori mean of the A i (0) s and B i (0) s can be set to zero if no knowledge about their values is available before any measurements are taken (the most likely case). On the other hand the usual choice of the initial variance of the A i s and B i s, denoted by P 0 is P 0 = ni, where n is a large integer. Let us now consider the case where the system model is not comletely known. The adative MMPF is one of the most widely used aroaches for similar roblems. This aroach was introduced by Lainiotis (1971, 1976a, 1976b) and summarises the arametric model uncertainty into an unknown, finite dimensional arameter vector whose values are assumed to lie within a known set of finite cardinality. In our roblem, assume that the model uncertainty is the lack of knowledge of the model order θ. Let us further assume that the model order θ lies within a known samle sace of finite cardinality, i.e. that 1 θ M, θ I, where I denotes the set of integers. The MMPF oerates on the following discrete-time model: x( k+ 1) = F( k+ 1,k/θ)x( k) + w( k) (2.5) y( k) = H( k/ θ) x( k) + v( k) M x( ˆ k / k) = x( ˆ k / k; θ ) ( θ / k) = 1 (2.6) where θ is the unknown arameter the model order in this case-f is the state transition matrix and w(k) is indeendent, zero mean, white noise not necessarily Gaussian with covariance Q which is usually set to a small ositive non zero constant. The otimal MMSE (minimum mean suare error) estimate of x(k) is given by (2.7) A finite set of models is designed, each matching one value of the arameter vector. If the rior robabilities (θ / k) for each model are already known, these are assigned to each model. In the absence of any rior knowledge, these are set to (θ / k) = 1/M, where M is the cardinality of the model set. A bank of conventional elemental filters (non-adative, e.g., Kalman) is then alied, one for each model, which can be run in arallel. At each iteration, the MMPF selects the model which corresonds to the maximum osterior robability as the correct one. This robability tends to one, while the others tend to zero. The overall otimal estimate can be taken either to be the individual estimate of the elemental filter exhibiting the maximum osterior robability (MAP) (Lainiotis et al., 1988), which is the case used in this aer, or the weighted average of the estimates roduced by the elemental filters, as described in euation (2.7). The weights are determined by the osterior robability that each model in the model set is in fact the true model. The robabilities are calculated online in a recursive manner as it is shown by euations (2.8) and (2.9). (θ /k)= L(k/k;θ ) M L(k/k;θ )(θ /k-1) =1 1 2 (θ /k-1) (2.8) Lk/k;θ ( ) = P y ( k/k-1;θ) (2.9) 1 T -1 ex[ y ( k/k-1;θ ) P y ( k/k-1;θ)y( k/k-1;θ)] 2 where the innovation rocess y ( k/k-1; θ ) = y( k) H( k; θ )x( ˆ k/k-1; θ ) is a zero mean white rocess with covariance matrix T y k/k 1; θ k; θ k/k; θ k; θ (2.10) P( ) = H( )P( )H ( ) + R(2.11) For euations (2.8) (2.11) = 1,2,, M.

4 Alication of the multi-model artitioning theory 245 Figure 1 MMPF block diagram Figure 2 deicts the osterior robabilities associated with each value of θ lotted against normalised time intervals. Figure 3 shows the criteria comarison for two data sets, one relatively small ( samles) and one larger ( samles); and Table 1 shows the estimated ARMA arameter coefficients. From Figure 2, it is obvious that the MMPF identifies the correct robability θ = (1, 1) = 2 very fast, in ust 17 stes. Convergence is taken to occur when the osterior robability of the model exceeds 0.9. Figure 3 Examle 1: criteria comarison number of correct model order identifications out of Monte Carlo Runs (see online version for colours) 3 Examles In order to assess the erformance of our method, several simulation exeriments were conducted. All of these exeriments were conducted for Monte Carlo Runs. For details for the alication of stochastic Monte Carlo techniues, see Shiryaev (1996) or Christakis (1998). The models used were those of (2.1) and (2.2), with cardinality M = 10. Examle 1: ARMA (1, 1). θ = (1, 1) = 2 and cardinality M = A 1 = , B 1 = , R= Figure 2 Examle 1: osterior robabilities seuence (see online version for colours) Monte Carlo Runs MMPF Criteria AIC BIC 92 Samles From Figure 3, we deduce that MMPF is % successful in selecting the correct model order for both data sets, while only BIC matches its erformance for the larger data set. Also, Table 1 shows that the arameter coefficient estimation is very accurate to the considered amount of noise. (RMSE root mean suare error is very small). Table 1 Examle 1: estimated ARMA coefficient arameters Estimated arameters RMS error Examle 2: ARMA (1, 1). θ = (1, 1) = 2. This is a more comlex MV ARMA since m = A 1 = , B 1 = R= diag [(0.42, 0.01, 0.16)]

5 246 S.S. Paas et al. Figure 4 deicts the osterior robabilities associated with each value of θ lotted against normalised time intervals. Figure 5 shows the criteria comarison for two data sets, one relatively small ( samles) and one larger ( samles) and Table 2 shows the estimated ARMA arameter coefficients. Figure 4 Examle 2: osterior robabilities seuence (see online version for colours) MMPF identifies the correct robability θ = (1, 1) = 2 very fast, in ust 18 stes (Figure 4). Convergence is taken to occur when the osterior robability of the model exceeds 0.9. Moreover, only MMPF is % successful in selecting the correct model order for both data sets (Figure 5). Examle 3: ARMA (2, 2). θ = (2, 2) = A 1 = , A 2 = , B 1 = , B 2 = , Figure Monte Carlo Runs Examle 2: criteria comarison number of correct model order identifications out of Monte Carlo Runs (see online version for colours) MMPF Criteria AIC BIC Samles R= Figure 6 deicts the osterior robabilities associated with each value of θ lotted against normalised time intervals. Figure 7 shows the criteria comarison for two data sets, one relatively small ( samles) and one larger ( samles) and Table 3 shows the estimated ARMA arameter coefficients. From Figure 6, it is obvious that the MMPF identifies the correct robability θ = (2, 2) = 4 very fast, in ust 24 stes. Convergence is taken to occur when the osterior robability of the model exceeds 0.9. Figure 6 Examle 3: osterior robabilities seuence (see online version for colours) Table 2 Examle 2: estimated ARMA coefficient arameters Estimated arameters RMS error From Figure 7, we deduce that MMPF is % successful in selecting the correct model order for both data sets, only BIC matches its erformance for the larger data set. As Table 3 clearly shows, the arameter estimation is again accurate since the RMSE is very small.

6 Alication of the multi-model artitioning theory 247 Figure Monte Carlo Runs Examle 3: criteria comarison number of correct model order identifications out of Monte Carlo Runs (see online version for colours) MMPF Criteria AIC BIC 92 Samles Examle 4: The aim of this examle is to show that the roosed method is able to track changes in the model structure in real time, and thus, the method should handle successfully the roblem of the time varying model order. At each iteration, the roosed algorithm selects the model that corresonds to the maximum a osteriori robability as the correct one. This robability tends (asymtotically) to one, while the remaining robabilities tend to zero (Figure 8). Thus, the algorithm is adative in the sense of being able to track model order changes in real time. A noisy set of data is used, which is generated as shown in Table 4. Table 3 Examle 3: es timated ARMA coefficients Estimated arameters RMS error Table 4 Time varying MV A RMA order seuence Stes Model order Stes Model order Figure 8 Examle 4 osterior robabilities seuence (see online version for colours)

7 248 S.S. Paas et al. The MV ARMA models used in order to create the data set are the ones used in Examles 1 and 3 lus a new MV ARMA (2,1), (i.e. θ = 3) with coefficients: A 1 =, A 2 =, B 1 = and The noise covariance matrix R and the cardinality M for all three models are R = and M = 10 resectively. Monte Carlo Runs were used; as Figure 8 shows, the MMPF is caable of identifying the correct model order (the corresonding osterior robability exceeds 0.9) for every time interval. When a model order change occurs, the MMPF algorithm needs M stes (in our case, M = 10) to erform variable initialisation. This is the reason for having a 10-ste ga before the MMPF starts to increase the osterior robability corresonding to the true model. This initialisation is erformed automatically, without user intervention. 4 Conclusions A new method for simultaneously selecting the order θ (, ) and for estimating the arameters of a MV ARMA model has been develoed, as an extension of the method roosed for the MV AR case (Paas et al., 2006). The roosed method successfully selects the correct model order in very few stes and identifies very accurately the ARMA arameters. Comarison with other established order selection criteria (AIC and BIC) shows that the method needs the shortest data set for successful order identification and accurate arameter estimation for all the simulated models, whereas the other criteria reuire longer data sets as the model order increases. The method erforms eually well when the comlexity of the MV ARMA model is increased. Finally, the method is caable of tracking, in real time, any model order changes. As a further ste to this research is left the task to otimise the algorithm in order to be able to identify the order of the AR comonent () and MA comonent () searately and to aly the algorithm on financial time series analysis (Liatsis and Hussain, 2001). References Akaike, H. (1969) Fitting autoregressive models for rediction, Ann. Inst. Stat. Math., Vol. 21, Akaike, H. (1973) Information theory and an extension of the maximum likelihood rincile, in B.N. Petrov and F. Csàki (Eds.): Proceedings of the Second International Symosium on Information Theory, Akadémia Kiadó, Budaest, Akaike, H. (1974) A new look at the statistical model identification, IEEE Trans. on Automatic Control, Vol. AC-19, Box, G.E.P, Jenkins, G.M. and Reinsel, G.C. (1994) Time Series Analysis: Forecasting and Control, ISBN: , 3rd ed., Prentice Hall, New Jersey. Chen, C.P. and Bilmes, J.A. (2007) MVA rocessing of seech features, audio, seech and language rocessing, IEEE Transactions on Audio Seech and Language Processing, Vol. 15, No. 1, Christakis, N. (1998) Modelling of the microhysical rocesses that lead to warm rain formation, PhD Thesis, UMIST Manchester. Demiris, E.N., Likothanassis, S.D. and Katsikas, S.K. (1998) Multivariate ARX model identification and control with unknown rocess order, in Proceedings, 2nd IMACS International Conference on Circuits, Systems and Comuters, Piraeus, Greece, Vol. 2, Derk, J., Weber, S. and Weber, C. (2007) Extended ARMA models for estimating rice develoments on day-ahead electricity markets, Electric Power Systems Research, Vol. 77, Nos. 5 6, Katsikas, S.K., Likothanassis, S.D. and Lainiotis, D.G. (1990) AR model identification with unknown rocess order, IEEE Transactions on Acoustics, Seech, and Signal Processing, Vol. ASSP-38, No. 5, Kourosh, M., Eslami, H.R. and Kahawita, R. (2006) Parameter estimation of an ARMA model for river flow forecasting using goal rogramming, Journal of Hydrology, Vol. 331, Nos. 1 2, Lainiotis, D.G. (1971) Otimal adative estimation: structure and arameter adatation, IEEE Trans. Automat. Contr., Vol. AC 16, Lainiotis, D.G. (1976a) Partitioning: a unifying framework for adative systems I: estimation, Proc. IEEE, Vol. 64, Lainiotis, D.G. (1976b) Partitioning: a unifying framework for adative systems II: control, Proc. IEEE, Vol. 64, Lainiotis, D.G., Katsikas, S.K. and Likothanassis, S.D. (1988) Adative deconvolution of seismic signals: Performance, comutational analysis arallelism, IEEE Trans Acoust Seech Signal Processing, Vol. 36, No. 11, Liatsis, P. and Hussain, A.J. (2001) Financial time series rediction using second-order ieline recurrent neural networks, Proc. IWSSIP 01, Bucharest, Romania, June 7 9, Likothanassis, S.D., Demiris, E.N. and Karelis, D.G. (1997) A real-time ARMA model structure identification method, Proceedings of 13th International Conference on Digital Signal Processing, Vol. 2, Lutkeohl, H. (1985) Comarison of criteria for estimating the order of a vector AR rocess, J. Time Ser. Anal. 6, Vol. 6, Mobarakeh, A.A., Rofooei, F.R. and Ahmadi, G. (2002) Simulation of earthuake record using time varying ARMA (2,1) model, Probabilistic Engineering Mechanics, Vol. 17,

8 Alication of the multi-model artitioning theory 249 Paakos, V. and Fassois, D.S. (2003) Multichannel identification of aircraft skeleton structures under unobservable excitation: a vector AR/ARMA framework, Mechanical Systems and Signal Processing, Vol. 17, No. 6, Paas, S.S., Leros, A.K. and Katsikas, S.K. (2006) Joint order and arameter estimation of multivariate autoregressive models using multi-model artitioning theory, Digital Signal Processing, Vol. 16, No. 6, Rissanen, J. (1978) Modeling by shortest data descrition, Automatica, Vol. 14, Rissanen, J. (1986) A redictive least suares rincile, IMA J. Math. Contr. Inform., Vol. 3, Sage, A. and Husa, G. (1969) Algorithms for seuential adative estimation of rior statistics, Proc. Joint. Automatic Control Conf., Boulder, Colorado, Schwarz, G. (1978) Estimation of the dimension of the model, Ann. Statist., Vol. 6, Sheng, L., Ki, H.J. and Chon, K.H. (2001) A new algorithm for linear and nonlinear ARMA model arameter estimation using affine geometry [and alication to blood flow/ressure data], Biomedical Engineering, IEEE Transactions on Biomedical Engineering, Vol. 48, No. 10, Shiryaev, A.N. (1996) Probability, Sringer-Verlag, New York Inc. Swider, D.J. and Weber C. (2007) Extended ARMA models for estimating rice develoments on day-ahead electricity markets, Electric Power Systems Research, Vol. 77, Nos. 5 6, Wax, M. (1988) Order selection for AR models by redictive least suares, IEEE Trans. Acoust., Seech, Signal Proc, Vol. 36,