The Recursive Fitting of Multivariate. Complex Subset ARX Models

Size: px

Start display at page:

Download "The Recursive Fitting of Multivariate. Complex Subset ARX Models"

Mae Doyle
6 years ago
Views:

1 lied Mathematical Sciences, Vol. 1, 2007, no. 23, The Recursive Fitting of Multivariate Comlex Subset RX Models Jack Penm School of Finance and lied Statistics NU College of Business & conomics The ustralian National University, CT, ustralia, 0200 bstract In this aer we roose an innovative recursive learning algorithm to seuentially estimate multivariate comlex subset autoregressive models with exogenous variables (VRX models), including full-order models. This aer suggests the use of the recursive fitting of multivariate comlex subset RX models in conjunction with order selection criteria to select an 'otimal' multivariate comlex subset RX model. The recursive rocedure can be embedded in a tree algorithm. We fit the necessary models associated with the bottom stage, and then recursively fit models which include more variables, until finally we fit recursively the full order RX model with maximum lags P and Q. Keywords: Recursion; Learning lgorithm; Multivariate Comlex Subset utoregressive modelling with xogenous Variables.

2 1130 J. Penm 1. INTRODUCTION Mathematical model researchers for alied time-series systems are often concerned that the coefficients of their established models may not be constant over time, but vary when the models are disturbed by changes arising from outside factors. This concern has motivated researchers to develo seuential estimation algorithms that allow users to udate subset time series models at consecutive time instants. This aroach will allow for the coefficients to slowly evolve, and can show evolutionary changes detected in model structures. Hannan and Deistler [1] roosed a recursive estimation of an autoregressive (R) model. Chen et al [2] suggested recursive udating rocedure for the learning rocess of a multi-layer neural network. The aim of this aer is to rovide an algorithm for the recursive fitting of multivariate comlex subset RX models, including full-order models. The algorithm is develoed for the selection of an otimal comlex subset RX model by emloying model selection criteria. The ultimate goal of this research is to investigate an efficient rocedure for selecting the otimal multivariate comlex subset RX model subject to ossible zero or absence entries in each existing coefficient matrix. It is unwise to neglect ossible zero constraints on the comlex coefficient matrices of the otimal comlex subset RX model selected, whether these constraints reresent the absence of a full comlex matrix or erhas simly a art of a comlex one. n algorithm for the recursive fitting of vector real subset RX models has been resented in [3]. In most of the engineering literature [4,5], systems ossessing multile inuts and multile oututs are referred to as vector systems where each inut reresents an inut channel and each outut reresents an outut channel. Mittnik [6] suggested rocedures for estimating an internally balanced state sace reresentation of VRX models. This work enriches model building techniues by using model reduction concets. In time series modelling, subset models [7,8] are often emloyed, esecially when the data exhibits some form of eriodic behaviour, erhas with a range of different natural eriods in terms of hours, days, months, and years, in alications involving weather,

3 Multivariate Comlex Subset RX Models 1131 humidity, and temerature recordings, and short-term and long-term electricity load data. Thus many researchers have drawn attention to vector subset time series system analysis. When the imulse-resonse matrix needs only an imaginary art, i.e. by imosing the constraint that the real art is null, the model becomes a vector imaginary subset model. When the imulse-resonse matrix is only real, the model will be a vector real subset model. In this aer, we roose rocedures for selecting the otimal multivariate comlex subset RX model by using the roosed recursions in conjunction with model selection criteria tailored to conform to multivariate comlex RX schemes. s stated in [3], the roosed recursions rovide a comutational rocedure which can be conventionally embedded in an inverse tree algorithm. The structure of the tree algorithm rovides great benefit in imlementing software on a multi-c..u. comuting machine, such as a suercomuter. Thus the roosed recursive algorithm is suerior to non-recursive algorithms. 2. TH RCURSIV STIMTION OF TH MULTIVRIT COMPLX SUBST RX MODLS Let y(t) and x(t) be jointly stationary, zero mean multivariate comlex stochastic rocesses. The dimension of y(t) is m and the dimension of x(t) is n. We consider the RX(,) model with the deleted lags i 1, i 2,..., i s for y(t) and deleted lags j 1,j 2,...,j r for x(t), so that a model is of the form i Is Bj ε ) 0 Is i Is εis ( )y(t-i) + ( )x(t-j) = (t, { ( )= I, ( )=0, as i, * B j ( J r )=0, as j εj r } (1) where I s reresents an integer set with elements i 1,i 2,...,i s, 1 i 1 i 2... i s -1, and J r reresents an integer set with elements j 1,j 2,...,j r, 0 j 1 j 2... j r -1. ε(t) is a m 1 zero mean stationary comlex disturbance rocess which is uncorrelated with any variables

4 1132 J. Penm included in (1) excet y(t). By using the orthogonality rincile [9], the estimates of arameters * i ( s) I and B * j( ) of the fitted RX (,) model are solutions of the following normal euations: * * i Is µ k-i Bj γk-j Is ( ) + ( ) = 0 k = 1,2,...,; k, * τ * i Is γi-l Bj ν l-j ( ) + ( ) = 0 l = 0,1,...,; l, and we have * * Is r i Is µ -i Bj r γ-j V (, J ) = ( ) + ( J ), and * * ( Is, ) = i ( Is) µ -i + Bj ( ) γ-j. (2) where τ denotes the conjugate transose, µ,, k ν k and γ k are the samle estimates of {y(t)y τ (t-k)}, (x(t)x τ (t-k)}, and {x(t)y τ τ τ (t-k)} resectively. µ k = µ -k and νk= ν -k. In addition, V (, ) I s is the estimate of the ower matrix and ( Is, J r) is the estimate of the cross-covariance matrix between ε(t) and y(t--1). Note that the orthogonality rincile has been adoted to estimate the coefficient matrices in (1) and y(t--1) is indeed not a variable included in (1), thus ( Is, J r) is not null [10]. Following [3], we consider an RX (-1,) model of the form -1 i s j ( I )y(t -i) + B ( )x(t - j) = ε(t ). The analogous normal euations, the associated estimated ower and cross-covariance matrices are the same as (2) with the excetion that is relaced by -1.

5 Multivariate Comlex Subset RX Models 1133 Now we need to introduce another RX(-1,) model of the form -i( Is)y(t+-i) + F-j( J r)x(t+-j) = ε(t ) { 0= I, -i( Is) = 0 as i ε Is i=1 j=0 * -j ( r ) = 0 as j ε r F J where ε(t) is a m 1 zero mean disturbance rocess. The highest subscrit for is -1, and for F is. We refer to this RX model as an F tye model. Similarly, by using the orthogonality rincile, we can obtain the analogous normal euations. fter solving the estimates of arameters -i ( I s ) and F -j ( J r ), we get J * * -1 ( Is, r) = -i -1() Is µ -i + F-j -1() r γ- j i=1 j=0, V J J and * * -1 Is -i -1 Is µ -i F-j -1 γ-j i=1 j=0 (, ) = ( ) + ( ). (3) Thus the following (-1,) to (,) recursions are available, ( I ) = ( I ) + ( I ) ( I ) i = 1,..., -1 (4) * * * * i s i -1 s s -i -1 s B ( ) = B ( ) + ( I ) F ( ) j = 0,..., (5) * * * * j j -1 s - j -1 * ( Is) = - -1 ( Is, ) / V-1 ( Is, ) (6) * V ( Is, ) = V-1 ( Is, ) + ( Is) -1 ( Is, ) (7) (, ) = (, ) τ -1 Is -1 Is. (8) In the secial case, where the consecutive coefficient matrices * -k for the lags of y(t-+k), k = 1,...,a (a -1) of the B tye RX(-1,) model are missing, the estimated coefficient matrices are null, i.e. * -k -1( s) I =0, k=1,2,...,a, and then the corresonding coefficient matrices and V from the B tye (-a-1,) model are sufficient to continue the recursive estimations. To develo the recursions for the F tye RX(,) model of the form

6 1134 J. Penm -i s -j ( I )y(t+-i) + F ( )x(t+-j+1) = ε(t ), (9) we introduce two models: a GH tye RX (-1,) model of the form -1 Gi Is Hj ε H ( )y(t -i -1) + ( )x(t - j) = (t) (10) where ε(t) H is a n 1 zero mean disturbance rocess, with τ H H H s r { ε(t ) ε (t) } = V ( I, J ) and H τ G s r { ε(t ) y (t - -1)} = δ ( I, J ) ; an F %% tye RX (,-1) model of the form % % -i Is F% -j ε ) % 0 Is % -i Is i=0 j=1 ( )y(t+-i) + ( )x(t+-j+1) = (t { ( ) = I, ( ) = 0 as * i εis F% -j( ) = 0 as j ε}, (11) where and % ε(t) is an m 1 zero mean disturbance rocess, with τ { ε(t ) % % ε(t ) } = V% ( Is, ), % τ F { ε(t ) x (t + +1)} = η% ( Is, ), % τ F { ε(t (t + - )} = % ( s, r). ) x I J Then we have the following recursive euations

7 Multivariate Comlex Subset RX Models 1135 * * -1 * * -i ( Is) = % -i ( Is) + F ( ) Gi -1( Is) i = 0,1,..., -1 * * -1 * * F-j ( ) = F% -j ( ) + F ( ) Hj -1( ) j = 1,2,..., F * F H r η% -1 Is V-1 ( J ) = - (, )/ * G Is r V -1 Is r F r η-1 Is r G F τ -1 Is η -1 ( Is, ). V (, J ) = % (, J ) + ( J ) (, J ) η (, ) = % ( Is, ). gain, if the consecutive coefficient matrices G * -k for the lags of y(t-+k-1), k = 1,2,...,b (b -1) of the GH tye RX(-1,) model are missing, this GH tye model is euivalent to a GH tye (-b-1,) model. lso note that the F tye RX (,) model of (9) is euivalent to the F %% tye RX (,-1) model of (11), i.e. as F of (9) is missing, we may substitute an F %% tye model from (11) for the F tye model of (9). Next, for the recursions to estimate the GH tye RX (,) model, we also need the information from the GH tye RX (-1,) model of the form (10) and from the F %% tye RX (,-1) model of the form (11). Thus, the following recursions are obtained: * * * * -1 Gi ( Is) = Gi -1( Is) + G ( Is) % -i ( Is) i = 0,1,..., -1 * * * * -1 Hj ( ) = Hj -1( ) + G ( Is) F% -j ( ) j = 1,2,..., G ( ) = - (, J )/ % (, J ) * G Is η-1 Is r V -1 Is r H H * F Is r -1 Is r Is -1 Is r V (, J ) = V (, J ) + G ( ) η% (, J ). To consider the recursive estimation of the F %% tye RX (,-1) model, (11), we need to introduce a CD tye RX (-1,) model of the form

8 1136 J. Penm D * C-1-i Is D+1-j ε ) D0 i=0 j=1 ( )y(t + -i) + ( )x(t + - j+1) = (t { ( ) = I, D -j+1 ( J r ) = 0, as j εj r, C -i-1 ( I s ) = 0 as i εi s }, (12) where ε(t) D is a n 1 zero mean disturbance rocess, with D D τ D s r { ε(t ) ε (t) } = V ( I, J ), and D τ C s r { ε(t ) y (t + - )} = ( I, J ). Now rewrite an B tye RX (-1,) model of (2) in the form i s j-1 J r i=0 j=1 ( I )y(t -i) + B ( )x(t - j+1) = ε(t ), and recall an F tye RX(-1,) model of the form +1 -i s +1-j i=1 j=1 ( I )y(t+-i) + F ( )x(t+-j+1) = ε(t ). We can derive the following formulae: * -1 * * -1 * * -1 * % -i ( Is) = -i -1( Is) - F% 0 ( ) C-i-1-1( Is) + % ( Is) i -1( Is) i = 1,2,..., -1 * -1 * * * F% -j ( ) = F-j+1-1( ) - F% 0 ( ) D-j+1-1( ) * * + % ( Is) Bj-1-1( J r) j = 1,2,..., -1-1 % -1( Is, ) = V-1 ( Is, ) - -1 ( Is, )[ V-1 ( Is, )] - 1 I s J r V F D -1 C + % -1( Is, )[ V-1 ( IS, JR) ] -1 ( Is, ) * -1 F D F% 0 ( ) = - % -1( Is, )/ V-1 ( Is, ) % ( ) = - (, ) / (, ) * -1 Is -1 Is V-1 Is C F τ τ -1 ( I s, J r ) = % -1 ( I s, J r ) -1 Is -1, Is (, ) = (, ). (, )

9 Multivariate Comlex Subset RX Models 1137 lso note that if both * ( Is) % and * F % of the F 0( ) %% tye RX (,-1) model of (11) are missing, this F %% tye model is euivalent to an F tye RX (-1,-1) model. The recursions for the CD tye RX (,) model of the form C-i Is D-j ε ( )y(t + -i) + ( )x(t + - j) = (t), arise from rewriting a CD tye RX (-1,) model from (12) so that we have D -1 C-1-i Is D-j ε ( )y(t+-i) + ( )x(t+-j) = (t). In addition, we need an F %% tye RX(,-1) model of the form D -1 % -i Is F% -1-j J r ε ( )y(t+-i) + ( )x(t+-j) = (t), to develo the following recursive formulae: % * * * * -1 C-i ( Is) = C-i-1-1( Is) + C0 ( Is) % -i ( Is) i = 0,1,..., -1 * * * * -1 D-j ( ) = D-j -1( ) + C0 ( Is) F% -j-1 ( ) j = 0,1,..., -1 C * 0 ( Is) = - C -1 ( Is, )/ V% -1( Is, ) D D * F V ( Is, ) = V-1,( Is, ) + C0 ( Is) % -1( Is, ). Therefore a chain of subset RX model recursions is available and forms a comlete cycle. In summary, we describe a (-1,) to (,) subset RX recursion algorithm as follows: 1. Comute (, ), (, ) C -1 Is -1 Is and η G ( -1 Is, ), 2. Comute * * * * * * ( Is, ), C0 ( Is, ), F ( Is, ), G ( Is, ), F % 0 ( Is, ), % ( Is, ),

10 1138 J. Penm 3. Comute D H V ( Is, ), V ( Is, ), V ( Is, ), V ( Is, ), V % -1( Is, ), 4. Comute * -1 * -1 % -i ( Is), i = 0,..., -1, F% -j ( ), j = 1,...,, C ( I ), i = 1,..., -1, B ( ), j = 0,...,, * * i s j ( ), i = 0,..., -1, ( ), j = 0,..., -1, * * -i Is D-j and G * -i * ( Is), i = 0,..., -1, F ( ), j = 1,...,, -j ( ), i = 0,..., -1, ( ), j = 1,..., * * i Is H j. In deriving the (,-1) to (,) recursive formulae, we introduce a CD % % tye RX (-1,) model of the form D% C% -i Is D% -j ε ) D% 0 D% -j i=1 j=0 * as j ε % -i( Is) = 0 as i εis}, where ( )y(t + - i) + ( )x(t + - j) = (t { ( ) = 0, ( ) = 0 C D ε (t) is a n 1 zero mean disturbance rocess. (13) Use the analogous relations for deriving the (-1,) to (,) recursive relations, we can have the (,-1) to (,) recursions, which can also be obtained by alying the (-1,) to (,) recursions the following exchange: i j, Is, η,, H, C F, D, G B. Note that i j means every i will be relaced by a j, and every j will also be relaced by an i. So far, we have constructed ascending recursions, where comlex (,) order RX

11 Multivariate Comlex Subset RX Models 1139 models associated with the k-th stage are estimated with information from (-1,) order or (,-1) order comlex RX models available at the (k-1)-th stage. This structure rovides great benefit in working within a arallel comuting environment. In fitting all comlex subset RX models u to lag P and lag Q for y(t) and x(t) resectively, the recursive comutational rocedure can be embedded in an inverse tree algorithm. The root of the tree reresents the comlex full order (P,Q) RX model at the to stage of the tree and the comlex RX models with only one y vector variable and one x vector variable make u the bottom stage. Further and denote the order of the fitted scheme, = 1,2,..,P, and = 0,1,...,Q. We fit the necessary models associated with the bottom stage, and then recursively fit comlex subset RX models which include more variables, moving to higher stages, until finally we fit recursively the comlex full order RX model with maximum lags P and Q. t each stage, the (,-1) to (,) recursions are erformed if ossible, and of course the (-1,) to (,) recursions are introduced when the k-th stage comlex RX model includes only one x variable, i.e. the (,-1) to (,) recursion cannot be utilised. Of course an alternative is to erform the (,-1) to (,) recursions and to follow with all necessary (-1,) to (,) recursions. The interested reader is referred to [3]. The ascending recursions can be alternatively written in the descending format, which, for simlicity, are not resented. Further, by imosing the constraint that all real matrices are null matrices, the resulting ascending recursions become the recursions for fitting vector imaginary subset RX models. Similarly, by constraining all imaginary matrices to null matrices, the ascending recursions for fitting vector real subset RX models can be derived, which have been resented in [3]. The roosed rocedures for selecting the otimal multivariate comlex subset RX model are then summarised in the following two stes:

12 1140 J. Penm Ste 1: Minimise kaike's IC to select the best comlex full order B tye RX model from all comlex full order RX models with the order of y from 1,...,K and the order of x from 0,1,...,L, where K > P and L > Q. Schwarz's BIC is not used, because we are aware of the BIC's arsimonious roensity [3]. We emloy IC to avoid missing any relevant arameters. The IC has the following form: IC = log V ˆ +[2/N] S, with N, the samle size, S, the number of indeendent arameters, and ˆV, the estimated ower matrix. Please note that, in this case, each existing coefficient matrix * i has 2m 2 indeendent arameters, and each existing B * j has 2mn arameters. Ste 2: fter the maximum lags P and Q are selected, we then obtain the otimal comlex subset RX model by using the roosed recursions for fitting comlex RX models in conjunction with the BIC criterion. The criterion has the form: BIC = log V ˆ +[logn/n] S, and the selected model has the minimum value of BIC. However, if the natural rocess can be fully described by an imaginary imulse-resonse matrix, the otimal model would be a vector imaginary subset model. Thus we also need to search for the otimal imaginary subset RX model by reeating the two stes above with the constraint that all real coefficient matrices are null. In this case, the ascending recursions for imaginary subset RX models will be used, and each existing coefficient matrix * i has only m 2 indeendent arameters, and each existing B * j has only mn arameters. nalogously, if the natural rocess can be fully described by a real imulse-resonse

13 Multivariate Comlex Subset RX Models 1141 matrix, the otimal model would be a vector real subset model and could be obtained from the recursions defined in [3]. Subseuently, we use the BIC criterion to evaluate the otimal comlex subset RX model, the otimal imaginary subset RX model, and the otimal real subset RX model to select the otimal subset RX model. fter the otimal subset RX model is selected, it is suggested that every indeendent arameter in each existing coefficient matrix be treated as a variable, then extend the tree runing method develoed in [11] in conjunction with the BIC criterion to select the overall otimal subset RX model with zero constraints. If the true otimal subset model is a vector subset RX model with some real coefficient matrices and some imaginary coefficient matrices, then the selected otimal subset model would still be a comlex subset RX model. To establish such a model as "otimal", the tree runing method would have to be used after the roosed recursions. t resent a detailed study for evaluating this tree runing algorithm for selecting the overall otimal subset RX model with zero constraints is being carried out by the authors, but excluded in the scoe of this aer. Please note that the roosed rocedure for selecting the otimal multivariate comlex subset RX model is different from [3]. The reasons are as follows: By imosing the constraint that all real coefficient matrices are null, the recursive fitting formulae for comlex subset models can be reduced to the recursive formulae for imaginary subset models. However, this cannot be achieved by imosing any constraint on the recursive fitting formulae for real subset models. Moreover, the uestion as to whether the natural rocess is comlex or imaginary in nature cannot be assessed using

14 1142 J. Penm only the recursive fitting formulae for real subset models to analyse any time series system. Further, the number of indeendent arameters is an imortant art of the selection criteria for evaluating the otimal model. In this analysis, any existing comlex coefficient matrix of y(t-i), i=1,.., has 2m 2 indeendent arameters, and any existing comlex coefficient matrix of x(t-j), j=0,.., has 2mn indeendent arameters., whereas a real or an imaginary coefficient matrix attached to y(t-i) has only m 2 indeendent arameters, and a real or an imaginary coefficient matrix of x(t-j) has only mn indeendent arameters. However, in [3], neither comlex nor imaginary coefficient matrices can be handled. 3. CONCLUSION In the revious sections, we have given an effective recursive algorithm for fitting multivariate comlex subset RX models. The algorithm widens the ossible use of the recursive method, and leads to a straightforward and neat analysis for a variety of signal rocessing and control theory alications. This algorithm is alicable to full-order model cases, allows users to udate otimal multivariate comlex subset RXs at consecutive time instants, and can show evolutionary changes detected in multivariate comlex subset RX structures. This new aroach is articularly useful in comlex relationshis where the relevant time series have been generated from structures subject to evolutionary changes in their environment. Further investigations will be undertaken to aly the roosed algorithms to the relevant fields (see Chen et al [2], O Neill et al [12]).

15 Multivariate Comlex Subset RX Models 1143 RFRNC [1].J., Hannan and M. Deistler, The statistical theory of linear systems, John Wiley and Sons, New York, (1988). [2]. Chen, J. Penm and R.D. Terrell, n evolutionary recursive algorithm in selecting statistical subset neural network/vdl filtering. Journal of lied Mathematics and Decision Sciences, (2006), rticle ID 46592, [3] J. Penm, J.H.C. Penm and R.D. Terrell, The recursive fitting of subset VRX models, Journal of Time Series nalysis. 14, 6, (1993), [4].. Robinson, Multichannel Time Series nalysis with Digital Comuter Programs. San Francisco: Holden-Day, (1978). [5] D. Manolakis, N. Kaloutsidis, and G. Carayannis, Fast algorithms for otimum lag multichannel Wiener filters, in Proc. I Int. Sym. Circuits Syst., Rome, Italy, (1982), [6] S. Mittnik, Multivariate time series analysis with vector state sace models, Comuters and Mathematics with lications, 17, 819, (1989), [7] C.W.J. Granger and P. Newbold, Forecasting conomic Time Series, Second dition, New York: cademic Press, (1986). [8] V. Haggan and O.B. Oyetunji, On the selection of subset autoregressive time series Models, J. Time Series nal. 5, 2, (1984), [9] B.D.O. nderson and J.B. Moore, Otimal Filtering. nglewood Cliffs, NJ: Prentice-Hall, (1979). [10] P.Whittle, On the fitting of multivariate autoregressions and the aroximate factorization of the sectral density matrix, Biometrika 50, (1963), [11] J. Penm and R.D.Terrell, Multivariate subset autoregressive modelling with zero constraints for detecting causality, Journal of conometrics, 3, (1984), [12] O Neill, T.J., Penm, J. and Terrell, R.D., The seuential estimation of subset vector autoregressive modelling with a forgetting factor and an intercet variable using the exact window case. International Journal of Theoretical and lied Finance 7, 8, (2004), Received: Setember 6, 2006

Using the Divergence Information Criterion for the Determination of the Order of an Autoregressive Process

Using the Divergence Information Criterion for the Determination of the Order of an Autoregressive Process P. Mantalos a1, K. Mattheou b, A. Karagrigoriou b a.deartment of Statistics University of Lund