2 A. Neumaier and T. Schneider. INTRODUCTION We consider modeling of stationary time series data by a multivariate autoregressive process of order p (

Transcription

1 Multivariate Autoregressive and Ornstein-Uhlenbeck Processes: Estimates for Order, Parameters, Spectral Information, and Condence Regions Arnold Neumaier Universitat Wien and Tapio Schneider Princeton University Fast methods are presented for identifying a multivariate autoregressive model that is appropriate to represent large, potentially high-dimensional time series data as they occur, e.g., in geophysical applications. The algorithms are based on the concept of least-squares estimation, which is known to yield consistent and asymptotically unbiased coecient matrix estimates that also perform well on small samples. For order selection, a new and useful modication of Schwarz' Bayesian Criterion is introduced. As the interpretation of a model is often facilitated by an analysis of its eigenmodes, the spectral decomposition of autoregressive processes of arbitrary order is considered. The discussion includes the computation of condence intervals for the estimated eigenmodes, eigenvalues, and other spectral information. Numerical experiments demonstrate the eciency of the proposed estimation techniques. Further, it is shown that a discrete sample of an Ornstein-Uhlenbeck process forms a rst order autoregressive (AR()) process. This fact reduces both the simulation and estimation of Ornstein-Uhlenbeck processes to the simulation and estimation of a corresponding AR() process. Categories and Subject Descriptors: G.3 [Mathematics of Computing]: Probability and Statistics statistical computing, statistical software; I.6 [Computing Methodologies]: Simulation and Modeling; I.6.4 [Simulation and Modeling]: Model Validation and Analysis; J.2 [Computer Applications]: Physical Sciences and Engineering Earth and atmospheric sciences General Terms: Algorithms, Performance, Reliability, Theory Additional Key Words and Phrases: condence regions, least squares, model identication, multivariate autoregressive process, order selection, Ornstein-Uhlenbeck process, parameter estimation, spectral decomposition A substantial part of this work was performed while the second author was with the Department of Physics at the University of Washington in Seattle. Support by the Fulbright Commission during that time and continuing support by the German National Scholarship Foundation are gratefully acknowledged. Name: Arnold Neumaier Aliation: Institut fur Mathematik, Universitat Wien Address: Strudlhofgasse 4, A-9 Wien, Austria, neum@cma.univie.ac.at Name: Tapio Schneider Aliation: AOS Program, Princeton University Address: P.O. Box CN7, Princeton, NJ 8544, U.S.A., tapio@splash.princeton.edu

2 2 A. Neumaier and T. Schneider. INTRODUCTION We consider modeling of stationary time series data by a multivariate autoregressive process of order p (AR(p) process), v = w + px l= A l v?l + " ; " = noise(c); = ; : : : ; N; () where v ( =?p; : : : ; N) is a time series of m-dimensional state vectors, observed at equally spaced instants, and " = noise(c) is shorthand for saying that the " are uncorrelated m-dimensional random vectors with zero mean and covariance matrix C. The m-dimensional parameter vector w of intercept terms is included to allow for a nonzero mean of the time series (cf., e.g., Lutkepohl [993, Chapter 2]), hv i = (I? A?? A p )? w; where hi stands for the expectation value operation. The process of identifying an AR model that is appropriate to represent observed time series data involves several intertwined stages (cf. Tiao and Box [98]): selecting the order of the model, estimating the model parameters, and diagnostic checking of the tted model's adequacy. In the following, we shall focus on the selection of the model order p and the estimation of the unknown parameters A ; : : : ; A p, w, and C. As this study was motivated by problems in geophysical climate research that involve large amounts of time series data, the emphasis is on computational eciency of the presented algorithms. If the process order p is known, various techniques are available to estimate the coecient matrices A ; : : : ; A p, the intercept vector w, and the covariance matrix C from an observed time series of state vectors v. The maximum likelihood (ML) approach [Ansley and Kohn 983] treats an AR model essentially by assuming that all states v consist of correlated noise whose covariance matrix derives from (). Since this state covariance matrix depends on the unknown parameters, maximizing the likelihood results in an iterative procedure, which leads to estimates that are not only asymptotically ecient as N!, but also show good small-sample performance [Brockwell and Davis 99]. In addition, with the reparameterization due to Ansley and Kohn [986], stability of the tted model can be enforced. While these are desirable properties for an estimation method, the computational complexity of the ML procedure renders it slow, so that for large problems, fast algorithms are called for that either are suciently accurate to replace it completely or provide good starting points for a subsequent ML iteration. Among commonly used estimates that can be computed via fast algorithms are the Yule-Walker estimates (cf., e.g., Brockwell and Davis [99], Honerkamp [994, pp. 458.]), Burg-type estimates [Burg 978a; Burg 978b], and least squares estimates (see, e.g., Lutkepohl [993, Chapter 3]), each one of them obtaining its theoretical justication from considerations of its asymptotic behavior as N!. While these estimates are all asymptotically unbiased and, in fact, can be shown to be asymptotically equivalent, their nite-sample behavior diers signicantly. Tjstheim and Paulsen [983] computed expressions for the large-sample bias of Yule-Walker and least squares estimates. Both their theory and simulations indicate that, at least in the univariate second-order case, the Yule-Walker estimates can be

3 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 3 severely biased when the process is only marginally stable, even when the sample is comparatively large. In fact, the bias of the estimates can be appreciably larger than their standard deviation. The least squares estimates, on the other hand, are free from this defect. Our own numerical experiments with these algorithms, reported in Section 7, conrm these ndings and demonstrate that, at least in our examples, the least squares estimates also have smaller variance than Burg's estimates. Indeed, in these experiments the least squares approach yields estimates of about the same quality as the ML procedure, typically at a small fraction of the ML method's computational expense. In the discussion so far, the model order p was assumed to be known a priori, which, however, is rarely the case in practice. On the contrary, the order usually has to be estimated from the data at hand, too. In selecting the model order one has to nd a trade-o between the exibility gained by increasing the number of adjustable parameters with increasing p and the danger of overtting, i.e., tailoring the t too closely to the particular process realization that was observed. It is common practice to select an optimal model order by consecutively tting models of increasing order p p max to the data, evaluating a suitably dened goodness-of- t measure (an order selection criterion) for each p, and eventually choosing as the best-tting model the one that optimizes the employed order selection criterion. However, for large data sets this may become a tedious and computationally costly procedure, as the evaluation of the order selection criterion at each order p requires estimation of the corresponding model parameters. Fortunately, as we shall show below, the least squares computations can be organized in an ecient manner: For a given p max, one can compute with essentially the same work as that needed for the computation of the least squares estimates for an AR(p max ) model both an optimal order p opt p max and the least squares estimate for the optimal model. If the resulting model has order p opt < p max, a small prize has to be paid for this eciency in that the rst p max? p opt data points of the time series are ignored. Frequently employed order selection criteria are Akaike's Final Prediction Error FPE [Akaike 97] and Schwarz' Bayesian Criterion SBC [Schwarz 978]. Comparing the relative merits of these and several other criteria in an extensive simulation study with low-order, low-dimensional autoregressive processes, Lutkepohl [985] found that SBC selected the correct order most frequently and also led to the smallest mean squared forecasting error. We shall present a modication MSC of SBC that, in an experiment analogous to one of Lutkepohl's, turns out to estimate the correct model order even more reliably than the original SBC, in particular for short time series. In many applications, after having obtained a model that adequately represents a given time series, one seeks to gain an understanding of the dynamics of the observed system by analyzing and interpreting the tted model. As suggested by Tiao and Box [98], to this end it is often informative to perform various eigenvalue and eigenvector analyses. For example, Box and Tiao [977] propose a decomposition of a multivariate autoregressive process into components that are ordered from least to most predictable, the most predictable components often being nearly unstable and representing dynamic growth characteristics of the time series. Box and Tiao [977] apply this analysis to the interpretation of economic time series that exhibit

4 4 A. Neumaier and T. Schneider dynamic growth. In the investigation of stochastic dynamical systems with oscillatory behavior, on the other hand, the spectral decomposition of autoregressive processes can be expected to be useful. For example in the atmospheric and oceanic sciences, the analysis of an AR() process' eigenmodes often called principal oscillation patterns, a term coined by Hasselmann [988] has become a widely used data analysis tool (see von Storch et al. [988], Blumenthal [99], Schnur et al. [993], Xue et al. [994], von Storch et al. [995], Penland and Sardeshmukh [995b], and references therein to name but a few examples). However, all these studies utilized rst-order processes, a fact that limits the applicability of this technique to the fairly restricted class of time series that can be modeled adequately by such processes. In order to gain the ability to model a wider class of time series data, we shall generalize the spectral decomposition of AR() processes to processes of arbitrary order. Furthermore, we shall introduce an index that orders the resulting eigenmodes according to their relative dynamical importance, and it will be demonstrated that condence intervals for the estimated spectral information can be obtained at low cost along with the estimates themselves. Our approach to nding condence intervals for the spectral information contrasts to that put forward in a recent paper by Penland and Sardeshmukh [995a], who relied on a costly Monte-Carlo simulation method to get condence intervals for estimated eigenvectors and eigenvalues of rst-order processes. Rather than autoregressive processes, Penland and Sardeshmukh [995a] and several of the other authors cited above were actually using Ornstein-Uhlenbeck processes, the continuous analogs of the discrete AR() processes. This often led to situations in which estimates with undesirable properties such as indenite covariance matrix estimates were employed and very expensive direct numerical integration schemes were used to simulate Ornstein-Uhlenbeck processes. In circumstances in which the modeling approach is purely empirical and no physical reason demands a model with a continuous time variable, the intricacies arising from modeling with continuous processes can be avoided by resorting to discrete models. In those cases, however, where one still desires to model data with Ornstein-Uhlenbeck processes, the abovementioned diculties can be circumvented by noting that, as we shall show, a discrete sample of an Ornstein-Uhlenbeck process, taken at regular intervals of arbitrary length, forms an AR() process. This fact reduces the parameter estimation problem for a multivariate Ornstein-Uhlenbeck process to that of a related AR() process, and it can be exploited to greatly simplify the simulation of Ornstein-Uhlenbeck processes. The paper is structured as follows: In Section 2 numerical techniques for least squares estimation of parameters in autoregressive models are presented, and in Section 3 the computation of condence regions for estimated parameters and functions thereof is discussed. Section 4 deals with criteria for the selection of an autoregressive model's order, including an account on their ecient computation that draws from the results of Section 2. This is followed by a discussion of the spectral decomposition of autoregressive processes in Section 5, which contains the relevant formulas for the calculation of condence intervals for eigenvalues, eigenmodes, the stationary state covariance matrix, and derived spectral information. In Section 6 we relate equidistant samples of an Ornstein-Uhlenbeck process to a corresponding AR() process and discuss the implications for the estimation, simulation, and spec-

5 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 5 tral decomposition of Ornstein-Uhlenbeck processes. Finally, we report in Section 7 results of numerical experiments with the presented algorithms. Most of the methods proposed below were implemented in the Matlab package ARfit which is described in a companion paper [Schneider and Neumaier 997]. Where appropriate in the subsequent discussions, we will point the reader to modules in ARfit that contain implementations of the algorithms under consideration. Notation. A :k denotes the kth column of the matrix A. A T is the transpose, and A y the conjugate transpose of A. The inverse of A y is written as A?y, and the superscript denotes complex conjugation. 2. LEAST SQUARES ESTIMATION FOR AUTOREGRESSIVE MODELS Throughout this section, we suppose that the time series v?p ; : : : ; v N is known to be a realization of a pth order autoregressive process. We consider the least squares estimation of the unknown model parameters A ; : : : ; A p, w, and C from the time series data, deferring the case of unknown model order to Section 4. The following Section 2. serves to introduce the notation by means of a brief review of the least squares estimation technique; for details of its derivation and a more general discussion of the estimates' properties, the reader may consult, for example, Lutkepohl [993, Chapter 3]. Subsequently, in Section 2.2, we shall present an algorithm for the numerical computation of the estimates. 2. Form and properties of least squares estimates The least squares estimates for autoregressive processes are most conveniently expressed when the AR model () is cast in the form with coecient matrix and v = Au + " ; " = noise(c); = ; : : : ; N; (2) u = A = (w A A p ); (3) v?. v?p C A 2 IRn ; n = mp + : (4) The key step in the derivation of the least squares estimates is to view (2) as a linear regression model with xed predictors u, which is an approximation as for measured time series the u are realizations of a random variable. However, (4) implies that the assumption of xed predictors amounts to treating u = v. v?p C A available from ~ neum/software/art/

6 6 A. Neumaier and T. Schneider as a vector of xed initial values, and since the relative impact of the initial condition vanishes as the time series length N approaches innity, using parameter estimates for the regression model (2) in the autoregressive model () can be expected to be asymptotically correct. For the linear regression model (2), the least squares procedure yields the best linear unbiased estimates. Introducing the matrices U = NX = u u T ; V = N X = v v T ; W = N X = the estimate for the coecient matrix A can be written as and an estimate for the covariance matrix C is given by ^C = N? n NX = v u T ; (5) ^A = W U? ; (6) ^" ^" T with ^" = v? ^Au : In the covariance matrix estimate ^C, an adjustment for degrees of freedom was applied, resulting in the factor =(N? n) in place of =N, as for the regression model this adjustment leads to an unbiased estimator. Alternatively, the covariance estimate can be expressed as ^C = N? n (V? W U? W T ); (7) which is proportional to the Schur complement of the matrix U W T W V and therefore assured to be positive semidenite. Though these estimates are derived for the regression model with xed predictors, which only approximates the AR model, it can nevertheless be shown that, under fairly general conditions on the noise term in (), the estimators (6) and (7) when applied to the AR model with coecient matrix (3) and \predictors" (4) are still consistent and asymptotically normal (see, e.g., Lutkepohl [993, Chapter 3]). In Section 3, we shall take advantage of these facts in the computation of condence intervals for the estimated parameters. However, rst we consider the numerical solution of the matrix equations (6) and (7) for the parameter estimates. 2.2 Numerical considerations As noted above, ^C is the Schur complement of the matrix U W T NX u? = u T W V v v T = K T K; (8) where = K = u T v T. u T N. vt N C A : (9)

7 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 7 A QR factorization of K, with R R Q orthogonal; R = 2 R 22 K = QR () upper triangular; provides an easy-to-implement way of computing the parameter estimates (6) and (7) numerically: Rewriting (8) as U W T = R T R T R = R RR T 2 W V R2 T R R2 T R 2 + R22 T R ; 22 one nds the representation ^A? = R? R T 2 ; ^C = N? n RT 22 R 22 () for the least squares estimates of the coecient matrix A and the covariance matrix C, respectively. Representation () yields ^A as the solution of triangular systems of equations and provides ^C in a factored form that exhibits the semideniteness explicitly. As an alternative to the QR factorization, one can obtain R = L T also from a Cholesky decomposition U W T W V = LL T : However, when U or ^C are ill-conditioned, using a QR factorization to calculate () is numerically more stable than a Cholesky decomposition. For this reason, the former is our recommended approach. To reduce the eect of rounding errors in extremely ill-conditioned cases, it is protable to calculate a Cholesky factorization of K T K + "D 2 = R T R, where D = Diag?P i jk ijj. This means that one has to use in place of () a regularized QR factorization of the form K p"d = QR: ARfit's module ar produces the least squares estimates for multivariate autoregressive models using this QR factorization technique. Probably the above crude regularization scheme could be made even more eective by using adaptive regularization techniques; see, e.g., Hanke and Hansen [993] or Neumaier [997]. 3. CONFIDENCE REGIONS Let = xa x C

8 8 A. Neumaier and T. Schneider be the parameter vector containing the elements of the matrices A and C, where x A = A :. A :n C A 2 IR mn consists of the components of the matrix A 2 IR mn, arranged as a vector by stacking adjacent columns A :j of A. The symmetry of the covariance matrix implies that it would suce for to contain the elements in the upper triangle of C only, but for notational simplicity in what follows, let consist of all elements of C. In the following, we shall discuss the computation of condence intervals for a realvalued function = () of the parameters. For example, may stand for one of the entries in A or C, or, in a later section, for a component of an eigenvector of an AR() process' coecient matrix. Consider the propagation of errors? ^ in the parameter estimate ^ into the estimate ^ (^). (Here and henceforth, the superscript ^ refers to estimated quantities, i.e., functions of the parameters with the respective parameter estimates (6) and (7) substituted for A and C.) Linearizing about ^ leads to? ^? r ^ T (? ^), where r ^ denotes the gradient of () at = ^. Using this linearization, one can approximate the mean squared error 2 (? ^) 2 as where 2? r ^ T? r ^ ; (2) = (? ^)(? ^) T is the covariance matrix of the estimator ^. The relation (2) becomes exact when the residues? ^ follow a normal distribution because in this case their higher cumulants vanish. If in addition to the estimator being normal its covariance matrix is known, 2 is a 2 distributed random variable, and this implies that a condence level associated with the condence interval j? ^j (3) is given by the probability P that a realization of a 2 distributed random variable is smaller than = 2 2 : Thus, for a given condence level P, one may assert (3) with = p ; (4) where is given by (2); the parameter is the solution of P = 2 ; 2 where (; x) =?() Z x t? e?t dt (5)

9 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 9 is the incomplete gamma function. Application of these formulas requires an expression for the covariance matrix of the estimates. Under fairly general conditions on the noise in the AR model, it can be shown that, as the sample size approaches innity, the residues? ^ are asymptotically normal distributed with zero mean and covariance matrix (cf. Lutkepohl [993, Chapter 3.2.2]) where ^ = ^ A ^ C ; (6) ^ A = U? ^C and ^ C = 2 N? n ^C ^C: (7) Here, A B denotes the Kronecker product of the matrices A and B, and in the estimate ^ C again a degree of freedom adjustment was applied. Substituting this asymptotic estimate ^ for in (2) enables one to compute approximate condence intervals for ^ by means of (3){(7). For the special cases = A jk and = C jk, the gradient r is a column of the identity matrix. In this case, (2) implies that 2 is just the corresponding diagonal element of the covariance matrix ^. More explicitly, using (4) and the Kronecker product property (A B) ll = A kk B jj if l = m(k? ) + j; one nds the following formulas for the condence intervals of the estimated parameters: ja jk? ^Ajk j jc jk? ^Cjk j q q (^ A ) ll = (U? ) kk ^Cjj ; (8) q (^ C ) ll = r 2 N? n ^C kk ^Cjj : (9) Here the parameter is the solution of (5) for a given condence level P. The routine arconf in the ARfit package uses these expressions to compute condence intervals for the estimated parameters of an autoregressive process. 4. ORDER SELECTION In practice, the model order is frequently unknown and must be estimated from the data, too. After discussing various order selection criteria in the following subsection, we show how to evaluate them eciently in Section 4.2. In Section 4.3, we verify that our combined order selection and parameter estimation scheme works with essentially the same cost as required to estimate parameters in the xed order case. 4. Selection criteria For the computation of order selection criteria, we explicitly consider the dependence on the model order p and write p = (N? n) ^C = R T 22 R 22 ;

10 A. Neumaier and T. Schneider n p = n = mp + : Equations () for the least squares estimates then become? ^A = R? R T 2 ; ^C = p : (2) N? n p According to Lutkepohl [993, Chapter 4], who discusses in detail a number of order selection criteria, a useful order estimate is the value of p that minimizes Schwarz' Bayesian Criterion (SBC, see Schwarz [978]), dened by where 2 SBC(p) = log 2 + n p log N N ; det p N =m = (det p ) =m =N is a scalar measure for the (not bias corrected) variance of the estimator. (We divided Lutkepohl's expression for SBC by the constant dimension m of the state vectors.) Using the abbreviation l p = log det p ; one nds the equivalent expression SBC(p) = l p m?? n p log N: (2) N Another order estimate based on the expected one-step prediction error is the value of p that minimizes Akaike's [97] Final Prediction Error criterion FPE, given by FPE(p) = log 2 + log N + n p N? n p : (We divided the logarithm of the standard expression N+np m N?n p det p N for FPE by m.) In terms of l p, this leads to FPE(p) = l p m? log N(N? n p) N + n p : (22) It is known that SBC is a strongly consistent order selector [Lutkepohl 993]. SBC generally predicts the order much better than FPE which possesses no such consistency property. To improve upon the small sample properties of SBC, we tried a number of modications of SBC that behave asymptotically like SBC. As a result of extensive tests, we propose the Modied Schwarz Criterion (MSC), dened by MSC(p) = l p m?? 2:5n p (log N? 2:5): (23) N? n p Just as SBC, MSC still satises the strong consistency conditions given in Lutkepohl [993, Chapter 4]. Indeed, for long time series MSC behaves in practice essentially as SBC, but for short time series, MSC consistently gives a signicantly reduced fraction of wrong order estmates. (See Section 7.3 for an illustrative example.)

11 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 4.2 Ecient evaluation of selection criteria Order selection criteria such as SBC and MSC can be computed eciently for a sequence of AR(p) models with p p max once the QR factorization () is available for some p = p max. There is no need to carry out the QR factorization () for each p in the sequence; as we shall show, from the factorization for p = p max one can with small additional eort also obtain good parameter estimates for models of order p < p max. To nd formulas that realize these savings, note that the matrix K, dened in (9), can be written as K = (K K 2 ); K = u T.. u T N C A ; K 2 B v T.. v T N C A : When reducing some order p to the order p = p?, one works instead with K = (K K 2 ); where K is obtained from K by removing the last m columns. Thus K = (K K ); where K contains the removed columns. If one partitions accordingly R R = R R ; R 2 = R 2 R 2 by treating the last m rows and columns of R and the last m rows of R 2 separately, the QR factorization () becomes (K K K 2) = Q@ R R R2 R R 2 R 22 A : Dropping in this equation the m superuous columns we get with K = (K K 2) = R R 2 R 22 = R2 R 22 R 22 : R 22 A = Q R R 2 R 22 This is a factorization of the same kind as before, except that R 22 is no longer triangular, the latter, however, being of no relevance in the derivation of (2). Thus we have and writing p = R T 22 R 22 = R T 22 R 22 + R T 2 R 2 ; R p = R 2;

12 2 A. Neumaier and T. Schneider we nd the update formula Using the Sherman-Morrison formulas and p? = p + R T p R p:? det p + Rp p? T R = det p det I + R p? p RT p? p + R T p R p? =? p?? p RT p ;? I + Rp? p RT p Rp? p [Sherman and Morrison 949; Duncan 944], the update for the determinant term and the inverse l p = log det p M p =? p can be computed from a Cholesky factorization via I + R p M p R T p = L p L T p (24) l p? = l p + 2 log det L p ; (25) N p = L? p R pm p ; (26) M p? = M p? N T p N p: (27) After having determined an optimal order p opt, one nds corresponding least squares parameter estimates from (2) on replacing the maximally sized R and R 2 by their leading submatrices of size n popt. This algorithm, using formulas (2){(27), is implemented in the Matlab module arord. Though the above procedure is very convenient when estimating an optimum model order p opt by minimizing order selection criteria, it is not quite optimal: In the order p least squares estimates, p max? p initial data points are ignored. However, asymptotically this does not aect the quality of the results, and for very short time series where this might be a serious loss of information, one may repeat the least squares t with p max = p opt. 4.3 Operation count As K is a N 2mp max matrix, the work for the QR factorization in the above algorithm is of the order O(m 2 p 2 maxn). The updates (24){(27) can be done with work cubic in the dimension 2mp max of R. Since N mp max, this contributes at most work of the same order as that required for the QR factorization, and for N mp max the contribution of the updating process becomes negligible. Thus the total operation count is a small multiple of m 2 p 2 maxn, implying that our algorithm handles high-dimensional models and large data sets much more eciently than standard procedures, which compute order selection criteria for a sequence of AR models by a separate factorization for each model order.

13 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 3 5. SPECTRAL DECOMPOSITION OF AR PROCESSES We shall now proceed to show how to estimate condence intervals for spectral information computed from a tted autoregressive model. To this end, we rst introduce some notation by briey reviewing the spectral decomposition of AR() processes; a similar account can, for example, be found in Honerkamp [994, pp. 426.]. Then formulas for the computation of condence intervals are obtained, and, in Section 5.2, the results for rst order processes are generalized to autoregressive processes of arbitrary order. For notational simplicity, we restrict ourselves for the remainder of this paper to processes that are known to have zero mean, so that the intercept term w in () is exactly zero. 5. Spectral decomposition of AR() processes Let us consider the AR() process v = Av? + " ; " = noise(c); v 2 IR m : (28) Suppose that A is nondefective, so that a basis of (complex) eigenvectors exists, given by the columns S :k of a matrix S. Then S is nonsingular and whence AS :k = k S :k ; A = SS? ; = Diag( k ): (29) In this spectral basis, one can represent the state vectors v and the noise vectors " as linear combinations v = mx k= v (k) S :k = Sv and " = mx k= " (k) S :k = S" (3) with coecient vectors v = (v () ; : : : ; v (m) ) T and " = (" () ; : : : ; " (m) ) T. For these coecient vectors, one obtains from (28) the dynamics with v = v? + " ; " = noise(c ); (3) C = S? CS?y : (32) Writing the model (3) componentwise, it is evident that it represents a system of uncoupled processes with correlated noise v (k) = k v (k)? + "(k) (33) h" (k) "(l) i = Ckl (34) (where = for = and = otherwise). From (33) follows that the expectation value for arg k 6= describes a spiral hv (k) hv (k) i = k hv (k)? i +t i = t khv (k) i = e?t= k e(arg k)it hv (k) i

14 4 A. Neumaier and T. Schneider in the complex plane with damping time scale and period k?= log j k j (35) T k 2 j arg k j : (36) Here, arg z = Im(log z) denotes an argument of the complex number z, and we employ the convention? arg z, which ensures that a pair of complex conjugate eigenvalues is associated with a single period. For a stable process with nonsingular A, we have < j k j < for all k = ; : : : ; m, hence k is positive and bounded. If k has nonvanishing imaginary part or if k is real but negative, then T k is bounded, too, and (33) is called a stochastically driven (damped) oscillator. The period attains its minimum value T k = 2 when k is real and negative, that is, j arg k j = ; this is the period corresponding to the well-known Nyquist frequency from Fourier analysis. On the other hand, if k is real and positive, then T k! and (33) describes a stochastically driven relaxator. Thus, (3) analyzes the process (28) in terms of linear combinations of relaxators and damped oscillators with relaxation and oscillation modes S :k associated with damping time scales k and periods T k. Real spectral decomposition. Since the matrix A is real, its eigenvalues and eigenvectors, when scaled appropriately, come in complex conjugate pairs, and one can also obtain a decomposition of an autoregressive process in terms of real rather than complex modes. Indeed, for any complex eigenvalue k = a k + ib k ; the corresponding process (33) can be written as a real bivariate AR() process with noise vectors Re v (k) Im v (k) " (k) =! ak?b = k b k a k! Re "(k) Im " (k) = 2 Re v (k)? Im v (k)? " (k ) i(" (k )! + " (k)? " (k) ) + " (k) ; Here, the quantities indexed by k are conjugates of those indexed by k. From (34), using the Hermiticity of C, one now nds the noise covariance matrix h" (k) " (l) T = 4 = 2 i C kl + Ck l + C k l + C kl?i(?c kl + C k l? C k l + C kl )?i(ckl? C k l? C k l + C kl ) C kl + C k l? C k l? C kl Re C kl + Re Ck l Im Clk + Im C lk : Im C kl + Im C k l Re C kl? Re C k l The corresponding eigenmodes of this process are given by the real and imaginary parts of the eigenvector S :k.! :

15 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 5 State covariance matrix. The spectral representation (29) also provides a convenient means of computing the state covariance matrix = hv v T i of a stationary autoregressive process. Using (28) in the denition of the state covariance matrix, we have so that = h(av? + " )(Av? + " ) T i = Ahv? v T?iA T + h" " T i; = AA T + C: (37) Substituting for A from (29) and for C from (32), one nds = S S y (38) with a solution of = y +C. Since is diagonal, this reads in components (? k l ) kl = C kl : For k = l, this equation shows that, at least when C is positive denite (so that C is denite, too, and has positive diagonal entries), all eigenvalues satisfy j k j <. If this condition holds, the equations can be solved for kl and give C kl kl = : (39)? k l Moreover, (3) and (38) imply that (39) is the covariance matrix of the amplitudes, hv (k) v(l) i = kl : (4) Relative dynamical importance of modes. From (33) we infer that, if the columns of S are normalized to norm, the magnitudes of the (estimated) v (k) indicate the relative importance of the various relaxation and oscillation modes. One can thus dene the excitation k hjv (k) j2 i (4) as an index of the relative importance of the various modes. From the representations (39) and (4) of the state covariance matrix in the spectral basis, it follows that k = C kk? j k j 2 ; and this expression has intuitive appeal: It is the ratio of the forcing strength C kk over the damping?j k j 2 of mode k. In the process of interpreting a tted autoregressive model, the excitation k may provide a useful measure of the dynamical importance of the modes. Our suggestion of sorting modes according to their excitation contrasts with many studies (inter alios von Storch et al. [988], von Storch et al. [995], Penland and Sardeshmukh [995b]), in which the modes were ordered from least damped to most strongly damped, i.e., in order of decreasing j k j. The tradition of viewing the least damped modes as the dynamically most important ones is probably based on the fact that in unforced deterministic linear systems the least damped mode, if excited, in the limit of long times eventually dominates the dynamics. However, in the presence of stochastic forcing this need not be the case

16 6 A. Neumaier and T. Schneider as the weakly damped modes may not be suciently excited by the noise. The excitation k therefore appears to be a more appropriate measure of dynamical importance. Condence intervals. Computing condence intervals by means of (3){(7) requires formulas for the derivatives of the estimated eigenvalues, of the eigenvectors, and of the state covariance matrix with respect to parameters in A and C. For condence regions to be meaningful in the rst place, the quantities under consideration must be uniquely dened. In order to obtain a unique representation of an eigenvector S :k = X :k + iy :k with X = Re S and Y = Im S, we impose the normalization conditions X T :kx :k + Y T :k Y :k = ; X T :ky :k = ; Y T :k Y :k < X T :kx :k : (42) Dierentiating these normalization conditions and the eigenvector-eigenvalue relation leads to where the symbol AS :k = k S :k ; A _S :k + _AS :k = _ k S :k + k _S :k ; X T :k _ X :k + Y T :k _ Y :k = ; X T :k _ Y :k + Y T :k _ X :k = l denotes the derivative with respect to a component l of the parameter vector from Section 3. Using the spectral decomposition (29), these equations can be rearranged in the form (? k I)S? _S :k? e (k) _ k =?S? _AS :k ; (43) X:k T X _ :k + Y:k T Y _ :k = ; (44) X:k T Y _ :k + Y:k T X _ :k = ; (45) where e (k) is the kth column of an identity matrix. The kth component of (43) now gives _ k = (S? _AS) kk (46) as an explicit formula for the derivative of the eigenvalue. Writing the derivative of the eigenvector as the remaining components of (43){(45) take the form _S :k = SZ :k ; (47) ( j? k )Z jk =?(S? _AS) jk for j 6= k; (X T :kx + Y T :k Y ) Re Z :k + (Y T :k X? X T :ky ) Im Z :k = ; (X T :ky + Y T :k X) Re Z :k + (X T :kx? Y T :k Y ) Im Z :k = :

17 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 7 Assuming that all eigenvalues are distinct, these equations can be solved for Z and yield and Re Z kk = X l6=k Z jk = (S? _ AS) jk k? j for j 6= k; (48)? (X T Y? Y T X) kl Im Z lk? (X T X + Y T Y ) kl Re Z lk ; (49) Im Z kk = P? l6=k (Y T Y? X T X) kl Im Z lk? (X T Y + Y T X) kl Re Z lk (X T X? Y T : (5) Y ) kk (In deriving (49) and (5) we used the normalization conditions (42) in the form (X T X + Y T Y ) kk = and (X T Y ) kk =.) In case of multiple eigenvalues, equations (43){(45) cannot be solved uniquely for the derivative of the eigenmodes with respect to parameters in A and C. However, in this degenerate case the eigenvectors are no longer uniquely determined, and it is not meaningful to give condence regions for them. Writing the eigenvalues as k = a k +ib k with respective real and imaginary parts a k and b k, we deduce from (46) that _a k = Re(S? _AS) kk ; _ bk = Im(S? _AS) kk : As is easily veried, the derivatives of the damping time scales (35) and periods (36) then take the form and _ k = 2 k a k _a k + b k _ bk a 2 k + b2 k _ T k =? T 2 k 2 Im _ k k = T 2 k 2 (5) b k _a k? a k bk _ a 2 k + ; (52) b2 k respectively. For the calculation of the derivative of the stationary covariance matrix we use its representation (38) to nd where the derivatives _ jk = _ = _S S y + S _ S y + S _S y ; (53) _C jk (? j k ) + C jk ( _ j k + j _ k ) (? j k )2 (54) are obtained by dierentiation of (39). As above, SS? = I implies SS _? + S(S? ) =, hence and (S? ) =?S? _ SS? (S?y ) =?S?y _ S y S?y :

18 8 A. Neumaier and T. Schneider Thus, taking the derivative of (32) gives _C = S? ( _C? _SS? C? CS?y _S y )S?y : (55) Equations (46){(55) give easily programmable formulas for all derivatives required. Note that for the spectral representation of the noise covariance matrix, C = T?T? ;? = Diag( k ); one can invoke the same arguments as above to get condence intervals for the eigenvectors T :k and eigenvalues k. For the derivatives one nds again (46){(5) but with A replaced by C, replaced by?, and S replaced by T. Since C is symmetric, both? and T are real. 5.2 Spectral decomposition of processes of arbitrary order To generalize the results for the AR() case to processes of arbitrary order, we utilize the fact that an AR(p) process v = px l= A l v?l + " ; " = noise(c); with v 2 IR m can be represented as an AR() process with augmented state and noise vectors ~v = and coecient matrix ~A = v v?. v?p+ ~v = ~ A~v? + ~" C A ; ~" = ". C A 2 IRmp ; A A 2 A p? A p I I... C A 2 IRmpmp : I Assuming that the coecient matrix ~ A of this AR() process is nondefective, one can again represent ~ A as ~ A = ~ S ~ S? ; = Diag(k); where the columns S:k ~ of S ~ are the mp-dimensional (complex) eigenvectors of A. ~ As above, for the coecients ~v (k) and ~" (k) in the expansions of the augmented state vectors ~v = Xmp k= ~v (k) ~S :k (56)

19 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 9 and of the noise vectors ~" = one obtains the decoupled dynamics with correlated noise where and ~v (k) h" (k) Xmp k= ~" (k) ~S :k (57) = k ~v (k)? + ~"(k) ; (58) "(l) i = C kl C = ~ S? ~ C ~ S?y C ~C = h~" ~" y i = 2 IR mpmp : Since the eigenvectors ~ S:k of the matrix ~ A satisfy ~S :k = p? k S :k. k S :k S :k where S :k is an m-dimensional (complex) vector, one nds from (56) that and similarly from (57) Introducing v?p+j+ = v (k) Xmp k= p? k " = C A ~v (k) j k S :k; j = ; : : : ; p? ; Xmp k= ~" (k) p? k S :k : ~v (k) and " (k) p? k ~" (k) ; one can now represent the original, m-dimensional state vectors v and noise vectors " as linear combinations v = Xmp k= X mp v (k) S :k and " = k= " (k) S :k: From (58) we conclude that the dynamics of the coecients v (k) system of uncoupled processes with correlated noise v (k) = k v (k)? + "(k) ; k = ; : : : ; mp h" (k) " (l) i = ( k l ) p? Ckl: are governed by a

20 2 A. Neumaier and T. Schneider As in the rst order case, each eigenmode is associated with a damping time scale (35) and a period (36); the excitation k = hjv (k) j 2 i = j k j 2(p?) hj~v (k) j 2 i of an eigenmode is obtained in a similar fashion as in the rst order case. Thus, the only dierence to the AR() case is that the AR(p) process has a larger number of eigenmodes; they still span the space but are no longer linearly independent. (This is analogous to what happens in higher order linear dierential equations with constant coecients.) Condence intervals. Imposing the normalization conditions ~X T :k ~ X :k + ~ Y T :k ~ Y :k = ; ~ X T :k ~ Y :k = ; ~ Y T :k ~ Y :k < (59) on the eigenvectors ~ S:k = ~ X:k + i ~ Y:k of ~ A, the derivative formulas (46){(52) apply with S replaced by ~ S and A by ~ A. This allows one to compute condence intervals by means of (3){(7). The module armode is a straightforward implementation of the procedures laid out in this section. For AR models of arbitrary order, it performs the estimation of eigenmodes, of periods, of damping time scales, and of condence intervals for the estimated spectral information. 6. PARAMETER ESTIMATES FOR LINEAR CONSTANT COEFFICIENT LANGEVIN EQUATIONS The multivariate, zero-mean Ornstein-Uhlenbeck process v(t) is dened by the Langevin equation d dt v(t) = A v(t) + "(t); "(t) WN(; C ); (6) where "(t) is a Gaussian white noise process with zero mean and covariance matrix C (see, e.g., Gardiner [983]). We consider sampling from a stable Ornstein-Uhlenbeck process v(t) at discrete steps t = h ( = ; ; : : :). In the following, we shall show that the resulting time series is a realization of an AR() process, and this will allow us to use the machinery developed for the identication of AR() processes to estimate the parameter matrices A and C appearing in the Langevin equation above. Theorem. A discrete sample v = v(h) (6) from an Ornstein-Uhlenbeck process v(t), dened by (6), forms an AR() process with coecient matrix v + = Av + " ; " = noise(c); (62) A = e A h and a constant noise covariance matrix C. Proof. Inserting t = ( + )h and t = h into the integral equation v(t) = e A (t?t ) v(t ) + Z t t (63) dt e A (t?t ) "(t ) (64)

21 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 2 that is equivalent to (6), yields, after a change of the integration variable, or, using (6), Here, the noise terms v(( + )h) = e A h v(h) + " = Z h dt e A t "? ( + )h? t ; v + = e A h v + " : (65) Z h dt e A t "? ( + )h? t are independent Gaussian noise vectors with covariance matrix C = h" " T i = Z h dt Z h which simplies to the constant matrix dt e A t h"? ( + )h? t " T? ( + )h? t ie AT t ; C = Z h Comparison of (65) with (62) now completes the proof. Rewriting (63) in the equivalent form dt e A t C e AT t : (66) A = h? log A; (67) one recognizes that the eigenvectors of A and A are the same and the eigenvalues k of A and k of A are related by Therefore, if as in Section 5 then k = h? log k : (68) A = SS? ; = Diag( k ); A = S(h? log )S? : (69) As in Section 5, one can associate a damping time scale k and a period T k with each eigenmode S :k. For the Ornstein-Uhlenbeck process these read and k =? Re k T k = 2 Im k ; respectively. Substitution of (68) and comparison with (35) and (36) yields T k = ht k ; k = h k : Thus, damping time scales and periods for the Ornstein-Uhlenbeck and corresponding AR() process dier only by the scaling factor h a result that could have been anticipated by physical reasoning.

22 22 A. Neumaier and T. Schneider A computationally more convenient relation between C and C than that given by (66) is obtained using the fact that the Ornstein-Uhlenbeck process and its corresponding AR() process must have the same stationary process covariance matrix. In order to nd a relation between the stationary process covariance matrix and the noise covariance matrix, we calculate the dynamics of the process covariance matrix (t) = hv(t)v(t) T i? hv(t)ihv(t)i T : Since the process is non-anticipating, i.e., hv(t )" T (t)i = for all t t, one nds from (64) (t) = e A (t?t ) (t )e AT (t?t ) + Dierentiation with respect to t yields Z t t dt e A (t?t ) C e AT (t?t) : d(t) = (t)a T + A (t) + C : dt In the limit t!, the process covariance matrix converges to the stationary covariance matrix (), and its derivative converges to zero. This gives the wellknown uctuation-dissipation theorem In Section 5, we showed that the stationary limit C =?()A T? A (): (7) = (); of the discretized process covariance matrices (h) for! can be computed from (38). It follows that after having identied the parameters of the AR() process one can nd its stationary covariance matrix and then use the uctuation-dissipation theorem (7) to get the covariance matrix of the white noise term in the corresponding Langevin equation (6). Thus, using also (38) and (69), one obtains C =?(A + A T ) =?h? S?? (log ) + (log ) y S y ; (7) which is easily computed once the spectral decomposition of the corresponding AR() process is available. Condence intervals. As in Section 5, the derivatives of A and C with respect to parameters in A and C are needed in order to compute condence intervals by means of (3){(7). From (67), one sees that the derivative of A is _A = h?? S(log )S? = h? ( _S log + S? _? S(log )S? _S S? ; (72) where the derivative _S of the eigenmodes is given by (47){(5), and the derivative of the eigenvalues, _ = Diag( _ k );

23 AR and Ornstein-Uhlenbeck Processes: Estimates and condence regions 23 follows from (46). Similarly, the derivative of the noise covariance matrix becomes _C =?( _A + _A T + A _ + _A T ) (73) with _A from (72) and _ from (53). Since the eigenmodes of the AR() process and the Ornstein-Uhlenbeck process are the same, so are the condence intervals for the estimated modes. For the eigenvalues we conclude from (68) that _ k = h? _ k = k with _ k again given by (46). Finally, one obtains condence intervals for the damping time scales and periods associated with the modes of the Ornstein-Uhlenbeck process by scaling the damping times scales and periods of the corresponding AR() process with the sampling step size h. Simulation of Ornstein-Uhlenbeck processes. Stability of a direct numerical simulation of the Langevin equation (6) often requires time steps that are much smaller than the desired sampling step size of the Ornstein-Uhlenbeck process. (See, e.g., Kloeden and Platen [992] for numerical methods for stochastic dierential equations.) In this case, the fact that a discrete sample from an Ornstein-Uhlenbeck process forms an AR() process is also an aid in the numerical simulation of Ornstein- Uhlenbeck processes. Instead of using expensive schemes to integrate the Langevin equation (6) directly, one simply simulates the corresponding AR() process (62) with coecient matrix A from (63) and noise covariance matrix (cf. (37)) C =? AA T ; (74) with = () satisfying the uctuation-dissipation theorem (7). The noise covariance matrix C can be computed by rst substituting A = SS? ; = Diag( k ) (75) into the uctuation-dissipation theorem (7) and then multiplying by S? from the left and by S?y from the right. Introducing and the uctuation-dissipation theorem (7) becomes C = S? C S?y (76) = S? S?y ; (77) C =? y? y ; which, using the Hermiticity of, reads in components Solving for kl yields (C ) kl =? kl l? k kl : kl =? (C ) kl k + l : (78) Now, computing (75), (76), (78), solving for the state covariance matrix from (77), and nally inserting the so obtained into (74), yields the needed noise

24 24 A. Neumaier and T. Schneider covariance matrix C of the AR() process (62) corresponding to the Ornstein- Uhlenbeck process (6). As the desired length of the integration and the sampling step size h increase, this approach of simulating a sample from an Ornstein-Uhlenbeck process becomes more and more ecient compared to integrating the Langevin equation directly. 7. NUMERICAL EXAMPLES Consideration of asymptotic properties forms the theoretical basis for the least squares method and other estimation procedures such as the Yule-Walker scheme and Burg's algorithm. The practitioner, however, usually has only a limited data set available. As analytical results for nite samples are dicult to obtain, we shall now study the small-sample performance of the procedures described in the previous sections resorting to Monte-Carlo experiments with time series of various length. In the following subsection, a bivariate process serves to illustrate both the consistency of the least squares estimates and the small-sample properties of the estimates for the parameters and derived spectral information. The subsequent Section 7.2 contains a performance comparison of the least squares procedure with other techniques in estimating a univariate AR(3) process. Finally, in Section 7.3, the performance of various order selection criteria is compared on a variety of simulated autoregressive processes. To ease the presentation of the results, we restricted ourselves in all these examples to low-dimensional data sets. However, we emphasize that the algorithms described above are ecient enough to handle large amounts of high-dimensional data. 7. A bivariate example We generated time series data by simulation of the bivariate AR(2) process v = A v? + A 2 v?2 + " ; " = WN(; C); = ; : : : ; N (79) with parameters A = :4 :2 :3 :7 ; A 2 = :35?:3 ; C =?:4?:5 : :5 : (8) :5 :5 Again, only processes with zero mean are considered, so that the intercept vector w in () is identically zero; it follows that the coecient matrix A and the predictor u in (3) and (4) must be replaced by and u = A = (A A p ) v?. v?p C A 2 IR n ; n = mp; respectively. We used the QR factorization technique from Section 2.2 (as implemented in ar) to estimate the parameters = fa ; A 2 ; Cg from realizations of the AR(2) processes (79) for various values of N. For all parameters j the approximate 68:3% condence intervals j j? ^ j j ( j ) were computed using the formulas (8) and