Time Series 2. Robert Almgren. Sept. 21, 2009

Transcription

1 Time Series 2 Robert Almgren Sept. 21, 2009 This week we will talk about linear time series models: AR, MA, ARMA, ARIMA, etc. First we will talk about theory and after we will talk about fitting the models to real data. Let {x t } t= be a scalar time series. That means that each x t is a real-valued random variable on some probability space, and the time series is completely defined by the distribution of each x t and of all the joint distributions. The index variable t has the interpretation of discrete time, and we consider the series to extend to infinity in both directions. This definition of a time series does not talk about any rule by which the values x t are generated; that is the subject of this section. The series {x t } is stationary if all joint distributions x t,..., x t p are independent of t, that is, of where we are in the sequence. It is weakly stationary if only the first and second moments are independent of t. The series is Gaussian if all joint distributions are Gaussian. If the series is Gaussian then weak stationarity is equivalent to stationarity. For the linear theory we only worry about first and second moments, it will not really matter whether a series is stationary or only weakly so, or whether it is Gaussian. People with a background in stochastic analysis may wonder what about filtrations? Do we need a detailed construction to describe how information is revealed in time? Generally speaking, that concept stays in the background in time series analysis, because we are not doing optimal control. It is still present, and we would prefer that any real-time analysis methods include only data observed to date rather than future data. But unlike trading strategies, here the payoff for forward-looking is less dramatic since we are only doing statistical modeling. Sometimes we even consider analysis methods such as Fourier transforms that use the entire series of observations both future and past. 1

2 Robert Almgren: Time Series 2 Sept. 21, Partial auto-correlations Last time, we defined the mean and autocovariance at lag l µ = E ( x t, γl = Cov ( x t, x t l = E ( (xt µ (x t l µ. Note that these formulas have a t on the right side, meaning that we should take the mean and covariances of some specific term in the sequence. But there is no t on the left side because of stationarity. The zero-order covariance γ 0 = Var(x t is the mean-square size of the individual terms. Then the autocorrelation is ρ l = γ l γ 0. which has ρ 0 = 1 and ρ l 1. The autocovariance and autocorrelation are defined for all l = 0, ±1, ±2,... and are even: ρ l = ρ l. Let us quickly review what correlation means. Correlation of ρ l between x t and x t l tells you how much you learn about the value of x t by knowing the value of x t l, without knowing anything about any other values. As usual in stochastic modeling, we assume we know all the parameters of the process µ and { } γ l l= ; the only unknown is the values x t themselves. In general, suppose that ξ and η are two random variables with E(ξ = E(η = 0, E(ξ 2 = σ 2 ξ, E(η2 = ση 2, and E(ξη = ρσ ξ σ η. If we had no information about the value of ξ, our best predictor for the value of η would be simply η = 0. To predict the value of η given ξ, we make a linear model η = βξ. We choose the value of β by minimizing the squared error: [ (η ] 2 min E βξ β ( = min σ 2 β η 2βρσ ξ σ η + β 2 σ 2 ξ β = ρ σ η σ ξ Applying this with η = x t µ and ξ = x t l µ, both of zero mean and with equal variances, the best predictor for x t given x t l (and knowing no other values is x t µ ρ l ( xt l µ. The error in this prediction will be somewhat less than the zeroinformation prediction x t = µ.

3 Robert Almgren: Time Series 2 Sept. 21, Now suppose we observe all the l preceding values x t 1,..., x t l, and let us set µ = 0 for simplicity. We make a model of the form x t = β 1 x t β l x t l. We choose the coefficients β 1,..., β l to minimize the square error: [ ( min E x t ( 2 ] β 1 x t β l x t l β 1,...,β l Differentiating with respect to β j for each j = 1,... and taking the expectation, we obtain the set of l linear conditions γ j = β 1 γ j β l γ j l, j = 1,..., l. (Note that the sequence of subscripts j 1,..., j l crosses zero somewhere in the middle, so we use the definition of γ l for l < 0 as well as l > 0. That is, we solve the linear system Γ β 1. β l = γ 1. γ l, with Γ ij = γ i j. The last coefficient β l is the lth partial autocorrelation of x t with x t l, sometimes denoted PACF(l. It is the additional value for predicting x t of knowing x t l if you already know x t 1,..., x t l+1. Note that to get the partial auto-correlation at each lag, you have to solve a different linear system for each l. More fully, we might denote the lth matrix as Γ l, and the l coefficients as β l,j, the jth coefficient in the fit of order l. 2 White noise A time series {w t } is a white noise if all finite collections of elements are identically distributed, uncorrelated random variables with zero mean. That is, the series {w t } is stationary, has mean zero E(w t = 0, and has zero covariances at all nonzero lags: E(w t w s = 0 if s t.

4 Robert Almgren: Time Series 2 Sept. 21, The individual variance σ 2 = E ( wt 2 must be constant if the series is stationary. The series is a strong white noise if w s and w t are independent for s t, which is a stronger than being uncorrelated. We do not require that w t be Gaussian; for example, a sequence of coin flips w t = ±1 is white noise. That said, a Gaussian sequence is the most common example and the most useful to keep in mind, as for example by Matlab w=randn(1,n or R w <- rnorm(n. If r t is a series of asset returns, then the theory of the efficient market suggests that the sign of r t should be very difficult to predict in terms of previous values r t 1, r t 2,... (and any other data available at the time. But this does not necessarily mean that the return series is white noise, since we might be able to estimate the size of likely price changes. If σ t is our estimate for E ( rt 2, then the series w t = r t /σ t might plausibly be a white noise (with unit variance. We construct time series by applying various transformations to a white noise input, and we analyse data by finding transformations that reduce the given data set to something that passe statistical tests for white noise. That is, we investigate the relationship {x t } {w t } (1 in both directions. The relationship is between the entire sequences on both side, not on the individual values at a specific time. In the leftward direction, {w t } {x t }, this is a model that specifies how the time series of interest might be generated from a white noise series. If you like, it is a specification for a program for generating sample realizations of the sequence. Just because this model exists does not mean that the original series really was generated in that way, just that this is one possible way to construct it. In the rightward direction, {x t } {w t }, this is a technique for data analysis. You start with an empirical data set consisting of one observation of all the x t, an apply a series of transformations to get another numerical series w t. If your analysis is correct, this derived series should satisfy the empirical tests to be a white noise. This is the same as usual regression, where you should keep improving your model until no observable structure is present in the residuals. As an example, consider the random walk model x t = x t 1 + w t. Stated in this way, it is a constructive formula for generating realizations. It can also be inverted as w t = x t x t 1, and in this formulation it can be used to test the residuals and thus the hypothesis that the input data is a random walk.

5 Robert Almgren: Time Series 2 Sept. 21, Moving average models A moving average model of order q 0, MA(q, has the form x t = µ + q θ i w t i = µ + θ 0 w t + θ 1 w t θ q w t q. (2 i=0 with E(w 2 t = σ 2. In theory, x t might also depend on future values w t+j with j > 0, but we will assume our model is causal, depending only on past values. (For finite q, a two-sided model of order q is equivalent to a one-sided model of order 2q, and if you were clever you could probably define this equivalence in the limit q. Clearly the mean is E(x t = µ. Let us calculate the autocorrelation function. Setting µ = 0 for convenience, γ l = E ( x t x t l = q i,j=0 θ i θ j E ( q l w t i w t l j = σ 2 θ j θ j+l. The key ingredient is the white noise definition E ( w i w j = σ 2 δ ij. By definition, the sum is zero when the upper limit is strictly less than the lower limit, that is, when l > q. Thus an MA(q model has autocorrelation strictly zero for lags greater than the order of the model. This is apparent from looking at (2, since x t depends on w t,..., w t q and x s depends on w s,..., w s q and if s t > q these sets do not overlap, so there is no possible mechanism for x t and x s to have any relationship. By the above, the variance is E(x 2 t = γ 0 = σ 2( θ θ 2 q which is certainly finite for any finite q. In the limit q we require that the sequence be summable ( θ j exists, which is stronger than square-summable ( θ j 2 exists. Because the variance is finite, the series x t is stationary for any finite set of values θ 1,..., θ q, with only the rather weak summability condition in the case q =. j=0 Non-example: The random walk x t = x t 1 + w t can formally be written as x t = w t + w t 1 + w t 2 +. This looks like a movingaverage modelbut we do not count it as such, because the sequence of coefficients θ = ( 1, 1, 1,... is not summable.

6 Robert Almgren: Time Series 2 Sept. 21, Figure 1: Sample path of MA(1 x t = 1 2( wt + w t 1. Small dots are w t, large dots and heavy lines are x t. The MA process smooths the jaggedness of the white noise. (We take Var(w t = 2 so Var(x t = 1. Example For q = 1, let us consider x t = w t + α w t 1, E ( wt 2 = σ 2. The mean is zero, and the autocovariances are γ 0 = ( 1 + α 2 σ 2, γ 1 = α σ 2, so the correlation (the only nonzero value is ρ = ρ 1 is ρ = α 1 + α 2. Note that the maximum possible value is ρ = ± 1 2, when α = ±1, although Cauchy-Schwarz requires only ρ 1. Larger values of ρ than 1 2 are certainly possible, but not within an MA(1 model.

7 Robert Almgren: Time Series 2 Sept. 21, Figure 1 shows a sample path, with α = 1. Thus (except for a factor of 2 each value x t is the average of two successive values of w t and is hence a smoothed version of w t. In general, the PACF of an MA(q model is not zero for l > q. For the above example, to compute the PACF at lag l = 2 the matrix (setting σ 2 = 1 is ( ( γ0 γ 1 1 ρ Γ = = γ γ 1 γ 0 0 ρ 1 with β 2 Γ 1 = γ 2 1 γ 0 (1 ρ 2 ( 1 ρ. ρ 1 Then for the fit x t = β 1 x t 1 + β 2 x t 2, the optimal values β 1, β 2 are β 1 = Γ 1 γ 1 = Γ 1 ργ 0 = ρ ρ 2 ρ so the partial auto-correlation at lag 2 is β 2 = ρ2 1 ρ 2 = α α 2 + α 4. You can easily check that the 3rd order PACF is also not zero. Here is what is going on. We want to predict x t, a combination of w t and w t 1, neither of which we can observe directly no matter what values of x s we are given. Knowing the value of x t 1 is useful, because it gives us some information about w t 1, though this is mixed in with w t 2. But knowing the value x t 2 only has no value because it involves w t 2 and w t 3, which are independent of the two values we care about. However, if we already know x t 1, then additional information about x t 2 is useful, because it helps us distinguish the two components w t 1 and w t 2 in x t 1. Recovery from autocorrelation Suppose we are given a sequence of autocorrelations γ 0,..., γ q. Can we reconstruct a model that would have generated them? The answer is yes, partially. First, the autocorrelations must be zero beyond a finite order q, if a MA(q model is to have any hope. Then we solve the above

8 Robert Almgren: Time Series 2 Sept. 21, equations for θ 0,..., θ q in terms of γ 0,..., γ q, but the answer will not be unique. As an example, consider q = 1 and fix σ = 1. The system is γ 0 = θ θ 2 1, γ 1 = θ 0 θ 1. It is clear that there are no solutions if γ 1 > 1 2 γ 0, although the standard conditions on autocorrelation require only γ 1 γ 0. And if γ 1 < 1 2 γ 0, there are four distinct solutions, corresponding to changing the sign of w t and w t 1 and to interchanging w t and w t 1. Normalization Having both θ 0 and σ 2 is redundant, and we may normalize the value θ 0 = 1 (or we could require E(wt 2 = 1, but θ 0 = 1 is the convention. Thus x t = µ + w t + θ 1 w t θ q w t q, E ( wt 2 = σ 2. This is the standard form of an MA(q model, although there are conflicting conventions for the sign of the θ j. 4 Auto-regressive models Exploiting the symmetry of the relationship (1, we now consider models that can be written like (2 exchanging x t w t : w t = x t c φ 1 x t 1 φ p x t p for some p 0 and eventually including the limit p. Or, in more standard form, x t = c + φ 1 x t φ p x t p + w t. (3 This is called an autoregressive model of order p or AR(p. It is not clear that x t has a stationary solution at all, or (what is the same thing that the influence of these starting values will decay. Indeed, the random walk from last week is an AR(1 process with φ 1 = 1 and c = 0, and we know that this is not stationary. Even worse, consider an AR(1 process x t = 2x t 1 + w t, whose solution explode exponentially. Clearly, we need to figure out a general way to tell when solutions are well behaved.

9 Robert Almgren: Time Series 2 Sept. 21, Linear dynamics To understand the behavior of solutions to (3, let us use a little formal notation from the general theory of linear systems. Let us write (3 as ( Lx t = R t with ( Lx t = x t φ 1 x t 1 φ p x t p and R t = c + w t. Here L is a homogeneous linear operator acting on an entire sequence x = {x t } t=, yielding another infinite sequence y = Lx, given by the above expression. R is the inhomogeneous term on the right side, which has a constant part and a random part. As is usual for linear systems, the solution may be written x t = s= I t s R s = I j R t j j=0 where the influence function I is the infinite sequence that satisfies 1 ( LI t = δ t, I t = 0 for t < 0. This tells you how the system responds to a unit perturbation at t = 0. The entire system behavior is simply a linear combination of the response to all these perturbations. All of this says that it is enough to look for solutions to the homogeneous problem Lx = 0, or x t = φ 1 x t φ p x t p with the same coefficients φ 1,..., φ p but with the constant c set to zero and the noise term w t eliminated. We look for pure exponential solutions to this problem in the form x t = λ t where λ is a real or complex number. This is a solution if and only if P(1/λ = 0, where the characteristic polynomial of degree p is P(z = 1 φ 1 z φ p z p. 1 The Kronecker delta is δ t = 1 for t = 0 and δ t = 0 for t 0.

10 Robert Almgren: Time Series 2 Sept. 21, Since P(0 = 1, the polynomial is not identically zero, and generically it has p distinct roots. The general solution to Lx = 0 is x t = a 1 λ t a p λ t p where the coefficients a 1,..., a p are determined by the initial conditions. Typically, all of the a j will be nonzero, if only by a small amount, so any growing exponential will cause the whole solution to grow exponentially. The behavior of solutions to the linear system is determined by the locations of the zeros of P in the complex plane. There are three cases: 1. All of the zeros z have z > 1. In this case, all solutions of the form x t = λ t have λ < 1. That is, they decay exponentially. Regardless of what the initial data is (the δ t in the definition of I, the effect of each input decays exponentially, with a rate that at worst is given by max z. Solutions are bounded and the process is stationary. 2. At least one of the zeros z has z < 1. Then there is at least one special solution with λ > 1, meaning that it grows exponentially. The typical solution will almost certainly contain a component of this growing solution, and hence it will grow. The process is not stationary and no reasonable solution exists. 3. All of the zeros z have z 1, and at least one has z = 1. This is a special borderline case that we will discuss later. Since P(0 = 1, the only way to get P(z = 0 for z < 1 (Case 2 is for the coefficients φ j to be large enough. For example, if φ φ p < 1, then for z 1 we have φ 1 z + + φ p z p φ φ p < 1, so it is impossible for P(z to be zero for any z < 1. Example Take p = 1, so the model is x t = c + φx t 1 + w t. The characteristic polynomial is P(z = 1 φz, which has a single root at z = 1/φ. This is strictly greater than one in absolute value if and only if φ < 1, which is the condition we found last time. If this is true then perturbations decay exponentially; if it is false then they explode exponentially.

11 Robert Almgren: Time Series 2 Sept. 21, Reduction to MA model Any stationary AR model can be converted to an MA model of infinite order. For example, consider the AR(1 model: x t = c + φ x t 1 + w t = c + φ ( c + φx t 2 + w t 1 + wt = (1 + φc + φ 2 ( c + φx t 3 + w t 3 + wt + φw t 1 = (1 + φ + φ c + w t + φ w t 1 + φ 2 w t 2 + c = 1 φ + w t + φ w t 1 + φ 2 w t 2 + if φ < 1, which we found to be exactly the condition for the model to have a stationary distribution at all. Thus the AR(1 model is equivalent to a MA( model, in which the coefficients are given by an infinite sequence with a very simple exponential structure. You can see that the same procedure would work for any finite AR(p model, though the details would be more complicated. Mean and autocorrelations Let us calculate the moments of an AR process. First, the mean µ = E(x t satisfies µ = c + φ 1 µ + + φ p µ by stationarity and since E(w t = 0, so µ = 1 P(1 = c 1 φ 1 φ p. This is finite as long as all of the zeros of P are outsize the unit disk. For the autocorrelation, we start with an AR(1 process with c = 0. Multiplying the dynamic equation x t = φx t 1 + w t by x t, we have x 2 t = φ x t x t 1 + ( φx t 1 + w t wt. Taking expectations and using E(w t x t 1 = 0, we find γ 0 = φ γ 1 + σ 2. For any l > 0, multiplying the dynamic equation by x t l and taking expectations gives γ l = φ γ l 1

12 Robert Almgren: Time Series 2 Sept. 21, In particular, γ 1 = φγ 0 and so γ 0 = φ 2 γ 0 + σ 2 or γ 0 = σ 2 1 φ 2 which is finite for φ < 1. Furthermore, all autocorrelations of positive order are a geometric series γ l = φ l γ 0. This suggests that an AR(1 model is a very natural fit for observed data whose autocorrelations decay exponentially. We recover the coefficient φ from the decay rate, and then the noise variance σ from the overall variance. For an AR(2 process, the same procedure gives γ 0 = φ 1 γ 1 + φ 2 γ 2 + σ 2 γ 1 = φ 1 γ 0 + φ 2 γ 1 γ l = φ 1 γ l 1 + φ 2 γ l 2 for l 2. The first three of these (l = 2 are an inhomogeneous linear system, the Yule-Walker equations, to be solved for γ 0, γ 1, and γ 2. Again, for l 2, the autocorrelations solve a linear recurrence relation in increasing lag that is the same recurrence relation as for the data series in increasing time. Its solutions are therefore decaying exponentially if the original series is stationary. Conversely, if you knew the first few autocorrelations of the series, you could solve these equations to determine φ 1, φ 2, and σ 2. In general, an AR(p model gives an inhomogeneous linear system of size p + 1, the Yule-Walker equations, to be solved for the first p + 1 covariances γ 0,..., γ p, followed by a recurrence relation with the same structure as the original problem for γ p+1, γ p+2,.... It is easy to see that the partial auto-correlations of an AR(p model are zero for l > p, since the dependence on previous values is explicitly included in the model. 5 ARMA models It is natural to combine the MA(q model with the AR(p model to construct an ARMA(p,q model: x t φ 1 x t 1 φ p x t p = c + w t + θ 1 w t θ q w t q. You can get quite rich dynamics with fairly small values of p and q.