Time Series HILARY TERM 2008 PROF. GESINE REINERT

Transcription

1 Time Series HILARY TERM 2008 PROF. GESINE REINERT reinert 1

2 Overview Chapter 1: What are time series? Types of data, examples, objectives. ffl Definitions, stationarity and autocovariances. Chapter 2: Models of stationary processes. Linear processes. Autoregressive, ffl moving average models, ARMA processes, the Backshift operator. Differencing, ARIMA processes. Second-order properties. Autocorrelation and partial autocorrelation function. Tests on sample autorcorrelations. Chapter 3: Statistical Analyis. Fitting ARIMA models: The Box-Jenkins ffl approach. Model identification, estimation, verification. Analysis in the frequency domain. Spectrum, periodogram, smoothing, filters. ffl Chapter 5: State space models. Linear models. Kalman filters. ffl Chapter 6: Nonlinear models. ARCH and stochastic volatility models. 2

3 Relevant books 1. Brockwell and Davis (2002). Introduction to Time Series and Forecasting. Springer. 2. Brockwell and Davis (1991). Time Series: Theory and methods. Springer. 3. Diggle (1990). Time Series. Clarendon Press. 4. Harvey (1993). Time Series Models. MIT Press. 5. Shumway and Stoffer (2000). Time Series Analysis and Its Applications. Springer. 6. R.L. SMITH (2001) Time Series. At 7. Venables and Ripley (2002). Modern Applied Statistics with S. Springer. 3

4 Lectures: Mondays and Fridays There will be one problem sheet, a Practical class Friday of Week 3, and an Examples class Tuesday 3-4 of Week 5. While the examples class will cover problems from the problem sheet, there may not be enough time to cover all the problems. You will benefit most from the examples class if you (attempt to) solve the problems on the sheet ahead of the examples class. Lecture notes are published at reinert/timeseries/timeseries.htm. The notes may cover more material than the lectures. The notes may be updated throughout the lecture course. Time series analysis is a very complex topic, far beyond what could be covered in an 8-hour class. Hence the goal of the class is to give a brief overview of the basics in time series analysis. Further reading is recommended. 4

5 1 What are Time Series? Many statistical methods relate to data which are independent, or at least uncorrelated. There are many practical situations where data might be correlated. This is particularly so where repeated observations on a given system are made sequentially in time. Data gathered sequentially in time are called a time series. 5

6 Examples Here are some examples in which time series arise: Economics and Finance ffl Environmental Modelling ffl Meteorology and Hydrology ffl Demographics ffl Medicine ffl Engineering ffl Quality Control ffl 6

7 The simplest form of data is a long-ish series of continuous measurements at equally spaced time points. That is observations are made at distinct points in time, these time points being equally ffl spaced ffl and, the observations may take values from a continuous distribution. The above setup could be easily generalised: for example, the times of observation need not be equally spaced in time, the observations may only take values from a discrete distribution,... 7

8 If we repeatedly observe a given system at regular time intervals, it is very likely that the observations we make will be correlated. So we cannot assume that the data constitute a random sample. The time-order in which the observations are made is vital. 8

9 Objectives of time series analysis: ffl description - summary statistics, graphs analysis and interpretation - find a model to describe the time dependence in the ffl data, can we interpret the model? forecasting or prediction - given a sample from the series, forecast the next ffl value, or the next few values control - adjust various control parameters to make the series fit closer to a ffl target adjustment - in a linear model the errors could form a time series of correlated ffl observations, and we might want to adjust estimated variances to allow for this 9

10 2 Examples: from Venables and Ripley, data from Diggle (1990) lh: a series of 48 observations at 10-minute intervals on luteinizing hormone levels for a human female lh Time 10

11 deaths: monthly deaths in the UK from a set of common lung diseases for the years 1974 to 1979 deaths year dotted series = males, dashed = females, solid line = total (We will not split the series into males and females from now on.) 11

12 1.1 Definitions Assume that the X series t runs throughout time, that (X is ) t=0;±1;±2;::: t, but is only observed at t times =1;:::;n. So we observe (X 1 ;:::;X n ). Theoretical properties refer to the underlying process t ) t2ż (X The X notations t X(t) and are interchangeable. The theory for time series is based on the assumption of second-order stationarity. Real-life data are often not stationary: e.g. they exhibit a linear trend over time, or they have a seasonal effect. So the assumptions of stationarity below apply after any trends/seasonal effects have been removed. (We will look at the issues of trends/seasonal effects later.) 12

13 1.2 Stationarity and autocovariances The process is called weakly stationary or second-order stationary if for all integers t; fi E(X t )=μ cov(x t+fi ;X fi )=fl t where μ is constant and fl t does not depend on fi. The process is strictly stationary or strongly stationary if (X t1 ;:::;X tk ) and (X t1 +fi ;:::;X tk +fi ) have the same distribution for all sets of time points t 1 ;:::;t k and all integers fi. 13

14 Notice that a process that is strictly stationary is automatically weakly stationary. The converse of this is not true in general. However, if the process is Gaussian, that is (X if ;:::;X tk ) t1 has a multivariate normal distribution for t all ;:::;t k 1, then weak stationarity does imply strong stationarity. Note that var(x t )=fl 0 and, by stationarity, fl t = fl t. The sequence (fl t ) is called the autocovariance function. The autocorrelation function (acf) (ρ t ) is given by t corr(x t+fi ;X fi )= fl t ρ : = 0 fl The acf describes the second-order properties of the time series. 14

15 We estimate fl t by c t, and ρ t by r t, where X min(n t;n) c t = 1 n s=max(1; t) [X s+t X][X s X] and r t = c t c 0 : For t>0, the covariance cov(x t+fi ;X fi ) is estimated from the n t observed ffl pairs (X t+1 ;X 1 );:::;(X n ;X n t ): If we take the usual covariance of these pairs, we would be using different estimates of the mean and variances for each of the (X subseries ;:::;X n ) t+1 (X and ;:::;X n t 1 ), whereas under the stationarity assumption these have the same mean and variance. So we X use (twice) in the above formula. 15

16 1 n t rather than, even though n t there are terms in the sum, to We use 1 n ffl ensure (c that ) t is the covariance sequence of some second-order stationary series. A plot r of t t against is called the correlogram. A (X series ) t is said to be lagged if its time axis is shifted: shifting fi by lags gives the (X series t fi ). r So t is the estimated autocorrelation at lag t; it is also called the sample autocorrelation function. 16

17 lh: autocovariance function Lag ACF (cov) Series lh 17

18 lh: autocorrelation function Lag ACF Series lh 18

19 deaths: autocorrelation function Series deaths ACF Lag 19

20 2 Models of stationary processes Assume we have a time series without trends or seasonal effects. That is, if necessary, any trends or seasonal effects have already been removed from the series. How might we construct a linear model for a time series with autocorrelation? 20

21 X t = μ + 1X c r ffl t r Linear processes The process (X t ) is called a linear process if it has a representation of the form where μ is a common mean, fc r g is a sequence of fixed constants and fffl t g are r= 1 independent random variables with mean 0 and common variance. We assume P c 2 r < 1 to ensure that the variance of X t is finite. Such a process is strictly stationary. c If r r<0 =0for it is said to be causal, i.e. the process at t time does not depend on the future, as yet unobserved, values ffl of t. The AR, MA and ARMA processes that we are now going to define are all special cases of causal linear processes. 21

22 X t = px 2.1 Autoregressive processes Assume that a current value of the series is linearly dependent upon its previous value, with some error. Then we could have the linear relationship t = ffx t 1 + ffl t X ffl where t is a white noise time series. [That is, ffl the t are a sequence of uncorrelated random variables (possibly normally distributed, but not necessarily normal) with mean 0 and ff variance.] 2 This model is called an autoregressive (AR) model, X since is regressed on itself. Here the lag of the autoregression is 1. More generally we could have an autoregressive model of order p, an AR(p) model, defined by ff i X t i + ffl t : i=1 22

23 X t = ffx t 1 + ffl t At first sight, the AR(1) process X t = ffx t 1 + ffl t is not in the linear form X t = μ + P c r ffl t r. However note that = ffl t + ff(ffl t 1 + ffx t 2 ) = ffl t + ffffl t 1 + ff 2 ffl t ff k 1 ffl t k+1 + ff k X t k ffl t + ffffl t 1 + ff 2 ffl t 2 + = which is in linear form. 23

24 If ffl t has variance ff 2, then from independence we have that t )=ff 2 + ff 2 ff ff 2(k 1) ff 2 + ff 2k Var(X t k ): Var(X The sum converges as we assume finite variance. But the sum converges only if jffj < 1. Thus jffj < 1 is a requirement for the AR(1) process to be stationary. We shall calculate the acf later. 24

25 2.2 Moving average processes Another possibility is to assume that the current value of the series is a weighted sum of past white noise terms, so for example that t = ffl t + fiffl t 1 : X Such a model is called a moving average (MA) model, X since is expressed as a weighted average of past values of the white noise series. Here the lag of the moving average is 1. We can think of the white noise series as being innovations or shocks: new stochastically uncorrelated information which appears at each time step, which is combined with other innovations (or shocks) to provide the observable series X. 25

26 X t = ffl t + Var(X t )=ff 2 + qx qx More generally we could have a moving average model of order q, an MA(q) model, defined by fi j ffl t j : j=1 If ffl t has variance ff 2, then from independence we have that fi 2 j ff 2 : We shall calculate the acf later. j=1 26

27 X t = px ff i X t i + qx fi j ffl t j 2.3 ARMA processes An autoregressive moving average process ARMA(p; q) is defined by where fi 0 =1. i=1 j=0 A slightly more general definition of an ARMA process incorporates a non-zero mean value μ, and can be obtained by X replacing t X by μ t X and t i by X t i μ above. 27

28 From its definition we see that an MA(q) process is second-order stationary for any 1 ;:::;fi q. fi However the AR(p) ARMA(p; and q) models do not necessarily define second-order stationary time series. For example, we have already seen that for an AR(1) model we need the condition < 1. This is the stationarity condition for an AR(1) process. All AR processes jffj require a condition of this type. Define, for any complex number z, the autoregressive polynomial ff (z) =1 ff 1 z ff p z p : ffi Then the stationarity condition for an AR(p) process is: all the zeros of the ffi function (z) ff lie outside the unit circle in the plane: complex This is exactly the condition that is needed fff on ;:::;ff p g 1 to ensure that the process is well-defined and stationary (see Brockwell and Davis 1991), pp

29 ff i B i! X t = ffl t 2.4 The backshift operator Define the backshift operator B by t = X t 1 ; B 2 X t = B(BX t )=X t 2 ; ::: BX We include the identity IX operator = B 0 X t = X t t. Using this notation we can write the AR(p) X process = P p i=1 ff ix t i + ffl t t as ψ I px or even more concisely i=1 ffi ff (B)X = ffl: 29

30 X t = ψ + I qx fi j B j! ffl t Recall that an MA(q) process X is = ffl t + P q j=1 fi jffl t j t. Define, for any complex number z, the moving average polynomial ffi fi (z) =1+fi 1 z + + fi q z q : Then, in operator notation, the MA(q) process can be written or j=1 X = ffi fi (B)ffl: 30

31 fl 0 = Var(X 0 ) = (1 + fi 2 )ff 2 fl 1 = Cov(X 0 ;X 1 ) ForanMA(q) process we have already noted that there is no need for a stationarity condition on the fi coefficients j, but there is a different difficulty requiring some restriction on the coefficients. Consider the MA(1) process X t = ffl t + fiffl t 1 : As ffl t has mean zero and variance ff 2, we can calculate the autocovariances to be = Cov(ffl 0 ;ffl 1 )+Cov(ffl 0 ;fiffl 0 )+Cov(fiffl 1 ;ffl 1 )+Cov(fiffl 1 ;fiffl 0 ) = Cov(ffl 0 ;fiffl 0 ) = fiff 2 ; fl k =0; k > 2: 31

32 0 =1; ρ 1 = fi ρ 2 ; ρ k =0 k > 2: 1+fi So the autocorrelations are 32

33 Now consider the identical process but fi with replaced by 1=fi. From above we can see that the autocorrelation function is unchanged by this transformation: the two processes defined fi by 1=fi and cannot be distinguished. It is customary to impose the following identifiability condition: all the zeros of the ffi function (z) fi lie outside the unit circle in the plane: complex 33

34 X t = px ff i X t i + qx fi j ffl t j The ARMA(p; q) process i=1 where fi 0 =1, can be written j=0 The conditions required are ffi ff (B)X = ffi fi (B)ffl: 1. the stationarity condition on fff 1 ;:::;ff p g 2. the identifiability condition on ffi 1 ;:::;fi q g 3. an additional identifiability condition: ffi ff (z) and ffi fi (z) have no common roots. Condition 3 is to avoid having ARMA(p; an q) model which can, in fact, be expressed as a lower order model, say as ARMA(p 1;q an 1) model. 34

35 r 2 X t = r(rx t )=X t 2X t 1 + X t Differencing The difference operator r is given by rx t = X t X t 1 These differences form a new time series rx (of length n 1 if the original series had length n). Similarly and so on. If our original time series is not stationary, we can look at the first order difference process rx, or second order r differences X, and so on. If we find that a differenced process is a stationary process, we can look for an ARMA model of that 2 differenced process. In practice if differencing is used, d usually =1, or d maybe =2, is enough. 35

36 2.6 ARIMA processes The X process t is said to be an autoregressive integrated moving average process d; q) if its dth difference r d X is an ARMA(p; q) process. ARIMA(p; ARIMA(p; d; An q) model can be written ffi ff (B)r d X = ffi fi (B)ffl or ffi ff (B)(I B) d X = ffi fi (B)ffl: 36

37 fl k = E(X t X t k ) " ( qx qx qx fi j ffl t j )( qx fi i ffl t k i ) # 2.7 Second order properties of MA(q) For the MA(q) X process = P q j=0 fi jffl t j t, fi where 0 =1, it is clear that E(X t )=0for all t. Hence, for k>0, the autocovariance function is = E j=0 i=0 fi j fi i E(ffl t j ffl t k i ): = Since the ffl t sequence is white noise, E(ffl t j ffl t k i )=0unless j = i + k. j=0 i=0 37

38 Hence the only non-zero terms in the sum are of the form ff 2 fi i fi i+k and we have fl k = 8 < : ff2 P q jkj i=0 fi i fi i+jkj jkj 6 q and the acf is obtained via ρ k = fl k =fl 0. 0 jkj >q In particular notice that the acf if zero for jkj >q. This cut-off in the acf after lag is a characteristic property of the MA process and can be used in identifying the q order of an MA process. 38

39 Simulation: MA(1) with fi =0:5 Series ma1.sim ACF Lag 39

40 Simulation: MA(2) with fi 1 = fi 2 =0:5 Series ma2.sim ACF Lag 40

41 To identify an MA(q) process: We have already seen that for an MA(q) time series, all values of the acf beyond lag are zero: i.e. ρ k =0for k>q. q So plots of the acf should show a sharp drop to near zero after the qth coefficient. This is therefore a diagnostic for an MA(q) process. 41

42 fl k = X t = px px 2.8 Second order properties of AR(p) Consider the AR(p) process ff i X t i + ffl t : For this model E(X t )=0(why?). Hence multiplying both sides of the above equation X by t k and taking expectations gives i=1 ff i fl k i ; k>0: i=1 42

43 ρ k = px ff i ρ k i ; k>0 In terms of the autocorrelations ρ k = fl k =fl 0 These are the Yule-Walker equations. i=1 The population ρ autocorrelations k are thus found by solving the Yule-Walker equations: these autocorrelations are generally all non-zero. Our present interest in the Yule-Walker equations is that we could use them to calculate ρ the k if we knew ff the i. However later we will be interested in using them to infer the values ff of i corresponding to an observed set of sample autocorrelation coefficients. 43

44 Simulation: AR(1) with ff =0:5 Series ar1.sim ACF Lag 44

45 Simulation: AR(2) with ff 1 =0:5;ff 2 =0:25 Series ar2.sim ACF Lag 45

46 To identify an AR(p) process: The AR(p) process ρ has k decaying smoothly k as increases, which can be difficult to recognize in a plot of the acf. Instead, the corresponding diagnostic for an AR(p) process is based on a quantity known as the partial autocorrelation function (pacf). The partial autocorrelation at k lag is the correlation X between t X and t k after regression X on ;:::;X t k+1 t 1. To construct these partial autocorrelations we successively fit autoregressive processes of 1; 2;::: order and, at each stage, define the partial autocorrelation a coefficient k to be the estimate of the final autoregressive coefficient: a so k is the estimate ff of k in an AR(k) process. If the underlying process is AR(p), ff then =0 k for k>p, so a plot of the pacf should show a cutoff after lag p. 46

47 ρ k = r k = px px ff i ρ jk ij a i;p r jk ij The simplest way to construct the pacf is via the sample analogues of the Yule-Walker equations for an AR(p) The sample analogue of these equations replaces ρ k by its sample value r k : i=1 k =1;:::;p where we write a i;p to emphasize that we are estimating the autoregressive i=1 ff coefficients ;:::;ff p 1 on the assumption that the underlying process is autoregressive of order p. k =1;:::;p So we p have equations in the a unknowns ;:::;a p;p 1;p, which could be solved, and the pth partial autocorrelation coefficient a is p;p. 47

48 ff 2 k = 1 n X t = kx a j;k X t j + ffl t Calculating the pacf In practice the pacf is found as follows. Consider the regression of X t on X t 1 ;:::;X t k, that is the model with ffl t independent of X 1 ;:::;X t 1. j=1 Given X data ;:::;X n 1, least squares estimates fa of ;:::;a k;k g 1;k are obtained by minimising the nx ψ t X kx a j;k X t j! 2 : t=k+1 j=1 48

49 a k;k = a j;k = a j;k 1 a k;k a k j;k 1 j =1;:::;k 1 These a j;k coefficients can be found recursively in k for k =0; 1; 2;:::. For k =0: ff 2 0 = c 0 ; a 0;0 =0, and a 1;1 = ρ(1). And then, given the a j;k 1 values, the a j;k values are given by k P k 1 j=1 a j;k 1ρ k j ρ P k 1 j=1 a j;k 1ρ j 1 and then ff 2 k = ff 2 k 1(1 a 2 k;k): 49

50 This recursive method is the Levinson-Durbin recursion. The a k;k value is the kth sample partial correlation coefficient. In the case of a Gaussian process, we have the interpretation that a k;k = corr(x t ;X t k j X t 1 ;:::;X t k+1 ): If the process X t is genuinely an AR(p) process, then a k;k =0for k>p. So a plot of the pacf should show a sharp drop to near zero after lag p, and this is a diagnostic for identifying an AR(p). 50

51 Simulation: AR(1) with ff =0:5 Partial ACF Series ar1.sim Lag 51

52 Simulation: AR(2) with ff 1 =0:5;ff 2 =0:25 Series ar2.sim Partial ACF Lag 52

53 Simulation: MA(1) with fi =0:5 Partial ACF Series ma1.sim Lag 53

54 Simulation: MA(2) with fi 1 = fi 2 =0:5 Series ma2.sim Partial ACF Lag 54

55 Tests on sample autocorrelations To determine whether the values of the acf, or the pacf, are negligible, we can use the approximation that they each have a standard deviation of 1= around n. So this would give p p n as approximate confidence bounds (2 is an ±2= approximation to 1.96). In R these are shown as blue dotted lines. Values outside the ±2= range n can be regarded as significant at about the 5% level. But if a large number of p k values, say, are calculated it is likely that some will r exceed this threshold even if the underlying time series is a white noise sequence. Interpretation is also complicated by the fact that r the k are not independently distributed. The probability of any r one k lying ±2= outside n depends on the values of the other p k. r 55

56 3 Statistical Analysis 3.1 Fitting ARIMA models: The Box-Jenkins approach The Box-Jenkins approach to fitting ARIMA models can be divided into three parts: Identification; ffl Estimation; ffl Verification. ffl 56

57 3.1.1 Identification This refers to initial preprocessing of the data to make it stationary, and choosing plausible values p of q and (which can of course be adjusted as model fitting progresses). To assess whether the data come from a stationary process we can look at the data: e.g. a time plot as we looked at for the lh series; ffl ffl consider transforming it (e.g. by taking logs;) consider if we need to difference the series to make it stationary. ffl For stationarity the acf should decay to zero fairly rapidly. If this is not true, then try differencing the series, and maybe a second time if necessary. (In practice it is rare to go d beyond =2stages of differencing.) 57

58 The next step is initial identification p of and q. For this we use the acf and the pacf, recalling that ffl for an MA(q) series, the acf is zero beyond lag q; ffl for an AR(p) series, the pacf is zero beyond lag p. We can use plots of the acf/pacf and the approximate ±2= p n confidence bounds. 58

59 r k = X t = px px ff i X t i + ffl t Estimation: AR processes For the AR(p) process we have the Yule-Walker ρ equations = P p i=1 ff iρ ji kj k, for k>0. i=1 We fit the parameters ff 1 ;:::;ff p by solving These p are equations for p the ff unknowns ;:::;ff p 1 which, as before, can be solved using a Levinson-Durbin recursion. i=1 ff i r ji kj ; k =1;:::;p 59

60 bff 2 p = 1 n The Levinson-Durbin recursion gives the residual variance nx ψ t X px bff j X t j! 2 : This can be used to guide our selection of the appropriate order p. Define an approximate log likelihood by t=p+1 j=1 = n log(bff 2 p): 2logL Then this can be used for likelihood ratio tests. Alternatively, p can be chosen by minimising AIC where AIC = 2logL +2k and k = p is the number of unknown parameters in the model. 60

61 p n ^ffmm (X If )t t is a causal AR(p) process with WN(0;ff i.i.d. ffl ), then (see Brockwell and Davis (1991), p.241) then the Yule-Walker estimator 2 is optimal with respect to the ^ff normal distribution. Moreover (Brockwell and Davis (1991), p.241) for the pacf of a causal AR(p) process we have that, for m>p, is asymptotically standard normal. However, the elements of the vector ^ff m =(^ff 1m ;:::;^ff mm ) are in general not asymptotically uncorrelated. 61

62 f (X 1 ;:::;X n )=f (X 1 ) ny Estimation: ARMA processes Now we consider ARMA(p; an q) process. If we assume a parametric model for the white noise this parametric model will be that of Gaussian white noise we can use maximum likelihood. We rely on the prediction error decomposition. That X is, ;:::;X n 1 have joint density f (X t j X 1 ;:::;X t 1 ): Suppose the conditional distribution of X t given X 1 ;:::;X t 1 is normal with t=2 mean b Xt and variance P t 1, and suppose that X 1 ο N ( b X1 ;P 0 ). (This is as for the Kalman filter see later.) 62

63 ( +logp t 1 + (X t b Yt ) 2 log(2ß) Then for the log likelihood we obtain 2logL = nx P t 1 ) : t=1 b Here Xt P and t 1 are functions of the ff parameters ;:::;ff p ;fi 1 ;:::;fi q 1, and so maximum likelihood estimators can be found (numerically) by 2logL minimising with respect to these parameters. The matrix of second derivatives 2 log of L, evaluated at the mle, is the observed information matrix, and its inverse is an approximation to the covariance matrix of the estimators. Hence we can obtain approximate standard errors for the parameters from this matrix. 63

64 In practice, for AR(p) for example, the calculation is often simplified if we condition on the m first values of the series for some small m. That is, we use a conditional likelihood, and so the sum in the expression 2 log L for is taken t = m over +1to n. ForanAR(p) we would use some small value of m > m, p. When comparing models with different numbers of parameters, it is important to use the same value of m, in particular when minimising = 2logL + 2(p + AIC q). In R this corresponds to n.cond keeping in arima the command fixed when comparing the AIC of several models. 64

65 3.1.4 Verification The third step is to check whether the model fits the data. Two main techniques for model verification are Overfitting: add extra parameters to the model and use likelihood ratio or t tests ffl to check that they are not significant. Residual analysis: calculate residuals from the fitted model and plot their acf, ffl pacf, spectral density estimates, etc, to check that they are consistent with white noise. 65

66 KX r 2 k 2 k r k : n Portmanteau test of white noise A useful test for the residuals is the Box-Pierce portmanteau test. This is based on Q = n where K>p+ q but much smaller than n, and r k is the acf of the residual series. If the model is correct then, approximately, k=1 Q ο χ 2 K p q so we can base a test on this: we would reject the model at level ff if Q>χ 2 K p q(1 ff). An improved test is the Box-Ljung procedure which replaces Q by ~Q = n(n +2) KX The distribution of ~ Q is closer to a χ 2 K p q than that of Q. 66 k=1

67 3.2 Analysis in the frequency domain We can consider representing the variability in a time series in terms of harmonic components at various frequencies. For example, a very simple model for a time X series t exhibiting cyclic fluctuations with a known p period, say, is t = ff cos(!t) +fisin(!t) +ffl t X ffl where t is a white noise! =2ß=p sequence, is the known frequency of the cyclic fluctuations, ff and fi and are parameters (which we might want to estimate). Examining the second-order properties of a time series via autocovariances/autocorrelations is analysis in the time domain. What we are about to look at now, examining the second-order properties by considering the frequency components of a series is analysis in the frequency domain. 67

68 fl k = Z ß e ik df ( ) The spectrum Suppose we have a stationary time series X t with autocovariances (fl k ). For any sequence of (fl autocovariances ) k generated by a stationary process, there exists a F function such that ß where F is the unique function on [ ß; ß] such that 1. F ( ß) =0 2. F is non-decreasing and right-continuous 3. the increments F of are symmetric about zero, meaning that for 6 a<b6 ß, 0 F (b) F (a) =F ( a) F ( b): 68

69 fl k = Z ß The F function is called the spectral distribution function or F spectrum. has many of the properties of a probability distribution function, which helps explain its name, F (ß) but =1is not required. The interpretation is that, for 0 6 a<b6 ß, F (b) F (a) measures the contribution to the total variability of the process within the frequency range b. a< 6 F If is everywhere continuous and differentiable, then f ( ) = ( ) df d is called the spectral density function and we have e ik f ( )d : ß 69

70 1X 1X It P jfl k j < 1, then it can be shown that f always exists and is given by fl k cos( k): ( ) = 1 f 2ß k e i k = fl 0 fl + 1 ß 2ß By the symmetry of fl k, f ( ) =f ( ). k= 1 k=1 From the mathematical point of view, the spectrum and acf contain equivalent information concerning the underlying stationary random (X sequence t ). However, the spectrum has a more tangible interpretation in terms of the inherent tendency for realizations (X of ) t to exhibit cyclic variations about the mean. [Note that some authors put constants 2ß of in different places. For example, some put a factor 1=(2ß) of in the integral expression fl for k in terms F; of f, and then they don t need 1=(2ß) a factor when f giving in terms fl of k.] 70

71 Example: WN(0;ff 2 ) Here, fl 0 = ff 2, fl k =0for k 6= 0, and so we have immediately f ( ) = ff2 for all which is independent of. 2ß The fact that the spectral density is constant means that all frequencies are equally present, and this is why the sequence is called white noise. The converse also holds: i.e. a process is white noise if and only if its spectral density is constant. 71

72 1X 1 + ffei 1X ff k e i k Example: AR(1): X t = ffx t 1 + ffl t. Here fl 0 = ff 2 =(1 ff 2 ) and fl k = ff jkj fl 0 for k 6= 0. So ( ) = 1 f fl 0 2ß 1X ff jkj e i k k= 1 fl 0 = + 1 2ß fl 0 2ß k e i k + 1 ff fl 0 2ß k=1 k=1 fl 0 = 2ß ffe i + i 1 ffe i ffe 1 0 (1 ff 2 ) fl 2ff cos + ff 2 ) 2ß(1 = 2 ff 2ff cos + ff 2 ) 2ß(1 = where we used e i + e i =2cos. 72

73 Simulation: AR(1) with ff =0:5 Series: ar1.sim AR (1) spectrum spectrum frequency 73

74 Simulation: AR(1) with ff = 0:5 Series: ar1b.sim AR (2) spectrum spectrum frequency 74

75 Plotting the spectral density f ( ), we see that in the case ff > 0 the spectral density f ( ) is a decreasing function of : that is, the power is concentrated at low frequencies, corresponding to gradual long-range fluctuations. ff < 0 For the spectral f ( ) density increases as a function of : that is, the power is concentrated at high frequencies, which reflects the fact that such a process tends to oscillate. 75

76 X t = px ff i X t i + qx fi j ffl t j ARMA(p; q) process The spectral density for an ARMA(p,q) process is related to the AR and MA ffi polynomials (z) ff ffi and fi (z). i=1 j=0 The spectral density of X t is f ( ) = ff2 fi (e i )j 2 jffi ff (e i )j 2 : jffi 2ß 76

77 f ( ) = ff2 Example: AR(1) Here ffi ff (z) =1 ffz and ffi fi (z) =1, so, for ß 6 <ß, 2ß j1 ffe i j 2 ff2 = j1 ff cos + iff sin j 2 2ß ff2 = f(1 ff cos )2 +(ffsin ) 2 g 1 2ß as calculated before. = 2 ff 2ff cos + ff 2 ) 2ß(1 77

78 ff2 ffl = (1 + 2 cos( ) + 2 ): 2ß Example: MA(1) Here ffi ff (z) =1;ffi fi (z) =1+ z, and we obtain, for ß 6 <ß, ( ) = ff2 ffl f j1+ e i j 2 2ß 78

79 Plotting the spectral f density ( ), we would see that in the >0 case the spectral density is large for low frequencies, small for high frequencies. This is not surprising, as we have short-range positive correlation, smoothing the series. <0 For the spectral density is large around high frequencies, and small for low frequencies; the series fluctuates rapidly about its mean value. Thus, to a coarse order, the qualitative behaviour of the spectral density is similar to that of an AR(1) spectral density. 79

80 ) 2 ( nx t= The Periodogram To estimate the spectral density we use the periodogram. For a! frequency we compute the squared correlation between the time series and the sine/cosine waves of frequency!. The I(!) periodogram is given by = 1 I(!) 2ßn fi fi fi fi nx fi X i!t fifififi 2 t e fi t=1 1 "( nx X = t sin(!t) t=1 2ßn ) 2 # : cos(!t) t X + 80

81 I(!) = 1 c t = Z ß 1X c t e i!t = c 0 1X The periodogram is related to the autocovariance function by c t cos(!t); 2ß 2ß + 1 ß t=1 t= 1 e i!t I(!)d!: So the periodogram and the autocovariance function contain the same information. ß For the purposes of interpretation, sometimes one will be easier to interpret, other times the other will be easier to interpret. 81

82 Simulation: AR(1) with ff =0:5 Series: ar1.sim Raw Periodogram spectrum 1e 03 1e 02 1e 01 1e+00 1e frequency bandwidth =

83 Simulation: AR(1) with ff = 0:5 Series: ar1b.sim Raw Periodogram spectrum 1e 03 1e 02 1e 01 1e+00 1e frequency bandwidth =

84 Simulation: MA(1) with fi =0:5 Series: ma1.sim Raw Periodogram spectrum 1e 03 1e 02 1e 01 1e+00 1e frequency bandwidth =

85 From asymptotic theory, at Fourier! =! frequencies j j =1; =2ßj=n, 2;:::, the periodogram fi(! ordinates );I(! 2 );:::g 1 are approximately independent with ff (! means );f(! 2 1 );:::g. That is for! these I(!) ο f (!)E where E is an exponential distribution with mean 1. Note var[i(!)] ß f (!) that, which does not tend to zero as n!1.soi(!) is NOT a consistent estimator. 2 In addition, the independence of the periodogram ordinates at different Fourier frequencies suggests that the sample periodogram, as a function of!, will be extremely irregular. For this reason smoothing is often applied, for instance using a moving average, or more generally a smoothing kernel. 85

86 3.2.3 Smoothing The idea behind smoothing is to take weighted averages over neighbouring frequencies in order to reduce the variability associated with individual periodogram values. The main form of a smoothed esimator is given by 1 =Z ^f K! (!) h I( )d : h K Here is some kernel (= function a probability density function), for example a standard normal pdf, h and is the bandwidth. The h bandwidth affects the degree to which this process smooths the periodogram. h = Small a little smoothing, h = large a lot of smoothing. 86

87 Z!j In practice, the smoothed esimate ^f (!) will be evaluated by the sum ^f (!) = 1 K! h X I( )d j! j 1 h X ß 2ß n 1 K!j! h I(! j ): h As the degree of h smoothing increases, the variance decreases but the bias increases. j 87

88 U (!) = The cumulative periodogram U (!) is defined by X bn=2c X I(! k ) = I(! k ): This can be used to test residuals in a fitted model, for example. If we hope that our residual series is white noise, the the cumulative periodogram of the residuals should increase linearly: i.e. we can plot the cumulative periodogram (in R) and look to see if the plot is an approximate straight line. 0<! k 6! 1 88

89 Example: Brockwell & Davis (p 339, 340) Data generated by t = cos(ßt=3) + ffl t t = :::;100 X fffl where g t is Gaussian white noise with variance 1. Peak in the periodogram at! 17 =0:34ß. [Figure from B& D] 89

90 lh Example series: lh Time 90

91 deaths Example series: deaths year 91

92 lh: unsmoothed periodogram lh: Raw Periodogram spectrum (db) frequency bandwidth = , 95% C.I. is ( 6.26,16.36)dB 92

93 deaths: unsmoothed periodogram deaths: Raw Periodogram spectrum (db) frequency bandwidth = , 95% C.I. is ( 6.26,16.36)dB 93

94 Suppose we have estimated the periodogram values I(! 1 );I(! 2 );:::, where j =2ßj=n, j =1; 2;:::.! An example of a simple way to smooth is to use a moving average, and so estimate j ) by I(! 1 I(! + j 4) j 3) +I(! 1 )+ + I(! j+3 )] + 1 j 2 I(! j+4): Observe that the sum of the weights above (i.e. the s and the s) is 1. [I(! Keeping the sum of weights equal to 1, this process could be modified by using more, or I(! fewer, ) k values to I(! estimate j ). Also, this smoothing process could be repeated. 94

95 If a series is (approximately) periodic, say with! frequency 0, then periodogram will show a peak near this frequency. It may well also show smaller peaks at 2! frequencies ; 3! 0 0 ;:::. The integer multiples! of 0 are called its harmonics, and the secondary peaks at these high frequencies arise because the cyclic variation in the original series is non-sinusoidal. (So a situation like this warns against interpreting multiple peaks in the periodogram as indicating the presence of several distinct cyclic mechanisms in the underlying process.) 95

96 In R, smoothing is controlled by the option spans to the spectrum function. The unsmoothed periodogram (above) was obtained via spectrum(lh) The smoothed versions below are spectrum(lh, spans = 3) spectrum(lh, spans = c(3,3)) spectrum(lh, spans = c(3,5)) All of the examples, above and below, from Venables & Ripley. V & R advise: ffl trial and error needed to choose the spans; ffl spans should be odd integers; ffl use at least two, which are different, to get a smooth plot. 96

97 lh: Smoothed Periodogram, spans=3 spectrum (db) frequency bandwidth = , 95% C.I. is ( 4.32, 7.73)dB 97

98 lh: Smoothed Periodogram, spans=c(3,3) spectrum (db) frequency bandwidth = , 95% C.I. is ( 3.81, 6.24)dB 98

99 lh: Smoothed Periodogram, spans=c(3,5) spectrum (db) frequency bandwidth = , 95% C.I. is ( 3.29, 4.95)dB 99

100 deaths: Smoothed Periodogram, spans=c(3,3) spectrum (db) frequency bandwidth = 0.173, 95% C.I. is ( 3.81, 6.24)dB 100

101 deaths: Smoothed Periodogram, spans=c(3,5) spectrum (db) frequency bandwidth = 0.241, 95% C.I. is ( 3.29, 4.95)dB 101

102 deaths: Smoothed Periodogram, spans=c(5,7) spectrum (db) frequency bandwidth = 0.363, 95% C.I. is ( 2.74, 3.82)dB 102

103 lh: cumulative periodogram Series: lh frequency 103

104 deaths: cumulative periodogram Series: deaths frequency 104

105 3.3 Model fitting using time and frequency domain Fitting ARMA models The value of ARMA processes lies primarily in their ability to approximate a wide range of second-order behaviour using only a small number of parameters. Occasionally, we may be able to justify ARMA processes in terms of the basic mechanisms generating the data. But more frequently, they are used as a means of summarising a time series by a few well-chosen summary statistics: i.e. the parameters of the ARMA process. 105

106 Now consider fitting an AR model to the lh series. Look at the pacf: Series lh Partial ACF Lag 106

107 Fit an AR(1) model: <- ar(lh, F, 1) lh.ar1 The fitted model is: with ff 2 =0:21. One residual plot we could look at is X t =0:58X t 1 + ffl t cpgram(lh.ar1$resid) 107

108 lh: cumulative periodogram of residuals from AR(1) model AR(1) fit to lh frequency 108

109 X t =0:65X t 1 0:06X t 2 0:23X t 3 + ffl t Also try select the order of the model using AIC: lh.ar <- ar(lh, order.max = 9) lh.ar$order lh.ar$aic This selects the AR(3) model: with ff 2 =0:20. The same order is selected when using lh.ar <- ar(lh, order.max = 20) lh.ar$order 109

110 lh: cumulative periodogram of residuals from AR(3) model AR(3) fit to lh frequency 110

111 By default, ar fits by using the Yule-Walker equations. We can also use in library(mass) arima to fit these models using maximum likelihood. (Examples in Venables & Ripley, and in the practical class) The tsdiag function produces diagnostic residuals plots. As mentioned in a previous lecture, the p-values from the Ljung-Box statistic are of concern if they go below 0.05 (marked with a dotted line on the plot). 111

112 lh: diagnostic plots from AR(1) model Standardized Residuals Time ACF of Residuals ACF Lag p values for Ljung Box statistic p value lag 112

113 lh: diagnostic plots from AR(3) model Standardized Residuals Time ACF of Residuals ACF Lag p values for Ljung Box statistic p value lag 113

114 3.3.2 Estimation and elimination of trend and seasonal components The first step in the analysis of any time series is to plot the data. If there are any apparent discontinuities, such as a sudden change of level, it may be advisable to analyse the series by first breaking it into a homogeneous segments. We can think of a simple model of a time series as comprising ffl deterministic components, i.e. trend and seasonal components ffl plus a random or stochastic component which shows no informative pattern. 114

115 m t = trend component (or mean level) at time t; s t = seasonal component at time t; We might write such a decomposition model as the additive model where X t = m t + s t + Z t t = random noise component at time t: Z Here the m trend t is a slowly changing function of t, and d if is the number of observations in a complete cycle s then = s t d t. In some applications a multiplicative model may be appropriate X t = m t s t Z t : After taking logs, this becomes the previous additive model. 115

116 It is often possible to look at a time plot of the series to spot trend and seasonal behaviour. We might look for a linear trend in the first instance, though in many applications non-linear trend is also of interest and present. Periodic behaviour is also relatively straightforward to spot. However, if there are two or more cycles operating at different periods in a time series, then it may be difficult to detect such cycles by eye. A formal Fourier analysis can help. The presence of both trend and seasonality together can make it more difficult to detect one or the other by eye. 116

117 Example: Box and Jenkins airline data. Monthly totals (thousands) of international airline passengers, 1949 to AirPassengers Time 117

118 airpass.log <- log(airpassengers) ts.plot(airpass.log) airpass.log Time 118

119 We can aim to estimate and extract the deterministic m components t s and t, and hope that the residual or noise Z component t turns out to be a stationary process. We can then try to fit an ARMA process, for example, Z to t. An alternative approach (Box-Jenkins) is to apply the difference r operator repeatedly to the X series t until the differenced series resembles a realization of a stationary process, and then fit an ARMA model to the suitably differenced series. 119

120 3.3.3 Elimination of trend when there is no seasonal component The model is t = m t + Z t X where we can E(Z assume t )=0. 120

121 1: Fit a Parametric Relationship We can take m t to be the linear trend m t = ff 0 + ff 1 t, or some similar polynomial trend, and estimate m t by minimising P (X t m t ) 2 with respect to ff 0 ;ff 1. Then consider fitting stationary models to Y t = X t bm t, where bm t = bff 0 + bff 1 t. Non-linear trends are also possible of course, say log m t = ff 0 + ff 1 k t (0 <k<1), t = ff 0 =(1 + ff 1 e ff 2t m ),... In practice, fitting a single parametric relationship to an entire time series is unrealistic, so we may fit such curves as these locally, by allowing the ff parameters to vary (slowly) with time. The resulting series Y t = X t bm t is the detrended time series. 121

122 Fit a linear trend: airpass.log time step 122

123 The detrended time series: time step 123

124 2: Smoothing If the aim is to provide an estimate of the local trend in a time series, then we can apply a moving average. That is, take a small sequence of the series values X t q ;:::;X t ;:::;X t+q, and compute a (weighted) average of them to obtain a smoothed series value at time t, say bm t, where t = 1 bm +1 2q qx X t+j : j= q It is useful to think fbm of g t as a process obtained f b from g Xt by application of a linear bm filter = P 1 j= 1 a jx t+j t, with a weights =1=(2q j q 6 j 6 +1), q, and a j =0, jjj >q. 124

125 This filter is a low pass filter since it takes X data t and removes from it the rapidly fluctuating Y component = X t bm t t, to leave the slowly varying estimated trend bm term t. We should not q choose too large since, m if t is not linear, although the filtered process will be smooth, it will not be a good estimate m of t. If we apply two filters in succession, for example to progressively smooth a series, we are said to be using a convolution of the filters. 125

126 (a 0 ;a 1 ;:::;a 7 )= 1 a j = a j jjj 6 7 a j =0 jjj > 7 By careful choice of the a weights j, it is possible to design a filter that will not only be effective in attenuating noise from the data, but which will also allow a larger class of trend functions. Spencer s 15-point filter has weights 320 (74; 67; 46; 21; 3; 5; 6; 3) and has the property that a cubic polynomial passes through the filter undistorted. spencer.wts <- c(-3,-6,-5,3,21,46,67,74,67,46,21,3,-5,-6,-3)/320 airpass.filt <- filter(airpass.log, spencer.wts) ts.plot(airpass.log, airpass.filt, lty=c(2,1)) 126

127 Original series and filtered series using Spencer s 15-point filter: log(airpassengers) Time 127

128 Detrended series via filtering: Time 128 airpass.log airpass.filt

129 3: Differencing Recall that the difference operator rx is = X t X t 1 t. Note that differencing is a special case of applying a linear filter. We can think of differencing as a sample derivative. If we start with a linear function, then differentiation yields a constant function, while if we start with a quadratic function we need to differentiate twice to get to a constant function. Similarly, if a time series has a linear trend, differencing the series once will remove it, while if the series has a quadratic trend we would need to difference twice to remove the trend. 129

130 Detrended series via differencing: Year 130

131 3.4 Seasonality After removing trend, we can remove seasonality. (Above, all detrended versions of the airline data clearly still have a seasonal component.) 1: Block averaging The simplest way to remove seasonality is to average the observations at the same point in each repetition of the cycle (for example, for monthly data average all the January values) and subtract that average from the values at those respective points in the cycle. 2: Seasonal differencing The seasonal difference operator is r s X t = X t X t s where s is the period of the seasonal cycle. Seasonal differencing will remove seasonality in the same way that ordinary differencing will remove a polynomial trend. 131

132 airpass.diff<-diff(airpass.log) airpass.diff2 <- diff(airpass.diff, lag=12) ts.plot(airpass.diff2) 132

133 airpass.diff Time 133

134 After differencing at lag 1 (to remove trend), then at lag 12 (to remove seasonal effects), log(airpassengers) the series appears stationary. That is, the series rr 12 X, or equivalently the series (1 B)(1 B 12 )X, appears stationary. R has a stl function which you can use to estimate and remove trend and seasonality using loess. is a complex function, you should consult the online documentation before you stl use it. The time series chapter of Venables & Ripley contains examples of how to use stl. As with all aspects of that chapter, it would be a good idea for you to work through the examples there. We could now look to fit an ARMA model rr to 12 X, or to the residual component extracted by stl. 134

135 px ff i X t i = ffl t + qx fi j ffl t j Seasonal ARIMA models Recall that X is an ARMA(p; q) process if X t i=1 j=1 and X is an ARIMA(p; d; q) process if r d X is ARMA(p; q). In shorthand notation, these processes are ffi ff (B)X = ffi fi (B)ffl and ffi ff (B)r d X = ffi fi (B)ffl: Suppose we have monthly observations, so that seasonal patterns repeat every s =12observations. Then we may typically expect X t to depend on such terms as X t 12, and maybe X t 24, as well as X t 1 ;X t 2 ;:::. 135

136 A general seasonal ARIMA (SARIMA) model, is Φ p (B)Φ P (B s )Y =Φ q (B)Φ Q (B s )ffl where Φp; ΦP ; Φq; ΦQ are polynomials of orders p; P; q; Q and where Here: Y =(1 B) d (1 B s ) D X: ffl s is the number of observarions per season, so s =12for monthly data; D is the order of seasonal differencing, i.e. differencing at lag s (we were ffl content D with =1for the air passenger data); d is the order of ordinary differencing (we were content with d =1for the air ffl passenger data). This model is often referred to as an ARIMA((p; d; q) (P; D; Q)s) model. 136

137 Examples 1. Consider a ARIMA model of order (1; 0; 0) (0; 1; 1) 12. This model can be written where (1 ffb)y t =(1+fiB 12 )ffl t Y t = X t X t 12 : 137

138 2. The airline model (so named because of its relevance to the air passenger data) is a ARIMA model of (0; 1; 1) (0; 1; 1) order 12. This model can be written t =(1+fi 1 B)(1 + fi 2 B 12 )ffl t Y Y where = rr 12 X t is the series we obtained after differencing to reach stationarity, i.e. one step of ordinary differencing, plus one step of seasonal (lag 12) differencing. 138

139 4 State space models State-space models assume that the (X observations ) t t are incomplete and noisy functions of some underlying unobservable (Y process ) t t, called the state process, which is assumed to have a simple Markovian dynamics. The general state space model is described by Y 1. ;Y 1 ;Y 2 0 ;:::is a Markov chain 2. Conditionally fy on g t t, X the t s are independent, X and t depends Y on t only. When the state variables are discrete, one usually calls this model a hidden Markov model; the term state space model is mainly used for continuous state variables. 139

140 Y t = g t (Y t 1 ;v t ) 4.1 The linear state space model A prominent role is played by the linear state space model Y t = G t Y t 1 + v t (1) t = H t Y t + w t ; (2) X G where t H and t are deterministic matrices, (v and ) t t fw and g t t are two independent white noise sequences v with t w and t being mean zero and having covariance V matrices t and W 2 t, respectively. The general case, 2 t = h t (Y t ;w t ); X is much more flexible. Also, multivariate models are available. The typical question on state space models is the estimation or the prediction of the (Y states )t t in terms of the observed data (X points ) t t. 140

141 Y t = ffiy t 1 + v t X t ffix t 1 = Y t ffiy t 1 + w t ffiw t 1 Example. Suppose the model (v where ) t t (w and ) t t are two independent white noise sequences v with t w and t being mean zero and having V covariance t and W 2 t, respectively. Then 2 X t = Y t + w t ; = v t + w t ffiw t 1 : The right-hand side shows that all correlations at lags > 1 are zero. Hence the right-hand side is equivalent to an MA(1) model, and X thus t follows an ARMA(1,1)-model. 141