Stat 248 Lab 2: Stationarity, More EDA, Basic TS Models

Transcription

1 Stat 248 Lab 2: Stationarity, More EDA, Basic TS Models Tessa L. Childers-Day February 8, Introduction Today s section will deal with topics such as: the mean function, the auto- and cross-covariance function, stationarity, and the sample acf. This material is drawn from chapters 1 and 2 from Shumway and Stoffer s book, which is available for free online through the university library system (there is a link on the section website). 2 Stochastic Processes and Stationarity A stochastic process is a collection of random variables {X i } i I. The set I over which the index ranges is called the index set. A time series is a type of stochastic process, where the index set I is a subset of the integers; i.e., time series data is of the form {X 1, X 2, X 3,...}. The index set I of a stochastic process may be quite general though such as continuous time (where I R) or spatial processes (where I R 2 or R 3 ). We often describe a time series (or stochastic process) by its first and second moments, the mean function and the covariance function. You can think of the covariance function as the average cross-product relative to the joint distribution f(x r, X s ). Definition: [mean function] The mean function is defined as: µ Xt = E(X t ) = xf t (x)dx (2.1) where F t (x) = P (X t x) and f t (x) = Ft(x) x. Note that F t(x) is a cdf, while f t (x) is a pdf. Definition: [autocovariance function (acvf)] If {X t, t T } is a process s.t. V ar(x t ) < for each t T, then its autocovariance function γ X (.,.) is defined as: γ X (r, s) = Cov(X r, X s ) = E[(X r E(X r ))(X s E(X s ))] r, s T (2.2) The autocovariance function measures the linear dependence between two points on the same series observed at different times. If X s and X r are independent then γ X (r, s) = 0. However, if γ X (r, s) = 0 then X s and X r are NOT necessarily independent. Remark: What happens when s=r? In theory, we observe many draws from the same distribution and can use these draws to make inferences about the underlying distribution. However, we typically only observe one realization of a time series. Stationarity is a key property that allows us to draw statistical inferences from a single stochastic process. Roughly, stationarity says that the distribution of {X t } looks the same at different locations t; this allows us to do statistical inference by treating the stochastic process as multiple correlated draws from the same distribution. 1

2 Definition: [strong stationarity] A stochastic process is called strong stationary or strictly stationary if the joint distributions of (X t1,..., X tk ) and (X t1+h,..., X tk +h) are the same for all positive integers k and for all t 1,..., t k, h Z. Strict stationarity means intuitively that the graphs over two equal-length time intervals of a realization of the time series should exhibit similar statistical characteristics. For example, the proportion of ordinates not exceeding a given level x should be roughly the same for both intervals. Definition: [(weak) stationarity]: The time series {X t, t Z} with index set Z = {0, ±1, ±2,...} is said to be (weakly) stationary if: 1. E X t 2 < t Z 2. E(X t ) = µ t Z 3. γ X (r, s) = γ X (r + t, s + t) r, s, t Z Intuitively a time series is stationary if it has a finite variance such that the mean value is constant and doesn t depend on time. The covariance function depends on s and t only through the difference s t. Remark: A strictly stationary process with finite second moments is stationary. true. The inverse is not necessarily Stationary processes play a crucial role in the analysis of time series. Of course many observed time series are decidedly non-stationary in appearance. Frequently such data sets can be transformed into series which can reasonably be modeled as realizations of some stationary process. The theory of stationary processes is then used for the analysis, fitting and prediction of the resulting series. In all of this the autocovariance function is a primary tool. Remark 1: Stationarity as defined is frequently referred to in the literature as weak stationarity, covariance stationarity, stationarity in the wide sense or second-order stationarity. For us when we refer to stationarity we will think about the three properties mentioned in the definition above. Remark 2: If {X t, t Z} is stationary then γ X (r, s) = γ X (r s, 0) for all r, s Z. It is therefore convenient to redefine the autocovariance function of a stationary process as the function of just one variable: γ X (h) = γ x (h, 0) = Cov(X t+h, X t ) t, h Z The function γ X (.) will be referred as the autocovariance function of {X t } and γ X (h) as its value at lag h. Note that γ X (s, t) = γ X (t, s) for all points s and t. Elementary properties of the covariance function. If γ(.) is the autocovariance function of a stationary process {X t, t Z}, then: γ(0) 0 γ(h) γ(0) γ(h) = γ( h) h Z h Z Definition: [autocorrelation function (acf)] The autocorrelation function of a stationary time series is the function whose value at lag h is: ρ X (h) = γ x(h) γ x (0) = Corr(X t+h, X t ) t, h Z (2.3) The Cauchy-Schwarz inequality shows that 1 ρ(h) 1 for all h. Further, ρ X (h) = 0 if X t and X t+h are not correlated, and ρ X (h) = ±1 if X t+h = α 0 + α 1 X t. The value ρ X (h) is a rough measure of the ability to forecast the series at time t + h from the value at time t. 2

3 Definition: [cross-covariance function (ccvf)] If {X t, t T } and {Y t, t T } are processes s.t. V ar(x t ) < & V ar(y t ) < for each t T, then the cross-covariance function γ XY (.,.) is defined as: γ XY (r, s) = Cov(X r, Y s ) = E[(X r E(X r ))(Y s E(Y s ))] r, s T (2.4) Of course there is also a scaled version of the cross-covariance function: Definition: [cross-correlation function (ccf)] The cross-correlation function of {X t, t T } and {Y t, t T } is defined as: ρ XY (s, t) = γ XY (s, t) γx (s, s)γ Y (t, t) s, t Z (2.5) 3 Estimation of Mean and Correlation Function As we just have one realization of our time series, the assumption of stationarity becomes critical. Somehow, we must use averages over this single realization to estimate the population means and covariance functions. If a time series is stationary, the mean function is constant µ t = µ and can be estimated by the sample mean. Definition: The sample mean is defined as: ˆµ = x t = 1 n n x t (3.1) Assuming stationarity, the autocovariance and autocorrelation function can be estimated using: t=1 Definition: The sample autocovariance function is defined as: n h ˆγ(h) = n 1 ((x t+h x)(x t x)) (3.2) t=1 with ˆγ( h) = ˆγ(h) for h = 0, 1,..., n 1. Dividing by n ensures that the function is non-negative definite (allowing for linear combinations of data to have non-negative variances), although it leads to a biased estimate of γ(h). Definition: The sample autocorrelation function is defined, analogously, as: ˆρ(h) = ˆγ(h) ˆγ(0) (3.3) The sample autocorrelation function has a sampling distribution that allows us to assess whether the data comes from a completely random or white series, or whether correlations are statistically significant at some lags Large sample distribution of the acf Under general conditions, if x t is white noise, then for large n, the sample acf, ˆρ X (h) for h = 1, 2,..., H, where H is fixed but arbitrary, is approximately normally distributed with zero mean and standard deviation given by: σˆρx(h) = 1 n Remark 1 Based on this result, we obtain a rough method of assessing whether peaks in ˆρ(h) are significant by determining whether the observed peak is outside the interval ±2/ n; for a white noise sequence, approximately 95% 3

4 of the sample acf s should be within these limits. After trying to reduce a time series to a white noise series the acf s of the residuals should then lie roughly within the limits given above. A plot of these values and the corresponding intervals is often called a correlogram. Remark 2 The sample autocovariance and autocorrelation functions can be computed for non-stationary process. For data containing a trend, ˆρ(h) will exhibit slow decay as h increases, and for data with a substantial deterministic periodic, ˆρ(h) will exhibit similar behavior with the same periodicity. Remark 3 The sample cross-covariance and sample cross-correlation function are defined analogously to the sample autocovariance and sample autocorrelation function. Correlogram of 'white noise' Correlogram of 'white noise with trend' Figure 1: Left: Sample acf of Gaussian white noise. Right: Sample acf of the series generated by X t = t + Z t, where Z t is Gaussian white noise (i.e. X t is white noise with a deterministic trend t). > white.noise = as.ts(rnorm(100)) > acf(white.noise, main = "Correlogram of white noise ") > t = 1:100 > Xt = t + white.noise > acf(xt, main = "Correlogram of white noise with trend ") 4 Some Basic Series Models 4.1 White Noise A sequence of uncorrelated random variables, {Z t ; t {1,..., n}}, with mean 0 and finite variance σz 2 is called a white noise process. It can be used as a model for noise in some engineering applications. The term white noise comes from the fact that a frequency analysis of this model shows that all frequencies enter equally, something that is also the case if you investigate the frequencies contained in white light. A white noise process will be denoted as Z t WN(0, σz 2 ). 4

5 Gaussian white noise (SD = 20) Correlogram white noise Zt Figure 2: Simulation (N=1000) of Gaussian white noise with σ Z = 20. > Zt = as.ts(rnorm(1000, sd = 20)) > par(mfrow = c(1,2)) > plot(zt, main = "Gaussian white noise (SD = 20)") > acf(zt, main = "Correlogram") Note that while the definition above fixes the first two moments of the random variables Z t, the rest of their distributions are left unspecified. It is even possible that Z i and Z j have different higher moments (and hence different distributions). However, sometimes we will require the random variables Z t, besides independent, also to be identically distributed with mean 0 and variance σz 2. In that case, we will denote the series as Z IID t (0, σ 2 Z ). A wide class of stationary processes can be generated by using white noise as the noise term in a set of linear difference equations. These processes are called autoregressive-moving average (ARMA), and will be discussed next week. 4.2 Moving Average If we replace the white noise series Z t by a moving average that smoothes the series, the resulting process is called a moving average. For example, let us consider a random variable V t that will average the current value with its immediate neighbors in the past and the future. That is, V t = 1 3 (Z t 1 + Z t + Z t+1 ) for t = 2, 3,..., n 1 (4.1) This is a 3-point moving average of the series Z t. If we look at the moving average it shows a smoother version of the first series, reflecting that the slower oscillations are more apparent and some of the faster oscillations are taken out. Note that in this example all involved points have equal weights. The use of different weights is possible as well, an example of that is the general MA(q)-model which will be discussed next week. 5

6 Moving 5 point average of above white noise series Correlogram 5 point moving average Vt Moving 21 point average of above white noise series Correlogram 21 point moving average Vt Figure 3: Moving averages of the white noise process of figure 1. Above: 5-points moving average. Below: 21-points moving average. > par(mfrow = c(2,2)) > Vt = filter(zt, rep(1/5, 5), sides = 2) > plot(vt, main = "Moving 5-point average of above white noise series") > acf(vt, main="correlogram 5-point moving average", lag.max=40, na.action=na.pass) > Vt = filter(zt, rep(1/21, 21), sides = 2) > plot(vt, main = "Moving 21-point average of above white noise series") > acf(vt, main="correlogram 21-point moving average", lag.max=40, na.action=na.pass) 4.3 Auto regressions Now, let us consider the process generated by the following equation: X t = φ 1 X t 1 + φ 2 X t 2 + Z t for t = 1, 2,..., n (4.2) Such a process is called a (second order) auto regression. Here, the current value of this series depends on the actual values at different time points. Actually, in the above example, the value X t depends on all previous shocks. (Why?) This equation represents a regression or prediction of the current value X t of a time series as a function of the past two values of the series. Note that along with above equation we should also provide startup values X 0 and X 1. 6

7 First order auto regression (phi = 0.9) Correlogram first order autoregression Xt Second order auto regression (phi_1 = 1, phi_2 = 0.8) Correlogram second order autoregression Xt Figure 4: Auto regressions with the white noise process of figure 1 as input, X 0 and X 1 are set to 0. Above: φ 1 = 0.8 and φ 2 = 0. Below: φ 1 = 0.8 and φ 2 = 0. > Xt1 = filter(zt, 0.9, method = "recursive") > Xt2 = filter(zt, c(1, -0.8), method = "recursive") > par(mfrow = c(2,2)) > plot(xt1, main = "First order auto regression (phi = 0.9)") > acf(xt1, main = "Correlogram first order autoregression") > plot(xt2, main = "Second order auto regression (phi1 = 1, phi2 = -0.8)") > acf(xt2, main = "Correlogram second order autoregression") 4.4 Random Walk If we take a first order auto regression with φ 1 = 1, the resulting process is called a random walk: Y t = Y t 1 + Z t for t = 1, 2,..., n, with initial condition Y 0 = 0 (4.3) The term random walk comes from the fact that the value of the time series at time t is the value of the series at time t 1 plus a completely random movement determined by Z t. We may rewrite it as a cumulative sum of white noise variates: Y t = t Z j for t = 1, 2,..., n (4.4) A variant of the random walk model is the random walk with drift, given by: j=1 Y t = δ + Y t 1 + Z t for t = 1, 2,..., n, with initial condition Y 0 = 0 (4.5) The constant δ is called the drift. This model can also be rewritten as a cumulative sum: Y t = δt + t Z j for t = 1, 2,..., n (4.6) j=1 7

8 The random walk with drift is clearly not stationary (why?), but what about the pure random walk process? This one is also not stationarity, which can be seen by calculating its variance, which turns out to be a function of the time t: > RW1 = as.ts(cumsum(zt)) V ar(y t ) = V ar( t Z j ) = tσz 2 for t = 1, 2,..., n (4.7) j=1 Random walk Correlogram random walk RW Random walk with drift (delta = 0.2) Correlogram random walk with drift RW Figure 5: Above: Simulated random walk. Below: Simulated random walk with drift δ = 0.2. > RW2 = 0.2*(1:length(Zt)) + as.ts(cumsum(zt)) > par(mfrow = c(2,2)) > plot(rw1, main = "Random walk") > acf(rw1, main = "Correlogram random walk") > plot(rw2, main = "Random walk with drift (delta = 0.2)") > acf(rw2, main = "Correlogram random walk with drift") 5 R functions mean(), acf(), acf(type = "covariance"), ccf(), ccf(type = "covariance"), filter(). 6 Bibliography This handout has benefitted by handouts prepared by Jack Kamm, Gido van de Ven, and Irma Hernandez-Magallanes, previous GSIs for this course. Additional sources that are used, and that could be useful for you: Series: Data Analysis and Theory by David R. Brillinger Series: Theory and Methods by Peter Brockwell & Richard Davis Series Analysis and Its Applications: With R Examples by Robert Schumway & David Stoffer 8