Time series and spectral analysis. Peter F. Craigmile

Transcription

1 Time series and spectral analysis Peter F. Craigmile Summer School on Extreme Value Modeling and Water Resources Universite Lyon 1, France Jun 2016 Thank you to The Camille Jordan Institute, the MultiRisk LEFE project, and the Labex Milyon. My research is partially supported by the US National Science Foundation under grants NSF-DMS and NSF-SES

2 Analyzing time series A time series is a set of observations made sequentially in time. R. A. Fisher: One damned thing after another. Time series analysis is the area of statistics which deals with analyzing dependencies between different observations in time. If we ignore the dependencies that we observe in time series data, then we can be led to incorrect statistical inferences. References: Good introductory text: Brockwell and Davis [2002] The theory version: Brockwell and Davis [1991] Other popular textbooks: Cryer and Chan [2008], Shumway and Stoffer [2010] 2

3 Time series processes and data A time series process (model) is a stochastic process {X t : t T } where the index set T is usually either a subset of the integers, Z (discrete-time time series), or a subset of the reals, R (continuous-time time series). For these lectures we will focus on discrete-time time series. A given realization or sample path of the process, {x t }, yields our time series data or just the time series. Sometimes a time series model will only be summaries of the process (for example, the means and covariances of the RVs). 3

4 Application 1: Global temperature series from Berkeley Earth project [Rohde et al., 2013] (a) (b) Temperature anomaly ( C) Estimated standard error ( C) Year Year (a) Global annual mean temperature anomalies with respect to the climatology. The gray shaded region denotes simultaneous 95% confidence intervals for the mean, using estimated standard errors from Rohde et al. [2013]; (b) A plot of the estimated standard errors for the mean by year. 4

5 Application 2: River discharge series Average monthly discharge Year Log10 average monthly discharge Year Average monthly discharge (in cubic feet per second), at Alum Creek, Columbus, Ohio, USA. Source: United States Geological Survey, 5

6 Exploratory data analysis (EDA) for time series Start with time series plot of {x t : t = 1,..., n} versus t. Carefully think about the time scale and the axes. Look for the following: 1. Are there changes in behavior over time? (e.g., are there shifts in mean and/or variance?) This is an indicator of nonstationarity often we try to model the nonstationarity, remove it, and try to model the remainder. 2. Are there outliers? (unusual values with respect to the rest of the data). 3. Are some data missing in time? (complicates EDA and analysis). 4. Is the data non-gaussian? Sometimes we can just transform the data, or consider non-gaussian time series models (e.g., Poisson or binomial time series models) 6

7 Classifying features of times series 1. Trend 2. Seasonal or periodic components 3. The noise (also called the residual, irregular, error, or random term). Detecting and modeling changepoints are currently in vogue. 7

8 Combining trend, seasonality, and noise Most common: The additive model, x t = µ t + s t + η t. Also common: the multiplicative model x t = µ t s t η t. If all the variables are positive then we obtain the additive model by taking logarithms: log x t = log µ t + log s t + log η t. Often want to transform back so that we can interpret on the original measurement scale. 8

9 Objectives of time series analysis 1. Analysis and modeling 2. Forecasting and prediction 3. Adjustment/transformation 4. Simulation/emulation 5. Control 9

10 Gaussian processes Gaussian processes are time series processes, for which all finite-dimensional distributions are multivariate normal. The joint probability density function of X = (X 1,..., X n ) T at x = (x 1,..., x n ) T is ( f X (x) = (2π) n/2 det(σ ) 1/2 exp 1 ) 2 (x µ)t Σ 1 (x µ), where µ = (µ 1,..., µ n ) T and the (t, t ) element of the covariance matrix Σ is cov(x t, X t ) 10

11 Strictly stationary processes Processes in which the joint distribution of the process at any collection of time points are unaffected by time shifts: Formally {X t : t T } is strictly stationary (also called narrow-sense) when (X t1,..., X tn ) is equal in distribution to (X t1 +h,..., X tn +h), for all integers h and n 1, and time points {t k T }. 11

12 (Weakly) stationary processes {X t : t T } is (weakly) stationary if (i) E(X t ) = µ X for some constant µ X which does not depend on t; and (ii) cov(x t, X t+h ) = γ X (h), a finite constant that can depend on the (time) lag h but not on t. Other terms for weakly stationary include second-order, covariance, wide-sense. γ X ( ) is called the autocovariance function (ACVF) of the stationary process. Also useful: the unitless autocorrelation function (ACF) of the stationary process, ρ X (h) = corr(x t, X t+h ) = γ X(h) γ X (0), h Z. 12

13 Properties of the ACVF Any ACVF of a stationary process has the following properties: (i) The ACVF is even: γ X (h) = γ X ( h) for all h; (ii) The ACVF is nonnegative definite (nnd): n j=1 n k=1 a jγ X (j k)a k 0 for all positive integers n and real-valued constants {a j : j = 1,..., n}. Bochner s theorem [Bochner, 1933, 1955] states that: a real-valued function γ X ( ) defined on the integers is the ACVF of a stationary process if and only if it is even and nnd. Rather than using Bochner s theorem, it is more common to create processes from measurable functions of other processes linear filtering is the most popular. 13

14 Linear filtering preserves stationarity A linear time invariant (LTI) filter is an operation that takes a time series process {W t : t T } and creates a new process {X t } via X t = j= ψ j W t j t T, where {ψ j : j Z} is a sequence of filter coefficients that do not depend on time and that are absolutely summable: j ψ j <. Then, if {W t } is a mean zero stationary process with ACVF γ W (h), {X t } is a mean zero stationary process with ACVF γ X (h) = ψ j ψ k γ W (j k + h), h Z. j= k= This is the linear filtering preserves stationarity result. We will use this result to define useful classes of stationary processes. 14

15 IID noise and white noise The simplest strictly stationary processes is IID noise: {X t : t T } is a set of independent and identically distributed RVs. There are no dependencies across time for this process. The analogous no correlation model that is weakly stationary is white noise: {X t } in this case satisfies E(X t ) = µ, a constant independent of time for all t, with σ 2, h = 0; γ X (h) = 0 otherwise. We write WN(µ, σ 2 ) for a white noise process with mean µ and variance σ 2 > 0. Neither process is a good model for dependent time series, but are good building blocks! 15

16 Linear processes Suppose that {ψ j : j Z} is an absolutely summable sequence that is independent of time and {Z t : t T } is a WN(0, σ 2 ) process. By the filtering preserves stationary result, the linear process {X t : t T }, defined by X t = j= ψ j Z t j, t T, is a stationary process with zero mean and an ACVF given by γ X (h) = σ 2 j= ψ j ψ j+h, h Z. This class is useful for theoretical studies of linear stationary processes, but in practice for statistical modeling we use linear filters of finite time support (i.e., ψ j 0 for a finite collection of indexes j). 16

17 Example: Moving average process of order q, MA(q) Let {Z t } be a WN(0, σ 2 ) process. For an integer q > 0 and constants θ 1,..., θ q with θ q 0 define X t = Z t + θ 1 Z t θ q Z t q where we let θ 0 = 1. = θ 0 Z t + θ 1 Z t θ q Z t q q = θ j Z t j, j=0 {X t } is known as the moving average process of order q, or the MA(q) process. Clearly, by definition, the MA(q) process is a linear process. 17

18 Defining linear processes with backward shifts The backward shift operator, B, is defined by BX t = X t 1. We can represent a linear process using the backward shift operator as where we let ψ(b) = j= ψ jb j. X t = ψ(b)z t, ψ(b) is a polynomial with the backward shift operator as the argument (not a number). We will think of it as a type of linear filter. Example: a mean zero MA(1) process can be written as where ψ(b) = 1 + θb. X t = ψ(b)z t, 18

19 Example: The MA(q) process is stationary By the filtering preserves stationarity result, the MA(q) process defined previously is a stationary process with mean zero and ACVF γ X (h) = σ 2 q θ j θ j+h j=0 (let θ j = 0 for j < 0 and j > q for this to make sense). Let I(A) denote the indicator function defined by 1, if the event A is true; I(A) = 0, if the event A is false. Then q q γ X (h) = θ j θ k γ Z (j k + h) j=0 k=0 = σ 2 q j=0 k=0 q θ j θ k I(k = j + h) = σ 2 q θ j θ j+h. j=0 19

20 Exercise: Differencing an MA(1) process Suppose Y t is an MA(1) process defined in the usual way. Let X t = Y t Y t 1 denote the differenced MA(1) process. 1. Is {X t } stationary? 2. What is the ACVF and ACF of {X t }? 20

21 What is spectral analysis? In probability theory we usually work with cumulative distribution functions (or probability density/mass functions, when they exist), But, we can also work with its Fourier transform: the characteristic function. 21

22 What is spectral analysis, continued? Similarly, for stationary processes we have used the autocovariance function (ACVF). We now consider its Fourier transform, the spectrum, which provides a frequency decomposition of the variance of a process. Good references: Priestley [1981a], Priestley [1981b], Brillinger [1981], Brockwell and Davis [1991, Section 4], and Percival and Walden [1993]. 22

23 Applications of spectral methods 1. Examining and modeling the dependence of a time series in the spectral domain. 2. Testing for significant periodicities [e.g., Fisher, 1929] 3. Building valid time series models [e.g., Bloomfield, 1973]. 4. Fast simulation of stationary processes [e.g. Davies and Harte, 1987]. 5. Approximate maximum likelihood estimation methods [e.g., Whittle, 1953, Dahlhaus, 1989] that are fully efficient as N. 23

24 Example: The harmonic process Consider the harmonic process defined by X t = L [A l cos(2πf l t) + B l sin(2πf l t)]. l=1 Assume that {f l } are sorted in increasing order, and that {A l } and {B l } are a sequence of (mutually) uncorrelated mean zero RVs with var(a l ) = var(b l ) = σ 2 l for each l. It can be shown that {X t } is a stationary mean zero process, with ACVF γ X (h) = L σl 2 cos(2πf l h). l=1 24

25 An analysis of variance for the harmonic process This implies that the variance is L var(x t ) = γ X (0) = σl 2. This is the first example of a spectral decomposition. We have decomposed the variance of the process {X t } in terms of the variances of sinusoids. This is an example of an analysis of variance. l=1 25

26 Relating sinusoids to complex exponentials The process {X t } can be rewritten as L X t = [C l cos(2πf l t + φ l )]. l=1 Using complex numbers, let i = 1. Employing the identity, cos(z) = (e iz + e iz )/2, it can be shown that L X t = D l e i2πflt. l= L where D 0 = 0, D l = (C l /2)e iφ l for l = 1,..., L, D l = Dl (the complex conjugate), and f l = f l. Then the ACVF is L γ X (h) = S l e i2πflh, l= L where S l = S l = σl 2/2 for l = 1,..., L and S 0 = 0. We call S l the integrated spectrum at frequency f l. 26

27 A discussion about notation and frequencies I will follow the notation of Percival and Walden [1993]. In particular I let the frequency be f [ 1/2, 1/2] Many use ω = 2πf [ π, π]. The use of f matches with the frequencies I used in the harmonic regression model. We start by assuming that the time series {X t } is collected with a sampling interval of = 1 (one time unit between observations). In practice, can vary. When changes, the range of frequencies changes to [ F, F ], where F = 1 2 is called the Nyquist frequency. 27

28 Herglotz s theorem (The spectrum is the Fourier transform of the ACVF) A real-valued function γ X ( ) defined on the integers is non-negative definite if and only if γ X (h) = 1/2 1/2 e i2πfh ds I X(f), for all h Z, where SX I ( ) is a right-continuous, non-decreasing and bounded function on [ 1/2, 1/2] with SX I ( 1/2) = 0. SX I ( ) is called the integrated spectrum or the spectral distribution function. The derivative of SX I ( ), when it exists, is called the spectral density function (SDF), S( ). 28

29 Combining with Bochner s theorem A real-valued function γ X ( ) defined on the integers is the ACVF of a stationary process {X t } if and only if 1. γ X (h) = 1/2 1/2 ei2πfh dsx I (f), for all h Z, where SI X ( ) is a right-continuous, nondecreasing and bounded function on [ 1/2, 1/2] with SX I ( 1/2) = γ X ( ) is even and non-negative definite. We say the S I X ( ) is the spectrum/spectral distribution function of {X t} and γ X ( ). (Whenever necessary we write S I X ( ) to indicate it is the spectrum of {X t}.) 29

30 So what does the spectrum measure? For a stationary process {X t }, taking γ X (h) = we let h = 0 to get 1/2 1/2 var(x t ) = γ X (0) = When the derivative exists, we have γ X (0) = 1/2 1/2 e i2πfh ds I X(f), 1/2 1/2 S X (f)df. ds I X(f). Thus we can think of the increments of the spectrum as decomposing the variance of a time series into a collection of pieces, each of which can be associated with a different frequency f. Different stationary time series models have different decompositions by frequency. Before we look at some examples let us discuss some further properties of the spectrum. 30

31 Existence of the spectral density function If γ X ( ) is any real-valued function defined on the integers that is absolutely summable (i.e., h γ X(h) < ), then with γ X (h) = 1/2 1/2 S X (f) = h Z e i2πfh S X (f)df, e i2πfh γ X (h). We say that the γ X ( ) and the S X ( ) are Fourier transform pairs, and write {γ X ( )} S X ( ). 31

32 For real-valued stationary processes A real-valued function S X ( ) defined on f 1/2 is the SDF of a real-valued stationary process {X t } if and only if 1. S X (f) = S X ( f) for all f 1/2 (the SDF is an even function). 2. S X (f) 0 for all f 1/ /2 1/2 S X(f) df <. Note that this result is if and only if: In particular it gives a way to create a stationary process without ever having to check that the ACVF is nonnegative definite. Once we have a SDF that satisfies these conditions then γ X (h) = 1/2 1/2 32 e i2πfh S X (f) df.

33 Changing the sampling interval Let {X t } be a stationary process, but suppose that we observe the process at sampling interval (not necessarily of value one). Remember that F = 1/(2 ) is the Nyquist frequency. With an absolutely summable ACVF we have S X (f) = e i2πfh γ X (h) df, h Z with γ X (h) = F F e i2πfh S X (f) df. As decreases we get more information about the SDF. We are less affected by aliasing of frequencies. 33

34 Aliasing: sinusoids An example of two aliased sinusoids: Delta = 1, f = 1/ Delta = 1, f = 1/ The amplitudes of the waves are same. Thus, the spectrum is the same. 34

35 Aliasing, in general Let {X t } be a stationary process, observed with sampling interval. If the ACVF is absolutely summable, then S X (f) = S X (f + k/ ), for all k Z. This leads us to imagine that S X (f), observed in any given interval [ F, F ], contains the accumulation of all spectral densities observed at frequencies {f + k/ : k Z}. 35

36 The SDF of a white noise process Let {Z t } be a WN(0, σ 2 ) process. Then for all f < 1/2, S Z (f) = e i2πfh γ Z (h) h= = 1. γ Z (0) = σ 2. The SDF of white noise is a constant function. Indeed, this is why it is called white noise. The SDF (and hence variance) of a WN process is an equal mix of all frequencies ( colors ). 36

37 Linear time-invariant filtering of stationary processes Suppose {X t } is a stationary mean zero process with ACVF γ X ( ) and let {a j } be absolutely summable. Remember {Y t }, defined as a linear filtering of {X t }, by Y t = j= a j X t j, is a stationary mean zero process with ACVF γ Y (h) = a j a k γ X (j k + h). j= k= Now suppose that the SDF of {X t } is S X ( ). What happens to the SDF under filtering? i.e., what is S Y ( )? 37

38 First, some definitions The transfer function of a filter {a j } is its Fourier transform: A(f) = j= a j e i2πfj. The gain function of {a j } is the modulus of the transfer function, A(f). The square or squared gain function of {a j } is the modulus squared of the transfer function: A(f) = A(f) 2 = A(f)A (f), where denotes, as usual, the complex conjugate. 38

39 What is the SDF when linear filtering? We have that 1/2 1/2 e i2πfh S Y (f)df = γ Y (h) = = = = = a j a k γ X (j k + h) j= k= j= k= 1/2 1/2 1/2 1/2 1/2 1/2 a j a k 1/2 e i2πfh S X (f) 1/2 e i2πf(j k+h) S X (f)df a j e i2πfj j= k= e i2πfh S X (f)a(f)a (f)df e i2πfh A(f)S X (f)df. a k e i2πfk df By uniqueness of Fourier transforms, S Y (f) = A(f)S X (f). 39

40 Example 1: The SDF of an MA(1) process For {Z t } a WN(0,σ 2 ) process, let X t = Z t + θz t 1 = a j Z t j, j= where 1, j = 0; a j = θ, j = 1; 0, otherwise. The transfer function is A(f) = j= a j e i2πfj = 1 + θe i2πf. 40

41 Example 1: The SDF of an MA(1) process, cont. The square gain function is A(f) = A(f)A (f) = (1 + θe i2πf )(1 + θe i2πf ) = 1 + θ ( e i2πf + e i2πf) + θ 2 = 1 + 2θ cos(2πf) + θ 2. Thus the SDF of an MA(1) process is S X (f) = A(f)S Z (f) = (1 + 2θ cos(2πf) + θ 2 )σ 2. 41

42 Example 2: ARMA processes Suppose {Z t } is a WN(0, σ 2 ) process. {X t } is an autoregressive moving average process of autoregressive order p and moving average order q, or an ARMA(p, q) process for short, if there exists a stationary solution to the following equation: p X t φ j X t j = Z t + j=1 assuming that φ p 0 and θ q 0. q θ k Z t k, t T, k=1 By there exists a stationary solution we mean that we can represent {X t } as a (stationary) linear process. 42

43 Special cases of the ARMA process With p = 0, the moving average process of order q (the MA(q) process) satisfies q X t = Z t + θ k Z t k, t T. k=1 When q = 0 we obtain the autoregressive process of order p (the AR(p) process; Yule [1921, 1927]) X t p φ j X t j = Z t, t T. j=1 43

44 The SDF of an ARMA process We have that 1 p φ j e i2πfj j=1 2 S X (f) = 1 + q 2 θ k e i2πfk S Z (f). k=1 Thus 1 + q S X (f) = σ 2 k=1 θ ke i2πfk 2 1 p j=1 φ je i2πfj 2, f < 1/2. This is often called the rational SDF. 44

45 Example SDFs for different AR (ARMA(p, 0)) processes AR(1) process φ 1 = 0.6 AR(1) process φ 1 = 0.2 SDF SDF frequency AR(2) process φ = (0.75, 0.5) frequency AR(4) process φ = (2.7607, , , ) SDF SDF frequency frequency Usually we change the y-scale to be decibels (db) a 10 log 10 transformation. 45

46 Example simulations of the AR(2) and AR(4) processes AR(2) ACF time Lag AR(4) ACF time Lag Exercise: Compare these realizations with the global temperature and water discharge series we examined previously. 46

47 Approximation theorems involving spectral densities [Brockwell and Davis, 1991, Coroll 4.4.2] if S Y ( ) is a symmetric continuous SDF and ɛ > 0, then there exists an AR(p) process {X t } such that S Y (f) S X (f) < ɛ, for all f [ 1/2, 1/2]. Brockwell and Davis [1991, Coroll 4.4.1] gives a similar result for MA(q) processes. An AR approximation for time series is used more often than the MA approximation in the literature. Combining both, gives us a motivation for why ARMA models can outperform AR and MA models, in terms of the ability to approximate the SDF (equivalently the ACVF). 47

48 Wavelet transformations We use spectral analysis to view a time series in terms of a sum of sinusoids at different frequencies. Unless we do something sneaky, we lose all information about time. A wavelet transformation uses wavelets ( small waves ) to decompose a time series into: 1. averages over a specified (typically long) time scale, and 2. changes of averages over a range of shorter time scales. It is a time-scale (some say frequency) decomposition. Thus wavelet methods are a natural way to explore and model non-stationarity. See Vidakovic [1998] and Percival and Walden [2000] we will discuss wavelets at further points in these lectures. 48

49 References P. Bloomfield. An exponential model for the spectrum of a scalar time series. Biometrika, 60: , S. Bochner. Monotone funktionen, Stieltjessche integrale und harmonische analyse. Mathematische Annalen, 108: , S. Bochner. Harmonic analysis and probability theory. University of California Press, Berkeley, CA, D. R. Brillinger. Time Series: Data Analysis and Theory. Holt, New York, NY, P. J. Brockwell and R. Davis. Introduction to Time Series and Forecasting (Second Edition). Springer, New York, NY, P. J. Brockwell and R. A. Davis. Time Series. Theory and Methods (Second Edition). Springer, New York, J. D. Cryer and K.-S. Chan. Time Series Analysis, with Applications in R. Springer, New York, NY, R. Dahlhaus. Efficient parameter estimation for self similar processes. The Annals of Statistics, 17: , R. B. Davies and D. S. Harte. Tests for Hurst effect. Biometrika, 74:95 101, URL R. Fisher. Tests of significance in harmonic analysis. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Math. or Phys. Character ( ), 125:54 59, D. Percival and A. Walden. Spectral Analysis for Physical Applications. Cambridge University Press, Cambridge, D. Percival and A. Walden. Wavelet Methods for Time Series Analysis. Cambridge University Press, Cambridge, England, M. B. Priestley. Spectral Analysis and Time Series. (Vol. 1): Univariate Series. Academic Press, London, UK, 1981a. M. B. Priestley. Spectral Analysis and Time Series. (Vol. 2): Multivariate Series, Prediction and Control. Academic Press, London, UK, 1981b. R. Rohde, R. A. Muller, R. Jacobsen, E. Muller, S. Perlmutter, A. Rosenfeld, J. Wurtele, D. Groom, and C. Wickham. A new estimate of the average earth surface land temperature spanning 1753 to Geoinfor Geostat: An Overview, 1, R. H. Shumway and D. S. Stoffer. Time series analysis and its applications: with R examples. Springer, New York, NY, B. Vidakovic. Statistical Modeling by Wavelets. Wiley, New York, NY, P. Whittle. Estimation and information in stationary time series. Arkiv för matematik, 2: , URL 2FBF G. U. Yule. On the time-correlation problem, with special reference to the variate difference correlation method. Journal of Royal Statistical Society, 84: , URL G. U. Yule. On a method of investigating periodicities in disturbed series, with special reference to Wolfer s sunspot numbers. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 226: , URL 49