Statistics of stochastic processes

Introduction Statistics of stochastic processes Generally statistics is performed on observations y 1,..., y n assumed to be realizations of independent random variables Y 1,..., Y n. In statistics of stochastic processes (= time series analysis) we will assume y 1,..., y n to be realizations of a stochastic process... Y 1,..., Y n,... with some rules for dependence. 14 settembre 2014 1 / 31

Introduction Aims of time series analysis Drawing inference from available data, but first we need to find an appropriate model. Once a model has been selected: provide a compact and correct description of data (trend, seasonal and random terms) adjust data (filtering, missing values) [separating noise from signal] test hypotheses (increasing trend? influence of factors) understand causes Remarks 14 settembre 2014 2 / 31

Introduction Stationary process Definition A stochastic process {X t } t Z is (strictly) stationary if the joint distribution of (X t1, X t2,..., X tk ) is equal to the distribution of (X t1 +h, X t2 +h,..., X tk +h) k N, h Z, t 1, t 2,..., t k Z. In particular, if a stationary stochastic process has finite second moment, then E(X t ) and Cov(X t, X t+h ) do not depend on t. 14 settembre 2014 3 / 31

Introduction Stationary process. 2 Linear time series analysis looks only at second-order properties. Then Definition A stochastic process {X t } t Z is stationary if it is in L 2 and E(X t ) = µ Cov(X t, X t+h ) = γ(h). If a Gaussian process is stationary, then it is strictly stationary. 14 settembre 2014 4 / 31

Introduction Reminders on multivariate normal Definition Y = (Y 1,..., Y n ) is multivariate normal if, a R n, a t Y is a univariate normal. 14 settembre 2014 5 / 31

Introduction Reminders on multivariate normal Definition Y = (Y 1,..., Y n ) is multivariate normal if, a R n, a t Y is a univariate normal. Equivalently, Y is multivariate normal there exists b R n A (n m) matrix, X = (X 1,..., X m ) independent standard normal r.v. such that Y = AX + b. = E(Y ) = b, Cov(Y ) = AA t, i.e. Y N(b, AA t ). Alternative characterization via characteristic function. 14 settembre 2014 5 / 31

Introduction Gaussian processes Definition A process {X t } is Gaussian, if for any n > 0 and any (t 1,..., t n ) the vector X = (X t1,..., X tn ) has a non-singular multivariate normal distribution. Then let µ = (µ t1,..., µ tn ) = E(X) and Cov(X) = Γ = {γ(t i, t j ), i, j = 1... n}. X has density function { g(x, µ, Γ) = (2π) n/2 Γ 1/2 exp 1 } 2 Γ 1 (x µ), x µ. {X t } is (weakly) stationary if µ t µ and γ(t i, t j ) = γ( t i t j ); then is also strictly stationary, as the distribution depends only on µ and Γ. 14 settembre 2014 6 / 31

Introduction Hilbert spaces Many time series problems can be solved using Hilbert space theory. Indeed space L 2 (Ω) is a Hilbert space with X, Y = E(XY ), X Y 2 = E( X Y 2 ). Restricting to the 0-mean subspace X, Y = Cov(X, Y ). 14 settembre 2014 7 / 31

Introduction Detrending data Often data do not appeat as arising from stationary processes. Estimating trend, and then study residuals (differences from trend) smoothing polynomial (esp. line) fitting Study differenced series In all cases, trasformations may be useful More systematic model fitting in the future. 14 settembre 2014 8 / 31

Johnson & Johnson quarterly earnings J & J Earnings per Share 0 5 10 15 data 3 points smoothing 5 points smoothing 1960 1965 1970 1975 1980 Time 14 settembre 2014 9 / 31

Johnson & Johnson data: deviations from trend Deviations from moving average -3-2 -1 0 1 2 1960 1965 1970 1975 1980 Time 14 settembre 2014 10 / 31

Johnson & Johnson data: deviations in log-scale Deviations (in log scale) from moving average -0.4-0.3-0.2-0.1 0.0 0.1 0.2 0.3 1960 1965 1970 1975 1980 Time 14 settembre 2014 11 / 31

Sunspots data 1700-1980 sunspots 0 50 100 150 1700 1750 1800 1850 1900 1950 year 14 settembre 2014 12 / 31

Sunspots data 1700-1980: square-root transformation sunspots 0 2 4 6 8 10 12 14 1700 1750 1800 1850 1900 1950 year 14 settembre 2014 13 / 31

PanAm international air passengers 1949-60 Passengers (1000's) 100 200 300 400 500 600 1950 1952 1954 1956 1958 1960 Time 14 settembre 2014 14 / 31

PanAm yearly data Annual air passengers aggregate(ap) 2000 3000 4000 5000 1950 1952 1954 1956 1958 1960 Time 14 settembre 2014 15 / 31

PanAm monthly variation Seasonal component in air passengers 100 200 300 400 500 600 1 2 3 4 5 6 7 8 9 10 11 12 Month 14 settembre 2014 16 / 31

Level of Lake Huron 1875-1972 Level of lake Huron ft 6 7 8 9 10 11 12 1880 1900 1920 1940 1960 Time 14 settembre 2014 17 / 31

Lake Huron level: deviations from trend Deviations from trend in level of lake Huron ft -2-1 0 1 2 0 20 40 60 80 100 Time 14 settembre 2014 18 / 31

sales of red wine in Australia 1980-91 Red wine sales in Australia kilolitres 500 1000 1500 2000 2500 3000 1980 1982 1984 1986 1988 1990 1992 Time 14 settembre 2014 19 / 31

Deviation from trend in wine sales Deviations from trend in sales of red wine kilolitres -1000-500 0 500 1000 0 20 40 60 80 100 120 140 Time 14 settembre 2014 20 / 31

PanAm monthly variation Seasonal variation in wine sales (AUS) 500 1000 1500 2000 2500 3000 1 2 3 4 5 6 7 8 9 10 11 12 14 settembre 2014 21 / 31

Global temperature data 1856-2005 Global temperature data Anomalies from 1961-90 mean -1.0-0.5 0.0 0.5 Monthly averages Yearly averages 1900 1950 2000 Time 14 settembre 2014 22 / 31

Global temperature: recent years and trend Global temperatures 1971-2005 (regression line in blue) Anomalies -0.4-0.2 0.0 0.2 0.4 0.6 0.8 1970 1975 1980 1985 1990 1995 2000 2005 Time 14 settembre 2014 23 / 31

Measles data in England 1944-1967 Measles in England cases per biweek 0 5000 10000 15000 20000 25000 1945 1950 1955 1960 1965 year 14 settembre 2014 24 / 31

EEG data from a subject with epilepsy EEG -150-50 50 150 0 1000 2000 3000 4000 time (arbitrary unit) 14 settembre 2014 25 / 31

De-trend and de-seasonalize (period T = 2q) yearly average: m t = 1 T ( 1 2 x t q + q 1 j= (q 1) x t+j + 1 2 x t+q ). 14 settembre 2014 26 / 31

De-trend and de-seasonalize (period T = 2q) yearly average: m t = 1 T ( seasonal deviation: w k = 1 n 1 2 x t q + n 1 j=0 q 1 j= (q 1) x t+j + 1 2 x t+q ) (x jt +k m jt +k ), k = 1... T.. 14 settembre 2014 26 / 31

De-trend and de-seasonalize (period T = 2q) yearly average: m t = 1 T ( seasonal deviation: w k = 1 n 1 2 x t q + n 1 j=0 seasonal component: ŝ k = w k 1 T ŝ t = ŝ t [ t 1 T ]T, t > T. q 1 j= (q 1) x t+j + 1 2 x t+q ) (x jt +k m jt +k ), k = 1... T. T w i, k = 1... T. i=1. 14 settembre 2014 26 / 31

De-trend and de-seasonalize (period T = 2q) yearly average: m t = 1 T ( seasonal deviation: w k = 1 n 1 2 x t q + n 1 j=0 seasonal component: ŝ k = w k 1 T ŝ t = ŝ t [ t 1 T ]T, t > T. deseasonalized data d t = x t ŝ t. q 1 j= (q 1) x t+j + 1 2 x t+q ) (x jt +k m jt +k ), k = 1... T. T w i, k = 1... T. i=1 ˆm t trend component on deseasonalized data.. 14 settembre 2014 26 / 31

Autocovariance and autocorrelation functions If a process {X t } is stationary, γ(h) := Cov(X t, X t+h ) is the Autocovariance function (ACVF). 14 settembre 2014 27 / 31

Autocovariance and autocorrelation functions If a process {X t } is stationary, γ(h) := Cov(X t, X t+h ) is the Autocovariance function (ACVF). Recall the correlation ρ(x, Y ) = Cov(X, Y ) V (X )V (Y ). For a stationary process V (X t ) = V (X t+h ) = γ(0). Hence ρ(h) = ρ(x t, X t+h ) = γ(h) γ(0) is the Autocorrelation function (ACF). 14 settembre 2014 27 / 31

Autocovariance and autocorrelation functions If a process {X t } is stationary, γ(h) := Cov(X t, X t+h ) is the Autocovariance function (ACVF). Recall the correlation ρ(x, Y ) = Cov(X, Y ) V (X )V (Y ). For a stationary process V (X t ) = V (X t+h ) = γ(0). Hence ρ(h) = ρ(x t, X t+h ) = γ(h) γ(0) First properties of ACVF: is the Autocorrelation function (ACF). 14 settembre 2014 27 / 31

Autocovariance and autocorrelation functions If a process {X t } is stationary, γ(h) := Cov(X t, X t+h ) is the Autocovariance function (ACVF). Recall the correlation ρ(x, Y ) = Cov(X, Y ) V (X )V (Y ). For a stationary process V (X t ) = V (X t+h ) = γ(0). Hence ρ(h) = ρ(x t, X t+h ) = γ(h) γ(0) First properties of ACVF: is the Autocorrelation function (ACF). γ(h) = γ( h) [stationarity = Cov(X t, X t+h ) = Cov(X t h, X t )] 14 settembre 2014 27 / 31

Autocovariance and autocorrelation functions If a process {X t } is stationary, γ(h) := Cov(X t, X t+h ) is the Autocovariance function (ACVF). Recall the correlation ρ(x, Y ) = Cov(X, Y ) V (X )V (Y ). For a stationary process V (X t ) = V (X t+h ) = γ(0). Hence ρ(h) = ρ(x t, X t+h ) = γ(h) γ(0) First properties of ACVF: is the Autocorrelation function (ACF). γ(h) = γ( h) [stationarity = Cov(X t, X t+h ) = Cov(X t h, X t )] γ(h) γ(0) [as ρ(x, Y ) 1] 14 settembre 2014 27 / 31

Simple stationary processes and their ACVF IID(0, σ 2 ): {X t } t Z independent and identically distributed r. v. with E(X t ) = 0, V(X t ) = σ 2 : γ(0) = σ 2, γ(h) = 0 for h > 0. 14 settembre 2014 28 / 31

Simple stationary processes and their ACVF IID(0, σ 2 ): {X t } t Z independent and identically distributed r. v. with E(X t ) = 0, V(X t ) = σ 2 : γ(0) = σ 2, γ(h) = 0 for h > 0. WN(0, σ 2 ) [white noise] {X t } t Z uncorrelated random variables with mean 0 and variance σ 2 : γ(0) = σ 2, γ(h) = 0 for h > 0. WN(0, σ 2 ) need not be independent. For instance if {Z t } t Z are IID and N(0,1) [normal r.v.], then { Z t t odd X t = (Zt 1 2 1)/ is WN(0, 1) but not IID(0, 1). 2 t even It is not IID, since (e.g.) X 1 and X 2 are obviously not independent. Left for exercise that X t is WN. 14 settembre 2014 28 / 31

Moving average processes and their ACVF. 2 MA(1): moving average {X t } t Z is MA(1) if X t = Z t + ϑz t 1, t Z where ϑ R, {Z t } WN(0, σ 2 ). A simple computation: γ(0) = σ 2 (1 + ϑ 2 ), γ(1) = ϑσ 2, γ(h) = 0 for h > 1. Similarly {X t } t Z MA(q) if X t = Z t + ϑ 1 Z t 1 + ϑ q Z t q, t Z, with ϑ 1,..., ϑ q R, {Z t } WN(0, σ 2 ). Another simple computation leads to γ(h) = 0 for h > q. 14 settembre 2014 29 / 31

AutoRegressive processes AR(1) [AutoRegressive] {X t } t Z is AR(1) if is stationary and X t = φx t 1 + Z t, t Z where φ R, {Z t } WN(0, σ 2 ). (1) 14 settembre 2014 30 / 31

AutoRegressive processes AR(1) [AutoRegressive] {X t } t Z is AR(1) if is stationary and X t = φx t 1 + Z t, t Z where φ R, {Z t } WN(0, σ 2 ). (1) (1) is an (infinite set of) equation. It is not obvious that a stationary process exists satisfying them (this will be discussed later). We are not saying {X t } t N is the Markov chain defined through X t = φx t 1 + Z t, t > 0 with X 0 some prescribed r.v. Now, assume a stationary process {X t } t Z exists satisfying (1) and E(X t Z s ) = 0 for t < s (this latter property seems natural as X t should be defined in terms of Z t and the previous ones). 14 settembre 2014 30 / 31

AutoRegressive processes. 2 Remarks: we have found φ 2 < 1 φ < 1 as a necessary condition for an AR(1) satisfying E(X t Z s ) = 0 for t < s. It will also be sufficient. Implicit assumption in the computations: E(X t ) = 0 (this can be proved analogously). More simply, one can then compute γ(h) for h > 0 (left for exercise). 14 settembre 2014 31 / 31