1. Fundamental concepts

Transcription

1 . Fundamental concepts A time series is a sequence of data points, measured typically at successive times spaced at uniform intervals. Time series are used in such fields as statistics, signal processing and financial mathematics. Examples of time series: the daily closing value of the Dow Jones index, the annual flow volume of the River Nile at Aswan, daily air temperature or monthly precipitation in a specific location, the annual yield of corn in Iowa, the size of an organism, measured daily,

2 annual U.S. population data, daily closing stock prices, weekly interest rates, national income, sales figures, innumerable other sequences based on industrial, economic, and social phenomena, and studies in medicine, geophysics, and engineering. 2

3 Time series analysis is carried out for several major objectives: (a) To describe and explain general characteristics of the series, for example to separate the series into components representing trend (long term movements), seasonality (periodic movements due to seasonal variation), and random fluctuations; (b) To forecast future values using some mathematical model; (c) To control the series, making sure that it remains within in certain bounds. Needed to carry out these tasks is a model for the series, which includes estimating model parameters. 3

4 Time series methods may be divided into two main classes: (a) frequency domain methods (spectral analysis to examine cyclic behavior which need not be related to seasonality) (b) time domain methods (autocorrelation and cross-correlation analysis to examine dependence over time) Although the two methods are mathematically equivalent in the sense that the autocorrelation function and the spectrum function form a Fourier transform pair, there are occasions when one approach is more advantageous than the other. 4

5 Time series data have a natural temporal ordering. This makes it distinct from common data problems, where there is no natural ordering of the observations, and from spatial data analysis, where the observations typically relate to geographic locations. A time series model will generally reflect the fact that observations close together in time will be more closely related than observations further apart. In addition, time series models will often make use of the natural one-way ordering of time so that values for a given period will be expressed as being derived in some way from past values, rather than future values. 5

6 Models for time series data can have many forms. When modeling variation in the level of the process, three broad classes of practical importance are autoregressive (AR) models, integrated (I) models, moving average (MA) models. These three classes depend linearly on previous data points. Combinations of these ideas produce autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) models. Extension of these classes to deal with vector-valued data is available. 6

7 Among non-linear models are models to represent changes of variance (heteroskedasticity) through time. These models are called autoregressive conditional heteroskedasticity (ARCH) and the collection comprises a wide variety of representations. Here, changes in variability are related to, or predicted by, recent past values of the observed series. After reviewing probability spaces, we give a more formal definition of time series and properties of basic tools for their analysis. 7

8 . Probability spaces Let Ω be the set of all possible outcomes of an experiment, and ω Ω the elementary events. Let ϒ be a collection of subsets of Ω, and A ϒ. If ω A occurs, then we say that A occurs. We define a function PA ( ) that gives the long-run relative frequency with which A will occur. P satisfies the axioms: AXIOM : PA ( ) 0 for every A ϒ. AXIOM 2: P( Ω ) = AXIOM 3: If A, A 2,, is a countable sequence from ϒ and Ai Aj = for all i j, then P ( A ) i = PA ( ) i= i. i= 8

9 Example..: We roll a fair die and record the number of dots on the side that shows. Then Ω= {, 2,3, 4,5,6}. We take PA ( ) = /6 η( A), where η ( A) is the number of elements ω in A. Let ϒ be the collection of all possible subsets of Ω. Then since there are a nonnegative number of elementary events in A ϒ, PA ( ) 0. P( Ω ) = ( / 6) 6 =. Also, a disjoint union k A i= i has exactly k η( Ai ) subsets, and thus i= ( k ) k i = i= i= P A ( / 6) η( A ) k k i= i i= i. = (/6) η( A) = PA ( ) i 9

10 So that we can define PA ( ) for all A, we restrict the collection ϒ of subsets of Ω so that it satisfies the following:. If A ϒ then C A ϒ 2. If A, A 2,, is a countable sequence from ϒ, then 3. ϒ A is in ϒ i= i ϒ is called a σ -field or σ -algebra. ( Ωϒ,,P) is called a probability space. A random variable X is a real valued function on Ω ( X : Ω ) such that the set { ω: X( ω) x} is in ϒ for every real number x. FX ( x) = P{ ω: X( ω) x} is called the distribution function of the random variable X. 0

11 .2 Defining Time Series Let (,,P) Ωϒ be a probability space, and T an index set. A real valued stochastic process is a real valued function X t ( ω ) such that for each fixed t T, X t ( ω ) is a random Ωϒ,,P. variable on ( ) When the index set T corresponds to time indices (discrete or continuous), we will call X ( ω ) a time series. t For fixed ω, ( ) t X ω is a real valued function of t that is called a realization of the time series.

12 When we look at the plot of a recorded time series, we are looking at one realization of the collection (ensemble) of all such realizations. We typically suppress the ω and write X() t. A time series is strictly stationary if the joint distribution function satisfies F ( x,, x ) = F ( x,, x ) X,, X n X,, X n t tn t+ h tn+ h for all possible (nonempty finite distinct) sets of indices t,, t n and t + τ,, t n + τ in T and all ( x,, x n ) in the range of X t. 2

13 If { X t } is strictly stationary with { } t E X <, then the expectation of X t is constant for all t, and if in addition E{ X 2 } t < then the variance of X t is constant for all t as well. A time series { X t } is second-order stationary or weakly stationary or simply stationary if. The expectation of X t exists and is constant for all t. 2. The covariance matrix of X,, t X tn exists and is the same as the covariance matrix of Xt,, + τ Xt n + τ for all nonempty finite sets of indices ( t,, tn ) and all τ such that t,, t n and t + τ,, t n + τ are in T. 3

14 Since E( X t ) = µ, without loss of generality it may be taken to be zero in analyses. The autocovariance function γ h EXX ( t t+ h) of a weakly stationary process depends only on the distance h. Not all time series are stationary. For example, an economic time series may show a trend, i.e. it may be increasing through time in the mean, variance, or both. 4

15 This series appears to be stationary in the mean as it varies about a fixed level. 5

16 This series does not vary about a fixed level and exhibits an overall upward trend instead. Morever, the variance of the series increases as the level of the series increases. The series is nonstationary. 6

17 This quarterly series is repetitive in nature due to seasonal variations. This is an example of a seasonal time series. This and the last example may be reduced to stationary series by a proper transformation. 7

18 We will consider basic properties of a general class of models, called ARIMA (autoregressive integrated moving average) models that are useful for describing stationary, nonstationary, seasonal and nonseasonal time series such as those shown in the last three figures. 8

19 This plot displays another phenomenon of nonstationarity due to a change in structure in the series from some external disturbance. Such extenrnal disturbances are termed interventions or outliers. This type of nonstationarity cannot be removed by a standard transformation. 9

20 .3 More examples of time series Example.3. An important time series is a white noise sequence of uncorrelated random variables { et : t = 0, ±, ± 2, }, where each e t has mean zero and finite variance 2 σ > 0. Its autocovariance function is γ h σ 2, h = 0 =. 0 otherwise 20

21 Example.3.2 Let Xt = Acos( λt) + Bsin( λt) + et, 2 ABe,, t iid... N(0, σ ). Then E( X t ) = 0 t and ( λ ) λ( ) ( ) 2 EXX ( t t+ h) = E Acos tcos t+ h ( ) ( ) ( ) [ ] + E B 2 sin λt sin λ t+ h + E ee t t + h ( λh) 2 σ h> cos 0 =. 2 2 σ h = 0 (Use the trigonometric identity ( A B) ( A) ( B) ( A) ( B) cos = cos cos + sin sin ). Since EXX ( t t + h) does not depend on t, the process is wide-sense stationary. 2

22 Example.3.3 Let the covariance of the X : t, be given by process { ( )} t γ {( µ )( µ )} h h= E X t X t h Ae α + =, A, α > 0 (time in hours). 2 The covariance between daily reports is thus 576α Ae. Also, the covariance between the change from 7:00 a.m. to 8:00 a.m. and the change from 8:00 a.m. to 9:00 a.m. is given by E{ ( Xt+ Xt)( Xt+ 2 Xt+ ) } {[( t+ µ ) ( t µ )][( t+ 2 µ ) ( t+ µ )]} = E{ [( X ] t+ µ )( Xt+ 2 µ ) ( Xt+ µ ) } = E X X X X [( µ )( µ )] [( µ )( µ )] E X X + E X X t t+ 2 t t+ = A + e e 4α α [ 2 ] 22

23 The variance of the change from t to t + h is { } h Var ( X t h X t) = 2 A( e α + ). Thus, Var{ X X } lim h 0 ( t+ h t) = 0. Such processes are called mean square continuous. The rate of change per unit of time, defined by 2 Xt+ h Xt Rt ( h) = h has variance { ( )} Var R h t 2 h 2A α e =. 2 h 23

24 By L Hospital s rule, αh 4α hae lim h 0Var { Rt ( h) } = limh 0 = 2Aα 2h 2 Stochastic processes for which this limit exists are called mean square differentiable. 24

25 .4 Properties of the autovariance and autocorrelation functions It is often useful to have a measure of association that doesn t depend on the units of measure (as is the case with correlation as opposed to covariance). We define the autocorrelation function of a stationary time series by h γ γ h =. 0 The autocovariance and autocorrelation functions of a stationary time series have some distinguishing characteristics. 25

26 Theorem The autocovariance function of a real stationary time series { Xt, t T} is an even function of t, i.e. γh = γ h. Proof. Without loss of generality, let E X =. By stationarity, { } 0 t { } E Xt X t+ h = γ h for all t and t + h contained in the index set. Now set t = t h. Then 0 γ { } { } = E X X = E X X = γ. h t t + h t h t h

27 Definition A function f( x ), x χ is said to be positive semi-definite (or nonnegative definite) if it satisfies n n j= k= aa f( t t) 0 j k k j for any set of real numbers ( a,, an ) for any ( t,, tn ) such that t k t j χ ( jk., ) and for all Theorem The autocovariance function of a stationary time series { Xt, t T} is positive semi-definite. 27

28 Proof. Without loss of generality, we assume that E{ X t} = 0. Let ( t,, tn ) T and (,, n a an ). Let γ tk t be the j autocovariance function at between observations at times t j and t k. Since the variance of any random variable is nonnegative, we have 0 n n n Var a X = E a a X X j t j k t t j= j= k= j j k n n =. j= k= aaγ j k t t k j 28

29 Now, set n = 2. Then aγ0+ a2γ0+ 2aa 2γt2 t, which implies that 2 2 aa 2γ t t a 2 + a2. γ 0 2 If t t2 = h and a = a2 =, this implies that h. If t t 2 = h and a = a2 =, then we have that h, i.e. h. Thus. h 29

30 .5 The partial autocorrelation function In addition to the autocorrelation between X t and X t + k, we may want to investigate the correlation between X t and X t + k after their mutual linear dependency on the intervening variables Xt+, Xt+ 2,, Xt+ k has been removed., without For the stationary time series { X t } loss of generality, assume that ( t ) 0 E X =. Let the linear dependence of X t + k on Xt+, Xt+ 2,, Xt+ k be defined as the best linear estimate in the mean square sense of X t + kas a linear function of Xt+, Xt+ 2,, Xt+ k 30

31 That is, if X ˆ t + k denotes the best linear estimate of X +, then t k ˆ t + = α k t + k + α2 t + k α k t + X X X X, where α i, ( i k ) are the mean square linear regression coefficients obtained by minimizing ( ˆ ) t+ k t+ k = ( t+ k α t+ k αk t+ ) 2 2 E X X E X X X. 3

32 Differentiating with respect to the coefficients and equating to zero, we obtain γ = αγ + αγ + + α γ ( i k i i 2 i 2 k i k+ ) (*) Hence, = α + α + + α ( i k ). i i 2 i 2 k i k+ In terms of matrix notation, the above system becomes 2 k 2 α 2 k 3 α 2 = k k 2 k 3 k 4 αk (**) 32

33 Similarly, ˆ = β t t + + β2 t β k t + k X X X X, where β i, ( i k ) are the mean square linear regression coefficients obtained by minimizing ( ˆ ) t t = ( t β t+ βk t+ k ) 2 2 E X X E X X X. A similar calculation shows that 2 k 2 β 2 k 3 β 2 = k k 2 k 3 k 4 βk. 33

34 This, along with the linear independence of the rows (columns) of the matrix, implies that αi = βi, i k. The partial autocorrelation between X t and X + is the ordinary autocorrelation between t k ( X ˆ ) t Xt and ( X ˆ ) t+ k Xt+ k. Then if we let P k denote the partial autocorrelation between X and X +, then t t k P k = ( ˆ ),( ˆ ) t t t+ k t+ k Cov X X X X ( ˆ ) ( ˆ ) t t t+ k t+ k Var X X Var X X. 34

35 Now, ( ˆ ) ( ) 2 t+ k t+ k = t+ k α t+ k αk t+ Var X X E X X X ( α α ) = E X X X X t+ k t+ k t+ k k t+ ( ) α E X X α X α X t+ k t+ k t+ k k t+ ( ) α E X X α X α X k t+ t+ k t+ k k t+ ( α α ) = E X X X X t+ k t+ k t+ k k t+ since all other remaining terms reduce to zero by virtue of (*). Hence, ( ˆ ) ( ˆ ) t+ k t+ k = t t = 0 k k Var X X Var X X γ αγ α γ 35

36 Next, using the fact that αi = βi, i k, ( ˆ ),( ˆ ) t t t+ k t+ k Cov X X X X ( t α t+ αk t+ k )( t+ k α t+ k αk t+ ) = E ( X α X α X ) X = E X X X X X X t t+ k t+ k t+ k = γ αγ α γ. k k k Therefore, P k γk αγ k αk γ k α k αk = = γ αγ α γ α α 0 k k k k 36

37 Solving the system (**) by Cramer s rule gives α = i i 2 i k 2 i 3 2 i k 3 k 2 k 3 k i k k i 2 i 2 i i k 2 i 3 i i k 3 k 2 k 3 k i k i k i 2 The matrix in the numerator is the same as that in the denominator except for its ith column being replaced by ( ).,,, k 2 37

38 Substituting the last expression for α i into the formula for P k and multiplying both the numerator and denominator of the derived expression by i 2 i k 2 i 3 2 i k 3 k 2 k 3 k i k k i 2 the resulting P k is P k = 2 k 2 k 3 2 k k 2 k 3 k 2 k 2 k k 3 k 2 k k 2 k 3, 38

39 As an example, for k = 3, we have P 3 = 3 α 2 α 2 α α 2 2, where α 2 = and α 2 2 =. 39

40 Substituting, P 3 = = =

41 The partial autocorrelation can also be derived as follows. Consider the regression model, where the dependent variable X t+ k from a mean zero stationary process is regressed on k lagged variables Xt+ k, Xt+ k 2,, Xt, i.e. X = φ X + φ X + + φ X + e, t+ k k t+ k k2 t+ k 2 kk t t+ k where φ ki denotes the ith regression parameter and e t+ k is a normal error term uncorrelated with X t + k jfor j. Multiplying by X t+ k j on both sides of the regression equation and taking expectations, we obtain γ j = φkγ j + φk 2γ j φkkγ j k and hence, = φ + φ + + φ. j k j k 2 j 2 kk j k 4

42 For j =,, k, we have the following system of equations: = φ + φ + + φ k 0 k 2 kk k = φ + φ + + φ 2 k k 2 0 kk k 2 = φ k + φ + + φ k k k 2 k 2 kk 0 Using Cramer s rule successively for k =, 2, we have φ 22 φ =, 2 =, 42

43 φ 33 = φ kk = 2 k 2 k 3 2 k k 2 k 3 k 2 k 2 k k 3 k 2 k k 2 k 3 Thus φ kk = Pk. 43

44 Example.5. For a white noise sequence of uncorrelated random variables e : t = 0, ±, ± 2,, { } t γ h σ 2, h = 0 =, and 0 otherwise h, h = 0 =. 0 otherwise Also, φ kk, k = 0 (by definition) =. 0 otherwise 44

45 .6 Time series models We now consider the modeling of time series as stochastic processes. The model defines the mechanism through which the observations are considered to have been generated. A simple example is the model Xt = et + θet, t =,, n, where e t, t = 0,,, n is a sequence of uncorrelated random variables drawn from a distribution 2 with mean zero and variance σ, and θ is a parameter. 45

46 A particular set of values of e 0, e,, en results in a corresponding sequence of observations X,, Xn. By drawing different sets of values e0, e,, en we can generate an infinite set of realizations over t =,, n. Thus, the model effectively defines a joint distribution for the random variables X,, Xn. For a general model, the mean of the process at time t is µ = E( X ), t =,, n, the average of X t taken over all possible realizations. Also, Cov( X, X ) = E[ ( X µ )( X µ )], t =,, n τ t t+ τ t t t+ τ t t t 46

47 If many realizations are available, the above quantities may be estimated by ensemble averages, for example m ( j) xt j= ˆ µ = m, t =,, n t where x denotes the jth observation on ( j ) t X and m is the number of realizations. t However, in most time series problems, only a single series of observations is available. 47

48 In these situations, to carry out meaningful inference on the mean and covariance functions one needs to make restrictions of the process generating the observations, so that instead of aggregating observations at a particular point in time, we do so over time. This is where stationarity becomes necessary. Under stationarity, we can use the estimates n X n Xt t= ˆ µ = =, t n h ˆ h= ch ( ) = n ( Xt X)( Xt+ h X) t= γ. () 48

49 Under certain conditions, these statistics give consistent estimates of the mean and autovariance functions. The conditions basically require that the observations sufficiently far apart be almost uncorrelated. The quantities () may be normalized to obtain sample autocovariances ˆ h = rh ( ) = ch ( )/ c(0), h =, 2, A plot of rh ( ) against non-negative values of h is known as the sample autocorrelation function or correlogram. 49

50 The sample autocorrelations are estimates of the corresponding theoretical autorrelations for the stochastic process that is assumed to have generated the observations. Although the correlogram will tend to mirror the properties of the theoretical autocorrelation function, it will not reproduce it exactly. 50

51 The correlogram is the main tool for analyzing the properties of a time series in the time domain. Partial autocorrelations can be helpful as well. We must, however, know the autocovariance and partial autocovariance functions of different stochastic processes. We then fit a model with time domain properties similar to that of the data. Also important is knowledge of the sampling variability of the estimated autocorrelations and partial autocorrelations. 5

52 We next study the nature of the autocovariance and partial autocovariance functions for models that may be represented as a linear combination of past and present values of { } t e. 52