Stochastic Processes: I. consider bowl of worms model for oscilloscope experiment:

Size: px

Start display at page:

Download "Stochastic Processes: I. consider bowl of worms model for oscilloscope experiment:"

Prudence Long
5 years ago
Views:

1 Stochastic Processes: I consider bowl of worms model for oscilloscope experiment: SAPAscope 2.0 / 0 1 RESET SAPA2e 22, 23 II 1

2 stochastic process is: Stochastic Processes: II informally: bowl + drawing mechanism formally: family of random variables (RVs) indexed by t t called time for convenience real-valued RV: mapping from sample space to real line denote individual real-valued RVs by X t or X(t) notation: discrete parameter stochastic process: {X t } = {X t : t 2 Z}, where Z = {..., 2, 1, 0, 1, 2,...} continuous parameter stochastic process: {X(t)} = {X(t) : t 2 R}, where R = {t : 1 < t < 1} SAPA2e 22, 23, 24 II 2

3 Basic Theory for Stochastic Processes: I cumulative probability distribution function (CPDF): F t (a) P[X t apple a] can define bivariate CPDF (and multivariate CPDF): note dependence on t F t1,t 2 (a 1, a 2 ) P[X t1 apple a 1, X t2 apple a 2 ] too complicated to use as models for time series summarize using moments (assuming they exist) SAPA2e 24, 25 II 3

4 Basic Theory for Stochastic Processes: II first moment: µ t E{X t } = = = X i Z 1 Z x df t (x) (see pages 26 7) xf t (x) dx x i p(x i ) second-order central moments: if f t (x) = df t(x) dx exists if F t ( ) is a step function t 0,t 1 cov{x t0, X t1 } = E{(X t0 µ t0 )(X t1 µ t1 )} 2 t var{x t} = still too complicated! Z 1 1 (x µ t ) 2 df t (x) SAPA2e 25, 26 II 4

5 Simplification: Stationary Processes {X t } is by definition stationary if: 1. E{X t } = µ, a finite constant independent of t 2. cov{x t+, X t } = s, a finite constant that can depend on (known as the lag), but not t autocovariance sequence (ACVS) for {X t }: {s } = {s : 2 Z} autocovariance function (ACVF) for {X(t)}: s( ) = function of defined by s( ) for 2 R several terms for this type of stationarity: second-order, covariance, wide sense, weak other type of stationarity: complete (strong, strict) stationarity SAPA2e 28, 29 II 5

6 Basic Properties: I var{x t } = cov{x t, X t } = s 0 (i.e., variance constant over t) s = s (i.e., symmetric about = 0) correlation between X t and X t+ : cov{x t+, X t } corr{x t+, X t } p var{xt } var{x t+ } = cov{x t+, X t } = s var{x t } s 0 autocorrelation sequence (ACS) for {X t }: { } = { : 2 Z} autocorrelation function (ACF) for {X(t)}: ( ) = function defined by ( ) for 2 R apple 1 implies s apple s 0 SAPA2e 29 II 6

7 Basic Properties: II {s } is positive semidefinite, i.e., n 1 X j=0 n 1 X k=0 s tj t k a j a k 0 (1) holds for any positive integer n, any set of n real numbers a 0,..., a n 1 and any set of n integers t 0,..., t n 1 proof: for W P n 1 j=0 a jx tj = a T V, have 0 apple var{w } = var{a T V} [2.2] = a T a is var/cov matrix for V, i.e., its (j, k)th element is cov{x tj, X tk } = s tj t k a T a = (1) imposes severe limitation (Makhoul, 1990) SAPA2e 29, 30 II 7

8 Basic Properties: III var/cov matrix for [X 0,..., X N 1 ] T is Toeplitz; when N = 6, have 2 3 s 0 s 1 s 2 s 3 s 4 s 5 s 1 s 0 s 1 s 2 s 3 s 4 s 2 s 1 s 0 s 1 s 2 s 3 6 s 3 s 2 s 1 s 0 s 1 s s 4 s 3 s 2 s 1 s 0 s 1 5 s 5 s 4 s 3 s 2 s 1 s 0 if {X t } Gaussian, completely characterized by µ and {s } extension: complex-valued stationary process {Z t } cov{z t+, Z t } E{(Z t+ µ)(z t µ) } = s (asterisk indicates complex conjugate) now have s = s (rather than s = s ) SAPA2e 30, 31 II 8

9 Example White Noise Process assume E{X t } = µ and var{x t } = 2 (< 1) {X t } called white noise if, when 6= 0, X t+ and X t are uncorrelated; i.e., cov{x t+, X t } = 0 white noise is stationary process with ACVS: ( 2, = 0; s = 0, otherwise six realizations of white noise, all with µ = 0 and 2 = 1 SAPA2e 32, 33 II 9

10 White Noise with Gaussian Distribution t SAPA2e 33 II 10

11 White Noise with Uniform Distribution t SAPA2e 33 II 11

12 White Noise with Double Exponential Distribution t SAPA2e 33 II 12

13 White Noise with Discrete Distribution t SAPA2e 33 II 13

14 White Noise with Random Distributions t SAPA2e 33 II 14

15 White Noise with Blocky Distribution t SAPA2e 33 II 15

16 White Noise and IID Noise all six examples have exactly the same second-order properties! as fifth and sixth examples illustrate, no need for distributions of X t and X t+1 to be the same for second-order stationarity to hold related concept: IID noise {X t : t 2 Z} are independent and identically distributed all IID noise with finite variance is white noise, but converse is false SAPA2e 33 II 16

17 Example Moving Average Process: I let { t } be white noise, mean 0, variance construct MA(q) process: qx X t = µ q,j t j = µ j=0 2 1X j= 1 q,j t j, where q,0 6= 0, q,q 6= 0 and q,j = 0 for j < 0 or j > q claim: {X t } is stationary, which follows if we can show that E{X t } does not depend on t cov{x t+, X t } does not depend on t (but can depend on ) now E{X t } = µ (why is this true?) SAPA2e 34 II 17

18 Example Moving Average Process: II since X t µ = P j q,j t j, have < cov{x t+, X t } = X : q,k t+ k k = X X q,k q,j E{ t+ k t j } k j = X q,j+ 2 q,j (why?) j s note: no restrictions on q,j s X j q,j t j 19 = A ; SAPA2e 34 II 18

19 Example Moving Average Process: III in terms of just q,0,..., q,q, can reexpress ACVS as ( 2 P q s = j=0 q,j+ q,j, apple q; 0, otherwise note: ACVS for MA(q) process is identically zero when > q note: variance of MA(q) process is qx s 0 = 2 j=0 2 q,j two realizations of MA(1) process X t = t 1,1 t 1 constructed from Gaussian white noise { t }, one with 1,1 = +1, and the other, 1,1 = 1 SAPA2e 34 II 19

20 MA(1) Process with 1,1 = +1; 1 = s 1 /s 0 = 1/ t SAPA2e 35 II 20

21 MA(1) Process with 1,1 = 1; 1 = 1/ t SAPA2e 35 II 21

22 Example Autoregressive Process let { t } be white noise, mean 0, variance construct AR(p) process px X t = µ + j=1 p,j(x t j µ) + t with p,p 6= 0 claim: {X t } is stationary under restrictions on p,j s proof: Priestley (1981) rich class of processes (Chapter 8) extension: ARMA(p,q) process (Box & Jenkins, 1970) 2 p SAPA2e 34, 35, 37 II 22

23 Second Order Autoregressive Process 1st example of realization of AR(2) process X t = 0.75X t 1 0.5X t 2 + t with 2 2 = t SAPA2e 36 II 23

24 Second Order Autoregressive Process 2nd example of realization of AR(2) process X t = 0.75X t 1 0.5X t 2 + t with 2 2 = t SAPA2e 36 II 23

25 Second Order Autoregressive Process 3rd example of realization of AR(2) process X t = 0.75X t 1 0.5X t 2 + t with 2 2 = t SAPA2e 36 II 23

26 Second Order Autoregressive Process 4th example of realization of AR(2) process X t = 0.75X t 1 0.5X t 2 + t with 2 2 = t SAPA2e 36 II 23

27 Fourth Order Autoregressive Process 1st example of realization of AR(4) process (here 2 4 = 0.002) X t = X t X t X t X t 4 + t t SAPA2e 36 II 24

28 Fourth Order Autoregressive Process 2nd example of realization of AR(4) process (here 2 4 = 0.002) X t = X t X t X t X t 4 + t t SAPA2e 36 II 24

29 Fourth Order Autoregressive Process 3rd example of realization of AR(4) process (here 2 4 = 0.002) X t = X t X t X t X t 4 + t t SAPA2e 36 II 24

30 Fourth Order Autoregressive Process 4th example of realization of AR(4) process (here 2 4 = 0.002) X t = X t X t X t X t 4 + t t SAPA2e 36 II 24

31 Example Harmonic Process: I for fixed L > 0, let A 1,..., A L, B 1,..., B L be set of uncorrelated zero mean RVs such that var {A l } = var {B l } l 2 construct harmonic process: LX X t = µ + A l cos (2 f l t) + B l sin (2 f l t), l=1 where µ and f l s are real-valued constants can reexpress harmonic process as LX X t = µ + D l cos (2 f l t + l ), l=1 where Dl 2 = A2 l + B2 l and tan ( l ) = B l /A l note: also have A l = D l cos ( l ) and B l = D l sin ( l ) SAPA2e 37 II 25

32 First Order Gaussian Harmonic Process 1st example of realization of harmonic process X t = A cos ( t) + B sin ( t), where A & B are independent Gaussian RVs with unit variance t SAPA2e 38 II 26

33 First Order Gaussian Harmonic Process 2nd example of realization of harmonic process X t = A cos ( t) + B sin ( t), where A & B are independent Gaussian RVs with unit variance t SAPA2e 38 II 26

34 First Order Gaussian Harmonic Process 3rd example of realization of harmonic process X t = A cos ( t) + B sin ( t), where A & B are independent Gaussian RVs with unit variance t SAPA2e 38 II 26

35 Example Harmonic Process: II claim: harmonic process LX X t = µ + A l cos (2 f l t) + B l sin (2 f l t) is stationary (?!) l=1 proof is subject of Exercise [38], which shows ACVS to be s = LX l=1 2 l cos (2 f l ) (same f l s as making up X t, but cosine terms are all in phase ) SAPA2e 37, 38 II 27

36 Example Harmonic Process: III specialize to case where RVs A l and B l are Gaussian recall, if Y 0, Y 1,..., Y 1 are IID zero mean Gaussian RVs with unit variance, 2 Y0 2 + Y Y 2 1 said to have chi-square distribution with degrees of freedom since A l / l and B l / l are independent zero mean, unit variance Gaussian RVs, Dl 2/ l 2 = (A 2 l + B2 l )/ l 2 has chi-square distribution with 2 degrees of freedom probability density function (PDF) for RV given by ( e u/2 /2, u 0; f 22 (u) 0, u < SAPA2e 38, 39 II 28

37 Example Harmonic Process: IV can deduce PDF for Dl 2 from fact that PDF for D2 l / l 2 is 2 2 : ( e u/(2 l 2) /(2 f D 2(u) l 2 ), u 0; l 0, u < 0 above special case of exponential PDF f(u) = exp( u/ )/ with mean value = 2 2 l random amplitude D l is thus the square root of an exponential RV and is said to obey a Rayleigh distribution symmetry of bivariate distribution for A l & B l dictates to be uniformily distributed over (, ] and independent of D l thus Gaussian harmonic processes involve Rayleigh distributed D l s and uniformily distributed l s, with all 2L RVs being independent of one another SAPA2e 39 II 29

38 Example Harmonic Process: V now consider X t = µ + LX D l cos (2 f l t + l ), l=1 l s are IID RVs uni- where D l s are positive constants, while formily distributed over (, ] X t is necessarily bounded and hence cannot have a Gaussian distribution SAPA2e 39 II 30

39 First-Order Non-Gaussian Process X t 1st example of realization of process X t = p 2 cos ( t + ), where is uniformily distributed over interval (, ] t SAPA2e 38 II 31

40 First-Order Non-Gaussian Process X t 2nd example of realization of process X t = p 2 cos ( t + ), where is uniformily distributed over interval (, ] t SAPA2e 38 II 31

41 First-Order Non-Gaussian Process X t 3rd example of realization of process X t = p 2 cos ( t + ), where is uniformily distributed over interval (, ] t SAPA2e 38 II 31

42 Example Harmonic Process: VI claim: X t = µ + is a stationary process LX D l cos (2 f l t + l ), l=1 to prove when L = 1, i.e., X t = µ + D cos (2 ft + show E{X t } & cov{x t+, X t } do not depend on t now E{X t } = µ + D E{cos (2 ft + )} = µ + D Z ), need to cos (2 ft + ) 1 2 d = µ (why?) SAPA2e 39 II 32

43 Example Harmonic Process: VII since X t µ = D cos (2 ft + ), have cov{x t+, X t } = E {(D Z cos (2 f[t + ] + )) (D cos (2 ft + ))} = D 2 cos (2 f[t + ] + ) cos (2 ft + ) 1 2 d Z trig = D2 cos (2 f[2t + ] + 2 ) + cos (2 f ) d 4 = D 2 cos (2 f )/2 s thus can dispense with random amplitudes! basis for spectral representation theorem (Chapter 4) Exercises [2.20] & [2.21]: non-uniformily distributed l doesn t work, and neither do uncorrelated uniformily distributed l s SAPA2e 39, 40, 49, 50 II 33

44 Stationary Processes as Models: I stationarity property of models, not data spectral analysis assumes stationarity need to examine assumption for each time series example: spinning rotor series (Figure 41) bin number SAPA2e 41, 42 II 34

45 Stationary Processes as Models: II can detrend using linear regression model: X t = + t + Y t letting ˆ and ˆ denote least squares estimates, detrended series gotten from Ŷ t = X t ˆ + ˆt bin number SAPA2e 41, 42 II 35

46 Stationary Processes as Models: III can also get detrended series by first di erencing (filtering): X (1) t X t X t 1 = + Y (1) t, where Y (1) t Y t Y t bin number SAPA2e 41, 42 II 36

47 Stationary Processes as Models: IV second example: standard resistor series (Figure 43) days (from 1/1/80) SAPA2e 43 II 37

Statistics of stochastic processes

Introduction Statistics of stochastic processes Generally statistics is performed on observations y 1,..., y n assumed to be realizations of independent random variables Y 1,..., Y n. 14 settembre 2014