Linear Stochastic Models. Special Types of Random Processes: AR, MA, and ARMA. Digital Signal Processing

Transcription

1 Linear Stochastic Models Special Types of Random Processes: AR, MA, and ARMA Digital Signal Processing Department of Electrical and Electronic Engineering, Imperial College c Danilo P. Mandic Digital Signal Processing

2 Motivation:- Wold Decomposition Theorem The most fundamental justification for time series analysis is due to Wold s decomposition theorem, where it is explicitly proved that any (stationary) time series can be decomposed into two different parts. Therefore, a general random process can be written a sum of two processes x[n] = x p [n] + x r [n] x r [n] regular random process x p [n] predictable process, with x r [n] x p [n], E{x r [m]x p [n]} = that is we can separately treat the predictable process (i.e. a deterministic signal) and a random signal. c Danilo P. Mandic Digital Signal Processing 2

3 What do we actually mean? a) Periodic oscillations b) Small nonlinearity c) Route to chaos d) Route to chaos e) small noise f) HMM and others c Danilo P. Mandic Digital Signal Processing 3

4 Example from brain science Electrode positions Raw EEG Useful signal c Danilo P. Mandic Digital Signal Processing 4

5 Linear Stochastic Processes It therefore follows that the general form for the power spectrum of a WSS process is P x (e jω ) = P xr (e jω ) + N α k u (ω ω k ) k= We look at processes generated by filtering white noise with a linear shift invariant filter that has a rational system function. These include the Autoregressive (AR) all pole system Moving Average (MA) all zero system Autoregressive Moving Average (ARMA) poles and zeros Notice the difference between shift invariance and time invariance c Danilo P. Mandic Digital Signal Processing 5

6 ACF and Spectrum of ARMA models Much of interest are the autocorrelation function and power spectrum of these processes. (Recall that ACF PSD in terms of the available information) Suppose that we filter white noise w[n] with a causal linear shift invariant filter having a rational system function with p poles and q zeros H(z) = B q(z) A p (z) = q k= b q(k)z k + p k= a p(k)z k Assuming that the filter is stable, the output process x[n] will be wide sense stationary and with P w = σ 2 w, the power spectrum of x[n] will be P x (z) = σw 2 B q (z)b q (z ) A p (z)a p (z ) Recall that ( ) in analogue frequency corresponds to z in digital freq. c Danilo P. Mandic Digital Signal Processing 6

7 Frequency Domain In terms of digital frequency θ (unit circle e jθ = e jωt ) B q (z)b q (z ) quadratic form and real valued A p (z)a p (z ) quadratic form and real valued B P z (e jθ ) = σw 2 q (e jθ ) 2 A p (e jθ ) 2 We are therefore using H(z) to shape the spectrum of white noise. A process having a power spectrum of this form is known as an autoregressive moving average process of order (p,q) and is referred to as an ARMA(p,q) process c Danilo P. Mandic Digital Signal Processing 7

8 Example Plot the power spectrum of an ARMA(2,2) process for which the zeros of H(z) are z =.95e ±jπ/2 poles are at z =.9e ±j2π/5 Solution: The system function is (poles and zeros resonance & sink) H(z) = +.925z z +.8z 2 Power Spectrum Frequency c Danilo P. Mandic Digital Signal Processing 8

9 Difference Equation Representation Random processes x[n] and w[n] are related by the linear constant coefficient equation x[n] p a p (l)x[n l] = l= q b q (l)w[n l] l= Notice that the autocorrelation function of x[n] and crosscorrelation between x[n] and w[n] follow the same difference equation, i.e. if we multiply both sides of the above equation by x[n k] and take the expected value, we have r xx (k) p a p (l)r xx (k l) = l= q b q (l)r xw (k l) l= Since x is WSS, it follows that x[n] and w[n] are jointly WSS. c Danilo P. Mandic Digital Signal Processing 9

10 General Linear Processes: Stationarity and Invertibility Consider a linear stochastic process output from a linear filter, driven by WGN w[n] x[n] = w[n] + b w[n ] + b 2 w[n 2] + = w[n] + that is, a weighted sum of past inputs w[n]. b j w[n j] For this process to be a valid stationary process, the coefficients must be absolutely summable, that is j= b j <. The model implies that under suitable condition, x[n] is also a weighted sum of past values of x, plus an added shock w[n], that is x[n] = a x[n ] + a 2 x[n 2] + + w[n] Linear Process is stationary if j= b j < Linear Process is invertible if j= a j < j= c Danilo P. Mandic Digital Signal Processing

11 Are these ARMA(p,q) processes? Unit response u[n] = {, n <, n If w[n] = δ[n] then u[n] = u[n ] + w[n], n Ramp function r[n] = {, n < n, n If w[n] = u[n] then r[n] = r[n ] + w[n], n c Danilo P. Mandic Digital Signal Processing

12 Autoregressive Processes A general AR(p) process (autoregressive of order p) is given by p x[n] = a x[n ] + + a p x[n p] + w[n] = a i x[n i] + w[n] i= Observe the auto regression above Duality between AR and MA processes: For instance the first order autoregressive process x[n] = a x[n ] + w[n] b j w[n j] j= Due to its all pole nature follows the duality between IIR and FIR filters. c Danilo P. Mandic Digital Signal Processing 2

13 ACF and Spectrum of AR Processes To obtain the autocorrelation function of an AR process, multiply the above equation by x[n k] to obtain x[n k]x[n] = a x[n k]x[n ] + a 2 x[n k]x[n 2] + +a p x[n k]x[n p] + x[n k]w[n] Notice that E{x[n k]w[n]} vanishes when k >. Therefore we have r xx (k) = a r xx (k ) + a 2 r xx (k 2) + + a p r xx (k p) k > On dividing throughout by r xx () we obtain ρ(k) = a ρ(k ) + a 2 ρ(k 2) + + a p ρ(k p) k > Parameters ρ(k) are correlation coefficients c Danilo P. Mandic Digital Signal Processing 3

14 Variance and Spectrum of AR Processes Variance: When k = the contribution from the term E{x[n k]w[n]} is σ 2 w, and r xx () = a r xx ( ) + a 2 r xx ( 2) + + a p r xx ( p) + σ 2 w Divide by r xx () = σ 2 x to obtain σ 2 x = σ 2 w ρ a ρ 2 a 2 ρ p a p Spectrum: P xx (f) = 2σ 2 w a e j2πf a p e j2πpf 2 f /2 Recall Spectrum of linear systems from the Course Introduction c Danilo P. Mandic Digital Signal Processing 4

15 Yule Walker Equations For k =, 2,..., p a set of equations:- from the general autocorrelation function, we obtain r xx () = a r xx () + a 2 r xx () + + a p r xx (p ) r xx (2) = a r xx () + a 2 r xx () + + a p r xx (p 2). =. r xx (p) = a r xx (p ) + a 2 r xx (p 2) + + a p r xx () These equations are called the Yule Walker or normal equations. Their solution gives us the set of autoregressive parameters a = [a,...,a p ] T. This can be expressed in a vector matrix form as a = R xxr xx Due to Toeplitz structure of R xx, its positive definitness enables matrix inversion c Danilo P. Mandic Digital Signal Processing 5

16 ACF Coefficients For the autocorrelation coefficients ρ k = r xx (k)/r xx () we have ρ = a + a 2 ρ + + a p ρ p ρ 2 = a ρ + a a p ρ p 2. =. ρ p = a ρ p + a 2 ρ p a p When does the sequence {ρ,ρ,ρ 2,...} vanish? Homework:- Try command xcorr in Matlab c Danilo P. Mandic Digital Signal Processing 6

17 Example:- Yule Walker modelling in Matlab In Matlab Power spectral density using Y W method pyulear Pxx = pyulear(x,p) [Pxx,w] = pyulear(x,p,nfft) [Pxx,f] = pyulear(x,p,nfft,fs) [Pxx,f] = pyulear(x,p,nfft,fs, range ) [Pxx,w] = pyulear(x,p,nfft, range ) Description:- Pxx = pyulear(x,p) implements the Yule-Walker algorithm, and returns Pxx, an estimate of the power spectral density (PSD) of the vector x. To remember for later This estimate is also an estimate of the maximum entropy. Se also aryule, lpc, pburg, pcov, peig, periodogram c Danilo P. Mandic Digital Signal Processing 7

18 Example:- AR(p) signal generation Generate the input signal x by filtering white noise through the AR filter Estimate the PSD of x based on a fourth-order AR model Solution:- randn( state,); x = filter(,a,randn(256,)); pyulear(x,4) % AR system output % Fourth-order estimate c Danilo P. Mandic Digital Signal Processing 8

19 Alternatively:- Yule Walker modelling AR(4) system given by y[n] = 2.237y[n ] 2.943y[n 2]+2.697y[n 3].966y[n 4]+w[n] a = [ ]; % AR filter coefficients freqz(,a) % AR filter frequency response title( AR System Frequency Response ) c Danilo P. Mandic Digital Signal Processing 9

20 From Data to AR(p) Model So far, we assumed the model (AR, MA, or ARMA) and analysed the ACF and PSD based on known model coefficients. In practice:- DATA MODEL This procedure is as follows:- * record data x(k) * find the autocorrelation of the data ACF(x) * divide by r_xx() to obtain correlation coefficients \rho(k) * write down Yule-Walker equations * solve for the vector of AR paramters The problem is that we do not know the model order p beforehand; we will deal with this problem later in Lecture 2. c Danilo P. Mandic Digital Signal Processing 2

21 Example:- Finding parameters of x[n] =.2x[n ].8x[n 2] + w[n] 6 AR(2) signal x=filter([],[,.2,.8],w) 2 ACF for AR(2) signal x=filter([],[,.2,.8],w) ACF for AR(2) signal x=filter([],[,.2,.8],w) AR(2) signal values 2 2 ACF of AR(2) signal 5 5 ACF of AR(2) signal Apply:- Sample number Correlation lag 2 2 Correlation lag for i=:6; [a,e]=aryule(x,i); display(a);end a () = [.6689] a (2) = [.246,.88] a (3) = [.759,.7576,.358] a (4) = [.762,.753,.456,.83] a (5) = [.763,.752,.562,.248,.4] a (6) = [.762,.758,.565,.98,.62,.67] c Danilo P. Mandic Digital Signal Processing 2

22 Special case:- AR() Process (Markov) Given below (Recall p(x[n],x[n ],...,x[]) = p(x[n] x[n ])) x[n] = a x[n ] + w[n] = w[n] + a x[n ] + a 2 w[n 2] + i) for the process to be stationary < a <. ii) Autocorrelation Function:- from Yule-Walker equations r xx (k) = a r xx (k ), k > or for the correlation coefficients, with ρ = ρ k = a k, k > Notice the difference in the behaviour of the ACF for a positive and negative c Danilo P. Mandic Digital Signal Processing 22

23 Variance and Spectrum of AR() process Can be calculated directly from a general expression of the variance and spectrum of AR(p) processes. Variance:- Also from a general expression for the variance of linear processes from Lecture σ 2 x = σ2 w ρ a = σ2 w a 2 Spectrum:- Notice how the flat PSD of WGN is shaped according to the position of the pole of AR() model (LP or HP) P xx (f) = 2σ 2 w a e j2πf 2 = 2σ 2 w + a 2 2a cos(2πf) c Danilo P. Mandic Digital Signal Processing 23

24 Example: ACF and Spectrum of AR() for a = ±.8 4 x[n] =.8*x[n ] + w[n] 5 x[n] =.8*x[n ] + w[n] Signal values 2 2 Signal values Sample Number ACF Sample Number ACF Correlation.5.5 Correlation.5 Power/frequency (db/rad/sample) Correlation lag Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) a < High Pass Power/frequency (db/rad/sample) Correlation lag Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) a > Low Pass c Danilo P. Mandic Digital Signal Processing 24

25 Special Case:- Second Order Autoregressive Processes AR(2) The input output functional relationship is given by x[n] = a x[n ] + a 2 x[n 2] + w[n] For stationarity- (to be proven later) a + a 2 < a 2 a < < a 2 < This will be shown within the so called stability triangle c Danilo P. Mandic Digital Signal Processing 25

26 Work by Yule Modelling of sunspot numbers Recorded for more than 3 years. In 927, Yule modelled them and invented AR(2) model 5 Sunspot series Signal values Sample Number ACF for sunspot series Correlation Correlation lag Sunspot numbers and its autocorrelation function c Danilo P. Mandic Digital Signal Processing 26

27 Autocorrelation function of AR(2) processes The ACF ρ k = a ρ k + a 2 ρ k 2 k > Real roots: (a 2 + 4a 2 > ) ACF = mixture of damped exponentials Complex roots: (a 2 + 4a 2 < ) ACF exhibits a pseudo periodic behaviour ρ k = Dk sin(2πf k + F) sinf D - damping factor, of a sine wave with frequency f and phase F. D = a 2 cos(2πf ) = a 2 a 2 tan(f) = + D2 D 2 tan(2πf ) c Danilo P. Mandic Digital Signal Processing 27

28 Stability Triangle a 2 ACF ACF II I m Real Roots m 2 2 a ACF ACF III m Complex Roots m IV i) Real roots Region : Monotonically decaying ACF ii) Real roots Region 2: Decaying oscillating ACF iii) Complex roots Region 3: Oscilating pseudoperiodic ACF iv) Complex roots Region 4: Pseudoperiodic ACF c Danilo P. Mandic Digital Signal Processing 28

29 Yule Walker Equations Substituting p = 2 into Y-W equations we have ρ = a + a 2 ρ ρ 2 = a ρ + a 2 which when solved for a and a 2 gives a = ρ ( ρ 2 ) ρ 2 or substituting in the equation for ρ a 2 = ρ 2 ρ 2 ρ 2 ρ = a a 2 ρ 2 = a 2 + a2 a 2 c Danilo P. Mandic Digital Signal Processing 29

30 Variance and Spectrum More specifically, for the AR(2) process, we have:- Variance Spectrum σ 2 x = σ 2 w ρ a ρ 2 a 2 = ( ) a2 + a 2 σ 2 w ( a 2 ) 2 a 2 P xx (f) = = 2σ 2 w a e j2πf a 2 e j4πf 2 2σ 2 w + a 2 + a2 2 2a ( a 2 cos(2πf) 2a 2 cos(4πf)), f /2 c Danilo P. Mandic Digital Signal Processing 3

31 Example AR(2): x[n] =.75x[n ].5x[n 2] + w[n] 2 x[n] =.75*x[n 2].5*x[n ] + w[n] Signal values Sample Number ACF Correlation.5 Power/frequency (db/rad/sample) Correlation lag Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) The damping factor D =.5 =.7, frequency f = cos (.533) The fundamental period of the autocorrelation function is π = 6.2 c Danilo P. Mandic Digital Signal Processing 3

32 Partial Autocorrelation Function:- Motivation Let us revisit example from page 2 of Lecture Slides. AR(2) signal values AR(2) signal x=filter([],[,.2,.8],w) ACF for AR(2) signal x=filter([],[,.2,.8],w) ACF for AR(2) signal x=filter([],[,.2,.8],w) ACF of AR(2) signal ACF of AR(2) signal Sample number Correlation lag 2 2 Correlation lag We do not know p, let us re-write the coefficients as [a_p,...,a_p p = [.6689] = a p = 2 [.246,.88] = [a 2,a 22 ] p = 3 [.759,.7576,.358] = [a 3, a 32, a 33 ] p = 4 [.762,.753,.456,.83] = [a 4,a 42,a 43, a 44 ] p = 5 [.763,.752,.562,.248,.4] = [a 5,...,a 55 ] p = 6 [.762,.758,.565,.98,.62,.67] = [a 6,...,a 66 ] c Danilo P. Mandic Digital Signal Processing 32

33 Partial Autocorrelation Function Notice: ACF of AR(p) infinite in duration, but can by be described in terms of p nonzero functions ACFs. Denote by a kj the jth coefficient in an autoregressive representation of order k, so that a kk is the last coefficient. Then ρ j = a kj ρ j + + a k(k ) ρ j k+ + a kk ρ j k j =, 2,...,k leading to the Yule Walker equation, which can be written as ρ ρ 2 ρ k ρ ρ ρ k ρ k ρ k 2 ρ k 3 a k a k2. a kk = ρ ρ 2. ρ k c Danilo P. Mandic Digital Signal Processing 33

34 Partial ACF Coefficients: Solving these equations for k =, 2,... successively, we obtain a = ρ, a 22 = ρ 2 ρ 2 ρ 2, a 33 = ρ ρ ρ ρ 2 ρ 2 ρ ρ 3 ρ ρ 2 ρ ρ ρ 2 ρ, etc The quantity a kk, regarded as a function of lag k, is called the partial autocorrelation function. For an AR(p) process, the PAC a kk will be nonzero for k p and zero for k > p tells us the order of an AR(p) process. c Danilo P. Mandic Digital Signal Processing 34

35 Importance of Partial ACF For a zero mean process x[n], the best linear predictor in the mean square error sense of x[n] based on x[n ], x[n 2],... is ˆx[n] = a k, x[n ] + a k,2 x[n 2] + + a k,k x[n k + ] (apply the E{ } operator to the general AR(p) model expression, and recall that E{w[n]} = ) (Hint: E{x[n]} = ˆx[n] = E {a k, x[n ] + + a k,k x[n k + ] + w[n]} = a k, x[n ] + + a k,k x[n k + ]) ) whether the process is an AR or not In MATLAB, check the function: ARYULE and functions PYULEAR, ARMCOV, ARBURG, ARCOV, LPC, PRONY c Danilo P. Mandic Digital Signal Processing 35

36 Model order for Sunspot numbers 5 Sunspot series ACF for sunspot series Signal values 5 Correlation.5 Correlation Sample Number.5.5 Partial ACF for sunspot series Correlation lag Correlation lag Power/frequency (db/rad/sample) Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) Sunspot numbers, their ACF and partial autocorrelation (PAC) After lag k = 2, the PAC becomes very small c Danilo P. Mandic Digital Signal Processing 36

37 Model order for AR(2) generated process 8 AR(2) signal ACF for AR(2) signal 6 Signal values Correlation Correlation Sample Number Partial ACF for AR(2) signal Correlation lag Power/frequency (db/rad/sample) Correlation lag Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) AR(2) signal, its ACF and partial autocorrelation (PAC) After lag k = 2, the PAC becomes very small c Danilo P. Mandic Digital Signal Processing 37

38 Model order for AR(3) generated process AR(3) signal ACF for AR(3) signal 5.8 Signal values 5 Correlation Correlation Sample Number Partial ACF for AR(3) signal Correlation lag Power/frequency (db/rad/sample) Correlation lag Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) AR(3) signal, its ACF and partial autocorrelation (PAC) After lag k = 3, the PAC becomes very small c Danilo P. Mandic Digital Signal Processing 38

39 Model order for a financial time series From:- Nasdaq ascending Nasdaq descending 26 Nasdaq composite June 23 February Nasdaq composite February 27 June Nasdaq value Nasdaq value Day number ACF of Nasdaq composite June 23 February 27 7 x Day number ACF of Nasdaq composite June 23 February 27 7 x ACF value 3 2 ACF value Correlation lag Correlation lag c Danilo P. Mandic Digital Signal Processing 39

40 Partial ACF for financial time series a = a = a = a = a = a = c Danilo P. Mandic Digital Signal Processing 4

41 Model Order Selection Practical issues In practice the greater the model order the higher the accuracy When do we stop? To save on computational complexity, we introduce penalty for a high model order. The criteria for model order selection are, for instance MDL (minimum description length - Rissanen), AIC (Akaike Information criterion), given by MDL = log(e) + p log(n) N AIC = log(e) + 2p/N E = the loss function (typically cumulative squared error, p = the number of estimated parameters N = the number of estimated data. c Danilo P. Mandic Digital Signal Processing 4

42 Example:- Model order selection MDL vs AIC Let us have a look at the squared error and the MDL and AIC criteria for an AR(2) model with a =.5 a 2 =.3.98 MDL Cumulative Squared Error MDL for AR(2) AIC for AR(2) AIC Cumulative Squared Error AR(2) Model Order (Model error) 2 versus the model order p c Danilo P. Mandic Digital Signal Processing 42

43 Moving Average Processes A general MA(q) process is given by x[n] = w[n] + b w[n ] + + b q w[n q] Autocorrelation function: The autocovariance function of MA(q) c k = E[(w[n] + b w[n ] + + b q w[n q])(w[n k] +b w[n k ] + + b q w[n k q])] Hence the variance of the process c = ( + b b 2 q)σw 2 The ACF of an MA process has a cutoff after lag q. Spectrum: All zeros struggles to model PSD with peaks P(f) = 2σw 2 + b e j2πf + b 2 e j4πf + + b q e j2πqf c Danilo P. Mandic Digital Signal Processing 43

44 Example:- MA(3) process 3 MA(3) signal.2 ACF for MA(3) signal Signal values Sample Number Correlation Correlation lag Correlation Partial ACF for MA(3) signal Correlation lag Power/frequency (db/rad/sample) Burg Power Spectral Density Estimate Normalized Frequency ( π rad/sample) MA(3) model, its ACF and partial autocorrelation (PAC) After lag k = 3, the ACF becomes very small c Danilo P. Mandic Digital Signal Processing 44

45 Analysis of Nonstationary Signals Signal values.5 W Speech Signal W2 W Sample Number Partial ACF for W Partial ACF for W2 Partial ACF for W3 Correlation Correlation.5.5 Correlation Correlation lag MDL calculated for W 25 5 Correlation lag MDL calculated for W Correlation lag MDL calculated for W3 MDL.8.6 Calculated Model Order = 3 MDL.5 Calculated Model Order > 5 MDL.8.6 Calculated Model Order = Model Order 25 5 Model Order Model Order Different AR models for different segments of speech To deal with nonstationarity we need short sliding windows c Danilo P. Mandic Digital Signal Processing 45

46 Duality Between AR and MA Processes i) A stationary finite AR(p) process can be represented as an infinite order MA process. A finite MA process can be represented as an infinite AR process. ii) The finite MA(q) process has an ACF that is zero beyond q. For an AR process, the ACF is infinite in extent and consits of mixture of damped exponentials and/or damped sine waves. iii) Parameters of finite MA process are not required to satisfy any condition for stationarity. However, for invertibility, the roots of the characteristic equation must lie inside the unit circle. ARMA modelling is a classic technique which has found a tremendous number of applications c Danilo P. Mandic Digital Signal Processing 46