Wavelet Methods for Time Series Analysis. Part IV: Wavelet-Based Decorrelation of Time Series

Wavelet Methods for Time Series Analysis Part IV: Wavelet-Based Decorrelation of Time Series DWT well-suited for decorrelating certain time series, including ones generated from a fractionally differenced (FD) process on synthesis side, leads to DWT-based simulation of FD processes wavelet-based bootstrapping on analysis side, leads to wavelet-based estimators for FD parameters test for homogeneity of variance test for trends (won t discuss see Craigmile et al., 24, for details) IV

Wavelets and FD Processes: I wavelet filters are approximate band-pass filters, with nominal pass-bands [/2 j+, /2 j ] (called jth octave band ) suppose { t } has S ( ) as its spectral density function (SDF) statistical properties of {W j,t } are simple if S ( ) has simple structure within jth octave band example: FD process with SDF S (f) = σ 2 ε [4 sin 2 (πf)] δ WMTSA: 28 284 IV 2

Wavelets and FD Processes: II FD process controlled by two parameters: δ and σ 2 ε for small f, have S (f) C f 2δ ; i.e., a power law log(s (f)) vs. log(f) is approximately linear with slope 2δ for large τ j, wavelet variance at scale τ j, namely ν 2 (τ j), satisfies ν 2 (τ j) C τ 2δ j log (ν 2 (τ j)) vs. log (τ j ) is approximately linear, slope 2δ approximately self-similar (or fractal ) FD process is stationary with long memory if < δ < /2: correlation between t & t+τ dies down slowly as τ increases WMTSA: 297, 284 IV 3

Wavelets and FD Processes: III power law model ubiquitous in physical sciences voltage fluctuations across cell membranes density fluctuations in hour glass traffic fluctuations on Japanese expressway impedance fluctuations in geophysical borehole fluctuations in the rotation of the earth -ray time variability of galaxies DWT well-suited to study FD and related processes self-similar filters used on self-similar processes key idea: DWT approximately decorrelates LMPs WMTSA: 34 IV 4

DWT of an FD Process: I 5 ˆρ,τ 5 256 52 768 24 32 t τ (lag) realization of an FD(.4) time series along with its sample autocorrelation sequence (ACS): for τ, P N τ ˆρ,τ t= t t+τ P N t= 2 t note that ACS dies down slowly WMTSA: 34 342 IV 5

DWT of an FD Process: II V 7 W 7 W 6 W 5 W 4 W 3 W 2 W 4 2 5 5 5 5 5 5 5 5 5 5 5 5 5 5 32 τ (lag) LA(8) DWT of FD(.4) series and sample ACSs for each W j & V 7, along with 95% confidence intervals for white noise WMTSA: 34 342 IV 6

ev 7 fw 7 fw 6 fw 5 fw 4 fw 3 fw 2 2 2 2 2 2 2 2 2 2 2 2 2 MODWT of an FD Process fw 2 2 2 32 τ (lag) LA(8) MODWT of FD(.4) series & sample ACSs for MODWT coefficients, none of which are approximately uncorrelated IV 7

DWT of an FD Process: III in contrast to, ACSs for W j consistent with white noise variance of W j increases with j can argue that var {W j,t } 2 j 2 j+ Z /2 j /2 j+ S (f) df C j, where C j is average value of S ( ) over [/2 j+, /2 j ] for FD process, have C j S (/2 j+ 2), where /2 j+ 2 is midpoint of interval [/2 j+, /2 j ] WMTSA: 343 344 IV 8

DWT of an FD Process: IV.. O O O O. O O O..... f plot shows cvar {W j,t } (circles) & S (/2 j+ 2) (curve) versus /2 j+ 2, along with 95% confidence intervals for var {W j,t } observed cvar {W j,t } agrees well with theoretical var {W j,t } WMTSA: 344 345 IV 9

Correlations Within a Scale and Between Two Scales let {s,τ } denote autocovariance sequence (ACVS) for { t }; i.e., s,τ = cov { t, t+τ } let {h j,l } denote equivalent wavelet filter for jth level to quantify decorrelation, can write cov {W j,t, W j,t } = L j l= L j l = h j,l h j,l s,2 j (t+) l 2 j (t +)+l, from which we can get ACVS (and hence within-scale correlations) for {W j,t }: cov {W j,t, W j,t+τ } = L j m= (L j ) s,2 j τ+m L j m l= h j,l h j,l+ m WMTSA: 345 IV

Correlations Within a Scale.2..2.2..2.2. j = j = 2 j = 3 j = 4 Haar D(4) LA(8).2 4 4 4 4 τ τ τ τ correlations between W j,t and W j,t+τ for an FD(.4) process correlations within scale are slightly smaller for Haar maximum magnitude of correlation is less than.2 WMTSA: 345 346 IV

Correlations Between Two Scales: I j = 2 j = 3 j = 4 8 8 τ 8 8 τ.2..2.2..2.2..2 8 8 τ j = j = 2 j = 3 correlation between Haar wavelet coefficients W j,t and W j,t from FD(.4) process and for levels satisfying j < j 4 WMTSA: 346 347 IV 2

Correlations Between Two Scales: II j = 2 j = 3 j = 4 8 8 τ 8 8 τ.2..2.2..2.2..2 8 8 τ j = j = 2 j = 3 same as before, but now for LA(8) wavelet coefficients correlations between scales decrease as L increases WMTSA: 346 347 IV 3

Wavelet Domain Description of FD Process DWT acts as a decorrelating transform for FD process (also true for fractional Gaussian noise, pure power law etc.) wavelet domain description is simple wavelet coefficients within a given scale approximately uncorrelated (refinement: assume st order autoregressive model) wavelet coefficients have scale-dependent variance controlled by the two FD parameters (δ and σ 2 ε) wavelet coefficients between scales also approximately uncorrelated (approximation improves as filter width L increases) WMTSA: 345 35 IV 4

DWT-Based Simulation properties of DWT of FD processes lead to schemes for simulating time series [,..., N ] T with zero mean and with a multivariate Gaussian distribution with N = 2 J, recall that = W T W, where W W 2. W = W j. W J V J WMTSA: 355 IV 5

Basic DWT-Based Simulation Scheme assume W to contain N uncorrelated Gaussian (normal) random variables (RVs) with zero mean assume W j to have variance C j S (/2 j+ 2) assume single RV in V J to have variance C J+ (see Percival and Walden, 2, for details on how to set C J+ ) approximate FD time series via Y W T Λ /2 Z, where Λ /2 is N N diagonal matrix with diagonal elements C /2,..., C /2 {z, C /2 } 2,..., C /2 {z 2,..., C /2 } J {z, C/2 J, C /2 } 2 of these N 2 of these N 4 of these J, C/2 J+ Z is vector of deviations drawn from a Gaussian distribution with zero mean and unit variance WMTSA: 355 IV 6

Refinements to Basic Scheme: I covariance matrix for approximation Y does not correspond to that of a stationary process recall W treats as if it were circular let T be N N circular shift matrix: Y Y Y T Y Y 2 = Y 2 Y 3 ; T 2 Y Y 2 = Y 3 Y Y 3 Y 2 Y 3 Y Y let κ be uniformily distributed over,..., N define e Y T κ Y e Y is stationary with ACVS given by, say, s ey,τ ; etc. WMTSA: 356 357 IV 7

Refinements to Basic Scheme: II Q: how well does {s ey,τ } match {s,τ }? due to circularity, find that s ey,n τ = s ey,τ for τ =,..., N/2 implies s ey,τ cannot approximate s,τ well for τ close to N can patch up by simulating e Y with M > N elements and then extracting first N deviates (M = 4N works well) WMTSA: 356 357 IV 8

Refinements to Basic Scheme: III 2.5 2. M = N M = 2N M = 4N.5..5 64 64 64 τ τ τ plot shows true ACVS {s,τ } (thick curves) for FD(.4) process and wavelet-based approximate ACVSs {s ey,τ } (thin curves) based on an LA(8) DWT in which an N = 64 series is extracted from M = N, M = 2N and M = 4N series WMTSA: 356 357 IV 9

Example and Some Notes 5 5 256 52 768 24 t simulated FD(.4) series (LA(8), N = 24 and M = 4N) notes: can form realizations faster than best exact method can efficiently simulate extremely long time series in realtime (e.g, N = 2 3 =, 73, 74, 824 or even longer!) effect of random circular shifting is to render time series slightly non-gaussian (a Gaussian mixture model) WMTSA: 358 36 IV 2

Wavelet-Domain Bootstrapping for many (but not all!) time series, DWT acts as a decorrelating transform: to a good approximation, each W j is a sample of a white noise process, and coefficients from different sub-vectors W j and W j are also pairwise uncorrelated variance of coefficients in W j depends on j scaling coefficients V J are still autocorrelated, but there will be just a few of them if J is selected to be large decorrelating property holds particularly well for FD and other processes with long-range dependence above suggests the following recipe for wavelet-domain bootstrapping of a statistic of interest, e.g., sample autocorrelation sequence ˆρ,τ at unit lag τ = IV 2

Recipe for Wavelet-Domain Bootstrapping. given of length N = 2 J, compute level J DWT (the choice J = J 3 yields 8 coefficients in W J and V J ) 2. randomly sample with replacement from W j to create bootstrapped vector W (b) j, j =,..., J 3. create V (b) J using st-order autoregressive parametric bootstrap 4. apply W T to W (b),..., W(b) J and V (b) J to obtain bootstrapped time series (b) and then form ˆρ (b), repeat above many times to build up sample distribution of bootstrapped autocorrelations IV 22

Illustration of Wavelet-Domain Bootstrapping V 4 W 4 W 3 W 2 W 32 64 96 28 32 64 96 28 Haar DWT of FD(.45) series (left-hand column) and waveletdomain bootstrap thereof (right-hand) IV 22

Illustration of Wavelet-Domain Bootstrapping V 4 W 4 W 3 W 2 W 32 64 96 28 32 64 96 28 Haar DWT of FD(.45) series (left-hand column) and waveletdomain bootstrap thereof (right-hand) IV 23

Wavelet-Domain Bootstrapping of FD Series approximations to true PDF using (a) Haar & (b) LA(8) wavelets (a) (b) ˆρ (m) ˆρ (m) using 5 FD time series and the Haar DWT yields: average of 5 sample means. =.35 (truth. =.53) average of 5 sample SDs. =.96 (truth. =.7) using 5 FD time series and the LA(8) DWT yields: average of 5 sample means. =.43 (truth. =.53) average of 5 sample SDs. =.98 (truth. =.7) vertical line indicates ˆρ, IV 24

MLEs of FD Parameters: I FD process depends on 2 parameters, namely, δ and σ 2 ε: S (f) = σ 2 ε [4 sin 2 (πf)] δ given = [,,..., N ] T with N = 2 J, suppose we want to estimate δ and σ 2 ε if is stationary (i.e. δ < /2) and multivariate Gaussian, can use the maximum likelihood (ML) method WMTSA: 36 IV 25

MLEs of FD Parameters: II definition of Gaussian likelihood function: L(δ, σε 2 Σ ) (2π) N/2 Σ /2e T /2 where Σ is covariance matrix for, with (s, t)th element given by s,s t, and Σ & Σ denote determinant & inverse ML estimators of δ and σε 2 maximize L(δ, σε 2 ) or, equivalently, mininize 2 log (L(δ, σ 2 ε )) = N log (2π) + log ( Σ ) + T Σ exact MLEs computationally intensive, mainly because of the need to deal with Σ and Σ good approximate MLEs of considerable interest WMTSA: 36 362 IV 26

MLEs of FD Parameters: III key ideas behind first wavelet-based approximate MLEs have seen that we can approximate FD time series by Y = W T Λ /2 Z, where Λ /2 is a diagonal matrix, all of whose diagonal elements are positive since covariance matrix for Z is I N, the one for Y is W T Λ /2 I N (W T Λ /2 ) T = W T Λ /2 Λ /2 W = W T ΛW e Σ, where Λ Λ /2 Λ /2 is also diagonal can consider e Σ to be an approximation to Σ leads to approximation of log likelihood: 2 log (L(δ, σ 2 ε )) N log (2π) + log ( e Σ ) + T e Σ WMTSA: 362 363 IV 27

MLEs of FD Parameters: IV Q: so how does this help us? easy to invert e Σ : eσ = W T ΛW = (W) Λ W T = W T Λ W, where Λ is another diagonal matrix, leading to T e Σ = T W T Λ W = W T Λ W easy to compute the determinant of e Σ : e Σ = W T ΛW = ΛWW T = ΛI N = Λ, and the determinant of a diagonal matrix is just the product of its diagonal elements WMTSA: 362 363 IV 28

MLEs of FD Parameters: V define the following three functions of δ: C j (δ) Z /2 j CJ+ NΓ( 2δ) (δ) Γ 2 ( δ) σ 2 ε(δ) N /2 j+ 2 j+ [4 sin 2 (πf)] δ df J j= V J, 2 J CJ+ (δ) + j= N 2 j C j (δ) C j (δ) Z /2 j 2 j+ df /2 j+ [2πf] 2δ N 2 j t= W 2 j,t WMTSA: 362 363 IV 29

MLEs of FD Parameters: VI wavelet-based approximate MLE δ for δ is the value that minimizes the following function of δ: l(δ ) N log(σ 2 ε(δ)) + log(c J+ (δ)) + J j= N 2 j log(c j (δ)) once δ has been determined, MLE for σ 2 ε is given by σ 2 ε( δ) computer experiments indicate scheme works quite well WMTSA: 363 364 IV 3

Other Wavelet-Based Estimators of FD Parameters second MLE approach: formulate likelihood directly in terms of nonboundary wavelet coefficients handles stationary or nonstationary FD processes (i.e., need not assume δ < /2) handles certain deterministic trends alternative to MLEs are least square estimators (LSEs) recall that, for large τ and for β = 2δ, have log (ν 2 (τ j)) ζ + β log (τ j ) suggests determining δ by regressing log (ˆν 2 (τ j)) on log (τ j ) over range of τ j weighted LSE takes into account fact that variance of log (ˆν 2 (τ j)) depends upon scale τ j (increases as τ j increases) WMTSA: 368 379 IV 3

Homogeneity of Variance: I because DWT decorrelates LMPs, nonboundary coefficients in W j should resemble white noise; i.e., cov {W j,t, W j,t } when t 6= t, and var {W j,t } should not depend upon t can test for homogeneity of variance in using W j over a range of levels j suppose U,..., U N are independent normal RVs with E{U t } = and var {U t } = σ 2 t want to test null hypothesis H : σ 2 = σ2 = = σ2 N can test H versus a variety of alternatives, e.g., H : σ 2 = = σ2 k 6= σ2 k+ = = σ2 N using normalized cumulative sum of squares WMTSA: 379 38 IV 32

Homogeneity of Variance: II to define test statistic D, start with P kj= Uj 2 P k, k =,..., N 2 max k N 2 P N j= U2 j and then compute D max (D +, D ), where µ D + k + N P k & D max k N 2 µ P k k N can reject H if observed D is too large, where too large is quantified by considering distribution of D under H need to find critical value x α such that P[D x α ] = α for, e.g., α =.,.5 or. WMTSA: 38 38 IV 33

Homogeneity of Variance: III once determined, can perform α level test of H : compute D statistic from data U,..., U N reject H at level α if D x α fail to reject H at level α if D < x α can determine critical values x α in two ways Monte Carlo simulations large sample approximation to distribution of D: P[(N/2) /2 D x] + 2 ( ) l e 2l2 x 2 l= (reasonable approximation for N 28) WMTSA: 38 38 IV 34

Homogeneity of Variance: IV idea: given time series { t }, compute D using nonboundary wavelet coefficients W j,t (there are Mj N j L j of these): P k P kt=l j W 2 j,t PN j t=l j W 2 j,t, k = L j,..., N j 2 if null hypothesis rejected at level j, can use nonboundary MODWT coefficients to locate change point based on P kt=lj W ep f j,t 2 k P N t=l j W f j,t 2, k = L j,..., N 2 along with analogs e D + k and e D k of D+ k and D k WMTSA: 38 38 IV 35

Example Annual Minima of Nile River: I 5 x 3 9 6 3 years 2 4 8 scale (years) left-hand plot: annual minima of Nile River.. new measuring device introduced around year 75 right: Haar ˆν 2 (τ j) before (x s) and after (o s) year 75.5, with 95% confidence intervals based upon χ 2 η 3 approximation o x o x o x o WMTSA: 326 327 IV 36

Example Annual Minima of Nile River: II l(δ ) 3 5...2.3.4.5 based upon last 52 values (years 773 to 284), plot shows l(δ ) versus δ for the first wavelet-based approximate MLE using the LA(8) wavelet (upper curve) and corresponding curve for exact MLE (lower) δ wavelet-based approximate MLE is value minimizing upper curve: δ. =.4532 exact MLE is value minimizing lower curve: ˆδ. =.4452 WMTSA: 386 388 IV 37

Example Annual Minima of Nile River: III. O. O O O O. O O O..... f using last 52 values again, variance of wavelet coefficients computed via LA(8) MLEs δ and σ 2 ε( δ) (solid curve) as compared to sample variances of LA(8) wavelet coefficients (circles) agreement is almost too good to be true! O WMTSA: 386 388 IV 38

Example Annual Minima of Nile River: IV results of testing all Nile River minima for homogeneity of variance using the Haar wavelet filter with critical values determined by computer simulations critical levels τ j Mj D % 5% % year 33.559.945.5.262 2 years 65.754.32.469.765 4 years 82..855.268.2474 8 years 4.233.2572.2864.3436 can reject null hypothesis of homogeneity of variance at level of significance.5 for scales τ & τ 2, but not at larger scales WMTSA: 386 388 IV 39

Example Annual Minima of Nile River: V 5 3 9.2...2.. 6 7 8 9 2 3 year Nile River minima (top plot) along with curves (constructed per Equation (382)) for scales τ & τ 2 (middle & bottom) to identify change point via time of maximum deviation (vertical lines denote year 75) WMTSA: 386 388 IV 4

Summary DWT approximately decorrelate certain time series, including ones coming from FD and related processes leads to schemes for simulating time series and bootstrapping also leads to schemes for estimating parameters of FD process approximate maximum likelihood estimators (two varieties) weighted least squares estimator can also devise wavelet-based tests for homogeneity of variance trends (see Craigmile et al., 24, for details) WMTSA: 388 39 IV 4

References: I bootstrapping A. C. Davison and D. V. Hinkley (997), Bootstrap Methods and their Applications, Cambridge, England: Cambridge University Press D. B. Percival, S. Sardy and A. C. Davison (2), Wavestrapping Time Series: Adaptive Wavelet-Based Bootstrapping, in Nonlinear and Nonstationary Signal Processing, edited by W. J. Fitzgerald, R. L. Smith, A. T. Walden and P. C. Young. Cambridge, England: Cambridge University Press, pp. 442 7 A. M. Sabatini (999), Wavelet-Based Estimation of /f-type Signal Parameters: Confidence Intervals Using the Bootstrap, IEEE Transactions on Signal Processing, 47(2), pp. 346 9 A. M. Sabatini (26), A Wavelet-Based Bootstrap Method Applied to Inertial Sensor Stochastic Error Modelling Using the Allan Variance, Measurement Science and Technology, 7, pp. 298 2988 B. J Whitcher (26), Wavelet-Based Bootstrapping of Spatial Patterns on a Finite Lattice, Computational Statistics & Data Analysis, 5(9), pp. 2399 42 IV 42

References: II decorrelation property of DWTs P. F. Craigmile and D. B. Percival (25), Asymptotic Decorrelation of Between-Scale Wavelet Coefficients, IEEE Transactions on Information Theory, 5, pp. 39 48 parameter estimation for FD processes P. F. Craigmile, P. Guttorp and D. B. Percival (25), Wavelet-Based Parameter Estimation for Polynomial Contaminated Fractionally Differenced Processes, IEEE Transactions on Signal Processing, 53, pp. 35 6 testing for homogeneity of variance B. J Whitcher, S. D. Byers, P. Guttorp and D. B. Percival (22), Testing for Homogeneity of Variance in Time Series: Long Memory, Wavelets and the Nile River, Water Resources Research, 38,.29/2WR59. wavelet-based trend assessment P. F. Craigmile, P. Guttorp and D. B. Percival (24), Trend Assessment in a Long Memory Dependence Model using the Discrete Wavelet Transform, Environmetrics, 5, pp. 33 35 IV 43