Distinguishing chaos from noise

Transcription

1 Distinguishing chaos from noise Jianbo Gao PMB InTelliGence, LLC, West Lafayette, IN Mechanical and Materials Engineering, Wright State University Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

2 Outline General background: fighting between chaos community and fractal community Distinguishing chaos from noise Characterization of chaotic signals Correlation dimension Largest positive Lyapunov exponent Kolmogorov and correlation entropy Criterion for chaos General background about random fractal theory Why 1/f α processes may be mis-interpreted as chaos Peace-keeping using multiscale approaches Scale-dependent Lyapunov exponent (SDLE) makes fighting impossible Applications (to be discussed in lecture 8) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

3 A classic issue: distinguishing chaos from noise Often adopted assumption: finite positive Lyapunov exponent (or Kolmogorov entropy) and non-integral fractal dimension Chaos in brain, chaos in heart, chaos in weather... Chaos is everywhere! Running on a wild beach, hunting for the beast of chaos: Here is a footprint, Here is another...turning around, astonishingly found those were but their own footprints! Counter-example: a 1/f process with Hurst parameter H has a fractal dimension 1/H and finite positive Lyapunov exponent (or entropy). Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

4 Characterization of chaos: correlation dimension In practice, when the dimension of the phase space is high and the length of the data is not very great, calculating dimension by partitioning the phase space into small boxes is not an efficient method Seminal Grassberger-Procaccia algorithm: 1 C(ɛ) = lim N N 2 N H(ɛ V i V j ), i,j=1 where V i and V j are points on the attractor, H(y) is the Heaviside function (1 if y 0 and 0 if y < 0), and N is the number of points randomly chosen from the entire dataset Scaling: C(ɛ) ɛ D 2, as ɛ 0 Removing autocorrelation or tangential motion: i j > w Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

5 An example: chaotic Lorenz attractor log 10 C( ) ε log 10 ε dlog 10 C(ε) dε m = 4 m = 6 m = 8 m = 10 m = log 10 ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

6 Quantifying chaos: Lyapunov exponents Lyapunov exponents are dynamical quantities Let l i (t) be the ith principal axis of the ellipsoid at time t, we have l i (t) dr e λ i t λ i = 1 lim dr 0, t t ln l i(t) dr Lyapunov exponents are conventionally listed in descending order: λ 1 λ 2 λ 3 dl 1 trajectory dl 2 dr Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

7 Exponential divergence in the chaotic logistic map x n+1 = rx n (1 x n ), r = 4 x x , x , x , x , x , x , x , x , x , x , x , Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

8 Lyapunov exponent for the chaotic logistic map ln x Time step n Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

9 Numerical computations of LEs: Wolf et al. s algorithm The basic idea is to select a reference trajectory, and follow the divergence of a neighboring trajectory from it Let the spacing between the two trajectories at time t i be d i, and the spacing at time t i+1 be d i+1 The rate of divergence of the trajectory over a time interval of t i+1 t i is then ln(d i+1 /d i ). t i+1 t i To ensure that the separation between the two trajectories is always small, when d i+1 exceeds certain threshold value, it has to be re-normalized: a new point in the direction of the vector of d i+1 is picked up so that d i+1 is very small compared to the size of the attractor After n repetitions of stretching and renormalizing the spacing, one obtains the following formula n 1 [ ][ t i+1 t i ln(di+1 /d i λ 1 = ) ] n 1 i=1 n 1 i=1 (t = ln(d i+1/d i ) i+1 t i ) t i+1 t i t n t 1 i=1 Con: The algorithm assumes but does not verify exponential divergence; any noise gives a positive LE Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

10 Numerical computations of LEs: Rosenstein et al s and Kantz s algorithm Choose a reference point, and find its ɛ-neighbors V j Then follow the evolution of all these points, and computes an average distance after certain time Finally, choose very many reference points, and take another average Λ(k) = ln V i+k V j+k average over j average over i where V i is a reference point, and V j are neighbors to V i, satisfying the condition V i V j < ɛ If Λ(k) k for certain intermediate range of k, then the slope is the largest Lyapunov exponent Con: In order for average over j to be well-defined, ɛ has to be small; then the method is sensitive to noise. Moreover, there has to be a condition on j, as shown by Gao and Zheng Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

11 Numerical computations of LEs: Gao and Zheng s method There are three basic equation: ( ) Vi+k V j+k Λ(k) = ln V i V j ɛ i V i V j ɛ i + ɛ i, i = 1, 2, 3, i j > w Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

12 Entropy of a biological trajectory The probability p i for the i th unit area being visited is n i /N, where n i is the times the i th unit area being visited and N is the length of the trajectory m Shannon entropy I = p i log p i i=1 Can be generalized to Renyi and Tsallis entropy A/P COP (m) i k M/L COP (m) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

13 Dynamical entropies and information flow: general considerations Partition the phase space into small boxes of size ɛ, compute the probability p i that box i is visited by the trajectory, and finally calculate Shannon entropy For many systems, when ɛ 0, information linearly increases with time I (ɛ, t) = I 0 + Kt where I 0 is the initial entropy and K is the Kolmogorov-Sinai (KS) entropy Choose I 0 = 0 if start from a unit area Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

14 Three general cases Deterministic, nonchaotic: no change in information since during the time evolution of the system, phase trajectories remain close together nearby phase points are still close to each other, and can be grouped into some other small region of the phase space Deterministic, chaotic Due to exponential divergence, the number of phase space region available to the system after a time T is N e (P λ + )T, where λ + are positive Lyapunov exponents Assuming that all of these regions are equally likely, then p i (T ) 1/N, and the information function becomes I (T ) = Therefore, K = λ + N p i (T ) ln p i (T ) = ( λ + )T i=1 Random: After a short time, the entire phase space may be visited; therefore, I ln N. When N, we have K = Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

15 Numerical computations of entropy: Grassberger-Procaccia s algorithm Basic idea: approximate K by estimating the correlation entropy K 2 (i.e., Renyi entropy of order 2) K 2 (ɛ) = lim m ln C (m) (ɛ) ln C (m+1) (ɛ) where Lδt δt is the sampling time C (m) (ɛ) is the correlation integral based on the m dimensional reconstructed vectors V i and V j C (m) 2 Nv 1 Nv (ɛ) = lim Nv N v (N v 1) i=1 j=i+1 H(ɛ V i V j ), where N v = N (m 1)L is the number of reconstructed vectors, H(y) is the Heaviside function (1 if y 0 and 0 if y < 0) Similarly compute C (m+1) (ɛ) for m + 1 dimensional reconstructed vectors Summary: C m (ɛ) ɛ D 2 e mτk 2 Approximate entropy, Sample entropy are special cases of Cohen-Procaccia, and Grassberger-Procaccia entropies Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

16 An example: chaotic Lorenz attractor for true low-dimensional chaotic dynamics, in a plot of ln C m (ɛ) vs. ln ɛ with m as a parameter, one observes a series of parallel straight lines, with the slope being the correlation dimension, D 2, and the spacing between the lines estimating K 2 (where lines for larger m lie below those for smaller m) log 10 C( ) ε log 10 ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

17 Detecting deterministic chaos Popular criterion: non-integral dimension, positive Lyapunov exponent or KS entropy How messy is the field? Can economic time series be chaotic? Shintani and Linton 2003, 2004; Hommes and Manzan 2006: No Correct conclusion, but wrong approach: they found a negative value for Lyapunov exponent (LE) using a neural-network-based LE estimator the LE becomes more and more negative when noise level is increased Random 1/f 2H+1 processes have dimension 1/H and positive LE and KS (Osborne and Provenzale, 1989, 1991) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

18 Detecting structures in time series data Sugihara and May (Nature 1990): Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series Tsonis and Elsner (Nature 1992): Again check if predictability is reduced when the original data is compared with some type of noise Kaplan and Glass (PRL 1992): Continuum implies determinism Salvino and Cawley (PRL 1994): Smoothness implies determinism Poon et al (Nature 1996; PNAS 2001): Noise titration Surrogate data test Major problem: these are indirect methods, tackling a smaller goal than distinguishing chaos from noise Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

19 Surrogate data test Basic idea: randomize some features of a time series, then check whether the original and the derived time series may significantly differ or not How to check whether the two processes the same or not? Compute some statistical quantities Use some prediction algorithm to check if prediction error is substantially increased Types of surrogate data Shuffling or permutation: this preserves the distribution of the data Randomize the phases of the Fourier transform of the data, then take inverse transform: this preserves the correlation of the data Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

20 Noise titration method The key element of the technique is the Volterra autoregressive representation k 1 m 1 =0 k 1 m 1 =0 x(n + 1) = k 0 + k 1 m d =0 k 1 m 2 =0 k 1 m 1 =0 k 1 (m 1 )x(n m 1 ) k 2 (m 1, m 2 )x(n m 1 )x(n m 2 ) k d (m 1, m 2,, m d )Π d i=1x(n m i ) x(n) is the time series, k is the length of memory (which may be considered equivalent of the embedding dimension), d is the order of the polynomial (when d > 1, it is nonlinear), and {k r (m 1, m 2,, m r )} are the coefficients of the Volterra series Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

21 Noise titration method (Cont ) This expansion creates M = (k + d)!/k!d! nonlinear models, with the order of polynomials ranging from 1 to d and embedding dimension ranging from 1 to k Such nonlinear models are then compared with linear models with order 1 and embedding dimension ranging from 1 to M, in terms of goodness of the fitting The latter is quantified by the Akaike cost function, an information theoretic metric. If nonlinear models are better, then it is concluded that the time series is nonlinear and chaotic To quantify chaos, noise is then added to the signal, till nonlinearity in the data can no longer be detected through such Volterra series expansion Noise Limit, the amount of noise needs to be added to make the time series linear, is used to indicate the strength of chaos Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

22 Problems with these methods y(t) y(t+6) Clean data Noisy data t slope = 2.15 ± (a) 2 (b) y(t) Noise Limit (%) λ(ε) 10 2 (c) ε (d) Sampling time Most are indirect methods, tackling a smaller goal than distinguishing chaos from noise The noise titration technique in particular only test nonlinearity, but not chaos Stochastic van der Pol s oscillator dx/dt = y + D 1 η 1 (t) dy/dt = (x 2 1)y x + D 2 η 2 (t) where < η i (t) >= 0, < η i (t)η j (t ) >= δ ij δ(t t ), i, j = 1, 2, and the parameters D i, i = 1, 2 characterize the strength of noise Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

23 Introduction to fractal and multifractal A part is (exactly or statistically) similar to another part, or the whole Clouds; mountains; trees; etc. (Images: not computer-made, but photos of Jiu Zhai Gou) Power-law relation a straight line in a log-log plot (scaling) Power-law relation is the origin of self-similarity Many (or possibly infinitely many) power-law relations Multifractal Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

24 Fractional Brownian motion (fbm) B H (t) Gaussian process with mean 0 & stationary increments Variance: E[(B H (t)) 2 ] = t 2H Power spectral density f (2H+1) H: Hurst parameter. 1/2 < H < 1: long memory (long-range-dependence (LRD)) H = 1/2: standard Brownian motion 0 < H < 1/2: anti-persistence Applications to a wide range of problems (including Hollywood movie making fancy landscapes) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

25 Examples of fbm processes with different H Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

26 Dimensions of Bm and fbm processes Consider the case t 1 and a fbm process defined over a unit interval of time Cover the curve by intervals of length l Number of intervals N( l) of rulers of size l amounts to partitioning the unit time interval by t; therefore, N( l) 1/ t l = when t 1, l t H t 2 + B 2H t 1 + t 2H 2, Fractal dimension is defined by N( l) l D. Therefore, D = 1/H. Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

27 Entropy and Lyapunov exponent (LE) for fbm processes On average, a small distance l grows with t as l t H ; using Wolf et al s algorithm, it is always positive Entropy estimated from finite data is always finite, but not infinite fbm processes can indeed be easily mis-interpreted as chaos! Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

28 Gao-Zheng s direct dynamical test for chaos Matter of fact: an excellent solution already exists A bigger goal: not only distinguish chaos from noise, but automatically find the defining parameter(s) of all known types of processes? Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

29 Scale dependent Lyapunov exponent (SDLE) (Gao et al. Phys Rev E 2006; Chaos 2009; ABME 2010; Cog NeuroDyn 2011) Consider an ensemble of trajectories in phase space Denote the initial separation between two nearby trajectories by ɛ 0, and their average separation at time t and t + t by ɛ t and ɛ t+ t, respectively Being defined in an average sense, ɛ t and ɛ t+ t can be readily computed from any processes, even if they are non-differentiable When t 0, we have ɛ t+ t = ɛ t e λ(ɛt) t or λ(ɛ t ) = ln ɛ t+ t ln ɛ t t or dɛ t dt = λ(ɛ t)ɛ t Given a time series data, the smallest t = sampling time Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

30 SDLE: Computation Basic idea: form proper ensemble averages Phase space reconstruction: V i = [x(i), x(i + L),..., x(i + (m 1)L], x(i): given time series, m: embedding dimension, L: delay time Introducing a sequence of shells ɛ k V i V j ɛ k + ɛ k, k = 1, 2, where ɛ k & ɛ k are arbitrarily chosen small distances ln V i+t+ t V j+t+ t ln V i+t V j+t λ(ɛ t ) = t Ensemble average is taken within a shell add condition i j > (m 1)L for chaotic systems Similar algorithm used in computing time-dependent exponent curves (Gao & Zheng, Phys. Lett. A 1993; Europhys. Lett. 1994; Phys. Rev. E, 1994) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

31 Classification of complex data Chaotic motions Deterministic chaos: λ(ɛ) const Noisy chaos & Noise-induced chaos (e.g., without noise, motion is regular: Gao et al. Phys. Rev. Lett. 1999, 2002; Hwang & Gao Phys. Rev. E 2000): λ(ɛ) γ ln ɛ γ determines the speed of loss of information 1/f 2H+1 processes: λ(ɛ) Hɛ 1/H α-stable Levy processes: λ(ɛ) 1 α ɛ α Stochastic oscillations: both λ(ɛ) γ ln ɛ and λ(ɛ) Hɛ 1/H Complex motions with multiple scaling behaviors Distinguishing chaos from noise is no longer a problem! Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

32 SDLE λ(ɛ) for chaos, noisy chaos, & noise-induced chaos Chaos: λ(ɛ) const (largest positive Lyapunov exponent) Noisy chaos & noise-induced chaos: λ(ɛ) γ ln ɛ on small scales (i) Stochastic Lorenz (63 ) system (ii) Noisy logistic map x n+1 = µx n (1 x n ) + P n, 0 < x n < 1, µ = 3.74, σ Pn = without noise, motion is periodic Noise-induced chaos 2.5 (a) Lorenz (b) Logistic (1): slope = 0.28 λ(ε) D = 0 D = 1 D = 2 D = 3 D = ε λ(ε) log 10 log 10 (2): slope = 0 ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

33 Chaotic systems with multiple positive Lyapunov exponents Mackey-Glass delay differential system (Science, 1977), dx/dt = ax(t + Γ) 1 + x(t + Γ) c bx(t) When a = 0.2, b = 0.1, c = 10, Γ = 30, two positive Lyapunov exponents (the largest 0.007) 15 x λ(ε) ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

34 Power-law scaling of λ(ɛ) for 1/f β processes Can prove λ(ɛ) Hɛ 1/H λ(ε) log (a) fbm (1) slope = 3.02 (2) slope = 2.04 H = 0.33 H = 0.50 H = 0.70 (3) slope = 1.45 log 10 λ(ε) log ε log ε 1 2 (b) ON/OFF (1) slope = 1.98 (2) slope = 1.42 µ = 2.0 µ = 1.6 µ = 1.2 (3) slope = 1.10 Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

35 Stable laws and Levy processes Stable laws: distributions for the sum of independent random variables (RV s) & those being summed have the same functional form Characteristic function for the distribution of a standard stable RV: { Φ Z (u) = E[e juz exp( u ] = α [1 jβ tan(πα/2)sign(u)], α 1 exp( u α [1 + jβ 2 π log u sign(u)], α = 1, where 0 < α 2: stability index; 1 β 1: skewness parameter; sign: sign function ( 1, 0, 1 depending on whether u < 0, = 0, > 0) In the case of strictly stable, n i=1 Y d i = n 1/α Y nvary = n 2/α VarY 0 < α 2 α = 2: normal distribution; 0 < α < 2: heavy-tailed, P(X x) x α Generalized central limit theorem: each stable law = attractor of the sum of independent RV s with infinite variance Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

36 Levy processes Levy flights: random processes consisting of many independent steps, each step being characterized by a stable law, and consuming a unit time regardless of its length H = 1/2 Symmetric Levy flights: β = 0 Levy walks: sampled from Levy flights with a uniform speed; each step takes time proportional to its length the increment process similar to power-law ON/OFF train A symmetric α stable Levy flight is 1/α self-similar Broad applications of Levy statistics: economics, fluid mechanics, device physics, ecology, art, etc. Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

37 Power-law scaling of λ(ɛ) for Levy flights Can prove λ(ɛ) 1 α ɛ α α plays the role of 1/H x (a) 0.4 (b) x(n+1) A B log 10 λ(ε) slope = x(n) log 10 ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

38 λ(ɛ) for stochastic oscillators Stochastic Van der Pol s oscillator: dx/dt = y + D 1 η 1 (t), dy/dt = (x 2 1)y x + D 2 η 2 (t) λ(ɛ) ln ɛ, when (m 1)L is small λ(ɛ) ɛ 1/H, H 1/2, when (m 1)L is large y(t+l) (a) (b) (c) λ(ε) λ(ε) slope = y(t) ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31 ε

39 Stochastic oscillations: H not necessary 1/2 Pathological tremor: involuntary, approximately rhythmic, and roughly sinusoidal movement of parts of the body often H > 1/2 (Gao & Tung, 2000; Gao 2002) Karman vortex street: When Re 137, H 1/2; When Re increases, H decreases (Lin et al. 1993; Gao 1997; Gao et al. 1999) (Left: Lim, University of Melbourne; Right: atmospheric Karman vortex street off Selkirk Island off the coast of Chile ) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

40 λ(ɛ) for motions with multiple scaling behaviors Cencini et al. (2000): x n+1 = [x n ] + F (x n [x n ]) + ση t [x n ]: integer part of x n ; η t : uniform noise on the interval [ 1, 1] { (2 + )y if y [0, 1/2) F (y) = (2 + )y (1 + ) if y (1/2, 1] λ 1 = ln(2 + ); [x n ] introduces a random walk on integer grids imagine a baby learning to walk 10 0 (a) 10 0 (b) λ(ε) 10 1 λ(ε) clean data noisy data ε ε Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

41 Characterizing large scale orderly motions by the SDLE Defining and characterizing large scale orderly motions, such as oscillatory ones, is a significant issue in many disciplines of science Hidden frequency phenomenon: Fourier analysis of one variable (say x(t)) may not suggest any oscillatory motions, while that of another variable (say z(t)) may How to reveal the hidden frequency? Reconstruct a phase space using x(t) to get information about z(t), then get a 1-D signal Existing methods Ortega (1996): density method; Chern et al. (1998): singular value decomposition Our approach: Take Fourier transform of the limiting scale, ɛ Our method is more effective, because of scale-isolation Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

42 Hidden frequency in the Lorenz ( 63) system (a) (b) (c) E x (f) 10 4 E y (f) 10 4 E z (f) Frequency f (Hz) Frequency f (Hz) Frequency f (Hz) (d) (e) (f) E (x) (f) ε 10 0 E (y) (f) ε 10 0 E (z) (f) ε Frequency f (Hz) Frequency f (Hz) Frequency f (Hz) Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

43 Detecting intermittent chaos in logistic map x(n) λ(ε) (a) n (b) ε A plateau is clearly visible chaos This capability is due to SDLE s scale separation property No other methods have this capability Intermittent chaos is the best model for river discharge Distributional difference in wet and dry seasons Correlation structure Nonlinear but non-chaotic behavior using standard chaos analysis methods Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31

44 Summary Empirical observation: chaos is more relevant to describe dynamics of well-controlled experiments; random fractal is more relevant to describe behaviors of open systems with many surprises Cross-talk between the chaos and the random fractal communities is essential, in order to most fruitfully solve significant real-world problems Multiscale analysis is one of the most promising ways to go SDLE is a multiscale complexity measure with the richest scaling behaviors that can be analytically proven SDLE can classify all known types of time series models by automatically finding the defining parameter(s) SDLE has the scale separation property SDLE can readily deal with nonstationarity SDLE can readily detect intermittent chaos Applications: geophysics; environmental engineering; economic time series; biosignal analysis; ensemble forecasting Gao, Jianbo (PMB InTelliGence) Chaos and noise June / 31