Characterizing chaotic time series

Transcription

1 Characterizing chaotic time series Jianbo Gao PMB InTelliGence, LLC, West Lafayette, IN Mechanical and Materials Engineering, Wright State University Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

2 Background A variable of a system is interesting only if it changes with time How does the signal change with time? Two major models Deterministic chaos: nonlinear interactions of a few degrees of freedom generate apparently random dynamics Noise or stochastic processes with infinite degrees of freedom Chaos is a preferred model, having a few degrees of freedom Except chaotic maps with a single variable, chaos involves a few variables In experiments or observations, only one or a few variables are observable How can the full chaotic dynamics be inferred from such partial observation? Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

3 Outline Time delay embedding: perhaps the biggest contribution of chaos theory to general science on data analysis Basic idea Examples Application: Internet worm detection Optimal embedding Static method for determining the embedding dimension: the false nearest neighbor method Statistical methods for determining the delay time A dynamical method for jointly determining the embedding dimension, the delay time, and other parameters Characterizing chaos Fractal dimension Lyapunov exponent Kolmogorov entropy Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

4 Phase space reconstruction: basic idea Basic question: Given a scalar time series x(t), can we learn the dynamics of the system without knowing other variables describing the system? Solution: Construct a suitable phase space from x(t) pioneering work by Packard et al., Takens, Mane, Sauer, Yorke... General consideration: Suppose a dynamical system is described by n first order ordinary differential equations The dynamical system can be equivalently described by a single ODE involving terms d n x/dt n, d n 1 x/dt n 1, etc. One way to construct a phase space is to estimate the derivatives of x(t) by finite differences dx dt x(t + t) x(t) t d 2 x x(t + 2 t) 2x(t + t) + x(t) dt 2 t 2 where t is the sampling time for x(t) This procedure is not viable, as it amplifies noise Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

5 Time delay embedding Simply construct the vector V i = [x(i), x(i + L),..., x(i + (m 1)L)] where m is called the embedding dimension and L the delay time More explicitly, we have V 1 = [x(t 1 ), x(t 1 + τ), x(t 1 + 2τ),..., x(t 1 + (m 1)τ], V 2 = [x(t 2 ), x(t 2 + τ), x(t 2 + 2τ),..., x(t 2 + (m 1)τ],. V j = [x(t j ), x(t j + τ), x(t j + 2τ),..., x(t j + (m 1)τ],. where t i+1 t i = t and τ = L t Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

6 Time delay embedding (cont ) This procedure then defines a mapping (i.e., dynamics), V n+1 = M(V n ). (1) Under the assumption that the dynamics of the system can be described by an attractor with boxing counting dimension D F (to be defined later), it can be proven that when m > 2D F, the dynamics of the original system is topologically equivalent to that described by Eq. (??) The basic idea behind this fundamental theorem is given an initial condition, the solution to a set of ODEs is unique, and the trajectory in the phase space does not intersect with itself When m is not large enough, however, self-intersection may occur; when m is too big, data may not be long enough Furthermore, L also needs to be carefully chosen Optimal embedding: determine the proper values of m and L Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

7 An example: phase plane of the equation ẍ = x (Rewrite as: ẋ = y, ẏ = x) x = y y = x Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

8 Optimal embedding: sine wave The dynamics of the harmonic oscillator can be described by Equivalently, dx dt = y, dy dt = ωx d 2 x dt 2 = ωx The general solution is: x(t) = A cos(ωt + φ 0 ), y(t) = A sin(ωt + φ 0 ) If we embed x(t) to a 2-D space, then V t = [x(t), x(t + τ)], τ = L Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

9 Quasi-periodic motion and torus x(t) = sin(t) sin( 3t) What happens if 3 is replaced by 1.732? x(t+l) x(t) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

10 Chaotic Lorenz system dx/dt = 16(x y) dy/dt = xz x y dz/dt = xy 4z 2500 ensemble members, represented by cyan, red, green, and blue colors at t = 0, 2, 4, and 6 units z x y 20 Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

11 Lorenz attractor represented by x(t) x(t+l) x(t) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

12 Application: Internet worm detection Motivations Enterprise networks are facing ever-increasing security threats from worms, viruses, intrusions, etc., causing billions of dollars of loss Existing monitoring methods can not detect worms effectively in real-time Purpose: to develop effective methods to defend against Internet worms in a real-time manner Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

13 Worm detection: state-of-the-art and challenges Internet worm: a self-propagating program that automatically replicates itself to vulnerable systems and spreads across the Internet. Basic strategies: Monitor network traffic data Look for specific byte sequences (attack signatures) that are known to appear in the attack traffic Challenges: How to find a good attack signature fast enough for real-time use? A good attack signature should be general enough to capture all attack traffic of certain type specific enough to avoid overlapping with normal traffic to reduce false alarms Existing methods (e.g., expectation maximization and hidden Markov models) need extensive training, thus can not be applied in real-time Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

14 Normal and worm traffic trace 300 datasets are collected from a double-honeypot system deployed in a local network for automatic detection of worm attacks from the Internet. Courtesy of Dr. Chen at the Univ. of Florida 300 Package Sequence 200 Normal Worm Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

15 Phase diagrams for normal and worm traffic Left: worm traffic; Right: normal traffic Region x(n+1) x(n+1) Region x(n) Region 1 Gao, Jianbo (PMB InTelliGence) Chaotic dynamics x(n) Region 4 June / 37

16 Accuracy of the phase space based method index based on percentage of number of points in region 1 35 index based on percentage of number of points in region Normal Worm 30 Normal Worm Frequency Frequency Percentage of points (%) Percentage of points (%) 70 index based on percentage of number of points in regions 3 and 4 60 Normal Worm 50 Frequency Percentage of points (%) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

17 Comparison with Tang & Chen (InfoCom, 2005) EM: Expectation-Maximization; Gibbs: Gibbs Sampling Algorithm Their methods are not only much slower, but also that the separation between normal and worm traffic data is much smaller than the phase space method! Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

18 Optimal embedding: heuristic and statistical methods Visual inspection: always helpful to check the data in a 2-D plane or 3-D space, to see whether the chosen delay time makes the plot stretch in the phase space uniformly (as in the harmonic oscillator case) No information on the embedding dimension m can be obtained When one gradually increases m, it is better to decrease the delay time, so that the embedding window (m 1)τ is kept approximately constant or increases slower than m Determining τ based on the autocorrelation function Empirical observation: the time corresponding to the first zero of the autocorrelation function of the signal is often a good estimate for τ No information on the embedding dimension m can be obtained Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

19 Heuristic and statistical methods (Cont ) Determining τ based on the mutual information: Denote the probability that the signal assumes a value inside the ith bin by p i, and let p ij (τ) be the probability that x(t) is in bin i and x(t + τ) is in bin j The mutual information for time delay τ is I (τ) = i,j p ij (τ) ln p ij (τ) 2 i p i ln p i when τ = 0, p ij = p i δ ij, and I yields the Shannon entropy of the data distribution When τ is large, x(t) and x(t + τ) are independent and pij factorizes to p i p j, therefore, I 0 A good τ corresponds to the first minimum of I (τ), when I (τ) has a minimum In practice, when calculating I (τ), it may be advantageous to use equal-probability bins instead of equal-size bins, i.e., p i = 1/n = const, when n is the number of bins used to partition the data. It should be emphasized that p ij (τ) is not a constant Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

20 Optimal embedding: False nearest neighbor method This is a geometrical method Consider the situation in which an m 0 -dimensional delay reconstruction is an embedding, but an m 0 1-dimensional reconstruction is not Passing from m 0 1 to m 0, self-intersection in the reconstructed trajectory is eliminated This feature can be quantified by the sharp decrease in the number of nearest neighbors when m is increased from m 0 1 by 1 Conclusion: the optimal value of m is m 0 Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

21 False nearest neighbor method (Cont ) More precisely, for each reconstructed vector V (m) i = [x(t i ), x(t i + τ), x(t i + 2τ),..., x(t i + (m 1)τ)], find its nearest neighbor V (m) j If m is not large enough, then V (m) [V (m) i j j may be a false neighbor of V (m) i If embedding can be achieved by increasing m by 1, then the embedding vectors become V (m+1) i = [x(t i ), x(t i + τ), x(t i + 2τ),..., x(t i + (m 1)τ, x(t i + mτ)] =, x(t i + mτ)] and V (m+1) = [V (m) j, x(t j + mτ)], and they will no longer be close neighbors Instead, they will be far apart (something like south and north poles) The criterion for optimal embedding is R f = x(t i + mτ) x(t j + mτ) > R V (m) i V (m) T j Where R T is a heuristic threshold value. Abarbanel (1996) recommends R T = 15 After m is chosen, τ can be found by minimizing R f What happens if there is noise? Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

22 Optimal embedding: Time dependent exponent curves This is a dynamical method developed by Gao and Zheng (1993, 1994) Basic idea: false neighbors will not stay as close neighbors during dynamical evolutions (m) Let the reconstructed trajectory be denoted by V 1, V (m) 2, Assume V (m) i and V (m) j are false neighbors (m) It is unlikely that points V i+k, V (m) j+k, where k is called the evolution time, will keep to be close neighbors (m) That is, the separation between V i+k than that between V (m) embedding i and V (m) j (m) and V j+k will be much larger, if the delay reconstruction is not an Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

23 Time dependent exponent curves (Cont ) The measure proposed by Gao and Zheng is ( ) Vi+k V j+k Λ(m, L, k) = ln V i V j The angle brackets denote the ensemble average of all possible (V i, V j ) pairs satisfying the condition ɛ i V i V j ɛ i + ɛ i, i = 1, 2, 3, where ɛ i and ɛ i are prescribed small distances Since the computation is carried out for a sequence of shells, the effect of noise can be largely eliminated Criterion: for a fixed small k, an optimal m is such that Λ(m, L, k) no longer decreases much when further increasing m. After m is chosen, L can be selected by minimizing Λ(m, L, k). Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

24 An example: Rossler attractor dx/dt = (y + z) dy/dt = x + ay dz/dt = b + z(x c) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

25 Characterizing chaos: Fractal dimension Dimension is a geometrical quantity characterizing the minimal number of variables needed to fully describe the dynamics of a motion Chaotic attractor is often called strange attractor Attractor means the motion is bounded Strange means exponential divergence of nearby trajectories Incessant divergence and folding back makes chaos a fractal There are many ways to define the dimensions of a chaotic attractor Capacity (or box-counting dimension) D F (or D 0 ): Partition the phase space containing the attractor into many cells of linear size ɛ. Denote the number of nonempty cells by n(ɛ) n(ɛ) ɛ D 0, ɛ 0 Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

26 Generalized dimension spectrum Assign a probability p i = n i /N to the ith nonempty cell ni : number of points within the ith cell; N: total number of points on the attractor Let the number of nonempty cells be n. Then for real q D q = 1 ( n ) log q 1 lim i=1 pq i ɛ 0 log ɛ Dq is a nonincreasing function of q D0 is simply the box-counting or capacity dimension, since n i=1 pq i = n D1 gives the information dimension D I, D I = lim ɛ 0 n i=1 p i log p i log ɛ Typically, D I is equal to the pointwise dimension defined as p(l) l α, l 0 where p(l) is the measure (i.e., probability) for a neighborhood of size l centered at a reference point D2 is called the correlation dimension, which is the most popular Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

27 Singular measure-based multifractal interpretation of D q Basic idea: a chaotic attractor is comprised of many interwoven fractals, each with a different fractal dimension Cover the attractor by boxes of size ɛ Let p i is a probability measure Associate each box with a singularity index α i via p i = ɛ α i Assume the number of boxes with singularity index in the range α to α + dα is ρ(α)ɛ f (α) dα Interpretation of f (α): the dimension of points with pointwise dimension α To express D q in terms of α and f (α), n i=1 pq i = α i ɛ qα i = dα ρ(α )ɛ qα f (α ) = dα ρ(α )(1/ɛ) qα +f (α ) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

28 Multifractal interpretation of D q (Cont ) When ɛ 0, 1/ɛ is very large, the integral is basically determined by the maximum value of (1/ɛ) qα +f (α ) = e f (α ) qα ln(1/ɛ) The maximum value is given by two conditions n i=1 pq i d dα [f (α ) qα ] α =α(q) = 0 or f (α(q)) = q d 2 d(α ) 2 [f (α ) qα ] α =α(q) < 0 or f (α(q)) > 0 e f (α) qα ln(1/ɛ) D q = 1 ( log n q 1 lim i=1 pq i ɛ 0 log ɛ ) = 1 [qα(q) f (α(q))] q 1 Differentiating w.r.t. q and using f (α(q)) = q yields α = d dq [(q 1)D q] f (α) = (1 q)d q + qα D q and f (α) spectra give the same amount of information Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

29 A practical algorithm: Grassberger-Procaccia algorithm In practice, when the dimension of the phase space is high and the length of the data is not very great, calculating dimension by partitioning the phase space into small boxes is not an efficient method Seminal Grassberger-Procaccia algorithm: 1 C(ɛ) = lim N N 2 N H(ɛ V i V j ), i,j=1 where V i and V j are points on the attractor, H(y) is the Heaviside function (1 if y 0 and 0 if y < 0), and N is the number of points randomly chosen from the entire dataset Scaling: C(ɛ) ɛ D 2, as ɛ 0 Removing autocorrelation or tangential motion: i j > w Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

30 An example: chaotic Lorenz attractor log 10 C( ) ε log 10 ε dlog 10 C(ε) dε m = 4 m = 6 m = 8 m = 10 m = log 10 ε Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

31 Quantifying chaos: Lyapunov exponents Lyapunov exponents are dynamical quantities Let l i (t) be the ith principal axis of the ellipsoid at time t, we have l i (t) dr e λ i t λ i = 1 lim dr 0, t t ln l i(t) dr Lyapunov exponents are conventionally listed in descending order: λ 1 λ 2 λ 3 dl 1 trajectory dl 2 dr Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

32 Exponential divergence in the chaotic logistic map x n+1 = rx n (1 x n ), r = 4 x x , x , x , x , x , x , x , x , x , x , x , Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

33 Lyapunov exponent for the chaotic logistic map ln x Time step n Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

34 Numerical computations of LEs: Wolf et al. s algorithm The basic idea is to select a reference trajectory, and follow the divergence of a neighboring trajectory from it Let the spacing between the two trajectories at time t i be d i, and the spacing at time t i+1 be d i+1 The rate of divergence of the trajectory over a time interval of t i+1 t i is then ln(d i+1 /d i ). t i+1 t i To ensure that the separation between the two trajectories is always small, when d i+1 exceeds certain threshold value, it has to be re-normalized: a new point in the direction of the vector of d i+1 is picked up so that d i+1 is very small compared to the size of the attractor After n repetitions of stretching and renormalizing the spacing, one obtains the following formula n 1 [ ][ t i+1 t i ln(di+1 /d i λ 1 = ) ] n 1 i=1 n 1 i=1 (t = ln(d i+1/d i ) i+1 t i ) t i+1 t i t n t 1 i=1 Con: The algorithm assumes but does not verify exponential divergence; any noise gives a positive LE Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

35 Numerical computations of LEs: Rosenstein et al s and Kantz s algorithm Choose a reference point, and find its ɛ-neighbors V j Then follow the evolution of all these points, and computes an average distance after certain time Finally, choose very many reference points, and take another average Λ(k) = ln V i+k V j+k average over j average over i where V i is a reference point, and V j are neighbors to V i, satisfying the condition V i V j < ɛ If Λ(k) k for certain intermediate range of k, then the slope is the largest Lyapunov exponent Con: In order for average over j to be well-defined, ɛ has to be small; then the method is sensitive to noise. Moreover, there has to be a condition on j, as shown by Gao and Zheng Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

36 Numerical computations of LEs: Gao and Zheng s method There are three basic equation: ( ) Vi+k V j+k Λ(k) = ln V i V j ɛ i V i V j ɛ i + ɛ i, i = 1, 2, 3, i j > w Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

37 Entropy of a biological trajectory The probability p i for the i th unit area being visited is n i /N, where n i is the times the i th unit area being visited and N is the length of the trajectory m Shannon entropy I = p i log p i i=1 Can be generalized to Renyi and Tsallis entropy A/P COP (m) i k M/L COP (m) Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

38 Dynamical entropies and information flow: general considerations Partition the phase space into small boxes of size ɛ, compute the probability p i that box i is visited by the trajectory, and finally calculate Shannon entropy For many systems, when ɛ 0, information linearly increases with time I (ɛ, t) = I 0 + Kt where I 0 is the initial entropy and K is the Kolmogorov-Sinai (KS) entropy Choose I 0 = 0 if start from a unit area Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

39 Three general cases Deterministic, nonchaotic: no change in information since during the time evolution of the system, phase trajectories remain close together nearby phase points are still close to each other, and can be grouped into some other small region of the phase space Deterministic, chaotic Due to exponential divergence, the number of phase space region available to the system after a time T is N e (P λ + )T, where λ + are positive Lyapunov exponents Assuming that all of these regions are equally likely, then p i (T ) 1/N, and the information function becomes I (T ) = Therefore, K = λ + N p i (T ) ln p i (T ) = ( λ + )T i=1 Random: After a short time, the entire phase space may be visited; therefore, I ln N. When N, we have K = Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

40 Basic definitions of entropies Consider a dynamical system with F degrees of freedom Suppose that the F -dimensional phase space is partitioned into boxes of size ɛ F Suppose that there is an attractor in phase space and consider a transient-free trajectory x(t) The state of the system is now measured at intervals of time τ Let p(i 1, i 2,, i d ) be the joint probability that x(t = τ) is in box i 1, x(t = 2τ) is in box i 2,, and x(t = dτ) is in box i d Block entropy: H d (ɛ, τ) = i 1,,i d p(i 1,, i d ) ln p(i 1,, i d ) It is on the order of dτk (ɛ, τ)-entropy h d (ɛ, τ) = 1 τ [H d+1(ɛ, τ) H d (ɛ, τ)] Kolmogorov-Sinai entropy: K = lim τ 0 lim ɛ 0 h(ɛ, τ) = 1 lim τ 0 lim ɛ 0 lim d τ [H d+1(ɛ, τ) H d (ɛ, τ)] = 1 lim τ 0 lim ɛ 0 lim d dτ i 1,,i d p(i 1,, i d ) ln p(i 1,, i d ) KS entropy can be extended to order-q Renyi entropies Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

41 Numerical computations of entropy: Grassberger-Procaccia s algorithm Basic idea: approximate K by estimating the correlation entropy K 2 (i.e., Renyi entropy of order 2) K 2 (ɛ) = lim m ln C (m) (ɛ) ln C (m+1) (ɛ) where Lδt δt is the sampling time C (m) (ɛ) is the correlation integral based on the m dimensional reconstructed vectors V i and V j C (m) 2 Nv 1 Nv (ɛ) = lim Nv N v (N v 1) i=1 j=i+1 H(ɛ V i V j ), where N v = N (m 1)L is the number of reconstructed vectors, H(y) is the Heaviside function (1 if y 0 and 0 if y < 0) Similarly compute C (m+1) (ɛ) for m + 1 dimensional reconstructed vectors Summary: C m (ɛ) ɛ D 2 e mτk 2 Approximate entropy, Sample entropy are special cases of Cohen-Procaccia, and Grassberger-Procaccia entropies Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37

42 An example: chaotic Lorenz attractor for true low-dimensional chaotic dynamics, in a plot of ln C m (ɛ) vs. ln ɛ with m as a parameter, one observes a series of parallel straight lines, with the slope being the correlation dimension, D 2, and the spacing between the lines estimating K 2 (where lines for larger m lie below those for smaller m) log 10 C( ) ε log 10 ε Gao, Jianbo (PMB InTelliGence) Chaotic dynamics June / 37