USING DELAY-COORDINATE EMBEDDING AS A DATA MINING TECHNIQUE: PART ONE

Transcription

1 James Popoff, Ph.D USIG DELAY-COORDIATE EMBEDDIG AS A DATA MIIG TECHIQUE: PART OE Data mining is an example of the theory of counting in applied form. The theory of counting governs counting processes, such as addition, multiplication, combination, and permutation. The Multiplication Principle is, after simple addition its most elementary idea. To illustrate it, suppose that an experiment called E 1 can have n 1 outcomes, and for each outcome a second experiment, called E 2 can have n 2 outcomes. The composite procedure that first performs E 1 and then performs E 2 on E 1 s outcome, will have n 1 *n 2 possible outcomes. On this simple concept, combinatorial mathematics is built. Data mining works by using combinatorial mathematics, constantly sifting each possible combination of the data values and looking for patterns in their repeated occurrence, or the occurrence of similar entities, and the occurrence of surrogates (other combinations that may not resemble the original combination but often appear alongside it or nearby when the original appears. ow, predictive analytics, including inferential statistics and data mining predominantly searches for causal relationships or at least probabilistic coincidences that can explain outcomes or at least predict them within some quantifiable potential error. But the conundrum that data mining also provides is that cause and effect or even just mutual occurrence among unknown, unknowable, and ephemeral patterns (typically, having a low signal to noise ratio, they frequently fade... cannot be established. Back-channel proclivities, emergent processes, and gradients and potentials are not the same as actual variables (begging the question; when is someone going to devise a theory of derivatives for analytics?. Enter the technique of delay-coordinate embedding (also known as the Takens Ruelle embedding theorem, and, to the engineering community, state space reconstruction. This methodology, although not the hoped-for theory of data derivatives, does manage to get its arms around the hidden metrification problem to an amazing degree. These remarks will give just the overview and briefest introductory mathematics surrounding the method, and hopefully our analytics community will find it interesting enough to delve deeper (an interesting and readable paper by Elizabeth Bradley introducing these ideas in detail is at: tuvalu.santafe.edu/events/workshops/images/1/1c/ida.pdf. Here s how it works. In DCE, a time series of observations of a single system variable are used to reconstruct the entire internal dynamics of the system, including not only the behavior of the unknown variables but also their actual number. Let s imagine a particularly simple time series, namely a sine wave. In the frequency domain it s represented simply as a vertical line (amplitude at a particular location on a coordinate axis (the horizontal or x- axis that represents a frequency. If we move to the time domain, the familiar sinusoidal 1

2 pattern emerges with amplitude on the vertical y-axis and angular distance (or time, in this case along the x-axis. But if we again shift our mindset to the domain that shows the states of the system, in two dimensions θ and dθ called the state space, we get a circular orbit about a point (the x-axis, seen end-on. If we include the more realistic situation of random noise and systemic errors to the time domain picture, we cause the smooth sine waveform to show quite irregular excursions as errors affect the curve. In the state space domain this will be seen as a messy superposition of orbit after orbit after orbit, none of which perfectly coincide to show the single circular orbit as before. The result is a band of superposed orbits that have about the same radius and have about the same, approximately-circular, shape; although they appear quite wobbly in their coincidence around that x-axis. For some parts of the system, it now seems the axis is not the point, but the orbit has assumed pride of place. The orbit is an attractor. The state space rendering is called the graph of an attractor for the system, and each of the trajectories can be enumerated, identified, and evaluated both mathematically and qualitatively. That is what a state space reconstruction accomplishes; it gives us an opportunity to identify, measure, propose a name for and reason about, and interpret each of the influences on our system. Hitherto they ve been called hidden variables, and been assumed to be random, systemic errors and unknown (and unknowable externalities, collectively producing model residuals. Attractors come in four varieties, and the different mathematical tools developed for coping with each of the known attractor types is the key to our being able to measure and discuss the components of residual error in our models. After Bradley, they are: Fixed or equilibrium points, periodic orbits (called limit cycles, quasiperiodic attractors, and chaotic or strange attractors. Attractors represent the idea of a dynamical invariant, which is some kind of fixed structure within the state space that remains after all transients have died out. The imaginary experiment we just worked through is, in fact, a chaotic system having a strange attractor. There are several mathematical approaches toward describing this system under its attractor. Perhaps the most important for hard-headed analysts is the famous method of Lyapunov Exponents. An heuristic helps illustrate the idea. We imagine a state vector X with an updating rule Γ, also in the form of a vector, corresponding to the x 1, x 2, x 3,..., x n elements of X. If we introduce a small change, Δ, called a perturbation, also in vector form, the result after the update rule can be found using a Taylor series expansion: "(x i +# i = "(x i +# i " (x i +K (1 where the remaining terms are so small they can be ignored, and " i, j = #$ i (2 #x j is the i, jth element of the derivative matrix (the partial derivative of γ with respect to x of the outer product of the update rule s row vector and its state s column vector. Then 2

3 "(x i +# i $ "(x i % # i " (x i (3 Of course, the square derivative matrix has d eigenvalues and orthonormal eigenvectors. The distance of the iteration from the original state to the updated state depends on the size of the perturbation applied, and we define the direction of the iteration in state space in terms of the normalized perturbation vector: M i = " i # (4 where M i is the relative amplitude of the particular perturbation applied to the particular state component i. It can be shown (Klein, 2001 that a particular direction, call it e 1, will maximize M i. That is, we want to find: % e 1 = sup " i #(x i ( ' * & $ We then have a remaining subset of e n - 1 (x i dimensional directions that are each perpendicular to e 1. ow we can serially find the direction in the remaining subset that next-maximizes M i and is perpendicular to all other directions, and so on until we have found all the directions, such that e n is the direction having the smallest perturbation. Sounds like principal components analysis, right? This stuff really is not too far a leap for our everyday methods to encompass with a little arm twisting. Since we repeatedly applied our analysis to the system, we can say: (5 " n (x i +# i $ " n (x i % = (M i (x i n = e n log M i (x i (6 For simplicity we can replace log M i by Λ i and the set {Λ 1 (x 1, Λ 2 (x 2,..., Λ n (x n } is called the local Lyapunov exponents at location X in the state space. If the Λ i is positive the state space is expanding in that direction (unstable; if the Λ i is negative the state space is shrinking in that direction (stable, and if Λ i = 0 the state space is constant (neutral. The time average of the Λ i over a trajectory i of the system results in: 1 T T #" i (x n n =1 which in the limit becomes the global Lyapunov exponent or just the Lyapunov exponent: " i = $ # i (x i %(x i dx (7 where ρ(x i is the variational distribution (here, simple relative-frequencies buckets. Chaos is characterized by extreme sensitivity to initial conditions; a raindrop a millimeter away from a cliff edge can take extremely different paths to ground depending on whether or not a puff of air occurs. The λ measure how fast extremely-close 3

4 trajectories can diverge (expand the multivariate state space volume or converge (shrink the volume along the serially-ordered, mutually-perpendicular dimensions or directions of the state space, signaled by the eigenvalues. Chaotic attractors have at least one positive λ, that tends to force trajectories apart, only one zero λ acting along the attractor, and negative λ perturbations that die off in all directions around the attractor. For now, assume we ve ascertained the correct dimension of the system (we ll cover that in detail below, and call it d E. We construct samples made up of d E dimensioned vectors from our observations of a single sensor taken at uniform intervals Δt apart. Depicting the system by its sensor values amounts to projecting the d E dimensions of the system onto the single axis of the serial sensor observations. Suppose we have a scalar time series of n = 20 once-daily observations that we ll identify by their observation number to keep things simple. Call it x(t = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}. Then the reconstruction vector X(τ, with, say, a delay of τ = 4 days, and, say, a dimension d E of 3, will look like this: X(τ = {(1, 5, 9, (2, 6, 10, (3, 7, 11, (4, 8,12, (5, 9, 13, (6, 10, 14, (7, 11, 15, (8, 12, 16, (9, 13, 17, (10, 14, 18, (11, 15, 19, (12, 16, 20, (13, 17, 1, (14, 18, 2, (15, 19, 3, (16, 20, 4, (17, 1, 5, (18, 2, 6, (19, 3, 7, (20, 4, 8, (1, 5, 9 }; n ordered tuples whose components are 4 days apart. We have expanded the scalar time series x(t into the vector time series X(τ. Embedding re-inflates the d E dimensional system, although topologically it then unfolds along different axes than the original (the delay-coordinate axes instead of the true state variables of the original complex system. evertheless, the reconstructed state space trajectory is identical with the behavior of the original system and inferences about the latter made on observations of the former are completely valid. We impose a minimum size of the reconstructed d E dimensional system of 2n + 1, which guarantees that as the system is unfolded no dimensions obscure one another by crossing (a potatochip-like warped polygon is still topologically equivalent to a circle, but viewed edge-on as a figure-eight that equality no longer applies; the dimensional limit we impose guarantees the appropriate point of view. So our problem becomes how to find the correct dimension and lag time to unfold the projection. Dimension is commonly found using the false near neighbor (F algorithm (Kennel et al, 1992, but that method requires us to provide a value for τ. An accessible method for finding the lag time τ was first proposed by Fraser and Swinney (1986; see also, Gershenfeld, 2000 and is called the method of mutual information redundancy (MIR. Information is measured using entropy, which is the expected value of bits required to describe the distribution of an observable quantized to one of integer values: y(t "{1,...,; where the distribution is approximated by binning; p 1 (y = n y nt and H 1 ( = "# p 1 (ylog 2 p 1 (y (8 y =1 Based on the distribution p(y the entropy H scores the amount of surprise the observable produces; a uniform distribution has the highest entropy (surprise value because the 4

5 likelihood of seeing any of the observables is equal, while a sharply peaked distribution has low entropy because we expect to see one or only a small subset of values. The entropy also measures the amount of structure (i.e., information versus pure energy, not to get too esoteric inherent in the data. The notion of mutual information refers to the degree to which we can say something about the next observation (which for ergodic data is in its own probability domain distinct from any other given what we already know about the previous observations (each of which also has a unique and independent distribution. In other words, some of the to-be-observed probability distribution may be redundant; if so, it provides us with nothing new to the extent that redundancy occurs. The mutual information is defined as the difference in information between two samples taken independently and taken together: I 2 (", = # $ p 1 log 2 p 1 and y t =1 " $ p 1 "# log 2 p 1 "# y t"# =1 + % " % p 2, y t #$ log 2 p 2, y t #$ (9 y t =1 y t#$ =1 I 2 (", = 2H 1 (", # H 2 (", (10 where the subscript 2 means base-2 where it is associated with the log and information where the result is in bits, and means the second observation s distribution elsewhere. Fraser and Swinney demonstrated that the total elapsed time associated with the first local minimum of the mutual information curve is the best value to use for the embedding lag τ. Various process conditions and errors emerge as the system matures over time, which accounts for their emphasis on an early (i.e., the first local minimum. Once the lag is known, we can use it to find the dimension D. There are many ways to go about finding D, including using the entropy (Gershenfeld, 2000, but the false near neighbor (F method is straightforward to apply and easy to explain. From the set of observations define multivariate vectors in d-dimensional space: y(n = (x(n, x(n + T,..., x(n + (d "1T (11 that within a given d identify the orbit of the system. Observations x(n on the orbit are identified as nearest neighbors in d-space, but we test the hypothesis by translating the system from d to d + 1 dimensions; if the distance separating the neighbors doesn t change, we have true nearest neighbors. Otherwise we have discovered a false near neighbor; it looks near because the embedding dimension is not yet sufficient to let us see a true picture. When, after iterating the procedure the number of false near neighbors falls to zero, we have found the minimum dimension d that will support embedding all the observations. So, in d dimensions denote the rth nearest neighbor of point x(n to be x (r (n, and the Euclidean distance between x(n and x (r (n is: 5

6 d "1 R 2 d (n,r = #[x(n + kt " x (r (n + kt] 2 (12 k =0 To increment the dimensionality we add a (d + 1th coordinate onto each of the y(n vectors, which is simply observation number x(n + Td. So, what is the Euclidean distance in dimension d + 1 between x(n and x (r (n? We recycle equation (12 by changing the subscript to d + 1 on the left-hand-side, and test the result with: # % $ 2 R d +1 (n,r " R d 2 (n,r R d 2 (n,r 1 & 2 x(n + Td " x (r (n + Td ( = > R ' R d (n,r tol (13 where R tol is a threshold error or change in the distance that we determine ahead of time. If equation (13 holds, we have a false near neighbor, and so on through the iterations. As we can plainly see, the method is stone simple, and since I promised to keep the math down to a dull roar, equation (13 will be our last. The reader who desires more details can find the fully-developed mathematics in the references below. For now, I want to end this note with a review of what we have here and how we can benefit by its use. Let s consider a single time series for a single variable. In this case, we actually have daily totals of online visits ( clicks and daily totals of conversions or selected actions by the visitor up to and including an actual purchase that can be monetized. As we will see, the correlation between these time series is significant, but not strong, and the time series we are actually considering as a result is the value of their daily differences. Why would the two input data sets be weakly correlated? The separate processes that might go into each data set, suggest some immediate differences. First, a visitor might not purchase immediately; some do, of course, but others might make several visits (including visits to other websites to view the item and compare its terms before they make the purchase decision on their final visit, which might come about quite some time after the initial visit (or visit cluster. A corollary of this is the visitor s personal characteristics; the presence of young children, for example, who may disrupt the session and generate a repeat visit in order to complete the transaction. Some processing time is required for some purchase decisions, such as memberships, subscriptions, and credit card applications that introduces a variable delay between the time of the visit and the time the purchase is attributed. Daily visitors and purchasers may be subject to regionally similar and very different outside influences at any given time. For example, local weather conditions may encourage or discourage internet activity quite differently in different regions. For some, there may be a local holiday, an election, or even a labor strike that affects services including internet access, or there may be one or more server failures in the chain between the host and the client. So we know that our simple time series of differences is actually a very complex system projected onto a single dimension. Why is this even a problem? Well, if we can model the time series of visitors (usually fairly easy to do and add (or subtract the corresponding time series of differences, we can make a forecast of revenues. However, as anyone who has actually tried to do this can attest, there are in reality sometimes quite huge errors among the model, the forecast, and the empirical outcome. The techniques for dealing with and controlling those errors are properly the subject of Part Two of this paper (forthcoming. However, we do want to see how our new knowledge applies. 6

7 Our theory gives us some important tools. We can easily find the number of degrees of freedom in our system (the actual number of variables producing the time series of differences. Our Lyapunov exponents give us insight into the naming and interpretation of the variables (remember, a negative sign means the system is damped and dying along that axis, a positive sign means the system is exploding along that axis, and a zero means the system is neutral along that axis. Some variables are easy to interpret and infer according to those criteria. For example, dissipative processes include processing lag times, and repeat visits and clusters of visits; while unstable, explosive processes (a perturbation that grows, like the Galloping Gerty Tacoma arrows Bridge disaster, or the Comet metal fatigue saga include demographic and personal influences, and geocoded events such as regional service interruptions due to social factors, or violent weather and natural phenomena. Although these are applied to a single purchase (i.e., they should, seemingly, be considered dissipative, not persistent let alone growing they remain as a class of influence for segments of the visitor base. eutral processes that simply follow the orbit can also be geocoded as regional seasonality; and appear as weather impacts on internet use (mild, sunny days predict a period of fewer visitors; while the opposite is true of heavy storms, up to a point, or winter cold. One is guided by the number of variables too, and their location in the hierarchy of eigenvalues in making these assignments. It s not hard to see how to proceed in making valuable strategic recommendations from such an analysis, even if making precise revenue forecasts for the system may still be a little beyond our analytic capability. Still, before we had nothing, and wouldn t have attempted the analysis in the first place. ow, rather than just discard those data as useless because we don t understand how to use them, the smart analyst can try this method and see if it pays off. REFERECES Bradley, Elizabeth Time Series Analysis, in D. Hand and M. Berthold, eds. Intelligent Data Analysis: An Introduction (ch. 5, 2 nd edition, ew York: Springer- Verlag Kennel, Matthew B. and Reggie Brown Determining embedding dimension for phase-space reconstruction using a geometrical construction. Physical Review Letters A, Vol. 45 o. 6; pp Klein, Dan Lagrange Multipliers without Permanent Scarring. Online tutorial. URL Fraser, Andrew M. and Harry L. Swinney, Independent coordinates for strange attractors from mutual information. Physical Review Letters Vol. 33, o. 2; pp Gershenfeld, eil A The ature of Mathematical Modeling, ew York: Cambridge University Press, pp