Exploratory Causal Analysis in Bivariate Time Series Data Abstract

Transcription

1 Exploratory Causal Analysis in Bivariate Time Series Data Abstract Many scientific disciplines rely on observational data of systems for which it is difficult (or impossible) to implement controlled experiments and data analysis techniques are required for identifying causal information and relationships directly from observational data. This need has lead to the development of many different time series causality approaches and tools including transfer entropy, convergent cross-mapping (CCM), and Granger causality statistics. A practicing analyst can explore the literature to find many proposals for identifying drivers and causal connections in times series data sets, but little research exists of how these tools compare to each other in practice. This work introduces and defines exploratory causal analysis (ECA) to address this issue. The motivation is to provide a framework for exploring potential causal structures in time series data sets. J. M. McCracken Defense talk for PhD in Physics, Department of Physics and Astronomy 10:00 AM November 20, 2015; Exploratory Hall, 3301 Advisor: Dr. Robert Weigel; Committee: Dr. Paul So, Dr. Tim Sauer J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

2 Exploratory Causal Analysis in Bivariate Time Series Data J. M. McCracken Department of Physics and Astronomy George Mason University, Fairfax, VA November 20, 2015 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

3 Outline 1. Motivation 2. Causality studies 3. Data causality 4. Exploratory causal analysis 5. Making an ECA summary Transfer entropy difference Granger causality statistic Pairwise asymmetric inference Weighed mean observed leaning Lagged cross-correlation difference 6. Computational tools for the ECA summary 7. Empirical examples Cooling/Heating System Data Snowfall Data 8. Times series causality as data analysis J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

4 Motivation Data Consider two sets of time series measurements, X and Y. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

5 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

6 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

7 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

8 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, had straightforward, preferably well-established, interpretations, J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

9 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, had straightforward, preferably well-established, interpretations, were reliable, J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

10 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, had straightforward, preferably well-established, interpretations, were reliable, and did not require studying the (vast) philosophical causality literature. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

11 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, had straightforward, preferably well-established, interpretations, were reliable, and did not require studying the (vast) philosophical causality literature. Essentially, we were looking for a plug-and-play analysis tool. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

12 Motivation Data Consider two sets of time series measurements, X and Y. Question Is there evidence that X drives Y? We were looking for a data analysis approach, i.e., we were looking for analysis tools that worked with time series data, had straightforward, preferably well-established, interpretations, were reliable, and did not require studying the (vast) philosophical causality literature. Essentially, we were looking for a plug-and-play analysis tool. This work stems from our search for such a tool. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

13 Causality studies The study of causality is as old as science itself Modern historians credit Aristotle with both the first theory of causality ( four causes ) and an early version of the scientific method The modern study of causality is broadly interdisciplinary; far too broad to review in a short talk. Illari and Russo s textbook 1 provides an overview of causality studies 1 Illari, P., & Russo, F. (2014). Causality: Philosophical theory meets scientific practice. Oxford University Press. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

14 Towards a taxonomy of causal studies Paul Holland identified four types of causal questions 2 : the ultimate meaningfulness of the notion of causality the details of causal mechanisms the causes of a given effect the effects of a given cause 2 Holland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

15 Towards a taxonomy of causal studies Paul Holland identified four types of causal questions 2 : the ultimate meaningfulness of the notion of causality the details of causal mechanisms the causes of a given effect the effects of a given cause Foundational causality Is a cause required to precede an effect? or How are causes and effects related in space-time? 2 Holland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

16 Towards a taxonomy of causal studies Paul Holland identified four types of causal questions 2 : the ultimate meaningfulness of the notion of causality the details of causal mechanisms the causes of a given effect the effects of a given cause Foundational causality Is a cause required to precede an effect? or How are causes and effects related in space-time? Data causality Does smoking cause lung cancer? or Are traffic accidents caused by rain storms? 2 Holland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

17 Data causality Data causality is data analysis to draw causal inferences J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

18 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

19 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) potential outcomes (Rubin s counterfactuals) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

20 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) potential outcomes (Rubin s counterfactuals) directed acyclic graphs (DAGs) with structural equation models (SEMs); popularized by Pearl as structural causal models (SCMs) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

21 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) potential outcomes (Rubin s counterfactuals) directed acyclic graphs (DAGs) with structural equation models (SEMs); popularized by Pearl as structural causal models (SCMs) time series causality J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

22 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) potential outcomes (Rubin s counterfactuals) directed acyclic graphs (DAGs) with structural equation models (SEMs); popularized by Pearl as structural causal models (SCMs) time series causality There is no consensus on the best approach to data causality Many authors consider their favored approach to be the exclusive correct approach. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

23 Data causality Data causality is data analysis to draw causal inferences Approaches to data causality studies include design of experiments (e.g., Fisher randomization) potential outcomes (Rubin s counterfactuals) directed acyclic graphs (DAGs) with structural equation models (SEMs); popularized by Pearl as structural causal models (SCMs) time series causality There is no consensus on the best approach to data causality Many authors consider their favored approach to be the exclusive correct approach. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

24 Time series causality Time series causality is data causality with time series data J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

25 Time series causality Time series causality is data causality with time series data Approaches to time series causality can be roughly divided into five categories, Granger (model based approaches) Information-theoretic State space reconstruction (SSR) Correlation Penchant J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

26 Exploratory causal analysis Language Exploring causal structures in data sets is distinct from confirming causal structures in data sets. Causal language used in ECA should not be conflated with other typical uses; i.e., cause, effect, drive, etc. are used as technical terms with definitions unrelated to their common, everyday definitions. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

27 Exploratory causal analysis Language Exploring causal structures in data sets is distinct from confirming causal structures in data sets. Causal language used in ECA should not be conflated with other typical uses; i.e., cause, effect, drive, etc. are used as technical terms with definitions unrelated to their common, everyday definitions. and will be used as shorthand for causal statements, e.g., A drives B will be written as A B. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

28 Exploratory causal analysis Assumptions A cause always precedes an effect. This assumption is required for the operational definitions of causality. A driver may be present in the data being analyzed. This assumption may lead to issues of confounding. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

29 Exploratory causal analysis ECA summary vector approach We will not favor a specific operational definition of causality we do not favor any particular tool Consider a time series pair (X, Y), J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

30 Exploratory causal analysis ECA summary vector approach We will not favor a specific operational definition of causality we do not favor any particular tool Consider a time series pair (X, Y), ECA summary vector Define a vector g where each element g i is defined as either 0 if X Y, 1 if X Y, or 2 if no causal inference can be made. The value of each g i comes from a specific time series causality tool. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

31 Exploratory causal analysis ECA summary vector approach We will not favor a specific operational definition of causality we do not favor any particular tool Consider a time series pair (X, Y), ECA summary vector Define a vector g where each element g i is defined as either 0 if X Y, 1 if X Y, or 2 if no causal inference can be made. The value of each g i comes from a specific time series causality tool. ECA summary The ECA summary is either X Y, Y X, or undefined, with g i = 0 g i g X Y and g i = 1 g i g Y X. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

32 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

33 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference Granger log-likelihood statistics pairwise asymmetric inference (PAI) average weighted mean observed leaning lagged cross-correlation difference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

34 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference information-theoretic Granger log-likelihood statistics pairwise asymmetric inference (PAI) average weighted mean observed leaning lagged cross-correlation difference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

35 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference information-theoretic Granger log-likelihood statistics Granger pairwise asymmetric inference (PAI) average weighted mean observed leaning lagged cross-correlation difference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

36 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference information-theoretic Granger log-likelihood statistics Granger pairwise asymmetric inference (PAI) SSR average weighted mean observed leaning lagged cross-correlation difference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

37 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference information-theoretic Granger log-likelihood statistics Granger pairwise asymmetric inference (PAI) SSR average weighted mean observed leaning penchant lagged cross-correlation difference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

38 Making an ECA summary Our focus is on time series, so each causal inference g i g will be drawn from a tool in one of each of the five time series causality categories. g 1 g 2 g 3 g 4 g 5 transfer entropy difference information-theoretic Granger log-likelihood statistics Granger pairwise asymmetric inference (PAI) SSR average weighted mean observed leaning penchant lagged cross-correlation difference correlation J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

39 Transfer entropy (g 1 ) Shannon entropy The uncertainty that a random variable X takes some specific value X n is given by the Shannon (or information) entropy, N X H X = P(X = X n ) log 2 P(X = X n ) n=1 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

40 Transfer entropy (g 1 ) Shannon entropy The uncertainty that a random variable X takes some specific value X n is given by the Shannon (or information) entropy, N X H X = n=1 P(X = X n ) log 2 P(X = X n ) P(X = X n ) is the probability that X takes the specific value X n J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

41 Transfer entropy (g 1 ) Shannon entropy The uncertainty that a random variable X takes some specific value X n is given by the Shannon (or information) entropy, H X = N X n=1 P(X = X n ) log 2 P(X = X n ) The sum is over all possible values of X n ; n = 1, 2,..., N X J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

42 Transfer entropy (g 1 ) Shannon entropy The uncertainty that a random variable X takes some specific value X n is given by the Shannon (or information) entropy, N X H X = P(X = X n ) log 2 P(X = X n ) n=1 The base of the logarithm sets the entropy units, which is bits here J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

43 Transfer entropy (g 1 ) Shannon entropy example Binary example (to help with intuition) Consider a coin C that take the value H with probability p H and T with probability p T. The Shannon entropy is H C = (p H log 2 p H + p T log 2 p T ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

44 Transfer entropy (g 1 ) Shannon entropy example Binary example (to help with intuition) Consider a coin C that take the value H with probability p H and T with probability p T. The Shannon entropy is H C = (p H log 2 p H + p T log 2 p T ) completely uncertain of outcome Fair coin p H = p T = 0.5 H C = 1 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

45 Transfer entropy (g 1 ) Shannon entropy example Binary example (to help with intuition) Consider a coin C that take the value H with probability p H and T with probability p T. The Shannon entropy is H C = (p H log 2 p H + p T log 2 p T ) completely uncertain of outcome Fair coin p H = p T = 0.5 H C = 1 completely certain of outcome Always heads (or tails) p H(T ) = 0, p T (H) = 1 H C = 0 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

46 Transfer entropy (g 1 ) Shannon entropy example Binary example (to help with intuition) Consider a coin C that take the value H with probability p H and T with probability p T. The Shannon entropy is H C = (p H log 2 p H + p T log 2 p T ) completely uncertain of outcome Fair coin p H = p T = 0.5 H C = 1 completely certain of outcome Always heads (or tails) p H(T ) = 0, p T (H) = 1 H C = 0 (Entropy calculations almost always assume 0 log 2 0 := 0.) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

47 Transfer entropy (g 1 ) Mutual information A pair of random variables (X, Y) have some mutual information given by I X ;Y = H X + H Y H X,Y = N X N Y n=1 m=1 P(X = X n, Y = Y m ) log 2 P(X = X n, Y = Y m ) P(X = X n )P(Y = Y m ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

48 Transfer entropy (g 1 ) Mutual information A pair of random variables (X, Y) have some mutual information given by I X ;Y = H X + H Y H X,Y = N X N Y n=1 m=1 P(X = X n, Y = Y m ) log 2 P(X = X n, Y = Y m ) P(X = X n )P(Y = Y m ) P(X = X n, Y = Y m ) is the probability that X takes the specific value X n and Y takes the specific value Y m J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

49 Transfer entropy (g 1 ) Mutual information A pair of random variables (X, Y) have some mutual information given by I X ;Y = H X + H Y H X,Y = N X N Y n=1 m=1 P(X = X n, Y = Y m ) log 2 P(X=X n,y=y m) P(X=X n)p(y=y m) If X and Y are independent, then P(X = X n, Y = Y m ) = P(X = X n )P(Y = Y m ) I X ;Y = 0 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

50 Transfer entropy (g 1 ) Mutual information A pair of random variables (X, Y) have some mutual information given by I X ;Y = H X + H Y H X,Y = N X N Y n=1 m=1 P(X = X n, Y = Y m ) log 2 P(X = X n, Y = Y m ) P(X = X n )P(Y = Y m ) The mutual information is symmetric; i.e., I X ;Y = I Y ;X J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

51 Transfer entropy (g 1 ) Mutual information A pair of random variables (X, Y) have some mutual information given by I X ;Y = H X + H Y H X,Y = N X N Y n=1 m=1 P(X = X n, Y = Y m ) log 2 P(X = X n, Y = Y m ) P(X = X n )P(Y = Y m ) Schreiber proposed an extension of the mutual information to measure information flow by making it conditional and including assumptions about the temporal behavior X and Y. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

52 Transfer entropy (g 1 ) Information flow Suppose X and Y are both Markov processes. The directed flow of information from Y to X is given by the transfer entropy, with T Y X = N X N Y n=1 m=1 p n+1,n,m log 2 p n+1 n,m p n+1 n p n+1,n,m = P(X(t + 1) = X n+1, X(t) = X n, Y(τ) = Y m ) p n+1 n,m = P(X(t + 1) = X n+1 X(t) = X n, Y(τ) = Y m ) p n+1 n = P(X(t + 1) = X n+1 X(t) = X n ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

53 Transfer entropy (g 1 ) Information flow There is no directed information flow from Y to X if X is conditionally independent of Y; i.e., p n+1 n,m = p n+1 n T Y X = 0 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

54 Transfer entropy (g 1 ) Information flow There is no directed information flow from Y to X if X is conditionally independent of Y; i.e., p n+1 n,m = p n+1 n T Y X = 0 Operational causality (information-theoretic) X causes Y if the directed information flow from X to Y is higher than the directed information flow from Y to X; i.e., T X Y T Y X > 0 X Y T X Y T Y X < 0 Y X T X Y T Y X = 0 no causal inference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

55 Granger causality (g 2 ) Granger s axioms Consider a discrete universe with two time series X = {X t t = 1,..., n} and Y = {Y t t = 1,..., n}, where t = n is considered the present time. All knowledge available in the universe at all times t n is denoted as Ω n. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

56 Granger causality (g 2 ) Granger s axioms Consider a discrete universe with two time series X = {X t t = 1,..., n} and Y = {Y t t = 1,..., n}, where t = n is considered the present time. All knowledge available in the universe at all times t n is denoted as Ω n. Axiom 1 The past and present may cause the future, but the future cannot cause the past. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

57 Granger causality (g 2 ) Granger s axioms Consider a discrete universe with two time series X = {X t t = 1,..., n} and Y = {Y t t = 1,..., n}, where t = n is considered the present time. All knowledge available in the universe at all times t n is denoted as Ω n. Axiom 1 The past and present may cause the future, but the future cannot cause the past. Axiom 2 Ω n contains no redundant information, so that if some variable Z is functionally related to one or more other variables, in a deterministic fashion, then Z should be excluded from Ω n. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

58 Granger causality (g 2 ) Granger s axioms Consider a discrete universe with two time series X = {X t t = 1,..., n} and Y = {Y t t = 1,..., n}, where t = n is considered the present time. All knowledge available in the universe at all times t n is denoted as Ω n. Granger s definition of causality Given some set A, Y causes X if P(X n+1 A Ω n ) P(X n+1 A Ω n Y) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

59 Granger causality (g 2 ) Granger s axioms Consider a discrete universe with two time series X = {X t t = 1,..., n} and Y = {Y t t = 1,..., n}, where t = n is considered the present time. All knowledge available in the universe at all times t n is denoted as Ω n. Granger s definition of causality Given some set A, Y causes X if P(X n+1 A Ω n ) P(X n+1 A Ω n Y) Granger s original goal was to make this notion of causality operational. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

60 Granger causality (g 2 ) VAR models Consider a time series pair (X, Y). Suppose there is a vector autoregressive (VAR) model that describes the pair, ( Xt Y t ) = n ( A i 11 A i ) ( ) 12 Xt i A i 21 A i + 22 Y t i i=1 ( ε1,t ε 2,t ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

61 Granger causality (g 2 ) VAR models Consider a time series pair (X, Y). Suppose there is a vector autoregressive (VAR) model that describes the pair, ( Xt Y t ) = n ( A i 11 A i ) ( ) 12 Xt i A i 21 A i + 22 Y t i i=1 ( ε1,t ε 2,t ) The current time step t of X and Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

62 Granger causality (g 2 ) VAR models Consider a time series pair (X, Y). Suppose there is a vector autoregressive (VAR) model that describes the pair, ( Xt Y t ) = n ( A i 11 A i ) ( ) 12 Xt i A i 21 A i + 22 Y t i i=1 ( ε1,t ε 2,t ) The current time step t of X and Y is modeled as a sum of n past steps J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

63 Granger causality (g 2 ) VAR models Consider a time series pair (X, Y). Suppose there is a vector autoregressive (VAR) model that describes the pair, ( Xt Y t ) = n i=1 ( A i 11 A i ) ( ) 12 Xt i A i 21 A i + 22 Y t i ( ε1,t ε 2,t ) The current time step t of X and Y is modeled as a sum of n past steps of X and Y, J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

64 Granger causality (g 2 ) VAR models Consider a time series pair (X, Y). Suppose there is a vector autoregressive (VAR) model that describes the pair, ( Xt Y t ) = n ( A i 11 A i ) ( ) 12 Xt i A i 21 A i + 22 Y t i i=1 ( ε1,t ε 2,t ) The current time step t of X and Y is modeled as a sum of n past steps of X and Y, plus uncorrelated noise terms. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

65 Granger causality (g 2 ) Comparison of VAR models Consider two different VAR models for the pair (X, Y), ( Xt Y t ( Xt Y t ) = ) = n ( ) ( ) Axx,i A xy,i Xt i + i=1 A yx,i A yy,i n ( A xx,i 0 i=1 0 A yy,i Y t i ) ( Xt i Y t i ( εx,t ε y,t ) ) ( ε ) + x,t ε y,t J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

66 Granger causality (g 2 ) Comparison of VAR models Consider two different VAR models for the pair (X, Y), ( Xt Y t ( Xt Y t ) = ) = n ( ) ( ) Axx,i A xy,i Xt i + i=1 A yx,i A yy,i n ( A xx,i 0 i=1 0 A yy,i Y t i ) ( Xt i Y t i The G-causality log-likelihood statistic is defined as F Y X = ln Σ xx Σ xx ( εx,t ε y,t ) ) ( ε ) + x,t ε y,t J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

67 Granger causality (g 2 ) Comparison of VAR models Consider two different VAR models for the pair (X, Y), ( Xt Y t ( Xt Y t ) = ) = n ( ) ( ) Axx,i A xy,i Xt i + i=1 A yx,i A yy,i n ( A xx,i 0 i=1 0 A yy,i Y t i ) ( Xt i Y t i The G-causality log-likelihood statistic is defined as F Y X = ln Σ xx Σ xx ( εx,t ε y,t ) ) ( ε ) + x,t ε y,t Covariance of X model residuals given no dependence on Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

68 Granger causality (g 2 ) Comparison of VAR models Consider two different VAR models for the pair (X, Y), ( Xt Y t ( Xt Y t ) = ) = n ( ) ( ) Axx,i A xy,i Xt i + i=1 A yx,i A yy,i n ( A xx,i 0 i=1 0 A yy,i Y t i ) ( Xt i Y t i The G-causality log-likelihood statistic is defined as F Y X = ln Σ xx Σ xx ( εx,t ε y,t ) ) ( ε ) + x,t ε y,t Covariance of X model residuals given a possible dependence on Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

69 Granger causality (g 2 ) G-causality log-likelihood statistic If both VAR models fit (or forecast ) the data equally well, then there is no G-causality; i.e., Σ xx = Σ xx F Y X = 0 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

70 Granger causality (g 2 ) G-causality log-likelihood statistic If both VAR models fit (or forecast ) the data equally well, then there is no G-causality; i.e., Σ xx = Σ xx F Y X = 0 Operational causality (Granger) X causes Y if the X-dependent forecast of Y decreases the Y model residual covariance (as compared to the X-independent forecast) more than the Y-dependent forecast of X decreases the X model residual covariance (as compared to the Y-independent forecast); i.e., F X Y F Y X > 0 X Y F X Y F Y X < 0 Y X F X Y F Y X = 0 no causal inference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

71 Pairwise asymmetric inference (g 3 ) State space reconstruction Consider an embedding of the time series X = {x t t = 0, 1..., L 1, L} constructed from delayed time steps as X = { x t t = 1 + (E 1)τ,..., L} with x t = ( x t, x t τ, x t 2τ,..., x t (E 1)τ ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

72 Pairwise asymmetric inference (g 3 ) State space reconstruction Consider an embedding of the time series X = {x t t = 0, 1..., L 1, L} constructed from delayed time steps as X = { x t t = 1 + (E 1)τ,..., L} with x t = ( x t, x t τ, x t 2 τ,..., x t (E 1) τ ) τ is the delay time step J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

73 Pairwise asymmetric inference (g 3 ) State space reconstruction Consider an embedding of the time series X = {x t t = 0, 1..., L 1, L} constructed from delayed time steps as X = { x t t = 1 + (E 1)τ,..., L} with x t = ( ) x t, x t τ, x t 2τ,..., x t ( E 1)τ τ is the delay time step E is the embedding dimension J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

74 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

75 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) 1. Find the n nearest neighbors to x t (in X), where nearest means smallest Euclidean distance, d; i.e., d 1 < d 2 <... < d n J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

76 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) 2. Create weights,w, from the nearest neighbors as w i = e d i d 1 n j=1 e d j d 1 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

77 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) 3. Construct the cross-mapped estimate of Y using the weights as { } n Y X = Y t X = t = 1 + (E 1)τ,..., L i=1 w i Yˆt i J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

78 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) Each cross-mapped point in the estimate of Y, i.e., Y t X = n i=1 e d i /d 1 n j=1 e d j /d 1 Yˆt i depends on comparisons of J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

79 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = (x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) Each cross-mapped point in the estimate of Y, i.e., Y t X = n i=1 e d i /d 1 n j=1 e d j /d 1 Yˆt i depends on comparisons of the pasts of X J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

80 Pairwise asymmetric inference (g 3 ) Cross-mapping Consider a time series pair (X, Y). The shadow manifold of X (labeled X) is constructed from the points x t = ( x t, x t τ, x t 2τ,..., x t (E 1)τ, y t ) Each cross-mapped point in the estimate of Y, i.e., Y t X = n i=1 e d i /d 1 n j=1 e d j /d 1 Yˆt i depends on comparisons of the pasts of X and the presents of X and Y. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

81 Pairwise asymmetric inference (g 3 ) Cross-mapped correlation A good cross-mapped estimate is defined as one that is strongly correlated with the original times series. The cross-mapped correlation is C YX = [ ] 2 ρ(y, Y X) where ρ ( ) is Pearson s correlation coefficient. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

82 Pairwise asymmetric inference (g 3 ) Cross-mapped correlation A good cross-mapped estimate is defined as one that is strongly correlated with the original times series. The cross-mapped correlation is C YX = [ ] 2 ρ(y, Y X) where ρ ( ) is Pearson s correlation coefficient. Cross-mapping interpretation If similar histories of X (i.e., nearest neighbors in the shadow manifold) capably estimate Y (i.e., lead to C YX 1, or at least C YX 0), then the presence (or action) of Y in the system has been recorded in X. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

83 Pairwise asymmetric inference (g 3 ) Cross-mapping interpretation of causality A time series pair (X, Y) will have two cross-mapped correlations, C YX and C XY. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

84 Pairwise asymmetric inference (g 3 ) Cross-mapping interpretation of causality A time series pair (X, Y) will have two cross-mapped correlations, C YX and C XY. Operational causality (SSR) X causes Y if similar histories of Y estimate X better than similar histories of X estimate Y, where the similar histories of one time series are used to estimate another time series through shadow manifold nearest neighbor weighting (cross-mapping); i.e., C YX C XY < 0 X Y C YX C XY > 0 Y X C YX C XY = 0 no causal inference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

85 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

86 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) P (E C) is the probability of some effect E given some cause C J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

87 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) P ( E C ) is the probability of some effect E given no cause C J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

88 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) So, the penchant is the probability of an effect E given a cause C minus the probability of that effect without the cause J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

89 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) In the psychology/medical literature, the causal penchant is known as the Eells measure of causal strength or probability contrast. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

90 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) If C drives E, then it is expected that ρ EC > 0. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

91 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P (E C) P ( E C ) The second term, P ( E C ), can be eliminated from the penchant formula using Bayes theorem. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

92 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P(E C) ( 1 + P(C) 1 P(C) ) P(E) 1 P(C) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

93 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P(E C) ( 1 + P(C) 1 P(C) ) P(E) 1 P(C) If E and C are independent, then P(E C) = P(E), which implies ρ EC = P(E) + P(E)P(C) P(E) 1 P(C) = P(E) P(E) = 0 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

94 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P(E C) ( 1 + P(C) 1 P(C) ) P(E) 1 P(C) Example (to help with intuition) Consider C and E to be two fair coins, c 1 and c 2, being heads ; i.e., P(c 1 = heads ) = 0.5 and P(c 2 = heads ) = 0.5. If the coins are independent, then P(c 2 = heads c 1 = heads ) = P(c 2 = heads ) = 0.5 ρ EC = 0 If they are completely dependent then P(c 2 = heads c 1 = heads ) = 1 or 0 ρ EC = 1 or 1 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

95 Weighted mean observed leaning (g 4 ) Causal penchant The causal penchant ρ EC [1, 1] is ρ EC = P(E C) ( 1 + P(C) 1 P(C) ) P(E) 1 P(C) This formula has the additional benefit of only needing to estimate one conditional probability from the data. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

96 Weighted mean observed leaning (g 4 ) Causal leaning A difference of penchants can be used to compare different cause-effect assignments (i.e., different assumptions of what should be considered a cause and what should be considered an effect). The leaning is λ EC = ρ EC ρ CE J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

97 Weighted mean observed leaning (g 4 ) Causal leaning A difference of penchants can be used to compare different cause-effect assignments (i.e., different assumptions of what should be considered a cause and what should be considered an effect). The leaning is λ EC = ρ EC ρ CE Leaning interpretation If λ EC > 0, then C drives E more than E drives C. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

98 Weighted mean observed leaning (g 4 ) Usefulness of the leaning The usefulness of the leaning depends on two things, J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

99 Weighted mean observed leaning (g 4 ) Usefulness of the leaning The usefulness of the leaning depends on two things, 1. Operational definitions of C and E (called the cause-effect assignment) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

100 Weighted mean observed leaning (g 4 ) Usefulness of the leaning The usefulness of the leaning depends on two things, 1. Operational definitions of C and E (called the cause-effect assignment) 2. Estimations of P(C), P(E), P(C E), and P(E C) from the data J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

101 Weighted mean observed leaning (g 4 ) Usefulness of the leaning The usefulness of the leaning depends on two things, 1. Operational definitions of C and E (called the cause-effect assignment) 2. Estimations of P(C), P(E), P(C E), and P(E C) from the data The primary cause-effect assignment will be the l-standard assignment, l-standard assignment Consider a time series pair (X, Y). The l-standard assignment initially assumes the cause is the l lagged time step of X and the effect is the current time step of Y; i.e., {C, E} = {x t l, y t }. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

102 Weighted mean observed leaning (g 4 ) Usefulness of the leaning The usefulness of the leaning depends on two things, 1. Operational definitions of C and E (called the cause-effect assignment) 2. Estimations of P(C), P(E), P(C E), and P(E C) from the data The primary cause-effect assignment will be the l-standard assignment, l-standard assignment Consider a time series pair (X, Y). The l-standard assignment initially assumes the cause is the l lagged time step of X and the effect is the current time step of Y; i.e., {C, E} = {x t l, y t }. Probabilities will estimated using data frequency counts. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

103 Weighted mean observed leaning (g 4 ) Leaning from the data The cause-effect assignment must be specific if the probabilities are to be estimated with frequency counts and need to include tolerance domains to account for noise in the measurements. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

104 Weighted mean observed leaning (g 4 ) Leaning from the data Consider the time series pair (X, Y). The penchant calculation depends on the conditional P(y t = a x t l = b), where a Y and b X. This conditional will be estimated as P(y t [a δ L y, a + δ R y ] x t l [b δ L x, b + δ R x ]) = n a b n b J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

105 Weighted mean observed leaning (g 4 ) Leaning from the data Consider the time series pair (X, Y). The penchant calculation depends on the conditional P(y t = a x t l = b), where a Y and b X. This conditional will be estimated as P(y t [a δ L y, a + δ R y ] x t l [b δ L x, b + δ R x ]) = n a b n b n a b is the number of times y t [a δ L y, a + δ R y ] and x t l [b δ L x, b + δ R x ] in (X, Y) J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

106 Weighted mean observed leaning (g 4 ) Leaning from the data Consider the time series pair (X, Y). The penchant calculation depends on the conditional P(y t = a x t l = b), where a Y and b X. This conditional will be estimated as P(y t [a δ L y, a + δ R y ] x t l [b δ L x, b + δ R x ]) = n a b n b n b is the number of times x t l [b δ L x, b + δ R x ] in X J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

107 Weighted mean observed leaning (g 4 ) Leaning from the data Consider the time series pair (X, Y). The penchant calculation depends on the conditional P(y t = a x t l = b), where a Y and b X. This conditional will be estimated as P(y t [a δ L y, a + δ R y ] x t l [b δ L x, b + δ R x ]) = n a b n b The tolerance domains are usually considered symmetric; i.e., δx L = δx R δy L = δy R and J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

108 Weighted mean observed leaning (g 4 ) Leaning from the data Consider the time series pair (X, Y). The penchant calculation depends on the conditional P(y t = a x t l = b), where a Y and b X. This conditional will be estimated as P(y t [a δ L y, a + δ R y ] x t l [b δ L x, b + δ R x ]) = n a b n b The causal inference implied by the leaning calculations are dependent on both the cause-effect assignment and the tolerance domains. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

109 Weighted mean observed leaning (g 4 ) Weighted mean Any time series pair (X, Y) will have many leanings; e.g., an l-standard assignment of {C, E} = {x t l = b ± δ x, y t = a ± δ y } will have a different leaning calculation for each x t 1 [b δ x, b + δ x ] and y t [a δ y, a + δ y ]. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

110 Weighted mean observed leaning (g 4 ) Weighted mean Consider a time series pair (X, Y) and some cause-effect assignment {C, E} for which reasonable tolerance domains have been defined. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

111 Weighted mean observed leaning (g 4 ) Weighted mean Consider a time series pair (X, Y) and some cause-effect assignment {C, E} for which reasonable tolerance domains have been defined. Any penchant calculation for which the (estimated) conditional P(E C) 0 (or P(C E) 0) is called an observed penchant. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

112 Weighted mean observed leaning (g 4 ) Weighted mean Consider a time series pair (X, Y) and some cause-effect assignment {C, E} for which reasonable tolerance domains have been defined. Any penchant calculation for which the (estimated) conditional P(E C) 0 (or P(C E) 0) is called an observed penchant. The weighed mean observed penchant, ρ EC w, is the weighed algebraic mean of the observed penchants. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

113 Weighted mean observed leaning (g 4 ) Weighted mean Consider a time series pair (X, Y) and some cause-effect assignment {C, E} for which reasonable tolerance domains have been defined. Any penchant calculation for which the (estimated) conditional P(E C) 0 (or P(C E) 0) is called an observed penchant. The weighed mean observed penchant, ρ EC w, is the weighed algebraic mean of the observed penchants. The weighed mean observed leaning, λ EC w, is the difference of the weighed mean observed penchants; i.e., λ EC w = ρ EC w ρ CE w J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

114 Weighted mean observed leaning (g 4 ) Causal inference Operational causality (penchant) X causes Y if the weighted mean observed leaning is positive given a cause-effect assignment (and reasonable tolerance domains) in which the assumed cause X precedes the assumed effect Y; i.e., λ EC w > 0 X Y λ EC w < 0 Y X λ EC w = 0 no causal inference given C X, E Y, and C precedes E. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

115 Lagged cross-correlation difference (g 5 ) Cross-correlation The cross-correlation between two time series X and Y is ρ xy = E [(x t µ X ) (y t µ Y )] σx 2 σ2 Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

116 Lagged cross-correlation difference (g 5 ) Cross-correlation The cross-correlation between two time series X and Y is ) ] E [(x t µ X (y t µ Y ) ρ xy = σx 2 σ2 Y Every point in X is compared to the mean of X J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

117 Lagged cross-correlation difference (g 5 ) Cross-correlation The cross-correlation between two time series X and Y is [ )] E (x t µ X ) (y t µ Y ρ xy = σx 2 σ2 Y Every point in Y is compared to the mean of Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

118 Lagged cross-correlation difference (g 5 ) Cross-correlation The cross-correlation between two time series X and Y is ρ xy = E [(x t µ X ) (y t µ Y )] σ 2 X σ2 Y The product of the individual variances of X and Y is used as a normalization J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

119 Lagged cross-correlation difference (g 5 ) Cross-correlation The cross-correlation between two time series X and Y is ρ xy = E [(x t µ X ) (y t µ Y )] σx 2 σ2 Y Example (to help with intuition) X = Y ρ xy = E [(x t µ X ) (y t µ Y )] σx 2 σ2 Y = E [(x t µ X ) 2] σ 2 X = σ2 X σ 2 X = 1 J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

120 Lagged cross-correlation difference (g 5 ) Lagged cross-correlation Consider a time series pair (X, Y). The past of Y may be compared to the present of X by introducing a lag l into the cross-correlation calculation, l = E [(x t µ X ) (y t l µ Y )] ρ xy σ 2 X σ2 Y J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

121 Lagged cross-correlation difference (g 5 ) Lagged cross-correlation Consider a time series pair (X, Y). The past of Y may be compared to the present of X by introducing a lag l into the cross-correlation calculation, l = E [(x t µ X ) (y t l µ Y )] ρ xy σ 2 X σ2 Y Operational causality (correlation) X causes Y (at lag l) if the past of X (i.e., X lagged by l time steps) is more strongly correlated with the present of Y than the past of Y (i.e., Y lagged by l time steps) is with the present of X; i.e., ρ xy l ρ yx ρ yx ρ xy l ρ xy l l < 0 X Y l > 0 Y X ρ yx l = 0 no causal inference J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

122 Computational tools for the ECA summary Open source packages are available for some of the mentioned times series causality tools and others required code to be develop from scratch. g 1 g 2 g 3 g 4 g 5 Java Information Dynamics Toolkit (JIDT) Multivariate Granger Causality (MVGC) MATLAB toolbox (C++) (MATLAB) (MATLAB) All the code is available at J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

123 Cooling/Heating System Data Time series data Consider a time series pair (X, Y) where X are indoor temperature measurements (in degrees Celsius) in a house with experimental environmental controls and Y is the temperature outside of that house, measured at the same time intervals (168 measurements in each series) x t t y t t X Y 3 This data was originally presented at a time series conference. The abstract is available here, The data is also available as part of the UCI Machine Learning Repository. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50

124 Cooling/Heating System Data Time series data Consider a time series pair (X, Y) where X are indoor temperature measurements (in degrees Celsius) in a house with experimental environmental controls and Y is the temperature outside of that house, measured at the same time intervals (168 measurements in each series) x t t y t t X Y The intuitive causal inference is Y X. 3 This data was originally presented at a time series conference. The abstract is available here, The data is also available as part of the UCI Machine Learning Repository. J. M. McCracken (GMU) ECA w/ time series causality November 20, / 50