External validity, causal interaction and randomised trials

Similar documents
Randomized trials for policy

12E016. Econometric Methods II 6 ECTS. Overview and Objectives

Comments on The Role of Large Scale Assessments in Research on Educational Effectiveness and School Development by Eckhard Klieme, Ph.D.

A Distinction between Causal Effects in Structural and Rubin Causal Models

Tools for causal analysis and philosophical theories of causation. Isabelle Drouet (IHPST)

From Causality, Second edition, Contents

CAUSALITY. Models, Reasoning, and Inference 1 CAMBRIDGE UNIVERSITY PRESS. Judea Pearl. University of California, Los Angeles

Online Appendix to Yes, But What s the Mechanism? (Don t Expect an Easy Answer) John G. Bullock, Donald P. Green, and Shang E. Ha

ESTIMATION OF TREATMENT EFFECTS VIA MATCHING

The Econometrics of Randomized Experiments

Applied Microeconometrics Chapter 8 Regression Discontinuity (RD)

Causal Inference Lecture Notes: Causal Inference with Repeated Measures in Observational Studies

Notes on causal effects

When Should We Use Linear Fixed Effects Regression Models for Causal Inference with Longitudinal Data?

On the Use of Linear Fixed Effects Regression Models for Causal Inference

Causality in Econometrics (3)

The Economics of European Regions: Theory, Empirics, and Policy

IsoLATEing: Identifying Heterogeneous Effects of Multiple Treatments

Technical Track Session I:

IDENTIFICATION OF TREATMENT EFFECTS WITH SELECTIVE PARTICIPATION IN A RANDOMIZED TRIAL

ECON Introductory Econometrics. Lecture 17: Experiments

Bioinformatics: Biology X

Quantitative Economics for the Evaluation of the European Policy

Intervention, determinism, and the causal minimality condition

MINIMAL SUFFICIENT CAUSATION AND DIRECTED ACYCLIC GRAPHS 1. By Tyler J. VanderWeele and James M. Robins. University of Chicago and Harvard University

Measurement Independence, Parameter Independence and Non-locality

When Should We Use Linear Fixed Effects Regression Models for Causal Inference with Longitudinal Data?

The decision theoretic approach to causal inference OR Rethinking the paradigms of causal modelling

EMERGING MARKETS - Lecture 2: Methodology refresher

Chapter Course notes. Experiments and Quasi-Experiments. Michael Ash CPPA. Main point of experiments: convincing test of how X affects Y.

Causal inference. Gary Goertz Kroc Institute for International Peace Studies University of Notre Dame Spring 2018

An Alternative Assumption to Identify LATE in Regression Discontinuity Designs

When Should We Use Linear Fixed Effects Regression Models for Causal Inference with Panel Data?

Causal Realism and Causal Analysis in the Social Sciences. Presentation by Daniel Little University of Michigan-Dearborn

Controlling for latent confounding by confirmatory factor analysis (CFA) Blinded Blinded

Comments on Best Quasi- Experimental Practice

Econometrics of causal inference. Throughout, we consider the simplest case of a linear outcome equation, and homogeneous

Modeling Mediation: Causes, Markers, and Mechanisms

Causality II: How does causal inference fit into public health and what it is the role of statistics?

Of Nickell Bias, Cross-Sectional Dependence, and Their Cures: Reply

A proof of Bell s inequality in quantum mechanics using causal interactions

Appendix A Lewis s Counterfactuals

Structure learning in human causal induction

Treatment Effects. Christopher Taber. September 6, Department of Economics University of Wisconsin-Madison

A Measure of Robustness to Misspecification

Cartwright on Causality: Methods, Metaphysics, and Modularity. Daniel Steel

The Role of Social Comparison in Reducing Residential Water Consumption: Evidence from a Randomized Controlled Trial

Inferring Causal Structure: a Quantum Advantage

CEPA Working Paper No

Selection on Observables: Propensity Score Matching.

Empirical Methods in Applied Microeconomics

Probabilistic Theories of Causality: Singular or Plural?

Research Note: A more powerful test statistic for reasoning about interference between units

Econometrics, Harmless and Otherwise

Introduction to Econometrics

PSC 504: Instrumental Variables

PROBLEMS OF CAUSAL ANALYSIS IN THE SOCIAL SCIENCES

Linear Regression with one Regressor

Research Design: Causal inference and counterfactuals

Technical Track Session I: Causal Inference

Social Science Counterfactuals. Julian Reiss, Durham University

Causal model search applied to economics: gains, pitfalls and challenges

AN EVALUATION OF PARAMETRIC AND NONPARAMETRIC VARIANCE ESTIMATORS IN COMPLETELY RANDOMIZED EXPERIMENTS. Stanley A. Lubanski. and. Peter M.

Empirically Adequate but Observably False Theories

Conceivability and Modal Knowledge

NBER WORKING PAPER SERIES A NOTE ON ADAPTING PROPENSITY SCORE MATCHING AND SELECTION MODELS TO CHOICE BASED SAMPLES. James J. Heckman Petra E.

MINIMAL SUFFICIENT CAUSATION AND DIRECTED ACYCLIC GRAPHS. By Tyler J. VanderWeele and James M. Robins University of Chicago and Harvard University

Program Evaluation and Research Designs

Independent and conditionally independent counterfactual distributions

Introduction to Statistical Inference

Introduction to Econometrics

Trygve Haavelmo and the Emergence of Causal Calculus

UNIVERSITY OF STELLENBOSCH Optional course in Advanced Econometrics: Cross section 2016

Regression Discontinuity Designs

Rigorous Science - Based on a probability value? The linkage between Popperian science and statistical analysis

The Econometric Evaluation of Policy Design: Part I: Heterogeneity in Program Impacts, Modeling Self-Selection, and Parameters of Interest

On Likelihoodism and Intelligent Design

ε X ε Y Z 1 f 2 g 2 Z 2 f 1 g 1

Introduction to Econometrics. Review of Probability & Statistics

Experiments and Quasi-Experiments

Potential Outcomes Model (POM)

Rigorous Science - Based on a probability value? The linkage between Popperian science and statistical analysis

Empirical approaches in public economics

Causal analysis after Haavelmo. 8th/last Lecture - Hedibert Lopes

An Alternative Assumption to Identify LATE in Regression Discontinuity Design

Teaching Causal Inference in Undergraduate Econometrics

Counterfactual Undoing in Deterministic Causal Reasoning

QCA: Strengths, Weaknesses, Policy Relevance

Covariate selection and propensity score specification in causal inference

UCLA Department of Statistics Papers

A Note on Adapting Propensity Score Matching and Selection Models to Choice Based Samples

Euler s Galilean Philosophy of Science

An Introduction to Causal Mediation Analysis. Xu Qin University of Chicago Presented at the Central Iowa R User Group Meetup Aug 10, 2016

Econometrics of Policy Evaluation (Geneva summer school)

Class 19. Daniel B. Rowe, Ph.D. Department of Mathematics, Statistics, and Computer Science. Marquette University MATH 1700

LECTURE 2: SIMPLE REGRESSION I

Single World Intervention Graphs (SWIGs): A Unification of the Counterfactual and Graphical Approaches to Causality

Introduction to Econometrics. Assessing Studies Based on Multiple Regression

Dover- Sherborn High School Mathematics Curriculum Probability and Statistics

The returns to schooling, ability bias, and regression

Transcription:

External validity, causal interaction and randomised trials Seán M. Muller University of Cape Town Evidence and Causality in the Sciences Conference University of Kent (Canterbury) 5 September 2012

Overview 1 Motivation: RCTs in economics

Overview 1 Motivation: RCTs in economics 2 From necessity and sufficiency to causal interaction 3 Causal interaction and the failure of external validity 4 Reduction in class size and educational outcomes 5

Caveats and notes Part of broader project: 1. Present work on RCTs; 2. Necessary & sufficient causes, econometrics & causal graphs; 3. Revisiting Suppes (1970) quantitative causality Economist interested in philosophy rather than philosopher interested in economics Rejection does not imply acceptance

From INUS causality to causal interaction

From INUS causality to causal interaction Two key concepts:

From INUS causality to causal interaction Two key concepts: INUS condition (Mackie, 1974): an insufficient but neccessary component of an unnecessary but sufficient condition

From INUS causality to causal interaction Two key concepts: INUS condition (Mackie, 1974): an insufficient but neccessary component of an unnecessary but sufficient condition Causal interaction: Given y = f (x, z) + u, no causal interaction if f (x, z) f (x, z) = g(x, x ) z (and vice versa) (Cartwright, 1989: 164)

From INUS causality to causal interaction Two key concepts: INUS condition (Mackie, 1974): an insufficient but neccessary component of an unnecessary but sufficient condition Causal interaction: Given y = f (x, z) + u, no causal interaction if f (x, z) f (x, z) = g(x, x ) z (and vice versa) (Cartwright, 1989: 164) Claim 1: INUS causality implies causal interaction

From INUS causality to causal interaction

From INUS causality to causal interaction Consider functional representation of a complex of INUS causes: y = x 0.x 1... x n + u (1)

From INUS causality to causal interaction Consider functional representation of a complex of INUS causes: y = x 0.x 1... x n + u (1) Then: (y x j = 1) (y x j = 0) =? (2)

From INUS causality to causal interaction Consider functional representation of a complex of INUS causes: y = x 0.x 1... x n + u (1) Then: (y x j = 1) (y x j = 0) =? (2) NB: I am not proposing some generic, statistical method of identifying causes (a la Spirtes et al. (1993/2000), Pearl (2000/2009) or even some philosophers)

From INUS causality to causal interaction Consider functional representation of a complex of INUS causes: y = x 0.x 1... x n + u (1) Then: (y x j = 1) (y x j = 0) =? (2) NB: I am not proposing some generic, statistical method of identifying causes (a la Spirtes et al. (1993/2000), Pearl (2000/2009) or even some philosophers) Claim 2: Interaction leads to failure of external validity

(Counterfactual notation of RCT literature: Y 1i and Y 0i for T = 1 and T = 0)

(Counterfactual notation of RCT literature: Y 1i and Y 0i for T = 1 and T = 0) Interest is in E[Y 1i Y 0i ]

(Counterfactual notation of RCT literature: Y 1i and Y 0i for T = 1 and T = 0) Interest is in E[Y 1i Y 0i ] Definition of external validity given binary location variable D E[Y i (1) Y i (0) D i = 1] = E[Y i (1) Y i (0) D i = 0] (3) Hotz et al. (2005)

(Counterfactual notation of RCT literature: Y 1i and Y 0i for T = 1 and T = 0) Interest is in E[Y 1i Y 0i ] Definition of external validity given binary location variable D E[Y i (1) Y i (0) D i = 1] = E[Y i (1) Y i (0) D i = 0] (3) Hotz et al. (2005) If T interacts with D the result follows straightforwardly. Statistical logic known since at least Cook and Campbell (1979: 70-74): external validity in essence [an issue of] statistical interaction

(Counterfactual notation of RCT literature: Y 1i and Y 0i for T = 1 and T = 0) Interest is in E[Y 1i Y 0i ] Definition of external validity given binary location variable D E[Y i (1) Y i (0) D i = 1] = E[Y i (1) Y i (0) D i = 0] (3) Hotz et al. (2005) If T interacts with D the result follows straightforwardly. Statistical logic known since at least Cook and Campbell (1979: 70-74): external validity in essence [an issue of] statistical interaction But also in the philosophy literature: Probably the most common reason for a capacity to fail to obtain in the new situation is causal interaction (Cartwright, 1989: 163)

Back to RCTs (in brief) Interaction entirely neglected in standard guides to practice (e.g. Angrist and Pischke (2009) and Duflo, Glennerster, and Kremer (2006)). Some allusion to the issue by critics - notably Ravallion (2008), Leamer (2010) and Keane (2010). Key difference: above authors conflate issues of selection, models of agent behaviour and interaction. Reflects an agnosticism, if not outright denial, in economics about functional form

Back to RCTs (in brief) Interaction entirely neglected in standard guides to practice (e.g. Angrist and Pischke (2009) and Duflo et al. (2006)). Some allusion to the issue by critics - notably Ravallion (2008), Leamer (2010) and Keane (2010). Key difference: above authors conflate issues of selection, models of agent behaviour and interaction. Reflects an agnosticism, if not outright denial, in economics about functional form Claim 3: Interaction - external validity issue not accounted for in current literature

Class size and the credibility revolution

Class size and the credibility revolution Angrist and Pischke (2010) argue that while extrapolation of causal effects to new settings is always speculative, estimated treatment effects may have external validity.

Class size and the credibility revolution Angrist and Pischke (2010) argue that while extrapolation of causal effects to new settings is always speculative, estimated treatment effects may have external validity. Specifically cite the case of class size: Across [the four cited] studies a ten-student reduction in class size produces about a 0.2 to 0.3 standard deviation increase in individual test scores (Angrist and Pischke, 2010: 24)

Class size and the credibility revolution

Class size and the credibility revolution I suggest the conceptual flaws here are intuitive:

Class size and the credibility revolution I suggest the conceptual flaws here are intuitive: 1 It makes no sense to think of class size as primarily an independent causal factor: class size matters because of what happens in the classroom.

Class size and the credibility revolution I suggest the conceptual flaws here are intuitive: 1 It makes no sense to think of class size as primarily an independent causal factor: class size matters because of what happens in the classroom. 2 Why would standard deviations in test scores relative to specific level changes be the correct measure?

Sibling (micro)econometrics paper(s) Focus on specific interaction: teacher quality and class size. Large, but orthogonal, literatures on both. Using existing data - Project STAR (RCT in Tennessee in 1980s) - but this presents various technical complications.. Two other issues emerge naturally from less simplistic specification of causal relationship: student absence and teacher absence (also the subject of various studies)

The critical contribution of the philosophy is to give us a reason to believe in interactive causal structures, implying:

The critical contribution of the philosophy is to give us a reason to believe in interactive causal structures, implying: 1 Failure of external validity as the null hypothesis

The critical contribution of the philosophy is to give us a reason to believe in interactive causal structures, implying: 1 Failure of external validity as the null hypothesis 2 A limited role for RCTs (as argued for other reasons by Cartwright, Heckman and others)

The critical contribution of the philosophy is to give us a reason to believe in interactive causal structures, implying: 1 Failure of external validity as the null hypothesis 2 A limited role for RCTs (as argued for other reasons by Cartwright, Heckman and others) 3 Little justification for the more experiments mantra; if you don t know/measure the interactive factors what basis is there for terminating the replication process?

Bibliography I Angrist, J. D., Pischke, J.-S., 2009. Mostly harmless econometrics. Princeton University Press, Princeton. Angrist, J. D., Pischke, J.-S., 2010. The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives 24 (2), 3 30. Cartwright, N., 1989. Nature s Capacities and their Measurement. Oxford University Press, Oxford. Cook, T. D., Campbell, D. T., 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Wadsworth. Duflo, E., Glennerster, R., Kremer, M., 2006. Using randomization in development economics research: A toolkitaccessed 24th January 2011 from http://www.povertyactionlab.org/methodology.

Bibliography II Hotz, V. J., Imbens, G. W., Mortimer, J. H., 2005. Predicting the efficacy of future training programs using past experiences at other locations. Journal of Econometrics 125, 241 270. Keane, M. P., 2010. A structural perspective on the experimentalist school. Journal of Economic Perspectives 24 (2), 47 58. Leamer, E., 2010. Tantalus on the road to asymptopia. Journal of Economic Perspectives 24 (2), 31 46. Mackie, J. L., 1974. The cement of the universe: A study of causation. OUP, Oxford. Pearl, J., 2000/2009. Causality: Models,reasoning and inference. Cambridge University Press, Cambridge. Ravallion, M., March 2008. Evaluation in the practice of development. World Bank Policy Research Working Paper 4547.

Bibliography III Spirtes, P., Glymour, C. N., Scheines, R., 1993/2000. Causation, prediction and search. MIT Press, Cambridge(MA). Suppes, P., 1970. A probabilistic theory of causality. North Holland Publishing Company, Amsterdam.

How is this different from the preceding literature? Link between causal interaction and external validity is not made explicit (at least not that I have seen). Economist critics focus on how various maximising behaviours of economic agents might compromise inference from RCTs (through selection into samples, compensating behaviour, etc), but little independent justification for particular models. Not an attempt to construct a positive general theory for hunting causes Present work: simple, plausible ontological assumptions about the nature of causal structures are enough to call into question the methodology of RCTs in economics. I do not commit myself to more than this.

2024 © sciencedocbox.com Privacy Policy | Terms of Service | Feedback