MATCHING FOR EE AND DR IMPACTS

Transcription

1 MATCHING FOR EE AND DR IMPACTS Seth Wayland, Opinion Dynamics August 12, 2015

2 A Proposal Always use matching Non-parametric preprocessing to reduce model dependence Decrease bias and variance Better understand your data EE, DR Quasi-Experiment Randomized Experiment IEPEC

3 Agenda Review current best practice for impact evaluation Review some matching methods Matching example IEPEC

4 Impact Estimation Best Practice RCT + Model (to reduce bias and variance) Quasi-experiment + Matching + Model IEPEC

5 Methods Model Difference-in-Difference Linear Fixed Effects Lagged Dependent Variable Match Propensity Score Mahalanobis Distance Coarsened Exact Matching Frontier IEPEC

6 Feeling Lucky? Randomized experiments are guaranteed to be unbiased over repeated experiments There is only one actual experiment How sure can we be that this one is unbiased? Check the balance of treatment versus control What can we do? Match to reduce imbalance Model to correct for dependence on known and (fixed) unknown covariates IEPEC

7 Applying the Rubin Causal Model For a particular unit, the causal effect of a treatment at time t is the difference between what would have happened at time t if the unit was exposed to the treatment and what would have happened at time t if the unit was not exposed to the treatment. IEPEC

8 Applying the Rubin Causal Model The customer cannot be simultaneously exposed to the treatment and not exposed to the treatment We need to make some assumptions SUTVA Ignorable treatment assignment IEPEC

9 Ignorable treatment assignment Model Parametrically adjust for the effect of covariates Match Non-parametrically improve balance of all included covariates Both also usually reduce variance Matching yields insight into the data IEPEC

10 Matching Procedure 1. Select a distance measure 2. Select and implement a matching method 3. Assess balance, return to 1 or 2 as necessary 4. Use the matched data to perform analysis IEPEC

11 Matching Procedure - Considerations Choice of treatment effect (ITT, ATE, ATT, SATT, FSATT, etc.) Choice of variables to include in matching Choice of matching method Choice of model in distance metric for Propensity Score matching Choice of balance checks IEPEC

12 Example Home energy report program with an RCT design IEPEC

13 Matching Methods Exact K nearest neighbors Coarsened Exact Matching Frontier Many others IEPEC

14 Balance Checks Difference in Means Check all variables (don t use statistical significance) Average Mahalanobis Imbalance Mean Mahalanobis distance between all matched pairs Median L1 Distance Distance between multivariate histograms IEPEC

15 When Matching Doesn t Help Coincident non-treatment changes Some whole-house programs Missing information about treatment assignment Opt-in bias? Modeling doesn t help either IEPEC

16 Coarsened Exact N = 9,408, Nc = 9,355 Median L1 distance: 0.09 Much better Average mean distance: 0.37 kwh/day Somewhat worse IEPEC

17 Coarsened Exact Feasible Group Non-Feasible Group IEPEC

18 Coarsened Exact FSATT (N f =9,408, N c =9,355) savings = 4.3% NFSATT (N nf =592, N c =644) savings = 9.6% Weighted SATT (N=N c =10,000) savings = 4.6% Full Sample SATT (N=N c =10,000) savings = 4.8% weighted SATT = FSATT N f + NFSATT N nf N IEPEC

19 A Second Proposal How do we evaluate what are the best methods/approaches for impact evaluation? We need published data and well-defined metrics Common Task Method Everyone works on the same problem Method Publish data Define evaluation metrics Periodic public evaluation of methods IEPEC

20 For More Information Seth Wayland, Associate Director Opinion Dynamics IEPEC

21 Thank you xkcd.com/925 IEPEC

22 Distance Metrics Exact Propensity Score Mahalanobis Euclidian is a special case IEPEC

23 Match Anyway Methods K nearest neighbors (1:1) with SATT Propensity score distance Mahalanobis distance Coarsened Exact with weighted SATT L1 distance IEPEC

24 Balance Metrics Treated group N = 10,000 Comparison group N c = 10,000 Average mean difference for the 12 months of the pre-period: 0.03 kwh/day Median L1 distance: 0.56 IEPEC

25 K Nearest Neighbors Propensity score metric Simple model with a variable for each month of pre-period usage N = 10,000 and N c = 5,580 Average mean difference: kwh/day Balance is a little worse IEPEC

26 K Nearest Neighbors Mahalanobis distance N = 10,000 and N c = 5,762 Average mean difference: 2.9 Balance is much worse IEPEC