Online Appendix for Targeting Policies: Multiple Testing and Distributional Treatment Effects

Transcription

1 Online Appendix for Targeting Policies: Multiple Testing and Distributional Treatment Effects Steven F Lehrer Queen s University, NYU Shanghai, and NBER R Vincent Pohl University of Georgia November 2016 Kyungchul Song University of British Columbia Abstract This is the online appendix for Lehrer, Pohl, and Song (2016) Three sections are included, which provide further details regarding (1) the asymptotic validity of the multiple testing procedure, (2) an additional test for treatment effect heterogeneity between subgroups, and (3) results from empirical analyses using alternative strategies to test the hypotheses of interest School of Policy Studies and Economics Department, lehrers@queensuca Department of Economics, Terry School of Business, pohl@ugaedu Vancouver School of Economics, kysong@mailubcca

2 A Supplemental Note on the Asymptotic Validity of Multiple Testing Procedures In this section, we provide a sketch of the asymptotic validity of the multiple testing procedures Since the proofs are fairly standard given the existing econometric literature of nonparametric and semiparametric models, we omit the details and instead provide relevant references to the literature For simplicity, let us first consider the case of treatment effect heterogeneity without subgroups Let us consider the case where T is finite, say, T tτ 1,, τ M u Define q rq τ1,, q τm s 1 and similarly, we let ˆq rˆq τ1,, ˆq τm s 1 and ˆq rˆq τ 1,, ˆq τ M s 1 For the bootstrap validity of testing in 411, it suffices to show the following two statements:? npˆq q q Ñ d Np0, V q, (A1) for some covariance matrix V, and for each t P R M, P t? npˆq ˆq q ď t X u Ñ P Φ M ptq, (A2) where Φ M ptq is the CDF of Np0, V q and X py i, D i, X i q n i 1 Then bootstrap validity of the test based on the test statistic T n in equation (3) in Lehrer, Pohl, and Song (2016) (henceforth, LPS) immediately follows by the continuous mapping theorem As for statement (A1), when the propensity score is estimated nonparametrically as in Hirano, Imbens, and Ridder (2003), we can use Theorem 1 of Firpo (2007) to derive for each τ P T,? npˆq τ q τ q 1? n nÿ i 1 ψ τ py i, D i, X i q ` o P p1q, (A3) where ψ τ is as given in Section 33 of Firpo (2007) 1 Then the result stated in (A1) immediately follows from the asymptotic linear representation of? npˆq τ qτ q in (A3) with covariance matrix V given by V ErψpY i, D i, X i qψpy i, D i, X i q 1 s, where ψ rψ τ1,, ψ τm s Since we do not use asymptotic critical values, the precise form of ψ τ is not used in the computation of critical values 2 Even if the propensity score is estimated from a parametric model such as a logit regression, we still have the asymptotic linear representation but with a different form of the influence function ψ Dealing with estimation error in the first step propensity score has been well known and studied extensively in the literature of generated regressors See for example Hahn and Ridder (2013), Song (2014), and Mammen, Rothe, and Schienle (2016) and references therein 2

3 The result stated in (A2) can be obtained by first establishing the bootstrap version of the asymptotic linear representation in (A3):? npˆq ˆq q 1? n nÿ i 1 ψ τ py i, D i, X i q ` o P p1q, (A4) where o P p1q is a term whose conditional distribution given X converges in probability to zero For example, the derivation of this bootstrap version of asymptotic linear representation can be done in a similar way as in the proof of Theorem 2 of Lee, Song, and Whang (2015) The proof in our case is simpler because T is a finite set, but the algebra can be more tedious because we have an estimated propensity score The asymptotic FWER control of our bootstrap multiple testing procedure can be shown as follows First, for any subset S Ă T, let us write a subvector q S to be a vector of q τ s with τ running in the set S Similarly, we define ˆq S arguments, we can show that for each subset S Ă T, and ˆq S Then using precisely the same? npˆq S q S q Ñ d Np0, V S q, (A5) and for each t P R M, where Φ S ptq denotes the CDF of Np0, V S q, and P t? npˆq S ˆq S q ď t X u Ñ P Φ S ptq, (A6) V S Erψ S py i, D i, X i qψ S py i, D i, X i q 1 s, where ψ S denotes the vector of ψ τ s with τ running in S Now define PpSq to be the class of probabilities P under which qs ď 0 and q T zs ą 0 By Theorem 21 of Romano and Shaikh (2010), for any subset S Ă T and any P P PpSq, the FWER at such P of the set T obtained through the step-down procedure is bounded by the rejection probability of the bootstrap test based on the test statistic T n psq max τps ˆq τ (after the re-centering in the bootstrap test statistic) Further, by the results stated in (A5) and (A6), it follows that the latter rejection probability is asymptotically bounded by α Hence we find that the FWER of our step-down procedure is asymptotically controlled by the nominal level α Finally, consider the tests for treatment effect heterogeneity presented in LPS, Section 412 Similar arguments can be made to the sketches presented above where the only difference is that when constructing a test statistic, we use a map f : R M Ñ R defined by fpxq max τpt x τ x, where x px τ q τpt and x 1 ř T τpt x τ, instead of the max function This new map is a continuous function, so that the same arguments based on the 3

4 continuous mapping theorem carry over to this case Further, the presence of subgroups in Section 42 does not pose difficulty in extending the same arguments to this case, because we can obtain the asymptotic linear representations similar to equations (A3) and (A4) using precisely the same arguments Extending the results to the case of T being an infinite, potentially uncountable set, requires substantially more work as it requires the use of empirical process theory Such an extension is feasible using similar arguments to those presented in Chernozhukov, Fernandez- Val, and Melly (2013) (henceforth, CFM) Since the length and the tediousness of the arguments dominate the innovativeness of the main idea in the proofs, and the practical implementation of the procedure remains the same (as we use a finite set T in practice), we omit these details B Additional Test for Treatment Effect Heterogeneity Between Subgroups In Section 422 of LPS, we test for treatment effect heterogeneity across quantiles and between subgroups Rejecting the null hypothesis (H5) implies that treatment effect heterogeneity across quantiles arises in some subgroups However, we may also be interested in testing if there is treatment effect heterogeneity in all subgroups To this end, we test the following hypothesis: H 0 : q τ pzq c z for all τ P T, for some c z P R, and some z P Z H 1 : q τ pzq c z for some τ P T, for all c z P R, and all z P Z (H5a) This test proceeds analogously to the test described in Section 422 with the test statistic replaced by T n min max zpz τpt and the bootstrap test statistic replaced by T n,b min max zpz τpt ˇ ˇˆq τ pzq q pzqˇˇ (B1) ˇ ˇˆq τ pzq ˆq τ pzq ` q pzq q pzq ˇˇ (B2) We reject the null hypothesis (H5a) if the test statistic (B1) exceeds the p1 αq-th quantile of the bootstrap test statistics (B2) Table B1 presents the test results for the same subgroups as in LPS We reject the null hypothesis (H5a) for all subgroup categories Hence, we find evidence that there is treatment effect heterogeneity in all subgroups for all categories of subgroups 4

5 Table B1: Testing For Treatment Effect Heterogeneity Between Subgroups Subgroup category Test statistic Critical value p-value Education Marital status Age of youngest child Number of children Earnings in quarter 7 pre-treatment Welfare receipt in quarter 7 pre-treatment Share of quarters with positive earnings Share of quarters on welfare Notes: This table shows test results for hypothesis (H5a), ie these tests show for which subgroups categories we can reject treatment effects that are homogenous within subgroups for some subgroups C Alternative Testing Results In this section, we provide test results using alternative methods to test our hypotheses (H1) to (H3), ie for the full sample C1 Using Abadie s (2002) Method to Test (H1) and (H2) Abadie (2002) develops a method to test distributional hypotheses, such as equality of distributions, using Kolmogorov-Smirnov statistics To test hypotheses (H1) and (H2) we use the same test statistics as in the paper (see test statistics in equations (3) and (5), respectively, in the paper) We use a bootstrap procedure, where we resample n observations with replacement and assign the first n 1 observations to the treatment (Jobs First) group and the remaining n 0 n n 1 observations to the control (AFDC) group, where n 1 and n 0 are the numbers of treatment and control group individuals, respectively, in the original sample We then calculate the critical value as the p1 αq-th quantile of the bootstrap test statistics and the associated p-value as the fraction of bootstrap test statistics that exceed the sample test statistic (also see Bitler, Gelbach, and Hoynes, 2006, Appendix B) Table C1 contains the results We obtain similar results to our tests of (H1) and (H2) and reject both null hypotheses using Abadie s (2002) as well 5

6 Table C1: Alternative Tests for Presence of Positive QTEs and QTE Heterogeneity Without Subgroups Using Abadie s (2002) Method Test statistic Critical value p-value Test of (H1) Test of (H2) Notes: This table shows test results for hypotheses (H1) and (H2) of LPS using Abadie s (2002) testing procedure, ie we test that there is no positive treatment effect for all quantiles and that the treatment effect is the same for all quantiles, respectively C2 Using Bonferroni and Holm Corrections to Test (H3) The simplest way to keep the family-wise error rate (FWER) at the desired level when testing multiple hypotheses is by using the correction initially proposed by Bonferroni (1936) and formalized by Dunn (1959a,b, 1961) This correction compares each p-value to the desired level of the test divided by the number of hypotheses being tested Hence, in our case, we reject hypothesis (H3) for quantile τ if ˆp τ ď α T, where T 97 is the number of quantiles and ˆp τ denotes the pointwise bootstrap p-value for quantile τ This p-value is defined as ˆp τ 1 B Bÿ b 1 1 ˆq τ ď ˆq τ,b (, (C1) where ˆq τ,b is the QTE for quantile τ from the b-th bootstrap replication While the Bonferroni controls the FWER at α, it is a very conservative testing procedure that my lead to a large type II error Based on this disadvantage, Holm (1979) proposed a stepdown procedure that compares p-values to increasingly larger constants The p-values are sorted from smallest to largest: ˆp p1q ď ď ˆp p T q and each p-value ˆp piq is compared to the constant c i α T i ` 1 (C2) 6

7 Then we reject the null hypothesis of no positive treatment effect at quantiles, for which ˆp piq ď c i Again, it has been shown that this procedure controls the FWER at α, but in comparison to the Bonferroni correction it leads to a smaller type II error Table C2 shows the results from using either Bonferroni or Holm corrections applied to the estimated QTEs for the full sample correspond to Figure 2 in the paper The QTE and pointwise confidence intervals The pointwise p-values are calculated according to equation (C1) based on B 9999 bootstrap replications (Note that the p-values refer to the one-sided tests in (H3)) The two columns Bonferroni and Holm display the results for these two multiple testing procedure, respectively, where a 1 indicates that we reject the null hypothesis for the corresponding quantile and 0 means that we fail to reject The last column shows the test results for our test of (H3) in column LPS 3 for comparison purposes A test of no positive treatment effects that does not adjust for multiple testing rejects the null hypothesis for the 48th to the 80th quantile (excluding the 50th quantile) Applying the Bonferroni or Holm correction limits the range of quantiles with statistically significant treatment effects as expected This result also highlights the need to adjust for multiple testing when doing inference on distributional treatment effects Comparing the results for the three multiple testing procedure, we make two observations First, the Bonferroni and Holm corrections reject for the same set of quantiles, so it appears that the Bonferroni procedure is not more conservative than the Holm correction finding is due to the fact that we have 23 quantiles where the p-value is exactly equal to zero because of the bootstrap (That is, out of the B 9999 bootstrap sample, no bootstrap QTE exceeded the sample QTE) The 24th smallest p-value equals 0001, which is already α larger than c 24 T 24` , the critical value according to equation (C2) Second, the large number of quantiles with p-values of zero also leads to our test of hypothesis (H3) to be more conservative than either the Bonferroni or the Holm correction This result does not hold in general however Comparing Bonferroni and Holm to the results of our tests also shows the advantage of using a stepdown method that iteratively rejects null hypotheses updating the critical value at each iteration (see Section 413 in the paper for a description of our algorithm) instead of setting critical values ex ante This more flexible approach leads to rejecting null hypotheses when the pointwise p-value equals zero (quantiles 51, 53, 54, 77, 79, and 80) Instead of attributing a positive treatment effect to, for example, the 80th quantile, our testing procedure recognizes that this result is likely due to random variation in the data Therefore, our procedure is better suited to guide policymakers than existing algorithms such as Bonferroni and Holm corrections This 7

8 C3 Chernozhukov, Fernandez-Val, and Melly s (2013) Approach for Inference on QTEs Our multiple testing procedures correct for dependency across individual test statistics, and hence lead to the exclusion of quantiles that could be mistakenly included in the estimated set of quantiles with positive QTEs when individual tests are used without correction Our approach is based on the inverse propensity score weighting (IPSW) method One might wonder whether our multiple testing results using the Jobs First data can be replicated or improved upon using individual tests based on an alternative method If that is true, then the choice of a proper inference method matters more empirically than using proper correction for multiple comparisons To investigate this, we use CFM as an alternative method, and compare our multiple testing results with the individual tests based on their approach CFM propose an alternative approach of making inference on counterfactual distributions based on conditional distribution functions and conditional quantile functions instead of using IPSW Specifically, CFM develop procedures for estimation and inference for counterfactual distributions that allows them to analyze how the distribution of an outcome variable changes under different policies These counterfactual distributions can be used to calculate QTEs Here we compare the results from our multiple testing procedure without subgroups (see LPS, Section 423) with the confidence sets based on CFM In our application, we use the CFM method to estimate QTEs and test the null hypothesis of no positive effect, which corresponds to our (H3) 3 Results from this test are found in the last two columns of Table C2 In particular, the last column indicates quantiles which exhibit significantly positive treatment effects We note that the range of positive QTEs is larger than for our test of (H3), ranging from the 46th to the 77th quantiles This result once again confirms that detecting quantiles with positive treatment effects using individual tests at each quantile can be misleading unless one corrects for dependency of the test statistics across the quantiles Indeed, the rejection of the null hypothesis of no positive treatment effect from the 46th to the 54th quantiles and from the 71st to the 78th quantiles in the individual tests of CFM cannot be used to claim the discovery of the positive treatment effect in those ranges of quantiles, as our multiple testing result shows 3 We implement CFM s method using the Stata command provided at The computation time of CFM exceeds our method since CFM estimates conditional distributions and we include a large number of covariates, whereas our method estimates unconditional QTEs and covariates enter through propensity score weights 8

9 Table C2: Alternative Methods For Testing For Which Quantiles the Treatment Effect Is Positive Pointwise QTE τ QTE p-value Bonferroni Holm LPS 3 (CFM) CFM Continued on next page 9

10 Continued from previous page Pointwise QTE τ QTE p-value Bonferroni Holm LPS 3 (CFM) CFM Notes: This table displays results for positive QTEs using various testing procedures The pointwise p-value if for the one-sided test that the corresponding QTE is non-positive Columns Bonferroni, Holm, LPS 3, and CFM show a 1 if we reject the null hypothesis of a non-positive QTE using the respective testing procedure (see text for details) The second to last column displays estimated QTEs using the method proposed by Chernozhukov, Fernandez-Val, and Melly (2013) 10

11 References Abadie, Alberto 2002 Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models Journal of the American Statistical Association 97 (457): Bitler, Marianne P, Jonah B Gelbach, and Hilary W Hoynes 2006 What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments American Economic Review 96 (4): Bonferroni, Carlo Emilio 1936 Teoria Statistica Delle Classi E Calcolo Delle Probabilità Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze Chernozhukov, Victor, Ivan Fernandez-Val, and Blaise Melly 2013 Inference on Counterfactual Distributions Econometrica 81 (6): Dunn, Olive Jean 1959a Confidence Intervals for the Means of Dependent, Normally Distributed Variables Journal of the American Statistical Association 54 (287): b Estimation of the Medians for Dependent Variables Annals of Mathematical Statistics 30 (1): Multiple Comparisons Among Means Journal of the American Statistical Association 56 (293):52 64 Firpo, Sergio 2007 Efficient Semiparametric Estimation of Quantile Treatment Effects Econometrica 75 (1): Hahn, Jinyong and Geert Ridder 2013 Asymptotic Variance of Semiparametric Estimators With Generated Regressors Econometrica 81 (1): Hirano, Keisuke, Guido W Imbens, and Geert Ridder 2003 Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score Econometrica 71: Holm, Sture 1979 A Simple Sequentially Rejective Multiple Test Procedure Scandinavian Journal of Statistics 6 (2):65 70 Lee, Sokbae, Kyungchul Song, and Yoon-Jae Whang 2015 Testing for a General Class of Functional Inequalities Lehrer, Steven F, R Vincent Pohl, and Kyungchul Song 2016 Targeting Policies: Multiple Testing and Distributional Treatment Effects Mammen, Enno, Christoph Rothe, and Melanie Schienle 2016 Semiparametric Estimation with Generated Covariates Econometric Theory 32 (5):

12 Romano, Joseph P and Azeem M Shaikh 2010 Inference for the Identified Set in Partially Identified Econometric Models Econometrica 78 (1): Song, Kyungchul 2014 Semiparametric Models with Single-Index Nuisance Parameters Journal of Econometrics 178 (3):