On Model Selection Criteria for Climate Change Impact Studies. Xiaomeng Cui Dalia Ghanem Todd Kuffner UC Davis UC Davis WUSTL

Transcription

1 1 / 47 On Model Selection Criteria for Climate Change Impact Studies Xiaomeng Cui Dalia Ghanem Todd Kuffner UC Davis UC Davis WUSTL December 8, 2017 University of Southern California

2 2 / 47 Motivation Empirical Literature Setting: For i = 1, 2,..., n, t = 1, 2,..., T, we observe a scalar outcome Y it ; in addition, for each i and t, we observe a regressor W itτ at a higher frequency τ = 1, 2,..., H. Weather and Climate Change Impacts (Review: Dell, Jones and Olken, JEL 2014) Climate Change and Agriculture: Deschenes and Greenstone (2007, AER), Fisher et al (2012, AER), Schlenker and Roberts (2009, PNAS), Schlenker and Lobell (2010, ERL), Feng, Krueger and Oppenheimer (2010, PNAS), Lobell, Schlenker and Costa-Roberts (2011, Science), Cui (2017),... Climate Change and Health: Deschênes, Greenstone and Guryan (2009, AER), Deschênes and Greenstone (2011, AEJ: Applied),... Pollution Impacts Boone et al (2013), Burney and Ramanathan (2014, PNAS), Carter et al (2016, Scientific Reports)

3 / 47 Motivation Empirical Practice: CHOOSE summary statistics of T k it ({W itτ } H τ=1 ) as regressors in a linear fixed effects model y it = K k=1 β k T k it ({W itτ } H τ=1 ) + a i + u it (1) Note: straightforward to accommodate year fixed effects and other control variables Annual Mortality Rate Daily Temperature Annual Mortality Rate Daily Mean Temperature Year Year

4 / 47 Motivation Empirical Practice: CHOOSE summary statistics of T k it ({W itτ } H τ=1 ) as regressors in a linear fixed effects model y it = K k=1 β k T k it ({W itτ } H τ=1 ) + a i + u it (1) Note: straightforward to accommodate year fixed effects and other control variables Annual Mortality Rate Daily Temperature Annual Mortality Rate Daily Mean Temperature Year Year Examples of Summary Statistics - temperature bins - various degree day measures - annual or seasonal means - quadratic function of annual mean -...

5 4 / 47 Motivation Goals of the Literature 1. Causal Inference identify the damage/response function that governs the relationship between Y it and the high-frequency regressor {W itτ } H τ=1 2. Prediction estimate the impact of future climate change on outcomes of interest This paper importance of model selection in this problem special features of the model selection problem faced by empirical researchers properties of cross-validation (CV) and information criteria (IC) as model selection tools for climate change impact studies - importance of guarding against over-fitting - mean squared error as a loss function Disclaimer: the paper will focus on linear fixed effects models only.

6 5 / 47 Outline of the Talk Why Model Selection in Climate Change Impact Studies? Model Selection Problem Overview of Model Selection Criteria Simulation Study Empirical Illustration

7 6 / 47 Why Model Selection in Climate Change Impact Studies? An Economic Reason Consider the following models: Annual Mean (A): Y it = 0.5 W it + a i + u it Quadratic in Annual Mean (QinA): Y it = 0.2 W it 0.02 W 2 it + a i + u it Quarterly Mean (Q): Y it = 0.5 W Q 1 it 0.75 W Q 2 it + a i + u it

8 6 / 47 Why Model Selection in Climate Change Impact Studies? An Economic Reason Consider the following models: Annual Mean (A): Y it = 0.5 W it + a i + u it Quadratic in Annual Mean (QinA): Y it = 0.2 W it 0.02 W 2 it + a i + u it Quarterly Mean (Q): Y it = 0.5 W Q 1 it 0.75 W Q 2 it + a i + u it Different damages functions have different policy implications! Differences in responses across space and seasons - A: an additional degree in annual mean temperature will affect all areas in the same way. - QinA: an additional degree in annual mean temperature will be more/less harmful in warmer regions - Q: additional warming in the summer, but beneficial in the winter Different adaptation mechanisms E.g., suppose the outcome is agricultural yield, - QinA: agriculture will move to historically cooler areas as they get warmer - Q: growing season can be shifted within the year in regions with warming patterns in the winter Different predictions for future damages due to climate change

9 7 / 47 Why Model Selection in Climate Change Impact Studies? A Statistical Reason Simulation Illustration Design n = 100, T = 5, S = 500 W it is a random sample of counties from the NCDC temperature dataset for the years Fixed Effects : a i W i N(0.5 W i, 1) Idiosyncratic Shocks: u it W i, a i are i.i.d. across i, mixture normal, mean 0, heteroskedastic and correlated across time Design Details Outcome is generated using the following models: (i) Annual Mean: Y it = W it + a i + u it (ii) Quadratic in Annual Mean: Y it = 0.2 W it 0.05 W 2 it + a i + u it (iii) Quarterly Mean: Y it = 0.5 W Q 1 it W Q 2 it + a i + u it

10 8 / 47 Why Model Selection in Climate Change Impact Studies? A Statistical Reason Models Considered - Seasonal Mean Models A Annual Mean Model B Bi-Annual Mean Model Q Quarterly Mean Model M Monthly Mean Model - Other Models QinA Quadratic in Annual Mean Model Bins 10 F bins Simulation Summary Statistics For each model considered, we report: - simulation Mean and S.D. of linear FE estimator - probability of rejecting a 5% significance test (ˆp 0.05 )

11 9 / 47 Why Model Selection in Climate Change Impact Studies? A Statistical Reason DGP (Y ) Annual Mean Quadratic in Quarterly Mean Annual Mean Model Mean S.D. ˆp 0.05 Mean S.D. ˆp 0.05 Mean S.D. ˆp 0.05 A A Mean B B1 Mean B2 Mean Q Q1 Mean Q2 Mean Q3 Mean Q4 Mean M M1 Mean M2 Mean M3 Mean M4 Mean M5 Mean M6 Mean M7 Mean M8 Mean M9 Mean M10 Mean M11 Mean M12 Mean Notes: Qj denotes the j th quarter, Mj denotes the j th month.

12 10 / 47 Why Model Selection in Climate Change Impact Studies? A Statistical Reason DGP(Y ) Annual Mean Quadratic in Quarterly Mean Annual Mean Model Mean S.D. ˆp 0.05 Mean S.D. ˆp 0.05 Mean S.D. ˆp 0.05 QinA A Mean A Mean Bins Bin (, 10) Bin [10, 20) Bin [20, 30) Bin [30, 40) Bin [40, 50) Bin [60, 70) Bin [70, 80) Bin [80, 90) Bin [90, + )

13 11 / 47 Why Model Selection in Climate Change Impact Studies? A Statistical Reason DGP(Y ) Quadratic in Annual Mean n =100 n=500 n=1000 A A Mean QinA A Mean A Mean Q Q Mean Q Mean Q Mean Q Mean Model Selection based on Stars (p-hacking) if one model is true, all other models will likely have significant coefficients less likely to choose QinA model when it is true

14 12 / 47 Model Selection Problem Set of Models Considered: {M j } J j=1 where J < For each j, M j where Y it =X j it βj + a j i + u j it - k j dim((x j it ) ) = dim(β j ) - W it {W itτ } H τ=1 - X j it = f j (W it ), where X j it is a 1 k j vector. Remarks All models considered are linear in the parameters and separable in a j i and u j it We assume that the true model M = M j for some j. The key difference between the models is the included regressors. Additional regressors and fixed effects can be readily accommodated. Special feature of this problem is that all models considered consist of different summary statistics of the same underlying high-frequency variables.

15 13 / 47 Model Selection Problem: Definitions For two models M 1 and M 2, assume wlog k 1 < k 2. nested: if M 1 is a restricted version of M 2. Example: annual and quarterly mean model overlapping: if restricted versions of M 1 and M 2 yield the same model. Example: quadratic in annual mean and quarterly mean model strictly non-nested: are neither nested nor overlapping

16 Overview of Model Selection Criteria Cross Validation MCCV (Monte Carlo Cross-Validation) - Create b random subsamples (training sets) of the original sample - For each subsample, estimate each model under consideration (training) - Use the complement of each subsample (test data) to predict each model out-of-sample - Calculate the mean squared error (MSE) across subsamples - Decision Rule: Choose model that minimizes the MSE of the MCCV procedure Source: 14 / 47

17 15 / 47 Overview of Model Selection Criteria Information Criteria For linear models GIC λn = n log(mse n) + λ nk j (2) - ˆɛ i = y i ŷ j i - MSE n = n i=1 ˆɛ2 i /n is the MSE of the original sample - λ n is the penalty term for the dimension of the model AIC: λ n = 2 BIC: λ n = log(n) - k j is the number of parameters in M j - Decision Rule: Choose model that minimizes the information criterion.

18 16 / 47 Overview of Model Selection Criteria: Consistent Model Selection Definition: P(choosing the true model) 1 as n MC Cross-validation (Shao 1993) - MCCV can deliver consistent model selection if training sample size (n c ) is proportionately smaller with larger samples and proportionately more to the testing sample. n c /n 0 - Formal results only available/possible for homoskedastic and serially uncorrelated errors. Information Criteria (Vuong 1989, Shao 1997, Sin and White 1996, Hong and Preston 2012) - AIC tends to overfit. - BIC is consistent, but not for non-nested model selection. - GIC λn with λ n = n log log(n) is consistent for all cases. - Formal results can accommodate heteroskedasticity and serial correlation.

19 16 / 47 Overview of Model Selection Criteria: Consistent Model Selection Definition: P(choosing the true model) 1 as n MC Cross-validation (Shao 1993) - MCCV can deliver consistent model selection if training sample size (n c ) is proportionately smaller with larger samples and proportionately more to the testing sample. n c /n 0 - Formal results only available/possible for homoskedastic and serially uncorrelated errors. Information Criteria (Vuong 1989, Shao 1997, Sin and White 1996, Hong and Preston 2012) - AIC tends to overfit. - BIC is consistent, but not for non-nested model selection. - GIC λn with λ n = n log log(n) is consistent for all cases. - Formal results can accommodate heteroskedasticity and serial correlation. In practice, we never know the true model, so what? if a model that nests the true model is considered, then a consistent procedure will choose that model. if no model nests the true model is considered, then a consistent procedure will choose the model with the smallest mean squared error asymptotically.

20 17 / 47 Simulation Study Preview When the true model is considered or the models considered nest it. - AIC and MCCV-p tend to overfit, with the former fairing worse in that regard than the latter. - BIC, MCCV-Shao, SW 1 and SW 2 choose the true model/the most parsimonious model that nests the true model. When the true model is not considered and the models considered do not nest it. - AIC, MCCV-p, BIC and MCCV-Shao tend to overfit. - SW 1 and SW 2 choose the most parsimonious model with the smallest MSE. However, in this case, minimizing MSE may be misleading in terms of what are important aspects of the damage function. What is interesting here is that the relationship between the considered models and the true model is what matters for consistency, not how models relate to each other as in classical model selection problems.

21 18 / 47 Simulation Study Back to the Previous Design Model Selection Problems Nested Models A Annual Mean Model B Bi-Annual Mean Model Q Quarterly Mean Model M Monthly Mean Model (X j it βj = 12 s=1 βj s Ms W, where it W Ms it = τ Ms W itτ / M s ) Overlapping/Non-nested Models A Annual Mean Model QinA Quadratic in Annual Mean Model Q Quarterly Mean Model Bins 10 F bins Model We will consider the choice between 2, 3 and all models in each problem.

22 19 / 47 Simulation Study Model Selection Criteria MCCV (n c/n) - p = 0.75 (MCCV-p) - p n = n 1/4 (MCCV-Shao) - λ nt = 2 (AIC) - λ nt = log(nt ) (BIC) - λ nt = nt log(log(nt )) (SW 1) - λ nt = nt log(nt ) (SW 2) Penalty SW 2 SW 1 BIC N

23 Simulation Study DGP=A (Annual Mean): Y it = W it + a i + u it Nested Models A B Q M Sim MSE n = 100 n = 500 MCCV GIC λnt MCCV GIC λnt Models p Shao AIC BIC SW 1 SW 2 p Shao AIC BIC SW 1 SW 2 A M A B A Q A B M A Q M A B Q B Q M A B Q M MCCV-p and AIC is not vanishing as n increases. Definition. Sim MSE is the mean squared error for the full sample of 3074 counties using the simulation pseudo-true, i.e. simulation mean of the estimated parameters of each model. 20 / 47

24 Simulation Study DGP=A (Annual Mean): Y it = W it + a i + u it Non-Nested Models A QinA Q Bins Sim MSE n = 100 n = 500 MCCV GIC λnt MCCV GIC λnt Models p Shao AIC BIC SW 1 SW 2 p Shao AIC BIC SW 1 SW 2 A QinA A Bins QinA Bins A QinA Bins A Q Bins A QinA Q QinA Q Bins A QinA Q Bins All models considered nest the true model, except the Bins, which has a larger MSE. MCCV-Shao, BIC, SW 1 and SW 2 exhibit consistency. 21 / 47

25 22 / 47 Simulation Study DGP=Q (Quarterly Mean): Y it = 0.25 W Q 1 it W Q 2 it + a i + u it Nested Models A B Q M Sim. MSE n = 100 n = 500 MCCV GIC λnt MCCV GIC λnt Models p Shao AIC BIC SW 1 SW 2 p Shao AIC BIC SW 1 SW 2 A M A B A Q A B M A Q M A B Q B Q M A B Q M SW 2 has a tendency to underfit for smaller sample sizes, but problem eventually subsides.

26 Simulation Study DGP=Q (Quarterly Mean): Y it = 0.25 W Q 1 it W Q 2 it + a i + u it Non-nested Models A QinA Q Bins Sim. MSE n = 100 n = 500 MCCV GIC λnt MCCV GIC λnt Models p Shao AIC BIC SW 1 SW 2 p Shao AIC BIC SW 1 SW 2 A QinA A Bins QinA Bins A QinA Bins A Q Bins A QinA Q QinA Q Bins A QinA Q B BIC and MCCV-Shao exhibit inconsistency issues in (A vs. QinA). SW 1 and SW 2 underfit for smaller samples (A vs. Bins). 23 / 47

27 Simulation Study DGP=Q (Quarterly Mean): Y it = 0.25 W Q 1 it W Q 2 it + a i + u it Non-nested Models A QinA Q Bins Sim. MSE n = 1000 n = 3000 Models AIC BIC SW 1 SW 2 AIC BIC SW 1 SW 2 A QinA A Bins QinA Bins A QinA Bins A Q Bins A QinA Q QinA Q Bins A QinA Q Bins BIC and MCC-Shao exhibit remain inconsistent for larger samples (A vs. QinA). Underfitting problem of SW 1 and SW 2 subsides (A vs. Bins). 24 / 47

28 Simulation Study DGP=QinA (Quadratic in Annual Mean): Y it = 0.2 W it 0.05 W 2 it + a i + u it Nested Models A B Q M Sim MSE n = 1000 n = 3000 Models AIC BIC SW 1 SW 2 AIC BIC SW 1 SW 2 A M A B A Q A B M A Q M A B Q B Q M A B Q M BIC is not consistent in the comparison between A and Q, which is interesting because they are nested models, but neither one of them nests the true model. MSE criterion may not be ideal for this problem, note that choosing monthly mean model is misleading from a policy perspective. 25 / 47

29 26 / 47 Simulation Study DGP=QinA (Quadratic in Annual Mean): Y it = 0.2 W it 0.05 W 2 it + a i + u it Non-nested Models A Q ina Q Bins Sim MSE n = 1000 n = 3000 AIC BIC SW 1 SW 2 AIC BIC SW 1 SW 2 A QinA A Bins QinA Bins A QinA Bins A Q Bins A QinA Q QinA Q Bins A QinA Q Bins BIC is inconsistent in the choice between A and Q.

30 27 / 47 Simulation Study Summary of Recommendations Information criteria are consistent under the most general conditions. AIC (MCCV-p) will tend to overfit. BIC (MCCV-Shao) leads to consistent model selection, unless no model considered nests the true model. Important to consider the physical relationship to ensure that we are close to the true model. SW 1 /SW 2 leads to consistent model selection in general, however it may underfit in some settings.

31 28 / 47 Empirical Illustration Data: a subsample of US counties in Deschênes & Greenstone (2011) (3074 counties in 49 states from ) Regression: y it = X j it βj + θp it + δ st + a i + γ tɛ it y it - overall annual mortality rate (number of deaths per 100,000, not age-adjusted) - age-specific annual mortality rate (number of deaths per 100,000) for < 1, 1 45, 45 65, > 65 X j it = f j (T it ): temperature variables constructed based on daily average temperature T it {T itτ } H τ=1 P it : annual total precipitation δ s: state-level linear trends γ t: year fixed effects Note: This is NOT a replication of Deschênes and Greenstone (2011). The dataset in Deschênes and Greenstone (2011) spans , use precipitation bins and age-adjusted mortality rate.

32 29 / 47 Empirical Illustration Temperature specifications: - No temperature variables (N) - Annual mean (A) - Quarterly mean (Q) - Monthly mean (M) - Quadratic in annual mean (QinA) - Annual 20 F bins (10 Bins) reference bin: F - Annual 10 F bins (20 Bins) reference bin: F

33 Empirical Illustration: Estimation Results A Q M Annual Mean (0.55) Q1 Mean -0.85*** (0.22) Q2 Mean 0.09 (0.43) Q3 Mean 0.92* (0.45) Q4 Mean (0.29) M1 Mean -0.50** (0.18) M2 Mean (0.18) M3 Mean (0.22) M4 Mean 0.50 (0.27) M5 Mean (0.26) M6 Mean -0.70** (0.26) M7 Mean 0.78* (0.33) M8 Mean 0.43 (0.32) M9 Mean 0.01 (0.29) M10 Mean 0.53* (0.25) M11 Mean 0.37 (0.22) M12 Mean -0.46* (0.19) Observations 64,554 64,554 64,554 Notes: All regressions control for total precipitation, state linear trends, county and year fixed effects. Standard errors (in parentheses) are clustered at county-level. Significance: * (p < 0.05), ** (p < 0.01), *** (p < 01). 30 / 47

34 31 / 47 Empirical Illustration: Estimation Results A QinA 20 F Bins 10 F Bins Annual Mean (0.55) (3.29) Annual Mean (0.03) [,20] 0.21* (0.09) [20,40] 0.16** (0.06) [60,80] (0.06) [80,+ ] 0.11 (0.08) [,10] 0.29* (0.13) [10,20] 0.19 (0.13) [20,30] 0.17* (0.08) [30,40] 0.20** (0.07) [50,60] 0.05 (0.07) [60,70] 0.01 (0.08) [70,80] 0.03 (0.08) [80,+ ] 0.15 (0.09) Observations 64,554 64,554 64,554 64,554 Notes: All regressions control for total precipitation, state linear trends, county and year fixed effects. Standard errors (in parentheses) are clustered at county-level. Significance: * (p < 0.05), ** (p < 0.01), *** (p < 01).

35 Empirical Illustration: Model Selection Nested Models N A Q M MSE 13,317 13,316 13,313 13,307 AIC 613, , , ,171 BIC 613, , , ,906 SW 1 640, , , ,923 SW 2 671, , , ,499 MCCV (p=0.75,b=1000) MCCV-Shao (p=0.13,b=6148) Non-Nested Models N A Q in A Bin 20F Bin 10F MSE 13,317 13,316 13,316 13,313 13,313 AIC 613, , , , ,192 BIC 613, , , , ,891 SW 1 640, , , , ,376 SW 2 671, , , , ,145 MCCV (p=0.75,b=1000) MCCV-Shao (p=0.13,b=6148) N: No temperature variables, A: Annual mean, Q: Quarterly mean, M: Monthly mean, QinA: Quadratic in annual mean, Bin 10F: Annual 10 F bins (reference bin: F), Bin 20F: Annual 20 F bins (reference bin: F) 32 / 47

36 Empirical Illustration: Estimation Results (Age: >65) A Q M Annual Mean *** (3.48) Q1 Mean *** (1.56) Q2 Mean (2.58) Q3 Mean 7.47** (2.86) Q4 Mean (1.85) M1 Mean -3.30** (1.13) M2 Mean -4.14*** (1.17) M3 Mean -3.87** (1.43) M4 Mean 0.81 (1.72) M5 Mean (1.59) M6 Mean -3.60* (1.72) M7 Mean 2.85 (2.04) M8 Mean 3.13 (1.93) M9 Mean 1.21 (1.85) M10 Mean (1.69) M11 Mean 1.59 (1.23) M12 Mean -2.40* (1.17) Observations 64,554 64,554 64,554 Notes: All regressions control for total precipitation, state linear trends, county and year fixed effects. Standard errors (in parentheses) are clustered at county-level. Significance: * (p < 0.05), ** (p < 0.01), *** (p < 01). 33 / 47

37 34 / 47 Empirical Illustration: Estimation Results (Age: >65) A QinA 20 F Bins 10 F Bins Annual Mean *** * (3.48) (18.79) Annual Mean (0.17) [,20] 2.74*** (0.62) [20,40] 0.84* (0.35) [60,80] -0.95* (0.37) [80,+ ] (0.49) [,10] 3.25*** [10,20] 1.91* [20,30] 0.57 (0.52) [30,40] 0.65 (0.45) [50,60] (0.45) [60,70] -1.46** (0.50) [70,80] -1.16* (0.49) [80,+ ] (0.57) Observations 64,554 64,554 64,554 64,554 Notes: All regressions control for total precipitation, state linear trends, county and year fixed effects. Standard errors (in parentheses) are clustered at county-level. Significance: * (p < 0.05), ** (p < 0.01), *** (p < 01).

38 Empirical Illustration: Model Selection (Age: >65) Nested Models N A Q M MSE 469, , , ,321 AIC 843, , , ,175 BIC 843, , , ,911 SW 1 870, , , ,927 SW 2 901, , , ,503 MCCV 472, , , ,333 (p=0.75,b=1000) MCCV-Shao 479, , , ,489 (p=0.13,b=6148) Non-Nested Models N A Q in A Bin 20F Bin 10F MSE 469, , , , ,579 AIC 843, , , , ,203 BIC 843, , , , ,902 SW 1 870, , , , ,387 SW 2 901, , , , ,156 MCCV 472, , , , ,491 (p=0.75,b=1000) MCCV-Shao 479, , , , ,376 (p=0.13,b=6148) N: No temperature variables, A: Annual mean, Q: Quarterly mean, M: Monthly mean, QinA: Quadratic in annual mean, Bin 10F: Annual 10 F bins (reference bin: F), Bin 20F: Annual 20 F bins (reference bin: F) 35 / 47

39 36 / 47 Empirical Illustration: Summary For the mortality rate > 65, BIC/MCCV-Shao choose the quarterly mean model. The result is intuitive, since it shows warming in the winter reduces the mortality rate but warming in the summer increases it. For overall mortality rate and for all other age groups, all consistent model selection criteria (BIC, etc.) choose a model with no temperature variables/annual mean model (which is not significant). The results are qualitatively consistent with Deschênes and Greenstone (2011), even though we use a different sample size and specification.

40 37 / 47 Concluding Remarks Summary The model selection problem examined here is relevant whether we are interested in examining climate change impacts or adaptation to climate change. Practical recommendations - BIC behaves similarly to MCCV-Shao, and it is less computationally intensive. - BIC is inconsistent when at least one of the models under consideration nests the true model. (This is different from the typical model selection setting.) - SW 1 can guard against overfitting, but it can underfit in some cases. Further Questions/Caveats Prediction of impacts of future climate change - Is MSE the right criterion? Post-selection inference - Data splitting? Nonseparability in unobservables - e.g. random coefficient models Infinite-dimensional true model

41 37 / 47 Concluding Remarks Summary The model selection problem examined here is relevant whether we are interested in examining climate change impacts or adaptation to climate change. Practical recommendations - BIC behaves similarly to MCCV-Shao, and it is less computationally intensive. - BIC is inconsistent when at least one of the models under consideration nests the true model. (This is different from the typical model selection setting.) - SW 1 can guard against overfitting, but it can underfit in some cases. Further Questions/Caveats Prediction of impacts of future climate change - Is MSE the right criterion? Post-selection inference - Data splitting? Nonseparability in unobservables - e.g. random coefficient models Infinite-dimensional true model Thank you!