Roberto Casarin University Ca Foscari of Venice Recent AdvancesDiscussion in Statistical Inference of the Padova, paper March 21-23, EMVS: 2013 The E

Transcription

1 Discussion of the paper EMVS: The EM Approach to Bayesian Variable Selection Veronika Ro cková and Edward I. George Roberto Casarin University Ca Foscari of Venice Recent Advances in Statistical Inference Padova, March 21-23, 2013

2 A Historical Perspective of BMS and BMA (Hoeting, Madigan, Raftery an Volinsky (1999), Stat. Science) Barnard, G. A. (1963), New Methods of quality control, JRSS A First mention of model combination in the statistical literature (airline passenger data)

3 A Historical Perspective of BMS and BMA (Hoeting, Madigan, Raftery an Volinsky (1999), Stat. Science) Barnard, G. A. (1963), New Methods of quality control, JRSS A First mention of model combination in the statistical literature (airline passenger data) Roberts, H. V. (1965), Probabilistic prediction, JASA Suggests a distribution which combines the opinion of two experts (or models)

4 A Historical Perspective of BMS and BMA (Hoeting, Madigan, Raftery an Volinsky (1999), Stat. Science) Barnard, G. A. (1963), New Methods of quality control, JRSS A First mention of model combination in the statistical literature (airline passenger data) Roberts, H. V. (1965), Probabilistic prediction, JASA Suggests a distribution which combines the opinion of two experts (or models) Bates, J. M. and Granger, C. W. J. (1969), The combination of forecasts, Operational Research Quarterly. Seminal forecasting paper about combining predictions from different models.

5 Alternative Approaches to BMA (BMS) All the stochastic methods that move simultaneously in the model and parameter spaces.

6 Alternative Approaches to BMA (BMS) All the stochastic methods that move simultaneously in the model and parameter spaces. Markov Chain Monte Carlo Model Comparison (MC 3 ). See for example Madigan, York (1995) Int. J. Stat. Review, the reversible jump in Green (1995) Bka, the product space search in Carlin and Chib (1995) JRSS B

7 Alternative Approaches to BMA (BMS) All the stochastic methods that move simultaneously in the model and parameter spaces. Markov Chain Monte Carlo Model Comparison (MC 3 ). See for example Madigan, York (1995) Int. J. Stat. Review, the reversible jump in Green (1995) Bka, the product space search in Carlin and Chib (1995) JRSS B Stochastic Search Variable Selection (SSVS) see George and McCulloch (1993) JASA and more recently see the model search approach for state space models in Frühwirth-Schnatter and Wagner (2009) JoE.

8 Bayesian Variable Selection This paper EMVS: The EM Approach to Bayesian Variable Selection proposes a suitable combination of EM algorithm with the spike-and-slab mixture underlying the SSVS. George and McCulloch (1993, 1997)

9 Bayesian Variable Selection This paper EMVS: The EM Approach to Bayesian Variable Selection proposes a suitable combination of EM algorithm with the spike-and-slab mixture underlying the SSVS. George and McCulloch (1993, 1997) I enjoyed the paper! I found it quite stimulating. It presents a really efficient method that can find many applications.

10 Discussion Part A Estimation issues Part B Modelling issues

11 Discussion Part A- Estimation issues

12 (Not so) related works Following the authors, the EM algorithm has been previously considered in the context of Bayesian shrinkage.

13 (Not so) related works Following the authors, the EM algorithm has been previously considered in the context of Bayesian shrinkage. I would add and also in BMA literature. More specifically in forecast combination, EM can be used to estimate the BMA parameters. From a BMA perspective, one assume a suitable combination of predictive densities and then try to find an optimal combination.

14 (Not so) related works Following the authors, the EM algorithm has been previously considered in the context of Bayesian shrinkage. I would add and also in BMA literature. More specifically in forecast combination, EM can be used to estimate the BMA parameters. From a BMA perspective, one assume a suitable combination of predictive densities and then try to find an optimal combination. A possible combination model is m w j g j (y st f jst,θ i ) (1) s,t j=1 where f jst is the j-th predictor at space-time point (s,t)

15 (Not so) related works Following the authors, the EM algorithm has been previously considered in the context of Bayesian shrinkage. I would add and also in BMA literature. More specifically in forecast combination, EM can be used to estimate the BMA parameters. From a BMA perspective, one assume a suitable combination of predictive densities and then try to find an optimal combination. A possible combination model is m w j g j (y st f jst,θ i ) (1) s,t j=1 where f jst is the j-th predictor at space-time point (s,t) Fraley, Raftery and Gneiting (2009), WP, Sloughter, Gneiting and Raftery (2009), JASA, Sloughter, Raftery, Gneiting and Fraley (2007).

16 (Not so) related works Following the authors, the EM algorithm has been previously considered in the context of Bayesian shrinkage. I would add and also in BMA literature. More specifically in forecast combination, EM can be used to estimate the BMA parameters. From a BMA perspective, one assume a suitable combination of predictive densities and then try to find an optimal combination. A possible combination model is m w j g j (y st f jst,θ i ) (1) s,t j=1 where f jst is the j-th predictor at space-time point (s,t) Fraley, Raftery and Gneiting (2009), WP, Sloughter, Gneiting and Raftery (2009), JASA, Sloughter, Raftery, Gneiting and Fraley (2007). Estimation: data augmentation, 0 1 variables, apply EM. Discussion of the paper EMVS: The E Roberto Casarin University Ca Foscari of Venice Recent Advances in Statistical Inference Padova, March 21-23, 2013

17 (Not so) related works

18 (Not so) related works In econometrics: the weights are random processes, in Timmermann (2009), HEF Markov-switching weights (apply EM), in Billio, Casarin, Ravazzolo, Van Dijk (2013), JoE logistic or Dirichlet weights (apply PF).

19 (Not so) related works In econometrics: the weights are random processes, in Timmermann (2009), HEF Markov-switching weights (apply EM), in Billio, Casarin, Ravazzolo, Van Dijk (2013), JoE logistic or Dirichlet weights (apply PF). Philosophical question: is the spike-and-slab prior specification the only difference between EM-BMA and EM-BMS?

20 (Not so) related works In econometrics: the weights are random processes, in Timmermann (2009), HEF Markov-switching weights (apply EM), in Billio, Casarin, Ravazzolo, Van Dijk (2013), JoE logistic or Dirichlet weights (apply PF). Philosophical question: is the spike-and-slab prior specification the only difference between EM-BMA and EM-BMS? In BMA, you can deal with missing values and with clustering of forecast within a EM approach. Can you incorporate this in your EM-BMS approach?

21 (Not so) related works From an historical perspective the two streams of literature evolve in parallel, but with relevant and sometimes unreconcilable differences: in BMA the combination (or mixture) of forecast (density) is the model, thus, feel free to choose your preferred source of forecast and your preferred combination model!

22 (Not so) related works From an historical perspective the two streams of literature evolve in parallel, but with relevant and sometimes unreconcilable differences: in BMA the combination (or mixture) of forecast (density) is the model, thus, feel free to choose your preferred source of forecast and your preferred combination model! in BMS the model is usually given within a family of distribution, and the ouput of the BMS can be eventually used to do averaging

23 (Not so) related works From an historical perspective the two streams of literature evolve in parallel, but with relevant and sometimes unreconcilable differences: in BMA the combination (or mixture) of forecast (density) is the model, thus, feel free to choose your preferred source of forecast and your preferred combination model! in BMS the model is usually given within a family of distribution, and the ouput of the BMS can be eventually used to do averaging in BMA the set of forecast densities does not necessary contain the true model and you can account for the mispecification error. In BMS, model mispecification is not usually part of the inference problem. But, see also the Bernardo and Smith (1994) classification.

24 (Not so) related works From an historical perspective the two streams of literature evolve in parallel, but with relevant and sometimes unreconcilable differences: in BMA the combination (or mixture) of forecast (density) is the model, thus, feel free to choose your preferred source of forecast and your preferred combination model! in BMS the model is usually given within a family of distribution, and the ouput of the BMS can be eventually used to do averaging in BMA the set of forecast densities does not necessary contain the true model and you can account for the mispecification error. In BMS, model mispecification is not usually part of the inference problem. But, see also the Bernardo and Smith (1994) classification.

25 Discussion Part B - Modelling issues

26 Bayesian Variable Selection Summary of the EMVS model: f(y t α,β,σ) = N(α+x t β,σ2 ) π(β σ,γ,ν 0,ν 1 ) = N p (0,D σ,γ ) D σ,γ = σ 2 diag(a 1,...,a p ) a i = (1 γ i )ν 0 +γ i ν 1 γ i Ber(θ)

27 Bayesian Variable Selection Summary of the EMVS model: f(y t α,β,σ) = N(α+x t β,σ2 ) π(β σ,γ,ν 0,ν 1 ) = N p (0,D σ,γ ) D σ,γ = σ 2 diag(a 1,...,a p ) a i = (1 γ i )ν 0 +γ i ν 1 γ i Ber(θ) and possibly θ Be(α,β) (parsimony see also Ley and Steel (2009) JoE)

28 Application of the EMVS to Time Series Analysis SSVS has been applied to: 1. VAR large dimension (George, Sun and Ni (2008), Jochmanna, Koop, and Strachan (2010))

29 Application of the EMVS to Time Series Analysis SSVS has been applied to: 1. VAR large dimension (George, Sun and Ni (2008), Jochmanna, Koop, and Strachan (2010)) 2. Multivariate GARCH or SV models (see Loddo, Ni and Sun (2011) JBES)

30 Application of the EMVS to Time Series Analysis SSVS has been applied to: 1. VAR large dimension (George, Sun and Ni (2008), Jochmanna, Koop, and Strachan (2010)) 2. Multivariate GARCH or SV models (see Loddo, Ni and Sun (2011) JBES) 3. Bayesian nonparametric

31 Some Modelling Issues Empirical findings when applying a time-varying BMA approach: 1. strong evidence of change over time of the relevance of the predictors (model instability). 2. similar models have similar weights (model clustering)

32 w M3t Model instability w M1t w M2t predictors

33 Model clustering EGARCH EGARCH G GARCH GARCH G 0.8 EGARCH T 0.16 GARCH T M M M M M M M M GJR GJR G GJR T predictors M M M M01 Casarin et al. (2012), MatCom Discussion of the paper EMVS: The E Roberto Casarin University Ca Foscari of Venice Recent Advances in Statistical Inference Padova, March 21-23, 2013

34 Model clustering and instability 15 SR (1992M12) 0.9 SR (1997M12) SR SR WN WN 0.1 SR (2008M06) 0.9 SR (2008M12) SR SR WN WN Billio, Casarin, Ravazzolo, Van Dijk (2013), JoE

35 Time-varying model probabilities Time-varying model probabilities f(y t α,β t,σ) = N(α+x tβ t,σ 2 ) (2) π(β t σ,γ t,ν 0,ν 1 ) = N p (0,D σ,γt ) (3) D σ,γt = σ 2 diag(a 1,t,...,a p,t ) (4) a it = (1 γ it )ν 0 +γ it ν 1 (5)

36 Time-varying model probabilities Time-varying model probabilities f(y t α,β t,σ) = N(α+x tβ t,σ 2 ) (2) π(β t σ,γ t,ν 0,ν 1 ) = N p (0,D σ,γt ) (3) D σ,γt = σ 2 diag(a 1,t,...,a p,t ) (4) a it = (1 γ it )ν 0 +γ it ν 1 (5) Alternative specifications of γ it γ it Ber(θ) i.i.d. i,t (dynamic mixture) γ it γ it 1 MC(P) (Markov-switching, and Markov-Random Fields, in case of persistence) Advantages: EM can be applied

37 Model clustering A flexible way to have clusters in the model space is to use Bayesian nonparametric with f(y t α,β,σ) = N(α+x tβ,σ 2 ) (6) π(β σ,γ,ν 0,ν 1 ) = N p (0,D σ,γ ) (7) D σ,γ = σ 2 diag(a 1,...,a p ) (8) a i = (1 γ i )ν 0 +γ i ν 1 (9) γ i Ber(θ) (10)

38 Model clustering A flexible way to have clusters in the model space is to use Bayesian nonparametric with θ f(y t α,β,σ) = N(α+x tβ,σ 2 ) (6) π(β σ,γ,ν 0,ν 1 ) = N p (0,D σ,γ ) (7) D σ,γ = σ 2 diag(a 1,...,a p ) (8) a i = (1 γ i )ν 0 +γ i ν 1 (9) γ i Ber(θ) (10) Be(α,β)dG(α,β) = w i Be(α i,β i ) Can EM be applied for a finite mixture approximation of the DP? i=1