Incorporating cost in Bayesian variable selection, with application to cost-effective measurement of quality of health care

Transcription

1 Incorporating cost in Bayesian variable selection, with application to cost-effective measurement of quality of health care Dimitris Fouskakis, Department of Mathematics, School of Applied Mathematical and Physical Sciences, National Technical University of Athens, Athens, Greece; Joint work with: Ioannis Ntzoufras & David Draper Department of Statistics Athens University of Economics and Business Athens, Greece; Department of Applied Mathematics and Statistics University of California Santa Cruz, USA; Presentation is available at: fouskakis/conferences/hwu/hwu.pdf.

2 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 2 Synopsis 1. Motivation. 2. Model Specification. 3. Decision Theoretic Cost-Benefit Analysis. 4. Bayesian Cost-Benefit Analysis. 5. Utility versus Cost-Adjusted BIC. 6. Discussion.

3 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 3 1 Motivation Health care quality measurements Indirect method: input-output approach. Construct a model on hospital outcomes (e.g., mortality within 30 days of admission) after adjusting for differences in inputs (sickness at admission). Compare observed and expected outcomes to infer for the health care quality. Data collection costs are available for each variable (measured in minutes or monetary units). We wish to incorporate cost in our analysis in order to reduce data collection costs but also have a well-fitted model.

4 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 4 Available data Data come form a major U.S. study constructed by RAND Corporation, with n = 2, 532 pneumonia patients (Keeler, et al., 1990). Response variable: mortality within 30 days of admission Covariates: p = 83 sickness indicators Construct a sickness scale using a logistic regression model. Benefit - Only Analysis (no costs): Classical variable selection techniques to find an optimal subset of indicators. The initial list of p = 83 sickness indicators was reduced to 14 significant predictors (Keeler, et al., 1990).

5 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 5 The 14-Variable Rand Pneumonia Scale The RAND admission sickness scale for pneumonia (p = 14 variables), with the marginal data collection costs per patient for each variable (in minutes of abstraction time). Variable Cost Variable Cost 1 Systolic Blood Pressure Score (2-point scale) (Minutes) Septic Complications (yes, no) 2 Age Prior Respiratory Failure (yes, no) (Minutes) 3 Blood Urea Nitrogen Recently Hospitalized (yes, no) APACHE II Coma Score (3-point scale) 5 Shortness of Breath Day 1 (yes, no) 6 Serum Albumin Score (3-point scale) 7 Respiratory Distress (yes, no) Initial Temperature Chest X-ray Congestive Heart Failure Score (3-point scale) Ambulatory Score (3-point scale) Total APACHE II Score (36-point scale)

6 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 6 Two different approaches The RAND Benefit - Only approach is sub-optimal: it does not consider differences in cost of data collection among available predictors. We propose a Cost - Benefit Analysis, in which variables are chosen only when they predict well enough given how much they cost to collect. In problems such as this, in which there are two desirable criteria that compete, and over which a joint optimisation must be achieved, there are two main ways to proceed: Both criteria can be placed on a common scale, and optimisation can occur on that scale (strategy (a)). One criterion can be optimised, subject to a bound on the other (strategy (b)).

7 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 7 Three methods for solving this problem (1) (strategy (a)) Draper and Fouskakis (2000) and Fouskakis and Draper (2002, 2008) proposed an approach to this problem based on Bayesian Decision Theory. They used stochastic optimisation methods to find (near-) optimal subsets of predictor variables that maximize an expected utility function which trades off data collection cost against predictive accuracy. (2) (strategy (a)) In this work, as an alternative to (1), we propose a prior distribution that accounts for the cost of each variable and results in a set of posterior model probabilities which correspond to a Generalized Cost-Adjusted version of the Bayesian Information Criterion (Fouskakis, Ntzoufras and Draper, 2009a). (3) (strategy (b)) We also implement a Cost - Restriction - Benefit Analysis, where the search is conducted only among models whose cost does not exceed a budgetary restriction (Fouskakis, Ntzoufras and Draper, 2009b), by the usage of a Population - Based Trans - Dimensional RJMCMC Method. Here we present results from methods (1) (Decision Theoretic Cost-Benefit Analysis) and (2) (Bayesian Cost-Benefit Analysis).

8 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 8 2 Model Specification Logistic regression model with Y i = 1 if patient i dies after 30 days of admission. X ij : j sickness predictor variable for the i patient. m γ = (γ 1,...,γ p ) T. γ j : Binary indicators of the inclusion of the variable X j in the model. Model space M = {0, 1} p ; p = total number of variables considered. Hence the model formulation can be summarized as (Y i γ) ( ) pi (γ) η i (γ) = log 1 p i (γ) indep = Bernoulli(p i (γ)), p β j γ j X ij, j=0 η(γ) = X diag(γ) β = Xγ βγ.

9 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 9 3 Decision Theoretic Cost-Benefit Analysis Utility Elicitation (1) We take Bayesian decision-theoretic approach (based on maximisation of expected utility). Utility function has 2 components: quantifying data collection costs and predictive successes and failures. Data-collection utility: p available sickness indicators X j ; γ j = 1 if X j is included in subset (0 otherwise). Dividing n patients at random into modeling and validation subsamples of size n M and n V, respectively, data-collection cost associated with subset γ = (γ 1,...,γ p ) for patients in validation subsample is U D (γ) = n V p c j γ j, (1) j=1 where c j is marginal cost per patient of data abstraction for variable j.

10 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 10 Utility Elicitation (2) Predictive Utility: (1) Apply logistic regression model, obtained from modeling subsample, to validation subsample to create predicted death probabilities ˆp γ i predictor subset γ. for patients using given (2) Classify patient i in the validation subsample as predicted dead or alive according to whether ˆp γ i exceeds or falls short of a cutoff p, which is chosen - by searching on a discrete grid from 0.01 to 0.99 by steps of to maximize the predictive accuracy of model γ. We then cross-tabulate actual versus predicted death status in a 2 2 contingency table, rewarding and penalizing model γ according to the numbers of patients in the validation sample which fall into the cells of the right-hand part of the following table.

11 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 11 Utility Elicitation (3) Rewards and Penalties Counts Predicted Predicted Died Lived Died Lived Actual Died C 11 C 12 n 11 n 12 Lived C 21 C 22 n 21 n 22 The predictive utility of model γ is then U P (γ) = 2 2 C lm n lm. (2) l=1 m=1 See Fouskakis and Draper (2008) for details on eliciting the utility values C lm.

12 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 12 Utility Elicitation (4) The overall expected utility function to be maximised over γ is then simply E[U(γ)] = E[U D (γ) + U P (γ)], (3) where this expectation is over all possible cross-validation splits of the data. The number of possible cross-validation splits is far too large to evaluate the expectation in (3) directly; in practice we therefore use Monte Carlo methods to evaluate it, averaging over N random modeling and validation splits.

13 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 13 Results We explored this approach in two settings: a Small World created by focusing only on the p = 14 variables in the original RAND scale (2 14 = 16, 384 is a small enough number of possible models to do brute-force enumeration of the estimated expected utility of all models). The Rand scale is nowhere near optimal when data collection costs are considered along with predictive accuracy: Estimated Expected Utility Number of Variables

14 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 14 The best model in this case is the one with the following 4 variables: 1. Systolic Blood Pressure - X 1, 2. Blood Urea Nitrogen - X 3, 3. APACHE II Coma Score - X 4 and 4. Shortness of Breath Day 1 Score - X 5. The 20 best models include the same 3 variables 18 or more times out of 20, and never include 6 other variables; the 5 best models are minor variations on each other, and include 4 6 variables. The best models save almost $8 per patient over the full 14-variable model (significant savings if input-output approach applied widely). the Big World defined by all p = 83 available predictors (2 83. = is far too large for brute-force enumeration; we compared a variety of stochastic optimisation methods - including simulated annealing, genetic algorithms, and tabu search - on their ability to find good variable subsets).

15 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 15 Drawback of the decision-theoretic approach Maximising expected utility, as we did before, is a natural Bayesian way forward in this problem, but (a) the elicitation process was complicated and difficult and (b) the utility structure we examine is only one of a number of plausible alternatives, with utility framed from only one point of view; the broader question for a decision-theoretic approach is whose utility should drive the problem formulation. It is well known (e.g., Arrow, 1963; Weerahandi and Zidek, 1981) that Bayesian decision theory can be problematic when used normatively for group decision-making, because of conflicts in preferences among members of the group; in the context of the problem addressed here, it can be difficult to identify a utility structure acceptable to all stakeholders (including patients, doctors, hospitals, citizen watchdog groups, and state and federal regulatory agencies) in the quality-of-care-assessment process.

16 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 16 4 Bayesian Cost-Benefit Analysis The aim is to identify well fitted models after taking into account the cost of each variable. Therefore we need to estimate posterior model probabilities f(γ y) = f(γ) f(y βγ, γ)f(βγ γ)dβγ γ {0,1} p f(γ ) f(y βγ, γ )f(βγ γ )dβγ after introducing a prior on model space f(γ) depending on the cost.

17 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University Preliminaries: Posterior model odds and penalty functions Information criteria (1) Information criterion for model γ f(y βγ) is the maximum likelihood. IC(γ) = 2 log f(y βγ, γ) + dγf dγ dimension of the model (number of parameters) F penalty for each model parameter used/estimated. dγf is the total penalty implemented to the maximum likelihood due to the use of a model with dγ parameters. Model with minimum IC is indicated as the best. The above criterion is a penalised likelihood measure.

18 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 18 Information criteria (2) When comparing two models γ (k) and γ (l) then IC kl = IC(γ (k) ) IC(γ (l) ) = 2 log f( y βγ (k), γ(k)) f ( y βγ (l), γ(l)) + ( dγ d ) (k) γ F (l) = Deviance kl + ( dγ (k) d γ (l) ) F We select model γ (k) if IC kl < 0, and model γ (l) if IC kl > 0.

19 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 19 Posterior model probabilities and information criteria The posterior model probability of a model γ is given by f(γ y) = f(y γ)f(γ) where f(y γ) is the marginal likelihood of model γ given by f(y βγ, γ)f(βγ γ)dβγ f(γ) prior probability of model γ It can be rewritten as 2 log f(γ y) = 2 log f(y γ) + [ 2 log f(γ)] IC(γ) = 2 log f(y βγ, γ) + dγf

20 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 20 Posterior model odds and information criteria Similarly if we consider the posterior odds of model γ (k) versus model γ (l). Then ( f(y γ (k) ) ) PO kl = f(γ(k) ) f(y γ (l) ) f(γ (l) ) = B kl PrO kl, B kl is the Bayes factor of model γ (k) versus model γ (l) (ratios of marginal likelihoods). PrO kl is the prior odds of model γ (k) versus model γ (l). It can be rewritten as 2 log PO kl = 2 log B kl + [ 2 log PrO lk ] IC kl = 2 log f( y βγ (k), γ(k)) f ( y βγ (l), γ(l)) + ( dγ (k) d γ (l) ) F

21 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 21 Uniform prior on model space If the prior model probabilities are defined via a negative function of the model dimension, then the prior model odds ξ kl = 2 log PrO kl = 2 log f(γ(k) ) f(γ (l) ) can be also interpreted as the extra penalty imposed to the Bayes factor. If the (usual) uniform prior distribution is used then ξ kl = 0 and PO kl = B kl for all models γ k, γ l M where M is the set of all models under consideration (model space). Bayesian benefit-only analysis can be assumed using the uniform prior on model space and hence base our variable selection procedure on Bayes factors.

22 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 22 Prior model odds interpretation Well-known rough approximation of log B kl (Schwartz, 1978): 2 log B kl = BIC kl + O(1) 2 log PO kl = BIC kl + ξ kl + O(1) = Deviance kl + (dγ d (k) γ (l)) log n + ξ kl + O(1) (4) where BIC kl is the Bayesian Information Criterion (e.g., Kass and Wasserman, 1996; Raftery, 1995, 1996) for choosing between models γ (k) and γ (l). BIC penalty equal to F = log n for each parameter used. The overall (posterior) penalty imposed to the deviance measure will be equal to (dγ (k) d γ (l)) log n + ξ kl.

23 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University Prior distributions Prior on model parameters βγ γ Normal ( ( ) ) 1 0, 4n X T γxγ Low information prior defined by Ntzoufras, Delaportas and Forster (2003). Can be derived using the power prior of Chen et al. (2000) and imaginary data supporting the simplest model included in our model space. It gives weight to the prior equal to one data-point. It is equivalent to the Zellner s g-prior (with g = 4n) used for normal regression models.

24 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 24 A cost-penalised prior on model space (1) Preliminaries We propose to specify our prior model probabilities via cost-dependent penalties for each variable. We denote by c j the cost of X j covariate and by c = (c 1, c 2,...,c p ) the vector of the costs of all variables under consideration. To specify this prior we define a baseline cost c 0 which is assumed to be a low acceptable cost for the collection of the data of a covariate. The cost of each variable can be then written as c j = k j c 0. For the Bayesian benefit only analysis we are using a uniform prior on model space.

25 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 25 A cost-penalised prior on model space (2) The five criteria We specify our prior distribution on γ to satisfy the following five criteria: (a) The prior must be unaffected to transformations c α c with α > 0, so that conversion between time and money or different monetary units (e.g., dollars and euros) leaves the prior unchanged; (b) the extra penalty ξ 1 for adding a variable X j with baseline cost c 0 is zero; (c) the extra penalty ξ 2 for adding a variable X j with cost c j = κ c 0 for some κ > 1 equals the BIC penalty of (κ 1) variables with cost c 0 ; (d) the extra penalty ξ 3 for adding any variable X j is greater or equal to zero; and (e) if all the variables have the same cost, then the prior must reduce to the uniform prior on γ.

26 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 26 A cost-penalised prior on model space (3) The five criteria - interpretation (a) ensures that the prior is invariant with respect to the manner in which cost is measured. (b) ensures that the penalty for adding a variable X j with baseline cost c 0 is the same as in the benefit-only analysis. (c) ensures that the posterior model odds will still have a BIC-like behavior. The induced extra penalty will be equal to the relative difference between the cost of X j and a variable with cost equal to c 0. (d) ensures that the cost-benefit analysis will support more parsimonious models than the corresponding ones supported by the benefit-only analysis. (e) requires that our prior should reproduce the benefit-only analysis if all costs are equal.

27 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 27 A cost-penalised prior on model space (4) The prior The following Theorem provides the only prior that meets the above five requirements, and defines the choice of c 0. Theorem 1. If a prior distribution f(γ) satisfies requirements (a-e) above, then it must be of the form f(γ j ) exp [ γ j 2 ( ) ] cj 1 log n c 0 where c j is the marginal cost per observation for variable X j and c 0 = min{c j, j = 1,...,p}. For proof see Fouskakis, Ntzoufras and Draper (2009a). for j = 1,...,p, (5)

28 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University Posterior model odds Cost-adjusted generalisation of BIC Under the above prior, if we consider the BIC-based approximation (4) then ( f(y ˆβγ 2 log PO kl = 2 log (k), ) γ(k) ) f(y ˆβγ (l), + C γ C (k) γ (l) log n + O(1). (6) γ(l) ) c 0 where Cγ = p j=1 γ jc j is the cost of model γ.

29 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 29 The penalty term dγ log n of model γ used in (4) has been replaced in the above expression by the cost-dependent penalty c 1 0 C γ log n; Ignoring costs is equivalent to c j = c 0 for all j, yielding c 1 0 C γ = dγ, the original BIC expression. We may interpret log n as the imposed penalty for each variable included in the model when no costs are considered. This baseline penalty term is inflated proportionally to the cost ratio c j c 0 for each X j ; for example, if the cost of a variable X j is twice the minimum cost (c j = 2 c 0 ) then the imposed penalty is equivalent to adding two variables with the minimum cost. For all these reasons, (6) can be considered as a cost-adjusted generalisation of BIC when prior model probabilities of type (5) are adopted.

30 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University Implementation and results Implementation details The procedure 1. Run RJMCMC (Green, 1995) for 100K iterations in the full model space. 2. Eliminate non-important variables (with marginal probabilities < 0.30) forming a new reduced model space. 3. Run RJMCMC for 100K iterations in the reduced model space to estimate posterior model odds and best models. Two setups: 1. Benefit only analysis (uniform prior on model space). 2. Cost - Benefit Analysis (cost penalised prior on model space).

31 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 31 Preliminary Results: Marginal Probabilities f(γ j = 1 y) Variable Variable Benefit Cost-Benefit Index Variable Name Cost Analysis Analysis 1 Systolic Blood Pressure (SBP) Score Age Blood Urea Nitrogen Apache II Coma Score Shortness of Breath Day Septic Complications Initial Temperature Heart Rate Day Chest Pain Day Cardiomegaly Score Hematologic History Score Apache Respiratory Rate Score Admission SBP Respiratory Rate Day Confusion Day Apache ph Score Morbid + Comorbid Score Musculoskeletal Score Number of variables 13 13

32 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 32 Reduced Model Space: Posterior Model Probabilities/Odds Common variables in both analyses: X 1 + X 2 + X 3 + X 5 + X 12 + X 70 Benefit-Only Analysis Common Variables Additional Model Posterior k Within Each Analysis Variables Cost Probabilities P O 1k 1 X 4 + X 15 + X 37 + X 73 +X 8 +X 27 +X X 8 +X X X 27 +X Cost-Benefit Analysis Common Variables Additional Model Posterior k Within Each Analysis Variables Cost Probabilities P O 1k 1 X 46 + X 51 +X 49 +X X 14 +X 49 +X X 13 +X 49 +X X 13 +X 14 +X 49 +X X 14 +X X X 37 +X X 13 +X 14 +X X above 3%. posterior odds of the best model within each analysis versus the current model k.

33 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 33 Reduced Model Space: Comparisons Comparison of measures of fit, cost and dimensionality between the best models in the reduced model space of the benefit-only and cost-benefit analysis; percentage difference is in relation to benefit-only. Analysis Difference Benefit-Only Cost-Benefit (%) Minimum Deviance Median Deviance Cost Dimension

34 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 34 5 Utility versus Cost-Adjusted BIC Method Variable Utility RJMCMC Cost Posterior Index Name (Minutes) Good? Good? Probability 1 Systolic Blood Pressure Score Age Blood Urea Nitrogen APACHE II Coma Score Shortness of Breath Day 1 (yes, no) Serum Albumin Score (3 point scale) Respiratory Distress (yes, no) Septic Complications (yes, no) Prior Respiratory Failure (yes, no) Recently Hospitalized (yes, no) Initial Temperature Chest X-ray Congestive Heart Failure Score 18 Ambulatory Score Total APACHE II Score It s clear that the Utility and Cost-Adjusted BIC approaches have reached nearly identical conclusions in the Small World of p = 14 predictors.

35 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 35 With p = 83 the agreement between the two methods is also strong (although not as strong as with p = 14): using a star system for variable importance given in Fouskakis, Ntzoufras and Draper (2009a), 60 variables were ignored by both methods, 8 variables had identical star patterns, 3 variables were chosen as important by both methods but with different star patterns, 10 variables were marked as important by the utility approach and not by RJMCMC, and 2 variables were singled out by RJMCMC and not by utility: thus the two methods substantially agreed on the importance of 71 (86%) of the 83 variables. Median p Method Model Cost Deviance LS CV 14 X 1 + X 2 + X 3 + X 4 + X 5 + X 6 + X 7 + X RJMCMC X 1 + X 2 + X 3 + X 4 + X 5 + X 7 + X Utility X 1 + X 3 + X 4 + X RJMCMC Utility X 1 + X 2 + X 3 + X 5 + X 12 +X 46 + X 49 + X 51 + X 70 + X X 1 + X 3 + X 4 + X 12 +X 46 + X 49 + X To the extent that the two methods differ, the utility method favors models that cost somewhat less but also predict somewhat less well.

36 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 36 6 Discussion The fact that the two methods may yield somewhat different results in high-dimensional problems does not mean that either is wrong; they are both valid solutions to similar but not identical problems. Both methods lead to noticeably better models (in a cost-benefit sense) than frequentist or Bayesian benefit-only approaches, when - as is often the case - cost is an issue that must be included in the problem formulation to arrive at a policy-relevant solution. In comparing two or more models, to say whether one is better than another I have to face the question: better for what purpose? This makes model specification a decision problem: I need to either (a) elicit a utility structure that s specific to the goals of the current study and maximise expected utility to find the best models, or (b) (if (a) is too hard, e.g., because the problem has a group decision character) I can look for a principled alternative (like the cost-adjusted BIC method described here) that approximates the utility approach while avoiding ambiguities in utility specification.

37 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 37 Authors related work Draper D, Fouskakis D (2000). A case study of stochastic optimization in health policy: problem formulation and preliminary results. Journal of Global Optimization, 18, Fouskakis D, Draper D (2002). Stochastic optimization: a review. International Statistical Review, 70, Fouskakis D, Draper D (2008). Comparing stochastic optimization methods for variable selection in binary outcome prediction, with application to health policy. Journal of the American Statistical Association, 103, Fouskakis D, Ntzoufras I, Draper D (2009a). Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care. Annals of Applied Statistics, 3, Fouskakis D, Ntzoufras I, Draper D (2009b). Population Based Reversible Jump MCMC for Bayesian Variable Selection and Evaluation Under Cost Limit Restrictions. Journal of the Royal Statistical Society C (Applied Statistics), 58,

38 23rd September 2009: School of Mathematical and Computer Sciences, Heriot-Watt University 38 Additional References Arrow KJ (1963). Social Choice and Individual Values. Wiley. New York. Chen MH, Ibrahim JG and Shao QM (2000). Power prior distributions for generalized linear models. Journal of Statistical Planning and Inference, 84, Green P (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika, 82, Keeler E, Kahn K, Draper D, Sherwood M, Rubenstein L, Reinisch E, Kosecoff J, Brook R (1990). Changes in sickness at admission following the introduction of the Prospective Payment System. Journal of the American Medical Association, 264, Ntzoufras I, Dellaportas P, Forster JJ (2003). Bayesian variable and link determination for generalized linear models. Journal of Statistical Planning and Inference, 111, Weerahandi S and Zidek JV (1983). Elements of multi-bayesian decision theory. Annals of Statistics, 11,