Structural Uncertainty in Health Economic Decision Models Mark Strong 1, Hazel Pilgrim 1, Jeremy Oakley 2, Jim Chilcott 1 December 2009 1. School of Health and Related Research, University of Sheffield, UK 2. Department of Probability and Statistics, University of Sheffield, UK Address for correspondence: Dr Mark Strong, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK. Email m.strong@sheffield.ac.uk Strong M, Pilgrim H, Oakley J, Chilcott J. (2009) Structural uncertainty in health economic decision models. ScHARR Occasional Paper. ISBN 1 900752 27 1 1
Structural Uncertainty in Health Economic Decision Models 2 Summary Within a health economic decision model there are multiple sources of uncertainty. These uncertainties must be properly accounted for if a decision maker is to have confidence in the model result. Structural uncertainty is present when we are uncertain about the model output because we are uncertain about the functional form of the model. We are not certain that our model adequately reflects reality, and we are therefore not certain that our result would be correct, even if the true values of all input parameters were known. Two broad approaches have been proposed for resolving the problem of structural uncertainty: the model averaging approach and the model discrepancy approach. In the model averaging approach we weight a set of plausible models by some measure of model adequacy, whereas in the discrepancy approach we focus on quantifying our judgements about the difference between the model output and the true value of the target quantity we are modelling. Both these approaches run into difficulties when applied in the health economics context. Model averaging implies that we have built a set of models, one of which we believe to be correct (we just don t know which one). This is a strong assumption given that models are in general only approximations of reality. Current model discrepancy approaches avoid having to make this assumption, but at the cost of having to make meaningful, direct judgements about the discrepancy between the model output and reality. This may simply not be feasible. In order to address this problem we propose a development of the discrepancy modelling approach whereby a model is first decomposed into a series of sub-units, revealing a set of intermediate parameters. We feel that it may be more fruitful to attempt to make useful judgements about the discrepancy between the modelled value and its true counterpart at the level of the intermediate parameters, than at the level of the model output. These judgements, once made, could then be combined with all other recognised sources of uncertainty within a coherent Bayesian framework.
Structural Uncertainty in Health Economic Decision Models 3 1 Introduction Within a health economic decision model there are inevitably multiple sources of uncertainty. These uncertainties must be fully accounted for in the output of the model if we are to truly acknowledge the limits of our knowledge. Only when a decision maker has a proper understanding of the range of model outputs that are consistent with the evidence can they make a properly informed decision. There is no single schema for characterising uncertainty within a health economic decision model. One suggestion is to think of uncertainty as being either parametric, methodological or structural (Briggs, 2000). Methods for assessing parametric and methodological uncertainty are well developed. Typically, a probabilistic sensitivity analysis is undertaken to quantify parametric uncertainty, and methodological uncertainties (for example, choices about how health outcomes are valued, or how time preference is incorporated) are, at least partially, dealt with by the increasing use of a reference case. Methods for quantifying structural uncertainty are rather less well described. Structural uncertainty, in broad terms, relates to whether or not all relevant processes are represented in the model; does the model adequately reflect reality? Standard probabilistic sensitivity analysis will not take into account any structural uncertainty that may be present, potentially leading to a spuriously precise estimate of the model output parameter. Moreover, structural uncertainty may well have a greater impact on the model results than either parameter or methodological uncertainty, yet methodologies for handling this source of uncertainty are relatively underdeveloped. This paper sets out some initial research on structural uncertainty currently being undertaken within The Centre for Bayesian Statistics in Health Economics (CHEBS) at the University of Sheffield. 1 1 http://www.shef.ac.uk/chebs/
Structural Uncertainty in Health Economic Decision Models 4 2 What is structural model uncertainty? In the context of health economics we often wish to predict the value of the population expected incremental cost effectiveness ratio (or the population expected net benefit) of choosing one course of action over an alternative. We take a Bayesian perspective and view the target entity as an uncertain quantity about which we seek to make probabilistic statements In order to predict this unknown, but in principle observable target quantity we build a deterministic model y = η(x). Our model is simply a function (possibly a highly complicated one), η(.), that takes a vector of input parameters x and generates an output y. (We assume here that y is scalar, but our conclusions are generalisable to vector model outputs). We usually consider our model input parameters, X, to be uncertain, and it is common practice to acknowledge this uncertainty in some form of uncertainty analysis. For example, in a probabilistic sensitivity analysis we would sample x 1,..., x n from p(x), and evaluate η(x 1 ),..., η(x n ) in order to get a sample from p(y ). This would quantify our uncertainty in Y due to our parameter uncertainty, but it would not tell us anything about our uncertainty in η(.). Imagine, however, that we do learn the true value of our input parameters X. Does our value of Y = η(x) represent the true target value, Ỹ? If we do not believe that this is the case, then we are acknowledging structural error, i.e. that η(.) is incorrect. There is model structural uncertainty if we are uncertain about Ỹ due to uncertainty about η(.). Finally, we note that if we do manage to specify a perfect model η(.) our evaluation of η(x) represents the mean of the target output parameter. The realised value in the population will not necessarily be equal to the mean due to random sampling variation (often known as first-order uncertainty). However, in this paper we are not concerned with this source of error, but instead in the error in the mean output value arising from the mis-specification of η(.).
Structural Uncertainty in Health Economic Decision Models 5 3 The M-open versus M-closed analysis of model uncertainty There is a rich statistical literature on model uncertainty motivated by the problem of choosing an optimal model from a set of stochastic models, conditional on some observed data (for example Draper (1995) and associated discussion). Although we generally build only a single health economic decision model, we can usually envisage building a number of other plausible models. Further models are plausible either because we are uncertain about the true structure our model should take (we don t fully understand the system), or because we have made pragmatic choices in order to simplify an otherwise complex model. We denote our notional set of buildable models as {M i, i I}, with M i = {η i (x (i) ), p i (X (i) )}. We draw here on Bernardo & Smith s (1994) notion of describing a set of possible models as M closed or M open. A set of models, {M i, i I}, is described as M closed if we believe that one of the models in {M i, i I} is true, but we do not know which. Conversely, a set of models is described as M open if we do not believe that one of the models in {M i, i I} is correct. 4 Two current approaches to resolving structural uncertainty We discuss two approaches that have been proposed for resolving the problem of model structural uncertainty, the model averaging approach and the model discrepancy approach. 4.1 Model averaging - taking an M closed view In the model averaging approach to structural uncertainty we consider that our models {M i, i I} represent a set of plausible models, and that our best approximation of the true value of the target parameter, Ỹ, is some weighted mean value of the individual model outputs. The weighting process could simply consist of
Structural Uncertainty in Health Economic Decision Models 6 choosing the model from the set that we believe is best while discarding the rest, effectively placing all the weight on a single model. Or, we may want to more formally assess our beliefs about how likely the different models are, and weight the outputs by these probabilities. If we have data, D, and can calculate some measure of the adequacy of the model, given the data, then we could weight the model outputs by (some function of) the adequacy measure. Within a Bayesian framework we would specify prior model probabilities, p(m i ), and calculate the posterior probabilities via p(m i D) = p(d M i )p(m i ) i I p(d M i)p(m i ), leading to a weighted mean output p(ỹ D) = i I p(ỹ M i, D)p(M i D). See Kadane and Lazar (2004) for a general discussion on this topic and Jackson et al (2009) for a more focussed discussion with respect to health economic decision model uncertainty. We recognise two problems with this approach. Firstly, the adequacy measure we construct relates only to the fit of the stochastic model elements of the economic model to the data; there is no assessment of the adequacy of the remaining structure of the model. Secondly, we may wish to consider competing models in the absence of data that would inform the choice between them. In this latter case we are left with just our prior model probabilities p(m i ), and our beliefs about Ỹ are simply p(ỹ ) = p(ỹ M i)p(m i ). i I Learning about p(m i ) might now be considered to be an expert elicitation problem, the extreme example being that of a modeller choosing a single model because they believe it to be best. Expert elicitation of prior model probabilities is not, however, a trivial problem. For example, how would an expert decide how much probability to place on two competing Markov models, one with three health states, and one with four?
Structural Uncertainty in Health Economic Decision Models 7 In a variation on the model averaging approach, Bojke et al (2009) have proposed a method in which the uncertainty in model structure is explicitly parameterised. A single general model is constructed in such a way that the models in {M i, i I} are just special cases of this general model. The parameters in the general model that reflect the structural modelling choices are considered to be just additional uncertain quantities along with other unknown inputs. Averaging over the distributions on the whole set of uncertain parameters is equivalent to model averaging as described above, but Bojke et al s approach does allow for more straightforward analysis of the value of reducing structural uncertainty. If we are prepared to believe the distribution on our averaged output value p(ỹ ), then we are implicitly claiming a M closed view of the problem. If this is not a realistic assumption, then we might be unwise in accepting p(ỹ ). 4.2 Model discrepancy - taking an M open view In the model discrepancy approach to structural uncertainty we assume that none of the models in our set {M i, i I} is correct. Each model is only an approximation of reality, and we focus on the discrepancy, δ, between the output of a model and the true target value Ỹ = η(x) + δ. We view the set of models as M open and it no longer makes sense to specify p(m i ). So, instead of trying to meaningfully specify model weights, the key question is now: can we usefully say anything about p(δ)? Much of the literature relating to model discrepancy has been in fields relating to the computer modelling of complex physical systems (for examples and discussions see Kennedy and O Hagan (2001) and Craig et al (2001)). In the context of modelling physical systems it often makes sense to partition the target model output Y = {Y o, Y u } into a portion for which we have (noisy) observations z on Y o, and a portion for which we have no observations. For example, we may have historic observations on the output variable, and wish to predict future observations (forecasting), or we may have observations at a set of points in
Structural Uncertainty in Health Economic Decision Models 8 space and wish to predict values at locations in between (interpolation). Kennedy and O Hagan (2001) propose a method for fully accounting for the uncertainty in Ỹ, given z and uncertain inputs X, within a Bayesian framework. The method relies on the existence of the observations z against which to calibrate the model. The model itself is considered as just a black box, the internal structure of which is not examined. A more advanced approach to modelling discrepancy is presented in Goldstein and Rougier (2009) within the context of what the authors term reified modelling. This involves eliciting expert judgements about the possible effects of improvements to a model, for example by including additional inputs or modifying the model structure. An example is given involving a model of the Atlantic Ocean, in which judgements are made about the effects of enhancing the structure of the model by increasing the number of compartments used to represent the ocean. Although the resulting reified model is never actually built, judgements regarding the difference between the built model and the reified model are used to inform beliefs about the discrepancy between the built model and reality. The difficulty in health economics with the discrepancy approach as described above is the lack of observations on the output Y. We do not directly measure incremental cost effectiveness ratios or net benefits; calibration of the model output against data is usually not possible. 5 Discussion and future direction We have identified problems in both the model averaging and model discrepancy approaches to resolving the problem of structural uncertainty within health economic models. In this section we outline one possible future direction for research. Firstly, we take the view that in the health economic context it is not safe to assume that one of our buildable models is necessarily correct. We therefore must, at least sometimes, take an M open view of the problem. Apart from in well understood and well defined problems, model averaging will not lead to p(ỹ ) that accurately reflects our true structural uncertainty.
Structural Uncertainty in Health Economic Decision Models 9 Secondly, we accept that we will almost never have adequate data against which to calibrate the output of a health economic decision model. In theory, a potential solution to this deficiency in data would be to elicit the views of a range of experts on the nature of the discrepancy Ỹ η(x). However, we think that it is unlikely that experts would have anything but a very qualitative sense of this. One possible direction for resolving the problem would be to retain an M open perspective, but instead of viewing the model as a black box and specifying the discrepancy at the level of the model output, open up the box and utilise the information contained within. If we can first decompose the model into a series of linked sub-units, we may be able to usefully specify discrepancy at the sub-unit level. Decomposing the model into sub-units exposes a set of intermediate parameters, W = (W 1,..., W n ) which, under certain decompositions will have clear real-world interpretations. If we have data relating to some subset of the intermediate parameters then we can partition W into observed and unobserved, W = {W o, W u }. Conditional on the observations z on W o, inference about the true output Ỹ could then be based on the joint distribution of {X, W o, W u, Ỹ }. Under many circumstances these calibration data will be absent, but it may still be possible to learn about the discrepancies between the modelled intermediate parameter values and their true values through elicitation. An expert may be able to meaningfully state their beliefs about the (joint distribution of the) discrepancies, δ i = W i W i, (i = 1,..., n), even if they cannot say anything about the discrepancy in the output Ỹ η(x). We will describe the proposed method in detail and present a case study in a future paper.
Structural Uncertainty in Health Economic Decision Models 10 References Bernardo, J. M. and Smith, A. F. M. (1994) Bayesian Theory. Chichester, John Wiley. Bojke, L., Claxton, K., Sculpher, M. and Palmer, S. (2009) Characterizing Structural Uncertainty in Decision Analytic Models: A Review and Application of Methods. Value in Health, 12, 739-749. Briggs, A. (2000) Handling Uncertainty in Cost-Effectiveness Models. Pharmacoeconomics, 17, 479-500. Craig, P. S., Goldstein, M., Rougier, J. C. and Seheult, A. H. (2001) Bayesian forecasting for complex systems using computer simulators. Journal of the American Statistical Association, 96, 717-729. Draper, D. (1995) Assessment and propagation of model uncertainty (with discussion). Journal of the Royal Statistical Society, Series B, 57, 45-97. Goldstein, M. and Rougier, J. C. (2009) Reified Bayesian modelling and inference for physical systems (with discussion and rejoinder). Journal of Statistical Planning and Inference, 139(3), 1221-1239. Jackson, C. H., Thompson, S. G. and Sharples, L. D. (2009) Accounting for uncertainty in health economic decision models by using model averaging. Journal of the Royal Statistical Society, Series A, 172(2), 383-404. Kadane, J. B. and Lazar, N. A. (2004) Methods and criteria for model selection. Journal of the American Statistical Association, 99, 279-290. Kennedy, M. and O Hagan, A. (2001) Bayesian calibration of computer models (with discussion). Journal of the Royal Statistical Society, Series B, 63, 425-464.