CREATE Research Archive Research Project Summaries 2009 Expert Judgment Elicitation Methods and Tools Stephen C. Hora University of Southern California, hora@usc.edu Follow this and additional works at: http://research.create.usc.edu/project_summaries Recommended Citation Hora, Stephen C., "Expert Judgment Elicitation Methods and Tools" (2009). Research Project Summaries. Paper 40. http://research.create.usc.edu/project_summaries/40 This Article is brought to you for free and open access by CREATE Research Archive. It has been accepted for inclusion in Research Project Summaries by an authorized administrator of CREATE Research Archive. For more information, please contact gribben@usc.edu.
Expert Judgment Elicitation Methods and Tools Stephen C. Hora, University of Southern California (formerly University of Hawaii at Hilo) hora@usc.edu 1. Expert Judgment Elicitation Methods and Tools...1 2. Publications...3 3. Presentations - Outreach...4 1. Expert Judgment Elicitation Methods and Tools This project focuses on the development of methods for assessing probabilities from subject matter experts. Such probabilities and probability distributions are often central to the quantification of risk models and play an indispensable role in risk assessment and resource allocation in uncertain environments. The work for 2008-2009 entailed two distinct research topics. The first of these topics dealt with assessment of probabilities for split fraction models while the second projected examined alternative methods for combining the judgments of experts. Split fractions assessed as random variables present a special challenge in probability elicitation. The problem was first encountered as part of the NBACC effort to quantify risks from terrorist use of biological agents. Four approaches for eliciting information about split fractions have been developed as part of this effort. The first approach entails using a Dirichlet density as a model of expert judgment. A second approach uses bifurcation resulting in a more general joint density over the split fraction but containing the Dirichlet density as a special case. A bifurcation tree resulting in six termini is shown in Figure 1. The task of assessing a five-variate density is exchanged for the simpler task of assessing five univariate densities. These first two approaches have advantages and disadvantages. 1 - q 1 1 - q 2 q 1 q 2 1 - q 3 1 - q 4 q 3 1 - q 5 q 4 q 5 x 1 x 2 x 3 x 4 x 5 x 6 Figure 1. Bifurcation Tree
A third approach that employs several independent Dirichlet densities was developed integrating some aspects of the first two approaches while mitigating some of their disadvantages. This approach employs a cascade on multiple Dirichlet densities as shown in Figure 2. q 1 1 - q 1 -q 2 q 11 1-q 11 1 - q 21 -q 22 q 21 q 2 q 111 1 q 111 -q 112 q 22 q 112 x 1 x 2 x 3 x 4 x 5 x 6 x 7 1 Figure 2. Cascaded Dirichlet Densities 7 x i i= 1 A fourth approach employs a bottom up assessment method based on bifurcation that is functionally similar to the second approach but entails elicitation of terminal branch means and variances rather than means and variances at bifurcation points in the tree. This approach eliminates the possibility of packing or ordering bias from influencing. This work is essentially complete and ready for publication. A second thrust has been the development of mathematical statistical tools to examine the performance of various methods of aggregating judgments from multiple SMEs. The methods employ infinite sequences of judgments from a fixed number of experts. The judgments are modeled to have given properties such as good calibration or, alternatively, overconfidence or underconfidence, varying amounts of information representing SMEs with varying levels of expertise, and varying levels of dependence among the experts. The performance of an aggregation is measured in terms of the spread of the aggregated density, the calibration of that density, and the expected score from a strictly proper scoring rule such as the Brier score. As an example, Figure 3 demonstrates the impact of inter-expert dependence upon calibration with 3 and 6 experts and three aggregation methods: arithmetic aggregation, geometric averaging (alpha = 1/m), and likelihood function method (alpha = 1). The horizontal axis is the dependence and the vertical axis is the miscalibration score where 0 is ideal and higher numbers mean poorer calibration. This work is completed and will appear in Operations Research in 2010. Page 2 of 4
Work is continuing along this path as the general expressions developed so far allow the consideration of differential weighting of experts based on performance on test questions. Tools will be developed that allow for the quantification of expertise, calibration and dependence to be used to develop optimal weights for multiple experts. Calibration m = 3, 6 Well Calibrated Dependent Experts 0.1 0.09 Geometric m = 6, alpha = 1 0.08 0.07 Geometric m = 3, alpha = 1 0.06 L2-norm 0.05 0.04 0.03 Geometric m = 3, alpha = 1/m Geometric m = 6, alpha = 1/m 0.02 Arithmetric m = 6 0.01 Arithmetric m = 3 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Expert Correlation Figure 3: Calibration as a Function of Interdependence Among SMEs 2. Publications CREATE PUBLICATIONS Hora, Stephen - University of Hawaii at Hilo 1. Hora, S. An Analytic Method for Evaluating the Performance of Aggregation Rules for Probability Densities, Management Science, submitted for publication, 2009 Page 3 of 4
3. Presentations - Outreach 1. Hora, S., The Elicitation of Densities over Split Fractions, presentation for CREATE, University of Southern California, Los Angeles, CA, 9-16-09 2. Hora, S., Probability Elicitation Training, workshop for CREATE, University of Southern California, Los Angeles, CA, 11-12-09 and 11-19-09 Page 4 of 4