Bayesian Melding. Assessing Uncertainty in UrbanSim. University of Washington

Transcription

1 Bayesian Melding Assessing Uncertainty in UrbanSim Hana Ševčíková University of Washington Joint work with Paul Waddell and Adrian Raftery University of Washington UrbanSim Workshop, Jan 6-7, 2005 p.1

2 What do we want? Express uncertainty about model output quantities of policy interest. UrbanSim provides point predictions. A lot of uncertainty enters the simulation. Predictions in the form of probability ranges may also be useful. Takes the form of a (predictive) probability distribution of the (future) quantity of interest. Method: Bayesian Melding UrbanSim Workshop, Jan 6-7, 2005 p.2

3 Bayesian Melding Goal: Put analysis of simulation models on a solid statistical footing. Initial motivation: Work for the International Whaling Commission Deterministic population dynamics model for estimating the natural rate of increase of bowhead whales. Raftery, Givens, Zeh, J. Amer. Statistical Assoc Poole, Raftery, J. Amer. Statistical Assoc UrbanSim Workshop, Jan 6-7, 2005 p.3

4 Formulation inputs Θ model M outputs Φ = M(Θ) Information sources: Prior distribution of inputs, q(θ) Data about outputs, which yield a likelihood L(Φ) = prob(data Φ) Quantities of interest: ψ = Ψ(Φ), a function of model outputs Posterior distribution of inputs: p(θ) prior likelihood = q(θ)l(φ) Can be translated into a posterior distribution of outputs. UrbanSim Workshop, Jan 6-7, 2005 p.4

5 Computing the Posterior Distribution (for a deterministic model) 1. Draw a sample {Θ 1,..., Θ I } of values of the inputs from the prior q(θ). 2. For each Θ i, run the model to obtain Φ i = M(Θ i ). 3. Compute weights w i = L(Φ i ) 4. Result: An approximate posterior distribution of inputs with values {Θ 1,..., Θ I } and probabilities proportional to {w 1,..., w I }. 5. The approximate posterior distribution of the quantities of interest has values {ψ 1,..., ψ I } where ψ i = Ψ(Φ i ), and probabilities proportional to {w 1,..., w I }. UrbanSim Workshop, Jan 6-7, 2005 p.5

6 Application to UrbanSim Model has a stochastic component. Quantifiable uncertainty: between-coefficient variation stochastic variation (different results with different seeds) other sources of uncertainty UrbanSim Workshop, Jan 6-7, 2005 p.6

7 Application to UrbanSim (cont.) Modification of Bayesian Melding: 1. Draw a sample {Θ 1,..., Θ I } from q(θ). 2. For each Θ i, run the model J times with different seeds to obtain Φ ij, j = 1,..., J. 3. Compute weights w i = L( Φ i ) where Φ i = 1 J J j=1 Φ ij. 4. Result: An approximate posterior distribution of inputs with values {Θ 1,..., Θ I } and probabilities proportional to {w 1,..., w I }. 5. ψ now has a distribution given Θ i, since the model is not deterministic. UrbanSim Workshop, Jan 6-7, 2005 p.7

8 Illustrative Example Model: Household Location Choice Model (HLCM). Test database: Eugene, OR in We have observed the number of households in each zone in 1994, y 1,..., y K (K = 295). Goal: Prediction for Inputs: HLCM logit parameters Θ. Prior on inputs: MVN( ˆΘ, diag(se( ˆΘ) 2 )) where ˆΘ is the estimator of Θ. Outputs: Number of households per traffic zone, Φ 1,..., Φ K. Several models not run (e.g. Developer, ELCM) = faster, but less realistic. UrbanSim Workshop, Jan 6-7, 2005 p.8

9 Bayesian Melding Model We are interested in w i p(y Θ i ) = K k=1 p(y k Θ i ) based on Φ ijk = µ ik + δ ijk, where δ ijk iid N(0, σ 2 δ ) (y k Θ = Θ i ) = µ ik + a + ɛ ik, where ɛ ik iid N(0, σ 2 ɛ ) Estimation of µ ik, σ 2 δ, σ2 ɛ, and a done by approximate maximum likelihood. Yields a predictive distribution of our quantity of interest. UrbanSim Workshop, Jan 6-7, 2005 p.9

10 Analysis of the data from 1994 Run the model for Θ 1,..., Θ 10, so I = 10, and also we choose J = 10. Square root transformation stabilizes the variances. Within-coefficient variance ˆσ 2 ɛ is much larger than the between-coefficient variance ˆσ 2 δ. ˆσ 2 δ = 0.04, ˆσ2 ɛ = (may be partly artificial due to our not running several models) UrbanSim Workshop, Jan 6-7, 2005 p.10

11 Simulated vs. observed data High correlation, ρ = rho = , rmse = 3.64 data simulated households per zone UrbanSim Workshop, Jan 6-7, 2005 p.11

12 Predictive distribution for 2000 Predictive distribution of ψ k = number of households in zone k in This is a mixture of normal distributions, one for each simulated value of the inputs, Θ i : p(ψ k ) = I i=1 w i N(âb a + ˆµ 2000 ik, (ˆσ ɛ 2 + ˆσ2 δ J )b v), k = 1,..., K We set the propagation factors to b a = b v = 20/14 = ( ). UrbanSim Workshop, Jan 6-7, 2005 p.12

13 Multiple runs Bayesian melding mean=26.2,sd=0.1,obs=30.3 mean=26.7,sd=4.5,obs= probability density households in zone 57 obs=30.3(red), start=23.7(green) households in zone 57 (BM marginal distr.) obs=30.3(red), start=23.7(green) UrbanSim Workshop, Jan 6-7, 2005 p.13

14 Coverage of 90% Predictive Interval Coverage = proportion of zones for which the truth fell in the interval Should be close to Actual coverage: Fixed coefficients: 0.10 Variable coefficients: 0.11 Bayesian melding: 0.88 Simulations with fixed and variable coefficients greatly underestimate uncertainty. Bayesian melding is well calibrated. UrbanSim Workshop, Jan 6-7, 2005 p.14

15 Verification rank histogram Assesses how well the predictive distribution is calibrated. Perfect calibration = uniform histogram. Fixed coeff. Variable coeff. Bayesian meld. Density Density Density observation ranking within the simulated values observation ranking within the simulated values observation ranking within the simulated values Bayesian melding much more uniform than others. UrbanSim Workshop, Jan 6-7, 2005 p.15

16 Verification rank histogram CDF Perfect calibration = CDF is on diagonal line. Fixed coeff. Variable coeff. Bayesian meld. F(ranking) F(ranking) F(ranking) ranking ranking ranking Fixed and variable coefficients poorly calibrated. Bayesian melding much better calibrated. UrbanSim Workshop, Jan 6-7, 2005 p.16

17 Aggregated quantities of interest Households within 15 minutes travel time from CBD. Variable coefficients Bayesian melding mean= , sd= 17.8, obs= 65862, ci = [ 61220, ] mean= , sd= , obs= 65862, ci = [ 64741, ] Households in 15 minutes distance from cbd (181 zones) Households in 15 minutes distance from cbd (181 zones) UrbanSim Workshop, Jan 6-7, 2005 p.17

18 Future work Running all models. Incorporating uncertainty about: coefficients of ELCM, Developer, Land price,... migration flow relocation rates Spatial correlation of model errors. What to do, if there is no data about outputs. UrbanSim Workshop, Jan 6-7, 2005 p.18

19 Summary Goal is to assess uncertainty about future quantities of policy interest. Fixed coefficients (varying seeds) and variable coefficients simulations seem to underestimate uncertainty (results only illustrative). Bayesian melding is a statistical method to get valid predicitve distribution based on model runs. Experiment on Eugene, OR, : Bayesian melding provided fairly well calibrated predictive distribution. UrbanSim Workshop, Jan 6-7, 2005 p.19