Physician Performance Assessment / Spatial Inference of Pollutant Concentrations

Transcription

1 Physician Performance Assessment / Spatial Inference of Pollutant Concentrations Dawn Woodard Operations Research & Information Engineering Cornell University Johns Hopkins Dept. of Biostatistics, April

2 Outline 1 Physician Performance Assessment Performance Metrics 2 Spatial Inference of Pollutant Concentrations Statistical Approaches for Pollutant Estimation Bayesian Moving-Average Models Application to Nitrates Data Conclusions and Future Work 2

3 Physician Performance Assessment Joint work with A. Gelfand, B. Barlow, J. Elmore, and the Breast Cancer Surveillance Consortium There is concern about large differences in false positive and false negative rates between radiologists in screening mammography Database of 500,000+ mammograms Demographic characteristics of the patient Outcome of the mammogram (false +, false -, true +, or true -) Radiologist surveys Demographic & practice characteristics Level of concern about malpractice 4

4 Physician Performance Assessment Goal: assess physician performance while accounting for: 1. Differences in patients (case mix) 2. Differences in sample size (e.g. few cancer cases for some radiologists)?. Differences in radiologist attributes 5

5 Physician Performance Assessment Can adjust for case mix (e.g. Salem-Schatz et al. 1994) Can test whether a physician is significantly above or below average Tests invalid for small sample sizes Not clear how to compare one physician to another We build on Normand, Glickman, Gatsonis (1997): performance metrics for hospitals based on patient survival rate We extend to metrics for sens. & spec. of physicians We use a Bayesian hierarchical modeling approach to estimate and explain accuracy differences among radiologists 6

6 Modeling Accuracy Logistic regression: logit(s ij )=X ij β + τ i τ i = W i γ + φ i φ i N(0,ψ) S ij = sensitivity or specificity on mammogram i,j X ij = risk factors of patient i,j W i = attributes of radiologist i i = 1,...,I radiologists j = 1,...,n i mammograms of radiologist i with cancer present 7

7 Performance on a Hypothetical Patient Predict the sensitivity and specificity of each radiologist for a typical patient Or a high-risk or low-risk patient For a hypothetical patient with attributes X 0, the measure is S(X 0,β,τ i )=logit 1 (X 0 β + τ i ) 9

8 Performance on a Hypothetical Patient Sensitivity and specificity on a typical patient: Sensitivity (%) Specificity (%) Radiologist ID Radiologist ID These measures do not adjust for differences in radiologist attributes 10

9 Performance Relative to a Standard Alternatively, take the predicted average accuracy (sensitivity or specificity) of a particular radiologist on her patients: μ i = 1 n i Σ n i j=1 S(X ij,β,τ i ) Compare to that expected for a radiologist with the same attributes and case mix: Take μ i μ i μ i = 1 n i Σ n i j=1 S(X ij,β,w i ) S(X ij,β,w i )=E τ Wi {S(X ij,β,τ)} Performance is evaluated while adjusting for radiologist attributes 11

10 Performance Relative to a Standard Sensitivity Difference Specificity Difference Radiologist Index Radiologist Index Many radiologists had predicted specificity significantly above or below that expected; not so for sensitivity 12

11 Conclusion Bayesian modeling of patient-level sensitivity and specificity provides estimates of performance measures while fully accounting for uncertainty 13

12 Spatial Inference of Pollutant Concentrations Joint work with R. Wolpert and M. O Connell. Measurements of nitrates in groundwater have been obtained from wells in the mid-atlantic states (Ator 1998): > 8.3 mg/l mid range < 0.75 mg/l 15

13 Spatial Inference of Pollutant Concentrations Desire geographic interpolation of nitrate levels Distinct regulatory goals require inference at distinct, non-nested geographic scales... fine-scale, regulatory units (e.g. counties), hydrologic units (e.g. watersheds)....as well as distinct risk measures average nitrate concentration, probability of exceeding a threshold, averaged by region, maximum nitrate concentration occurring in each region. 16

14 Spatial Inference of Pollutant Concentrations We utilize a nonparametric spatial statistical model for nitrate concentrations at all locations Bayesian approach: uncertainty about the nitrate concentration and its average over various regions are all random variables......for which we can compute expected values (best overall estimates) and probabilities of exceeding specified thresholds 17

15 Existing Approaches When inference is desired at a single spatial partition (e.g. counties) or nested partitions, lattice models can be used. Kriging allows smooth spatial interpolation: models the pollutant concentration Λ(x) at x X as: log Λ(x) = JX X j (x)β j + Z (x) j=1 where Z (x) is a mean-zero Gaussian process. 19

16 Existing Approaches A kriged surface with only an intercept term β 0 : > 8.3 mid range < Latitude Longitude 20

17 Existing Approaches The confidence intervals are very wide in many locations, even where there is much data: Lower Bound: Upper Bound: > 8.3 mid range < > 8.3 mid range < Latitude Latitude Longitude Longitude 21

18 Moving-Average Models Ickstadt and Wolpert (1997) and Wolpert and Ickstadt (1998) introduced methods for interpolating intensities of spatial point processes by modeling the intensity Λ(x) as a moving average of an unobserved stochastic process The approach has been used in non-point-process applications: identifying proteins in mass spectroscopy (House, Clyde, and Wolpert 2006) spatio-temporal inference of sulfur dioxide air pollution (Tu 2006) We apply this model to obtain inferences of multiple risk measures, at multiple spatial scales 23

19 Moving-Average Models The concentration Λ(x) at location x X is modeled as: Λ(x) = JX X j (x)β j + j=1 MX k(x, s m )γ m m=1 for k(x, s) a kernel function on X S. The parameters s m are taken to be the centers of the mixture components, so that S = X The number M, locations s m, and magnitudes γ m > 0ofthe components are uncertain The ith measurement Y i is assumed to follow: log Y i N(log Λ(x i ),σ 2 ) 24

20 Moving-Average Models Interpretation of the spatial portion of the model, m k(x, s m)γ m, for pollutant level estimation: the pollutant surface is the sum of an unknown number of point sources with unknown locations and magnitudes......where the concentration decreases with distance from each source in a manner consistent with the kernel k(, ) 25

21 Moving-Average Models The kernel form is specified as: where d > 0 is a constant k(x, s) =exp j 1 ff x s 2 2d 2 Can be generalized to use unknown scale, eccentricity, and asymmetry For the nitrates analysis we do not include covariates, so β is not in the model 26

22 Prior Specification The spatial term in the model can be rewritten MX Z k(x, s m )γ m = m=1 S k(x, s)γ(ds) where MX Γ(ds) = γ m δ sm (ds) is a discrete measure on S. m=1 Γ is given a Lévy random field prior: Parameterized by a measure ν(dγ,ds) on R + S M Pois(ν + ) where ν + = ν(r + S) Conditional on M, (γ m, s m ) iid ν(dγ,ds)/ν + 27

23 Prior Specification We use the gamma random field on a bounded set S R 2 Its Levy density is ν(γ,s) =αγ 1 e ργ for α, ρ > 0 For A Swe have Γ(A) Ga(α A,ρ) In order for ν + <, must truncate ν(γ,s) by setting to zero for γ<ɛwhere ɛ>0 28

24 Prior Specification This prior implies that: The number of mixture components M satisfies: M Pois(α S E 1 (ρɛ)) where E 1 is the exponential integral function Conditional on M, the locations s m are independently uniformly distributed on S......and the magnitudes γ m are independently distributed according to the density f (γ) γ 1 e ργ 1(γ >ɛ) 29

25 Prior Specification These choices lead to prior surfaces Λ(x) like this one: Latitude Longitude 30

26 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 31

27 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 32

28 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 33

29 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 34

30 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 35

31 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 36

32 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 37

33 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 38

34 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 39

35 Prior Specification The areas with high concentrations have random (unknown) locations a priori: Latitude Longitude 40

36 Computation Computation is performed using reversible jump Markov chain Monte Carlo (Green 1995), wherein samples ω t of the parameter vector ω are obtained approximately from the posterior distribution Each iteration updates a single parameter, or adds, deletes, or updates a single mixture component A posterior estimate can be obtained for any function g(ω) of the parameters ω, since E[g(ω)] = lim T 1 T X g(ω t ) t T Ex: for the average of Λ(x,ω) P over x A, sample {a i } K i=1 uniformly in A and use the estimate 1 Λ(a TK i,ω t ) i,t 41

37 Nitrate Inferences The posterior mean of the concentration: Latitude > 8.3 mid range < Longitude 43

38 Nitrate Inferences The posterior standard deviation of the concentration: Latitude > 8.3 mid range < Longitude 44

39 Nitrate Inferences This is a measure of estimation uncertainty. Latitude > 8.3 mid range < Longitude 45

40 Nitrate Inferences Most areas with numerous measurements have low uncertainty. Latitude > 8.3 mid range < Longitude 46

41 Nitrate Inferences Average nitrate concentrations over counties: Latitude > 5 mg/l mid range < 1 mg/l Longitude 47

42 Nitrate Inferences The probability that the nitrate concentration exceeds the regulatory limit, averaged by county: Latitude > 8 % mid range < 2 % Longitude 48

43 Conclusions The Bayesian moving-average model allows inference of a variety of risk measures at a variety of spatial scales. Uncertainty measures are available for all these estimates. The model is nonparametric. It has a desirable interpretation in the context of pollutant level estimation. 50

44 Conclusions The moving-average model has a computational advantage over kriging for large data sets Likelihood evaluation for the moving-average model is O(NM), where N is the number of data points and M is the number of mixture components. Likelihood evaluation is O(N 3 ) for kriging. 51

45 Future Work Covariates such as climatic, geologic, and land use factors could be added. The fixed kernels could be replaced with kernels that have priors on the scale, eccentricity, and asymmetry. This would allow the model to capture, e.g., pollutant point sources that have spread out more in one direction than another due to flow patterns. 52

46 Published in Woodard, Gelfand, Barlow, and Elmore (2007, Statistics in Medicine) and Woodard, Wolpert, and O Connell (2009, JABES). More details (references, this talk in.pdf, related work) available at or on request from dbw59@cornell.edu 53