Linking data and model quality in macromolecular crystallography. Kay Diederichs

Transcription

1 Linking data and model quality in macromolecular crystallography Kay Diederichs

2 Crystallography is highly successful Can we do better?

3 Error in experimental data Error = random + systematic Multiplicity of n reduces the Random = counting + detector random error by n Systematic = Radiation damage + absorption + nonmultiplicity does not necessarily linearities + vibrations + reduce the systematic error! instabilities + If you don t have good data, then you have no data at all. -Sung-Hou Kim If you don t have good data, then you must learn statistics. - James Holton

4 Crystallographic statistics - which indicators are being used? Data R-values: Rpim < Rmerge=Rsym < Rmeas n n 1 /n 1 I i ( hkl ) I ( hkl ) R pim= hkl i= 1 n I i ( hkl ) hkl i= 1 I i ( hkl ) I ( hkl ) R merge = hkl i= 1 n R meas = I i ( hkl ) hkl n n I ( hkl ) I ( hkl ) n 1 i= 1 i n I i (hkl ) hkl i=1 hkl i= 1 Model R-values: Rwork/Rfree F obs (hkl ) F calc (hkl ) R work / free= hkl F obs (hkl ) hkl I/σ (for unmerged or merged data!)

5 In search for a proxy

6 Decisions and compromises Which high-resolution cutoff for refinement? Higher resolution means better accuracy and maps But: high resolution yields high Rwork/Rfree! Basic questions: what to optimize? Is the data/model R-value a good predictor/indicator of model quality? How good need the data be; to what extent do the data influence the refinement result? What to refine, and when to stop? Overfitting? Which datasets/frames to include into scaling? Optimize consistency (Rmeas) or quality of merged data (Rpim, <I/σ>)? Systematic differences/errors? Completeness versus radiation damage? Anomalous signal (CCano)? Reject negative observations or unique reflections? The reason why this is difficult to answer R-value questions is that no proper mathematical theory exists that uses absolute differences; concerning the use of R-values, Crystallography is disconnected from mainstream Statistics

7 Conflicting views An appropriate choice of resolution cutoff is difficult and sometimes seems to be performed mainly to satisfy referees... Ideally, we would determine the point at which adding the next shell of data is not adding any statistically significant information... Rmerge is not a very useful criterion. P. Evans (2011) An introduction to data reduction: space-group determination, scaling and intensity statistics. Acta Cryst. D67, 282 "At the highest resolution shell, the Rmerge can be allowed to reach 30 40% for low-symmetry crystals and up to 60% for high-symmetry crystals, since in the latter case the redundancy is usually higher." A. Wlodawer, W. Minor, Z. Dauter and M. Jaskolski (2008) Protein crystallography for non-crystallographers, or how to get the best (but not more) from published macromolecular structures. FEBS J. 275, 1

8 2010 PDB depositions

9 The asymptotic behaviour of model and data R-values is different at high resolution Data R-values (Rpim, Rmerge, Rmeas): with resolution, go to infinity since the numerator is constant (determined by background), and the denominator approaches zero Model R-values (Rwork/free): the ratio of numerator and denominator approaches a constant for a random or wrong model (Wilson 1950) This means that at high resolution, a quantitative relation cannot exist between model and data R-values!

10 Crystallographic reasoning 1. Better data allow to obtain a better model 2. A better model has a lower Rfree, and a lower Rfree-Rwork gap 3. Comparison of model R-values is only meaningful when using the same data 4. Taken together, this leads to the paired refinement technique : compare models in terms of their R-values against the same data.

11 Example: Cysteine DiOxygenase (CDO; PDB 3ELN) re-refined against 15-fold weaker data Rmerge Rpim Rfree Rwork I/sigma

12 Is there information beyond the conservative hi-res cutoff? Paired refinement technique : refine at (e.g.) 2.0Å and at 1.9Å using the same starting model and refinement parameters since it is meaningless to compare R-values at different resolutions, calculate the overall R-values of the 1.9Å model at 2.0Å (main.number_of_macro_cycles=1 strategy=none fix_rotamers=false ordered_solvent=false) ΔR=R1.9(2.0)-R2.0(2.0)

13 Do the maps really get better? cooperation with JCSG: Henry van den Bedem, Ashley M. Deacon, Abhinav Kumar identified 5 structures where re-processing of existing raw data permits to extend the resolution by another Å, with full completeness reprocessing with XDS/MOSFLM; refinement with refmac/phenix.refine calculate real-space CC of map and model, using EDSTATS (CCP4) or phenix.resolve

14 Example 1/5

15 Summary I Rmerge should no longer be considered as useful for deciding on a high-resolution cutoff The paired refinement technique can prove that data should be used to higher resolution than a (Rmerge-based) conservative cutoff suggests

16 In search for a proxy

17 measuring data quality with a correlation coefficient Correlation coefficient has clear meaning and well-known statistical properties Significance of its value can be assessed by Student's t-test (e.g. CC>0.3 is significant at p=0.01 for n>100; CC>0.08 is significant at p=0.01 for n>1000) Apply this idea to crystallographic intensity data: use random half-datasets CC1/2 (called CC_Imean by SCALA/aimless, now CC1/2 ) From CC1/2, we can analytically estimate CC of the full dataset against the true (usually unmeasurable) intensities using 2 CC 1/ 2 CC*= 1+CC 1/2 (Karplus and Diederichs (2012) Science 336, 1030)

18 Data CCs CC1/2 CC* Δ I/sigma

19 What is a suitable resolution cutoff when using CC1/2? Results from Student's t-test tell statistical significance (p=0.001 marked w/ * in CORRECT.LP/XSCALE.LP) Data are usually insignificant if CC1/2 < 0.15 It depends on refinement program if it can make good use of weak data; in practice a bit higher cutoff is best (remember, for CCanom a value of 0.3 proved useful) Operational definition: resolution should be cut such that the best model results use paired refinement!

20 Model CCs We can define CCwork, CCfree as CCs calculated on Fcalc2 of the working and free set, against the experimental data CCwork and CCfree can be directly compared with CC* CC* Dashes: CCwork, CCfree against weak exp tl data Dots: CC work, CC free against strong 3ELN data

21 Quantitative relation between data and model CCs Refinement should make CCwork converge towards CC* (from lower values) Inadequate model, or systematic errors in data processing: CCwork remains < CC* If CCwork > CC*: the model is closer to the data, than the truth is to the data : overfitting Error Exp l data True data l e d o m First Distance to truth

22 Summary II CC1/2 assesses the statistical significance of data 2 CC 1/ 2 CC*= 1+CC 1/2 predicts the agreement of experimental data and true data; CC* is the upper limit for the CCwork /CCfree model quality indicators CC1/2, CC*, CCwork/CCfree table can be obtained from Phenix distribution; the routine is called phenix.cc_star

23 Four new concepts for improving crystallographic procedures Image courtesy of P.A. Karplus

24 Further uses of the paired refinement technique

25 Does merging with weak data improve the resulting data? dataset name CDO3 CDO4 CDO5 CDO3+4 CDO3+5 CDO4+5 CDO3+4+5 # observations (10837) (2389) (10838) (13657) (21717) (13655) (24518) # unique reflections (2008) (1316) (1982) (2013) (2013) (1995) (2013) completeness [%] 99.9 (98.7) 93.8 (79.3) 95.7 (98.0) 99.9 (98.8) 99.9 (98.8) 95.7 (98.0) 99.9 (98.8) Rmeas [%] 10.2 (294.0) 23.1 (431.1) 26.0 (395.6) 15.0 (314.7) 15.9 (332.6) 26.0 (401.4) 15.9 (339.8) <I/σ> (0.64) (0.21) 9.88 (0.19) (0.69) (0.87) (0.49) (0.91) CC1/2 in highest shell; # of pairs ; ; ; ; ; ; ; 2008 CC* in highest shell

26 Refinement: which data give the best model? dataset name CDO3 CDO4 CDO5 CDO3+4 CDO3+5 CDO4+5 CDO3+4+5 model CDO3 0,1855, CDO , CDO , CDO , CDO , CDO4+5 CDO , , , , , , , , , , , , ,

27 Summary III Data R-values of merged data do not in any useful way predict the refinement outcome CC and CC* do 1/2

28 Does removal of negative reflections improve the data? All data Only positive observations Only positive unique reflections

29 Why reject negative observations or unique reflections? Negative observations/unique reflections increase the data R-value Assuming that data R-values are a predictor of data quality, rejecting them should improve model quality Make the data look better to reviewers

30 Does removal of negative reflections improve the model? Results from paired refinement R-value against Refinement against All reflections Only positive unique reflections Only positive observations All reflections , , , Only positive unique , , , Only positive observations , , ,

31 Summary IV Throwing away negative observations / unique reflections beautifies the data R-values and <I/σ>, but lowers CC1/2 and produces worse models as shown by the paired refinement technique!

32 Acknowledgement Andy Karplus, Oregon State University (Corvallis, OR) References P.A. Karplus and K. Diederichs (2012) Linking Crystallographic Data with Model Quality. Science 336, K. Diederichs and P.A. Karplus (2013) Better models by discarding data? Acta Cryst D, in the press see also: P.R. Evans (2012) Resolving Some Old Problems in Protein Crystallography. Science 336,

33 Thank you! If you would like to obtain a PDF of the slides, send to kay.diederichs@uni-konstanz.de

34 Checking your imagination! Which data are better? Rmerge=80%, multiplicity=4 or Rmerge=100%, multiplicity=8? Model A is refined at 2.0 Å, Rwork/Rfree=18.0/22%, model B at 2.2 Å, Rwork/Rfree=16.0/20.5%. a) Which model is better? b) If you don't know the answer: which procedure to decide? Can one compare R-values that do not refer to the same sets of reflections? Can one estimate model error by formulas that disregard experimental errors (sigmas)? Can a model give Fcalc2 that are closer to the true Fcalc2 than the Iobs are against which it was refined? Why is there a gap between CC* and CCwork at low res (only)?

35 General references Data-collection strategies Z. Dauter (1999) Acta Cryst. D55, Integration of macromolecular diffraction data A.G.W. Leslie (1999) Acta Cryst. D55, An introduction to data reduction: space-group determination, scaling and intensity statistics P. R. Evans (2011) Acta Cryst. D67,

36 Outlook anisotropy: elliptic cutoff or use all data? a refinement program that does not use experimental sigmas suffers from noise amplification, and benefits from throwing away data in the weak directions phenix.refine does not use the experimental sigmas; Refmac and CNS do.

37 How to obtain good data, given a total dose Old/classic/ conservative better Statistics Rmerge CC1/2 Exposure Reflections visible; tolerate some overloads Low-dose, high multiplicity Oscillation range 1 Pilatus: CCD: Rotation range strategy, xplan 180 /360 native/anomalous