Effectve plots to assess bas and precson n method comparson studes Bern, November, 016 Patrck Taffé, PhD Insttute of Socal and Preventve Medcne () Unversty of Lausanne, Swtzerland Patrck.Taffe@chuv.ch
Outlne Bland & Altman s lmts of agreement method (1986) Extenson to proportonal bas and heteroscedastcty (1999) A new methodology to quantfy bas and precson Illustraton wth a smulated example
How to measure agreement between two measurement methods? Ex: blood pressure STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT J. Martn Bland, Douglas G. Altman Department of Clncal Epdemology and Socal Medcne, St. George's Hosptal Medcal School, London SW17 ORE; and Dvson of Medcal Statstcs, MRC Clncal Research Centre, Northwck Park Hosptal, Harrow, Mddlesex (Lancet, 1986; : 307-310) Statstcal methods for assessng agreement between two... www.ncb.nlm.nh.gov/pubmed/86817 3 by JM Bland - 1986 - Cted by 35451 - Related artcles Lancet. 1986 Feb 8;1(8476):307-10. Statstcal methods for assessng agreement between two methods of clncal measurement. Bland JM, Altman DG.
Bland & Altman (1986) : They wanted a measure of agreement whch was easy to estmate and to nterpret for a measurement on an ndvdual patent. An obvous startng pont was a plot of the dfferences versus the mean of the measurements by the two methods : 4
The bas (dfferental bas) between the two measurement methods s estmated by the mean dfference : bas mean 5
If the dfferences are normally dstrbuted, we would expect about 95% of the dfferences to le between the mean +- 1.96*SD, the so called lmts of agreement (LoA) (Bland & Altman, 1986): 6
The decson about what s acceptable agreement s a clncal one: We can see that the blood pressure machne (S) may gve values between 55mmHg above the sphygmomanometer (J) readng to mmhg below t, => such dfferences would be unacceptable for clncal purposes 7
However, these estmates are meanngful only f we can assume bas and varablty are unform throughout the range of measurement, assumptons whch can be checked graphcally: 8 => assumptons approxmatvely met
In some cases the varablty of the measurements ncreases wth the magntude of the latent trat (heteroscedastcty), as well as wth the mean dfference (proportonal bas): Plasma volume data (Bland & Altman, 1999) dfference: Nadler-Hurley - 6 10 14 18 60 80 100 10 140 average: (Nadler+Hurley)/ Plasma volume expressed n percentage of normal value: as measured by Nadler and Hurley 9
In ths case, a lnear regresson of the dfferences on the averages can be estmated along wth the LoA (Bland & Altman, 1999): Plasma volume data (Bland & Altman, 1999) dfference: Nadler-Hurley - 6 10 14 18 60 80 100 10 140 average: (Nadler+Hurley)/ dfference upper 95% LoA lnear predcton lower 95% LoA Plasma volume expressed n percentage of normal value: as measured by Nadler and Hurley 10
In that case, the LoA are more dffcult to nterpret (wdth not constant), and more mportantly, dfference: Nadler-Hurley - 6 10 14 18 Plasma volume data (Bland & Altman, 1999) 60 80 100 10 140 average: (Nadler+Hurley)/ dfference upper 95% LoA lnear predcton lower 95% LoA there are settngs where Bland & Altman s plots are msleadng! Indeed, we wll show that when varances of the measurement errors of the two methods are dfferent, Bland and Altman s plots may be msleadng 11
Smulated examples where the regresson lne shows an upward or a downward trend but there s no bas LoA mu_y1=9.61 mu_y=9.99 sg_y1=60.56 sg_y=34.69 sg_e1=6.5 sg_e=1.00 bas = 0.38 dfference: y1-y -0-10 0 10 0-10 0 10 0 30 average: (y1+y)/ dfference upper zero Lnear predcton lower dfference: y1-y -0-10 0 10 0 LoA mu_y1=10.33 mu_y=10.8 sg_y1=34.68 sg_y=69.1 sg_e1=0.95 sg_e=36.41 bas = -0.04 1 dfference: y1-y -0-10 0 10 0 LoA mu_y1=3.77 mu_y=4.05 sg_y1=60.90 sg_y=84.75 sg_e1=3.53 sg_e=30.44 bas = 0.8 0 10 0 30 40 50 average: (y1+y)/ 0 10 0 30 average: (y1+y)/ dfference upper zero Lnear predcton lower
or a zero slope and there s a bas LoA mu_y1=3.74 mu_y=3.73 sg_y1=101.30 sg_y=97.90 sg_e1=3.47 sg_e=3.93 bas = -0.01 dfference: y1-y -0-10 0 10 0 LoA mu_y1=5.99 mu_y=6.05 sg_y1=93.75 sg_y=9.50 sg_e1=5.0 sg_e=5.73 bas = 0.06 0 10 0 30 40 50 average: (y1+y)/ dfference: y1-y -0-10 0 10 0 0 10 0 30 40 50 average: (y1+y)/ 13
Therefore, the goal of my presentaton s to ntroduce a new methodology for the evaluaton of the agreement between two methods of measurement, where the frst s the reference standard and the other the new method to be evaluated: Effectve plots to assess bas and precson n method comparson studes Patrck Taffé Insttute for Socal and Preventve Medcne, Unversty of Lausanne, Swtzerland Patrck.Taffe@chuv.ch Accepted September 015, onlne October 016 14
More specfcally, the objectves of ths new methodology are to dentfy and quantfy the amounts of dfferental and proportonal bases, develop a method of recalbraton n order to correct the bas of the new measurement method, and compare ts precson wth that of the reference standard. The methodology requres several measurements by the reference standard and possbly only one by the new method for each ndvdual. It s applcable n all crcumstances wth or wthout dfferental and/or proportonal bases and when the measurement errors are ether homoscedastc or heteroscedastc. 15
Get ready! 16
The measurement error model.1 Formulaton of the model Consder the measurement error model: y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 j 1j 1j j 1 y = α + βx + ε, ε ~ N(0, σ ε j j j j j x f µ σ j ~ x( x, x ) 1 where y 1j be the jth replcate measurement by method 1 on ndvdual, j = 1,..., n and = 1,..., N, whereas y j s obtaned by method, x j s a latent varable wth densty f x representng the true unknown trat, and ε 1j and ε j represent measurement errors by method 1 and. 17
y1 j = α1+ β1xj + ε1 j, ε1 j ~ N(0, σ ε ( xj ; θ 1)) y = α + βx + ε, ε ~ N(0, σ ε j j j j j 1 x f µ σ ~ x ( x, x ) It s assumed that the varances of these errors,.e. σε ( x ; 1 j θ 1) and σ ( ; θ ), are heteroscedastc and depend on the level of the true unknown ε x j varable x j, as well as on the vectors θ 1 and θ of unknown parameters. 18
For the reference method, for nstance method, α = 0 and β = 1, whereas for method 1 the dfferental α 1 and proportonal β 1 bases have to be estmated from the data. The mean value of the latent varable x j s µ x and ts varance σ x. It s assumed that the latent varable s constant for ndvdual,.e. j (ths assumpton may be relaxed). x x When method s the reference standard and method 1 the new method to be evaluated, the model reduces to: y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 19
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 We have consdered a smple lnear relatonshp between y 1j and x to dentfy the dfferental and proportonal bases. It s possble, however, n our framework to consder nstead a non-lnear functon of x but n that case the bas no longer decomposes nto two components wth clear nterpretatons. 0
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 Nawarathna and Choudhary (Stat n Med, 015) estmate the parameters of ths model by bvarate maxmum lkelhood. Ther approach s complcated by the evaluaton of the ntegrals n the margnal lkelhood functon and requres specal numercal methods such as Laplace approxmaton or Gauss-Hermte quadrature. We have developed another more smple way to estmate ths model by a twostage procedure, whch performs effectvely as demonstrated by the smulaton study. 1
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1. Estmaton of the model In the frst stage, we estmate the regresson model for y j, by margnal maxmum lkelhood accountng non-parametrcally for the heteroscedastcty.
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 Then, we adopt an emprcal Bayes approach to predct x by the mean of ts posteror dstrbuton, whch s the best lnear unbased predcton (BLUP) for x : xˆ = E( x y ) = x f f y ( y x ) f ( x ) x ( y x ) f ( x ) dx y x dx 3
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 In the second stage, we proceed to the estmaton of the regresson equaton for y 1j and of the dfferental α 1 and proportonal β 1 bases smply by OLS after havng substtuted the BLUP x ˆ for the true unmeasured trat x. 4
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 * * Based on the estmates ˆ1 α and ˆ β 1 of the dfferental and proportonal bases one can compute an estmate of the bas of the new method: bas = ˆ α + xˆ ( ˆ β 1) * * 1 1 5
A very useful fgure to vsualze the bas of the new method (.e. method 1) s the bas plot. Bas plot y1 and y 0 0 40 60 80 Bas = 0% - 0 4 bas Bas = 10% 10 0 30 40 BLUP of x 6 y y1 Bas Reference standard New method
.3 Recalbraton of the new method To remove the dfferental and proportonal bases of the new method we proceed to ts recalbraton by computng: y = ( y ˆ α ) / ˆ β * * * 1j 1j 1 1 Now that y j and * y 1j are on the same scale we can compare the varances of the measurement errors to determne whch method s more precse. 7
We proceed to the comparson of the varances by makng a scatter plot of the estmated standard devatons ˆ σ ˆ ε ( x ; 1 θ 1) and ˆ σ ˆ ε ( x ; θ ) versus x ˆ, whch we call precson plot : Precson plot after recalbraton 4 6 8 10 standard devaton of the measurement errors 10 0 30 40 BLUP of x 8 estmated value true value (reference standard y) estmated value true value (new method recalbrated y1_corr)
.4 Why Bland and Altman s plot may be msleadng Bland and Altman have suggested to plot the dfferences Dj = y1 j yj versus the averages Aj = ( y1 j + yj ) /, and add to the plot the regresson lne of the relatonshp between D j and A j n addton to the LoA. The problem s that the regresson lne may show a postve or negatve slope when there s no bas or have a zero slope n the presence of a bas. The reason s related to the fact that n the regresson of D j on A j : D = α + βa + ε j j j A j cannot be consdered as beng exogenous t s, rather, endogenous. Plasma volume data (Bland & Altman, 1999) 9 dfference: Nadler-Hurley - 6 10 14 18 60 80 100 10 140 average: (Nadler+Hurley)/ dfference upper 95% LoA lnear predcton lower 95% LoA
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 OLS estmaton provdes unbased estmates only when: cov( A, ε ) 0 j j σ ( x ; θ ) ε 1 = = σ ε ( x ; θ ) β β 1 1.e. there s no bas whenever the varances of the measurement errors are strctly equal to the proportonal bas, a specal condton that has lttle chance to truly hold n practce 30
3 A smulaton study Extensve smulatons demonstrated that our methodology to assess bases, recalbrate the new method, and compare the precson of the two measurement methods performed very well for sample szes of 100 ndvduals and between 10 to 15 measurements per ndvdual by the reference standard and as few as only 1 by the new method. 31
For our smulatons we consdered the followng data generatng process: y x N x 1 = 4+ 1. + ε1, ε1 ~ (0,(1+0.1 ) ) j = + ε j, ε j ~ (0,(+0. ) ) y x N x x ~ Unform[10 40] where =,1,...,100 and the number of repeated measurements of ndvdual from the reference standard was n ~ Unform[10 15]. The new method has dfferental bas of -4 and a proportonal bas of 1.. The varance of the measurement errors from method 1 s smaller than that of the reference method. 3
The Bland and Altman LoA plot extended to the settng where there s heteroscedastcty of the measurement errors does not seem to ndcate any bas: LoA dfference: y1-y -40-0 0 0 40 10 0 30 40 50 60 average: (y1+y)/ 33
whereas the bas plot llustrates that the new method underestmates the trat up to and then overestmates t, thereby clearly llustratng the occurrence of dfferental and proportonal bases: Bas plot y1 and y 0 0 40 60 80-0 4 bas 10 0 30 40 BLUP of x y y1 Bas Reference standard New method 34
y = α + βx + ε, ε ~ N(0, σ ε 1j 1 1 1j 1j 1 y = x + ε, ε ~ N(0, σ ε j j j x f µ σ ~ x ( x, x ) 1 Actually, estmaton of the measurement error model allowed us to dentfy a dfferental bas of -3.85 95%CI= [-6.81; -0.88] (true value s -4) and a proportonal bas of 1.19 95%CI = [1.08; 1.9] (true value s 1.). 35
The varances of the measurement errors can already be well estmated wth 10~15 measurements by the reference standard and only 1 by the new method: Precson plot after recalbraton true value 4 6 8 10 standard devaton of the measurement errors estmated value 10 0 30 40 BLUP of x estmated value true value (reference standard y) estmated value true value (new method recalbrated y1_corr) 36
Fnally, the comparson plot allows us to vsualze the performance of our recalbraton procedure: Comparson of the methods measurement method 0 0 40 60 10 0 30 40 BLUP of x y y1 y1_corr Ftted values Ftted values Ftted values 37
We computed Bland and Altman LoA plot for the recalbrated method to llustrate that n the absence of bas the fgure may mslead the reader nto belevng that there s a bas: LoA for the recalbrated method dfference: y1-y -30-0 -10 0 10 0 0 10 0 30 40 50 average: (y1+y)/ 38
In summary, We have developed a new methodology to assess the bas and precson of a new measurement method relatve to the reference standard, whch does not have the shortcomngs of Bland and Altman s LoA methodology. It s, however, n sprt of the orgnal paper n the sense that new graphcal representatons of the bas and of the performance of the method to be evaluated are proposed. In addton, we have shown a very smple way to recalbrate the new method to be able to use t n place of the more complex and costly reference standard. 39
basplot: A Stata package to effectve plots to assess bas and precson n method comparson studes Patrck Taffé Insttute for Socal and Preventve Medcne, Unversty of Lausanne, Swtzerland Patrck.Taffe@chuv.ch Mngka Peng Department of Communty Health Scences, Unversty of Calgary, Canada Mngka.peng@ucalgary.ca Vck Stagg Calgary Statstcal Support, Canada Vck@calgarystatstcalsupport.com Tyler Wllamson Department of Communty Health Scences, Unversty of Calgary, Canada Tyler.wllamson@ucalgary.ca 40 Wll appear soon n the Stata Journal
Thank you for your attenton 41