( ) ( Statistical Equivalence Testing

Transcription

1 ( Downloaded via on November 1, 018 at 13:8: (UTC). See for option on how to legitimately hare publihed article. 0 BEYOND Gielle B. Limentani Moira C. Ringo Feng Ye Mandy L. Bergquit Ellen O. MCSorley GlaxoSmithKline the t-tet: Statitical Equivalence Teting One of the mot common quetion conidered by analytical chemit i whether replicate meaurement are the ame or ignificantly different from each other. The determination of ignificantly different reult can be ued to argue that a phenomenon i novel or to jutify a claim of a ignificant improvement in a technique, proce, or product. Science and technology are alo driven by determination of amene, uch a equivalence, control, or ruggedne. Given the variability inherent to mot intrument ytem, the quetion of whether a meaurable difference i real can be difficult to anwer. In ome cae, intuition, experience, and knowledge of the practical context Statitical equivalence teting can be ued to make better technical and buine deciion from analytical data. of the data can be ued to inpect or eyeball the data to ae whether a true difference exit. For example, mot of u would agree that a difference of 100% in a meaurement that typically exhibit a preciion of 1% i a real difference, and we would alo agree that a difference of 0.01% i not ignificant for the ame meaurement. But what about le clear-cut cae in which the difference between two et of data i imilar to the preciion? How much difference i too much? Not only doe imple inpection fail in thee circumtance, but a ubjective proce can be biaed, i difficult to jutify, and, mot importantly, can lead to the wrong concluion. JUNE 1, 005 / ANALYTICAL CHEMISTRY 1

2 Statitical hypothei teting offer a rigorou, objective approach to ditinguihing truly ignificant difference in meaurement from noie. Although many tet exit that are uitable for different ituation, the tatitical tet that i mot familiar i the imple-to-perform twoample t-tet (t i a probability ditribution that i cloely related to the tandard normal ditribution). However, the t-tet ha everal limitation and may not be the mot appropriate technique when the objective i to how equivalence between two data et. We will demontrate that the two one-ided t-tet (TOST) i a better option in many cae. The two-ample t-tet and TOST are ditinct approache for aeing a difference or equivalence in data; which tet i ued can have a ignificant impact on the outcome of the comparion a well a on the cientific and buine deciion made a a reult. The baic Mot analytical cientit are familiar with the mean and tandard deviation and how thee meaurement are ued to make imple data comparion. However, the calculated mean and tandard deviation value, y and repectively, are merely etimate of the real mean and tandard deviation value for a population of poible meaurement. For example, for a ample data et of n independent meaurement, y and can be calculated. Thee value are etimate of the mean and tandard deviation of the entire population of meaurement from which the ample wa taken. A more informative decription of the population mean i the range of probable true value, or confidence interval, for the mean, which i y + t(1 /, n 1) for a two-ided 100(1 )% confidence interval. Note that the width of the confidence interval increae a increae and n decreae. In other word, a data et for which i large (noiy data) or n i mall (few meaurement) reult in a wider confidence interval. (A narrow interval i more deirable.) The width of the confidence interval i alo determined by the Student t-value, known a t, for a given n and a given ignificance level, which i related to the probability that the confidence interval include the true mean. For mot analytical application, i typically et at 0.05, producing a 95% confidence interval. Student t-value may be obtained from publihed table or by uing the TINV(, n 1) function in Excel. (Note that for ome verion of Excel, it i neceary to ue in place of for thi function.) (a) (b) (c) (d) (e) n (1) 0 y 1 y FIGURE 1. Comparion of two-ample t-tet and TOST in term of confidence interval. The concluion for each cenario with a t-tet and TOST, repectively, would be (a) equal and equivalent, (b, c) equal but not equivalent, (d) not equal but equivalent, (e) not equal and not equivalent. The mean i not the only parameter that i etimated with error. The etimated, or meaurement preciion, can vary with the number of meaurement that are made, the ample that i meaured, and the manner in which the data are collected. With a 95% confidence limit and the aumption that the data follow a normal ditribution, the true tandard deviation for n = 10 can be a high a 1.8 the meaured, and thi error increae to 6.3 the meaured for n = 3 (1). Moreover, the meaurement repeatability (among meaurement taken by a ingle analyt in a ingle laboratory) can differ ignificantly from the meaurement reproducibility, which i a broader etimate of preciion baed on meaurement by multiple analyt in multiple laboratorie. In one tudy of everal method from a compendium for pharmaceutical, the mean analytical repeatability wa 1.5% RSD, while reproducibility wa etimated at nearly double that amount (). In the abence of a coniderable amount of experience with the meaurement under a range of condition, it may be neceary to ue a tatitical procedure for etimating the meaurement preciion. To conduct a tatitical analyi, the analyt mut conider the experimental objective and decide upon a null hypothei for the tet. An appropriate tatitical tet mut then be choen to prove that the null hypothei i fale. If ufficient evidence doe not exit to prove falene, the tet default to the concluion that the null hypothei i correct but doe not actually prove that it i correct. It i therefore critical to chooe a null hypothei that i the revere of the tatement the analyt wihe to prove. For example, if the analyt wihe to prove that the mean from two group of data are not equal, he or he hould chooe a null hypothei in which the mean are equal and then perform a tet to demontrate that thi hypothei i fale. In addition, the analyt mut determine how much rik of error i acceptable. In general, when a tatitical tet i conducted, two type of potential error can occur. The probability of a type 1 error, or, repreent the rik of rejecting the null hypothei when it i true. A type error, of probability, occur when the experimenter fail to reject the null hypothei when it i fale. Typical value are 5 10%, and typical value are 5 0%. After the analyt ha determined the appropriate error rik, it i uually neceary to etimate the ample ize required for the data comparion. The procedure for etimating the neceary n depend on everal factor. If i large (poor meaurement preciion) and and are mall (low rik of error i deired), the neceary n can be prohibitively large. In thee cae, it may be neceary to refine the meaurement ytem or revie the tudy deign before proceeding. On the other hand, a mall can reult A ANALYTICAL CHEMISTRY / JUNE 1, AMERICAN CHEMICAL SOCIETY

3 in a calculated required n of or 3. Although thee value may reult from tatitically valid calculation, the accuracy of the experimentally determined y and value i often very poor with mall n value, and the tet may reult in an incorrect concluion. It may be ueful to examine rik of error that are achievable with different n value and preciion o that the analyt can make the mot informed compromie between rik and laboratory efficiency. Once valid data have been obtained, the experimenter ha everal choice on how to proceed with the data evaluation, depending on the reearch objective. Two-ample t-tet The two-ample t-tet allow comparion of the mean value of two data et by the calculation of the tet tatitic T= y 1 in which y 1 and y are the mean value from group 1 and, p i an etimate of the pooled of the meaurement, and n 1 and n are the number of obervation for each group. The p of replicate et of meaurement i p = n1 1 1 n 1 + n + 1 n in which 1 and are the etimated value for each et of meaurement. The abolute value of the calculated T-value i then compared with the critical t-value for the elected ignificance level (obtained from tatitic table or by uing TINV in Excel). If the abolute value of the calculated T-value i greater than or equal to the critical t-value, then the data et are declared tatitically different. Thi tet can alo be performed by contructing a 100(1 )% confidence interval for the difference between two mean uing ( y 1 y ) + t ( 1 /, n + n ) p 1 1 n + n y 1 1 p n + n () (3) (4) and determining whether the reulting confidence interval contain 0. If the confidence interval doe not contain 0, then the mean are declared not equal. The null hypothei of the two-ample t-tet i that the mean value of the two data et are equal; thi place the burden on the analyt to prove that the mean value are in fact different. Although it i an appropriate tet for proving that two data et are different, problem arie when the two-ample t-tet i ued to how equivalence. Firt, the traditional two-ample t-tet can reward the analyt for having poor preciion and/or a mall n. Equation indicate that an increae in p or a decreae in n reult in a maller calculated T-value, which make it more difficult to declare that the mean value are not equal. In the abence of ubtantial evidence to conclude that the mean value are different, the analyt can mitakenly default to the hypothei that they are equal. Another problem aociated with the ue of the two-ample t-tet i that it may lead the analyt to conclude that a tatitically ignificant difference exit between the mean value when the magnitude of the difference i of no practical importance. Thi i a particular problem when the preciion of the meaurement i very good; a pot hoc explanation of tatitical ignificance may be required when the difference i of no practical importance. Therefore, the two-ample t-tet i not well uited for howing the equivalence of mean value from two group. Equivalence tet An alternative to the two-ample t-tet i TOST, deigned pecifically for bioequivalence teting of pharmaceutical product (3 6). It ha recently been expanded into broader application in pharmaceutical cience (1, 7 10), proce engineering (11, 1), pychology (13), medicine (14), chemitry (15), and environmental cience (16). TOST begin with a null hypothei that the two mean value are not equivalent, then attempt to demontrate that they are equivalent within a practical, preet limit; thi i conceptually oppoite to the two-ample t-tet procedure. Unlike the two-ample t-tet, TOST appropriately penalize poor preciion and/or mall n value and place the burden on the analyt to prove that the data et are equivalent. The deign of an equivalence tet can be challenging becaue the analyt mut define an acceptance criterion on the bai of prior knowledge of the meaurement a well a it intended application. Thi acceptance criterion i the limit outide which the difference in mean value hould be conidered practically and tatitically ignificant. The analyt then contruct a 100(1 )% confidence interval for the difference between the two mean value and compare it with. If the confidence interval i completely contained within the interval [, ], the mean value of the two data et are equivalent. The ue of etablihe a priori what level of difference i acceptable. Figure 1 diplay five data comparion and illutrate the different outcome that arie from uing the traditional two-ample t-tet and TOST. The center of each confidence interval i the difference between the oberved y value. The width of the interval, which depend on the meaurement preciion, repreent the range of plauible true difference in mean value between the data et. If thee interval were created with the traditional two-ample t-tet, for Figure 1a c the analyt would conclude that there i no difference between the mean value becaue the confidence interval include a difference of 0. The confidence interval in Figure 1d and 1e do not include a difference of 0; therefore, the mean value would be declared different. By contrat, if thee interval were created with TOST, the mean value of the two data et would be declared equivalent only for Figure 1a and 1d becaue thee confidence interval are completely contained within the range from to. The mean value for Figure 1d are declared equivalent even though the JUNE 1, 005 / ANALYTICAL CHEMISTRY 3 A

4 confidence interval doe not include 0, becaue the bia repreented by the difference in mean i mall and within the interval [, ]. The confidence interval in Figure 1b and 1c are too wide for the mean value of the data et to be declared equivalent. Making it eay What i an acceptable difference between the mean value of two data et? Chooing an appropriate can be a challenge. It mut be greater than n, let the tet fail imply becaue of impreciion rather than becaue of a true difference. However, mut alo be le than any pecification or tandard that the teting i deigned to challenge, or the tet become too eay and will not adequately dicriminate. Although ome tatitical oftware package include TOST (17), our dicuion provide a tep-by-tep proce for performing equivalence teting with a preet with Excel or other commonly available computational oftware package. The firt parameter that mut be pecified before an analyt perform tatitical teting i, the abolute value of the true difference between the group mean value; i a hypothetical value uch that if the abolute value of the oberved difference i no more than, there i a trong probability of concluding that the two data et repreent equivalent reult. An acceptable level of bia can be conidered and included in ; however, the choice of a nonzero can decreae the ability of the tet to ditinguih a mall but important difference between data et. In addition, in the abence of extenive data, no objective bai exit for chooing a nonzero. Therefore, the mot conervative approach i to aign a value of 0 to. The econd tep i determining the n value needed for the tet. Becaue the required n for TOST i related to and to other parameter dicued earlier, it may be helpful at thi early tage to aume a range of potential value a a fraction of the pecification or tandard that the tet i deigned to challenge or a a multiple of ; can be refined later. Then, with thi erie of potential value, along with value for,,, and, the relationhip between thee variable may be olved iteratively to yield an appropriate n. Although many approache exit for determining n (1, 10, 11, 18, 19), Excel and the following equation (0) can be ued to eaily calculate a implified approximation of the required n for each group: n = ( z + z ) ( ) in which the z value are the percentile of the tandard normal ditribution, which are available in tatitic table or from the NORMSINV( ) function in Excel. + 1 (5) Table 1. for variou n and upper limit of method preciion *. ( = = 0.05, = 0) * for n = 5 for n = 10 for n = 1 for n = It may be helpful to compile a table of n and value for variou combination of,,, and to ae the trade-off between n and acceptable level of rik that the tet will lead to the wrong concluion. Table 1 how the acceptance criteria that are achievable for different combination of and n, with =0 and type 1 and error rate of 5% ( = = 0.05). A expected, a maller n increae the acceptance criterion within which the data would be conidered equivalent, eentially making the tet eaier to pa but le cientifically defenible. A with any form of tatitical teting, cientific and practical judgment i the key to proper implementation. Conider a ituation in which i initially et at 0.9%; n = 10 would be adequate for value 0.5% becaue i 0.9%. If i 1.0%, n 30 i neceary to ue = 0.9%. After the appropriate ample ize n i choen, the third tep i to take the firt et of replicate meaurement and etimate. For better etimate of meaurement preciion, ome approache to equivalence teting have included multiple analyt and multiple day in data comparion tudie (1). However, becaue of contraint on ample ize, time, or reource, it i fairly common to ue independent, replicate meaurement from a ingle analyt or a ingle laboratory to etimate the preciion of a meaurement. Although thi approach can lead to an underetimate of preciion, it i a pragmatic compromie between appropriate tatitical application and real-world contraint on reource. To enure that adequately repreent the true meaurement preciion, it i recommended that an upper confidence limit (e.g., the upper limit from a one-ided 80% confidence interval) be ued a an etimate of meaurement preciion. The upper 100(1 )% confidence limit * for may be calculated a n * = 1 (, n 1) in which (, n 1) i the (100 )th percentile of a ditribution with n 1 degree of freedom (1, ). The i available in tatitic reference table () or from the CHIINV(1, n 1) function in Excel. Thi accommodation for error in the etimate eentially enable a econd lab or method to produce data with le preciion, provided that the preciion of the data i till equivalent to that from the original lab or method. Next, with (6) 4 A ANALYTICAL CHEMISTRY / JUNE 1, 005

5 Table. Statitical ignificance v practical relevance in the tranfer of a diolution method. ( calculated on the bai of the development laboratory of 1.9%, with = = 0.05, = 0, n = 1) % Label trength diolved Tablet number Development lab Manufacturing QC lab y % RSD TOST: 90% confidence interval for difference ( = 3.7%) (0.5,.7) Two-ample t-tet p-value 0.0 the deired,, n (for each group),, and * value, can be calculated by = * + [ t ( 1, n ) + t ( 1 /, n ) ] n For example, uppoe a reearcher wihe to et an acceptance criterion for an aay that compare the content of a ample meaured at two different laboratorie, and the data are expreed a a percentage of the labeled content. If = = 0.05, = 0, and a predetermined n = 10 ample preparation exit for each ite, tatitically appropriate value for the acceptance criterion can be calculated for different level of preciion (Table 1). In all cae, the calculated value hould be critically examined to determine whether it ha practical relevance for the tet that i performed (i.e., i not too mall) yet i cientifically defenible for the intended application (i.e., i not too large). After an appropriate acceptance criterion i choen, the econd et of meaurement i taken and a lightly modified verion of Equation 4 i ued to calculate the confidence interval for the difference in mean value. Note that it i neceary to ue intead of / for the equivalence tet. ( y 1 y 1 ) + t ( 1, n + n ) p n + n Alternatively, the Excel Analyi ToolPak can be ued to implify thi procedure (1). Finally, thi confidence interval i compared with the determined in the previou tep. If the confidence interval for the difference in mean value i completely contained within [, ], the mean value are conidered equivalent. If the confidence interval contain ome value outide [, ], the tet (7) (8) ha not provided ufficient evidence that the mean value are equivalent. It i poible to declare the mean value equivalent if the confidence interval for the difference in mean value doe not include 0, provided that the interval doe not include any value outide [, ]. A confidence interval that doe not include 0 ugget bia in the meaurement, which may need to be examined further; however, the TOST concluion that the mean value are equivalent ay that the bia i le than the acceptable. Note that for Equation 8 to be valid, the value of the two data et hould be imilar. An appropriate variance tet, uch a Levene (3), hould be ued to further evaluate ignificant difference in meaurement preciion. Practical difference v tatitical ignificance One of the more common criticim of the traditional t-tet i that it cannot ditinguih between tatitically ignificant and cientifically relevant difference. To evaluate how equivalence teting work in practice, a erie of data comparion tudie known a method tranfer were conducted. In each of thee tudie, the analyi of a pharmaceutical product by a econd laboratory wa compared with the analyi by the original laboratory to ae whether the econd laboratory applied the method in an equivalent manner. Table how the data for the diolution analyi of a tablet product from the development laboratory and the manufacturing QC laboratory. From data from the development lab, wa etimated at 1.9%. With the 80% upper confidence limit for thi, = 0, and = = 0.05, i calculated with Equation 7 to be 3.7% diolved. Then, with the data in Table and Equation 8, Table 3. Comparion of the effect of poor preciion on the outcome of TOST v a twoample t-tet. ( calculated on the bai of the development laboratory * from previou experiment of 1.5%, with = = 0.05, = 0, n = 6) % Label trength diolved Tablet number Development lab Contract lab y % RSD Difference between y value 3 TOST: 90% confidence interval for difference ( = 3.5%) ( 3.1, 10.5) Two-ample t-tet p-value 0.35 JUNE 1, 005 / ANALYTICAL CHEMISTRY 5 A

6 a 90% confidence interval for the difference between the laboratory mean value i calculated to be 0.5.7%. Becaue the difference between the mean i le than the required 3.7%, the null hypothei that the mean value are not equivalent i diproved, and the laboratory method are declared equivalent. Thu, the method i uccefully tranferred to the manufacturing QC laboratory. When a traditional two-ample t-tet with = 0.05 i ued to compare the data in Table, ufficient evidence exit (p-value = 0.0) for the analyt to conclude that the laboratory mean are not equal. In thi cae, the p- value i the probability of oberving a T-value that i more extreme than the oberved T- value when the mean are equal. The concluion reached with the traditional two-ample t-tet i problematic, becaue the difference in y value (1.6%) i a cientifically acceptable difference for thi tet. Thi example highlight a key advantage of TOST over a twoample t-tet for howing equivalence TOST allow mall, cientifically irrelevant difference to exit without leading to the concluion that the laboratory mean are not equivalent. Conequence of poor preciion Table 3 i an example of a tablet diolution method that wa tranferred from a development laboratory to a contract laboratory during the early tage of product development. In thi tudy, n = 6 for each laboratory becaue of limited ample availability. With an initial * etimate of 1.5% from previou analye, wa calculated at 3.5%, which would generally be conidered acceptable for a method of thi type. Notice that the actual from each laboratory wa much larger than the initial etimate of 1.5%. Thi wa determined to be the reult of poor ample homogeneity caued by degradation during torage. When the data are compared via an equivalence tet with = 3.5%, ufficient evidence doe not exit to declare the laboratorie method equivalent. Moreover, when the upper limit of the experimentally oberved of 5.6% i ued to calculate the minimum achievable for the tet, = 19%, which i coniderably larger than would be acceptable for a diolution tet. Even without performing the econd et of meaurement, the analyt can ee that the method preciion and tudy deign are inufficient to meet the objective, and it i appropriate to conclude that the method tranfer i a failure. If the laboratory mean value had been compared with a twoample t-tet ( = 0.05), the p-value of 0.35 indicate that the data would not have provided ufficient evidence to conclude that the laboratorie were different. In other word, the laboratory mean value would have been declared equal and the method tranfer would have been deemed a ucce. In thi cae, the traditional two-ample t-tet would not have rejected the hypothei that the data et are equal, becaue wa too large relative to the difference between the y value. Thi example highlight another Statitical hypothei teting offer a rigorou, objective approach to ditinguihing truly ignificant difference in meaurement from noie. key advantage of TOST over a two-ample t- tet TOST appropriately penalize the analyt if the oberved variance i too large. We thank Bob Barraza, Sue Long, Chri Kidd, Gerald Monger, John Allen, and Charle Go for helpful uggetion. Gielle B. Limentani i a director in product development for GlaxoSmithKline in Reearch Triangle Park, N.C. She ha more than 0 year of experience developing new drug in the pharmaceutical indutry. She recently preented a paper on analytical method tranfer at the Pharmaceutical and Biomedical Analyi 004 conference. Moira C. Ringo i an invetigator in product development at GlaxoSmithKline. Feng Ye i a enior manager in quality engineering for Amgen in Thouand Oak, Calif. Mandy L. Bergquit i a principal tatitician in reearch and development at GlaxoSmithKline. Ellen O. McSorley i an independent tatitical conultant in Cary, N.C. Addre correpondence to Limentani at 5 Moore Dr., Reearch Triangle Park, NC 7709 or Gielle.B.Limentani@gk.com. Reference (1) Kringle, R.; et al. Drug Inf. J. 001, 35, () Horwitz, W. J. AOAC Int. 1977, 60, (3) Wetlake, W. J. Biometric 1976, 3, (4) Schuirmann, D. J. Biometric 1981, 37, 617. (5) Wetlake, W. J. Biometric 1981, 37, (6) Schuirmann, D. J. J. Pharmacokinet. Biop. 1987, 15, (7) Hartmann, C.; et al. Anal. Chem. 1995, 67, (8) Average, Population, and Individual Approache to Etablihing Bioequivalence (Draft Guidance); U.S. Department of Health and Human Service, Food and Drug Adminitration, Center for Drug Evaluation and Reearch: Wahington, DC, Aug (9) Statitical Approache to Etablihing Bioequivalence; U.S. Department of Health and Human Service, Food and Drug Adminitration, Center for Drug Evaluation and Reearch: Wahington, DC, Feb 001. (10) Tubert-Bitter, P.; et al. J. Clin. Epidemiol. 000, 53, (11) Stein, J.; Doganakoy, N. Qual. Engin , 1, (1) Richter, S. J.; Richter, C. Qual. Engin. 00, 14, (13) Roger, J. L.; Howard, K. I.; Veey, J. T. Pychol. Bull. 1993, 113, (14) Munk, A.; Hwang, J. T.; Brown, L. D. Biometr. J. 000, 4, (15) Roy, T. J. Math. Chem. 1997, 1, (16) McBride, G. Aut. NZ. J. Stat. 1998, 41, (17) Luzar-Stiffler, V.; Stiffler, C. J. Comput. Inf. Tech. 00, 3, (18) Chow, S.-C.; Shao, J.; Wang, H. Stat. Med. 003,, (19) Zhang, P. J. Biopharm. Stat. 003, 13, (0) Britol, D. R. Commun. Statit. Theory Method 1993,, (1) Seely, R. J.; Munyakazi, L.; Haury, J. Biopharm. 001, 10, () Chow, S. C.; Liu, J. P. Statitical Deign and Analyi in Pharmaceutical Science; Marcel Dekker: New York, (3) Milliken, G. A.; Johnon, D. E. Analyi of Mey Data, Volume 1: Deigned Experiment; Chapman & Hall: New York, 199; pp A ANALYTICAL CHEMISTRY / JUNE 1, 005