Report and Manage Post Marketing Changes to an Approved NDA, ANDA and BLA. Jane Weitzel Independent Consultant

Transcription

1 Report and Manage Post Marketing Changes to an Approved NDA, ANDA and BLA Jane Weitzel Independent Consultant

2 IVT S Analytical Procedures & Methods Validation December 2016 San Diego

3 Jane Weitzel Biosketch Jane Weitzel has been working in analytical chemistry for over 35 years for mining and pharmaceutical companies with the last 5 years at the director/associate director level. She is currently a consultant, auditor, and trainer. Jane has applied Quality Systems and statistical techniques, including the estimation and use of measurement uncertainty, in a wide variety of technical and scientific businesses. She has obtained the American Society for Quality Certification for both Quality Engineer and Quality Manager. Jane has assisted several laboratories with implementing the Lifecycle Approach to Analytical Procedures In 2014 she was pointed to the Chinese National Drug Reference Standards Committee and attended their inaugural meeting in Beijing For the cycle, Jane is a member of the USP Statistics Expert Committee and Expert Panel on Method Validation and Verification. mljweitzel@msn.clm 3

4 Disclaimer This presentation reflects the speaker s perspective on this topic and does not necessarily represent the views of USP or any other organization. mljweitzel@msn.clm 4

5 Provides Goal and Acceptance Criteria for Analytical Procedure Throughout its Life LIFECYCLE OF ANALYTICAL PROCEDURE 5

6 Stimuli Articles Lifecycle Management of Analytical Procedures: Method Development, Procedure Performance Qualification and Procedure Performance Verification (USPPF 39(5)) Fitness for Use: Decision Rules and Target Measurement Uncertainty (USPPF 42(2)) Analytical Target Profile. Structure and Application Throughout The Analytical Lifecycle (USPPF 42(5)) Analytical Control Strategy (USPPF 42(5)) 6

7 <1210> Statistical Tools for Procedure Validation 7

8 Proposed General Chapter <1220> Will be published in the Pharmacopeial Forum 43(1) Published as Stimuli Article to promote discussion Decision to leave <1225>, <1226> and <1224> intact Industry has a lot invested in these chapters The current systems works, although sometimes not as well as we would like. mljweitzel@msn.clm 8

9 Explains the terminology AMC/TechnicalBriefs.asp NEXT SOME TERMINOLOGY 9

10 Improving Trueness a) Error b) Accuracy Error Bias Trueness Accuracy contains both bias and precision components c) d) Bias Bias is the total systematic error Improving Precision Uncertainty includes all random effects (including the uncertainty of the bias) How uncertainty relates to accuracy and precision 10

11 VIM International Vocabulary of Metrology 2.26 (3.9) measurement uncertainty uncertainty of measurement uncertainty non-negative parameter characterizing the dispersion of the quantity values being attributed to a measurand, based on the information used (Standard uncertainty is analogous to a standard deviation. It is abbreviated as u ) mljweitzel@msn.clm 11

12 Target Measurement Uncertainty 2.34 target measurement uncertainty target uncertainty Measurement uncertainty specified as an upper limit and decided on the basis of the intended use of measurement results guides/vim.html 12

13 GUM (Guide to the expression of uncertainty in measurement) MU is a critical part of a system for the international traceability of chemical measurements. In this era of the global marketplace, it is imperative that the method for evaluating and expressing uncertainty be uniform throughout the world so that measurements performed in different countries can be easily compared. Evaluation of measurement data Guide to the expression of uncertainty in measurement, 2008 mljweitzel@msn.clm 13

14 BIPM.org 14

15 Analytical Target Profile LIFECYCLE APPROACH 15

16 Three Stage Approach to Analytical Lifecycle Stage 1 Procedure Design and Development Stage 2 Procedure Performance Qualification Stage 3 Continued Procedure Performance Verification Risk assessment Knowledge management Analytical Control Strategy Changes mljweitzel@msn.clm 16

17 Stage 3 Continued Procedure Performance Verification To provide ongoing assurance that the analytical procedure remains in a state of control throughout its lifecycle Routine Monitoring: an ongoing program to collect and process data that relate to method performance, e.g. from analysis / replication of samples or standards during batch analysis by trending system suitability data by assessing precision from stability studies [J. Ermer et al.: J. Pharm. Biomed. Anal. 38/4 (2005) ] mljweitzel@msn.clm 17

18 Continual Improvements (Changes) Risk assessment to evaluate Impact of the respective change Required actions to demonstrate (continued) appropriate performance Feedback loop As needed, return to Stage 2 or Stage 1 ATP establishes criteria for acceptability mljweitzel@msn.clm 18

19 Risk Assessment Design of experiments (DOE) is a fundamental methodology for the QRM process. It is a systematic method to determine the relationships between variables affecting a process, and it is used to find cause-and-effect relationships Understand the procedure variables and their impact on the reportable value Detect te presence and degree of variation Understand the impact of variation on the analytical procedure performance and ultimately on data attributes mljweitzel@msn.clm 19

20 Report and Manage Post Marketing Changes to an Approved NDA, ANDA and BLA Lifecycle approach provides a structure, a language and techniques to evaluate, manage and report changes These will be understood by industry and regulatory bodies All understand the probability, the risk, the evaluations Understand how they are identified, evaluated and managed mljweitzel@msn.clm 20

21 ANALYTICAL TARGET PROFILE 21

22 Wording for ATP Assay The procedure must be able to quantify the analyte in presence of (X, Y, Z) over a range of A% to B% of the nominal concentration with an accuracy and uncertainty so that the reportable result falls within ±C% of the true value with at least P% probability. Target The variables in orange are Measurement specific for each reportable result. Uncertainty (TMU) mljweitzel@msn.clm 22

23 TMU and ATP The target measurement uncertainty becomes part of the analytical target profile. The TMU defines the acceptance criteria for the method. Remember, the uncertainty includes all random effects (including the uncertainty of the bias). 23

24 Analytical Target Profile (ATP) A predefined objective that states the performance requirements for the analytical procedure The output of the procedure is a reportable result that must be fit for its purpose. Applies throughout the life of the analytical procedure, including stage 3 Report and manage post marketing changes to an approved NDA, ANDA and BLA mljweitzel@msn.clm 24

25 DECISION RULES AND TARGET MEASUREMENT UNCERTAINTY (TMU) 25

26 Intended Use can be Linked to Clinical Requirement 26

27 Intended Use can be Linked to Clinical Requirement - MU MU mljweitzel@msn.clm 27

28 Used to explain fitness for intended purpose as part of change control Decision rules and their relevance to analytical procedure qualification will be presented DECISION RULES 28

29 What is the role manufacturing and clinical play in defining that use? Fitness for intended use needs to be known Decision rules proved that link Through probability 29

30 Types of Decisions What is the drug concentration in the blood (during a clinical study). Does this batch of drug product meet specification for potency? Does this lot of drug substance meet specification for impurity A? Does this in-process solution have correct concentration, e.g. for excipient concentration? Is this environmental monitoring sample in specification? mljweitzel@msn.clm 30

31 During change control and reporting to regulatory bodies, it is clear the user of the data is involved. Ensuring the reportable result is fit for use Purpose of a decision Rule HOW THE USER OF THE DATA IS KEY TO DEFINING THE DECISION RULE mljweitzel@msn.clm 31

32 The USER Develops the Decision Rule For an example, consider the case for a brand new measurement The best source for the prescription of the decision rule is the person/organization that will use the output of the analytical procedure Can be one person (the expert) or a group of people (clinical studies, production, stability) The group can include management (financial risks) Called Decision Makers in ICH Q8 mljweitzel@msn.clm 32

33 Analytical may develop decision rule If the end user of the data is not available E.g. often the case for a commercial contract laboratory The laboratory can create a decision rule to assist with ensuring its test results are suitable. This is especially useful for testing according to USP monographs. The approach provides a language for communication. mljweitzel@msn.clm 33

34 Decision Rule A documented rule... that describes how measurement uncertainty will be allocated with regard to accepting or rejecting a product according to its specification and the result of a measurement. ASME B (reaffirmed 2006) mljweitzel@msn.clm 34

35 Decision Rules References 35

36 Decision Rule Acceptance or Rejection Decision rules give a prescription for the acceptance or rejection of a product based on the measurement result, its uncertainty and the specification limit or limits, taking into account the acceptable level of the probability of making a wrong decision. E X C E L mljweitzel@msn.clm 36

37 Meaning of Product The term product in the decision rule definition refers to the whatever is tested. Does not mean drug product lot only. Could be: In-process sample (buffer solution) Lot of drug substance Batch of drug product Lot of excipient Environmental monitoring sample 37

38 Why Use Decision Rules Decision rules clearly state the intended use of the procedure Risk and probability are used to develop the decision rule This means there is a defined process to define the intended use of the procedure Risk and Probability are consistent with QbD A guard band can be created using the uncertainty mljweitzel@msn.clm 38

39 Decision Rule To decide whether a result indicates compliance or non-compliance with a specification, it is necessary to take into account the measurement uncertainty. Upper Limit 1 Result is above the limit. Limit is below expanded uncertainty. 2 Result above the limit. Limit is within the expanded uncertainty. 3 Result is below the limit. Limit is within the expanded uncertainty. 4 Result is below the limit. Limit is above expanded uncertainty. mljweitzel@msn.clm 39

40 Decision Rule To decide whether a result indicates compliance or non-compliance with a specification, it is necessary to take into account the measurement uncertainty. Upper Limit How much overlap is acceptable? That is the acceptable probability of making a wrong decision. 1 Result is above the limit. Limit is below expanded uncertainty. 2 Result above the limit. Limit is within the expanded uncertainty. 3 Result is below the limit. Limit is within the expanded uncertainty. 4 Result is below the limit. Limit is above expanded uncertainty. mljweitzel@msn.clm 40

41 Decision Rules Require 4 Components Decision rules give a prescription for the acceptance or rejection of a product based on 1. the measurement result, 2. its uncertainty and 3. the specification limit or limits, 4. taking into account the acceptable level of the probability of making a wrong decision. mljweitzel@msn.clm 41

42 Example Decision Rule The lot of drug substance will be considered compliant with the specification of 95.0% to 105.0% if the probability of being above the upper limit is less than 2.5% and below the lower limit is less than 2.5%. Lower Limit 95 Nominal Concentration (Central Value) 100 Upper Limit 105 Measurement Uncertainty 2.5 % Below Lower Limit Total % Outside Limits % Above Upper Limit 2.28% 4.55% 2.28% Concentration LL UL mljweitzel@msn.clm 42

43 Setting TMU This document discusses how to set a maximum admissible uncertainty, defined in the third edition of the International Vocabulary of Metrology as the target uncertainty, to check whether measurement quality quantified by the measurement uncertainty is fit for the intended purpose. mljweitzel@msn.clm 43

44 Acceptance Criteria ANALYTICAL PROCEDURE DESIGN AND PROCEDURE PERFORMANCE QUALIFICATION 44

45 ATP - Reminder Assay The procedure must be able to quantify the analyte in presence of (X, Y, Z) over a range of A% to B% of the nominal concentration with an accuracy and uncertainty so that the reportable result falls within ±C% of the true value with at least P% probability. The variables in orange are specific for each reportable result. mljweitzel@msn.clm 45

46 Accuracy and uncertainty are defined Accuracy is represented by the bias. The uncertainty is the Target measurement uncertainty 46

47 TMU and ATP The TMU defines the acceptance criteria for the method. Remember, the uncertainty includes all random effects (including the uncertainty of the bias). Uncertainty Bias Intermediate Precision Repeatability 47

48 When an analytical procedure is changed, the change is evaluated against the fitness for intended use requirement. Set Requirements PERFORMANCE CHARACTERISTICS 48

49 TMU TMU is an overreaching acceptance criterion for the analytical procedure qualification The source of the uncertainty is not important as long as the combined uncertainty is acceptable 49

50 Acceptance Criteria for Qualification Combination of Bias and Uncertainty meets target measurement uncertainty and meets the decision rule requirements Decision rule requirements include the acceptable probability of making a wrong decision mljweitzel@msn.clm 50

51 These techniques provide documented risk analysis, evaluation of impact on reportable value and probability. Misclassification Gage R&R ANOVA DOE MAKING THE ANALYTICAL PROCEDURE ROBUST 51

52 Taking variability of the product being tested ensures proper assignment of TMU CALCULATING TMU USING MISCLASSIFICATION RATES 52

53 References Burdick RK, Park Y-J, Montgomery DC, Borror CM. Confidence intervals for misclassification rates in a gauge R&R study. J Qual Tech. 2005;37(4): Burdick RK, Borror CM, Montgomery DC. Design and Analysis of Gauge R&R Studies; Making Decisions with Confidence Intervals in Random and Mixed ANOVA Models. ASA-SIAM Series on Statistics and Applied Probability, SIAM, Philadelphia, ASA, Alexandria, VA, mljweitzel@msn.clm 53

54 True Value & Measured Value True Value is the actual value which cannot be known Measured Value is the result of a measurement made Is an indication of the true value The measured value and the uncertainty associated with it are used to make the decision mljweitzel@msn.clm 54

55 Probability For each true value that is inside the specification, there is a probability the measured value will be outside the specification. Lower Limit 95 Nominal Concentration (Central Value) 100 Upper Limit 105 Measurement Uncertainty 3 % Below Lower Limit Total % Outside Limits % Above Upper Limit 4.78% 9.56% 4.78% LL UL Concentration mljweitzel@msn.clm 55

56 Probability Alternately, for each true value that is outside the specification, there is a probability the measured value will be inside the specification. Lower Limit 95 Nominal Concentration (Central Value) 92 Upper Limit 105 Measurement Uncertainty 3 % Below Lower Limit Total % Outside Limits % Above Upper Limit 84.13% 84.14% 0.00% LL UL Concentration mljweitzel@msn.clm 56

57 Two types of risks of wrong decision False failure (FF): A lot whose true value (measurand quantity or lot mean) is within product limits, but is assigned an OOS potency and rejected. Missed fault (MF): A lot whose true value (measurand quantity or lot mean) is outside product limits, but is judged acceptable. mljweitzel@msn.clm 57

58 Monte Carlo acceptable probabilities are decided for an MF and FF error The probabilities for these misclassification errors can be calculated using a Monte Carlo simulation simulating 100,000 true values (i.e., 100,000 simulated production lots) from the production process normal distribution, using expected values for the production mean and standard deviation. mljweitzel@msn.clm 58

59 Use EXCEL for Simulation We can simulate the results of testing the 100,000 lots using the characteristics, bias, and standard deviation of the analytical procedure. 59

60 EXCEL Inputs Variables for calculation of misclassification probabilities using Monte Carlo simulation Instructions: Enter values into cells shaded blue. Product Limits Lower Limit 95.0 Upper Limit Production Distribution true mean true SD 2.0 Analytical Uncertainty true bias 0.0 true SD (uncertainty) 1.2 Decision Rule Lower Limit 95.0 Upper Limit

61 EXCEL Effect See the effect on the probability of misclassification below: Type of Misclassification Probability % Probability of a Missed Fault % Probability of a False Failure % Probability of Failure (MF & FF) % mljweitzel@msn.clm 61

62 True Value compared to limits Good product in blue and bad in red Count Lower Limit Upper Limit True Value 62

63 Measured Values Good Product False Failures (FF) 10 Percent of Total in Each Category 10 Bad Product Missed Faults (MF) Measured Value mljweitzel@msn.clm 63

64 MF and FF True state of product: Bad Good Bad FF Product Disposition: Fail Upper Decision Limit Measured Value MF MF Pass Lower Decision Limit Fail FF Lower Product Limit True Value Upper Product Limit 64

65 USING MISCLASSIFICATION TO SET TMU 65

66 Assumptions If We know the true product distribution (this can be achieved by using process capability). We have product limits to distinguish truly good from truly bad product. These are a priori limits that unambiguously discriminate good from bad lots in terms of potency. These limits apply to the measurand quantity true value which we can never know. We know the maximum allowable MF and FF probabilities. mljweitzel@msn.clm 66

67 Then We can conduct simulations We can vary the MU and decision rule and predict the MF and FF probabilities. We can find ranges for MU and DR that provide acceptable MF and FF probabilities. We can specify the maximum MF and FF probabilities in the DR and determine the TMU. 67

68 Example 1 from Stimuli Article Potency Test for a Drug Product (DP) USP monograph specification Lower Specification is 95.0% (38 mg) Upper Specification is 105.0% (40 mg) Target Specification is 100.0% (42 mg) For simplicity No bias for production or analytical mljweitzel@msn.clm 68

69 Manufacturing a process that will produce 95% of the product with an assay value centered on 100.0% with the standard deviation of 2.0%. This is determined from the formal experimental designs in the manufacturing process development program including manufacturing robustness studies. mljweitzel@msn.clm 69

70 FF & FF The probability of harm (making a wrong decision) assigned for the DR was the acceptable probability for the individual FF and MF events. The decision makers assessed the acceptable level of risk and assigned the FF as 2.5% (0.025) and the MF as 1.5% (0.015). mljweitzel@msn.clm 70

71 Decision Rule The DR is therefore the lot of DP will be considered compliant if the probability of the FF is less than 2.5% and the probability for a MF is less than 1.5%. I added the probabilities as numbers so it is easier to compare to the data. mljweitzel@msn.clm 71

72 Find TMU that meets DR Needs Using an MS Excel spreadsheet with Monte Carlo formula, the true SD (uncertainty) is varied until the maximum value that yields the DR requirements is found. 72

73 LOOK AT MONTE CARLO EXCEL 73

74 TMU using FF & MF Conclude TMU determined using misclassification approach: An uncertainty of 1.2% met the goal or DR requirements of FF < and MF < This becomes the TMU. The target uncertainty can also be expressed using a coverage factor to calculate a coverage interval. Using a coverage factor of 2 for a 95% coverage interval the TMU is 2.4%. mljweitzel@msn.clm 74

75 GAGE R&R 75

76 Example Based on Paper Volume XX Issue 4 mljweitzel@msn.clm 76

77 Experimental Design Samples from five API batches were tested on four setups (run) by two analysts using each of two instruments each instrument used a unique column for total of two columns New mobile phase was prepared for each setup and each analyst prepared a fresh set of standards for each run. Samples were prepared and analyzed in duplicate on each run, producing a total of 5x2x2x2 = 40 Potency(%) test results. mljweitzel@msn.clm 77

78 API Batch Analyst HPLC/ Column Test Potency (%) API Batch Analyst HPLC Column Test Potency (%) 1 A A A A B B B B A A A A B B B B A A A A B B B B A A A A B B B B A A A A B B B B mljweitzel@msn.clm 78

79 Data Organized by Run/Setup (1& 2) Run/Set-up Analyst HPLC/Column Lot/Sample Replicate API Batch Test Potency(%) 1 A A A A A A A A A A B B B B B B B B B B mljweitzel@msn.clm 79

80 Data Organized by Run/Setup (3&4) Run/Set-up Analyst HPLC/Column Lot/Sample Replicate API Batch Test Potency(%) 3 A A A A A A A A A A B B B B B B B B B B mljweitzel@msn.clm 80

81 Variance components analysis Used Minitab Estimates repeatability (within set-up variability) Estimates reproducibility (setup to setup variability) multiple analysts, instruments and columns were used to obtain more realistic variability estimates, and could not be separately evaluated for statistically significant effects 81

82 Gage R &R Variance Components results of the variance components analysis for the potency measurements are summarized Source of Variation Variance % of Total Std Dev Repeatability Reproducibility Total Gage R&R mljweitzel@msn.clm 82

83 Intermediate Precision Reportable value is average of the two potencies analyzed in a single setup The standard deviation is 0.71% With an 80%upper confidence limit of 1.16% Acceptance criterion is NMT 1.0% The standard deviation passes but the upper confidence limit does not mljweitzel@msn.clm 83

84 Pool results for all the batches Make an assumption there are no significant differences between the batches Support this assumption with detailed analytical results from the supplier of the drug substance The standard deviation becomes 0.63% with an 80% Upper Confidence Limit of 1.03% Rounds to 1.0% that meets the acceptance criterion 84

85 Impact of Replication Another way to reduce the standard deviation is to use replication Equation to calculate intermediate precision SD 85

86 0.71% Source of Variation Variance % of Total Std Dev Repeatability Reproducibility Total Gage R&R (( ) 2 /1 + ( ) 2 /(1*2)) = 0.71% mljweitzel@msn.clm 86

87 Impact of Replication Number of Setups Number of Within Setup Replicates Method Standard Deviation (Intermediate Precision) % % % % Increasing number of replicates within a run/setup has little impact Since between setup standard deviation is larger component of variance, replicate between setup/runs. mljweitzel@msn.clm 87

88 Benefit of Using R&R Study You understand the variance components Don t waste effort by preparing replicates at the repeatability level (within setup/run). Waste resources Have a false sense of assurance that precision is improved mljweitzel@msn.clm 88

89 Analysis of Variance ANOVA 89

90 Useful ANOVA is a powerful and useful statistical tool that is underused Often the analytical procedure contains the data for an ANOVA assessment, but it is not done It allows the estimation of relative sizes of variances Can quantify the size of uncertainty components mljweitzel@msn.clm 90

91 Typical Experiment for Accuracy Replicate Spike 1 Spike 2 Spike Average Std Dev Accuracy Experiment 3 spikes done on same day Look at data Could calculate average and standard deviation Could do t test Only tell if 2 spikes were different Only tells if they are different, no probability It looks good.?? mljweitzel@msn.clm 91

92 ANOVA One-way EXCEL Print Out Anova: Single Factor SUMMARY Groups Count Sum Average Variance Spike Spike Spike ANOVA Source of Variation SS df MS F P-value F crit Between Groups Within Groups Total mljweitzel@msn.clm 92

93 Calculate Precisions Precision Symbol Formula repeatability s r SQRT(MS Within Groups) Between Group Standard Deviation s BG SQRT((M S Between Groups - M S Within Groups)/Count ) Intermediate Precision s IP SQRT(S r 2 +S BG 2 ) ANOVA Source of Variation SS df MS F P-value F crit Between Groups Within Groups Precisions s r s BG #NUM! The MS's do not differ significantly. Both estimate repeatability. s IP #NUM! mljweitzel@msn.clm 93

94 Repeatability Anova: Single Factor SUMMARY Groups Count Sum Average Variance Std Dev Spike Spike Spike Pooled 15 Degrees of ANOVA Freedom Source of Variation SS df MS F P-value F crit Between Groups Within Groups Total Precisions s r mljweitzel@msn.clm 94

95 Between Groups & Intermediate Precision (IP) Precision Symbol Formula repeatability s r SQRT(MS Within Groups) Between Group Standard Deviation s BG SQRT((M S Between Groups - M S Within Groups)/Count ) Intermediate Precision s IP SQRT(S r 2 +S BG 2 ) ANOVA Source of Variation SS df MS F P-value F crit Between Groups Within Groups Precisions s r s BG #NUM! The MS's do not differ significantly. Both estimate repeatability. s IP #NUM! The between group difference is not significant. Set s BG to 0. Not unexpected given 3 spikes done on same day. mljweitzel@msn.clm 95

96 Condition Experiment Replicate Condition 1 Conditon 2 Condition Average Std Dev What could condition be? Condition could be day Instrument reagents Look at average & standard deviation mljweitzel@msn.clm 96

97 ANOVA Anova: Single Factor Between Group is significant The condition is significant. What does this mean? SUMMARY Groups Count Sum Average Variance Condition Conditon Condition ANOVA Source of Variation SS df MS F P-value F crit Between Groups Within Groups Total Precisions s r s BG s IP mljweitzel@msn.clm 97

98 Uncertainty Components Both repeatability and between condition will impact the variability the uncertainty The conditions are critical variables and will need some form of operational control Precisions s r 2.21 s BG 2.25 s IP 3.15 mljweitzel@msn.clm 98

99 Uncertainty Components If replication is needed (the test portion is analyzed more than a singlicate), where are the replicates placed? Between runs is best Precisions s r 2.21 s BG 2.25 s IP 3.15 mljweitzel@msn.clm 99

100 Replicate Condition 1 Conditon 2 Condition Average Std Dev Anova: Single Factor How should replicates be placed? The repeatability is not significant. The between conditions is the critical variable and needs operational control. SUMMARY Groups Count Sum Average Variance Condition Conditon Condition ANOVA Source of Variation SS df MS F P-value F crit Between Groups E Within Groups Total Precisions s r 2.21 s BG 9.71 s IP 9.96 mljweitzel@msn.clm 100

101 Data Format for ANOVA As recovery (normalized) Target Concentration Replicate Replicate Replicate Replicate Replicate Replicate Replicate mljweitzel@msn.clm 101

102 ANOVA (s r uncertainty component) Anova: Single Factor SUMMARY Groups Count Sum Average Variance Column Column Column ANOVA Source of Variation SS df MS F P-value F crit Between Groups Within Groups Total Precisions s r 0.54 s BG #NUM! The MS's do not differ significantly. Both estimate repeatability. s IP #NUM! Precision Symbol Formula repeatability s r SQRT(MS Within Groups) Between Group Standard Deviation s BG SQRT((M S Between Groups - M S Within Groups)/Count ) Intermediate Precision s IP SQRT(S 2 r +S 2 BG ) mljweitzel@msn.clm 102

103 ANOVA example worksheet ANOVA IN EXCEL 103

104 DOE 104

105 Acceptance Criteria 105

106 Experimental Results The values are automatically transferred from the appropriate worksheets. Compare the uncertainty components and combine appropriately. 106

107 Experimental Results Linearity is an example of an uncertainty component that is included in other components. It is useful to know its magnitude. 107

108 Procedure Design - Robustness DESIGN OF EXPERIMENTS mljweitzel@msn.clm 108

109 DOE DOE is a useful, efficient tool for procedure validation Well planned experiments Hint: finalize and clearly define how the data will be assessed/analyzed as part of the protocol. Include actual formulas, charts, tables, etc. Doing so makes for a well thought out, easily understood protocol. It also makes writing the report faster and easier. You are less likely to have to repeat experiments. Penny wise is often pound foolish. mljweitzel@msn.clm 109

110 Ruggedness DOE - Definitions USP <1225> Validation of Compendial Procedures Ruggedness -Intermediate precision (also known as ruggedness) expresses within-laboratory variation, as on different days, or with different analysts or equipment within the same laboratory. Robustness -The robustness of an analytical procedure is a measure of its capacity to remain unaffected by small but deliberate variation in procedural parameters listed in the procedure documentation... mljweitzel@msn.clm 110

111 DOE Presented Here and in Workbook It is an incomplete factorial design Also known as AOAC Ruggedness Test Youden There are many published examples that you can use as guides. Determine which factors were varied. 111

112 Robustness test Often done during the design stage of the procedure If so, the test can be referenced and summarized in the final report of the analytical procedure qualification Allows much information to be learned from a relatively small number of tests mljweitzel@msn.clm 112

113 Run Design Experiment (Run) Factor A B C D E F G Result s t u v w x y z mljweitzel@msn.clm 113

114 Calculations for Factor A Experiment (Run) Factor A B C D E F G Result s t u v w x y z Effect of altering level is calculated: E.g. Factor A (s+t+u+v)/4 (w+x+y+z)/4 mljweitzel@msn.clm 114

115 Other Factors Cancel Out eg. B & b Experiment (Run) Factor A B C D E F G Result s t u v w x y z Effect of altering level is calculated: E.g. Factor A (s+t+u+v)/4 (w+x+y+z)/4 mljweitzel@msn.clm 115

116 Calculations for Factor D Experiment (Run) Factor A B C D E F G Result s t u v w x y z Effect of altering level is calculated: E.g. Factor D (s+t+y+z)/4 (u+v+w+x)/4 mljweitzel@msn.clm 116

117 Other Factors Cancel Out eg. E and e Experiment (Run) Factor A B C D E F G Result s t u v w x y z Effect of altering level is calculated: E.g. Factor D (s+t+y+z)/4 (u+v+w+x)/4 mljweitzel@msn.clm 117

118 Comparison of Factors Compare the difference for factors. Any difference that is substantially larger than the others is significant. If no factors are significant, take the standard deviation of all experiments as the robustness standard deviation. Even if a factor is significant, if the overall standard deviation is less than the TMU, the procedure is still fit for use. The factor can be left as is. mljweitzel@msn.clm 118

119 Example of Factors Selected for HPLC Robustness Study Factors extraction heating time, extraction temperature, HCl concentration, extraction volume, column temperature, flow rate, and test sample weight 119

120 Critical Variable Identified Robustness Study Factor - + units Differ ence A extraction temperature C 0.3 B flow rate 5 10 ml/min 0.1 C HCl concentration M 0.1 D extraction volume ml 0.0 E column temperature C 0.8 F extraction heating time min 3.3 G test sample weight g 0.2 Overall STDEV: n: 8 DF 7 Operational control implemented for extraction heating time require 30 minutes heating time, enter that in SOP, require actual heating time to be recorded The overall standard deviation is less than TMU of 2.1% so no further experiments are needed mljweitzel@msn.clm 120

121 What to do with Significant Factor? When a factor is found to be significant there are a few options Control the factor Change the factor slightly Worst case, go back to procedure development mljweitzel@msn.clm 121

122 Overall Standard Deviation The overall standard deviation from all the tests can be a good estimate of precision (uncertainty). Must know what was varied during and between experiments (beside the factors), eg. Different instruments? Different days? Different calibration solutions? mljweitzel@msn.clm 122

123 DOE EXAMPLES 123

124 Factors Selected for HPLC Factors extraction heating time, extraction temperature, HCl concentration, extraction volume, column temperature, flow rate, and test sample weight 124

125 Example 1 Factor A Experim en t (Ru n ) Repeatability: 0.34 n for Repeatability: 21 Degrees of Freedom: 20 t 95%: 2.09 Fact or Sum + Sum - Differ ence t exp A B C D E F G Result s t u v w x y z Overall standard deviation: 2.25 Calcu lation s mljweitzel@msn.clm 125

126 Factor A Statistically Significant NOT Practically Significant Experim en t (Ru n ) Repeatability: 0.34 n for Repeatability: 21 Degrees of Freedom: 20 t 95%: 2.09 Fact or Sum + Sum - Differ ence t exp A B C D E F G Result s t u v w x y z Overall standard deviation: 0.90 Calcu lation s Over all standard deviation < Target Measurement Uncertainty mljweitzel@msn.clm 126

127 Example 2 Factor D Experim en t (Ru n ) Repeatability: 0.34 n for Repeatability: 21 Degrees of Freedom: 20 t 95%: 2.09 Fact or Sum + Sum - Differ ence t exp A B C D E F G Result s t u v w x y z Overall standard deviation: 1.75 Calcu lation s mljweitzel@msn.clm 127

128 No Factor Significant Experim en t (Ru n ) Repeatability: 0.34 n for Repeatability: 21 Degrees of Freedom: 20 t 95%: 2.09 Fact or Sum + Sum - Differ ence t exp A B C D E F G Result s t u v w x y z Overall standard deviation: 0.33 Calcu lation s mljweitzel@msn.clm 128

129 Examples in Literature 129

130 REFERENCES 130

131 DOE References Design of experiments is a discipline in itself. More information on DOE and the robustness test presented above can be found in the AOAC book Use of Statistics to Develop and Evaluate Analytical Methods An Introduction to Design of Experiments, A Simplified Approach, by Larry B. Barrentine. Application of ISO/IEC Technical Requirements in Industrial Laboratories; Method Validation, M. L. Jane Weitzel and Wesley M. Johnson mljweitzel@msn.clm 131

132 Based on Approach store/title/

133 Reference NIST div898/handbook/ 133

134 ASQ Source for References Statistical Quality Control Using Excel, Second Edition And other references 134

135 Useful web page Web Pages that Perform Statistical Calculations! 135

136 FIT FOR USE 136

137 Example from REAL data 137

138 Conclusion The Lifecycle approach provides terminology, techniques and tools to identify need for change Evaluate impact of change Risk analysis Probability Report changes to regulatory bodies 138

139 When you become comfortable with uncertainty, infinite possibilities open up in your life. Eckhardt Tolle When you become comfortable with uncertainty, infinite possibilities open up in you analyses. 139

140 Thank 140