Using Gage R&R Studies to Quantify Test Method Repeatability and Reproducibility

Transcription

1 Using Gage R&R Studies to Quantify Test Method Repeatability and Reproducibility Ronald D. Snee, PhD IVT Lab Week 2016 San Diego, CA December 12-14,

2 About the Speaker.. He is also an Adjunct Professor in the pharmaceutical programs at Temple and Rutgers Universities. He worked at DuPont for 24 years prior to starting his consulting career. Ron Snee, PhD is Founder of Snee Associates, LLC, a firm dedicated to the successful implementation of process and organizational improvement initiatives. He provides guidance to pharmaceutical and biotech senior executives in their pursuit of improved business performance using Quality by Design, Lean Six Sigma and other improvement approaches that produce bottom line results. He has authored several articles on how to successfully implement QbD, coauthored 2 books on QbD tools and speaks regularly at pharmaceutical and biotech conferences. Ron received his BA from Washington and Jefferson College and MS and PhD degrees from Rutgers University. He is an academician in the International Academy for Quality and Fellow of the American Society of Quality, American Statistical Association, and American Association for the Advancement of Science. He has been awarded ASQ s Shewhart, Grant and Distinguished Service Medals, and ASA s Deming Lecture and Dixon Consulting Awards. He is a frequent speaker and has published 6 books and more than 300 papers in the fields of quality, performance improvement, management, and statistics. He recently received the Institute of Validation Technology s Speaker of the Year Award. 2

3 Abstract Pharmaceutical industry is experiencing a growing need to improve performance driven by global competition and the increasing impact of information technology. How to go about this improvement in a systematic, focused and sustainable manner is the question. Fortunately a world-class body of improvement technology exists known as Quality by Design (QbD); a science and data-based approach that builds quality into products and processes during development and validation as well as operations. A characteristic of good science is good data. Quality data are arguably more important today than ever before. Using a series of case studies and examples it will be shown how QbD can be integrated into a holistic approach that can be used to effectively design and improve laboratory processes enabling prompt, successful method validation. Methods discussed in this session include assessment of method repeatability and reproducibility, improving method robustness and measurement process control. Methods for assessing that amount of product variation that can be attributed to the manufacturing process, sampling procedures and test method are also presented. The concepts and methods involved will be introduced and illustrated with pharmaceutical and biotech case studies and examples. 3

4 Agenda Today s Realities Importance and need to improve measurement quality Need to Reduce Risk in measurement process The Promise of QbD Building Quality into Test Methods Using Gage R&R Studies to Assess Measurement Repeatability and Reproducibility Understanding Sources of Test Method Variation Destructive Test Method Gage R&R Monitoring Test Method Performance over Time Summary Tips, Traps and Guidelines 4

5 Why Do You Need Measurements? When you can measure what you are speaking about and express it in numbers you know something about it, but when you cannot measure it, when you cannot express it in numbers, your knowledge is of the meager and unsatisfactory kind. Lord Kelvin 5

6 Lab Results Tell Us About Product Quality and Process Performance Test labs produce the data that are used to: Develop products and processes Control and improve processes Assess product quality Improving test methods improves the quality of the data used to design, control and improve products and processes: Risk of poor quality product getting to patients and ineffective and inefficient process performance is reduced in the process Improved Test Methods Performance Reduces Risk 6

7 Improving Lab Performance Reducing Risk in Test Method Measurements Developing test method design Improve the quality of measurements process: Gage R&R Study Improve the stability of the measurement process: Process control charting of control samples over time Reducing the sensitivity of the measurement process to uncontrollable variables: Robustness studies using 2-level fractional-factorial designs Improving flow through the lab Use lean operating principles for the flow of individual methods as well as the overall lab flow Quality by Design Addresses These Issues 7

8 Quality by Design Generic Definition A Systematic Process for Building Quality into a Product from the Inception to Final Output Product of Analytical Laboratory Processes: Test results that have the desired precision and accuracy delivered in a cost-effective and timely manner compliant with GLPs 8

9 Analytical Method QbD Building Blocks Method R&R Process Control Critical Quality Attributes (Ys) Critical Method Parameters (Xs) Method Design Method Robustness Design Space Risk Level Raw Materials (Xs) Failure Modes and Effects Analysis Life Cycle Management and Continuous Improvement Continued Process Verification 9

10 Use of QbD in Analytical Methods Method Design Meet method design performance criteria Method Evaluation Risk assessment using Failure Modes and Effects Analysis (FMEA) Assess method repeatability and reproducibility (Gage R&R) Assess robustness of method design space (DOE) Assess sampling and testing variation Method Control and Continued Verification Monitor test method repeatability and reproducibility (Gage R&R) using control (reference) samples Continued process verification using an integrated system of method control and improvement techniques 10

11 Y = f(x) Cause (X) and Effect (Y) Process Inputs (Xs) Raw Materials Instruments Personnel Measurement Is a Process Controlled Variables (Xs) Test Time and Temperature Mixing Speed and Time Reagent Concentrations Apparatus Test Method Process Outputs (Y) Quality Measurements Test Results on Time Cost Effective Operations Environmental Variables (Xs) Ambient Temperature and Humidity Reagent Quality Analyst Day of Week Season of Year Shift 11

12 FDA Process Validation Guidance Importance of Variation A successful validation program depends upon information and knowledge from product and process development. This knowledge and understanding is the basis for establishing an approach to control of the manufacturing process that results in products with the desired quality attributes. Manufacturers should: Understand the sources of variation Detect the presence and degree of variation Understand the impact of variation on the process and ultimately on product attributes Control the variation in a manner commensurate with the risk it represents to the process and product Each manufacturer should judge whether it has gained sufficient understanding to provide a high degree of assurance in its manufacturing process to justify commercial distribution of the product. 12

13 It s About Variation All processes--human and non human--exhibit variability. This variability is measurable. Joseph M. Juran If I had to reduce my message for management to just a few words, I d say it all had to do with reducing variation. W. Edwards Deming State of statistical control is not a natural state for a manufacturing (and measurement) process. It is instead an achievement, arrived at by elimination, one by one, by determined effort, of special causes of excess variation. W. Edwards Deming Test Method Variation is an Important Component of Process Measurement 13

14 Variation Drives Risk, Quality, Cost and Customer Satisfaction Risk Quality Variation Variation Costs ($$) Customer Satisfaction Variation Variation Understanding and Reducing Variation is a Good Thing 14

15 Variation Also Drives Process Understanding Process Understanding Variation Increased Process Understanding Enhanced Process Control and Improvement It Starts with Understanding Variation: You Can t Successfully Control, Improve and Transfer a Process that You Don t Understand 15

16 Aspects of Statistical Problem Solving Practical: Issue Context Objectives, Process, Variation, Data Pedigree Sequence - Small Data Sets Practical, Graphical and Analytical - Large Data Sets Practical, Analytical and Graphical Analytical: Variable Reduction, Data Smoothing, Averaging Graphical Analytical 16

17 Example - Potency Test Method Robustness Potency Test Method evaluated over six years 48 batches Blind control samples periodically submitted for lab analysis Individuals Moving Range Charts used to study variation Analyst A results were more variable Reproducibility acceptable Repeatability within goal Batches Analysts Repeatability Std Dev All All A B, C, D, E, F 0.43 Goal < 1.00 Method Not Rugged With Respect to Analyst 17

18 Example Continued Process Verification Potency Test Method Control Sample 48 Batches Over Six Years 1 I-MR Chart of Potency Sample Average (n=2) UCL= _ X= LCL= Moving Range Batch Process Stable Variation Decreases UCL=1.773 after Batch 23 MR= LCL= Batch 18

19 Example Continued Process Verification Potency Test Method Control Sample 48 Batches Over Six Years Analyst A Analysts B, C, D, E, F 19

20 Tablet Content Variation Study Time Series Plot of % ACTIVE INGREDIENT Variation Due to: Raw Material Lot? No Tablet Press? No Analyst? % ACTIVE INGREDIENT Batch

21 Tablet Content Variation Study Results Reported by Analysts A, B, C and D % ACTIVE INGREDIENT A A A B B A B A A B A B A BB B B C A B A B A BB B A A A A A 12 Time Series Plot of % ACTIVE INGREDIENT B A A A A 24 A 36 B B A A A B A B B B 48 A A B 60 Batch C C B B B B B B B B BB B BB B BB B B B B B B B B B B B B B B B B B B B A Work of Analyst B C C C C 96 B B B B BB B B D B C B D DD B D C D B B D

22 Tablet Content Variation Study Analyst B Has the Least Amount of Variation Dotplot of % ACTIVE INGREDIENT vs ANALYST ANALYST A B C Work of Analyst B D % ACTIVE INGREDIENT

23 Tablet Content Variation Study Analyst B Has the Least Amount of Variation Boxplot Display Work of Analyst B 23

24 Test Method Design - Experimentation Strategy Screening Experiment Optimization Experiment Robustness Experiment Screening Experiment for 6 or more method variables Identify variables with largest effects Study selected variables in an optimization experiment Are often sufficient when the method variables and ranges have been determined Optimization experiment Identify method Design Space Variable settings that will give the best method performance Robustness Experiment Assure that small variations in method use do not affect the performance of the method 24

25 Example Test Method Development Screening Experiment 11 Design Variables evaluated Each at 2 levels Experiment Design 24 run Plackett-Burman design in 4 blocks 6 runs in each of 4 weeks 4 critical variables were identified Optimization Experiment 4 variables from screening experiment were optimized Experiment Design 28-Run subset of a 3x3x2x3 factorial design Runs identified by computer-aided selection procedure enabling estimation of main effects and two-factor interactions 25

26 Example Test Method Development Results Summary Assay Method is robust to variation in the four principal control parameters. Assay precision is 2-4% depending on the number of samples tested and the number of tests made on each sample. Reaction time and reaction temperature had the largest effects Increasing reaction temperature significantly decreases method variation Method design space can be expressed as a function of the four principal control parameters Next Step: Method raw material optimization 26

27 Reducing Test Method Risk Improving Test Method Repeatability and Reproducibility Using Gage R&R Studies 27

28 Sources of Process Variation Excessive variation in process output results may be due to the performance of the: Manufacturing Process Sampling Method Measurement Method A combination of these sources of variation Measurement 10% Sampling 35% Manufacturing 55% Gage R&R Studies Assess Measurement Variation Test Method Repeatability and Reproducibility 28

29 Reducing Test Method Risk Improving Data Quality Measure Repeatability and Reproducibility of Test Results Using Gage R&R Studies Study Variation Due To: Analysts and Instruments (Reproducibility) Repeat Tests (Repeatability) Gage R&R Studies Can Be Done: During Test Method Development After Test Method Has been in Use 29

30 Sources of Variation Total Variation Process or Product Variation Measurement System Variation Accuracy Precision 30

31 Sources of Variation Process or Product Variation Total Variation Accuracy Measurement System Variation Gage R&R Studies Focus on Repeatability and Reproducibility Precision Repeatability Reproducibility 31

32 Gage Repeatability and Reproducibility (R&R) Gage R&R Study: Procedure used to determine how much of the observed total variation is due to the measurement system precision variation Gage is any device used to obtain measurements Why is Gage R&R Important? Pharmaceutical R&D and Operations are data driven Before data are collected, it is crucial that the gage is adequate Gage R&R should also be used whenever a new measurement device is released to manufacturing 32

33 Repeatability and Reproducibility Gage Repeatability Gage Reproducibility Operator B Operator C Repeatability Operator A Reproducibility Gage Repeatability is the variation in measurements obtained when one operator uses the same gage several times for measuring the identical characteristics of the same sample or part. Gage Reproducibility is the variation in the average of measurements made by different operators using the same gage when measuring identical characteristics of the same sample or part. 33

34 Gage R&R Study Design? Select N Samples that represent the full range of long-term variation for your process Select K Analysts that customarily test the samples Each Analyst will measure each sample R Times Sample T1 T2 T1 T2 T1 T2 1 Analyst 1 Analyst 2 Analyst Example Design 10 Samples 3 Analysts 2 Tests by Each Analyst on Each Sample 10x3x2=60 Test Results

35 HPLC Gage R&R Study Sampling Design - 6 Samples - 4 Analysts per Sample Total of 24 sub-samples - 4 Tests per sub-sample Total of 96 tests Sample Sample 1 Sample 6 Analyst A1 A2 A3 A4 A21 A22 A23 A24 Tests T1 T5 T9 T13 T81 T85 T89 T93 T2 T6 T10 T14 T82 T86 T90 T94 T3 T7 T11 T15 T83 T87 T91 T95 T4 T8 T12 T16 T84 T88 T92 T96 Adapted from Ahuga and Dong (2005) Handbook of Pharmaceutical Analysis by HPLC 35

36 HPLC Gage R&R Study Partial Data List - Samples 1 and 6 Sample Analyst Test Assay% Sample Analyst Test Assay%

37 HPLC Gage R&R Study Two-Way ANOVA Table Source DF SS MS F P Sample Analyst Sample*Analyst Repeatability Total P<.05 indicates significant effects Significant differences between Samples (as expected) Analysts Analyst differences are the same for all Samples Sample-Analyst interaction is not significant 37

38 HPLC Gage R&R Study Variance Components Source VarComp % of Total Total Gage R&R Repeatability Reproducibility Analyst Analyst*Sample Part-To-Part Total Variation Total Gage R&R should be <30% Majority of Gage Variation is due to Test-to-Test Number of Distinct Categories = 2 Number of categories should be >4) 38

39 1 80 HPLC Gage 40 R&R Study Repeatability Test-to-Test Variation Percent 0 Gage R&R Repeat Reprod Part-to-Part % Study Var R Chart by Analyst Sample Range UCL=3.802 _ R=1.667 LCL= Control Chart for Range Sample Mean Xbar Chart by Analyst Repeatability is stable across Analysts UCL= _ X= LCL= Average

40 HPLC Gage R&R Study Reproducibility: Analyst to Analyst Variation Analyst Assay % Analyst Differences Are Small Range =

41 Number of Distinct Categories Reflects measurement resolution Ability to detect differences between samples Number of distinct categories part to part measurement 1.41 >4 distinct data categories is desired 41

42 Number of Distinct Categories The number of distinct levels that can be measured by the measuring device also known as resolution 1 Distinct Category unacceptable for estimating process parameters; only indicates whether the process is producing conforming or nonconforming samples 2-4 Distinct Categories generally unacceptable for estimating process parameter; provides only coarse estimates 5 or more Distinct Categories acceptable for estimating process parameters 42

43 HPLC Gage R&R Study Measurement Variation vs Total Variation Measurement Variation Measurement Variation Takes up 63% of the Total Variation Method Not Useful for Distinguishing Between Different Samples 43

44 Thickness Measurement System Number of Categories = 5 (Acceptable Resolution) Measurement Variation Measurement Variation Takes up 23% of the Total Variation Method Useful for Distinguishing Between Different Samples 44

45 Sample and Operator Differences - Examples Study A Operator 2 Test-to-Test Variation Is Higher than Other Operators Study B Operators Do Not Agree on Length of Sample 6 45

46 Potency Potency Sample Operator Day Repeat Lab 1 Lab Test Method Transfer - Sender Lab1 - Receiver Lab2-3 Samples: Potency 100, 200, 400 (IU/ml) 2 Operators 2 Days 2 Repeat Tests 24 Tests/Lab Adapted From: PDA Technical Report No. 57 Analytical Method Validation and Transfer for Biotech Products (2012) 46

47 Test Method Transfer Lab Potency Lab 1 Lab 2 Source Var Comp Pct Std Dev Var Comp Pct Std Dev TotalGageRR Repeat Reproduc Operator O*S Parts TotalVariation Average CV % (RSD) Labs Exhibit Same Amount of Variation Repeatability Is Major Source of Variation

48 Measuring the Effect of Training on Test Method Repeatability and Reproducibility Gage R&R Design Measurement of Weld Bead Quality 8 welds, 3 operators, one test result, 24 results Study repeated 3 times for total of 72 measurements Study 1 Operator difference for: Average value and repeatability Trend across the study Study 2 - After Operator Training Operator differences nor significant; Repeatability and reproducibility variation decreased Not trends over study Measurement resolution increased Coefficient of Variation (RSD) decreased 48

49 Weld Bead Quality Measurement Repeatability Control Chart of 24 Std Dev: 8 Parts x 3 Operators Repeatability Variation Between 3 Repeat Tests Control Chart of 24 Standard Deviations: 8 Parts x 3 Operators Operator 3 has Best Repeatability (Lowest Std Dev) 49

50 Weld Bead Variance Components Study 1 (Before) Var. Component Pct Var. Std Dev Componet Pct Std Dev Total Gage Repeatability Reproducibility Part-to-Part Total Variation No. Categories 5 15 Average Coeff Var (RSD) Study 1 (Before) Operator Effects for Repeatability and Reproducibility Trend in Test Results Study 2 (After Training) Study 2 (After Training) No Significant Differences Between Operators No Trends observed Variation and CV Smaller Resolution Improved 50

51 Weld Bead Quality Measurement Training Reduces Test Method Variation 3.4 Individual Value Plot of Quality Quality Study Parts_

52 Weld Bead Quality Measurement Residuals Versus Test Order Residual = Test Result Model Prediction = Test Result (Operator Effect + Part Effect) 52

53 Weld Bead Quality Measurement Residuals Versus Test Order Residual = Observed Value Predicted Value Variation After Operator and Part Effects Have Been Removed 53

54 Destructive Tests In some test methods the sample is consumed in the test Sample cannot be tested multiple times by multiple operators Estimated test method repeatability becomes Material Variance + Test Method Variance Solutions: A. Make variation between test samples as small as possible Perform Gage Studies as usual B. Use a Two phased approach (Ref: Phillips, etal 1997) Phase I: Estimate material variance (S^2 Material) Phase II: Test Method Variance = Gage S^2 Material S^2 Reference: Phillips, etal (1997) Using Repeatability and Reproducibility Studies to Evaluate a Destructive test Method, Quality Engg, 10(2),

55 Chemical Analysis Test Variation Runs were made by 3 operators to locate a source of variation in a chemical analysis. A Run consisted of: Heat treating a specimen Performing a chemical analysis. In the study: 3 operators Each made 6 combustion runs and Titrated each run in duplicate. Total of 36 test results were obtained Specimen is Consumed in the Analysis Nested Gage R&R Study Will Be Used 55

56 Chemical Analysis Test Variation O R A Y O R A Y O R A Y O = Operator R = Run A = Analysis Y = Test Result 56

57 Chemical Analysis Test Variation Plot of Individual Test Results Individual Value Plot of TEST RESULT Operator 1 TEST RESULT Operator 2 Operator 3 Large Operator Effects Operator 2 Has: Lower Average More Run-to-Run Variation Than Other Operators Run OPERATOR

58 Data Structure Operator Run Analysis 1 Analysis 2 O1 R1 A1 A2 R2 A1 A2 R3 A1 A2 R4 A1 A2 R5 A1 A2 R6 A1 A2 O2 R1 A1 A2 R2 A1 A2 R3 A1 A2 R4 A1 A2 R5 A1 A2 R6 A1 A2 O2 R1 A1 A2 R2 A1 A2 R3 A1 A2 R4 A1 A2 R5 A1 A2 R6 A1 A2 R1 A1 A2 Chemical Test Analysis Variation R2 A1 A2 Operator R3 A1 A2 R4 A1 A2 R5 A1 A2 R6 Nested ANOVA Runs Nested in Operators Analyses Nested in Runs A1 A2 58

59 Chemical Analysis Test Variation Analysis of Variance of TEST RESULT Source DF SS MS F P OPERATOR Run ANALYSIS Total Variance Components % of Source Var Comp. Total StDev OPERATOR Run ANALYSIS Total Significant Effects Operator and Run Operator is the Largest Source (94.58%) 59

60 Raw Material Supplier Selection Study ( Adapted from Gonzalez-de la Parra and Rodriguez-Loaiza, Quality Engg 2003) Purpose Identify and quantify the sources of variability in a critical raw material used to manufacture an API Nested Sampling Design was used Material from 2 suppliers was analyzed 3 lots were evaluated from each supplier 4 containers were sampled from each lot 3 measurements (assay %) were made on each container Total of 2x3x4x3 = 72 assay results 60

61 Raw Material Supplier Selection Study Nested Analysis of Variance Results Source DF SS MS F P Supplier Lot Container Assay Total Variance Components Source Var Comp. % of Total StDev Supplier Lot Container Assay Total Let s Look at a Plot of the Data Significant Container Variation 61

62 Raw Material Supplier Selection Study I Chart of Result(%) by Supplier Supplier 1 Supplier 2 1 UCL= Individual Value _ X= LCL= Observation

63 Source Raw Material Supplier Selection Study Variance Component Supplier 1 Supplier 2 % of Total Std Dev Variance Component % of Total Std Dev Batch Container Assay Total Supplier 2 Has Greater Variation than Supplier 1 Variation is due to sampling (59%) and testing (35%) Next Steps Assess sampling and test methods used by Supplier 2 Consider evaluating what Supplier 1 is doing differently 63

64 Method Control Reducing Analytical Risk Maintaining Stable Analytical Methods Using Shewhart Control Charts Enables Continued Measurement Process Verification 64

65 Use of Blind Controls (Reference Samples) Blind Control Samples from a common source are regularly submitted for analysis along with routine production samples in a way that the analyst can not determine the difference between the production samples and the control samples The use of blind controls is rare in the pharmaceutical industry. The roots of this probably lie in the compliance aspects of the study. However there is no better way to understand the true variability of the analytical method B. K Nunnally and J. S McConnell (2007) Six Sigma in the Pharmaceutical Industry Understanding, Reducing and Controlling Variation in Pharma and Biologics 65

66 Measurement of Potency (%) on a QC Control Material Potency (%) on a QC Control Material 84 Rows of Data 42 QC Runs 2 Measurements / Run VII-66

67 Measurement of Potency (%) on a QC Control Material (Continued) Variables Control Chart XBar of Potency 96.0 Mean of Potency UCL= Av g= LCL= Out-of-Control Runs Detected. Process Is Not Stable; Long-Term Variation = 58% Run Note: T he s igma was calc ulated us ing the range. R of Potency Range of Potency UCL=1.604 Av g=0.491 Good Repeatability Within-Run Variation in Control 0.0 LCL= Run VII-67

68 Measurement of Potency (%) on a QC Control Material Nested ANOVA: Potency versus Run 42 Runs, 2 Tests per Run Analysis of Variance for Potency Source DF SS MS F P Run Error Total Variance Components % of Source Var Comp. Total StDev Run Error Total Long-Term Variation = 58% Method Reproducibility is Poor 68

69 Assessing Measurement Process Stability Q: When Should I worry about measurement stability? A: When Long-Term Variation represents more than 20% of the total variation Total Variation = Long-Term Variation (Reproducibility) Measurement Process Stability + Short-Term Variation (Repeatability) This guideline is based on the assumption that Well-controlled process will detect a shift of 1.5 shortterm standard deviations Long-Term Variation Not a Problem < 20% May be A Problem 20 30% Corrective Action May Needed? > 30% Ref: Snee and Hoerl, Quality Progress, May 2012,

70 Assessing Measurement Process Stability Example Stable and Unstable Processes Stable Process Good Reproducibility Long-Term Variance = 21% Un-Stable Process Poor Reproducibility Long-Term Variance = 58% 70

71 Tips and Traps Samples should cover the range of production Assess method in-use performance Define method resolution Use control (reference) samples to monitor method performance over time Report Coefficient of variation (RSD) Analyze the model residuals Variation after Operator and Parts effects have been accounted for by the model: Resid = Obs d Pred) Plot the Data in All Aspects of Exploration, Analysis and Communication of Test Method Results 71

72 Using QbD Techniques Improves Test Method Performance Measurement quality can be improved using Gage Repeatability and Reproducibility studies Measurement process can be controlled using Control samples and Statistical Process Control techniques Data plots are essential to understanding test method variation Product variation: Be sure to separate sampling and process variation from test method variation Risk is Reduced when Test Method Performance is Stable, Capable, Efficient and Cost-Effective 72

73 References Improving Measurement Systems Borman, P., M., etal (2007) Application of Quality by Design to Analytical Methods, Pharmaceutical Technology, October 2007, Box, G. E. P., J. S. Hunter and W. G. Hunter (2005), Statistics for Experimenters, 2 nd Edition, John Wiley and Sons, New York, NY, Montgomery, D. C. (2009), Design and Analysis of Experiments,7th Edition, John Wiley and Sons, New York, NY, Chapter 13. Schweitzer, M., etal (2010) Implications and Opportunities of Applying QbD Principles to Analytical Measurements, Pharma Tech, Feb 2010, Snee, R. D. (1983), Graphical Analysis of Process Variation Studies, J. Quality Technology, 15, Snee, R. D. (2005) Are We Making Decisions in a Fog? The Measurement Process Must Be Continually Measured, Monitored and Improved, Quality Progress, December 2005, Snee, R. D. (2010) Crucial Considerations in Monitoring Process Performance and Product Quality, Pharmaceutical Technology, October 2010, Snee, R. D. & R. W. Hoerl (2012) Going on Feel: Monitor and Improve Process Stability to Make Customers Happy, Quality Progress, May 2012, Snee, R. D. (2015) Management Holds the Key to Continued Process Verification, Pharmaceutical Manufacturing, January/February 2015, Weitzel, J., R. A. Forbes and R. D. Snee (2015) Use of the Analytical Target Profile in the Lifecycle of an Analytical Procedure: with an example for an HPLC Procedure, J. of Validation Technology, Vol. 20, Issue 4, Jan

74 For Further Information, Please Contact: Ronald D. Snee, PhD Newark, DE (610) Please visit our website at: 74

75 Process Variation Is Too High Where is the Source of the Variation? Process? Sampling Method? Analytical Lab? Some or All of the Above? 75

76 Process Variation Issues Excessive variation in process output results may be due to the performance of the: Manufacturing Process Sampling Method Measurement Method A combination of these sources of variation Need a procedure for identifying the key sources of variation Measurement 10% Sampling 35% Manufacturing 55% 76

77 Analyzing Process Variation Strategy Process variation is a function of many variables, both internal and external to the process. Understanding the magnitude and nature of the sources of process variation can identify opportunities for improvement Tactics Collect data to quantify the various sources of variation in a process Analyze the resulting components of process variation to identify ways to improve the process 77

78 Chemical Production Box, Hunter and Hunter, pp Batch Process: A sample is taken form each batch and analyzed in the lab to determine the moisture content of the product It is desired to understand the sources of variation for the moisture content measurement so that it can be determined the nature and magnitude of the needed improvements Three sources of variation are identified: Batch-to-batch Sample-to-sample within a batch Test-to-test within a sample 78

79 Chemical Production 15 Batches, 2 Samples/Batch, 2 Tests/ Sample Schematic of Sampling Design Batches B1. B15 Samples S1 S2. S29 S30 Tests T1 T2 T1 T2 T57 T58 T59 T60 79

80 Chemical Moisture Variation Due to Batches, Samples and Tests 40 Individual Value Plot of MOISTURE CONTENT 35 MOISTURE CONTENT SAMPLE BATCH Ref: Snee

81 Chemical Moisture Variation Analysis of Variance for MOISTURE Source DF SS MS F P BATCH SAMPLE TEST Total Sampling Variation is Statistically Significant Variance Components (p=0.000) Source VarComp. %oftotal Std Dev BATCH SAMPLE TEST Sampling is the Largest Source of Variation 81

82 Chemical Moisture Variation Interpretation of Results Largest variance component is due to Sampling Sampling procedure: 1. Randomly select 5 drums from a batch of 80 drums 2. Use a special sampling tube to sample material from all levels of the drum 3. Thoroughly mix the 5 samples 4. Test a sample of the aggregate mixture 82

83 Chemical Moisture Variation Interpretation of Results Investigation found: Procedures not followed and unknown Test samples were routinely taken from a single drum When the proper sampling method was used the sample variance component decreased Before After Source Variance Std Dev Variance Std Dev Batches Samples Analyses Total