Standard Guide for Conducting Ruggedness Tests 1

Transcription

1 Deignation: E (Reapproved 996) Standard Guide for Conducting Ruggedne Tet AMERICA SOCIETY FOR TESTIG AD MATERIALS 00 Barr Harbor Dr., Wet Conhohocken, PA 948 Reprinted from the Annual Book of ASTM Standard. Copyright ASTM An American ational Standard Thi tandard i iued under the fixed deignation E 69; the number immediately following the deignation indicate the year of original adoption or, in the cae of reviion, the year of lat reviion. A number in parenthee indicate the year of lat reapproval. A upercript epilon (e) indicate an editorial change ince the lat reviion or reapproval.. Scope. In tudying a tet method, it i neceary to conider the effect of environmental factor on the reult obtained uing the tet method. If thi effect i not conidered, the reult from the original developmental work on the tet method may not be a accurate a expected.the purpoe of a ruggedne tet i to find the variable (experimental factor) that trongly influence the meaurement provided by the tet method, and to determine how cloely thee variable need to be controlled. Ruggedne tet do not determine the optimum condition for the tet method.. The experimental deign mot often ued in ruggedne teting are the o called Plackett-Burman deign (). Other experimental deign alo can be ued. Thi guide, however, will retrict itelf to Plackett-Burman deign with two level per variable becaue thee deign are particularly eay to ue and are efficient in developing the information needed for improving tet method. The deign require the imultaneou change of the level of all of the variable, and allow the determination of the eparated effect of each of the variable on the meaured reult. In ruggedne tet the two level for each variable are et o a not to be greatly different. For uch ituation, the calculated effect for any given variable i generally not greatly affected by change in the level of any of the other variable. A detailed example involving gla electrode meaurement of the ph of dilute acid olution i ued to illutrate ruggedne tet procedure. A method i preented for evaluating the experimental uncertaintie..3 The information in thi guide i arranged a follow: Section Scope Summary of Guide Significance and Ue 3 Plackett-Burman Deign Applied to Ruggedne Tet 4 Plackett-Burman Deign Calculation 5 Plackett-Burman Deign Conideration 6 Interpretation of Reult 7 Example 8 Thi guide i under the juridiction of ASTM Committee E- on Statitical Method and i the direct reponibility of Subcommittee E.0 on Tet Method Evaluation and Quality Control. Current edition approved ov. 0, 989. Publihed January 990. Originally publihed a E Lat previou edition E The boldface number in parenthee refer to the lit of reference at the end of thi guide. Editorially corrected. Teting Effect from Repeated (ph) Experiment 9 Controllable veru Uncontrollable 0 Additional Information Table Figure Appendixe Additional Plackett-Burman Deign Short-Cut Calculation Reference Appendix X. X.3..4 Thi tandard doe not purport to addre all of the afety concern, if any, aociated with it ue. It i the reponibility of the uer of thi tandard to etablih appropriate afety and health practice and determine the applicability of regulatory limitation prior to ue.. Summary of Guide. A ruggedne tet i conducted by making ytematic change in the variable aociated with the tet method and oberving the ize of the aociated change in the tet method reult. Generally, the deign (ytematic plan of experimentation) aociated with ruggedne tet are taken from the field of tatitic. 3. Significance and Ue 3. The ruggedne tet of a tet method hould precede an interlaboratory tudy. The interlaboratory (round robin) tudy hould be the final proof tet for determining the preciion of the tet method. If a ruggedne tet ha not been run to determine, and ubequently to retrict the allowable range of the critical variable in the tet method, then the preciion from the round robin may be poor. It may not be known what went wrong, or how to fix the tet method. The ruggedne tet, by tudying the influence of the tet method variable and by indicating the need for elective tightening of tet method pecification, help avoid uch ituation. The ue of ruggedne tet encourage the orderly development of a tet method. 3. Ruggedne teting hould be done within a ingle laboratory o the effect of the variable are eaier to ee. Only the effect of change in the tet method variable from high level to low level need to be determined. umerou variable uch a temperature, preure, relative humidity, etc., may need to be tudied. The influence of thee change are bet tudied under the hort-term, high-preciion condition found within a ingle laboratory.

2 E Plackett-Burman Deign Applied to Ruggedne Tet 4. A erie of Plackett-Burman (P-B) deign are available for ue with ruggedne tet for determining the effect of the tet method variable (ee 4.3 and Appendix X). The effect for each variable i calculated on the bai that a given change in a variable from a high to a low level reult in a fixed change in the tet reult. It i common in ruggedne teting to aume that the oberved effect of the imultaneou change of a number of variable can be decribed a the imple addition of the fixed effect for each variable. It i alo aumed that the effect for each variable i independent of the effect of other variable, that i, there are no coupled influence. The effect that are calculated on the bai of thi aumption are called main effect. If a coniderable lack of independence among the effect of the variable i oberved, the oberver i than forced to recognize additional factor, which are called interaction. The ruggedne tet procedure for dealing with interaction are more complex, and are given in Ref () and (3). Thee more involved procedure, however, require additional meaurement to develop information about the interaction. Thi guide i written only for evaluating main effect. 4. P-B deign require that mut be an integer multiple of four, for example, 4, 8,, 6, etc. P-B deign for meaurement per replicate can be ued to etimate up to - main effect. The calculated main effect, however, will be confounded (contaminated) with the interaction. If the interaction are relatively mall, then the uer may be atified in making only ruggedne tet meaurement and obtaining lightly contaminated etimate for the - main effect. 4.3 A P-B deign for even factor (A through G) and eight meaurement i given in Fig.. Thi deign i uitable for ue whenever an independent etimate of meaurement variability i available. ote that each column of the deign contain an equal number of plu ( + ) and minu ( ) factor etting. A ( + ) for a given factor indicate that the meaurement i made with that factor et at the high level, and a ( ) indicate the factor i to be at the low level. All even factor are et for each meaurement (tet reult). The eight meaurement hould be made in a random order. Typical tet reult are hown at the far right of the deign in Fig.. If lightly le than even factor are being invetigated, imply drop the exce column from the deign. For uch ituation, the experimenter hould conult a tatitician to evaluate the meaurement variability. In thi regard, Ref () (pp. 30 to 30) may be of interet. The experimenter can, however, till ue the technique decribed in Section 6, 7, and 8 of thi guide. Tet Reult FIG. A Plackett-Burman Deign for A P-B deign i contructed uch that the four A( + ) and the four A( ) term will each be aociated with an equal number of B( + ) and B( ) term. The A effect i orthogonal to the B effect, that i, it i not affected by the B effect. In the P-B deign, all main effect (column) are orthogonal to all other main effect (column). Thi orthogonality of the main effect, and the acceptance of poible contamination of etimate for the main effect (by the interaction) are the major characteritic of mot ruggedne tet. For many practical problem thee characteritic are acceptable. 5. P-B Deign Calculation 5. The effect of any factor, uch a A, i calculated a the average of the meaurement made at the high level minu the average of the meaurement made at the low level, for example: Effect A 5 (A~! (A~! 5 ~/!@(A~! (A~!# () Effect A ! ~ !# For the P-B deign, the tandard deviation for an effect, uch a A, i eaily derived by uing Eq along with the tandard deviation of a ingle meaurement,. effect A 5 =~/! (A~!# 5 =~/! () effect A 5 /= The ame equation for the P-B deign apply when the tandard deviation i replaced by it ample etimate,, a follow: effect A 5 /= (3) Section 7 and 9 preent two method for determining a ample etimate of the tandard deviation of a ingle meaurement,. 6. P-B Deign Conideration 6. Eq 3 how that the tandard deviation of an effect i inverely proportional to =, the number of meaurement made. The uer may therefore be tempted to ue large P-B deign. Practical experience, however, favor moderate ize deign. Overly large deign require the correct etting of too many factor, and thi increae the chance for blunder. In addition, large deign require more time to complete and other factor not being conidered in the deign can change and ditort the reult. The effect of incorrect factor etting and of hifting experimental condition are propagated into all of the calculated reult (ee Eq ). The ( 5 8) P-B deign in Fig. i a uitable ize for many experiment. If more factor need to be tudied, a econd ( 5 8) P-B deign may be ued. Thi latter procedure may involve the repeated teting of ome of the more important factor from the firt deign. 6. Ruggedne tet that have mall or only moderate change in the level of the factor tend to have interaction that are relatively mall, that i, the interaction tend to be unimportant relative to the main effect. For uch ituation,

3 E 69 ueful information may be obtained by invetigating additional main effect rather than by invetigating the numerou interaction. 6.3 In general, the ize of all effect in a P-B deign will increae with increaed eparation of the high and low etting of the factor. It eem prudent to ue only moderate eparation of the high and low etting o that the meaured effect will be approximately additive and, at the ame time, reaonably large relative to the meaurement error. For the high and low etting of the factor, it i uggeted that the extreme limit that may be expected to be oberved between different qualified laboratorie be ued. 7. Interpretation of Reult 7. Since the main effect are expreed in the unit of the meaurement, direct judgment can be made a to whether or not the change aociated with the hift of the factor from a high level to a low level i too large. Other, more quantitative method of judgment that analyze the variance of the meaurement are given in 7., 7.3, and 7.4. Thee quantitative method till only give tentative anwer and follow-up or confirmatory experiment are frequently needed. 7. If m auxiliary meaurement, all made under the ame condition a each other are available from other experimentation, the within-laboratory meaurement variability,, can be calculated. A t-tet (with m- df) can be ued to judge if a main effect, uch a A, i tatitically ignificant relative to the meaurement variability, for example: t m 5 effect A (4) effecta ote that the m from the auxiliary meaurement will not generally be the ame a the of the ruggedne tet. Uing Eq 3, the t-tet can be calculated a follow: t m 5 effect A /= A proper etimate for the -value in Eq 5 hould include all of the uncertaintie of a ingle ruggedne tet meaurement. It i therefore deirable that the auxiliary meaurement be made a independently a poible with experimental condition being reet for each meaurement. 7.3 Tighten the tet method pecification if the calculated t-value from Eq 5 i tatitically ignificant, and if the ize of the effect i of practical importance. Thi change hould help reduce the interlaboratory variability. 7.4 The complete P-B-experiment can be replicated to obtain better etimate of the effect of the factor and to get a current etimate of the meaurement variability,. In etimating the meaurement variability, it i neceary to guard againt the occurrence of a poible meaurement hift between the running of the two deign. Thi can be handled mathematically (ee Section 8). 8. Example 8. Thi ruggedne teting example deal with factor that may influence the determination of the ph in dilute acid olution when meaurement are made by ue of a gla electrode. The meaurement procedure ued with the gla electrode have been decribed in Ref 4. The even factor ( (5) 5 8) P-B deign that wa ued i given in Fig.. Thi convenient deign wa firt uggeted by F. Yate (5) and wa frequently ued by W. J. Youden (6) who did much of the pioneering work in ruggedne teting. For thoe experienced with the ue of fractional factorial deign, it i a 7-4 deign. It ha been hown (), by a rearrangement of the row and column, that thi deign i equivalent to the previouly lited P-B deign. 8. The even factor that were tudied are lited in The firt lited level for each factor ha been arbitrarily aigned the poitive ign: 8.. A Temperature, 5 or 30 C, 8.. B Stirring during the ph meaurement: ye or no (denoted a Y or in Table ), 8..3 C Dilution (0.5 ml ditilled H O/0 ml of olution), ye or no, 8..4 D Depth of electrode immerion, or 3 cm below liquid urface, 8..5 E Addition of odium nitrate (ao meq/0 ml olution), ye or no, 8..6 F Addition of potaium chloride (KCl 5.34 meq/0 ml of olution), ye or no, and 8..7 G Electrode equilibration time before meauring the ph, 0 or 5 min. 8.3 The even factor are only a partial lit of factor that may change the oberved value of the ph. Obviouly, all other factor that are not lited above need to be kept contant. The particular, contant level of thee other factor will reult in ome pecific offet in the ph meaurement. In the ruggedne tet, however, thi fixed offet need not be of concern ince meaurement change (the effect) that occur when the even factor ( ) are changed i the primary interet. 8.4 Reult from a ruggedne tet with a hydrochloric acid (HCl) olution are given in Table. The complete experiment wa replicated on a econd day. A different random order of meaurement wa ued for each day. The two et of meaurement reult are given at the far right of Table For the firt et of meaurement in Table, the effect of factor A i calculated from Eq a the difference of the average value when 5 C i ued and the average value when 30 C i ued, for example ( )/4 ( )/ The average and difference of the average (the effect) are given for A-G ( ) in the third and fourth column of Table. Similar calculation for the econd et of meaurement are given in the fifth and ixth column of the table. A hort-cut method for doing thee calculation i given in X FIG. Alternate Form of Plackett-Burman Deign for 5 8 3

4 E 69 TABLE Experimental Deign and ph Meaurement Oberved ph (milli-ph Unit) Y 3 Y Y Y Y Y Y Y Y Y Y Y 3 Y Y Y Y Average TABLE Arithmetic Treatment of ph Meaurement Data (milli-ph Unit) A Level Firt Data Set Second Data Set Average Effect Average Effect Difference, (d) between Effect A A B Y B C Y C D D E Y E F Y F G G A For the purpoe of clarity of preentation, the calculated value of thi table are not given pat the nearet milli-ph unit. The concluion from thi example are not affected by thi rounding of the value ince the tandard deviation are of the order of 0 milli-ph unit. In actual practice, however, at leat one or two additional digit hould be carried o a to keep the rounding error quite mall relative to the difference (d) given in Column 7 of thi table. 9. Teting Effect from Repeated (ph) Experiment 9. In Table, generally good agreement i oberved between the calculated effect from the two et of meaurement. Effect A, D, E, and F are relatively large and are of interet. The average C effect i (6 + )/ To help decide if the C effect value i real, or if it might imply be due to impreciion in the meaurement, ue a t-tet a follow: t 5 (6) average effect c Since the etimate for each effect i now the average of two experiment, each involving obervation, the t-tet given in Eq 5 mut be modified a follow: t 5 (7) /= 9.. Calculation of the Standard Deviation The etimate of the tandard deviation,, and the aociated degree of freedom for the t-tet of Eq 7 are obtainable from the two et of meaurement. Since the two et of meaurement were run on different day, the firt et of meaurement could be offet relative to the econd et. Therefore, the value hould be calculated by a method that i not vulnerable to a poible offet between the two et of meaurement When calculating, note that an offet between the et of meaurement will not affect the value of the calculated effect. The effect for each et are calculated eparately. Therefore, conider the difference between the effect a calculated for the example in 9. (ee alo Table, Column 7). Since the ame effect from the two et of experiment are being conidered, the tatitically expected value of the difference between the effect are zero. The variance of the difference i therefore the expected value of the quared difference, for example, Variance of ~d! 5 Expected value of d ' ( d /~!. (8) An etimate of the expected value of d i obtained by averaging the quare of the difference lited in Table, Column 7. The calculated etimate i 384/ ext, note that the variance of the difference (between the independently determined duplicated effect) i the um of the variance of the two effect. Eq 3 (ee 5.) decribed the ample etimate for the quare root of the variance of an effect. Therefore, Etimated variance of ~d! 5 4 / 4 / 5 8 /. (9) By combining Eq 8 and Eq 9 and rearranging, an etimate of the tandard deviation of a ingle meaurement that ha - df aociated with it can be obtained a follow: 5 =@(d /~!#~/8! (0) 9. The deired t-tet decribed by Eq 7 i obtained by combining Eq 7 and Eq 0 a follow: t 5 () =@(d /~!#~/8!/= Thi t-tet i for the oberved. In the current example, equal eight and: t 7 5 =(d /7 / =6 8.5 t 7 5 =384/7 / = () () Thi quantity, in abolute value, i lightly le than.36, the 5 % critical t-value aociated with 7 df. The calculated t-value i not tatitically ignificant. The C factor decribe the effect of a mall dilution, a may reult from not properly wiping dry the gla electrode. 9.3 A mentioned in 7., if the effect of any factor i too large (for example, A, D, E, and F ) tightening the tet method pecification for that factor i uggeted. The goal, of coure, i to reduce the interlaboratory variability. 0. Controllable veru Uncontrollable 0. In the example in 8., all even factor were controllable (fixed factor), that i, they could be et at pecified high or low level. In the development of a tet method there alo may be factor that cannot be controlled or et. They are called random factor. uch a operator or intrument fit thi decription. A high or a low operator, or a high or low intrument cannot be obtained from the laboratory helf. The ruggedne tet deign cannot efficiently handle uch factor ince the particular two level that are ued in the tet cannot be controlled. 4

5 E Random factor will be encountered in many interlaboratory tudie. In Interlaboratory tet the combined impreciion of many uch factor are meaured. If the combined impreciion of thee factor i expected to be mall, ignoring the eparate factor and imply oberving their combined impreciion in the interlaboratory tet may be done. If their effect i large, however, thi approach i not practical. 0.3 A econd approach to the problem i to make a erie of meaurement in which one random factor i repeatedly ampled while the other factor are kept contant. If large random factor effect are expected, thi approach may be reaonable. The thorough tudy of random effect, however, i a ubject area beyond the cope of thi guide. 0.4 A third approach, which i omething of a compromie, i to repeat a few of the ruggedne tet while changing one of the random variable. In the Section 9 cae, an (uncontrolled) random variable wa the gla electrode ued in the ph meaurement. The original electrode (o. ) wa broken in ue, and a econd gla electrode had to be ued. Thi econd electrode wa of the ame model and manufacturer a the firt electrode. Table 3, however, how that for HCl olution, the Column F (KCl) effect wa quite different for the two electrode. The ruggedne tet experiment were repeated everal time uing different HCl olution to confirm the unexpected Column F electrode effect. 0.5 In Table 3, generally good agreement were obtained for the repeated ruggedne tet experiment uing Electrode and two different HCl olution (o. 5, o. 3). The repective TABLE 3 Calculated Effect for Variou HCl Solution Milli-pH Unit A A Legend for poitive ( + ) factor: A 5 5 C; B 5 olution tirred; C mL dilution; D 5 -cm electrode immerion; E 5 ao 3 added; F 5 KCl added; and G 5 ph meaured at 0 min. B See alo Table reult with Electrode and a third HCl olution (o. 4) are alo in general agreement, except for the Column F effect. The tandard deviation for a ingle meaurement,, are given in the lat column of Table 3, and are of the order of 0 milli-ph unit. Thi i conidered good preciion. 0.6 The Column A number in Table 3 indicate that a 5 C temperature change reult in le than a 50 milli-ph unit change. Auming that the effect are linear, a tet method tolerance of 60.5 C hould allow ph meaurement reult to agree within 0 milli-ph unit. The large and highly variable Column F number of 0-0 milli-ph unit argue againt the ue of KCl. It i realized that the Column F data, by themelve, only tell the difference in the effect of adding or not adding KCl. Auxiliary experience (and data) how good meaurement tability for ph meaurement made without KCl. Therefore, the addition of KCl i excluded from the tet method. A more extenive dicuion of the practical experimental tolerance for ph meaurement, baed on more extenive ruggedne tet reult i reported in Ref 4.. Additional Information. The ruggedne tet experiment can alo be ued to etimate effect for different acid. Repeat ruggedne tet experimental ph reult with two different (controlled) ulfuric acid (H SO 4 ) olution are hown in Table 4.. Both the preciion,, and the agreement of reult between the two H SO 4 olution are good. There are phyical chemical reaon for expecting the ao 3 and KCl (E and F ) effect to be larger with H SO 4 olution than with HCl olution (ee Ref 4). The large effect oberved for ao 3 and KCl in thee ruggedne tet ha reulted in the excluion of both ao 3 and KCl from the ph meaurement tet method. TABLE 4 Calculated Effect for Variou H SO 4 Solution ph Solution ph number 5 B B Electrode Solution number Electrode Milli-pH Unit A APPEDIX (onmandatory Information) X. Additional Plackett-Burman Deign X. Plackett-Burman deign () are available for -value that are integer multiple of four. The following i a method for contructing the deign for 5 4, 8,, 6, 0, and 4. The firt row of each of thee deign i given below for the aociated -value. Each row pecifie which of the - factor will be et at the high level (+) or the low level ( ) X. For any elected -value, the correponding et of - (+) and ( ) ign i written down a the firt row of the deign. The econd row of the deign i obtained by copying the firt row after hifting it one place to the right and putting the lat ign of Row in the firt poition of Row. Thi type 5

6 E 69 of cyclic hifting hould be done a total of - time, after which a final row of all minu ign i added. The reult of thi procedure for the 5 8 Plackett-Burman deign i given in the firt lited deign of thi guide. X.3 Short-Cut Calculation X.3. All of the ruggedne teting calculation are conceptually imple, but tediou to perform. Hand calculator that have at leat nine memory regiter allow hortcut that minimize the arithmetic operation and the keying of the data. It i aumed that the calculation are made on et of eight meaurement. Let the average of thee meaurement be X. Starting from Eq, the derivation of the hortcut method i a follow: where: 5 8, Effect A 5 ( A~! (A~! 5 (A~! F (A~! (A~! G (A~! Effect A 5 4 ( A~! Effect A 5 ( A~!/ X (X.) X (X.) (X.3) X.3. ow rewrite the econd lited deign of thi guide for the firt et of eight ph meaurement, ubtituting the ordered meaurement number for the poitive ign. For the et of eight meaurement, key the meaurement into memory Regiter through 8, repectively, and then calculate the lat term of Eq X.3, which i two time the average of the eight meaurement. Thi quantity i tored in memory Regiter 9. In order to minimize the chance of error, it i adviable to ue the meaurement reult that are tored in memory Regiter through 8 to calculate thi latter quantity. Then imply ue Eq X.3 and the column of the deign table of number hown in Fig. X. to calculate the variou effect a follow: Effect A 5 ~Regiter !/ Regiter milliph (X.4) Effect B 5 ~Regiter !/ Regiter milliph (X.5) Oberved ph (milli-ph Unit) Average FIG. X. Second Alternate Form of Plackett-Burman Deign for 5 8 REFERECES () Plackett, R. L., and Burman, J. P. (946), The Deign of Optimum Multifactorial Experiment, Biometrika, Vol 33, pp () Paule, R. C., Marinenko, G., Knoerdel, M., and Koch, W. F., Ruggedne Teting Part I: Ignoring Interaction, Journal Reearch of the ational Bureau of Standard, Vol 9, 986, pp (3) Paule, R. C., Marinenko, G., Knoerdel, M., and Koch, W. F., Ruggedne Teting Part II: Recognizing Interaction, Journal Reearch of the ational Bureau of Standard, Vol 9, 986, pp (4) Marinenko, G., Paule, R. C., Koch, W. F., and Knoerdel, M., Effect of Variable on ph Meaurement in Acid-Rain-Like Solution a Determined by Ruggedne Tet, Journal Reearch of the ational Bureau of Standard, Vol 9, 986, pp. 7. (5) Yate, F. Complex Experiment, J. Roy, Statitical Society (Supplement), Vol, 935, pp (6) Youden, W. J., Deign for Multifactor Experimentation, Indutrial an Engineering Chemitry, Vol 5, pp. 79A 80A. The American Society for Teting and Material take no poition repecting the validity of any patent right aerted in connection with any item mentioned in thi tandard. Uer of thi tandard are exprely advied that determination of the validity of any uch patent right, and the rik of infringement of uch right, are entirely their own reponibility. Thi tandard i ubject to reviion at any time by the reponible technical committee and mut be reviewed every five year and if not revied, either reapproved or withdrawn. Your comment are invited either for reviion of thi tandard or for additional tandard and hould be addreed to ASTM Headquarter. Your comment will receive careful conideration at a meeting of the reponible technical committee, which you may attend. If you feel that your comment have not received a fair hearing you hould make your view known to the ASTM Committee on Standard, 00 Barr Harbor Drive, Wet Conhohocken, PA