Robust design, modeling and optimization of measurement systems

Transcription

1 Robust design, modeling and optimization of measurement systems Tirthankar Dasgupta Arden Miller C. F. Jeff Wu Department of Statistics Department of Statistics School of Industrial and Systems Engineering Harvard University University of Auckland Georgia Institute of Technology Abstract An integrated approach for estimation and reduction of measurement variation (and its components) through a single parameter design experiment is developed. Systems with a linear signal-response relationship are considered. The noise factors are classified into a few distinct categories based on their impact on the measurement system. A random coefficients model that accounts for the effect of control factors and each category of noise factors on the signalresponse relationship is proposed. A suitable performance measure is developed using this general model, and conditions under which it reduces to the usual dynamic signal-to-noise (SN) ratio are discussed. Two different data analysis strategies response function modeling (RFM) and performance measure modeling (PMM) for modeling and optimization are proposed and compared. The effectiveness of the proposed method is demonstrated with a simulation study and Taguchi s drive shaft experiment. KEY WORDS: Parameter design, measurement error, signal-to-noise ratio, random coefficients model. 1 Introduction In industry, the ability to obtain accurate estimates of certain critical quantities is often an important part of quality management and particularly of process monitoring. For example, the production of a chemical compound may require a complicated process. It is important to have accurate estimates of the composition of the process stream at various stages of the process so that adjustments can be made to maintain a high product yield. 1

2 Consider a measurement system in which Y denotes the observed measurement and M the true value (unknown). The measurement process consists of observing Y, and estimating M from a calibration function Y = g(x,z,m), where X and Z denote respectively the control factors and the unobservable noise variables (which constitute the stochastic component of Y ). Although some components of Z may be controlled and observed during the experiment, none of them will be observed in the actual measurement process. Therefore, it is reasonable to obtain an estimate of the true value M from an average calibration function. Let y be a realization of Y and let h(x,m) = E Z (Y ) = E Z (g(x,z,m)). Assuming h to be an invertible function, an estimator of M can be obtained as ˆM = h 1 (X,y). Measurement variation or Measurement error is typically defined as E( ˆM M) 2, which clearly consists of a bias component E( ˆM) M and a variance (precision) component var( ˆM). Various components of measurement error are defined in the literature, some of which are related to the bias component (bias, linearity) and some to the variance component (repeatability and reproducibility, more commonly abbreviated as R&R). See Joglekar (2003, Ch. 9) for a detailed description of different components of measurement variation. Clearly, measurement variation is the manifestation of all controllable (X) and uncontrollable noise factors (Z) associated with the measurement system. Measurement systems analysis (MSA) is a systematic approach to measure and reduce measurement variation (Carey 1993, Liggett 1997). Traditional MSA consists of two distinct steps - (i) estimation of different components of measurement variation (e.g., gage R&R) using designed experiments (Vardeman and VanValkenburg 1999, Burdick, Borror and Montgomery 2003) and (ii) identification of root causes using a typical problem-solving approach (e.g., Dasgupta and Murthy 2002). However, Taguchi (1987) addressed the more ambitious engineering objective of improving the precision of a measurement system by identifying interactions between control factors (X) and noise factors (Z) through designed experiments and consequently identifying settings of X that will reduce measurement variation. This approach is a special case of parameter design with signal-response systems (Wu and Hamada, 2000, Ch. 11), where the true value M is the signal, the observed value Y is the response, and reducing the measurement variation is equivalent to maximizing the signal-to-noise ratio in some sense. A critical study of Taguchi s parameter design approach for signal-response systems was done by Miller and Wu (1996). 2

3 Although the traditional MSA approach and the parameter design approach share the common objective of reducing measurement variation, a link between them has not been established. This can, however, be done if we (i) recognize the fact that different types of noise factors account for different components of measurement variation, (ii) incorporate effects of different classes of noise factors into the signal-response model, (iii) develop appropriate performance measures, the optimization of which will lead to reduction of measurement variation, and (iv) develop a strategy for estimation and optimization of such performance measures from designed experiments. The bottom line is that an appropriate classification of noise factors can result in a much better understanding of measurement variation, subsequently leading to its reduction. This possibility has not been explored earlier in robust design literature. In this paper, we discuss an integrated approach for estimation and reduction of measurement variation (and its components) through a single parameter design experiment for a linear calibration function. We classify the noise factors into a few distinct categories based on their impact on the measurement system, citing practical examples for each category. Next, we propose a random coefficients statistical model that distinctly accounts for the effects of control factors and of each category of noise factors on the signal-response relationship (assumed to be linear). Our model differs from standard random coefficients models such as those found in Longford (1993) in that the expected values of the random coefficients are modeled as functions of the experimental factors. A suitable performance measure is developed for the generalized model. A comprehensive data analysis strategy for modeling and optimization is proposed. In the following section we describe the role of different types of factors in the context of the experiment, propose a classification of the noise factors, and discuss how the experiment should be designed. In Section 3, we develop the random coefficients model based on our classification of noise factors and derive appropriate performance measures under different assumptions. The analysis strategy is discussed in detail in Section 4. This section consists of two subsections in which two analysis strategies response function modeling (RFM) and performance measure modeling(pmm) are discussed. In Section 5, the proposed procedures are demonstrated and evaluated using a simulation study. An application to real-life data is demonstrated using Taguchi s drive shaft experiment in Section 6. The principal findings and contributions are summarized in Section 7. 3

4 2 Types of Factors and the Experimental Set-up Figure 1 summarizes the different types of factors in a parameter design experiment for measurement systems. As in any parameter design experiment for signal-response systems (Miller and Wu 1996), here we shall consider the following three classes of factors: X M g( X, Z, M) Y P Z Q N U Figure 1: Different types of factors in parameter design for measurement systems 1. Control factors X, which will be varied in the experiment with the objective of determining their best settings. 2. Noise factors Z constituting the stochastic component of the signal-response model. A key element of our approach is that two distinct types of variability occur for most measurement systems, which we will call measurement-to-measurement (short-term) variability and application-to-application (longer-term) variability. The first type is the variability observed when the same object is measured twice in succession by the same person (appraiser) and is the result of factors like the appraiser s concentration or within-product variation. The second type is the variability observed when the measurement process is applied by different appraisers on different occasions and is the result of factors like the appraiser s skill level, environmental conditions, variation over time. In measurement terminology the first type of variability is called repeatability. The between-appraisers component of the second type of variability is called reproducibility. We shall use P to represent sources of measurement-tomeasurement variability and Q to represent sources of application-to-application variability. 4

5 In our experience, it is extremely difficult to identify and control sources of measurementto-measurement variability and thus the noise factors that are included in experiments are almost always members of Q rather than P. Thus we split Q into [N,U], where N is the set of noise factors included in the experiment and U is the set of uncontrolled factors. Examples of N include appraiser-to-appraiser variation, day-to-day variation, change of reagents in a measurement process involving chemicals; whereas change in ambient temperature is an example of U (see Joglekar 2003, Ch. 9 for more examples). 3. Signal factor M which, in most of the experiments will be the true value of the characteristic being measured. Although typically gage R&R studies do not require measurement of the true value of the characteristic, it is usually possible to obtain a reference value for each part being used in the study. An in-depth discussion of designing experiments for signal-response systems can be found in Miller and Wu (1996). In this section, the basic results are presented in terms of measurement systems. To facilitate the discussion we adopt the following notation: let D(a) denote a design for the set of factors a and let D(a) D(b) denote a cross-array design between two sets of factors a and b. The cross-array design (Wu and Hamada, 2009, Ch. 11; Mukherjee and Wu, 2006, Ch. 9) is the design for a b which is constructed by combining each run of D(a) with every run of D(b). For signal-response systems the modeling strategy dictates the structure of the design array. If a PMM strategy is used, then the design array should be of the form D(X) D(N) D(M). To reduce run size, the practitioner can use a fractional factorial design for D(X) or D(N) or both. For an RFM strategy, the design array should be of the form D(X,N) D(M) note that this class of designs includes all of the D(X) D(N) D(M) designs and thus has greater flexibility. To reduce run size in this case, the practitioner can use a fractional factorial design for D(X,N) which means that a smaller design is often feasible see Wu and Hamada (2009, Ch. 11). As will be discussed later in Section 4.1, the number of levels of the signal factor M should preferably be larger than 4. For signal-response experiments the signal factor is often easier to adjust than either the control factors or the noise factors and thus a split-plot design with the signal factor levels assigned to the split-plots is more cost-efficient than a completely randomized design, which is compatible with both types of design array discussed above. We believe that this will usually be the case for measurement system experiments. For measurement systems changing control factors settings 5

6 typically involves reconfiguring the measurement apparatus or modifying the measurement procedure; changing noise factor levels typically involves changing operators (or batches of reagents) or adjusting environmental conditions; but changing the level of the signal factor usually means that the measurement process is repeated on different standards. The models we propose in the next section are specifically designed for experiments that employ this type of split-plot design. 3 Statistical Models and Performance Measures In this section, we develop statistical models and performance measures for measurement systems. In order to distinguish between the two types of variability, we propose the following additive model Y = µ(x,n;m) + σ(x,n;m)ǫ, (1) where ǫ s are independent N(0,1) variables. The above model is similar to the functional response model proposed by Nair, Taam and Ye (2002), but has a couple of key differences. Nair et al. (2002) assumed that all the noise factors whose effects can be moderated by adjusting the control factor settings are included in the experiment. Further, these noise factors were assumed to affect the mean part of model (1) only. This means the error term σ in their model was independent of X and N. This is clearly not the case in our model. Further, in model (1), µ(x,n;m) and σ(x,n;m) are random functions because of U. This model therefore allows for the possibility of finding control factor settings that reduce both application-to-application variability and measurement-tomeasurement variability. For measurement systems it makes sense to base performance measures on the precision of the estimates of M and, in particular, on V ( ˆM). As defined in Section 1, let h(x,m) = E Z (Y ) = E Q E P [( µ(x,n;m) + σ(x,n;m)ǫ ) Q ] = EQ ( µ(x,n;m) ). ( ) ( ) Note that E Q µ(x,n;m) = EU E N X,N;M) U ; hence, the expectation is only a function of X and M. The estimate of M for Y = y o is obtained from ˆM = ψ(x,y o ), where ψ(x,y ) is the inverse function of h(x,m). Using the delta method we can approximate V ( ˆM) as (dψ(x,y )/dy ) 2 V (Y ), giving the expression V ( ˆM) (dψ(x,y )/dy ) 2 [ V Q (µ(x,n;m)) + E Q ( σ 2 (X,N;M) )]. (2) 6

7 This provides a general basis for creating performance measures for measurement systems. It has three key components: the sensitivity of ψ(x, Y ) to changes in Y, the variability in the calibration function and measurement-to-measurement variability. Observe that in general V ( ˆM) is a function of the the signal factor M as well as the control factors. The goal of an experiment is to identify settings of X that make this function as small as possible across values of M. Therefore, as pointed out by Joseph (2003), a summary measure such as the mean of V ( ˆM) over the distribution of M needs to be identified and minimized with respect to X. For measurement systems, the calibration function is usually described by a simple parametric function. As a result, the effect of the experimental factors on V ( ˆM) can be assessed via their impact on the parameters in the calibration function. Suppose the calibration function depends on parameter vector β = (β (1),...,β (B) ) t whose values depend on X and N (if noise factors are utilized). To emphasize this relationship, we can write the calibration function as µ (β (X,N) ;M). A two-stage analysis strategy is adopted, where the data obtained from an experiment is used to model the elements of β and σ as functions of X and N and then these models are applied to (2) to identify preferred settings for the control factors. It is readily apparent that this will work for dψ(x,y )/dy and E Q ( σ(x,n;m) ) as these components of V ( ˆM) will be functions of β and σ. For the V Q (µ(x,n;m)) component, the connection is less apparent but can be made by thinking of the variability in µ(x,n;m) as being transmitted through variation in β. If β is thought of as a random vector with covariance matrix Σ β, then an application of the multivariate version of the delta method gives: [ ] µ(β;m) t [ ] µ(β;m) V (h(x,m)) = Σ β, (3) β β where [ µ(β;m)/ β] is a vector containing the partial derivatives of µ(β;m) with respect to each of the parameters. Thus for V Q (µ(x,n;m)) the experimental data is used to determine how changing control factors settings will affect the elements of Σ β and then (3) is applied to translate into the impact on V Q (µ(x,n;m)). For a complicated parametric function this would be a daunting prospect but most calibration functions are very simple (often linear) and this is quite straightforward. The set of statistical models we propose are based on a split-plot structure as discussed in Section 2 (levels of the signal factor are assigned to split-plots) and allow the parameters of the calibration function to vary between applications. Let y ijk denote the observed response corre- 7

8 sponding to the i th level combination of the control factors X i, the j th level of the signal factor M j and the k th level combination of the noise array N k. Further let β (b) ik and σ2 ik represent the actual values (at the time the data, is collected) of β (b) and σ 2 for X i combined with N k. Estimates are obtained by fitting the model for the calibration function to the subset of the data where the control and noise factors are fixed as specified and the signal factor is varied across its levels. Then we propose: y ijk = µ (β (X i,n k ) ;M j ) + σ (X i,n k ;M j ) ǫ ijk, (4) β (b) ik = Z ik Ω(b) β + σ(b) β τ ik, (5) ln(σ 2 ik ) = Z ik Ω σ + σ σ ζ ik, (6) i = 1,2,... I, j = 1,2,... J, k = 1,2,... K, b = 1,2,... B Note the following for (4)-(6): ǫ ijk, τ ik and ζ ik are independent N(0,1) variables. Z ik denotes a combined vector of terms involving X i and N k (including interaction terms) and Ω the corresponding vector of coefficients. For a given setting of control factors X i and noise factors N k, the observed variation in y ijk is due to unobservable causes of measurement-to-measurement variability P. In some situations, this variation depends on M j, e.g., a larger value of the signal factor may lead to a larger variation (Joseph 2003). The randomness of β and σ comes from two different sources : (i) the observable noise N, which is itself a random variable, and (ii) the unobservable noise U, the effect of which is manifested in the variance components σ β and σ σ. A standard split-plot model would allow the intercept to vary from application to application but would fix all the remaining parameters in the calibration function. As a result, it only accommodates variation caused by the calibration function being shifted up or down. Our model allows all of these parameters to vary which improves its ability to detect different types of variability in the calibration function - it can also detect variation caused by changes in the shape of the calibration function. 8

9 We now address the case of linear calibration function in detail as this is very common for measurement systems. Consider the case, where the calibration function has a fixed intercept α (possibly 0) and a slope β that may vary from application to application, i.e., µ(x;n,m) = α(x)+β(x,n)m, where β(x,n) is a random function due to U. Thus equations (4)-(6) become: y ijk = α i + β ik M j + σ ik ǫ ijk, (7) β ik = Z ik Ω β + σ β τ ik, (8) ln(σik 2 ) = Z ik Ω σ + σ σ ζ ik, (9) i = 1,2,... I, j = 1,2,... J, k = 1,2,... K. Figure 2 shows how the different categories of noise components are connected to the corresponding components of (7)-(9). In the remainder of this article, we shall study only models (7)-(9) in detail. var(y ijk ) ALL NOISE FACTORS y ijk M i ik j ik ijk P Q ik ln( 2 ik Z ik ) Z ik ik ik N U Figure 2: Classification of noise factors and their impact on measurement variation The signal corresponding to Y = y o can be estimated as ˆM = y o α E Q (β), (10) where E Q is used to emphasize that the expectation is taken over Q (N and U). For fixed M and with respect to the expression for V ( ˆM) in (2), we can make the following replacements: dψ(x,y )/dy = 1/E Q (β ik ), V Q (µ(x,n;m) = (d h(y )/dβ) 2 V Q (β) = M 2 V Q (β) = M 2 ( V N (Z Ω β ) + σ 2 β), E Q [ σ 2 (X,N;M) ] = E Q (σ 2 ) = e σ2 σ/2 E N (e Z Ω σ ). 9

10 Thus we obtain the following proposition: PROPOSITION 1. For fixed M, the variance of ˆM for a measurement system that can be adequately modeled using (7)-(9) can be approximated by V ( ˆM) eσ2 σ /2 E N (e Z Ω σ ) + V N (Z Ω β )M 2 + σβ 2 ( ) M2 2. E N (Z Ω β ) As mentioned previously, this variance is a function of the signal factor M in addition to the control factors X. Therefore, a summary measure such as average V ( ˆM) needs to be taken over ( ) the distribution of M before minimizing it over X. Since minimizing E M V ( ˆM) is equivalent to ( ( ) maximizing E M V ( ˆM) ) 1, an appropriate performance measure that needs to be maximized is given by PM = ( ) 2 E N (Z Ω β ) e σ2 σ/2 E N (e Z Ω σ) + VN (Z Ω β )E(M 2 ) + σ 2 β E(M2 ). (11) However, this will not be necessary if one assumes that the error variance in (7) is proportional to the square of the level of the signal factor. In that case the first term in the numerator of V ( ˆM) will also contain M 2 ; and after taking expectation on M, the term E(M 2 ) can be ignored. We thus have the following corollary. COROLLARY 1.1. Under the assumption that the error variance in (7) is proportional to M 2, i.e., y ijk = α i + β ik M j + σ ik M j ǫ ijk, the performance measure is independent of M and is given by ( ) 2 E N (Z Ω β ) PM 1 = e σ2 σ/2 E N (e Z Ω σ) + VN (Z Ω β ) + σβ 2. (12) Next we consider another special case. If the effect of U is negligible, which means all the important components of Q are included in the experiment, then we have σβ 2 0 and σ2 σ 0. Further, if the slope β is not affected by N, we have E N (Z Ω β ) = Z Ω β and V N (Z Ω β ) = 0. In such a case the performance measure in (11) is further simplified and we have the following corollary. COROLLARY 1.2. If the effect of U is negligible and the slope β is not affected by N, then maximizing the performance measure in (11) is equivalent to maximizing PM 2 = (Z Ω β ) 2 E N (e Z Ω σ). (13) 10

11 Estimated performance measure Assume that the signal factor is scaled such that its mean is zero. If we fit models ˆβ = Z ˆΩβ and ln(ˆσ 2 ) = Z ˆΩσ, and obtain ˆσ 2 β and ˆσ2 σ as estimates of σ 2 β and σ2 σ respectively (estimation methods discussed later), the performance measures in (11) - (13) can be estimated as PM = PM 1 = PM 2 = Note that maximizing PM 2 is equivalent to maximizing ( ) 2 E N (ˆβ) eˆσ2 σ /2 E N (ˆσ 2 ) + V N (ˆβ) V (M) + ˆσ β 2 V, (14) (M) ( ) 2 E N (ˆβ), (15) eˆσ2 σ/2 E N (ˆσ 2 ) + V N (ˆβ) + ˆσ β 2 ˆβ 2 E N (ˆσ 2 ). (16) PM 2 = ln(ˆβ 2 ) ln E N (ˆσ 2 ). (17) The following points may be noted with respect to the above performance measures and their estimates: (i) In general (except in (16)), both the numerator and denominator of the estimated performance measure will depend on X and the distribution of N. Each component of N is a random variable in reality, and some idea about its distribution (at least its first two moments) needs to be formed from historical data and/or practical experience. Without this knowledge, it will not be possible to compute expectation or variance with respect to N. The idea is similar to obtaining transmitted variance from a response model (Wu and Hamada, Chap. 10). (ii) The performance measure PM 2 given by (16) is the same as the usual dynamic signal-to-noise (SN) ratio given by SN = ˆβ 2 ˆσ 2, (18) if N is not considered. Even if N is considered, its form is equivalent to the dynamic SN ratio, where the denominator represents a pooled variance estimated across levels of the noise factor. Thus, if the assumptions of Corollary 1.2 hold good, it is reasonable to use the dynamic SN ratio as performance measure. 11

12 4 Analysis Strategy We shall now develop an analysis strategy for the set of models given by (7)- (9). The eventual goal of analysis is to model the chosen performance measure as a function of control factors and then determine the settings that optimize the performance measure. There are two distinct approaches to developing such a model: response function modeling (RFM) and performance measure modeling (PMM). In the former, the experimental data is used to model the signal-response relationship as a function of the control and noise factors. The specified performance measure is then evaluated with respect to the fitted models in order to select preferred levels of the control factors. PMM requires a two-stage procedure. First, obtain an estimate of the PM for each combination of control factors used in the experiment. Then use these estimates to model the PM as a function of the control factors. An overview of these two approaches in the context of parameter design experiments for signal-response systems can be found in Miller and Wu (1996). Both of these approaches can be handled by the set of models given in (7)-(9). In this section we compare the procedures using this common framework. 4.1 Response Function Modeling In this approach we need to consider a design matrix, Z, which includes both control and noise factors (X and N). Let Z i denote the ith row of Z. The data obtained from such an experiment can be modeled using the framework (7)-(9) by dropping the suffix k. y ij = α i + β i M j + σ i ǫ ij, (19) β i = Z i Ω β + σ β τ i, (20) ln(σ 2 i ) = Z i Ω σ + σ σ ζ i, (21) i = 1,2,... I, j = 1,2,... J. First, consider the modeling of σ 2. Let s 2 i denote the residual variance obtained from the linear regression of y on M for the ith row of Z. Bartlett and Kendall (1946) have used ln s 2 to study variance. Adapting their approach to the current situation gives ln(s 2 i ) = ln(σ2 i ) + (2/ν)1/2 ǫ 1i, (22) 12

13 where ν = J 2 and ǫ 1i is approximately N(0,1) this result is valid for J 5. Substituting (21) into (22) gives ln(s 2 i) = Z iω σ + (σσ 2 + 2/ν) 1/2 ς i, (23) where ς is approximately N(0,1). So a half-normal plot can be used to evaluate significant effects. If Z contains replication then a direct estimate of σσ 2 can be obtained and utilized in the analysis. It will be possible to obtain maximum likelihood estimates for unreplicated data. For β, we have the following model ˆβ i = β i + (σi 2 /S mm ) 1/2 ǫ 2i, (24) where S mm = j (M j M) 2. Substituting in the model for β i gives ˆβ i = Z iω β + (σ 2 β + σ2 i /S mm ) 1/2 ǫ 3i, (25) where the error terms are independent N(0,1). If Z contains replication, then it is possible to estimate σ 2 β by plugging in estimators of Ω σ and σ σ into var(ˆβ i ) = σ 2 β + exp(z Ω σ + σ 2 σ/2)/s mm. This is not possible if Z does not contain replication. In this case, the estimates ˆβ i have nearly constant variance if σβ 2 is large with respect to the σ2 i s or the σ2 i s have similar values and so a halfnormal plot is a valid form of analysis only under these circumstances. We recommend identifying a reduced model based on a half-normal plot and then checking the fitted model for non-constant variance. If evidence of non-constant variance is found, alternative analyses are recommended. The RFM approach can thus be summarized by the following stages: 1. Compute ˆβ i and s 2 i corresponding to the ith row. 2. Use half-normal plots for β and ln(σ 2 ) and a joint effects plot to detect significant effects for β and ln(σ 2 ). 3. Fit separate models for ˆβ and ln(s 2 ) in terms of significant control and noise factors. 4. Estimate σσ 2 directly and σβ 2 as described earlier from replicated data. For unreplicated data, maximum likelihood estimators may be obtained. 5. Compute a performance measure given by (14), (15) or (16) depending upon the experimental set-up. 6. Determine control factor settings that maximize the performance measure. 13

14 4.2 Performance Measure Modeling In this approach, an estimate of the selected performance measure is calculated for each combination of control factors in the experiment. Then the performance measure is treated as a response and is modeled directly as a function of control factors. Taguchi s approach is a specific example of this method. Consider the cross-array data and the model for PMM analysis using the framework described by (7)-(9). The control array consists of I rows, the signal factor has J levels and the noise array has K rows. For the JK data points corresponding to each row of the control array, Taguchi s approach is to fit K straight lines corresponding to the different rows of the noise array where the lines have a common slope but different intercepts. Next, a modified version of the dynamic SN ratio ˆω = β 2 /s 2 is computed for each row as the performance measure, where s 2 denotes the residual variance obtained for the fitted model. Taguchi s approach thus involves fitting a model ŷ ijk = ˆα ik + ˆβ i M j for the ith row (i = 1,2,... I) of the control array. In the following discussion, we shall drop the suffix i, since the computation of the SN ratio is identical for each row of the control array. The least squares estimators of α k and β in the fitted model ŷ jk = ˆα k + ˆβM j are given by ˆα k = ȳ.k ˆβ M, j k ˆβ = (y jk ȳ.k )(M j M) k (M j M) 2 = j j k (y jk ȳ.k )(M j M), (26) KS mm where ȳ.k = j y jk/j and S mm = j (M j M) 2. It can easily be shown that E(ˆβ) = k β k/k (= β, say) and var(ˆβ) = k σ2 k /(K2 S mm ), where σk 2 = var(y jk U). The residual sum of squares has degrees of freedom (J 1)K 1 and is given by SSE = (y jk ȳ.k ) 2 K ˆβ 2 S mm. j k For a given row of the control array, the estimated performance measure is obtained as PM = ˆβ 2 s 2, (27) where ˆβ ( ) is the estimated slope for that row, s 2 = SSE/ (J 1)K 1. Note that both ˆβ 2 and s 2 are functions of the control factors, although the suffix i is removed. The following proposition (proof in Appendix) is useful in studying whether s 2 can be a reasonable estimate for the denominator of the performance measures defined in (11)-(13). Note that 14

15 β 2 may be considered reasonable for the numerator, as the fitted slope is, in effect, an average over the K different combinations of noise factor levels. PROPOSITION 2. For chosen levels M 1,...,M J of the signal factor, the conditional expectation of s 2 for fixed U (which means β k is a constant) is E(s 2 U) = γs 2 1 m K 1 where γ = (J 1)(K 1) (J 1)K 1 and s2 m = S mm /(J 1). K (β k β) K k=1 K σk 2, (28) Recall that the denominator of PM given by (11) consists of the sum of two parts E(M 2 )V Q (β) and E Q (σ 2 ). Clearly, K k=1 (β k β) 2 /(K 1) measures the variance of β and K k=1 σ2 k /K the average of σ 2 over different noise combinations. Therefore from Proposition 2, heuristically s 2 seems to be a reasonable estimator. A theoretical comparison of E(s 2 ) and the denominator of PM in (11) is possible if we assume that the noise combinations and levels of the signal factor are chosen at random, which is usually not the case in practice. However, there are situations where this is still possible. For example, if the noise factor is of spatial type (e.g., operator, machine, or location) and there are numerous choices, the experimenter may choose a few out of them either by random or systematic sampling. For the signal factor, this is also plausible when a few products are randomly chosen for measurement from a large lot. For such situations we have the following corollaries, all of which assume that the levels of each noise factor and the signal factor are chosen at random. k=1 COROLLARY 2.1. Assume that the levels of each noise factor and the signal factor are chosen at random. Then, E(s 2 ) = γσ 2 M ( ) var N (Z Ω β ) + σβ 2 + e σ2 σ/2 E N (e Z Ω σ ), where the expectation is over P,U,N and M, and σ 2 M denotes the population variance of M. It is easy to see that 0 < γ 1, with equality holding if and only if J = 2. Thus, for J > 2, we have γσ 2 M < σ2 M E(M2 ), which means that s 2 is a negatively biased estimator of the denominator of PM in (11). The following corollary helps us obtain an unbiased estimator. COROLLARY 2.2. Assume that the levels of each noise factor and the signal factor are chosen at random. Consider K individual regression lines ŷ jk = ˆα k + ˆβ k M j fitted with K sets of paired 15

16 observations (y jk,m j ), k = 1,...,K, j = 1,...,J. Then, an unbiased estimator of the denominator of PM in (11) is given by ( 1 s 2 adj = s2 + J J )( K Mj 2 γs ˆβ 2 k=1 k 2 K ˆβ 2 m K 1 j=1 1 KS mm K k=1 s 2 k ), (29) where s 2, s 2 m and ˆβ 2 are as defined earlier, and s 2 k the fitted model ŷ jk = ˆα k + ˆβ k M j. is the residual sum of squares associated with Note that, if the conditions stated in Corollary 1.2 are met, that is, the main effect of U is negligible, and the slope β is not affected by N, Corollary 2.1 leads to E(s 2 ) = E N (e Z Ω σ ), which is the same as the denominator of the performance measure PM 2 defined in (13). This leads to another corollary: COROLLARY 2.3. Assume that the levels of each noise factor and the signal factor are chosen at random. If the main effect of U is negligible, and the slope β is not affected by N, then s 2 defined in Proposition 2 is an unbiased estimator of the denominator of PM given by (11), which is the same as the denominator of PM 2 defined by (13). 5 A Simulation Study We examine the effectiveness of the proposed estimation methods using a simulation study. We assume the true signal-response relationship to pass through the origin (that is α i = 0 in (7)). The simulation model includes two control variables X 1 and X 2, and two noise variables N 1 and N 2 that are deliberately varied during the experiment. The following model is considered: y = β(x,n)m + σ(x,n)ǫ, β(x,n) = X N X 1 N 1 + σ β τ, ln σ 2 (X,N) = X N X 2 N 2 + σ σ ζ, (30) where ǫ, τ and ζ are N(0,1) variables, σ β = 0.1 and σ σ = We assume the following two-point discrete distribution for both the noise variables N 1 and N 2 : 1 with probability 0.5 N k = +1 with probability 0.5, 16

17 X 1 X 2 True PM Table 1: True Values of the Performance Measure for k = 1,2. It is also assumed that the signal factor M can take values -2,-1,0,1,2 with equal probabilities, so that E(M) = 0 and V ar(m) = E(M 2 ) = 2. Factor levels and designs for RFM and PMM: Two levels (-1 and +1) of each of the four variables (two control and two noise) are considered in the experiment. Thus, the design for RFM is a 2 4 design, while the design for PMM is a cross array design. The performance measure and its true values: It can easily be seen that the true performance measure (as a function of the control variables) given by (11) takes the form PM true (X 1,X 2 ) = ( X 1 ) ( ) 2 0.5e X 2 e X 2 + e X 2 + 2( X1 ) (31) Table 1 shows the true values of the performance measure corresponding to the four possible combinations of the two control variables. Clearly, X 1 = 1 and X 2 = 1 is the most desirable combination (that maximizes the performance measure). Note that this is evident from (30), since this choice removes the effects of N 1 and N 2 from the response. Also, X 1 = 1 maximizes the numerator of (31) since its coefficient (0.35) is positive. Response Function Modeling: As mentioned earlier, a 2 4 design was used with four factors X 1, X 2, N 1 and N 2, each at two levels, -1 and +1. For each of these 16 runs, β and σ were simulated using the second and third equations of (30). Next, five values of y corresponding to M = 2, 1,0,1,2 were generated by substituting the simulated values of β and σ into the first equation of model (30). Thus, the data was a 16 5 matrix of observations. For each control noise combination, i.e., for each row of the design matrix, a linear regression equation ŷ = ˆβM was fit and the residual variance σ 2 was estimated as the mean square error s 2 of the fitted model. Next, models ˆβ = Z ˆΩβ and log( ˆσ 2 ) = Z ˆΩσ were fit. 17

18 First, we present the result of one particular simulation. Variable selection was done using half-normal plot (See Figure 3). The fitted reduced models were: ˆβ = X N X 1 N 1, log(ˆσ 2 ) = X N X 2 N 2. absolute effects N1 x1:n1 x1 absolute effects N2 x2:n2 x half normal quantiles half normal quantiles Half-normal plot for β Half-normal plot for log(σ 2 ) Figure 3: Simulated experiment(rfm) Replacing the coefficients in (31) by the corresponding components of ˆΩ β and ˆΩ β, the estimated performance measure (given by (14)) for the four combinations (-1,-1), (-1,1), (1,-1), (1,1) are 1.208, 0.170, 9.296, Thus, the best combination is clearly identified. Note that the half-normal plot for β can be somewhat misleading and lead to the choice of only X 1 as the significant effect. However, even in that case (i.e., ˆβ = X 1 ), the estimated values of the performance measure for the four combinations (-1,-1), (-1,1), (1,-1), (1,1) are 1.962, 0.180, 9.58 and respectively, so the best combination is still identified without much trouble. This is due to the positive coefficient of X 1 in the model for β, and will not be true in general. The summary statistics for the estimated P M values from 500 such simulations (done by directly fitting the reduced model) using the RFM approach is shown in Table 2. Performance Measure Modeling: For each of the four rows of the control array, 20 observations on y were generated corresponding to the four combinations of levels of the noise factors and five levels of the signal factor using model (30). As described in Section 5.2, a single straight line passing 18

19 X 1 X 2 True PM Mean( PM) Median( PM) Min( PM) Max( PM) sd( PM) Table 2: Results of 500 simulations using the RFM approach X 1 X 2 True PM Mean( PM) Median( PM) Min( PM) Max( PM) sd( PM) Table 3: Results of 500 simulations using the PMM approach through the origin was fit for each of the four rows, and the performance measure for the ith row (i = 1,...,4) was estimated as PM i = β ˆ i 2/s2 i. The results of 500 such simulations are summarized in Table 3. Comparison of RFM and PMM: Comparing Tables 2 and 3, we find that both methods are useful in identifying the best combination of X 1 and X 2. Distributions of PM have their locations close to the true value of PM. In general, the estimators from RFM are seen to be more efficient than the estimators obtained from PMM. Thus, the RFM approach is capable of producing better results. Although PMM produces satisfactory results in this example, it is important to note that there are several situations, especially with fractional factorial designs, where PMM may lead to incorrect conclusions. For example, it may fail to capture dispersion effects under certain aliasing patterns in the control factors (Steinberg and Burzstyn 1994). Our simulation study does not run into such problems as it uses full factorial designs. However, the simplicity of the PMM approach and the fact that it involves much less computation sometimes make it attractive to practitioners. 19

20 A: Testing Machine A 1 : New A 2 : Old B: Master Rotor B 1 : #1 B 2 : #2 B 3 : #3 B 4 : #4 C: Rotations at Handling Time C 1 : Current C 2 : New D: Rotations at Measurement D 1 : Current D 2 : New E: Signal Sensitivity E 1 : 10 E 2 : 20 E 3 : 30 E 4 : 40 F: Sequence of Correction of Imbalance F 1 : Current F 2 : Reverse F 3 : New #1 F 4 : New #2 G: Imbalance Correction Location G 1 : Current G 2 : New Table 4: Control Factors for Drive Shaft Case Study 6 Analysis of Drive Shaft Data The data from an experiment (described in Taguchi 1987) that investigated a system of measuring the residual imbalance of automobile drive shafts will be used to illustrate our methodology. Manufactured drive shafts are often not adequately balanced which results in noise and vibration. This problem can be corrected using balance weights if an accurate measurement of the amount of imbalance can be made. An experiment was undertaken with the goal of identifying control factor settings that would produce the most precise measurements. Table 4 lists the control factors included in the experiment. As it was thought that the relationship between Y and M may be affected by differences between drive shafts, drive shaft was included as a noise factor at three levels, corresponding to three different drive shafts (DS 1, DS 2, DS 3 ) which had varying degrees of imbalance. The control factors were varied according to the design matrix in Table 5. For each combination of control factors, the imbalance of each drive shaft was measured as is and with 10g, 20g and 30g of attached weight. A linear relationship was assumed between the response Y and the amount of attached weight M (M 1 = 0g, M 2 = 10g, M 3 = 20g and M 4 = 30g): Y = α + βm + ǫ, which was estimated for each drive shaft using these four observations. There are two key features of this data which are not typical of most measurement system experiments. First, although the attached weight M is treated as the signal factor, it does not represent the actual signal. For this measurement system the actual signal is the amount of im- 20

21 Control Factors Control Factors Run A B C D E F G Run A B C D E F G Table 5: Control Array for Drive Shaft Case Study balance in a drive shaft. Thus for a drive shaft with an attached weight, the signal is equal to the imbalance of the drive shaft itself plus the imbalance induced by the attached weight: let M represent the actual signal, then M = z +M where z is the unknown imbalance in the drive shaft. A typical calibration experiment would involve measurements on a set of standards for which the actual signal (M ) is known with a high degree of precision in this case a set of drive shafts with known values of imbalance. Adding known weights to a single drive shaft, in effect creates a set of pseudo-standards which can be used to estimate the calibration function parameters. To see this, note that the calibration function is assumed to be a straight line through the origin E(Y ) = βm and we wish to get estimates of the slope β and the residual standard deviation σ. If we were using data obtained using an ordinary set of standards, this would be done by fitting a regression through the origin model. It can also be done for data obtained using the pseudo-standards. Note that, by substituting M = z + M into the calibration model, we get E(Y ) = α + βm where α = β z. Thus by fitting a linear model (with intercept) to data obtained using the pseudo-standards, we can estimate β and σ for the calibration model the estimate of α is ignored since it depends on the level of imbalance in the drive shaft which is not of interest when estimating the calibration model. The second unusual feature is that the levels of the noise factor represent different units. Since measurements corresponding to different levels of signal are made on the same unit, unit-specific calibration curves can be estimated and used to investigate the impact of differences between units. For a typical calibration experiment, each standard would be a separate unit and thus the impact 21

22 of unit-to-unit differences would show up as part of the residual variation for each fitted calibration and could not be isolated. Although the nature of this noise factor is unusual, it can still extract a component of the residual variation and thus, for the purposes of analysis, it is treated in the same way that any other noise factor would be treated. So although we would not view the drive shaft example as a typical measurement system experiment, its unusual features have little impact on the application of our methodology. We now demonstrate the PMM and RFM approaches using the (flange-side) drive shaft data in Table 6. This data contains several unusual observations (denoted by brackets) which were deleted and for the affected runs the analysis was done using the three remaining observations. Plots (not shown here) of the measured imbalance Y versus the load M for each drive shaft/experimental combination run confirm a linear relationship. These plots also show that for DS3 there is a large negative intercept indicating a high degree of imbalance, whereas for both DS1 and DS2 the intercepts are close to the origin. DS1 DS2 DS3 Run M 1 M 2 M 3 M 4 M 1 M 2 M 3 M 4 M 1 M 2 M 3 M (-14) (17) Table 6: Drive Shaft Data (flange side) First, consider an analysis using the PMM approach. Following Taguchi s approach (described in Section 4.2), for each control factor combination a linear model was fitted that had a different intercept for each drive shaft but a common slope. The estimated performance measure, denoted 22

23 RFM Analysis PMM Analysis DS1 DS2 DS3 Run ˆβ s 2 ˆβ s 2 ˆβ s 2 ˆβ s 2 ˆ PM i Table 7: Estimated Parameters for Drive Shaft Data by ˆ PM i, was taken as ln(ˆβ 2 /s 2 ) see Table 7. The natural log transformation was used to stabilize the variance of the estimates (see Miller and Wu 1991). Control factors which have more than two levels were treated by considering a set of suitable contrasts. Control factor E (signal sensitivity) is a quantitative factor with four levels so orthogonal contrasts corresponding to the linear, quadratic, and cubic components (E l, E q, E c ) were used. Control factors B (master rotor) and F (correction sequence) are qualitative. The contrasts used for these control factors are designated by B 1, B 2, B 3, F 1, F 2, and F 3. B 1 contrasts levels 1 and 2 with levels 3 and 4, B 2 contrasts levels 1 and 3 with levels 2 and 4, and B 3 contrasts levels 1 and 4 with levels 2 and 3. The contrasts for F are designated in the same manner. Figure 4 contains the half-normal plot for the estimated effects. We observe that A has the largest absolute values and is clearly separated from the rest of the estimates. The performance measure is higher when the new testing machine is used rather than the old one. There is also a cluster of five effects (E c, CD, B 1, D, and B 2 ) which are marginal. Of these, we are skeptical about E c as it is rare for a cubic effect to be significant when the linear and quadratic effects are not. Further the confounding pattern for the control array indicates that control factor E is aliased with several two-factor interactions. In particular, E c is partially aliased with the AD interaction which may provide a more plausible explanation of the observed effect. The remaining effects in this group (CD, B 1, D, and B 2 ) may represent further opportunities for improvement. The estimates for these effects suggest using the current setting for C (rotations at 23

24 handling), the new setting for D (rotations at measurement), and rotor number 3 for B (master rotor). PMM Analysis A Effects F3 B3 AG F2 C El F1 Eq G B D B1 CD Ec Figure 4: Half-normal Plot for lnβ 2 /σ 2 Next we consider the RFM approach which models the estimated parameters (contained in Table 7) or suitable transformations as functions of the control and noise factors. For each drive shaft/experimental combination a linear model of the form Y = α + βm + σǫ was fitted and the estimated values of β and σ 2 are recorded in Table 7. Effects were calculated for ˆβ and ln s 2. The factor representing drive shaft was split into two orthogonal contrasts: N 1 and N 2. The first, N 1, contrasts DS1 with DS2, whereas N 2 contrasts DS3 with DS1 and DS2. The effects involving this second contrast will be particularly important in investigating whether the results for DS3 are consistent with those for DS1 and DS2. For this experiment, it appears that each time the control factors were set to one of the combinations of levels in the control array (Table 5), measurements were taken on all three drive shafts (i.e. at all three levels of the noise factor). Thus the experiment has a split-plot structure with the control factors applied to main units, and the noise factor (drive shaft represented by contrasts N 1 and N 2 ) applied to sub-units. As a result estimates of effects which involve N 1 and N 2 will have a smaller error variance than those which only involve control factors (see Box and Jones 1992) and separate half-normal plots are needed. Figures 5 and 6 contain the half-normal plots for ˆβ and 24

25 Control Effects Noise Effects Effects B3 Eq F3 F2 Ec B2 F1 AG C El CD G A B1 D Effects N2CD N1F2 N1CD N1B2 2D N2Ec N1F3 N1C N1El N1G N1B3 N1B1 N1Ec N1A N2F1 N1N1D N2Eq N1F1 N2G N2F2 N1Eq N2F3 N2B2 N1AG N2CN2AG N2B3N2El N2A N2B Figure 5: Half-normal Plots for β Control Effects Noise Effects Effects AG B3 CD F2 El F3 C F1 A Eq G B1 B2 Ec D Effects N2A N1Eq N1F3N1 N2B2 N1B1 N1D N1B3 N2El N1El N1Ec N1F1 N1C N2C N2Eq N2F1 N2B1 N2Ec N2CD N1AG N1CD N1G N1F2 N2AG 2F3 N2G N2D N1B2 N1A N2F2 N2B3 N Figure 6: Half-normal Plots for ln σ 2 25