Experimental designs for precise parameter estimation for non-linear models

Transcription

1 Minerals Engineering 17 (2004) This article is also available online at: Experimental designs for precise parameter estimation for non-linear models Z. Xiao a, *, A. Vien b a Mining, Metals and Materials Engineering Department, McGill University, Wong Building, 3610 University Street, Montreal, Que., Canada H3A 2B2 b Metso Mineral Process Technology, Metso Minerals Canada Inc., 2281 Hunter Road, Kelowna, B.C., Canada V1X 7C5 Received 6 October 2003; accepted 30 November 2003 Abstract This paper is concerned with the experimental design for precise parameter estimation in a model of known form. Emphasis is primarily placed on three flotation models in which the parameters appear in a non-linear fashion. The first part of the paper consists of a brief statement of the statistical concepts and procedures involved in obtaining precise estimates of parameters, the logic of the design criterion, and the general sequential design procedures. Based on of this background, the second part describes the experimental design problem of estimating the parameters in three different flotation recovery models. The numbers of sampling and different measuring times for these models to obtain the precise parameter estimates are present in the conclusion. Ó 2004 Elsevier Ltd. All rights reserved. Keywords: Flotation kinetics; Modelling; Sampling 1. Introduction Experimental design is a systematic, rigorous approach to engineering problem solving that applies principles and techniques at the data collection stage so as to ensure the generation of valid, precise, and accurate engineering conclusions. Different experimental designs are used for different objectives. For example, randomized block designs can be used to compare data sets, full or fractional factorial design can be used for screening relevant factors. This paper is concerned with the experimental design for precise parameter estimation in a non-linear model and, more specifically, for flotation kinetic models. Before the model can be used to predict or optimize the system it describes, it is usually necessary to run experiments that will provide the data necessary to estimate precisely the unknown model parameters. As an example, suppose that an experimenter is studying the exponential flotation recovery model described by R ¼ R max ð1 e kt Þ ð1þ * Corresponding author. addresses: zhixian.xiao@mail.mcgill.ca (Z. Xiao), andre. vien@metso.com (A. Vien). where R is the recovery of the mineral or metal of interest, R max is the ultimate recovery and k is a first order rate constant. How can the parameters R max and k be estimated precisely? Of course, there are several methods of experimental designs at our disposal. However, this paper is concerned with a problem that can be defined as: at what times should measurements of R be taken in order to obtain estimates of the parameters that are as precise as possible? The general form of a model can be expressed as y ¼ f ðx; bþþe ð2þ where y is dependent or response variable, f ð Þ is a mathematical function and e denotes a random experimental error. The mathematical function, f, depends on two sets of input values. These set consist of n predictor variables (sometimes called independent variables), x 1 ; x 2 ;...; x n, and p parameters, b 1 ; b 2 ;...; b p. The predictor variables are observed along with the response variable. The collection of all of the predictor variables is denoted by x. x ðx 1 ; x 2 ;...; x n Þ ð3þ The parameters are the quantities that will be estimated during the modeling process. Their true values are /$ - see front matter Ó 2004 Elsevier Ltd. All rights reserved. doi: /j.mineng

2 432 Z. Xiao, A. Vien / Minerals Engineering 17 (2004) unknown and unknowable, except in simulations. The collection of all of the parameters is denoted by b. b ðb 1 ; b 2 ;...; b p Þ ð4þ Like the parameters in the mathematical function, the random errors e are unknown. They are simply the difference between the data and the mathematical function. They are assumed to follow a particular probability distribution. The parameters and predictor variables are combined in different ways to give different forms of models. Usually, the models are classified into two categories: linear and non-linear. A linear model is one in which each of the partial derivatives of ðx; bþ ob i, i ¼ 1; 2;...; p is a function of the x s only and not a function of any of the parameters. For example, the model of a straight line (Eq. (5)) is a linear model. f ðx; bþ¼b 1 þ b 2 x þ e ð5þ A non-linear model is one in which one or more of the derivatives is a function of one or more parameter b i. Many mineral processing models are non-linear in the parameters. For example, the exponential model of Eq. (1) is a non-linear model. In the statistical treatment of a program of N trials or experiments, we assume, for simplicity, that the errors in setting or determining the x values in the experiments are either zero or negligible. Generally, the program of N experiments can be represented by an N n matrix X ¼jx ui j (sometimes referred to as the design matrix). The ith member of the uth row of this matrix gives the level of the ith variable for the uth experiment. Usually, the range of values the predictor variable can take is restricted by physical constraints on the system. For example, a negative value for time has no physical meaning in the exponential model. Atkinson and Hunter (1968) defined this range as the region of operability. Outside this region experiments cannot be performed. Least squares parameter fitting is arguably one of the best parameter estimation methods for linear models when the experimental random errors are identically and independently distributed following a normal distribution with mean value of zero and variance r 2. It can be shown, under these assumptions, that the parameters will be estimated with minimum variance (maximum precision). The least squares estimates are obtained by selecting those values of b that minimize S 2 ð bþ¼ XN u¼1 ðy u f ðx u ; bþþ 2 ð6þ where S 2 ð bþ is referred to as the residual sum of squares, and y u are the observed responses. The precision of the parameter estimates, as measured by the variances and covariances, is calculated from ðx T X Þ 1 r 2, where the variances are on the diagonal and the covariances on the off-diagonals. The variance r 2 can be estimated by s 2 ¼ S 2 ðb Þ=ðN pþ, where S 2 ðb Þ is the residual sum of squares at the least squares estimate b. In order to obtain precise parameter estimates, ðx T X Þ 1 r 2 should be minimized. This principle will be applied to non-linear models. The least squares estimates of the parameters can always be obtained analytically for a linear model, while that is generally not the case with non-linear models. 2. The experimental design criterion for precise parameter estimation Generally, the least square method is concerned with finding the best parameter estimates in a proposed model. Considerable improvement in the precision of these estimates may be possible if the experimental conditions are located appropriately. To illustrate, suppose 10 observations, i.e. 10 (x, y) pairs, can be taken and measured for the purpose of determining precise parameters in a model. Where should these values be taken along the horizontal axis so as to minimize the variance of the estimated parameters of the model? Should the values be: 1. Ten equi-space values across the range of interest? 2. Five valued at the minimum of the X range and five values at the maximum of the range. 3. One value at the minimum, eight values at the midrange, and one value at the maximum? 4. Four values at the minimum, two values at midrange, and four values at the maximum? 5. Something else? Experimental designs for precise parameter estimation are intended to resolve this problem. Thus, parameter estimation designs are not model fitting techniques, but designs to obtain to the extent that it is possible, for a given specific model form, parameter estimates with minimum variance (Bacon, 1982). Box and Lucas (1959) were among the first to propose a solution for the problem of design for precise parameter estimation for non-linear models. In their paper, application of the method to several models were examined. The following can summarize the logic of the design criterion for the non-linear models. A first order Taylor series expansion of f ðx; bþ about b 0 is expressed as f ðx; bþf ðx; b 0 Þþ Xp of ðx;! bþ ðb b 0 Þ ð7þ ob i¼1 i b¼b 0 Now let Z ¼½z u Š¼fðx u ; bþ fðx u ; b 0 Þ ð8þ X ¼½x ui Š¼ of ðx u; bþ ob i b¼b 0 ð9þ

3 Z. Xiao, A. Vien / Minerals Engineering 17 (2004) B ¼½b i Š¼b i b 0 i ð10þ The subscript u denotes the run number, u ¼ 1; 2;...; N. Therefore, the expansion can be reduced to the matrix form Z X B. Then, the problem is transferred to that of finding the most precise design in the X -space to estimate the vector of coefficient B in a linear model, where X is an N p matrix of partial derivatives of the response, as shown in Eq. (9). Generally, each x ui will be a function of the vector of the preliminary estimates of the parameter b 0 and also of the vector of predictor variables, x u. The vector b 0 may consist of initial estimates of the parameters available before any data are obtained from the current investigation, or it could be composed of the current best estimates of the parameters based on some data that have already been collected. Uncertainty about the true parameter values results from the random errors associated with the measured responses. All basic information about the precision of parameter estimates for a fitted model is contained in the matrix ðx T X Þ 1 r 2, as mentioned previously. If, further, the random errors are normally distributed, the boundary of a region with confidence coefficient 1 a is the space of parameters is formed by the values of b which satisfy the relationship ðb b ÞX T X ðb b Þ 6 s 2 pf a ðp; mþ ð11þ where F a ðp; mþ is the a percent point of the F -distribution with p and m degrees of freedom and s 2 is an independent estimate of the error variance r 2, based on the m degrees of freedom. The design criterion suggested by Box and Lucas is to choose the settings of the predictor variables x for N trials or experiments so that the determinant jðx T X Þ 1 j is made as small as possible. This should ensure maximum precision for the parameters due to the minimum overall variance. Box and Hunter (1965) have shown that under suitable reasonable assumptions for nonlinear models, minimization of jðx T X Þ 1 j is equivalent to maximizing the posterior distribution of parameters. Thus there seems to be suitable justification for using the minimization of the determinant jðx T X Þ 1 j as a design criterion for obtaining precise parameter estimates. Since the determinant of an inverse matrix is equal to the reciprocal of the determinant of the original matrix, the criterion can be considered equivalently as maximizing the determinant D ¼jX T X j P N P N u¼1 x2 u1 u¼1 x P N u1x u2... u¼1 x u1x up PN ¼ u¼1 x P N P N u2x u1 u¼1 x2 u2... u¼1 x u2x up P N u¼1 x P N upx u1 u¼1 x upx u2... P N u¼1 x2 up ð12þ The design criterion is, therefore, to find that set of N experimental conditions which maximizes D. Because non-linear models are being considered, it must be remembered that the x ui s, and hence D, will be a function of the current estimates of the parameter values. Thus, before a design can be formed, some preliminary estimates of the parameters are required. This is indeed a paradox! Because of the non-linearity in the model it is necessary to have some initial guesses or estimates of the parameter values in order to obtain an efficient design to produce precise estimates of these same parameters (Behnken, 1964). Now, in practice, such preliminary information will often be available. Box and Lucas (1959) considered in particular the case in which the number of trials or experiments to be designed is equal to the number of parameters to be estimated, that is, N ¼ p. For this special situation, X is a p p matrix and D ¼jX T X j¼jxj 2 ð13þ where x 11 x x 1p x jx j¼ 21 x x 2p ð14þ x p1 x p2... x pp Therefore, maximizing the absolute value of the determinant jx j is equivalent to maximizing D. 3. The general design procedure As mentioned previously, it is necessary and important to have initial estimates of the parameters in order to apply the design criterion. If the initial estimates upon which the design is based are poor, the design may be inefficient. Therefore, in most cases in order to obtain the maximum information per experiment, some experiments could be performed, from which the initial parameters could be re-estimated. Thereafter, the experiments could be designed one at a time, using the current best estimates of the parameters, which could be recalculated after each experiment. This is usually called a sequential procedure. Then, given a single response model, Bacon (1982) provided a possible sequential strategy: 1. Select initial estimates or perform least square method to obtain the p parameters in the proposed model. 2. Compute the partial derivatives of the response function with respect to each of the parameters. 3. Find the maximum value of the determinant D to determine the Box Lucas design conditions for n ¼ p experimental runs. 4. Carry out these p runs at values of b which are set as close as possible to those for maximum D.

4 434 Z. Xiao, A. Vien / Minerals Engineering 17 (2004) Estimate the p parameters in the model from the observed response using non-linear least squares. 6. Design m additional experimental runs for the model using the latest parameter estimates, b, by maximizing the augmented determinant jx T X j. 7. Carry out these m additional runs. 8. Return to step 5 and continue the investigation until the parameters have been estimated with satisfactory precision. When some experimental data is available prior to considering such a strategy, it is possible to enter the scheme at step 5 directly. 4. Application of design criterion in flotation recovery models Let s assume that a set of data was obtained on the basis of experiments, with the predictor variable being time and the response variable being the observed flotation recovery. The data is shown in Table 1. Three different flotation kinetics models are being considered: exponential model (Eq. (1)), Klimpel s model (Eq. (19)), and Agar s model (Eq. (24)). We are concerned with how to design experiments to obtain the most precise parameters for each of these three flotation models. Specifically, the question is: at what times should measurements of R be taken in order to obtain the most precise parameters in these models? Another question of interest is: are the time intervals used to take measurements the same for all three models? Because of the non-linearity of these three models, it is necessary to obtain efficient initial values for the parameters. The initial parameter values shown in Table 2 were obtained by performing the least square method on the data shown in Table 1. In the following, we only consider the experimental design for N ¼ p (Step 3 above). For the exponential model (Eq. (1)), the problem is to choose a set of time t u, u ¼ 1; 2 at which to observe the flotation recovery R so that from these observations R max and k can be estimated as precisely as possible. The partial derivatives of the response function with respect to R max and k can be obtained as: Table 1 Flotation recovery data Time (min) Flotation recovery (%) Table 2 Initial parameter values for three models Exponential Klimpel Agar R max k b 0.1 x u1 ¼ or ¼ð1 e kt Þ ð15þ or max x u2 ¼ or ok ¼ R max t e kt ð16þ For two different sets of times t 1 and t 2, the maximization of D can be accomplished by maximizing the absolute value of the determinant X. X ¼ ð1 e kt 1 Þ Rmax t 1 e kt 1 ð17þ ð1 e kt 2 Þ Rmax t 2 e kt 2 Therefore, the formula of determinant jx j can be calculated as following: jx j¼jð1 e kt 1 ÞR max t 2 e kt 2 ð1 e kt 2 Þ R max t 1 e kt 1 j ð18þ The determinant is maximized for t 1 ¼ 0:72, t 2 ¼ 25:5 min. i.e. measurements should be carried out close to these times in order to obtain precise parameter estimates. Note that the selection of t 2 is somewhat arbitrary, as any value that is sufficiently large (larger than 10) will numerically maximize the determinant. The reason for this can be seen in Fig. 1, where the two derivatives are flat after about 10 min. The arrows shown in Fig. 1 indicate the optimal sample times. Of course, according to the sequential procedure, additional experiments could be run to refine parameter estimates. This can be repeated until the satisfactory precision is attained. Let s now consider Klimpel s flotation recovery model, described as Time (min) dr/drmax dr/dk Fig. 1. The derivatives of exponential model as function of time.

5 R ¼ R max 1 1 kt ð1 expð k tþþ ð19þ The objective of experimental design is again to obtain precise estimates of maximum recovery R max and the rate constant k. The partial derivatives of the above equation with respect to the two parameters R max and k are: x u1 ¼ or ¼ 1 1 or max kt ð1 e kt Þ ð20þ x u2 ¼ or ok ¼ R 1 ðkt þ 1Þe kt max ð21þ k 2 t The initial values of k and R max are shown in Table 2. In this case, for two different experimental times t 1, t 2, the determinant X can be obtained as: X ¼ 1 1 kt 1 ð1 e kt 1 Þ Rmax 1 ðkt 1þ1Þe kt 1 k 2 t kt 2 ð1 e kt 2 Þ Rmax 1 ðkt 2þ1Þe kt 2 ð22þ k 2 t 2 So that the absolute value of determinant jx j is calculated by: jx j¼ 1 1 ð1 e kt 1 Þ kt 1 R max 1 ðkt 2 þ 1Þe kt 2 k 2 t ð1 e kt 2 Þ R max 1 ðkt 2 þ 1Þe kt2 kt 2 k 2 t 2 ð23þ The values that maximize the determinant are t 1 ¼ 0:78 and t 2 ¼ min. i.e. a very large time. In practice, it means that the two trials should be run at two extreme side of the operability region. In other words, the second experimental trial should be run at the maximum time limit. Fig. 2 shows the derivatives of Klimple s model as function of time. Finally, consider Agar s flotation kinetics model, R ¼ R max ð1 e kðtþbþ Þ ð24þ This model consists of three parameters, the maximum recovery R max, the kinetics rate constant k, and constant Z. Xiao, A. Vien / Minerals Engineering 17 (2004) dr/drmax dr/dk dr/db Time (min) Fig. 3. The derivatives of Agar s model as function of time. b. Obviously, the number of experiments needed to run at least is equal to the number of parameter. The initial parameter values are shown in Table 2. The partial derivatives of the Agar model with respect to the three parameters R max, k and b are: x u1 ¼ or ¼ 1 e kðtþbþ ð25þ or max x u2 ¼ or ok ¼ R max ðt þ bþe kðtþbþ ð26þ x u3 ¼ or ob ¼ R max k e kðtþbþ ð27þ So that the matrix of X for three different run time t 1, t 2, and t 3 is: 1 e kðt 1þbÞ R max ðt 1 þ bþe kðt 1þbÞ R max k e kðt 1þbÞ X ¼ 1 e kðt 2þbÞ R max ðt 2 þ bþe kðt 2þbÞ R max k e kðt 2þbÞ 1 e kðt 3þbÞ R max ðt 3 þ bþe kðt 3þbÞ R max k e kðt 3þbÞ ð28þ The values that maximize the determinant are: t 1 ¼ 0, t 2 ¼ 0:85, and t 3 ¼ 128:8 min. This choice is made evident in Fig. 3. Each sample time correspond to the maximum of a derivative. For this model, the best sample times are: as soon as possible, at 0.85 min. and at very long time dr/drmax dr/dk Time (min) Fig. 2. The derivatives of Klimpel s model as function of time. 5. Conclusion The key underlying point with respect to the experimental design for precise parameter estimation is that some distribution of data in X vector may yield better (smaller variation) parameter estimates than others. The aim of experimental design for precise parameter estimation is to find what values should be selected for the independent variables. The optimal sets of times for the three flotation kinetics models are shown in Table 3. The last row is the inverse of the rate constant for each model. Some rules of thumb emerge from this data. If there is a time shift parameter in the model (Agar) then the first sample

6 436 Z. Xiao, A. Vien / Minerals Engineering 17 (2004) Table 3 The final results of precise parameter estimation Time Exponential Klimpel Agar t t Maximum 0.85 t =k should be taken as soon as practically possible. A sample should be taken at a time that corresponds roughly to the peak of the derivative with respect to the rate constant. For the Exponential and Agar models this simplifies to the inverse of the rate constant. For the Klimpel model this expression is more complex and does not lend itself to a simple analytical solution. The last sample should be taken as far as practically possible in time. Once the parameters have been estimated, the procedure can be repeated to select additional data points using these new estimates. Additional experiments can then be run to improve the precision of the parameter estimates. An interesting aspect not covered in this paper is that these additional points are at similar sample times as for the initial time selections i.e. if the initial parameter estimates are good then additional samples should be taken at the same sample times to improve precision. This is counter-intuitive as it indicates that it is better to concentrate the data points at key times rather than spread them throughout the time range. References Atkinson, A.C., Hunter, W.G., The design of experiments for parameter estimation. Technometrics 10, 271. Behnken, D.W., Estimation of copolymer reactivity ratios: an example of nonlinear estimation. Journal of Polymer Science A, 2, 645. Bacon, D.W., Collection and Interpretation of Industrial Data, Department of Chemical Engineering. Queen s University, Ontario. pp Box, G.E.P., Lucas, H.L., Designs of experiments in nonlinear situations. Biometrika 46, 77. Box, G.E.P., Hunter, W.G., Sequential design of experiments for nonlinear models. IBM Scientific Computing System In Statistics, 113.