Pointwise RMS bias error estimates for design of experiments

Transcription

1 44th AIAA Aerospace Scienc Meeting and Exhiit 9-1 January 006, Reno, Nevada AIAA Pointwise RMS ias error timat for dign of experiments Tushar Goel *, Raphael T. Haftka Department of Mechanical and Aerospace Engineering, University of Florida, Gainville, FL 3611 Wei Shyy Department of Aerospace Engineering, University of Michigan, Ann Aror, MI Layne T Watson Department of Computer Science and Mathematics, Virginia Polytechnic Institute, Blacksurg, VA 4061 Astract Rponse surface approximations offer an effective way to solve complex prolems. However limitations on computational expense pose rtrictions to generate ample data and a simple model is typically used for approximation leaving the possiility of errors due to insufficient model known as ias errors. This paper prents a method to timate pointwise RMS ias errors in rponse surface models. Prior to generation of data, RMS ias error timat can e used to construct dign of experiments (DOEs to minimize the imal RMS ias errors or compare different DOEs. It is demonstrated that for high dimensional dign spac, central composite dign giv a reasonale trade-off etween variance and ias errors under conventional assumption of quadratic model and cuic true function. The rults indicate that compared to ounds on ias error, RMS timat are ls affected y increase in the dimensionality of the prolem. Latin Hypercue Sampling (LHS, though very popular, may yield large unsampled regions which can e succsfully detected y implementing an alternate criterion like imum standard error or imum RMS ias error in dign space. We further demonstrate that poor LHS digns can e eliminated y using more than one DOE. With the help of a non-polynomial example prolem, it is demonstrated that the proposed method is correctly ale to identify the regions of high errors even when assumed true model is not accurate. I. NOMENCLATURE A Alias matrix Vector of timated coefficients of asis functions c l, c u Vectors of ounds on the coefficients vector β DOE Dign of experiment E(x Expected value of x e Vector of true errors at the data points e(x Error at dign point x e Actual RMS error at dign point x act e RMS ias error at dign point x I e Data independent ias error ound at dign point x e (x True ias error at dign point x * Graduate Student, Student Memer AIAA Distinguished Profsor and Fellow AIAA Clarence L Kelly Johnson Collegiate Profsor and Fellow AIAA Profsor American Institute of Aeronautics and Astronautics 1 Copyright 006 y the paper authors. Pulished y the American Institute of Aeronautics and Astronautics, Inc., with permission.

2 e (x Estimated standard error at the dign point x f(x, f (1 (x, f ( (x Vectors of asis functions at x f j (x Basis functions N(µ,σ Normal distriution with mean µ and standard deviation σ N P Numer of true candidate polynomials N s Numer of dign data points N v Numer of dign varial n 1 Numer of asis functions in the regrsion model n Numer of missing asis functions in the regrsion model r Parameter used to construct DOEs V Dign space X, X (1, X ( Gramian (Dign matric x Dign point (vector x 1, x,,x n Dign varial x (i 1, x (i (i,, x n Dign varial for i th dign point y Vector of oserved rpons y(x Oserved rponse at dign point x ŷ(x Rponse surface approximation at a dign point x α 1, α,, α m Parameters used to construct DOEs (1 ( β, β, β Vectors of asis function coefficients β j, β (1 ( j, β j Coefficients of asis functions γ ( β Bound on coefficient vector ε Noise in the rponse η(x True mean rponse at x θ Parameter used to construct DOEs σ Noise variance σ a Root mean square ( error ( Maximum value of the quantity over the dign space ( avg Space averaged value of the quantity Suscript c An instance of a true polynomial II. INTRODUCTION AND LITERATURE REVIEW Polynomial rponse surface approximations (RSAs are widely adopted for solving optimization prolems with high computational or experimental cost, as they offer a computationally ls expensive way of evaluating digns. It is important to ensure the accuracy of RSAs efore using it for dign and optimization. The accuracy of RSA, constructed using a limited numer of simulations, is primarily affected y two factors: (i noise in the data; and (ii inadequacy of the fitting model (called modeling error or ias error. In experiments noise may appear due to measurement errors and other experimental errors. Numerical noise in computer simulations is usually small, ut it can e high if there are some unconverged solutions such as those encountered in CFD or structural optimization. The true model reprenting the data is rarely known and due to limitation on the amount of data availale, usually a simple model is fitted to the data. For simulation-ased RSAs, modeling/ias error is mainly rponsile for the error in prediction. The effect of different errors can e reduced y careful sampling of the points using dign of experiment techniqu. There are two considerations for a good dign of experiment: (i dominant source of error in experiment and (ii uniformity (called staility y Myers and Montgomery [1] over the dign space (small variation of error in dign space. When noise is the dominant source of error, there are a numer of experiments which minimize the effect of variance (noise on the rulting approximation, for example, the central composite (Box and Wilson, [] and the Box-Behnken digns for regularly shaped dign spac and, the D-optimal dign (Myers and Montgomery [1], chapters 7 and 8 which minimiz the variance associated with the timat of coefficients of rponse surface model for irregularly shaped dign spac. Traditional variance-ased digns minimize the effect of noise and attempt to otain uniformity (staility over dign space ut they do not addrs ias errors. Recently Goel et al. [3] have demonstrated that prediction variance is not an appropriate tool to timate errors when ias errors are dominant. American Institute of Aeronautics and Astronautics

3 It is more difficult to achieve uniformity when ias errors are dominant. Traditional minimum ias digns consider only integrated error measur (Myers and Montegomery [1] in dign of experiments. A good amount of work has een done on developing dign of experiments which minimize mean squared errors averaged over the dign space that comin variance and ias errors (averaged or integrated mean squared error [4 7]. The ias component of the averaged or integrated mean squared error was also minimized to otain so-called minimum ias digns. The fundamentals of minimizing integrated mean squared error and its components can e found in Myers and Montgomery [1] (chapter 9 and Khuri and Cornell [8] (chapter 6. Venter and Haftka [9] developed an algorithm implementing a minimum-ias criterion, necsary for an irregularly shaped dign space where no closed form solution exists for minimum-ias experimental dign. They compared minimum-ias and D-optimal experimental digns for two prolems with two and three varial. The minimum-ias experimental dign was found to e more accurate than D-optimal for average error ut not for imum error. Montepiedra and Fedorov [10] invtigated experimental digns minimizing the ias component of the integrated mean square error suject to a constraint on the variance component or vice versa. Fedorov et al. [11] later studied dign of experiments via weighted regrsion prioritizing regions where the approximation is dired to predict the rponse. Their approach considered oth variance and ias components of the timation error. Palmer and Tsui [1] studied minimum-ias Latin hypercue experimental dign for sampling from deterministic procs simulators. Qu et al. [13] implemented Gaussian quadrature-ased minimum ias dign and prented minimum ias central composite digns for up to six varial. Since minimum-ias digns do not achieve uniformity, digns that distriute points uniformly in dign space (space filling digns like Latin-Hypercue Sampling are popular even though the digns have no claim to optimality. However due to random component the digns occasionally can leave large hol in dign space which may lead to poor predictions. Bias error averaged over the dign space has een studied extensively ut there is relatively small amount of work done to account for pointwise variation of ias errors ecause of inherent difficulti. Pointwise error timat are useful to identify regions of large errors and to reduce the imal asolute ias error rather than the spaceaveraged error. An approach for timating pointwise ounds on ias errors in RSAs following the pointwise decomposition of the mean squared error into variance and the square of ias was developed y Papila and Haftka [14]. They used the ounds to otain dign of experiments that minimize the imum asolute ias error. Papila et al. [15] extended the approach to account for the data and proposed data-dependent ounds. They assumed that the true model is a higher degree polynomial than the approximating polynomial and it satisfi the given data exactly. Goel et al. [3] generalized this ias error ounds timation method to account for inconsistenci etween the assumed true model and actual data, and rank deficienci in the matrix of equations used in linear regrsion. They demonstrated that the ounds can e used to develop adaptive dign of experiment to reduce the effect of ias errors in region of intert. For high dimensional prolems due to increase in numer of coefficients not modeled y rponse surface model, ias error ound timat pose two main prolems: (i computational cost of timating ounds y solving optimization prolem increas (polynomial growth rate with increase in numer of varial in optimization prolem, and (ii the practical utility of error ounds diminish as the proaility of occurrence of the error predicted y ounds ecom very small (refer to Appendix A for demonstration. To addrs the two issu, a method to timate pointwise root mean square (RMS ias errors (instead of ounds is proposed in this work. The RMS ias error timat can e used as a possile metric to construct DOEs or to compare different dign of experiments. The paper is organized as follows: The next section prents the theoretical formulation of the pointwise RMS ias error timation method. Different error metrics to construct and compare DOEs are also given. Major rults of this study are given in Section 4. Specifically, the RMS ias error timat are used to construct and compare different dign of experiments for a quadratic rponse surface when assumed true model is cuic. One nonpolynomial example involving multiple sinusoidal inputs of different wave lengths was used to study the application of RMS ias errors when assumed true model was incorrect. Conclusions and issu identified are summarized in Section 5. III. THEORETICAL MODEL FOR ESTIMATING POINTWISE RMS BIAS ERRORS Let the true rponse η(x at a dign point x e reprented y a polynomial of asis functions and β is the vector of coefficients. The vector f(x has two components: f asis functions used in RSA or fitting model, and f ( T f (xβ, where f(x is the vector American Institute of Aeronautics and Astronautics 3 (1 (x is the vector of (x is the vector of additional asis functions which are

4 missing in the linear regrsion model. Similarly, the coefficient vector β can e written as a comination of vectors (1 β and ( β which reprent the true coefficients associated with the asis function vectors rpectively. That is: f (1 (x and f ( (x ( 1 β T ( 1 T ( T ( 1 T (1 ( T ( ( β η = f β= f f = ( f β + ( f β (1 Assuming normally distriuted noise ε with zero mean and variance at a dign point x is given as σ ( N(0, σ, the oserved rponse y(x y = η + ε ( The predicted rponse at a dign point x, y ˆ is given as a linear comination of approximating asis functions vector f (1 and corrponding timated coefficients vector. ˆ( ( ( (1 T y x = f x (3 The timated coefficients vector is evaluated using the data (y for N s dign points as [1] (chapter : = (1T (1-1 (1 T ( X X X y (4 (1 where X is the Gramian dign matrix constructed using f (1 ( x (refer to Appendix B. Error at a general dign point x is the difference of the true rponse and the predicted rponse e = η yˆ. When noise is dominant, timated standard error e is used to appraise error and its exprsion is given as [1]: T ( = [ ] = ( X X 1 e x Var y σ (5 ˆ f (1 (1T (1 f (1 a where σ is variance. a When ias error is dominant, e = e, where e is the ias error at dign point x. Sustituting valu from Equations (1 & (4, the exprsion for e can e given as follows: e (1 T (1 ( T ( (1 T ( = ( yˆ ( = ( ( + ( ( ( ( x x x f x β f x β f x η (6 When data is consistent with the assumed true model, it can e shown that (refer to Appendix B β Aβ A = X X X X (7 (1 ( (1 T (1 = +, where ( -1 (1 T ( Using the aove exprsion, Equation (6 can e rearranged as follows, e ( = ( ( ( ( T β x f ( x T (1 ( A f x (8 American Institute of Aeronautics and Astronautics 4

5 This shows that for given data (set of dign points, ias error at a general dign point x depends only on the ( true coefficient vector β. The root mean square of ias error at dign point x e ( is otained y computing its L norm ( E e (E(x is the expected value of x considering all possile valu of x as follows: T ( ( T ( ( T (1 T ( ( ( ( ( ( T ( ( (1 T e = e x e x = f x f x β f x f β E x E E A A (9 ( T (1 T Note that the term f f ( x A depends only on the dign point x and DOE; hence, the aove equation can e rewritten as, Then, ( T (1 T T ( T (1 ( e = ( f f β β ( ( f x f x ( T (1 T T ( T (1 = f f A E( β β f A f E E A A e ( e ( ( T (1 T T ( T (1 = E ( x = f f A E β β f A f (11 It is ovious from Equation (11 that RMS ias error on any point depends on the coefficient vector β. This exprsion to timate pointwise RMS ias error is very easy to evaluate once the true polynomial and the distriution of coefficient vector β is known. A. RMS ias error prior to the generation of data an error metric Prior to generation of data, order of the fitting polynomial and the true model is assumed to e known ut the coefficient vector β is unknown. With no data on the distriution, the principle of imum entropy was followed and it was assumed that all components of coefficient vector β have uniform distriution ** etween -γ and γ (γ is a T constant. Then with simple algera, it can e shown that E( β β Sustituting this in Equation (11, pointwise RMS ias error is given as, = γ 3 x (10 I, where I is identity matrix. ( T (1 T ( T (1 e = ( ( A γ f x f x I ( ( 3 f x A f x ( T (1 T ( T (1 = γ ( ( A I ( A ( 3 f x f x f x f x (1 It is clear from Equation (1 that RMS ias error at a dign point x can e otained from the location of data points (defin alias matrix A used to fit the rponse surface, the form of the assumed true function (f 1 and f (which is a higher order polynomial than the approximating polynomial and the constant γ. Since γ has a uniform scaling effect, prior to generation of data it can e taken as unity for sake of simplicity. RMS ias error timat given y Equation (1 can e used to compare/construct different DOEs. A few possile RMS ias error ased metric to construct DOE are: imum of ias error in dign space, average of ias error in dign space, median of ias error in dign space, or the ratio of imum- to minimum- ias errors in dign space. In this study, ** In future with improved understanding of pointwise ias error timat more suitale distriutions can e identified. American Institute of Aeronautics and Astronautics 5

6 imum pointwise RMS ias error in dign space V, ( e given y Equation (13 is used as the error metric to construct and compare different DOEs. ( e ( e = (13 V i.e. the DOE, which effects pointwise RMS ias error timat via alias matrix A, is selected such that the imum value of RMS ias error ( e at any point in dign space V is minimized. B. Different metrics for constructing and comparing DOEs Many criteria used to construct/compare different DOEs are prented in literature. A few commonly used error metrics are enlisted as follows: ( (i Maximum standard error in dign space, ( e e (ii Space averaged standard error V e (Equation (5 with σ 1 = (14 ( a = e 1 = e dv avg V (Equation (5 with σ = 1 a. I I (iii Maximum asolute ias error ound in dign space ( e ( e Equation (15 with c ( = 1. V = (otained from V n n1 ( (1 ( f A f c j j= 1 i= 1 I e ( = j ij i x (15 I (iv Space averaged asolute ias error ound ( ( with c ( = 1. 1 I e x = e ( dv avg V x (otained from Equation (15 (v Maximum RMS ias error in dign space ( e ( e (vi Space averaged RMS ias error ( x criterion is the same as space-averaged ias error. V = (Equation (1 with γ=1. V 1 e ( = e dv avg V (Equation (1 with γ=1. This Among all the criteria, standard error ased criteria are most commonly used. The criteria will e compared to RMS ias error ased metric. For all prolems, a N v -dimensional cuoidal coded dign space V = 1 x, x,..., x 1 is used. 1 N v V IV. APPLICATION OF RMS BIAS ERROR ESTIMATES This section is divided in three parts. The first susection deals with the construction of DOE using pointwise RMS ias error as error metric. As is commonly done, the true function was assumed to e cuic polynomial and rponse surface model was quadratic polynomial. This dign was compared with other standard DOEs. Next, different error timat (RMS ias error and standard error were used to compare DOEs. Specifically, an attempt was made to identify potentially poor Latin Hypercue Sampling (LHS digns which may lead to poor predictions. Finally, the applicaility of RMS ias error timat to complex prolems, where assumed true model was inaccurate, was demonstrated with the help of a sinusoidal function with multiple wavelengths. American Institute of Aeronautics and Astronautics 6

7 A. Construction of dign of experiments To construct dign of experiments ased on RMS ias error timat, the imum RMS ias error in dign space ( e is minimized (min- RMS ias dign instead of minimizing the average square error as was done with minimum ias digns (Myers and Montgomery [1], chap 8, Khuri and Cornell [8], chapter 6 and Venter and Haftka [9]. A procedure to construct DOEs is given as follows: consider a dign of experiment in a N v - dimensional cuoidal coded dign space V = 1 x, x,..., x 1 for which data points are determined y parameters α α α 1 m 1 0,, L, 1 which define the coordinat of sampled data points. For instance, two-level factorial points associated with the parameter α in the coded hyper-cuoidal space are Nv cominations at m x =± α, i = 1, N. Figure 1(A shows an example of DOE as a function of parameters α, α in two i m v 1 dimensions. The min- RMS ias dign can e otained y identifying parameters α, α such that 1 ( ( x e given y Equation (13 is minimized. Mathematically it is written as a two-level optimization prolem as follows: ( e = ( e min min α,..., α 0 α,..., α 1 1 m 1 m V The inner/first level prolem is the imization of pointwise RMS ias error over dign space (this is a function of α and pointwise errors are timated y Equation (1 and outer level prolem is minimization m prolem over parameters α. This approach of constructing DOEs, though demonstrated using quadratic-cuic m example, can e extended to any higher order polynomial approximation/true function y appropriately choosing f 1 and f Min- RMS ias dign for -dimensional space For the two-dimensional example prolem, let the true model e a cuic polynomial: N v (16 η = β + β x + β x + β x + β x x + β x + (1 (1 (1 (1 (1 ( β x + β x x + β x x + β x ( 3 ( ( ( (17 And the fitted model is a quadratic polynomial: yˆ = + x + x + x + x x + x ( (1 T ( , i.e. f = 1 x x x x x x and f = x x x x x x β (1 (1 (1 (1 (1 (1 (1 T ( ( ( ( ( = β β β β β β and β = β β β β (1 ( Note that, the information aout vector, β, β is not required to generate DOE. The experimental dign includ a center point (0, 0, four factorial and four axial points (central composite dign in two-dimension determined y parameters α and α rpectively as shown in Figure 1(A. Points corrponding to vertex were 1 placed at x =± ; i = 1, N x = 0, x =± α ; i, j = 1, N ; i j. The optimal i 1 v i j v min- RMS ias dign was otained y solving Equation (16, where RMS ias error (inner/first level α and axial points were located at ( optimization prolem for a given comination of parameters α and α was otained y evaluating the pointwise 1 RMS ias error on a uniform structured grid of 41x41 points. The min- RMS ias dign was T T American Institute of Aeronautics and Astronautics 7

8 ( e x α = 0.954, α = min ( = (19 1 The aove choice of parameter α rtricted the search space to axial digns however the digns which minimize the imum RMS ias error may exist away from the axis. To explore the dign points another DOE was constructed using three parameters α, r and, θ as shown in Figure 1(B. As efore parameter α was used to select four factorial points; however four interior points were selected y choosing radius r and angle θ as shown in Figure 1(B. The points were: ( r cos θ, r sin θ,( r cos θ, r sin θ, ( r cos θ, r sin θ,( r cos θ, r sin θ [ ] where θ = θ + π, θ = θ + π, θ = θ π r 0,1, θ = 0, π 1 3 (0 The optimal DOE (considering min- RMS ias error was otained y solving the prolem ( e x min ( where imum RMS ias error for each dign was otained y evaluating pointwise RMS ias α, r, θ error on a uniform structured grid of 41x41. The min- RMS ias dign otained was: ( e x α = 0.954, r = θ = 0, π ( = (1 This dign was the same as the one otained with two-parameter dign (given y Equation (19, which confirmed that the dign otained indeed minimized imum RMS ias error Comparison of different experimental digns for -dimensional space The min- RMS ias dign in two-dimensional space given y Equation (16 was compared with other standard digns availale in literature. Here, minimum variance dign (DOE 1 minimiz variance, Myers and Montgomery [1], minimum space averaged ias dign (DOE, minimiz space averaged ias error, Qu et al. [13] and min- ias error dign (DOE 3, minimiz imum ias error ound, Papila et al. [15] were compared with the min- RMS ias dign (DOE 4 using metrics defined in Section 3.. For all the prolems, the assumed true function was cuic and rponse surface model was quadratic. Rults are given in Tale 1. Intertingly, the minimum space averaged ias dign (DOE performed poort on all metrics ut space averaged ias errors (RMS and ound. Since for DOE, sampled points were located in the interior, there was a significantly large extrapolation region which explained high imum errors. This suggts that averaging of ias error over the entire space may not e the t criterion to create dign of experiments. On the other hand, all other digns gave comparale performance on all metrics. As expected, the DOEs ased on ias errors (min- ias error ound dign (DOE 3 and min- RMS ias dign (DOE 4 performed etter on ias errors and the DOE ased on min-variance dign (DOE 1 reduced standard error more than other digns. The differenc etween the min- ias error ound dign (DOE 3 and the min- RMS ias error dign (DOE 4 were small Min- RMS ias digns for higher-dimensional spac Next, min- RMS ias digns of experiments were constructed for higher dimensional spac with the conventional assumption of modeling cuic function y a quadratic RSA. The central composite digns (CCD, which are popular minimum variance digns, were adapted to minimize imum RMS ias error. The numer of N v points in a CCD is given y + N + 1 where N v is the dimensionality of the prolem. Particularly, N v = to 5 v dimensional digns were studied in this work. To construct different DOEs, two parameters Points corrponding to vertex were placed at ( α i j v α and α were used. 1 x = ± α ; i = 1, N and axial points were located at 1 i x = 0, x = ± ; i, j = 1, N ; i j. The optimal min- RMS ias dign was otained y solving Equation (16, where imum RMS ias error for a given comination of parameters v α and 1 α was otained y American Institute of Aeronautics and Astronautics 8

9 evaluating pointwise RMS ias errors on a uniform structured grid of 11 Nv points. The min- RMS ias digns for different dimensional spac are given in Tale along with different error metrics. For two- and three- dimensional spac, the axial points (given y α were located at the face and the vertex points (given y α 1 were placed slightly inwards to minimize the imum RMS ias errors. The RMS ias digns performed very reasonaly on all the error metrics. An interting rult was otained for optimal digns for more than three-dimensional spac that is, while the parameter corrponding to vertex points (α 1 was at its upper limit, the parameter corrponding to the location of axial points (α hit the corrponding lower ound. This meant that to minimize imum RMS ias error the points should e placed near the center which was quite contrary to the rults otained for two and three dimensional spac. For this dign the standard error was expectedly very high. Notaly with increase in dimension, imum and average ias error ound grew quickly compared to imum and average of RMS ias error. This means that the effect of increase in numer of coefficients in vector β was higher on ias error ounds than RMS ias errors (same was oserved in Appendix A. To etter understand the unexpected rult otained for higher dimensional cas, a trade-off study etween the imum errors (RMS ias error, standard error and ias error ound was conducted for a four-dimensional dign space y varying the location of axial point (α from center to the face of the dign space while keeping the vertex location (α 1 fixed. The trade-off etween imum RMS ias error and imum standard error is shown in Figure (A and the trade-off etween imum RMS ias error and imum ias error ound is shown in Figure (B. As the axial point moved away from the center, the imum ias error ound and the imum standard error reduced while the imum RMS ias error increased. The variation in imum RMS ias error was small compared to the variation in imum standard error and imum ias error ound. It is clear from Figure that with a small increase in imum RMS ias error a large reduction in imum standard error could e otained. Also small increase in imum standard error can facilitate sustantial decrement in imum RMS ias error. Trade-off etween imum ias error ound and imum RMS ias error also reflected similar rults though the gradients were relatively small. Same conclusions were otained for N v = 5 dimension space. Figure 3 shows the variation of the imum- and the space-averaged- ias error ounds and, the imumand the space-averaged- RMS ias error with change in the location of axial point. The curve for imum- and space averaged- RMS ias error was flat which meant that RMS ias error was not sensitive to the location of dign points. This explains the succs of central-composite digns in handling the digns with ias errors. On the other hand, imum- and space averaged ias error ound reduced as the axial points were moved towards the face ut still the reduction was smaller compared to the reduction in standard error. The magnitude of imum ias error ound remained sustantially higher compared to imum RMS ias error. This confirmed the finding that RMS ias error was relatively ls affected y increase in the dimensionality of the prolem than ias error ound. To further illustrate the trade-off etween ias and standard error, min- RMS ias dign and face-centered cuic dign was compared with D-optimal dign, which minimiz the variance of coefficients, in fourdimensional space (Appendix C. The determinant of X X and different error metrics are given in Tale 3. As (1 T (1 can e seen, the D-optimal dign had hight determinant and min- RMS ias dign had the smallet determinant. While the min- RMS ias counterpart of CCD performed poorly on standard error ased criteria, D-optimal dign performed poorly on ias error ased criteria. Face-centered cuic dign, otained y placing the axial points far from the center, performed reasonaly on all the metrics. The most important implication of the rults prented in this section is that it may not e wise to select dign of experiment ased on a single criterion. Instead trade-off etween different metrics should e explored to find a reasonale dign of experiments Verification of dign of experiments Further insight in rults for higher dimensional space min- RMS ias DOE was otained y conducting a tt to compare predicted RMS ias errors with actual RMS errors. Four-dimensional space was used as the reprentative higher dimensional space and cuic true function was approximated y quadratic polynomial. A structured uniform grid of 11 4 points was used to compute the errors. Pointwise RMS ias error was timated using Equation (1 with constant γ=1. To compute actual RMS errors, a large numer of true polynomials were generated (1 ( y randomly selecting the coefficient vectors β and β from a uniform distriution of [-1, 1]. For each true polynomial, true rponse was otained using Equation (1 and vector was evaluated using Equation (4. Actual error at a point due to each true polynomial was computed y taking the difference etween actual and predicted rponse as, American Institute of Aeronautics and Astronautics 9

10 (1 (1 ( ( (1 ( c ( c ( T T T e = y yˆ = f β + f β f c c ( where suscript c reprents an instance of the true polynomial. Pointwise actual RMS errors e were timated y averaging actual errors over a large numer of polynomials in root mean squar sense as, act N P act c P c= 1 e = e N ( true polynomials were used to compute the actual errors (N P = Two digns of experiments with extreme valu of axial location parameter α were compared. Actual errors and predicted errors along with the correlations etween actual and predicted error timat are given in Tale 4. Maximum and space-averaged valu of actual RMS errors and RMS ias errors compared very well. The correlations etween pointwise actual RMS errors and RMS ias errors were also high. The small change in imal actual RMS error with the location of axial dign point confirmed the outcome that imal ias errors had low variation with rpect to the parameter α and placing the axial points close to the center minimized the imal actual RMS error. B. Comparison of different LHS DOEs Space filling digns like LHS are popular ecause they tend to reduce ias errors. Due to random components LHS digns occasionally leave large hol in dign space which may corrpond to poor approximation. Typically the digns use either distance (imize minimum distance etween points or correlation (minimize correlation etween points measur to otain uniform distriution in dign space however often the measur are not sufficient to avoid poor distriution. In low dimensional spac a ad dign of experiment can e identified y visual inspection ut additional measur are required to filter ad digns in high dimension spac. In this work pointwise RMS ias error and standard error were considered as two possile criteria to identify ad digns. Explicitly, LHS digns which yielded high imum RMS ias error or high imum standard error in dign space were considered poor digns and were required to e identified. To demonstrate that poor digns could e identified using the criteria, 100 LHS dign of experiments for a two-variale prolem with 1 points (rponse surface model and assumed true model were rpectively quadratic and quartic polynomial were generated using MATLAB routine lhsdign with imin criterion which imiz minimum distance etween points (a imum of 0 iterations were assigned to find optimal dign. A two-variale example was selected here to enale visualization of good and ad digns. For each dign, imum RMS ias error and imum standard error in dign space were computed using a uniform grid of 41x41 points. Digns with the hight and the least imum RMS ias error and imum standard error, are shown in Figure 4 and corrponding error timat are taulated in Tale 5. The digns which yielded the hight ( e Figure 4(A and the hight ( e ( x Figure 4(B indeed had a poor distriution of points (convex hull of points shown to help see that. Many points were aligned along the diagonal leaving large hol in dign space which corrponded to large errors. Digns with least ( e Figure 4(C and least ( e ( x Figure 4(D had relatively more uniform distriution of points. It is noteworthy that the distriution of points while using min- standard error criterion (Figure 4(D was more uniform than that otained y min- RMS ias error (Figure 4(C. Comparison of errors corrponding to different DOEs in Tale 5 shows that staility (ratio of imum to minimum error in space was poor for ias error (~70 than standard error (~10 (worst case. The effect of DOE on imum error was more for standard error (the ratio of worst to t ( e ( x was ~3.3 than RMS ias error (the ratio of worst to t ( e ( x was ~1.8. Noticealy, the variations in magnitude of RMS ias error were smaller compared to standard error, however the percent increase in RMS ias error y selecting a DOE ased on least ( e ( x rather than least ( e ( x was more than the percent increase in standard error when DOE was selected ased on least ( e ( x criterion. The difference etween the worst and t of 100 digns was more American Institute of Aeronautics and Astronautics 10

11 noticeale when the metric was imum error instead of space-averaged error measur. The rults clearly show that the distance ased criterion was not sufficient to otain uniform distriution of points and additional criteria like RMS ias error or standard error are required to identify poor LHS digns. In practical scenario, it is not feasile to generate a large numer of DOEs and select the t. Instead, one can generate N LHS DOEs using LHS and pick the t of the N LHS according to the suitale criterion. To illustrate the improvements y using N LHS DOEs than a single DOE two criteria: imum RMS ias error and imum standard error were used. According to each criterion the t of N LHS DOEs was selected. Different DOEs were compared y computing the following ratios: e of N LHS a r(be;se: ratio of imum RMS ias error of the selected dign and the worst ( digns when the dign was selected y minimizing imum standard error in dign space. e of N LHS digns when the dign was selected y minimizing ( e. r(se;se: ratio of imum standard error of the selected dign and the worst ( c r(be;be: ratio of imum RMS ias error of the selected dign and the worst ( digns when the dign was selected y minimizing ( e. e of N LHS d r(se;be: ratio of imum standard error of the selected dign and the worst ( American Institute of Aeronautics and Astronautics 11 e of N LHS digns when the dign was selected y minimizing ( e. High value of ratios indicat significant improvement in the quality of DOE y using N LHS DOEs. The minimum value of any ratio is one. For illustration purpose, here N LHS = 3 was used and 100 such experiments were conducted. The oxplot of ratios over 100 experiments is shown in Figure 5. The performance of standard error and RMS ias error criterion was comparale. When imum RMS ias error was selected as the criterion to differentiate etween DOEs, the reduction in ias error was more than the reduction otained when imum standard error was the chosen criterion and vice-versa. More importantly, oth criteria helped eliminate poor dign of experiments. Actual magnitude of imum RMS ias error and imum standard error for all 300 digns and the 100 digns otained after filtering using min- RMS ias error criterion or min- standard error criterion are plotted in Figure 6 and statistics of magnitude of error is summarized in Tale 6. The variation of imum RMS ias error was smaller compared to imum standard error. After implementing the filtering, high imum RMS ias error and imum standard error were eliminated and the upper tails of the oxplots were shortened. It can e concluded from this exercise that potentially poor digns can e filtered out y considering a small numer of LHS digns with an (imum standard or RMS ias error criterion. C. Application to non-polynomial prolems The ias error timat are criticized due to requirement of true function which is mostly unknown and a high order polynomial than the fitting model is assumed to e true function. In this susection, it is demonstrated that this assumption on assumed true model is practically useful if it reprents the main characteristics of the prolem. A case is considered when true function is a sinusoidal function given y Equation (4. ijx ( ( 1 ikx x = Real ia e e j, k 0,integer; x, x [ 1,1] η (4 j+ k 3 jk Constants a were assumed to e uniformly distriuted etween [-1, 1]. This function reprents the jk cumulative rponse of sine wav of different wavelengths. The choice of parameters j, k, x 1, and x ensured the dign space to e rtricted to one cycle of hight frequency (shortt wavelength such that the function could e approximated y a low order polynomial. To timate actual errors (N p = cominations of a were used. This function was approximated using a quadratic polynomial and ias errors were timated assuming the true model to e cuic. Min- RMS ias error DOE (Equation (19 was used for sampling (9 points and a uniform 1 If a large numer of iterations are allowed, the rults may e different. jk

12 grid of 1x1 points was used as tt points to timate errors. The distriution of actual RMS error and RMS ias error in dign space is shown in Figure 7. The RMS ias error was scaled y a factor of three to compare the actual errors and RMS ias error distriution in dign space. Note that prior to generation of data, actual magnitude of error was not important as the timated errors were scaled y a unknown factor γ (refer to Equation (1 which is aritrarily set as one. Estimated RMS ias error correctly identified the prence of high error zon along the diagonals and a low error zone in the central region. However predictions were inaccurate close to the oundary where the effect of higher order te in the true function was significant. The correlation etween actual RMS error and RMS ias error was The error contours and correlation coefficient indicated a reasonale agreement etween actual RMS error and RMS ias error timat considering the fact that the accurate modeling of the true function required a higher order polynomial and/or more sampling points. ( For this example prolem, the actual distriution of coefficient vector β can e otained from the distriution of constants in true function sin x= n 1 ( 1 ( n a y expanding each sine term in algeraic seri as jk n 1 x (5 n= 1 1! ( ( and comparing the coefficients of cuic te. It was found that coefficients β and β (corrponding to x and 1 (corrponding to 3 x rpectively followed a uniform distriution with range [-8,8] and coefficients x x and 1 modified distriution of points, ( T ( β and ( β 3 x x rpectively followed a uniform distriution with range [-4,4]. Using the 1 E β β and RMS ias errors were timated. Corrponding actual RMS error and scaled RMS ias error (scaled y 0.4 contours are shown in Figure 8. For this case, the agreement etween the actual RMS error and RMS ias error improved significantly (compare Figure 7(A and Figure 8(A and the correlation etween actual RMS errors and RMS ias errors increased to This indicat that supplying correct distriution of coefficient vector is very important to the asssment of errors. The rults also show that ias error timat can e relied upon even though the assumed true model is only partially correct. V. CONCLUDING REMARKS AND FUTURE SCOPE A formulation to timate pointwise RMS errors due to incorrect data model (ias error in rponse surface approximation is prented in this paper. It was demonstrated that in asence of noise, the errors depend on the coefficients associated with the missing asis functions in the true function and a simple exprsion to timate pointwise RMS ias error was derived. Prior to generation of data, this exprsion can e used to compute a data independent RMS ias error field which serv as an error metric to construct/compare different dign of experiments. Using imum RMS ias error as the metric, a min- RMS ias dign for two dimensional space and min RMS ias dign counterpart of central composite dign for two-five dimensional spac was developed when the assumed true function was cuic and a quadratic rponse surface model was used. For low dimension spac, the min- RMS ias dign yielded reasonale error timate of all considered error metrics. However for higher dimensional spac min- ias dign was otained y placing the axial point close to the center which corrponded to very high normalized standard errors. A trade-off study was conducted to study the effect of the location of axial point on the variation of different error metric. The most important outcome of this trade-off study was that more than one criterion should e considered while constructing dign of experiments and central composite digns offer a reasonale trade-off etween ias and noise errors. It was oserved that the imum RMS ias error was ls sensitive to the position of axial points compared to the imum standard error. Also it was shown that compared to ias error ounds, RMS ias errors were ls sensitive to increase in the dimensionality of the prolem. Popular space filling dign like LHS, occasionally leave large hol in dign space which may yield poor approximation dpite using measur like imization of minimum distance etween points. The potentially poor digns were identified using filtering measur namely imum RMS ias and standard error. Further, it was American Institute of Aeronautics and Astronautics 1

13 demonstrated that using a small numer of LHS digns with a filtering criterion like imum RMS ias error or imum standard error, the risk of using potentially poor digns can e reduced. Finally the effect of incorrect assumed true function on ias error timat was asssed with the help of an example where true function was a complex sinusoidal function and assumed true model was a cuic polynomial. ( RMS ias errors timat correctly identified regions of high errors when all components of β were assumed to follow same uniform distriution. The characterization of actual error field was improved y supplying more ( information aout the actual distriution of different components of β. Rults indicate that ias error timat can e relied if the assumed true polynomial captur the main characteristics of the true function. ( ( The method proposed here requir the expected value of matrix β β T which was computed y assuming ( same uniform distriution for all components of coefficient vector β. However as was demonstrated for nonpolynomial true function example, this assumption need to e further proed for error timation. This issue is ( particularly important posterior to the generation of data since all possile cominations of β need to satisfy data. The invtigations to improve modeling of ias errors are the suject of future rearch. VI. ACKNOWLEDGEMENTS The prent efforts have een supported y the Institute for Future Space Transport, under the NASA Constellation University Institute Program (CUIP, Ms. Claudia Meyer program monitor and National Science Foundation (Grant # DDM VII. REFERENCES 1. Myers, R.H., Montgomery, D.C., Rponse Surface Methodology-Procs and Product Optimization Using Digned Experiments, New York: John Wiley & Sons, Inc., 1995, pp Box, G.E.P., Wilson, K.B., On the Experimental Attainment of Optimum Conditions, Journal of Royal Statistical Society, 1951, pp Goel, T., Papila, M., Haftka, R.T., Shyy, W., Generalized Pointwise Bias Error Bounds for Rponse Surface Approximation, to appear in International Journal of Numerical Methods in Engineering. 4. Box, G.E.P., Draper, N.R., The Choice of a Second Order Rotatale Dign", Biometrika, 50 (3, 1963, pp Draper, N.R., Lawrence, W.E., Digns which Minimize Model Inadequaci: Cuoidal Regions of Intert", Biometrika, 5 (1-, 1965, pp Kupper, L.L., Meydrech, E.F., A New Approach to Mean Squared Error Estimation of Rponse Surfac", Biometrika, 60 (3, 1973, pp Welch, W.J., A Mean Squared Error Criterion for the Dign of Experiments", Biometrika, 70(1, 1983, pp Khuri, A.I., Cornell, J.A., Rponse Surfac: Digns and Analys, nd edition, New York, Marcel Dekker Inc., 1996, pp Venter, G., Haftka, R.T., Minimum-Bias Based Experimental Dign for Constructing Rponse Surfac in Structural Optimization, Proceedings of 38 th AIAA/ASME/ASCE/AHS/ASC Structur, Structural Dynamics, Materials Conference, Kissimmee FL, 1997, pp , AIAA Montepiedra, G., Fedorov, V.V., Minimum Bias Digns with Constraints", Journal of Statistical Planning and Inference, 63, 1997, pp Fedorov, V.V., Montepiedra, G., Nachtsheim C.J., Dign of Experiments for Locally Weighted Regrsion", Journal of Statistical Planning and Inference, 81, 1999, pp Palmer, K., Tsui, K.L., A Minimum Bias Latin Hypercue Dign", Institute of Industrial Engineers Transactions, 33(9, 001, pp Qu, X., Venter, G., Haftka, R.T., New Formulation of a Minimum-Bias Experimental Dign ased on Gauss Quadrature", Structural and Multidisciplinary Optimization, 8(4, 004, pp Papila, M., Haftka, R.T., Uncertainty and Rponse Surface Approximations", in Proceedings, 4 nd AIAA/ASME/ASCE/ASC Structur, Structural Dynamics, and Material Conference, Seattle, WA, 001, AIAA American Institute of Aeronautics and Astronautics 13

14 15. Papila, M., Haftka, R.T., Watson, L.T., Bias Error Bounds for Rponse Surface Approximations and Min-Max Bias Dign, AIAA Journal, 43(8, 005, pp MATLAB, The Language of Technical Computing, Version 6.5, Release 13, , The MathWorks Inc. 17. JMP, The Statistical Discovery Software TM, Version 5, , SAS Institute Inc., Cary NC, USA. Figur: (-α1, α1 (0.0, -α (α1, α1 (-α, α (α, α (rcos θ 1, rsin θ 1 (-α,0.0 (0.0, 0.0 (α, 0.0 (rcos θ, rsin θ (0.0, 0.0 θ r (rcos θ, rsin θ (rcos θ 3, rsin θ 3 (0.0, -α (- α1, -α1 (α1, -α1 (- α, -α (α, -α (A First set of dign points (B Second set of dign points Figure 1 Datasets used to construct the rponse surfac for polynomial example prolem Axial point on face α = 1 Axial point on face α = 1 Axial point near center α = α Axial point on face α = 0.05 α (A ( e vs. ( e (B ( e vs. ( e Figure Trade-offs etween (Aimum standard error and imum RMS ias error, (B ias error ound and imum RMS ias error, for four-dimensional space Figure 3 Variation of different error metrics with variation of axial point location (α American Institute of Aeronautics and Astronautics 14

15 (a Max-imum RMS ias error ( Max-imum standard error (c Min-imum RMS ias error (d Min-imum standard error Figure 4 Bt and worst LHS digns out of 100 LHS DOEs along with corrponding convex hull of points, filtered using different criteria Figure 5 Ratio of errors corrponding to selected and worst digns: r(x;y denot the ratio of type x error for the selected and the worst of N LHS digns when the selected dign is chosen using type y error as selection criterion. BE denot imum RMS ias error and SE denot imum standard error. American Institute of Aeronautics and Astronautics 15

16 Figure 6 Boxplots of imum RMS ias and standard errors in dign space considering all LHS (300 digns, (100 digns filtered using min BE as a criterion and (100 digns filtered using min SE as a criterion (BE denot imum RMS ias error, SE denot imum standard error (A Scaled RMS ias error (B Actual RMS error Figure 7 Contours of scaled RMS ias error and actual RMS error when assumed true model to compute ias error was cuic while true model was sinusoidal (Scaled ias error = 3*RMS ias error (A Scaled RMS ias error (B Actual RMS error Figure 8 Contours of scaled RMS ias error and actual RMS error when actual distriutions of different ( coefficients of β were specified (Assumed true model was cuic while true model was sinusoidal, Scaled ias error = 0.4*RMS ias error American Institute of Aeronautics and Astronautics 16

17 Tal: Tale 1 Comparison of different DOEs (1 Minimize variance dign ( Minimum space averaged ias dign (3 Min- ias error ound dign (4 Min- RMS ias dign. Metric of measurement were imum standard error, average standard error, imum ias error ound, average ias error ound, imum RMS ias error, average RMS ias error. Rults are computed on a uniform grid of 41x41 in space x 1, x = [-1.0, 1.0]. α 1 and α define the location of sampled points in Figure 1(A. ( DOE α 1 α e ( e ( I a vg I e ( e avg ( e ( e ( ( ( ( avg Tale Min- RMS ias CCD digns for higher dimension spac and corrponding dign metrics (N v is numer of varial N v α 1 α ( e e ( x ( a vg I I e e ( e ( e Tale 3 Comparison of RMS ias CCD, FCCD and D-optimal digns for 4-dimensional space ( * Appendix C α 1 α det(x (1T X (1 ( a ( e e vg avg I I e e ( e e ( avg e e D-optimal * 1.4e Tale 4 Comparison of actual RMS errors and RMS ias errors for DOEs in four dimensional space α 1 α ( e ( e act ( e ( e avg act r( e, avg e act avg avg American Institute of Aeronautics and Astronautics 17

18 Tale 5 Maximum and minimum RMS ias error and standard error for t/worst of 100 LHS DOEs. Min ( e corrponds to the dign with least imum RMS ias error among all 100 LHS digns. ( ( Similarly, Max ( e x corrpond to digns which yield worst RMS ias error, least standard error and worst standard error rpectively. x e x ( e e x Min ( e Max ( e e, Min e ( x and Max ( DOE ( Min ( ( x e ( ( e ( min ( min e ( American Institute of Aeronautics and Astronautics 18 avg ( ( e e Max ( Tale 6 Summary of actual magnitude of errors for different LHS digns. SE indicat imum standard error, BE is imum RMS ias error, All DOE refers to actual error corrponding to all 300 LHS digns, 100-BE and 100-SE refer to actual error data from 100 selected DOEs when filtering criterion is BE and SE rpectively All DOE 100-BE 100-SE SE Min Max Mean Median COV BE Min Max Mean Median COV Appendix A: Comparison of ias error ound and RMS ias error Bias error ounds timate the worst case scenario of the error timat. This giv reasonale timate of errors in low dimensional spac however as the numer of varial or the order of fitting/true polynomial increas, the numer of unknown coefficients increas and corrpondingly the error ounds correctly increase to account for the worst case scenario. However it fails to account for the decrease in proaility of the occurrence of worst case event, diminishing the utility of ounds. On the other hand, RMS ias errors timate RMS error which is a more realistic asssment of actual errors. To corroorate this point, two exampl were considered here: (i errors were timated for various dimension prolems when assumed true model is cuic and rponse surface model is quadratic, and (ii errors were timated for a four dimensional prolem when rponse surface model is quadratic and true polynomial is assumed to e a polynomial of higher degree. For oth cas, ias error ound (Equation (15 with c ( = 1, RMS ias error (Equation (1 with γ=1, actual imum error ( e = e ; c= 1, N and actual RMS error (Equation (3 were timated. 6 act c p c N = 10 polynomials were used to timate actual errors. For all prolems, n 1 p (n 1 is numer of coefficients in rponse surface model dign points and five tt points were sampled using LHS (using MATLAB routine lhsdign with imin criterion which imiz minimum distance etween points. The comparison of actual and ias error timat for prolems of varying dimensionality is given in Tale 7. With increase in dimensionality of the prolem, ias error ound (BEB at tt locations increased rapidly and the difference etween imum actual error and ias error ound grew significantly. This means that with increase in dimensionality practical utility of error ounds diminish. On the other hand, RMS ias error scaled well with the actual RMS error for all tt cas considered. The comparison of error timat with actual errors for a four avg