COMPARATIVE STUDIES OF METAMODELING TECHNIQUES UNDER MULTIPLE MODELING CRITERIA

Transcription

1 AIAA COMPARATIVE STUDIES OF METAMODELING TECHNIQUES UNDER MULTIPLE MODELING CRITERIA Ruchen Jn * and We Chen Department of Mechancal Engneerng Unversty of Illnos at Chcago Chcago, Illnos Tmothy W. Smpson Department of Mechancal & Nuclear Engneerng The Pennsylvana State Unversty Unversty Par PA 680 Abstract Despte the advances n computer capacty, the enormous computatonal cost of complex engneerng smulatons makes t mpractcal to rely exclusvely on smulaton for the purpose of desgn optmzaton. To cut down the cost, surrogate models, also known as metamodels, are constructed from and then used n leu of the actual smulaton models. In the paper, we systematcally compare four popular metamodelng technques Polynomal Regresson, Multvarate Adaptve Regresson Splnes, Radal Bass Functons, and Krgng based on multple performance crtera usng fourteen test problems representng dfferent classes of problems. Our obectve n ths study s to nvestgate the advantages and dsadvantages these four metamodelng technques usng multple modelng crtera and multple test problems rather than a sngle measure of mert and a sngle test problem. Introducton Smulaton-based analyss tools are fndng ncreased use durng prelmnary desgn to explore desgn alternatves at the system level. In spte of advances n computer capacty and speed, the enormous computatonal cost of complex, hgh fdelty scentfc and engneerng smulatons makes t mpractcal to rely exclusvely on smulaton codes for the purpose of desgn optmzaton. A preferable strategy s to utlze approxmaton models whch are often referred to as metamodels as they provde a model of the model (Klenen, 987, replacng the expensve smulaton model durng the desgn and optmzaton process. Metamodelng technques have been wdely used for desgn evaluaton and optmzaton n many engneerng applcatons; a comprehensve revew of metamodelng Copyrght 000 by We Chen. Publshed by the, Inc. wth permsson. * Graduate research assstant. Assstant Professor, Member AIAA, correspondng author, wechen@uc.edu, Assstant Professor, Member AIAA. applcatons n mechancal and aerospace systems can be found n (Smpson, et al., 997 and wll therefore not be repeated here. For the nterested reader, a revew of metamodelng applcatons n structural optmzaton can be found n (Barthelemy and Haftka, 993; for metamodelng applcatons n multdscplnary desgn optmzaton, see (Sobeszczansk-Sobesk and Haftka, 997. A varety of metamodelng technques exst; the Response Surface Methodology (Box, et al.; 978; Myers and Montgomery, 995 and Artfcal Neural Network (ANN methods (Smth, 993; Cheng and Ttterngton, 994 are two well known approaches for constructng smple and fast approxmatons of complex computer codes. An nterpolaton method known as Krgng s becomng wdely used for the desgn and analyss of computer experments (Sacks, et al., 989; Booker, et al., 999. Fnally, other statstcal technques that hold a lot of promse, such as Multvarate Adaptve Regresson Splnes (Fredman, 99 and radal bass functon approxmatons (Hardy, 97; Dyn, et al., 986 are begnnng to draw the attenton of many researchers. An mmedate queston that a desgner may have s on what bass the varous technques should be used? Moreover, s one technque superor to the others? Numerous examples that demonstrate the applcaton of one metamodelng technque or the other, typcally for a specfc applcaton exst; however, our survey reveals a lack of comprehensve comparatve studes of the varous technques, let alone standard procedures for testng the relatve merts of dfferent methods. In Smpson, et al. (998, krgng methods are compared aganst polynomal regresson models for the multdscplnary desgn optmzaton of an aerospke nozzle. Gunta, et al. (998 also compare krgng models and polynomal regresson models for two 5 and 0 varable test problems. In Varadaraan, et al. (000, Artfcal Neural Network (ANN methods are compared wth polynomal regresson models for the engne desgn problem n modelng the nonlnear thermodynamc behavor. In Yang, et al., (000, four approxmaton methods enhanced Multvarate Adaptve Regresson Splnes (MARS, Stepwse

2 Regresson, ANN, and the Movng Least Square are compared for the constructon of safety related functons n automotve crash analyss, for a relatve small samplng sze. Smpson (999 presents the results of ongong work nvestgatng dfferent metamodelng technques response surfaces, krgng models, radal bass functons, and MARS models on a varety of engneerng test problems. Although the exstng studes provde useful nsghts nto the varous approaches consdered, a common lmtaton s that the tests are restrcted to a very small group of methods and test problems, and n many cases only one problem due to the expenses assocated wth testng. Moreover, when usng multple test problems, t s often dffcult to make comparsons between the test problems when they belong to dfferent classes of problems (e.g., lnear, quadratc, nonlnear, etc.. It s our belef that varous factors contrbute to the success of a gven metamodelng technque, rangng from the nonlnearty of the model behavor, to the dmensonalty and data samplng technque, to the nternal parameter settngs of the varous technques. We contend that nstead of usng accuracy as the only measure, multple metrcs for comparson should be consdered based on multple modelng crtera, such as accuracy, effcency, robustness, model transparency, and smplcty. Overall, the knowledge of the performance of dfferent metamodelng technques wth respect to dfferent modelng crtera s of utmost mportance to desgners when tryng to choose an approprate technque for a partcular applcaton. In ths wor we present prelmnary results from a systematc comparatve study whch provdes nsghtful observatons nto the performance of varous metamodelng technques under dfferent modelng crtera, and the mpact of the contrbutng factors to ther success. A set of mathematcal and engneerng problems has been selected to represent dfferent classes of problems wth dfferent degrees of nonlnearty, dfferent dmensons, and nosy versus smooth behavors. Relatve large, small, and scarce sample sets are also used for each test problem. Four promsng metamodelng technques, namely, Polynomal Regresson (PR, Krgng (KG, Multvarate Adaptve Regresson Splnes (MARS, and Radal Bass Functons (RBF, are compared n ths study. Although ANN s a well-known technque, t s not ncluded n our study due to the large amount of tral-and-error assocated wth the use of ths technque. Metamodelng Technques The prncple features of the four metamodelng technques compared n our study are descrbed n the followng sectons.. Polynomal Regresson (PR PR models have been appled by a number of researchers (Engelund, et al, 993; Unal, et al., 996; Vtal, et al., 997; Venkataraman, et al., 997; Venter, et al., 997; Chen, et al., 996; Smpson, et al., 997 n desgnng complex engneerng systems. A secondorder polynomal model can be expressed as: k k yˆ = β + β x + β x + β x x ( o = = When creatng PR models, t s possble to dentfy the sgnfcance of dfferent desgn factors drectly from the coeffcents n the normalzed regresson model. For problems wth a large dmenson, t s mportant to use lnear or second-order polynomal models to narrow the desgn varables to the most crtcal ones. In optmzaton, the smoothng capablty of polynomal regresson allows quck convergence of nosy functons (see, e.g., Gunta, et al., 994. In spte of the advantages, there s always a drawback when applyng PR to model hghly nonlnear behavors. Hgher-order polynomals can be used; however, nstabltes may arse (Barton, 99, or t may be too dffcult to take suffcent sample data to estmate all of the coeffcents n the polynomal equaton, partcularly n large dmensons. In ths wor lnear and second-order PR models are consdered.. Krgng Method (KG A krgng model postulates a combnaton of a polynomal model and departures of the form: k yˆ = β f ( x + Z( x, ( = where Z(x s assumed to be a realzaton of a stochastc process wth mean zero and spatal correlaton functon gven by Cov[Z(x,Z(x ] = σ R(x, x, (3 where σ s the process varance and R s the correlaton. A varety of correlaton functons can be chosen (cf., Smpson, et al., 998; however, the Gaussan correlaton functon proposed n (Sacks, et al., 989 s the most frequently used. Furthermore, f (x n Eqn. s typcally taken as a constant term. In our study, we use a constant term for f (x and a Gaussan correlaton functon wth p= and k θ parameters, one θ for each of the k dmensons n the desgn space. In addton to beng extremely flexble due to the wde range of the correlaton functons, the krgng method has advantages n that t provdes a bass for a stepwse algorthm to determne the mportant factors, and the same data can be used for screenng and buldng the predctor model (Welch, et al., 99. The maor dsadvantage of the krgng process s that model constructon can be very tme-consumng. Determnng the maxmum lkelhood estmates of the θ parameters used to ft the model s a k-dmensonal optmzaton

3 problem, whch can requre sgnfcant computatonal tme f the sample data set s large. Moreover, the correlaton matrx can become sngular f multple sample ponts are spaced close to one another or f the sample ponts are generated from partcular desgns. Fttng problems have been observed wth some full factoral desgns and central composte desgns when usng krgng models (Meckeshemer, et al., 000; Wlson, et al., 000. Fnally, the complexty of the method and the lack of commercal software may hnder ths technque from beng popular n the near term (Smpson, et al., Multvarate Adaptve Regresson Splnes (MARS Multvarate Adaptve Regresson Splnes (Fredman, 99 adaptvely selects a set of bass functons for approxmatng the response functon through a forward/backward teratve approach. A MARS model can be wrtten as: M y ˆ = a B m m (x, (4 m= where a m s the coeffcent of the expanson, and B m, the bass functons, can be represented as: Km q B m (x= [ s ( x t ] k= k, m v( m m (5 + where K m s the number of factors (nteracton order n the m-th bass functon, s m =+/-, x v(m s the v-th varable, v(m n, and t m s a knot locaton on each of the correspondng varables. The subscrpt + means the functon s a truncated power functon: q q [ ( ] [ s m ( xv( m t m ] s m( xv( m t m s x t = > 0 m v( m m + 0 otherwse (6 Compared to other technques, the use of MARS for engneerng desgn applcatons s relatvely new. Bua, et al. (990 use MARS for extensve analyss of data concernng memory usage n electronc swtches. Wang, et al. (999, compare MARS to lnear, secondorder, and hgher-order regresson models for a fve varable automoble structural analyss. Fredman (99 uses the MARS procedure to approxmate behavor of performance varables n a smple alternatng current seres crcut. The maor advantages of usng the MARS procedure appears to be accuracy and maor reducton n computatonal cost assocated wth constructng the metamodel..4 Radal Bass Functons (RBF Radal bass functons (RBF have been developed for scattered multvarate data nterpolaton (Hardy, 97; Dyn, et al., 986. The method uses lnear combnatons of a radally symmetrc functon based on Eucldean dstance or other such metrc to approxmate response functons. A radal bass functon model can be expressed as: yˆ = x x (7 a 0 where a s the coeffcent of the expresson and x 0 s the observed nput. Radal bass functon approxmatons have been shown to produce good fts to arbtrary contours of both determnstc and stochastc response functons (Powell, 987. Tu and Barton (997 found that RBF approxmatons provde effectve metamodels for electronc crcut smulaton models. Meckeshemer, et al. (000 use the method for constructng metamodels for a desk lamp desgn example, whch has both contnuous and dscrete response functons. 3 Test Problems and Test Scheme 3. Features of Test Problems To test the effectveness of varous approaches to dfferent classes of problems, 4 test problems are selected and classfed based on the followng representatve features of engneerng desgn problems. Problem Scale. Two relatve scales are consdered: large (the number of varables 0 and small (the number of varables =,3. Nonlnearty of the performance behavor. For convenence, we classfy the problems nto two categores: low-order nonlnearty (f the square regresson 0.99 when usng frst or second-order polynomal model and hgh-order nonlnearty (otherwse. Nosy versus smooth behavor. In some cases, numercal smulaton error or other nosy causes cannot be elmnated. In our study, the nosy behavor s artfcally created usng local varatons of a smooth functon. A summary of the features of the 4 test problems s gven n Table ; the test problems are descrbed n more detal n the next secton. Type Mathematcal Vehcle Handlng Table. Features of Test Problems Problem No. Nonlnearty Order Scale (# of Inputs Hgh (n=0 NO Low (n=0 NO 3 Hgh (n=0 NO 4 Low (n=0 NO 5 Low (n=6 NO 6 Hgh (n= NO 7 Hgh (n= NO 8 Low (n= NO 9 Hgh (n=3 NO 0 Hgh (n=3 NO Low (n=3 NO Low (n= NO Nosy Behavor 3 Low (n= YES 4 Hgh (n=4 NO 3

4 3. Descrpton of Test Problems Thrteen mathematcal problems are utlzed n our study (see Appendx. These test problems are chosen from (Hock and Schttkows 98 whch offers 80 problems for testng nonlnear optmzaton algorthms. Provded n the Appendx are the mathematcal functons for each of these problems. Whle some of the functons exhbt smooth low-order nonlnear behavor, the others are hghly nonlnear functons that pose challenges for many metamodelng technques. Fgure A. are the grd plots of hghly nonlnear problems except problem 7, of whch the plots from dfferent approaches are compared n Fgure. Meanwhle, Problem 4 s a real engneerng problem that calls for better vehcle desgn to mprove a vehcle s handlng characterstcs, partcularly the preventon of rollover. The problem statement s frst gven n (Chen, et. al., 999. The smulator used s the ntegrated computer tool ArcSm (ArcSm, 997; Sayers and Rley, 996 developed at the Unversty of Mchgan for smulatng and analyzng the dynamc behavor of 6-axle tractor-semtralers. Each smulaton takes more than three mnutes to run on a Sun UltraSparc workstaton. The use of ArcSm for the purpose of optmzaton demands heavy computatonal costs. In ths study, 4 nput varables are consdered whch nclude nne suspenson and vehcle parameters as desgn varables and fve uncontrollable factors for steerng and brakng. The response of nterest s the vehcle handlng performance, whch s measured by the rollover metrc. The prevous studes (Chen, et al., 999 ndcate that the rollover metrc has a hghly nonlnear dependence on the control and nose varables, especally for brake and steerng levels. 3.3 Data Samplng We are nterested n examnng the performance of varous metamodelng technques when dfferent szes of data samples are used for model formaton, as shown n Table. For dfferent problem scales, dfferent sets of sample data such as scarce, small, and large sets are used as tranng ponts for model formaton. For large-scale problems, Latn Hypercubes (McKay, et al., 979 are used to generate the tranng ponts n all cases because ths method provdes good unformty and flexblty on the sze of the sample. The second order polynomal models have k = (n+(n+/ coeffcents for n desgn varables. Gunta, et al. (994 and Kaufman, et al. (996 found that.5k sample ponts for 5-0 varable problems to 4.5k sample ponts for 0-30 varable problems are necessary to obtan reasonably accurate second-order polynomal models. Therefore, for large-scale problems, 3k sample ponts are selected and are referred to as a large sample set. For complex and tme-consumng problems, t s preferable to use fewer samples. For ths reason, scarce sample sets wth 3n ponts are tested. In addton to large and scarce sample sets, small sample sets wth 0n are also used. For small-scale problems, only small and large sample sets are consdered. Also shown n Table s the number of confrmaton ponts used for checkng the accuracy of each model. The Monte Carlo method s used to generate confrmaton ponts. Table. Expermental Desgns for Test Problems Tranng Ponts Scarce Set Set Set Confrmaton Ponts Scale Problem Latn Hypercube (3n Scale Problem N/A Latn Hypercube Latn Hypercube (0n (9 f n=, 7 f n=3 Latn Hypercube Latn Hypercube 3 ( n + ( n + (00 f n=, 5 f ( n=3 Monte-Carlo Method (500 for vehcle problem, for others 3.4 Metrcs for Performance Measures In accordance wth havng multple metamodelng crtera, the performance of each metamodelng technque s measured from the followng aspects. Accuracy the capablty of predctng the system response over the desgn space of nterest. Robustness the capablty of achevng good accuracy for dfferent the dfferent problem types and sample szes. Effcency the computatonal effort requred for constructng the metamodel and for predctng the response for a set of new ponts by metamodels. Transparency the capablty of llustratng explct relatonshps between nput varables and responses. Conceptual Smplcty ease of mplementaton. Smple methods should requre mnmum user nput and be easly adapted to each problem. For accuracy, the goodness of ft obtaned from tranng data s not suffcent to assess the accuracy of newly predcted ponts. For ths reason, addtonal confrmaton samples (see Table are used to verfy the accuracy of the metamodels. To provde a more complete pcture of metamodel accuracy, three dfferent metrcs are used: R Square, Relatve Average Absolute Error, and Relatve Maxmum Absolute Error. The equatons for these three measures are gven n Eqns. (8 to (0, respectvely. 4

5 a R Square R = n ( y yˆ = = n ( y y = MSE Varance (8 where ŷ s the correspondng predcted value for the observed value y ; y s the mean of the observed values. Whle MSE (Mean Square Error represents the departure of the metamodel from the real smulaton model, the varance captures how rregular the problem s. The larger the value of R Square, the more accurate the metamodel. b Relatve Average Absolute Error (RAAE n y yˆ RAAE = =, (9 n * STD where STD stands for standard devaton. The smaller the value of RAAE, the more accurate the metamodel. c Relatve Maxmum Absolute Error (RMAE max( y yˆ y yˆ yn yˆ,,..., n RMAE = (0 STD Whle the RAAE s usually hghly correlated wth MSE and thus R Square, RMAE s not necessarly. RMAE ndcates large error n one regon of the desgn space even though the overall accuracy ndcated by R Square and RAAE can be very good. Therefore, a small RMAE s preferred; however, snce ths metrc cannot show the overall performance n the desgn space, t s not as mportant as R Square and RAAE. 4 Results and Comparson Based on the proposed scheme for comparatve study, 36 metamodels are created for the 4 test problems (see Table, usng dfferent sets of sample data (see Table, and based on four dfferent metamodelng technques (see Sectons.-.4. Dfferent technques are compared based on the results from confrmaton ponts. 4. Accuracy and Robustness To llustrate the performance of the metamodelng technques under dfferent crcumstances (e.g., nonlnearty, problem sze, and sample sze, multple bar-charts are provded. Whle the mean ndcates the average accuracy of a technque, the varance llustrates the robustness of the accuracy. Fnally, whle the heght of a bar ndcates the magntude of accuracy, the dfferences between heghts of multple bars llustrate the mpact of a partcular contrbutng factor. 4.. Overall Performance Illustrated n Fgures and are the mean and varance of the three accuracy metrcs for all metamodels whch consder dfferent orders of nonlnearty, dfferent problem szes, and dfferent sample szes. For the mean values, the larger the R Square, the better the accuracy s; however, for both RAAE and RMAE, the smaller value ndcates better accuracy. For varance, a smaller value always ndcates hgher robustness. R Square and RAAE R Square and RAAE 9.00E E E E E E E-0.00E-0.00E-0 R Square RAAE RMAE 3.00E+00.50E+00.00E+00.50E+00.00E E-0 Fgure. The Mean of Accuracy Metrcs 4.50E E E E-0.50E-0.00E-0.50E-0.00E E-0 R Square RAAE RMAE 3.50E E+00.50E+00.00E+00.50E+00.00E E-0 Fgure. The Varance of Accuracy Metrcs Fgure shows that the average accuraces of RBF and KG for all test cases are among the best n the group; ther values are very close to each other. RBF s slghtly better than KG n R Square (all close to 0.8, but KG s better than RBF n both RAAE and RMAE. The average accuracy of PR s thrd n the group. As revealed n Secton 4..3, the poor average performance of MARS s due to the defcency of usng MARS when scare set of samples s avalable. In terms of the robustness of the accuracy for all test cases, RBF s dstnctly the best for all three accuracy measures. Overall, RBF s shown to be the best approach n terms of ts average accuracy and robustness when handlng all types of problems for any amount of samples. 4.. Performance for Dfferent Types of Problems Fgures 3-6 show the mean and varance of R Square of the metamodels for dfferent types of problems. In Fgures 3 and 4, Hgh and Low represent the nonlnearty of problems, whle RMAE RMAE 5

6 and represent problem scale. So for example, Hgh means a hgh-order nonlnear and large scale problem. The values n Fgures 3 and 4 are derved based on the data from all sample szes (large, small, and scare. It s noted that for hgh-order nonlnear and large scale problems, RBF performs best n terms of both average accuracy and robustness. For low-order nonlnear and large scale problems, KG performs best n terms of both average accuracy and robustness. For hgh-order nonlnear and small scale problems, RBF performs best n terms of both average accuracy and robustness. For low-order nonlnear and small scale problems, PR performs best n terms of both average accuracy and robustness. better than PR. We also observe that each method has dstnctvely better accuracy for low-order nonlnear problems than for hgh-order nonlnear problems, whch matches well wth our ntuton. The dfference s the most sgnfcant for PR: whle the mean of R Square s close to for low-order nonlnear problems, t s less than 0.35 for hgh-order nonlnear problems. Except for MARS (due to the defcency for scare set of samples, the accuracy of the other three methods s acceptable for low-order nonlnear problems wth any sze sample set. However, the robustness, although small for RBF, KG, and PR when the model s loworder nonlnear, becomes larger when the problems are hgh-order nonlnear. The mpact s the most sgnfcant for KG and PR..0E+00.00E+00.0E+00.00E E E E E E E-0.00E-0.00E-0 Hgh Low Hgh Low Hgh Order Low Order Scale Scale Fgure 3. The Mean of R Square for Dfferent Types of Problems (a Fgure 5. The Mean of R Square for Dfferent Types of Problems (b 7.00E E E E E E E E E-0.00E-0.00E-0.00E-0.00E-0 Hgh Low Hgh Low Hgh Order Low Oreder Scale Scale Fgure 4. The Varance of R Square for Dfferent Types of Problems (a In Fgures 5 and 6, the average accuracy and ts robustness are derved for sngle contrbutng factors (e.g., hgher-order nonlnear based on all the test data belong that that category. It ndcates that for hghorder nonlnear problems, RBF performs best n terms of both average accuracy and robustness. For loworder nonlnear problems, the average accuracy of KG and PR s very close, whle the robustness of KG s slghtly better than PR. So, overall, KG s slghtly Fgure 6. The Varance of R Square for Dfferent Types of Problems (b For large scale problems, the average accuracy of RBF and KG s very close, whle the robustness of RBF s better than KG. So, overall, RBF s the best. For small scale problems, RBF s the best agan n terms of both average accuracy and robustness. It s also found that problem scale has lttle mpact on the performance of RBF. Although the mpact of problem scale on the average accuracy of KG s also small, the mpact on robustness for KG s rather large. 6

7 4..3 Performance under Dfferent Sample Sze Fgures 7 and 8 show the performance of metamodelng technques for types of problems under dfferent sample szes (large, small and scarce. For large set samples, the performances of MARS, RBF and KG are very close not only n average accuracy but also n robustness, whle the performance of PR s worse than the others. We cannot tell overall whch s the best because KG performs slghtly better than RBF and MARS n average accuracy whle MARS s more robust than KG and RBF. RBF performs best for small set samples both n average accuracy and robustness. Although KG also performs well n average accuracy, t s not as robust. For scarce sample sets, the average accuracy of RBF, KG and PR are close but the robustness of RBF s the best. Therefore, for scarce set samples, RBF performs best overall. It s also noted that the sample sze has the largest mpact on MARS, for both the mean and varance of accuracy. When small or scarce sample sets are used, the accuracy of MARS s low (R < Ths s because MARS fals to work when the sze of samples s too small. The varances of the accuracy (robustness are shown to be very small for MARS, RBF, and KG when large sample sets are used. Fgures 9 and 0 further llustrate the performance of metamodelng technques for dfferent sample szes when handlng the most dffcult stuaton,.e., large scale and hgh-order nonlnear problems. It shows the average accuracy of MARS s the best when large sample sets are used for ths type of problem. For small sample sets, MARS also performs best f average accuracy and robustness are both consdered (although RBF performs best n average accuracy. However, ts performance deterorates sgnfcantly when the sample sze becomes scarce, under whch RBF performs best. The mpact of sample sze on average accuracy and robustness s the smallest for RBF. The accuracy and robustness of PR s not stable, as for hgh nonlnear problem, ts performance s very problem-dependent and sample-dependent (not only the sample sze. 9.00E E E E E E E-0.00E E E E E E E E-0.00E-0.00E-0 Set Set Scarce Set.00E-0.00E-0 Set Set Scarce Set Fgure 9. The Mean of R Square under Dfferent Sample Scales for Scale and Hgh-Order Nonlnear Problems 6.00E E-0 Fgure 7. The Mean of R Square under Dfferent Sample Scales 4.00E E E E E-0.00E-0.00E-0 Set Set Scarce Set 3.00E-0.00E-0.00E-0 Set Set Scarce Set Fgure 8. The Varance of R Square under Dfferent Sample Scales Fgure 0. The Varance of R Square under Dfferent Sample Scales for Scale an d Hgh- Order Nonlnear Problems 4..4 Impact of Nosy Behavor Fgure shows the nfluence of nose on the performance of dfferent metamodelng technques. Only problems and 3 are compared here snce the functon n Problem 3 s the result of local varatons of the functon n problem 7

8 , a low-order nonlnear problem. From Fgure, t s found that Krgng s very senstve to the nose snce t nterpolates the data. Consequently, when estmatng the accuracy of the krgng metamodel for Problem 3 usng the non-nosy data from Problem, the krgng model does not yeld good predctons. PR performs the best because ts tendency to gve a smooth metamodel. MARS and RBF also perform well n ths test problem Smooth(Prob Nosy(Prob 3 Fgure. R Square Smooth Vs Nosy Problems a. From Analytcal Model b. From MARS c. From RBF d. From KG f. From PR Fgure Grd Plots for Problem 7 Due to the space lmtaton, we only provde here the sample grd plots of problem 7, whch has a loworder hghly nonlnear (wavng behavor, for comparng the accuracy of dfferent approaches. It s noted that KG s extremely accurate for modelng the wavng behavor for ths partcular case, whle the RBF s the second best. We also found that PR s not sutable at all for ths type of behavor, whle MARS captures the general trend but falls short n ts accuracy n local regons. 4. Effcency The effcency of each metamodelng technque s measured by the tme used for model constructon and new predctons. The tme depends on the problem scale and the sample sze, whch also depends hghly on the computer platform (MARS, PR, RBF are tested on a PC-pentum III 500 MHz machne whle KG s run on a Sun Ultra60 workstaton. Rough tme statstcs needed for model constructon and new predcton are provded n Tables 3 and 4, respectvely. Table 3. Tme Needed to Construct Metamodel Problem Scale/ Sample sze / / / / MARS 5-0s -5s <s <<s RBF 5-0s -s <s <<s KG -3h 0-5m -5m 0-30s PR -s <s <<s <<s Table 4. Tme Needed for 000 New Predctons Problem Scale/ Sample sze / / / MARS <<s <<s <<s <<s RBF 0-0s -5s -5s <s KG 5-0s -3s -3s <s PR <<s <<s <<s <<s / It s obvous that PR s the most effcent method both n model constructon and response predcton. It s found that, relatvely speakng, model constructon s very tme-consumng for Krgng. Krgng requres a k- dmensonal optmzaton to fnd the maxmum lkelhood estmates of the parameters used to ft the model, whch can become computatonally expensve when the problem scale and the sample sze are large. For predctons, RBF and krgng are relatvely slow because spatal dstances are needed for predcton. However, all the reported tme scales are consdered to be small compared to the tme needed for smulatons to obtan the tranng data n complex applcatons. 4.3 Transparency PR provdes the best transparency n terms of the functon relatonshp and the factor contrbutons (see 8

9 Eqn.. When usng MARS, models can be recast nto the form (Fredman, 99: y = a0 + f ( x + f ( x, x + fk ( x, x, xk + Km= Km= Km= 3 ( The frst sum s over all bass functons that nvolve only a sngle varable. The second sum s over all bass functons that nvolve exactly two varables, representng (f present two-varable nteractons. Smlarly, the thrd sum represents (f present the contrbutons from three-varable nteractons and so on. Therefore, MARS also provdes some model transparency. For RBF, an explct functon exsts (Eqn. 9; however, the factor contrbutons are not clear. The same holds true for Krgng; however, the theta parameters can be nterpreted wth some practce large theta values ndcate a hghly non-lnear functon whle small thetas ndcate a smooth functon wth lttle varaton. The nfluence of each nput varable on the fnal output response cannot be readly ascertaned from a krgng model, however. 4.4 Smplcty Applyng PR and RBF s relatvely straghtforward, and no parameters need to be specfed by a user. They are both easy to mplement. MARS and Krgng are more sophstcated n theory. For MARS, nternal parameters can be specfed whch may mprove or deterorate ts performance dependng on the problem. For krgng models, the user has the capablty to manpulate the optmzaton parameters used when constructng the model, and there are a varety of choces for the correlaton functon used n the model and for the underlyng global porton of the model. Whle the Gaussan correlaton functon and a constant underlyng global model are the most frequently used, t s currently unclear the extend to whch more complex krgng models mprove the accuracy of the model. 5 Summary The systematc comparatve study presented n ths paper has provded nsghtful observatons nto performance of varous metamodelng technques under dfferent modelng crtera. Based on the dscussons n Secton 4, we provde here a summary of our observatons and conclusons. In terms of the accuracy and robustness of the varous technques for dfferent types of problems (.e., dfferent orders of nonlnearty and problem scales, we can summarze our results as shown n Table 5. As shown, RBF excels n most of the categores. When large sample szes are used (assumng desgners have enough computatonal resource, MARS, RBF, and KG perform equally well n both average accuracy and robustness. For small and scarce sample sets, RBF performs the best when both average accuracy and robustness are consdered. It s noted that the performance of MARS deterorates when small or... scarce sample sets are used. For the most dffcult problems,.e., large scale and hgh-order nonlnear problems, the average accuracy of MARS s the best when the large sample sets are used. MARS also performs best f average accuracy and robustness are both consdered. However, ts performance deterorates sgnfcantly when the sample sze reduces to scare, under whch RBF performs best. For RBF, the mpact of sample sze on average accuracy and robustness s the least. Fnally, n our test problem wth nose, PR performs the best and MARS also works well; however, KG s very senstve to the nose because t nterpolates the sample data. From the above observatons, we can conclude that RBF s the most dependable method n most stuatons n terms of accuracy and robustness. Table 5. Summary of Best Methods Hgh-Order Nonlnear Low-Order Nonlnear Overall Scale RBF Krgng RBF Scale RBF PR RBF Overall RBF PR RBF In terms of effcency for metamodel constructon, KG can be very tme-consumng, especally for large scale problems wth large sample szes. Meanwhle, PR takes the least amount of tme for model buldng. For all technques, the tme needed for new predctons s consdered to be trval compared to the tme used for smulaton and model constructon. PR and MARS have good transparency, whch means we can obtan the contrbutons of each nput factor and the nteracton among them. Both RBF and KG are less transparent n ths sense. In terms of smplcty, PR and RBF are the easest to mplement, whle the users need to confgure the parameters for MARS and KG to acheve better accuracy. Although PR s not accurate for hghly nonlnear problems, t s easy to use and s very accurate for low order nonlnearty. It s therefore proposed that when constructng m etamodels, PR should mplemented frst to see f a reasonable ft can be obtaned. It s also observed that expermental desgn (data samplng also plays an mportant role, especally when buldng KG models. If the samples are not properly selected, KG may not work well and sometmes even fal to obtan the metamodel. Data samplng also has some nfluence on RBF. It s found for small scale problems, when usng full factoral desgn, the accuracy wll mprove. Although not tested, data samplng s 9

10 expected to mpact the performance of other technques. It s desred to obtan unform samples not only n onedmensonal proectons but also hgher dmensonal proectons, to mprove metamodelng performance. Adaptve samplng methods, whch sample each varable accordng ts contrbutons to the response and other varable nteractons, should also be nvestgated. Fnally, to mprove our study, more test problems wth large dmensons and some medum dmensons (between 3 to 0 varables need to be consdered. For the same sze of data sample, the effectveness of dfferent sample technques s also of nterest. Acknowledgements The support from the NSF/DMII s gratefully acknowledged. Support from Dr. Kam Ng, ONR 333, through the Naval Sea Systems Command under Contract No. N G-0058 s also acknowledged. References ArcSm User s Gude, Automotve Research Center, The Unversty of Mchgan, Ann Arbor, July 997. Barthelemy, J.-F. M. and Haftka, R. T., 993, "Approxmaton Concepts for Optmum Structural Desgn - A Revew," Structural Optmzaton, Vol. 5, pp Barton, R. R., 99, December 3-6, "Metamodels for Smulaton Input-Output Relatons," Proceedngs of the 99 Wnter Smulaton Conference (Swan, J. J., et al., eds., Arlngton, VA, IEEE, pp Booker, A. J., Denns, J. E., Jr., Fran P. D., Serafn, D. B., Torczon, V. and Trosset, M. W., 999, "A Rgorous Framework for Optmzaton of Expensve Functons by Surrogates," Structural Optmzaton, 7(, pp. -3. Box, G. E. P., Hunter, W. G. and Hunter, J. S., 978, Statstcs for Expermenter s, John Wley & Sons, NY. Chen, W., Allen, J. K., Mavrs, D. and Mstree, F., 996, "A Concept Exploraton Method for Determnng Robust Top-Level Specfcatons," Engneerng Optmzaton, Vol. 6, pp Chen, W., Garmella, R., and Mchelena, N., 999, Robust Desgn for Improved Vehcle Handlng Under a Range of Maneuver Condtons, 999 ASME Desgn Techncal Conference, Las Vegas, NV, Paper No. DAC Cheng, B. and Ttterngton, D. M., 994, "Neural Networks: A Revew from a Statstcal Perspectve," Statstcal Scence, Vol. 9, No., pp Cresse, N. A. C., 993, Statstcs for Spatal Data, Revsed, John Wley & Sons, New York. Dyn, N., Levn, D. and Rppa, S., 986, "Numercal Procedures for Surface Fttng of Scattered Data by Radal Bass Functons," SIAM Journal of Scentfc and Statstcal Computng, Vol. 7, No., pp Engelund, W. C., Douglas, O. S., Lepsch, R. A., McMllan, M. M. and Unal, R., 993, "Aerodynamc Confguraton Desgn Usng Response Surface Methodology Analyss," AIAA Arcraft Desgn, Sys. & Oper. Mngmt., Monterey, CA, Paper Fredman, J. H., 99, Multvarate Adaptve Regresson Splnes, The Annals of Statstcs, 9(, pp. -4. Gunta, A.A., Dudley, J.M., Narducc, R., Grossman, B., Haftka, R.T., Mason, W.H, and Watson, L. T., 994, Nosy Aerodynamc Response and Smooth Approxmaton n HSCT Desgn, Proceedngs Analyss and Optmzaton (Panama Cty, FL, Vol., AIAA, Washngton, DC, pp Gunta, A., Watson, L. T. and Koehler, J., 998, September -4, "A Comparson of Approxmaton Modelng Technques: Polynomal Versus Interpolatng Models," 7th AIAA/USAF/NASA/ISSMO Symposum on Multd scplnary Analyss & Optmzaton, St. Lous, MO, AIAA, Vol., pp AIAA Hardy, R.L., 97, Multquadratc Equatons of Topography and Other Irregular Surfaces, J. Geophys. Res., Vol. 76, pp Hock and Schttkows 98, Test Examples for Nonlnear Programmng Codes, New York: Sprnger-Verlag. Kaufman, M., Balabanov, V., Burgee, S.L., Gunta, A.A., Grossman, B., Mason, W. H., and Watson, L. T., 996, Varable-Complexty Response Surface Approxmatons for Wng Structural Weght n HSCT Desgn, 34th Aerospace Scences Meetng and Exhbt, Reno, NV, AIAA Paper Klenen, J.P.C., 987, Statstcall Tools for Smulaton Practtoners, Marcel Dekker, NY. McKay, M.D., Beckman, R.J. and Conover, W.J., 979, A comparson of three methods for selectng values of nput varables n the analyss of output from a computer code, Technometrcs, (, pp Meckeshemer, M., Barton, R. R., Lmayem, F. and Yannou, B., 000, "Metamodelng of Combned Dscrete/Contnuous Responses," Desgn Theory and Methodology DTM 00 (Allen, J.K., Ed., ASME, Baltmore, MD, Paper No. DETC000/DTM Myers, R. H. and Montgomery, D. C., 995, Response Surface Methodology: Process and Product Optmzaton Usng Desgned Experments, Wley & Sons, New York. Powell, M. J. D., 987, "Radal Bass Functons for Multvarable Interpolaton: a Revew," Algorthms for Approxmaton (J.C. Mason and M.G. Cox, Eds., London, Oxford Unversty Press. Sacks, J., Welch, W. J., Mtchell, T. J. and Wynn, H. P., 989, "Desgn and Analyss of Computer Experments," Statstcal Scence, 4(4, pp Sayers, M. W. and Rley, S. M., 996, Modelng Assumptons for Realstc Multbody Smulatons of the Yaw and Roll Behavor of Heavy Trucks, SAE Paper No 96073, Socety of Automotve Engneers, Warrendale, PA. Smpson, T. W., Mauery, T. M., Korte, J. J. and Mstree, F., 998, September -4, "Comparson of Response Surface and Krgng Models for Multdscplnary Desgn Optmzaton," 7th AIAA/USAF/NASA/ISSMO Symposum on Multdscplnary Analyss & Optmzaton, St. Lous, MO, AIAA, Vol., pp AIAA Smpson, T. W., 999, November 7-, "A Comparson of Metamodelng Strateges for Computer-Based Engneerng Desgn," INFORMS Phladelpha Fall 999 Meetng, Phladelpha, PA, INFORMS. Smpson, T. W., Peplns J., Koch, P. N. and Allen, J. K., 997, "On the Use of Statstcs n Desgn and the Implcatons for Determnstc Computer Experments," Desgn Theory and Methodology - DTM'97, Sacramento, CA, ASME, Paper No. DETC97/DTM

11 Sobeszczansk-Sobes J. and Haftka, R. T., 997, "Multdscplnary Aerospace Desgn Optmzaton: Survey of Recent Developments," Structural Optmzaton, 4, pp. -3. Smth, M., 993, Neural Networks for Statstcal Modelng, Von Nostrand Renhold, New York. Tu, C. H. and Barton, R.R. (997, Producton Yeld Estmaton by the Metamodel Method wth a Boundary focused Experment Desgn, Desgn Theory and Methodology Conference DTM 97, DETC97/DTM3870. Unal, R., Lepsch, R. A., Engelund, W. and Stanley, D. O., 996, "Approxmaton Model Buldng and Multdscplnary Desgn Optmzaton Usng Response Surface Methods," 6th AIAA/USAF/NASA/ISSMO Symposum on Multdscplnary Analyss and Optmzaton, Bellevue, WA, AIAA, Vol., pp Varadaraan, S., Chen, W., and Pelka, C., 000, The Robust Concept Exploraton Method wth Enhanced Model Approxmaton Capabltes, Engneerng Optmzaton, 3(3, Venter, G., Haftka, R. T. and Starnes, J. H., Jr., 996, "Constructon of Response Surfaces for Desgn Optmzaton Applcatons," 6th AIAA/USAF/NASA/ISSMO Symposum on Multdscplnary Analyss and Optmzaton Bellevue, WA, AIAA, Vol., pp Wang, X., Lu, Y. and Antonsson, E. K., 999, September -5, "Fttng Functons to Data n Hgh Dmensonal Desgn Spaces," Advances n Desgn Automaton, Las Vegas, NV, ASME, Paper No. DETC99/DAC-86. Welch, W. J., Buc R. J., Sacks, J., Wynn, H. P., Mtchell, T. J. and Morrs, M. D., 99, "Screenng, Predctng, and Computer Experments," Technometrcs, 34(, pp Yang, R.J., Gu, L., Law, L., Gearhart, C., Tho, C.H., Lu, X., and Wang, B.P., 000, Approxmatons for Safety Optmzaton of Systems, ASME Desgn Automaton Conference, September 0-3, Baltmore, MD, Paper No. DETC-00/DAC f ( x = [(ln( x. x 9.9 Appendx A Test Problems (ln( 0 x ] = = ( x 0 x. f ( x = x ( c + ln, c = ,-7.64,- = x x ,-5.94,-4.7,-4.986,-4.00,-0.708,-6.66,-.79; =,,, f ( x = exp( x ( c + x ln( exp( x, 0 = k= c = ,-7.64, ,-5.94,-4.7,-4.986,-4.00, ,-6.66,-.79; =,,,0. 4. f ( x = x + x + x x 4x 6x + ( x 0 + 4( x 5 + ( x 3 + ( x6 + 5x7 + 7( x8 + ( x9 0 + ( x f ( x = a ( x + x + ( x + x +,, =,,, 6. = = [a ]= [ k ]; x 6. f ( x = (30+ x *sn( x *(4 + exp( 7. f x = sn( πx /cos( πx /6 ( 8. f x = sn( x + x + ( x x.5x +.5x ( + f ( x = ( x x f ( x = =,..., ( x x + ( x 3 99 u = 5+ f ( x f( x = exp( ( u x x = ( 50ln(.0 / 3 x3. f x = x x x x ( 3. f ( x = 0.5x + x xx 7x 7x 3. f ( x = 0.5x + x xx 7x 7x + ε( x, x where ε(x,x s a normal nose. The mean s equal to 0 and the varance s equal to /00 of that of the smooth part of f(x. When - 5 x,x 5, the varance s of the nose: /00*54.9=.55. a. P (x, x 3 other x =3 b. P3 (x, x 3 other x =3 c. P6 (x, x d. P9 (x, x 3 x =0 80 Rollover 60 Metrc 40 (degree Brake 80 Level (ps Steer level (degree e. P0 (x, x 3 x =.8 f. P4 Vehcle handlng Fgure A. Grd Plots of Hghly Nonlnear Models (P stands for Problem