* Corresponding author at: Imperial College London, London, SW7 2AZ, UK. address: (S. Kucherenko).

Transcription

1 Explorg mult-dmesoal spaces: a Comparso of Lat ypercube ad Quas Mote Carlo Samplg Techques Serge Kuchereko *, Dael Albrecht, Adrea Saltell 3 CPSE, Imperal College Lodo, Lodo, SW7 AZ, UK The Europea Commsso, Jot Research Cetre, TP 36, 07 ISPRA (VA), Italy 3 Europea Cetre for Goverace Complexty, Uverstat Autooma de Barceloa, Cataloa, Spa Abstract Three samplg methods are compared for effcecy o a umber of test problems of varous complexty for whch aalytc quadratures are avalable. The methods compared are Mote Carlo wth pseudo-radom umbers, Lat ypercube Samplg, ad Quas Mote Carlo wth samplg based o Sobol sequeces. Geerally results show superor performace of the Quas Mote Carlo approach based o Sobol sequeces le wth theoretcal predctos. Lat ypercube Samplg ca be more effcet tha both Mote Carlo method ad Quas Mote Carlo method but the latter equalty holds for a reduced set of fucto typology ad at small umber of sampled pots. I cocluso Quas Mote Carlo method would appear the safest bet whe tegratg fuctos of ukow typology. Keywords: Mote Carlo, Lat ypercube Samplg, Quas Mote Carlo, Sobol sequeces, gh Dmesoal Itegrato.. Itroducto I may practcal applcatos umercal models are used a parametrc bootstrap fasho as to obta a dstrbuto of the ferece, from whch e.g. some estmate of model statstcs (average, percetles) ca be obtaed. Ths mples propagatg through the model assumed determstc here - the ucertaty from a set of the model s put varables or factors. Sce models ca be expesve to ru, t s sometmes mpossble to have as may model rus as eeded to acheve the desred accuracy. I such cases a effcecy of the Mote Carlo samplg techque ca make the dfferece betwee a relable ferece ad a msleadg result. The smple Mote Carlo techque s kow to have low effcecy. Varace reducto methods ca be employed to crease ts effcecy. Oe of the most popular varace reducto techque s stratfed samplg usg Lat ypercube Samplg (LS) []. May studes have bee made over the years to develop LS wth better space fllg propertes. Some authors have proposed to mprove LS space fllg ot oly oe dmesoal projecto, but also hgher dmesos []. Dfferet optmalty crtero such as etropy, tegrated mea square error, mmax ad maxm dstaces, ad others ca be used for optmzg LS [3]. The maxm crtero, cosstg maxmzg the mmal dstace betwee pots * Correspodg author at: Imperal College Lodo, Lodo, SW7 AZ, UK. E-mal address: s.kuchereko@mperal.ac.uk (S. Kuchereko).

2 to avod samplg desgs wth pots too close to oe aother, was used [4]. Iooss et al [5] compared the performace of dfferet types of space fllg desgs, the class of the optmal LS, terms of the effcecy of the subsequet metamodel fttg. They cocluded that optmzed LS wth mmal wraparoud dscrepacy (defed later ths work) are partcularly well-suted for the Gaussa process metamodel fttg. A detaled study o LS ad varous modfed desgs of LS was gve by Owe [6], [7], [8], Tag, [9], Ka-Ta Fag ad co-authors [0-] ad some other authors. Kalagaam ad Dwekar [] compared the performace of the Lat hypercube ad ammersley samplg techques o a umber of test problems. The umber of samples requred for covergece to mea ad varace was used as a measure of performace. It was show that the samplg techque based o the low dscrepacy ammersley pots requres far fewer samples to coverge to the varace of the tested dstrbutos: for the selected test cases ammersley requred up to 40 tmes fewer sampled pots tha LS. Wag et al. [3] developed a ew samplg techque combg LS ad ammersley sequece samplg ad compared t wth MC, LS ad the ammersley sequece samplg methods for calculatg ucertaty rsk aalyss. They foud that the ew (LS+ammersley) samplg techque ad the ammersley sequece samplg techque behaved cosstetly better tha MC ad pure LS. Whle ts applcato areas lke expermetal desg ad metamodel fttg s well studed, the effcecy of LS other areas such as hgh dmesoal tegrato has ot bee aalyzed systematcally. I hgh dmesoal tegrato, effcecy depeds upo the degree of uformty of the sampled pots the -dmesoal space. I ths work we compare effceces of three samplg methods: the MC method wth pseudo-radom, LS samplg ad the QMC method wth samplg based o Sobol sequeces. Although we dscuss three dfferet LS desgs, oly u-optmzed LS s used our umercal expermet. I spte of the recet advaces optmzato of hgh dmesoal propertes of LS, due to the P complexty of the optmzato, these advaces are oly practcal for relatvely small desgs. The most effcet ad hece expesve optmzed LS desgs caot be used for hgh dmesoal tegrato. Furthermore we ote a wdespread practce of use of o-optmzed LS whch justfes the preset exercse. QMC samplg s performed by usg the hgh-dmesoal Sobol Low Dscrepacy Sequece (LDS) geerator wth advaced uformty propertes: Property A for all dmesos ad Property A for adjacet dmesos (see Secto 5 for detals). Three dfferet classes of test fuctos are cosdered for hgh dmesoal tegrato, where the classfcato of fuctos s based o ther effectve dmeso as dscussed [4]. We report the absolute values of tegrato errors as well as the covergece rate. Ths paper s orgazed as follows: the ext Secto we descrbe Mote Carlo tegrato method. Secto 3 cotas a bref descrpto of LS. The QMC method s preseted Secto 4. Secto 5 troduces Sobol Low Dscrepacy Sequeces (LDS). A frst ddactc set of results comparg dfferet techques for populatg a ut square s gve Secto 6. Dscrepacy as a quattatve measure for the devato of sampled pots from the deal (desred) uform dstrbuto s descrbed Secto 7, where L dscrepacy for the three cosdered samplg methods s also reported. Global sestvty aalyss ad

3 effectve dmesos are troduced Secto 8. The results from the three samplg techques are compared ad dscussed Sectos 9 ad 0. Secto cocludes our aalyss.. Mote Carlo tegrato hypercube Cosder the tegral of the square-tegrable fucto f (x) over the -dmesoal ut Fucto f ( x ) s assumed to be tegrable I[ f] = f( x) dx. (.). The MC quadrature formula s based o the probablstc terpretato of a tegral. For a radom varable x that s uformly dstrbuted I[ f] = E[ f( x)], where E[ f ( x )] s the mathematcal expectato. A approxmato to ths expectato s I [ f] = f( x ), (.) = where (,..., x = x x ), =,..., s a sequece of depedet radom pots of legth. I other words, s a umber of sampled pots. Approxmato (.) s kow as the smple (or crude) Mote Carlo estmate of the tegral. The approxmato I[ f ] coverges to I[ f ] wth probablty. It follows from the Cetral Lmt Theorem that the expectato of a tegrato error ε, where ε = I[ f ] I [ f ] s where σ ( f ) s the varace gve by σ ( f ) E( ε ) =,. σ ( f ) = f ( x ) dx ( f ( x ) dx ) The expresso for the root mea square error of the MC method s / σ ( f ) ε = ( E( ε )) =. (.3) / The covergece rate of MC does ot deped o the umber of varables although t s rather low. Varous varace reducto techques (e.g. atthetc method, cotrol varates, stratfed samplg, mportace samplg) ca be appled to reduce the value of the umerator (.3), whch does ot chage the MC tegrato covergece rate of / O(/ ). These techques are ot cosdered here. The effcecy of MC methods s determed by the propertes of radom umbers. It s kow that radom umber samplg s proe to clusterg: for ay samplg there are always empty areas as well as regos whch radom pots are wasted due to clusterg; as ew pots are added radomly, they do ot ecessarly fll the gaps betwee already sampled pots. The qualty of a quadrature by a fte umber of sampled pots depeds o the uformty of the pots dstrbuto, ot ther radomess. Samplg 3

4 strateges amed at placg pots more uformly clude the LS ad LDS (also kow as quas radom) desgs addressed the preset aalyss. 3. Lat ypercube Samplg LS s wdely appled computatoal egeerg. It was developed by McKay et al. []. It s oe form of stratfed samplg that ca reduce the varace the Mote Carlo estmate of the tegrad. It was further aalyzed by Ima ad Shortecarer [5] ad the by Ste [6]. It has bee show that LS ca mprove the effcecy compared to the Mote Carlo approach, though we show here (Secto 9) that ths oly holds for certa classes of fuctos. Cosder the rage [0,] dvded to tervals of the equal legth /. Oe pot s selected at radom from each terval formg a sequece of pots { },,.., x =. Smlarly but depedetly we costruct aother sequece{ x }, =,..,. The two sequeces { },,.., x = ad { },,.., x = ca be pared to populate a bdmesoal space. These pars ca tur radomly be combed wth the values of sequece of s formed. 3 { },,..., x = to form trplets, ad so o utl a -dmesoal The algorthm formally ca be preseted as follows. Let { π k}, k =,..., be depedet radom permutatos of {,..., } each uformly dstrbuted over all! possble permutatos. Set k k π k() + U x =, =,...,, k =,...,, (3.) k where U are depedet radomly sampled pots o [0,] terval. k It s easy to see that oly oe pot of { x }, =,...,, k =,..., falls betwee (-)/ ad /, =,.., for each dmeso k =,..,. owever, ths stratfcato scheme s bult by supermposg well stratfed oe-dmesoal samples, ad caot be expected to provde prcple good uformty propertes a -dmesoal ut hypercube. I ths work we used the code for the LS from [7], referred to as stadard LS the followg. The effcecy of the stadard LS ca be mproved by takg the stadard LS as a startg desg ad the optmzg t accordg to some optmzato crtero: e.g. maxmzg the mmum dstace betwee ay two pots (maxm crtero), mmzg the dstace betwee a pot of the put doma ad the pots of the desg or mmzg the dscrepacy. Optmzato ca be doe by usg dfferet methods: choce of the best ( terms of the chose crtera) LS amogst a large umber of dfferet LS, colum wse par wse exchage algorthms or optmzato techques such as geetc algorthms, smulated aealg, ad others []. It has bee show by Iooss et al. [5] that mmzg the dscrepacy leads to a better space-fllg desg compared to the oe where the mmum dstace s maxmzed. I partcular they compared the two-dmesoal projectos of the maxm LS ad low wrap-aroud dscrepacy LS wth = 00 pots ad dfferet tal dmesos. They foud that the tal LS 4

5 desg optmzed wth the wrap-aroud dscrepacy offers the best results. These results were further developed [8]. We use ther code whch produces optmzed LS va smulated aealg. We also used maxm LS obtaed by selectg a LS desg from a set of four dfferet desgs accordg to the maxm crtero. The umber of chose desgs was lmted by the CPU tme ad a requremet to use a very large umber of sampled pots (up to = 0 averaged over 50 depedet replcatos, see Secto 9). 4. Quas Mote Carlo LDS are specfcally desged to place sample pots as uformly as possble. LDS are also kow as quas radom umbers. The QMC algorthm for the evaluato of the tegral (.) has a form smlar to (.) I = f( q ). (4.) = ere { q } s a set of LDS pots uformly dstrbuted a ut hypercube, q q q (,..., ) =. The Koksma- lawka equalty [9] gves a upper boud for the QMC tegrato error: * ε V f D. (4.) ( ) ere, V (f) s the varato of f ( x ) the sese of ardy ad Krause, * D s the sample dscrepacy (ts defto s gve Secto 7). For a oe-dmesoal fucto wth a cotuous frst dervatve t s smply V( f) = df( x)/ dx dx. I hgher dmesos, the ardy-krause varato may be defed terms of the tegral of partal dervatves. Further t s assumed that f ( x ) s a fucto of bouded varato. The smaller the dscrepacy D, the better the covergece of the QMC tegrato. umercal expermets suggest that the QMC tegrato error s determed by the varace of the tegrad ad ot by ts varato [0]. It s geerally accepted that the rate of the dscrepacy meag by ths the rate at whch dscrepacy coverges as a fucto of, determes the expected rate of the accuracy. I fact oe ca use the followg estmate of the QMC covergece rate (see Secto 7 for detals): O(l ) ε QMC =. (4.3) The QMC rate of covergece (4.3) s much faster tha that for the MC method (.3), although t depeds o the dmesoalty. Cosequetly, the smaller the value of, the better ths estmate. I practce at > the rate of covergece appears to be approxmately equal to O ( α ), 0< α. ece, the QMC method most cases outperforms MC terms of covergece. 5

6 5. Sobol sequeces There are a few well-kow ad commoly used LDS. May practcal studes have prove that the Sobol LDS s may aspects superor to other LDS. "Prepoderace of the expermetal evdece amassed to date pots to Sobol' sequeces as the most effectve quas-mote Carlo method for applcato facal egeerg." [] (see also []). For ths reaso Sobol sequeces were used ths work. Sobol sequeces are dgtal (t; s)-sequeces base the sese of ederreter [9]. Sobol LDS am to meet three ma requremets [3]:. Best uformty of dstrbuto as.. Good dstrbuto for farly small tal sets. 3. A very fast computatoal algorthm. Pots geerated by the Sobol LDS produce a uform-lookg fllg of the space, eve for rather small umbers of pots. Best uformty of dstrbuto s defed terms of low dscrepacy ad addtoal uformty propertes A ad A [4]: Defto. A low-dscrepacy sequece s sad to satsfy Property A f for ay bary segmet (ot a arbtrary subset) of the -dmesoal sequece of legth there s exactly oe pot each hyper-octat that results from subdvdg the ut hypercube alog each of ts legth extesos to half. Defto. A low-dscrepacy sequece s sad to satsfy Property A f for ay bary segmet (ot a arbtrary subset) of the -dmesoal sequece of legth 4 there s exactly oe pot each 4 hyper-octat that results from subdvdg the ut hypercube alog each of ts legth extesos to four equal parts. 6. Comparso of sample dstrbutos geerated by dfferet techques o a ut square I ths secto we compare dstrbutos of = 4 (Fg. ) ad = 6 (Fg. ) pots o a ut square ( =) gve by fve dfferet samplg techques: MC, stadard LS, maxm LS, optmzed LS ad Sobol LDS. Ths provdes a qualtatve pcture of the uformty propertes of these samplg techques. I the frst case the ut square s dvded to ad respectvely. I the secod case the ut square s dvded to ad / respectvely. squares of measure / ad / squares of measure / ad Fgs., ad, j show -dmesoal dstrbutos of Sobol pots satsfyg Property A. Oe ca see that each of the 4 small squares cotas exactly oe Sobol pot. Ths s ot the case for stadard LS (Fg.,c). Projectos to both -dmesoal subspaces also cota pot each of the 4 tervals for stadard LS (Fg., d), for maxm LS (Fg., f), for optmsed LS (Fg., h) ad for LDS-Sobol (Fg., j). Radom samplg (Fg.,b) does ot possess ether of these propertes. 6

7 LDS also possess a more geeral property: all projecto of the -dmesoal LDS o s- dmesoal subspaces (s < ) form s-dmesoal LDS [9, 4]. These addtoal stratfcato propertes of Sobol LDS result creased uformty of Sobol samplg. The stadard LS samplg does ot possess addtoal stratfcato propertes hgher dmesos. The LS desg ca be mproved by optmzg t further. owever, ths optmzato beg CPU tme cosumg s oly possble for a small umber of pots ad lmted umber of dmesos. Although maxm LS (Fgs.,e ad f) vsually shows a mprovemet a space dstrbuto over stadard LS, sampled pots do t satsfy Property A. owever, pots produced by optmzed LS do satsfy Property A (Fgs.,g, ad,h). 7

8 (a) (b) (c) (d) (e) (f) (g) (h) () (j) Fg.. Dstrbutos of 4 pots two dmesos. The ut square s dvded to 4 (o the left) ad 6 (o the rght) squares. (a,b) MC, (c,d) stadard LS, (e,f) maxm LS, (g,h) optmzed LS, (,j) Sobol LDS. 8

9 (a) (b) (c) (d) (e) (f) (g) (h) () (j) Fg.. Dstrbutos of 6 pots two dmesos. The ut square s dvded to 6 (o the left) ad 56 squares (o the rght). (a,b) MC, (c,d) stadard LS, (e,f) maxm LS, (g,h) optmzed LS, (,j) Sobol LDS. 9

10 Fg. shows dstrbutos of 6 pots two dmesos. From Fg.,, ad, j t s clear that Sobol pots satsfy Property A dmesos: each of the =6 subsquares cotas exactly Sobol pot (Fg., ). Ths s ot true for all types of LS (Fg., c, e, g) ad MC (Fg., a) samplgs: clusterg ad empty subsquares are clearly vsble from these plots. Optmzed LS gves the best dstrbuto amog all other LS desgs. Projectos of the -dmesoal samplg to both -dmesoal subspaces also cota pot each of the 6 tervals (Fgs., j). Ths s also true for LS (Fg., d, f, h) but ot for MC (Fg., b). I summary t would appear that Sobol LDS samplg gves a better way of arragg pots dmesos tha MC ad stadard LS. Although LS samplg ca be mproved through optmzato, ths procedure s lmted oly for small sample szes ad low dmesos. Aother mportat lmtato of optmzed LS s that the sample sze caot be creased cremetally wthout re-executg the optmzato step, whch has mplcatos for the motorg of the covergece of the tegrato [5]. 7. Dscrepacy Dscrepacy s a quattatve measure for the devato of sampled pots from the uform dstrbuto. Cosder a umber of pots Qt () from a sequece { x }, =,.., a -dmesoal rectagle Qt () wth the org 0, whose sdes are parallel to the coordate axes ad whch s a subset of : Qt (). () Qt has a volume mqt ( ( )) = t... t, where t = ( t,..., t ) are the rght had sde coordates of the rectagle wth the org the cetre of coordates. The local dscrepacy s defed as (see f.e. [9]) where s the total umber of pots sampled two dmesos. The star-dscrepacy s defed as Qt () () ( ()) ht = mqt, (7.). Fg. 3 llustrates the oto of the local dscrepacy D * = sup ht ( ). (7.) Qt () By defto a low dscrepacy sequece (LDS) s oe satsfyg the upper boud codto: * (l ) D c( ). (7.3) Costat c ( ) depeds upo the sequece as well as upo the dmeso, but does ot deped o. For radom umbers the expected dscrepacy s D = O((l l ) / ). * / From the Koksma-lawka equalty (4.) t follows that asymptotcally at QMC has a covergece rate O(/ ) whch s much better tha the rate O(/ ) that ca be acheved by MC ad 0

11 LS. owever, ths rate ca be attaed oly at the umber of sampled pots hgher tha the threshold * ~exp( ) whch s practcally ot possble at large. The value of dscrepacy depeds equally o all dmesos. A crucal cosderato here s that practcal applcatos, whe oe attempts to calculate the quadrature of a gve fucto, some varables (dmesos) are more mportat tha others for ay gve fucto. Ths explas why eve for hgh dmesoal problems wth hudreds of varables ad a umber of sampled pots < * stll QMC has a better covergece rate tha MC. The theory behd ths behavor s based o global sestvty aalyss ad s preseted the ext secto. * Calculato of the star-dscrepacy D s very dffcult ad the L dscrepacy s used stead for practcal purposes. The L dscrepacy s defed as L D has a closed form [5]: whch was used ths work. ( ) / D = h t dt. (7.4) L () D x x x L k k k = ( max( )) ( ( ) ) 3 j +, j= k= = k=, (7.5) Fg. 3. Illustrato of local dscrepacy ht () two dmesos wth Sobol LDS samplg of = 56 pots. Q s a umber of pots the shaded rectagular.

12 There are also other deftos of dscrepaces. ckerell [6] poted out that the dscrepacy (7.) s achored to the org because the terval [0; t) appears ts defto ad also there s referece to t = (; ; ) sce t appears the formula for the varato (4.). e defed the so-called cetered dscrepacy ad varato that refer to the ceter of the hypercube at t = (0.5,, 0.5). Ths dscrepacy ad varato are varat uder reflectos of a set of pots about ay plae t j = 0.5. The cetered dscrepacy has some advatages over the stadard L dscrepacy (7.4) whe the objectve s to detfy dffereces varous samplg desgs. ckerell also troduced the wrap-aroud dscrepacy whch measures uformty ot oly over the -dmesoal ut hypercube a set of pots to s-dmesoal subspaces of s but also over all the projecto of, s <. These results were further developed by Fag et al [] who appled searchg algorthms by mmzg dscrepacy as the objectve fucto for optmzg LS desgs. Fag et al [0] showed that LS has a lower expectato of square cetered dscrepacy tha that of smple radom desgs. We compared uformty propertes of MC, LS ad Sobol sequeces for dfferet dmesos usg L D defed by (7.5). Results preseted Fg. 4 were obtaed wth a stadard radom umber geerator as provded by MATLAB, the LS geerator developed [7] ad the SobolSeq89 geerator [7,8]. The maxmum umber of sampled pots used for calculatos was 3768.

13 0. MC LS QMC 0.0 D_L e (a) 0.0 MC LS QMC D_L (b) e-07 MC LS QMC D_L e-08 e L Fg. 4. D dscrepacy dmesos = 5 (a), = 0 (b), = 40 (c) versus for MC (blue broke le), LS (thck sold red le) ad QMC (th sold gree le) samplgs. (c) It ca be see that for low dmesos ( < 0) LDS s superor to MC ad LS as t shows the lowest values for the dscrepacy. I hgher dmesos ( > 40) the L dscrepacy of Sobol LDS becomes 3

14 comparable to that of MC ad LS. owever as atcpated, dscrepacy values are ot suffcet to judge the performace of the cosdered samplg techques whe appled to practcal problems such as hgh dmesoal tegrato. The performace s determed by the effectve dmeso values for a problem at had. The oto of the effectve dmeso s troduced the ext secto. Due to the large umber of requred sampled pots ad hgh dmesos t was ot possble to cosder optmzed LS. We also cosdered maxm LS usg procedure descrbed above. Comparso showed that the results obtaed wth stadard ad maxm LS are smlar, therefore we preset oly the result of stadard LS also because stadard LS s the method most see the ltarature. These fdgs also cocer the results for fucto tegratos preseted Secto 9. It s qute possble that optmzed LS would show a lower L D dscrepacy tha that of MC. owever, optmzed LS ca oly be used at very low umber of sample pots ad relatvely low dmesos, whle all these tests requred a very hgh umber of. 8. Global Sestvty Aalyss ad Effectve dmesos Global Sestvty Aalyss (SA) provdes formato o the geeral structure of a fucto by quatfyg the varato the output varables respose to varato the put varables. Cosder a tegrable fucto f ( x ) defed the ut hypercube the followg form: f ( x) f f ( x,..., x ) 0... s s= <... < s s. It ca be expaded = +. (8.) Expaso (8.) s a sum of compoets. It ca also be preseted as Each of the compoets s s f ( x) = f + f ( x) + f ( x, x ) f ( x, x,..., x ). 0 j j... < j f... ( x,..., x ) s a fucto of a uque subset of varables from x. The compoets f ( x ) are called frst order terms, f ( x, x ) - secod order terms ad so o. j j It ca be prove [9] that the expaso (8.) s uque f... (,..., ) 0,, (8.) f x x dx = k s s s k whch case t s called a decomposto to summads of dfferet dmesos. Ths decomposto s also kow as the AOVA (Aalyss Of VAraces) decomposto. The AOVA decomposto s orthogoal,.e. for ay two subsets u v a er product It follows from (8.) ad (8.) that f ( x ) f ( x ) dx= 0. u v f ( xdx )... dx = f, 0 4

15 f ( x) dx = f + f ( x ), (8.3) k 0 k f ( x) dx = f + f ( x ) + f ( x ) + f ( x, x ) k 0 j j, j j k, j ad so o. For square tegrable fuctos, the varaces of the terms the AOVA decomposto add up to the total varace of the fucto where σ = (,..., ),..., s f x x dx x s s. s σ σ... s s= < < s Sobol defed the global sestvty dces as the ratos =, (8.4) S σ σ... =... s /. s All S are o egatve ad add up to oe... s... s s= <... < s S... s =. S ca be vewed as a atural sestvty measure of a set of varables fracto of the total varace gve by f... ( x,..., x ). s s x,..., s x. It correspods to a Sobol also troduced sestvty dces for subsets of varables [9]. Cosder two complemetary subsets of varables y ad z: x = ( yz, ). Let y = ( x,..., x ), <... <, K = (,..., ). The varace correspodg to y s defed as m m m σ y cludes all partal varaces σ m y σ... s= ( < < ) K =. s σ, σ,, σ such that ther subsets of dces... s s (,..., ) K s. The total varace ( ) tot σ y s defed as ( ) tot σ y cossts of all... s tot ( σ ) = σ σ y z σ such that at least oe dex p K whle the remag dces ca belog to the complemetary to K set K [9]. The correspodg global sestvty dces are defed as S S y tot y = σ = σ / σ, y σ tot ( y ) /. 5

16 S tot y = S, z S tot y S accouts for all teractos betwee y ad z. y The mportat dces practce are S ad S tot [3,3]. Samplg based computatoal strateges to compute these measures are dscussed [33,34]. The frst order dex, wrtte as ( ( ( ) )) Vx E x f x x S = V( f( x)) s detcal to Pearso s correlato rato [35]. ere from dex. I most cases kowledge of S ad S tot x deotes a set of varables wth dces dfferet provdes suffcet formato to determe the sestvty of the fucto beg vestgated to dvdual put varables. Ther values also ca be used to determe fucto effectve dmesos. The oto of the effectve dmeso was troduced [36]. Let y be a cardalty of a subset y. such that Defto. The effectve dmeso of f ( x ) the superposto sese s the smallest teger S y (8.5) 0 < y < d s Defto. The effectve dmeso the trucato sese d s the smallest teger defed as T S y (8.6) y {,,..., d } T The threshold 0.99 s arbtrary. Codto (8.5) meas that the fucto f ( x ) s almost a sum of d dmesoal fuctos. A small value of S d mples that there are o hgh-order teractos. A low S value of d mples that there are few mportat varables. The detfcato of mportat ad ot mportat T varables allows o mportat varables to be fxed at ay arbtrary value wth ther doma of exstece. The resultat model has thus lower complexty wth dmesoalty reduced from to d T. Codto dt << s ofte verfed practcal problems. The value d does ot deped o the order whch the put varables are sampled, whle S d T does. The followg equalty s always satsfed: ds d. For some problems chagg the order whch T put varables are sampled ca dramatcally decrease d (see e.g. [36], [37], [38]). T A straghtforward evaluato of the effectve dmesos from ther deftos s ot practcal geeral. Global sestvty aalyss allows to estmate the effectve dmesos at reasoable computatoal costs as dscussed [4]. d S 9. Results. Itegrato 6

17 It ths secto we cosder the tegral evaluato for dfferet classes of fuctos, reportg the tegrato error ad the rate of covergece. A taxoomy for fuctos dffculty was suggested [4] ad s based o the fuctos effectve dmesos: Type A: Fuctos wth ot equally mportat varables. For type A models the effectve dmeso the trucato sese d <<. T Type B: Fuctos wth equally or almost equally mportat varables ad wth domat low order teracto terms AOVA decomposto. For such models the effectve dmeso the superposto sese d <<, whle S d. T Type C: Fuctos wth equally or almost equally mportat varables ad wth domat hgh order teracto terms AOVA decomposto. For type C models ds dt. Ths classfcato s summarzed Table. Due to the supermposto of hgh cardalty ad hgh teracto class C represets the most dffcult class of fuctos for tegrals evaluato. Iformato about relatve effceces of the three samplg techques s preseted the last two colums. 7

18 Table Classfcato of fuctos based o the effectve dmesos. Two complemetary subsets of varables y ad z are cosdered: x = ( yz, ). Fucto type A B C Descrpto A few domat varables o umportat subsets; oly low-order teracto terms are preset o umportat subsets; hghorder teracto terms are preset Relatoshp betwee S ad S tot S y tot / y >> S z tot / z S S j,, j S / S tot, S S j,, j S / S tot <<, d T d S QMC s more effcet tha MC << << Yes o << Yes Yes o o LS s more effcet tha MC For types A ad B fuctos QMC tegrato ca atta the rate of covergece close to the theoretcal lmt O(/ ) regardless of the omal dmeso, although the presece of hgh order teracto terms the AOVA decomposto ca somewhat decrease the covergece rate. The AOVA decomposto a geeral case ca be preseted as f ( x) = f0 + f( x) + r( x), where rx ( ) are the AOVA terms correspodg to hgher order teractos. Ste [6] showed that varace computed wth the LS desg s E( ε LS ) = [ r( x)] dx O( ) +, whle for MC E( ε MC ) = [ f ( )] [ ( )] ( ) x dx+ r x dx+ O. I the AOVA decomposto of type B fuctos, the effectve dmeso d S s small, hece r( x ) s also small comparg to the ma effects. I the extreme case t s equal to, ad a fucto f(x) ca be preseted as a sum of oe-dmesoal fuctos f ( x) f f ( x) = +. Ths meas that oly oe-dmesoal projectos of the sampled pots play a role the fucto approxmato. For type B fuctos LS ca acheve a much hgher covergece rate tha that of the stadard MC. To test the classfcato preseted above MC, LS ad QMC tegrato methods were compared cosderg test fuctos preseted Tables -4. All fuctos are defed 0. The theoretcal values 8

19 of all tegrals apart from the fucto A are equal to. For the fucto A a value of a tegral s equal to I[ f ] = ( ( ) ). 3 Table Covergece results for Type A fuctos Idex Fucto f ( x ) Dm Slopeα MC A ( ) A Slopeα QMC Slopeα LS x j = j = 4x + a = + a a = a = 0 a 3 = = a 00 = 6.5 Table 3 Covergece results for Type B fuctos Idex Fucto f ( x ) Dm Slopeα B B B B x = 0. 5 x = MC Slopeα QMC Slopeα LS x = + x = Table 4 Covergece results for Type C fuctos Idex Fucto f ( x ) Dm Slopeα C = / MC Slopeα QMC Slopeα LS x C ( ) / = x Fgs. 5-8 show the root mea square error (RMSE) versus the umber of sampled pots. The root mea square error s defed as K / k ([ I f] I[ f]), k= ε = K k where K s a umber of depedet rus, I [ f ] s a value of the MC/QMC estmate of I[ f ] wth the use of sampled pots at k-th depedet ru. For the MC ad LS method all rus were statstcally depedet. For QMC tegrato for each ru a dfferet part of the Sobol' LDS was used. For all tests 9

20 K=50. The RMSE s approxmated by the formula c α,0< α <. Smaller c meas smaller RMSE. The value of α defes the covergece rate. The tred les ad correspodg values for α parethess are preseted Fgures 5-8 ad the last three colums of Tables -4. (a) (b) Fg. 5. RMSE versus the umber of sampled pots for type A models. (a) fucto A ( = 360 ); (b) fucto A ( = 00 ). 0

21 (a) (b) Fg. 6. RMSE versus the umber of sampled pots for type B models, fucto B. (a) fucto B ( = 30 ); (b) fucto B ( = 00 ).

22 (a) (b) Fg. 7. RMSE versus the umber of sampled pots for type B models, fucto B. (a) fucto B ( = 30 ); (b) fucto B ( = 00 )

23 (a) (b) Fg. 8. RMSE versus the umber of sampled pots for type C models. (a) fucto C ( = 0 ); (b) fucto C ( = 0 ) 3

24 Fg. 9. Values of a tegral for Fucto A versus for MC, QMC ad LS samplg methods ( = 360 ). These results show that agreemet wth theory, QMC tegrato s superor to that of MC ad LS both terms of the rate of covergece (larger α ) ad the absolute value of the tegrato error (costat c ) all test cases. For fucto A at =360 (a very hgh dmesoal problem) the expoet for algebrac decay the case of QMC tegrato α QMC = 0.94 s very close to the theoretcally predcted asymptotcal value α =. The costat c s lower for the QMC method tha that for both MC ad LS methods. The LS method shows the same covergece rate α 0.5 as the MC method ad t has oly a margally smaller costat c tha that of MC. For fucto A at = 00 the effcecy of QMC s hgher tha that of both MC ad LS, although α QMC s oly close to 0.7. As for the prevous case effceces of MC ad LS methods are smlar wth LS havg a margally smaller costat c. Itegrals for type B fuctos are cosdered at two dfferet dmesos ( = 30 ad = 00) to show a terestg effect of terplay betwee the effceces of QMC ad LS methods. At = α QMC 0.96, whle 0.6 α LS < 0.69 ad αmc 0.5. The costat c s lower for the QMC 0 method tha that for the MC method at all ad the LS method at >. At = 00 the effcecy of the QMC method slghtly drops, wth 0.79 α QMC 0.94, whle the effcecy of the LS method slghtly creases wth 0.7 α LS The effcecy of the MC method remas uchaged: αmc 0.5. The costat c s lower for the LS method at 4 5 tha that for the QMC method, whle at hgher 5 > due to the hgher gradet the QMC method becomes more effcet tha the LS method.. We ote, that type B s the oly type of fuctos for whch LS s sgfcatly more effcet tha MC. For the most dffcult type C fuctos we preseted results oly for a relatvely low dmesoal case of = 0. ere the QMC method remas the most effcet method, whle LS shows practcally o 4

25 better performace tha MC. owever, hgher dmesos the covergece rate of QMC tegrato drops ad becomes close to that of the other two samplg methods. Fg. 9 presets covergece plots for the fucto A wthout averagg over depedet rus. For the QMC method the covergece s mootoc ad very fast. A covergece curve reaches the theoretcal value at 4, whle for both MC ad LS covergece curves are oscllatg ad approach the 9 theoretcal value oly at. The QMC method s approxmately 30 tmes faster tha the LS method. The advatage of usg MC or QMC schemes over LS s that ew pots ca be added cremetally: formulas (.) ad (4.) ca be wrtte as I = I + f( x). ere f ( x ) s the ew updated value of a tegrad (for the QMC method ts value s equal to f ( q )). Thus whe usg QMC t s straghtforward to code a termato crtero that ca be voked cremetally. For LS stead t s ot possble to cremetally add a ew pot whle keepg the old LS desg: formula (3.) radom permutatos of {,..., } are dfferet from radom permutatos of {,..., }. Ths meas that addg a ew th pot to a already sampled set of pots ad a set { f( x )}, =,..., of fucto values requres to resample all pots ad to recalculate the whole set { f( x )}, =,..., of fucto values at ew { x}, =,...,. 0. Results. Evaluato of quatles for ormally dstrbuted varates May practcal smulato problems requre the geerato of ormally dstrbuted varables. I ths secto we compare MC, LS ad QMC samplg methods for evaluato of quatles. We cosder evaluato of low 5% ad hgh 95% percetles for the cumulatve dstrbuto fucto of a radom varable f ( x) = x, where x are depedet stadard ormal varates, s the dmeso, = 5. The = cosdered fucto s dstrbuted accordg to the ch-squared dstrbuto wth 5 degrees of freedom: f( x) χ (5). The values of 5% ad hgh 95% percetles are.46 ad.07, respectvely. The algorthm for geeratg -dmesoal radom ormal varables usg uformly dstrbuted varables cossts of the followg steps: a) geerate -dmesoal radom vector u uformly dstrbuted betwee 0 ad usg radom umbers, LS or quas radom sequeces; b) trasform every elemet of u to a stadard ormal vector x% wth zero mea ad ut varace usg the verse ormal cumulatve dstrbuto fucto: % x F ( u ) =. Fg. 0 presets covergece of umercal estmates to the true values of quatles. 5

26 Log(MeaError Low) "MC" "QMC" "LS" Expo. ("QMC") Expo. ("MC") Expo. ("LS") Log() (a) Log(MeaError gh) "MC" "QMC" "LS" Expo. ("MC") Expo. ("QMC") Expo. ("LS") Log() (b) Fg. 0. Comparso betwee MC, LS ad QMC methods for evaluato of quatles: RMSE error versus. (a) Low quatle = 0.05; (b) gh quatle = 0.95, dmeso = 5. MC damods, LS tragles, QMC squares. Results show a superor covergece of the QMC method. LS provdes margally hgher covergece tha MC.. Coclusos 6

27 We compared the effceces of three samplg methods: MC, LS ad QMC based o Sobol sequeces. It was show that Sobol sequeces possess uformty propertes whch other samplg techques do ot have (Propertes A ad A ). Optmzed LS satsfes Property A low dmesos but hgh dmesos due to the cost hgh of optmzato t s very dffcult to verfy ths property. I ay case whle for optmzed LS the fulflmet of the property has to be estmated emprcally case by case, Sobol sequeces possess these addtoal uformty propertes by costructo. Comparso of L dscrepaces revealed that the QMC method has the lowest dscrepacy up to dmeso 0. We used a umber of test fuctos of varous complextes for hgh dmesoal tegrato. Comparso showed that for types A ad C fuctos LS shows oly a slght mprovemet over MC. The covergece rate of the QMC method for types A ad B fuctos s close to O(/), whle the MC method has the covergece rate close to O(/ ) ad the LS method outperformg MC oly for type B fuctos. For type B fuctos the superorty of the QMC method over the LS method ca oly be see at some large umber of sampled pots, whle at small umber of sampled pots the LS method ca have a smaller varace tha the QMC method. For type C fuctos covergece of the QMC method sgfcatly drops, however t stll remas the most effcet method amog the three samplg techques. We also compared MC, LS ad QMC samplg methods for evaluato of quatles ad showed that QMC remas the most effcet method amog the three techques. Although there has bee a sgfcat progress mprovg space fllg propertes of LS, due to the hgh cost of optmzato volved, eve the most effcet techques caot be appled for solvg hgh dmesoal problems. O the other sde optmzed LS ot tested the preset work - have prove to be very effectve applcato to metamodellg where ofte oly a small umber of sampled pots are requred ad a samplg desg of such pots ca be effcetly optmzed. A mportat practcal dsadvatage of LS over MC or QMC schemes s that ew pots caot be added sequetally, whch makes t very dffcult to use tegrato where termato crtera are ofte voked cremetally ad other methods such as adaptve samplg. ote that most staces the evaluato of tegrals eed to be obtaed for fuctos of ukow typology (e.g. the fucto s a computer code). LS performs better tha QMC oly for class B fuctos. These fuctos where all varable are a sese mportat but the teractos are low are hardly the most frequet they eed bult ad hoc because the mportace of factors mathematcal models customarly follow a Pareto law wth few mportat factors ad may o-mportat oes (Saltell et al., 008). It would thus appear that a safe bet the choce of a samplg algorthm for quadratures s provded by Quas Radom umbers ad ths work has made the case that t s so usg Sobol sequeces. The preset work may be put perspectve by otg that by far the majorty of applcato see the lterature use o-optmzed LS. Ackowledgemets 7

28 The authors would lke to thak Bertrad Iooss for hs valuable commets ad provdg codes for producg optmzed LS. The Europea Cetre for Goverace Complexty s a jot udertakg of the Cetre for the Study of the Sceces ad the umates (SVT), Uversty of Berge (UIB) ad of the Isttut de Cèca Tecologa Ambetals (ICTA), Uverstat Autooma de Barceloa. Refereces [] M. McKay, R. Beckma, W. Coover, A comparso of three methods for selectg values of put varables the aalyss of output from a computer code, Techometrcs () (979) [] J.S. Park, Optmal Lat-hypercube desgs for computer expermets, J. of Stat. Pla. ad Iferece 39 (993) 95. [3] R. J, W. Che, A. Sudjato, A effcet algorthm for costructg optmal desg of computer expermets, J. of Stat. Pla. ad Iferece 34 (005) [4] M.D. Morrs, T.J. Mtchell, Exploratory desgs for computatoal expermets, J. of Stat. Pla. ad Iferece 43 (995) [5] B. Iooss, L. Boussouf, V. Feullard, A. Marrel, umercal studes of the metamodel fttg ad valdato processes, It. J. of Adv. Syst. ad Meas. 3 (00) -. [6] A. Owe, A cetral lmt theorem for Lat hypercube samplg, J. of the R. Stat. Soc. Ser. B 54(3) (99) [7] A. Owe, Cotrollg correlatos Lat hypercube samples, J. of Am. Stat. Assoc. 89 (994) [8] J.R. Koehler, A.B. Owe, Computer expermets, adbook of Stat. S. Ghosh ad C.R. Rao ed., Elsever Scece B.V. 3, 996, [9] B. Tag, Orthogoal array-based Lat hypercube, J. of Am. Stat. Assoc. 88 (993) [0] K.T. Fag, M. Chag-Xg, P. Wker, Cetered L-dscrepacy of radom samplg ad Lat hypercube desg, ad costructo of uform desgs, Math. Comput. 7(37) (00) [] K.T. Fag, R. L, A. Sudjato, Desg ad Modelg for Computer Expermets, Chapma ad all/crc, ew York, 006. [] J. Kalagaam, U. Dwekar, A effcet samplg techque for off-le qualty cotrol, Techometrcs 39(3) (997) [3] R. Wag, U. Dwekar, C.E. Grégore Padró, Effcet samplg techques for ucertates rsk aalyss, Evro. Prog. 3() (004) [4] S. Kuchereko, B. Fel,. Shah, W. Mautz, The detfcato of model effectve dmesos usg global sestvty aalyss, Relab. Eg. ad Syst. Saf. 96 (0)

29 [5] R. Ima, M. Shortecarrer, A FORTRA 77 program ad users gude for the geerato of Lat hypercube ad radom samples for use wth computer models, Sada atoal Laboratores, Albuquerque (M), Rep. SAD83, 984. [6] M. Ste, Large sample propertes of smulatos usg Lat hypercube samplg, Techometrcs 9 (987) [7] J. Burkardt, Departmet of Scetfc Computg at Florda State Uversty; GU LGPL lcese [8] G. Dambl, M. Couplet, B. Iooss, umercal studes of space fllg desgs: optmzato algorthms ad subprojecto propertes, J. of Smul. 7(03) [9]. ederreter, Radom umber geerato ad quas-mote Carlo methods, Soc. for Id. ad Appl. Math, 99. [0] W. Morokoff, R.E. Caflsch, Quas Mote-Carlo tegrato, J. of Comput. Phys. (995) [] P. Glasserma, Mote Carlo methods facal egeerg, Sprger, 003. [] P. Jaeckel, Mote Carlo methods face, Lodo, Joh Wley & Sos, 00. [3] I. Sobol', O the dstrbuto of pots a cube ad the approxmate evaluato of tegrals, Comput. Math. ad Math. Phys. 7 (967) 86-. [4] I. Sobol', Uformly dstrbuted sequeces wth addtoal uformty propertes, USSR Comput. Math. ad Math. Phys. 6(5) (976) [5] T.T Warock, Computatoal vestgatos of low-dscrepacy pot sets, Applcatos of umber Theory to umercal Aalyss, Academc Press, ew York, 97, pp [6] F. ckerell, A geeralzed dscrepacy ad quadrature error boud, Math. Comp. 67 (998) [7] BRODA Ltd [8] I. Sobol', D. Asotsky, A. Kre, S. Kuchereko, Costructo ad comparso of hgh-dmesoal Sobol geerators, Wlmott J. ov (0) [9] I. Sobol', Sestvty estmates for olear mathematcal models, Mat. Modelrovae () (990) -8 ( Russa), traslated I.M. Sobol', Sestvty estmates for olear mathematcal models, Math. Model. ad Comput. Exp. (993) [30] T. omma, A. Saltell, Importace measures global sestvty aalyss of olear models, Relab. Eg. Syst. Saf. 5 (996) 7. [3] A. Saltell, S. Taratola, F. Campologo, Sestvty aalyss as a gredet of modelg, Stat. Sc. 5(4) (000) [3] A. Saltell, S. Taratola, O the relatve mportace of put factors mathematcal models: Safety assessmet for uclear waste dsposal, J. of Am. Stat. Assoc. 97 (00) [33] A. Saltell, Makg best use of model valuatos to compute sestvty dces, Comput. Phys. Commu. 45 (00)

30 [34] A. Saltell, P. Ao, I. Azz, F. Campologo, M. Ratto, S. Taratola, Varace based sestvty aalyss of model output, Desg ad estmator for the total sestvty dex, Comput. Phys. Commu. 8() (00) [35] K. Pearso, O the geeral theory of skew correlato ad o-lear regresso, Vol. IV of Math. Cotrb. to the Theory of Evol., Drapers' Compay Research Memors, Reprted : Early, Lodo:Dulau & Co., 905. [36] R.E. Caflsch, W. Morokoff, A.B. Owe, Valuato of mortgage backed securtes usg Browa brdges to reduce effectve dmeso, The J. of Comput. Fac. (997) [37] I. Sobol, S. Kuchereko, O global sestvty aalyss of quas-mote Carlo algorthms, Mt.-Carlo Method ad Smul. () (005) -9. [38] S. Kuchereko,. Shah, The Importace of beg Global, Applcato of global sestvty aalyss Mote Carlo opto prcg, Wlmott J. July (007)