Design and analysis of computer experiments

Transcription

1 Desgn and analyss of comuter exerments Bertrand Iooss 9//0

2 Uncertanty management - The generc methodology Venez course 0 Desgn and analyss - B. Iooss

3 Man objectves of senstvty analyss Reducton of the uncertanty of the model oututs by rortzaton of the sources Varables to be fxed n order to obtan the largest reducton or a fxed reducton of the outut uncertanty A urely mathematcal varable orderng Most nfluent varables n a gven outut doman f reducbles, then R&D rortzaton else, modfcaton of the system Smlfcaton of a model determnaton of the non-nfluent varables, that can be fxed wthout consequences on the outut uncertanty buldng a smlfed model, a metamodel Venez course 0 Desgn and analyss - B. Iooss 3

4 Outlne.Desgn of numercal exerments Sace fllng desgns. Analyss of numercal exerments Venez course 0 Desgn and analyss - B. Iooss 4

5 Tycal engneerng ractce : One-At-a-Tme OAT desgn P3 P P P Man remars : OAT brngs some nformaton, but otentally wrong Exloraton s oor: Non monotoncty? Dscontnuty? Interacton? Leave large unexlored zones of the doman curse of dmensonalty Venez course 0 Desgn and analyss - B. Iooss 5

6 Model exloraton goal GOAL : exlore as best as ossble the behavour of the code Put some onts n the whole nut sace n order to «maxmze» the amount of nformaton on the model outut Contrary to an uncertanty roagaton ste, t deends on Regular mesh wth n levels N n smulatons Ex:, n 3 N 9 0, n3 N To mnmze N, needs to have some technques ensurng good «coverage» of the nut sace Smle random samlng Monte Carlo does not ensure ths Ex: N 0 V V V Monte Carlo V Otmzed desgn Venez course 0 Desgn and analyss - B. Iooss 6

7 Objectves When the objectves s to dscover what haens nsde the model and when no model comutatons have been realzed, we want to resect the two followng constrants: To sread the onts over the nut sace n order to cature non lneartes of the model outut, To ensure that ths nut sace coverage s robust wth resect to dmenson reducton. Therefore, we loo some desgn whch nsures the «best coverage» of the nut sace Man queston: How to defne ths «best»? Venez course 0 Desgn and analyss - B. Iooss 7

8 Exloraton n hyscal exermentaton Desgn of exerments develos strateges to defne exerments n order to obtan the requred nformaton as effcently as ossble Desgns for real exerments Estmate arameters of lnear regresson wth a mnmal number of onts Examles : Full factoral desgn 3 Fractonal factoral desgn 3- Desgns for numercal exerments Characterstcs Determnstc exerments no error, Large number of nut varables, Large range of nut varaton doman, Multle outut varables, arameter Strong nteractons between nuts, Hgh non lnearty n the model arameter arameter 3 sace fllng desgns unform coverage n the nut sace Venez course 0 Desgn and analyss - B. Iooss 8

9 Sace fllng desgns Sarsty of the sace of the nut varables n hgh dmenson The learnng desgn choce s made n order to have an otmal coverage of the nut doman The sace fllng desgns are good canddates. Smle Random Samle SRS V V Sace Fllng Desgn SFD V V Examle: Sobol sequence Two ossble crtera:. Dstance crtera between the onts: mnmax, maxmn,. Unformty crtera of the desgn dscreancy measures Venez course 0 Desgn and analyss - B. Iooss 9

10 Geometrcal crtera / Mnmax desgn D MI : Mnmze the maxmal dstance between one ont of the doman and one ont of the desgn mn max d x, D D x where d x, D max d x, D mn d x, x 0 D All onts n [0,] are not too far from a desgn ont x x MI 0 [ Johnson et al. 990 ] [ Koehler & Owen 996 ] > One of the best desgn, but too exensve to fnd D MI Venez course 0 Desgn and analyss - B. Iooss 0

11 Mnmax desgn ; -/N ; φ mm / N > : shere recoverng [ ] Venez course 0 Desgn and analyss - B. Iooss

12 Geometrcal crtera / φ Ξ N mn d x, x N - Mndst dstance: L norm for examle x, x Ξ Maxmn desgn Ξ N Mm : maxmze mnmal dstance between two onts of the desgn max Ξ N x mn d x, x mn d x, x, x Ξ N x, x Ξ N Mm - Venez course 0 Desgn and analyss - B. Iooss

13 Maxmn desgn ; -/N- ; φ mm / N- > : shere acng [ ] [ ] Venez course 0 Desgn and analyss - B. Iooss 3

14 Sace fllng measure of a desgn: the dscreancy Measure of the maxmal devaton between the dstrbuton of the samle s onts to an unform dstrbuton Measure of devaton from the unformty Geometrcal nterretaton: Comarson between the volume of ntervals and the number onts wthn these ntervals Q t [0,[, Q t [0, t [ [0, t [ K [0, t [ dsc D su Q t [0,[ N Q t N t Lower the dscreancy s, the more the onts of the desgn D fll the all sace Venez course 0 Desgn and analyss - B. Iooss 4

15 Ln wth the ntegraton roblem I [0,[ f x dx Monte Carlo: I wth MC N x N... N N f x a sequence of random onts n [0,[ MC MC Var N Ε I N I ; Var I N ε O N N General roerty Kosma-Hlawa nequalty: ε V f dsc D Wth a low dscreancy sequence D quas Monte Carlo sequence : Well-nown choce: Sobol sequence ε O ln N N Venez course 0 Desgn and analyss - B. Iooss 5

16 L dscreancy Several defntons, deendng on consdered norms and ntervals D * N Ξ su t [0,[ N N x Q t Volume Q t Choce allowng comutatons : L dscreancy [ Hcernell 998 ] L dscreancy at orgn : D * N Ξ N N [ 0,[ x Q t Volume Q t dt / Mssng roerty: tang nto account unformty of the ont rojectons On lower-dmensonal subsaces of [0,[ > Modfed L dscreances N D Ξ wth u and Q {,..., } N u Ø u C xu u N t rojecton of Q u t Q t on C Volume Q u u t unt cube of dt coordnates n u Venez course 0 Desgn and analyss - B. Iooss 6

17 Dscreancy comutaton n ractce Modfed L -dscreancy ntervals wth mnmal boundary 0 Centered L -dscreancy ntervals wth boundary one vertex of the unt cube Venez course 0 Desgn and analyss - B. Iooss 7 Symetrc L -dscreancy ntervals wth boundary one «even» vertex of the unt cube N j j j N x x x x N x x N D, 3 dsc

18 Sobol sequence vs. Random samle vs. regular grd [ From: Kuchereno, 00 ] Venez course 0 Desgn and analyss - B. Iooss 8

19 Examle - N 50 - Dmenson 8 Sobol Sobol scramblng Owen Venez course 0 Desgn and analyss - B. Iooss 9

20 Examle - N 50 - Dmenson 8 Halton Venez course 0 Desgn and analyss - B. Iooss 0

21 Pathologes on D rojectons Halton Venez course 0 Desgn and analyss - B. Iooss

22 Imortant roerty: robustness n terms of subrojectons Most of the tmes, the functon f has low effectve dmensons: - n the truncaton sense number of nfluent nuts << - n the sueroston sense hgher order of nfluent nteracton << Then, we need SFD whch ees ther sace-fllng roertes n low-dmensonal subsaces by mortance: n dmensons, then,... LHS ensures good D rojecton roertes good bad In ther defnton, the modfed L -dscreancy crtera tae nto account subrojectons In contrary desgn onts dstance crtera are not robust at all Venez course 0 Desgn and analyss - B. Iooss

23 Latn Hyercube Samle LHS Most often, only a small number of varables are nfluent [ McKay et al. 979 ] Proerty: Unform rojectons on margns Prncle: varables, N onts LHS,N Dvde each dmenson n N ntervals Tae one ont n each stratum Exemle :, N 4 Each level s taen only one tme by each varable Each column of the desgn s a ermutaton of {,,..,N } Venez course 0 Desgn and analyss - B. Iooss 3

24 Algorthm of LHS,N Sten method ran matrxrunfn*,nrown,ncol #trage de N x valeurs selon lo U[0,] x matrx0,nrown,ncol # constructon de la matrce x for n : { dx samle:n #vecteur de ermutatons des enters {,,,N} P dx-ran[,] / N # vecteur de robabltés x[,] <- quantle_selon_la_lo P } Examle :, N 0, ~ U[0,], ~ N0, Venez course 0 Desgn and analyss - B. Iooss 4

25 Otmsaton of LHS > Sace-fllng LHS [ Par 993; Morrs & Mtchell 995 ] Smle method: roduce a large number for ex 000 of dfferent LHS. Then, choos the best wth resect to a crteron φ. «sace fllng» Examle : LHS,6 BUT: the number of LHS s huge : Maxmn crteron N! Methods va otmzaton algo ex: mnmsaton of φ. va smulated annealng :. Intalsaton of a desgn Ξ LHS ntal and a temerature T. Whle T > 0 :.Produce a neghbor Ξ new of Ξ ermutaton of comonents n a column.relace Ξ by Ξ new wth roba φ Ξnew φ Ξ mn ex, T 3.decrease T 3. Sto crteron > Ξ s the otmal soluton Venez course 0 Desgn and analyss - B. Iooss 5

26 Examles of otmzed LHS Jonng the two roertes sace fllng and LHS Examle: N Maxmn LHS Low wra-around For comarson: dscreancy LHS Sobol sequence Venez course 0 Desgn and analyss - B. Iooss 6

27 Summary on the desgn of numercal exerments Goal: Samle a hgh dmensonam sace n an «otmal» manner obtan the maxmum of nformaton on the behavour of the oututz / є R Problem: a ure random samle Monte Carlo badly flls the sace.«sace fllng» desgns are good canddates: - Based on a dstance crteron between onts mnmax, maxmn, - Based on a cteron of unform dstrbuton of the onts dscreancy.proerty of unform rojectons on margns can be obtaned va the Latn hyercube desgns LHS 3.It s ossble to coule and Venez course 0 Desgn and analyss - B. Iooss 7

28 Outlne.Desgn of numercal exerments. Analyss of numercal exerments Senstvty analyss Venez course 0 Desgn and analyss - B. Iooss 8

29 Senstvty analyss notons Senstvty, for examle Y Donne une dée de la manère dont eut réondre la réonse en foncton de varatons otentelles des facteurs Contrbuton senstvty x mortance, for examle Y σ Permet de détermner le ods d une varable d entrée ou groue de varables sur l ncerttude de la varable d ntérêt la sorte f Y D D Y Venez course 0 Desgn and analyss - B. Iooss 9

30 Man objectves of senstvty analyss Reducton of the uncertanty of the model oututs by rortzaton of the sources Varables to be fxed n order to obtan the largest reducton or a fxed reducton of the outut uncertanty A urely mathematcal varable orderng Most nfluent varables n a gven outut doman f reducbles, then R&D rortzaton else, modfcaton of the system Smlfcaton of a model determnaton of the non-nfluent varables, that can be fxed wthout consequences on the outut uncertanty buldng a smlfed model, a metamodel Venez course 0 Desgn and analyss - B. Iooss 30

31 Overall classfcaton of senstvty analyss methods quantty of nterest varablty of the outut Three tyes of answers:. Screenng : - classcal desgn of exerments, - numercal desgn of exerments Morrs, sequental bfurcaton. Quanttatve measures of global nfluence : - correlaton/regresson on values/rans - statstcal tests, - functonal varance decomoston Sobol, - other measures : entroy, dstrbuton dstances 3. Dee exloraton of senstvtes - smoothng technques aram./non arametrc - metamodels Y 3 Venez course 0 Desgn and analyss - B. Iooss 3

32 Screenng wth n < suersaturated desgns Many nuts >> 0 and cu tme costly comuter code Objectve: less comutatons than number of nuts Hyotheses: Number of nfluent nuts << total number of nuts Monotony of the model, no nteracton between nuts Knowledge of the drecton of the outut varaton / each nut Examle: method of sequental bfurcatons runs 3 Y + 4 Y - Venez course 0 Desgn and analyss - B. Iooss 3

33 Screenng wth n < suersaturated desgns Many nuts >> 0 and cu tme costly comuter code Objectve: less comutatons than number of nuts Hyotheses: Number of nfluent nuts << total number of nuts Monotony of the model, no nteracton between nuts Knowledge of the drecton of the outut varaton / each nut Examle: method of sequental bfurcatons runs run 3 Y + s Y + Y Y 4 Y Venez course 0 Desgn and analyss - B. Iooss 33

34 Screenng wth n < suersaturated desgns Many nuts >> 0 and cu tme costly comuter code Objectve: less comutatons than number of nuts Hyotheses: Number of nfluent nuts << total number of nuts Monotony of the model, no nteracton between nuts Knowledge of the drecton of the outut varaton / each nut Examle: method of sequental bfurcatons runs run 3 4 Y + Y Y s Y + Y - s Y ~ Y A few nfluent Number of runs: n ~ / Venez course 0 Desgn and analyss - B. Iooss 34

35 Screenng wthout hyothess on functon: Morrs method Dscretzaton of nut sace Needs + exerments P3 P P OAT One-at-A-Tme comutaton of one elementary effect for each nut Venez course 0 Desgn and analyss - B. Iooss 35

36 Morrs method 4 3 OAT desgn s reeated R tmes total: n R*+ exerments It gves an R-samle for each elementary effect 5 Senstvty measures: Venez course 0 Desgn and analyss - B. Iooss 36

37 Morrs: senstvty measures s a measure of the senstvty: Imortant value mortant effects n mean senstve model to nut varatons s a measure of the nteractons and of the non lnear effects: mortant value dfferent effects n the R-samle effects whch deend on the value: of the nut > non lnear effect or of the other nuts > nteracton the dstncton between the two cases s mossble Venez course 0 Desgn and analyss - B. Iooss 37

38 Morrs : examle 0 factors 0 smulatons Grah mu*, sgma σ 3 Dstncton between 3 grous: µ*. Neglgble effects. Lnear effects 3. Non lnear effects and/or wth nteractons Cas test : non monotonc functon of Morrs Venez course 0 Desgn and analyss - B. Iooss 38

39 Examle : fuel rradaton comutaton n HTR Comuter code ATLAS CEA : smulaton of the HTR fuel fuel artcles behavour under rradaton < µµ φ Noyau de matère fssle Carbone yrolytque oreux Carbone yrolytque dense Carbure de Slcum Contamnaton sources: falure of artcles Relablty studes are needed Number of artcles nsde a reactor : 0 9 to 0 0! The falure of a artcle can be caused by the falure of the external thc layers IPyC, SC, OPyC Outut varables are rersentatve of falure henomena: maxmal orthoradal strans n external layers Venez course 0 Desgn and analyss - B. Iooss 39

40 3 uncertanty tyes for the nuts 0 arameters of fuel artcle manufacturng rocess thcness, Secfcatons truncated Gaussan dstrbutons 5 arameters of rradaton temerature, Interval [mn,max] unform dstrbutons 8 behavour laws functons of temerature, flux, Exert judgment multlcatve constants ~ U[0.95,.05] Examle : Law of PyC densfcaton Venez course 0 Desgn and analyss - B. Iooss 40

41 Results of Morrs 43 nuts, 0 reettons, n 860 runs, untary cost ~ mn > total4h Large senstvtes to these nuts thcness, rradaton temerature Small nteracton effects Influence of cree and densfcaton laws of PyC Concluson: Morrs method rovdes qualtatve nformaton about outut varatons due to otental varatons of nuts Useful n order to dentfyles otental nfluent nuts Venez course 0 Desgn and analyss - B. Iooss 4

42 Overall classfcaton of senstvty analyss methods quantty of nterest varablty of the outut Three tyes of answers:. Screenng : - classcal desgn of exerments, - numercal desgn of exerments Morrs, sequental bfurcaton. Quanttatve measures of global nfluence : - correlaton/regresson on values/rans - statstcal tests, - functonal varance decomoston Sobol, - other measures : entroy, dstrbuton dstances 3. Dee exloraton of senstvtes - smoothng technques aram./non arametrc - metamodels Y 3 Venez course 0 Desgn and analyss - B. Iooss 4

43 Senstvty analyss for one scalar outut Samle R, Y R of sze N > Prelmnary ste: grahcal vzualsaton for ex: scatterlots Remar: t can be a Monte Carlo samle, a quas-monte Carlo samle or any other desgns Venez course 0 Desgn and analyss - B. Iooss 43

44 Grahal reresentaton : scatterlots Measure the lnear characher of the cloud N runs Grahs Outut wth resect to each nut cov, Y ρ σ σ Y π Examle : N ι3 ˆρ N N x s x y x s y y Venez course 0 Desgn and analyss - B. Iooss 44

45 Flood model - Scatterlots Outut S Q rver flowrate ~ Gumbel on [500,3000] Ks frcton coeffcent ~ normal on [5,50] Zv downstream rver bed hegth ~ trangular on [49,5] Hd dye hegth ~ trangular on [7,9] Cb ban hegth ~ trangular on [55,56] Monte Carlo samle N 00 Venez course 0 Desgn and analyss - B. Iooss 45

46 Flood model - Scatterlots Outut C Monte Carlo samle N 00 Major drawbac: only frst order relatons between nuts are analyzed and not ther nteractons > needs of other data anlyss tools Venez course 0 Desgn and analyss - B. Iooss 46

47 Senstvty analyss for one scalar outut Senstvty ndces Samle R, Y R of sze N > Prelmnary ste: grahcal vzualsaton for ex: scatterlots Quanttatve senstvty analyss methodology? Lnear relaton? Ou R² Lnear regresson between and Y Regresson coeffcents [Saltell et al. 00, Helton et al. 06 ] Non? Monotonc relaton? Ou Regresson on rans R²* Non Sobol ndces Venez course 0 Desgn and analyss - B. Iooss 47

48 Senstvty ndces n case of lnear nuts/outut relaton Indeendent nut varables Samle : n realzatons of SRC ndex: Y + β 0 β SRC :,Y β,..., Var Var Sgn of β gves the drecton of varaton of Y n fct of Y SRC s smlar to the lnear correlaton coeffcent Pearson Valdty of the lnear model va The resduals dagnostcs and R² : We have R SRC R n n Yˆ Y Y Y > nce nterretaton of SRC Venez course 0 Desgn and analyss - B. Iooss 48 48

49 Flood model - Outut S Monte Carlo samle N 00 3% 5% 8% 9% 6% Senstvty ndces SRC The model s lnear R²0.99 SRC coeffcents are suffcent for the quanttatve sensvty analyss Venez course 0 Desgn and analyss - B. Iooss 49

50 Senstvty analyss for one scalar outut Senstvty ndces Samle R, Y R of sze N > Prelmnary ste: grahcal vzualsaton for ex: scatterlots Quanttatve senstvty analyss methodology? Lnear relaton? Ou R² Lnear regresson between and Y Regresson coeffcents [Saltell et al. 00, Helton et al. 06 ] Non? Monotonc relaton? Ou Regresson on rans R²* Non S Sobol ndces Var[E Y Var Y ] Venez course 0 Desgn and analyss - B. Iooss 50

51 Functonal decomoston y f x f 0 + f x + fj x, x j f,,..., x, x,..., x wth f x L x x [0; ] j> Infnty of ossble decomostons BUT, uncty of decomoston f:... x,..., x dx j 0 j,..., s s Proertes x ~ U[O,] for,,, the x s are ndeendent 0 f x y x d E f f f x f x dx f0 E y x f0 x, x j E y x, x j E y x E y x j f0 j + f s Examle : f x f 0, x ; f x x + x x ; x ; ~ U[0;] f x ; x x ~ U[0;] ; f x, x 0 Venez course 0 Desgn and analyss - B. Iooss 5

52 Another examle f 0 0 f x x 4 f x 3x f f x + x, x 4x 3 x, x U[ / ; ] E y 4x + 3x dx dx 0 / / / / f x E y x f0 4 x + 3 x dx 4 x / 3 3 f x, x 0 [ JRC 00 ] f x x E y x f0 3 f x, x 0 Venez course 0 Desgn and analyss - B. Iooss 5

53 Senstvty ndces wthout model hyotheses Functonal ANOVA [Efron & Sten 8] hy. of ndeendent s : Var Y V Y + V j Y +L + V Sobol ndces defnton: < j where V j L Y V Y Var Var[ E Y ] [ E Y ] V V,... j j Frst order senstvty ndces: S V VarY Second order senstvty ndces: S j V j VarY... Venez course 0 Desgn and analyss - B. Iooss 53 53

54 Another examle y f x + x, x 4x 3 x, x U[, ] On a vu : f 0 E y Var f x 3 S V f x E y x f0 4x 3 Var f x f x E y x f0 3x S V f x, x 0 [ ] [ ] [ JRC 00 ] Venez course 0 Desgn and analyss - B. Iooss 54

55 Grahcal nterretaton Frst order Sobol ndces measure the varablty of condtonal exectatons mean trend curves n the scatterlots Null ndex Hgh ndex er d Small ndex d Venez course 0 Desgn and analyss - B. Iooss 55

56 Sobol ndces roertes S S + S + S j + S j... S,,..., j j Always Addtve model S Measure the degree of nteractons between varables S Examles : 4 gves 4 ndces S, 6 ndces S j, 4 ndces S j, ndce S jl General case : - ndces to be estmated Total senstvty ndex: S T S + Sj + Sj +... S~ j j, [ Homma & Saltell 996 ] Venez course 0 Desgn and analyss - B. Iooss 56

57 Flood model Venez course 0 Desgn and analyss - B. Iooss 57

58 Sobol ndces comutaton Indces for st order and total : S V Var Y and S T V~ Var Y Formulatons of the condtonal varances: Let ~, and ' an ndeendent coy of Y d E Y V Y Var[E Y ] E ' [ f, f, ] V Y Var[E Y ~ ] Cov, ~ ~ ~ ' d Cov[ f, f ], ~, ~ Venez course 0 Desgn and analyss - B. Iooss 58

59 Drect estmaton va Monte Carlo..d. samles : Varance classcal estmator : Condtonal varances estmaton: Indces st order : cost n + n n f n f f f n Y V 0 0 ˆ avec ˆ ˆ 0 ',..., ',, ',..., ',...,,,,..., ˆ f f f n Y V n + + n j j n j j,.., ;,..,,.., ;,.., ' and Venez course 0 Desgn and analyss - B. Iooss 59 Indces st order + total ndces : cost n +, by nvertng In ractce, n ~ e4 > roblem of the cost n terms of requred model runs Other formula Jansen-Sobol estmator: 0 ~,...,, ',,...,,...,,,,..., ˆ f f f n Y V n + + ˆ n ' and ~,.., ;,..,,.., ;,.., Y V n j j n j j [ ] + n f f f n V,,,,,,,,,,...,,...,,,,...,,..., ˆ

60 The samlng-based aroaches Samle R, Y R of sze N >? Lnear relaton? Senstvty ndces Yes Lnear regresson between and Y Regresson coeffcents R² No? Monotonc relaton? Yes Regresson on rans Monte Carlo N > 000 R²* No neglgble small Quas-MC, FAST, RBD, N > 00 Sobol ndces? CPU tme cost of the model? Venez course 0 Desgn and analyss - B. Iooss 60 large Smoothng Metamodel N > 0

61 Flood model Outut C From the 00-sze Monte Carlo samle, a Gaussan rocess metamodel s ftted Predctvty of the G metamodel : Q 99% Ne5 00 relcates N x + x 00 7e7 evaluatons Venez course 0 Desgn and analyss - B. Iooss 6

62 Classfcaton of senstvty analyss methods Calculatons of all tyes of ndces Sobol, dstrbuton-based, + man effects EY Comlexty/regularty of model f Non monotonc Screenng Morrs Metamodel Varance decomoston Sobol ndces Monotonc + nteractons Monotonc wthout nteracton Lnear st degree Suer screenng Desgn of exerment Monte-Carlo samlng Ran regresson Lnear regresson Number of model f number of nut varables evaluatons Venez course 0 Desgn and analyss - B. Iooss 6

63 Bblograhy Fang et al., Desgn and modelng for comuter exerments, Chaman & Hall, 006 J.C. Helton, J.D. Johnson, C.J. Salaberryet C.B. Storle: Survey of samlngbased methods for uncertanty and senstvty analyss. Relablty Engneerng and System Safety, 9:75 09, 006. Klejnen, The desgn and analyss of smulaton exerments, Srnger, 008 Koehler & Owen, Comuter exerments, 996 A. Saltell, K. Chan & E.M. Scott, Senstvty analyss, Wley, 000 A. Saltell et al., Global senstvty analyss - The rmer. Wley, 008. Venez course 0 Desgn and analyss - B. Iooss 63