Quantifying uncertain system outputs via the multilevel Monte Carlo method Part I: Central moment estimation

Transcription

1 MATHICSE Institute of Mathematics Schoo of Basic Sciences MATHICSE Technica Report Nr October 017 Quantifying uncertain system outputs via the mutieve Monte Caro method Part I: Centra moment estimation Michee Pisaroni, Sebastian Krumscheid, Fabio Nobie Address: EPFL - SB INSTITUTE of MATHEMATICS Mathicse Bâtiment MA Station 8 - CH Lausanne - Switzerand

2

3 Quantifying uncertain system outputs via the mutieve Monte Caro method Part I: Centra moment estimation M. Pisaroni a,, S. Krumscheid a, F. Nobie a a Cacu Scientique et Quantication de 'Incertitude CSQI, Institute of Mathematics, Écoe Poytechnique Fédérae de Lausanne, 1015 Lausanne, Switzerand Abstract In this work we introduce and anayze a nove mutieve Monte Caro MLMC estimator for the accurate approximation of centra moments of system outputs aected by uncertainties. Centra moments pay a centra roe in many discipines to characterize a random system output's distribution and are of primary importance in many prediction, optimization, and decision making processes under uncertainties. We detai how to eectivey tune the MLMC agorithm for centra moments of any order and present a compete practica agorithm that is impemented in our accompanying Python ibrary cmmc-py 1. In fact, we vaidate the methodoogy on seected reference probems and appy it to an aerodynamic reevant test case, namey the transonic RAE 8 airfoi aected by operating and geometric uncertainties. Keywords: centra moments, mutieve Monte Caro, uncertainty quantication, h-statistic 1. Introduction Probabiistic modes are widey used across many discipines, incuding engineering and nancia appications. Inevitaby any quantity of interest QoI Q, say, that is derived from a probabiistic mode's output is a random variabe. Consequenty, important tasks such as predictions, decision making, or optimization based on the avaiabe system knowedge Q need to be carried our subject to uncertainties. To reiaby account for the eects of these uncertainties, it is indispensabe to characterize the distribution of Q or some statistics of it. Whie the accompanying Part II of this work wi be devoted to approximations to the cumuative distribution function CDF and robustness indicators such as quanties aso known as vaue-at-risk and the conditiona vaue-at-risk, here we focus on the commony used approach of characterizing a random system output's distribution by a carefuy chosen seection of moments of Q. In fact, in addition to the mean, many important features of distribution, such as ocation, dispersion, or asymmetry, can be assessed through moments of the deviation from the mean. These statistica moments that are computed about the mean Corresponding author Emai addresses: michee.pisaroni@epf.ch M. Pisaroni, sebastian.krumscheid@epf.ch S. Krumscheid, fabio.nobie@epf.ch F. Nobie 1 The cmmc-py python ibrary is avaiabe at Preprint submitted to Journa of Computationa Physics October 6, 017

4 are caed centra moments. Specicay, the p-th centra moment µ p of a random variabe Q is dened as µ p = E [ Q µ p], with µ = E[Q], provided the right-hand side exists. Foowing the denition of µ p, the rst centra moment i.e. p = 1 is equa to zero. The second centra moment µ is the variance often aso denoted as σ and, together with the mean µ = E[Q], is one of the most commony used quantities in appications to characterize a random variabe. The third centra moment µ 3 oers insight into the asymmetry of a random variabe's distribution about its mean. Specicay, the skewness γ = µ 3 / µ 3, which is the standardized counterpart of the third centra moment, is commony used as a measure of probabiity distribution's asymmetry. Indeed, the skewness or equivaenty the third centra moment µ 3 of a symmetric distribution about the mean is zero. Negative vaues of the skewness indicate that the probabiity distribution has a eft tai that is onger compared to the right one. Anaogousy, positive vaues indicate a onger right tai. A measure of a probabiity distribution's asymmetry is very important in many engineering risk/reiabiity assessments and nancia appications reated to stock prices and assets. In fact, Mandebrot et a. [14] observed that the majority of nancia assets returns are non-norma. This is due to the appearance of extreme events more ikey than predicted by a norma distribution [4] and due to the fact that crashes occur more often than booms [17]. For this reason, investment decisions based ony on the mean and variance cannot discriminate whether a given future event wi be more or ess ikey to appear on the eft or right side of the mean [13, 15]. Appied to investment returns, negativey skewed distributions indicate greater chance of extremey negative outcomes, whie in positive skewed distributions extremey bad scenarios are not as ikey. Assuming a norma distribution, when in fact data sets are skewed, can ead to the so caed skewness risk [5]. Simiar probems arise in many appications across science and technoogy where decisions based on a reiabiity or risk measure need to be taken. The fourth centra moment µ 4 and its standardized counterpart, which is known as kurtosis Kurt = µ 4 /µ, provide some further important insights into a random variabe's distribution. In fact, the kurtosis can be used to measure whether the output random variabe are heavy-taied high eve of kurtosis or ight-taied ow eve of kurtosis compared to a norma distribution, for which Kurt = 3. Heavy-taied distributions are common in probems where extreme events are ikey to appear. Random variabes with ow eves of kurtosis tend to have ight tais and ack of extreme events. In other words high eves of kurtosis indicate that most of the variabiity in the distribution is due to extreme deviations from the mean. In this work we consider ecient samping-based estimators for centra moments of a quantity of interest Q output of a compex probabiistic mode such as a uid ow with random inow conditions, a stochastic dynamica system, a random partia dierentia equation, etc.. We address in particuar probabiistic modes that invove dierentia equations such as system of SDEs, SPDEs, or PDEs with random input parameters for which typicay the random system output Q cannot be samped exacty and ony approximate samping can be accessed with a given accuracy e.g. by soving the dierentia equation via some numerica scheme. As a consequence of this inexact samping, a bias is introduced that has to be accounted for. In the context of estimating the mean E[Q] of Q, the mutieve Monte Caro MLMC method has estabished itsef as a versatie and ecient samping-based method

5 when ony inexact samping is possibe [6, 9, 10, 19]. Specicay, et Q M denote an approximation of the unknown random variabe Q obtained via a numerica scheme with M degrees of freedom, i.e. via a numerica scheme that reduces the origina innite dimensiona probem to a M-dimensiona one and has a computationa cost proportiona to some power of M. The key idea of the MLMC method then is to consider not just approximations obtained from said scheme with M degrees of freedom, but aso approximations obtained with different numbers of degrees of freedoms. In fact, a hierarchy of L + 1 approximation eves is considered, where each approximation eve is dened by its size and the dierent eves are reated by M 0 < M 1 < < M L. Using N independent and identicay distributed i.i.d. reaizations ω i,, i = 1,, N, of the system's random input parameter on each eve 0 L, the MLMC estimator for the mean µ = E[Q] of Q is E MC Q 0 N 0,M 0 + L =1 E MC Q N,M E MC Q N,M 1, 1 where E MC denotes the sampe average operator and Q N,M := Q M ω i, is the i=1,...,n sampe of N i.i.d. repications of Q M. Here, the same ω i, are used in both Q N,M and Q N,M 1, which yieds a strong correation and, hopefuy, an estimator with smaer variance. In this work we extend the MLMC concept to the estimation of arbitrary order centra moments µ p. Specicay, we introduce and anayze a nove mutieve Monte Caro method that aows an ecient samping-based estimation from inexact/approximate sampes. One of the method's key ingredients is the use of h-statistics [8] as unbiased centra moment estimators with minima variance for the eve-wise contributions. That is, instead of the eve-wise contributions E MC Q N,M E MC Q N,M 1 that are used in the estimation of the mean c.f. 1, here we use terms of the form h p Q N,M hp Q N,M 1, where hp denotes an appropriate h-statistic of order p. Consequenty, the MLMC estimator p for arbitrary order p centra moment considered here is of the form p = h p Q 0 N0,M 0 + L =1 h p Q N,M hp Q N,M 1. We note that a mutieve Monte Caro estimator for the variance µ of a random variabe Q has aready been introduced in []. There, the authors dene the mutieve Monte Caro estimator by teescoping on the unbiased sampe variance estimator for the eve-wise contributions. Our approach based on h-statistics thus oers an aternative derivation of said estimator, which aows for a more straightforward compexity anaysis in fact. Moreover, the approach introduced here is easiy generaized to arbitrary order centra moments, as we wi iustrate in the foowing. In fact, the resuts presented here for estimating µ p for p 3 appear to be nove. Finay, we mention that somewhat reated work on mutieve Monte Caro techniques for arbitrary order centra moment estimators can be found in [3]. However, there the authors construct the estimators for p 3 based on biased estimators for the eve-wise contributions. Consequenty, the method introduced in the aforementioned work requires to carefuy contro this additiona bias. Moreover, the mean squared error anaysis is aso aected by this bias, in the sense that the error is quantied using worstcase bounds based on triange inequaities. Instead our work, as mentioned earier aready, 3

6 uses h-statistics for eve-wise contributions, which are unbiased estimators with minima variance. In fact, these unbiased estimators can be straightforwardy derived in cosed-form, aowing for a possiby sharper mean squared error bound. The cost of working with unbiased estimators is that deriving these estimators in cosed-form requires somewhat tedious cacuations. However, these cacuations can be easiy carried out automaticay by symboic computer agebra systems, such as Mape and Mathematica, as we wi describe in the foowing. Lasty, we present a compete agorithm and detai how to tune the MLMC method for centra moments to achieve optima compexity. In fact, we demonstrate the eectiveness of the deveoped methodoogy, which has been impemented in our Python ibrary cmmc-py, by appying it to a number of seected exampes. These exampes incude a reevant probem in compressibe aerodynamics, namey a transonic airfoi aected by both operating and geometric uncertainties, which we anayze based on estimates of the rst four statistica centra moments. The rest of this work is organized as foows. In Sects. 3 we present and anayze the samping-based estimation of centra moments. Specicay, in Sect. we rst consider the cassic Monte Caro method, before introducing the nove mutieve Monte Caro estimator in Sect. 3. Foowing these theoretica considerations, we discuss various practica aspects and impementation detais for the mutieve Monte Caro methods in Sect. 4. In Sect. 5 we appy the MLMC methodoogy to the above mentioned numerica exampes. Finay, we oer a summary and a discussion of our resuts in Sect. 6.. Monte Caro estimation of centra moments The p-th centra moment µ p µ p Q of a random variabe Q aso known as the p-th moment about the mean is given by [ Q ] p µ p Q := E E[Q], for any p N provided it exists, athough the vaue for p = 1 is trivia µ 1 Q = 0. Any centra moment can, of course, be expressed in terms of non-centered so-caed raw moments or moments about the origin as a consequence of the binomia theorem and the inearity of the expected vaue: [ Q ] p µ p Q E E[Q] = n j=0 p 1 p j E [ Q j] E[Q] p j. j However, approximating the p-th centra moment µ p Q by a combination of approximated non-centered moments can be numericay unstabe. This may be especiay severe if the centra moments are sma whereas the raw moments are not. To avoid these numerica instabiities, here we present Monte Caro samping based estimators for centra moments directy. We begin by reviewing cassic singe-eve Monte Caro estimators in this Section, before addressing the mutieve estimators in Sect. 3. Starting point for the construction of ecient samping based estimators for centra moments are the so-caed h-statistics [8]. That is, in the cassic singe-eve setting we consider an i.i.d. sampe Q N := Qω i, of size N, where each Qω i=1,...,n i has the same 4

7 distribution as Q. The h-statistic h p h p Q N then is an unbiased estimator of µ p Q, in the sense that E [ h p Q N ] = µ p Q. Moreover, the h-statistic has the favorabe property that its variance Var [ h p Q N ] = E [ h p Q N µ p Q ] is minima compared to a other unbiased estimators [1]. Based on the sampe Q N of size N, the h-statistic h p Q is commony expressed in terms of power sums S a S a Q N := N i=1 Qω i a. For exampe, the rst three h-statistics are h = NS S 1 N 1N, h 3 = N S 3 3NS S 1 + S 3 1 N N 1N, h 4 = 4N + 8N 1 S 3 S 1 + N 3 N + 3N S 4 + 6NS S NS 3S 4 1 N 3N N 1N where we have used the shorthand notation h p h p Q N and S a S a Q N for brevity see, e.g., [8] for the construction of h p for arbitrary p. In practice, samping the random variabe Q usuay requires the soution of a compex probem e.g. uid ow with random initia/boundary conditions, random dynamica system, etc., which inevitaby invoves a discretization step. That is, it is often not possibe to sampe Q exacty and instead we assume that one can ony draw approximate i.i.d. random variabes Q M ω i, i = 1,..., N, from a random variabe Q M, which is a suitabe approximation in a sense made precise beow of the unknown random variabe Q. In this case the natura Monte Caro MC estimator for the p-th centra moment µ p Q by means of an i.i.d. sampe Q N,M := Q M ω i of the approximate, computabe random variabe Q i=1,...,n M is simpy the h-statistic based on Q N,M : m MC p := h p Q N,M. That is, there are two eves of approximations: the rst one due to approximate samping µ p Q µ p Q M and the second one due to the Monte Caro error µ p Q M m MC p. Consequenty, the mean squared error of this Monte Caro estimator is MSE m MC p := E [ m MC p µ p Q ] = µ p Q M µ p Q + Var [ hp Q N,M ], from which we identify the bias µp Q M µ p Q and the statistica error Var h p Q N,M. Under appropriate assumptions, the statistica error is of order ON 1 as usua. In fact, for the rst three centra moment estimators h 1 0 not incuded, the MC estimator's variance reads Var h = µ 4 N µ N 3 N 1N, 3a Var h 3 = 3µ 3 3N 1N + 0 N N 1N 3µ 4µ N 5 N 1N + µ 6 N µ 3N 10 N 1N, 3b Var 7µ 4 h 4 = N 6N + 1 N 3N N 1N + 16µ 3µ N 4N + 13 N N 1N 4µ 4µ 4N 11 N N 1N + 16µ 6µ N 1N + µ 8 N 8µ 3µ 5 N µ 4N 17 N 1N, 5, 3c

8 where we have suppressed the arguments of h p h p Q N,M and µ p µ p Q M for brevity again. It is noteworthy that these quantities can be computed combinatoria probem straightforwardy for any p using the Mathematica package mathstatica [16], due to the h-statistic's power sum representation. If one assumes that the approximate random variabe Q M is such that the bias term µp Q M µ p Q decays at a certain rate when increasing the discretization parameter M, then it is possibe to baance the squared bias and statistica error contributions to the MSE in. Such a bias assumption is pausibe since the bias term is reated to the numerica method assumed to be consistent used to approximate the underying compex system. At the same time, generating reaizations of Q M typicay becomes more expensive as M increases. The foowing resut thus quanties the computationa cost to estimate the p-th centra moment by the MC method, when using optima discretization parameter M and optima sampe size N to achieve a prescribed accuracy. As a matter of fact, the theoretica resut beow is the centra moment anaog of the standard resut for expectations. Proposition.1. Let p N, p, and assume that the p-th centra moment of Q M is bounded, so that µ p Q M < for M 1. Furthermore, suppose that there exist constants α and γ such that i the bias decays with order α > 0, in the sense that µp Q M µ p Q cα M α for some constant c α > 0, ii the cost to compute each i.i.d. reaization of Q M is bounded by costq M c γ M γ for some constants c γ, γ > 0. The MC estimator m MC p = h p Q N,M with N = Oε and M = Oε 1/α satises MSE m MC p = Oε and the cost associated with computing this estimator is bounded by cost m MC p = N costqm cε γ/α, where c is independent of ε > 0. Note that the constants appearing in the resut above i.e. c, c α, and c γ depend on the order p of the centra moment. In fact, aso the rates α and γ may depend on p in principe. However, numerica evidence suggests that the rates may, in fact, not depend on p for a arge cass of probems; cf. the numerica studies presented in Sect Practica aspect: MSE and unbiased variance estimation A robust impementation of the MC estimator m MC p shoud aso provide an estimation of the associated MSE. This is aso the rst step towards buiding an adaptive MC agorithm in which the sampe size N and/or the discretization parameter M are progressivey increased to achieve a MSE smaer than a prescribed toerance. The bias term µp Q M µ p Q reates ony to the numerica discretization of the underying dierentia probem. Possibe ways of estimating the bias incude: i the cacuation on a sequence of rened discretizations with parameters M 1 < M <... and extrapoation of the error; 6

9 ˆV N ˆV 3 N N N 1 NS 4 N 3S +6 4NS1 4+4NN 3S S1 4N 1 NS 3 S 1 N 3N N 1 N 1 13N N 5N 4N 3N N 1 N 15N + 0S N3N 15N + 0S S1 4 4N N 9N + 11S 3 S NS1 7N 4 36N N 90N + 40S 4 6N4N 1N + 9S 6NS 1 N 3 3N + 6N 8N 1 S 5 + 5N N N 90N + 40S S 3 + N N 1 NN 3 3N + 6N 8S N 4 4N N 90N + 40S 3 3N 5 11N N 3 + 5N 70N + 40S 4 S N 5 + 4N 4 41N N + 100N 80S3 Tabe 1: Cosed-form expressions of the unbiased estimators ˆV p /N for Var [ h p Q N,M ] = V p /N, p =, 3, as poynomia functions of the power sums S a S a Q N,M. ii error estimations based on a-posteriori error estimators see e.g. [1], [1] avaiabe for certain type of equations. We wi not detai further this aspect here, as the main goa of this work is on the estimation of the statistica error. For this, a possiby unbiased estimator for the variance Var [ h p Q N,M ] based on the same sampe Q N,M of size N is needed. We discuss hereafter the derivation of one such estimator. As we have seen in 3, it hods that Var [ h p Q N,M ] = O1/N. It is thus convenient to set V p := N Var [ h p Q N,M ] and derive unbiased estimators ˆV p of V p. However, the naive approach of simpy repacing µ k, for k =,..., p, in 3 by its unbiased estimator h k wi not resut in an unbiased estimator for V p, since the statistica error Var [ h p Q N,M ] depends non-ineary on the centra moments. Instead, we do not ony substitute h k for µ k but aso introduce an additiona mutipicative coecient for each substitution. For exampe, inspecting equation 3a suggests to make the ansatz ˆV = a 1 h 4 + a h for p =. Simiary, 3b impies the ansatz ˆV 3 = a 1 h 3 + a h h 4 + a 3 h 6 + a 4 h 3 for the case p = 3 and so on. For an ansatz of this form the expected vaue of ˆVp, E[ ˆV p ], can be computed as a poynomia function of the centra moments µ k, k =,..., p, using mathstatica. Consequenty, we can derive unbiased estimators by equating the coecients of such poynomia with the corresponding ones in the expression of E[V p ].For exampe, for p = we nd E[ ˆV ] = µ 4a + a 1 N N + a µ N N + 3 N 1N which, after comparing with equation 3a, yieds a 1 = N 1 and a N N+3 = N 3. The N N+3 unbiased variance estimators ˆV p /N of Var [ h p Q N,M ] obtained by foowing this procedure are summarized in Tabe 1, where the na expression has been given directy in terms of the power sums S a S a Q N,M instead of the h-statistics. For the sake of a cear presentation, we present the unbiased estimator for the case p = 4 in AppendixA. It is noteworthy, that athough these formuas are rather engthy, they are in cosed-form, so that they are easiy impementabe and are avaiabe in the accompanying Python ibrary cmmc-py., 7

10 3. Mutieve Monte Caro estimation of centra moments. Using the resuts presented in the previous Section and foowing the genera construction of MLMC estimators, we introduce the MLMC estimator for the p-th centra moment as: p := h p Q 0 N0,M 0 + L =1 h p Q N,M hp Q N,M 1 L =0 h p Q N,M hp Q N,M 1, 4 with the convention that h p Q 0 N0,M 1 0. Here, the sampe Q N,M of i.i.d. reaizations is given by Q N,M := Q M ω i, for any eve. The superscript of both sampes i=1,...,n Q N,M and Q N,M 1 is used to indicate the correation across two consecutive eves, which is the key ingredient for any mutieve Monte Caro method. Specicay, the reaizations of the sampe Q, and those of Q k,, k, are independent, whie the N reaizations of Q N,M and Q N,M 1 are correated, in the sense that the approximate quantities of interest computed on the ner discretization i.e. sampe Q N,M and those computed on the coarser discretization i.e. sampes of Q N,M 1 correspond to the same reaizations of the uncertain inputs. Consequenty, the MLMC estimator's mean squared error is MSE p where we have introduced the shorthand notation = µp Q ML µ p Q L + Var [ ] h p, 5 h p h p Q N,M, Q N,M 1 := hp Q N,M hp Q N,M 1. The bias term µ p Q ML µ p Q in 5 corresponds to the bias of the cassic Monte Caro method described in Sect. on discretization eve L, cf. equation. The anaysis of the variances Var[ h p ] and their dependence on N as we as on the centra moments of Q N,M and Q N,M 1 is more cumbersome than for the cassic MC estimator. In particuar we need to quantify the correation between Q N,M and Q N,M 1. To do so, it is convenient to introduce both the sampe sum and the sampe dierence of these sampes: X,+ N X, N := X,+ i with X i=1,...,n,+ := X, i =0 i := Q M ω i, + Q M 1 ω i,, with X i=1,...,n, i := Q M ω i, Q M 1 ω i,. In other words, we have that X,+ N = Q N,M + Q N,M 1 and X, N = Q N,M Q N,M 1. Moreover, we introduce the bivariate power sums S a,b anaogousy to the power sums S a in the previous Section, that is N S a,b Xi i=1,...,n, Y i i=1,...,n := X a i Y b i, for any two sampes X i i=1,...,n and Y i i=1,...,n of the same size N. Then we can compute the variance Var h p for each eve as foows: i=1 8

11 1. For each, we express the term h p h p Q N,M hp Q N,M 1 in terms of bivariate power series S a,b in X,+ N and X, N, that is in terms of Sa,b S a,b X,+ N, X, N. This can, of course, be achieved by using the identities Q N,M = 1 X,+ N Q N,M 1 = 1 X,+ N X, N and some agebra. + X, N and. The obtained representation of h p in terms of these bivariate power sums in X,+ N and X, N is then amenabe for further treatment by the mathstatica software. In fact, the software provides an ecient agorithm for treating the combinatoria probem of computing the desired variances, due to the power series representation. Foowing this procedure, the rst step yieds for exampe h = N S 1,1 S 0,1S 1,0 N 1N, h 3 = N S 0,3 3N S,1 + 3N S 0,S 0,1 + 3N S,0S 0,1 + 6N S 1,0S 1,1 S 3 0,1 6S 1,0S 0,1 4N N 1N, where we have again omitted the arguments for brevity. The same procedure can aso be used to derive cose-form expressions for h p with p 4, which become rather engthy and are thus not presented here for the sake of a cear presentation. Based on these cosedform expressions for h p, the required expression of the variance Var [ h p ] on eve then foows accordingy as Var [ ] N µ 1,1 h = + µ 0,µ,0 + µ,, 6a N 1N N 1N N Var [ h 3 ] = 3 3N 1N + 0 µ 3 0, 16N N 1N N 4N + 1 µ,0µ 0, 16N N 1N + 9 N 4N + 8 µ 1,1µ 0, 4N N 1N N 4N + 6 µ 1,1µ,0 N N 1N + 9N µ,0 µ 0, + 9µ 4,0µ 0, 9N µ, µ 0, 8N 1N 16N 1N 8N 1N 3N 5µ 0,4 µ 0, 9µ 1, + + µ 0,6 + 3µ,4 16N 1N 4N 1N 16N 8N + 9µ 1,µ 3,0 4N 1N + 9µ 4, 16N 3µ 0,4µ,0 8N 3N 4µ 1,1 µ 1,3 4N 1N 9N µ 1,1 µ 3,1 4N 1N 3N 4µ 0,3 µ,1 8N 1N 9N 3µ,0 µ, 8N 1N 6b N 10µ 0,3 16N 1N 9N 6µ,1 16N 1N, 9

12 where we present Var [ ] h 4 in AppendixA for a cearer presentation. Here, µp,q µ p,q X,+, X, denotes the bivariate centra moment of order p, q of X,+ and X,, where the bivariate centra moment is given by [ X p ] q µ p,q X, Y := E EX Y EY, for any two random variabes X and Y. Inspection of the variance expressions for Var h p in 6 reveas that Var h p = O1/N for any xed. Setting V,p := N Var h p, the mean squared error of the MLMC estimator p can then be written in the somewhat more famiiar form MSE p = µp Q ML µ p Q L V,p +, N =0 which indicates the usua interpay of bias and statistica error. Due to the identities for the variance expressions, the compexity resut for the MLMC estimator for centra moments foows by the same arguments as the ones used in the standard MLMC resut; see, e.g., [10]. In fact, the ony dierence to the standard MLMC compexity resut is that the notion of bias and variance have to be modied. Then even the formuas for the optima number of eves and sampe size on each eve foow immediatey; see Sect. 4 for further detais. Proposition 3.1. Let p N, p, and assume that the p-th centra moment of Q M is bounded, so that µ p Q M <, for 0. Furthermore, suppose that there exist constants α, β, and γ such that α minβ, γ and i the bias decays with order α > 0, in the sense that µp Q M µ p Q cα M α for some constant c α > 0, ii the variance V,p Var [ h p ] N decays with order β > 0, in the sense that V,p c β M β for some constant c β > 0, iii the cost to compute each i.i.d. reaization of Q M is bounded by costq M c γ M γ for some constants c γ, γ > 0. For any 0 < ε < e 1, the MLMC estimator p with maximum eve L N 0 such that µp Q ML µ p Q ε and with sampe size N N on eve given by V,p N = ε costq M L j=0 costq Mj V j,p, 0 L, satises MSE p ε at a computationa cost that is bounded by cost p where c is independent of ε > 0. ε nε 1, if β = γ, c γ β + ε α, if β < γ, ε, if β > γ, 10

13 As remarked after Prop..1 aready, it is aso the case for the MLMC estimator that the appearing constants depend on the order p. It may aso be the case that the rates depend on p, athough numerica experiments suggest that this is not the case for a arge cass of probems; see Sect. 5. Finay, we mention that the proposition above can be stated in terms of the costq M instead of the cost h p due to the avaiabiity of the cosed-form expressions for h p, whose evauation cost is negigibe compared to costq M Practica aspect: MSE and unbiased eve-wise variance estimation As for the cassic Monte Caro method described in Sect., aso a robust impementation of the MLMC estimator shoud provide an estimation of the associated MSE. Moreover, estimations of Var [ ] h p are further needed to determine the optima sampe size N on each eve to achieve a prescribed toerance ε, and we detai hereafter a practica construction of unbiased estimators for V,p. Concerning the bias term, the same considerations made for the cassic MC estimator hod here as we. However, since the MLMC estimator aready uses a sequence of discretizations, the situation is somewhat simpied as a natura way to estimate the bias is µ p Q ML µ p Q L h p Q L N L,M L, Q L N L,M L 1. Next, we discuss how to construct an unbiased estimator of the variance Var [ ] h p on each eve, or equivaenty of V,p Var [ ] h p N, based on the sampes Q N,M and Q N,M 1. Simiary to the derivation of an unbiased variance estimator for the MC method cf. Sect..1, an unbiased estimator of the eve-wise variance V,p is not straightforward to construct. In fact, here the situation is even sighty more compicated due to the highy noninear combination of the bivariate centra moments µ k,, cf. the expressions in 6. However, aso for the bivariate centra moments µ k, there exist unbiased estimators, namey the h k, -statistic [16]. As a consequence, the procedure to construct unbiased variance estimators described in Sect..1 can be foowed for the most parts with ony minor modications. Specicay, to construct unbiased estimators of Var [ ] h p V,p /N, we proceed as foows: 1. We make an initia generic ansatz for the estimator ˆV,p of V,p based upon repacing the centra moments µ k, in 6 by their mutivariate h k, -statistics, so that ˆV,p = i a ih m i p i,q i h n i r i,s i with the same powers m i and n i appearing in 6.. We compute the expectation E[ ˆV,p ] of the considered ansatz expicity as a poynomia function of the centra moments µ k,. Again, this combinatoria manipuation can be carried out ecienty using the mathstatica software. 3. We assembe a inear system of equations for the unknown coecients a i i in the considered ansatz by equating the coecients in 6 with those of E[ ˆV,p ]/N, obtained by ordering with respect to the centra moments µ k,. 4. If the inear system is not uniquey sovabe, then we augment the ansatz for the estimator to account for the newy introduced centra moment terms by computing E[ ˆV,p ] and repeat steps 4. Obviousy, it is aso possibe to directy consider an ansatz that contains a unique combinations of µ k 1 p 1,q 1 µ k p,q, such that k 1 p 1 + q 1 + k p + q = p. However, the procedure 11

14 described above oers the advantage that it may resut in a ower dimensiona inear system, which needs to be soved. We detai here the procedure for p =. In view of 6a we rst make the initia ansatz ˆV, = a 1 h 1,1 + a h 0, h,0 + a 3 h,. Next, we compute the expectation of this ansatz, which can be written as E[ ˆV, ] = a + N a 3 N 1N µ 1,1 + a N 1 + a 3 N 1N µ 0, µ,0 + a 1N + a + a 3 N µ,. By equating the coecients of the right-hand side above and those in 6a we then obtain a inear system of equations for the coecients a 1, a, and a 3. Finay, soving this inear N system yieds a 1 = 1, a N N N +3 = 1, and a N 3 4N +7N 6 3 = N +4N 5 N 3 4N +7N. Using these 6 coecients, we can eventuay express the unbiased sampe-based estimator of Var [ ] h = V, /N as a poynomia function of the bivariate power sums as ˆV, N = 1 N N 3 N N 1 N + N + S 1,1 N + N 1 N S, S1,0S 1, + N 1 S0,S 1,0 S,0 + N 1 N N S,0 N S0,1 1 S, N S 1,0 S1,1, 6 + S0,1 4N S 1,0 7 where S a,b S a,b X,+ N, X, N for brevity. The same procedure can aso be appied to obtain unbiased estimators for higher order i.e. for any p centra moments, which become rather engthy though. However, we emphasize that the obtained unbiased variance estimators are in cosed-form, so that an ecient impementation is possibe. For exampe, in AppendixB we present the unbiased estimator for the case p = 3, whie we refer to our impementation detais avaiabe in the Python ibrary cmmc-py for the formua for p = 4. We reiterate that the procedure introduced here yieds unbiased sampe-based variance estimators, which are needed for the practica error contro and tuning of the MLMC approach introduced in this work; see Section 4 for detais. The fact that these variance estimators are unbiased and not just asymptoticay unbiased is particuary important on ner eves, on which the sampe size N wi be sma. For exampe, for p = the bias of the naive variance estimator, which is obtained by simpy repacing the bivariate centra moments µ k, by the corresponding h k, -statistics, is N 4N +6µ 1,1 +3 N µ 0, µ,0 N 4N +3µ, N 1 N. Athough this additiona bias as a function of the sampe size N is of order ON, it may sti contribute to a non-negigibe error of the MLMC estimator, in particuar due to ne eves for which N wi be sma. Finay, we aso emphasize that, as a consequence of being based on unbiased estimators, the MLMC method for centra moments introduced in this work does not come at the expense of introducing an additiona systematic error i.e. a bias that needs to be accounted for, unike other works on centra moment estimators, such as [3]. 3.. From mean squared errors to condence intervas The discussion of both the MC method and the MLMC method above was soey based on the mean squared error as an accuracy measure. However, for some appications it is often 1

15 aso desirabe to associate condence intervas or, equivaenty, faiure probabiities to an estimator. Specicay, et ˆθ be a generic estimator of the deterministic vaue θ with mean squared error given by MSEˆθ = E [ ˆθ θ ]. For a condence pc 0, 1, the associated condence interva can then be characterized by the vaue δ > 0, such that P ˆθ θ < δ p c. In the absence of any further knowedge of the probabiity distribution of ˆθ θ, a sucient condition for the ength δ of the condence interva can be derived using Chebyshev's inequaity: P ˆθ θ δ MSEˆθ MSEˆθ = 1 p δ c δ =. 8 1 p c That is, the condence interva can be directy inked to the estimator's mean squared error. Consequenty, the mean squared error based anaysis considered in this work can straightforwardy be used to quantify condence regions or faiure probabiities of estimators. It is noteworthy however, that the condence region identied in 8 may be rather conservative due to the use of Chebyshev's inequaity. We wi revisit this fact in the forthcoming Part II of this work, where we wi introduce methodoogies that aow for sharper approximations of condence regions. 4. Impementation detais and compete agorithm In this Section, we address important practica aspects needed for the impementation of the MLMC methodoogy presented in this work and eventuay oer pseudo-code of the compete MLMC agorithm. In fact, here we present a unied framework for the estimation of both the expectation E[Q] and any order centra moment µ p Q of a random variabe Q subject to prescribed mean squared error toerance. As the centra moment µ p is triviay zero for p = 1, it wi be convenient to denote by 1 the MLMC estimator for E[Q]. Specicay, equation 4 denes the MLMC centra moment estimator for any non-trivia order p > 1. For p = 1 we sti use the denition in equation 4 but with a sight abuse of notation by setting h 1 Q N,M := 1 N N i=1 Q Mω i to denote the sampe average operator, so that equation 4 yieds the usua MLMC estimator of the expected vaue for p = 1. In the absence of theoretica estimates for the rates and constants that characterize the bias and statistica error decays as we as the cost mode for the probem under investigation cf. Prop. 3.1, these rates and constants need to be estimated as they are required to optimay tune the MLMC method. That is, to be abe to compute the optima number of eves and sampe sizes, a common practice is to perform an initia screening procedure. Such a screening procedure consists, for exampe, of the evauation of a predened number of N reaizations on few coarse eves {0,..., L}. Based on these simuations, it is possibe to t these rates and constants e.g. via a east squares procedure, which then determine the modes for the bias, statistica error, and cost per sampe. Once the rates and constants are determined, the pivota step for achieving the theoretica compexity of the MLMC method subject to a prescribed mean squared error toerance, is 13

16 the choice of both the number of eves L and the sampe size N required on each eve 0 L. To determine these parameters a precise estimation of the mean squared error MSE, MSE = B + SE, specicay of its two error contributions bias B and statistica error SE, is required as described in Sect. 3. In order to present a genera procedure for the unied MLMC approach to both the expectation and centra moments, we reca that h p = h p Q N,M hp Q N,M 1, hp Q N,M = { 1 N N i=1 Q Mω i, if p = 1, p-th h-statistic, if p > 1. 9 In practice the bias contribution B is thus estimated by On the other hand, the statistica error SE is approximated by B L h p. 10 SE L =0 V,p N, 11 where V,p denotes the estimated variance Var [ h p ] on eve. Specicay, we use V,p = ˆV,p on those eves for which simuations have been run during the screening procedure i.e. L. Here, ˆV,p is the unbiased sampe-based variance estimator introduced in Sect On eves for which no sampe exists yet i.e. for > L, we extrapoate the tted mode and use V,p = c β M β as an estimator. To achieve a prescribed mean squared error of ε, we thus require B 1 θε, SE θε, 1a 1b where we have additionay introduced a spitting parameter θ 0, 1 to oer the possibiity of weighting the two MSE contributions dierenty. Specicay, the bias constraint 1a is satised for L N such that 1 θε M L c α 1 α, 13 in view of Prop. 3.1i. Moreover, the theoretica compexity resut in Prop. 3.1 aso suggests that the statistica error constraint 1b is satised with optima compexity by seecting the sampe size N N on eve as N = 1 θε V,p C L Ck V k,p, = 0, 1,..., L, 14 k=0 where C = costq M. In Agorithm 1 we provide a detaied pseudo-code of the fu MLMC agorithm, which is based on the discussion above. There SOLVE denotes a back-box sover that, for a 14

17 Agorithm 1: MLMC Agorithm for the expectation and centra moments of order p. SCREENINGN, L, p, ε r, θ for = 0 : L do for i = 0 : N do Generate random sampe: ω i, Q M ω i, SOLVE ω i, Q M 1 ω i, SOLVE 1 ω i, h p = h p Q N,M hp Q N,M, where hp Q N,M as in 9 1 estimate ε = ε r p [Q] estimate {C } L =0, {V,p} L =0 compute P = {c α, c β, c γ, α, β, γ} using LS t extrapoate {C } >L, {V,p } >L compute L using 13 and N using 14 return L, {N } L =0 MLMCL, {N } L =0, p for = 0 : L do for i = 0 : N do Generate random sampe: ω i, Q M ω i, SOLVE ω i, Q M 1 ω i, SOLVE 1 ω i, compute V,p compute L h p as in 9, return p [Q], MSE = B + SE, given reaization ω i of the random parameters, returns the approximation Q M ω i on the discretization eve. For the sake of competeness, the pseudo-code aso contains a possibe screening procedure. Eventuay, Agorithm 1 returns the MLMC estimator p [Q] of the QoI for a prescribed mean squared error toerance. We emphasize that the impementation presented here takes as an input a reative MSE toerance ε r, which is reated to the commony used absoute MSE toerance ε via { E[Q], if p = 1, ε = ε r µ p Q, if p > 1. The unied MLMC framework described in this Section, and much more, is avaiabe via our Python ibrary cmmc-py. 5. Numerica Experiments In this section we appy the introduced mutieve Monte Caro technique to various exampes. In fact, we have used Agorithm 1 with θ = 1/ in a simuations that foow. We begin by scrutinizing the methodoogy for rather simpe toy probems for which exact or 15

18 centra moment µ p Q EQ p = p = 3 p = Tabe : Reference vaues for the expected vaue E[Q] and various centra moments µ p Q for the QoI Q derived form the geometric Brownian motion SDE. highy accurate soutions are easiy avaiabe; see Sects. 5.1 and 5.. Then we move on to study a more chaenging probem in Sect. 5.3, namey the one of a transonic airfoi under operationa and/or geometric uncertainties Stochastic dierentia equation mode: a nancia option Let us begin with a simpe exampe invoving a stochastic dierentia equation SDE. Specicay, we consider the case that the SDE modes exampe borrowed from [10, Sect. 5] a nancia ca option with the asset being a geometric Brownian motion, viz. ds = rs dt + σs dw, S0 = S Here, r, σ, and S 0 are given positive numbers. For this asset we are interested in quantifying the uncertainties in the discounted payo, so that we set the quantity of interest Q as Q := e rt max ST K, 0, where K > 0 denotes the agreed strike price and T > 0 the pre-dened expiration date. Due to the fact that the soution to 15 at time T, i.e. ST, is a og-normay distributed random variabe with mean S 0 e rt and variance S 0 e rt e σt 1, it is straightforward to compute highy accurate approximations to statistics of Q. In fact, Tabe ists approximated reference vaues for the expected vaue and for the rst three centra moments of Q corresponding to the parameter vaues r = 1, σ = 1, T = 1, K = 10, and S = 10. These reference vaues were obtained using a high precision numerica quadrature. For the numerica experiments based on the mutieve Monte Caro method that wi foow, we discretize the SDE 15 via the Mistein method. S n+1 = S n + δ rs n + σs n δ ξ n + σ δ S n ξn 1, S 0 = S 0, so that S n+1 Snδ, where δ = T and ξ n n 0 denotes a sequence of i.i.d. standard normay distributed random variabes. That is, we empoy a discretization with a nested grid hierarchy with M = T/δ = DOFs, which corresponds to the number of time steps needed to integrate the SDE from time t = 0 to the na time T. In order to vaidate the MLMC methodoogy discussed in this work, we provide in Tabe 3 a sampe based estimation of the MSE of the MLMC estimators, using 100 independent repetitions of the agorithm. Specicay, the tabe oers a comparison between the required reative root MSE toerance and the sampe based root mean squared error achieved by Agorithm 1 for both the expected vaue and the rst three centra moments for various 16

19 To ε r = ε r = ε r = ε r = Tabe 3: Sampe estimate of reative root MSE based on 100 repetitions of the MLMC agorithm for computing p for dierent reative toerance requirements. toerances. The resuts in Tabe 3 demonstrate that the MLMC impementation described in the previous Section does indeed provide estimators that satisfy the toerance requirement. Additionay, in the top row of Figure 1 we show the actua computed vaues of these 100 repetitions of the MLMC agorithm red circes compared with the reference soution green stars. To quantify the range of the MLMC estimators, we aso indicate the 90% condence intervas based on a Chebyshev bound bue bars; see Eq. 8 in these pots. The bottom row in Figure 1 presents the corresponding MLMC hierarchies both number of eves and sampe size per eve required to achieve prescribed reative toerance requirements when estimating the expectation and various centra moments of the QoI Q. It is interesting to observe that the computationa cost required to compute centra moments is proportiona to that for the expectation up to a mutipicative constant. The atter can be further observed in Figure, where we pot the computed bias and variance of the estimators, respectivey, for various toerance demands. It can be inferred that the decay rate for the estimator's bias and variances is the same, whie the constants are increasing with increasing p. 5.. Eiptic PDE in two spatia dimensions We consider a random Poisson equation in two spatia dimensions, u = f, in D = 0, 1, 16 with homogeneous Dirichet boundary conditions. Here, the forcing term f is given by fx = Kξx 1 + x x 1 x, with ξ being a non-negative random variabe and K > 0 a positive constant. For this forcing term the soution to the PDE can be computed expicity and reads ux 1, x = Kξx 1 x 1 x 1 1 x /. As quantity of interest we consider the spatia average of the soution, that is Q := u dx = K 7 ξ. D This expicit representation of Q in terms of the random input ξ to the PDE mode 16 aows us to easiy compute the exact mean as we as centra moments of Q, which we wi use to verify the numerica experiments that foow. Specicay, here we use ξ Beta, 6 and K = 43. Tabe 4 then ists approximations to the corresponding mean and the rst three centra moments of Q. For the numerica experiments based on mutieve Monte Caro method we discretize the PDE 16 using a second order nite dierence scheme on a reguar 17

20 µ[q] 1 [Q] Cheb-bound σ [Q] [Q] Cheb-bound µ3[q] 3 [Q] Cheb-bound µ4[q] 4 [Q] Cheb-bound ε r ε r ε r ε r εr =0.1 εr =0.05 εr =0.05 εr = εr = εr =0.05 εr = εr = εr = εr =0.05 εr = εr = εr = εr =0.05 εr = εr =0.01 Neve eve eve eve eve Figure 1: Computed vaues of 100 repetitions of the MLMC agorithm compared with the reference soution rst row and MLMC hierarchies number of eves and sampe size per eve required to achieve prescribed reative toerance requirements when estimating the expectation and various centra moments of the QoI Q second row for the SDE probem B L h p L h 1, c α = L h, c α = 0.0 L h 3, c α = 1.01 L h 4, c α = 7.54 rate α = V,p V,1, cβ = V,, c β = 0.6 V,3, c β = V,4, c β = rate β = Lhp 10 V,p eve eve Figure : Decay rates for the bias and variances of MLMC estimator with increasing p for the SDE probem. grid. That is, we empoy a nested grid hierarchy with M = 5 DOFs, which correspond to the vaues of the soution u at grid points that are not on the boundary D. 18

21 centra moment µ p Q EQ p = p = 3 p = Tabe 4: Reference vaues for the expected vaue E[Q] and the rst three centra moments µ p Q for the QoI Q derived form the random Poisson probem. As for the previous exampe, in Tabe 5 we present the sampe based root mean squared errors obtained by repeating the MLMC agorithm for the expectation and centra moments 100 times and for various toerances. Aso for this exampe we nd that the MLMC impe- To ε r = ε r = ε r = ε r = Tabe 5: Sampe estimate of reative root MSE of 100 repetitions of the MLMC estimators p reative toerance requirements. for dierent mentation does indeed satisfy the required toerance goas. In the top row of Figure 3 the actua computed vaues for 100 repetitions of the MLMC agorithm red circes are compared with the reference vaues green stars. Aso for this exampe we observe an accurate estimation within the imposed toerance goa and within the condence region bue bars, 90% condence; see Eq. 8. In the second row of Figure 3 we report the hierarchies required to achieve the prescribed toerances. As it is possibe to observe the number of eves and sampes per eve and hence the cost increase consistenty with the centra moment we are computing. Such can be aso inferred by ooking at the decays of the bias and variance of the MLMC estimators for moments presented in Figure 4. These pots moreover conrm the observation from the previous exampe, namey that the decay rates for the estimator's bias and variance are the same for dierent vaues of p, and ony the constants vary Transonic Airfoi: d We consider hereafter a transonic supercritica RAE-8 airfoi [11, 0], which has become a standard test-case for transonic ows, subject to both operating and geometric uncertainties. The uid owing around an airfoi generates a oca force on each point of the body. The norma and tangentia components of such force are the pressure and the shear stress. By integrating the force and stress distribution around the surface of the airfoi we obtain a tota force F and a moment M about a reference point so caed center of pressure. The parae and perpendicuar component of F with respect to the free-stream direction M are the ift L and drag D forces respectivey. Figure 5a shows a sketch of this concept. For an airfoi shape with surface S we dene the foowing ift, drag, and moment dimensioness 19

22 µ[q] 1 [Q] Cheb-bound σ [Q] [Q] Cheb-bound µ3[q] 3 [Q] Cheb-bound 3.4. µ4[q] 4 [Q] Cheb-bound ε r ε r ε r ε r 10 7 εr = εr = εr = εr = εr =0.05 εr = εr =0.05 εr = εr =0.05 εr = εr =0.05 εr =0.05 εr =0.01 εr =0.01 εr =0.01 εr = Neve eve eve eve eve Figure 3: Computed vaues of 100 repetitions of the MLMC agorithm compared with the reference soution rst row and MLMC hierarchies number of eves and sampe size per eve required to achieve prescribed reative toerance requirements when estimating the expectation and various centra moments of the QoI Q second row for the Eiptic PDE probem. 10 B L h p L h 1, c α = V,p V,1, cβ = L h, c α = 8.8 L h 3, c α = 5.94 L h 4, c α = 36.8 rate α = V,, c β = 15.8 V,3, c β = 749. V,4, c β = rate β = Lhp 10 0 V,p eve eve Figure 4: Decay rates for the bias and variances of MLMC estimator with increasing p for the Eiptic PDE probem. coecients: C L = L q S, C D = D q S, and C M = M q SL ref, 17 0

23 respectivey. Here, q = 1M γ g p denotes the dynamic pressure and γ g = 1.4 is the ratio of specic heats of the gas. As we are considering D normaized airfois we set the reference ength L ref = 1 and the reference surface S = 1. α M L D F M α M R s R p y y p RAE8 x p C p C s y s x s θ s θ p x a Aerodynamic forces and moments acting on the airfoi. b Geometry of the RAE-8 transonic airfoi and PARSEC parameters that dene the geometry of the airfoi. Figure 5: Description of the airfoi set-up considered in this work. The nomina geometry of the RAE-8 airfoi is dened by a set of PARSEC parameters see [18] for detais. The advantage of the PARSEC approach over other parametrizations i.e. Bezier, NURBS, FFD is that we can easiy perturb the geometrica parameters on the suction and pressure side of the airfoi see Figure 5b which are most reevant for the study that foows. Among other things, Tabe 6 summarizes the geometric denition of the airfoi as we as the set of operating parameters for three dierent ow conditions considered here. Specicay, CASE-6 denotes the mid transonic case corresponding to experimenta case 6 from AGARD [0], CASE-S is a subsonic case with M = 0.6, and CASE-R is a higher Reynods number case. Operating Geometric Name Nomina vaue Uncertainty CASE-6 CASE-S CASE-R Re c 6.5e6 6.5e6 10e6 M B4,, 0.05, M α B4,, 0.,.16 R p e 03 U98%, 10% R s e 03 U98%, 10% x p e 01 U98%, 10% x s e 01 U98%, 10% y p e 0 U98%, 10% y s e 0 U98%, 10% C p e 01 U98%, 10% C s e 01 U98%, 10% θ p e + 00 U98%, 10% θ s e + 01 U98%, 10% Tabe 6: Operationa and geometrica parameters as we as a description of the uncertainties for the RAE- 8 airfoi. In what foows, we consider the RAE-8 airfoi in three dierent operating regimes with increasing number of uncertain parameters. Specicay, we use the etter G to denote 1