Sobol-Hoeffding Decomposition with Application to Global Sensitivity Analysis

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Sobol-Hoeffding Decomposition with Application to Global Sensitivity Analysis Olivier Le Maître with Colleague & Friend Omar Knio LIMSI, CNRS UPR-325, Orsay, France https://persolimsifr/olm/ PhD course on UQ - DTU

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Table of content Sobol-Hoeffding decomposition Notations S-H decomposition 2 Application to Global SA Analysis of the variance (ANOVA) Sensitivity indices Example 3 Computation of the SI Monte-Carlo estimation SI from PC expansion

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Notations L 2 functions over unit-hypercubes Let L 2 (U d ) be the space of real-valued squared-integrable functions over the d-dimensional hypercube U: ˆ f : x U d f (x) R, f L 2 (U d ) f (x) 2 dx < U d L 2 (U d ) is equipped with the inner product,, f, g L 2 (U d ), u, v := ˆ f (x)g(x)dx, U d and norm 2, f L 2 (U d ), f 2 := f, f /2

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Notations L 2 functions over unit-hypercubes Let L 2 (U d ) be the space of real-valued squared-integrable functions over the d-dimensional hypercube U: ˆ f : x U d f (x) R, f L 2 (U d ) f (x) 2 dx < U d NB: all subsequent developments immediately extend to product-type situations, where x A = A A d R d, and weighted spaces L 2 (A, ρ), ρ : x A ρ(x) 0, ρ(x) = ρ (x ) ρ d (x d ) (eg: ρ is a pdf of a random vector x with mutually independent components)

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Notations Ensemble notations Let D = {, 2,, d} Given i D, we denote i := D \ i its complement set in D, such that i i = D, i i = Given x = (x,, x d ), we denote x i the vector having for components the x i i, that is D i = {i,, i i } x i = (x i,, x i i ), where i := Card(i) For instance ˆ ˆ f (x)dx i = f (x,, x d ) dx i, U i U i i i and ˆ ˆ i / i f (x)dx i = f (x,, x d ) dx i, U d i U d i i D

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI S-H decomposition Sobol-Hoeffding decomposition Any f L 2 (U d ) has a unique hierarchical orthogonal decomposition of the form d d f (x) = f (x,, x d ) =f 0 + f i (x i ) + d i= d d i= j=i+ k=j+ i= j=i+ d f i,j (x i, x j )+ f i,j,k (x i, x j, x k ) + + f,,d (x,, x d ) Hierarchical: st order functionals (f i ) 2nd order functionals (f i,j ) 3rd order functionals (f i,j,l ) d-th order functional (f,,d ) Decomposition in a sum of 2 k functionals Using ensemble notations: f (x) = f i (x i ) i D

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI S-H decomposition Sobol-Hoeffding decomposition Any f L 2 (U d ) has a unique hierarchical orthogonal decomposition of the form f (x) = f i (x i ) i D Orthogonal: the functionals if the S-H decomposition verify the following orthogonality relations: ˆ f i (x i )dx j = 0, i D, j i, U ˆ f i (x i )f j (x j )dx = f i, f j = 0, i, j D, i j U d It follows the hierarchical construction ˆ f = f (x)dx = f =D U ˆ d f {i} = f (x)dx {i} f = f D\{i} f i D U ˆ d f i = f (x)dx i f j = f i f j i D, i 2 U i j i j i

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI S-H decomposition Table of content Sobol-Hoeffding decomposition Notations S-H decomposition 2 Application to Global SA Analysis of the variance (ANOVA) Sensitivity indices Example 3 Computation of the SI Monte-Carlo estimation SI from PC expansion

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Analysis of the variance (ANOVA) Parametric sensitivity analysis Consider x as a set of d independent random parameters uniformly distributed on U d, and f (x) a model-output depending on these random parameters It is assumes that f is a 2nd order random variable: f L 2 (U d ) Thus, f has a unique S-H decomposition f (x) = f i (x i ) i D Further, the integrals of f with respect to i are in this context conditional expectations, ˆ E [f x i ] = f (x)dx i = g(x i ) U i i D, so the S-H decomposition follows the hierarchical structure f = E [f ] f {i} = E [ ] f x {i} E [f ] i D f i = E [f x i ] f j j i i D, i 2

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Analysis of the variance (ANOVA) Variance decomposition Because of the orthogonality of the S-H decomposition the variance V [f ] of the model-output can be decomposed as i V [f ] = V [f i ], V [f i ] = f i, f i i D V [f i ] is interpreted as the contribution to the total variance V [f ] of the interaction between parameters x i i The S-H decomposition thus provide a rich mean of analyzing the respective contributions of individual or sets of parameters to model-output variability However, as there are 2 d contributions, so one needs more "abstract" characterizations

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Analysis of the variance (ANOVA) Sensitivity indices To facilitate the hierarchization of the respective influence of each parameter x i, the partial variances V [f i ] are normalized by V [f ] to obtain the sensitivity indices: S i (f ) = V [f i] V [f ], i S i (f ) = i D The order of the sensitivity indices S i is equal to i = Card(i) st order sensitivity indices The d first order indices S {i} D characterize the fraction of the variance due the parameter x i only, ie without any interaction with others Therefore, d S {i} (f ) 0, i= measures globally the effect on the variability of all interactions between parameters If d i= S {i} =, the model is said additive, because its S-H decomposition is d f (x i,, x d ) = f 0 + f i (x i ), i= and the impact of the parameters can be studied separately

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Sensitivity indices Sensitivity indices To facilitate the hierarchization of the respective influence of each parameter x i, the partial variances V [f i ] are normalized by V [f ] to obtain the sensitivity indices: S i (f ) = V [f i] V [f ], i S i (f ) = i D The order of the sensitivity indices S i is equal to i = Card(i) Total sensitivity indices The first order SI S {i} measures the variability due to parameter x i alone The total SI T {i} measures the variability due to the parameter x i, including all its interactions with other parameters: T {i} := i i S i S {i} Important point: for x i to be deemed non-important or non-influent on the model-output, S {i} and T {i} have to be negligible Observe that i D T {i}, the excess from characterizes the presence of interactions in the model-output

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Sensitivity indices

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Sensitivity indices Sensitivity indices In many uncertainty problem, the set of uncertain parameters can be naturally grouped into subsets depending on the process each parameter accounts for For instance, boundary conditions BC, material property ϕ, external forcing F, and D is the union of these distinct subsets: D = D BC D ϕ D F The notion of first order and total sensitivity indices can be extended to characterize the influence of the subsets of parameters For instance, S Dϕ = i D ϕ S i, measures the fraction of variance induced by the material uncertainty alone, while T DF = S i i D F measures the fraction of variance due to the external forcing uncertainty and all its interactions

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Example Let (ξ, ξ 2 ) be two independent centered, normalized random variables ξ i N(0, ), i =, 2 Consider the model-output f : (ξ, ξ 2 ) R 2 R given by f (ξ, ξ 2 ) = (µ + σ ξ ) + (µ 2 + σ 2 ξ 2 ) Determine the S-H decomposition of f 2 Compute the st order and total sensitivity indices of f 3 Comment 4 Repeat for f (ξ, ξ 2 ) = (µ + σ ξ ) (µ 2 + σ 2 ξ 2 )

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI E [f ξ ] = (µ + µ 2 ) + σ ξ f (ξ ) = E [f ξ ] E [f ] = σ ξ E [f ξ 2 ] = (µ + µ 2 ) + σ 2 ξ 2 f 2 (ξ 2 ) = E [f ξ 2 ] E [f ] = σ 2 ξ 2 Example f (ξ, ξ 2 ) = µ + µ 2 + σ ξ + σ 2 ξ 2 E [f ] = (µ + µ 2 ) E [f ξ, ξ 2 ] = f (ξ, ξ 2 ) f,2 (ξ, ξ ) = E [f ξ, ξ 2 ] E [f ] f (ξ ) f 2 (ξ 2 ) = 0 Then, V [f ] = σ 2 + σ2 2, so σ 2 σ2 2 S = T = σ 2 + and S 2 = T 2 = σ2 2 σ 2 + σ2 2 Comment: obvious case, as f is a linear (additive) model

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI E [f ξ ] = µ µ 2 + µ 2 σ ξ f (ξ ) = E [f ξ ] E [f ] = µ 2 σ ξ E [f ξ 2 ] = µ µ 2 + µ σ 2 ξ 2 f 2 (ξ 2 ) = E [f ξ 2 ] E [f ] = µ σ 2 ξ 2 Example E [f ] = µ µ 2 f (ξ, ξ 2 ) = µ µ 2 + µ 2 σ ξ + µ σ 2 ξ 2 + σ σ 2 ξ ξ 2 E [f ξ, ξ 2 ] = f (ξ, ξ 2 ) f,2 (ξ, ξ ) = E [f ξ, ξ 2 ] E [f ] f (ξ ) f 2 (ξ 2 ) = σ σ 2 ξ ξ 2 Then, V [f ] = µ 2 σ2 2 + µ2 2 σ2 + σ2 σ2 2, so µ 2 2 S = σ2 µ 2 σ2 2 + µ2 2 σ2 + σ2 σ2 2 µ 2 and S 2 = σ2 2 µ 2 σ2 2 + µ2 2 σ2 +, σ2 σ2 2 µ 2 2 T = σ2 + σ σ 2 µ 2 µ 2 σ2 2 + µ2 2 σ2 + and T 2 = σ2 2 + σ σ 2 σ2 σ2 2 µ 2 σ2 2 + µ2 2 σ2 +, σ2 σ2 2 Comment: fraction of variance due to interactions is σ 2 σ2 2 /(µ2 σ2 2 + µ2 2 σ2 + σ2 σ2 2 )

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Example f (ξ, ξ 2 ) = µ µ 2 + µ 2 σ ξ + µ σ 2 ξ 2 + σ σ 2 ξ ξ 2 Then, V [f ] = µ 2 σ2 2 + µ2 2 σ2 + σ2 σ2 2, so µ 2 2 S = σ2 µ 2 σ2 2 + µ2 2 σ2 + σ2 σ2 2 µ 2 and S 2 = σ2 2 µ 2 σ2 2 + µ2 2 σ2 +, σ2 σ2 2 µ 2 2 T = σ2 + σ σ 2 µ 2 µ 2 σ2 2 + µ2 2 σ2 + and T 2 = σ2 2 + σ σ 2 σ2 σ2 2 µ 2 σ2 2 + µ2 2 σ2 +, σ2 σ2 2 Comment: fraction of variance due to interactions is σ 2 σ2 2 /(µ2 σ2 2 + µ2 2 σ2 + σ2 σ2 2 ) Example: (µ, σ ) = (, 3) and (µ 2, σ 2 ) = (2, 2), so S = 9/9, S 2 = /9, S,2 = 9/9 One can draw the conclusions: ξ is the most influential variable as S > S 2 and T > T 2 interactions are important as S S 2 = 9/9 05, especially for ξ 2 for which (T 2 S 2 )/T 2 = 9/0

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Example Table of content Sobol-Hoeffding decomposition Notations S-H decomposition 2 Application to Global SA Analysis of the variance (ANOVA) Sensitivity indices Example 3 Computation of the SI Monte-Carlo estimation SI from PC expansion

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation -st order sensitivity indices by Monte-Carlo sampling The S i can be computed by MC sampling as follow Recall that S {i} (f ) = V [ f {i} ] V [f ] Observe: E [E [f x i ]] = E [f ] = V [E [f x i ])] V [f ] E = [ E [f x i ] 2] E [E [f x i ]] 2 E [f ] and V [f ] can be estimated using MC sampling Let χ M = {x (),, x (M) } be a set of independent samples drawn uniformly in U d, the mean and variance estimators are: V [f ] Ê [f ] = M M l= f (x (l) ), V [f ] = M M f (x (l) ) 2 Ê [f ]2 l= It now remains to compute the variance of conditional expectations, V [ E [ f x {i} ]] Any idea?

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation Monte-Carlo estimator for variance of conditional expectation Observe: ˆ f (x i, x i )f (x i, x U d+ i i )dx idx i dx i ˆ ˆ ˆ = dx U i i f (x i, x i )dx i f (x i, x U i U i i )dx i ˆ [ˆ ] 2 = dx U i i f (x i, x i )dx i U i V [ E [ ]] [ f x {i} = E E [ ] ] 2 f x {i} E [ E [ ]] 2 f x {i} = E [E [ ] ] 2 f x {i} E [f ] 2

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation Monte-Carlo estimators for st order SI S {i} Draw 2 independent sample sets, with size M, χ M and χ M 2 Compute estimators Ê [f ] and V [f ] from χ M (or χ M ) [M model evaluations] 3 For i =, 2,, d: Estimate variance of conditional expectation through V [ E [ ]] f x {i} = M Estimator of the -st order SI: M l= ( ) ( ) f x (l) {i}, x (l) f x (l) {i} {i}, x (l) Ê [f ]2 {i} V [ E [ ]] f ξ {i} Ŝ {i} (f ) = V [f ] [M new model evaluations at ( x (l) {i}, x (l) {i} )] Requires (d + ) M model evaluations

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation χ M : x () x () i x () d x (l) x (l) i x (l) d x (M) x (M) i x (M) d χ M : x () x () i x () d x (l) x (l) i x (l) d x (M) x (M) i x (M) d χ {i} M : x () x () i x () d x (l) x (l) i x (l) d x (M) x (M) i x (M) d V [ E [ f x {i} ]] M M l= f (x {i}, x {i} )f ( x {i}, x {i} ) ( M M l= f (x {i}, x {i} )) 2

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation Total sensitivity indices by Monte-Carlo sampling The T {i} can be computed by MC sampling as follow Recall that T {i} (f ) = S i (f ) = i {i} i D\{i} S i (f ) = V [ E [ f x {i} ]] V [f ] As for V [ E [ f x {i} ]], we can derive the following Monte-Carlo estimator for V [ E [ f x {i} ]], using the two independent sample sets χm and χ M V [ E [ ]] f x {i} = M M f l= ( ) ( ) x (l), x (l) f x (l), x (l) E [f ] 2, {i} {i} {i} {i} so finally ( T {i} (f ) = V [f ] M M f l= ) ( ) ( ) x (l), x (l) f x (l), x (l) E [f ] 2 {i} {i} {i} {i} The MC estimation of the T {i} needs d M additional model evaluations

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI Monte-Carlo estimation MC estimation of st order and total sensitivity indices: requires M (2d + ) simulations convergence of estimators in O(/ M) slow convergence, but d-independent convergence not related to smoothness of f works also (in practice) with advanced sampling schemes (QMC, LHS, ) to proceed with QMC/LHS? How

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition from PC expansions Consider the model-output f : ξ Ξ R d R, where ξ = (ξ,, ξ d ) are independent real-valued rv with joint-probability density function p ξ (x,, x d ) = d p i (x i ) Let { Ψα} be the set of d-variate orthogonal polynomials, Ψα(ξ) = d i= i= ψ (i) α i (ξ i ), with ψ (i) l 0 π l the uni-variate polynomials mutually orthogonal with respect to the density p i If f L 2 (Ξ, p ξ ), it has a convergent PC expansion d f (ξ) = lim Ψα(ξ)fα, α = α i No α No i=

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition from PC expansions Given the truncated PC expansion of f, ˆf (ξ) = Ψα(ξ)fα, α A one can readily obtain the PC approximation of the S-H functionals through ˆfi (ξ i ) = Ψα(ξ i )fα, where the multi-index set A(i) is given by α A(i) A(i) = {α A; α i > 0 for i i, α i = 0 for i / i} A For the sensitivity indices it comes α A(i) S i (ˆf ) = f α 2 Ψα, Ψα α A f α 2 Ψα, Ψα, T α T (i) {i}(ˆf ) = f α 2 Ψα, Ψα α A f α 2, Ψα, Ψα where T (i) = {α A; α i > 0 for i i}

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Subsets of PC multi-indices for S-H functionals

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Example: Let (ξ, ξ 2 ) be two independent random variables with uniform distributions on the unit interval Consider the model-output f : (ξ, ξ 2 ) [0, ] 2 R given by f (ξ, ξ 2 ) = tanh(3(ξ 02ξ 2 5)( + ξ 2 )) f(ξ, ξ 2 ) 05 0-05 - 08 ξ 2 06 04 02 02 04 06 ξ 08

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition at No = 3 (P = 0): 5 05 0-05 - -5 5 05 0-05 - -5 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08 02 05 0 005-005 0-0 -05-02 04 02 0-02 -04 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition at No = 5: (P = 2) 5 05 0-05 - -5 5 05 0-05 - -5 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08 02 05 0 005-005 0-0 -05-02 04 02 0-02 -04 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition at No = 7: (P = 36) 5 05 0-05 - -5 5 05 0-05 - -5 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08 02 05 0 005-005 0-0 -05-02 04 02 0-02 -04 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion S-H decomposition at No = 9: (P = 55) 5 05 0-05 - -5 5 05 0-05 - -5 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08 02 05 0 005-005 0-0 -05-02 04 02 0-02 -04 08 ξ 2 06 04 02 02 04 06 ξ 08 08 ξ 2 06 04 02 02 04 06 ξ 08

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Convergence of sensitivity indices: No Card(A) S {} (ˆf ) S {2} (ˆf ) S {,2} (ˆf ) ˆf f 2 3 0 9604 ( ) 2567 ( 2) 390 ( 2) 7 ( 2) 5 2 9578 ( ) 2545 ( 2) 676 ( 2) 26 ( 2) 7 36 9572 ( ) 2542 ( 2) 734 ( 2) 96 ( 3) 9 55 957 ( ) 2542 ( 2) 745 ( 2) 36 ( 3) 5 36 957 ( ) 2542 ( 2) 748 ( 2) 9 ( 4)

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Convergence of sensitivity indices for naive MC sampling: 0962 004 096 0035 003 0958 0025 S 0956 S 2 002 0954 005 00 0952 0005 095 000 0000 00000 e+06 e+07 M 0 000 0000 00000 e+06 e+07 M

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Homework: Homma-Saltelli function f (x, x 2, x 3 ) = sin(x ) + a sin 2 (x 2 ) + bx 4 3 sin(x ), where x, x 2 and x 3 are iid random variables uniformly distributed on [ π, π] Letting be the expectation (over [ π, π] 3 ), we have sin = 0, sin 2 = /2, sin 4 = 3/8, x n = { π n n+, n even, 0, n odd Compute the SH decomposition of f and the associated sensitivity indices Solution: f (x, x 2, x 3 ) = a/2+sin(x )(+bπ 2 /4)+a(sin 2 (x 2 ) /2)+b sin(x )(x 4 3 π4 /5)) 2 Compute PC expansion of f by non-intrusive spectral projection for polynomial degrees to 6 and extract st order and total sensitivity indices for a = 7 and b = 0 3 Compute -st order and total sensitivity indices by Monte-Carlo simulation

Sobol-Hoeffding decomposition Application to Global SA Computation of the SI SI from PC expansion Further readings: Hoeffding, W (948) A class of statistics with asymptotically normal distribution The annals of Mathematical Statistics, 9, pp 293-325 Sobol, I M (993) Sensitivity estimates for nonlinear mathematical models Wiley,, pp 407-44 Sobol, I M (200) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates Mathematics and Computers in Simulations, 55, pp 27-28 Homma T, Saltelli A, (996) Importance measures in globla sensitivity analysis of nonlinear models Reliab Eng Syst Safety, 52:, pp -7 Crestaux T, Le Maître O and Martinez JM, (2009) Polynomial Chaos expansion for sensitivity analysis Reliabg Eng Syst Safety, 94:7, pp 6-72