Gaussian process regression for Sensitivity analysis

Transcription

1 Gaussian process regression for Sensitivity analysis GPSS Workshop on UQ, Sheffield, September 2016 Nicolas Durrande, Mines St-Étienne, GPSS workshop on UQ GPs for sensitivity analysis 1 / 43

2 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 2 / 43

3 We assume we are interested in a function f : R d R with d 2. We want to get some understanding on the structure of f : What is the effect of each input variables on the output? Do some variables have more influence than other? Do some variables interact together? The talk will be illustrated on the following test function : f : [0, 1] 6 R x 10 sin(πx 1 x 2 ) + 20(x 3 0.5) x 4 + 5x 5 GPSS workshop on UQ GPs for sensitivity analysis 3 / 43

4 First thing one can do is to plot the output versus each input. For 100 samples uniformly distributed over [0, 1] 6 we get : x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 4 / 43

5 In a similar fashion, we can fix all variables except one. In graph bellow, all non plotted variables are set to x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 5 / 43

6 In order to get an insight on the interaction between variables, we can look at the influence of changing the reference value x 2 = 0 x 2 = 0.2 x 2 = 0.4 x 2 = 0.5 x 2 = 0.7 x 2 = 0.9 x x 2 = 0 x 2 = 0.2 x 2 = 0.4 x 2 = 0.5 x 2 = 0.7 x 2 = 0.9 x GPSS workshop on UQ GPs for sensitivity analysis 6 / 43

7 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 7 / 43

8 One common tool for analysing the structure of f is to look at its FANOVA representation : d f (x) = f 0 + f i (x i ) + f i,j (x i, x j ) + + f 1,...,d (x) i=1 i<j This decomposition is such that : f 0 accounts for the constant term all f I are zero mean (I 0) f 1 accounts for all signal that can be explained just by x 1 f I (x)dx 1 = 0 for all I / {0, 1} f I (x)dx 1 = 0 for all I 1 In other words, this decomposition is such that all terms are orthogonal in L 2. GPSS workshop on UQ GPs for sensitivity analysis 8 / 43

9 The expressions of the f I are : f 0 = f (x)dx f i (x i ) = f (x)dx i f 0 f i,j (x i, x j ) = f (x)dx ij f i (x i ) f j (x j ) + f 0 It can also be interesting to look at the total effect of some inputs : f 1 (x 1 ) = f (x)dx 1 f 1,2 (x 1, x 2 ) = f (x)dx {1,2} GPSS workshop on UQ GPs for sensitivity analysis 9 / 43

10 On the previous example we obtain : x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 10 / 43

11 We can also look at 2nd order interactions f 1,2 (x 1, x 2 ) f 1,3 (x 1, x 3 ) Interaction x 1, x 2 Interaction x 1, x 3 GPSS workshop on UQ GPs for sensitivity analysis 11 / 43

12 The total effect of (x 1, x 2 ) is thus f 1,2 (x 1, x 2 ) = f 0 + f 1 (x 1 ) + f 2 (x 2 ) + f 1,2 (x 1, x 2 ) GPSS workshop on UQ GPs for sensitivity analysis 12 / 43

13 In practical application f is not analytical so the above method require numerical computations of the integrals. If there is a cost associated with the evaluation of f, surrogate models are useful. Some models are naturally easy to interpret, for example but it soon becomes more tricky. m(x) = β 0 + β 1 x 1 + β 2 x 2 m(x) = β 0 + β 1 x 1 + β 2 x 2 + β 1,2 x 1 x 2 GPSS workshop on UQ GPs for sensitivity analysis 13 / 43

14 In GPR, the mean can be seen either as a linear combination of the observations : m(x) = α t F the kernel evaluated at X : m(x) = k(x, X)β For example, we have for a squared exponential kernel k(x, X) x Z(x) Z(X) = F x The basis function have a local influence which makes the interpretation difficult. GPSS workshop on UQ GPs for sensitivity analysis 14 / 43

15 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 15 / 43

16 The principle of polynomial chaos is to project f onto a basis of orthonormal polynomials. One dimension For x R, the h i are of order i. Starting from the constant function h 0 = 1, the following ones can be obtain using Gram-Schmidt orthogonalisation. h 0 (x) = 1 1, h 1(x) = x x, h 0 h 0 x x, h 0 h 0, h 2(x) = x 2 x, h 0 h 0 x, h 1 h 1 x 2 x, h 0 h 0 x, h 1 h 1 d-dimension In R d, the basis is obtained by a tensor product of one dimensional basis. For example, if d = 2 : h 00 (x) = 1 1 h 10 (x) = h 1 (x 1 ) 1 h 01 (x) = 1 h 1 (x 2 ) h 11 (x) = h 1 (x 1 ) h 1 (x 2 ) h 20 (x) = h 2 (x 1 ) 1. =. GPSS workshop on UQ GPs for sensitivity analysis 16 / 43

17 The orthonormal basis H depends on the measure over the input space D. A uniform measure over D = [ 1, 1] gives the Legendre basis : h 0 (x) = 1/2 h 1 (x) = 3/2 x h 2 (x) = 5/4 (3x 2 1) h 3 (x) = 7/4 (5x 3 3x) h 4 (x) = 9/16 (35x 4 30x 2 + 3). =. A standard Gaussian measure over R gives the Hermite basis : h 0 (x) = 1/ 2π h 1 (x) = 1/ 2π x h 2 (x) = 1/(2 2π) (x 2 1) h 3 (x) = 1/(6 2π) (x 3 3x) h 4 (x) = 1/(24 2π) (x 4 6x 2 + 3). =. GPSS workshop on UQ GPs for sensitivity analysis 17 / 43

18 Legendre basis in 1D x x 3 x 2 x 1 x GPSS workshop on UQ GPs for sensitivity analysis 18 / 43

19 Legendre basis in 2D h00 h10 h01 x 1 x 2 x 1 x 2 x 1 x 2 h11 h20 h21 x 1 x 2 x 1 x 2 x 1 x 2 GPSS workshop on UQ GPs for sensitivity analysis 19 / 43

20 If we consider linear regression model based on polynomial chaos basis functions m(x) = βh I (x) I {0,...p} the FANOVA representation of m is straightforward. For example in 2D : m 0 = m(x)dx 1 dx 2 = β 00 h 00 (x)dx 1 dx 2 = β 00 m 1 (x 1 ) = m(x)dx 2 m 0 = β 00 h 00 (x) + β 10 h 10 (x) + β 20 h 20 (x)dx 2 m 0 = β 10 h 10 (x 1 ) + β 20 h 20 (x 1 ) m 1,2 (x) =... = β 11 h 11 (x) + β 12 h 12 (x) + β 21 h 21 (x) + β 22 h 22 (x) GPSS workshop on UQ GPs for sensitivity analysis 20 / 43

21 We obtain on the motivating example : x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 21 / 43

22 Same figure without cheating : x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 22 / 43

23 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 23 / 43

24 A first idea is to consider ANOVA kernels [Stitson 97] : k(x, y) = d (1 + k i (x i, y i )) i=1 = 1 + The associated GP is d k i (x i, y i ) + d k i (x i, y i )k j (x j, y j ) + + k i (x i, y i ) i=1 i<j i=1 }{{}}{{}}{{} additive part 2 nd order interactions full interaction d Z(x) = Z }{{} 0 + Z i (x i ) + Z i,j (x i, x j ) + + Z 1...d (x) }{{} cst i=1 i<j }{{}}{{} full interaction additive part 2 nd order interactions However, the Z I do not satisfy Z I (x)dx i = 0. GPSS workshop on UQ GPs for sensitivity analysis 24 / 43

25 If we build a GPR model based on this kernel, we obtain : m(x) = k(x, X)k(X, X) 1 F ( d m(x) = 1 + k i (x i, y i ) + ) k i (x i, y i )k j (x j, y j ) k(x, X) 1 F i=1 i<j d = 1 t k(x, X) 1 F + k(x i, X i )k(x, X) 1 F i=1 } {{ } m i (x i ) + k i (x i, X i )k(x j, X j )k(x, X) 1 F i<j }{{} m i,j (x i,x j ) +... As previously, the m I do not satisfy m I (x)dx i = 0. GPSS workshop on UQ GPs for sensitivity analysis 25 / 43

26 samples with zero integrals We are interested in building a GP such that the integral of the samples are exactly zero... Let s consider the associated conditional GP : Z 0 law = Z Z(s)ds =0 Let µ 0 (x) = E ( Z(x) Z(s)ds =0 ) denote the conditional expectation and k 0 (x, x ) = cov ( Z(x), Z(x ) Z(s)ds =0 ) ( 1 µ 0 (x) = k(x, s)ds k(s, t)dsdt) 0 k 0 (x, y) = k(x, x ) ( k(x, s)ds ) 1 k(s, t)dsdt k(x, s)ds GPSS workshop on UQ GPs for sensitivity analysis 26 / 43

27 Samples from Z 0 have the required property k(x, s)ds k(y, s)ds µ 0 (x) = 0 k 0 (x, y) = k(x, y) k(s, t)dsdt Z(x) x GPSS workshop on UQ GPs for sensitivity analysis 27 / 43

28 These 1-dimensional kernels are of great importance to create ANOVA kernels dedicated to sensitivity analysis : k SA (x, y) = d (1 + k 0 (x i, y i )) i=1 = 1 + d k(x i, y i ) + d k(x i, y i )k(x j, y j ) + + k(x i, y i ) i=1 i<j i=1 }{{}}{{}}{{} additive part 2 nd order interactions full interaction The associated GP naturally writes d Z SA (x) = Z }{{} 0 + Z i (x i ) + Z i,j (x i, x j ) + + Z 1...d (x) }{{} cst i=1 i<j }{{}}{{} full interaction additive part 2 nd order interactions Now, the Z I do satisfy Z I (x)dx i = 0. GPSS workshop on UQ GPs for sensitivity analysis 28 / 43

29 Z We get the following decomposition of samples Z00 Z10 x 1 x 2 x 1 x 2 x 1 x 2 Z01 Z11 x 1 x 2 x 1 x 2 GPSS workshop on UQ GPs for sensitivity analysis 29 / 43

30 Furthermore, the GPR model inherits this properties 2D example 2 k(x, y) = (1 + k 0 (x i, y i )) i=1 = 1 + k 0 (x 1, y 1 ) + k 0 (x 2, y 2 ) + k 0 (x 1, y 1 )k 0 (x 2, y 2 ) The mean writes m(x) = (1 + k 0 (x 1, X 1 ) + k 0 (x 2, X 2 ) + k 0 (x 1, X 1 )k 0 (x 2, X 2 )) t k(x, X) 1 F = m 0 + m 1 (x 1 ) + m 2 (x 2 ) + m 12 (x) These terms correspond to the FANOVA representation of m. The sub-models are conditional expectations : m I (x) = E (Z I (x) Z(X)=F ) We can thus associate a predictive covariance to each sub-model! GPSS workshop on UQ GPs for sensitivity analysis 30 / 43

31 We obtain on the motivating example : x 1 x 2 x x 4 x 5 x GPSS workshop on UQ GPs for sensitivity analysis 31 / 43

32 We obtain on the motivating example : GPSS workshop on UQ GPs for sensitivity analysis 32 / 43

33 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 33 / 43

34 The principle of sensitivity analysis is to quantify how much each input or group of inputs has an influence on the output : local sensitivity analysis global sensitivity analysis Probabilistic framework has proven to be very interesting : If we introduce randomness in the inputs, how random is the output? Hereafter, we focus on variance based global sensitivity analysis GPSS workshop on UQ GPs for sensitivity analysis 34 / 43

35 Let X be the random vector representing our uncertainty on the inputs. We assume its probability distribution factorises (i.e. the X i are independents) : µ(x) = µ(x 1 ) µ(x 2 ) µ(x n ) This factorization of µ allows to use it in the FANOVA representation of f : d f (x) = f 0 + f i (x i ) + f i,j (x i, x j ) + + f 1,...,d (x) i=1 i<j in this expression, the f I are orthogonal for µ. We know plug X into this expression : d f (X) = f 0 + f i (X i ) + f i,j (X i, X j ) + + f 1,...,d (X) i=1 i<j and we get interesting results... GPSS workshop on UQ GPs for sensitivity analysis 35 / 43

36 The f I (X I ) are centred and independent E (f I (X I )) = f I (x I )dµ(x) = 0 (for I 0) cov (f I (X I ), f J (X J )) = E (f I (X I )f J (X J )) = f I (x I )f J (x J )dµ(x) = 0 As a consequence, we get d var (f (X)) = var (f i (X i ))+ var (f i,j (X i, X j ))+ +var (f 1,...,d (X)) i=1 i<j The Sobol indices are defined as S I = var (f I(X I )) var (f (X)) These indices are in [0, 1], and their sum is 1. GPSS workshop on UQ GPs for sensitivity analysis 36 / 43

37 In practice, these indices can be computed using Monte Carlo methods. This requires lots of observations Another approach is to use surrogate models. If the model is well chosen, computational cost is almost free! Polynomial Chaos GPR with k sa kernels GPSS workshop on UQ GPs for sensitivity analysis 37 / 43

38 Polynomial Chaos In the case of polynomial Chaos, Sobol indices are given by the squares of the β coefficients D i = Var X [E X (H(X)β X i )] = Var X [H i (X i )β i ] = β 2 i S i = D i k D k For mode details, see the work from Bruno Sudret GPSS workshop on UQ GPs for sensitivity analysis 38 / 43

39 GPR with k sa kernel The sensitivity indices can be obtained analytically : S I = var (m I(X I )) var (m(x)) F T K 1 ( i I = Γ i) K 1 F ( di=1 ) F T K 1 (1 n n + Γ i ) 1 n n K 1 F where Γ i is the matrix Γ i = D i k 0 i (s i)k 0 i (s i) T ds i, and is an entry-wise product. GPSS workshop on UQ GPs for sensitivity analysis 39 / 43

40 The computation of Sobol indices on the mean gives : S 1 S 2 S 3 S 4 S 5 S 6 S 12 model model truth For this test-function 50 observations are enough! GPSS workshop on UQ GPs for sensitivity analysis 40 / 43

41 Introduction FANOVA, ie HDMR, ie Sobol-Hoeffding representation Polynomial Chaos Gaussian process Regression Sensitivity Analysis Conclusion GPSS workshop on UQ GPs for sensitivity analysis 41 / 43

42 Sensitivity analysis There are interesting tools to get an insight of what s happening inside high dimensional functions The effective dimensionality can be much smaller Monte Carlo or model based approach Some modelling tips What is the purpose of the model? GPR models are not necessarily black-box... It is possible to include fancy observations in GPR (integrals, derivatives,...) This talk unfortunately focused on sensitivity analysis on the mean... the proper way is to perform SA on the conditional sample paths. See work from Marrel, Iooss et Al, SAMO GPSS workshop on UQ GPs for sensitivity analysis 42 / 43

43 Using appropriate kernels, the computation of Sobol indices on the samples gives : D i = Var X [E X (Z(X) X i )] = Var X [Z i (X i )] We can easily sample from this distribution D 1 prior D 2 prior D12 prior Similarly, we can sample from the posterior to get an uncertainty measure on the indices. GPSS workshop on UQ GPs for sensitivity analysis 43 / 43