Constrained Gaussian processes: methodology, theory and applications

Transcription

1 Constrained Gaussian processes: methodology, theory and applications Hassan Maatouk Workshop on Gaussian Processes, November 6-7, 2017, St-Etienne (France) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 1 / 47

2 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 2 / 47

3 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 3 / 47

4 Gaussian Process Regression (GPR) or Kriging The following nonparametric function estimation is considered y = f(), R d. Observations : f( (i) ) = y i, i = 1,..., n. In statistical framework, y is viewed as a realization of a GP Y : Y () := η() + Z(), where η is the mean and Z is a zero-mean GP with covariance function K. The conditional process remains a GP { } Y () Y ( (1) ) = y 1,..., Y ( (n) ) = y n N ( ζ(), τ 2 () ), with { ζ() = η() + k() K 1 (y µ) τ 2 () = K(, ) k() K 1 k() µ i = η( (i) ), K i,j = K( (i), (j) ), i, j = 1,..., n and (k()) i = ( K(, (i) ) ). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 4 / 47

5 Gaussian Process Regression Let f() = sin(5π), [0, 1] be a real function vraie fonction moyenne de krigeage Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 5 / 47

6 Motivating eample y() GP sample paths true function posterior mean Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 6 / 47

7 Motivating eample y() GP sample paths true function posterior mean y() true function posterior mean Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 6 / 47

8 Motivating eample y() GP sample paths true function posterior mean y() true function posterior mean monotone GP true function inequality mode Hassan Maatouk / Rennes 2 Gaussian processes withinequality constraints November 7, 2017, St-Etienne 6 / 47

9 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 7 / 47

10 Gaussian processes with inequality constraints I is the space of interpolation conditions : f( (i) ) = y i, i = 1,..., n. C is the space of conveity constraints (such as boundedness, monotonicity and conveity). (Y ()) [0,1] d is a zero-mean GP with covariance function : K (, ) = Cov (Y (), Y ( )) = E (Y ()Y ( )). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 8 / 47

11 Gaussian processes with inequality constraints I is the space of interpolation conditions : f( (i) ) = y i, i = 1,..., n. C is the space of conveity constraints (such as boundedness, monotonicity and conveity). (Y ()) [0,1] d is a zero-mean GP with covariance function : K (, ) = Cov (Y (), Y ( )) = E (Y ()Y ( )). Formulation of the problem : simulate the Gaussian process Y conditionally to : Y ( (i) ) = y i, i = 1,..., n, Y C. (I C) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 8 / 47

12 Gaussian processes with inequality constraints I is the space of interpolation conditions : f( (i) ) = y i, i = 1,..., n. C is the space of conveity constraints (such as boundedness, monotonicity and conveity). (Y ()) [0,1] d is a zero-mean GP with covariance function : K (, ) = Cov (Y (), Y ( )) = E (Y ()Y ( )). Formulation of the problem : simulate the Gaussian process Y conditionally to : Y ( (i) ) = y i, i = 1,..., n, Y C. (I C) Remark This is a difficult problem because the conditional process Y Y I C is not a GP in general. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 8 / 47

13 Finite-dimensional approimation of GPs Methodology : we develop a finite-dimensional approimation of GPs of the form N Y N () := ξ j φ j (), with ξ = (ξ 0 ξ N ) N (0, Γ N ) and {φ j } the deterministic basis functions. j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 9 / 47

14 Finite-dimensional approimation of GPs Methodology : we develop a finite-dimensional approimation of GPs of the form N Y N () := ξ j φ j (), with ξ = (ξ 0 ξ N ) N (0, Γ N ) and {φ j } the deterministic basis functions. j=0 Fundamental property of the basis functions (φ j ) j=0,,n we choose the basis functions {φ j } such that Y N (.) is positive ξ j 0 ; 0 j N. Y N () [a, b] a ξ j b ; 0 j N. Y N (.) is non-decreasing ξ j 0 ; 0 j N. Y N (.) is conve ξ j 0 ; 0 j N. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 9 / 47

15 Boundedness constraints 1 C = {f : [0, 1] R : a f() b}. 2 The basis functions (h j ) j are the hat functions associated to the knots (u j ) j=0,...,n such that : h j (u k ) = δ j,k. hi Y N () = N j=0 ξ jh j () and Y N (u k ) = N j=0 ξ jh j (u k ) = N j=0 ξ jδ j,k = ξ k. Y N N () = j=0 Y N (u j )h j () is a piecewise linear function. N Y N () = Y N (u j )h j () [a, b], ξ j = Y N (u j ) [a, b], j = 0,..., N. j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 10 / 47

16 Covariance matri of the random coefficients ξ j By the special choose of the basis functions, we have N N Y N () := ξ j h j () = Y N (u j )h j (). j=0 j=0 To ensure the almost surly uniform convergence of Y N to Y we suppose : Y N (u j ) = Y (u j ). Thus, N Y N () = Y (u j )h j (). j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 11 / 47

17 Covariance matri of the random coefficients ξ j By the special choose of the basis functions, we have N N Y N () := ξ j h j () = Y N (u j )h j (). j=0 j=0 To ensure the almost surly uniform convergence of Y N to Y we suppose : Y N (u j ) = Y (u j ). Thus, N Y N () = Y (u j )h j (). j=0 Γ N i,j = Cov(ξ i, ξ j ) = Cov(Y (u i ), Y (u j )) = K(u i, u j ), with K the covariance function of the original Gaussian process Y. Cov ( Y N (), Y N ( ) ) = h() Γ N h(). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 11 / 47

18 Monotonicity constraints The basis functions are choose as the primitive of the hat functions : φ j () := 0 h j(t)dt. φ i In that case, the finite-dimensional approimation of GPs can be reformulated as N N Y N () := ζ + ξ j φ j () = Y (0) + Y (u j )φ j (). j=0 Thus, Γ N j,k = Cov(ξ j, ξ k ) = Cov (Y (u j ), Y (u k )) = 2 K(u j,u k ). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 12 / 47 j=0

19 New formulation of the problem - case of monotonicity constraints The simulation of Y N conditionally to interpolation and inequality constraints is equivalent to the simulation of the Gaussian vector ξ such that N ( ) (Aξ) i := ζ + ξ j φ j (i) = y i, i = 1,..., n (n interpolation conditions) I ξ j=0 ξ j 0, j = 0,..., N (N+1 inequality constraints) C coef with A i,j := φ j ( (i) ) and A i,1 = 1, i = 1,..., n. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 13 / 47

20 New formulation of the problem - case of monotonicity constraints The simulation of Y N conditionally to interpolation and inequality constraints is equivalent to the simulation of the Gaussian vector ξ such that N ( ) (Aξ) i := ζ + ξ j φ j (i) = y i, i = 1,..., n (n interpolation conditions) I ξ j=0 ξ j 0, j = 0,..., N (N+1 inequality constraints) C coef with A i,j := φ j ( (i) ) and A i,1 = 1, i = 1,..., n. The problem is reduced to simulate a truncated Gaussian vector restricted to conve sets. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 13 / 47

21 New formulation of the problem - case of monotonicity constraints The simulation of Y N conditionally to interpolation and inequality constraints is equivalent to the simulation of the Gaussian vector ξ such that N ( ) (Aξ) i := ζ + ξ j φ j (i) = y i, i = 1,..., n (n interpolation conditions) I ξ j=0 ξ j 0, j = 0,..., N (N+1 inequality constraints) C coef with A i,j := φ j ( (i) ) and A i,1 = 1, i = 1,..., n. The problem is reduced to simulate a truncated Gaussian vector restricted to conve sets. truncated normal distribution ξ I = µ ξ C ξ I µ ξ C truncated normal distribution Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 13 / 47

22 Mean and maimum of the posterior distribution Definition (mean a posteriori estimate) The mean of the posterior distribution ( is defined as ) m N pos () := E Y N () Y N ( (i) ) = y i, ξ C coef = φ() ξ pos, ( ) where ξ pos = E ξ ξ Iξ C coef. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 14 / 47

23 Mean and maimum of the posterior distribution Definition (mean a posteriori estimate) The mean of the posterior distribution is defined as m N pos () := E ( Y N () Y N ( (i) ) = y i, ξ C coef ) = φ() ξ pos, where ξ pos = E ( ξ ξ Iξ C coef ). { } Let µ be the mode of the truncated Gaussian vector ξ ξ Iξ C coef. Then ( ) 1 µ = arg min ξ I ξ C coef 2 ξ (Γ N ) 1 ξ, (1) with Γ N the covariance matri of the Gaussian vector ξ. Definition (maimum a posteriori estimate) The maimum a posterior (MAP) estimate is defined as N Mpos() N := µ j φ j (), µ = (µ 0,..., µ N ) is computed from (1). j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 14 / 47

24 Truncated normal variables Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 15 / 47

25 Truncated normal variables Proposition Let f and g be two pseudo-density functions defined as f() := f( 0, Σ)1 C and g() := g( µ, Σ)1 C. The optimal constant k such that f() k g(), C is equal to k = ep ( 12 ) (µ ) Σ 1 µ. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 15 / 47

26 Simulation of the truncated multivariate Gaussian random variables Accept/Rejection algorithm 1 Generate X with density g while X / C. 2 Generate U uniformly on [0, 1] and accept X if U e (µ ) T Σ 1 µ X T Σ 1 µ. Otherwise, repeat from step 1. y[,2] [,2] y[,1] [,1] Figure: Crude rejection sampling with 2% acceptance rate (left) and the so-called rejection sampling from the mode (RSM) with 20% acceptance rate (right) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 16 / 47

27 Illustrative eample - monotonicity constraints cases PG monotone Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

28 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

29 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

30 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

31 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction PG monotone Figure: 100 sample paths taken from the GP approimation conditionally to interpolation condition and monotonicity constraints. The unconstrained kriging mean coincides with the MAP estimate in the left figure but not in the right one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

32 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction PG monotone moyenne de krigeage usuelle Figure: 100 sample paths taken from the GP approimation conditionally to interpolation condition and monotonicity constraints. The unconstrained kriging mean coincides with the MAP estimate in the left figure but not in the right one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

33 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone Figure: 100 sample paths taken from the GP approimation conditionally to interpolation condition and monotonicity constraints. The unconstrained kriging mean coincides with the MAP estimate in the left figure but not in the right one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

34 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone mode Figure: 100 sample paths taken from the GP approimation conditionally to interpolation condition and monotonicity constraints. The unconstrained kriging mean coincides with the MAP estimate in the left figure but not in the right one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

35 Illustrative eample - monotonicity constraints cases PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone 95% intervalle de prédiction PG monotone moyenne de krigeage usuelle moyenne de krigeage monotone mode 95% intervalle de prédiction Figure: 100 sample paths taken from the GP approimation conditionally to interpolation condition and monotonicity constraints. The unconstrained kriging mean coincides with the MAP estimate in the left figure but not in the right one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 17 / 47

36 Illustrative eample - boundedness constraints case Gaussian process approimation : N N Y N () := ξ j h j () = Y (u j )h j (). (2) j=0 j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 18 / 47

37 Illustrative eample - boundedness constraints case Gaussian process approimation : N N Y N () := ξ j h j () = Y (u j )h j (). (2) j=0 j=0 PG borné moyenne de krigeage usuelle moyenne de krigeage bornée mode 95% intervalle de prédiction observation Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 18 / 47

38 Illustrative eample - boundedness constraints case Gaussian process approimation : N N Y N () := ξ j h j () = Y (u j )h j (). (2) j=0 The basis functions h j, j = 0,..., N are the hat functions. j=0 Y N () [a, b] if and only if Y (u j ) [a, b]. The sample paths verify boundedness constraints in the entire domain. The mean and the maimum of the posterior distribution respect boundedness constraints contrarily to the unconstrained kriging mean. PG borné moyenne de krigeage usuelle moyenne de krigeage bornée mode 95% intervalle de prédiction observation Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 18 / 47

39 Illustrative eample - conveity constraints case The Gaussian process approimation is equal to N Y N () := Y (0) + Y (0) + Y (u j )ϕ j (). (3) j=0 ϕ j := ( ) t 0 0 h j(u)du dt. Y N is conve if and only if Y (u j ) 0. The simulated paths respect conveity constraints in the entire domain. The mean and the maimum of the posterior distribution respect conveity constraints contrarily to the unconstrained kriging mean. PG convee moyenne de krigeage usuelle moyenne de krigeage convee mode Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 19 / 47

40 Isotonicity in two dimensions The input = ( 1, 2 ) is supposed to be in [0, 1] 2. The monotonicity constraints with respect to the two inputs is defined as 1 [0, 1], 2 f( 1, 2 ) is monotone and 2 [0, 1], 1 f( 1, 2 ) is monotone. 2 1 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 20 / 47

41 Isotonicity in two dimensions The finite-dimensional approimation of GPs is defined as N Y N ( 1, 2 ) := Y (u i, u j )h i ( 1 )h j ( 2 ), (4) i,j=0 where h j, j = 0,..., N are the hat functions. Y N is monotone with respect to the two inputs if and only if the random coefficients Y (u i, u j ) verify Y (u i 1, u j ) Y (u i, u j ) Y (u i, u j 1 ) Y (u i, u j ), i, j = 1,..., N. Y (u i 1, u 0 ) Y (u i, u 0 ), i = 1,..., N. Y (u 0, u j 1 ) Y (u 0, u j ), j = 1,..., N. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 21 / 47

42 Isotonicity with respect to only one variable The finite-dimensional approimation of GPs Y N ( 1, 2 ) = N Y (u i, u j )h i ( 1 )h j ( 2 ) (5) i,j=0 respects monotonicity constraints for only the first variable if and only if : Y (u i 1, u j ) Y (u i, u j ), i = 1,..., N and j = 0,..., N. X2 X1 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 22 / 47

43 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 23 / 47

44 Kimeldorf-Wahba correspondence We consider the following optimization problem : min h H I h 2 H, (Q) H the RKHS with r.k. K which is the covariance kernel of Y, I the space of functions which verify the interpolation condition. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 24 / 47

45 Kimeldorf-Wahba correspondence We consider the following optimization problem : min h H I h 2 H, (Q) H the RKHS with r.k. K which is the covariance kernel of Y, I the space of functions which verify the interpolation condition. Theorem (Kimeldorf and Wahba 1970) The problem (Q) has the kriging mean as an unique solution : h opt () = k() K 1 y, (6) where y = (y 1,..., y n ), k() = ( K( (i), ) ) i and K i,j = ( K( (i), (j) ) ). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 24 / 47

46 Generalization of the Kimeldorf-Wahba correspondence We consider the following conve optimization problem : min h H I C h 2 H, (P ) H the RKHS with r.k. K which is the covariance kernel of Y, I the space of functions which verify the interpolation condition, C the space of functions which verify inequality constraints (such as monotonicity, conveity,...). Theorem (Bay, Grammont, Maatouk, Electron. J. Statist.) The maimum a posteriori estimate (MAP) converges to the constrained spline M N pos() := N µ j φ j () h opt := arg min N + h H I C h 2 H. j=0 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 25 / 47

47 Numerical illustration of the new correspondence boundedness constraints unconstrained mean mean a posteriori maimum a posteriori Figure: The unconstrained kriging mean coincides with the MAP in the right figure but not in the left one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 26 / 47

48 Numerical illustration of the new correspondence boundedness constraints unconstrained mean mean a posteriori maimum a posteriori boundedness constraints unconstrained mean mean a posteriori maimum a posteriori Figure: The unconstrained kriging mean coincides with the MAP in the right figure but not in the left one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 26 / 47

49 boundedness constraints 0.2 unconstrained mean mean a posteriori maimum a posteriori unconstrained mean mean a posteriori maimum a posteriori 30 boundedness constraints Numerical illustration of the new correspondence Figure: The unconstrained kriging mean coincides with the MAP in the right figure but not in the left one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 26 / 47

50 Constrained cubic spline interpolation The cubic spline is the function minimises the linear enegry criterion (LE) (see, e.g., [Wolberg and Alfy, 2002]) : Subject to h I C : { 1 min 0 E L = 1 0 (h (t)) 2 dt. (7) } (h (t)) 2 dt, h H 2 I C, where H 2 = { h L 2 ([0, 1]) tel que h, h L 2 ([0, 1]) }. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 27 / 47

51 Constrained cubic spline interpolation The cubic spline is the function minimises the linear enegry criterion (LE) (see, e.g., [Wolberg and Alfy, 2002]) : Subject to h I C : { 1 min 0 E L = 1 0 (h (t)) 2 dt. (7) } (h (t)) 2 dt, h H 2 I C, where H 2 = { h L 2 ([0, 1]) tel que h, h L 2 ([0, 1]) }. This is equivalent to min h H α+β (i) +h( (i) )=y i α+β+h() C 1 0 (h (t)) 2 dt = h 2 H. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 27 / 47

52 Monotone cubic spline interpolation [Wolberg and Alfy, 2002] Table: Wolberg s data used to compare different methods f() approimation hyman Figure: Monotone cubic spline interpolation using Wolberg s data : Hyman approach (blue curve) and the proposed approach M N pos() with N = 1000 (red curve) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 28 / 47

53 Monotone cubic spline interpolation [Wolberg and Alfy, 2002] Table: Linear Energy (LE) criterion using Wolberg s data Method E L approimation Hyman CSE FE LE SDDE MDE FB N LE approimation LE = Figure: LE using our approach and Wolberg s data Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 29 / 47

54 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 30 / 47

55 Discount factors Discount factor is known to be a non-increasing function with respect to time-to-maturities. It verifies the following linear equality constraints : ( A f(x) = b, f(x) = f( (1) ),..., f( )) (n), (8) where A is a m n matri and b R m. 1 Bounds for OIS discounting curves 0.9 OIS discount factors time to maturity Figure: Cousin and Niang 2014 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 31 / 47

56 Discount factors using Swap vs Euribor 6M market quotes as of 02/06/2010 discount factor monotone GP 95% confidence interval mode Nelson Siegel Nelson Siegel Svensson discount factor monotone GP 95% confidence interval mode Nelson Siegel Nelson Siegel Svensson Figure: Simulated paths (gray lines) taken from the conditional GP with non-increasing constraints and market-fit constraints using the Gaussian covariance function with nugget equal to 10 5 (left) and the Matérn 5/2 covariance function without nugget (right). Swap vs. Euribor 6M market quotes as of 02/06/2010 Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 32 / 47

57 Forward rates using Swap vs Euribor 6M market quotes as 02/06/2010 Forward rate TF sample paths 95% confidence interval TF mode Nelson Siegel Nelson Siegel Svensson Forward rate TF sample paths 95% confidence interval TF mode Nelson Siegel Nelson Siegel Svensson Figure: Forward rates obtained from sample paths of previous figures with Gaussian covariance function (left) and Matérn 5/2 covariance function (right) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 33 / 47

58 Several quotation dates Swap vs Euribor discount factors time to maturity dates Figure: Swap-vs.-Euribor discount factors as a function of time-to-maturities and quotation dates Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 34 / 47

59 Default probabilities Credit Default Swaps (CDS) on 06/01/2005 survival probability survival probability Figure: CDS implied survival curves (gray lines) given as simulated paths of a conditional GP with non-increasing constraints using a Gaussian covariance function (left) and a Matérn 5/2 covariance function (right) Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 35 / 47

60 Several quotation dates Credit Default Swaps (CDS) protection maturities dates Figure: CDS implied survival probabilities as a function of time-to-maturities and quotation dates Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 36 / 47

61 Plan 1 General introduction of GP regression and motivating eample 2 Gaussian processes with inequality constraints Finite-dimensional approimation of GPs Simulation of truncated Gaussian vectors 3 Generalization of the Kimeldorf-Wahba correspondence 4 Real application in Insurance and Finance : estimation of discount factors and default probabilities Discount factors Default probabilities Credit Default Swaps (CDS) 5 Noisy case Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 37 / 47

62 Noisy In many application situations, an approimate response is available f( (i) ) = y i + ɛ i = ỹ i, i = 1,..., n, where ɛ N (0, σnoise 2 I), with σ2 noise the noise variance and I the identity matri. Conditionally to noisy ỹ = (ỹ 1,..., ỹ n ), the process remains a GP Y () Y (X) = ỹ GP ( ζ(), τ 2 () ), where ζ() = η() + k() (K + σ 2 noisei) 1 (ỹ µ) ; τ 2 () = K(, ) k() (K + σ 2 noisei) 1 k(), and µ = η(x) is the vector of trend values at the design of eperiments, K i,j = K( (i), (j) ), i, j = 1,..., n is the covariance matri of Y (X) and k() = (K(, (i) )) i is the vector of covariance between Y () and Y (X). Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 38 / 47

63 Illustrative eample - boundedness constraints Bounded GP unconstrained mean mean a posteriori maimum a posteriori Positive GP unconstrained mean mean a posteriori maimum a posteriori Figure: The GP approimation with positivity constraints (right) and boundedness constraints (left). The unconstrained mean coincides with the maimum a posteriori in the right figure but not in the left one Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 39 / 47

64 Illustrative eample - monotonicity constraints Monotone GP unconstrained mean mean a posteriori maimum a posteriori Figure: The GP approimation with monotonicity constraints for sinusoidal function f() = 0.32( + sin()). The unconstrained mean does not coincide with the maimum a posteriori Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 40 / 47

65 Illustrative eample - isotonicity in two dimensions f(1,2) 2 1 X2 X1 Figure: The maimum a posteriori estimate respecting monotonicity (non-decreasing) constraints for the two inputs, and the associated contour levels Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 41 / 47

66 Simulation study The real non-decreasing functions proposed by [Holmes and Heard, 2003, Neelon and Dunson, 2004] and used in a comparative study by [Shively et al., 2009, Lin and Dunson, 2014] are considered 1 flat function f 1() = 3, (0, 10] ; 2 sinusoidal function f 2() = 0.32{ + sin()}, (0, 10] ; 3 step function f 3() = 3 if (0, 8] and f 3() = 8 if (8, 10] ; 4 linear function f 4() = 0.3, (0, 10] ; 5 eponential function f 5() = 0.15 ep(0.6 3), (0, 10] ; 6 logistic function f 6() = 3/{1 + ep( )}, (0, 10]. The root-mean-square error (RMSE) of the estimates is computed at the one hundred values taken uniformly (equidistant) in the interval (0, 10] : RMSE = 1 n ( 2, f( i ) n ˆf( i )) i=1 where ˆf() is the estimate of f() and i are the n equally-spaced -values. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 42 / 47

67 Simulation study Table: Root-mean-square error ( 100) for data of size n = 100. The results are obtained by repeating the simulation 5000 times Flat Step Linear Eponential Logistic Sinusoidal GP GP projection Regression spline GP approimation GP projection : Lin, L. and Dunson, D. B. (2014). Bayesian monotone regression using Gaussian process projection. Biometrika, 101(2) : Regression spline : Shively, T. S., Sager, T. W., and Walker, S. G. (2009). A Bayesian approach to non-parametric monotone function estimation. J. R. Stat. Soc. B, 71(1) : GP approimation : Maatouk, H. and Bay, X. (2017). Gaussian process emulators for computer eperiments with inequality constraints. Math.Geosci., 49 : Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 43 / 47

68 Simulation study f() % credible intervals true sinusoidal function maimum a posteriori Figure: The 95% credible intervals of the GP approimation together with the sinusoidal function, the (grey crosses) and the maimum a posteriori estimate Table: Empirical coverage (%) for 95% credible intervals at different values. The simulations are repeated 1000 times GP GP projection GP approimation Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 44 / 47

69 Figure: The root-mean-square error at different sample sizes together with the optimal values obtained in [Riihimäki and Vehtari, 2010]. The results are based on 1000 simulation replicates using the logistic function : 2/{1 + ep( 8 + 4)}, [0, 1] Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 45 / 47 Simulation study (methodology based on the knowledge of the derivatives of the GP at some input locations) root mean square error root mean square error RMSE=0.062 RMSE= sample size

70 References Holmes, C. and Heard, N. (2003). Generalized monotonic regression using random change points. Stat. Med., 22(4) : Lin, L. and Dunson, D. B. (2014). Bayesian monotone regression using Gaussian process projection. Biometrika, 101(2) : Maatouk, H. and Bay, X. (2016). A new rejection sampling method for truncated multivariate Gaussian random variables restricted to conve sets. In Cools, R. and Nuyens, D., editors, Monte Carlo and Quasi-Monte Carlo Methods, pages Springer International Publishing, Cham. Neelon, B. and Dunson, D. B. (2004). Bayesian isotonic regression and trend analysis. Biometrics, 60(2) : Riihimäki, J. and Vehtari, A. (2010). Gaussian processes with monotonicity information. J. Mach. Learn. Res., 9 : Shively, T. S., Sager, T. W., and Walker, S. G. (2009). A Bayesian approach to non-parametric monotone function estimation. J. R. Stat. Soc. B, 71(1) : Wolberg, G. and Alfy, I. (2002). An energy-minimization framework for monotonic cubic spline interpolation. Journal of Computational and Applied Mathematics, 143(2) : Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 46 / 47

71 Thank you for your attention. Hassan Maatouk / Rennes 2 Gaussian processes with inequality constraints November 7, 2017, St-Etienne 47 / 47