Deep Gaussian Processes for Multi-fidelity Modeling

Transcription

1 Deep Gaussian Processes for Muti-fideity Modeing Kurt Cutajar EURECOM Sophia Antipois, France Mark Puin Andreas Damianou Nei Lawrence Javier Gonzáez Abstract Muti-fideity modes are prominenty used in various science and engineering appications where cheapy-obtained, but possiby biased and noisy observations must be effectivey combined with imited or epensive true data in order to construct reiabe modes. The notion of appying deep Gaussian processes DGPs to this setting has recenty shown great promise by capturing compe noninear correations across fideities. However, the architectures epored thus far are burdened by structura assumptions and constraints which deter such modes from performing to the best of their epected capabiities. In this paper we propose a nove approach for DGP muti-fideity modeing which treats DGP ayers as fideity eves and uses a variationa inference scheme to propagate uncertainty across them. In our eperiments, we show that this approach makes substantia improvements in quantifying and propagating uncertainty in muti-fideity set-ups, which in turn improves their effectiveness in decision-making pipeines. 1 Introduction Muti-fideity modes [4, 7] are designed to fuse imited true observations high-fideity with cheapy-obtained ower granuarity representations ow-fideity. Gaussian processes [GPs; 9] are we-suited to muti-fideity probems due to their abiity to encode prior beiefs about how fideities are reated, yieding predictions accompanied by uncertainty estimates. GPs formed the basis of semina autoregressive modes AR1 investigated by [4] and [6], and are suitabe when the mapping between fideities is inear, i.e. the high-fideity function f t can be modeed as: f t = ρf t 1 + δ t, 1 where ρ is a constant scaing the contribution of sampes f t 1 drawn from the GP modeing the data at the preceding fideity, and δ t modes the bias between fideities. However, this is insufficient when the mapping is noninear, i.e. ρ is now a noninear transformation such that: f t = ρ t f t 1 + δ t. 2 The additive structure and independence assumption between the GPs for modeing ρ t f t 1 and δ t permits us to combine these as a singe GP that takes as inputs both and f t 1, which here denotes a sampe from the posterior of the GP modeing the preceding fideity evauated at. This can be epressed as f t = g t f t 1,. Third workshop on Bayesian Deep Learning NeurIPS 2018, Montréa, Canada. Work carried out during an internship at, Cambridge.

2 high-fideity AR1 ow-fideity NARGP a Left: Overfitting in the NARGP mode. Right: b Left: AR1 cannot capture noninear mappings. We-caibrated fit using proposed MF - DGP mode. Right: Fied by compositiona structure of MF - DGP. Figure 1: Limitations addressed and resoved jointy by MF - DGP. Bue and red markers denote ow and high-fideity observations respectivey. Shaded regions indicate the 95% confidence interva. Deep Gaussian processes [DGPs; 2] are a natura candidate for handing such reationships, aowing for uncertainty propagation in a nested structure of GPs where each GP modes the transition from one fideity to the net. However, DGPs are cumbersome to deveop and approimations are necessary for enabing tractabe inference. Whie motivated by the structure of DGPs, the noninear muti-fideity mode NARGP proposed in [8] amounts to a disjointed architecture whereby each GP is fitted in an isoated hierarchica manner, preventing GPs at ower fideities from being updated once they have been fit. Consider the eampe given in Figure 1a. In the boed area, we woud epect the mode to return high uncertainty to refect the ack of data avaiabe, but overfitting in NARGP resuts in predicting an incorrect resut with reasonaby high confidence. Contribution: In this work, we propose the first compete interpretation of muti-fideity modeing using DGPs, which we refer to as MF - DGP. In particuar, we everage the sparse DGP approimation proposed in [10] for constructing a muti-fideity DGP mode which can be trained end-to-end, overcoming the constraints that hinder eisting attempts at using DGP structure for this purpose. Returning to the eampe given in Figure 1a, we see that our mode fits the true function propery whie aso returning sensiby conservative uncertainty estimates. Additionay, our mode aso inherits the compositiona structure of NARGP, aeviating a crucia imitation of AR1 Figure 1b. 2 Muti-fideity Deep Gaussian Process The appication of DGPs to the muti-fideity setting is particuary appeaing because if we assume that each ayer corresponds to a fideity eve, then the atent functions at the intermediate ayers are given a meaningfu interpretation which is not aways avaiabe in standard DGP modes. The first attempt at using compositions of GPs in a muti-fideity setting [8] reied on structura assumptions on the data to circumvent the intractabiity of DGPs, but this heaviy impairs their epected feibiity. Recent advances in the DGP iterature [1, 10] have everaged traditiona GP approimations to construct scaabe DGP modes which are easier to specify and train; we buid our etension atop the mode presented in [10] to avoid the constraints imposed on seecting kerne functions in [1]. 2.1 Mode Specification Let us assume a dataset D having observations at T fideities, where Xt and yt denote the nt inputs and corresponding outputs observed with fideity eve t: { } D = X1, y1,..., Xt, yt,..., XT, yt. For enhanced interpretabiity, we assume that each ayer of our MF - DGP mode corresponds to the process modeing the observations avaiabe at fideity eve t, and that the bias or deviation from the true function decreases from one eve to the net. We use the notation Ft to denote the evauation at ayer for inputs observed with fideity t; for eampe, the evauation of the process at ayer 1 for the inputs observed with fideity 3 is denoted as F31. A conceptua iustration of the proposed MF DGP architecture is given in Figure 2 eft for a dataset with three fideities. Note that the GP at each ayer is conditioned on the data beonging to that eve, as we as the evauation of that same input data at the preceding fideity. This gives greater purpose to the notion of feeding forward the origina inputs at each ayer, as originay suggested in [3] for avoiding pathoogies in deep architectures. 2

3 X 1 f X 2 X 3 1 GP f 2 GP f 3 GP {F t {F t 1 }3 t=1 2 }3 t=2 {F t 3 }3 t=3 f 1 GP f 1 f 2 GP f 2 f 3 GP f 3 y 1 y 2 y 3 y 1 y 2 y 3 Figure 2: Left: architecture with 3 fideity eves. Right: Predictions using same. At each ayer we rey on the sparse variationa approimation of a GP for inference, thus obtaining the foowing variationa posterior distribution: q F t U = p F t U ; {F t 1, X t }, Z 1 q U, 3 where Z 1 denotes the inducing inputs for, U their corresponding function evauation, and q U = N U µ, Σ is the variationa approimation of the inducing points. The mean and variance defining this variationa approimation, i.e. µ and Σ, are optimized during training. Furthermore, if U is marginaized out from Equation 3, the resuting variationa posterior is once again Gaussian and fuy defined by its mean, m, and variance, S : q F t µ, Σ ; {F t 1, X t }, Z 1 = N F t m t, S t, 4 which can be derived anayticay. The ikeihood noise at ower fideity eves is encoded as additive white noise in the kerne function of the GP at that ayer. We can then formuate the variationa ower bound on the margina ikeihood as foows: L = T n t t=1 i=1 E qf i,t t [ og p y i,t f i,t t ] + L D KL q U p U ; Z 1, where we assume that the ikeihood is factorized across fideities and observations, and D KL denotes the Kuback-Leiber divergence. Sampes from the mode are obtained recursivey using the reparameterization trick [5] to draw sampes from the variationa posterior. Mode predictions with different fideities are aso obtained recursivey by propagating the input through the mode up to the chosen fideity. At a intermediate ayers, the output from the preceding ayer is augmented with the origina input, as wi be made evident by the choice of kerne epained in the net section. The output of a test point can then be predicted with fideity eve t as foows: q f t 1 S S s=1 =1 q f s, t µ t, Σ t ; {f s, t 1, }, Z t 1, 5 where S denotes the number of Monte Caro sampes and t repaces as the ayer indicator. This procedure is iustrated in Figure 2 right. 2.2 Muti-fideity Covariance For every GP at an intermediate ayer, we opt for the muti-fideity kerne function proposed in [8], since this captures both the potentiay noninear mapping between outputs as we as the correation in the origina input space: k = k ρ i, j ; θ ρ k f 1 f 1 i, f 1 j ; θ f 1 + k δ i, j ; θ δ, 6 3

4 Linear 1 Linear 2 Noninear 1 Noninear 2 AR1 NARGP defaut aternate Figure 3: Comparison across methods and benchmarks for chaenging muti-fideity scenarios. The importance of choosing an appropriate kerne for is aso reinforced here. where k f 1 denotes the covariance between outputs obtained from the preceding fideity eve, k ρ is a space-dependent scaing factor, and k δ captures the bias at that fideity eve. At the first ayer this reduces to k 1 = k1 δ i, j ; θ δ 1. In [8], it was assumed that each individua component of the composite kerne function is an RBF kerne, and we sha aso assume this to be the defaut setting for. However, this may not be appropriate when the mapping between fideities is inear. In such instances, we propose to repace k f 1 with an aternate inear kerne such that the composite intermediate ayer covariance becomes: k = k ρ i, j ; θ ρ f 1 i f 1 j + k δ i, j ; θ δ. 7 3 Eperimenta Evauation In the preceding sections, we demonstrated how the formuation of state-of-the-art DGP modes can be adapted to the muti-fideity setting. Through a series of eperiments, we vaidate that beyond its novety and theoretic appea, the proposed mode aso works we in practice. Improved UQ: We empiricay vaidate s we-caibrated uncertainty quantification by considering eperimenta set-ups where the avaiabe data is generay insufficient to yied confident predictions, and higher uncertainty is prized. In Figure 3, we consider muti-fideity scenarios where the aocation of high-fideity data is imited or constrained to ie in one area of the input domain. In a of the eampes, our mode yieds appropriatey conservative estimates in regions where insufficient observations are avaiabe. As evidenced by the overfitting ehibited by AR1 for the LINEAR 2 eampe, deep modes can aso be usefu for probems having inear mappings. Tabe 1: Mode comparison on muti-fideity benchmark eampes. Defaut indicates use of the kerne isted in Equation 6, whie aternate indicates that the covariance in Equation 7 was used. Mean Squared Error Benchmark n ow n high AR1 NARGP Linear aternate Linear aternate Noninear e defaut Noninear defaut 4

5 Mean Squared Error AR Iterations NARGP high-fideity gp Predicted Water Fow Actua Water Fow Figure 4: Eperimenta design oop. Figure 5: fit to Borehoe function. Benchmark Comparison: We aso compare the predictive performance of to AR1 and NARGP on the same seection of benchmark eampes. Twenty randomy-generated training sets are prepared for each eampe function, foowing the aocation of ow and high-fideity points isted in Tabe 1. The resuts denote the average mean squared error obtained using each mode over a fied test set covering the entire input domain. The obtained resuts give credence to our intuition that baances out issues in the two modeing approaches; it performs as we as NARGP on the noninear eampes where AR1 faters, and outperforms the former on inear eampes. Muti-fideity in the Loop: We further assess using an epository eperimenta design oop whereby points are sequentiay chosen to reduce uncertainty about a function of interest. Starting with 20 ow-fideity and 3 high-fideity observations, we earn the NONLINEAR 1 function by seecting to observe points where the variance of the predictive distribution at the high fideity is argest. Figure 4 shows how the mean squared error against a constant test set evoves as more points are coected, averaged over 5 runs with different initia training data. Here we aso compare against a standard GP trained on the high-fideity observations ony. As epected, NARGP and MF- DGP perform best as the mode structure better represents the underying data. Athough NARGP and both converge to a simiar soution once enough points are samped, the benefit of using is evidenced in the initia steps of the procedure, whereby it fits the data sensiby after ony few iterations. Rea-word Simuation: We fit to a two-eve function that simuates stochastic water fow through a borehoe [11] and depends on eight input parameters, for which a dataset of 150 ow and 40 high-fideity points was generated. Figure 5 iustrates the performance of for a test set containing 1000 high-fideity points, where it achieves an R 2 of Concusion Reiabe decision making under uncertainty is a core requirement in muti-fideity scenarios where unbiased observations are scarce or difficut to obtain. In this paper, we proposed the first compete specification of a muti-fideity mode as a DGP that is capabe of capturing noninear reationships between fideities with reduced overfitting. By providing end-to-end training across a fideity eves, yieds superior quantification and propagation of uncertainty that is crucia in iterative methods such as eperimenta design. In spite of being prevaent in engineering appications, we beieve that muti-fideity modeing has been under-epored by the machine earning community, and hope that this work can reignite further interest in this direction. References [1] K. Cutajar, E. V. Bonia, P. Michiardi, and M. Fiippone. Random feature epansions for deep Gaussian processes. In Proceedings of the 34th Internationa Conference on Machine Learning, ICML 2017, Sydney, NSW, Austraia, 6-11 August 2017, pages , [2] A. C. Damianou and N. D. Lawrence. Deep Gaussian processes. In Proceedings of the Siteenth Internationa Conference on Artificia Inteigence and Statistics, AISTATS 2013, Scotts- 5

6 dae, AZ, USA, Apri 29 - May 1, 2013, pages , [3] D. K. Duvenaud, O. Rippe, R. P. Adams, and Z. Ghahramani. Avoiding pathoogies in very deep networks. In Proceedings of the Seventeenth Internationa Conference on Artificia Inteigence and Statistics, AISTATS 2014, Reykjavik, Iceand, Apri 22-25, 2014, pages , [4] M. C. Kennedy and A. O Hagan. Predicting the output from a compe computer code when fast approimations are avaiabe. Biometrika, 871:1 13, [5] D. P. Kingma and M. Weing. Auto-encoding variationa Bayes. In Proceedings of the Second Internationa Conference on Learning Representations, ICLR 2014, Banff, Canada, Apri 14-16, 2014, [6] L. Le Gratiet and J. Garnier. Recursive co-kriging mode for design of computer eperiments with mutipe eves of fideity. Internationa Journa for Uncertainty Quantification, 45, [7] B. Peherstorfer, K. Wico, and M. Gunzburger. Survey of mutifideity methods in uncertainty propagation, inference, and optimization. SIAM Review, 603: , [8] P. Perdikaris, M. Raissi, A. Damianou, N. D. Lawrence, and G. E. Karniadakis. Noninear information fusion agorithms for data-efficient muti-fideity modeing. Proceedings of the Roya Society A: Mathematica, Physica and Engineering Sciences, : , [9] C. E. Rasmussen and C. K. I. Wiiams. Gaussian processes for machine earning. Adaptive computation and machine earning. MIT Press, [10] H. Saimbeni and M. P. Deisenroth. Douby stochastic variationa inference for deep Gaussian processes. In Advances in Neura Information Processing Systems 30: Annua Conference on Neura Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages , [11] S. Xiong, P. Z. G. Qian, and C. F. J. Wu. Sequentia design and anaysis of high-accuracy and ow-accuracy computer codes. Technometrics, 551:37 46,