Nonstationary Covariance Functions for Gaussian Process Regression

Transcription

1 Nonstationar Covariance Functions for Gaussian Process Regression Christopher J. Paciorek Department of Biostatistics Harvard School of Public Health Mark J. Schervish Department of Statistics Carnegie Mellon Universit Neural Information Processing Sstems December 9,

2 ABSTRACT We introduce a class of nonstationar covariance functions for Gaussian process (GP) regression. Nonstationar covariance functions allow the model to adapt to functions whose smoothness varies with the inputs. The class includes a nonstationar version of the Matérn stationar covariance, in which the differentiabilit of the regression function is controlled b a parameter, freeing one from fiing the differentiabilit in advance. In eperiments, the nonstationar GP regression model performs well when the input space is two or three dimensions, performing comparabl to a Baesian neural network and outperforming an optimized neural network model and Baesian free-knot spline models. In one dimension, it is outperformed b a state-of-the-art Baesian free-knot spline model and b the Baesian neural network model. The model readil generalizes to non-gaussian data. Use of computational methods for speeding GP fitting allows for implementation of the method on somewhat larger datasets. 2

3 THE GENERAL PROBLEM OF VARIABLE SMOOTHNESS For functions whose smoothness varies with the input values, most nonparametric methods, including Gaussian process (GP) regression with a stationar covariance function, will oversmooth in some regions and undersmooth in others. Below I show the posterior mean regression function from a GP regression implementation with two different fied values (blue and red lines) of the correlation scale hperparameter. The top panels are for 100 data points using Gaussian error around the true function (black line) while the bottom panels are for 500 data points. κ = 0.2 κ =

4 CURRENT APPROACHES Man other approaches to this problem have been presented, but good approaches for multivariate features and comparison amongst the approaches are still needed. Some current approaches are: Free-knot regression spline models: number and location of knots optimized, with more knots in locations where function varies more quickl Baesian Adaptive Regression Splines (BARS) (DiMatteo, Genovese, and Kass 2002) (single feature onl) Baesian Multivariate Adaptive Regression Splines (BMARS) (Denison, Mallick, and Smith 1998) and Baesian Multivariate Linear Splines (BMLS) (Holmes and Mallick 2001) (multiple features) Adaptive penalt smoothing spline models (MacKa and Takeuchi 1995) Neural network models (Neal, etc.) Mitures of stationar Gaussian processes (Tresp 2001; Rasmussen and Ghahramani 2002) Stationar Gaussian process regression in a deformed feature space (Damian, Sampson, and Guttorp 2001, Schmidt and O Hagan 2000) (used for spatial features) In this poster and accompaning paper, we describe an approach to the variable smoothness problem using Gaussian process regression with nonstationar covariance functions. 4

5 STATIONARY CORRELATION FUNCTIONS Here are two stationar correlation functions, with eamples of the correlation function (left) and sample functions drawn from a Gaussian process parameterized b the correlation function (right). The squared eponential correlation function (top) gives sample functions with infinitel man derivatives, while the Matérn correlation function (bottom) gives sample functions whose number of derivatives varies with ν: ν 1 derivatives. For ν, one recovers the squared eponential form. Squared eponential: R(τ) = ep ( ( ) ) τ 2 κ Correlation κ = 0.2 κ = 0.04 f() Distance Matérn form: R(τ) = ( 1 2 ντ 2 Γ(ν)2 ν 1 κ ) ν Kν ( 2 ντ 2 κ ) Correlation ν = 30 ν = 2 f() Distance Correlation function 5 Sample functions

6 A NONSTATIONARY COVARIANCE FUNCTION Higdon, Swall, and Kern (1999) (HSK) introduced the following nonstationar correlation function, where c ij is a normalizing term and k i (u) is a function (called the kernel function) centered at i. R NS ( i, j ) = c ij R P k i (u)k j (u)du Guaranteed positive definite Using Gaussian kernels, one gets a closed form for the correlation: k i (u) ep ( (u i ) T Σ 1 i (u i ) ) ( R NS ( i, j ) = c ij ep ( i j ) T ( Σi + Σ j 2 ) 1 ( i j )) Gibbs (1997) gave a special case with diagonal Σ i, Σ j f( ) GP(µ, σ 2 R NS (, ; Σ( ))) is a nonstationar Gaussian process 6

7 NONSTATIONARY GPS IN 1D Here are four sample functions (right) drawn from a nonstationar Gaussian process distribution whose Gaussian kernels are defined based on their standard deviation (left). Note that the sample functions are more wiggl in the left part of the input space where the kernels are less broad. Some sample functions Kernel standard deviation Some kernels

8 NONSTATIONARY GPS IN 2D Here (right) is one sample function (of two inputs) from a nonstationar Gaussian process distribution whose Gaussian kernels are depicted using ellipses of constant densit (left). Note the smoothness of the function where the kernels are large and the directionalit of the smoothness where the kernels have strong directionalit. Kernel Structure Sample Function z

9 GENERALIZING THE HSK KERNEL METHOD The stationar squared eponential form (left) has the same form as the HSK nonstationar covariance (right): ( ( τ ) ) ( ( 2 ep c ij ep ( i j ) T Σi + Σ j κ 2 ) 1 ( i j )) The HSK nonstationar covariance merel replaces Euclidean distance with a quadratic form in the eponential. Gaussian process distributions with this nonstationar covariance have infinitel-differentiable sample paths if Σ( ) var smoothl. Consider the following distance measures (the nonstationar one does not isotrop satisf the triangle inequalit): τ 2 ij = ( i j ) T ( i j ) anisotrop τ 2 ij = ( i j ) T Σ 1 ( i j ) ( ) 1 Q ij = ( i j ) T Σi + Σ nonstationarit j ( i j ) 2 Can we replace τij 2 with Q ij in other stationar correlation functions and still retain positive definiteness? 9

10 GENERALIZED NONSTATIONARY COVARIANCE Theorem 1 (Paciorek 2003): if a stationar correlation function, R(τ), is positive definite for R P, P = 1, 2,..., then R NS ( i, j ) = Σ i 1 4 Σ j 1 4 Σ 1 i+σ j 2 2 ( Qij ) R is positive definite for R P, P = 1, 2,... Theorem 2 (Paciorek 2003): Smoothness (differentiabilit) properties of original stationar correlation R(τ) are retained if elements of Σ( ) var smoothl in the feature space. 10

11 PROOF OF THEOREM 1 (SKETCH) If R(τ) is positive definite for R P, P = 1, 2,..., then R(τ) = 0 ep( τ 2 w)h(w)dw (Schoenberg 1938) Reepress the proposed nonstationar correlation function: R NS ( i, j ) = 2 P 2 Σ i 1 4 Σ j 1 4 Σ i + Σ j ep( Q ij w)h(w)dw = 2 P 2 Σ i 1 4 Σ j 1 4 Σ i + Σ j 1 2 = 0 0 ep ( 1 2 ( i j ) T ( Σi +Σ j 2w R P k i,w (u)k j,w (u)duh(w)dw ) 1 (i j )) h(w)dw 11

12 Now check the definition of positive definiteness directl: n i=1 n a i a j C( i, j ) = j=1 = = n i=1 j=1 0 0 n a i a j R P R P 0 n a i k i,w (u) i=1 R P k i,w (u)k j,w (u)duh(w)dw n a j k j,w (u)duh(w)dw j=1 ( n 2 a i k i,w (u)) duh(w)dw 0. The ke is that the covariance must depend onl on location-specific kernels i=1 12

13 NONSTATIONARY MATÉRN COVARIANCE A particular nonstationar covariance of interest is a Matérn form, which satisfies the conditions of Theorem 1. 1 Γ(ν)2 ν 1 stationar form ( ) ν ( 2 ντ 2 ντ K ν κ κ ) nonstationar form 1 ( Γ(ν)2 ν 1 2 ) ν ( νq ij Kν 2 ) νq ij Provided Σ( ) varies smoothl, b Theorem 2, this form will give sample functions whose differentiabilit varies with ν. B not constraining the differentiabilit, this gives a more fleible form of the correlation function than the original HSK nonstationar correlation, and has asmptotic advantages (Stein 1999). 13

14 A BAYESIAN NONSTATIONARY GP REGRESSION MODEL Baesian model Y i N(f( i ), η 2 ), i R P f( ) GP(µ, σ 2 R NS (, ; Σ( ), ν)) Let R NS be the nonstationar Matérn correlation Kernels, Σ( ), are constructed based on stationar GP priors, as described net. 14

15 SMOOTHLY-VARYING KERNEL MATRICES Goals: Define multiple kernel matrices, Σ i one for each training observation and test/prediction observation Matri elements should be smoothl-varing in input space Matrices must be positive definite Use spectral decomposition (Σ i = Γ T i Λ iγ i ) Eigenvector matri, Γ i, is parameterized as first eigenvector plus successive orthogonal vectors in reduced-dimension subspaces stationar GP priors on unnormalized eigenvector coordinates [(a i, b i ) in 2-d cartoon ] Eigenvalue matri, Λ i, parameterized b diagonal elements stationar GP priors on logarithms of diagonal elements (log λ 1,i, log λ 2,i in 2-d cartoon) gets unwield and highl-parameterized for large P λ 2, i i λ 1,i a i, b i 15

16 EFFICIENT REPRESENTATIONS OF STATIONARY GPS Let Φ be a vector of process values 1. Matérn basis functions (Kammann and Wand 2003) Φ = µ + σzω 1 2 w Z = (C( i κ k )), 1 i n,1 k K Ω = (C( κ j κ k )), 1 j K,1 k K C( ) a stationar covariance function matri operations based on K knots, {κ k }, so more efficient motivation: if {κ k } = { i }, Cov(Φ) = C( ) When sample hperparameters in MCMC, sample process as well: Φ = µ + σzω 1 2 w 16

17 2. Fourier basis functions (Wikle 2002) Φ dat = µ + σaφ grid (Φ dat is process values at data locations) Φ grid = Ψw (Φ grid is discretized process on a grid) w elements are independent, comple-valued RVs their variance is based on spectral densit of stationar C( ) Ψw is the inverse FFT (Ψ are Fourier basis vectors) propose blocks of values of w with focus on low-frequenc coefficients motivation: Cov(Φ) = C( ) asmptoticall as grid gets finer When sample hperparameters in MCMC, sample process as well: Φ dat = µ + σaψw 17

18 NONPARAMETRIC REGRESSION COMPARISON - 1D Compare to: stationar GP free-knot regression spline model (BARS) (DiMatteo et al. 2002) neural network (with number of hidden units that give best result) (R nnet librar) Baesian neural network (R. Neal software as implemented b A. Vehtari) Simulate 50 datasets and fit each model to each dataset Compare results based on mean squared error (relative to true function) 18

19 REGRESSION RESULTS - 1D test function 1 test function 2 test function Method Function 1 Function 2 Function 3 BARS.0081 (.0071,.0092).012 (.011,.013).0050 (.0043,.0056) Baesian neural network.0082 (.0072,.0093).011 (.010,.014).015 (.014,.016) Nonstat. GP.0083 (.0073,.0093).015 (.013,.016).026 (.021,.030) Stat. GP.0083 (.0073,.0093).026 (.024,.029).071 (.067,.074) neural network.0108 (.0095,.012).013 (.012,.015).0095 (.0086,.010) Free-knot spline (BARS) performs best, followed b Baesian neural network and nonstationar GP Most methods are similar on smoothl varing Function 1 19

20 NONPARAMETRIC REGRESSION COMPARISON - 2/3D Compare to: stationar GP free-knot regression spline with tensor products of univariate splines (BMARS) (Denison et al. 1998) free-knot regression spline with multivariate linear splines (BMLS) (Holmes and Mallick 2001) neural network (optimal number of hidden units)(r nnet librar) Baesian neural network (R. Neal software as implemented b A. Vehtari) 20

21 EXAMPLES 1.) Simulated dataset with 2 inputs: P = 2, n = 225 simulate 50 datasets and compare using MSE (relative to true function) True function NSGP estimate z 4 z ) Real dataset of Dec mean temperatures in Americas, n = 109 P = 2: longitude, latitude MSE based on 50-fold cross-validation (MSE relative to test data) 3.) Real dataset of dail ozone in NY, n = 111 P = 3: radiation, temperature, wind speed MSE based on 50-fold cross-validation (MSE relative to test data) 21

22 MEAN SQUARED ERROR Method Function with 2 inputs Temp. data Ozone data Baesian neural network.020 (.019,.022) Nonstat. GP.023 (.020,.026) Stat. GP.024 (.021,.026) BMARS.076 (.065,.087) BMLS.033 (.029,.038) neural network.040* (.033,.047) * Holmes and Mallick (2001) report a value of.07 for a neural network Baesian neural network and nonstationar Gaussian process appear to outperform other approaches. 22

23 GENERALIZED NONPARAMETRIC REGRESSION Model: Y i D(g(f( i ))) f( ) GP(µ, σ 2 R NS (, ; Σ( ), ν)) D is a probabilit distribution and g( ) is a link function Eamples: count data (D = Poisson, g 1 = log) binar data (D = Bernoulli, g 1 = logit) 23

24 TOKYO RAINFALL DATA EXAMPLE Data are presence/absence of rainfall for the calendar das, (Y i {0, 1, 2}). Top panel shows posterior mean probabilit of rainfall as a function of calendar da, with pointwise 95% uncertaint intervals, and bottom panel shows standard deviation of the Gaussian kernels as a function of calendar da. The probabilit of rainfall appears to be more variable toward the end of the ear calendar da Kernel size Prob. of rainfall calendar da

25 CONCLUSIONS We introduce a class of nonstationar covariance functions that can be used in Gaussian process regression (and classification) models and allow the model to adapt to variable smoothness in the unknown function. The nonstationar GPs improve on stationar GP models on several test datasets. In test functions on one-dimensional spaces, a state-of-the-art free-knot spline model and Baesian neural network outperform the nonstationar GP, but in higher dimensions, the nonstationar GP outperforms two free-knot spline approaches and a non-baesian neural network while being comparable to a Baesian neural network. The nonstationar GP ma be of particular interest for data indeed b spatial coordinates. Unfortunatel, the nonstationar GP requires man more parameters than a stationar GP, particularl as the dimension grows, losing the attractive simplicit of the stationar GP model. Use of stationar GP priors in the hierarch of the model to parameterize the nonstationar covariance results in slow computation. More efficient representations of these stationar GPs improves efficienc but still limits the model to approimatel n < The slow part of the computation for Gaussian data is that calculating the marginal likelihood requires the Cholesk decomposition of an n b n matri. Our approach provides a general modelling framework; other low-rank approimations to the covariance matri (e.g., Smola and Bartlett 2001; Seeger and Williams 2003) ma further speed fitting. 25

26 References Damian, D., Sampson, P., and Guttorp, P. (2001), Baesian estimation of semi-parametric non-stationar spatial covariance structure, Environmetrics, 12, Denison, D., Mallick, B., and Smith, A. (1998), Baesian MARS, Statistics and Computing, 8, DiMatteo, I., Genovese, C., and Kass, R. (2002), Baesian curve-fitting with free-knot splines, Biometrika, 88, Gibbs, M. (1997), Baesian Gaussian Processes for Classification and Regression, unpublished Ph.D. dissertation, Universit of Cambridge. Higdon, D., Swall, J., and Kern, J. (1999), Non-stationar spatial modeling, in Baesian Statistics 6, eds. J. Bernardo, J. Berger, A. Dawid, and A. Smith, Oford, U.K.: Oford Universit Press, pp Holmes, C. and Mallick, B. (2001), Baesian regression with multivariate linear splines, Journal of the Roal Statistical Societ, Series B, 63, Kammann, E. and Wand, M. (2003), Geoadditive models, Applied Statistics, 52, MacKa, D. and Takeuchi, R. (1995), Interpolation models with multiple hperparameters,. Paciorek, C. (2003), Nonstationar Gaussian Processes for Regression and Spatial Modelling, unpublished Ph.D. dissertation, Carnegie Mellon Universit, Department of Statistics. Rasmussen, C. and Ghahramani, Z. (2002), Infinite mitures of Gaussian process eperts, in Advances 26

27 in Neural Information Processing Sstems 14, eds. T. G. Dietterich, S. Becker, and Z. Ghahramani, Cambridge, Massachusetts: MIT Press. Schmidt, A. and O Hagan, A. (2000), Baesian Inference for Nonstationar Spatial Covariance Structure via Spatial Deformations, Technical Report 498/00, Universit of Sheffield, Department of Probabilit and Statistics. Schoenberg, I. (1938), Metric spaces and completel monotone functions, Ann. of Math., 39, Seeger, M. and Williams, C. (2003), Fast forward selection to speed up sparse Gaussian process regression, in Workshop on AI and Statistics 9. Smola, A. and Bartlett, P. (2001), Sparse greed Gaussian process approimation, in Advances in Neural Information Processing Sstems 13, eds. T. Leen, T. Dietterich, and V. Tresp, Cambridge, Massachusetts: MIT Press. Tresp, V. (2001), Mitures of Gaussian Processes, in Advances in Neural Information Processing Sstems 13, eds. T. K. Leen, T. G. Dietterich, and V. Tresp, MIT Press, pp Wikle, C. (2002), Spatial modeling of count data: A case stud in modelling breeding bird surve data on large spatial domains, in Spatial Cluster Modelling, eds. A. Lawson and D. Denison, Chapman & Hall, pp