Hilbert Space Methods for Reduced-Rank Gaussian Process Regression

Transcription

1 Hilbert Space Methods for Reduced-Rank Gaussian Process Regression Arno Solin and Simo Särkkä Aalto University, Finland Workshop on Gaussian Process Approximation Copenhagen, Denmark, May 2015 Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

2 Contents 1 Problem formulation 2 Covariance and Laplacian operators 3 Hilbert-space approximation 4 Application to GP regression 5 Experimental results 6 Summary Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

3 Problem formulation Gaussian process (GP) regression problem: f GP(0, k(x, x )), y i = f (x i ) + ε i. The GP-regression has cubic computational complexity O(n 3 ) in the number of measurements. This results from the inversion of an n n matrix: µ(x ) = k(x, x 1:n ) (k(x 1:n, x 1:n ) + σ 2 ni) 1 y V (x ) = k(x, x ) k(x, x 1:n ) (k(x 1:n, x 1:n ) + σ 2 ni) 1 k(x 1:n, x ). In practice, we use Cholesky factorization and do not invert explicitly but still the O(n 3 ) problem remains. Various sparse, reduced-rank, and related approximations have been developed for mitigating this problem. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

4 Covariance operator For covariance function k(x, x ) we can define covariance operator: K φ = k(, x ) φ(x ) dx. For stationary covariance function k(x, x ) k( r ); r = x x we get S(ω) = k(r) e i ωtr dr. The transfer function corresponding to the operator K is S(ω) = F [K]. The spectral density S(ω) also gives the approximate eigenvalues of the operator K. We can now represent the operator K as a series of Laplace operators. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

5 Laplacian operator series In isotropic case S(ω) S( ω ), and we can expand S( ω ) = a 0 + a 1 ω 2 + a 2 ( ω 2 ) 2 + a 3 ( ω 2 ) 3 +. The Fourier transform of the Laplace operator 2 is ω 2, i.e., K = a 0 + a 1 ( 2 ) + a 2 ( 2 ) 2 + a 3 ( 2 ) 3 +. Defines a pseudo-differential operator as a series of differential operators. Let us now approximate the Laplacian operators with a Hilbert method... Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

6 Background: series expansions of GPs Assume a covariance function k(x, x ) and an inner product, say, f, g = f (x) g(x) w(x) dx. Ω The inner product induces a Hilbert-space of (random) functions. If we fix a basis {φ j (x)}, a Gaussian process f (x) can be expanded into a series f (x) = f j φ j (x), where f j are jointly Gaussian. j=1 If we select φ j to be the eigenfunctions of k(x, x ) w.r.t.,, then this becomes the Karhunen Loève series. In the Karhunen Loève case the coefficients f j are independent Gaussian. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

7 Hilbert-space approximation of Laplacian Consider the eigenvalue problem for the Laplacian operators: { 2 φ j (x) = λ j φ j (x), x Ω, φ j (x) = 0, x Ω. The eigenfunctions φ j ( ) are orthonormal w.r.t. inner product f, g = f (x) g(x) dx, Ω φ i (x) φ j (x) dx = δ ij. Ω The negative Laplacian has the formal kernel l(x, x ) = j λ j φ j (x) φ j (x ) in the sense that 2 f (x) = l(x, x ) f (x ) dx. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

8 Hilbert-space approximation of covariance function Recall that we have the expansion K = a 0 + a 1 ( 2 ) + a 2 ( 2 ) 2 + a 3 ( 2 ) 3 +. Substituting the formal kernel gives k(x, x ) a 0 + a 1 l 1 (x, x ) + a 2 l 2 (x, x ) + a 3 l 3 (x, x ) + = [ a0 + a 1 λ 1 j + a 2 λ 2 j + a 3 λ 3 j + ] φ j (x) φ j (x ). j Evaluating the spectral density series at ω 2 = λ j gives S( λ j ) = a 0 + a 1 λ 1 j + a 2 λ 2 j + a 3 λ 3 j +. This leads to the final approximation k(x, x ) j S( λ j ) φ j (x) φ j (x ). Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

9 Accuracy of the approximation ν = 1 2 ν = 3 2 ν = 5 2 ν = 7 2 ν Exact m = 12 m = 32 m = 64 m = l Approximations to covariance functions of the Matérn class of various degrees of smoothness; ν = 1/2 corresponds to the exponential Ornstein Uhlenbeck covariance function, and ν to the squared exponential (exponentiated quadratic) covariance function. Approximations are shown for 12, 32, 64, and 128 eigenfunctions. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

10 Reduced-rank method for GP regression Recall the GP-regression problem f GP(0, k(x, x )) y i = f (x i ) + ε i. Let us now approximate f (x) m f j φ j (x), j=1 where f j N (0, S( λ j )). The approximating GP f m (x) now approximately has the desired covariance function. Via the matrix inversion lemma we then get µ φ T (Φ T Φ + σ 2 nλ 1 ) 1 Φ T y, V σ 2 nφ T (Φ T Φ + σ 2 nλ 1 ) 1 φ. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

11 Computational complexity The computation of Φ T Φ takes O(nm 2 ) operations. The covariance function parameters do not enter Φ and we need to evaluate Φ T Φ only once (nice in parameter estimation). If the observations are on a grid, we can use FFT-kind of methods. The scaling in input dimensionality can be quite bad but depends on the chosen domain. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

12 Convergence analysis Theorem There exists a constant C d such that k(x, x ) k m (x, x ) C d L + 1 π d ω π ˆm 2L S(ω) dω, where L = min k L k, which in turn implies that uniformly [ ] lim lim k m (x, x ) = k(x, x ). L 1,...,L d m Corollary The uniform convergence of the prior covariance function also implies uniform convergence of the posterior mean and covariance in the limit m, L 1,..., L d. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

13 Toy example 2 Observations Mean 95% region Exact interval True curve Approximation True value Output, y σ 2 = 1 2 l = Input, x σ 2 n = SMSE m MSLL FULL NEW SOR DTC VAR FIC SSGP m Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

14 Correlation contours in two-dimensions (m = 16) L FIC (grid) L FIC (random) Sparse spectrum (random) L x2 0 x2 0 x2 0 L L 0 L x1 L L 0 L x1 L L 0 L x1 L Laplacian (Cartesian) Laplacian (Cartesian extended) L L Laplacian (polar) x2 0 x2 0 x2 0 L L 0 L x1 L L 0 L x1 L L 0 L Correlation contours computed for three locations corresponding to the squared exponential covariance function (exact contours dashed). x1 Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

15 Precipitation data Interpolation of yearly precipitation (n = 5776) using a full GP and the Laplacian GP mm Evaluation time (s) Solin & Sa rkka (Aalto University, Finland) MSLL SMSE Hilbert Space Methods for GPs Evaluation time (s) May, / 24

16 Surface temperature of the globe Random draws (Mate rn, ν = {1/2, 3/2, }) in a spherical domain. 20 C 0 C 30 C 0 C 2 C 4 C 6 C 8 C GP regression (mean and standard deviation) of temperature data (n = 11028). Solin & Sa rkka (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

17 Summary Hilbert-space method for reduced-rank Gaussian process regression. The covariance function is approximated by eigenfunctions of the Laplace operator. The eigenvalues are approximated by spectral density values. The approximation converges to exact limit in well-defined conditions. Experimental results show that the method works well in practice. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24

18 Related and future work Spatio-temporal Gaussian processes combination with Kalman filtering and smoothing. Periodic covariance functions and other specific classes of covariances. Static and spatio-temporal inverse problems. Non-Gaussian processes. Optimization for high-dimensional inputs. Solin & Särkkä (Aalto University, Finland) Hilbert Space Methods for GPs May, / 24