Kernel-based Approximation. Methods using MATLAB. Gregory Fasshauer. Interdisciplinary Mathematical Sciences. Michael McCourt.

Transcription

1 SINGAPORE SHANGHAI Vol TAIPEI - Interdisciplinary Mathematical Sciences 19 Kernel-based Approximation Methods using MATLAB Gregory Fasshauer Illinois Institute of Technology, USA Michael McCourt University of Colorado Denver, USA ^ World Scientific NEW JERSEY LONDON BEIJING HONG KONG CHENNAI

2 Contents Preface vii An Introduction to Kernel-Based Approximation Methods and Their Stable Computation 1 1 Introduction 3 11 Positive Definite Kernels: Where Do They Fit in the Mathematical Landscape? 3 12 A Historical Perspective 5 13 The Fundamental Application: Scattered Data Fitting The Haar-Mairhuber-Curtis theorem: Why using kernels is a "natural" approach Variations of scattered data fitting Other Applications Statistical data fitting Machine learning Numerical solution of PDEs Computational finance Topics We Do Not Cover 15 2 Positive Definite Kernels and Reproducing Kernel Hilbert Spaces Positive Definite Kernels Hilbert-Schmidt, Mercer and Karhunen-Loeve Series Hilbert-Schmidt operators The Hilbert Schmidt eigenvalue problem Mercers theorem Examples of Hilbert-Schmidt integral eigenvalue problems and Mercer series Iterated kernels 30 xi

3 xii Contents 226 Fourier and Karhunen-Loeve expansions Reproducing Kernel Hilbert Spaces 24 Feature Maps Examples of Kernels Radial Kernels Isotropic radial kernels Anisotropic radial kernels Translation Invariant Kernels Series Kernels Power series and Taylor series kernels Other series kernels General Anisotropic Kernels Dot product kernels Zonal kernels Tensor product kernels Compactly Supported Radial Kernels Multiscale Kernels Space-Time Kernels Learned Kernels Designer Kernels Periodic, kernels Chebyshev kernels 58 4 Kernels in Matlab Radial Kernels in MATLAB Symmetric distance matrices in Matlab General distance matrices in Matlab Anisotropic distance matrices in Matlab Evaluating radial kernels and interpolants in Matlab Compactly Supported Kernels in Matlab Zonal Kernels in Matlab Tensor Product Kernels in Matlab Series Kernels in Matlab 79 5 The Connection to Kriging Random Fields and Random Variables Duality of Spaces Modeling and Prediction via Kriging Kriging as best linear unbiased predictor Bayesian framework Confidence intervals 101

4 Contents xiii 534 Semi-variograms Karhunen-Loeve Expansions and Polynomial Chaos Generalized Polynomial Chaos The Connection to Green's Kernels Introduction Ill 62 Green's Kernels Defined Differential Eigenvalue Problems Computing Green's Kernels An example: Computing the Brownian bridge kernel as Green's kernel Generalizations of the Brownian bridge kernel Classical Examples of Green's Kernels Sturm-Liouville Theory Eigenfunction Expansions 68 The Connection Between Hilbert-Schmidt and Sturm-Liouville Eigenvalue Problems Limitations Summary Iterated Brownian Bridge Kernels: A Green's Kernel Example Derivation of Piecewise Polynomial Spline Kernels Recall some special Green's kernels A family of piecewise polynomial splines of arbitrary odd degree Benefits of using a kernel representation for piecewise polynomial splines 72 Derivation of General Iterated Brownian Bridge Kernels Properties of Iterated Brownian Bridge Kernels Truncation of the Mercer series Effects of the boundary conditions Convergence orders Iterated Brownian bridge kernels on bounded domains "Flat" limits Summary for functions satisfying homogeneous boundaryconditions Generalized Sobolev Spaces How Native Spaces Were Viewed Until Recently Generalized Sobolev Spaces on the Full Space Rd Two different kernels for H2(E) Higher-dimensional examples 156

5 An xiv Contents 823 Summary for full-space generalized Sobolev spaces Generalized Sobolev Spaces on Bounded Domains Modifications of the Brownian bridge kernel: A detailed investigation Summary for generalized Sobolev spaces on bounded domains An alternative framework for boundary value problems on [a,b] Conclusions Accuracy and Optimality of Reproducing Kernel Hilbert Space Methods Optimality Different Types of Error The "Standard" Error Bound Error Bounds via Sampling Inequalities How sampling inequalities lead to error bounds Univariate sampling inequalities and error bounds Application to iterated Brownian bridge kernels Sampling inequalities in higher dimensions Dimension-independent error bounds Traditional dimension-dependent error bounds Worst-case weighted L2 error bounds "Flat" Limits Introduction Kernels with Infinite Smoothness Kernels with Finite Smoothness Summary and Outlook The Uncertainty Principle - Unfortunate Misconception Accuracy vs Stability Accuracy and Stability Alternate Bases Data-dependent Basis Functions Standard basis functions Cardinal basis functions Alternate bases via matrix factorization Newton-type basis functions SVD and weighted SVD bases Analytical and Numerical Eigenfunctions 217

6 Contents xv 1221 Eigenfunctions given analytically Eigenfunctions obtained computationally Approximation Using Eigenfunctions 124 Other Recent Preconditioning and Alternate Basis Techniques Stable Computation via the Hilbert-Schmidt SVD A Formal Matrix Decomposition of K Obtaining a Stable Alternate Basis via the Hilbert Schmidt SVD Summary: How to use the Hilbert-Schmidt SVD Iterated Brownian Bridge Kernels via the Hilbert-Schmidt SVD 134 Issues with the Hilbert-Schmidt SVD Truncation of the Hilbert-Schmidt series Invertibility of 4>i Comparison of Alternate Bases for Gaussian Kernels Parameter Optimization 141 Modified Golomb-Weinberger Bound and Kriging Variance How to avoid cancelation while computing the power function (kriging variance) How to stably compute the native norm space of the interpolant (Mahalanobis distance) Cross-Validation Maximum Likelihood Estimation MLE independent of process variance MLE with process variance A deterministic derivation of MLE Other Approaches to the Selection of Good Kernel Parameters 145 Goals for a Parametrization Judgment Tool 269 Advanced Examples Scattered Data Fitting Approximation Using Smoothing Splines Low-rank Approximate Interpolation Interpolation on the Unit Sphere Computational Considerations for Scattered Data Fitting The cost of computing/implementing an alternate basis Exploiting structure in kernel computations Computer Experiments and Surrogate Modeling Surrogate Modeling 295

7 xvi Contents 162 Experimental Design Surrogate Models for Standard Test Functions Piston simulation function Borehole function Modeling From Data Fitting Empirical Distribution Functions Statistical Data Fitting via Gaussian Processes Geostatistics Anisotropic Data Fitting Data Fitting Using Universal Kriging and Maximum Likelihood Estimation Machine Learning Regularization Networks Radial Basis Function Networks Numerical experiments for regression with RBF networks Support Vector Machines Linear classification Kernel classification Numerical experiments for classification with kernel SVMs Computational consideration for classification with kernel SVMs Linear support vector regression Nonlinear support vector regression Derivatives of Interpolants and Hermite Interpolation Differentiating Interpolants Cardinal function representation of derivatives Error bounds for simultaneous approximation Global differentiation matrices Local differentiation matrices Hermite Interpolation 1921 Nonsymmetric kernel-based Hermite interpolation Symmetric kernel-based Hermite interpolation Generalized Hermite interpolation via the Hilbert -Schmidt SVD 1924 An example: Gradient interpolation 1925 Kriging interpretation 193 Doing Hermite Interpolation via Derivatives of Eigenfunctions Differentiation of a low-rank eigenfunction approximate interpolant

8 Contents xvii 1932 An example: Derivatives of Gaussians eigenfunctions Multiphysics Coupling Meshfree coupling An example: coupled 2D heat equation Computational considerations Kernel-Based Methods for PDEs Collocation for Linear Elliptic PDEs Nonsyinmetric collocation in the standard basis Nonsymmetric collocation using the Hilbert-Schmidt SVD Method of Lines Method of Fundamental Solutions Method of Particular Solutions Kernel-based Finite Differences Space-Time Collocation Finance Brownian motion Brownian motion and the Brownian motion kernel Geometric Brownian motion Pricing options and high-dimensional integration A generic error formula for quasi-monte Carlo integration via reproducing kernels Example of asset pricing through quasi-monte Carlo Black-Scholes PDEs Single-asset European option through Black-Scholes PDEs Pricing American options 445 Appendix A Collection of Positive Definite Kernels and Their Known Mercer Series 447 Al Piecewise Linear Kernels 447 A 11 Brownian bridge kernel 447 A12 Brownian motion kernel 448 A 13 Another piecewise linear kernel 448 A2 Exponential Kernel 448 A21 Domain: [0,1] 449 A22 Domain: 449 [ L, L] A23 Domain: 449 [0, oo) A3 Other Continuous Kernels 450 A31 Tension spline kernel 450 A32 Relaxation spline kernel 451 A33 Legendre kernel 451

9 xviii Contents A4 Modified Exponential Kernel 451 A5 Families of Iterated Kernels 452 A51 Iterated Brownian bridge kernels 452 A52 Periodic spline kernels 452 A53 Periodic kernels 453 A54 Chebyshev kernels 453 A6 Kernel for the First Weighted Sobolev Space 454 A 7 Gaussian Kernel 455 A8 Sine Kernel 455 A 9 Zonal Kernels 456 A91 Spherical inverse multiquadric 456 A92 Abel-Poisson kernel 456 Appendix B How To Choose the Data Sites 457 B l Low Discrepancy Designs 458 B2 Optimal Designs in Statistics 460 B3 Optimal Points in Approximation Theory 461 Appendix C A Few Facts from Analysis and Probability 463 Appendix D The GaussQR Repository in Matlab 467 Dl Accessing GaussQR 467 D2 Common functions in GaussQR 468 D3 Full Hilbert-Schmidt SVD sample solver 469 Bibliography 473 Index 505