SINGAPORE SHANGHAI Vol TAIPEI - Interdisciplinary Mathematical Sciences 19 Kernel-based Approximation Methods using MATLAB Gregory Fasshauer Illinois Institute of Technology, USA Michael McCourt University of Colorado Denver, USA ^ World Scientific NEW JERSEY LONDON BEIJING HONG KONG CHENNAI
Contents Preface vii An Introduction to Kernel-Based Approximation Methods and Their Stable Computation 1 1 Introduction 3 11 Positive Definite Kernels: Where Do They Fit in the Mathematical Landscape? 3 12 A Historical Perspective 5 13 The Fundamental Application: Scattered Data Fitting 7 131 The Haar-Mairhuber-Curtis theorem: Why using kernels is a "natural" approach 9 132 Variations of scattered data fitting 11 14 Other Applications 12 141 Statistical data fitting 12 142 Machine learning 13 143 Numerical solution of PDEs 13 144 Computational finance 14 15 Topics We Do Not Cover 15 2 Positive Definite Kernels and Reproducing Kernel Hilbert Spaces 17 21 Positive Definite Kernels 17 22 Hilbert-Schmidt, Mercer and Karhunen-Loeve Series 20 221 Hilbert-Schmidt operators 20 222 The Hilbert Schmidt eigenvalue problem 22 223 Mercers theorem 24 224 Examples of Hilbert-Schmidt integral eigenvalue problems and Mercer series 25 225 Iterated kernels 30 xi
xii Contents 226 Fourier and Karhunen-Loeve expansions 31 23 Reproducing Kernel Hilbert Spaces 24 Feature Maps 32 36 3 Examples of Kernels 41 31 Radial Kernels 41 311 Isotropic radial kernels 41 312 Anisotropic radial kernels 44 32 Translation Invariant Kernels 45 33 Series Kernels 46 331 Power series and Taylor series kernels 47 332 Other series kernels 48 34 General Anisotropic Kernels 49 341 Dot product kernels 49 342 Zonal kernels 50 343 Tensor product kernels 52 35 Compactly Supported Radial Kernels 53 36 Multiscale Kernels 54 37 Space-Time Kernels 55 38 Learned Kernels 56 39 Designer Kernels 56 391 Periodic, kernels 57 392 Chebyshev kernels 58 4 Kernels in Matlab 61 41 Radial Kernels in MATLAB 62 411 Symmetric distance matrices in Matlab 63 412 General distance matrices in Matlab 64 413 Anisotropic distance matrices in Matlab 66 414 Evaluating radial kernels and interpolants in Matlab 68 42 Compactly Supported Kernels in Matlab 72 43 Zonal Kernels in Matlab 76 44 Tensor Product Kernels in Matlab 77 45 Series Kernels in Matlab 79 5 The Connection to Kriging 89 51 Random Fields and Random Variables 90 52 Duality of Spaces 94 53 Modeling and Prediction via Kriging 96 531 Kriging as best linear unbiased predictor 96 532 Bayesian framework 99 533 Confidence intervals 101
Contents xiii 534 Semi-variograms 105 54 Karhunen-Loeve Expansions and Polynomial Chaos 106 55 Generalized Polynomial Chaos 107 6 The Connection to Green's Kernels 111 61 Introduction Ill 62 Green's Kernels Defined 112 63 Differential Eigenvalue Problems 114 64 Computing Green's Kernels 115 641 An example: Computing the Brownian bridge kernel as Green's kernel 115 642 Generalizations of the Brownian bridge kernel 117 65 Classical Examples of Green's Kernels 118 66 Sturm-Liouville Theory 120 67 Eigenfunction Expansions 68 The Connection Between Hilbert-Schmidt and Sturm-Liouville Eigenvalue Problems 123 69 Limitations 124 610 Summary 125 121 7 Iterated Brownian Bridge Kernels: A Green's Kernel Example 127 71 Derivation of Piecewise Polynomial Spline Kernels 127 711 Recall some special Green's kernels 127 712 A family of piecewise polynomial splines of arbitrary odd degree 129 713 Benefits of using a kernel representation for piecewise polynomial splines 72 Derivation of General Iterated Brownian Bridge Kernels 132 73 Properties of Iterated Brownian Bridge Kernels 134 731 Truncation of the Mercer series 134 732 Effects of the boundary conditions 136 733 Convergence orders 139 734 Iterated Brownian bridge kernels on bounded domains 139 735 "Flat" limits 143 736 Summary for functions satisfying homogeneous boundaryconditions 146 131 8 Generalized Sobolev Spaces 147 81 How Native Spaces Were Viewed Until Recently 147 82 Generalized Sobolev Spaces on the Full Space Rd 152 821 Two different kernels for H2(E) 155 822 Higher-dimensional examples 156
An xiv Contents 823 Summary for full-space generalized Sobolev spaces 158 83 Generalized Sobolev Spaces on Bounded Domains 158 831 Modifications of the Brownian bridge kernel: A detailed investigation 160 832 Summary for generalized Sobolev spaces on bounded domains 167 833 An alternative framework for boundary value problems on [a,b] 167 84 Conclusions 168 9 Accuracy and Optimality of Reproducing Kernel Hilbert Space Methods 171 91 Optimality 171 92 Different Types of Error 172 93 The "Standard" Error Bound 172 94 Error Bounds via Sampling Inequalities 175 941 How sampling inequalities lead to error bounds 175 942 Univariate sampling inequalities and error bounds 176 943 Application to iterated Brownian bridge kernels 181 944 Sampling inequalities in higher dimensions 183 95 Dimension-independent error bounds 184 951 Traditional dimension-dependent error bounds 185 952 Worst-case weighted L2 error bounds 185 10 "Flat" Limits 189 101 Introduction 189 102 Kernels with Infinite Smoothness 191 103 Kernels with Finite Smoothness 193 104 Summary and Outlook 197 11 The Uncertainty Principle - Unfortunate Misconception 199 111 Accuracy vs Stability 199 112 Accuracy and Stability 201 12 Alternate Bases 203 121 Data-dependent Basis Functions 204 1211 Standard basis functions 204 1212 Cardinal basis functions 206 1213 Alternate bases via matrix factorization 208 1214 Newton-type basis functions 210 1215 SVD and weighted SVD bases 215 122 Analytical and Numerical Eigenfunctions 217
Contents xv 1221 Eigenfunctions given analytically 218 1222 Eigenfunctions obtained computationally 221 123 Approximation Using Eigenfunctions 124 Other Recent Preconditioning and Alternate Basis Techniques 226 230 243 13 Stable Computation via the Hilbert-Schmidt SVD 231 131 A Formal Matrix Decomposition of K 232 132 Obtaining a Stable Alternate Basis via the Hilbert Schmidt SVD 235 1321 Summary: How to use the Hilbert-Schmidt SVD 241 133 Iterated Brownian Bridge Kernels via the Hilbert-Schmidt SVD 134 Issues with the Hilbert-Schmidt SVD 248 1341 Truncation of the Hilbert-Schmidt series 248 1342 Invertibility of 4>i 250 135 Comparison of Alternate Bases for Gaussian Kernels 252 14 Parameter Optimization 141 Modified Golomb-Weinberger Bound and Kriging Variance 255 256 1411 How to avoid cancelation while computing the power function (kriging variance) 257 1412 How to stably compute the native norm space of the interpolant (Mahalanobis distance) 258 142 Cross-Validation 260 143 Maximum Likelihood Estimation 263 1431 MLE independent of process variance 264 267 1432 MLE with process variance 265 1433 A deterministic derivation of MLE 266 144 Other Approaches to the Selection of Good Kernel Parameters 145 Goals for a Parametrization Judgment Tool 269 Advanced Examples 273 15 Scattered Data Fitting 275 151 Approximation Using Smoothing Splines 276 152 Low-rank Approximate Interpolation 280 153 Interpolation on the Unit Sphere 286 154 Computational Considerations for Scattered Data Fitting 290 1541 The cost of computing/implementing an alternate basis 291 1542 Exploiting structure in kernel computations 292 16 Computer Experiments and Surrogate Modeling 295 161 Surrogate Modeling 295
xvi Contents 162 Experimental Design 297 163 Surrogate Models for Standard Test Functions 298 1631 Piston simulation function 298 1632 Borehole function 304 164 Modeling From Data 306 165 Fitting Empirical Distribution Functions 307 17 Statistical Data Fitting via Gaussian Processes 315 171 Geostatistics 315 172 Anisotropic Data Fitting 324 173 Data Fitting Using Universal Kriging and Maximum Likelihood Estimation 327 18 Machine Learning 335 181 Regularization Networks 336 182 Radial Basis Function Networks 337 1821 Numerical experiments for regression with RBF networks 339 183 Support Vector Machines 343 1831 Linear classification 344 1832 Kernel classification 346 1833 Numerical experiments for classification with kernel SVMs 350 1834 Computational consideration for classification with kernel SVMs 354 1835 Linear support vector regression 358 1836 Nonlinear support vector regression 359 19 Derivatives of Interpolants and Hermite Interpolation 361 191 Differentiating Interpolants 362 1911 Cardinal function representation of derivatives 362 1912 Error bounds for simultaneous approximation 363 1913 Global differentiation matrices 364 1914 Local differentiation matrices 369 192 Hermite Interpolation 1921 Nonsymmetric kernel-based Hermite interpolation 378 1922 Symmetric kernel-based Hermite interpolation 381 1923 Generalized Hermite interpolation via the Hilbert -Schmidt SVD 1924 An example: Gradient interpolation 1925 Kriging interpretation 193 Doing Hermite Interpolation via Derivatives of Eigenfunctions 387 1931 Differentiation of a low-rank eigenfunction approximate interpolant 377 383 384 386 388
Contents xvii 1932 An example: Derivatives of Gaussians eigenfunctions 389 194 Multiphysics Coupling 392 1941 Meshfree coupling 395 1942 An example: coupled 2D heat equation 396 1943 Computational considerations 401 20 Kernel-Based Methods for PDEs 403 201 Collocation for Linear Elliptic PDEs 403 2011 Nonsyinmetric collocation in the standard basis 404 2012 Nonsymmetric collocation using the Hilbert-Schmidt SVD 407 202 Method of Lines 411 203 Method of Fundamental Solutions 416 204 Method of Particular Solutions 420 205 Kernel-based Finite Differences 423 206 Space-Time Collocation 425 21 Finance 431 211 Brownian motion 431 2111 Brownian motion and the Brownian motion kernel 432 2112 Geometric Brownian motion 433 2113 Pricing options and high-dimensional integration 434 2114 A generic error formula for quasi-monte Carlo integration via reproducing kernels 436 2115 Example of asset pricing through quasi-monte Carlo 437 212 Black-Scholes PDEs 440 2121 Single-asset European option through Black-Scholes PDEs 441 2122 Pricing American options 445 Appendix A Collection of Positive Definite Kernels and Their Known Mercer Series 447 Al Piecewise Linear Kernels 447 A 11 Brownian bridge kernel 447 A12 Brownian motion kernel 448 A 13 Another piecewise linear kernel 448 A2 Exponential Kernel 448 A21 Domain: [0,1] 449 A22 Domain: 449 [ L, L] A23 Domain: 449 [0, oo) A3 Other Continuous Kernels 450 A31 Tension spline kernel 450 A32 Relaxation spline kernel 451 A33 Legendre kernel 451
xviii Contents A4 Modified Exponential Kernel 451 A5 Families of Iterated Kernels 452 A51 Iterated Brownian bridge kernels 452 A52 Periodic spline kernels 452 A53 Periodic kernels 453 A54 Chebyshev kernels 453 A6 Kernel for the First Weighted Sobolev Space 454 A 7 Gaussian Kernel 455 A8 Sine Kernel 455 A 9 Zonal Kernels 456 A91 Spherical inverse multiquadric 456 A92 Abel-Poisson kernel 456 Appendix B How To Choose the Data Sites 457 B l Low Discrepancy Designs 458 B2 Optimal Designs in Statistics 460 B3 Optimal Points in Approximation Theory 461 Appendix C A Few Facts from Analysis and Probability 463 Appendix D The GaussQR Repository in Matlab 467 Dl Accessing GaussQR 467 D2 Common functions in GaussQR 468 D3 Full Hilbert-Schmidt SVD sample solver 469 Bibliography 473 Index 505