Positive Definite Kernels: Opportunities and Challenges

Transcription

1 Positive Definite Kernels: Opportunities and Challenges Michael McCourt Department of Mathematical and Statistical Sciences University of Colorado, Denver CUNY Mathematics Seminar CUNY Graduate College April 10, 2014 (CUDenver) Positive Definite Kernels April 10, / 55

2 Acknowledgements I would like to extend my thanks to John Loustau Marcello Lucia michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

3 Solving problems numerically Opportunity As applications continue to increase in complexity, and computer hardware becomes more powerful and inexpensive, better numerical methods will be required to conduct the required simulations. (CUDenver) Positive Definite Kernels April 10, / 55

4 Solving problems numerically Opportunity As applications continue to increase in complexity, and computer hardware becomes more powerful and inexpensive, better numerical methods will be required to conduct the required simulations. My Strategy Kernel-based methods are a useful tool for computations including: Function approximation and interpolation, Integration, Partial differential equations (BVPs), Machine and statistical learning, and Optimization and surrogate modeling. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

5 Topics of Discussion Introduce positive definite kernels and their properties Discuss the scattered data interpolation problem Explain how a kernel basis can be used to solve that problem Explore the impact of different kernel choices in the quality of the approximation Explore various kernel-based schemes for solving PDEs Method of fundamental solutions Kernel collocation RBF-based finite differences michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

6 What is a kernel? Kernel functions Essentially, any function of two variables: K (x, z) is a kernel. I only care about symmetric kernels: K (x, z) = K (z, x). Positive definite kernels can be interpreted as functional analysis analogs of positive definite matrices. v T Kv > 0, or K (x, z)v(x)v(z) dx dz > 0. R n R n A reproducing kernel Hilbert space H K is a space of functions which are in some way generated by the kernel K : K (x, ), f HK = f (x), if f H K. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

7 What is a kernel? Kernel functions Essentially, any function of two variables: K (x, z) is a kernel. I only care about symmetric kernels: K (x, z) = K (z, x). Positive definite kernels can be interpreted as functional analysis analogs of positive definite matrices. v T Kv > 0, or K (x, z)v(x)v(z) dx dz > 0. R n R n A reproducing kernel Hilbert space H K is a space of functions which are in some way generated by the kernel K : K (x, ), f HK = f (x), if f H K. Note: This term K (x, z) is unrelated to the kernel (null space) of a matrix. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

8 Positive Definite Kernels Positive Definite Kernel and Radial Basis Function (RBF) are used interchangeably... but Not all kernels are radial: K (x, z) = min(x, z) xz Not all RBFs are positive definite: φ(r) = r 2 + c 2 Even so, I may switch back and forth between the terms, thus unless otherwise specified. kernel = RBF michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

9 Sample Kernels Because all kernels (for now) are radial, we define r = x z. Gaussians Inverse Multiquadrics C 2 Matérn K (x, z) = exp ( ε 2 r 2) K (x, z) = ε 2 r 2 K (x, z) = (1 + r) exp ( εr) C 4 Wendland K (x, z) = (1 εr) 6 + ( ) 35(εr) εr + 3 michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

10 Gaussian exp ( ε 2 r 2) Inv. Multiquadric ε 2 r 2 C 2 Matérn These RBFs my be thought of as a similarity measure. (1 + r) exp ( εr) C 4 Wendland (1 εr) 6 ( + 35(εr) εr + 3 ) michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

11 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

12 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. Consider kernel-based interpolation - the approximation is a linear combination of RBFs (not polynomials) s(x) = N c j K (x, x j ), j=1 x Ω R d with K : Ω Ω R a positive definite kernel. To find c j solve the interpolation equations s(x i ) = f (x i ) = y i, i = 1,..., N which gives a linear system Kc = y; K is symmetric positive definite. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

13 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

14 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

15 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

16 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

17 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

18 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

19 Fundamental Application (Scattered Data Fitting) Given data (x j, y j ), j = 1,..., N, with x j R d, y j R, find a (continuous) function s such that s(x j ) = y j, j = 1,..., N. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

20 Kernels vs. Polynomials A polynomial interpolation scheme (in 1D) looks like: p f (x) = a 1 + a 2 x + a 3 x a n x n 1. The kernel-based scheme takes the form s(x) = c 1 K (x, x 1 ) + c 2 K (x, x 2 ) + c 3 K (x, x 3 ) c n K (x, x n ). Why is this kernel (RBF) approach preferable? michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

21 Kernels vs. Polynomials A polynomial interpolation scheme (in 1D) looks like: p f (x) = a 1 + a 2 x + a 3 x a n x n 1. The kernel-based scheme takes the form s(x) = c 1 K (x, x 1 ) + c 2 K (x, x 2 ) + c 3 K (x, x 3 ) c n K (x, x n ). Why is this kernel (RBF) approach preferable? Uniqueness is guaranteed in higher dimensions. RBF methods are often referred to as meshfree. Optimality of these methods is provable in some circumstances. Properties such as smoothness are variable. The locality of the basis functions can be parameterized. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

22 RBFs in applications: Why aren t there more? For many problems, kernel-based approximations are appropriate; sometimes they would be optimal. Why are they not more prevalent? michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

23 RBFs in applications: Why aren t there more? For many problems, kernel-based approximations are appropriate; sometimes they would be optimal. Why are they not more prevalent? Possible barriers to widespread RBF usage 1 Mathematical complexity 2 Stability concerns 3 Computational cost 4 Presence of one or more free parameters michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

24 Gaussian Kernel Interpolation Probably the most common kernel in applications is the Gaussian ( K (x, z) = exp ε 2 x z 2), The ε value determines the locality of the basis functions. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

25 Gaussian Kernel Interpolation Probably the most common kernel in applications is the Gaussian ( K (x, z) = exp ε 2 x z 2), The ε value determines the locality of the basis functions. Opportunity Gaussians are great choices for interpolating very smooth functions. Problem For some (potentially very accurate) choices of ε, the interpolation matrix K may be very ill-conditioned. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

26 Scattered Data Fitting with Gaussians Gaussians can be spectrally accurate, allowing for high accuracy with smaller N (number of input points). For some applications, the cost of producing a data point is significant, so this is useful. The choice of shape parameter ε can have a great effect on the accuracy and condition of the interpolant. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

27 Naive Gaussian Interpolation Results As ε 0 the K (, x i ) and K (, x j ) start to resemble each other, producing a matrix which looks increasingly like a matrix of all ones michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

28 Better Gaussian Interpolation Results This is what we want to see: No instability for smaller ε Recovering the ε 0 polynomial limit [Fornberg, 2002] How can we produce this result? michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

29 Eigenfunction (Mercer) Series Using Hilbert-Schmidt Theory K (x, z) = e ε2 (x z) 2 = λ n ϕ n (x)ϕ n (z) n=0 michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

30 Eigenfunction (Mercer) Series Using Hilbert-Schmidt Theory K (x, z) = e ε2 (x z) 2 = λ n ϕ n (x)ϕ n (z) where α λ n = 2 ( ε 2 ) n α 2 + δ 2 + ε 2 α 2 + δ 2 + ε 2, ϕ n (x) = γ n e δ2 x 2 H n (αβx) with H n Hermite polynomials, ( ( ) ) 1 2ε 2 4 β = 1 +, γ n = α n=0 β 2 n Γ(n + 1), δ2 = α2 2 ( ) β 2 1. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

31 Eigenfunction (Mercer) Series Using Hilbert-Schmidt Theory K (x, z) = e ε2 (x z) 2 = λ n ϕ n (x)ϕ n (z) where α λ n = 2 ( ε 2 ) n α 2 + δ 2 + ε 2 α 2 + δ 2 + ε 2, ϕ n (x) = γ n e δ2 x 2 H n (αβx) with H n Hermite polynomials, ( ( ) ) 1 2ε 2 4 β = 1 +, γ n = α n=0 β 2 n Γ(n + 1), δ2 = α2 2 ( ) β 2 1. These eigenpairs solve the Hilbert-Schmidt eigenvalue problem, e ε2 (x z) 2 ϕ n (z)ρ(z) dz = λ n ϕ n (x), ρ(x) = α π e α2 x 2. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

32 Eigenfunctions in the ε 0 limit Studying the limits as ε 0 shows ( lim β = lim ε 0 ε ( ) ) 1 2ε 2 4 α ) α 2 ( lim ε 0 δ2 = lim β 2 1 ε 0 2 which in turn allows us to conclude that = 1, lim ε 0 ε 2, lim ϕ n(x) = lim γ n e δ2 x 2 H n (αβx) = γ n H n (αx). ε 0 ε 0 This supports the polynomial limit proved more than 10 years ago. Infinite Smoothness Only This limit is true only for analytic kernels - kernels with finite smoothness have (often) a piecewise polynomial spline limit. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

33 Eigenvalues in the ε 0 limit As ε 0, the eigenvalues λ n decay increasingly fast lim λ α n = lim 2 ( ε 2 ε 0 ε 0 α 2 + δ 2 + ε 2 α 2 + δ 2 + ε 2 ( ) α 2 1 n = α 2 α 2 lim ε 0 ε2n This decay is a major cause of the ill-conditioning in the kernel interpolation problem (we will see this). Avoiding Instability We can use the eigenexpansion of the Gaussians to create a stable basis for them. ) n michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

34 Developing the Hilbert-Schmidt SVD (GaussQR) The eigenfunction structure is K (x, z) = λ m ϕ m (x)ϕ m (z), m=0 λ 1 = (... ϕ 1 (x) ϕ N (x) ) ϕ 1 (z). λ N ϕ N (z),.... In nice, compressed, vector notation, K (x, z) = φ(x) T Λφ(x) michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

35 Developing the Hilbert-Schmidt SVD (GaussQR) The scattered data fitting problem asks us to find an interpolant s(x) = N c k K (x, x k ), k=1 = k(x) T c. The coefficient vector c is defined by Kc = y: K (x 1, x 1 ) K (x 1, x N ) c 1 k(x 1 ) T.. =. K (x N, x 1 ) K (x N, x N ) k(x N ) T which is just s(x j ) = y j, 1 j N. We can write c N s(x) = k(x) T K 1 y. c = y 1. y N, michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

36 Developing the Hilbert-Schmidt SVD (GaussQR) We can rewrite each row of K (our basis functions) as k(x) T = ( K (x, x 1 ) K (x, x N ) ) = ( φ(x) T Λφ(x 1 ) φ(x) T Λφ(x N ) ) = φ(x) T Λ ( φ(x 1 ) φ(x N ) ). Stacking these gives the K matrix k(x 1 ) T φ(x 1 ) T K = =. k(x N ) T. φ(x N ) T which I will write as a matrix eigenexpansion Λ ( φ(x 1 ) φ(x N ) ), K = ΦΛΦ T, and k(x) T = φ(x) T ΛΦ. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

37 Developing the Hilbert-Schmidt SVD (GaussQR) K = ΦΛΦ T involves infinite sized matrices... Break Φ and Λ into blocks. ( ) Λ1 Φ = (Φ 1 Φ 2 ), Λ =, Λ 2 so that Φ 1 and Λ 1 are N N. Let s rewrite our basis using this structure ( ) ( ) k(x) T = φ(x) T ΛΦ T = φ(x) T Λ1 Φ T 1 Λ 2 Φ T 2 ( ) = φ(x) T I N Λ 2 Φ T 2 Φ T 1 Λ 1 1. Λ 1 Φ T 1, Define a new function to write ψ(x) T = φ(x) T ( I N Λ 2 Φ T 2 Φ T 1 Λ 1 1 k(x) T = ψ(x) T Λ 1 Φ T 1. ) michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

38 Developing the Hilbert-Schmidt SVD (GaussQR) Use this to rewrite K, which was just rows of k: k(x 1 ) T ψ(x 1 ) T K = =. k(x N ) T. ψ(x N ) T Λ 1 Φ T 1 = ΨΛ 1Φ T 1. This K = ΨΛ 1 Φ T 1 is called the Hilbert-Schmidt SVD. Using this in to s... s(x) = k(x) T K 1 y = ψ(x) T Λ 1 Φ T 1 Φ T 1 Λ 1 1 Ψ 1 y = ψ(x) T Ψ 1 y. and all the ill-conditioning from Λ 1 which would normally have appeared in K 1 has been resolved. Our new stable basis is ψ. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

39 A New, More Stable, Basis (CUDenver) Positive Definite Kernels April 10, / 55

40 A New, More Stable, Basis This stable basis ψ allows us to evaluate the interpolant s(x) = ψ(x) T Ψ 1 y, rather than s(x) = k(x) T (ΨΛ 1 Φ T 1 ) 1 y. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

41 Solving PDEs with Kernels We have used a linear combination of kernels to solve a scattered data approximation problem. A similar strategy can be used to solve boundary value problems of the form Ls = f, Bs = g, on interior, on boundary, for L a linear differential operator. Note Nonlinear and time-stepping problems can also be solved, though we will not talk about them here. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

42 Solving PDEs numerically Question What does it mean to solve a PDE numerically? Lu = f, Bu = g, on interior, on boundary. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

43 Solving PDEs numerically Question What does it mean to solve a PDE numerically? Lu = f, Bu = g, on interior, on boundary. Answer Different communities view different results as solutions. Approximate solution values at chosen points. Approximate Quantities of Interest (QoI) within larger simulations. Approximate solution function which can be evaluated as needed. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

44 Solving PDEs numerically: Approximate solution Our focus in this talk will be on constructing a function which approximates the true solution. Goal Create a function s(x) = N c k K (x, z k ) k=1 which approximately solves the PDE in some sense we deem appropriate. Appropriate approximation How do we enforce the PDE and BC on our s? What are these z k centers? michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

45 Method of Fundamental Solutions The first (1950s??) meshfree method for solving PDEs is called the Method of Fundamental Solutions (MFS). Assumption Our L operator and our domain support an analytically known Green s function G(x, z), such that LG(, z) = δ( z). Strategy L = 2 - G(x, z) = 1/ x z L = 2 λ 2 I - G(x, z) is a Bessel Function (2nd kind) Use K (x, z k ) = G(x, z k ) as the basis functions. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

46 Method of Fundamental Solutions MFS is appropriate for homogeneous PDEs: Ls = 0, Bs = g, on interior, on boundary. By choosing M s(x) = c k G(x, z k ) k=1 we see the interior condition is immediately satisfied: M M M Ls(x) = L c k G(x, z k ) = c k LG(x, z k ) = c k δ(x z k ) = 0, k=1 k=1 k=1 so long as z k is not in the interior. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

47 Method of Fundamental Solutions Because of this design, only the c k are chosen to satisfy the BC. To do this, we fix N collocation points on the boundary, at which we enforce Bs = g. BG(x 1, z 1 ) BG(x 1, z M ) c 1 g(x 1 ).. =. BG(x N, z M ) BG(x N, z M ) g(x N ) c M The shorthand BG(x i, z j ) means BG(x, z J ) x=xi. If M = N this is a square system. Usually, we set M < N and solve the least square problem. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

48 Method of Fundamental Solutions: Overview 1 Choose N collocation points on the boundary 2 Form a solution which is a linear combination of Green s functions There should be M N such functions 3 Center the Green s function outside of the solution domain 4 Solve the appropriate linear system (least squares system) to correctly form the solution A theorem exists which guarantees convergence for M = N as N, subject to possible ill-conditioning. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

49 Method of Fundamental Solutions - Point Distribution Collocation points are located on the boundary. G(, z k ) is centered outside of the domain. No collocation points needed in the interior. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

50 Method of Fundamental Solutions - Point Distribution michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

51 MFS Convergence - Modified Helmholtz Problem michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

52 Method of Fundamental Solutions - Review MFS benefits An R d PDE is solved as a R d 1 interpolation. Cheaper, and easier analysis. Choosing M N is cheaper and (often) more stable. No specific point distribution is required. MFS problems Collocation and source points must be chosen. There may be bad choices. The collocation matrix is probably dense. Only appropriate for homogeneous PDEs. The Method of Particular Solutions exists for nonhomogeneous problems. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

53 Kernel Collocation Instead of Green s functions as our solution basis, we will choose other positive definite kernels (Gaussians). Unlike MFS, the collocation points must exist on the interior as well as the boundary, because Ls = f will not be automatically satisfied. Again, two sets of points must be determined: x i - The collocation points (interior/boundary) z j - The kernel centers michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

54 Kernel Collocation - Point Selection Historical note Almost all papers use x k = z k, that is, the centers and collocation points are the same - this matches the interpolation setting. Recently, researchers have challenged this strategy, but it is still the standard and the only setting we consider here. Matching the centers and collocation points (N = N L + N B ) yields: LK (x 1, x 1 ) LK (x 1, x N ) c 1 f (x 1 ).. LK (x NL, x 1 ) LK (x NL, x N ). BK (x NL +1, x 1 ) BK (x NL +1, x N ) = f (x NL ). g(x NL +1).. BK (x NL +N B, x 1 ) BK (x NL +N B, x N ) g(x NL +N B ) c N michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

55 Kernel Collocation - Stable Basis Using appropriate shorthand, we can rewrite this system as ( ) ( ) KL f c = g K B The HS-SVD allows us to write K L = Ψ L Λ 1 Φ T 1 K B = Ψ B Λ 1 Φ T 1 because Λ 1 Φ T 1 is defined by the centers, not the collocation points. This produces ( ) ( ) ΨL Λ 1 Φ T f 1 c = g Ψ B and the solution s is evaluated as ( ) 1 ( ) s(x) = ψ(x) T ΨL f Ψ B g = ψ(x) T Ψ 1 L,B y f,g. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

56 Stable Gaussians for Boundary Value Problems Implementing the stable basis for this system will avoid the ill-conditioning which would otherwise plague the collocation solution. This example is a 2D Helmholtz problem, comparing Gaussian collocation to Trefethen s polynomial method and the standard Gaussian kernel collocation method. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

57 Evaluating the Collocation Solution Suppose that we wanted to evaluate our PDE solution at our collocation points x 1,..., x N, similarly to a finite difference method: s = s(x 1 ). s(x N ) = = ψ(x 1 ) T Ψ 1. ψ T (x N )Ψ 1 ψ(x 1 ) T. ψ(x N ) T L,B y f,g L,B y f,g Ψ 1 L,B y f,g = ΨΨ 1 L,B y f,g This relationship s = ΨΨ 1 L,B y f,g suggests that ΨΨ 1 L,B plays a similar role as the standard finite difference matrix (plus boundary conditions). michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

58 RBF Differentiation Matrices Observation Applying ΨΨ 1 L,B y f,g evaluates a function (the approximate solution s) when given data (the PDE s RHS y f,g ). Idea Could we reverse the process... that is, given function values, could we evaluate derivatives of that function? Given data y at points x 1,..., x N, we can form an associated interpolant s = ψ( ) T Ψ 1 y. Applying a derivative operator D to this produces Ds = Dψ( ) T Ψ 1 y. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

59 RBF Differentiation Matrices Evaluating the derivative of our interpolant at the interpolation points: Ds(x 1 ) Dψ(x 1 ) T Ψ 1 y s D =. =. Ds(x N ) Dψ(x N ) T Ψ 1 y = Dψ(x 1 ) T. Dψ(x N ) T Ψ 1 y = Ψ D Ψ 1 y Our differentiation matrix Ψ D Ψ 1 takes in data from a function f (evaluated at x 1,..., x N and stored in y) and returns an approximation of Df at the same locations. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

60 RBF Pseudospectral Solver We can expand our notation to evaluate Ds at any location, not just the input data locations. Ψ I L Ψ B B - Evaluates Lψ at the interior collocation points - Evaluates Bψ at the boundary collocation points Using this, the PDE, in pseudospectral form could be written as ( Ψ I L Ψ 1 ) ( ) f Ls = f s =, because g Bs = g Ψ B B Ψ 1 This used by Trefethen to describe the differentiation matrix structure of polynomial collocation methods. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

61 RBF Finite Differences This design of RBF Differentiation matrices motivates one of the most promising new developments in meshfree PDE solvers: RBF-FD. Strategy Define a localized differential operator (analogous to finite differences) which can be used approximate derivatives. Benefits No prescribed point distribution would be required (e.g., uniform points for finite differences, Chebyshev nodes for polynomials) The locality of the operator is a free parameter, and could even vary within a single PDE. Unlike collocation based operators, RBF-FD would be sparse. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

62 RBF Finite Differences - Point Selection Each point uses nearby points to approximate a differential operator. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

63 RBF Finite Differences - Structure For each x k, nearby points, X k, are chosen to approximate the derivatives, and a one row differentiation matrix is computed for each point: Ψ x k L ( Ψ X k ) 1. This matrix has the effect of taking in nearby function values and producing a linear combination of them to approximate the L operator at x k. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

64 RBF Finite Differences - Sparsity Pattern This seems like a really crummy structure which will be much worse than finite differences... michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

65 RBF Finite Differences - Improved Sparsity Pattern But a simple reordering of our indices produces the familiar banded structure. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

66 RBF Finite Differences - Convergence Multiple meanings As the number of points used in computing the finite difference stencil increases, this method should converge to traditional collocation. For a fixed number of neighbors, but an increasingly large number of collocation points, convergence should occur until saturation. Eventually the quality will be bounded by the limited domain over which points are used to approximate the derivatives. A problem of scale For increasingly dense points, different ε shape parameters (which control the kernel locality) may be needed to maintain convergence order. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

67 RBF Finite Differences - Convergence michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

68 RBF Finite Differences - Overview 1 A neighborhood is defined around each collocation point. 2 A linear combination of the points in that neighborhood are used to approximate the derivative at that point. 3 This produces a sparse operator which can be used to solve boundary value problems. 4 Choosing a neighborhood optimally, and then choosing the kernel to use for the approximation in the neighborhood (ε) is still an open problem. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55

69 Review Positive definite kernels (RBFs) can be used as a basis for scattered data interpolation and PDE collocation. Some kernels see severe ill-conditioning. A change of basis using the Hilbert-Schmidt SVD can circumvent the traditional ill-conditioning; mostly Gaussians, for now... The Method of Fundamental Solutions is the original kernel-based PDE solver. Collocation schemes can be constructed using the same strategy as interpolation. Less theory exists in this setting. Using this concept, we can define RBF differentiation matrices. RBF-FD produce localized differential operators, comparable to standard PDE solvers, but in a meshfree setting. michael.mccourt@ucdenver.edu (CUDenver) Positive Definite Kernels April 10, / 55