MATH 590: Meshfree Methods

Transcription

1 MATH 590: Meshfree Methods Chapter 2: Radial Basis Function Interpolation in MATLAB Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Fall 2010 MATH 590 Chapter 2 1

2 Outline 1 Radial (Basis) Functions 2 Radial Basis Function Interpolation fasshauer@iit.edu MATH 590 Chapter 2 2

3 Outline Radial (Basis) Functions 1 Radial (Basis) Functions 2 Radial Basis Function Interpolation fasshauer@iit.edu MATH 590 Chapter 2 3

4 Next goals Radial (Basis) Functions Get a feel for RBFs before we study them in detail Want to overcome limitations of distance matrix interpolation, but keep overall structure: combine distance matrix interpolants with good basic functions fasshauer@iit.edu MATH 590 Chapter 2 4

5 Radial (Basis) Functions Example The Gaussian ϕ(r) = e (εr)2, r R, has a shape parameter ε related to the variance σ 2 of the normal distribution: ε 2 = 1/(2σ 2 ). fasshauer@iit.edu MATH 590 Chapter 2 5

6 Radial (Basis) Functions Example The Gaussian ϕ(r) = e (εr)2, r R, has a shape parameter ε related to the variance σ 2 of the normal distribution: ε 2 = 1/(2σ 2 ). Compose the Gaussian with Euclidean distance function 2 and get Φ k (x) = e ε2 x x k 2 2, x R s, where x k R s is some fixed center or knot. fasshauer@iit.edu MATH 590 Chapter 2 5

7 Radial (Basis) Functions Example The Gaussian ϕ(r) = e (εr)2, r R, has a shape parameter ε related to the variance σ 2 of the normal distribution: ε 2 = 1/(2σ 2 ). Compose the Gaussian with Euclidean distance function 2 and get Φ k (x) = e ε2 x x k 2 2, x R s, where x k R s is some fixed center or knot. Remark Note that ϕ is univariate, but Φ k is a multivariate function such that Φ k (x) = ϕ( x x k 2 ) radial basis function (RBF). fasshauer@iit.edu MATH 590 Chapter 2 5

8 Radial (Basis) Functions Definition A function Φ : R s R is called radial provided there exists a univariate function ϕ : [0, ) R such that Φ(x) = ϕ(r), where r = x, and is some norm on R s usually the Euclidean norm. fasshauer@iit.edu MATH 590 Chapter 2 6

9 Radial (Basis) Functions Definition A function Φ : R s R is called radial provided there exists a univariate function ϕ : [0, ) R such that Φ(x) = ϕ(r), where r = x, and is some norm on R s usually the Euclidean norm. Thus, for a radial function x 1 = x 2 = Φ(x 1 ) = Φ(x 2 ), x 1, x 2 R s. fasshauer@iit.edu MATH 590 Chapter 2 6

10 Radial (Basis) Functions Definition A function Φ : R s R is called radial provided there exists a univariate function ϕ : [0, ) R such that Φ(x) = ϕ(r), where r = x, and is some norm on R s usually the Euclidean norm. Thus, for a radial function Example x 1 = x 2 = Φ(x 1 ) = Φ(x 2 ), x 1, x 2 R s. The Euclidean distance function Φ(x) = x 2 (or ϕ(r) = r) is a special case of a radial (basis) function. Terminology: ϕ: basic function, Φ k ( 2 ) (centered at x k ): radial basis function fasshauer@iit.edu MATH 590 Chapter 2 6

11 Radial (Basis) Functions Figure: Gaussian with ε = 1 (left) and ε = 3 (right) centered at the origin. A smaller value of ε (i.e., larger variance) causes the function to become flatter, so it s like an inverse length scale. Increasing ε leads to a more peaked RBF. fasshauer@iit.edu MATH 590 Chapter 2 7

12 Radial (Basis) Functions Figure: Gaussian with ε = 1 (left) and ε = 3 (right) centered at the origin. A smaller value of ε (i.e., larger variance) causes the function to become flatter, so it s like an inverse length scale. Increasing ε leads to a more peaked RBF. The choice of ε influences both accuracy and numerical stability of the solution to our interpolation problem (details later). fasshauer@iit.edu MATH 590 Chapter 2 7

13 Radial (Basis) Functions Some properties of radial function interpolants Invariance under all Euclidean transformations (i.e., translations, rotations, and reflections). It doesn t matter whether we first compute the RBF interpolant and then apply a Euclidean transformation, or if we first transform the data and then compute the interpolant. True since Euclidean transformations are characterized by orthogonal transformation matrices and are therefore 2-norm-invariant. Invariance under translation, rotation and reflection is often desirable in applications. fasshauer@iit.edu MATH 590 Chapter 2 8

14 Radial (Basis) Functions Some properties of radial function interpolants Invariance under all Euclidean transformations (i.e., translations, rotations, and reflections). It doesn t matter whether we first compute the RBF interpolant and then apply a Euclidean transformation, or if we first transform the data and then compute the interpolant. True since Euclidean transformations are characterized by orthogonal transformation matrices and are therefore 2-norm-invariant. Invariance under translation, rotation and reflection is often desirable in applications. Insensitive to the dimension s of the space in which the data sites lie. Instead of having to deal with a multivariate function Φ we can work with the same univariate function ϕ for all choices of s. fasshauer@iit.edu MATH 590 Chapter 2 8

15 The Kernel Notation Radial (Basis) Functions Sometimes we use more general (reproducing) kernels K. A kernel is a function of two variables K : R s R s R, K : (x, y) K (x, y) We will discuss the reproducing concept later. fasshauer@iit.edu MATH 590 Chapter 2 9

16 Radial (Basis) Functions Kernels can be radial: (also called isotropic) K (x, y) = κ( x y ), x, y R s, κ : R + 0 R fasshauer@iit.edu MATH 590 Chapter 2 10

17 Radial (Basis) Functions Kernels can be radial: (also called isotropic) K (x, y) = κ( x y ), x, y R s, κ : R + 0 R (only) translation invariant: (also called stationary) K (x, y) = K (x y), x, y R s, K : R s R fasshauer@iit.edu MATH 590 Chapter 2 10

18 Radial (Basis) Functions Kernels can be radial: (also called isotropic) K (x, y) = κ( x y ), x, y R s, κ : R + 0 R (only) translation invariant: (also called stationary) K (x, y) = K (x y), x, y R s, K : R s R some other special form, e.g., zonal: K (x, y) = κ(x y), x, y S 2, κ : [ 1, 1] R anisotropic (but stationary): K (x, y) = s K d (x d y d ), x d, y d R, K d : R R d=1 fasshauer@iit.edu MATH 590 Chapter 2 10

19 Radial (Basis) Functions Kernels can be radial: (also called isotropic) K (x, y) = κ( x y ), x, y R s, κ : R + 0 R (only) translation invariant: (also called stationary) K (x, y) = K (x y), x, y R s, K : R s R some other special form, e.g., zonal: K (x, y) = κ(x y), x, y S 2, κ : [ 1, 1] R anisotropic (but stationary): K (x, y) = s K d (x d y d ), x d, y d R, K d : R R d=1 very general: K (x, y), x, y R s, K : R s R s R A nice paper that discusses different classes of kernels from a statistical perspective is [Genton (2001)]. fasshauer@iit.edu MATH 590 Chapter 2 10

20 Examples Radial (Basis) Functions Radial kernel: K (x, y) = κ( x y ) = e ε2 x y 2, κ(r) = e ε2 r 2 fasshauer@iit.edu MATH 590 Chapter 2 11

21 Examples Radial (Basis) Functions Radial kernel: K (x, y) = κ( x y ) = e ε2 x y 2, κ(r) = e ε2 r 2 Anisotropic translation invariant kernel: s K (x, y) = e ε2 d (x d y d ) 2 = e s d=1 ε2 d (x d y d ) 2 d=1 becomes radial if ε d = ε for all d = 1,..., s fasshauer@iit.edu MATH 590 Chapter 2 11

22 Examples Radial (Basis) Functions Radial kernel: K (x, y) = κ( x y ) = e ε2 x y 2, κ(r) = e ε2 r 2 Anisotropic translation invariant kernel: s K (x, y) = e ε2 d (x d y d ) 2 = e s d=1 ε2 d (x d y d ) 2 d=1 becomes radial if ε d = ε for all d = 1,..., s Zonal kernel: 1 K (x, y) = κ(x y) = 1 + ε 2 2εx y, κ(t) = ε 2 2εt fasshauer@iit.edu MATH 590 Chapter 2 11

23 Examples Radial (Basis) Functions Radial kernel: K (x, y) = κ( x y ) = e ε2 x y 2, κ(r) = e ε2 r 2 Anisotropic translation invariant kernel: s K (x, y) = e ε2 d (x d y d ) 2 = e s d=1 ε2 d (x d y d ) 2 d=1 becomes radial if ε d = ε for all d = 1,..., s Zonal kernel: 1 K (x, y) = κ(x y) = 1 + ε 2 2εx y, κ(t) = ε 2 2εt General kernel: K (x, y) = min{x, y} xy, s = 1 (Brownian bridge) fasshauer@iit.edu MATH 590 Chapter 2 11

24 Outline Radial Basis Function Interpolation 1 Radial (Basis) Functions 2 Radial Basis Function Interpolation fasshauer@iit.edu MATH 590 Chapter 2 12

25 Radial Basis Function Interpolation Instead of distance matrices we now use a radial basis function expansion to solve the scattered data interpolation problem by assuming N P f (x) = c k ϕ ( x x k 2 ), x R s. (1) k=1 fasshauer@iit.edu MATH 590 Chapter 2 13

26 Radial Basis Function Interpolation Instead of distance matrices we now use a radial basis function expansion to solve the scattered data interpolation problem by assuming N P f (x) = c k ϕ ( x x k 2 ), x R s. (1) k=1 Using the interpolation conditions P f (x i ) = f (x i ), i = 1,..., N, we get c k from ϕ ( x 1 x 1 2 ) ϕ ( x 1 x 2 2 )... ϕ ( x 1 x N 2 ) c 1 ϕ ( x 2 x 1 2 ) ϕ ( x 2 x 2 2 )... ϕ ( x 2 x N 2 ) c = ϕ ( x N x 1 2 ) ϕ ( x N x 2 2 )... ϕ ( x N x N 2 ) c N f (x 1 ) f (x 2 ). f (x N ) fasshauer@iit.edu MATH 590 Chapter 2 13

27 Radial Basis Function Interpolation Instead of distance matrices we now use a radial basis function expansion to solve the scattered data interpolation problem by assuming N P f (x) = c k ϕ ( x x k 2 ), x R s. (1) k=1 Using the interpolation conditions P f (x i ) = f (x i ), i = 1,..., N, we get c k from ϕ ( x 1 x 1 2 ) ϕ ( x 1 x 2 2 )... ϕ ( x 1 x N 2 ) c 1 ϕ ( x 2 x 1 2 ) ϕ ( x 2 x 2 2 )... ϕ ( x 2 x N 2 ) c = ϕ ( x N x 1 2 ) ϕ ( x N x 2 2 )... ϕ ( x N x N 2 ) c N f (x 1 ) f (x 2 ). f (x N ) Question: For what type of basic functions ϕ is the system matrix non-singular? fasshauer@iit.edu MATH 590 Chapter 2 13

28 Radial Basis Function Interpolation Setup for Example Use s = 2 Basic function: Gaussians or linear function ϕ(r) = r Code in RBFInterpolation2D.m written for 2D (can easily be generalized to sd), and uses DistanceMatrix.m. Use Franke s function f (x, y) = 3 4 e 1/4((9x 2)2 +(9y 2) 2) e (1/49)(9x+1)2 (1/10)(9y+1) e 1/4((9x 7)2 +(9y 3) 2) 1 5 e (9x 4)2 (9y 7) 2 Use Halton data sites (get others easily from CreatePoints) Compute errors on uniform grid fasshauer@iit.edu MATH 590 Chapter 2 14

29 Radial Basis Function Interpolation Figure: Franke s test function. fasshauer@iit.edu MATH 590 Chapter 2 15

30 Radial Basis Function Interpolation Program (RBFInterpolation2D.m) 1 rbf exp(-(e*r).^2); ep = 21.1; 2 f1 0.75*exp(-((9*x-2).^2+(9*y-2).^2)/4); 3 f2 0.75*exp(-((9*x+1).^2/49+(9*y+1).^2/10)); 4 f3 0.5*exp(-((9*x-7).^2+(9*y-3).^2)/4); 5 f4 0.2*exp(-((9*x-4).^2+(9*y-7).^2)); 6 testfunction f1(x,y)+f2(x,y)+f3(x,y)-f4(x,y); 7 N = 1089; gridtype = h ; 8 dsites = CreatePoints(N,2,gridtype); 9 ctrs = dsites; 10 neval = 40; M = neval^2; 11 epoints = CreatePoints(M,2, u ); 12 rhs = testfunction(dsites(:,1),dsites(:,2)); 13 DM_data = DistanceMatrix(dsites,ctrs); 14 IM = rbf(ep,dm_data); 15 DM_eval = DistanceMatrix(epoints,ctrs); 16 EM = rbf(ep,dm_eval); 17 Pf = EM * (IM\rhs); 18 exact = testfunction(epoints(:,1),epoints(:,2)); 19 maxerr = norm(pf-exact,inf) 20 rms_err = norm(pf-exact)/neval fasshauer@iit.edu MATH 590 Chapter 2 16

31 Radial Basis Function Interpolation Stationary and Non-Stationary Approximation Remark Note the conflicting use of the term stationary: here vs. statistics = translation-invariant fasshauer@iit.edu MATH 590 Chapter 2 17

32 Radial Basis Function Interpolation Stationary and Non-Stationary Approximation Remark Note the conflicting use of the term stationary: here vs. statistics = translation-invariant We usually verify convergence of a method numerically by looking at a sequence of experiments with increasingly larger sets of data. fasshauer@iit.edu MATH 590 Chapter 2 17

33 Radial Basis Function Interpolation Stationary and Non-Stationary Approximation Remark Note the conflicting use of the term stationary: here vs. statistics = translation-invariant We usually verify convergence of a method numerically by looking at a sequence of experiments with increasingly larger sets of data. Non-stationary approximation: Use one fixed value of ε for all of the experiments. fasshauer@iit.edu MATH 590 Chapter 2 17

34 Radial Basis Function Interpolation Stationary and Non-Stationary Approximation Remark Note the conflicting use of the term stationary: here vs. statistics = translation-invariant We usually verify convergence of a method numerically by looking at a sequence of experiments with increasingly larger sets of data. Non-stationary approximation: Use one fixed value of ε for all of the experiments. Stationary approximation: Scale the shape parameter ε according to the fill distance (or meshsize) h so that we end up using peaked basis functions for densely spaced data and flat basis functions for coarsely spaced data. More details later fasshauer@iit.edu MATH 590 Chapter 2 17

35 Fill distance Radial Basis Function Interpolation The fill distance (or covering radius) is usually defined as h = h X,Ω = sup min x x j 2. (2) x j X x Ω fasshauer@iit.edu MATH 590 Chapter 2 18

36 Fill distance Radial Basis Function Interpolation The fill distance (or covering radius) is usually defined as h = h X,Ω = sup min x x j 2. (2) x j X x Ω It indicates how well the data in the set X fill out the domain Ω. fasshauer@iit.edu MATH 590 Chapter 2 18

37 Fill distance Radial Basis Function Interpolation The fill distance (or covering radius) is usually defined as h = h X,Ω = sup min x x j 2. (2) x j X x Ω It indicates how well the data in the set X fill out the domain Ω. In MATLAB: hx = max(min(dm_eval )) (3) fasshauer@iit.edu MATH 590 Chapter 2 18

38 Fill distance Radial Basis Function Interpolation The fill distance (or covering radius) is usually defined as h = h X,Ω = sup min x x j 2. (2) x j X x Ω It indicates how well the data in the set X fill out the domain Ω. In MATLAB: hx = max(min(dm_eval )) (3) Remark Since min works along columns of a matrix, transposition of the non-symmetric evaluation matrix corresponds to finding for each evaluation point the distance to the corresponding closest data site, and then setting h X,Ω as the worst of those distances. fasshauer@iit.edu MATH 590 Chapter 2 18

39 Radial Basis Function Interpolation y h X x Figure: The fill distance for N = 25 Halton points (h X,Ω ). It s the radius of the largest possible empty ball that can be placed among the data locations inside Ω. fasshauer@iit.edu MATH 590 Chapter 2 19

40 Radial Basis Function Interpolation Test of non-stationary interpolation Fix large ε to prevent severe ill-conditioning with large N (if ε = 1, then N = 25 already very ill-conditioned) Consequence: very localized basis functions that don t capture enough information on small point sets (see left plot) Figure: Gaussian RBF interpolant with ε = 21.1 at N = 289 (left) and at N = 1089 Halton points (right). fasshauer@iit.edu MATH 590 Chapter 2 20

41 Radial Basis Function Interpolation Gaussian Distance matrix k N RMS-error max-error RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-003 Table: Non-stationary RBF interpolation to Franke s function using Gaussians (ε = 21.1) and Euclidean distance matrices. fasshauer@iit.edu MATH 590 Chapter 2 21

42 Radial Basis Function Interpolation Gaussian Distance matrix k N RMS-error max-error RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-003 Table: Non-stationary RBF interpolation to Franke s function using Gaussians (ε = 21.1) and Euclidean distance matrices. Remark Surprising observation in above example: distance matrix fit more accurate than Gaussian. fasshauer@iit.edu MATH 590 Chapter 2 21

43 Radial Basis Function Interpolation Gaussian Distance matrix k N RMS-error max-error RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-003 Table: Non-stationary RBF interpolation to Franke s function using Gaussians (ε = 21.1) and Euclidean distance matrices. Remark Surprising observation in above example: distance matrix fit more accurate than Gaussian. Would expect Gaussian to be more accurate find a better ε fasshauer@iit.edu MATH 590 Chapter 2 21

44 Radial Basis Function Interpolation Same test function and Gaussians as before. Figure: Maximum (blue) and RMS (red) errors vs. ε for 81 (top left), 289 (top right), 1089 (bottom left), and 4225 Halton points (bottom right). MATH 590 Chapter 2 22

45 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). fasshauer@iit.edu MATH 590 Chapter 2 23

46 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). Error curves are not monotonic. fasshauer@iit.edu MATH 590 Chapter 2 23

47 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). Error curves are not monotonic. Optimal value of ε seems to exist which minimizes errors. fasshauer@iit.edu MATH 590 Chapter 2 23

48 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). Error curves are not monotonic. Optimal value of ε seems to exist which minimizes errors. For small enough values of ε the computational results become unpredictable, and the error curves become erratic (ill-conditioning) fasshauer@iit.edu MATH 590 Chapter 2 23

49 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). Error curves are not monotonic. Optimal value of ε seems to exist which minimizes errors. For small enough values of ε the computational results become unpredictable, and the error curves become erratic (ill-conditioning) We ll consider ε to be safe if we don t get a MATLAB warning about ill-conditioning. Note that for small N optimal ε is safe, but for larger N not. We obtain highly accurate solutions from severely ill-conditioned linear systems! This is known as the uncertainty or trade-off principle. fasshauer@iit.edu MATH 590 Chapter 2 23

50 Radial Basis Function Interpolation What do we learn from the ε error curves? Errors decrease with decreasing ε (not the same as convergence for h 0). Error curves are not monotonic. Optimal value of ε seems to exist which minimizes errors. For small enough values of ε the computational results become unpredictable, and the error curves become erratic (ill-conditioning) We ll consider ε to be safe if we don t get a MATLAB warning about ill-conditioning. Note that for small N optimal ε is safe, but for larger N not. We obtain highly accurate solutions from severely ill-conditioned linear systems! This is known as the uncertainty or trade-off principle. Interesting problem: how to compute optimal solution in a stable way. One possible approach is [Fornberg & Wright (2004)] with details later, also ongoing work with Mike McCourt (see next slide). fasshauer@iit.edu MATH 590 Chapter 2 23

51 Radial Basis Function Interpolation Figure: Preliminary results for stable Gaussian evaluation. QR corresponds to new method, Direct to standard method. MATH 590 Chapter 2 24

52 Radial Basis Function Interpolation Best possible errors for Gaussian interpolation smallest safe ε smallest RMS-error k N ε RMS-error max-error ε RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-007 Table: Optimal RBF interpolation to Franke s function using Gaussians. fasshauer@iit.edu MATH 590 Chapter 2 25

53 Radial Basis Function Interpolation Best possible errors for Gaussian interpolation smallest safe ε smallest RMS-error k N ε RMS-error max-error ε RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-007 Table: Optimal RBF interpolation to Franke s function using Gaussians. Remark Errors for safe ε now comparable (or smaller) than those for distance matrix interpolation. fasshauer@iit.edu MATH 590 Chapter 2 25

54 Radial Basis Function Interpolation Best possible errors for Gaussian interpolation smallest safe ε smallest RMS-error k N ε RMS-error max-error ε RMS-error max-error e e e e e e e e e e e e e e e e e e e e e e e e-007 Table: Optimal RBF interpolation to Franke s function using Gaussians. Remark Errors for safe ε now comparable (or smaller) than those for distance matrix interpolation. Much better for optimal ε. However, ε = 6.2 with N = 1089 Halton points yields RCOND = e-020. fasshauer@iit.edu MATH 590 Chapter 2 25

55 Radial Basis Function Interpolation Summary Remark If the data are not sampled from a known test function, then we will not be able to choose an optimal shape parameter by monitoring the RMS error. fasshauer@iit.edu MATH 590 Chapter 2 26

56 Radial Basis Function Interpolation Summary Remark If the data are not sampled from a known test function, then we will not be able to choose an optimal shape parameter by monitoring the RMS error. New challenge: how to find optimal ε based only on the data? fasshauer@iit.edu MATH 590 Chapter 2 26

57 Radial Basis Function Interpolation Summary Remark If the data are not sampled from a known test function, then we will not be able to choose an optimal shape parameter by monitoring the RMS error. New challenge: how to find optimal ε based only on the data? We will study later. ill-conditioning and preconditioning, optimal shape parameter selection, and alternate stable evaluation methods via a Contour-Padé algorithm or eigenfunction expansions fasshauer@iit.edu MATH 590 Chapter 2 26

58 References I Appendix References Buhmann, M. D. (2003). Radial Basis Functions: Theory and Implementations. Cambridge University Press. Fasshauer, G. E. (2007). Meshfree Approximation Methods with MATLAB. World Scientific Publishers. Higham, D. J. and Higham, N. J. (2005). MATLAB Guide. SIAM (2nd ed.), Philadelphia. Iske, A. (2004). Multiresolution Methods in Scattered Data Modelling. Lecture Notes in Computational Science and Engineering 37, Springer Verlag (Berlin). Wendland, H. (2005a). Scattered Data Approximation. Cambridge University Press (Cambridge). fasshauer@iit.edu MATH 590 Chapter 2 27

59 References II Appendix References Fornberg, B. and Wright, G. (2004). Stable computation of multiquadric interpolants for all values of the shape parameter. Comput. Math. Appl. 47, pp Genton, M. C. (2001). Classes of kernels for machine learning: a statistics perspective. J. Machine Learning Research 2, pp MATLAB Central File Exchange. available online at fasshauer@iit.edu MATH 590 Chapter 2 28