MATH 590: Meshfree Methods Chapter 2 Part 3: Native Space for Positive Definite Kernels Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Fall 2014 fasshauer@iit.edu MATH 590 Chapter 2 1
Outline 1 Native Spaces for Positive Definite Kernels 2 Examples of Native Spaces for Popular RBFs fasshauer@iit.edu MATH 590 Chapter 2 2
In this section we will show that every positive definite kernel can indeed be associated with a reproducing kernel Hilbert space its native space. First, we note that the definition of an RKHS tells us that H K (Ω) contains all functions of the form f = N c j K (, x j ) j=1 provided x j Ω. fasshauer@iit.edu MATH 590 Chapter 2 4
Using the properties of RKHSs established earlier along with the form of f just mentioned we have that N N f 2 H K (Ω) = f, f HK (Ω) = c i K (, x i ), c j K (, x j ) = = N i=1 j=1 N i=1 j=1 i=1 j=1 N c i c j K (, x i ), K (, x j ) HK (Ω) N c i c j K (x i, x j ) = c T Kc. H K (Ω) So for these special types of f we can easily calculate the Hilbert space norm of f. In particular, if f = s is a kernel-based interpolant, i.e., c = K 1 y, then we also have s 2 H K (Ω) = y T K T KK 1 y = y T K 1 y. fasshauer@iit.edu MATH 590 Chapter 2 5
Therefore, we define the (possibly infinite-dimensional) space of all linear combinations H K (Ω) = span{k (, z) : z Ω} (1) with an associated bilinear form, K given by N M N M c i K (, x i ), d j K (, z j ) = c i d j K (x i, z j ) = c T Kd. i=1 Remark j=1 K i=1 j=1 Note that this definition implies that a general element in H K (Ω) has the form (where N = is allowed) f = N c j K (, x j ). j=1 However, not only the coefficients c j, but also the specific value of N and choice of points x j will vary with f. fasshauer@iit.edu MATH 590 Chapter 2 6
Theorem If K : Ω Ω R is a symmetric strictly positive definite kernel, then the bilinear form, K defines an inner product on H K (Ω). Furthermore, H K (Ω) is a pre-hilbert space with reproducing kernel K. Remark A pre-hilbert space is an inner product space whose completion is a Hilbert space. fasshauer@iit.edu MATH 590 Chapter 2 7
Proof., K is obviously bilinear and symmetric. We just need to show that f, f K > 0 for nonzero f H K (Ω). Any such f can be written in the form Then N f = c j K (, x j ), x j Ω. j=1 N N f, f K = c i c j K (x i, x j ) > 0 i=1 j=1 since K is strictly positive definite. The reproducing property follows from f, K (, x) K = N c j K (x, x j ) = f (x). j=1 fasshauer@iit.edu MATH 590 Chapter 2 8
Since we just showed that H K (Ω) is a pre-hilbert space, i.e., need not be complete, we now first form the completion H K (Ω) of H K (Ω) with respect to the K -norm K ensuring that f K = f HK (Ω) for all f H K (Ω). In general, this completion will consist of equivalence classes of Cauchy sequences in H K (Ω), so that we can obtain the native space N K (Ω) of K as a space of continuous functions with the help of the point evaluation functional (which extends continuously from H K (Ω) to H K (Ω)), i.e., the (values of the) continuous functions in N K (Ω) are given via the right-hand side of δ x (f ) = f, K (, x) K, f H K (Ω). Remark The technical details concerned with this construction are discussed in [Wen05]. fasshauer@iit.edu MATH 590 Chapter 2 9
In summary, we now know that the native space N K (Ω) is given by (continuous functions in) the completion of H K (Ω) = span{k (, z) : z Ω} a not very intuitive definition of a function space. In the special case when we are dealing with strictly positive definite (translation invariant) functions Φ(x z) = K (x, z) and when Ω = R d we get a characterization of native spaces in terms of Fourier transforms. We present the following theorem without proof (for details see [Wen05]). fasshauer@iit.edu MATH 590 Chapter 2 10
Theorem Suppose Φ C(R d ) L 1 (R d ) is a real-valued strictly positive definite function. Define G = {f L 2 (R d ) C(R d ) : ˆf L 2 (R ˆΦ d )} and equip this space with the bilinear form f, g G = 1 (2π) d ˆf ˆΦ, ĝ L2 (R ˆΦ d ) = 1 (2π) d ˆf (ω)ĝ(ω) R ˆΦ(ω) d dω. Then G is a real Hilbert space with inner product, G and reproducing kernel Φ( ). Hence, G is the native space of Φ on R d, i.e., G = N Φ (R d ) and both inner products coincide. In particular, every f N Φ (R d ) can be recovered from its Fourier transform ˆf L 1 (R d ) L 2 (R d ). fasshauer@iit.edu MATH 590 Chapter 2 11
Mercer s theorem allows us to construct the native space/rkhs H K (Ω) for any continuous positive definite kernel K by representing the functions in H K as infinite linear combinations of the eigenfunctions ϕ n of the Hilbert Schmidt integral operator K, i.e., { } H K = f : f = c n ϕ n. Thus the eigenfunctions {ϕ n } n=1 of K provide an alternative basis for H K (Ω) instead of the standard {K (, z) : z Ω}. For any fixed x, the corresponding basis transformation is given by the Mercer series K (, z) = λ n ϕ n ϕ n (z). n=1 n=1 This shows that indeed K (, z) H K (Ω). fasshauer@iit.edu MATH 590 Chapter 2 12
The inner product for H K (Ω) can now be written as f, g HK (Ω) = c m ϕ m, d n ϕ n = m=1 n=1 H K (Ω) n=1 where we used the fact that the eigenfunctions are not only L 2 -orthonormal, but also orthogonal in H K (Ω), i.e., ϕ m, ϕ n HK (Ω) = δ mn λm λn. c n d n λ n, fasshauer@iit.edu MATH 590 Chapter 2 13
We can also verify that K is indeed the reproducing kernel of H K since the Mercer series of K and the orthogonality of the eigenfunctions imply f, K (, x) HK (Ω) = c m ϕ m, λ n ϕ n ϕ n (x) = = m=1 n=1 n=1 c n λ n ϕ n (x) λ n c n ϕ n (x) n=1 = f (x). H K (Ω) fasshauer@iit.edu MATH 590 Chapter 2 14
Finally (cf. [Wen05]), we can also describe the RKHS H K as { } 1 H K (Ω) = f L 2 (Ω) : f, ϕ n λ L2 (Ω) 2 < n with inner product n=1 f, g HK (Ω) = n=1 1 λ n f, ϕ n L2 (Ω) g, ϕ n L2 (Ω), f, g H K (Ω). Remark Since H K (Ω) is a subspace of L 2 (Ω) this latter interpretation corresponds to the identification of the coefficients in the eigenfunction expansion of an f H K (Ω) with the generalized Fourier coefficients of f, i.e., c n = f, ϕ n L2 (Ω). fasshauer@iit.edu MATH 590 Chapter 2 15
Examples of Native Spaces for Popular RBFs The theorem characterizing the native spaces of translation invariant functions on all of R d shows that these spaces can be viewed as a generalization of standard Sobolev spaces. For m > d/2 the Sobolev space W2 m can be defined as (see, e.g., [AF03]) W m 2 (Rd ) = {f L 2 (R d ) C(R d ) : ˆf ( )(1 + 2 2 )m/2 L 2 (R d )}. (2) Remark One also frequently sees the definition W m 2 (Ω) = {f L 2(Ω) C(Ω) : D α f L 2 (Ω) for all α m, α N d }, (3) which applies whenever Ω R d is a bounded domain. fasshauer@iit.edu MATH 590 Chapter 2 17
Examples of Native Spaces for Popular RBFs Example Using the notation r = x and modified Bessel functions of the second kind K d/2 β, the Matérn kernels have Fourier transform κ β (r) = K d/2 β(r) r d/2 β, β > d 2, ˆκ β ( ω ) = ( 1 + ω 2) β. So it can immediately be seen that their native space is N K (R d ) = W β 2 (Rd ) with β > d/2, which is why some people refer to the Matérn kernels as Sobolev splines. fasshauer@iit.edu MATH 590 Chapter 2 18
Examples of Native Spaces for Popular RBFs The native spaces for Gaussians is rather small. Example According to the Fourier transform characterization of the native space, for Gaussians the Fourier transform of f N Φ (Ω) must decay faster than the Fourier transform of the Gaussian (which is itself a Gaussian). The native space of Gaussians was recently characterized in [FY11] in terms of an (infinite) vector of differential operators. In fact, the native space of Gaussians is contained in the Sobolev space W m 2 (Rd ) for any m. It is known that, even though the native space of Gaussians is small, it contains the important class of so-called band-limited functions, i.e., functions whose Fourier transform is compactly supported. Band-limited functions play an important role in sampling theory. fasshauer@iit.edu MATH 590 Chapter 2 19
References I Appendix References [AF03] [FY11] Robert Alexander Adams and John J. F. Fournier, Sobolev Spaces, Academic Press, 2003. G. E. Fasshauer and Qi Ye, Reproducing kernels of generalized Sobolev spaces via a Green function approach with distributional operators, Numerische Mathematik 119 (2011), 585 611. [Wen05] H. Wendland, Scattered Data Approximation, Cambridge Monographs on Applied and Computational Mathematics, vol. 17, Cambridge University Press, 2005. fasshauer@iit.edu MATH 590 Chapter 2 20