Solving Support Vector Machines in Reproducing Kernel Banach Spaces with Positive Definite Functions

Transcription

1 Solving Suort Vector Machines in Reroducing Kernel Banach Saces with Positive Definite Functions Gregory E. Fasshauer a, Fred J. Hickernell a, Qi Ye b, a Deartment of Alied Mathematics, Illinois Institute of Technology, Chicago, Illinois b Deartment of Mathematics, Syracuse University, Syracuse, NY Abstract In this aer we solve suort vector machines in reroducing kernel Banach saces with reroducing kernels defined on nonsymmetric domains instead of the traditional methods in reroducing kernel Hilbert saces. Using the orthogonality of semi-inner-roducts, we can obtain the exlicit reresentations of the dual (normalized-duality-maing) elements of suort vector machine solutions. In addition, we can introduce the reroduction roerty in a generalized native sace by Fourier transform techniques such that it becomes a reroducing kernel Banach sace, which can be even embedded into Sobolev saces, and its reroducing kernel is set u by the related ositive definite function. The reresentations of the otimal solutions of suort vector machines (regularized emirical risks) in these reroducing kernel Banach saces are formulated exlicitly in terms of ositive definite functions, and their finite numbers of coefficients can be comuted by fixed oint iteration. We also give some tyical examles of reroducing kernel Banach saces induced by Matérn functions (Sobolev slines) so that their suort vector machine solutions are well comutable as the classical algorithms. Moreover, each of their reroducing bases includes information from multile training data oints. The concet of reroducing kernel Banach saces offers us a new numerical tool for solving suort vector machines. Keywords: suort vector machine, regularized emirical risk, reroducing kernel, reroducing kernel Banach sace, ositive definite function, Matérn function, Sobolev sline. Corresonding author addresses: fasshauer@iit.edu (Gregory E. Fasshauer), hickernell@iit.edu (Fred J. Hickernell), qiye@syr.edu (Qi Ye) Prerint submitted to Alied and Comutational Harmonic Analysis Aril 21, 2013

2 1. Introduction The theory and ractice of kernel-based methods is a fast growing research area. They have been used for both scattered data aroximation and machine learning. Alications come from such different fields as hysics, biology, geology, meteorology and finance. The books [4, 7, 20, 21] show how to use (conditionally) ositive definite kernels to construct interolants for observation data samled from some unknown functions in the native saces induced by the kernel functions. In the books [2, 18], the otimal suort vector machine solutions are obtained in reroducing kernel Hilbert saces (RKHSs), and these solutions are formulated in terms of the related reroducing kernels and given data values. Actually, as long as the same inner roduct is used, the concets of native saces and RKHSs are interchangeable. It is just that researchers in numerical analysis and statistical learning use different terminology and techniques to introduce those saces. Moreover, the recent contributions [9, 10, 22] develo a clear and detailed framework for generalized Sobolev saces and RKHSs by establishing a connection between Green functions and reroducing kernels. Related to the current research work, [5, 6, 23] all generalize classical native saces (RKHSs) to Banach saces in different ways. However, the reroducing roerty in generalized native saces is not discussed in [5, 6], and [23] does not mention how to use reroducing kernels to introduce the exlicit forms of their reroducing kernel Banach saces (RKBSs) analogous to the tyical cases of RKHSs induced by Gaussian kernels and Sobolev-sline kernels, etc. Using [23] it is therefore difficult to obtain exlicit and simle suort vector machine (SVM) solutions and erform ractical comutations. Following the results of these earlier authors, [22, Section 6] tries to combine both of these ideas, and uses Fourier transform techniques to construct RKBSs. In this aer we want to comlete and extend the theoretical results in [22, Section 6]. In addition, the RKBS given in Definition 4.1 is different from that of [23]. Our RKBS can be one-sided or two-sided and its reroducing kernel K can be defined on nonsymmetric domains, i.e., K : Ω 2 Ω 1 C, where Ω 1 and Ω 2 can be various subsets of R d 1 and R d 2, resectively (see Definition 4.1). Our RKBS is an extension of the RKHS and it does not require the reflexivity condition. The RKBS defined in [23] can be seen as a secial case of the RKBS defined in this aer. According to Lemma 4.1, we can still obtain the otimal solution in the one-sided RKBS using the techniques of semi-inner-roducts. It is well known that for given training data D := { (x j, y j ) } N the classical 2

3 SVM (regularized emirical risk) in the RKHS H has the form min f H L ( x j, y j, f (x j ) ) + R ( ) f H, where L is a loss function and R is a regularization function (see Theorem 3.1). In the same way we are able to aly an otimal recovery of RKBSs to solve SVMs in RKBSs. Theorem 4.2 establishes that the SVM in the right-sided RKBS B with the reroducing kernel K : Ω 2 Ω 1 C based on the training data D Ω 1 C satisfies min f B L ( x j, y j, f (x j ) ) + R ( ) f B. Moreover, this roblem has a unique otimal solution s D,L,R and its dual (normalizedduality-maing) element s D,L,R is a linear combination of the reroducing kernel centered at the training data oints {x 1,..., x N } Ω 1, i.e., s D,L,R(x) = c k K(x, x k ), x Ω 2. k=1 According to Corollary 4.3, the coefficient vector c := (c 1,, c N ) T of s D,L,R is a fixed oint of the function FD,L,R : RN R N deendent of the differential loss function L and the differential regularization function R, i.e., FD,L,R (c) = c. From this it is obvious that the SVM in the RKBS is the generalization of the classical method in the RKHS. In Section 5, we show how to use a ositive definite function Φ to set u different RKBSs B Φ (Rd ) and B Φ (Ω) with > 1 whose two-sided reroducing kernel is given by K(x, y) = Φ(x y) (see Theorems 5.1 and 5.6). We can observe that B Φ (Rd ) is a kind of generalized native sace. Furthermore, B Φ (Rd ) and B Φ (Ω) coincide with the definition of RKBSs given in [23]. The SVM solution s D,L,R in B Φ (Rd ) can be reresented by the ositive definite function Φ, which means that we can obtain an exlicit formula for the SVM solution s D,L,R in B Φ (Rd ) (see Theorem 5.4). Corollary 5.5 shows that the finite dimensional coefficients of the SVM solution s D,L,R can even be obtained by solving a fixed oint iteration roblem for differentiable loss functions and regularization functions. Theorem 5.6 and Corollary 5.7 give some examles of reroducing kernels defined on nonsymmetric domains. Corollary 5.3 and 5.8 rovide that RKBSs can be embedded into Sobolev saces for some secial reroducing kernels, e.g., Sobolev-sline kernels (Matérn functions). 3

4 The Matérn functions reresent a fast growing research area which has frequent alications in aroximation theory and statistical learning, and moreover, they are ositive definite functions and (full-sace) Green functions (see [7, 9, 14, 22]). In Section 6, we solve the SVMs in the RKBSs of Matérn functions. If G θ,n is the Matérn function with arameter θ > 0 and degree n > 3d/2 then, according to our theoretical results, B 2 G θ,n (R d ) is an RKHS, while B 4 G θ,n (R d ) is only an RKBS. Their reroducing kernels, however, are the same Sobolevsline kernel K θ,n (x, y) := G θ,n (x y). It is well known that the SVM solution in B 2 G θ,n (R d ) H Gθ,n (R d ) has the exlicit exression s D,L,R (x) := c k K θ,n (x, x k ), x R d, k=1 (see Theorem 3.1). In this aer we discover a new fact that the SVM solution in B 4 G θ,n (R d ) also has an exlicit form, namely s D,L,R (x) = N,N,N k 1,k 2,k 3 =1 c k1 c k2 c k3 K θ,3n ( x, xk1, x k2, x k3 ), x R d, where K θ,3n (x, y 1, y 2, y 3 ) := G θ,3n (x y 1 + y 2 y 3 ). Section 6 shows that several other exlicit reresentations of SVM solutions in the RKBS B G θ,n (R d ) are easily comutable when is an even number. This discovery could lead to a new numerical tool for SVMs. For the binary classification roblems, it is well-known that the classical hinge loss is designed to maximize the 2-norm margins by using the linear functions. However, we can not emloy the hinge loss to set u the SVMs in order to maximize other -norm margins. We guess that for alications to the roblems that arise in current ractice it will be necessary to construct loss functions deending on different kinds of RKBSs. Remark 1.1. In this aer, the third author hoes to correct a mistake concerning the otimal recovery of RKBS B Φ (Rd ) mentioned in [22, Section 6.2]. Theorem 5.4 is the correction of [22, Theorem 6.5], which was the result of a misconcetion that the normalized duality maing is linear. The main ideas and techniques used in the corrected version below are still the same as in [22]. An udated version of [22] has been osted on Ye s webage. 4

5 2. Banach Saces In this section, we review some classical theoretical results for Banach saces from [11, 13, 15, 16]. We denote the dual sace (the collection of all bounded linear functionals) of a Banach sace B by B and its dual bilinear roduct as, B, i.e., f, T B := T( f ), for all T B and all f B. [16, Theorem ] states that B is also a Banach sace. If the Banach saces B 1 and B 2 are isometrically isomorhic (equivalent), i.e., B 1 B 2, then we can think of both saces as being identical in the sense that their norms and their elements can be seen to be the same in both saces (see [16, Definition ]). We say that B 1 is embedded into B 2 if there exists a ositive constant C such that f B2 C f B1 for all f B 1 B 2 (see [1, Section 1.25]). If the Banach sace B is reflexive (see [16, Definition ]), then we have B B and f, g B = g, f B for all f B and all g B. For examle, the function sace L (Ω; µ) defined on the ositive measure sace (Ω, B Ω, µ) is a reflexive Banach sace and its dual sace is isometrically equivalent to L q (Ω; µ) where, q > 1 and 1 + q 1 = 1 (see [16, Examle and Theorem ]). For the comlex situation, the isometric isomorhism from L (Ω; µ) onto L q (Ω; µ) is antilinear. We say that B is uniformly convex if, for every ɛ > 0, there is δ > 0 such that f + g 2 1 δ, whenever f B = g B = 1 and f g B ɛ B (see [16, Definition 5.2.1]). According to [16, Definition 5.4.1, and Corollary ], B is said to be smooth or Gâteaux differentiable if f + λg lim B f B exists, for all f, g B. λ 0 λ A tyical case is that L (Ω; µ) is uniformly convex and smooth if 1 < <. It is well known that we can discuss the orthogonality in Banach saces with a more general axiom system than that in Hilbert saces. The aers [11, 13, 15] show that every Banach sace can be reresented as a semi-inner-roduct sace in order that the theories of Banach sace can be enetrated by Hilbert sace tye arguments. A semi-inner-roduct [, ] B : B B C defined on a Banach sace B is given by (i) [ f + g, h] B = [ f, h] B + [g, h] B, (ii) [ f, f ] B = f 2 B, (iii) [λ f, g] B = λ[ f, g] B, [ f, λg] B = λ[ f, g] B, (iv) [ f, g] B [ f, f ] B [g, g] B, 5

6 for all f, g, h B and all λ C. However, Hermitian symmetry of the semi-innerroduct may not hold, i.e., [ f, g] B [g, f ] B. This indicates that the generality of the semi-inner-roduct in Banach sace is a serious limitation for any extensive develoment that arallels the inner roduct of Hilbert sace. For examle, a semi-inner-roduct of L (Ω; µ) with 1 < < is given by 1 [g, f ] L (Ω;µ) = f 2 L (Ω;µ) Ω g(x) f (x) f (x) 2 dµ(x), for all f, g L (Ω; µ), (see examles in [11, 13]). We say that f is orthogonal to g in a Banach sace B if f + λg B f B, for all λ C, (see the definitions in [11, 13]). Suose that the Banach sace B is smooth. Using [11, Theorem 2], we can determine that f is orthogonal to g if and only if f is normal to g, i.e., [g, f ] B = 0. We can also obtain a reresentation theorem in Banach sace by an adatation of the reresentation theorem in Hilbert sace. Suose that the Banach sace B is uniformly convex and smooth. According to [11, Theorem 3 and 6], for every bounded linear functional T B, there exists a unique f B such that T(g) = g, T B = [g, f ] B, for all g B, and T B = f B. This maing is also surjective. We call T the normalizedduality-maing element of f and rewrite it as f := T. For convenience we simlify normalized-duality-maing element to dual element in this aer. The normalized duality maing is a one-to-one and norm-reserving maing from B onto B. Note that this maing is usually nonlinear. According to [11, Theorem 7], the semi-inner-roduct of B has the form [ f, g ] B = [g, f ] B for all f, g B. For examle, the dual element of f L (Ω; µ) with 1 < < is given by f f (x) f (x) 2 = L f 2 q (Ω; µ), L (Ω;µ) where q is the conjugate exonent of. Let N be a subset of B. We can check that f is orthogonal to N if and only if its dual element f N = {η B : h, η B = 0, for all h N}, i.e., [h, f ] B = h, f B = 0, for all h N. 6

7 3. Reroducing Kernels and Reroducing Kernel Hilbert Saces Most of the material resented in this section can be found in the monograhs [7, 18, 21]. For the reader s convenience we reeat here what is essential to our discussion later on. Definition 3.1 ([21, Definition 10.1]). Let Ω R d and H be a Hilbert sace consisting of functions f : Ω C. H is called a reroducing kernel Hilbert sace (RKHS) and a kernel function K : Ω Ω C is called a reroducing kernel for H if (i) K(, y) H and (ii) f (y) = ( f, K(, y)) H, for all f H and all y Ω, where (, ) H is used to denote the inner roduct of H. Remark 3.1. In order to simlify our discussion and roofs, we let all kernel functions be comlex-valued and all function saces be comosed of comlex-valued functions in this aer. According to [16, Proosition 1.9.3], it is not difficult for us to restrict the theoretical results to real kernel functions and function saces Otimal Recovery in Reroducing Kernel Hilbert Saces Theorem 3.1 (Reresenter theorem [18, Theorem 5.5]). Let H be a reroducing kernel Hilbert sace with a reroducing kernel K defined on Ω R d, and a regularization function R : [0, ) [0, ) be convex and strictly increasing. We choose the loss function L : Ω C C [0, ) such that L(x, y, ) is a convex ma for any fixed x Ω and any fixed y C. Given the data D := {(x 1, y 1 ),..., (x N, y N )}, with airwise distinct data oints X = {x 1,..., x N } Ω and associated data values Y = {y 1,..., y N } C, the otimal solution (suort vector machine solution) s D,L,R of min f H L ( x j, y j, f (x j ) ) + R ( ) f H, has the exlicit reresentation s D,L,R (x) = c k K(x, x k ), x Ω, k=1 for some coefficients c 1,..., c N C. 7

8 3.2. Constructing Reroducing Kernel Hilbert Saces by Positive Definite Functions Definition 3.2 ([21, Definition 6.1]). A continuous even function Φ : R d C is called ositive definite if, for all N N and all sets of airwise distinct centers X = {x 1,..., x N } R d, the quadratic form c j c k Φ(x j x k ) = c A Φ,X c > 0, for all c C N \{0}. k=1 Here the interolation matrix A Φ,X := ( Φ(x j x k ) ) N,N j,k=1 CN N and c = c T. We say Φ is even if Φ(x) = Φ( x). This shows that Φ is a ositive definite function if and only if A Φ,X is a ositive definite matrix for any airwise distinct finite set X of data oints in R d. The alication and history of ositive definite functions can be seen in the review aer [8]. [21, Section 10.2] shows how to use ositive definite functions to construct RKHSs. Theorem 3.2 ([21, Theorem 6.11]). Suose that Φ C(R d ) L 1 (R d ). Then Φ is ositive definite if and only if Φ is bounded and its Fourier transform ˆΦ is nonnegative and nonvanishing (nonzero everywhere). Remark 3.2. In this aer, the Fourier transform of f L 1 (R d ) is defined by ˆ f (x) := (2π) d/2 R d f (y)e ixt y dy, where i is the imaginary unit, i.e., i 2 = 1. Theorem 3.3 ([21, Theorem 10.12]). Suose that Φ C(R d ) L 1 (R d ) is a ositive definite function. Then the sace H Φ (R d ) := { f L 2 (R d ) C(R d ) : ˆ f / ˆΦ 1/2 L 2 (R d ) }, equied with the norm f HΦ (R d ) := (2π) d/2 R d f ˆ(x) 2 ˆΦ(x) dx 1/2 is a reroducing kernel Hilbert sace (native sace) with reroducing kernel given by K(x, y) := Φ(x y), x, y R d, 8

9 where ˆΦ and fˆ are the Fourier transforms of Φ and f, resectively. The inner roduct in H Φ (R d ) has the form ( f, g) H = (2π) d/2 f ˆ(x)ĝ(x) dx, f, g H Φ (R d ). R ˆΦ(x) Using Fourier transform techniques similar to those in Theorem 3.3, we can emloy ositive definite functions to set u RKBSs (see Section 5). 4. Reroducing Kernels and Reroducing Kernel Banach Saces Now we give the definition of RKBSs as a natural generalization of RKHSs by viewing the inner roduct as a dual bilinear roduct. Definition 4.1. Let Ω 1 and Ω 2 be two subsets of R d 1 and R d 2 resectively, and B be a Banach sace comosed of functions f : Ω 1 C, whose dual sace B is isometrically equivalent to a function sace F with g : Ω 2 C. Denote that K : Ω 2 Ω 1 C is a kernel function. We call B a reroducing kernel Banach sace (RKBS) and K its right-sided reroducing kernel if (i) K(, y) F B and (ii) f (y) = f, K(, y) B, for all f B and all y Ω 1. If the Banach sace B reroduces from the other side, i.e., (iii) K(x, ) B and (iv) g(x) = K(x, ), g B, for all g F B and all x Ω 2, then B is called a reroducing kernel Banach sace and K its left-sided reroducing kernel. For two-sided reroduction as above we say that B is a reroducing kernel Banach sace with the two-sided reroducing kernel K. Remark 4.1. We know that the Riesz reresenter ma on comlex Hilbert sace H is antilinear, i.e., T λg ( f ) = f, λg H = λ( f, g) H = λ f, g H = λt g ( f ), for all f, g H and all λ C. Here we also let the isometrical isomorhism from the dual sace B onto the related function sace F be antilinear. Thus, the format of two-sided RKBSs coincides with comlex RKHSs, i.e., K(y, ), f H = (K(y, ), f ) H = ( f, K(, y)) H = f (y), for all f H and all y Ω, which indicates that the RKHS is a secial case of a two-sided RKBS. 9

10 Why do we define our RKBSs differently from [23, Definition 1]? The reason is that we can show the otimal recovery in an RKBS even if it is only onesided. We do not require a reflexivity condition for the definition of our RKBS. Moreover, since the dual sace of a Hilbert sace is isometrically equivalent to itself, we can choose the equivalent function sace F H such that the domain of the reroducing kernel K is symmetric, i.e., Ω 2 = Ω 1. Actually, the Banach sace B is usually not equal to any equivalent function sace F of its dual B even though we only require them to be isomorhic. We naturally do not need any symmetry conditions in the Banach sace. Therefore the nonsymmetric domain is used to define the RKBS B and its reroducing kernel K, i.e., Ω 2 Ω 1. The domain of K is related to both B and F B. If we choose a different F which is isometrically equivalent to the dual B, then we can obtain a different reroducing kernel K of the RKBS B deendent on its equivalent dual sace F. The functional K(, y) can be seen as a oint evaluation function δ y defined on B. This imlies that δ y is a bounded linear functional on B, i.e., δ y B. If the Banach sace B is further uniformly convex and smooth, then its semi-innerroduct and its normalized duality maing are well-defined, which can be used to set u the equivalent conditions of right-sided RKBSs, i.e., δ y B F which indicates that f (y) = f, δ y B = [ f, δ y] B, for all f B and all y Ω 1 (see the discussions of the semi-inner roducts in Section 2). If B is a reflexive two-sided RKBS, then the equivalent dual sace F of B is also a reflexive two-sided RKBS. All RKBSs and reroducing kernels set u in Section 5 satisfy the two-sided definition but their domains can be symmetric or nonsymmetric. If a sequence { f n } n=1 B and f B such that f f n B 0 when n, then f (y) f n (y) = f f n, K(, y) B K(, y) B f f n B 0, y Ω 1, when n. This means that convergence in the right-sided RKBS B imlies ointwise convergence. Suose that B is a reflexive right-sided RKBS. We show that {K(, y) : y Ω 1 } is a linear vector sace basis of F and san {K(, y) : y Ω 1 } is dense in F. Let N be a comletion (closure) of san {K(, y) : y Ω 1 } F B with its dual norm. Now we rove that N F B. Since [16, Theorem ] rovides that F is also a Banach sace, we have N F. Assume that N F. According 10

11 to [16, Corollary 1.9.7] (alication of Hahn-Banach extension theorems) there is an element f B B F such that f B = 1 and f (y) = f, K(, y) B = 0 for all y Ω 1. We find the contradiction between f B = 1 and f = 0. Thus the first assumtion is not true and then we can conclude that N F B, which indicates that {K(, y) : y Ω 1 } is a linear vector sace basis of F and { δ y : y Ω 1 } is a linear vector sace basis of B. Examle 4.1. We give a simle examle of a two-sided RKBS. Let Ω 2 = Ω 1 := {1,, n} and A C n n be a symmetric ositive definite matrix. It can be decomosed into A = VDV, where D is a ositive diagonal matrix and V is an orthogonal matrix. We choose, q > 1 such that 1 + q 1 = 1. Define B := { f : Ω 1 C} equied with the norm f B := D 1/q V f q, where f := ( f (1),, f (n)) T. We can check that B is a Banach sace and its dual sace B is isometrically equivalent to F := {g : Ω 2 C} equied with the norm g B := D 1/ V g, where g := (g(1),, g(n)) T. Moreover, its dual bilinear form is given by f, g B = g A 1 f, for all f B and all g B. If the kernel function is defined by K( j, k) := A jk, j Ω 2, k Ω 1, then the reroduction can easily be verified, i.e., f, K(, k) B = f (k), k Ω 1, and K( j, ), g B = g( j), j Ω 2. Therefore B is indeed a two-sided RKBS. (In the same way, we can also emloy the singular value decomosition of a nonsymmetric and nonsingular square matrix A to introduce the two-sided RKBS.) 4.1. Otimal Recovery in Reroducing Kernel Banach Saces It is well-known that any Hilbert sace is uniformly convex and smooth. It is natural for us to assume the right-sided RKBS is further uniformly convex and smooth to discuss otimal recovery in it. The definitions of uniform convexity and smoothness of Banach saces are given in Section 2. 11

12 Given the airwise distinct data oints X = {x 1,..., x N } Ω 1 and the associated data values Y = {y 1,..., y N } C, we define a subset of the right-sided RKBS B by N B (X, Y) := { f B : f (x j ) = y j, for all j = 1,..., N }. If N B (X, Y) is the null set, then there is no meaning for the SVMs. So we need to assume that N B (X, Y) is always non-null for the given data sites. Actually we can show that N B (X, Y) is non-null for any data values Y if and only if δ x1,..., δ xn are linearly indeendent on B because N k=1 c kδ xk = 0 if and only if N k=1 c k f (x k ) = 0 for all f B, and moreover, c = (c 1,, c N ) T = 0 if and only if b c = 0 for all b C N. In this section, we suose that δ x1,..., δ xn are always linearly indeendent on B for the given airwise distinct data oints X, which is equivalent to the fact that K(, x 1 ),..., K(, x N ) are linearly indeendent. We use the techniques of [23, Theorem 19] to verify the following lemma. Lemma 4.1. Let B be a reroducing kernel Banach sace with a right-sided reroducing kernel K defined on Ω 2 Ω 1 R d 2 R d 1. Suose that B is uniformly convex and smooth. Given the data D := {(x 1, y 1 ),..., (x N, y N )} with airwise distinct data oints X = {x 1,..., x N } Ω 1 and associated data values Y = {y 1,..., y N } C, the dual element s D of the unique otimal solution { s D := argmin f B : f (x j ) = y j, for all j = 1,..., N }, (4.1) f B is the linear combination of K(, x 1 ),..., K(, x N ), i.e., s D(x) = c k K(x, x k ), x Ω 2. k=1 Proof. We first rove the uniqueness of the otimal solution of the minimization roblem (4.1). Let us assume that the minimization roblem (4.1) has two otimal solutions s 1, s 2 B with s 1 s 2. Since B is uniformly convex, [16, Corollary ] rovides that 1 (s s 2 ) B < 1 s 2 1 B + 1 s 2 2 B. Then s 1 B = s 2 B shows for s 3 := 1 (s s 2 ) that s 3 B < s 1 B and s 3 N B (X, Y), i.e., s 1 is not an otimal solution of the minimization roblem (4.1). The assumtion that there are two minimizers is false. Next we show the existence of the minimizer. The minimization roblem (4.1) is equivalent to min f NB (X,Y) f B. Since convergence in a one-sided RKBS B imlies ointwise convergence, we can check that N B (X, Y) is a closed convex subset of B. Combining this with the uniform convexity of B, [16, Corollary ] 12

13 shows that N B (X, Y) is a Chebyshev set (see [16, Definition ]). Thus an otimal solution min f NB (X,Y) f B exists. Because N B (X, Y)+N B (X, {0}) = N B (X, Y) and N B (X, {0}) is a closed subsace of B we can determine that the otimal solution s D is orthogonal to N B (X, {0}), i.e., s D + h B s D B for all h N B (X, {0}). Since B is uniformly convex and smooth, the dual element s D of s D is well-defined and which imlies that [h, s D ] B = h, s D B = 0, for all h N B (X, {0}), s D N B (X, {0}) = {g F B : h, g B = 0, for all h N B (X, {0})}. It is obvious that N B (X, {0}) = { f B : f (x j ) = f, K(, x j ) B = 0, j = 1,..., N } = { f B : f, h B = 0, for all h san {K(, x k )} N k=1} = san {K(, x k )} N k=1. According to [16, Proosition ], we have s D ( san {K(, x k )} N k=1) = san {K(, x1 ),..., K(, x N )}. Here N 1 and N 2 denote the annihilator of N 1 in B and the annihilator of N 2 in B, resectively, where N 1 B and N 2 B (see [16, Definition ]). Now we verify the reresenter theorem for SVMs in a right-sided RKBS. Theorem 4.2. Let B be a reroducing kernel Banach sace with a right-sided reroducing kernel K defined on Ω 2 Ω 1 R d 2 R d 1, and a regularization function R : [0, ) [0, ) be convex and strictly increasing. Suose that B is uniformly convex and smooth. We choose the loss function L : Ω 1 C C [0, ) such that L(x, y, ) is a convex ma for any fixed x Ω 1 and any fixed y C. Given the data D := {(x 1, y 1 ),..., (x N, y N )} with airwise distinct data oints X = {x 1,..., x N } Ω 1 and associated data values Y = {y 1,..., y N } C, the dual element of the unique otimal solution (suort vector machine solution) s D,L,R of min f B L ( x j, y j, f (x j ) ) + R ( ) f B, (4.2) 13

14 has the exlicit reresentation s D,L,R(x) = c k K(x, x k ), x Ω 2, k=1 for some coefficients c 1,..., c N C. Proof. Let T D,L,R ( f ) := L ( x j, y j, f (x j ) ) + R ( ) f B, f B. The minimization roblem (4.2) is equivalent to min f B T D,L,R ( f ). Since B is uniformly convex and R is convex and strictly increasing, the regularization f R ( f B ) is continuous and strictly convex. Because the B-norm convergence imlies the ointwise convergence and L ( x j, y j, ) is convex for all j = 1,..., N, the maing f N L ( x j, y j, f (x j ) ) is also continuous and convex. This indicates the continuity and strict convexity of T D,L,R. Using the increasing roerty of R, we can check that the set { f B : T D,L,R ( f ) T D,L,R (0) } is nonemty and bounded. Moreover, the uniformly convex norm imlies its reflexivity by the Milman-Pettis Theorem [16, Theorem ]. Thus the existence of minimizers theorem [18, Theorem A.6.9] gives the existence of the unique solution s D,L,R to minimize T D,L,R over B. We fix any f B and let D f := {(x k, f (x k ))} N k=1. According to Lemma 4.1, there exists an element s D f whose dual element s D f san {K(, x k )} N k=1 such that s D f interolates the data values { f (x k )} N k=1 at the centers oints X = {x k} N k=1 and s B D f f B. This indicates that T D,L,R (s D f ) T D,L,R ( f ). Therefore the dual element s D,L,R of the otimal solution s D,L,R of the minimization roblem (4.2) belongs to san {K(, x k )} N k=1. Remark 4.2. Since K(, x j ) can be seen as a oint evaluation functional δ x j defined on B, it indicates that the dual element of s D,L,R can be also written as a linear combination of δ x1,..., δ xn, i.e., s D,L,R = N c jδ x j. The uniform convexity and smoothness of B imly the uniform convexity and smoothness of its dual B F. If B is a left-sided RKBS satisfying uniform 14

15 convexity and smoothness conditions, then we can further erform otimal recovery in F in the same way, i.e., the dual element of the otimal solution (SVM solution) of min g F B L ( x j, y j, g(x j ) ) + R ( g B ), is a linear combination of K(x 1, ),..., K(x N, ), where X = {x 1,..., x N } Ω 2 and L : Ω 2 C C [0, ). Moreover, since the normalized duality maing is an identity maing on the Hilbert sace and the reroducing kernel of an RKHS is symmetric, otimal recovery in RKBSs as in Theorem 4.2 can be seen as a generalization of otimal recovery in RKHSs as in Theorem 3.1. Since the normalized duality maing is one-to-one, for any fixed c C N, there exists an unique s c B such that its dual element has the form s c = N k=1 c kk(, x k ) = k T Xc, where k X := (K(, x 1 ),, K(, x N )) T and c := (c 1,, c N ) T. According to Theorem 4.2, the SVM (4.2) can be transformed to solve a finitedimensional otimization roblem, i.e., c ot := argmin c C N L ( x j, y j, s c (x j ) ) + R ( ) s c B, and the dual element of the SVM solution has the form s D,L,R = kt Xc ot. Now we want to show that these otimal coefficients c ot can be comuted by a fixed oint iteration method similar as in [17]. Suose that L(x, y, ) C 1 (C) for all x Ω 1 and all y C, and R C 1 ([0, )). Let φ j(c) := [K(, x j ), k T Xc] B = [K(, x j ), s c] B, c C N, j = 1,..., N, and L (x, y, t) := d dt L(x, y, t), x Ω 1, y C, where d reresents the Wirtinger derivative defined by dt d dt := 1 ( d 2 du i d ), where t = u + iv with i 2 = 1 and u, v R. dv Thus we have s c (x j ) = s c, K(, x j ) B = [s c, K(, x j ) ] B = [K(, x j ), s c] B = φ j(c), j = 1,..., N, 15

16 and s c 2 B = [s c, s c ] B = s c, s c B = c j s c, K(, x j ) B = c j φ j(c) = c φ (c), where φ := ( φ 1,, φ N) T. Denote that T D,L,R(c) := L ( x j, y j, φ j(c) ) +R ( c φ (c) ) = L ( x j, y j, s c (x j ) ) +R ( ) s c B. Since c ot is the global minimizer of TD,L,R over CN, c ot is a stationary oint of TD,L,R, i.e., T D,L,R (c ot) = 0. We comute the gradient of TD,L,R by Wirtinger artial derivatives, i.e., ( TD,L,R(c) T = l D (φ (c)) T φ (c) + R c φ (c) ) 4 c φ (c), c φ (c) where l D (φ ) := ( L (x 1, y 1, φ 1 ),, L (x N, y N, φ N )) T and φ := ( ) N,N c k φ j is the j,k=1 Jacobian (gradient) matrix of φ by Wirtinger artial derivatives. The otimal solution c ot is also a fixed oint of the function FD,L,R, i.e., F D,L,R(c ot ) = c ot, where F D,L,R(c) := c + T D,L,R(c), c C N \{0}. (4.3) Corollary 4.3. Suose that the loss function L(x, y, ) C 1 (C) for all x Ω 1 and all y C, and the regularization function R C 1 ([0, )). Then the coefficients c of the dual element s D,L,R of the suort vector machine solution s D,L,R given in Theorem 4.2 is a fixed oint of the function FD,L,R defined in Equation (4.3), i.e., (c) = c. F D,L,R Remark 4.3. Even though we can obtain the coefficients of s D,L,R by the fixed oint iteration method, it is still difficult for us to recover the exlicit form s D,L,R in many cases. In Section 5 we discuss how to obtain the SVM solutions in RKBSs induced by ositive definite functions (see Theorem 5.4). In that setting the coefficients of the exlicit form are also comutable by a fixed oint iteration method for differentiable loss functions and regularization functions. 16

17 5. Constructing Reroducing Kernel Banach Saces by Positive Definite Functions Now we construct RKBSs based on ositive definite functions in a way similar to the construction of RKHSs in Theorem 3.3. Let 1 <, q < and 1 +q 1 = 1. Suose that Φ C(R d ) L 1 (R d ) is a ositive definite function. According to Theorem 3.2, we know that ˆΦ L 1 (R d ) C(R d ) is nonnegative and nonvanishing. We define B Φ (Rd ) := { f C(R d ) SI : the distributional Fourier transform ˆ f of f is a measurable function defined on R d such that ˆ f / ˆΦ 1/q L q (R d ) }, (5.1) equied with the norm f B Φ (Rd ) := (2π) d/2 R d f ˆ(x) q ˆΦ(x) dx 1/q, where SI is the collection of all slowly increasing functions (see [21, Definition 5.19]). We define B q Φ (Rd ) in an analogous way as above. Remark 5.1. Following the theoretical results of [12, Section 7.1] and [19, Section 1.3] we can define the distributional Fourier transform ˆT S of the temered distribution T S by γ, ˆT S := ˆγ, T S, for all γ S, where S is the Schwartz sace (see [21, Definition 5.17]) and S is its dual sace with the dual bilinear form, S. We can also verify that C(R d ) SI L loc 1 (Rd ) SI is embedded into S. When q, then ˆΦ L 1 (R d ) C(R d ) imlies that ˆΦ /q L 1 (R d ) which will be used in the roof of the following theorem. We also need to imose an additional symmetry condition on ˆΦ q/ L 1 (R d ) which is needed in the roof. Since /q = 1 and q/ = q 1, this condition can be reresented as ˆΦ min{,q} 1 L 1 (R d ). Since we can denote the ositive measure µ on R d as µ(a) := (2π) d/2 dx ˆΦ(x), for any oen set A of Rd. A 17

18 [16, Examle 1.2.6] rovides that the sace L q (R d ; µ) is well-defined on the ositive measure sace (R d, B R d, µ), i.e., { } L q (R d ; µ) := f : R d C : f is measurable and f (x) q dµ(x) <, R d equied with the norm ( ) 1/q f Lq (R d ;µ) := f (x) q dµ(x). R d L (R d ; µ) is also defined in an analogous way. [16, Examle and Theorem ] show that L q (R d ; µ) is a Banach sace and its dual sace L q (R d ; µ) is isometrically equivalent to L (R d ; µ). In analogy to the reresentation theorem on Hilbert sace, the bounded linear functional T g L q (R d ; µ) associated with g L (R d ; µ) is given by T g ( f ) := f (x)g(x)dµ(x), for all f L q (R d ; µ). R d Here, this isometric isomorhism from L q (R d ; µ) onto L (R d ; µ) is antilinear, just as the dual of comlex Hilbert saces, i.e., T λg ( f ) = f (x)λg(x)dµ(x) = λt g ( f ), for all f L q (R d ; µ) and all λ C. R d If we can show that B Φ (Rd ) and L q (R d ; µ) are isometrically isomorhic, then B Φ (Rd ) is a Banach sace and its dual sace B Φ (Rd ) is isometrically equivalent to L (R d ; µ). One can argue analogously for B q Φ (Rd ) L (R d ; µ). If we can further verify the two-sided reroduction of B Φ (Rd ), then B Φ (Rd ) is a two-sided RKBS. Theorem 5.1. Let 1 <, q < and 1 + q 1 = 1. Suose that Φ L 1 (R d ) C(R d ) is a ositive definite function on R d and that ˆΦ min{,q} 1 L 1 (R d ). Then B Φ (Rd ) given in Equation (5.1) is a reroducing kernel Banach sace with the two-sided reroducing kernel K(x, y) := Φ(x y), x, y R d. Its dual sace B Φ (Rd ) and B q Φ (Rd ) are isometrically isomorhic. Moreover, B Φ (Rd ) is uniformly convex and smooth. In articular, when = 2 then B 2 Φ (Rd ) = H Φ (R d ) is a reroducing kernel Hilbert sace as in Theorem

19 Proof. For convenience, we assume that q. We first rove that B Φ (Rd ) and L q (R d ; µ) are isometrically isomorhic. The Fourier transform ma can be seen as a one-to-one ma from B Φ (Rd ) into L q (R d ; µ). We can check the equality of their norm f ˆ(x) q 1/q ( f B Φ (Rd ) = (2π) d/2 R ˆΦ(x) dx = f ˆ(x) ) 1/q q dµ(x) = fˆ Lq. d R d (R d ;µ) So the Fourier transform ma is an isometric isomorhism. Now we verify that the Fourier transform ma is surjective. Fix any h L q (R d ; µ). We want to find an element in B Φ (Rd ) whose Fourier transform is equal to h. We conclude that h L 1 (R d ) because R d h(x) dx ( Rd h(x) q ) 1/q ( ˆΦ(x) dx R d ) 1/ ˆΦ(x) /q dx <. Thus, the inverse Fourier transform of h given as ȟ(x) = (2π) d/2 R d h(y)e ixt y dy is well-defined and an element of C(R d ) SI. This indicates that ˆȟ = h and ȟ B Φ (Rd ) because ˆȟ, γ S = h, ˇˆγ S = h, γ S for all γ S. Therefore B Φ (Rd ) is isometrically equivalent to L q (R d ; µ). Using ˆΦ q/ = ˆΦ q 1 L 1 (R d ) we can also rove that B q Φ (Rd ) L (R d ; µ) in an analogous way. Therefore B q Φ (Rd ) is isometrically equivalent to the dual sace of B Φ (Rd ). We fix any y R d. The Fourier transform of K(, y) is equal to ˆk y (x) := ˆΦ(x)e ixt y. Since ˆΦ 1 L 1 (R d ) we have ˆk y L (R d ; µ). Thus K(, y) can be seen as an element of B q Φ (Rd ) B Φ (Rd ). In addition, K(x, ) B Φ (Rd ) for any x R d because ˆΦ q 1 L 1 (R d ) and ( K(x, ) )ˆ= ˆk x L q (R d ; µ) by Φ = Φ( ). Finally, we verify the right-sided reroduction. Fix any f B Φ (Rd ) and y R d. We can verify that fˆ L 1 (R d ) as in the above roof. Moreover, the continuity of f and f ˇˆ allows us to recover f ointwise from its Fourier transform via Thus, we have f (x) = ˇˆ f (x) = (2π) d/2 R d ˆ f (y)e ixt y dy. f, K(, y) B Φ (Rd ) = ˆ f, ˆk y Lq (R d ;µ) = =(2π) d/2 R d R d ˆ f (x)ˆk y (x)dµ(x) f ˆ(x) ˆΦ(x)e ixt y dx = (2π) d/2 f ˆ(x)e ˆΦ(x) R ixt y dx = f (y). d 19

20 In the same way, we can also verify that B Φ (Rd ) satisfies the left-sided reroduction roerty, i.e., K(x, ), g B Φ (Rd ) = ˆk x, ĝ Lq (R d ;µ) = ĝ(y)ˆk x (y)dµ(y) = g(x), R d for all g B q Φ (Rd ) B Φ (Rd ) and all x R d. Therefore B Φ (Rd ) is an RKBS with the two-sided reroducing kernel K. Since B Φ (Rd ) L q (R d ; µ) is reflexive and K is even, the dual sace B Φ (Rd ) B q Φ (Rd ) is also an RKBS with the two-sided reroducing kernel K. Because L q (R d ; µ) and L (R d ; µ) are uniformly convex and smooth by [16, Theorem and Examle 5.4.8]. B Φ (Rd ) and B q Φ (Rd ) are also uniformly convex and smooth. Remark 5.2. We can combine our result with [16, Proosition 1.9.3] to conclude that the restriction of B Φ (Rd ) to the reals is also an RKBS with the two-sided reroducing kernel K and its dual is isometrically equivalent to the restriction of B q Φ (Rd ) to the reals. It is well-known that the RKHS of a given reroducing kernel is unique. Theorem 5.1, however, shows that different RKBSs may have the same reroducing kernel. We will rovide an examle for this in Section 6. Moreover, the roof of Theorem 5.1 rovides that B Φ (Rd ) with 2 is still a right-sided RKBS without the additional condition ˆΦ q 1 L 1 (R d ). According to [21, Theorem 10.10] any ositive definite kernel can be used to construct an RKHS. We may extend the ositive definite kernel into an RKBS. Corollary 5.2. Let B Φ (Rd ) with 2 be defined in Theorem 5.1. Then B Φ (Rd ) L (R d ). Proof. We fix any f B Φ (Rd ). According to the roof of Theorem 5.1, we have fˆ L q (R d ) because f ˆ(x) q f ˆ(x) q ( ) dx (2π) qd/2 R d R ˆΦ(x) dx su ˆΦ(x) <. d x R d The Hausdorff-Young inequality [12, Theorem ] rovides that f = L (R d ) because 1 < q 2. Remark 5.3. The RKBS B Φ (Rd ) with 2 can be recisely written as B Φ (Rd ) := { f L (R d ) C(R d ) : the distributional Fourier transform ˆ f of f is a measurable function defined on R d such that ˆ f / ˆΦ 1/q L q (R d ) }. 20 f ˇˆ

21 However, B Φ (Rd ) L (R d ) with 1 < < 2 because the Hausdorff-Young inequality does not work for q > 2. We fix any ositive number m > d/2. According to [21, Corollary 10.13], if there are two ositive constants C 1, C 2 such that ( ) C x 2 m/2 ( 2 ˆΦ(x) 1/2 C x 2 m/2 2), x R d, then the RKHS B 2 Φ (Rd ) H Φ (R d ) and the classical L 2 -based Sobolev sace W m 2 (Rd ) H m (R d ) of order m are isomorhic, i.e., H Φ (R d ) H m (R d ). Following the ideas of RKHSs, we can also find a relationshi between RKBSs and Sobolev saces. Let f m (x) := ( 1 + x 2) 2 m/2 f ˆ(x) with 2. The theory of singular integrals then shows that f belongs to the classical L -based Sobolev sace W m (R d ) of order m if any only if the function f m is the Fourier transform of some function in L (R d ), and the L -norm of the inverse Fourier transform f m is equivalent to the W m -norm of f (much more detail is mentioned in [1, Section 7.63] and [12, Section 7.9]). Using the Hausdorff-Young inequality, we can get f W m (R d ) C fˇ L m C f (R d ) m Lq (R d ) for some ositive constant C indeendent of f. Following these statements, we can introduce the following corollary. Corollary 5.3. Let the ositive definite function Φ be as in Theorem 5.1 and W m (R d ) be the classical L -based Sobolev sace of order m > d/q d/q. Here q is the conjugate exonent of 2. If there are two ositive constants C 1, C 2 such that ( ) C x 2 m/2 ( 2 ˆΦ(x) 1/q C x 2 m/2 2), x R d, then B Φ (Rd ) is embedded into W m (R d ), i.e., f W m (R d ) C f B Φ (Rd ), f B Φ (Rd ) W m (R d ), for some ositive constant C indeendent on f. Remark 5.4. Here the lower bound for m is induced by the condition that ˆΦ q/ L 1 (R d ). According to Corollary 5.3, the dual sace Wq m (R d ) of the Sobolev sace W m (R d ) is embedded into the dual sace B Φ (Rd ) of the RKBS B Φ (Rd ). It is well-known that the oint evaluation functional δ x belongs to Wq m (R d ) (see [1, Section 3.25]) which coincides with δ x B Φ (Rd ). Since K(, x 1 ),..., K(, x N ) are linearly indeendent in B q Φ (Rd ) B Φ (Rd ) for any airwise distinct data oints X = {x 1,..., x N } R d, δ x1,..., δ xn are linearly indeendent on B Φ (Rd ). Combining Theorems 4.2 and 5.1, we can solve the emirical SVM solution in B Φ (Rd ) with > 1. 21

22 Theorem 5.4. Let B Φ (Rd ) with > 1 be defined as in Theorem 5.1 and the regularization function R : [0, ) [0, ) be convex and strictly increasing. We choose the loss function L : R d C C [0, ) such that L(x, y, ) is a convex ma for any fixed x R d and any fixed y C. Given the data D := {(x 1, y 1 ),..., (x N, y N )} with airwise distinct data oints X = {x 1,..., x N } R d and associated data values Y = {y 1,..., y N } C, the unique otimal solution (suort vector machine solution) s D,L,R of min f B Φ (Rd ) L ( x j, y j, f (x j ) ) + R ( ) f B Φ (Rd ), (5.2) has the exlicit reresentation 2 s D,L,R (x) = (2π) d/2 ˆΦ(y) 1 c k e i(x x k) T y c l e ixt l y dy, x R R d k=1 l=1 d, (5.3) for some coefficients c 1,..., c N C and i 2 = 1. Proof. Using Theorems 4.2 and 5.1, the dual element of the SVM solution s D,L,R of the SVM (5.2) is a linear combination of K(, x 1 ),..., K(, x N ), i.e., s D,L,R(x) = b k K(x, x k ) = b k Φ(x x k ), x R d, b := (b 1,, b N ) T C N. k=1 k=1 Suose that s D,L,R is not trivial. According to the roof of Theorem 5.1, the identity element of s D,L,R Bq Φ (Rd ) in L (R d ; µ) is the Fourier transform of s D,L,R, i.e., f s (x) := F ( ) s D,L,R (x) = b k ˆΦ(x)e ixt x k, x R d. The dual element of f s L (R d ; µ) in L q (R d ; µ) has the form k=1 fs (x) = f s(x) f s (x) 2, x R d. f s 2 L (R d ;µ) Because the dual element of s D,L,R in B Φ (Rd ) is equal to the identity element of fs L q (R d ; µ) in B Φ (Rd ), which is the inverse Fourier transfer of fs, we can determine that s D,L,R (x) = F 1 ( fs ) 2 (x) = (2π) d/2 ˆΦ(y) 1 c k e i(x x k) T y c l e ixt l y dy, R d 22 k=1 l=1

23 2 1 and the coefficients are given by c k := f s b L (R d ;µ) k k = 1,..., N, where q is the conjugate exonent of. = sd,l,r q 2 B Φ (Rd ) b k for all Remark 5.5. In articular, if is an even ositive integer, then s D,L,R is also a linear combination of some kernel function translated to the data oints X. For examle, when = 4, then s D,L,R = N,N,N k 1,k 2,k 3 =1 c k1 c k2 c k3 Φ 3 ( xk1 + x k2 x k3 ) = N,N,N k 1,k 2,k 3 =1 c k1 c k2 c k3 K 3 (, xk1, x k2, x k3 ), where the kernel function K 3 (x, y 1, y 2, y 3 ) := Φ 3 (x y 1 + y 2 y 3 ) and Φ 3 is the inverse Fourier transform of ˆΦ 3. Moreover, s 4/3 D,L,R = B sd,l,r 2/3 [s Φ (Rd ) B Φ (Rd ) D,L,R, s D,L,R ] B Φ (Rd ) = sd,l,r 2/3 s B Φ (Rd ) D,L,R, s D,L,R B Φ (Rd ) = c j s D,L,R, K(, x j ) B Φ (Rd ) = N,N,N,N j,k 1,k 2,k 3 =1 c j c k1 c k2 c k3 K 3 ( x j, x k1, x k2, x k3 ). We can observe that the coefficients of the SVM solution s D,L,R given in Theorem 5.4 differ from the coefficients of its dual element s D,L,R only by a constant factor. As in Corollary 4.3, the coefficients of s D,L,R can also be comuted by the fixed oint iteration method. For any fixed c := (c 1,, c N ) T C N, we can define a unique function s c B Φ (Rd ) as in Equation (5.3). Let φ j (c) := s c (x j ) = (2π) d/2 R d ˆΦ(y) 1 for all j = 1,..., N, and φ := (φ 1,, φ N ) T. Thus we have 2 c k e i(x j x k ) T y c l e ixt l y dy, c C N, k=1 l=1 s c q B Φ (Rd ) = s c q 2 B Φ (Rd ) s c, s c B Φ (Rd ) = c j s c, K(, x j ) B Φ (Rd ) = c φ(c). Here q is the conjugate exonent of. Denote that T D,L,R (c) := L ( x j, y j, φ j (c) ) +R ( (c φ(c)) 1/q) = L ( x j, y j, s c (x j ) ) +R ( ) s c B Φ (Rd ). 23

24 It is easy to check that the coefficients of s D,L,R are the minimizers of T D,L,R over C N, i.e., c ot := argmin c C N T D,L,R (c) such that s D,L,R = s cot. Suose that L(x, y, ) C 1 (C) for all x R d and all y C, R C 1 ([0, )) and 2. We can comute the gradient of T D,L,R by Wirtinger artial derivatives in the form T D,L,R (c) T = l D (φ(c))t φ(c) + R ( (c φ(c)) 1/q ) 2q (c φ(c)) 1/ c φ(c), where l D (φ) := (L (x 1, y 1, φ 1 ),, L (x N, y N, φ N )) T and the entries of the Jacobian (gradient) matrix φ := ( ) N,N c k φ j by Wirtinger artial derivatives have the forms j,k=1 φ j (c) = c k 2 (2π) d/2 R d 2 ˆΦ(y) 1 e i(x j x k )T y c l e ixt l y dy. Moreover, c ot is the stationary oint of T D,L,R which indicates that c ot is a fixed oint of the function F D,L,R (c) := c + T D,L,R (c), c C N. (5.4) Therefore, we can introduce the following corollary. Corollary 5.5. Suose that the loss function L(x, y, ) C 1 (C) for all x R d and all y C, the regularization function R C 1 ([0, )) and 2. Then the coefficient vector c of the suort vector machine solution s D,L,R given in Theorem 5.4 is a fixed oint of the function F D,L,R defined in Equation (5.4), i.e., F D,L,R (c) = c. Remark 5.6. The coefficients c := (c 1,, c N ) T of the SVM solution s D,L,R in B Φ (Rd ) differ from the coefficients b := (b 1,, b N ) T of its dual element s D,L,R in B q Φ (Rd ) only by a constant factor. Both coefficient vectors b and c are fixed oints of the functions F D,L,R as in Equation (4.3) and F D,L,R as in Equation (5.4), resectively. Roughly seaking, F D,L,R can be seen as a conjugate of F D,L,R. Much more contents of these fixed oint iteration algorithms for the binary classification roblems will be deely discussed in our next aers. We now use the techniques of [3, Theorem 6] to set u a two-sided RKBS defined on a subset Ω of R d. 24 l=1

25 Theorem 5.6. Let the ositive definite function Φ be as in Theorem 5.1 and Ω R d. Then the function sace B Φ (Ω) := { h : there exists a function h B Φ (Rd ) such that f Ω = h }, equied with the norm h B Φ (Ω) := inf f f B B Φ (Rd ) Φ (Rd ) s.t. f Ω = h, is a reroducing kernel Banach sace with the two-sided reroducing kernel K R d Ω(x, y) := Φ(x y), x R d, y Ω, where f Ω stands for the restriction of f to Ω. Its dual sace B Φ (Ω) is isometrically equivalent to a closed subsace of B q Φ (Rd ) (the annihilator of N 0 in B q Φ (Rd )) N 0 = { g B q Φ (Rd ) B Φ (Rd ) : f, g B Φ (Rd ) = 0, for all f N 0 }, where q is the conjugate exonent of > 1 and N 0 := { f B Φ (Rd ) : f Ω = 0 }. Moreover, B Φ (Ω) is uniformly convex and smooth. Proof. Since convergence in a two-sided RKBS B Φ (Rd ) imlies ointwise convergence, we can determine that N 0 is a closed subsace of B Φ (Rd ). According to the construction of B Φ (Ω), B Φ (Ω) is isometrically equivalent to the quotient sace B Φ (Rd ) / N 0 (see [16, Definition and 1.7.3]). Thus B Φ (Ω) is a Banach sace by [16, Theorem and Corollary ]. Next we use the identification of ( B Φ (Rd ) / ) N 0 N 0 to verify the two-sided reroduction (see [16, Theorem ]). Let K be the reroducing kernel of B Φ (Rd ) given in Theorem 5.1. We fix any y Ω. Since f, K(, y) B Φ (Rd ) = f (y) = 0, for all f N 0, we have K(, y) N 0 ( B Φ (Rd ) / ) N 0 B Φ (Ω). Combining this with the rightsided reroduction of B Φ (Rd ), we have h, K(, y) B Φ (Ω) = Eh, K(, y) B Φ (Rd ) = (Eh)(y) = h(y), 25

26 for all h B Φ (Ω) and all y Ω, where E is the extension oerator from B Φ (Ω) into B Φ (Rd ) such that Eh Ω = h and Eh B Φ (Rd ) = h B Φ (Ω). Since K(x, ) Ω B Φ (Ω) for all x R d, we can also obtain the left-sided reroduction of B Φ (Ω), i.e., K(x, ) Ω, g B Φ (Ω) = K(x, ), g B Φ (Rd ) = g(x), for all g N 0 B Φ (Ω). Therefore B Φ (Ω) is an RKBS with the two-sided reroducing kernel K R d Ω. Since B Φ (Rd ) is uniformly convex, [16, Theorem ] rovides that B Φ (Ω) B Φ (Rd ) / N 0 is uniformly convex. We also know that B Φ (Rd ) L q (R d ; µ) is uniformly convex and N 0 is a closed subsace of Bq Φ (Rd ) B Φ (Rd ) by [16, Proosition ]. Combining with [16, Proosition and 5.4.5], we can also check that B Φ (Ω) is smooth. Remark 5.7. When = 2, then we know that B 2 Φ (Ω) is a Hilbert sace by Theorem 5.1. Thus the dual sace and the sace itself are isometrically isomorhic such that the reroducing kernel becomes K Ω Ω. Since B 2 Φ (Rd ) = N 0 N 0, we can determine that { } g Ω : g N 0 = B 2 Φ (Ω) and g B 2 Φ (Rd ) = g Ω B 2 Φ (Ω) for all g N 0 which imlies that B2 Φ (Ω) N 0 B 2 Φ (Ω) and B 2 Φ (Ω) has the inner roduct (h 1, h 2 ) B 2 Φ (Ω) = h 1, h 2 B 2 Φ (Ω) = Eh 1, Eh 2 B 2 Φ (R d ) = (Eh 1, Eh 2 ) B 2 Φ (R d ), for all h 1, h 2 B 2 Φ (Ω). Therefore B2 Φ (Ω) is an RKHS. Moreover, since K(, y) N 0 for any y Ω, we have E (K(, y) Ω) = K(, y). This shows that K Ω Ω is a reroducing kernel of B 2 Φ (Ω). This conclusion is the same as in [3, Theorem 6]. If the RKBS is even a Hilbert sace, then we can choose an equivalent function sace of its dual as itself such that its reroducing kernel has symmetric domains. The difficulty to find an equivalent function sace of the dual of RKBS, which is defined on the same domain of the RKBS, causes the domains of its reroducing kernel to be nonsymmetric. Theorems 5.1 and 5.6 rovide us with examles of symmetric and nonsymmetric reroducing kernels of RKBSs, resectively. Suose that the ositive definite function Φ given in Theorem 5.1 has a comact suort Ω Φ. Because of the ositive definite roerties of Φ, its suort su(φ) = Ω Φ with the origin is symmetric and bounded. Let Ω 1 and Ω 2 be two subsets of R d such that the comlement Ω c 1 includes Ωc 2 +Ω Φ. We fix any γ S so that its suort su(γ) Ω c 2. Since the convolution function γ Φ B Φ (Rd ) and 26

27 its suort su(γ Φ) su(γ) + su(φ) Ω c 2 + Ω Φ Ω c 1, we can determine that γ Φ N 0 with Ω := Ω 1. For any g N 0, we have γ Φ(x)ĝ(x) γ(x)g(x)dx = ˆγ(x)ĝ(x)dx = dx = γ Φ, g B R d R d R ˆΦ(x) d Φ (Rd ) = 0 which indicates that g Ω c 2 = 0. According to this result we can deduce that g = 0 if and only if g N 0 and g Ω 2 = 0. This means that the restriction ma of N 0 to Ω 2 is one-to-one. Thus the normed sace B(Ω 2 ) := { φ : Ω 2 C : φ = g Ω2 for some g N } 0 equied with the norm φ B(Ω2 ) := g q B Φ (Rd ) is well-defined and it is obvious that B(Ω 2 ) N 0. Under these additional conditions, the dual sace of B Φ (Ω 1) defined in Theorem 5.6 can be even isometrically equivalent to a sace comosed of functions defined on Ω 2, i.e., B Φ (Ω 1) N 0 B(Ω 2). In this case B Φ (Ω 1) is also an RKBS with the two-sided reroducing kernel K Ω2 Ω 1. Corollary 5.7. Suose that the ositive definite function Φ given in Theorem 5.1 has a comact suort Ω Φ in R d. Let Ω 1 and Ω 2 be two subsets of R d such that the comlement Ω c 1 includes Ωc 2 +Ω Φ. Then B Φ (Ω 1) with > 1 defined in Theorem 5.6 is a reroducing kernel Banach sace with the two-sided reroducing kernel K Ω2 Ω 1 (x, y) := Φ(x y), x Ω 2, y Ω 1. If the subset Ω is a regular domain, then the definition of weak derivatives (see [1, Section 1.62]) rovides that f Ω W m (Ω) and f Ω W m (Ω) f W m (R d ) for all f W m (R d ), where W m (Ω) is the L -based Sobolev sace of order m. Now we use the embeddings of B Φ (Rd ) given in Corollary 5.3 to derive the embeddings of B Φ (Ω). We fix any h B Φ (Ω). According to Corollary 5.3, we have h W m (Ω) Eh W m (R d ) C Eh B Φ (Rd ) = C h B Φ (Ω), for some ositive constant C indeendent on h. h B Φ (Ω) Wm (Ω), Corollary 5.8. Let Φ be a ositive definite function and m > d/q d/q be as in Corollary 5.3. Here q is the conjugate exonent of 2. Suose that Ω R d is regular. Then B Φ (Ω) defined in Theorem 5.6 is embedded into the L -based Sobolev sace of order m, W m (Ω), i.e., h W m (Ω) C h B Φ (Ω), for some ositive constant C indeendent on h. 27 h B Φ (Ω) Wm (Ω),