Yoram Gat. Technical report No. 548, March Abstract. A classier is said to have good generalization ability if it performs on

Transcription

1 A bound concerning the generalization ability of a certain cla of learning algorithm Yoram Gat Univerity of California, Berkeley Technical report No. 548, March 999 Abtract A claier i aid to have good generalization ability if it perform on tet data almot a well a it doe on the training data. The main reult of thi paper provide a ucient condition for a learning algorithm to have good nite ample generalization ability. Thi criterion applie in ome cae where the et of all poible claier ha innite VC dimenion. We apply the reult to prove the good generalization ability of upport vector machine. Introduction I conider the claical problem of learning a claier from example which can be formalized a follow: Let Z i =( i ;Y i );; 2;:::be iid random variable taking value in Z = f,; +g. The problem i predicting Y l+ given ;:::; l+ and Y ;:::;Y l. The olution to the problem i a map M : Z l!f, where F i a pace of claier function, i.e., each f 2Fi a function f :!f,; +g. Thu the prediction i Y = f l+ ( l+ ) where f = M(Z ;:::;Z l ). The quality of the olution may be meaured uing the expected error rate (alo called expected rik): EER = P(Y l+ 6= Y l+ ): The olution M i uually geared toward nding a function which ha low empirical error rate (alo called empirical rik): EMER = 2l jf ( i ), Y i j : Therefore, it i often deirable to be able to obtain bound for the dierence between the empirical and the expected error rate. The behavior of the dierence will depend on the underlying, unknown probability meaure. The term generalization ability i ued to decribe the wort-cae behavior of the dierence between the empirical and expected error rate for a pecic algorithm. The AMS 99 ubject claication. Primary 62H30. Key word and phrae. Generalization ability, upport vector machine, VC dimenion, perceptron algorithm.

2 maller the probability for a large dierence, the better i the generalization ability of the algorithm. One map M commonly ued i M(Z ;:::;Z l ) = arg min f2f jy i, f( i )j : Thi i known a the Empirical Rik Minimization (ERM) method. It ha been hown that the generalization ability of the algorithm can be determined by uing the VC dimenion of the et of function F ([] ec. 4.9). Other learning algorithm ue map of the form M(Z ;:::;Z l )=arg min f2 M(Z ~ ;:::;Z l) jy i, f( i )j ; where ~ M i an auxiliary map ~ M : l! 2 F. I call thi type of algorithm Retricted Empirical Rik Minimization (RERM) rule. The main reult The following theorem guarantee the generalization ability of certain learning algorithm even when F ha an innite VC dimenion: Theorem Denote M(z ;:::;z 2l ) = M(z i() ;:::;z i(l) ):the i(j)' are l ditinct indice in the range ;:::;2lg : If then up z ;:::;z 2l2Z M(z ;:::;z 2l ) = c(l); P(jEER, EMERj >) < 2c(l)exp,(l 2, 2): Proof: Since for any Binomial variable, B, P(B > EB +) < 0:5, it i enough to bound p 0 = P( 2l 2l jf ( i ), Y i j,emer >0 ); where 0 =, l. Thi i done by conditioning on the value of z i;;:::;2l and then taking the expectation over the dierent poible order. 2

3 To implify the formula, I ue below f (z) a horthand for jf(x), yj =2, where z =(x; y). Thu f (z) i either 0 or. p 0 = E ( f (Z (i) ), 2l f (Z (i) ) >l0 ): P Here, a below, mean umming over all permutation of the number ;:::;2l. p 0 E E E f2m (Z ;:::;Z 2l) c(l) exp,l 02 ( up f2m (Z ;:::;Z 2l) f2m(z ;:::;Z 2l) c(l) exp,(l 2, 2): ( ( f (Z (i) ), f (Z (i) ), f (Z (i) ), 2l 2l 2l f (Z (i) ) >l0 ) f (Z (i) ) >l0 ) f (Z (i) ) >l0 ) The bound for the fraction of permutation giving a dierence greater than 0 wa calculated by Vapnik ([] ec. 4.3). 2 The proof above follow the argument of Theorem 4. of [], which deal with the generalization ability of ERM algorithm. The main dierence i the reference to the random et M(Z ;:::;Z 2l ) rather than to a xed et of function. Two variant of the reult tated in Theorem 4. of [] are Theorem 4.2 of [] and the main reult of [2]. The rt give better bound when the empirical error rate i mall, and the other give a better rate of convergence when c(l) i polynomial. Both can be adapted and proven for the etup here in a manner imilar to that of Theorem. The next reult follow immediately from Theorem : Corollary For map M of the RERM type, the bound of Theorem hold provided that up M(z ~ ;:::;z 2l ) = c(l); with z ;:::;z 2l2Z ~M(z ;:::;z 2l )= [ i();:::;i(l) ~M(z i() ;:::;z i(l) ): where the i(j)' are l ditinct indice in the range ;:::;2l. Example (r-determined rule): Corollary can be ued to obtain a nontrivial generalization property for any rule of the RERM type where ~ M i of the form ~M(z ;:::;z l )= f zj();:::;z j(r) : j(i) 2f;:::;lg; ;:::;r ; 3

4 ince for any map M of thi type, M(z ~ ;:::;z 2l ) (2l) r. Below, I refer to uch rule a r-determined rule. The upport-vector etup The upport-vector machine (SVM) ([]) create a linear dicriminant claier in a ball within a high dimenional, or an innite dimenional, Euclidean pace: = fx 2R n : jxj g ; F = ff a;b (x) = ign(a x, b) :a 2R n ;b2r; jaj =g : To put the denition of an SVM into the framework preented here, I introduce the following denition: Denition The et S(x ;:::;x l ;t ;:::;t l );x i 2;t i 2f,; +g i the et of claier f a;b 2F uch that f a;b (x i )=i t i =for all i =;:::;l. In other word, the et S(x ;:::;x l ;t ;:::;t l ) i the et of claier which predict Y = t i when preented with = x i, for all i =;:::;l. Denition 2 The margin of a claier f a;b 2F with repect to a et of point x ;:::;x l 2 i dened a min ;:::;l ja x i + bj : The maximum margin claier (MMC) i the member, f a;b, of the et S with the property that it margin i the larget in the et. The value of the margin of the MMC i denoted bymarg(x ;:::;x l ;t ;:::;t l ). Uing the denition above, the SVM can now be dened a a RERM type rule with: ~M(z ;:::;z l ) = ff a;b = (x ;:::;x l ;t ;:::;t l ):t i 2f,; +g;;:::;l; marg(x ;:::;x l ;t ;:::;t l ) hg ; where (x ;:::;x l ;t ;:::;t l ) i ome member of S(x ;:::;x l ;t ;:::;t l ) and h i ome xed contant. The et M(z ~ ;:::;z l )mayormay not contain a repreentative fromthe et S(x ;:::;x l ;y ;:::;y l ). If it doe contain uch a repreentative, f, then f will have zero empirical error rate, and therefore M(z ;:::;z l ) = f will hold. If uch a repreentative i not in M(z ~ ;:::;z l )then M(z ;:::;z l ) = (x ;:::;x l ;t ;:::;t l ) for ome t ;:::;t l and the empirical error rate of the algorithm will be equal to the cardinality of the et fi : t i 6= y i g. Baed on heuritic appeal and experimental reult, i uually choen to be equal to the MMC. Here, however, I propoe a dierent way to elect a 4

5 repreentative, for which the generalization ability can be determined. Note that the empirical rik achieved i the ame for any choice of a repreentative. The algorithm below, known a the perceptron algorithm ([3]), may be ued to obtain a member of S(x ;:::;x l ;t ;:::;t l ). Let the repreentative,, bethe one produced by the algorithm. Thi algorithm had been previouly conidered in thi context by Freund and Schapire ([4]). Initialization: Set a 0;b 0;k Update: If t k (a x k + b) > 0 then go to tep Loop Correction: Set a a + t k x k ;b b + t k Loop: If a Correction tep wa not carried out in the lat l loop, top. Otherwie, et k k +(modl) and go to tep Update The Perceptron Convergence Theorem ([3]) tate that if the point x i all lie inide the unit phere, and marg(x ;:::;x l ;t ;:::;t l ) h then the algorithm will execute at mot b=h 2 c correction, after which the reulting a; b parameter will provide a member f a;b of S(x ;:::;x l ;t ;:::;t l ). By contruction the reulting claier i r-determined with r b=h 2 c. Applying the bound for r-determined rule lead to the following concluion: For any xed h, if a upport-vector method i employed and a claier with a margin of h and empirical error rate R i found, then there exit an r(h)- determined claier for which the following tatement hold: P(EER >R+ ) < 2(2l) b=h2c exp,(l 2, 2): () The perceptron algorithm can be ued to obtain uch a claier. An important point about the perceptron algorithm i that it can be executed without reference to the training vector themelve but rather making ue only of the inner product between training vector. The importance of thi property tem from the fact that often in application of the upport vector machine calculating inner product between training vector i feaible, but any explicit repreentation of the vector i prohibitively expenive. Equation () can be converted into a, upper condence bound. With probability of at leat,, the following inequality hold: log 2l EER <R+ l h 2 +log +log2e2 : (2) The upper condence bound (2) hold under the aumption that h i xed in advance. It i common practice, however, to have h random. Thi i, for example, the cae when the empirical error rate i pre-pecied (e.g. zero). A reult uitable for the cae of a random h will have the form of imultaneou upper condence bound for r = h 2 =;:::;l. Thi i obtained by imply 5

6 replacing by =l in (2), obtaining a, an upper condence bound of the following form: EER <R+ log 2l +log l l h 2 +log2e2 : (3) Since initing on a pre-pecied empirical error rate may lead to a large upper condence bound, dierent procedure may be followed. One uch procedure would be an adaptation of the perceptron algorithm: Initialization: Set a(0) 0;b(0) 0;k ;j 0;R(0) l Update: If t k (a(j) x k + b(j)) > 0 go to tep Loop Correction: Set a(j +) a(j)+t k x k ;b(j +) b(j)+t k ; j j +;R(j) jfi : t k (a x k + b) 0gj Loop: If R(j) = 0orj = l, go to tep Optimization. Otherwie, et k k +(modl) and go to tep Update Optimization: Set j = arg min R(i)+ 0ij l Set f = f a(j );b(j ). Stop i log 2l +log l +log2e2 : At the termination of the algorithm, f i a claier with empirical error rate R(j ), and which with probability ofatleat, ha expected error rate no greater than R(j )+ l j log 2l +log l + log 2e2 : Experimental reult The ue of variant of the perceptron algorithm in the upport vector context had been previouly uggeted and implemented by Freund and Schapire ([4]). They carried out experiment uing the perceptron algorithm for claifying image of handwritten digit and report error rate which are omewhat larger than thoe obtained with maximum margin claier. Acknowledgment I thank Peter Bickel for pointing out the problem which reulted in thi paper, for hi valuable comment and for having reviewed the paper. 6

7 Reference [] Vapnik, V. N. (998). Statitical Learning Theory. John Wiley & Son, Inc., New York. [2] Devroye, L. (982). Bound for the uniform deviation of empirical meaure. J. Multivariate Anal., [3] Minky, M. L. and Papert S. A. (988). Perceptron. The MIT Pre, Cambridge. [4] Freund, Y. and Schapire, R. E. (998). Large margin claication uing the perceptron algorithm. COLT '98: Proceeding of the eleventh annual conference on computational learning theory, Yoram Gat Univerity of California, Berkeley 367 Evan Hall Berkeley, CA, yoram@tat.berkeley.edu 7