Explicit overall risk minimization transductive bound

Transcription

1 1 Expicit overa risk minimization transductive bound Sergio Decherchi, Paoo Gastado, Sandro Ridea, Rodofo Zunino Dept. of Biophysica and Eectronic Engineering (DIBE), Genoa University Via Opera Pia 11a, Genoa, Itay Summary. Aside cassica inductive methods transduction has reached an aways increasing attention from the scientific community because of its earning paradigm. Expicit error bounds for inductive methods are we estabished resuts and stem from Vapnik theory or Rademacher compexity. In this work we address the probem of buiding an expicit form of the transductive bound presented in Vapnik Overa Risk Minimization approach. 1 Introduction In recent years, approaches aternatives to fu induction have reached an aways increasing attention from the machine earning research community [1]. Inductive methods find a goba soution from empirica data and buid genera mode appicabe a over the popuation. Beside inductive earning schemes, exist the so caed transductive earning: in this environment is not required generaization for every possibe input, instead ony achieving the best possibe performance on a particuar and known test data. This, intuitivey, makes transduction simper than induction, since what is request are vaues at given points [] and not a goba predictive function.

2 Sergio Decherchi, Paoo Gastado, Sandro Ridea, Rodofo Zunino Transduction and semi-supervised earning are quite different concepts: in the first setting we are interested in finding vaues at given points and no more, in semisupervised earning we are interested in producing a decision function by using abeed and unabeed data: a transductive agorithm can perform predictions ony on working set, a semi-supervised one can predict a over the popuation so it is competey inductive. The importance of transduction is due to different reasons: one of them is its fundamenta part over the inductive approach itsef. We known Vapnik cassica bound, impicity makes use of transduction when concerned with ghost set. In transductive setting, the ghost set is rea and is the set of given points in which predictions are performed. Another reason stems from the possibiity to take advantage of this new simper setting to get tighter bounds on generaization error over a particuar working set. In this work this second aspect wi be studied: adapting the machinery of Theorem 4. [] and a reativey recent resut [3] an expicit formua wi be obtained for overa risk minimization bound. In the first part of the paper computationa issues wi be discussed over the numerica evauation of transductive bound in its impicit origina form and a cosed form formua wi be obtained; in the second one the resut wi be compared to other existing bounds. The same symboic conventions of [] wi be used throughout the paper: + k is the tota number of patterns; are abeed and k are unabeed ν τ is the transductive error (the error over test or working sampe); ν is the error on training set; ν is the error on the ghost set; than we ca ν 0 = ν + ν τ k, and ν α = ν+ν m is the tota numnber of errors and can be epxressed as ν + ν τ k G() is the Grow Function computed for ; Hann() Λ is the anneaed Entropy 1 δ is the confidence eve of the bound Cm r is the binomia coefficient Γ,k (ε, m) is a quantity derived from the hypergeometric distribution; than we ca Γ,k (ε) = max m Γ,k (ε, m); E(Γ ) is the expectation of the hypergeometric ; N is the finite number of equivaence casses Overa Risk Minimization Transduction The Overa Risk Minimization framework is part of the more genera Statistica Learning Theory and is one of the possibe approaches to transduction. In [] transduction is introduced and two possibe settings (Setting 1 and Setting ) are exposed: it can be shown that both of them are equivaent []. The fundamenta idea, on which ORM is buit up, is that is useess soving a more difficut probem when a simper one is needed to be soved. From a mathematica point of view in ORM we are endowed with a training abeed set, an unabeed working set on which we want to perform predictions and it is aowed to use both of them during training. This fundamenta theorem gives an impicit bound for transduction error Theorem.1. (Theorem 8. in []) Let the set of decision rues f(x, α), α Λ on the compete set of vectors have N equivaence casses. Then the probabiity that the

3 1 Expicit overa risk minimization transductive bound 3 reative size of deviation for at east one rue in f(x, α), α Λ exceeds ε is bounded by: { } ν ν τ P sup > ε < N Γ (ε) (1) α Λ ν0 Now, as said in the introductory section, we want to ink the equipment of induction to Theorem 8. []: in Statistica Learning Theory the main inductive resut is Theorem 4.1 []; the key part in which we are interested in, is Lemma 4. []. Lemma 1. (Lemma 4. in []) For any > ε is vaid the foowing bound: { } { } ν ν 1 P να + 1/() > ε < exp Hann() Λ ε 4 sup α Λ further { by using } the property Hann() Λ < G() for right hand side we get exp G() ε 4. To make ORM approach consistent with machinery of Lemma 4. [] we need to repace origina Vapnik gamma function argument ε m with ε m+1 m+1 (as in Lemma 4. [] proof where we have ε ). This margina modification eads aso to modify (1) into P sup ν ν τ > ε α Λ ν < N Γ (ε) (3) () and the fina expicit formua becomes: ν τ ν + kε ( + k) + ε [ ] kε + ν + 1 ( + k) + k (4) This modification makes Theorem 8. [] consisten with Lemma 4. [] at price of adding the term 1/( + k) over the origina formuation; as wi be seen ater this adaptation open the possibiity to buid up a pain proof that makes ε term expicit. After this simpe variation we tried to appraise the impicit bound derived in Theorem 8. []. For its evauation one has to find the smaest soution of n N + n Γ (ε) < n δ and pug it into the bound; so one has to sove this equation for trias performing discretization on ε. Before proceeding it is necessary to expicity compute the number of equivaence casses: for this purpose we used the fact that n N < G( + k). This approach presents some performance probems when the number of patterns ( or k) is over 1e3. The main issue consists in the expicit evauation of the gamma function: its cacuation pans the evauation of three binomia coefficients. We impemented this computation using Stiring and Ramanujan formuas and ogarithmic representations, but despite this, the execution time is quite high when deaing with data mining probems. As suggested by Vapnik itsef, gamma function can be tabuated but it shoud be preferabe having a simper and expicit way of computing the bound. Athough these concerns, evauation of the bound has been possibe via iterative search of the soution. For the exposed reason a more practica soution consists in deriving an expicit bound.

4 4 Sergio Decherchi, Paoo Gastado, Sandro Ridea, Rodofo Zunino 3 Bound derivation The subsequent theorem is the centra resut of this work: it foows Vapnik demonstration for the cassica inductive bound (Theorem 4. [], Lemma 4. []) and readapts it to the transductive issue (Theorem 8. []) using a quite recent statistica resut. Theorem 3.1. (Expicit bound). Assured that G( + k) n δ > 6, and setting ε = ( + k) 3, with probabiity 1 δ, the bound in (4) is vaid. G() n δ+ (k) Proof. Suppose having a popuation of +k patterns in which there are m miscassified patterns. We seect randomy of them. The probabiity that among the seected patterns there are r errors equas Cr m C r m C k. The probabiity that the frequency of miscassified patterns in the first group () deviates from the frequency of errors in the second group (k) by the amount exceeding ε equas: { r P m r } k > ε = CmC r r m C k = Γ,k ( ε, m) (5) r Where the sum is taken over the vaue of r such that: max(0, m k) r min(, m), r m + k > ε k + k Note that both sides are aways greater than 0. From [3] is known that: if (6) is true, than: ( where α = max m+1 Expressing ε = ε k+1, 1 m m+1 ( r E(Γ ) (7) n Γ,k ( ε, m) < α((r E(Γ )) 1) (8) ). Knowing that E(Γ ) = m we get: Γ,k ( ε, m) < exp α ( r r we get: m > ε k m ) 1) + k. Note that beacause we ( ) ( ) need the square in (9), we can observe that by using (6) r m > ε k hods. For proceeding we have to assure that the hypothesis on hypergeometric bound (7) and (6) both hod. A simpe way for achieving this goa is to request ( ) ( ) that: r m > max ( ε k, ). Now observe that asking ε k > is a suf- ( ) ( ) ficient condition to resort to the ony r m > ε k origina condition; in this way, at the end of the proof, we wi have to check for what vaues the expression ε k > is true. With these hypothesis we can bound the hypergeometric on (3) { } { { ( ( ) )}} ν ν getting: P sup τ α Λ > ε < N max m exp α r m 1 than we get: ν m+1 (9)

5 P sup α Λ ν ν τ ν Expicit overa risk minimization transductive bound 5 ( > ε < N max m exp α ε k ) m k + k (10) It can be easiy shown, e.g. by potting the function for different m,, k vaues, (see fig.1) that hypergeometric dependent part of the previous formua is maximized for m = 0 (as happens in Vapnik inductive proof). This fact (m = 0) makes α = ( max k+1, ), that is the same that saying that α > 1; for this reason we can repace α with 1. Observe that this operation on α sighty affects the quaity of the bound, in facts in amost a rea word probems (e.g, k > 10) α 1 hods. Fig. 1 Exponentia part of the bound for different vaues of and k with m variabe and ε = 0.4. Soid ine represents the case k =. Setting m = 0 and resembing that n N < G( + k) we obtain: P sup ν ν τ ( )) > ε α Λ ν < exp G( + k) (ε (k) ( + k) 3 1 (11) Remember that the right part of the above inequaity is δ. So expressing a in terms of ε, we get: ε = ( + k) 3. Puggin this formua in (4) we get the fina G() n δ+ (k) expression of the bound where ε is expicit. Finay we have to check the correctness of ε k we have to verify that ε > 4 ()3 (k). So we get ε = > hypothesis. In other terms G() n δ+ (k) that produces the condition of the theorem: G( + k) n δ > 6. ( + k) 3 > 4 ()3 (k)

6 6 Sergio Decherchi, Paoo Gastado, Sandro Ridea, Rodofo Zunino 4 Vauation and experimenta resuts The obtained resut is vaid when the Grow function is expicity known. If Grow Function is not exacty known, Sauer emma can be used to get a bound in terms of Vapnik-Chervonenkis dimension d vc that eads to: ν τ ν + β 4( k) ( + k) + β ( + k)3 (k) ν + k β ( + k) (1) 8(k) ) where β = d vc (1 + n d vc n δ +. Note aso that a when typica confidence vaue of.95 is used, the theorem hypothesis is G( + k) > 3, that is very ikey to happen in practice. There are others aspects that need anaysis: first of a it is appropriate to observe that for k = the bound becomes: G() n δ + ν τ ν + G() n δ + + ν + G() n δ + 4 From a cognitive and mathematica point of view keeping k = and requesting, k big enough makes the above bound quite simiar to origina Vapnik inductive bound; these bounds became very simiar when the Grow Function is far ess, in absoute vaue, than the number of patterns (e.g. this can happen in custering based cassifiers). For competeness of information origina Vapnik formuation was: G() n δ + n π ν + + G() n δ + n ν + (13) G() n δ + n (14) It is important to note down that in this case (k = ) the obtained bound is aways convenient over induction (see figure). Roughy speaking expicit transduction bound is convenient over induction in this case because we did not pay the price of the ghost set, because ghost set in this setting exists and it is represented by k patterns. When k and are unbaanced this advantage is ost due to the behaviour of the hypergeometric distribution. Now we want to present a comparison of the obtained resut respect the bounds obtained in [4] via PAC-Bayesian arguments. In our experimenta environment we aways choose 1 δ =.95 and p(h) = 1/3 so getting N = exp(3) for the number of possibe functions to satisfy hypothesis of our theorem. The bounds that we are going to compare are: the bound of this work, coroary 3 bound [4] and Serfing bound [4]. We propose the same experiments performed on figure 1 in [4]. As can be seen the obtained bound is tigther respect those obtained in [4] in 3 cases over 4 and this confirms the vauabiity of the resut. 5 Concusions Here we presented a possibe approach for buiding an expicit and simpe to use transductive bound by using ony Vapnik theory and without requiring any bayesian

7 1 Expicit overa risk minimization transductive bound 7 Fig. Experiment for k = variabe. Note the advantage of transductive bound (soid ine) over induction (dashed ine) approach. We don t caim that is the best possibe transductive bound, instead we want to underine the feasibiity of the resut respect to the transductive issue and respect more compicated arguments. Other improvements are possibe in two directions: by using better concentration inequaities of the hypergeometric distribution or by using a Rademacher compexity approach, that at our knowedge, for now is ony conceived in induction probems. Another interesting direction of research is trying to reproduce Vapnik machinery over the probem of buiding semi-supervised generaization error bounds: this aspect is a competey open probem and much theory acks for a broad understanding. References 1. Chapee, O., B.Schokopf, A.Zien: A Discussion of Semi-Supervised Learning and Transduction. In: Semi-Supervised Learning. MIT Press (006). Vapnik, V.: Estimation of Dependences Based on Empirica Data. Springer-Verag (198) 3. D.Hush, C.Scove: Concentration of the hypergeometric distribution. Statistics Probabiity Letters 75 (005) P.Derbeko, R.E-Yaniv, R.Meir: Expicit earning curves for transduction and appication to custering and compression agorithms. Journa of Artificia Inteigence Research (004)

8 8 Sergio Decherchi, Paoo Gastado, Sandro Ridea, Rodofo Zunino Fig. 3 Same experiments as in [4]. Note the advantage of the obtained bound (dashed ine) over the other bounds (3 cases over 4).