(Not) Bounding the True Error

Transcription

1 (No) Bounding he True Error John Langford Deparmen of Compuer Science Carnegie-Mellon Universiy Pisburgh, PA Rich Caruana Deparmen of Compuer Science Carnegie-Mellon Universiy Pisburgh, PA Absrac We presen a new approach o bounding he rue error rae of a coninuous valued classifier based upon PAC-Bayes bounds. The mehod firs consrucs a disribuion over classifiers by deermining how sensiive each parameer in he model is o noise. The rue error rae of he sochasic classifier found wih he sensiiviy analysis can hen be ighly bounded using a PAC-Bayes bound. In his paper we demonsrae he mehod on arificial neural neworks wih resuls of a order of magniude improvemen vs. he bes deerminisic neural ne bounds. 1 Inroducion In machine learning i is imporan o know he rue error rae a classifier will achieve on fuure es cases. Esimaing his error rae can be suprisingly difficul. For example, all known bounds on he rue error rae of arificial neural neworks end o be exremely loose and ofen resul in he meaningless bound of always err (error rae = 1.0). In his paper, we do no bound he rue error rae of a neural nework. Insead, we bound he rue error rae of a disribuion over neural neworks which we creae by analysing one neural nework. (Hence, he ile.) This approach proves o be much more fruiful han rying o bound he rue error rae of an individual nework. The bes curren approaches [1][2] ofen require,, or more examples before producing a nonrivial bound on he rue error rae. We produce nonrivial bounds on he rue error rae of a sochasic neural nework wih less han examples. Our approach uses he PAC-Bayes bound [4]. The approach can be hough of as a redivision of he work beween he experimener and he heoreician: we make he experimener work harder so ha he heoreician s rue error bound becomes much igher. This exra work on he par of he experimener is significan, bu racable, and he resuling bounds are much igher. An alernaive viewpoin is ha he classificaion problem is finding a hypohesis wih a low upper bound on he fuure error rae. We presen a pos-processing phase for neural neworks which resuls in a classifier wih a much lower upper bound on he fuure error rae. The pos-processing can be used wih any arificial neural ne rained wih any opimizaion mehod; i does no require he learning procedure be modified, re-run, or even ha he hreshold funcion be differeniable. In fac, his pos-processing sep can easily be adaped o oher learning algorihms. The pos-processing sep finds a large disribuion over classifiers, which has a small average empirical error rae. Given he average empirical error rae, i is sraighforward o apply he PAC-Bayes bound in order o find a bound on he average rue error rae. We find his large disribuion over classifiers by performing a simple noise sensiivy analysis on he learned model. The noise model allows us o generae a disribuion of classifiers

2 ; ] v wih a known, small, average empirical error rae. In his paper we refer o he disribuion of neural nes ha resuls from his noise analysis as a sochasic neural ne model. Why do we expec he PAC-Bayes bound o be a significan improvemen over sandard covering number and VC bound approaches? There exis learning problems for which he difference beween he lower bound and he PAC-Bayes upper bound are igh up o where is he number of raining examples. This is superior o he guaranees which can be made for ypical covering number bounds where he gap is, a bes, known up o an (asympoic) consan. The guaranee ha PAC-Bayes bounds are someimes quie igh encourages us o apply hem here. The nex secions will: 1. Describe he bounds we will compare. 2. Describe our algorihm for consrucing a disribuion over neural neworks. 3. Presen experimenal resuls. 2 Theoreical seup We will work in he sandard supervised bach learning seing. This seing sars wih he assumpion ha all examples are drawn from some fixed (unknown) disribuion,, over (inpu, oupu) pairs,. The oupu is drawn from he space "! and he inpu space is arbirary. The goal of machine learning is o use a sample se # of pairs o find a classifier, $, which maps he inpu space o he oupu space and has a small rue error, % &$')(+*-,/.0&$21 3. Since he disribuion is unknown, he rue error rae is no observable. However, we can observe he empirical error rae, % 4 5$67(8*-,9&$':1 3 =< ; $ A1 3. Now ha he basic quaniies of ineres are defined, we will firs presen a modern neural nework bound, hen specialize he PAC-Bayes bound o a sochasic neural nework. A sochasic neural nework is simply a neural nework where each weigh in he neural nework is drawn from some disribuion whenever i is used. We will describe our echnique for consrucing he disribuion of he sochasic neural nework. 2.1 Neural Nework bound We will compare a specializaion of he bes curren neural nework rue error rae bound [2] wih our approach. The neural nework bound is described in erms of he following parameers: 1. A margin, )BDCEBF. 2. A funcion G defined by GH' 3 if FBI, GH' 3 if FJK, and linear in beween. 3. L, an upper bound on he sum of he magniude of he weighs in he M h layer of he neural nework 4. N, a Lipschiz consan which holds for he M h layer of he neural nework. A Lipschiz consan is a bound on he magniude of he derivaive. 5. O, he size of he inpu space. Wih hese parameers defined, we ge he following bound. Theorem 2.1 (2 layer feed-forward Neural Nework rue error bound) where ^ Ch 3 ; g< *P,. QSR GVi'jk lnmo ]qpsruwv yx $2TVUW % &$'XJDY@Z\[ ]_^ `Cacbedgf z {} ; N ; N L ; L r~ Proof: Given in [2]. The heorem is acually only given up o a universal consan. migh be he righ choice, bu his is jus an educaed guess. The neural nework rue error bound above is

3 4 º Œ? ; v (perhaps) he ighes known bound for general feed-forward neural neworks and so i is he naural bound o compare wih. This 2 layer feed-forward bound is no easily applied in a igh manner because we can calculae a priori wha our weigh bound L should be. This can be pached up using he principle of srucural risk minimizaion. In paricular, we can sae he bound for L ; 3 a where is some non-negaive ineger and Je is a consan. If he h bound holds wih probabiliy ˆ Š x, hen all bounds will hold simulaneously wih probabiliy f, since, we ge he following heorem: Theorem 2.2 (2 layer feed-forward Neural Nework rue error bound) 3I Ž Applying his approach o he values of boh L ; and L where ^ C } 3 ; < *-,. Q R $ TVUW % &$'XJ=YnZ[ ] / ^ `C 5 } b ddf G i jk lnmo ] p r8 v ] yx z {} ; N ; N h š œ y }žÿ 5 r Proof: Apply he union bound o all possible values of and as discussed above. In pracice, we will use 3F 3 for all and repor he value of he ighes applicable bound 2.2 Sochasic Neural Nework bound Our approach will sar wih a simple refinemen [3] of he original PAC-Bayes bound [4]. We will firs specialize his bound o sochasic neural neworks and hen show ha he use of his bound in conjuncion wih a pos-processing algorihm resuls in a much igher rue error rae upper bound. Firs, we will need o define some parameers of he heorem. 1. is a disribuion over he hypoheses which can be found in an example dependen manner. 2. is a disribuion over he hypoheses which is chosen a priori wihou dependence on he examples. 3. % &$' 3I P% kª &$' is he rue error rae of he sochasic hypohesis which, in any evaluaion, draws a hypohesis $ from, and oupus $. 4. % 4 &$' 3«% k ª 4 5$6 is he average empirical error rae of he same sochasic hypohesis. Now, we are ready o sae he heorem. Theorem 2.3 (PAC-Bayes Relaive Enropy Bound) For all priors,, *-,. Q6R W KLœ4 % 5$6 n % &$' KL& ² r ³ Z bed f where KL5 ² 3 µ &$' ³ Z¹ l@ko O $ is he Kullback-Leibler divergence beween he disribuions and and KLœ4 % % &$' is he KL divergence beween a coin of bias kp l»k o % &$' and a coin of bias % &$'. Proof: Given in [3]. We need o specialize his heorem for applicaion o a sochasic neural nework wih a choice of he prior. Our prior will be zero on all neural ne srucures oher han he one we rain and a mulidimensional isoropic gaussian on he values of he weighs in our neural nework. The mulidimensional gaussian will have a mean of and a variance in each dimension of ^. This choice is made for convenience and happens o work.

4 R Q < Ó É Ç b Ì x The opimal value of ^ is unknown and dependen on he learning problem so we will wish o parameerize i in an example dependen manner. We can do his using he same rick as for he original neural ne bound. Use a sequence of bounds where ^ 3u¼ for ¼ and some consans and a nonnegaive number. For he h bound se f²½ ˆ. Now, he union bound will imply ha all bounds hold simulaneously wih probabiliy a leas f. Now, assuming ha our poserior is also defined by a mulidimensional gaussian wih he mean and variance in each dimension defined by ¾ and, we can specialize o he following corollary: Corollary 2.4 (Sochasic Neural Nework bound) Le be he number of weighs in a neural ne, ¾ be he M h weigh and be he variance of he M h weigh. Then, we have *-,.ÁÀ W KLœ4 % 5$6 n % &$'à DY@Z\[ n? ; Ä ³ ZÆÅcÇ ÈcÉ r È É aê ; Å r ³ Z x Ë ÍÎ d f (1) Proof: Analyic calculaion of he KL divergence beween wo mulidimensional Gaussians and he union bound applied for each value of. We will choose and ¼73 S as reasonable defaul values. One more sep is necessary in order o apply his bound. The essenial difficuly is evaluing %" 4 &$'. This quaniy is observable alhough calculaing i o high precision is difficul. We will avoid he need for a direc evaluaion by a mone carlo evaluaion and a bound on he ail of he mone carlo evaluaion. Le %sð 4 5$6 (*-, ÒÑ Ð 9 &$'±1 3 be he observed rae of failure of a Ó random hypoheses drawn according o and applied o a random raining example. Then, he following simple bound holds: Theorem 2.5 (Sample Convergence Bound) For all disribuions,, for all sample ses #, *-, KLœ4 %XÐ 4 % ³ Z 5$6yX dgf where Ó is he number of evaluaions of he sochasic hypohesis. Proof: This is simply an applicaion of he Chernoff bound for he ail of a Binomial where a head occurs when an error is observed and he bias is % 4 5$6. In order o calculae a bound on he expeced rue error rae, we will firs bound he expeced empirical error rae % 4 5$6 wih confidence hen bound he expeced rue error rae % &$' wih confidence, using our bound on % 4 5$6. Since he oal probabiliy of failure is only r 3 f our bound will hold wih probabiliy f. In pracice, we will use Ó 3 evaluaions of he empirical error rae of he sochasic neural nework. 2.3 Disribuion Consrucion algorihm One criical sep is missing in he descripion: How do we calculae he mulidimensional gaussian,? The variance of he poserior gaussian needs o be dependen on each weigh in order o achieve a igh bound since we wan any meaningless weighs o no conribue significanly o he overall sample complexiy. We use a simple greedy algorihm o find he appropriae variance in each dimension. 1. Train a neural ne on he examples 2. For every weigh, ¾, search for he variance,, which reduces he empirical accuracy of he rained nework by Ô Õ (for example) while holding all oher weighs fixed. 3. The sochasic neural nework defined by¾ 7! will generally have a oo-large empirical error. Therefore, we calculae a global muliplier Ö=B such ha he sochasic neural nework defined by ¾ 7Ö'! decreases he empirical accuracy by only Ô Õ. 4. Then, we evaluae he empirical error rae of he resuling sochasic neural ne wih samples from he sochasic neural nework.

5 Ø 0.8 error SNN bound NN bound SNN Train error NN Train error SNN Tes error NN Tes error error paern presenaions paern presenaions Figure 1: Plo of errors and rue error bounds for he neural nework (NN) and he sochasic neural nework (SNN). The graph exhibis overfiing afer approximaely 6000 paern presenaions. Noe ha a rue error bound of 100 implies ha a leas ÙÚ Ú Û more examples are required in order o make a nonvacuous bound. The graph on he righ expands he verical scale by excluding he poor rue error bound. 3 Experimenal Resuls How well can we bound he rue error rae of a sochasic neural nework? The answer is much beer han we can bound he rue error rae of a neural nework. Our experimenal resuls ake place on a synheic daase which has 25 inpu dimensions and one oupu dimension. Mos of hese dimensions are useless simply random numbers drawn from a Ü ÝHÚSÞÙß Gaussian. One of he 25 inpu dimensions is dependen on he label. Firs, he label à is drawn uniformly from á â0ù Þ Ù"ã, hen he special dimension is drawn from a Ü ÝHà'Þ Ùß Gaussian. Noe ha his learning problem can no be solved perfecly because some examples will be drawn from he ail of he gaussian. The ideal neural ne o use in solving his problem is a single node percepron. We will insead use a 2 layer neural ne wih 2 hidden nodes. This overly large neural ne will resul in he poenial for significan overfiing which makes he bound predicion problem ineresing. I is also somewha more realisic if he neural ne srucure does no exacly fi he learning problem. All of our daases will use jus Ù Ú Ú examples. Consrucing a nonvacuous bound for a coninuous hypohesis space a Ù Ú Ú examples is quie challenging as indicaed by figure 1. Convenional bounds are hopelessly loose while he sochasic neural nework bound is sill no as igh as migh be desired. There are several noable hings abou his figure. 1. The SNN upper bound is 2-3 orders of magniude lower han he NN upper bound. 2. The SNN performs beer han expeced. In paricular, he SNN rue error rae is a mos äæå worse han he rue error rae of he NN. This is suprising considering ha we fixed he difference in empirical error raes a ç å. 3. The SNN bound has a minimum a paern presenaions which weakly predics he overfiing poin of 6000 for boh he SNN and he NN. The comparison beween he neural nework bound and he sochasic neural nework bound is no quie fair due o he form of he bound. In paricular, he sochasic neural nework bound can never reurn a value greaer han always err. This implies ha when he bound is near he value of Ù, i is difficul o judge how rapidly exra examples will improve he sochasic neural nework bound. We can judge he sample complexiy of he sochasic bound by ploing he value of he numeraor in equaion 1. Figure 2 plos he complexiy versus he number of paern presenaions in raining. The sochasic bound is a radical improvemen on he neural nework bound bu i is no ye a perfecly igh bound. Given ha we do no have a perfecly igh bound, one imporan consideraion arises: does he minimum of he sochasic bound predic he minimum of he rue error rae (as prediced by a large holdou daase). In paricular, can we use

6 è 100 Complexiy Complexiy paern presenaions Figure 2: We plo he complexiy of he sochasic nework model (numeraor of 1) vs. raining epoch. Noe ha he complexiy increases wih more raining as expeced and says below, implying nonvacuous bounds on a raining se of size. he sochasic bound o deermine when we should cease raining? The sochasic bound depends upon (1) he complexiy which increases wih raining ime and (2) he raining error which decreases wih raining ime. This dependence resuls in a minima which for our problem occurs a approximaely paern presenaions. The poin of minimal rue error (for he sochasic and deerminisic neural neworks) occurs a approximaely 6000 paern presenaions indicaing ha he sochasic bound weakly predics he poin of minimum error. The neural nework bound has no such minimum. Is he choice of 5% increased empirical error opimal? In general, he opimal choice of he exra error rae depends upon he learning problem. Since he sochasic neural nework bound (corollary 2.4) holds for all mulidimensional gaussian disribuions, we are free o opimize he choice of disribuion in anyway we desire. Figure 3 shows he resuling bound for differen choices of. The bound has a minimum a exra error indicaing ha our iniial choice of Ô is somewha large. Also noe ha he complexiy always decreases wih increasing enropy in he disribuion of our sochasic neural ne. The exisence of a minimum in Figure 3 is he righ behaviour: he increased empirical error rae is significan in he calculaion of he rue error bound. 4 Conclusion We have applied a PAC-Bayes bound for he rue error rae of a sochasic neural nework. The sochasic neural nework bound resuls in a radically igher ( é= orders of magniude) bound on he rue error rae of a classifier while increasing he empirical and rue error raes only a small amoun. Alhough, he sochasic neural ne bound is no compleely igh, i is no vacuous wih jus examples and he minima of he bound weakly predics he poin where overraining occurs. The resuls wih a synheic daa se are exremely promising he bounds are orders of magniude beer. Our nex sep will be o es he mehod on a few real world daases o insure ha he bounds remain igh. In addiion, here remain many opporuniies for

7 ê rue error bound or complexiy Sochasic NN bound Complexiy exra raining error Figure 3: Plo of he sochasic neural ne (SNN) bound for poserior disribuions chosen according o he exra empirical error hey inroduce. improving he applicaion of he bound. For example, i is possible ha shifing he weighs when finding a maximum accepable variance will resul in a igher bound. Also, We have no aken ino accoun symmeries wihin he nework which allow for a igher bound calculaion. References [1] Peer Barle, The Sample Complexiy of Paern Classificaion wih Neural Neworks: The Size of he Weighs is More Imporan han he Size of he Nework, IEEE Transacions on Informaion Theory, Vol. 44, No. 2, March [2] V. Kolchinskii and D. Panchenko, Empirical Margin Disribuions and Bounding he Generalizaion Error of Combined Classifiers, preprin, hp://cieseer.nj.nec.com/ hml [3] John Langford and Mahias Seeger, Bounds for Averaging Classifiers. CMU ech repor, [4] David McAlleser, Some PAC-Bayes bounds, COLT 1999.