M. Opper y. Institut fur Theoretische Physik. Justus-Liebig-Universitat Giessen. D-6300 Giessen, Germany. physik.uni-giessen.dbp.

Transcription

1 Query by Committee H. S. Seung Racah Institute of Physics and Center for eural Comutation Hebrew University Jerusalem 9194, Israel M. Oer y Institut fur Theoretische Physik Justus-Liebig-Universitat Giessen D-63 Giessen, Germany manfred.oer@ hysik.uni-giessen.db.de H. Somolinsky Racah Institute of Physics and Center for eural Comutation Hebrew University Jerusalem 9194, Israel haim@galaxy.huji.ac.il Abstract We roose an algorithm called query by committee, in which a committee of students is trained on the same data set. The next query is chosen according to the rincile of maximal disagreement. The algorithm is studied for two toy models: the high-low game and ercetron learning of another ercetron. As the number of queries goes to innity, the committee algorithm yields asymtotically nite information gain. This leads to generalization error that decreases exonentially with the numberof examles. This in marked contrast to learning from randomly chosen inuts, for which the information gain aroaches zero and the generalization error decreases with a relatively slow inverse ower law. We suggest that asymtotically nite information gain may be an imortant characteristic of good query algorithms. 1 Introduction Although query algorithms have been roosed for a variety of learning roblems[bau91], little work has gone into understanding the general rinciles by which these algorithms should be constructed. In this work, we argue that the Shannon information of a query can be a suitable guide[fed72]. We further show that the degree of disagreement among a committee of learners can serve as an estimate of this information value. The al- Present address: AT&T Bell Laboratories, 6 Mountain Ave., Murray Hill, J 7974, seung@hysics.att.com y Present address: Institut fur Theoretische Physik, Julius Maximilians Universitat Wurzburg, Am Hubland, D- 87 Wurzburg, Germany, oer@vax.rz.uni-wuerzburg. db.de gorithms considered in this work focus on minimizing the number of queries required, and hence are most relevant to situations in which queries carry the heaviest comutational cost. We consider the aradigm of incremental query learning, in which the training set is built u one examle at a time. We restrict our scoe to arametric learning models with continuously varying weights, learning erfectly realizable, boolean-valued target rules. The rior distribution on the weight sace is assumed to be at. An incremental learning rocedure consists of two comonents: a training algorithm and a query algorithm. Given a set of P examles, the training algorithm roduces a set of weights satisfying the training set. The query algorithm is then used to select examle P +1. Then the training algorithm is run again on the newly incremented training set, and so on. In this aer, the only training algorithm that we consider is the (zero temerature) Gibbs algorithm, which selects a weight vector at random from the version sace, the set of all weight vectors that are consistent with the training set. This will enable us to use techniques from statistical mechanics[sst92]. After training 2k students on the same training set, the query by committee algorithm selects an inut that is classied as ositive by half of the committee, and negative by the other half. By maximizing disagreement among the committee, the information gain of the query can be made high. In the k!1limit, each query bisects the version sace, so that the information gain saturates the bound of 1 bit er query. In the following, the query by committee algorithm is rst illustrated using a very simle model, the high-low game. We thenmove on to a more comlicated model, ercetron learning of another ercetron[gd89]. For both models, the information gain aroaches a nite value as the number of queries goes to innity. This asymtotically nite information gain leads to generalization error that decreases exonentially with the number of queries. This is in marked contrast to the case of learning with random inuts, in which the information gain

2 aroaches zero as the number of examles increases. In the random inut case, the generalization error decreases relatively slowly, as an inverse ower law in the number of examles. 2 The information content of a query Denote the target function, or \teacher," by (X), and the arametric learning model, or \student," by (W X). Both teacher and student are boolean-valued functions, i.e. mas into the set f1g. We further assume that the target function is erfectly realizable by the student, which means that there exists some weight vector W such that (X) = (W X) for all X. The inut vector X t is called a ositive or negative examle deending on the sign of the teacher's outut t = (X t ). The training set of inut-outut airs t =(X t t ) determines the version sace W P = fw : (W X t )= t t =1 ::: Pg (1) which is the set of all W consistent with the training set. If the rior distribution P (W) is assumed to be at, then the osterior distribution is uniform on the version sace, and vanishes outside[tls89]. This is written as P(Wj 1 ::: P )= V ;1 P W 2W P, otherwise, (2) where V P is the volume of W P.We consider the Gibbs training algorithm[hks91], in which theweight vector W is drawn at random from this osterior distribution. A query algorithm secies a way of choosing another inut X P +1 with which to query the teacher. In general, it can be written as a conditional robability P(X P +1 j 1 ::: P ) (3) which weleave unsecied for now. An inut X P +1 is chosen from this distribution and receives a label P +1 from the teacher. The results of this query are then added to the training set. The entroy of the osterior distribution (2) on the weight sace is S = log V P : (4) Since the entroy quanties our uncertainty about W, the information gained from query P + 1 can be dened as the reduction in the entroy, or I P +1 = ;S = ; log P +1 : (5) Here we have dened the volume ratio P +1 V P +1 V P : (6) The information gain I P +1 deends on the query sequence f 1 ::: P +1 g, although the deendence is not exlicit in the notation of (5). This deendence can be eliminated by considering averaged quantities. For examle, one can average the information gain (and other quantities of interest) over query sequences, and then over the rior distribution of W. For our uroses, only a artial average will suce. Holding the query sequence 1 ::: P constant, we will average over the inut X P +1 with resect to (3) and the teacher weight vector W with resect to the osterior distribution (2). In other words, the average is over all teacher vectors W that are consistent with the rst P examles, and over all inuts X P +1 given by the query algorithm. So the average information gain is given by hi P +1 i = ;hlog P +1 i W X P +1 (7) Similarly,we can calculate the comlete robability distribution for the volume ratio, which is P( P +1 j 1 ::: P )= P +1 ; V P +1 : V P W X P +1 (8) ote that these quantities still contain a deendence on the query sequence 1 ::: P. Performing the W average in (7) leads to a bayesian interretation of the formula. Any inut X P +1 divides the version sace W P into two arts, W + = fw 2W P : (W X P +1 )=+1g (9) W ; = fw 2W P : (W X P +1 )=;1g : (1) Averaging over the osterior distribution of W,we nd that the average information gain (7) is given by hi P +1 i = ; V + log V + ; V ; log V ; (11) V P V P V P V P XP +1 where V are the volumes of W and hence deend uon X P +1 imlicitly. After the teacher answers the query, P +1 is known with certainty. Before the answer arrives, the value of P +1 is uncertain: according to the bayesian, it is +1 with robability V + =V P, and ;1 with robability V ; =V P. The entroy of this distribution is recisely the information value of the query, and is the exression inside the average (11). The average information gain is maximized by X P +1 such that V + = V ;, i.e. by queries that divide the version sace in half. In this case of exact bisection, I = 1 bit exactly. Unfortunately, for most nontrivial learning models, the geometry of the version sace is comlex, and one cannot ractically calculate the volumes V for any given inut, much less nd an inut for which V + = V ;. Training algorithms tyically yield single oints in the version sace, not global information about the version sace. However, a committee of students can be used to obtain global information. Train a committee of 2k weight vectors using the Gibbs algorithm. Find an inut vector that is classied as a ositive examle by k members of the committee, and classied as negative by the other k. Query the teacher about this inut vector. Train the committee again using the new enlarged training set, and reeat. As k! 1 this algorithm aroaches the bisection algorithm. This algorithm is very much in the sirit of [OH91], in which consensus was used to imrove generalization erformance. Here we use lack of consensus to choose a query, or a rincile of maximal disagreement.

3 We dene the generalization function by g (W W )=h(;(w X)(W X))i X (12) where the average is taken over the rior distribution P (X) of inuts. This measures the robability oferror by a student W on inut-outut airs drawn from a teacher W. Given a query sequence 1 ::: P, the generalization error is dened by g ( 1 ::: P )=h g (W W )i W W (13) where the averages over W and W are taken over the osterior distribution. The average generalization error g (P ) is dened as the average of (13) over the query sequence. 3 High-Low The game of high-low, erhas the simlest model of query learning, can be formalized within our arametric learning framework. The teacher and student are (X) = sgn(x ; W ) (14) (X W) = sgn(x ; W ) : (15) The inut and weight saces are the unit interval [ 1], and the rior distributions on these saces are at, P (W )=1 (16) P (X) =1: (17) Given any query X, the teacher will resond with +1 or ;1, deending on whether X is higher or lower than W. Suose that there already exists a training set of P 2 examles. Let X L be the largest negative examle, and X R the smallest ositive examle. Then the version sace is W P =[X L X R ] (18) with volume V P = X R ; X L : (19) The osterior weight distribution P P (W )= V ;1 P W 2 [X L X R ], otherwise, (2) has entroy S = log V P. The Gibbs training algorithm simly icks a W at random from the version sace, i.e. from the interval [X L X R ]. We rst consider the case of randomly chosen inuts. With robability 1 ; V P the next inut X P +1 will fall outside the version sace, so that the volume ratio will be 1. If X P +1 falls inside the version sace, the robability distribution of the volume ratio is given by 2, where this result is obtained by averaging over all X P +1 W 2 [X L X R ]. Combining these two alternatives, we nd that the robability distribution of the volume ratio is P( P +1 jv P )=(1; V P )( P +1 ; 1) + V P 2 P +1 : (21) The exected information gain is given by the average of ; log P +1 with resect to this distribution, or I P +1 = V P 2 (nat) (22) As the number of examles increases, the volume V P shrinks, and the information gain tends to zero. The generalization error (13) is linear in the volume of the version sace, g ( 1 ::: P )= 1 3 V P : (23) Averaging this over all ossible training sets, one can show that the average generalization error satises g (P )= P +2 (24) an inverse ower law in the number of examles. Maximal information gain can be attained by the bisection algorithm, in which X P +1 is chosen to be halfway between X L and X R. In this case, the robability distribution of the volume ratio is P() = ; 1 (25) 2 so that the information gain is 1 bit er query. volume decreases exonentially, The V P =2 ;P : (26) Indeed, our knowledge of the binary reresentation of W increases by one bit er query. For a committee with 2k members, the robability distribution of the volume ratio is given by P() / 2 dy 1 dz yk;1 (1 ; z) k;1 z ; y (27) where we have omitted the normalization constant. Here we have written P() rather than P( P ), since the distribution is indeendent of the query history, unlike the random inut case (21). Figure 1 shows (27) for various values of k. ote that as k aroaches innity, the curves aroach the delta function (25) of the bisection algorithm. The average information gain in nats is given by I(k) = 1 d log P() (28) = (2k +2); (k +2); (k +2) (k +2); (k) where (x) =; (x)=;(x) is the Euler -function. As k!1, this saturates the bound of 1 bit er query. Since g = V P =3 is the roduct of the volume ratios, log g = PX t=1 log t + const : (29)

4 P() k =1 k =2 k =4 k = Figure 1: Probability distribution of the volume ratio for a 2k-member committee laying the high-low game. The volume ratios t are indeendent, identically distributed random variables drawn from (27). Hence by the central limit theorem g aroaches a log-normal random variable as P!1. The most robable g (P ) scales like g (P ) e ;PI(k) (3) where I(k) is given by (28). 1 Thus for query learning of high-low there is a very simle exonential relationshi between information gain and generalization error. In the next section we will see that a similar relationshi can be written for ercetrons. 4 Percetron Learning We now consider ercetron learning of another ercetron, where the teacher and student are given by (X) = sgn(w X) (31) (X W) = sgn(w X) : (32) The vectors W, W, and X all have comonents. The weight sace is taken to be the hyershere W = fw : W W = g (33) and the inut distribution is Gaussian P (X) =(2) ;=2 ex ; 1 2XX : (34) Given P examles the osterior distribution is uniform on the version sace W P = fw 2W:(W X t )(W X t ) > 8t =1 :::Pg (35) 1 We could also calculate the average of g rather than log g. The coecient of P in the resulting exonential is given by loghi rather than hlog i. 4.1 Random inuts When all inuts are chosen at random from the distribution (34), the relica method can be used to calculate the entroy of the osterior distribution[gt9]. The calculation is exact in the thermodynamic limit, where P!1with = P= constant. The entroy er weight s S= is then s() = 1 2 log(1 ; q)+1 2 q +2 Dx H(x) log H(x) where Dx H(y) (36) r q 1 ; q (37) dx e ;x2 =2 2 1 (38) Dx : (39) The entroy must be extremized with resect to the order arameter q, from which the average generalization error can be obtained using the relation y g = ;1 cos ;1 q: (4) In the large limit, this leads to the inverse ower law behavior g () :625 (41) The large asymtotics can also be obtained by examining the scaling of the entroy with the generalization error, similar to the arguments in [SST92] using the microcanonical high-t limit for a general classication of learning curves. As q! 1, the rst term of (36)

5 dominates, so that the entroy has a simle logarithmic deendence on the generalization error s log g (42) where we have used the asymtotic result g ;1 2(1 ; q). The information gain is given by = ;2 ;2 1 ; q Dx H(x)logH(x) dxh(x) log H(x) 1:6 g (43) This condition combined with (42) can only be satised by the inverse ower law g 1=, as in (41). We have erformed a Monte Carlo simulation of this algorithm. The deterministic ercetron algorithm[mp88] was used to obtain a weight vector in the version sace. Then zero-temerature Monte Carlo was used to random walk inside the version sace. To maintain accetance rates of aroximately 5%, the size of the Monte Carlo ste was scaled downward with increasing. The resulting learning curve is shown in Fig. 2, and ts the analytic results obtained through the relica method. 4.2 Query by committee Because each query deends on the revious history of queries, the committee algorithm is a dynamical rocess, where the number of examles lays the role of \time." In the relica calculations for this algorithm, we must know the overlas between weight vectors at dierent \times." This makes the calculations much more dicult than for the case of random-inuts. The relica calculation in the aendix makes the simlifying assumtion that the tyical overla between weight vectors at dierent \times" is equal to the tyical overla of two vectors at the earlier \time." In other words, we assume that the overlas satisfy W t W t = q(minft t g) (44) and are self-averaging quantities. Taking the thermodynamic limit!1turns the discrete-time dynamics in t into a continuous-time dynamics in t=. The entroy of the weight sace at time is calculated by treating q( ) for revious \times" <as an external eld. The resulting saddle oint equation for q() is an integral equation that can be solved numerically. The relica results for the two-member committee are shown in Fig. 2, along with Monte Carlo simulations. In the simulations, zero-temerature Gibbs training was imlemented by training two ercetrons using the standard ercetron algorithm, and then equilibrating them with zero-temerature Monte Carlo. Inut vectors were then selected at random from the rior distribution (34) until an inut that roduced disagreement was found. The teacher was queried on this inut, and (nat) (bit) Table 1: Information gain of the query by committee algorithm for ercetron learning the inut and the teacher's outut were added to the training set. ote that the generalization error of the two-member committee is very close to the random inut results for of order 1 or less. Even for relatively large, dierent committee sizes (not shown) erform almost identically. However, the algorithms are quite dierent in their large asymtotic roerties. By taking the derivative of the committee entroy (66) with resect to, we nd that the information gain aroaches the limit I()!;2 R dxhk (x)h k (;x)h(x) log H(x) R dxhk (x)h k (;x) (45) as!1. These limiting values for some small k can be found in Table 1. As k!1, the information gain saturates the bound of one bit er query. How does the information gain aect the generalization erformance? For the committee algorithm, we have found that the information gain aroaches a nite constant value ds!;i(1) (46) d We assume that the entroy is still given by s() log g as in (42). This assumtion, which isshown to be consistent in the aendix, leads to the asymtotic result g e ;I(1) : (47) The generalization error is asymtotically exonential in, with a decay constant determined by the information gain. This is markedly faster than the inverse ower law (41) for random inuts. In our imlementation of the committee algorithms, a source of random inuts is screened by a committee until one is found that rovokes disagreement. A ractical drawback ofthisscheme is that the time to nd a query diverges like the inverse of the generalization error. In this resect, algorithms which construct queries directly may be suerior. For examle, the query algorithm roosed by Kinzel and Rujan[KR9, WR92] for ercetron learning constructs inut vectors that are erendicular to the student weight vector. In conjunction with the Gibbs training algorithm, this method yields generalization erformance only slightly worse than that of the committee algorithms for moderate, but much faster query times. However, the committee algorithms aear to ossess a generality that other query algorithms do not.

6 g committee of two random inuts relica relica Figure 2: Generalization curves for ercetron learning using the random-inut, committee, and minimal training set algorithms. The Monte Carlo simulations were done with = 25, averaged over 64 samles. After each query, the ercetron was equilibrated for 124 stes. The standard error of measurement is smaller than the size of the symbol. The solid lines are analytic results from relica theory. 5 Conclusion We have comared query by committee and random{ inut training for the case of the ercetron. For random inuts, the information gain aroaches zero with the generalization error, I() ; ds d g : (48) For the committee algorithm, the information gain is asymtotically nite where I(1) > is given by (45). I()! I(1) (49) For ercetron learning, the entroy behaves asymtotically as s S log g (5) for both random{inut and query algorithms. Given ds=d and s in terms of g, one can derive the generalization curve as a function of. For random inuts, (48) and (5) imly g 1 : (51) For the committee algorithms, (49) and (5) imly log g ;I(1) : (52) The high-low game and the ercetron have rovided simle illustrations of the query by committee algorithm, and of the relationshi between information gain and generalization error. However, we have not addressed the issue of whether the behaviors exhibited by these toy models are general. For what architectures does query by committee lead to asymtotically nite information gain? Under what conditions does asymtotically nite information gain lead to exonential generalization curves? These questions are currently under investigation. Acknowledgements We acknowledge helful discussions with Y. Freund, M. Griniasty, D. Hansel,. Tishby, and T. Watkin. References [Bau91] E. Baum. eural net algorithms that learn in olynomial time from examles and queries. IEEE Trans. in eural etworks, 2:5{19, [Fed72] V. V. Fedorov. Theory of Otimal Exeriments. Academic Press, ew York, [GD89] E. Gardner and B. Derrida. Three unnished works on the otimal storage caacity of networks. J. Phys., A22:1983{1994, [GT9] G. Gyorgyi and. Tishby. Statistical theory of learning a rule. In W. K. Theumann and R. Koberle, editors, eural etworks and Sin Glasses, ages 3{36, Singaore, 199. World Scientic. [HKS91] D. Haussler, M. Kearns, and R. Schaire. Bounds on the samle comlexity ofbayesian

7 learning using information theory and the VC dimension. In M. K. Warmuth and L. G. Valiant, editors, Proceedings of the Fourth Annual Worksho on Comutational Learning Theory, ages 61{74, San Mateo, CA, Morgan Kaufmann. [KR9] W. Kinzel and P. Rujan. Imroving a network generalization ability by selecting examles. Eurohys. Lett., 13:473{477, 199. [MP88] M. L. Minsky and S. Paert. Percetrons. MIT Press, Cambridge, exanded edition, [OH91] M. Oer and D. Haussler. Generalization erformance of bayes otimal classication algorithm for learning a ercetron. Phys. Rev. Lett., 66:2677{268, [SST92] H. S. Seung, H. Somolinsky, and. Tishby. Statistical mechanics of learning from examles. Phys. Rev., A45:656{691, [TLS89]. Tishby, E. Levin, and S. Solla. Consistent inference of robabilities in layered networks: Predictions and generalization. In Proc. Int. Joint Conf. on eural etworks, volume 2, ages 43{49, Washington, DC, IEEE. [WR92] T. L. H. Watkin and A. Rau. Selecting examles for ercetrons. J. Phys., A25:113{121, Aendix: Relica calculations for query by committee The volume V P of the version sace determines the entroy S =lnv P for a xed teacher W. The average of the entroy over all ossible query sequences 1 ::: P consistent with the teacher is denoted by S av. From this quantity we shall nd the overla q between two random weight vectors inside the version sace W P.If we take one them to be the target vector, this in turn will give usthegeneralization error on a new random inut. We begin with the denition S av = hln V P i fx t g (53) = ln hhv n P i fx t g i W (54) In the second line we have introduced the relica trick, and an average of W over the rior distribution of teachers has been added, as in [OH91]. This makes the symmetry of the teacher and the students exlicit, so that the teacher vector W can be treated as another S av = lim ln Tr f t g * PY t=1 ny = ny = dw a ( t W X t ) + fx t g (55) ote that for all integer n we have now n + 1 relicas. Recall that for the k = 2 committee, the t + 1st inut X t+1 is selected so as to cause disagreement between two committee members W t and + Wt ; drawn at random from the version sace W t.thus the robability density for X t+1 is P(X t+1 jw t + Wt ; ) /P (X t+1 ) X =1 (W t + Xt+1 )(;W; t Xt+1 )(56) where a refactor R must be added to achieve the correct normalization dx t+1 P(X t+1 jw+ t W;) t = 1. The rior distribution of atterns P (X) is the Gaussian P (X) =(2) =2 e ; 1 2XX : (57) Let us consider the average over X t+1, with the committee weights W t + and W; t xed for the time being. * ny = = = ( t+1 W X t+1 ) X =1 + X t+1 dx t+1 n Y = ( t+1 W X t+1 )P(X t+1 jw t + W t ;) dx t+1 P (X t+1 ) X =1 ny = ( t+1 W X t+1 ) (W t + X t+1 )(;W t ; X t+1 ) (58) dx(w t + X)(;W t ; X)P (X) The average with resect to P (X t+1 )involves n +3 quantities W a X t =, where a canbeany of the symbols 1 ::: n + ;. These quantities are Gaussian random variables with zero mean, and covariance given by Wa X! ;1 W b X = W a W b = q ab : (59) P (X) The average (58) deends on the vectors W a only through these overlas q ab. In accord with the ansatz (44) and the ansatz of relica symmetry, we assume q = +(1; )q (6) q = q(t) (61) q = 1 (62) q = q(t) : (63) We now make the relacements W t X t+1! x q(t)+z 1 ; q(t) (64) W X t+1! x q(t)+y q ; q(t) +z 1 ; q: (65)

8 where x y z z are uncorrelated Gaussian random variables of unit variance. This change of variables yields the roer covariances. This average must be done for every X t, and yields an entroy that deends on the order arameters q(t), t = 1 to P ; 1, and on the overla q at time P of the weight vectors W. We treat the q(t) as external elds and determine q variationally. This is valid rovided that the q(t) are self-averaging quantities, so that the uctuations in the committee members at times t<pdo not aect the calculation of the entroy at time P. To take the continuum limit!1,we de- ne P = and t = and relace the roduct Q P t=1 (:::)by ex( R d ln(:::). Finally, the W integrals are done, as usual, by introducing entroic terms containing q. Generalizing to a committee of 2k members, we nd for the entroy er weight s = 1 2 q + 1 ln(1 ; q) (66) 2 where + 2 R R d Dx DyHk ( x)h k (; x)h(u)lnh(u) R DxHk ( x)h k (; x) s u x q( ) 1 ; q( ) (67) s q( ) 1 ; q + y q ; q( ) : (68) 1 ; q s The value of q = q() is determined variationally from (66). The information gain can be calculated by dierentiating (66) with resect to, R DxHk (x)h k (;x)h(x) log H(x) I(q) =;2 R DxHk (x)h k (;x) (69) where is dened as in (37). As q! 1, this yields the asymtotic result (45). Assuming that the generalization error is asymtotically exonential in, one can show that the disorder term in (66), the integral over, aroaches a constant. Hence the ln(1;q) term dominates, s log g, and the derivation in the text of asymtotically exonential g is selfconsistent.