Two Spirals Two Gaussians Letters

Transcription

1 Two Spirals Two Gaussians Letters Figure 8: Number of examples needed for average error to reac.3. From left to rigt: random, uncertainty, maximal distance and lookaead sampling metods. contains more tan one region of some class. Ten, te selective sampling algoritm must consider not only te examples from te ypotesis boundary, butmust also explore large unsampled regions. Te lack of'exploration' element in 'uncertainty' and 'maximal distance' sampling metods often results in a failure in suc cases. Te benet of a lookaead selective sampling metod can be seen by comparing te number of examples needed to reac some pre-dened accuracy, Figure 8. Counting te classication of one point (including nding 1 or 2 labeled neigbors) as a basic operation, te uncertainty and maximal distance metods ave time complexity ofo(jxj) wile te straigtforward implementation oflookaead selective sampling as a time complexity ofo(jxj 2 )(we need to compute class probabilities for all points in te instance space after eac lookaead ypotesis). Tis iger complexity, owever, is well justied for a natural setup, were we are ready to invest computational resources to save time for a uman expert wose role is to label an examples. References Aa, D. W. Kibler, D. and Albert, M. K Instance-based learning algoritms. Macine Learning 6(1):37{66. Angluin, D Queries and concept learning. Macine Learning 2(3):319{42. Blake, C. Keog, E. and Merz, C UCI repository of macine learning databases [ttp:// University of California, Irvine, Dept. of Information and Computer Sciences. Con, D. A. Atlas, L. and Lander, R Improving generalization wit active learning. Macine Learning 15(2):21{21. Cover, T. M., and Hart, P. E Nearest neigbor pattern classication. IEEE Transactions on Information Teory 13(1):21{27. Dagan, I., and Engelson, S. P Committee-based sampling for training probabilistic classiers. In Macine Learning - International Worksop ten Conference conf 12, 15{157. Morgan Kaufmann. Davis, D. T., and Hwang, J.-N Attentional focus training by boundary region data selection. In IJCNN, volume 1, 676{81. IEEE. Eldar, Y. Lindenbaum, M. Porat, M. and Zeevi, Y. Y Te fartest point strategy for progressive image sampling. IEEE Transactions on Image Processing 6(9):135{15. Freund, Y. Seung, H. S. Samir, E. and Tisbi, N Selective sampling using te query by committeee algoritm. Macine Learning 28(2-3):133{68. Frey, P. W., and Slate, D. J Letter recognition using olland-style adaptive classiers. Macine Learning 6(2):161{82. Hasenjager, M., and Ritter, H Active learning of te generalized ig-low-game. In ICANN, xxv+922, 51{6. Springer-Verlag. Hasenjager, M., and Ritter, H Active learning wit local models. Neural Processing Letters 7(2):17{ 17. Krog, A., and Vedelsby, J Neural network ensembles, cross validation, and active learning. In NIPS, volume 7, 231{8. MIT Press. Lang, K. J., and Witbrock, M. J Learning to tell two spirals apart. In Proceedings of te Connectionist Models Summer Scool, 52{59. Morgan Kaufmann. Lewis, D. D., and Catlett, J Heterogeneous uncertainty sampling for supervised learning. In Macine Learning - International Worksop ten Conference conf 11, 148{156. Lindenbaum, M. Markovic, S. and Rusakov, D Selective sampling by random eld modelling. Tecnical Report CIS996, Tecnion - Israel Institute of Tecnology. MacKay, D. J Introduction to gaussian processes. NATO ASI series. Series F, Computer and system sciences. 168:133. Papoulis, A Probability, Random Variables, and Stoastic Processes. McGraw-Hill series in electrical engineering, Communications and signal processing. McGraw-Hill, Inc., 3rd edition. RayCauduri, T., and Hamey, L Minimisation of data collection by active learning. In IEEE ICNN, volume 3, 6 vol. l+3219, 1338{41. IEEE. Seung, H. S. Opper, M. and Sompolinsky, H Query by committee. In Proceedings of te Fift Annual ACM Worksop on Computational Learning Teory, v+452, 287{94. ACM New York, NY, USA. Williams, C. K. I., and Barber, D Bayesian classication wit gaussian processes. IEEE PAMI 2(12):1342. Wong, E., and Hajek, B Stoastic Processes in Engineering Systems. Springer-Verlag.

2 .5.45 Random Uncertainty Maximal Distance Lookaead Random Uncertainty Maximal Distance Lookaead Error rate.35 Error rate Number of examples Number of examples Figure 4: Learning rate graps for various selective sampling metods applied to te \two spirals" data Figure 6: Learning rate graps for various selective sampling metods applied to te \two gaussians" data. to and all te letters from 'n' to 'z' to 1. Te learning rate of te various selective sampling metods is sown in Figure 7. Te lookaead selective sampling algoritm outperforms oter selective sampling metods in tis particularly ard domain, were every class consists of many dierent (associated wit te dierent letters) Figure 5: A feature space wit bayes decision boundaries (only 4 points are sown) for \two gaussians" data Random Uncertainty Maximal Distance Lookaead.4 'Two Gaussians' Data Te test set of \two gaussians" consists of two dimensional vectors belonging to two classes wit equal a priori probability (:5). Te distribution of class 1 is uniform over te region [ 2] [ 2] and te distribution of class consists of two symmetric gaussians, wit means in points (5 5) and (15 15) and covariance matrix = 2 2 I, illustrated in Figure 5. Te bayes error is :1827. Te learning rate of te various selective sampling metods is sown in Figure 6. We can see tat apparently te uncertainty and maximal distance selective sampling metods fail to detect one of te gaussians, resulting in iger error rates. Tis is due to fact tat tese metods consider sampling only at te existing boundary. Letters Data Te letter recognition database (contributed to UCI Macine learning repository (Blake, Keog, & Merz 1998) by Frey and Slate (1991) consists of 2 feature vectors belonging to 26 classes tat represent capital letters of Latin alpabet. Since our current implementation works only wit binary classication, we converted te database to suc by canging all letters from 'a' to 'm' Error rate Number of examples Figure 7: Learning rate graps for various selective sampling metods applied to te letters dataset. Discussion Nearest neigbor classiers are often used wen little or no information is available about te instance space structure. Tere, te loose, minimalistic specication of te instance space labeling structure, wic is implied by te distance based random eld model, seems to be adequate. We also observe tat large canges in te covariance function ad no signicant eect on te classication performance. Te experiments sow tat lookaead sampling metod performs better or comparatively to oter selective sampling algoritms on bot articial and real domains. It is especially strong wen te instance space

3 See te experimental part for an evaluation of some covariance function and for teir use in estimating te parameters. Wit tis metod, every sampled point inuences te estimated probability. In practice, suc long range inuence is non-intuitive and is also computationally expensive. Terefore, in practice, we neglect te inuence of all except te two closest neigbors. Tis coice gives a iger probability to te nearest neigbor class and is terefore consistent wit1-nn classication. One deciency of tis estimation process is tat te estimated probabilities are not guaranteed to lie in te required [ 1] range. Wen suc over- ows indeed appen (very rarely), we correct tem by clipping te estimate. Tis deciency is corrected in more complex estimation procedures, described in te full version (Lindenbaum, Markovic, & Rusakov 1999). (Te framework we use is similar to Bayesian Classi- cation via Gaussian Process Modeling (MacKay 1998 Williams & Barber 1998) Experimental Evaluation We ave implemented our random-eld based lookaead algoritm and tested it on several problems, comparing its performance wit several oter selective sampling metods. Experimental Metodology Te algoritm described in te previous sections allows us to euristically coose te covariance function, (d). In te experiments described ere, every class contained a nearly equal number of examples and terefore we assume tat te a priori class probabilities are equal. Tis implies tat () = :25. We coose an exponentially decreasing covariance function (common in image processing) (d) =:25e ;d=. We tested te eect of a range of values on te performance of te algoritm and found tat canging ad almost no eect (tese results are included in te full version (Lindenbaum, Markovic, & Rusakov 1999) Te lookaead algoritm was compared wit te following tree selective sampling algoritms, wic represent te most common coices (see introduction): Random sampling: Te algoritm randomly selects te next example. Wile tis metod looks unsopisticated, it as te advantage of yielding a uniform exploration of te instance space. Tis metod actually corresponds to a passive learning model. Uncertainty sampling: Te metod selects te example wic te current classier is most uncertain about. Te uncertainty for eac example depends on te ratio between te distances to te closest labeled neigbors of dierent classes Tis metod tends to sample on te existing border, and wile for some decision boundaries tat may be benecial, for oters it may be a source for serious failure (as will be sown in te following subsections). Maximal distance: An adaptation of te metod described by Hasenjager and Ritter (1998). Tis metod selects te example from te set of all unlabeled points tat ave dierent labels among teir tree nearest classied neigbors. Te example selected is te one wic is most distant from its closest labeled neigbor. Te basic measurement used for te experiments is te expected error rate. For eac selective sampling metod and for eac dataset te following procedure was applied: 1. 1 examples from te dataset were drawn randomly - tis is a set used for selective sampling and learning,x, te rest 19 examples (all datasets included 2 examples) were used only for te evaluation of error rates of te resulting classiers. 2. Te selective sampling algoritm was applied to te cosen set, X. After selection of eac example, te error rate of te current ypotesis, (wic iste nearest neigbor classier), was calculated using te test set of 19 examples put aside. 3. Steps 1 2were performed 1 times and te average error rate was calculated Class Class Figure 3: Te feature space of te \two spirals" data. Te 'Two Spirals' Problem Te two spirals problem was studied by a number of researcers (Lang & Witbrock 1988 Hasenjager & Ritter 1998). Tis is an articial problem were te task is to distinguis between two spirals of uniform density in XY -plane, as sown in Figure 3. (Te code for generating tese spirals was based on (Lang & Witbrock 1988)) Te Bayes error of suc classication is zero since te classes are perfectly separable. Te learning rate of te various selective sampling metods is sown in Figure 4. All tree non-random metods demonstrated comparable performance, better tan random sampling. In te next experiment we will sow tat oter metods lack one of te basic properties required from selective sampling algoritms - exploration - and fail in te datasets consisting of separated regions of te same classication.

4 distribution of consistent target functions. First, consider a specic target f. Let I f be a binary indicator function, were I f (x) =1if(x) = (x), and let f () denote te accuracy of ypotesis relative to f: f () =f \ = R x2r d I f (x)p(x)dx. Recall tat p(x) is te probability density function specifying te instance distribution over R d. Let A L (D) denote te expected accuracy of a ypotesis produced by learning algoritm L: A L (D) = E fjd [ f ( = L(D))] = E fjd [ R x2r d I f (x)p(x)dx]= R x2r d P (f(x) =(x)jd)p(x)dx (1) were P (f(x) = (x)jd) is te probability tat a random target function f consistent wit D will be equal to in te point x, i.e P (f(x) =(x)jd) =E fjd [f(x) = (x)]. Note tat P (f(x) = (x)jd) is te probability tat a particular point x gets te correct classication. Terefore, for every given ypotesis, estimating te class probabilities P (f(x) =jd) P(f(x) =1jD), gives also te accuracy estimate (from Equation 1): A L (D) X x2x P (f(x) =(x)jd)=jxj: (2) (Te number of examples in X is assumed to be nite). Tus te problem of evaluating te utility measure as te classier accuracy is translated into te problem of estimating te class probabilities. Assuming tat te probability computation model is correct, te optimal selective sampling strategy is one tat uses U L (D), A L (D) as te utility function. Random Field Model for Feature Space Classication Feature vectors from te same class tend to cluster in te feature space (toug sometimes te clusters are quite complex). Terefore close feature vectors sare te same label more often tan not. Tis intuitive observation, wic is te rationale for te nearest neigbor classication approac, is used ere to estimate te classes of unlabeled feature points and teir uncertainties. Matematically, tis observation is described by assuming tat te label of every point is a random variable, and tat tese random variables are mutually dependent. Suc dependencies are usually described (in a iger tan 1-dimensional space) by random eld models. In te probabilistic setting, estimating te classi- cation of unlabeled vectors and teir uncertainties is equivalent to calculating te conditional class probabilities from te labeled data, relying on te random eld model. In te full version of te paper (Lindenbaum, Markovic,& Rusakov 1999), we consider several options for suc estimates. Tis sorter version focuses on one particular model. Tus, we assume tat te classication of an instance space is a sample function of a binary valued omogeneous isotropic random eld (Wong & Hajek 1985) caracterized by a covariance function decreasing wit a distance. (see (Eldar et al. 1997) were a similar metod was used for progressive image sampling.) Tat is: let x x 1 be points in X and let 1 be teir classications, i.e. random variables tat can ave values of or 1. Te omogeneity and isotropy properties imply tat te expected values of and 1 are equal, i.e. E[ ]=E[ 1 ]=, and te covariance between and 1 is specied only by te distance between x and x 1 : C[ 1 ]=E[( ; )( 1 ; )], (d(x x 1 )) (3) were : R +! (;1 1) is a covariance function wit () = Var[] =E[( ; ) 2 ]=P P 1,wereP, P 1 = 1;P are te a priori class probabilities. Usually we will assume tat is decreasing wit te distance and tat lim r!1 (r) =. Note tat te random eld model species (indirectly) a distribution of target functions. In estimation, one tries to nd te value of some unobserved random variable, from observed values of oter, related, random variables, and prior knowledge about teir joint statistics. Te class probabilities associated wit some feature vector are uniquely specied by te conditional mean of its associated random variable (r.v.) Tis conditional mean is also te best estimator for te r.v. value in te least squares sense (Papoulis 1991). Terefore, te widely available metods for mean square error (MSE) estimation can be used for estimating te class probabilities. We coose a linear estimator, for wic a closed form solution, described below, is available. Let be te binary r.v. associated wit some unlabeled feature vector, x, and let 1 ::: n be te known labels r.v. associated wit te feature vectors, x 1 ::: x n,tatwere already sampled. Now let ^ = + nx i=1 i i (4) be te estimate of te unknown label. Te estimate uses te known labels and relies on unknown coecients wic sould be set so tat te MSE, mse = E[(^ ; ) 2 ] is minimized. Te optimal linear approximation in te MS sense (Papoulis 1991) is described by: ^ = E[ ]+~a ( ~ ; E[ ]]) ~ t (5) were ~a is an n-dimensional vector specied by te covariance values: ~a = R ;1 ~r R ij = E [( i ; E[])( j ; E[])] (6) r i = E [( ; E[])( j ; E[])] : (R is an n n matrix, and ~a ~r are n;dimensional vectors). Te values of R and ~r are specied by te random eld model: R ij = (d(x i x j )) (7) r i = (d(x x i )):

5 a teacer (also called an oracle or an expert) wic labels instances by or 1, f : X! f 1g. A learning algoritm takes a set of classied examples, fx 1 f(x 1 )i ::: x n f(x n )ig, and returns a ypotesis, : X!f 1g. Trougout tis paper we assume tat X = R d. Let X be an instance space - a set of objects drawn randomly from X according to distribution p. Let D X be a nite set of classied examples. Aselective sampling algoritm S L wit respect to learning algoritm L takes X and D, and returns an unclassi- ed element ofx. Anactive learning process can be described as follows: 1. D 2. L( ) 3. Wile stop-criterion is not satised do: (a) Apply S L and get te next example, x S L (X D). (b) Ask te teacer to label x,! f(x) (c) Update S te labeled examples set, D D fx!ig (d) Update te classier, L(D) 4. Return classier Te stop criterion may be a limit M on te number of examples tat te teacer is willing to classify or a lower bound on te classier accuracy. We will assume ere te rst case. Te goal of te selective sampling algoritm is to produce a sequence of lengt M wic leads to a best classier according to some given criterion. Lookaead Algoritms for Selective Sampling Knowing tat we are allowed to ask for exactly M labels allows, in principle, to consider all object sequences of lengt M. Not knowing te labeling of tese objects, owever, prevents us from evaluating te resulting classiers directly. One way to overcome tis diculty is to consider te selective sampling process as an interaction between te learner and te teacer. At eac stage te learner must select an object from te set of unclassied instances and te teacer assigns one of te possible labels to te selected object. Tis interaction can be represented by a \game tree" of 2M levels suc as te one illustrated Figure 1. We can use suc a tree representation to develop a lookaead algoritm for selective sampling. Let U L (D) be a utility evaluation function tat is capable of appraising a set D as examples for a learning algoritm L. Let us dene a k-deep lookaead algoritm for selective sampling wit respect to learning algoritm L as illustrated on Figure 2. Note tat tis algoritm is a specic case of a decision teoretic agent, and tat, wile it is specied for maximizing te expected utility, one can be, for exam x Q 1 x 2 QQs x n ) + a a a Q Q f(x 1 )= QQs 1 QQs Q QQs P PPPP Pq x 11 x 12 x 21 x 22 + a a a a a a Q QQs. QQQs Q QQs Figure 1: Selective sampling as a game. S k L (X D) : Select x 2 X wit maximal expected utility: x = arg max x2x E! [U L (X D [fx!ig k; 1)] were U L (X D k) is a recursive utility propagation function: U L (X D k) = UL (D) k = max x E! [U L (X D k; 1)] k> were D = D [fx!ig and te expected value E! [] istaken according to conditional probabilities for classication of x given D, P (f(x) =!jd). Figure 2: Lookaead algoritm for selective sampling. ple, pessimistic and consider a minimax approac. In our implementation we use a simplied one-step lookaead algoritm: S (X D) : L Select x 2 X wit maximal expected utility, E!2f 1g [U L (D [fx!ig)], wic is equal to: P (f(x) =jd) U L (D [fx ig)+ P (f(x) =1jD) U L (D [fx 1ig) Te actual use of te lookaead example selection sceme relies on two coices: Te utility function U L (D). Te metod for estimating P (f(x) = jd) (and P (f(x) =1jD)). Two particular coices are considered in te next sections. Te Classier Accuracy Utility Function Taking a Bayesian approac, we specify te utility of te classier as its expected accuracy relative to te

6 Selective Sampling for Nearest Neigbor Classiers Micael Lindenbaum Saul Markovic Computer Science Department, Tecnion - Israel Institute of Tecnology, 32, Haifa, Israel Dmitry Rusakov rusakov@cs.tecnion.ac.il Abstract In te passive, traditional, approac to learning, te information available to te learner is a set of classied examples, wic are randomly drawn from te instance space. In many applications, owever, te initial classication of te training set is a costly process, and an intelligently selection of training examples from unlabeled data is done by an active learner. Tis paper proposes a lookaead algoritm for example selection and addresses te problem of active learning in te context of nearest neigbor classiers. Te proposed approac relies on using a random eld model for te example labeling, wic implies a dynamic cange of te label estimates during te sampling process. Te proposed selective sampling algoritm was evaluated empirically on articial and real data sets. Te experiments sow tat te proposed metod outperforms oter metods in most cases. Introduction In many real-world domains it is expensive tolabela large number of examples for training, and te problem of reducing training set size, wile maintaining te quality of te resulting classier, arises. A possible solution to tis problem is to give te learning algoritm some control over te inputs on wic it trains. Tis paradigm is called active learning, and is rougly divided into two major subelds: learning wit membersip queries and selective sampling. In learning wit membersip queries (Angluin 1988) te learner is allowed to construct articial examples, wile selective sampling deals wit selection of informative examples from a large set of unclassied data. Selective sampling metods ave been developed for various classication learning algoritms: for neural networks (Davis & Hwang 1992 Con, Atlas, & Lander 1994), for te C4:5 rule-induction algoritm (Lewis & Catlett 1994) and for HMM (Dagan & Engelson 1995). Te goal of te researc described in tis paper is to develop a selective sampling metodology for nearest neigbor classication learning algoritms. Te Copyrigt c1999, American Association for Articial Intelligence ( All rigts reserved. nearest neigbor (Cover & Hart 1967 Aa, Kibler, & Albert 1991) algoritm is a non-parametric classication metod, useful especially wen little information is known about te structure of te distribution, implying tat parametric classiers are arder to construct. Te problem of active learning for nearest neigbor classi- ers was considered by Hasenjager and Ritter (1998). Tey proposed querying in points wic are te fartest from previously sampled examples, i.e. in te vertices of Voronoi diagram of te points labeled so far. Tis metod, owever, falls under te membersip queries paradigm and is not suitable for selective sampling. Most existing selective sampling algoritms focus on coosing examples from regions of uncertainty. One approac to dene uncertainty is to specify a committee (Seung, Opper, & Sompolinsky 1992) or an ensemble (Krog & Vedelsby 1994) of ypoteses consistent wit te sampled data and ten to coose an exampleonwic te committee members most disagree. Query By Committee is an active researc topic, and strong teoretical results (Freund et al. 1997) along wit practical justications (Dagan & Engelson 1995 Hasenjager & Ritter 1996 RayCauduri & Hamey 1995) were acieved. It is not clear, owever, ow to apply tis metod to nearest-neigbor classication. Tis paper introduces a lookaead approac to selective sampling tat is suitable for nearest neigbor classication. We start by formalizing te problem of selective sampling and continue wit a lookaead based framework wic cooses te next example (or sequence of examples) in order to maximize te expected utility (goodness) of te resulting classier. Te major components needed to apply tis framework are an utility function for appraising classiers and a posteriori class probability estimates for points in te instance space. We propose a random eld model for te feature space classication structure. Tis model serves as te basis for a class probability estimation. Te merit of our approac is empirically demonstrated on articial and real problems. Te Selective Sampling Process We consider ere te following selective sampling paradigm. Let X be a set of objects. Let f be