Combining Classifiers and Learning Mixture-of-Experts

Transcription

1 38 Combining Classiiers and Learning Mixture-o-Experts Lei Xu Chinese University o Hong Kong Hong Kong & Peing University China Shun-ichi Amari Brain Science Institute Japan INTRODUCTION Expert combination is a classic strategy that has been widely used in various problem solving tass. A team o individuals with diverse and complementary sills tacle a tas ointly such that a perormance better than any single individual can mae is achieved via integrating the strengths o individuals. Started rom the late 980 in the handwritten character recognition literature studies have been made on combining multiple classiiers. Also rom the early 990 in the ields o neural networs and machine learning eorts have been made under the name o ensemble learning or mixture o experts on how to learn ointly a mixture o experts (parametric models) and a combining strategy or integrating them in an optimal sense. The article aims at a general setch o two streams o studies not only with a re-elaboration o essential tass basic ingredients and typical combining rules but also with a general combination ramewor (especially one concise and more useul one-parameter modulated special case called a-integration) suggested to uniy a number o typical classiier combination rules and several mixture based learning models as well as max rule and min rule used in the literature on uzzy system. BACKGROUND Both streams o studies are eatured by two periods o developments. The irst period is roughly rom the late 980s to the early 990s. In the handwritten character recognition literature various classiiers have been developed rom dierent methodologies and dierent eatures which motivate studies on combining multiple classiiers or a better perormance. A systematical eort on the early stage o studies was made in (Xu Krzyza & Suen 992) with an attempt o setting up a general ramewor or classiier combination. As reelaborated in Tab. not only two essential tass were identiied and a ramewor o three level combination was presented or the second tas to cope with dierent types o classiier s output inormation but also several rules have been investigated towards two o the three levels especially with Bayes voting rule product rule and Dempster-Shaer rule proposed. Subsequently the rest one (i.e. ran level) was soon studied in (Ho Hull & Srihari 994) via Borda count. Interestingly and complementarily almost in the same period the irst tas happens to be the ocus o studies in the neural networs learning literature. Encountering the problems that there are dierent choices or the same type o neural net by varying its scale (e.g. the number o hidden units in a three layer net) dierent local optimal results on the same neural net due to dierent initializations studies have been made on how to train an ensemble o diverse and complementary networs via cross-validation- partitioning correlation reduction pruning perormance guided re-sampling etc such that the resulted combination produces a better generalization perormance (Hansen & Salamon 990; Xu Krzyza & Suen 99; Wolpert 992; Baxt 992 Breiman 992&94; Drucer et al 994). In addition to classiication this stream also handles unction regression via integrating individual estimators by a linear combination (Perrone & Cooper 993). Furthermore this stream progresses to consider the perormance o two tass in Tab. ointly in help o the mixture-o-expert (ME) models (Jacobs et al 99; Jordan & Jacobs 994; Xu & Jordan 993; Xu Jordan & Hinton 994) which can learn either or both o the combining mechanism and individual experts in a maximum lielihood sense. Two stream studies in the irst period ointly set up a landscape o this emerging research area together Copyright 2009 IGI Global distributing in print or electronic orms without written permission o IGI Global is prohibited.

2 Combining Classiiers and Learning Mixture-o-Experts Table. Essential tass and their implementations C with a number o typical topics or directions. Thereater urther studies have been urther conducted on each o these typical directions. First theoretical analyses have been made or deep insights and improved perormances. For examples convergence analysis on the EM algorithm or the mixture based learning are conducted in (Jordan & Xu 995; Xu & Jordan 996). In Tumer & Ghosh (996) the additive errors o posteriori probabilities by classiiers or experts are considered with variances and correlations o these errors investigated or improving the perormance o a sum based combination. In Kittler et al (998) the eect o these errors on the sensitivity o sum rule vs product rule are urther investigated with a conclusion that summation is much preerred. Also a theoretical ramewor is suggested or taing several combining rules as special cases (Kittler 998) being unaware o that this ramewor is actually the mixture-o-experts model that was proposed irstly or combining multiple unction regressions in (Jacobs et al 99) and then or combining multiple classiiers in (Xu & Jordan 993). In addition another theoretical study is made on six classiier usion strategies in (Kuncheva 2002). Second there are urther studies on Dempster- Shaer rule (Al-Ania 2002) and other combing methods such as ran based boosting based as well as local accuracy estimates (Woods Kegelmeyer & Bowyer 997). Third there are a large number o applications. Due to space limit details are reerred to Ranawana & Palade (2006) and Sharey & Sharey (999). A GENERAL ARCHITECTURE TWO TASKS AND THREE INGREDIENTS We consider a general architecture shown in Fig.. There are { e } experts with each e (x) as either a classiier or an estimator. As shown in Tab.2 a classiier outputs one o three types o inormation on which we have three levels o combination. The irst two can be regarded as special cases o the third one that outputs a vector o measurements. A typical example T is [ p ( x) p ( m x)] with each p ( x) 0 expressing a posteriori probability that x is classiied to the -th class. Also p ( x) p ( y x) can be urther extended to p that describes a distribution or a m regression x y R. In Figure there is also a gating net that generates signals { ( x)} to modulate experts by a combining mechanism M(x). 39

3 Combining Classiiers and Learning Mixture-o-Experts Figure. A general architecture or expert combination Table 2. Three levels o combination Based on this architecture two essential tass o expert combination could be still quoted rom (Xu Krzyza & Suen 992) with a slight modiication on Tas as shown in Tab. that the phrase `or a speciic application? should be deleted in consideration o the previously introduced studies (Hansen & Salamon 990; Xu Krzyza & Suen 99; Wolpert 992; Baxt 992 Breiman 992&94; Drucer et al 994; Tumer & Ghosh 996). Insights can be obtained by considering three basic ingredients o two streams o studies as shown in Fig.2. Combinatorial choices o dierent ingredients lead to dierent speciic models or expert combination and dierences in the roles by each ingredient highlight the dierent ocuses o two streams. In the stream o neural networs and machine learning provided with a structure or each e (x) a gating structure and a combining structure M(x) all the rest unnowns are determined under guidance o a learning theory in term o minimizing an error cost. Such a minimization is implemented via an optimizing procedure by a learn- N ing algorithm based on a training set { } that xt y t t m teaches a target y t or each mapping xt R. While in the stream o combing classiiers all { p } are nown without unnowns let to be speciied. Also M is designed according to certain heuristics or principles with or without help o a training set and studies are mainly placed on developing and analyzing dierent combining mechanisms or which we will urther dis- 320

4 Combining Classiiers and Learning Mixture-o-Experts cuss subsequently. The inal combining perormance is empirically evaluated by the misclassiication rate but there is no eort yet on developing a theory or one M that minimizes the misclassiication rate or a cost unction though there are some investigations on how estimated posteriori probabilities can be improved by a sum rule and on error sensitivity o estimated posteriori probabilities (Tumer & Ghosh 996; Kittler et al 998). This under-explored direction also motivate uture studies subsequently. -COMBINATION The arithmetic geometric and harmonic mean o nonnegative number b 0 has been urther extended into one called: -mean m ( ( b )) where (r) is a monotonic scalar unction and a > 0 (Hardy Littlewood & Polya 952). We can urther generalize this -mean to the general architecture shown in Fig. resulting in the ollowing -combination: M ( ( p )) or ( M ) where ( p a > 0. ) In the ollowing we discuss to use it as a general ramewor to uniy not only typical classiier combining rules but also mixture-o-expert learning and RBF net learning as shown in Tab.3. We observe the three columns or three special cases o (r). The irst column is the case (r) r we return to the ME model: M p which was proposed irstly or combining multiple regressions in (Jacobs et al 99) and then or combining classiiers in (Xu & Jordan 993). For dierent special cases o a (x) we are lead to a number o existing typical examples. As already pointed out in (Kittler et al 998) the irst three rows are our typical classiier combining rules (the 2 nd row directly applies to the min-rule too). The next three rows are three types o ME learning models and a rather systematic summary C Figure 2. Three basic ingredients 32

5 Combining Classiiers and Learning Mixture-o-Experts Table 3. Typical examples (including several existing rules and potential topics) is reerred to Sec. 4.3 in (Xu 200). The last row is a recent development o the 3 rd row. The 2nd row o the 2nd column is the geometric mean: M p is ust a marginal probability M p which is equal to the product rule (Xu Krzyza and Suen 992; Kittler et al 998 Hinton 2002) i each a priori is equal i.e. a (x) /m. Generally i a (x) /m there is a dierence by a scaling actor a (x) /. The product rule wors in a probability theory sense under a condition that classiiers are mutually independent. In (Kittler et al 998) attempting to discuss a number o rules under a uniied system the sum rule is approximately derived rom the product rule under an extra condition that is usually diicult to satisy. Actually such an imposed lin between the product rule and the sum rule is unnecessary the sum: p( y x) which is already in the ramewor o probability theory. That is both the sum rule and the product rule already coexist in the ramewor o probability theory. On the other hand it can be observed that the sum: ln p is dominated by a p i it is close to 0. That is this combination expects that every expert should cast enough votes otherwise the combined votes will be still very low ust because there is only one that casts a very low vote. In other words this combination can 322

6 Combining Classiiers and Learning Mixture-o-Experts be regarded as a relaxed logical AND that is beyond the ramewor o probability theory when a (x) /m. However staying within the ramewor o probability theory does not mean that it is better not only because it requires that classiiers are mutually independent but also because there lacs theoretical analysis on both rules in a sense o classiication errors or which urther investigations are needed. In Tab.2 the 2nd row o the third column is the harmonic mean. It can be observed that the problem o combining the degrees o support is changed into a problem o combining the degrees o disagree. This is interesting. Unortunately eorts o this ind are seldom ound yet. Exceptionally there are also examples that can not be included in the -combination such as Dempster-Shaer rule (Xu Krzyza and Suen 992; Al-Ania 2002) and ran based rule (Ho Hull Srihari 994). a-integration Ater completed the above -combination the irst author becomes aware o the wor by (Hardy Littlewood & Polya 952) through one coming paper (Amari 2007) that studies a much concise and more useul one-parameter modulated special case called a-integration. With help o a concrete mathematical oundation rom an inormation geometry perspective. Imposing an additional but reasonable nature that the -mean should be linear scale-ree i.e.: cm ( ( cb )) or any scale c alternative choices o (r) reduces into the ollowing only one: 0.5( ( r) r ln r ). It is not diicult to chec that r ( r) ln r / r 3. Thus the discussions on the examples in Tab.2 are applicable to this a (r). Moreover the irst row in Tab.2 holds when and + or whatever a gating net which thus includes two typical operators o the uzzy system as special case too. Also the amily is systematically modulated by a parameter + which provides not only a spectrum rom the most optimistic integration to the most pessimistic integration as varying rom to + but also a possibility o adapting a or a best combining perormance. Furthermore Amari (2007) also provides a theoretical ustiication that a-integration is optimal in a sense o minimizing a weighted average o a-divergence. Moreover it provides a potential road or studies on combining classiiers and learning mixture models rom the perspective o inormation geometry. FUTURE TRENDS Further studies are expected along several directions as ollows: Empirical and analytical comparisons on perormance are needed or those unexplored or less explored items in Tab.2. Is there a best structure or a (x)? comparisons need to be made on dierent types o a (x) especially the ones by the MUV type in the last row and the ME types rom the 4 th to the 7 th rows Is it necessary to relax the constraint: a (x) > 0 ( ) x e.g. removing non-negative requirement and to relax the distribution p to other types o unctions? How weights a (x) can be learned under a generalization error bound. As discussed in Fig.2 classiier combination and mixture based learning are two aspects with dierent eatures. How to let each part to tae their best roles in an integrated system? CONCLUSION Updating the purpose o (Xu Krzyza & Suen 992) the article provides not only a general setch o studies 323 C

7 Combining Classiiers and Learning Mixture-o-Experts on combining classiiers and learning mixture models but also a general combination ramewor to uniy a number o classiier combination rules and mixture based learning models as well as a number o directions or urther investigations. Acnowledgment The wor is supported by Chang Jiang Scholars Program by Chinese Ministry o Education or Chang Jiang Chair Proessorship in Peing University. REFERENCES Al-Ania A. (2002) A New Technique or Combining Multiple Classiiers using The Dempster-Shaer Theory o Evidence Journal o Artiicial Intelligence Research Amari S. (2007) Integration o stochastic models by minimizing a-divergence Neural Computation 9(0) Baxt W. G. (992) Improving the accuracy o an artiicial neural networ using multiple dierently trained networs Neural Computation Breiman L. (992) Staced Regression Department o Statistics Bereley TR-367. Breiman L. (994) Bagging Predictors Department o Statistics Bereley TR-42. DrucerH. CortesC. JacelL.D. LeCunY.& Vapni V. (994) Boosting and other ensemble methods Neural Computation Hardy G.H. Littlewood J.E. and Polya G. (952) Inequalities 2nd edition Cambridge University Press. Hansen L.K. & Salamon P. (990) Neural networ ensembles. IEEE Transactions Pattern Analysis Machine Intelligence 2(0) Hinton G.E.(2002) Training products o experts by minimizing contrastive divergence Neural Computation Ho T.K. Hull. J.J. Srihari S. N. (994) Decision combination in multiple classiier systems IEEE Transactions Pattern Analysis Machine Intelligence 6() Jacobs R. A. Jordan M. I. Nowlan S. J. & Hinton G. E. (99) Adaptive mixtures o local experts Neural Computation Jordan M. I. & Jacobs R. A. (994) Hierarchical mixtures o experts and the EM algorithm Neural Computation Jordan M.I. & Xu L. (995) Convergence Results or The EM Approach to Mixtures o Experts Architectures Neural Networs Kittler J. Hate M. Duinand R. P. W. Matas J. (998) On combining classiiers IEEE Trans. Pattern Analysis Machine Intelligence 20(3) Kittler J. (998) Combining classiiers: A theoretical ramewor Pattern Analysis and Applications Kuncheva L.I. (2002) A theoretical study on six classiier usion strategies IEEE Transactions Pattern Analysis Machine Intelligence 24(2) Lee C. Greiner R. Wang S. (2006) Using query-speciic variance estimates to combine Bayesian classiiers Proc. o the 23rd International Conerence on Machine Learning (ICML) Pittsburgh June ( ). Perrone M.P. & Cooper L.N.(993) When networs disagree: Ensemble methods or neural networs Neural Networs or Speech and Image Processing Mammone R. J. editor Chapman Hall. Ranawana R. & Palade V. (2006) Multi-Classiier Systems: Review and a roadmap or developers International Journal o Hybrid Intelligent Systems 3() Shaer G. (976) A mathematical theory o evidence Princeton University Press. Sharey A.J.& Sharey N.E. (997) Combining diverse neural nets Knowledge Engineering Review 2(3) Sharey A.J.& Sharey N.E. (999.) Combining Artiicial Neural Nets: Ensemble and Modular Multi- Net Systems Springer-Verlag NewYor Inc. Tumer K. & Ghosh J. (996) Error correlation and error reduction in ensemble classiiers Connection Science 8 (3/4)

8 Combining Classiiers and Learning Mixture-o-Experts Wolpert D.H. (992) Staced generalization Neural Networs 5(2) WoodsK. KegelmeyerK.P. & Bowyer K. (997) Combination o multiple classiiers using local accuracy estimates IEEE Transactions Pattern Analysis Machine Intelligen 9(4) Xu L (2007) A uniied perspective and new results on RHT computing mixture based learning and multilearner based problem solving Pattern Recognition Xu L (200) Best harmony uniied RPCL and automated model selection or unsupervised and supervised learning on Gaussian mixtures three-layer nets and ME-RBF-SVM Models International Journal o Neural Systems () Xu L. (998) RBF Nets Mixture Experts and Bayesian Ying-Yang Learning Neurocomputing Xu L. & Jordan M.I. (996) On convergence properties o the EM algorithm or Gaussian mixtures Neural Computation 8() Xu L. Jordan M.I. & Hinton G.E. (995) An Alternative Model or Mixtures o Experts Advances in Neural Inormation Processing Systems 7 Cowan Tesauro and Alspector editors MIT Press Xu L. Jordan M.I. & Hinton G.E. (994) A Modiied Gating Networ or the Mixtures o Experts Architecture Proc. WCNN94 San Diego CA (2) Xu L. & Jordan M.I. (993) EM Learning on A Generalized Finite Mixture Model or Combining Multiple Classiiers Proc. o WCNN93 (IV) Xu L. Krzyza A. & Sun C.Y. (992) Several methods or combining multiple classiiers and their applications in handwritten character recognition IEEE Transactions on System Man and Cybernetics Xu L. Krzyza A. & Sun C.Y. (99) Associative Switch or Combining Multiple Classiiers Proc. o IJCNN9 July 8-2. Seattle WA (I) Xu L. Oa E. & Kultanen P. (990) A New Curve Detection Method Randomized Hough transorm (RHT) Pattern Recognition Letters Key TERMS Conditional Distribution p(y x): Describes the uncertainty that an input x is mapped into an output y that simply taes one o several labels. In this case x is classiied into the class label y with a probability p(y x). Also y can be a real-valued vector or which x is mapped into y according density distribution p(y x). Classiier Combination: Given a number o classiiers each classiies a same input x into a class label and the labels maybe dierent or dierent classiiers. We see a rule M(x) that combines these classiiers as a new one that perorms better than anyone o them. Sum Rule (Bayes Voting): A classiier classiies x to a label y can be regarded as casting one vote to this label a simplest combination is to count the votes received by every candidate label. The -th classiier classiies x to a label y with a probability p means that one vote is divided to dierent candidates in ractions. We can sum up: p to count the votes on a candidate label y which is called Bayes voting since p(y x) is usually called Bayes posteriori probability. Product Rule: When classiiers { e } are mutually independent a combination is given by p( x C y x) p( x C ) y p( x C y e p( x C ) y C 325

9 Combining Classiiers and Learning Mixture-o-Experts or concisely p( y x) p ( y) p which is also called product rule. Mixture o Experts: Each expert is described by a conditional distribution p either with y taing one o several labels or a classiication problem or with y being a real-valued vector or a regression problem. A combination o experts is given by: M p (x) p( x) > 0 which is called a mixture-o-experts model. Particularly or y in a real-valued vector its regression orm is E( y x) ( y) yp ( y x dy. ) -Mean: Given a set o non-negative numbers b 0 the -mean is given by: m ( ( b )) where (r) is a monotonic scalar unction and a > 0. Particularly one most interesting special case is that (r) satisies cm ( ( cb )) or any scale c which is called a -mean. Perormance Evaluation Approach: It usually wors in the literature on classiier. Combination with a chart low that considering a set o classiiers { e } designing a combining mechanism M(x) according to certain principles evaluating perormances o combination empirically via misclassiication rates in help o samples with nown correct labels. Error-Reduction Approach: It usually wors in the literature on mixture based learning where what needs to be pre-designed is the structures o classiiers or experts as well as the combining structure M(x) with unnown parameters. A cost or error measure is evaluated via a set o training samples and then minimized through learning all the unnown parameters. 326