Dynamically Weighted Majority Voting for Incremental Learning and Comparison of Three Boosting Based Approaches

Transcription

1 Proceedngs of Inernaonal Jon Conference on Neural Neworks, Monreal, Canada, July 3 - Augus 4, 2005 Dynamcally Weghed Majory Vong for Incremenal Learnng and Comparson of Three Boosng Based Approaches Alasgar Gangardwala Elecrcal and Compuer Engneerng Rowan Unversy Glassboro, NJ USA gangar34@sudens.rowan.edu Rob Polkar Elecrcal and Compuer Engneerng Rowan Unversy Glassboro, NJ USA polkar@rowan.edu Absrac - We have prevously nroduced Learn++, an ensemble based ncremenal learnng algorhm for acqurng new knowledge from daa ha laer become avalable, even when such daa nroduce new classes. In hs paper, we descrbe a modfcaon o hs algorhm, where he vong weghs of he classfers are updaed dynamcally based on he locaon of he es npu n he feaure space. The new algorhm provdes mproved performance, sronger mmuny o caasrophc forgeng and fner balance o he sably-plascy dlemma han s predecessor, parcularly when new classes are nroduced. The modfed algorhm and s performance, as compared o Adaboos.M and he orgnal Learn++, on real and benchmark daases are presened. I. INTRODUCTION A. Incremenal Learnng Supervsed classfers are effecve and powerful learnng ools for paern recognon and machne learnng applcaons. As mos machne learnng and paern recognon professonals are panfully aware of, however, he generalzaon performance of any learnng algorhm reles heavly on adequae and represenave ranng daa. Snce daa collecon s an expensve, me consumng and a edous process for mos praccal applcaons, such daa are ofen acqured n small baches over me. Wang for he enre daase o be avalable for ranng may prove neffecve, uneconomcal and nflexble. In such cases, would be more desrable o ran a classfer on avalable daa and ncremenally updae he classfer as new daa become avalable, whou compromsng he performance on prevously learned daa. Learnng new daa ncremenally whou forgeng prevously acqured knowledge rases he ssue of sablyplascy dlemma []: acqurng new knowledge requres plascy, whereas reanng prevously acqured knowledge requres sably. The challenge s hen o acheve a meanngful balance beween hese wo conflcng properes. Many of he commonly used supervsed classfers such as mullayer percepron (MLP), radal bass funcon, probablsc neural neworks, ec. are very sable classfers, unable o learn new nformaon. The praccal approach generally aken wh hese classfers for ncremenal learnng s o dscard he prevously raned classfer, combne and use he enre ranng daa accumulaed hus far o creae a new classfer from scrach. Ths approach effecvely causes he prevously learned nformaon o be enrely los, a phenomenon known as caasrophc forgeng [2,3]. For he purpose of hs work, we defne an ncremenal learnng algorhm as one ha has: () he capably of learnng novel nformaon conen from consecuve daases whou requrng access o prevously used daa; (2) he capably of reanng prevously learned knowledge; and (3) he ably o learn new classes nroduced by new daases. Learn++, based on weghed majory vong of an ensemble of classfers, sasfes he above lsed crera for ncremenal learnng, ye ressan o aforemenoned drawbacks [4, 5, 6]. In essence, he dea s o generae an ensemble of classfers wh he nal daa, and generae addonal classfers as new daases are acqured. We have recenly noced ha he way n whch he vong weghs are assgned o classfers based on her performance durng ranng s subopmal, because hese weghs are se durng ranng and reman consan hereafer. A dynamc approach ha assgns vong weghs o classfers based on he esmaed performance of each classfer on ha nsance may be more opmal. B. Ensemble of Classfers and Weghed Majory Vong Ensemble approaches have drawn much neres snce Hansen and Salamon s semnal work [7]. In essence, a group of classfers are raned usng dfferen dsrbuons of ranng samples, and oupus of hese classfers are hen combned n some manner o oban he fnal classfcaon rule. Learn++ uses he synergsc power of such an ensemble for ncremenal learnng of novel conen provded by consecuve daases. The algorhm was nspred from Freund and Schapre s adapve boosng (Adaboos.M) algorhm [8], whch also was orgnally proposed for mprovng he performance of weak classfers. I s based on he weghed majory vong [9] of hypoheses ha are generaed by sequenally ranng a se of weak classfers on dfferen ds /05/$ IEEE 3

2 rbuons of he ranng daa. Usng weak classfers allow dfferen classfers o make dfferen errors, a combnaon of whch hrough weghed majory vong hen effecvely averages ou he ndvdual errors resulng n a sronger classfer wh a much mproved generalzaon performance. The orgnal verson of Learn++ followed he AdaBoos approach n deermnng vong weghs, whch were assgned durng ranng dependng on he classfers performance on her own ranng daa. Whle hs approach makes perfec sense when he enre daa come from he same daabase, does have a handcap when used n an ncremenal learnng seng: snce each classfer s raned o recognze (slghly) dfferen porons of he feaure space, classfers performng well on a regon represened by her ranng daa may no do so well when classfyng nsances comng from dfferen regons of he space. Therefore, assgnng vong weghs prmarly on he ranng performance of each classfer s subopmal. Esmang he poenal performance of a classfer on a es nsance usng a sascal dsance merc, and assgnng vong weghs based on hese esmaes may be more opmal. In hs paper, we presen a modfed verson of Learn++ along wh s smulaon resuls as compared o he orgnal Learn++ o show he mprovemen on generalzaon performance and sably n ncremenally learnng new nformaon conen. Also new n hs sudy, we evaluae AdaBoos for ncremenal learnng and compare boh versons of Learn++. We show ha, whle no orgnally nended for such applcaons, AdaBoos s capable of ncremenal learnng, albe wh a lower performance, effcency and sably hen eher versons of Learn++. Overvews of oher ensemble and classfer combnaon echnques can be found n [0~5] and references whn. II. LEARN++ WITH DYNAMICALLY UPDATED VOTING WEIGHTS Learn++, smlar o AdaBoos, generaes an ensemble of weak classfers by choosng a subse of he ranng daa from he daabase usng an eravely updaed wegh dsrbuon rule, no o be confused wh he vong weghs. Learn++, however, combnes all classfers a each eraon o oban a compose hypohess before updang he dsrbuon. The dsrbuon updae rule of Learn++ s hen based on he performance of hs compose hypohess, whch represens he performance of he enre ensemble ha has been generaed hus far. The dsrbuon weghs of hose nsances ha are correcly denfed by he ensemble are reduced. Ths dsrbuon updae rule s desgned specfcally o accommodae ncremenal learnng of addonal daases, especally hose ha nroduce prevously unseen classes. The pseudocode of he algorhm s gven n Fg.. For each daabase D k, k=,,k ha becomes avalable, he npus o Learn++ are: () labeled ranng daa S k ={[(x, y )], =,,m k }, where x s he ranng nsance and y s he correc label; (2) a weak learnng algorhm BaseClassfer; and (3) an neger T k, he oal number of weak classfers o be generaed. The BaseClassfer can be any supervsed algorhm ha can oban a mnmum of 50% classfcaon performance on ranng daa, ensurng he classfer s relavely weak, ye reasonably srong o have a meanngful performance. Usng a weak classfer has he addonal advanage of rapd learnng, snce he me-consumng fne-unng sep, whch could poenally cause overfng, s avoded. Unless here s compellng reason o choose oherwse, he dsrbuon weghs are nalzed o be unform, so ha all nsances have he same probably of beng seleced no he frs ranng subse. If k> (ha s, new daabase has been nroduced), a dsrbuon nalzaon sequence renalzes he daa dsrbuon (he If block n Fg. ) based on he performance of he curren ensemble on he new daa. A each eraon, he dsrbuon D s obaned by normalzng he weghs w of he nsances based on her ndvdual classfcaon by he curren ensemble (sep ). m k D = w w () () = The ranng daase S k s dvded no a ranng (TR ) and a esng subse (TE ) accordng o D (sep 2). Learn++ han calls BaseClassfer (sep 3) and rans wh TR o generae a weak hypohess h. The error of hs hypohess s calculaed on he curren ranng daa S k by addng he dsrbuon weghs of he msclassfed nsances (sep 4) [ h ( x ) y ] ε D = D (2) = : h y = where [ ] s f he predcae s rue, and 0 oherwse. If ε>/2, curren h s deemed oo weak, and s replaced wh a new h generaed from a fresh se of TR and TE. If ε</2, he curren hypohess s added o he prevous hypoheses and all hypoheses generaed durng he prevous eraons are hen combned usng he weghed majory vong o consruc he compose hypohess H (sep 5). In orgnal Learn++, he vong weghs were calculaed based on he error ε, so ha hypoheses wh lower error were gven hgher weghs, resulng n classes predced by hese hypoheses o be weghed more heavly. Snce he hypohess weghs are assgned pror o esng based on her ndvdual ranng performance, hs wegh assgnmen s subopmal. Ths s because, hypoheses are raned wh dfferen (and possbly overlappng) porons of he feaure space, and may no be reasonable o expec a classfer o perform well on es nsances ha may come from dfferen porons of he feaure space. Ths s no lkely o be a major ssue when only a sngle daabase s used (as n AdaBoos); however, s a vald concern n an ncremenal learnng seng. A more opmal rule would be o dynamcally esmae whch hypoheses are more lkely o correcly classfy any gven nsance and gve hem hgher vong weghs accordngly. Therefore, we modfy he expresson for compose hypoheses, represenng ensemble decson, as 32

3 Inpu: For each daase drawn from D k k=,2,,k Sequence of m k examples S k = {( x, y ) =,, } Weak learnng algorhm BaseClassfer. Ineger T k, specfyng he number of eraons. Do for each k=,2,,k: Inalze w = D =,, =,2,, If k>, Go o Sep 5, evaluae curren ensemble on new daa se D k, updae wegh dsrbuon; End If Do for =,2,...,T k : m. Se D = w w() so ha D s a dsrbuon. = 2. Draw ranng TR and esng TE subses from D. 3. Call BaseClassfer o be raned wh TR. 4. Oban a hypohess h and calculae s error ε = D on S k. : h ( x ) y If ε > ½, dscard h and go o sep Call dynamcally weghed majory vong (DWMV) o oban he compose hypohess H = arg max DW ( x ) : h 6. Compue he error of he compose hypohess E = D = D [ H ( x ) y ] : H ( x ) y = If E > ½, dscard H and go o sep Se B = E /(-E ), and updae he weghs: B, f H ( x ) w + ( ) = w, oherwse [ H ( ) ] ( ) x y = w B End Call DWMV and oupu he fnal hypohess: K H fnal( x ) = argmax DW k= : h Fg.. Pseudocode for he modfed Learn++ algorhm H = arg max DW ( x ) : h where DW (x) s he dynamc wegh assgned o hypohess h for he nsance x. As descrbed below, dynamc weghs are deermned by usng Mahalanobs-dsance based esmaed lkelhood of h o correcly classfy he nsance x. The compose error E made by H, ha s, he performance of he enre ensemble consruced hus far, s hen deermned by summng up he dsrbuon weghs of all nsances msclassfed by he ensemble (sep6). E = D = D [ H ( x ) y ]. : H ( x ) y = Fnally, he compose normalzed error s deermned as (3) (4) B E E ), 0< E < and 0< B < (5) = ( 2 and dsrbuon weghs are updaed accordng o he ensemble performance (sep 7). B, f H ( x ) w + ( ) = w, oherwse [ H ( ) ] ( ) x y = w B. Ths expresson reduces he weghs of hose nsances correcly classfed by he compose hypohess H, by a facor of B, whle he weghs of he msclassfed nsances are kep unchanged. A + s eraon, afer normalzaon of he weghs n sep, he probably of choosng prevously correcly classfed nsances for TR + s reduced, whle ha of msclassfed nsances s effecvely ncreased. Ths would be a logcal place o pause and pon ou o some of he man dfference beween AdaBoos and Learn++. The dsrbuon updae rule n AdaBoos s based on he performance of he prevous hypohess [8], whch focuses he algorhm on dffcul nsances wh respec o dfferen samplng of a gven sngle daabase, whereas ha of Learn++ s based on he performance of he enre ensemble [4], whch focuses hs algorhm on nsances ha carry novel nformaon wh respec o consecuve daabases. Ths becomes parcularly crcal when new daabase nroduces nsances from a prevously unseen class. Snce none of he prevous classfers n he ensemble has seen he nsances from he new class, H nally msclassfes hem, forcng he algorhm o focus on hese nsances ha carry novel nformaon. The procedure would no work nearly as effcenly, however, f he wegh updae rule were based on he performance of h only (as AdaBoos does) nsead of H. Ths s because he ranng performance of he frs h on nsances from he new class s ndependen of ha of he prevously generaed classfers. Therefore, h s lkely o correcly classfy new class nsances ha has jus seen, bu only a he me hey are frs nroduced. Ths would cause he algorhm o focus on oher dffcul o learn nsances, such as oulers, raher hen he nsances wh novel nformaon conen. Once T k hypoheses are generaed for each daabase D k, he fnal hypohess H fnal can be obaned by combnng all hypoheses by dynamcally weghed majory vong, choosng he class ha receves he hghes oal voe among all hypoheses: H fnal K k= : h (6) ( x ) = arg max DW. (7) The nuon n usng dynamcally updaed vong weghs s as follows: f we knew whch hypoheses would perform bes ahead of me, we would gve hose hypoheses hgher weghs. We canno have hs nformaon a pror, 33

4 however, we can esmae whch classfers are more lkely o correcly denfy a gven nsance based on he locaon of ha nsance n he feaure space wh respec o he nsances used o ran ndvdual classfers. If an nsance s spaally close n a dsance merc sense o he ranng daa used o ran a classfer, hen s reasonable o expec ha ha classfer wll perform well on he gven nsance. We use he class-specfc Mahalanobs dsance merc o compue he dsance beween he ranng daa and he unknown nsance for each classfer. Classfers whose ranng daase are closer o he unknown nsance are weghed hgher. We noe ha prevously seen daa need no be sored n order o compue he desred dsances, bu only he means and covarance marces of he ranng ses. We formalze he compuaon of hese weghs as follows: Le us defne TR c as he subse of TR, he ranng daase used durng he h eraon, o nclude only hose nsances ha belongs o class c, ha s, TR C c = { x x TR & y = c} TR = TRc (8) c= where C s he oal number of classes. Class-specfc Mahalanobs dsance s hen compued as, T Mc ( x ) = ( x mc) Cc ( x mc), c =,.., C (9) where m c s he mean and C c s he covarance merc of TR c. For any nsance x, he Mahalanobs dsance based dynamc wegh of he h hypohess s hen compued as DW ( x ) =, c =,.., C; =,..., T mn( Mc( x)) (0) where T s he oal number of hypoheses generaed. The Mahalanobs dsance mplcly assumes ha he underlyng daa dsrbuon s Gaussan, whch n general s no he case. Ye s more nformave hen oher dsance mercs as akes he daa covarance no consderaon, and provdes promsng resuls demonsrang s effecveness. III. SIMULATION RESULTS In hs paper, we presen smulaon resuls of Learn++ wh dynamc vong wegh updae along wh Learn++ and Adaboos.M on one real world and wo benchmark daases. All resuls are gven as 95% confdence nerval obaned hrough 0-fold cross valdaon. To smulae ncremenal learnng, he ranng s done n sessons, where only he mos recenly avalable daabase s shown o he algorhm durng he curren ranng sesson (TS). A. Volale Organc Compounds (VOC) Daabase Ths daabase was generaed from responses of sx quarz crysal mcrobalances (QCMs) o varous concenraons of fve volale organc compounds, Ehanol (ET), Ocane (OC), Toluene (TL), Xylene (XL) and Trchloroehylene (TCE). The daabase was paroned no four ses, S ~S 3 for ranng, where each se nroduces one new class, and TEST for valdaon. The daa dsrbuon s shown n Table. The base classfer used for all hree algorhms was a sngle layer MLP wh jus enough hdden layer nodes and a raher oleran error goal o make a reasonably weak classfer for hs daabase. Tables 2, 3, and 4 llusrae he percen ranng and generalzaon performances of Learn++ wh dynamcally updaed vong weghs (DUVW), orgnal Learn++ and Adaboos.M, respecvely, on VOC daa, afer each ranng sesson, TS ~TS 3. TABLE. DATA DISTRIBUTION FOR VOC DATABASE Daase ET OC TL TCE XL S S S TEST TABLE 2. DUVW- LEARN++ PERFORMANCE ON VOC DATA Daase TS TS 2 TS 3 S 99.37~ ~ ~87.47 S ~ ~90.4 S ~94.78 TEST 59.7~ ~ ~88.60 TABLE 3. ORIGINAL LEARN++ PERFORMANCE ON VOC DATA Daase TS TS 2 TS 3 S 99.9~ ~ ~8.93 S ~ ~93.49 S ~95.68 TEST 6.70~ ~ ~88.46 TABLE 4. ADABOOST.M PERFORMANCE ON VOC DATA Daase TS TS 2 TS 3 S ~ ~77.88 S ~ ~84.30 S ~94.87 TEST 6.2~ ~ ~82.58 Whle all algorhms acheved ncremenal learnng, Learn++ wh dynamcally updaed vong weghs performed bes, jus slghly beer han he orgnal verson of Learn++, and sgnfcanly beer han Adaboos.M. I s also worh nong ha he confdence nerval of he modfed Learn++ was also narrower han ha of s predecessor, ndcang less varably and ncreased sably n he performance of he modfed algorhm. Tables 2~4 also show some declne n ranng performances over hree ranng sessons (on daases S ~S 3 ). Ths s expeced due o sably-plascy dlemma. We noe, however ha he loss of prevously acqured knowledge as measured by ranng daa performance s much less n he modfed Learn++ hen s n ohers. B. Wsconsn Breas Cancer (BC) Daabase Ths daabase, orgnally creaed a The Unversy of Wsconsn, Madson [6], was obaned from he UCI repos- 34

5 ory [7]. The daabase consss of nne feaures and a oal of 683 nsances from wo classes of breas umors: bengn and malgnan. The daa dsrbuon s shown n Table 5. The base classfer was agan a MLP ype neural nework wh smlar characerscs as descrbed earler. Tables 6~8 presen he 0-fold cross valdaon percen ranng and generalzed performances. TABLE 5. DATA DISTRIBUTION FOR BC DATA Daase Bengn Malgnan S S TEST TABLE 6. DUVW LEARN++ PERFORMANCE ON BC DATA Daase TS TS 2 S 93.97~ ~96.8 S ~95.76 TEST 94.82~ ~98.4 TABLE 7. ORIGINAL LEARN++ PERFORMANCE ON BC DATA Daase TS TS 2 S 94.7~ ~96.44 S ~96.08 TEST 96.34~ ~98.7 TABLE 8. ADABOOST.M PERFORMANCE ON BC DATA Daase TS TS 3 S 94.72~ ~97.57 S ~95.63 TEST 94.95~ ~98.5 The ranng and generalzaon performances n Tables 6~8 ndcae ha all hree algorhms do equally well n learnng addonal nformaon f no new classes are nroduced. In hs applcaon, all algorhms have acqured mos of her knowledge from S durng he frs ranng sesson, however, hey were sll able o exrac ncremenal amoun of new knowledge from he second daase, S 2. The performance dfference beween he modfed Learn++, orgnal Learn++ and AdaBoos.M become less sgnfcan under such scenaros, where no new classes are nroduced, or no subsanal novel conen s provded wh he new daabase. Ths s expeced, as he man dfference beween he orgnal Learn++ and AdaBoos.M s he dsrbuon updae rule ha s geared owards learnng new classes. Smlarly, he modfcaon wh he dynamc vong weghs becomes more meanngful when dfferen daases cover subsanally dfferen porons of he feaure space, whch happens more drascally when eher a new class s nroduced, or he new nsances carry subsanal amoun of novel nformaon conen. C. Vehcle Slhouees Daabase Vehcle daabase was also obaned from he UCI reposory [7]. Ths daabase consss of 8 feaures n 946 nsances from four vehcle classes. Ths daabase s known o be challengng daabase, as ypcal performances on varous algorhms on hs daabase has reporedly been around 65~75% on non-ncremenal learnng [7]. The vehcle daabase was dvded no hree ranng daase S ~ S 3 and one es daase, TEST. The daa dsrbuon s shown n Table 9, whch was specfcally based owards new classes. Tables 0~2 summarze 0-fold cross valdaon percen ranng and generalzaon performances of Learn++ wh dynamc vong wegh updae, Learn++ and Adaboos.M, respecvely afer each ranng sesson, TS ~ TS 3. The base classfer used was agan a sngle layer MLP ype neural nework, wh smlar characerscs as descrbed above. TABLE 9. DATA DISTRIBUTION FOR THE VEHICLE DATABASE Daase Opel Saab Bus Van S S S TEST TABLE 0. DUVW LEARN++ PERFORMANCE ON VEHICLE Daase TS TS 2 TS 3 S 88.66~ ~ ~79.68 S ~ ~73.66 S ~87.43 TEST 47.00~ ~ ~75.46 TABLE. ORIGINAL LEARN++ PERFORMANCE ON VEHICLE Daase TS TS 2 TS 3 S 89.60~ ~ ~76.78 S ~ ~64.03 S ~86.92 TEST 47.8~ ~ ~73.20 TABLE 2. ADABOOST.M PERFORMANCE ON VEHICLE Daase TS TS 2 TS 3 S 67.20~ ~ ~83.2 S ~ ~52.88 S ~80.90 TEST 35.37~ ~ ~63.48 The generalzaon (TEST) performances n Tables 0~2 ndcae ha he modfed Learn++ has ouperformed oher wo algorhms boh n performance and n he confdence nerval of he performance. Based on 0-fold cross valdaon, he generalzaon performance of he modfed Learn++ was n 72~75% range, compared o 68~73% for orgnal Learn++ and 52~63% for AdaBoos.M. Furhermore, he modfed Learn++ places self much more favorably along he plascy sably specrum, as was able o rean sgnfcanly more of s prevously acqured knowledge hen he oher algorhms. 35

6 IV. DISCUSSION AND CONCLUSIONS In hs paper, we presened a modfed approach o weghed majory vong rule, where he classfers are weghed dynamcally for each nsance, dependng upon he esmaed lkelhood of he hypoheses o correcly classfy he unknown nsance. The nuve dea behnd hs approach s ha he classfer whose ranng daase s closes o he gven nsance, has more nformaon abou ha parcular nsance and herefore s more lkely o classfy ha nsance correcly. Smulaon resuls ndcae ha all hree algorhms are capable of ncremenal learnng; however, he resuls were mos favorable and promsng for he modfed Learn++ usng dynamcally updaed vong weghs. We noe ha he generalzaon performances obaned by he modfed Learn++ durng ncremenal learnng were very smlar, f no beer, hen he generalzaon performances obaned by several oher algorhms on hese daases when used n a non-ncremenal learnng seng as repored n [6]. Learnng n a non-ncremenal seng allows he enre daa o be made avalable o he algorhm a once, whch s a much smpler problem. The modfed Learn++ algorhm exhbed no only a beer generalzaon performance, bu also a sgnfcanly narrower confdence nerval. The mproved confdence nerval s n fac worh aenon. Ths s because a narrower confdence nerval ndcaes mproved sably and robusness, quales of consderable concern n ncremenal learnng. In parcular, mproved generalzaon performance coupled wh a narrower confdence nerval s a sasfyng oucome, snce hs combnaon places he modfed Learn++ very favorably on he sably-plascy specrum. We also run all algorhms mulple mes under several oher scenaros, such as changng he order n whch he ranng daa are presened, and changng he base classfer ranng parameers (such as number of hdden layer nodes, error goal, ec.). We have found ou ha all algorhms are robus o he order n whch he daases are presened, as well as o reasonable modfcaons n he ranng parameers. Also, none of he algorhms suffer from caasrophc forgeng, snce prevously generaed classfers are reaned. Loss of some nformaon s nevable due o he sablyplascy dlemma whle new nformaon s beng learned. However, hs loss of prevously acqured knowledge was very margnal wh he modfed Learn++, bu mos promnen wh AdaBoos, when he daa nroduced sgnfcan amoun of novel nformaon conen, such as a new class. We conclude by resang ha n applcaons where he addonal nformaon conen s mnmal, he performance dfferences beween he algorhms become less sgnfcan. The promsng resuls of he modfed Learn++ wh dynamcally updaed vong weghs are mos meanngful and benefcal when he algorhm s used under he scenaros for whch s specfcally desgned, ha s, when he addonal daa provde sgnfcan novel nformaon conen. ACKNOWLEDGEMENT Ths maeral s based upon work suppored by he Naonal Scence Foundaon under Gran No. ECS , CAREER: An Ensemble of Classfers Approach for Incremenal Learnng. REFERENCES [] S. Grossberg, Nonlnear neural neworks: prncples, mechansms and archecures, Neural Neworks, vol., no., pp. 7-6, 988. [2] M. McCloskey and N. Cohen, Caasrophc nerference n connecons neworks: he sequenal learnng problem, n The Psychology of Learnng and Movaon, G.H. Bower, ed., vol. 24, pp , Academc Press, San Dego, 989. [3] R. French, Caasrophc forgeng n connecons neworks, Trends n Cognve Scences, vol. 3, no.4, pp , 999. [4] R. Polkar, L. Udpa L, S. Udpa, and V. Honavar, Learn++: An ncremenal learnng algorhm for supervsed neural neworks, IEEE Trans. on Sysem, Man and Cybernecs (C), vol. 3, no. 4, pp , 200. [5] R. Polkar, J. Byorck, S. Krause, A. Marno, and M. Moreon, Learn++: A classfer ndependen ncremenal learnng algorhm for superv. neural new., Proc. of In. Jon Conf. on Neural Neworks., vol. 2, pp , Honolulu, HI, [6] R. Polkar, L. Udpa L, S. Udpa, and V. Honavar, An ncremenal learnng algorhm wh confdence esmaon for auomaed denfcaon of NDE sgnals, IEEE Trans.Ulraso., Ferro., Freq. Con., vol. 5, no. 8, pp , [7] L. Hansen and P. Salamon, Neural nework ensembles, IEEE Trans. on PAMI, vol. 2, no. 0, pp , 990. [8] Y. Freund and R. Schapre, A decson heorec generalzaon of onlne learnng and an appl. o boosng, Comp. and Sysem Scences, vol. 57, no., pp. 9-39, 997. [9] N. Llesone and M. Warmuh, Weghed majory algorhm, Infor. and Compu., vol. 08, pp , 994. [0] M.I. Jordan and R.A. Jacobs, Herarchcal mxures of expers and he EM algorhm, Neural Compuaon, vol. 6, no. 2, pp. 8-24, 994. [] J. Kler, M. Haef, R.P. Dun, J. Maas, On combnng classfers, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 20, no.3, pp , 998. [2] L. Breman, Combnng predcors, Combnng Arfcal Neural Nes, A.Sharkey, ed., pp. 3-50, NY: Sprnger 999. [3] T. Deerch, Ensemble mehods n machne learnng, Proc. s In. Wkshop on Mul. Class. Sys., LNCS, J. Kler, F. Rol, ed., vol. 857, pp.-5, NY, Sprnger [4] J. Ghosh, Mulclassfer sysems: back o he fuure, 3rd In. Work. on Mul. Classfer Sys., LNCS (J. Kler & F. Rol, eds), vol. 2364, p. -5, NY: Sprnger, [5] T. Wndea and F. Rol (eds), In Proc. 4h In. Workshop on MCS, LNCS, vol. 2709, NY, Sprnger, [6] W. H. Wolberg and O.L. Mangasaran, Mulsurface mehod of paern separaon for medcal dagnoss appled o breas cyology, Proc. of he Naonal Academy of Scences, vol. 87, pp , 990. [7] C.L. Blake and C.J. Merz, Unv. of Calforna, Irvne, Reposory of Machne Learnng Daabases a Irvne, CA. 36