Ensemble of Classifiers Based Incremental Learning with Dynamic Voting Weight Update

Transcription

1 Ensemble of Classfers Based Incremenal Learnng wh Dynamc Vong Wegh Updae Rob Polkar, Sefan Krause and Lyndsay Burd Elecrcal and Compuer Engneerng, Rowan Unversy, 36 Rowan Hall, Glassboro, NJ 828, USA. Absrac An ncremenal learnng algorhm based on weghed majory vong of an ensemble of classfers s nroduced for supervsed neural neworks, where he vong weghs are updaed dynamcally based on he curren es npu of unknown class. The algorhm s dynamc vong wegh updae feaure s an enhancemen o our prevously nroduced ncremenal learnng algorhm, Learn++. The algorhm s capable of ncremenally learnng new nformaon from addonal daases ha may laer become avalable, even when he new daases nclude nsances from addonal classes ha were no prevously seen. Furhermore, he algorhm reans formerly acqured knowledge whou requrng access o daases used earler, aanng a delcae balance on he sably-plascy dlemma. The algorhm creaes addonal ensembles of classfers based on an eravely updaed dsrbuon funcon on he ranng daa ha favors ranng wh ncreasngly dffcul o learn, prevously no learned and/or unseen nsances. The fnal classfcaon s made by weghed majory vong of all classfer oupus n he ensemble, where he vong weghs are deermned dynamcally durng acual esng, based on he esmaed performance of each classfer on he curren es daa nsance. We presen he algorhm n s enrey, as well as s promsng smulaon resuls on wo real world applcaons. I. INTRODUCTION A. Incremenal Learnng As mos researchers n machne learnng are panfully aware, he generalzaon performance of any learnng algorhm s acuely conngen upon he avalably of an adequae and represenave ranng daase. Ofen mes, however, acquson of such daa s edous, me consumng and expensve. Therefore, s no unusual for such daa o become avalable n baches over a perod of me. Furhermore, s also no unusual for daa belongng o dfferen classes o be acqured n separae daa acquson epsodes. Dependng on he exac naure of he applcaon, wang for he enre daa o become avalable can be neffecve, fnancally uneconomcal, echncally mprovden, or even unfeasble parcularly f he exac naure of fuure daa s unknown. Under such scenaros, would be more effecve o be able o sar ranng a classfer wh he exsng daa, and hen ncremenally updae hs classfer o accommodae new daa whou, of course, compromsng classfcaon performance on prevously seen daa. Snce mos of he commonly used classfers, ncludng he ubquous mullayer percepron (MLP) and he radal bass funcon (RBF) neworks are unable o accommodae such ncremenal learnng, he praccal approach has radonally been dscardng he exsng classfer and sarng from scrach by combnng all daa accumulaed hus far every me a new daase becomes avalable. Ths approach resuls n loss of all prevously learned knowledge, a phenomenon known as caasrophc forgeng. Furhermore, he combnaon of old and new daases s no even always possble f prevous daases are los, dscarded, corruped, naccessble, or oherwse unavalable. Incremenal learnng of new nformaon whou forgeng wha s prevously learned rases he so-called sably plascy dlemma []: some nformaon may have o be los o learn new nformaon, as learnng new paerns wll end o overwre formerly acqured knowledge. Thus, a sable classfer can preserve exsng knowledge, bu canno accommodae new nformaon, whereas a plasc classfer can learn new nformaon, bu canno rean pror knowledge. The ssue a hand s hen, f, when and how much should one be sacrfced for he oher o acheve a meanngful balance. Varous defnons and nerpreaons of ncremenal learnng can be found n leraure. A represenave, ye ceranly no exhausve, ls of references can be found n [2]. For he purposes of hs work, an algorhm possesses ncremenal learnng capables, f mees he followng crera: () ably o acqure addonal knowledge when new daases are nroduced, whou requrng access o prevously seen daa; (2) ably o rean a meanngful poron of he prevously learned nformaon; and (3) ably o learn new classes f nroduced by new daa. Algorhms referenced n [2], as well as many ohers are all capable of learnng new nformaon; hough hey sasfy he above-menoned crera only a varyng degrees: hey eher requre access o prevously seen daa, forge subsanal amoun of pror knowledge along he way, or canno accommodae new classes. One promnen excepon s he (fuzzy) ARTMAP algorhm [3] along wh s many recen varaons. However, has long been known ha ARTMAP s very sensve o selecon of s vglance parameer, o nose levels n he ranng daa and o he order n whch he ranng daa are presened o he algorhm. Furhermore, he algorhm ofen suffers from overfng problems, f he vglance parameer s no chosen appropraely. Varous approaches have been and are beng suggesed o overcome hese dffcules [4, 5, 6, 7, 8]. Recenly we have suggesed Learn++ as an alernae approach o he growng ls of ncremenal learnng algorhms [2,9]. Learn++, based on he weghed majory vong of an ensemble of classfers, sasfes he above menoned crera. However, he weghs for he majory vong are se durng ranng and reman consan (hence sac /3/$7. 23 IEEE 277

2 weghs) afer he ranng. In hs paper, we presen a modfed verson of he algorhm, where vong weghs are updaed dynamcally by esmang whch of he classfers are lkely o correcly classfy any gven es nsance. B. Ensemble of Classfers Learn++ akes advanage of he synergsc power of an ensemble of classfers n learnng a concep usng he dvde and conquer approach. The algorhm s n par nspred by he AdaBoos (adapve boosng) algorhm [], orgnally developed o mprove he classfcaon performance of weak classfers. In essence, an ensemble of weak classfers are raned usng dfferen dsrbuons of ranng samples, whose oupus are hen combned usng he weghed majory-vong scheme [] o oban he fnal classfcaon rule. The approach explos he so-called nsably of he weak classfers, whch allows he classfers o consruc suffcenly dfferen decson boundares for mnor modfcaons n her ranng daases, causng each classfer o make dfferen errors on any gven nsance. A sraegc combnaon of hese classfers hen elmnaes he ndvdual errors, generang a srong classfer. Usng ensemble of classfers has been well researched for mprovng classfer accuracy [2, 3, 4, 5, 6]; however, s poenal for addressng he ncremenal learnng problem has been mosly unexplored. Learn++ was developed n response o recognzng he poenal feasbly of ensemble of classfers n solvng he ncremenal learnng problem. Learn++ was frs nroduced n [7, 8], as an ncremenal learnng algorhm for MLP neworks. More recenly we showed ha Learn++ s acually que versale, as works wh any supervsed classfcaon algorhm [9]. In s orgnal form, Learn++ combnes he classfers usng weghed majory vong, where he vong weghs are deermned by ndvdual performances of he classfers on her own ranng daa. We realze ha hs s sub opmal, as a classfer ha performs well on s own ranng daa need no perform equally well on daa comng from a dfferen poron of he npu space. In hs paper, we frs nroduce he modfed Learn++ algorhm, whch uses a sascal-dsance-merc based procedure for esmang whch classfers are lkely o correcly classfy a gven unknown nsance. Hgher weghs are hen assgned o hose classfers esmaed o perform well on he gven nsance. II. LEARN++ Learn++ generaes a se of classfers (hypoheses) and combnes hem hrough weghed majory vong of he classes predced by he ndvdual hypoheses. The hypoheses are generaed by ranng a weak classfer, usng nsances drawn from eravely updaed dsrbuons of he ranng daabase. The dsrbuon updae rule used by Learn++ s desgned o accommodae addonal daases, n parcular hose ha nroduce prevously unseen classes. Each classfer s raned usng a subse of examples drawn from a weghed dsrbuon ha gves hgher weghs o examples msclassfed by he prevous ensemble. The pseudocode of he algorhm s provded n Fg.. For each daabase D k, k=,,k ha becomes avalable o he algorhm, he npus o Learn++ are () labeled ranng daa S k = {( x, y ) =, L, } where x and y are ranng nsances and her correc classes, respecvely ; (2) a weaklearnng algorhm BaseClassfer; and (3) an neger T k, he maxmum number of classfers o be generaed. For brevy we drop he subscrp k from all oher varables. BaseClassfer can be any supervsed algorhm ha acheves a leas 5% correc classfcaon on S k afer beng raned on a subse of S k. Ths ensures ha he classfer s suffcenly weak, ye srong enough o ensure a leas a meanngful classfcaon performance. We also noe ha usng weak classfers has he addonal advanage of rapd ranng and overfng avodance, snce hey only generae a gross approxmaon of he underlyng decson boundary. A each eraon, Learn++ frs nalzes a dsrbuon D, by normalzng a se of weghs, w, assgned o nsances based on her ndvdual classfcaon by he curren ensemble (sep ) m D = w w. () = Learn++ hen dvdes S k no wo muually exclusve subses by drawng a ranng subse TR and a es subse TE accordng o D (sep 2). Unless here s pror reason o choose oherwse, D s nally se o be unform, gvng equal probably o each nsance o be seleced no TR. Learn++ hen calls BaseClassfer o generae hypohess h (sep 3). The error of h s compued on S k = TR U TE by addng he dsrbuon weghs of msclassfed nsances (sep 4) ε [ h ( x ) y ] (2) h ( x ) y = where [ ] s f he predcae s rue, and oherwse. If ε > ½, curren h s dscarded and a new h s generaed from a fresh se of TR and TE. If ε < ½, hen normalzed error β s compued as β = ε ( ε ), < β <. (3) All hypoheses generaed durng he prevous eraons are hen combned usng weghed majory vong (sep 5), o consruc he compose hypohess H H = arg max log : h β. (4) H decdes on he wnnng class ha receves he hghes oal voe. In he orgnal Learn++ algorhm, he vong weghs were deermned based on he normalzed errors β : hypoheses wh lower normalzed errors are gven larger weghs, so ha he classes predced by a hypohess wh a proven record are weghed more heavly. The log funcon s used o conrol he explosve effec of very low β values assocaed wh classfers ha perform well durng ranng. 277

3 Algorhm Learn++ Inpu: For each daase drawn from D k k=,2,,k Sequence of m k examples S = {( x, y ) =, L, m } Weak learnng algorhm BaseClassfer. Ineger T k, specfyng he number of eraons. Do for each k=,2,,k: Inalze w = D = m,, =,2, L, m Do for =,2,...,T k : m. Se D = w w so ha D s a dsrbuon. = 2. Draw ranng TR and esng TE subses from D. 3. Call BaseClassfer o be raned wh TR. 4. Oban a hypohess h : X Y, and calculae he error of h : ε on TR + TE. (x ) y h k If ε > ½, dscard h and go o sep 2. Oherwse, compue normalzed error as β =ε / (-ε ). 5. Call dynamcally weghed majory vong o oban compose hypohess H = arg max DW : h 6. Compue he error of he compose hypohess E [ H ( x ) y ] H ( x ) y = 7. Se B = E /(-E ), and updae he weghs: B, f H ( x ) w + = w, oherwse [ H ( ) w B x y = ] Call Dynamcally weghed majory vong and Oupu he fnal hypohess: K H fnal ( x ) = arg max DW k= : h Fg. Algorhm Learn++ Whle hs rule makes nuve sense, and n fac performed remarkably well n a number of smulaons [9], s neverheless sub opmal. Ths s because he weghs are deermned and fxed pror o esng based on ndvdual performances of hypoheses on her own ranng daa subse. A rule ha dynamcally esmaes whch hypoheses are lkely o correcly classfy an unlabeled nsance, o gve hgher vong weghs o hose hypoheses would be more opmal. We herefore change he compose hypohess expresson as k H = arg max DW (5) : h where DW (x) s he nsance-specfc dynamc wegh assgned o nsance x by he hypohess h. Dynamc weghs are deermned usng a Mahalanobs-dsance-based esmaed lkelhood of h for correcly classfyng x, as descrbed below. The compose error E made by H s hen compued as he sum of dsrbuon weghs of nsances msclassfed by H (sep 6) E [ H ( x ) y ]. (6) H ( x ) y = The compose normalzed error s smlarly compued as B = E E, < B <. (7) The weghs w () are hen updaed, for compung he nex dsrbuon D +, whch n urn s used n selecng he nex ranng and esng subses, TR + and TE +, respecvely (sep 7) B, f H ( x ) w + = w, oherwse. (8) [ H ( ) w ( ) x y = ] B Ths rule reduces he weghs of hose nsances correcly classfed by he compose hypohess H by a facor of B (snce < B < ), whereas leaves he weghs of msclassfed nsances unchanged. Afer normalzaon (n sep of eraon +), he probably of correcly classfed nsances beng chosen no TR + s reduced, whle hose of msclassfed ones are effecvely ncreased. Therefore, he algorhm focuses on nsances ha are dffcul o classfy, or nsances ha have no ye been properly learned. Ths approach allows ncremenal learnng by concenrang on newly nroduced nsances, parcularly hose comng from prevously unseen classes, as hese are precsely hose nsances ha have no been learned ye. We emphasze he nroducon of he compose hypohess H by Learn++, whch along wh he wegh updae rule, unquely allows Learn++ o learn new classes. I s emprcally observed ha he procedure fals o learn new classes, f nsead he wegh updae rule were based on he performance of h only (as AdaBoos does). The wegh updae rule based on compose hypohess performance allows Learn++ o focus on hose nsances ha have no been learned by he curren ensemble, raher hen he prevous hypohess. Afer T k hypoheses are generaed for each daabase D k, he fnal hypohess H fnal s obaned by combnng all hypoheses ha have been generaed hus far usng he dynamcally weghed majory-vong rule choosng he class ha receves he hghes oal voe among all classfers, K H fnal ( x ) = arg max DW. (9) k= : h 2772

4 To deermne he dynamc vong weghs, we use he Mahalanobs dsance, o compue he dsance of he unknown nsance o he daases used o ran ndvdual classfers. Classfers raned wh daases closer o he unknown nsance are hen gven larger weghs. Noe ha hs approach does no requre he ranng daa o be saved, bu only he mean and covarance marces, whch are ypcally much smaller han he orgnal daa. In Learn++, we frs defne TR c as a subse of TR, he ranng daa used durng he h eraon, where TR c ncludes only hose nsances of TR ha belong o class c, ha s, { x x TR & c} U C TRc = y = TR TR c c= = () where C s he oal number of classes. The class specfc Mahalanobs dsance from an unknown nsance x o TR c, s hen compued as M c x T ( ) = ( x m ) C ( x m ) c c c, c =,2, L, C () where m c s he mean and C c s he covarance marx of TR c. For any nsance x, he Mahalanobs dsance based dynamc wegh of he h hypohess can hen be obaned as DW =, c =,2, L, C. =, L, T (2) mn( M c ) where T s he oal number of hypoheses generaed. The Mahalanobs dsance merc mplcly assumes ha he daa s drawn from a Gaussan dsrbuon, whch n general s no he case. However, hs merc provded promsng resuls demonsrang s effecveness. Oher dsance mercs ha do no make hs assumpon wll also be evaluaed. III. LEARN++ SIMULATION RESULTS The orgnal Learn++ algorhm usng he sac weghed majory vong has been esed on a number of benchmark and oher real world daabases, whose resuls are provded n [2,9]. In hs paper, we presen smulaon resuls of usng Learn++ wh dynamcally updaed weghed majory vong on wo real-world ncremenal learnng problems ha nroduce new classes wh addonal daa. A. Ulrasonc Weld Inspecon (UWI) Daabase Ths raher challengng daabase was obaned by ulrasonc scannng of weldng regons of varous sanless or carbon seel srucures. The weldng regons, known as heaaffeced zones, are hghly suscepble o growng a varey of defecs, ncludng poenally dangerous cracks. The dsconnues whn he maeral, such as he ar gaps due o cracks, cause he ulrasonc wave o be refleced back and receved by he ransducer. The refleced ulrasonc wave, also called he A-scan, serves as he sgnaure paern of he dsconnuy, whch s hen analyzed o deermne s ype. Ths analyss, however, s hampered by he presence of oher ypes of dsconnues, such as porosy (POR), slag and lack of fuson (LOF), all of whch generae very smlar A- scans, creang hghly overlappng paerns n he feaure space. Represenave A-scans are shown n Fg.2 Fg. 2 Represenave normalzed A-scans of (a) crack, (b) Lack of fuson, (c) slag, (d) porosy Ths four-class daabase was dvded no hree ranng daases, S ~ S 3, and a valdaon se, TEST. Table presens he daa dsrbuon n each daase. We noe ha each addonal daabase nroduced a new class: n parcular, S had nsances only from crack and LOF, S 2 nroduced slag nsances and S 3 nroduced porosy nsances. TEST daase ncluded nsances from all classes. Only S k was used durng he k h ranng sesson TS k. Therefore, prevously used daa were no made avalable o Learn++ n fuure ranng sessons. Table 2 summarzes he ranng and generalzaon performances of he algorhm afer each ranng sesson. The classfcaon performances on ranng and esng daases are shown n rows labeled by S, S 2, S 3, and TEST as obaned afer each ranng sesson, TS k. The numbers n parenheses ndcae he number of weak classfers generaed n each ranng sesson. Classfer generaon connued unl he generalzaon performance on he TEST daase, shown n he las row, reached a seady sauraon value. The weak learner used o generae ndvdual hypoheses was a sngle hdden layer MLP wh 5 hdden layer nodes. The mean square error goals of all MLPs were prese o a value of.2 o preven over-fng and o ensure suffcenly weak learnng. We noe ha any neural nework can be urned no a weak learnng algorhm by selecng s number of hdden layers and he number of hdden layer nodes small, and he error goal hgh, wh respec o he complexy of he problem. 2773

5 TABLE. DATA DISTRIBUTION FOR THE UWI DATABASE Daase LOF SLAG CRACK POR S 3 3 S S TEST TABLE 2. LEARN++ PERFORMANCE ON THE UWI DATABASE Daase TS (8) TS 2 (27) TS 3 (43) S 99.2% 89.2% 88.2% S % 88.% S % TEST 57.% 7.5% 83.8% Several observaons can be made from Table 2: () The generalzaon performances on TEST daase seadly ncrease, ndcang ha Learn++ s ndeed able o learn new nformaon, as well as new classes as hey become avalable; () In general, larger numbers of classfers are requred for ncremenal learnng of new classes. () There s an occasonal declne on ranng daa performances ndcang some loss, albe mnor, of prevously acqured knowledge as new nformaon s acqured. Ths s expeced due o sably-plascy dlemma; (v) Near 5% generalzaon performance afer TS makes nuve sense, snce a ha me he algorhm had only seen wo of he four classes ha appear n he TEST daa. The performance gradually ncreases proporonal o he rao of he classes seen, as hey become avalable. As a performance comparson, he same daabase was also used o ran and es a sngle srong learner, a 49x4x2x4 wo hdden layer MLP wh an error goal of.. The bes es daa classfcaon performance of he srong learner has been around 75%, despe he fac ha he srong learner was raned wh nsances from all classes. B. Volale Organc Compound (VOC)Daabase Ths daabase was obaned from responses of sx quarz crysal mcrobalances (QCMs) o varous concenraons of fve volale organc compounds (VOCs), ncludng ehanol (ET), xylene (XL), ocane (OC), oluene (TL), and rchloroehelene (TCE). When QCMs are exposed o VOCs, he molecular mass deposed on her crysal surface alers her resonan frequency, whch can be measured usng a frequency couner or a nework analyzer. By usng an array of QCMs, each coaed wh a dfferen polymer sensve o specfc VOCs, he collecve response of he array can be used as a sgnaure paern of he VOC. However, QCMs have very lmed selecvy, makng he denfcaon a challengng ask. Represenave paers of sensor responses for each VOC are shown n Fg. 3. Smlar o he UWI daabase, he VOC denfcaon daabase was dvded no hree ranng daases S ~ S 3 and one valdaon daase, TEST. The daa dsrbuon s shown n Table 3, whch was specfcally based owards he new class: daabase S had nsances from ET, OC and TL, S 2 added nsances manly from TCE (and very few from he prevous hree), and S 3 added nsances from XL (and very few from he prevous four). TEST se ncluded nsances from all classes. The base classfer used for hs daabase was also a MLP ype neural nework wh a 6x3x5 archecure, and an error goal of.5. Table 4 presens he ranng and generalzaon performances of he algorhm on hs daabase. Ehanol TCE Toluene Xylene Ocane Fg. 3 Represenave QCM sensor responses o fve VOCs TABLE 3.DATA DISTRIBUTION FOR THE VOC DATABASE Daase ET OC TL TCE XL S S2 25 S3 5 4 TEST TABLE 4. LEARN++ PERFORMANCE ON THE VOC DATABASE Daase TS (7) TS 2 () TS 3 (6) S % 97.5% 92.5% S % 96.4% S % TEST 57.3% 67.% 89.6% Performance fgures lsed above follow a smlar rend o hose of he UWI daabase. The seady ncrease on he generalzaon performance ndcaes ha Learn++ was able o ncremenally learn addonal nformaon provded by he new classes. We have evaluaed Learn++ numerous mes for each daabase wh slghly dfferen algorhm parameers, such as he order of nroducon of he classes, base classfer archecure and/or error goal, he percenage of ranng daa used as ranng and esng subses, ec. and even o a grea exend he choce of base classfer. In all cases, he algorhm seemed o be remarkably ressan o such changes. IV. DISCUSSIONS & CONCLUSIONS In hs paper, we nroduced a new verson of he Learn++ algorhm. Learn++ essenally adds he ncremenal learn- 2774

6 ng capably o any supervsed neural nework ha normally does no possess hs propery. As demonsraed by he resuls presened n hs and our prevous papers, Learn++ s ndeed able o learn new nformaon provded by addonal daabases, even when he new classes are nroduced by he consecuve daases. Learn++ akes advanage of he synergsc expressve power of an ensemble of weak classfers, where each classfer s raned wh a ranng daa subse drawn from a sraegcally updaed dsrbuon of he ranng daa. Indvdual classfers are hen combned usng he weghed majory-vong rule o oban he fnal classfer. In hs paper, we proposed an alernae weghed majory vong sraegy, where he vong weghs are deermned dynamcally for each nsance, based on he esmaed lkelhood of he hypoheses o correcly classfy ha nsance. The nuve dea behnd hs approach s ha hose classfers raned wh a daase ha ncluded nearby nsances n he Mahalanobs dsance sense o he unknown nsance are more lkely o correcly classfy he unknown nsance. I can be argued ha he classfcaon decson s already beng made by he choce of he class provdng he mnmum Mahalanobs dsance, snce f nsance x belongs o a parcular class, and nsances from ha class have been used n he curren daase, hen he Mahalanobs dsance beween x and TR c s lkely o be mnmum among all ohers. Ths s ndeed rue for daases wh non-overlappng classes wh nose-free well-behavng dsrbuons. In pracce, however, hs s rarely he case, and a decson based on he Mahalanobs dsance only nvarably acheves poor classfcaon performances on challengng daases, such as he UWI and VOC daases. We emphasze ha he Mahalanobs dsance s no used drecly for makng a classfcaon decson, bu raher o assgn a wegh o compeng hypoheses. Through ou he smulaons, a number of addonal observaons were made, no readly apparen from he ables. In parcular, we have esed he sensvy of Learn++ o he order of presenaon of he daa, as well as o mnor changes n s parameers. We found ou ha he classfcaon performance of Learn++ was vrually he same, regardless of he order n whch daabases (and herefore he classes) were presened o he algorhm. Furhermore, he algorhm was consderably robus o changes n s nernal parameers, such as he nework sze, error goal, he number of hypoheses generaed, and even he ype of base classfer. Fnally, unlke mos oher supervsed classfers, Learn++ does no suffer from caasrophc forgeng, snce prevously generaed classfers are reaned. Due o he sably plascy dlemma, some knowledge s ndeed forgoen whle new nformaon s beng learned; however, hs appears o be nsgnfcan, as ndcaed by he seady mprovemen n he generalzaon performance. One mgh also wonder wha generalzaon performance could be acheved f he enre daabase were avalable for srong learnng. Tranng a srong classfer usng he enre ranng daabase, we oban performances n he md 7% o hgh 8% range, smlar o, or even slghly worse han hose of Learn++. Ths raher sasfyng resul furher ndcaes he feasbly of Learn++ as an alernave o oher ncremenal learnng algorhms, as well as o non-ncremenal srong classfers. ACKNOWLEDGEMENT Ths maeral s based upon work suppored by he Naonal Scence Foundaon under Gran No. ECS REFERENCES [] S. Grossberg, Nonlnear neural neworks: prncples, mechansms and archecures, Neural Neworks, vol., no., pp. 7-6, 988. [2] R. Polkar, L. Udpa L, S. Udpa, and V. Honavar, Learn++: An ncremenal learnng algorhm for supervsed neural neworks, IEEE Transacons on Sysem, Man and Cybernecs (C), Specal Issue on Knowledge Managemen, vol. 3, no. 4, pp , 2. [3] G.A. Carpener, S. Grossberg, N. Markuzon, J.H. Reynolds, and D.B. Rosen, Fuzzy ARTMAP: A neural nework archecure for ncremenal supervsed learnng of analog muldmensonal maps, IEEE Trans. on Neural Neworks, vol. 3, no. 5, pp , 992. [4] J.R. Wllamson, Gaussan ARTMAP: a neural nework for fas ncremenal learnng of nosy muldmensonal maps, Neural Neworks, vol. 9, no. 5, pp , 996. [5] C.P. Lm and R.F. Harrson, An ncremenal adapve nework for onlne supervsed learnng and probably esmaon, Neural Neworks, vol., no. 5, pp , 997. [6] F. H. Haer, Lfe-long learnng cell srucures connuously learnng whou caasrophc nerference, Neural Neworks, vol. 4, no. 4-5, pp , 2. [7] G.C. Anagnosopoulos and M. Georgopoulos, Ellpsod ART and ARTMAP for ncremenal cluserng and classfcaon, In. Jon Conf. on Neural Neworks (IJCNN 2), vol. 2, pp , 2. [8] E. Gomez-Sanchez, Y.A. Dmrads, J.M. Cano-Izquerdo, J. Lopez- Coronado, Safe-µARTMAP: A new soluon for reducng caegory prolferaon n Fuzzy ARTMAP, Proc. of In. Jon Conf. on Neural Neworks (IJCNN 2), vol.2, pp , 2. [9] R. Polkar, J. Byorck, S. Krause, A. Marno, and M. Moreon, Learn++: A classfer ndependen ncremenal learnng algorhm for supervsed neural neworks, Proc. of In. Jon Conference on Neural Neworks (IJCNN 22), vol.2, pp , Honolulu, HI, 22. [] Y. Freund and R. Schapre, A decson heorec generalzaon of onlne learnng and an applcaon o boosng, Compuer and Sysem Scences, vol. 57, no., pp. 9-39, 997. [] N. Llesone and M. Warmuh, Weghed majory algorhm, Informaon and Compuaon, vol. 8, pp , 994. [2]L.K. Hansen and P. Salamon, Neural nework ensembles, IEEE Transacons on Paern Analyss and Machne Inellgence, vol. 2, no., pp. 993-, 99. [3] M.I. Jordan and R.A. Jacobs, Herarchcal mxures of expers and he EM algorhm, Proc. of In. Jon Conf. on Neural Neworks (IJCNN 993), pp , 993. [4] J. Kler, M. Haef, R.P. Dun, J. Maas, On combnng classfers, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 2, no.3, pp , 998. [5] T.G. Deerch, An expermenal comparson of hree mehods for consrucng ensembles of decson rees: Baggng, boosng and randomzaon, Machne Learnng, vol. 4, no. 2, pp. -9, 2. [6] L.I. Kuncheva, A heorecal sudy on sx classfer fuson saraeges, IEEE Transacons on Paern Analyss and Machne Inellgence, vol. 24, no. 2, pp , 22. [7] R. Polkar, Algorhms for enhancng paern separably, feaure selecon and ncremenal learnng wh applcaons o gas sensng elecronc nose sysems, Ph.D. dsseraon, Iowa Sae Unversy, Ames, IA, 2. [8] R. Polkar, L. Udpa, S. Udpa, V. Honavar, Learn++: An ncremenal learnng algorhm for mullayer neural neworks, Proc. of IEEE In. Conf. on Acouscs, Speech and Sgnal Proc., vol. 6, pp ,