Confidence Estimation Using the Incremental Learning Algorithm, Learn++

Transcription

1 Confdence Esmaon Usng he Incremenal Learnng Algorhm, Learn++ Jeffrey Byorck and Rob Polkar Elecrcal and Compuer Engneerng, Rowan Unversy, 136 Rowan Hall, Glassboro, NJ 08028, USA. Absrac. Paern recognon problems span a broad range of applcaons, where each applcaon has s own olerance on classfcaon error. The varyng levels of rsk assocaed wh many paern recognon applcaons ndcae he need for an algorhm wh he ably o measure s own confdence. In hs work, he supervsed ncremenal learnng algorhm Learn++ [1], whch explos he synergsc power of an ensemble of classfers, s furher developed o add he capably of assessng s own confdence usng a weghed exponenal majory vong echnque. 1 Inroducon 1.1 Incremenal Learnng I s wdely recognzed ha he recognon accuracy of a classfer s heavly ncumben on he avalably of an adequae and represenave ranng daase. Acqurng such daa s ofen edous, me-consumng, and expensve. In pracce, s no uncommon for such daa o be acqured n small baches over a perod of me. A ypcal approach n such cases s combnng new daa wh all prevous daa, and ranng a new classfer from scrach. Ths approach resuls n loss of all prevously learned knowledge, a phenomenon known as caasrophc forgeng. Furhermore, he combnaon of old and new daases s no even always a vable opon f prevous daases are los, dscarded, corruped, naccessble, or oherwse unavalable. Incremenal learnng s he soluon o such scenaros, whch can be defned as he process of exracng new nformaon whou losng pror knowledge from an addonal daase ha laer becomes avalable. Varous defnons and nerpreaons of ncremenal learnng can be found n leraure, ncludng onlne learnng [2,3], relearnng of prevously msclassfed nsances [4,5], and growng and prunng of classfer archecures [6,7]. For he purposes of hs work, an algorhm possesses ncremenal learnng capables, f mees he followng crera: (1) ably o acqure addonal knowledge when new daases are nroduced; (2) ably o rean prevously learned nformaon; (3) ably o learn new classes f nroduced by new daa. O. Kaynak e al. (Eds.): ICANN/ICONIP 2003, LNCS 2714, pp , Sprnger-Verlag Berln Hedelberg 2003

2 182 J. Byorck and R. Polkar 1.2 Ensemble of Classfers Ensemble sysems have araced a grea deal of aenon over he las decade due o her emprcal success over sngle classfer sysems on a varey of applcaons. Such sysems combne an ensemble of generally weak classfers o ake advanage of he so-called nsably of he weak classfer, whch causes he classfers o consruc suffcenly dfferen decson boundares for mnor modfcaons n her ranng parameers, causng each classfer o make dfferen errors on any gven nsance. A sraegc combnaon of hese classfers, such as weghed majory vong [8], hen elmnaes he ndvdual errors, generang a srong classfer. A rch collecon of algorhms have been developed usng mulple classfers, such as AdaBoos [9], wh he general goal of mprovng he generalzaon performance of he classfcaon sysem. Usng mulple classfers for ncremenal learnng, however, has been largely unexplored. Learn++, n par nspred by AdaBoos, was developed n response o recognzng he poenal feasbly of ensemble of classfers n solvng he ncremenal learnng problem. Learn++ was nally nroduced n [1] as an ncremenal learnng algorhm for MLP ype neworks. A more versale form of he algorhm was presened n [10] for all supervsed classfers. We have recenly recognzed ha nheren vong mechansm of he algorhm can also be used n effecvely deermnng he confdence of he classfcaon sysem n s own decson. In hs work, we descrbe he algorhm Learn++, along wh represenave resuls on ncremenal learnng and confdence esmaon obaned on one real world and one benchmark daabase from he Unv. of Calforna, Irvne (UCI) machne learnng reposory [11]. 2 Learn++ The Learn++ algorhm, gven n Fg. 1, explos he synergsc power of an ensemble of classfers o ncremenally learn new nformaon ha may laer become avalable. Learn++ generaes mulple weak classfers, each raned wh dfferen subses of he daa. For each daabase ' k, k1,,k ha becomes avalable, he npus o Learn++ are () Sk {( x, y) 1, L, mk}, a sequence of m k ranng daa nsances x, along wh her correc labels y, () a weak classfcaon algorhm BaseClassfer o generae weak classfers, and () an neger T k specfyng he number of classfers (hypoheses) o be generaed for ha daabase. We requre ha BaseClassfer oban a leas 50% correc classfcaon performance on s own ranng daase, o ensure a meanngful classfcaon performance for each classfer. Learn++ sars by nalzng a se of weghs for he ranng daa, w, and a dsrbuon D obaned from w, accordng o whch a ranng subse TR and a es subse TE are drawn a he h eraon of he algorhm, 1,,T k, where Sk TR U TE. Unless a pror nformaon ndcaes oherwse, hs dsrbuon s nally se o be unform, gvng equal probably o each nsance o be seleced no he frs ranng subse. The varaon of nsances whn he ranng daa subses s acheved by eravely updang he dsrbuon of weghs D. A each eraon, he weghs adjused a eraon -1 are normalzed o ensure ha a legmae dsrbuon, D, s obaned. TR and TE are hen drawn accordng o D and BaseClassfer s raned wh he ranng subse. A hypohess h s obaned as he h classfer, whose error s

3 Confdence Esmaon Usng he Incremenal Learnng Algorhm, Learn compued on he enre (curren) daabase S k smply by addng he dsrbuon weghs of he msclassfed nsances ε D ( ) (1) : h ( x ) y If ε > ½, h s dscarded and a new TR and TE are seleced. If he error s less hen half, hen he error s normalzed and compued as β ε ( 1 ε ), 0 β 1 (2) Hypoheses generaed n all prevous eraons are hen combned usng weghed majory vong o form a compose hypohess H usng H arg max y Y : h y 1 log β where he sum of weghs assocaed wh each classfer s compued for every class presen n he classfcaon ask. A hgher wegh s gven o classfers ha perform beer on her specfc ranng ses. The compose hypohess H s obaned by assgnng he class label o an nsance x ha receves he larges oal voe. The compose error made by H s hen compued as : H ( x ) y D ( ) m 1 D ( ) [ H ( x ) y ] where [ ] evaluaes o 1, f he predcae holds rue. Smlar o he calculaon of β, a normalzed compose error B s compued as ( 1 E ), 0 B 1 B E The weghs w () are hen updaed o oban D, whch s used for he selecon of +1 he nex ranng and esng subses, TR +1 and TE +1, respecvely. The dsrbuon updae rule whch comprses he hear of he algorhm s gven by w B, f H ( x ) y 1, oherwse [ H ( x ) y ] ( ) w ( ) w ( ) B. Ths rule reduces he weghs of hose nsances ha are correcly classfed by he compose hypohess H, so ha her probably of beng seleced no he nex ranng subse s reduced. When normalzed durng eraon +1, he weghs of msclassfed nsances are ncreased relave o he res of he daase. We emphasze ha unlke AdaBoos and s varaons, he wegh updae rule n Learn++ looks a he classfcaon oupu of he compose hypohess, no o ha of a specfc hypohess. Ths wegh updae procedure forces he algorhm o focus more on nsances ha have no been properly learned by he ensemble. When Learn++ s learnng ncremenally, he nsances nroduced by he new daabase are precsely hose no learned by he ensemble. Afer T k hypoheses are generaed for each daabase ' N, he fnal hypohess s obaned by he weghed majory vong of all compose hypoheses: (3) (4) (5) (6)

4 184 J. Byorck and R. Polkar H fnal arg max K y Y k 1 : H y 1 log B (7) Inpu: For each daase drawn from ' N k1,2,,k Sequence of m k examples S k {( x, y ) 1, L, mk } Weak learnng algorhm BaseClassfer Ineger T k, specfyng he number of eraons Inalze w 1 () D 1 ()1/m k,, 1,2,,m k Do for each k1,2,,k: Do for 1,2,,T k : m 1. Se D w w ( ) so ha D s a dsrbuon 1 2. Draw ranng TR and esng TE subses from D. 3. Call BaseClassfer o be raned wh TR. 4. Oban a hypohess h : XÅY, and calculae he error of h : ε D ( ) : h ( x ) y on TR +TE. If ε > ½, dscard h and go o sep 2. Oherwse, compue normalzed error as β ε ( 1 ε ). 5. Call weghed majory vong and oban he compose hypohess H argmax log 1 β y Y : h y ( ) 6. Compue he error of he compose hypohess E D ( ) m : H ( x ) y 1 7. Se B E ( E ) D ( ) [ H ( x ) y ] 1, and updae he weghs: Call Weghed majory vong and Oupu he fnal hypohess: H fnal y Y K k 1 : h y w + 1 ( ) w ( ) B ( ) arg max log 1 β Fg. 1. Learn++ Algorhm 1 [ H ( x ) y ] 3 Confdence Esmaon An nmaely relevan ssue s he confdence of he classfer n s decson, wh parcular neres n wheher he confdence of he algorhm mproves as new daa becomes avalable. The vong mechansm nheren n Learn++ hns o a praccal approach for esmang confdence: decsons made wh a vas majory of voes have beer confdence hen hose made by a slgh majory. We have mplemened McIver

5 Confdence Esmaon Usng he Incremenal Learnng Algorhm, Learn and Fredl s weghed exponenal vong based confdence merc [12] wh Learn++ as F j (8) e C j P( y j x) N, 0 C j 1 Fk e k 1 where C j (x) s he confdence assgned o nsance x when classfed as class j, F j (x) s he oal voe assocaed wh he j h class for he nsance x, and N s he oal number of classes. The oal voe F j (x) class j receves for any gven nsance s compued as N 1 log h j Fj β 1 0 oherwse The confdence of wnnng class s hen consdered as he confdence of he algorhm n makng he decson wh respec o he wnnng class. Snce C j (x) s beween 0 and 1, he confdences can be ranslaed no lngusc ndcaors, such as hose shown n Table 1. These ndcaors are adoped and used n abulang he resuls. (9) Table 1. Confdence percenages represened by lngusc ndcaors Confdence Percenage Range Confdence Level 90 C 100 Very Hgh (VH) 80 C < 90 Hgh (H) 70 C < 80 Medum (M) 60 C < 70 Low (L) C < 60 Very Low (VL) Equaons (8) and (9) allow Learn++ o deermne s own confdence n any classfcaon makes. The desred oucome of he confdence analyss s o observe a hgh confdence on correcly classfed nsances, and a low confdence on msclassfed nsances, so ha he low confdence can be used o flag hose nsances ha are beng msclassfed by he algorhm. A second desred oucome s o observe mproved confdences on correcly classfed nsances and reduced confdence on msclassfed nsances, as new daa becomes avalable, so ha he ncremenal learnng ably of he algorhm can be furher confrmed. 4 Smulaon Resuls on Learn++ Learn++ has been esed on a dverse se of benchmark daabases acqured from he UCI Machne Learnng Reposory, as well as a few real-world applcaons, boh for ncremenal learnng where new daases ncluded new classes and for esmang he confdence of Learn++ n s own decsons. The ncremenal learnng resuls wh new classes are presened n [1]. In hs paper, we presen he resuls on confdence

6 186 J. Byorck and R. Polkar esmaon, and we use wo daabases, one benchmark daabase from UCI, and one real world on gas sensng, as represenave smulaons. 4.1 Volale Organc Compound (VOC) Daabase The VOC daabase s a real world daabase for he odoran denfcaon problem. The nsances are responses of sx quarz crysal mcrobalances o fve volale organc compounds, ncludng ehanol (ET), ocane (OC), oluene (TL), rchloroehelene (TCE), and xylene (XL), consung a fve class, sx feaure daabase. Three daases, S 2 and, where each daase ncluded approxmaely one hrd of he enre ranng daa, were provded o Learn++ n hree ranng sessons for ncremenal learnng. The daa dsrbuon and he percen classfcaon performance are gven n Table 2. The performances lsed are on he valdaon daa, TEST, followng each ranng sesson. Table 3 provdes an acual breakdown of correcly classfed and msclassfed nsances fallng no each confdence range afer each ranng sesson. The rends of he confdence esmaes afer subsequen ranng sessons are gven n Table 4. The desred oucome on he acual confdences s hgh o very hgh confdences on correcly classfed nsances, and low o very low confdences on msclassfed nsances. The desred oucome on confdence rends s ncreasng or seady confdences on correcly classfed nsances, and decreasng confdences on msclassfed nsances, as new daa s nroduced. Table 2. Daa dsrbuon and performance on VOC daabase Ehanol Ocane Toluene TCE Xylene Tes Perf. (%) S TEST Correcly Classfed Msclassfed Table 3. Confdence resuls on VOC daabase VH H M L VL S S Table 4. Confdence rends for he VOC daabase. Increasng/Seady Decreasng Correcly Classfed Msclassfed 9 16 The performance fgures n Table 2 ndcae ha he algorhm s mprovng s generalzaon performance as new daa becomes avalable. The mprovemen s modes, however, as majory of he new nformaon s already learned n he frs ranng sesson. Oher expermens, where new daa nroduced new classes, showed remarkable performance ncrease as repored n [1]. Table 3 ndcaes ha he vas majory

7 Confdence Esmaon Usng he Incremenal Learnng Algorhm, Learn of correcly classfed nsances end o have very hgh confdences, wh connually mproved confdences a consecuve ranng sessons. Whle a consderable poron of msclassfed nsances also had hgh confdence for hs daabase, he general desred rends of ncreased confdence on correcly classfed nsances and decreasng confdence on msclassfed ones were noable and domnan, as shown n Table Glass Daabase The glass daabase, rereved from UCI reposory, s a 10-feaure, 6-class daabase wh samples of glass from buldngs, vehcles, conaners, ableware, and headlamps. The buldngs nclude wo ypes of glass, whch are floa processed and non-floa processed. Ths daabase was also provded ncremenally n hree ranng sessons, wh each sesson usng one of he daases, ~. The dsrbuon of daa, as well as he performance on he valdaon daase, s shown n Table 5. The confdence resuls are shown n Table 6, whle he confdence rends are provded n Table 7. Table 5. Daa dsrbuon and generalzaon performance on glass daabase Floa Non- Floa Vehcle Conaner Table Lamp Tes Perf. (%) S TEST Table 6. % Confdence Resuls on Glass Daabase VH H M L VL Correcly Classfed S Msclassfed S Table 7. Confdence rends for glass daabase Increasng/Seady Decreasng Correcly classfed 63 1 Msclassfed 3 2 For he glass daabase, he above-menoned desrable ras are even more remarkable. The majory of correcly classfed nsances fell no a very hgh confdence range, whle he msclassfed nsances fell no he very low confdence range. Posve arbues were also seen n he confdence rends where he majory of correcly classfed nsances had an ncreasng or seady confdence hrough consecuve ranng sessons. Furhermore, he ncremenal learnng ably of he algorhm s also demonsraed hrough he mproved generalzaon performance (from 83% o 93%) on he TEST daase wh avalably of addonal daa.

8 188 J. Byorck and R. Polkar 5 Dscusson and Conclusons Apar from he ncremenal learnng ably of Learn++, was found ha he algorhm can also assess he confdence of s own decsons. In general, majory of correcly classfed nsances had very hgh confdence esmaes whle lower confdence values were assocaed wh msclassfed nsances. Therefore, classfcaons wh low confdences can be used as a flag o furher evaluae hose nsances. Furhermore, he algorhm also showed ncreasng confdences n correcly classfed nsances and decreasng confdences n msclassfed nsances afer subsequen ranng sessons. Ths s a very comforng oucome, whch furher ndcaes ha algorhm can ncremenally acqure new and novel nformaon from addonal daa. Work s n progress o furher es he algorhm s capables on a more dverse se of real world and benchmark daabases. Acknowledgemen. Ths maeral s based upon work suppored by he Naonal Scence Foundaon under Gran No References [1] R. Polkar, L. Udpa, S. Udpa, V. Honavar. Learn++: An ncremenal learnng algorhm for supervsed neural neworks. IEEE Tran. on Sysems, Man, and Cybernecs(C)., Vol. 31, no. 4, November [2] P. Wnson, Learnng srucural descrpons from examples, In The Psychology of Compuer Vson P. Wnson (ed.), pp , McGraw-Hll: New York, NY, [3] N. Llesone, Learnng quckly when rrelevan arbues abound, Machne Learnng, vol. 2, pp , [4] P. Janke, Types of ncremenal learnng, Tranng Issues n Incremenal Learnng, A. Cornuejos (ed.) pp.26-32, The AAAI Press: Menlo Park, CA [5] M.A. Maloof and R.S. Mchalsk, Selecng examples for paral memory learnng, Machne Learnng, vol. 41, pp , [6] F.S. Osaro and B. Amy, INSS: A hybrd sysem for consrucve machne learnng, Neurocompung, vol. 28, pp , [7] J. Ghosh and A,c, Nag, Knowledge enhancemen and reuse wh radal bass funcon neworks, Proceedng of Inernaonal Jon Conference on Neural Neworks, vol. 9, no. 2, pp , [8] N. Llesone and M. Warmuh, "Weghed Majory Algorhm," Informaon and Compuaon, vol. 108, pp , [9] Y. Freund and R. Schapre, A decson-heorec generalzaon of on-lne learnng and an applcaon o boosng, Journal of Compuer and Sysem Scences, [10] R. Polkar, J. Byorck, S. Krause, A. Marno, and M. Moreon, Learn++: A classfer ndependen ncremenal learnng algorhm, Proceedngs of Inernaonal Jon Conference on Neural Neworks, May [11] C. L. Blake and C. J. Merz. (1998) UCI reposory of machne learnng daabases. Dep. Inform. and Compu. Sc., Unversy of Calforna, Irvne. [Onlne]. Avalable: hp:// [12] D. McIver and M. Fredl, Esmang Pxel-Scale Land Cover Classfcaon Confdence Usng Nonparamerc Machne Learnng Mehods, IEEE Transacons on Geoscence and Remoe Sensng, Vol. 39, No. 9, Sepember 2001.