AdaBoost. AdaBoost: Introduction

Transcription

1 Slides modified from: MLSS 03: Guar Räsch, Iroducio o Boosig hp:// : Iroducio 2 Classifiers Supervised Classifiers Liear Classifiers Percepro, Leas Squares Mehods Liear SVM Noliear Classifiers Par I: Muli Layer Neural Neworks Par II: Pol. Class., RBF, Noliear SVM Nomeric Mehods - Decisio Trees Usupervised Classifiers

2 : Ageda 3 Idea (Adapive Boosig, R. Scharpire, Y. Freud, ICML, 996): Combie may low-accuracy classifiers (weak learers) o creae a high-accuracy classifier (srog learers ) : Iroducio 4 2 classes of apples The World: Daa: Ukow arge fucio: Ukow disribuio: Objecive: N ( x, y ), x, y { } y f ( x ) ( or y P ( y x )) x p ( x) Give ew x, predic y d Problem: P(x, y) is ukow! 2

3 : Iroducio 5 The Model: Hypohesis class: Loss: h h : { } d l y, h ( x ) ( e.g. I [ y h( x)] ) Objecive: Miimize he rue (expeced) loss ( geeralizaio error ) * h arg m i L ( h) w ih h L( h) : E l, h( ) Problem: Oly a daa sample is available, P(x, y) is ukow! Soluio: Fid empirical miimizer N ˆ h m i, ( ) N l y h x h N How ca we efficiely cosruc complex hypoheses wih small geeralizaio errors? : Frame work 6 Algorihm Idea: Simple Hypoheses are o perfec! Hypoheses combiaio icreased accuracy Problems: How o geerae differe hypoheses? How o combie hem? Mehod: Compue disribuio d,..., d N o examples Fid hypohesis o he weighed raiig sample (x, y, d ),..., (x N, y N, d N ) Combie hypoheses h, h 2,... liearly: f T h 3

4 : Frame work 7 Ipu: N examples {(x, y ),..., (x N, y N )}, L a learig algorihm geeraig hypohesis h (x) (classifiers) T maxnumber of hypoheses i he esemble ( ) Iiialize: d weigh of example (d is a disribuio wih d ) () Do for =,..., T, d / N for all,, N. Trai base learer accordig o example disribuio d () ad obai hypohesis 2. compue weighed error 3. compue hypohesis weigh 4. updae example disribuio h : x { }. N ( ) I - l 2 d ( y h ( x ) ( ) ( d d ) exp y h ( x ) Z Z is a ormalizaio facor N Oupu: fial hypohesis f ( x) h ( x) Es T : Decisio Sumps 8 A family of weak learers, e.g. Decisio sump: ca perform a sigle es o a sigle aribue wih hreshold Θ. parameerize all decisio sumps as follows: if j x j f ( x; ), j,..., d else 4

5 : Example 9 aural apples vs. plasic apples class B How o classify? class A 0 aural apples vs. plasic apples s hypohesis Weak classifier (cus o coordiae axes) 5

6 Recompuig weighigs of he raiig paers 2 2 d hypohesis 6

7 3 Recompue weighig 4 3 rd hypohesis 7

8 5 Recompue weighig 4 h hypohesis 6 Combiaio of hypoheses 8

9 7 Decisio surface 8 Example Fial decisio fucio 9

10 : Frame work 9 Ipu: N examples {(x, y ),..., (x N, y N )}, L a learig algorihm geeraig hypohesis h (x) (classifiers) T maxnumber of hypoheses i he esemble ( ) Iiialize: d weigh of example (d is a disribuio wih d ) () Do for =,..., T, d / N for all,, N. Trai base learer accordig o example disribuio d () ad obai hypohesis 2. compue weighed error 3. compue hypohesis weigh 4. updae example disribuio h : x { }. N ( ) I - l 2 d ( y h ( x ) ( ) ( d d ) exp y h ( x ) Z Z is a ormalizaio facor N Oupu: fial hypohesis f ( x) h ( x) Es T 20 = d () /0 N 0 N ( ) d I y h - 2 l = ( ( x )) = 0.3 f ( x) h ( x) Es 0

11 2 =2 d ( 2 ) () h x d exp y ( ) Z Z is a ormalizaio facor N ( 2 ) d I y h l ( ( x )) f ( x) h ( x) h ( x) Es =3.

12 : Frame work 23 Weak Learers used wih Boosig Decisio sumps (axis parallel splis) Decisio rees (e.g. C4.5 by Quila 996) Muli-layer Neural eworks (e.g. for OCR) Radial basis fucio eworks (e.g. UCI bechmarks, ec) Decisio rees: Hierarchical ad recursive pariioig o he ipu space May approaches, usually axis parallel splis : vs. SVM 24 Compariso vs. SVM s decisio lie SVM s decisio lie These decisio lies are for a low oise case wih similar geeralizaio errors. I, RBF eworks wih 3 ceers were used. 2

13 : Applicaio 25 Applicaio DT C4.5 as weak classifier Spam, Zip Code OCR Tex classificaio: Schapire ad Siger - Used sumps wih ormalized erm frequecy ad muli-class ecodig OCR: Schwek ad Begio (eural eworks) Naural laguage Processig: Collis; Haruo, Shirai ad Ooyama Image rerieval: Thieu ad Viola Medical diagosis: Merle e al. Fraud Deecio: Räsch & Müller 200 Drug Discovery: Räsch, Demiriz, Bee 2002 Elec. Power Moiorig: Ooda, Räsch & Müller 2000 : Iformaio 26 Iroducio hp://iformaik.uibas.ch/lehre/ws06/cs232/_dowloads/ Schapire_A_Shor_Iroducio_o_Boosig.pdf Iere hp:// hp:// Cofereces Compuaioal Learig Theory (COLT), Neural Iformaio Processig Sysems (NIPS), I. Coferece o Machie Learig (ICML),... Jourals Machie Learig, Joural of Machie Learig Research, Iformaio ad Compuaio, Aals of Saisics People Lis available a hp:// Sofware Oly few implemeaios (algorihms oo simple ) (cf. hp:// 3