Boosting MIT Course Notes Cynthia Rudin

Transcription

1 Credi: Freund, Schapire, Daubechies Boosing MIT Course Noes Cynhia Rudin Boosing sared wih a quesion of Michael Kearns, abou wheher a weak learning algorih can be ade ino a srong learning algorih. Suppose a learning algorih is only guaraneed, wih high probabiliy, o be slighly ore accurae han rando guessing. Is i possible, despie how weak his algorih is, o use i o creae a classifier whose error rae is arbirarily close o 0? (There are soe oher consrains ha ade he proble harder, regarding how uch calculaion is allowed.) The answer o his quesion was given by Rob Schapire, wih an algorih ha could do i, bu wasn very pracical. Rob and Yoav Freund developed soe ore algorihs, e.g., Boos-by-Majoriy, and hen developed AdaBoos. Given:. exaples S = {x i, y i }, where y i {, }. easy access o weak learning algorih A, producing weak classifiers h H, h : X {, }. 3. ɛ > 0. Goal: Produce a new classifier H : X {, } wih error ɛ. Noe: H is no required o be in H. Wha we igh do is ask he weak learning algorih A o produce a collecion of weak classifiers and figure ou how o cobine he. Bu running A wih he sae inpu uliple ies won be useful, for insance, if A is deerinisic, i will always produce he sae weak classifiers over and over again. So we need o odify A s inpu o give new inforaion each ie we ask for a weak classifier. AdaBoos does his by producing a discree probabiliy disribuion over he exaples, using ha as inpu o he weak learning algorih and changing i a each round.

2 Ouline of a generic boosing algorih: for =...T consruc d, where d is a discree probabiliy disribuion over indices {...}. run A on d, producing h () : X {, }. calculae ɛ = error d (h () ) = P r i d [h () (xi) = y i ] =: γ, where by he weak learning assupion, γ > γ W LA. (Of course, A ries o iniize he error hrough he choice of h ().) end oupu H How do we design he d s? How do we creae H? Le s see wha AdaBoos does. AdaBoos (Freund and Schapire 98) is one of he op 0 algorihs in daa ining, also boosed decision rees raed # in Caruana and Niculescu-Mizil s 006 epirical survey. The idea is ha a each round, we increase he weigh on he exaples ha are harder o classify - hose are he ones ha have been previously isclassified a previous ieraes. So he weighs of an exaple go up and down, depending on how easy he exaple was o classify. The easy exaples can evenually ge iny weighs and he hard exaples ge all he weigh. In our noaion, d,i is he weigh of he probabiliy disribuion on exaple i. d is called he weigh vecor. d,i = for all i { d,i e α if y i = h () (x i ) (saller weighs for easy exaples) d +,i = Z e α if y i = h () (x i ) (larger weighs for hard exaples) where Z is a noralizaion consan for he discree disribuion ha ensures d +,i = d,i = e y iα h () (x i ) Z i

3 Tha s how AdaBoos s weighs are defined. I s an exponenial weighing schee, where he easy exaples are down-weighed and he hard exaples are upweighed. H is a linear cobinaion of weak classifiers: ( ) T H(x) = sign α h () (x). = I s a weighed voe, where α is he coefficien assigned o h (). Here, α is direcly relaed o how well he weak classifier h () perfored on he weighed raining se: ( ) ɛ α = ln. () ɛ where ɛ = P i d [h () (x i ) = y i ] = d,i [h() (x i )=y i ] () AdaBoos sands for Adapive Boosing. I doesn depend on he weak learning algorih s assupion, γ W LA, i adaps o he weak learning algorih. Deo Try o reeber hese hings abou he noaion: d are he weighs on he exaples, and α are he coefficiens for he linear cobinaion ha is used o ake predicions. These in soe sense are boh weighs, so i s kind of easy o ge he confused. i Saisical View of AdaBoos I going o give he saisical view of AdaBoos, which is ha i s a coordinae descen algorih. Coordinae descen is jus like gradien descen, excep ha you can ove along he gradien, you have o choose jus one coordinae a a ie o ove along. 3

4 .5 5 x -6 x y+5 y = 0 y x AdaBoos was designed by Freund and Schapire, bu hey weren he ones who cae up wih he saisical view of boosing, i was 5 differen groups siulaneously (Breian, 997; Friedan e al., 000; Räsch e al., 00; Duffy and Helbold, 999; Mason e al., 000). Sar again wih {(x i, yi)} and weak learning algorih A ha can produce weak classifiers {h j } n j=, h j : X {, }. Here n can be very large or even infinie. Or, h j could be very siple and produce he j h coordinae when x i is (j) a binary vecor, so h j (x i ) = x i. Consider he isclassificaion error: Miscl. error = [yi f(x i ) 0] which is upper bounded by he exponenial loss: e yi f(x i ). Choose f o be soe linear cobinaion of weak classifiers, Iage by MIT OpenCourseWare. f(x) = n j= λ j h j (x). We re going o end up iniizing he exponenial loss wih respec o he λ j s. 4

5 The noaion is going o ge coplicaed fro here on in, so ry o reeber wha is wha! Define an n arix M so ha M ij = y i h j (x i ). So arix M encodes all of he raining exaples and he whole weak learning algorih. In oher words, M conains all he inpus o AdaBoos. The ij h enry in he arix is whenever weak classifier j correcly classifies exaple i. (Noe: we igh never wrie ou he whole arix M in pracice!) Then Then he exponenial loss is: y i f(x i ) = λ j y i h j (x i ) = λ j M ij = (Mλ) i. j j R rain (λ) = e y if(x i ) = e (Mλ) i. (3) i Le s do coordinae descen on he exp-loss R rain (λ). A each ieraion we ll choose a coordinae of λ, called j, and ove α in he j h direcion. So each weak classifier corresponds o a direcion in he space, and α corresponds o a disance along ha direcion. To do coordinae descen, we need o find he direcion j in which he direcional derivaive is he seepes. Le e j be a vecor ha is in he j h enry and 0 elsewhere. Choose direcion: i 5

6 [ ] R rain (λ + αe j ) j argax j α [ [ α=0 λ = argaxj e (M( α [ = argax e (Mλ ) i α j α [ = argax e (Mλ ) i j α [ = argax ] M e (Mλ ) i j ij. +αe j )) i [ α=0 (Me j ) i ] ] ] ] ] ] [ α=0 αm ij Creae a discree probabiliy disribuion (we ll see laer i s he sae as AdaBoos s): α=0 d,i = e (Mλ ) i /Z where Z = e Mλ ) i (4) Muliplying by Z doesn affec he argax. Sae wih. So fro above we have: j argax j Mij d,i = argax(d T M) j. So ha s how we choose weak classifier j. How far should we go along direcion j? Do a linesearch, se derivaive o 0. R rain (λ + αe j 0 = ) α α = M e (Mλ ) i α M ij ij = e (Mλ ) i e α i:mij = i:m ij = ( e (Mλ ) i e α. 6

7 Ge rid of he / and uliply by /Z : 0 = d e α α d,i e,i i:m ij = i:m ij = =: d e α + de α de α = d + e α e α = d + d α = d + ln = ln d. d d So he coordinae descen algorih is: d,i = / for i =... λ = 0 loop =...T T j argax j(d M) j d = M ij = d,i ( ) α = ln d d λ + = λ + α ej d = e (Mλ +) i / Z for each i, where Z = e (Mλ +) i +,i + + end So ha algorih ieraively iniizes he exp-loss. Bu how is i AdaBoos? Sar wih f(x). Fro coordinae descen, we noice ha λ,j is jus he su of he α s where our chosen direcion j is j. λ,j = In oher words, i s he oal aoun T = α [j =j] we ve raveled along direcion j. Thus n f(x) = n T T n T λ,j h j (x) = α [j =j]h j (x) = α hj (x) [j =j] = α h j (x). j= j= = = j= = Look failiar? There is a sligh difference in noaion beween his and AdaBoos, bu ha s i. (You can jus se AdaBoos s h () o coord descen s h j.) 7

8 Le s look a d. AdaBoos has: d e M ij,ie M ij α α e M ijα d j +,i = = = = Z Z Z e M ijλ,j Z This eans he denoinaor us be i e j M ijλ j because we know he d + vecor is noralized. So he d s for AdaBoos are he sae as for coordinae descen (4) as long as j s and α s are he sae. Le s ake sure AdaBoos chooses he sae direcions j as he coordinae descen algorih. AdaBoos s weak learning algorih hopefully chooses he weak classifier ha has he lowes error. This cobined wih he definiion for he error () eans: j argin j d,i [hj (x i )=y i ] = argin j d,i = argax j i i:mij= d,i = argax j d,i + i:m ij= i:mij= = argax j d,i i:m ij= d,i i:mij= i:mij= d,i = argax j d,i T d,i = argaxj(d M) j. i:m ij= i:m ij= Does ha also look failiar? So AdaBoos chooses he sae direcions as coordinae descen. Bu does i go he sae disance? Look a α. Sar again wih he error rae ɛ : ɛ = d,i [hj (x i )=y i ] = d,i = d,i = d Saring fro AdaBoos, i i:h j (x i )=y i i:m ij = d α = ln ɛ = ln, ɛ d as in coordinae descen. So AdaBoos iniizes he exponenial loss by coordinae descen. 8

9 Probabilisic Inerpreaion This is no quie he sae as logisic regression. Since AdaBoos approxiaely iniizes he expeced exponenial loss over he whole disribuion D, we can ge a relaionship beween i s oupu and he condiional probabiliies. As in logisic regression, assue ha here s a disribuion on y for each x. Lea (Hasie, Friedan, Tibshirani, 00) is iniized a: Proof. Y f(x) E Y D(x) e P (Y = x) f(x) = ln. P (Y = x) 0 = de(e Y f(x) x) df(x) = P (Y = x)e f(x) + P (Y = x)e f(x) Ee Y f(x) = P (Y = x)e f(x) + P (Y = x)e f(x) P (Y = x)e f(x) = P (Y = x)e f(x) P (Y = x) ( P = = e f x) (Y x) f(x) = ln. P (Y = x) P (Y = x) In logisic regression, we had so we re differen by jus a facor of. P (Y = x) f(x) = ln P (Y = x) Fro he Lea, we can ge probabiliies ou of AdaBoos by solving for P (Y = x), which we denoe by p: e f(x) pe f(x) P (Y = x) p f(x) = ln =: ln P (Y = x) p p e f(x) = p = p e f(x) = p( + e f(x) ) 9

10 and finally, e f(x) p = P (Y = x) =. + e f(x) Recall logisic regression had he sae hing, bu wihou he s. So now we can ge probabiliies ou of AdaBoos. This is helpful if you wan a probabiliy of failure, or probabiliy of spa, or probabiliy of discovering oil. Even hough logisic regression iniizes he logisic loss and AdaBoos iniizes he exponenial loss, he forulas o ge probabiliies are soehow very siilar. Training Error Decays Exponenially Fas Theore If he weak learning assupion holds, AdaBoos s isclassificaion error decays exponenially fas: Proof. Sar wih [y i =H(x i )] e γ W LA T. R rain (λ + ) = R rain (λ + α e j ) = e [M(λ +α e j )] i = e (Mλ ) i α M ij α λ = e e (M ) i + e α e (Mλ ) i. (5) i:m ij = i:m ij = To suarize his proof, we will find a recursive relaionship beween R rain (λ + ) and R rain (λ ), so we see how uch he raining error is reduced a each ieraion. Then we ll uncoil he recursion o ge he bound. Le s creae he recursive relaionship. Think abou he disribuion d ha we defined fro coordinae descen: d = e (Mλ ) i,i /Z where Z = e (Mλ ) i (6) 0

11 So you should hink of e (Mλ ) i as an unnoralized version of he weigh d,i. So ha eans: Z Z (Mλ d + = d ) i,i = e. i:m ij = i:m ij = and we could do he sae hing o show Z d = e (Mλ ) i i:m ij =. Plugging ha ino (5), R rain (λ ) = e αz Z + d + + e α d. I s kind of nea ha Z / = R rain (λ ), which we can see fro he definiion of Z in (6) and he definiion of R rain in (3). Le s use ha. Reeber: α = ln R rain (λ + ) = R rain (λ)[e d + + e α d ] = R rain (λ )[e α ( d ) + e α d ]. ( ) ( ) / ( ) / d d d, so e α = and e α = d d d Plugging: [ ( ) / ( rain d R (λ ) = R rain d + (λ ) ( d ) + d = R rain (λ ) [d ( d )] / α / = R rain (λ ) [ɛ ( ɛ )]. d / d. ) ] rain Uncoil he recursion, using for he base case λ = 0, so ha R (0) =, hen T R rain (λ T ) = ɛ ( ɛ ). = Fro all he way back a he beginning (on page in he pseudocode) ɛ = / γ. So, T R rain (λ T ) = ( ) ( ) γ + γ = 4 γ = 4γ. =

12 Using he inequaliy + x e x, which is rue for all x, T 4γ e 4γ = e γ = e = T = γ. Now, we ll use he weak learning assupion, which is ha γ > γ W LA for all. We ll also use ha he isclassificaion error is upper bounded by he exponenial loss: T γ [y ) = =H(x )] R rain (λ T e e γ W LA T. i i And ha s our bound. Inerpreing AdaBoos Soe poins:. AdaBoos can be used in wo ways: (j) where he weak classifiers are ruly weak (e.g., h j (x i ) = x i ). where he weak classifiers are srong, and coe fro anoher learning algorih, like a decision ree. The weak learning algorih is hen, e.g., C4.5 or CART, and each possible ree i produces is one of he h j s. You can hink of he decision ree algorih as a kind of oracle where AdaBoos asks i a each round o coe up wih a ree h j wih low error. So he weak learning algorih does he argax j sep in he algorih. In realiy i igh no ruly find he argax, bu i will give a good direcion j (and ha s fine - as long as i chooses a good enough direcion, i ll sill converge ok).. The WLA does no hold in pracice, bu AdaBoos works anyway. (To see why, you can jus hink of he saisical view of boosing.) 3. AdaBoos has an inerpreaion as a -player repeaed gae.

13 weak learning algorih chooses j colun player chooses a pure sraegy d ixed sraegy for row player 4. Neiher logisic regression nor AdaBoos has regularizaion... bu AdaBoos has a endency no o overfi. There is los of work o explain why his is, and i sees o be copleely due o AdaBoos s ieraive procedure - anoher ehod for opiizing he exponenial loss probably wouldn do as well. There is a argin heory for boosing ha explains a lo. 3

14 MIT OpenCourseWare hp://ocw.i.edu Predicion: Machine Learning and Saisics Spring 0 For inforaion abou ciing hese aerials or our Ters of Use, visi: hp://ocw.i.edu/ers.