Machine Learning 4771

Transcription

1 Machne Leanng 4771 Instucto: Tony Jebaa

2 Topc 6 Revew: Suppot Vecto Machnes Pmal & Dual Soluton Non-sepaable SVMs Kenels SVM Demo

3 Revew: SVM Suppot vecto machnes ae (n the smplest case) lnea classfes that do stuctual sk mnmzaton (SRM) Dectly maxmze magn to educe guaanteed sk J(θ) Assume fst the 2-class data s lnealy sepaable: have ( x 1,y 1 ),, x N,y N { } { } whee x D and y 1,1 = sgn w T x +b f x;θ Decson bounday o hypeplane gven by w T x +b = 0 Note: can scale w & b whle keepng same bounday Many solutons exst whch have empcal eo R emp (θ)=0 Want wdest o thckest one (max magn), also t s unque!

4 Suppot Vecto Machnes Defne: w T x +b = 0 H + =postve magn hypeplane H - =negatve magn hypeplane q =dstance fom decson plane to ogn q = mn x x 0 subject to w T x +b = 0 x 0 2 λ w T x +b mn x 1 2 1) gad x 1 = 0 2 xt x λ w T x +b x λw = 0 x = λw 3) Sol n ˆx = ( b )w w T w 4) dstance q = ˆx 0 = b w = w T w 5) Defne wthout loss of genealty snce can scale b & w 2) plug nto constant b w T w w T w = b w H w T x +b = 0 H + w T x +b = +1 H w T x +b = 1 w T x +b = 0 w T ( λw) +b = 0 λ = b w T w

5 Suppot Vecto Machnes d +d H The constants on the SVM fo R emp (θ)=0 ae thus: w T x +b +1 y = +1 w T x +b 1 y = 1 y ( w T x +b) 1 0 O moe smply: The magn of the SVM s: H + H H + w T x +b = +1 H w T x +b = 1 m = d + + d Dstance to ogn: Theefoe: d + = d = 1 w H q = b and magn Want to max magn, o equvalently mnmze: SVM Poblem: mn 1 w 2 subject to y 2 ( w T x +b) 1 0 Ths s a quadatc pogam! Can plug ths nto a matlab functon called qp(), done! w H + q + = b 1 m = w 2 w H q = 1 b w o w 2 w

6 SVM n Dual Fom We can also solve the poblem va convex dualty Pmal SVM poblem L P : Ths s a quadatc pogam, quadatc cost functon wth multple lnea nequaltes (these cave out a convex hull) Subtact fom cost each nequalty tmes an α Lagange multple, take devatves of w & b: Plug back n, dual: Also have constants: mn 1 w 2 subject to y 2 ( w T x +b) L P = mn w,b max w 2 α 0 α 2 ( y ( w T x +b) 1) L = w α w P y x = 0 w = α y x L = α b P y = 0 L D = α 1 α 2 α j y y j x T j x j α y = 0 & α 0 Above L D must be maxmzed! convex dualty also qp()

7 SVM Dual Soluton Popetes We have dual convex pogam: 1 2 α α α j y y j x T x j subject to α y = 0 & α 0,j Solve fo N alphas (one pe data pont) nstead of D w s Stll convex (qp) so unque soluton, gves alphas Alphas can be used to get w: w = α y x Suppot Vectos: have non-zeo alphas shown wth thcke ccles, all lve on the magn: w T x +b = ±1 Soluton s spase, most alphas=0 these ae non-suppot vectos SVM gnoes them f they move (wthout cossng magn) o f they ae deleted, SVM doesn t change (stays obust)

8 SVM Dual Soluton Popetes Pmal & Dual Illustaton: w T x +b ±1 Recall we could get w fom alphas: O could use as s: f x w = α y x +b Kaush-Kuhn-Tucke Condtons (KKT): solve value of b on magn (fo nonzeo alphas) have: w T x +b = y usng known w, compute b fo each suppot vecto then Spasty (few nonzeo alphas) s useful fo seveal easons Means SVM only uses some of tanng data to lean Should help mpove ts ablty to genealze to test data Computatonally faste when usng fnal leaned classfe = sgn ( x T w +b) = sgn ( α y x T x ) b = y w T x : α > 0 b = aveage b

9 Non-Sepaable SVMs What happens when non-sepaable? Thee s no soluton and convex hull shnks to nothng Not all constants can be esolved, the alphas go to Instead of pefectly classfyng each pont: y ( w T x +b) 1 we Relax the poblem wth (postve) slack vaables x s allow data to (sometmes) fall on wong sde, fo example: w T x +b 0.03 f y = +1 w T x +b +1 ξ f y = +1 whee ξ 0 New constants: w T x +b 1 + ξ f y = 1 whee ξ 0 But too much x s means too much slack, so penalze them L P : mn 1 w 2 +C ξ 2 subject to y ( w T x +b) 1 + ξ 0

10 Non-Sepaable SVMs Ths new poblem s stll convex, stll qp()! Use chooses scala C (o coss-valdates) whch contols how much slack x to use (how non-sepaable) and how obust to outles o bad ponts on the wong sde Lage magn Low slack On ght sde L P : mn 1 w 2 +C ξ 2 α ( y ( w T x +b) 1 + ξ ) β ξ L and L asbefoe... L b P w P ξ P = C α β = 0 Can now wte dual poblem (to maxmze): L D : max α 1 2 α = C β but... α & β 0 0 α C Fo x postvty α α j y y j x T x,j j subject to α y = 0 and α 0,C Same dual as befoe but alphas can t gow beyond C

11 Non-Sepaable SVMs As we ty to enfoce a classfcaton fo a data pont ts Lagange multple alpha keeps gowng endlessly Clampng alpha to stop gowng at C makes SVM gve up on those non-sepaable ponts The dual pogam s now: Solve as befoe wth exta constants that alphas postve AND less than C gves alphas fom alphas get w = α y x Kaush-Kuhn-Tucke Condtons (KKT): solve value of b on magn fo not=zeo alphas AND not=c alphas fo all othes have suppot vectos, assume and use fomula y w T x + ξ = 0 ( b ) 1 + ξ = 0 to get b and b = aveage ( b ) Mechancal analogy: suppot vecto foces & toques

12 Nonlnea SVMs What f the poblem s not lnea?

13 Nonlnea SVMs What f the poblem s not lnea? We can use ou old tck Map d-dmensonal x data fom L-space to hgh dmensonal H (Hlbet) featue-space va bass functons Φ(x) Fo example, quadatc classfe: x Φ( x ) va Φ( x ) = Call ph s featue vectos computed fom ognal x nputs Replace all x s n the SVM equatons wth ph s Now solve the followng leanng poblem: L D : max α 1 2 Whch gves a nonlnea classfe n ognal space: f x x vec x x T T φ( x j ) α α j y y j φ x,j s.t. α 0,C, α y = 0 +b = sgn α y φ( x) T φ( x ) L Η

14 Kenels (see One mpotant aspect of SVMs: all math nvolves only the nne poducts between the ph featues! = sgn α y φ( x) T φ( x ) f x +b Replace all nne poducts wth a geneal kenel functon Mece kenel: accepts 2 nputs and outputs a scala va: k ( x, x ) = φ( x),φ( x ) = Example: quadatc polynomal t T φ( x ) fo fnte φ φ x φ( x,t)φ( x,t )dt φ x othewse 2 = x 1 = φ( x) T φ( x ) k x, x 2x 1 x 2 x 2 2 T = x 1 2 x x 1 x 1 x 2 x 2 + x 2 2 x 2 2 = ( x 1 x 1 + x 2 x 2 ) 2

15 Kenels Sometmes, many Φ(x) wll poduce the same k(x,x ) Sometmes k(x,x ) computable but featues huge o nfnte! Example: polynomals If explct polynomal mappng, featue space Φ(x) s huge d-dmensonal data, p-th ode polynomal, mages of sze 16x16 wth p=4 have dm(h)=183mllon but can equvalently just use kenel: = ( x T x ) p = ( x x ) p k x, x φ( x)φ x k x,y dm Η = ( x T y) p = d + p 1 Multnomal Theoem p! 1! 2! 3! ( p 1 2 )! x 1 x x d d x 1 1 x 2 2 x d d ( w x 1 1 x 2 2 x d )( d w x 1 1 x 2 2 x d ) w=weght on tem d Equvalent! p

16 Kenels Replace each x T x j k x,x j, fo example: P-th Ode Polynomal Kenel: RBF Kenel (nfnte!): Sgmod (hypebolc tan) Kenel: Usng kenels we get genealzed nne poduct SVM: L D : max α 1 α 2 α j y y j k x,x,j j s.t. α 0,C, α y = 0 f ( x) = sgn ( α y k ( x,x) +b) Stll qp solve, just use Gam matx K (postve defnte) K = k ( x 1,x 2 ) k ( x 1,x 3 ) k ( x 2,x 2 ) k ( x 2,x 3 ) k ( x 2,x 3 ) k ( x 3,x 3 ) k x 1,x 1 k x 1,x 2 k x 1,x 3 = ( x T x + 1) p = exp 1 2σ x x 2 2 = tanh κx T x δ k x, x k x, x k x, x K j = k x,x j

17 Kenelzed SVMs Polynomal kenel: Radal bass functon kenel: Polynomal Kenel = ( x T x j + 1) p k x,x j = exp 12σ x x 2 j k x,x j RBF kenel 2 Least-squaes, logstc-egesson, pecepton ae also kenelzable

18 SVM Demo SVM Demo by Steve Gunn: In svc.m eplace [alpha lambda how] = qp( ); wth [alpha lambda how] = quadpog(h,c,[],[],a,b,vlb,vub,x0); Ths eplaces the old Matlab command qp (quadatc pogammng) wth the new one fo moe ecent vesons