Department of Computer Science Artificial Intelligence Research Laboratory. Iowa State University MACHINE LEARNING

Transcription

1 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory MACHINE LEARNING Vasant Honavar Artfcal Intellgence Research Laboratory Department of Computer Scence Bonformatcs and Computatonal Bology Program Center for Computatonal Intellgence, Learnng, & Dscovery Iowa State Unversty Copyrght Vasant Honavar, 2006.

2 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The Generalzaton Problem Capacty of the machne ablty to learn any tranng set wthout error related to VC dmenson Excellent memory s not an asset when t comes to learnng from lmted data A machne wth too much capacty s lke a botanst wth a photographc memory who, when presented wth a new tree, concludes that t s not a tree because t has a dfferent number of leaves from anythng she has seen before; a machne wth too lttle capacty s lke the botanst s lazy brother, who declares that f t s green, t s a tree C. Burges Copyrght Vasant Honavar,

3 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory A Lttle Learnng Theory Suppose: We are gven l observatons (,). x y Tran and test ponts drawn randomly (..d) from some unknown probablty dstrbuton D(x,y) The machne learns the mappng x y hypothess h( x, α). A partcular choce of and outputs a α generates traned machne The expectaton of the test error or expected rsk s R( α) = y h(, α) dd(, y) () 2 x x Copyrght Vasant Honavar,

4 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory A Bound on the Generalzaton Performance The emprcal rsk s: l Remp ( α) = y h( x, α). (2) 2l = Choose some δ such that 0 < δ <. Wth probablty δ the followng bound rsk bound of h(x,a) n dstrbuton D holds (Vapnk,995): d(log(2 l/ d) + ) log( δ / 4) R( α) Remp ( α) + (3) l d 0 where s called VC dmenson s a measure of capacty of machne. Copyrght Vasant Honavar,

5 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory A Bound on the Generalzaton Performance The second term n the rght-hand sde s called VC confdence. Three key ponts about the actual rsk bound: It s ndependent of D(x,y) It s usually not possble to compute the left hand sde. If we know d, we can compute the rght hand sde. The rsk bound gves us a way to compare learnng machnes! Copyrght Vasant Honavar,

6 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Copyrght Vasant Honavar,

7 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The VC Dmenson Defnton: the VC dmenson of a set of functons H = {(, h x α)} s d f and only f there exsts a set of ponts { } x d = such that these ponts can be labeled n all 2 d possble confguratons, and for each labelng, a member of set H can be found whch correctly assgns those labels, but that no set satsfyng ths property. { } x q = exsts where q > d Copyrght Vasant Honavar,

8 Iowa State Unversty The VC Dmenson Department of Computer Scence Artfcal Intellgence Research Laboratory VC dmenson of H s sze of largest subset of X shattered by H. VC dmenson measures the capacty of a set H of hypotheses (functons). If for any number N, t s possble to fnd N ponts x,.., xn that can be separated n all the 2 N possble ways, we wll say that the VC-dmenson of the set s nfnte Copyrght Vasant Honavar,

9 Iowa State Unversty The VC Dmenson Example Department of Computer Scence Artfcal Intellgence Research Laboratory Suppose that the data lve n 2 R space, and the set {(, h x α)} conssts of orented straght lnes, (lnear dscrmnants). I It s possble to fnd three ponts that can be shattered by ths set of functons It s not possble to fnd four. 2 So the VC dmenson of the set of lnear dscrmnants n R s three. Copyrght Vasant Honavar,

10 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The VC Dmenson of Hyperplanes Theorem Consder some set of m ponts n n R. Choose any one of the ponts as orgn. Then the m ponts can be shattered by orented hyperplanes f and only f the poston vectors of the remanng ponts are lnearly ndependent. Corollary: The VC dmenson of the set of orented n hyperplanes n R s n+, snce we can always choose n+ ponts, and then choose one of the ponts as orgn, such that the poston vectors of the remanng ponts are lnearly ndependent, but can never choose n+2 ponts Copyrght Vasant Honavar,

11 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The VC Dmenson VC dmenson can be nfnte even when the number of parameters of the set {(, h x α)} of hypothess functons s low. Example: h( x, α) sgn(sn( αx)), x, α R For any nteger l wth any labels y,..., yl, y {,} we can fnd l ponts x,..., xl and parameter a such that those ponts can be shattered by h( x, α) Those ponts are: x = 0, =,..., l. and parameter a s: α l ( y )0 π( ) 2 = + = Copyrght Vasant Honavar, 2006.

12 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Vapnk-Chervonenks (VC) Dmenson Let H be a hypothess class over an nstance space X. Both H and X may be nfnte. We need a way to descrbe the behavor of H on a fnte set of ponts S X. { X X } S =..., 2 X m For any concept class H over X, and any S X, Π H ( S) = { h S : h H} Equvalently, wth a lttle abuse of notaton, we can wrte H ( S ) = {( h( X )... h( X )) h H} Π : m Π H (S) s the set of all dchotomes or behavors on S that are nduced or realzed by H Copyrght Vasant Honavar,

13 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Vapnk-Chervonenks (VC) Dmenson ( ) { } m If Π S = 0, H where m, or equvalently, we say that S s shattered by H. m S = Π ( ) = H S 2 A set S of nstances s sad to be shattered by a hypothess class H f and only f for every dchotomy of S, there exsts a hypothess n H that s consstent wth the dchotomy. Copyrght Vasant Honavar,

14 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory VC Dmenson of a hypothess class Defnton: The VC-dmenson V(H), of a hypothess class H defned over an nstance space X s the cardnalty d of the largest subset of X that s shattered by H. If arbtrarly large fnte subsets of X can be shattered by H, V(H)= How can we show that V(H) s at least d? Fnd a set of cardnalty at least d that s shattered by H. How can we show that V(H) = d? Show that V(H) s at least d and no set of cardnalty (d+) can be shattered by H. Copyrght Vasant Honavar,

15 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory VC Dmenson of a Hypothess Class - Examples Example: Let the nstance space X be the 2-dmensonal Eucldan space. Let the hypothess space H be the set of axs parallel rectangles n the plane. V(H)=4 (there exsts a set of 4 ponts that can be shattered by a set of axs parallel rectangles but no set of 5 ponts can be shattered by H). Copyrght Vasant Honavar,

16 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Copyrght Vasant Honavar, Some Useful Propertes of VC Dmenson ( H H ) V ( H ) V ( H ) If ( H = { X h : h H} ) V ( H ) = V ( H ) ( H = H H ) V ( H ) V ( H ) + V ( H ) If H If VH H s a fnte concept class, V( H ) lg H l concepts from H, V Π Φ s formed by a unon or ntersecton of l d = d, Π H 2 H 2 ( m) Φ ( m) = max d ( m) ( H ) Proof: Left as an exercse l where 2 = O( V ( H ) l lgl) { Π ( S) : S = m}, H m d ( m) = 2 f m d and Φ ( m) = O( m ) f m < d d 2 + 6

17 Iowa State Unversty The VC Dmenson Department of Computer Scence Artfcal Intellgence Research Laboratory Defnton: the VC dmenson of a set of functons H = {(, h x α)} s d f and only f there exsts a set of ponts { x } d = such that these ponts can be labeled n all 2d possble confguratons, and for each labelng, a member of set H can be found whch correctly assgns those labels, but that no set x exsts where q > d satsfyng ths property. { } d = Copyrght Vasant Honavar,

18 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Rsk Bound What s VC dmenson and emprcal rsk of the nearest neghbor classfer? Any number of ponts, labeled arbtrarly, can be shattered by a nearest-neghbor classfer, thus d = and emprcal rsk =0. So the bound provde no useful nformaton n ths case. Copyrght Vasant Honavar,

19 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The Generalzaton Problem Capacty of the machne ablty to learn any tranng set wthout error related to VC dmenson Excellent memory s not an asset when t comes to learnng from lmted data A machne wth too much capacty s lke a botanst wth a photographc memory who, when presented wth a new tree, concludes that t s not a tree because t has a dfferent number of leaves from anythng she has seen before; a machne wth too lttle capacty s lke the botanst s lazy brother, who declares that f t s green, t s a tree C. Burges Copyrght Vasant Honavar,

20 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmzng the Bound by Mnmzng d d(log(2 l/ d) + ) log( δ / 4) R( α) Remp ( α) + (3) l VC confdence (second term n (3)) dependence on d/l gven 95% confdence level ( δ = 0.05 ) and assumng tranng sample of sze One should choose that learnng machne whose set of functons has mnmal d For d/l>0.37 (for δ = 0.05 and l=0000) VC confdence >. Thus for hgher d/l the bound s not tght. Copyrght Vasant Honavar,

21 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Bounds on Error of Classfcaton Vapnk proved that the error ε of of classfcaton functon h for separable data sets s d ε = O l where d s the VC dmenson of the hypothess class and l s the number of tranng examples Ths means the error depends on the VC dmenson of the hypothess class beng searched, and Copyrght Vasant Honavar,

22 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Structural Rsk Mnmzaton Fndng a learnng machne wth the mnmum upper bound on the actual rsk leads us to a method of choosng an optmal machne for a gven task. Ths s the essental dea of the structural rsk mnmzaton (SRM). Let H H2 H3... be a sequence of nested subsets of hypotheses whose VC dmensons satsfy d < d 2 < d 3 < SRM then conssts of fndng that subset of functons whch mnmzes the upper bound on the actual rsk. Ths can be done by tranng a seres of machnes, one for each subset, where for a gven subset the goal of tranng s to mnmze the emprcal rsk. One then takes that traned machne n the seres whose sum of emprcal rsk and VC confdence s mnmal. Copyrght Vasant Honavar,

23 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Bounds on Error of Classfcaton Margn based bound ε L = O l γ 2 L = max p X p γ = mn yf ( x) w f ( x) = w, x + b Important nsght Error of the classfer traned on a separable data set s nversely proportonal to ts margn, and s ndependent of the dmensonalty of the nput space! Copyrght Vasant Honavar,

24 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmal Margn Classfer The bounds on error of classfcaton suggest the possblty of mprovng generalzaton by maxmzng the margn Mnmze the rsk of overfttng by choosng the maxmal margn hyperplane n feature space SVMs control capacty by ncreasng the margn, not by reducng the number of features Copyrght Vasant Honavar,

25 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Improvng generalzaton requres controllng capacty The emprcal rsk s: l Remp ( α) = y h( x, α). (2) 2l = Choose some δ such that 0 < δ <. Wth probablty δ the followng bound rsk bound of h(x,a) n dstrbuton D holds (Vapnk,995): d(log(2 l/ d) + ) log( δ / 4) R( α) Remp ( α) + (3) l d 0 where s called VC dmenson s a measure of capacty of machne. Copyrght Vasant Honavar,

26 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmzng the Rsk Bound by Mnmzng d d / l Copyrght Vasant Honavar,

27 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmzng the Bound by Mnmzng d VC confdence (second term n (3)) dependence on d/l gven 95% confdence level ( δ = 0.05 ) and assumng tranng sample of sze One should choose that learnng machne whose set of functons has mnmal d For d/l>0.37 (for δ = 0.05 and l=0000) VC confdence >. Thus for hgher d/l the bound s not tght. Copyrght Vasant Honavar,

28 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Margn Lnear separaton of the nput space w f ( x) = w, x + b hx ( ) = sgn( f( x)) b / w Copyrght Vasant Honavar,

29 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Functonal and Geometrc Margn The functonal margn of a lnear dscrmnant (w,b) w.r.t. a labeled pattern d ( x, y ) R {,} s defned as If the functonal margn s negatve, then the pattern s ncorrectly classfed, f t s postve then the classfer predcts the correct label. The larger γ the further away x s from the dscrmnant. Ths s made more precse n the noton of the geometrc margn γ whch measures the γ Eucldean dstance of a pont w from the decson boundary. Copyrght Vasant Honavar, γ y ( wx, + b) 29

30 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Geometrc Margn X S + γ j γ X j S The geometrc margn of two ponts Copyrght Vasant Honavar,

31 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Geometrc Margn γ j X j S γ (,) X S + W X Y γ γ b = ( = Example ) = = ( ) = = = ( Y )() ( x + (). x2 ) ()( + ) = γ W = 2 Copyrght Vasant Honavar,

32 Iowa State Unversty Margn of a tranng set Department of Computer Scence Artfcal Intellgence Research Laboratory γ The functonal margn of a tranng set γ = mn γ The geometrc margn of a tranng set γ = mn γ Copyrght Vasant Honavar,

33 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmum Margn Separatng Hyperplane γ mn γ s called the (functonal) margn of (w,b) w.r.t. the data set S={(x,y )}. The margn of a tranng set S s the maxmum geometrc margn over all hyperplanes. A hyperplane realzng ths maxmum s a maxmal margn hyperplane. Copyrght Vasant Honavar, Maxmal Margn Hyperplane 33

34 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmal Margn Classfer The bounds on error of classfcaton suggest the possblty of mprovng generalzaton by maxmzng the margn Mnmze the rsk of overfttng by choosng the maxmal margn hyperplane n feature space SVMs control capacty by ncreasng the margn, not by reducng the number of features Copyrght Vasant Honavar,

35 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmzng Margn Mnmzng W Defnton of hyperplane (w,b) does not change f we rescale t to (σw, σb), for σ>0. Functonal margn depends on scalng, but geometrc margn γ does not. If we fx (by rescalng) the functonal margn to, the geometrc margn wll be equal / w. Then, we canmaxmze the margn by mnmzng the norm w. Copyrght Vasant Honavar,

36 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmzng Margn Mnmzng W Dstance between the two convex hulls + wx, + b=+ o x x γ o o o o x x o x wx, + b= + w,( x x ) = 2 w + 2,( x x ) = w w Copyrght Vasant Honavar,

37 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Learnng as optmzaton Mnmze ww, (2) subject to: y ( w, x + b) (3) Copyrght Vasant Honavar,

38 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Consder f ( x) ( x) Dgresson Mnmzng / Maxmzng Functons s convex over somesub - doman D D the chord f the graph of f, a functon of a scalar varable x wth doman D jonng the ponts ( x) f ( X ) and f ( X ) 2 x f X, X les above 2 x D, f X ( x) a has a local mnmumat x = X such that x U, f ( x) > f ( X ) a a f neghborhoodu D x around We say that lm x a x such that x-a < δ f ( x) = A f, for any ε > 0, δ > 0 such that f ( x) A < ε Copyrght Vasant Honavar,

39 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory We say that f lm Mnmzng/Maxmzng Functons f ( x) ( x) { } { ( )} lm f = lm lm f x ε 0 x a+ ε 0 s contnuous at x = x a a The dervatve of df dx = lm Δx 0 f the functon ( x + Δx) f ( x) ( Δx) f ( x) s defned as df dx x= X 0 = 0 f X 0 s a local maxmum or a local mnmum Copyrght Vasant Honavar,

40 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory ( u + v) d dx d dx u d v dx ( uv) Mnmzng/Maxmzng Functons du dv = + dx dx dv du = u + v dx dx du v u dx = 2 v dv dx Copyrght Vasant Honavar,

41 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory If, ( x) ( x) Taylor Seres Approxmaton of Functons Taylor seres approxmaton of df dx f f f f s dfferentable ( x) 2 n d f d df d f =,... exst at x X 2 = 0 and n dx dx dx dx s contnuous n the neghborhood of x = X ( x) = f ( X ) + ( x X ) ( x X ) 0 ( x) f ( X ) + ( x X ) 0 df dx df dx x= X x= X 0 f.e., ts dervatves 0 0! 0 n df dx n n 0, then x= X 0 0 n Copyrght Vasant Honavar,

42 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Let f f x Chan rule ( X) = f ( x x, x,... x ) 0, s obtaned by treatng all 2 n x j as constant. Chan rule Then k ( ) Let z = φ u...u Let u = f ( x x... x ) 0, z x k = m m n z u u x = k Copyrght Vasant Honavar,

43 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Taylor Seres Approxmaton of Multvarate Functons Let f 0 Then ( X) = f ( x x, x,... x ) dfferentable X f = ( x x, x,... x ) 00, 0 ( X) f ( X ) + ( x x ) 0 0, 20 and n = 0 2 contnuous n0 f x n X= X be 0 at 0 Copyrght Vasant Honavar,

44 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmzng / Maxmzng Multvarate Functons To fnd X * that mnmzes ( X) n the drecton of the negatve gradent of f, we change current guess X f ( X) evaluated at C X C X C X C f η x 0 f, x f... x n X= X C (why?) for small (deally nfntesmally small) Copyrght Vasant Honavar,

45 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmzng / Maxmzng Functons Gradent descent / ascent s guaranteed to fnd the mnmum / maxmum when the functon has a sngle mnmum / maxmum x f (x, x 2 ) X C= (x C, x 2C ) X * x 2 Copyrght Vasant Honavar,

46 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Copyrght Vasant Honavar, Constraned Optmzaton Prmal optmzaton problem Gven functons f, g, = k; h j j=..m; defned on a doman Ω R n, mnmze subject to Shorthand f g h ( w) ( w) ( w) j 0 h = 0 g w Ω =... k, j =... m { { objectvefuncton nequaltyconstrants { equalty constrants ( w) 0 denotes g ( w) 0 =...k ( w) = 0 denotes h ( w) 0 j =...m Feasble regon F = { w Ω : g( w) 0, h( w) = 0} j 46

47 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Optmzaton problems Lnear program objectve functon as well as equalty and nequalty constrants are lnear Quadratc program objectve functon s quadratc, and the equalty and nequalty constrants are lnear Inequalty constrants g (w) 0 can be actve.e. g (w) = 0 or nactve.e. g (w) < 0. Inequalty constrants are often transformed nto equalty constrants usng slack varables g (w) 0 g (w) +ξ = 0 wth ξ 0 We wll be nterested prmarly n convex optmzaton problems Copyrght Vasant Honavar,

48 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Convex optmzaton problem If functon f s convex, any local mnmum of an unconstraned optmzaton problem wth objectve functon * f s also a global mnmum, snce for any * u w f ( w ) f ( u) A set n Ω R s called convex f, wu, Ω and for any θ (0,), the pont ( θ w+ ( θ ) u) Ω A convex optmzaton problem s one n whch the set, the objectve functon and all the constrants are convex * w Ω Copyrght Vasant Honavar,

49 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Lagrangan Theory Gven an optmzaton problem wth an objectve functon f (w) and equalty constrants h j (w) = 0 j =..m, we defne a Lagrangan as m L ( w,β) = f ( w) + ( w) j= β j h j where β j are called the Lagrange multplers. The necessary Condtons for w * to be mnmum of f (w) subject to the Constrants h j (w) = 0 j =..m s gven by L * * * * ( w, β ) L( w, β ) 0, = 0 w = β The condton s suffcent f L(w,β * ) s a convex functon of w Copyrght Vasant Honavar,

50 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Lagrangan theory example Fnd the lengths u, v, w of sdes of the box that has the largest volume for a gven surface area c mnmze c subject to wu + uv + vw = 2 c L = uvw + β wu + uv + vw 2 L L L = 0 = uv + β(u + v); = 0 = vw + β(v + w); w u v βv -uvw ( w u) = 0 and β( u v) 0 u = v = w = c 6 = 0 = wu + β(u + w) ; Copyrght Vasant Honavar,

51 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Lagrangan Optmzaton - Example Copyrght Vasant Honavar, The entropy of a probablty dstrbuton p=(p...p n ) over a fnte set {, 2,...n } s defned as n ( p) = p log p H 2 = The maxmum entropy dstrbuton can be found by mnmzng H(p) subject to the constrants n = p p = 0 5

52 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Copyrght Vasant Honavar, Generalzed Lagrangan Theory Gven an optmzaton problem wth doman Ω R n where f s convex, and g and h j are affne, we can defne the generalzed Lagrangan functon as: L mnmze subject to f g h ( w) ( w) ( w) j 0 = 0 k ( w, α, β) = f ( w) + α g ( w) + β h ( w) = w Ω =... k, j =... m m j= { { nequalty constrants { equalty constrants j objectve functon An affne functon s a lnear functon plus a translaton F(x) s affne f F(x)=G(x)+b where G(x) s a lnear functon of x and b s a constant j 52

53 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Copyrght Vasant Honavar, Generalzed Lagrangan Theory Gven an optmzaton problem wth doman Ω R n mnmze f ( w) w Ω { objectve functon subject to g ( w) 0 =... k, { nequalty constrants h ( w) = 0 j =... m { equalty constrants where f s convex, and g and h j are affne. The necessary and Suffcent condtons for w* to be an optmum are the exstence of α* and β* such that L j * *, * * *, * ( w, α β ) L( w, α β ) w = 0, β = 0, * * * ( w ) = 0; g ( w ) 0 ; α 0; = k * α g... 53

54 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Dual optmzaton problem A prmal problem can be transformed nto a dual by smply settng to zero, the dervatves of the Lagrangan wth respect to the prmal varables and substtutng the results back nto the Lagrangan to remove dependence on the prmal varables, resultng n θ ( α, β ) = nf L ( w, α, β ) w Ω whch contans only dual varables and needs to be maxmzed under smpler constrants The nfmum of a set S of real numbers s denoted by nf(s) and s defned to be the largest real number that s smaller than or equal to every number n S. If no such number exsts then nf(s) =. Copyrght Vasant Honavar,

55 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmum Margn Hyperplane The problem of fndng the mnmal margn hyperplane s a constraned optmzaton problem Use Lagrange theory (extended by Karush, Kuhn, and Tucker KKT) Lagrangan: Lp( w) = w, w α[ y( w, x 2 + b) ] (4) α 0 Copyrght Vasant Honavar,

56 Iowa State Unversty Copyrght Vasant Honavar, From Prmal to Dual w = α yx (7) Department of Computer Scence Artfcal Intellgence Research Laboratory Mnmze L p (w) wth respect to (w,b) requrng that dervatves of L p (w) wth respect to α all vansh, subject to the constrants α 0 Lp Dfferentatng L p (w) : = 0, w (5) Lp = 0 b (6) α y = 0 (8) Substtutng ths equalty constrants back nto L p (w) we obtan a dual problem. 56

57 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Maxmze: subject to The Dual Problem L = α αα y y xx (9) D j j j 2, j Dualty permts the use of kernels! The value of b does not appear n the dual problem and so b needs to be found from prmal constrants α 0 (0) α y = 0 () b = max ( w x ) + mn ( w x ) y = y = 2 Copyrght Vasant Honavar,

58 Iowa State Unversty KKT condtons Department of Computer Scence Artfcal Intellgence Research Laboratory Karush-Kuhn-Tucker Condtons w ν L = w α y x = 0 (2) α [ y ( w,x + b) ] = 0 (6) p ν ν L b = α y = 0 (3) p y ( w, x + b) 0 (4) α 0 (5) Solvng the SVM problem s equvalent to fndng a soluton to the KKT condtons Copyrght Vasant Honavar,

59 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Karush-Kuhn-Tucker Condtons for SVM The KKT condtons state that optmal solutons α ( w, b) must satsfy α [ y ( w,x + b) ] = 0 (6) Only the tranng samples x for whch the functonal margn = can have nonzero α. They are called Support Vectors. The optmal hyperplane can be expressed n the dual representaton n terms of ths subset of tranng samples the support vectors l f ( x, α, b) = yα x x + b = yα x x + b = sv Copyrght Vasant Honavar,

60 60 Copyrght Vasant Honavar, Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory x j SV = = SV x w α γ ( ),,, = + = SV j j j j b y y b f y x x α x α ( ) = = = = = SV SV j j j j SV SV j j j j j j l j b y y y y y, α α α α α α,,, x x x x w w So Because for

61 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Support Vector Machnes Yeld Sparse Solutons γ Support vectors from postve (yellow) class are shown n green and the negatve class are shown n purple Copyrght Vasant Honavar,

62 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Summary of Maxmal margn classfer Good generalzaton when the tranng data s nose free and separable n the kernel-nduced feature space Excessvely large Lagrange multplers often sgnal outlers data ponts that are most dffcult to classfy SVM can be used for dentfyng outlers Focuses on boundary ponts f they happen to be mslabeled (nosy tranng data), the result s a terrble classfer Nosy tranng data often means a non separable tranng set (unless we use hghly nonlnear kernel transformatons whch n turn ncrease the lkelhood of over fttng despte maxmzng the margn) Copyrght Vasant Honavar,

63 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Non-Separable Case In the case of non-separable data n feature space, the objectve functon grows arbtrarly large. Soluton: Relax the constrant that all tranng data be correctly classfed but only when necessary to do so Cortes and Vapnk, 995 ntroduced slack varables n the constrants: ξ, =,.., l ξ = max(0, γ y ( w, x + b)) (7) Copyrght Vasant Honavar,

64 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Non-Separable Case ξ = max(0, γ y ( w, x + b)). Generalzaton error can be shown to be γ ξ j ε l 2 ( R + ξ ) 2 γ 2 (8) ξ Copyrght Vasant Honavar,

65 Iowa State Unversty Non-Separable Case Department of Computer Scence Artfcal Intellgence Research Laboratory For error to occur, the correspondng must exceed (whch was chosen to be unty) So ξ s an upper bound on the number of tranng 2 errors. So the objectve functon s changed from w / 2 2 k to w / 2 + C ξ where C s a parameter. For any postve k the result s a convex problem ( for k =2 or t s also a quadratc programmng problem). ξ γ Copyrght Vasant Honavar,

66 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The Soft-Margn Classfer Prmal problem for k=2 s to mnmze: 2 ww, + C ξ (20) 2 Subject to: y ( w, x + b) ξ (2) Prmal problem for k= s to mnmze: Subject to: 2 ww, + C ξ (22 ) y ( w, x + b) ξ (23) For k = ξ ξ 0 do not appear n the dual problem Copyrght Vasant Honavar,

67 Iowa State Unversty The Soft-Margn Classfers Department of Computer Scence Artfcal Intellgence Research Laboratory Lagrangan for -norm soft margn problem s l l l Lp( w, b, ξ, α, r) = w, w + C ξ α[ y( x w + b) + ξ] rξ (24 ) 2 wth α 0 and r 0 Dfferentatng w.r.t. w, ξ and b l Lp = w αyx = 0 (25) w = Lp = C α r = 0 (26) ξ l Lp = αy = 0 (27) b Copyrght Vasant Honavar,

68 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory The KKT condtons are Soft Margn-Dual Lagrangan Dual problem L = α αα y y x x (28) D j j j 2, j 0 α C (29) α y = 0 (30) α [ y ( w,x + b) + ξ ] = 0 (3) ξ ( α ) 0 C = (32) The slack varables are nonzero only when α = C Copyrght Vasant Honavar,

69 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Implementaton Technques Maxmzng a quadratc functon, subject to a lnear equalty and nequalty constrants W ( α) = α αα yyk( x, x ) 0 α C α y W ( α) α = 0 j j j 2, j = y α y K( x, x ) j j j j Copyrght Vasant Honavar,

70 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory On-lne algorthm for the -norm soft margn (bas b=0) Gven tranng set D α 0 η = K x repeat, x for = to l ( ) α α + η ( y α y K( x, x )) j j j j f α < 0 then α 0 else f α > C then α C end for untl stoppng crteron satsfed return α Copyrght Vasant Honavar,

71 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Implementaton Technques Use QP packages (MINOS, LOQO, quadprog from MATLAB optmzaton toolbox). They are not onlne and requre that the data are held n memory n the form of kernel matrx Stochastc Gradent Ascent. Sequentally update weght at the tme. So t s onlne. Gves excellent approxmaton n most cases ˆ α α + y α jyjk( x, xj) K( x, x) j Copyrght Vasant Honavar,

72 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Chunkng and Decomposton Gven tranng set D α 0 Select an arbtrary workng set ˆD D repeat solve optmzaton problem on ˆD select new workng set from data not satsfyng KKT condtons untl stoppng crteron satsfed return α Copyrght Vasant Honavar,

73 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Sequental Mnmal Optmzaton S.M.O. At each step SMO: optmze two weghts α, α j smultaneously n order not to volate the lnear constrant α y = 0 Optmzaton of the two weghts s performed analytcally Realzes gradent dssent wthout leavng the lnear constrant (J.Platt) Onlne versons exst (L, Long; Gentle) Copyrght Vasant Honavar,

74 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory SVM Implementatons SVMLght one of the frst practcal mplementatons of SVM (Platt) Matlab toolboxes for SVM LIBSVM one of the best SVM mplementatons s by Chh- Chung Chang and Chh-Jen Ln of the Natonal Unversty of Sngapore WLSVM LIBSVM ntegrated wth WEKA machne learnng toolbox Yasser El-Manzalawy of the ISU AI Lab Copyrght Vasant Honavar,

75 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Example Suppose we have 5 D data ponts x =, x 2 =2, x 3 =4, x 4 =5, x 5 =6, wth, 2, 6 as class and 4, 5 as class 2 y =, y 2 =, y 3 =-, y 4 =-, y 5 = We use the polynomal kernel of degree 2 K(x,y) = (xy+) 2 C s set to 00 We frst fnd α (=,, 5) by Copyrght Vasant Honavar,

76 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Example By usng a QP solver, we get α =0, α 2 =2.5, α 3 =0, α 4 =7.333, α 5 =4.833 Note that the constrants are ndeed satsfed The support vectors are {x 2 =2, x 4 =5, x 5 =6} The dscrmnant functon s b s recovered by solvng f(2)= or by f(5)=- or by f(6)=, as x 2, x 4, x 5 le on and all gve b=9 Copyrght Vasant Honavar,

77 Iowa State Unversty Example Department of Computer Scence Artfcal Intellgence Research Laboratory Value of dscrmnant functon class class 2 class Copyrght Vasant Honavar,

78 Iowa State Unversty Copyrght Vasant Honavar, Why does SVM Work? Department of Computer Scence Artfcal Intellgence Research Laboratory The feature space s often very hgh dmensonal. Why don t we have the curse of dmensonalty? A classfer n a hgh-dmensonal space has many parameters and s hard to estmate Vapnk argues that the fundamental problem s not the number of parameters to be estmated. Rather, the problem s the capacty of a classfer Typcally, a classfer wth many parameters s very flexble, but there are also exceptons Let x =0 where ranges from to n. The classfer can classfy all x correctly for all possble combnaton of class labels on x Ths -parameter classfer s very flexble 78

79 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Why does SVM work? Vapnk argues that the capacty of a classfer should not be characterzed by the number of parameters, but by the capacty of a classfer Ths s formalzed by the VC-dmenson of a classfer The mnmzaton of w 2 subject to the condton that the geometrc margn = has the effect of restrctng the VCdmenson of the classfer n the feature space The SVM performs structural rsk mnmzaton: the emprcal rsk (tranng error), plus a term related to the generalzaton ablty of the classfer, s mnmzed SVM loss functon s analogous to rdge regresson. The term ½ w 2 shrnks the parameters towards zero to avod overfttng Copyrght Vasant Honavar,

80 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Choosng the Kernel Functon Probably the most trcky part of usng SVM The kernel functon should maxmze the smlarty among nstances wthn a class whle accentuatng the dfferences between classes A varety of kernels have been proposed (dffuson kernel, Fsher kernel, strng kernel, ) for dfferent types of data - In practce, a low degree polynomal kernel or RBF kernel wth a reasonable wdth s a good ntal try for data that lve n a fxed dmensonal nput space Low order Markov kernels and ts relatves are good ones to consder for structured data strngs, mages, etc. Copyrght Vasant Honavar,

81 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Other Aspects of SVM How to use SVM for mult-class classfcaton? One can change the QP formulaton to become multclass More often, multple bnary classfers are combned One can tran multple one-versus-all classfers, or combne multple parwse classfers ntellgently How to nterpret the SVM dscrmnant functon value as probablty? By performng logstc regresson on the SVM output of a set of data that s not used for tranng Some SVM software (lke lbsvm) have these features bultn Copyrght Vasant Honavar,

82 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Recap: How to Use SVM Prepare the data set Select the kernel functon to use Select the parameter of the kernel functon and the value of C --- You can use the values suggested by the SVM software, or you can set apart a valdaton set to determne the values of the parameter Execute the tranng algorthm and obtan the α Unseen data can be classfed usng the α and the support vectors Copyrght Vasant Honavar,

83 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Strengths and Weaknesses of SVM Strengths Tranng s relatvely easy No local optma It scales relatvely well to hgh dmensonal data Tradeoff between classfer complexty and error can be controlled explctly Non-tradtonal data lke strngs and trees can be used as nput to SVM, nstead of feature vectors Weaknesses Need to choose a good kernel functon. Copyrght Vasant Honavar,

84 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Recent developments Better understandng of the relaton between SVM and regularzed dscrmnatve classfers (logstc regresson) Knowledge-based SVM ncorporatng pror knowledge as constrants n SVM optmzaton (Shavlk et al) A vertable zoo of kernel functons Extensons of SVM to mult-class and structured label classfcaton tasks One class SVM (anomaly detecton) Copyrght Vasant Honavar,

85 Iowa State Unversty Department of Computer Scence Artfcal Intellgence Research Laboratory Representatve Applcatons of SVM Handwrtten letter classfcaton Classfcaton of tssues healthy versus cancerous - based on gene expresson data Text classfcaton Image classfcaton e.g., face recognton Proten functon classfcaton Proten structure classfcaton Proten sub-cellular localzaton Copyrght Vasant Honavar,