An Introduction to Support Vector Machines

Transcription

1 An Introducton to Support Vector Mchnes

2 Wht s good Decson Boundry? Consder two-clss, lnerly seprble clssfcton problem Clss How to fnd the lne (or hyperplne n n-dmensons, n>)? Any de? Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

3 Perceptron Perceptrons cn be used to fnd the seprtng hyperplne Feed-forwrd rchtectures, where the neurons re orgnzed nto herrchcl lyers nd the sgnl flows n just drecton. Perceptrons lyers: Input nd Output w j z j g wj x j Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

4 Response of n output neuron Gven step-lke trnsfer functon, the output neuron of perceptron s ctvted f the ctvton s postve: d z 0 w x 0 The nput spce s then dvded nto two regons by hyperplne wth equton w x In vectorl notton: d 0 W X 0 where X s the vector of nput vlues nd W s the vector of weghts connectng the nput neurons wth the output. Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

5 Wht s good Decson Boundry? Consder two-clss, lnerly seprble clssfcton problem Mny decson boundres! The Perceptron lgorthm cn be used to fnd such boundry Are ll decson boundres eqully good? Clss Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

6 Exmples of Bd Decson Boundres Clss Clss Clss Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

7 Lrge-mrgn Decson Boundry The decson boundry should be s fr wy from the dt of both clsses s possble We should mxmze the mrgn, m Clss Clss m Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

8 Vectors..

9 Vectors: nottons x B A A vector n n-dmensonl spce n descrbed by n-uple of rel numbers Vector symbols cn be wrtten: Wth n rrow up to the vector nmes: Wth bold chrcter: As column mtrces. A B A A A B A B x B B A T B T A B A B Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

10 Vectors: sum The components of the sum vector re the sums of the components C A B C C A A B B x C C B A A B A B C x Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

11 Vectors: dfference The components of the sum vector re the sums of the components x C B A C C B B A A B C A A C B A C B x -A Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

12 Vectors: product by sclr The components of the sum vector re the dfference of the components C A C C A A x C 3A A A A C x Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

13 Vectors: Norm The most smple defnton for norm s the euclden module of the components A A. X Y X Y x. 3. X X X 0 se X 0 A A A A A A x Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

14 Vectors: dstnce between two ponts The dstnce between two ponts s the norm of the dfference vector d A, B A B B A x B C A A B C d A, B B A B A A C B x -A Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

15 Vectors: Sclr product c AB A T B A B. X, Y Y, X x. 3. X Y,Z X, Z X,Y X, Y Y, Z nd nd X, Y X,Y Z X, Y X, Y X, Z 4. X, X 0 B A B A θ A B x c A B cos Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

16 Vectors: Sclr product x A Consder reference frme where B s collner to the x xs (you cn ALWAYS fnd t) θ A A B A A A x B B 0 c AB A T B A B A B B A cos Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

17 Vectors: Sclr product A A B 90 A, B 0 90 A, B 0 B A B 90 A, B 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

18 Vectors: Norm nd sclr product The components of the sum vector re the sums of the components A T A A A A, A Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

19 Defnton of hyperplne pssng through the orgn In R, hyperplne s lne A lne pssng through the orgn cn be defned wth s the set of ponts defned by the vectors tht re perpendculr to gven vector W x X W x W XW X W W T X 0 X 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

20 Defnton of hyperplne pssng through the orgn In R 3, hyperplne s plne A plne pssng through the orgn cn be defned wth s the set of the vectors tht re perpendculr to gven vector W x 3 XW W T X 0 W X W X W 3 X 3 0 W x x Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

21 Defnton of hyperplne pssng out the orgn In R, hyperplne s lne Consder vector W: t defnes n nfnty of strght lnes perpendculr to t. A prtculr lne s fxed when the projecton of ponts of the lne on vector W s fxed to vlue p: x X W p x X cos( ) p T XW W X p W W p 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

22 Defnton of hyperplne pssng out the orgn In R, hyperplne s lne Consder vector W: t defnes n nfnty of strght lnes perpendculr to t. A prtculr lne s fxed when the projecton of ponts of the lne on vector W s fxed to vlue p: X x X cos( ) p T XW W X p W W p W x p 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

23 Defnton of hyperplne pssng out the orgn In R, hyperplne s lne Cllng: 0 b X W X W W b W X W W XW T Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn p W X W W XW T W b p

24 Defnton of hyperplne pssng through the orgn In R, hyperplne s lne W X W X b 0 x x X X W -b/ W x W -b/ W x b<0 b>0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

25 Generle defnton of hyperplne In R n, n hyperplne s defned by XW b W T X b 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

26 An hyperplne dvdes the spce x X <BW>/ W W A <AW>/ W -b/ W x AW W T A B b AW b 0 BW W T B b BW b 0 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

27 Dstnce between hyperplne nd pont A x <AW>/ W <BW>/ W X W B -b/ W x d( A, r) d( B, r) AW W BW W b b Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

28 Dstnce between two prllel hyperplnes W T X b 0 x W T X b' 0 -b / W W -b/ W x d ( r, r') b b' W Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

29 Wht s good Decson Boundry? Consder two-clss, lnerly seprble clssfcton problem Mny decson boundres! The Perceptron lgorthm cn be used to fnd such boundry Are ll decson boundres eqully good? Clss Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

30 Lrge-mrgn Decson Boundry The decson boundry should be s fr wy from the dt of both clsses s possble We should mxmze the mrgn, m Clss Mxmze m Mnmze w Clss m Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

31 Hyperplne Clssfers() W W X X b b for for y y - Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

32 Fndng the Decson Boundry Let {x,..., x n } be our dt set nd let y {,-} be y=- y=- y=- y=- Clss the clss lbel of x y= y= y= y= y=- y=- m y= Clss For y = For y =- So: T w x b T w x b T w x b, x y y, Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

33 Fndng the Decson Boundry The decson boundry should correctly clssfy ll ponts The decson boundry cn be found by solvng the followng constrned optmzton problem Ths s constrned optmzton problem. Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

34 Constrned optmzton problems: Lgrnge Multplers

35 Am We wnt to mxmse the functon z = f(x,y) subject to the constrnt g(x,y) = c (curve n the x,y plne) 9/6/05 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn 35

36 Smple soluton Solve the constrnt g(x,y) = c nd express, for exmple, y=h(x) The substtute n functon f nd fnd the mxmum n x of f(x, h(x)) Anlytcl soluton of the constrnt cn be very dffcult Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

37 Contour lnes of functon Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

38 Contour lnes of f nd constrnt Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

39 Lgrnge Multplers Suppose we wlk long the constrnt lne g (x,y)= c. In generl the contour lnes of f re dstnct from the constrnt g (x,y)= c. Whle movng long the constrnt lne g (x,y)= c the vlue of f vry (tht s, dfferent contour levels for f re ntersected). Only when the constrnt lne g (x,y)= c touches the contour lnes of f n tngentl wy, we do not ncrese or decrese the vlue of f: the functon f s t ts locl mx or mn long the constrnt. Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

40 Geometrcl nterpretton Contour lne nd constrnt re tngentl: ther locl perpendculr to re prllel Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

41 Gven curve g(x,y) = c the grdent of g s: Consder ponts of the curve: (x,y); (x+ε x, x+ε y ), for smll ε The locl perpendculr to curve: Grdent y g x g g, (x,y) (x+ε x, y+ε y ) ), ( ), ( ), (,,, y x T y x y y x x y x g y x g y g x g y x g y x g ε Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

42 The locl perpendculr to curve: Grdent Gven curve g(x,y) = c the grdent of g s: (x,y) (x+ε x, x+ε y ) ε grd (g) Snce both ponts stsfy the curve equton: T c ε T g ( x, y) g The grdent s perpendculr to ε. For smll ε, ε s prllel to the curve nd,by consequence, the grdent s perpendculr to the curve c ε 0 ( x, y) Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

43 Norml to curve Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

44 Lgrnge Multplers On the pont of g(x,y)=c tht Mx-mn-mze f(x,y), the grdent of f s perpendculr to the curve g(x,y) =c, otherwse we should ncrese or decrese f by movng loclly on the curve. So, the two grdents re prllel for some sclr λ (where s the grdent). Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

45 Lgrnge Multplers Thus we wnt ponts (x,y) where g(x,y) = c nd, To ncorporte these condtons nto one equton, we ntroduce n uxlry functon (Lgrngn) nd solve. F( x, y, ) f ( x, y) g( x, y) c Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

46 Recp of Constrned Optmzton Suppose we wnt to: mnmze/mxmze f(x) subject to g(x) = 0 A necessry condton for x 0 to be soluton: - : the Lgrnge multpler For multple constrnts g (x) = 0, =,, m, we need Lgrnge multpler for ech of the constrnts - Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

47 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn Constrned Optmzton: nequlty We wnt to mxmze f(x,y) wth nequlty constrnt g(x,y)c. The serch must be confned n the red porton (grdent of functon ponts towrds the drecton long whch t ncreses) g(x,y) c

48 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn Constrned Optmzton: nequlty mxmze f(x,y) wth nequlty constrnt g(x,y)c. If the grdents re opposte (<0) the functon ncreses n the llowed porton The mxmum cnnot be on the curve g(xy)=c (the constrnt do not ct) The mxmum s on the constnt only f >0 g(x,y) c f ncreses, 0 F( x, y, ) f ( x, y) g( x, y) c

49 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn Constrned Optmzton: nequlty Mnmze f(x,y) wth nequlty constrnt g(x,y)c. If the grdents re prllel (>0) the functon decreses n the llowed porton The mnmum cnnot be on the curve g(xy)=c (the constrnt do not ct) The mnmum s on the constrnt only f <0 g(x,y) c f decrese, 0 F( x, y, ) f ( x, y) g( x, y) c

50 Constrned Optmzton: nequlty mxmze f(x,y) wth nequlty constrnt g(x,y) c. If the grdents re prllel (>0) the functon ncreses n the llowed porton The mnmum cnnot be on the curve g(xy)=c (the constrnt do not ct) The mxmum s on the constrnt only f <0 F( x, y, ) f ( x, y) g( x, y) c g(x,y) c 0 f ncreses, Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

51 Constrned Optmzton: nequlty Mnmze f(x,y) wth nequlty constrnt g(x,y) c. If the grdents re opposte (<0) the functon decreses n the llowed porton The mnmum cnnot be on the curve g(xy)=c (the constrnt do not ct) The mnmum s on the constrnt only f >0 F( x, y, ) f ( x, y) g( x, y) c g(x,y) c 0 f decreses, Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

52 Krush-Kuhn-Tucker condtons wth α stsfyng the followng condtons: nd The functon f(x) subject to constrnts g (x) 0 or g (x) 0 s mx-mnmzed by optmzng the Lgrnge functon F( x, ) f ( x) g (x) 0 g (x) 0 MIN α 0 α 0 MAX α 0 α 0 ( x 0 ) 0, g g ( x) Ether the constrnt ct (x 0 s on the curve:g=0) or not (=0) Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

53 Constrned Optmzton: nequlty Krush-Kuhn-Tucker complementrty condton ( x 0 ) 0, mens tht g 0 g ( x ) o 0 The constrnt s ctve only on the border, nd cncel out n the nternl regons Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

54 Dul problem If f(x) s functon Is solved by: From the frst equton we cn fnd x s functon of the These reltons cn be substtuted n the Lgrngn functon obtnng the dul Lgrngn functon L( ) nf L( x, ) nf f ( x) g( x) x x Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

55 Dul problem KKT condtons mposes to serch for 0 L( ) nf L( x, ) nf f ( x) g( x) x x 0 for ech Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

56 Concve-Convex functons Concve Convex Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

57 Dul problem: f convex dul L concve L( ) nf L( x, ) nf f ( x) g( x) x x The dul Lgrngn s concve: mxmsng t wth respect to, wth >0, solve the orgnl constrned mnmzton problem. We compute s: mx L( ) mx nf L( x, ) mx nf f ( x) g ( x) x x Then we cn obtn x by substtutng usng the expresson of x s functon of Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

58 Dul problem:trvl exmple Mnmze the functon f(x)=x wth the constrnt x - (trvl: x=-) The Lgrngn s L( x, ) x ( x ) Mnmsng wth respect to x L 0 x 0 x x The dul Lgrngn s Mxmsng t gves: = Then substutng, L( ) 4 - x 4 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

59 Wht s good Decson Boundry? Consder two-clss, lnerly seprble clssfcton problem Mny decson boundres! The Perceptron lgorthm cn be used to fnd such boundry Are ll decson boundres eqully good? Clss Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

60 Fndng the Decson Boundry Let {x,..., x n } be our dt set nd let y {,-} be y=- y=- y=- y=- Clss the clss lbel of x y= y= y= y= y=- y=- m y= Clss For y = For y =- So: T w x b T w x b T w x b, x y y, Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

61 Fndng the Decson Boundry The decson boundry should clssfy ll ponts correctly The decson boundry cn be found by solvng the followng constrned optmzton problem Ths s constrned optmzton problem. Solvng t requres to use Lgrnge multplers Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

62 Fndng the Decson Boundry The Lgrngn s 0 Note tht w = w T w Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

63 Settng the grdent of w.r.t. w nd b to zero, we hve Grdent wth respect to w nd b 0 0, b L k w L k n m k k k m k k k n T T b x w y w w b x w y w w L n: no of exmples, m: dmenson of the spce Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

64 The Dul Problem If we substtute to, we hve Snce Ths s functon of only Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

65 The Dul Problem The new objectve functon s n terms of only It s known s the dul problem: f we know w, we know ll ; f we know ll, we know w The orgnl problem s known s the prml problem The objectve functon of the dul problem needs to be mxmzed The dul problem s therefore: Propertes of when we ntroduce the Lgrnge multplers The result when we dfferentte the orgnl Lgrngn w.r.t. b Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

66 The Dul Problem Ths s qudrtc progrmmng (QP) problem A globl mxmum of cn lwys be found w cn be recovered by Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

67 Chrcterstcs of the Soluton Mny of the re zero w s lner combnton of smll number of dt ponts Ths sprse representton cn be vewed s dt compresson s n the constructon of knn clssfer x wth non-zero re clled support vectors (SV) The decson boundry s determned only by the SV Let t j (j=,..., s) be the ndces of the s support vectors. We cn wrte Note: w need not be formed explctly Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

68 A Geometrcl Interpretton Clss 8 =0.6 0 =0 5 =0 7 =0 =0 4 =0 9 =0 Clss 3 =0 6 =.4 =0.8 Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

69 Chrcterstcs of the Soluton For testng wth new dt z Compute nd clssfy z s clss f the sum s postve, nd clss otherwse Note: w need not be formed explctly Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

70 The Qudrtc Progrmmng Problem Mny pproches hve been proposed Loqo, cplex, etc. (see Most re nteror-pont methods Strt wth n ntl soluton tht cn volte the constrnts Improve ths soluton by optmzng the objectve functon nd/or reducng the mount of constrnt volton For SVM, sequentl mnml optmzton (SMO) seems to be the most populr A QP wth two vrbles s trvl to solve Ech terton of SMO pcks pr of (, j ) nd solve the QP wth these two vrbles; repet untl convergence In prctce, we cn just regrd the QP solver s blck-box wthout botherng how t works Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

71 Exmple X: (0;0), clss + X: (;), clss - X(+) X(-) X(+) 0 0 X(-) 0 Sclr products 0 ) ( 0 0,, ), ( L L L Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

72 Exmple X: (0;0), clss + X: (;), clss - X(+) X(-) X(+) 0 0 X(-) 0 Sclr products Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn b b b X X hyperplne: W

73 Exmple X: (0;0), clss + X: (;), clss - Sclr products X(+) X(-) X(+) 0 0 X(-) 0 m W W hyperplne Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

74 Exmple X: (0;0), clss + X: (;), clss - X3: (-;0), clss + X(+) X(-) X3(+) X(+) X(-) 0 - X3(+) 0 - Sclr products ), ( 0 0, ),, ( L wth L Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

75 3 Exmple X: (0;0), clss + X: (;), clss - X3: (-;0), clss + L(, ) L L Sclr products X(+) X(-) X3(+) X(+) X(-) 0 - X3(+) 0 - Cnnot be both strctly > 0 Ether α or α 3 s equl to 0 (t lest one constrnt cts) 3 let s try free mxmzton Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

76 Exmple X: (0;0), clss + X: (;), clss - X3: (-;0), clss + Sclr products X(+) X(-) X3(+) X(+) X(-) 0 - X3(+) 0 - L 3, 3 0 If α =0 nd α 3 >0 α <0 If α 3 =0 nd α >0 α >0 NO OK α 3 =0, α =, α = Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

77 Exmple Sclr products X: (0;0), clss + X: (;), clss - X3: (-;0), clss + X(+) X(-) X3(+) X(+) X(-) 0 - X3(+) 0 - Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn b b b X X hyperplne: W

78 Exmple X: (0;0), clss + α =, X: (;), clss - α =, X3: (-;0), clss + α 3 =0 Sclr products X(+) X(-) X3(+) X(+) X(-) 0 - X3(+) 0 - NOT Support W hyperplne Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

79 Exmple 3 X: (0;0), clss + X: (;), clss + X3: (0;), clss - X4: (/;0), clss - Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

80 Non-lnerly Seprble Problems We llow error x n clssfcton; t s bsed on the output of the dscrmnnt functon w T x+b Clss Clss Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

81 Soft Mrgn Hyperplne The new condtons become x re slck vrbles n optmzton Note tht x =0 f there s no error for x x s n upper bound of the llowed errors We wnt to mnmze w C n x C : trdeoff prmeter between error nd mrgn Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

82 The Optmzton Problem n n T n T b x w y C w w L x x x 0 n j j j x y w w L 0 n y x w 0 j j j C L x 0 n y b L Wth α nd μ Lgrnge multplers, POSITIVE Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

83 The Dul Problem n j n n j j y y L j T x x n n n j j j n j n n j j b y y C y y L x x x T j j T x x x x j j C 0 n y Wth Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

84 The Optmzton Problem The dul of ths new constrned optmzton problem s New constrns derve from re postve. w s recovered s C j j snce μ nd α Ths s very smlr to the optmzton problem n the lner seprble cse, except tht there s n upper bound C on now Once gn, QP solver cn be used to fnd Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

85 w C n x The lgorthm try to keep ξ null, mxmsng the mrgn The lgorthm does not mnmse the number of error. Insted, t mnmses the sum of dstnces fron the hyperplne When C ncreses the number of errors tend to lower. At the lmt of C tendng to nfnte, the soluton tends to tht gven by the hrd mrgn formulton, wth 0 errors Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn

86 Soft mrgn s more robust Per Lug Mrtell - Systems nd In Slco Bology Unversty of Bologn