Machine Learning. Support Vector Machines. Eric Xing , Fall Lecture 9, October 6, 2015

Transcription

1 Machne Learnng 0-70 Fall 205 Support Vector Machnes Erc Xng Lecture 9 Octoer Readng: Chap. 6&7 C.B ook and lsted papers Erc CMU

2 What s a good Decson Boundar? Consder a nar classfcaton task th = ± laels not 0/ as efore. When the tranng eaples are lnearl separale e can set the paraeters of a lnear classfer so that all the tranng eaples are classfed correctl Man decson oundares! Generatve classfers Logstc regressons Are all decson oundares equall good? Class Class 2 Erc CMU

3 What s a good Decson Boundar? Erc CMU

4 Not All Decson Boundares Are Equal! Wh e a have such oundares? Irregular dstruton Ialanced tranng szes outlners Erc CMU

5 Classfcaton and Margn Paraeterzng decson oundar Let denote a vector orthogonal to the decson oundar and denote a scalar "offset" ter then e can rte the decson oundar as: 0 Class 2 Class d - d + Erc CMU

6 Classfcaton and Margn Paraeterzng decson oundar Let denote a vector orthogonal to the decson oundar and denote a scalar "offset" ter then e can rte the decson oundar as: 0 Margn +/ > +c/ for all n class 2 +/ < c/ for all n class Class Class 2 d - d + Or ore copactl: + / >c/ he argn eteen an to ponts = d + d + = Erc CMU

7 7 Mau Margn Classfcaton he nu perssle argn s: Here s our Mau Margn Classfcaton prole: c 2 * * c c / / s.t 2 a Erc CMU

8 Mau Margn Classfcaton con'd. he optzaton prole: a s.t But note that the agntude of c erel scales and and does not change the classfcaton oundar at all! h? So e nstead ork on ths cleaner prole: a s.t c he soluton to ths leads to the faous Support Vector Machnes - -- eleved an to e the est "off-the-shelf" supervsed learnng algorth c Erc CMU

9 Support vector achne A conve quadratc prograng prole th lnear constrans: a s.t he attaned argn s no gven Onl a fe of the classfcaton constrants are relevant support vectors d + d - Constraned optzaton We can drectl solve ths usng coercal quadratc prograng QP code But e ant to take a ore careful nvestgaton of Lagrange dualt and the soluton of the aove n ts dual for. deeper nsght: support vectors kernels ore effcent algorth Erc CMU

10 0 Dgresson to Lagrangan Dualt he Pral Prole Pral: he generalzed Lagrangan: the 's 0 and 's are called the Lagarangan ultplers Lea: A re-rtten Pral: l h k g f s.t. n 0 0 l k h g f L o/ constrants pral satsfes f a f L 0 a n L 0 Erc CMU

11 Lagrangan Dualt cont. Recall the Pral Prole: n a 0 L he Dual Prole: heore eak dualt: a 0 n L d * a 0 n L n a 0 L p * heore strong dualt: Iff there est a saddle pont of L e have d * p * Erc CMU

12 A sketch of strong and eak dualt No gnorng h for splct let's look at hat's happenng graphcall n the dualt theores. d * a n f g n a 0 0 f g p * f g Erc CMU

13 A sketch of strong and eak dualt No gnorng h for splct let's look at hat's happenng graphcall n the dualt theores. d * a n f g n a 0 0 f g p * f g Erc CMU

14 A sketch of strong and eak dualt No gnorng h for splct let's look at hat's happenng graphcall n the dualt theores. d * a n f g n a 0 0 f g p * f f g g Erc CMU

15 he KK condtons If there ests soe saddle pont of L then the saddle pont satsfes the follong "Karush-Kuhn-ucker" KK condtons: L L α g g k l Copleentar slackness Pral feaslt Dual feaslt heore: If * * and * satsf the KK condton then t s also a soluton to the pral and the dual proles. Erc CMU

16 6 Solvng optal argn classfer Recall our opt prole: hs s equvalent to Wrte the Lagrangan: Recall that * can e reforulated as No e solve ts dual prole: s.t a s.t n L * a n L 0 n a L 0 Erc CMU

17 7 *** he Dual Prole We nze L th respect to and frst: Note that * ples: Plug *** ack to L and usng ** e have: n a L 0 0 L 0 L * 2 L ** 2 L Erc CMU

18 he Dual prole cont. No e have the follong dual opt prole: hs s agan a quadratc prograng prole. A gloal au of can alas e found. But hat's the g deal?? Note to thngs:. can e recovered 2. he "kernel" a s.t. J 0 2 k 0. See net More later Erc CMU

19 Support vectors Note the KK condton --- onl a fe 's can e nonzero!! α g 0 5 =0 Class 2 8 =0.6 0 =0 7 =0 2 =0 Call the tranng data ponts hose 's are nonzero the support vectors SV 4 =0 9 =0 Class 3 =0 6 =.4 =0.8 Erc CMU

20 Support vector achnes Once e have the Lagrange ultplers { } e can reconstruct the paraeter vector as a eghted conaton of the tranng eaples: SV For testng th a ne data z Copute z SV z and classf z as class f the su s postve and class 2 otherse Note: need not e fored eplctl Erc CMU

21 Interpretaton of support vector achnes he optal s a lnear conaton of a sall nuer of data ponts. hs sparse representaton can e veed as data copresson as n the constructon of knn classfer o copute the eghts { } and to use support vector achnes e need to specf onl the nner products or kernel eteen the eaples We ake decsons coparng each ne eaple z th onl the support vectors: * sgn SV z Erc CMU

22 Non-lnearl Separale Proles Class 2 Class We allo error n classfcaton; t s ased on the output of the dscrnant functon + approates the nuer of sclassfed saples Erc CMU

23 23 Soft Margn Hperplane No e have a slghtl dfferent opt prole: are slack varales n optzaton Note that =0 f there s no error for s an upper ound of the nuer of errors C : tradeoff paraeter eteen error and argn s.t 0 C 2 n Erc CMU

24 24 he Optzaton Prole he dual of ths ne constraned optzaton prole s hs s ver slar to the optzaton prole n the lnear separale case ecept that there s an upper ound C on no Once agan a QP solver can e used to fnd 2 a J 0. 0 s.t. C Erc CMU

25 he SMO algorth Consder solvng the unconstraned opt prole: We ve alread see three opt algorths!??? Coordnate ascend: Erc CMU

26 Coordnate ascend Erc CMU

27 27 Sequental nal optzaton Constraned optzaton: Queston: can e do coordnate along one drecton at a te.e. hold all [-] fed and update? 2 a J 0. 0 s.t. C Erc CMU

28 he SMO algorth Repeat tll convergence. Select soe par and to update net usng a heurstc that tres to pck the to that ll allo us to ake the ggest progress toards the gloal au. 2. Re-optze J th respect to and hle holdng all the other k 's k ; fed. Wll ths procedure converge? Erc CMU

29 29 Convergence of SMO Let s hold 3 fed and reopt J.r.t. and 2 2 a J. 0 s.t. k C 0 KK: Erc CMU

30 Convergence of SMO he constrants: he oectve: Constraned opt: Erc CMU

31 Cross-valdaton error of SVM he leave-one-out cross-valdaton error does not depend on the densonalt of the feature space ut onl on the # of support vectors! Leave - one - out CV error # support vectors # of tranng eaples Erc CMU

32 Suar Ma-argn decson oundar Constraned conve optzaton Dualt he K condtons and the support vectors Non-separale case and slack varales he SMO algorth Erc CMU