CS 75 Mache Learg Lecture Support vector maches II Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Learl separable classes Learl separable classes: here s a hperplae that separates trag staces th o error Normal or drecto of a plae Class + Class -
Learg learl separable sets Fdg eghts for learl separable classes: Lear program LP soluto It fds eghts that satsf the follog costrats: For all, such that For all, such that ogether: Propert: f there s a hperplae separatg the eamples, the lear program fds the soluto Optmal separatg hperplae Problem: multple hperplaes that separate the data ests Whch oe to choose? Mamum marg choce: mamum dstace of d d here s the shortest dstace of a postve eample from the hperplae smlarl for egatve eamples d Note: a marg classfer s a classfer for hch e ca calculate the dstace of each eample from the decso boudar d d d
Mamum marg hperplae For the mamum marg hperplae ol eamples o the marg matter ol these affect the dstaces hese are called support vectors Mamum marg hperplae We at to mamze d We do t b mmzg d L, - varables / L / But e also eed to eforce the costrats o all data staces:, 3
Mamum marg hperplae Soluto: Icorporate costrats to the optmzato Optmzato problem Lagraga Data staces J,, /, - Lagrage multplers Mmze th respect to, prmal varables Mamze th respect to α dual varables What happes to α: f else Actve costrat α > α = Ma marg hperplae soluto Set dervatves to Kuh-ucker codtos J, J,,, No e eed to solve for Lagrage parameters Wolfe dual J, Subect to costrats Quadratc optmzato problem: soluto for all for all, ad mamze 4
Mamum marg soluto he resultg parameter vector ŵ ca be epressed as: s the soluto of the optmzato he parameter s obtaed from Soluto propertes for all pots that are ot o the marg he decso boudar: SV he decso boudar defed b support vectors ol α > α = Support vector maches: soluto propert Decso boudar defed b a set of support vectors SV ad ther alpha values Support vectors = a subset of datapots the trag data that defe the marg SV Classfcato decso for e : sg SV Note that e do ot have to eplctl compute ŵ hs ll be mportat for the olear kerel case Lagrage multplers 5
6 Support vector maches he decso boudar: Classfcato decso: SV sg SV Support vector maches: er product Decso o a e depeds o the er product betee to eamples he decso boudar: Classfcato decso: Smlarl, the optmzato depeds o SV sg SV, J
7 Ier product of to vectors he decso boudar for the SVM ad ts optmzato deped o the er product of to datapots vectors: 6 5 3? Ier product of to vectors he decso boudar for the SVM ad ts optmzato deped o the er product of to data pots vectors: 5 6* 5*3 * 3 * 6 5 6 5 3
Ier product of to vectors he decso boudar for the SVM ad ts optmzato deped o the er product of to data pots vectors: he er product s equal * If the agle betee them s the: * If the agle betee them s 9 the: cos he er product measures ho smlar the to vectors are Eteso to a learl o-separable case Idea: Allo some fleblt o crossg the separatg hperplae 8
Learl o-separable case Rela costrats th varables for Error occurs f, s the upper boud o the umber of errors Itroduce a pealt for the errors soft marg mmze Subect to costrats for / C C set b a user, larger C leads to a larger pealt for a error Learl o-separable case mmze for / C for Rerte ma, Regularzato pealt / C ma, Hge loss / C 9
he parameter s obtaed through KK codtos Learl o-separable case Lagrage multpler form prmal problem Dual form after are epressed s cacel out, J C J /,,, Subect to: C for all, ad Soluto: he dfferece from the separable case: C Support vector maches: soluto he soluto of the learl o-separable case has the same propertes as the learl separable case. he decso boudar s defed ol b a set of support vectors pots that are o the marg or that cross the marg he decso boudar ad the optmzato ca be epressed terms of the er product betee pars of eamples SV, J sg sg SV
Nolear decso boudar So far e have see ho to lear a lear decso boudar But hat f the lear decso boudar s ot good. Ho e ca lear o-lear decso boudares th the SVM? Nolear decso boudar he o-lear case ca be hadled b usg a set of features. Essetall e map put vectors to larger feature vectors φ Note that feature epasos are tpcall hgh dmesoal Eamples: polomal epasos Gve the olear feature mappgs, e ca use the lear SVM o the epaded feature vectors ' φ φ ' Kerel fucto K,' φ φ '
Support vector maches: soluto for olear decso boudares he decso boudar: Classfcato: sg K, SV Decso o a e requres to compute the kerel fucto defg the smlart betee the eamples Smlarl, the optmzato depeds o the kerel J sg K,, K, SV Kerel trck he o-lear case maps put vectors to larger feature space φ Note that feature epasos are tpcall hgh dmesoal Eamples: polomal epasos Kerel fucto defes the er product the epaded hgh dmesoal feature vectors ad let us use the SVM ' K,' φ φ ' Problem: after epaso e eed to perform er products a ver hgh dmesoal space Kerel trck: If e choose the kerel fucto sel e ca compute lear separato the hgh dmesoal feature space mplctl b orkg the orgal put space!!!!
Kerel fucto eample Assume [ ad a feature mappg that maps the put, ] to a quadratc feature set φ [,,,,,] Kerel fucto for the feature space: Kerel fucto eample Assume [ ad a feature mappg that maps the put, ] to a quadratc feature set φ [,,,,,] Kerel fucto for the feature space: K ', φ ' φ ' ' ' ' ' ' ' ' ' he computato of the lear separato the hgher dmesoal space s performed mplctl the orgal put space 3
Kerel fucto eample Lear separator the epaded feature space No-lear separator the put space Nolear eteso Kerel trck Replace the er product th a kerel A ell chose kerel leads to a effcet computato 4
Kerel fuctos Lear kerel K,' ' Polomal kerel K, ' ' k Radal bass kerel K,' ep ' Kerels ML researchers have proposed kerels for comparso of varet of obects Strgs rees Graphs Cool thg: SVM algorthm ca be o appled to classf a varet of obects 5