S 75 Mchne ernng ecture Mos Huskrecht mos@cs.ptt.edu 539 Sennott Squre SVMs for regresson Non-prmetrc/nstnce sed cssfcton method S 75 Mchne ernng Soft-mrgn SVM Aos some fet on crossng the seprtng hperpne S 75 Mchne ernng
Soft-mrgn SVM mnmze / n for for Rerte m n Regurzton pent n / m Hnge oss n / S 75 Mchne ernng Gener form: ssfcton ernng mn D Q oss functon Regurzton pent oss functons: Negtve ogkehood used n the R Hnge oss used n SVM Regurzton terms: sso rdge S 75 Mchne ernng
he decson oundr: he decson: Support vector mchnes ˆ ˆ SV ˆ sgn ˆ SV!!: Decson on ne requres to compute the nner product eteen the empes Smr the optmzton depends on j n n J j j j j S 75 Mchne ernng Nonner cse he ner cse requres to compute he non-ner cse cn e hnded usng set of fetures. Essent e mp nput vectors to rger feture vectors φ It s posse to use SVM formsm on feture vectors Kerne functon φ φ ' ruc de: If e choose the kerne functon se e cn compute ner seprton n the feture spce mpct such tht e keep orkng n the orgn nput spce!!!! K ' φ φ ' S 75 Mchne ernng 3
Kerne functon empe Assume [ nd feture mppng tht mps the nput ] nto qudrtc feture set φ [ ] Kerne functon for the feture spce: K ' φ ' φ ' ' ' ' ' ' ' ' ' he computton of the ner seprton n the hgher dmenson spce s performed mpct n the orgn nput spce S 75 Mchne ernng Nonner etenson Kerne trck Repce the nner product th kerne A e chosen kerne eds to n effcent computton S 75 Mchne ernng 4
Kerne functons ner kerne K ' ' Ponom kerne K ' ' k Rd ss kerne K ' ep ' S 75 Mchne ernng Kernes Kernes defne smrt mesure : defne dstnce n eteen to ojects Desgn crter: e nt kernes to e vd Stsf Mercer condton of postve semdefnteness good emod the true smrt eteen ojects pproprte generze e effcent the computton of K s fese NP-hrd proems ound th grphs S 75 Mchne ernng 5
Kernes Reserch hve proposed kernes for comprson of vret of ojects: Strngs rees Grphs oo thng: SVM gorthm cn e no pped to cssf vret of ojects S 75 Mchne ernng Regresson = fnd functon tht fts the dt. A dt pont m e rong due to the nose Ide: Error from ponts hch re cose shoud count s vd nose ne shoud e nfuenced the re dt not the nose. Support vector mchne for regresson ε ε S 75 Mchne ernng 6
rnng dt: n {... } R R Our go s to fnd functon f tht hs t most ε devton from the ctu otned trget for the trnng dt. f ner mode ε ε S 75 Mchne ernng ner mode ner functon: f We nt functon tht s: ft: mens tht one seeks sm dt ponts re thn ts ε neghorhood he proem cn e formuted s conve optmzton proem: mnmze suject to A dt ponts re ssumed to e n the ε neghorhood S 75 Mchne ernng 7
8 S 75 Mchne ernng f ner mode Re dt: not dt ponts s f nto the ε neghorhood Ide: penze ponts tht f outsde the ε neghorhood ε ε S 75 Mchne ernng f ner mode ner functon: Ide: penze ponts tht f outsde the ε neghorhood suject to mnmze
9 S 75 Mchne ernng ε-ntensve oss functon otherse for ner mode S 75 Mchne ernng grngn tht soves the optmzton proem Optmzton Suject to Prm vres
S 75 Mchne ernng Optmzton Dervtves th respect to prm vres S 75 Mchne ernng Optmzton
S 75 Mchne ernng Optmzton S 75 Mchne ernng ] [ : suject to - j j j - Optmzton Mmze the du Inner product
SVM souton We cn get: Inner product f t the optm souton the grnge mutpers re non-zero on for ponts outsde the ε nd. S 75 Mchne ernng Nonprmetrc vs Prmetrc Methods Nonprmetrc modes: More fet no prmetrc mode s needed But requre storng the entre dtset nd the computton s performed th dt empes. Prmetrc modes: Once ftted on prmeters need to e stored he re much more effcent n terms of computton But the mode needs to e pcked n dvnce S 75 Mchne ernng
Non-prmetrc ssfcton methods Gven dt set th N k dt ponts from css k nd e hve nd correspondng Snce Bes theorem gves S 75 Mchne ernng K-Nerest-Neghours for ssfcton K= 3 S 75 Mchne ernng 3
4 S 75 Mchne ernng Nonprmetrc kerne-sed cssfcton Kerne functon: k Modes smrt eteen Empe: Gussn kerne e used n the kerne denst estmton Kerne for cssfcton ' ' ': ' ' k k k p k / ' ep ' h h k D N k N p