Lecture SVMs for regresson Muter neur netors Mos Husrecht mos@cs.ptt.edu 539 Sennott Squre Support vector mchne SVM SVM mmze the mrgn round the seprtng hperpne. he decson functon s fu specfed suset of the trnng dt the support vectors.
he decson oundr: ˆ he decson: Support vector mchnes ˆ α SV ˆ sgn αˆ SV!!: Decson on ne requres to compute the nner product eteen the empes Smr the optmzton depends on n n J α α α α Nonner cse he ner cse requres to compute he non-ner cse cn e hnded usng set of fetures. Essent e mp nput vectors to rger feture vectors φ It s posse to use SVM formsm on feture vectors Kerne functon φ φ ' Cruc de: If e choose the erne functon se e cn compute ner seprton n the feture spce mpct such tht e eep orng n the orgn nput spce!!!! K ' φ φ '
Kerne functon empe Assume [ nd feture mppng tht mps the nput ] nto qudrtc feture set φ [ ] Kerne functon for the feture spce: K ' φ ' φ ' ' ' ' ' ' ' ' ' he computton of the ner seprton n the hgher dmenson spce s performed mpct n the orgn nput spce Nonner etenson Kerne trc Repce the nner product th erne A e chosen erne eds to n effcent computton
Ponom erne Kerne functons Lner erne K ' ' [ ] ' K ' Rd ss erne K ' ep ' Kernes Kernes defne smrt mesure : defne dstnce n eteen to oects Desgn crter: e nt ernes to e vd Stsf Mercer condton of postve semdefnteness good emod the true smrt eteen oects pproprte generze e effcent the computton of K s fese NP-hrd proems ound th grphs
Kernes Reserch hve proposed ernes for comprson of vret of oects: Strngs rees Grphs Coo thng: SVM gorthm cn e no pped to cssf vret of oects Regresson fnd functon tht fts the dt. A dt pont m e rong due to the nose Ide: Error from ponts hch re cose shoud count s vd nose Lne shoud e nfuenced the re dt not the nose. Support vector mchne for regresson
Lner mode rnng dt: n {... } R R Our go s to fnd functon f tht hs t most devton from the ctu otned trget for the trnng dt. f Lner mode Lner functon: f We nt functon tht s: ft: mens tht one sees sm dt ponts re thn ts neghorhood he proem cn e formuted s conve optmzton proem: mnmze suect to A dt ponts re ssumed to e n the neghorhood
f Lner mode Re dt: not dt ponts s f nto the neghorhood Ide: penze ponts tht f outsde the neghorhood f Lner mode Lner functon: Ide: penze ponts tht f outsde the neghorhood suect to mnmze C
-ntensve oss functon ntensve oss functon otherse for Lner mode Lgrngn tht soves the optmzton proem Optmzton C L η η η η Suect to Prm vres
L L C L η Optmzton C L η Dervtves th respect to prm vres C C L η η ω ω Optmzton
C C C C L 443 4 43 4 4 4 443 4 443 ω η η ω η η Optmzton ] [ : suect to - C L L - Optmzton Mmze the du
SVM souton L We cn get: f t the optm souton the Lgrnge mutpers re non-zero on for ponts outsde the nd. Muter neur netors Or nother of modeng nonnertes for regresson nd cssfcton proems
Lner unts Lner regresson f d d f Logstc regresson f p g d z d f p d On-ne grdent updte: α f d he sme On-ne grdent updte: α f α f α f Lmttons of sc ner unts Lner regresson f d Logstc regresson f p g d f z p d d d d Functon ner n nputs!! Lner decson oundr!!
Regresson th the qudrtc mode. Lmtton: ner hper-pne on non-ner surfce cn e etter Cssfcton th the ner mode. Logstc regresson mode defnes ner decson oundr Empe: csses ue nd red ponts Decson oundr.5.5 -.5 - -.5 - - -.5 - -.5.5.5
Lner decson oundr ogstc regresson mode s not optm ut not tht d 5 4 3 - - -3-4 -4-3 - - 3 4 5 6 When ogstc regresson fs? Empe n hch the ogstc regresson mode fs 5 4 3 - - -3-4 -4-3 - - 3 4 5
Lmttons of ner unts. Logstc regresson does not or for prt functons - no ner decson oundr ests.5.5 -.5 - -.5 - - -.5 - -.5.5.5 Souton: mode of non-ner decson oundr f Etensons of smpe ner unts use feture ss functons to mode nonnertes Lner regresson m φ φ φ φ - n rtrr functon of Logstc regresson f g φ m d φ m m
f Lernng th etended ner unts Feture ss functons mode nonnertes Lner regresson m φ φ φ Logstc regresson f m g φ d φ m m Importnt propert: he sme proem s ernng of the eghts for ner unts the nput hs chnged ut the eghts re ner n the ne nput Proem: too mn eghts to ern Mut-ered neur netors An terntve to ntroduce nonnertes to regresson/cssfcton modes Ke de: Cscde sever smpe neur modes th ogstc unts. Much e neuron connectons.
Muter neur netor Aso ced muter perceptron MLP d Cscdes mutpe ogstc regresson unts Empe: er cssfer th non-ner decson oundres z z z p Input er Hdden er Output er Muter neur netor Modes non-nert through ogstc regresson unts Cn e pped to oth regresson nd nr cssfcton proems Input er d Hdden er z z Output er regresson f f z cssfcton f p opton
Muter neur netor Non-nertes re modeed usng mutpe hdden ogstc regresson unts orgnzed n ers he output er determnes hether t s regresson or nr cssfcton proem Input er Hdden ers Output er regresson f f cssfcton d opton f p Lernng th MLP Ho to ern the prmeters of the neur netor? Grdent descent gorthm Weght updtes sed on the error: J D α J D We need to compute grdents for eghts n unts Cn e computed n one crd seep through the net!!! he process s ced c-propgton
Bcpropgton --th eve -th eve -th eve z z z - output of the unt on eve - nput to the sgmod functon on eve z g z - eght eteen unts nd on eves - nd
Bcpropgton δ u u n u f K δ Updte eght usng dt pont D J α D J z δ Let hen: z z D J D J δ S.t. s computed from nd the net er δ δ δ Lst unt s the sme s for the regur ner unts: It s the sme for the cssfcton th the og-ehood mesure of ft nd ner regresson th est-squres error!!! } { > < D Lernng th MLP Grdent descent gorthm Weght updte: D J α z z D J D J δ αδ δ - -th output of the - er - dervtve computed v c-propgton α - ernng rte
Lernng th MLP Onne grdent descent gorthm Weght updte: α J onne D u J onne Du z J onne Du z δ αδ - -th output of the - er δ - dervtve computed v cpropgton α - ernng rte Onne grdent descent gorthm for MLP Onne-grdent-descent D numer of tertons Intze eghts for :: numer of tertons do seect dt pont D u <> from D set ernng rte α compute outputs for ech unt compute dervtves δ v cpropgton updte eghts n pre αδ end for return eghts
Xor Empe. ner decson oundr does not est.5.5 -.5 - -.5 - - -.5 - -.5.5.5 Xor empe. Lner unt
Xor empe. Neur netor th hdden unts Xor empe. Neur netor th hdden unts
MLP n prctce Optc chrcter recognton dgts Automtc sortng of ms 5 er netor th mutpe output functons outputs 9 er Neurons Weghts 5 3 4 3 3 5 784 336 4 nputs 336 784