CS 7 Fouatos of Mache Lear Lecture 4 Lear reresso cot Lostc reresso Mlos Hausrecht mlos@cs.ptt.eu 539 Seott Square Lear reresso Vector efto of the moel Iclue bas costat the put vector f - parameters ehts Iput vector f
Lear reresso. Error. Data: D Fucto: f We oul le to have f for all.. Error fucto measures ho much our prectos evate from the esre asers Mea-square error.. Lear: We at to f the ehts mm the error! f Lear reresso. Eample mesoal put 3 5 5 5-5 - -5 -.5 - -.5.5.5
Lear reresso. Eample. mesoal put 5 5-5 - -5 - -3 - - 3-4 - 4 Lear reresso. Optmato. We at the ehts mm the error f.. For the optmal set of parameters ervatves of the error th respect to each parameter must be Vector of ervatves: ra.. 3
4 Solv lear reresso B rearra the terms e et a sstem of lear equatos th + uos A b Solv lear reresso he optmal set of ehts satsfes: Leas to a sstem of lear equatos SLE th + uos of the form Soluto to SLE:? A b
5 Solv lear reresso he optmal set of ehts satsfes: Leas to a sstem of lear equatos SLE th + uos of the form Soluto to SLE: matr verso A b b A Graet escet soluto Goal: the eht optmato the lear reresso moel A alteratve to SLE soluto: Graet escet Iea: Aust ehts the recto that mproves the Error he raet tells us hat s the rht recto - a lear rate scales the raet chaes Error.. f Error
Graet escet metho Desce us the raet formato Error Error * * Drecto of the escet Chae the value of accor to the raet Error Graet escet metho Error Error * * Ne value of the parameter Error * * For all - a lear rate scales the raet chaes 6
Graet escet metho Iteratvel approaches the optmum of the Error fucto Error 3 Batch vs Ole reresso alorthm he error fucto efe o the complete ataset D Error f.. We sa e are lear the moel the batch moe: All eamples are avalable at the tme of lear Wehts are optmes th respect to all tra eamples A alteratve s to lear the moel the ole moe Eamples are arrv sequetall Moel ehts are upate after ever eample If eee eamples see ca be forotte - 7
Ole raet alorthm he error fucto s efe for the complete ataset D Error f Error for oe eample.. ole Error f Ole raet metho: chaes ehts after ever eample Error vector form: D Error - Lear rate that epes o the umber of upates Ole raet metho Lear moel f O-le error ole Error f O-le alorthm: eerates a sequece of ole upates -th upate step th : -th eht: D Error f Fe lear rate: - Use a small costat C Aeale lear rate: - Grauall rescales chaes 8
Ole reresso alorthm Ole-lear-reresso stopp_crtero Itale ehts tale =; hle stopp_crtero = FALSE select the et ata pot D set lear rate upate eht vector e retur ehts f Avataes: ver eas to mplemet cotuous ata streams O-le lear. Eample 4.5 4.5 4 4 3.5 3.5 3 3.5.5.5.5-3 - - 3-3 - - 3 5.5 5 4.5 5.5 3 5 4 4.5 4 4 3.5 3.5 3 3.5.5.5.5.5-3 - - 3.5-3 - - 3 9
Aaptve moels Lear moel f O-le error ole Error f O-le alorthm: Sequece of ole upates oe eample at the tme Useful for cotuous ata streams Aaptve moels: the uerl moel s ot statoar a ca chae over tme Eample: seasoal chaes O-le alorthm ca be mae aaptve b eep the lear at some costat value c Etesos of smple lear moel Replace puts to lear uts th feature bass fuctos to moel oleartes f m - a arbtrar fucto of f m m he same techques as before to lear the ehts!!!!
Etesos of the lear moel Moels lear the parameters e at to ft f Bass fuctos eamples: a hher orer polomal oe-mesoal put 3 3 Multmesoal quaratc 3 4 Other tpes of bass fuctos s cos m... m - parameters... - feature or bass fuctos m 5 Eample. Reresso th polomals. Reresso th polomals of eree m Data pots: pars of Feature fuctos: m feature fuctos m Fucto to lear: m f m m m m
Multmesoal moel eample 5 5-5 - -5 - -3 - - 3-4 - 4 Multmesoal moel eample
Reulare lear reresso If the umber of parameters s lare relatve to the umber of ata pots use to tra the moel e face the threat of overft eeralato error of the moel oes up he precto accurac ca be ofte mprove b sett some coeffcets to ero Icreases the bas reuces the varace of estmates Solutos: Subset selecto Re reresso Lasso reresso Prcpal compoet reresso Net: re reresso Re reresso Error fucto for the staar least squares estmates:.. * We see: ar m Re reresso: Where.... a What oes the e error fucto o? 3
Re reresso Staar reresso: Re reresso:.. L.. L peales o-ero ehts th the cost proportoal to a shrae coeffcet If a put attrbute has a small effect o mprov the error fucto t s shut o b the pealt term Icluso of a shrae pealt s ofte referre to as reularato. re reresso s relate to hoov reularato Reulare lear reresso Ho to solve the least squares problem f the error fucto s erche b the reularato term? Aser: he soluto to the optmal set of ehts s obtae aa b solv a set of lear equato. Staar lear reresso: Soluto: * X X X Reulare lear reresso: here X s a matr th ros correspo to eamples a colums to puts * I X X X 4
Lasso reresso Staar reresso:.. Lasso reresso/reularato: L.. L peales o-ero ehts th the cost proportoal to. L s more aressve push the ehts to compare to L. Data: D {.. } Classfcato represets a screte class value Goal: lear f : X Y Bar classfcato A specal case he Y {} Frst step: e ee to evse a moel of the fucto f 5
Dscrmat fuctos A commo a to represet a classfer s b us Dscrmat fuctos Wors for both the bar a mult-a classfcato Iea: For ever class efe a fucto mapp X Whe the ecso o put shoul be mae choose the class th the hhest value of * ar ma Dscrmat fuctos A commo a to represet a classfer s b us Dscrmat fuctos Wors for both the bar a mult-a classfcato Iea: For ever class efe a fucto mapp X Whe the ecso o put shoul be mae choose the class th the hhest value of * ar ma So hat happes th the put space? Assume a bar case. 6
Dscrmat fuctos.5.5 -.5 - -.5 - - -.5 - -.5.5.5 Dscrmat fuctos.5.5 -.5 - -.5 - - -.5 - -.5.5.5 7
Dscrmat fuctos.5.5 -.5 - -.5 - - -.5 - -.5.5.5 Dscrmat fuctos Decso bouar: scrmat fuctos are equal.5.5 -.5 - -.5 - - -.5 - -.5.5.5 8
Quaratc ecso bouar 3 Decso bouar.5.5.5 -.5 - -.5 - - -.5 - -.5.5.5 Lostc reresso moel Defes a lear ecso bouar Dscrmat fuctos: here / e f - s a lostc fucto Iput vector f Lostc fucto 9
Fucto: Lostc fucto e Is also referre to as a smo fucto taes a real umber a outputs the umber the terval [] Moels a smooth stch fucto; replaces har threshol fucto.9.8.7.6.5.9.8.7.6.5.4.4.3.3.... - -5 - -5 5 5 Lostc smooth stch - -5 - -5 5 5 hreshol har stch Lostc reresso moel Dscrmat fuctos: Values of scrmat fuctos var terval [] Probablstc terpretato f p p Iput vector
Lostc reresso We lear a probablstc fucto f : X [] here f escrbes the probablt of class ve f p Note that: p p Ma ecsos th the lostc reresso moel:? Lostc reresso We lear a probablstc fucto f : X [] here f escrbes the probablt of class ve f p Note that: p p Ma ecsos th the lostc reresso moel: If p / the choose Else choose
Lear ecso bouar Lostc reresso moel efes a lear ecso bouar Wh? Aser: Compare to scrmat fuctos. Decso bouar: For the bouar t must hol: lo o lo lo o ep ep lo ep lo ep Lostc reresso moel. Decso bouar LR efes a lear ecso bouar Eample: classes blue a re pots Decso bouar.5.5 -.5 - -.5 = - - -.5 - -.5.5.5
3 Lelhoo of outputs Let he F ehts that mame the lelhoo of outputs Appl the lo-lelhoo trc. he optmal ehts are the same for both the lelhoo a the lo-lelhoo Lostc reresso: parameter lear D l lo lo P D L p lo lo D Lostc reresso: parameter lear Notato: Lo lelhoo Dervatves of the lolelhoo Graet escet: lo lo D l f D l ] [ D l Nolear ehts!! f ] [ D l p
4 Dervato of the raet Lo lelhoo Dervatves of the lolelhoo lo lo D l f D l D l lo lo lo lo Dervatve of a lostc fucto Lostc reresso. Ole raet escet O-le compoet of the lolelhoo O-le lear upate for eht th upate for the lostc reresso a ] [ ole D ole D D f ] [ lo lo ole D
Ole lostc reresso alorthm Ole-lostc-reresso stopp_crtero tale ehts hle stopp_crtero = FALSE o select et ata pot D set upate ehts parallel e retur ehts [ f ] Ole alorthm. Eample. 5
Ole alorthm. Eample. Ole alorthm. Eample. 6