Advanced Introduction to Machine Learning

Size: px

Start display at page:

Download "Advanced Introduction to Machine Learning"

Isabella Preston
5 years ago
Views:

1 Advaced Itroducto to Mache Learg 075, Fall 04 Lear Regresso ad Sparst Erc g Lecture, September 0, 04 Readg: Erc CMU, 04

2 Mache learg for apartmet hutg ow ou've moved to Pttsburgh!! Ad ou wat to fd the most reasoabl prced apartmet satsfg our eeds: square-ft., # of bedroom, dstace to campus Lvg area ft # bedroom Ret $ ? 70.5? Erc CMU, 04

3 he learg problem Features: Lvg area, dstace to campus, # bedroom Deote as =[,, k ] arget: Ret Deoted as rag set: ret Lvg area ret Lvg area Locato k k k Y or Erc CMU, 04 3

4 Lear Regresso Assume that Y target s a lear fucto of features: e.g.: let's assume a vacuous "feature" 0 = ths s the tercept term, wh?, ad defe the feature vector to be: the we have the followg geeral represetato of the lear fucto: Our goal s to pck the optmal. How! We seek that mmze the followg cost fucto: 0 ˆ J ˆ Erc CMU, 04 4

5 he Least-Mea-Square LMS method he Cost Fucto: J Cosder a gradet descet algorthm: t t j j J j t Erc CMU, 04 5

6 he Least-Mea-Square LMS method ow we have the followg descet rule: t j t j t j For a sgle trag pot, we have: hs s kow as the LMS update rule, or the Wdrow-Hoff learg rule hs s actuall a "stochastc", "coordate" descet algorthm hs ca be used as a o-le algorthm Erc CMU, 04 6

7 Geometr ad Covergece of LMS t t = = =3 t Clam: whe the step sze satsfes certa codto, ad whe certa other techcal codtos are satsfed, LMS wll coverge to a optmal rego. Erc CMU, 04 7

8 Steepest Descet ad LMS Steepest descet ote that: hs s as a batch gradet descet algorthm k J J J,, t t t Erc CMU, 04 8

9 he ormal equatos Wrte the cost fucto matr form: o mmze Jθ, take dervatve ad set to zero: J 0 J tr tr tr tr he ormal equatos * Erc CMU, 04 9

10 Commets o the ormal equato I most stuatos of practcal terest, the umber of data pots s larger tha the dmesoalt k of the put space ad the matr s of full colum rak. If ths codto holds, the t s eas to verf that s ecessarl vertble. he assumpto that s vertble mples that t s postve defte, thus at the crtcal pot we have foud s a mmum. What f has less tha full colum rak? regularzato later. Erc CMU, 04 0

11 Drect ad Iteratve methods Drect methods: we ca acheve the soluto a sgle step b solvg the ormal equato Usg Gaussa elmato or QR decomposto, we coverge a fte umber of steps It ca be feasble whe data are streamg real tme, or of ver large amout Iteratve methods: stochastc or steepest gradet Covergg a lmtg sese But more attractve large practcal problems Cauto s eeded for decdg the learg rate Erc CMU, 04

12 Covergece rate heorem: the steepest descet equato algorthm coverge to the mmum of the cost characterzed b ormal equato: If A formal aalss of LMS eed more math-mussels; practce, oe ca use a small, or graduall decrease. Erc CMU, 04

13 A Summar: LMS update rule Pros: o-le, low per-step cost, fast covergece ad perhaps less proe to local optmum Cos: covergece to optmum ot alwas guarateed Steepest descet t Pros: eas to mplemet, coceptuall clea, guarateed covergece Cos: batch, ofte slow covergg ormal equatos t j * t t, Pros: a sgle-shot algorthm! Easest to mplemet. t j t Cos: eed to compute pseudo-verse -, epesve, umercal ssues e.g., matr s sgular.., although there are was to get aroud ths Erc CMU, 04 3

14 Geometrc Iterpretato of LMS he predctos o the trag data are: ote that ad s the orthogoal projecto of to the space spaed b the colums of ˆ * I ˆ 0 I ˆ!! ŷ Erc CMU, 04 4

15 Probablstc Iterpretato of LMS Let us assume that the target varable ad the puts are related b the equato: where ε s a error term of umodeled effects or radom ose ow assume that ε follows a Gaussa 0,σ, the we have: B depedece assumpto: ep ; p p L ep ; Erc CMU, 04 5

16 Probablstc Iterpretato of LMS, cot. Hece the log-lkelhood s: l log Do ou recogze the last term? Yes t s: J hus uder depedece assumpto, LMS s equvalet to MLE of θ! Erc CMU, 04 6

17 Case stud: predctg gee epresso he geetc pcture causal SPs CGCACGACAA a uvarate pheotpe:.e., the epresso test of a gee Erc CMU, 04 7

18 Assocato Mappg as Regresso Idvdual Idvdual Idvdual Pheotpe BMI Geotpe.. C C C..... A.. C G..... A.. G A..... C C G C G G Beg SPs Causal SP Erc CMU, 04 8

19 Assocato Mappg as Regresso Pheotpe BMI Geotpe Idvdual Idvdual Idvdual = J j j j SPs wth large β j are relevat Erc CMU, 04 9

20 Epermetal setup Asthama dataset 543 dvduals, geotped at 34 SPs Dplod data was trasformed to 0/ for homozgotes or for heterozgotes =54334 matr Y=Pheotpe varable cotuous A sgle pheotpe was used for regresso Implemetato detals Iteratve methods: Batch update ad ole update mplemeted. For both methods, step sze α s chose to be a small fed value 0-6. hs choce s based o the data used for epermets. Both methods are ol ru to a mamum of 000 epochs or utl the chage trag MSE s less tha 0-4 Erc CMU, 04 0

21 Covergece Curves For the batch method, the trag MSE s tall large due to uformed talzato I the ole update, updates for ever epoch reduces MSE to a much smaller value. Erc CMU, 04

22 he Leared Coeffcets Erc CMU, 04

23 Multvarate Regresso for rat Assocato Aalss rat Geotpe Assocato Stregth G A A C C A G A A G A. =? = β Erc CMU, 04 3

24 Multvarate Regresso for rat Assocato Aalss rat Geotpe Assocato Stregth G A A C C A G A A G A. = Ma o-zero assocatos: Whch SPs are trul sgfcat? Erc CMU, 04 4

25 Sparst Oe commo assumpto to make sparst. Makes bologcal sese: each pheotpe s lkel to be assocated wth a small umber of SPs, rather tha all the SPs. Makes statstcal sese: Learg s ow feasble hgh dmesos wth small sample sze Erc CMU, 04 5

26 Sparst: I a mathematcal sese Cosder least squares lear regresso problem: Sparst meas most of the beta s are zero But ths s ot cove!!! Ma local optma, computatoall tractable. Erc CMU, 04 6

27 L Regularzato LASSO bshra, 996 A cove relaato. Costraed Form Lagraga Form Stll eforces sparst! Erc CMU, 04 7

28 heoretcal Guaratees Assumptos Depedec Codto: Relevat Covarates are ot overl depedet Icoherece Codto: Large umber of rrelevat covarates caot be too correlated wth relevat covarates Strog cocetrato bouds: Sample quattes coverge to epected values quckl If these are assumptos are met, LASSO wll asmptotcall recover correct subset of covarates that relevat. Erc CMU, 04 8

29 Cosstet Structure Recover [Zhao ad Yu 006] Erc CMU, 04 9

30 Lasso for Reducg False Postves rat Geotpe Assocato Stregth. = G A A C C A G A A G A Lasso Pealt for sparst J + β j j Ma zero assocatos sparse results, but what f there are multple related trats? Erc CMU, 04 30

31 Rdge Regresso vs Lasso Rdge Regresso: Lasso: HO! βs wth costat Jβ level sets of Jβ βs wth costat l orm β βs wth costat l orm β Lasso l pealt results sparse solutos vector wth more zero coordates Good for hgh dmesoal problems do t have to store all coordates! Erc CMU, 04 3

32 Baesa Iterpretato reat the dstrbuto parameters also as a radom varable he a posteror dstrbuto of after seem the data s: hs s Baes Rule lkelhood margal pror lkelhood posteror d p D p p D p D p p D p D p he pror p. ecodes our pror kowledge about the doma Erc CMU, 04 3

33 Regularzed Least Squares ad MAP What f s ot vertble? log lkelhood log pror I Gaussa Pror 0 Rdge Regresso Closed form: HW Pror belef that β s Gaussa wth zero mea bases soluto to small β Erc CMU, 04 33

34 Regularzed Least Squares ad MAP What f s ot vertble? log lkelhood log pror II Laplace Pror Lasso Closed form: HW Pror belef that β s Laplace wth zero mea bases soluto to small β Erc CMU, 04 34

35 ake home message Gradet descet O-le Batch ormal equatos Geometrc terpretato of LMS Probablstc terpretato of LMS, ad equvalece of LMS ad MLE uder certa assumpto what? Sparst: Approach: rdge vs. lasso regresso Iterpretato: regularzed regresso versus Baesa regresso Algorthm: cove optmzato we dd ot dscuss ths LR does ot mea fttg lear relatos, but lear combato or bass fuctos that ca be o-lear Weghtg pots b mportace versus b ftess Erc CMU, 04 35

36 After class materal Erc CMU, 04 36

37 Advaced Materal: Beod basc LR LR wth o-lear bass fuctos Locall weghted lear regresso Regresso trees ad Multlear Iterpolato We wll dscuss ths et class after we set the state rght! f we ve got tme Erc CMU, 04 37

38 LR wth o-lear bass fuctos LR does ot mea we ca ol deal wth lear relatoshps We are free to desg o-lear features uder LR 0 m j j where the j are fed bass fuctos ad we defe 0 =. Eample: polomal regresso: :, 3,, We wll be cocered wth estmatg dstrbutos over the weghts θ ad choosg the model order M. Erc CMU, 04 38

39 Bass fuctos here are ma bass fuctos, e.g.: Polomal Radal bass fuctos Sgmodal j j j j j s ep j s Sples, Fourer, Wavelets, etc Erc CMU, 04 39

40 D ad D RBFs D RBF After ft: Erc CMU, 04 40

41 Good ad Bad RBFs A good D RBF wo bad D RBFs Erc CMU, 04 4

42 Overfttg ad uderfttg j 0 j j Erc CMU, 04 4

43 Bas ad varace We defe the bas of a model to be the epected geeralzato error eve f we were to ft t to a ver sa, ftel large trag set. B fttg "spurous" patters the trag set, we mght aga obta a model wth large geeralzato error. I ths case, we sa the model has large varace. Erc CMU, 04 43

44 Locall weghted lear regresso he algorthm: Istead of mmzg ow we ft θ to mmze Where do w 's come from? J J w ep w where s the quer pot for whch we'd lke to kow ts correspodg Essetall we put hgher weghts o errors o trag eamples that are close to the quer pot tha those that are further awa from the quer Erc CMU, 04 44

45 Parametrc vs. o-parametrc Locall weghted lear regresso s the secod eample we are rug to of a o-parametrc algorthm. what s the frst? he uweghted lear regresso algorthm that we saw earler s kow as a parametrc learg algorthm because t has a fed, fte umber of parameters the θ, whch are ft to the data; Oce we've ft the θ ad stored them awa, we o loger eed to keep the trag data aroud to make future predctos. I cotrast, to make predctos usg locall weghted lear regresso, we eed to keep the etre trag set aroud. he term "o-parametrc" roughl refers to the fact that the amout of stuff we eed to keep order to represet the hpothess grows learl wth the sze of the trag set. Erc CMU, 04 45

46 Robust Regresso he best ft from a quadratc regresso But ths s probabl better How ca we do ths? Erc CMU, 04 46

47 LOESS-based Robust Regresso Remember what we do "locall weghted lear regresso"? we "score" each pot for ts mpotece ow we score each pot accordg to ts "ftess" Courtes to Adrew Moor Erc CMU, 04 47

48 Robust regresso For k = to R Let k, k be the kth datapot Let est k be predcted value of k Let w k be a weght for data pot k that s large f the data pot fts well ad small f t fts badl: w k k est k he redo the regresso usg weghted data pots. Repeat whole thg utl coverged! Erc CMU, 04 48

49 Robust regresso probablstc terpretato What regular regresso does: Assume k was orgall geerated usg the followg recpe: k k 0, Computatoal task s to fd the Mamum Lkelhood estmato of θ Erc CMU, 04 49

50 Robust regresso probablstc terpretato What LOESS robust regresso does: Assume k was orgall geerated usg the followg recpe: wth probablt p: k 0, k but otherwse k ~, huge Computatoal task s to fd the Mamum Lkelhood estmates of θ, p, µ ad σ huge. he algorthm ou saw wth teratve reweghtg/refttg does ths computato for us. Later ou wll fd that t s a stace of the famous E.M. algorthm Erc CMU, 04 50

51 Regresso ree Decso tree for regresso Geder Rch? um. Chldre # travel per r. Age Geder? F o 5 38 M o 0 5 M Yes 0 7 : : : : : Female Predcted age=39 Male Predcted age=36 Erc CMU, 04 5

52 A coceptual pcture Assumg regular regresso trees, ca ou sketch a graph of the ftted fucto * over ths dagram? Erc CMU, 04 5

53 How about ths oe? Multlear Iterpolato We wated to create a cotuous ad pecewse lear ft to the data Erc CMU, 04 53

54 ake home message Gradet descet O-le Batch ormal equatos Geometrc terpretato of LMS Probablstc terpretato of LMS, ad equvalece of LMS ad MLE uder certa assumpto what? Sparst: Approach: rdge vs. lasso regresso Iterpretato: regularzed regresso versus Baesa regresso Algorthm: cove optmzato we dd ot dscuss ths LR does ot mea fttg lear relatos, but lear combato or bass fuctos that ca be o-lear Weghtg pots b mportace versus b ftess Erc CMU, 04 54

55 Apped Erc CMU, 04 55

56 Parameter Learg from d Data Goal: estmate dstrbuto parameters from a dataset of depedet, detcall dstrbuted d, full observed, trag cases D = {,..., } Mamum lkelhood estmato MLE. Oe of the most commo estmators. Wth d ad full-observablt assumpto, wrte L as the lkelhood of the data: L P,,, ; P ; P P 3. pck the settg of parameters most lkel to have geerated the data we saw: * ; ;,, P ; argma L argma log L Erc CMU, 04 56

57 Eample: Beroull model Data: We observed d co tossg: D={, 0,,, 0} Represetato: Bar r.v: { 0, } Model: for 0 P for How to wrte the lkelhood of a sgle observato? P P he lkelhood of datasetd={,, }: P,,..., P #head #tals Erc CMU, 04 57

58 Mamum Lkelhood Estmato Objectve fucto: h t l ; D log P D log log log We eed to mamze ths w.r.t. ake dervatves wrt h h l h h 0 MLE h or MLE Suffcet statstcs h, where k Frequec as sample mea he couts, are suffcet statstcs of data D, Erc CMU, 04 58

59 Overfttg Recall that for Beroull Dstrbuto, we have head ML head head tal What f we tossed too few tmes so that we saw zero head? We have head ad we wll predct that the probablt of ML 0, seeg a head et s zero!!! he rescue: "smoothg" Where ' s kow as the pseudo- magar cout head ML But ca we make ths more formal? head ' tal ' head Erc CMU, 04 59

Rule lkelhood margal pror lkelhood posteror d p D p p D p D p p D p D

60 Baesa Parameter Estmato reat the dstrbuto parameters also as a radom varable he a posteror dstrbuto of after seem the data s: hs s Baes Rule lkelhood margal pror lkelhood posteror d p D p p D p D p p D p D p he pror p. ecodes our pror kowledge about the doma Erc CMU, 04 60

61 Frequetst Parameter Estmato wo people wth dfferet prors p wll ed up wth dfferet estmates p D. Frequetsts dslke ths subjectvt. Frequetsts thk of the parameter as a fed, ukow costat, ot a radom varable. Hece the have to come up wth dfferet "objectve" estmators was of computg from data, stead of usg Baes rule. hese estmators have dfferet propertes, such as beg ubased, mmum varace, etc. he mamum lkelhood estmator, s oe such estmator. Erc CMU, 04 6

62 Dscusso or p, ths s the problem! Baesas kow t Erc CMU, 04 6

63 Baesa estmato for Beroull Beta dstrbuto: Whe s dscrete Posteror dstrbuto of : otce the somorphsm of the posteror to the pror, such a pror s called a cojugate pror ad are hperparameters parameters of the pror ad correspod to the umber of vrtual heads/tals pseudo couts t h t h p p p P,...,,...,,...,,, ; P B! Erc CMU, 04 63

64 Baesa estmato for Beroull, co'd Posteror dstrbuto of : Mamum a posteror MAP estmato: Posteror mea estmato: Pror stregth: A=+ A ca be teroperated as the sze of a magar data set from whch we obta the pseudo-couts t h t h p p p P,...,,...,,..., d C d D p h Baes t h Bata parameters ca be uderstood as pseudo-couts,..., ma log arg MAP P Erc CMU, 04 64

65 Effect of Pror Stregth Suppose we have a uform pror ==/, ad we observe h, 8 Weak pror A =. Posteror predcto: Strog pror A = 0. Posteror predcto: However, f we have eough data, t washes awa the pror. e.g., h 00,. he the estmates uder t weak ad strog pror are 000 ad, respectvel, 0000 both of whch are close to 0. t p h h, t 8, ' p h h, t 8, ' Erc CMU, 04 65

66 Eample : Gaussa dest Data: We observed d real samples: D={-0., 0,, -5.,, 3} Model: P Log lkelhood: / ep / D P D l ; log log MLE: take dervatve ad set to zero: l l / 4 MLE MLE ML Erc CMU, 04 66

67 MLE for a multvarate-gaussa It ca be show that the MLE for µ ad Σ s where the scatter matr s he suffcet statstcs are ad. ote that = ma ot be full rak eg. f <D, whch case Σ ML s ot vertble S ML ML MLE MLE ML ML ML ML S K Erc CMU, 04 67

68 Baesa estmato ormal Pror: Jot probablt: Posteror: / ep / P ~ ad, / / / / / / ~ where Sample mea / ep ep, / / P ~ / ~ ep ~ / P Erc CMU, 04 68

69 Baesa estmato: ukow µ, kow σ he posteror mea s a cove combato of the pror ad the MLE, wth weghts proportoal to the relatve ose levels. he precso of the posteror /σ s the precso of the pror /σ 0 plus oe cotrbuto of data precso /σ for each observed data pot. Sequetall updatg the mea µ = 0.8 ukow, σ = 0. kow Effect of sgle data pot Uformatve vague/ flat pror, σ ~, / / / / / / Erc CMU, 04 69

Machine Learning. Introduction to Regression. Le Song. CSE6740/CS7641/ISYE6740, Fall 2012

Machine Learning. Introduction to Regression. Le Song. CSE6740/CS7641/ISYE6740, Fall 2012 Mache Learg CSE6740/CS764/ISYE6740, Fall 0 Itroducto to Regresso Le Sog Lecture 4, August 30, 0 Based o sldes from Erc g, CMU Readg: Chap. 3, CB Mache learg for apartmet hutg Suppose ou are to move to