Machine Learning. Introduction to Regression. Lecture 3, September 19, Reading: Chap. 3, CB

Transcription

1 ache Learg 0-70/5 70/ all 006 Iroduco o Regresso Erc g Lecure 3 Sepember Readg: Chap. 3 C Iferece wh he Jo Compue Codoals P lu eadhead P lu eadhead P eadhead Geeral dea: compue dsrbuo o quer arable b fg edece arables ad summg oer hdde arables

2 Codoal depedece Wre ou full o dsrbuo usg cha rule: Peadache;lu;Vrus;Drkeer Peadache lu;vrus;drkeer Plu;Vrus;Drkeer Peadache lu;vrus;drkeer Plu Vrus;Drkeer PVrus Drkeer PDrkeer Assume depedece ad codoal depedece Peadache lu;drkeer Plu Vrus PVrus PDrkeer I.e.? depede parameers I mos cases he use of codoal depedece reduces he sze of he represeao of he o dsrbuo from epoeal o lear. Codoal depedece s our mos basc ad robus form of kowledge abou ucera eromes. Rules of Idepedece --- b eamples PVrus Drkeer PVrus ff Vrus s depede of Drkeer Plu Vrus;Drkeer Plu Vrus ff lu s depede of Drkeer ge Vrus Peadache lu;vrus;drkeer Peadache lu;drkeer ff eadache s depede of Vrus ge lu ad Drkeer

3 argal ad Codoal Idepedece Recall ha for ees E.e. ad sa Y he codoal probabl of E ge wre as PE s PE ad /P he probabl of boh E ad are rue ge s rue E ad are sascall depede f PE PE.e. prob. E s rue does' deped o wheher s rue; or equalel PE ad PEP. E ad are codoall depede ge f PE PE or equalel PE PE P Wh kowledge of Idepedece s useful Lower comple me space search oaes effce ferece for all kds of queres Sa ued!! Srucured kowledge abou he doma eas o learg boh from eper ad from daa eas o grow 3

4 Where do probabl dsrbuos come from? Idea Oe: uma Doma Epers Idea wo: Smpler probabl facs ad some algebra e.g. P P P P Idea hree: Lear hem from daa! A good chuk of hs course s esseall abou arous was of learg arous forms of hem! Des Esmao A Des Esmaor lears a mappg from a se of arbues o a Probabl Ofe kow as parameer esmao f he dsrbuo form s specfed omal Gaussa hree mpora ssues: aure of he daa d correlaed Obece fuco LE AP Algorhm smple algebra grade mehods E Ealuao scheme lkelhood o es daa predcabl cossec 4

5 Parameer Learg from d daa Goal: esmae dsrbuo parameers from a daase of depede decall dsrbued d full obsered rag cases D {... } amum lkelhood esmao LE. Oe of he mos commo esmaors. Wh d ad full-obserabl assumpos wre L as he lkelhood of he daa: L P K ; P ; P ; K P P ; 3. pck he seg of parameers mos lkel o hae geeraed he daa we saw: ; * arg ma L arg ma log L Eample : eroull model Daa: We obsered d co ossg: D{ 0 0} Represeao: ar r.: odel: p P p for 0 for ow o wre he lkelhood of a sgle obserao? P { 0 } P he lkelhood of daased{ }: P... P #head #als 5

6 LE Obece fuco: h l ; D log P D log log log We eed o mamze hs w.r.. ake deraes wr h h l h h 0 h LE or LE Suffce sascs h where k requec as sample mea he cous are suffce sascs of daa D LE for dscree o dsrbuos ore geerall s eas o show ha # records whch ee s rue P ee oal umber of records hs s a mpora bu somemes o so effece learg algorhm!

7 Eample : uarae ormal Daa: We obsered d real samples: odel: D{ } P Log lkelhood: / πσ ep{ µ / σ } LE: ake derae ad se o zero: µ l ; D logp D log πσ σ l µ / σ µ µ LE l 4 σ σ σ µ σ LE µ L Oerfg Recall ha for eroull Dsrbuo we hae Wha f we ossed oo few mes so ha we saw zero head? We hae head ad we wll predc ha he probabl of L 0 seeg a head e s zero!!! he rescue: head L head Where ' s kow as he pseudo- magar cou head L u ca we make hs more formal? head al head ' head al ' 7

8 he aesa heor he aesa heor: e.g. for dae D ad model P D PD P/PD he poseror equals o he lkelhood mes he pror up o a cosa. hs allows us o capure ucera abou he model a prcpled wa erarchcal aesa odels are he parameers for he lkelhood p are he parameers for he pror p. We ca hae hper-hper-parameers ec. We sop whe he choce of hper-parameers makes o dfferece o he margal lkelhood; pcall make hperparameers cosas. Where do we ge he pror? Iellge guesses Emprcal aes pe-ii mamum lkelhood compug po esmaes of : LE arg ma p 8

9 9 aesa esmao for eroull ea dsrbuo: Poseror dsrbuo of : oce he somorphsm of he poseror o he pror such a pror s called a cougae pror β β h h p p p P Γ Γ Γ β β β β β β ; P aesa esmao for eroull co'd Poseror dsrbuo of : amum a poseror AP esmao: Poseror mea esmao: Pror sregh: Aβ A ca be eroperaed as he sze of a magar daa se from whch we oba he pseudo-cous β β h h p p p P β β d C d D p h aes h aa parameers ca be udersood as pseudo-cous... ma log arg AP P

10 Effec of Pror Sregh Suppose we hae a uform pror β/a ad we obsere h 8 Weak pror A. Poseror predco: p h h 8 ' Srog pror A 0. Poseror predco: 0 p h h 8 ' oweer f we hae eough daa washes awa he pror. e.g. h he he esmaes uder weak ad srog pror are 000 ad respecel boh of whch are close o 0. aesa esmao for ormal dsrbuo ormal Pror: Jo probabl: Poseror: / πτ ep{ µ µ τ } P µ / P µ / πσ ep µ σ / πτ ep{ µ µ / τ } 0 0 omework!!! 0

11 ache Learg 0-70/5 70/ all 006 Iroduco o Regresso Erc g Lecure 3 Sepember Readg: Chap. 3 C ache learg for aparme hug ow ou'e moed o Psburgh!! Ad ou wa o fd he mos reasoabl prced aparme sasfg our eeds: square-f. # of bedroom dsace o campus Lg area f # bedroom.5 Re $ ??

12 he learg problem eaures: Lg area dsace o campus # bedroom Deoe as [ k ] arge: Re Deoed as rag se: re Lg area re Lg area Locao k k k K K K Y or Lear Regresso Assume ha Y arge s a lear fuco of feaures: e.g.: le's assume a acuous "feaure" 0 hs s he ercep erm wh? ad defe he feaure ecor o be: he we hae he followg geeral represeao of he lear fuco: Our goal s o pck he opmal. ow! We seek ha mmze he followg cos fuco: 0 ˆ J ˆ

13 he Leas-ea-Square LS mehod he Cos uco: J Cosder a grade desce algorhm: J he Leas-ea-Square LS mehod ow we hae he followg desce rule: or a sgle rag po we hae: hs s kow as he LS updae rule or he Wdrow-off learg rule hs s acuall a "sochasc" "coordae" desce algorhm hs ca be used as a o-le algorhm 3

14 he Leas-ea-Square LS mehod Seepes desce oe ha: J J K J k hs s as a bach grade desce algorhm Some mar deraes or R m f : a R defe: f A A f A f A m race: ra A L O L A A m ra a f f r AC rca rca Some fac of mar deraes whou proof ra raa C CA C A A A A A A A 4

15 5 he ormal equaos Wre he cos fuco mar form: o mmze J ake derae ad se o zero: J 0 J r r r r he ormal equaos * A recap: LS updae rule Pros: o-le low per-sep cos Cos: coordae mabe slow-coergg Seepes desce Pros: fas-coergg eas o mpleme Cos: a bach ormal equaos Pros: a sgle-sho algorhm! Eases o mpleme. Cos: eed o compue pseudo-erse - epese umercal ssues e.g. mar s sgular.. *

16 Geomerc Ierpreao of LS he predcos o he rag daa are: ˆ * oe ha ˆ I ad ˆ I 0!! ŷ s he orhogoal proeco of o he space spaed b he colums of Probablsc Ierpreao of LS Le us assume ha he arge arable ad he pus are relaed b he equao: where ε s a error erm of umodeled effecs or radom ose ow assume ha ε follows a Gaussa 0σ he we hae: ε ep πσ σ ; p depedece assumpo: L p ; ep πσ σ 6

17 Probablsc Ierpreao of LS co. ece he log-lkelhood s: l log πσ σ Do ou recogze he las erm? Yes s: J hus uder depedece assumpo LS s equale o LE of! eod basc LR LR wh o-lear bass fucos Locall weghed lear regresso Regresso rees ad ullear Ierpolao 7

18 LR wh o-lear bass fucos LR does o mea we ca ol deal wh lear relaoshps We are free o desg o-lear feaures uder LR m 0 φ φ where he φ are fed bass fucos ad we defe φ 0. Eample: polomal regresso: 3 [ ] φ : We wll be cocered wh esmag dsrbuos oer he weghs ad choosg he model order. ass fucos here are ma bass fucos e.g.: Polomal Radal bass fucos Sgmodal φ φ µ φ σ s ep µ s Sples ourer Waeles ec 8

19 D ad D Rs D R Afer f: Good ad ad Rs A good D R wo bad D Rs 9

20 Locall weghed lear regresso Oerfg ad uderfg Locall weghed lear regresso he algorhm: Isead of mmzg ow we f o mmze Where do w 's come from? J J w ep τ w where s he quer po for whch we'd lke o kow s correspodg Esseall we pu hgher weghs o errors o rag eamples ha are close o he quer po ha hose ha are furher awa from he quer Do we also hae a probablsc erpreao here as we dd for LR? 0

21 Paramerc s. o-paramerc Locall weghed lear regresso s he frs eample we are rug o of a o-paramerc algorhm. he uweghed lear regresso algorhm ha we saw earler s kow as a paramerc learg algorhm because has a fed fe umber of parameers he whch are f o he daa; Oce we'e f he ad sored hem awa we o loger eed o keep he rag daa aroud o make fuure predcos. I coras o make predcos usg locall weghed lear regresso we eed o keep he ere rag se aroud. he erm "o-paramerc" roughl refers o he fac ha he amou of suff we eed o keep order o represe he hpohess grows learl wh he sze of he rag se. Robus Regresso he bes f from a quadrac regresso u hs s probabl beer ow ca we do hs?

22 LOESS-based Robus Regresso Remember wha we do "locall weghed lear regresso"? we "score" each po for s mpoece ow we score each po accordg o s "fess" Coures o Adrew oor Robus regresso or k o R Le k k be he kh daapo Le es k be predced alue of k Le w k be a wegh for daa po k ha s large f he daa po fs well ad small f fs badl: w φ k es k k he redo he regresso usg weghed daa pos. Repea whole hg ul coerged!

23 Robus regresso probablsc erpreao Wha regular regresso does: Assume k was orgall geeraed usg he followg recpe: k k 0 σ Compuaoal ask s o fd he amum Lkelhood esmao of Robus regresso probablsc erpreao Wha LOESS robus regresso does: Assume k was orgall geeraed usg he followg recpe: wh probabl p: k k 0 σ bu oherwse k ~ µ σ huge Compuaoal ask s o fd he amum Lkelhood esmaes of p µ ad σ huge. he algorhm ou saw wh erae reweghg/refg does hs compuao for us. Laer ou wll fd ha s a sace of he famous E.. algorhm 3

24 Regresso ree Decso ree for regresso Geder Rch? um. Chldre # rael per r. Age Geder? o o emale ale : Yes : : 0 : 7 : Predced age39 Predced age36 A cocepual pcure Assumg regular regresso rees ca ou skech a graph of he fed fuco * oer hs dagram? 4

25 ow abou hs oe? ullear Ierpolao We waed o creae a couous ad pecewse lear f o he daa ake home message Grade desce O-le ach ormal equaos Equalece of LS ad LE LR does o mea fg lear relaos bu lear combao or bass fucos ha ca be o-lear Weghg pos b mporace ersus b fess 5