Generative vs. Discriminative Classifiers
|
|
- Ethan Pope
- 5 years ago
- Views:
Transcription
1 Geerate s. Dscrmate Classfers Goal: Wsh to lear f: Y, e.g., P(Y ) Geerate classfers (e.g., Naïe Baes): Assume some fuctoal form for P( Y), P(Y) hs s a geerate model of the data! Estmate parameters of P( Y), P(Y) drectl from trag data Use Baes rule to calculate P(Y ) Y Dscrmate classfers: Drectl assume some fuctoal form for P(Y ) hs s a dscrmate model of the data! Estmate parameters of P(Y ) drectl from trag data Y Naïe Baes s Logstc Regresso Cosder Y boolea, cotuous, <... m > Number of parameters to estmate: NB: LR: p( ) π k ep π k ' ep µ ( ) + e j ( ) log j µ k, j σ k, j C σ k, j ( ', ) log j µ k j σ k ' C σ k ', j, ' k j j ** Estmato method: NB parameter estmates are ucoupled LR parameter estmates are coupled
2 Naïe Baes s Logstc Regresso Asmptotc comparso (# trag eamples ft) whe model assumptos correct NB, LR produce detcal classfers whe model assumptos correct LR s less based does ot assume codtoal depedece therefore epected to outperform NB 3 Naïe Baes s Logstc Regresso No-asmptotc aalss (see [Ng & Jorda, 00] ) coergece rate of parameter estmates how ma trag eamples eeded to assure good estmates? NB order log m (where m # of attrbutes ) LR order m NB coerges more quckl to ts (perhaps less helpful) asmptotc estmates 4
3 Rate of coergece: logstc regresso Let h Ds,m be logstc regresso traed o eamples m dmesos. he wth hgh probablt: Implcato: f we wat for some small costat ε 0, t suffces to pck order m eamples Coergeces to ts asmptotc classfer, order m eamples result follows from Vapk s structural rsk boud, plus fact that the "VC Dmeso" of a m-dmesoal lear separators s m 5 Rate of coergece: aïe Baes parameters Let a ε, δ>0, ad a 0 be fed. Assume that for some fed ρ 0 >0 0, we hae that Let he wth probablt at least -δ, after eamples:. For dscrete put, for all ad b. For cotuous puts, for all ad b 6 3
4 Some epermets from UCI data sets 7 Summar Naïe Baes classfer What s the assumpto Wh we use t How do we lear t Logstc regresso Fuctoal form follows from Naïe Baes assumptos For Gaussa Naïe Baes assumg arace For dscrete-alued Naïe Baes too But trag procedure pcks parameters wthout the codtoal depedece assumpto Gradet ascet/descet Geeral approach whe closed-form solutos uaalable Geerate s. Dscrmate classfers Bas s. arace tradeoff 8 4
5 ache Learg 0-70/5-78, 78, Fall 0 Lear Regresso ad Sparst Erc g Lecture 4, September, 0 Readg: 9 ache learg for apartmet hutg Now ou'e moed to Pttsburgh!! Ad ou wat to fd the most reasoabl prced apartmet satsfg our eeds: square-ft., # of bedroom, dstace to campus Lg area (ft ) # bedroom Ret ($) ? 70.5? 0 5
6 6 he learg problem Features: Lg area, dstace to campus, # t Lg area, dstace to campus, # bedroom Deote as [,, k ] arget: Ret Deoted as rag set: ret Lg area k ret Lg area Locato k k K K K Y or Lear Regresso Assume that Y (target) s a lear fucto of (features): e.g.: ˆ g let's assume a acuous "feature" 0 (ths s the tercept term, wh?), ad defe the feature ector to be: the we hae the followg geeral represetato of the lear fucto: 0 ˆ + + Our goal s to pck the optmal. How! We seek that mmze the followg cost fucto: J ) ) ( ˆ ( ) (
7 he Least-ea-Square (LS) method he Cost Fucto: J ( ) ( ) Cosder a gradet descet algorthm: t+ t j j α J ( ) j t 3 he Least-ea-Square (LS) method Now we hae the followg descet rule: t+ j t j + α t ( ) j For a sgle trag pot, we hae: hs s kow as the LS update rule, or the Wdrow-Hoff learg rule hs s actuall a "stochastc", "coordate" descet algorthm hs ca be used as a o-le algorthm 4 7
8 8 Geometrc ad Coergece of LS N N N3 Clam: whe the step sze α satsfes certa codto, ad whe certa other techcal codtos are satsfed, LS wll coerge to a optmal rego. t t t ) ( α Steepest Descet ad LS Steepest descet Note that: k J J J ) (,, K + + t t t ) ( α hs s as a batch gradet descet algorthm 6
9 9 he ormal equatos Wrte the cost fucto matr form: J ) ( ) ( o mmze J(), take derate ad set to zero: ( ) ( ) ( ) J + ) ( ) ( ( ) ( ) ( ) J tr tr tr tr he ormal equatos ( ) * 7 Some matr derates For, defe: R a R m f : race: f A f A f A f A A f m m A L O L ) (, tr A A, tr a a BCA CAB ABC tr tr tr Some fact of matr derates (wthout proof), tr A B AB, tr A AB C CAB C ABA + ( ) A A A A 8
10 Commets o the ormal equato I most stuatos of practcal terest, the umber of data pots N s larger tha the dmesoalt k of the put space ad the matr s of full colum rak. If ths codto holds, the t s eas to erf that s ecessarl ertble. he assumpto that s ertble mples that t s poste defte, thus the crtcal pot we hae foud s a mmum. What f has less tha full colum rak? regularzato (later). 9 Drect ad Iterate methods Drect methods: we ca achee the soluto a sgle step b solg the ormal equato Usg Gaussa elmato or QR decomposto, we coerge a fte umber of steps It ca be feasble whe data are streamg real tme, or of er large amout Iterate methods: stochastc or steepest gradet Coergg a lmtg sese But more attracte large practcal problems Cauto s eeded for decdg the learg rate α 0 0
11 Coergece rate heorem: the steepest descet equato algorthm coerge to the mmum of the cost characterzed b ormal equato: If A formal aalss of LS eed more math-mussels; practce, oe ca use a small α, or graduall decrease α. A Summar: LS update rule t+ j t j t ), + α( Pros: o-le, low per-step cost, fast coergece ad perhaps less proe to local optmum Cos: coergece to optmum ot alwas guarateed Steepest descet t + t t + α ( ) Pros: eas to mplemet, coceptuall clea, guarateed coergece Cos: batch, ofte slow coergg Normal equatos * ( ) Pros: a sgle-shot algorthm! Easest to mplemet. Cos: eed to compute pseudo-erse ( ) -, epese, umercal ssues (e.g., matr s sgular..), although there are was to get aroud ths
12 Geometrc Iterpretato of LS he predctos o the trag data are: Note that ˆ ( ( ) I ) ad ( ˆ ) ( ( ) I ) ( ( ) ) 0!! ŷ ˆ * ( ) s the orthogoal projecto of to the space spaed b the colums of 3 Probablstc Iterpretato of LS Let us assume that the target arable ad the puts are related b the equato: + where ε s a error term of umodeled effects or radom ose Now assume that ε follows a Gaussa N(0,σ), the we hae: p ε ( ) ep πσ σ ( ; ) B depedece assumpto: L( ) p( ; ) ep πσ ( ) σ 4
13 Probablstc Iterpretato of LS, cot. Hece the log-lkelhood s: l( ) log ( ) πσ σ Do ou recogze the last term? Yes t s: J ( ) ( ) hus uder depedece assumpto, LS s equalet to LE of! 5 Case stud: predctg gee epresso he geetc pcture causal SNPs CGCACGACAA a uarate pheotpe:.e., the epresso test of a gee 6 3
14 Assocato appg as Regresso Iddual Iddual Pheotpe (BI) Geotpe.. C C C..... A.. C G..... A.. G A..... C C Iddual d N G C G G Beg SNPs Causal SNP 7 Assocato appg as Regresso Pheotpe (BI) Geotpe Iddual Iddual Iddual d N J j j β j SNPs wth large β j are releat 8 4
15 Epermetal setup Asthama dataset 543 dduals, geotped at 34 SNPs Dplod data was trasformed to 0/ (for homozgotes) or (for heterozgotes) matr YPheotpe arable (cotuous) A sgle pheotpe was used for regresso Implemetato detals Iterate methods: Batch update ad ole update mplemeted. For both methods, step sze α s chose to be a small fed alue (0-6 ). hs choce s based o the data used for epermets. Both methods are ol ru to a mamum of 000 epochs or utl the chage trag SE s less tha Coergece Cures For the batch method, the trag SE s tall large due to uformed talzato I the ole update, N updates for eer epoch reduces SE to a much smaller alue. 30 5
16 he Leared Coeffcets 3 ultarate Regresso for rat Assocato Aalss rat Geotpe Assocato Stregth G A A C C A G A A G A.? β 3 6
17 ultarate Regresso for rat Assocato Aalss rat Geotpe Assocato Stregth G A A C C A G A A G A. a o-zero assocatos: Whch SNPs are trul sgfcat? 33 Sparst Oe commo assumpto to make sparst. akes bologcal sese: each pheotpe s lkel to be assocated wth a small umber of SNPs, rather tha all the SNPs. akes statstcal sese: Learg s ow feasble hgh dmesos wth small sample sze 34 7
18 Sparst: I a mathematcal sese Cosder least squares lear regresso problem: Sparst meas most of the beta s are zero. β β β β 3 β But ths s ot coe!!! a local optma, computatoall tractable. 35 L Regularzato (LASSO) (bshra, 996) A coe relaato. Costraed Form Lagraga Form Stll eforces sparst! 36 8
19 Lasso for Reducg False Postes rat Geotpe Assocato Stregth. G A A C C A G A A G A Lasso Pealt for sparst J + λ β j j a zero assocatos (sparse results), but what f there are multple related trats? 37 Rdge Regresso s Lasso Rdge Regresso: Lasso: HO! βs wth costat J(β) (leel sets of J(β)) βs wth costat l orm β β βs wth costat l orm β Lasso (l pealt) results sparse solutos ector wth more zero coordates Good for hgh dmesoal problems do t hae to store all coordates! 38 9
20 Baesa Iterpretato reat the dstrbuto parameters also as a radom arable he a posteror dstrbuto of after seem the data s: hs s Baes Rule p( D ) p( ) p( D) p( D) lkelhood pror posteror margal lkelhoodlh p( D ) p( ) p( D ) p( ) d he pror p(.) ecodes our pror kowledge about the doma 39 Regularzed Least Squares ad AP What f ( ) s ot ertble? log lkelhood log pror I) Gaussa Pror 0 Rdge Regresso Closed form: HW Pror belef that β s Gaussa wth zero mea bases soluto to small β 40 0
21 Regularzed Least Squares ad AP What f ( ) s ot ertble? log lkelhood log pror II) Laplace Pror Lasso Closed form: HW Pror belef that β s Laplace wth zero mea bases soluto to small β 4 Beod basc LR LR wth o-lear bass fuctos Locall weghted lear regresso Regresso trees ad ultlear Iterpolato 4
22 No-lear fuctos: 43 LR wth o-lear bass fuctos LR does ot mea we ca ol deal wth lear relatoshps We are free to desg (o-lear) features uder LR m 0 + φ( ) ( ) j j φ where the φ j () are fed bass fuctos (ad we defe φ 0 () ). Eample: polomal regresso: 3 [,, ] φ( ) :, We wll be cocered wth estmatg (dstrbutos oer) the weghts ad choosg the model order. 44
23 Bass fuctos here are ma bass fuctos, e.g.: Polomal l φ () Radal bass fuctos Sgmodal j j φ ( ) µ j φ j ( ) σ s j ep ( µ ) j s Sples, Fourer, Waelets, etc 45 D ad D RBFs D RBF After ft: 46 3
24 Good ad Bad RBFs A good D RBF wo bad D RBFs 47 Oerfttg ad uderfttg j j 0 j 48 4
25 Bas ad arace We defe the bas of a model to be the epected geeralzato error ee f we were to ft t to a er (sa, ftel) large trag set. B fttg "spurous" patters the trag set, we mght aga obta a model wth large geeralzato error. I ths case, we sa the model has large arace. 49 Locall weghted lear regresso he algorthm: Istead of mmzg J ( ) ( ) ow we ft to mmze J ( ) w ( ) Where do w 's come from? ( ) ep τ w where s the quer pot for whch we'd lke to kow ts correspodg Essetall we put hgher weghts o (errors o) trag eamples that are close to the quer pot (tha those that are further awa from the quer) 50 5
26 Parametrc s. o-parametrc Locall weghted lear regresso s the secod eample we are rug to of a o-parametrc algorthm. (what s the frst?) he (uweghted) lear regresso algorthm that we saw earler s kow as a parametrc learg algorthm because t has a fed, fte umber of parameters (the ), whch are ft to the data; Oce we'e ft the ad stored them awa, we o loger eed to keep the trag data aroud to make future predctos. I cotrast, to make predctos usg locall weghted lear regresso, we eed to keep the etre trag set aroud. he term "o-parametrc" (roughl) refers to the fact that the amout of stuff we eed to keep order to represet the hpothess grows learl wth the sze of the trag set. 5 Robust Regresso he best ft from a quadratc But ths s probabl better regresso How ca we do ths? 5 6
27 LOESS-based Robust Regresso Remember what we do "locall weghted lear regresso"? we "score" each pot for ts mpotece Now we score each pot accordg to ts "ftess" (Courtes to Adrew oor) 53 Robust regresso For k to R Let ( k, k ) be the kth datapot Let est k be predcted alue of k Let w k be a weght for data pot k that s large f the data pot fts well ad small f t fts badl: w k φ est ( ) ) k k he redo the regresso usg weghted data pots. Repeat whole thg utl coerged! 54 7
28 Robust regresso probablstc terpretato What regular regresso does: Assume k was orgall geerated usg the followg recpe: k k + N( 0, σ ) Computatoal task s to fd the amum Lkelhood estmato of 55 Robust regresso probablstc terpretato What LOESS robust regresso does: Assume k was orgall geerated usg the followg recpe: wth probablt p: k k + N( 0, σ ) but otherwse k ~ N ( µ, σ huge) Computatoal task s to fd the amum Lkelhood estmates of, p, µ ad σ huge. he algorthm ou saw wth terate reweghtg/refttg does ths computato for us. Later ou wll fd that t s a stace of the famous E.. algorthm 56 8
29 Regresso ree Decso tree for regresso Geder Rch? Num. Chldre # trael per r. Age Geder? F No 5 38 No 0 5 Yes 0 7 : : : : : Female Predcted age39 ale Predcted age36 57 A coceptual pcture Assumg regular regresso trees, ca ou sketch a graph of the ftted fucto *() () oer ths dagram? 58 9
30 How about ths oe? ultlear Iterpolato We wated to create a cotuous ad pecewse lear ft to the data 59 ake home message Gradet descet O-le Batch Normal equatos Equalece of LS ad LE LR does ot mea fttg lear relatos, but lear Wdows arketplace combato or bass fuctos (that ca be olear) Weghtg pots b mportace ersus b ftess 60 30
Machine Learning. Introduction to Regression. Le Song. CSE6740/CS7641/ISYE6740, Fall 2012
Mache Learg CSE6740/CS764/ISYE6740, Fall 0 Itroducto to Regresso Le Sog Lecture 4, August 30, 0 Based o sldes from Erc g, CMU Readg: Chap. 3, CB Mache learg for apartmet hutg Suppose ou are to move to
More informationAdvanced Introduction to Machine Learning
Advaced Itroducto to Mache Learg 075, Fall 04 Lear Regresso ad Sparst Erc g Lecture, September 0, 04 Readg: Erc g @ CMU, 04 Mache learg for apartmet hutg ow ou've moved to Pttsburgh!! Ad ou wat to fd the
More informationCS 2750 Machine Learning. Lecture 8. Linear regression. CS 2750 Machine Learning. Linear regression. is a linear combination of input components x
CS 75 Mache Learg Lecture 8 Lear regresso Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Learg Lear regresso Fucto f : X Y s a lear combato of put compoets f + + + K d d K k - parameters
More informationGenerative classification models
CS 75 Mache Learg Lecture Geeratve classfcato models Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Data: D { d, d,.., d} d, Classfcato represets a dscrete class value Goal: lear f : X Y Bar classfcato
More informationKernel-based Methods and Support Vector Machines
Kerel-based Methods ad Support Vector Maches Larr Holder CptS 570 Mache Learg School of Electrcal Egeerg ad Computer Scece Washgto State Uverst Refereces Muller et al. A Itroducto to Kerel-Based Learg
More informationBayes (Naïve or not) Classifiers: Generative Approach
Logstc regresso Bayes (Naïve or ot) Classfers: Geeratve Approach What do we mea by Geeratve approach: Lear p(y), p(x y) ad the apply bayes rule to compute p(y x) for makg predctos Ths s essetally makg
More informationDimensionality Reduction and Learning
CMSC 35900 (Sprg 009) Large Scale Learg Lecture: 3 Dmesoalty Reducto ad Learg Istructors: Sham Kakade ad Greg Shakharovch L Supervsed Methods ad Dmesoalty Reducto The theme of these two lectures s that
More informationCS 1675 Introduction to Machine Learning Lecture 12 Support vector machines
CS 675 Itroducto to Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Mdterm eam October 9, 7 I-class eam Closed book Stud materal: Lecture otes Correspodg chapters
More informationSupervised learning: Linear regression Logistic regression
CS 57 Itroducto to AI Lecture 4 Supervsed learg: Lear regresso Logstc regresso Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 57 Itro to AI Data: D { D D.. D D Supervsed learg d a set of eamples s
More information( ) = ( ) ( ) Chapter 13 Asymptotic Theory and Stochastic Regressors. Stochastic regressors model
Chapter 3 Asmptotc Theor ad Stochastc Regressors The ature of eplaator varable s assumed to be o-stochastc or fed repeated samples a regresso aalss Such a assumpto s approprate for those epermets whch
More informationBinary classification: Support Vector Machines
CS 57 Itroducto to AI Lecture 6 Bar classfcato: Support Vector Maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 57 Itro to AI Supervsed learg Data: D { D, D,.., D} a set of eamples D, (,,,,,
More informationObjectives of Multiple Regression
Obectves of Multple Regresso Establsh the lear equato that best predcts values of a depedet varable Y usg more tha oe eplaator varable from a large set of potetal predctors {,,... k }. Fd that subset of
More informationLinear Regression Linear Regression with Shrinkage. Some slides are due to Tommi Jaakkola, MIT AI Lab
Lear Regresso Lear Regresso th Shrkage Some sldes are due to Tomm Jaakkola, MIT AI Lab Itroducto The goal of regresso s to make quattatve real valued predctos o the bass of a vector of features or attrbutes.
More informationSupport vector machines II
CS 75 Mache Learg Lecture Support vector maches II Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square Learl separable classes Learl separable classes: here s a hperplae that separates trag staces th o error
More informationSTA 108 Applied Linear Models: Regression Analysis Spring Solution for Homework #1
STA 08 Appled Lear Models: Regresso Aalyss Sprg 0 Soluto for Homework #. Let Y the dollar cost per year, X the umber of vsts per year. The the mathematcal relato betwee X ad Y s: Y 300 + X. Ths s a fuctoal
More informationPoint Estimation: definition of estimators
Pot Estmato: defto of estmators Pot estmator: ay fucto W (X,..., X ) of a data sample. The exercse of pot estmato s to use partcular fuctos of the data order to estmate certa ukow populato parameters.
More informationLecture 8: Linear Regression
Lecture 8: Lear egresso May 4, GENOME 56, Sprg Goals Develop basc cocepts of lear regresso from a probablstc framework Estmatg parameters ad hypothess testg wth lear models Lear regresso Su I Lee, CSE
More informationi 2 σ ) i = 1,2,...,n , and = 3.01 = 4.01
ECO 745, Homework 6 Le Cabrera. Assume that the followg data come from the lear model: ε ε ~ N, σ,,..., -6. -.5 7. 6.9 -. -. -.9. -..6.4.. -.6 -.7.7 Fd the mamum lkelhood estmates of,, ad σ ε s.6. 4. ε
More informationRegression and the LMS Algorithm
CSE 556: Itroducto to Neural Netorks Regresso ad the LMS Algorthm CSE 556: Regresso 1 Problem statemet CSE 556: Regresso Lear regresso th oe varable Gve a set of N pars of data {, d }, appromate d b a
More informationCS 2750 Machine Learning. Lecture 7. Linear regression. CS 2750 Machine Learning. Linear regression. is a linear combination of input components x
CS 75 Mache Learg Lecture 7 Lear regresso Mlos Hauskrecht los@cs.ptt.edu 59 Seott Square CS 75 Mache Learg Lear regresso Fucto f : X Y s a lear cobato of put copoets f + + + K d d K k - paraeters eghts
More informationChapter 4 (Part 1): Non-Parametric Classification (Sections ) Pattern Classification 4.3) Announcements
Aoucemets No-Parametrc Desty Estmato Techques HW assged Most of ths lecture was o the blacboard. These sldes cover the same materal as preseted DHS Bometrcs CSE 90-a Lecture 7 CSE90a Fall 06 CSE90a Fall
More informationModel Fitting, RANSAC. Jana Kosecka
Model Fttg, RANSAC Jaa Kosecka Fttg: Issues Prevous strateges Le detecto Hough trasform Smple parametrc model, two parameters m, b m + b Votg strateg Hard to geeralze to hgher dmesos a o + a + a 2 2 +
More informationSupport vector machines
CS 75 Mache Learg Lecture Support vector maches Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Learg Outle Outle: Algorthms for lear decso boudary Support vector maches Mamum marg hyperplae.
More informationECON 482 / WH Hong The Simple Regression Model 1. Definition of the Simple Regression Model
ECON 48 / WH Hog The Smple Regresso Model. Defto of the Smple Regresso Model Smple Regresso Model Expla varable y terms of varable x y = β + β x+ u y : depedet varable, explaed varable, respose varable,
More informationRadial Basis Function Networks
Radal Bass Fucto Netorks Radal Bass Fucto Netorks A specal types of ANN that have three layers Iput layer Hdde layer Output layer Mappg from put to hdde layer s olear Mappg from hdde to output layer s
More informationIntroduction to local (nonparametric) density estimation. methods
Itroducto to local (oparametrc) desty estmato methods A slecture by Yu Lu for ECE 66 Sprg 014 1. Itroducto Ths slecture troduces two local desty estmato methods whch are Parze desty estmato ad k-earest
More informationLecture Notes 2. The ability to manipulate matrices is critical in economics.
Lecture Notes. Revew of Matrces he ablt to mapulate matrces s crtcal ecoomcs.. Matr a rectagular arra of umbers, parameters, or varables placed rows ad colums. Matrces are assocated wth lear equatos. lemets
More informationEconometric Methods. Review of Estimation
Ecoometrc Methods Revew of Estmato Estmatg the populato mea Radom samplg Pot ad terval estmators Lear estmators Ubased estmators Lear Ubased Estmators (LUEs) Effcecy (mmum varace) ad Best Lear Ubased Estmators
More informationPGE 310: Formulation and Solution in Geosystems Engineering. Dr. Balhoff. Interpolation
PGE 30: Formulato ad Soluto Geosystems Egeerg Dr. Balhoff Iterpolato Numercal Methods wth MATLAB, Recktewald, Chapter 0 ad Numercal Methods for Egeers, Chapra ad Caale, 5 th Ed., Part Fve, Chapter 8 ad
More informationLecture Notes Forecasting the process of estimating or predicting unknown situations
Lecture Notes. Ecoomc Forecastg. Forecastg the process of estmatg or predctg ukow stuatos Eample usuall ecoomsts predct future ecoomc varables Forecastg apples to a varet of data () tme seres data predctg
More informationFeature Selection: Part 2. 1 Greedy Algorithms (continued from the last lecture)
CSE 546: Mache Learg Lecture 6 Feature Selecto: Part 2 Istructor: Sham Kakade Greedy Algorthms (cotued from the last lecture) There are varety of greedy algorthms ad umerous amg covetos for these algorthms.
More informationLecture 7. Confidence Intervals and Hypothesis Tests in the Simple CLR Model
Lecture 7. Cofdece Itervals ad Hypothess Tests the Smple CLR Model I lecture 6 we troduced the Classcal Lear Regresso (CLR) model that s the radom expermet of whch the data Y,,, K, are the outcomes. The
More informationBayesian Classification. CS690L Data Mining: Classification(2) Bayesian Theorem: Basics. Bayesian Theorem. Training dataset. Naïve Bayes Classifier
Baa Classfcato CS6L Data Mg: Classfcato() Referece: J. Ha ad M. Kamber, Data Mg: Cocepts ad Techques robablstc learg: Calculate explct probabltes for hypothess, amog the most practcal approaches to certa
More informationUnsupervised Learning and Other Neural Networks
CSE 53 Soft Computg NOT PART OF THE FINAL Usupervsed Learg ad Other Neural Networs Itroducto Mture Destes ad Idetfablty ML Estmates Applcato to Normal Mtures Other Neural Networs Itroducto Prevously, all
More informationLecture 9: Tolerant Testing
Lecture 9: Tolerat Testg Dael Kae Scrbe: Sakeerth Rao Aprl 4, 07 Abstract I ths lecture we prove a quas lear lower boud o the umber of samples eeded to do tolerat testg for L dstace. Tolerat Testg We have
More informationChapter 14 Logistic Regression Models
Chapter 4 Logstc Regresso Models I the lear regresso model X β + ε, there are two types of varables explaatory varables X, X,, X k ad study varable y These varables ca be measured o a cotuous scale as
More informationLecture 3. Least Squares Fitting. Optimization Trinity 2014 P.H.S.Torr. Classic least squares. Total least squares.
Lecture 3 Optmzato Trt 04 P.H.S.Torr Least Squares Fttg Classc least squares Total least squares Robust Estmato Fttg: Cocepts ad recpes Least squares le fttg Data:,,,, Le equato: = m + b Fd m, b to mmze
More informationOverview. Basic concepts of Bayesian learning. Most probable model given data Coin tosses Linear regression Logistic regression
Overvew Basc cocepts of Bayesa learg Most probable model gve data Co tosses Lear regresso Logstc regresso Bayesa predctos Co tosses Lear regresso 30 Recap: regresso problems Iput to learg problem: trag
More informationLine Fitting and Regression
Marquette Uverst MSCS6 Le Fttg ad Regresso Dael B. Rowe, Ph.D. Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Coprght 8 b Marquette Uverst Least Squares Regresso MSCS6 For LSR we have pots
More informationLecture 16: Backpropogation Algorithm Neural Networks with smooth activation functions
CO-511: Learg Theory prg 2017 Lecturer: Ro Lv Lecture 16: Bacpropogato Algorthm Dsclamer: These otes have ot bee subected to the usual scruty reserved for formal publcatos. They may be dstrbuted outsde
More informationCHAPTER VI Statistical Analysis of Experimental Data
Chapter VI Statstcal Aalyss of Expermetal Data CHAPTER VI Statstcal Aalyss of Expermetal Data Measuremets do ot lead to a uque value. Ths s a result of the multtude of errors (maly radom errors) that ca
More informationMultiple Choice Test. Chapter Adequacy of Models for Regression
Multple Choce Test Chapter 06.0 Adequac of Models for Regresso. For a lear regresso model to be cosdered adequate, the percetage of scaled resduals that eed to be the rage [-,] s greater tha or equal to
More informationL5 Polynomial / Spline Curves
L5 Polyomal / Sple Curves Cotets Coc sectos Polyomal Curves Hermte Curves Bezer Curves B-Sples No-Uform Ratoal B-Sples (NURBS) Mapulato ad Represetato of Curves Types of Curve Equatos Implct: Descrbe a
More informationChapter Two. An Introduction to Regression ( )
ubject: A Itroducto to Regresso Frst tage Chapter Two A Itroducto to Regresso (018-019) 1 pg. ubject: A Itroducto to Regresso Frst tage A Itroducto to Regresso Regresso aalss s a statstcal tool for the
More informationSection 2 Notes. Elizabeth Stone and Charles Wang. January 15, Expectation and Conditional Expectation of a Random Variable.
Secto Notes Elzabeth Stoe ad Charles Wag Jauar 5, 9 Jot, Margal, ad Codtoal Probablt Useful Rules/Propertes. P ( x) P P ( x; ) or R f (x; ) d. P ( xj ) P (x; ) P ( ) 3. P ( x; ) P ( xj ) P ( ) 4. Baes
More informationLecture 3 Probability review (cont d)
STATS 00: Itroducto to Statstcal Iferece Autum 06 Lecture 3 Probablty revew (cot d) 3. Jot dstrbutos If radom varables X,..., X k are depedet, the ther dstrbuto may be specfed by specfyg the dvdual dstrbuto
More information9.1 Introduction to the probit and logit models
EC3000 Ecoometrcs Lecture 9 Probt & Logt Aalss 9. Itroducto to the probt ad logt models 9. The logt model 9.3 The probt model Appedx 9. Itroducto to the probt ad logt models These models are used regressos
More informationMaximum Likelihood Estimation
Marquette Uverst Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Coprght 08 b Marquette Uverst Maxmum Lkelhood Estmato We have bee sag that ~
More informationCSE 5526: Introduction to Neural Networks Linear Regression
CSE 556: Itroducto to Neural Netorks Lear Regresso Part II 1 Problem statemet Part II Problem statemet Part II 3 Lear regresso th oe varable Gve a set of N pars of data , appromate d by a lear fucto
More informationSimple Linear Regression
Statstcal Methods I (EST 75) Page 139 Smple Lear Regresso Smple regresso applcatos are used to ft a model descrbg a lear relatoshp betwee two varables. The aspects of least squares regresso ad correlato
More informationQR Factorization and Singular Value Decomposition COS 323
QR Factorzato ad Sgular Value Decomposto COS 33 Why Yet Aother Method? How do we solve least-squares wthout currg codto-squarg effect of ormal equatos (A T A A T b) whe A s sgular, fat, or otherwse poorly-specfed?
More information6. Nonparametric techniques
6. Noparametrc techques Motvato Problem: how to decde o a sutable model (e.g. whch type of Gaussa) Idea: just use the orgal data (lazy learg) 2 Idea 1: each data pot represets a pece of probablty P(x)
More informationBig Data Analytics. Data Fitting and Sampling. Acknowledgement: Notes by Profs. R. Szeliski, S. Seitz, S. Lazebnik, K. Chaturvedi, and S.
Bg Data Aaltcs Data Fttg ad Samplg Ackowledgemet: Notes b Profs. R. Szelsk, S. Setz, S. Lazebk, K. Chaturved, ad S. Shah Fttg: Cocepts ad recpes A bag of techques If we kow whch pots belog to the le, how
More informationSTATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1
STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Recall Assumpto E(Y x) η 0 + η x (lear codtoal mea fucto) Data (x, y ), (x 2, y 2 ),, (x, y ) Least squares estmator ˆ E (Y x) ˆ " 0 + ˆ " x, where ˆ
More informationDimensionality reduction Feature selection
CS 750 Mache Learg Lecture 3 Dmesoalty reducto Feature selecto Mlos Hauskrecht mlos@cs.ptt.edu 539 Seott Square CS 750 Mache Learg Dmesoalty reducto. Motvato. Classfcato problem eample: We have a put data
More information13. Parametric and Non-Parametric Uncertainties, Radial Basis Functions and Neural Network Approximations
Lecture 7 3. Parametrc ad No-Parametrc Ucertates, Radal Bass Fuctos ad Neural Network Approxmatos he parameter estmato algorthms descrbed prevous sectos were based o the assumpto that the system ucertates
More information4. Standard Regression Model and Spatial Dependence Tests
4. Stadard Regresso Model ad Spatal Depedece Tests Stadard regresso aalss fals the presece of spatal effects. I case of spatal depedeces ad/or spatal heterogeet a stadard regresso model wll be msspecfed.
More informationKLT Tracker. Alignment. 1. Detect Harris corners in the first frame. 2. For each Harris corner compute motion between consecutive frames
KLT Tracker Tracker. Detect Harrs corers the frst frame 2. For each Harrs corer compute moto betwee cosecutve frames (Algmet). 3. Lk moto vectors successve frames to get a track 4. Itroduce ew Harrs pots
More informationCorrelation and Simple Linear Regression
Correlato ad Smple Lear Regresso Berl Che Departmet of Computer Scece & Iformato Egeerg Natoal Tawa Normal Uverst Referece:. W. Navd. Statstcs for Egeerg ad Scetsts. Chapter 7 (7.-7.3) & Teachg Materal
More informationESS Line Fitting
ESS 5 014 17. Le Fttg A very commo problem data aalyss s lookg for relatoshpetwee dfferet parameters ad fttg les or surfaces to data. The smplest example s fttg a straght le ad we wll dscuss that here
More informationENGI 4421 Joint Probability Distributions Page Joint Probability Distributions [Navidi sections 2.5 and 2.6; Devore sections
ENGI 441 Jot Probablty Dstrbutos Page 7-01 Jot Probablty Dstrbutos [Navd sectos.5 ad.6; Devore sectos 5.1-5.] The jot probablty mass fucto of two dscrete radom quattes, s, P ad p x y x y The margal probablty
More informationMultivariate Transformation of Variables and Maximum Likelihood Estimation
Marquette Uversty Multvarate Trasformato of Varables ad Maxmum Lkelhood Estmato Dael B. Rowe, Ph.D. Assocate Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Copyrght 03 by Marquette Uversty
More informationAn Introduction to. Support Vector Machine
A Itroducto to Support Vector Mache Support Vector Mache (SVM) A classfer derved from statstcal learg theory by Vapk, et al. 99 SVM became famous whe, usg mages as put, t gave accuracy comparable to eural-etwork
More informationENGI 3423 Simple Linear Regression Page 12-01
ENGI 343 mple Lear Regresso Page - mple Lear Regresso ometmes a expermet s set up where the expermeter has cotrol over the values of oe or more varables X ad measures the resultg values of aother varable
More informationECON 5360 Class Notes GMM
ECON 560 Class Notes GMM Geeralzed Method of Momets (GMM) I beg by outlg the classcal method of momets techque (Fsher, 95) ad the proceed to geeralzed method of momets (Hase, 98).. radtoal Method of Momets
More informationRecall MLR 5 Homskedasticity error u has the same variance given any values of the explanatory variables Var(u x1,...,xk) = 2 or E(UU ) = 2 I
Chapter 8 Heterosedastcty Recall MLR 5 Homsedastcty error u has the same varace gve ay values of the eplaatory varables Varu,..., = or EUU = I Suppose other GM assumptos hold but have heterosedastcty.
More informationCorrelation and Regression Analysis
Chapter V Correlato ad Regresso Aalss R. 5.. So far we have cosdered ol uvarate dstrbutos. Ma a tme, however, we come across problems whch volve two or more varables. Ths wll be the subject matter of the
More informationMultiple Linear Regression Analysis
LINEA EGESSION ANALYSIS MODULE III Lecture - 4 Multple Lear egresso Aalyss Dr. Shalabh Departmet of Mathematcs ad Statstcs Ida Isttute of Techology Kapur Cofdece terval estmato The cofdece tervals multple
More informationLecture 1: Introduction to Regression
Lecture : Itroducto to Regresso A Eample: Eplag State Homcde Rates What kds of varables mght we use to epla/predct state homcde rates? Let s cosder just oe predctor for ow: povert Igore omtted varables,
More informationRegresso What s a Model? 1. Ofte Descrbe Relatoshp betwee Varables 2. Types - Determstc Models (o radomess) - Probablstc Models (wth radomess) EPI 809/Sprg 2008 9 Determstc Models 1. Hypothesze
More informationLecture Notes Types of economic variables
Lecture Notes 3 1. Types of ecoomc varables () Cotuous varable takes o a cotuum the sample space, such as all pots o a le or all real umbers Example: GDP, Polluto cocetrato, etc. () Dscrete varables fte
More information0/1 INTEGER PROGRAMMING AND SEMIDEFINTE PROGRAMMING
CONVEX OPIMIZAION AND INERIOR POIN MEHODS FINAL PROJEC / INEGER PROGRAMMING AND SEMIDEFINE PROGRAMMING b Luca Buch ad Natala Vktorova CONENS:.Itroducto.Formulato.Applcato to Kapsack Problem 4.Cuttg Plaes
More informationThe Mathematical Appendix
The Mathematcal Appedx Defto A: If ( Λ, Ω, where ( λ λ λ whch the probablty dstrbutos,,..., Defto A. uppose that ( Λ,,..., s a expermet type, the σ-algebra o λ λ λ are defed s deoted by ( (,,...,, σ Ω.
More informationLecture 3. Sampling, sampling distributions, and parameter estimation
Lecture 3 Samplg, samplg dstrbutos, ad parameter estmato Samplg Defto Populato s defed as the collecto of all the possble observatos of terest. The collecto of observatos we take from the populato s called
More informationOrdinary Least Squares Regression. Simple Regression. Algebra and Assumptions.
Ordary Least Squares egresso. Smple egresso. Algebra ad Assumptos. I ths part of the course we are gog to study a techque for aalysg the lear relatoshp betwee two varables Y ad X. We have pars of observatos
More informationBayesian belief networks
Lecture 14 ayesa belef etworks los Hauskrecht mlos@cs.ptt.edu 5329 Seott Square Desty estmato Data: D { D1 D2.. D} D x a vector of attrbute values ttrbutes: modeled by radom varables { 1 2 d} wth: otuous
More informationStatistics MINITAB - Lab 5
Statstcs 10010 MINITAB - Lab 5 PART I: The Correlato Coeffcet Qute ofte statstcs we are preseted wth data that suggests that a lear relatoshp exsts betwee two varables. For example the plot below s of
More informationParametric Density Estimation: Bayesian Estimation. Naïve Bayes Classifier
arametrc Dest Estmato: Baesa Estmato. Naïve Baes Classfer Baesa arameter Estmato Suppose we have some dea of the rage where parameters θ should be Should t we formalze such pror owledge hopes that t wll
More information7. Joint Distributions
7. Jot Dstrbutos Chrs Pech ad Mehra Saham Ma 2017 Ofte ou wll work o problems where there are several radom varables (ofte teractg wth oe aother. We are gog to start to formall look at how those teractos
More informationChapter 13 Student Lecture Notes 13-1
Chapter 3 Studet Lecture Notes 3- Basc Busess Statstcs (9 th Edto) Chapter 3 Smple Lear Regresso 4 Pretce-Hall, Ic. Chap 3- Chapter Topcs Types of Regresso Models Determg the Smple Lear Regresso Equato
More informationExample: Multiple linear regression. Least squares regression. Repetition: Simple linear regression. Tron Anders Moger
Example: Multple lear regresso 5000,00 4000,00 Tro Aders Moger 0.0.007 brthweght 3000,00 000,00 000,00 0,00 50,00 00,00 50,00 00,00 50,00 weght pouds Repetto: Smple lear regresso We defe a model Y = β0
More informationMultiple Regression. More than 2 variables! Grade on Final. Multiple Regression 11/21/2012. Exam 2 Grades. Exam 2 Re-grades
STAT 101 Dr. Kar Lock Morga 11/20/12 Exam 2 Grades Multple Regresso SECTIONS 9.2, 10.1, 10.2 Multple explaatory varables (10.1) Parttog varablty R 2, ANOVA (9.2) Codtos resdual plot (10.2) Trasformatos
More informationAssignment 5/MATH 247/Winter Due: Friday, February 19 in class (!) (answers will be posted right after class)
Assgmet 5/MATH 7/Wter 00 Due: Frday, February 9 class (!) (aswers wll be posted rght after class) As usual, there are peces of text, before the questos [], [], themselves. Recall: For the quadratc form
More informationMidterm Exam 1, section 2 (Solution) Thursday, February hour, 15 minutes
coometrcs, CON Sa Fracsco State Uverst Mchael Bar Sprg 5 Mdterm xam, secto Soluto Thursda, Februar 6 hour, 5 mutes Name: Istructos. Ths s closed book, closed otes exam.. No calculators of a kd are allowed..
More informationMATH 247/Winter Notes on the adjoint and on normal operators.
MATH 47/Wter 00 Notes o the adjot ad o ormal operators I these otes, V s a fte dmesoal er product space over, wth gve er * product uv, T, S, T, are lear operators o V U, W are subspaces of V Whe we say
More informationPart I: Background on the Binomial Distribution
Part I: Bacgroud o the Bomal Dstrbuto A radom varable s sad to have a Beroull dstrbuto f t taes o the value wth probablt "p" ad the value wth probablt " - p". The umber of "successes" "" depedet Beroull
More informationChapter 8. Inferences about More Than Two Population Central Values
Chapter 8. Ifereces about More Tha Two Populato Cetral Values Case tudy: Effect of Tmg of the Treatmet of Port-We tas wth Lasers ) To vestgate whether treatmet at a youg age would yeld better results tha
More informationClassification : Logistic regression. Generative classification model.
CS 75 Mache Lear Lecture 8 Classfcato : Lostc reresso. Geeratve classfcato model. Mlos Hausrecht mlos@cs.ptt.edu 539 Seott Square CS 75 Mache Lear Bar classfcato o classes Y {} Our oal s to lear to classf
More informationSampling Theory MODULE V LECTURE - 14 RATIO AND PRODUCT METHODS OF ESTIMATION
Samplg Theor MODULE V LECTUE - 4 ATIO AND PODUCT METHODS OF ESTIMATION D. SHALABH DEPATMENT OF MATHEMATICS AND STATISTICS INDIAN INSTITUTE OF TECHNOLOG KANPU A mportat objectve a statstcal estmato procedure
More informationStatistics. Correlational. Dr. Ayman Eldeib. Simple Linear Regression and Correlation. SBE 304: Linear Regression & Correlation 1/3/2018
/3/08 Sstems & Bomedcal Egeerg Departmet SBE 304: Bo-Statstcs Smple Lear Regresso ad Correlato Dr. Ama Eldeb Fall 07 Descrptve Orgasg, summarsg & descrbg data Statstcs Correlatoal Relatoshps Iferetal Geeralsg
More informationTESTS BASED ON MAXIMUM LIKELIHOOD
ESE 5 Toy E. Smth. The Basc Example. TESTS BASED ON MAXIMUM LIKELIHOOD To llustrate the propertes of maxmum lkelhood estmates ad tests, we cosder the smplest possble case of estmatg the mea of the ormal
More informationSummary of the lecture in Biostatistics
Summary of the lecture Bostatstcs Probablty Desty Fucto For a cotuos radom varable, a probablty desty fucto s a fucto such that: 0 dx a b) b a dx A probablty desty fucto provdes a smple descrpto of the
More informationLecture 1: Introduction to Regression
Lecture : Itroducto to Regresso A Eample: Eplag State Homcde Rates What kds of varables mght we use to epla/predct state homcde rates? Let s cosder just oe predctor for ow: povert Igore omtted varables,
More informationSTK3100 and STK4100 Autumn 2018
SK3 ad SK4 Autum 8 Geeralzed lear models Part III Covers the followg materal from chaters 4 ad 5: Cofdece tervals by vertg tests Cosder a model wth a sgle arameter β We may obta a ( α% cofdece terval for
More informationChapter 5 Transformation and Weighting to Correct Model Inadequacies
Chapter 5 Trasformato ad Weghtg to Correct Model Iadequaces The graphcal methods help detectg the volato of basc assumptos regresso aalss. Now we cosder the methods ad procedures for buldg the models through
More information9 U-STATISTICS. Eh =(m!) 1 Eh(X (1),..., X (m ) ) i.i.d
9 U-STATISTICS Suppose,,..., are P P..d. wth CDF F. Our goal s to estmate the expectato t (P)=Eh(,,..., m ). Note that ths expectato requres more tha oe cotrast to E, E, or Eh( ). Oe example s E or P((,
More informationCLASS NOTES. for. PBAF 528: Quantitative Methods II SPRING Instructor: Jean Swanson. Daniel J. Evans School of Public Affairs
CLASS NOTES for PBAF 58: Quattatve Methods II SPRING 005 Istructor: Jea Swaso Dael J. Evas School of Publc Affars Uversty of Washgto Ackowledgemet: The structor wshes to thak Rachel Klet, Assstat Professor,
More information{ }{ ( )} (, ) = ( ) ( ) ( ) Chapter 14 Exercises in Sampling Theory. Exercise 1 (Simple random sampling): Solution:
Chapter 4 Exercses Samplg Theory Exercse (Smple radom samplg: Let there be two correlated radom varables X ad A sample of sze s draw from a populato by smple radom samplg wthout replacemet The observed
More informationA Study of the Reproducibility of Measurements with HUR Leg Extension/Curl Research Line
HUR Techcal Report 000--9 verso.05 / Frak Borg (borgbros@ett.f) A Study of the Reproducblty of Measuremets wth HUR Leg Eteso/Curl Research Le A mportat property of measuremets s that the results should
More information6.867 Machine Learning
6.867 Mache Learg Problem set Due Frday, September 9, rectato Please address all questos ad commets about ths problem set to 6.867-staff@a.mt.edu. You do ot eed to use MATLAB for ths problem set though
More information