Maximum Likelihood Estimation (MLE)
|
|
- Morgan Reynolds
- 6 years ago
- Views:
Transcription
1 Maxmum Lkelhood Estmaton (MLE) Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 01 UCSD
2 Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y ), fnd an approxmatng functon f (x) y x f () ŷ f( x) y hs s called tranng or learnng. wo major types of learnng: Unsupervsed Classfcaton (aka Clusterng) or Regresson ( blnd curve fttng): only X s known. Supervsed Classfcaton or Regresson: both X and target value Y are known durng tranng, only X s known at test tme.
3 Optmal Classfers Performance depends on the data/feature space metrc Some metrcs are better than others he meanng of better s connected to how well adapted the metrc s to the propertes of the data But can we be more rgorous? What do we mean by optmal? o talk about optmalty we need to talk about cost or loss x yˆ f ( x) () f L( y, yˆ ) Average Loss (Rsk) s the functon that we want to mnmze Rsk depends on true y and the predcton ells us how good our predctor/estmator s ŷ 3
4 Data-Condtonal Rsk, R(x,), for 0/1 Loss An mportant specal case of nterest: zero loss for no error and equal loss for two error types hs s equvalent to the zero/one loss : L j 0 1 j j snake predcton dart frog regular frog regular 1 0 dart 0 1 Under ths loss * x L j PYX j x j ( ) arg mn ( ) arg mn P ( j x) j YX 4
5 Data-Condtonal Rsk, R(x,), for 0/1 Loss Note, then, that n the 0/1 loss case, ] j R( x, ) E L[ Y x P ( Y j x). I.e., the data-condtonal rsk under the 0/1 loss s equal to the data-condtonal Probablty of Error, hus the optmal Bayesan decson rule (BDR) under 0/1 loss mnmzes the condtonal probablty of error. hs s gven by the MAP BDR : Y X R( x, ) P( x) 1 P( x) * ( x) argma x P( x). 5
6 Data-Condtonal Rsk, R(x,), for 0/1 Loss Summarzng: * x PYX j x j ( ) arg mn ( ) arg mn 1 PYX ( x) arg max P ( x) YX he optmal decson rule s the MAP Rule : Pck the class wth largest probablty gven the observaton x hs the Bayes Decson Rule (BDR) for the 0/1 loss We wll often smplfy our dscusson by assumng ths loss But you should always be aware that other losses may be used 6
7 BDR (under 0/1 Loss) For the zero/one loss, the followng three decson rules are optmal and equvalent 1) ) 3) * x PYX x ( ) arg max ( ) ( ) argmax ( ) ( ) * x PX Y x PY ( ) arg max log ( ) log ( ) * x PX Y x PY Form 1) s usually hard to use, 3) s frequently easer than ) 7
8 Gaussan BDR Classfer (0/1 Loss) A very mportant case s that of Gaussan classes he pdf of each class s a Gaussan of mean m and covarance S 1 1 P x x x 1 X Y ( ) exp ( m ) S ( m ) d ( ) S he Gaussan BDR under 0/1 Loss s * 1 1 ( x) argmax ( ) ( ) x m S x m 1 log( ) d S log PY ( ) 8
9 Gaussan Classfer (0/1 Loss) hs can be wrtten as ( x) argmn d ( x, m ) a * ( ) = 0.5 wth d ( x, y) ( x y) S ( x y) 1 a d log( ) S log P ( ) Y and can be nterpeted as a nearest class-neghbor classfer whch uses a funny metrc Note that each class has ts own dstance functon whch s related to the sum of the square of the Mahalanobs dstance for that class plus the a term for that class we effectvely use dfferent metrcs n dfferent regons of the space 9
10 Gaussan Classfer (0/1 Loss) A specal case of nterest s when all classes have the same S =S ( ) = 0.5 x d x m a * ( ) argmn (, ) wth d x y x y x y Note: 1 (, ) ( ) S ( ) a log P ( ) Y a can be dropped when all classes have equal probablty (the case shown n the above fgure). In ths case the classfer s close n form to a NN classfer wth Mahalanobs dstance, but nstead of fndng the nearest tranng data pont, t looks for the nearest class prototype m usng the Mahalanobs dstance 10
11 Gaussan Classfer (0/1 Loss) Bnary Classfcaton wth S =S One mportant property of ths case s that the decson boundary s a hyperplane (Homework) hs can be shown by computng the set of ponts x such that d ( x, m ) a d ( x, m ) a and showng that they satsfy dscrmnant for ( ) = 0.5. w ( x x ) 0. 0 hs s the equaton of a hyperplane wth normal w. x 0 can be any fxed pont on the hyperplane, but t s standard to choose t to have mnmum norm, n whch case w and x 0 are then parallel x n x 1 x 3 x x x 0 0 x w 11
12 Gaussan Classfer (0/1 Loss) Furthermore, f all the covarances are the dentty S =I x d x m a * ( ) argmn (, ) wth d ( x, y) x y a log P ( ) hs s just Eucldean Dstance emplate Matchng wth class means as templates E.g. for dgt classfcaton Y *? Compare complexty to nearest neghbors! 1
13 he Sgmod n 0/1 Loss Detecton We have derved all of ths from the log-based 0/1 BDR ( ) arg max log ( ) log ( ) * x PX Y x PY When there are only two classes, t s also nterestng to look at the orgnal defnton n an alternatve form: wth * ( x) argmax g ( x) g ( x) P ( x) Y X P ( x ) P ( ) X Y X Y P ( x) P ( x ) P ( ) P ( x 0) P (0) P ( x 1) P (1) X Y Y X Y Y X Y Y 13
14 he Sgmod n MAP Detecton Note that ths can be wrtten as * ( x) argmax g ( x) g ( x) 1 g ( x) 1 0 g 0 ( x) 1 1 P ( x 1) P (1) X Y P ( x 0) P (0) X Y Y Y For Gaussan classes, the posteror probablty for 0 s g 0 1 ( x) 1 exp d ( x, m ) d ( x, m ) a a where, as before, d ( x, y) ( x y) S ( x y) 1 a d log( ) S log P ( ) Y 14
15 he Sgmod n MAP detecton he posteror densty for class 0, g 0 1 ( x) 1 exp d ( x, m ) d ( x, m ) a a s a sgmod and looks lke ths ( 1 ) =
16 he Sgmod n Neural Networks he sgmod functon also appears n neural networks In neural networks, t can be nterpreted as a posteror densty for a Gaussan problem where the covarances are the same. 16
17 he Sgmod n Neural Networks But not necessarly when the covarances are dfferent 17
18 Implementaton All of ths s appealng, but n practce one doesn t know the values of the parameters m, S, P Y (1) In the homework we use an ntutve soluton to desgn a Gaussan classfer: Start from a collecton of datasets: D () = {x 1 (),..., x n () } = set of examples from class For each class estmate the Gaussan BDR parameters usng, 1 () 1 ˆ m x j S Pˆ Y () n j ˆ ( ) ( ) ( ˆ )( ˆ xj m xj m ) n j where s the total number of examples (over all classes) E.g., below are sample means computed for dgt classfcaton: n 18
19 A Practcal Gaussan MAP Classfer Instead of the deal BDR * 1 1 ( x) argmax ( ) ( ) x m S x m 1 log( ) d S log PY ( ) use the estmate of the BDR found from ˆ* 1 ˆ 1 ( x) argmax ( x ˆ m ) ( ˆ S x m ) 1 d log( ) ˆ log ˆ S PY( ) 19
20 Important Warnng: at ths pont all optmalty clams for the BDR cease to be vald!! he BDR s guaranteed to acheve the mnmum loss only when we use the true probabltes When we plug n probablty estmates, we could be mplementng a classfer that s qute dstant from the optmal E.g. f the P X Y (x ) look lke the example above one could never approxmate t well by usng smple parametrc models (e.g. a sngle Gaussan). 0
21 Maxmum lkelhood Estmaton (MLE) Gven a parameterzed pdf how should one estmate the parameters whch defne the pdf? here are many technques of parameter estmaton. We shall utlze the maxmum lkelhood (ML) prncple. hs has three steps: 1) We choose a parametrc model for all probabltes. o make ths clear we denote the vector of parameters by and the class-condtonal dstrbutons by PX Y ( x ; ), p Note: hs s a classcal statstcs approach, whch means that s NO a random varable. It s a determnstc but unknown parameter, and the probabltes are a functon of ths unknown parameter. 1
22 Maxmum Lkelhood Estmaton (MLE) he three steps contnued: ) Assemble a collecton of datasets: D () = {x 1 (),..., x n () } = set of examples from each class 3) Select the values of the parameters of class to be the ones that maxmze the probablty of the data from that class ˆ ( ) arg max P ; ( ) D Y ( ) argmaxlog P ( ) ; D Y Note that t does not make any dfference to maxmze probabltes or ther logs.
23 Maxmum Lkelhood Estmaton (MLE) Snce Each sample D () s consdered ndependently Each parameter vector s estmated only from sample D () we smply have to repeat the procedure for all classes. So, from now on we omt the class varable : ˆ arg max P ; ML argmaxlog P ; he functon L( ; D) = P X (D; ) s the lkelhood of the parameter gven the data D, or smply the lkelhood functon. X X 3
24 he Lkelhood Functon Gven a parameterzed famly of pdf s (aka known as a statstcal model) for the data D, we defne a Lkelhood of the parameter vector gven D : L ( ) L( ; ) a( ) P ( ) D where a(d ) > 0 for all D, and a(d ) s ndependent of the parameter. he choce a(d ) = 1 yelds the Standard Lkelhood: L( ; D) = P D (D ; ) whch was shown on the prevous slde. 4
25 Maxmum Lkelhood Prncple p ( x) 1 Lx ) P ( x) P x Lx 1 ( ( ) ( ) 1 x X
26 he Lkelhood Functon Note that the lkelhood functon s a functon of the parameters It does not have the same shape as the densty tself E.g. the lkelhood functon of a Gaussan s not bell-shaped he lkelhood s defned only after we have a data sample P X 1 ( d m) ( d; ) exp ( ) 6
27 Maxmum Lkelhood Estmaton (MLE) Gven a sample, to obtan ML estmate we need to solve ˆ ; M L arg max P D When s a scalar, ths s hgh-school calculus: We have a local maxmum of f(x) at a pont x when he frst dervatve at x s zero. (x s a statonary pont.) he second dervatve s negatve at x. 7
28 MLE Example Gaussan wth unknown mean & standard devaton: Gven a data sample D = { 1,, N } of ndependent and dentcally dstrbuted (d) measurements, the (standard) lkelhood s L(, ;,, ) 1 N 8
29 MLE Example he log-lkelhood s he dervatve wth respect to the mean s zero when yeldng Note that ths s just the sample mean 9
30 MLE Example he log-lkelhood s he dervatve wrt the standard devaton s zero when or Note that ths s just the sample varance. 30
31 MLE Example Numercal example: If sample s {10,0,30,40,50} 31
32 he Gradent In hgher dmensons, the generalzaton of the dervatve s the gradent he (Cartesan) gradent of a functon f(w) at z s f f f f ( z) ( z) ( z),, ( z) w w w 0 n 1 he gradent has a nce geometrc nterpretaton It ponts n the drecton of maxmum growth of the functon. (Steepest Ascent Drecton.) Whch makes t perpendcular to the contours where the functon s constant. he above s the gradent for the smple (unweghted) Eucldean Norm (aka the Cartesan Gradent). f ( x 0, y0) f f(x,y) f ( x, y 1 1) 3
33 he Gradent Note that f f(x) = 0 here s no drecton of growth at x also f(x) = 0, and there s no drecton of decrease at x We are ether at a local mnmum or maxmum or saddle pont at x Conversely, f there s a local mn or max or saddle pont at x here s no drecton of growth or decrease at x f (x) = 0 hs shows that we have a statonary pont at x f and only f f(x) = 0 o determne whch type holds we need second order condtons max mn saddle 33
34 he Hessan he extenson of the scalar second-order dervatve s the Hessan matrx of second partal dervatves: ( x) f f ( x) ( x) x0 x0 xn1 ( x) f f ( x) ( x) xn1 x0 x n1 f f ( x) x x x Note that the Hessan s symmetrc. he Hessan gves us the quadratc functon 1 ( ) x x ( 0 x )( 0 x x ) 0 that best approxmates f(x) at a statonary pont x 0. 34
35 Hessan as a Quadratc Approxmaton E.g. ths means that f the gradent s zero at x 0, we have a maxmum when the functon f(x) can be locally approxmated by an upwards pontng quadratc bowl (H (x 0 ) s neg-def) max a mnmum when the functon can be locally approxmated by a downwards pontng quadratc bowl (H (x 0 )s pos-def) a saddle pont otherwse (H (x 0 ) s ndefnte) saddle mn 35
36 Hessan Gves Local Behavor max hs s somethng that we already saw: For any matrx M, the quadratc functon x M x s an upwards pontng quadratc quadratc bowl at the pont x = 0 when M s negatve defnte s a downwards pontng quadratc bowl at x = 0 when M s postve defnte s a saddle pont at x = 0 otherwse Hence, smlarly, what matters s the defnteness property of the Hessan at a statonary pont x 0 saddle mn E.g., we have a maxmum at a statonary pont x 0 when the Hessan s negatve defnte at x 0 36
37 Optmalty Condtons In summary: w 0 s a local mnmum of f(w) f and only f f has zero gradent at w 0 f( w ) 0 0 and the Hessan of f at w 0 s postve defnte where ( x) d ( ) 0, n x, 0 0 d d d f f ( x) ( x) x0 x0xn1 f f ( x) ( x) xn1 x0 x n1 37
38 Maxmum Lkelhood Estmaton (MLE) Gven a sample, to obtan an MLE we want to solve ˆ ML arg max P D ; max Canddate solutons are the parameter values ˆ such that P D ( ; ˆ ) 0 ( ˆ ) 0, 0 Note that you always have to check the second-order Hessan condton 38
39 MLE Example Back to our Gaussan example Gven d samples { 1,, N } the lkelhood s L(, ;,, ) 1 N 39
40 MLE Example he log-lkelhood s he dervatve of wth respect to the mean s from whch we compute the second-order dervatves N 3 40
41 41 MLE Example he dervatve of wth respect to the standard devaton s whch yelds the the second-order dervatves he statonary parameter values are, N 4 3 3
42 MLE Example he elements of the Hessan are: N 3 hus the Hessan s ( ) N N N whch s clearly negatve defnte at the statonary pont. hs we have determned the MLE of the parameters. 3 N
43 nd MLE Example o fnd the MLE s of the two pror class probabltes P Y () note that, 1 PY () 1, 0 can be wrtten as x 1 x P ( x ) 1 x 0, 1 Y where x s the so-called ndcator (or 0-1) functon. Gven d ndcator samples D = {x 1,..., x N }, we have x L( ; ) P ( ; ) 1 Y N 1 1 x 43
44 nd MLE Example herefore n 1 log P ( ; ) x log 1 x log1 Y Settng the dervatve of the log-lkelhood wth respect to equal to zero, N log PY ( ) 1 N 1 x N 1 N x (1 ) 1 1 N 0, x 44
45 nd MLE Example yelds the MLE estmate ˆ ML 1 n x where n x N N N 1 N 1 Note that ths s just the relatve frequency of occurrence of the value 1 n the sample. I.e. the MLE s just the count of the number of 1 s over the total number of ponts! Agan we see that the MLE yelds an ntutvely pleasng estmate of the unknown parameters. 45
46 nd MLE Example Check that the second dervatve s negatve: for < 1. N log PY ( D) 1 N x (1 ) 1 1 N N
47 Combnng the MLE Examples For Gaussan Classes all of the above formulas can be generalzed to the random vector case as follows: D () = {x 1 (),..., x n () } = set of d vector examples from each class, = 1,, d. he MLE estmates n the vector random data case are: ˆ 1 () m x j Pˆ n Y () j 1 S ˆ ( ) ( ) ( ˆ )( ˆ xj m xj m ) n j hese are the sample estmates gven earler wth no justfcaton. he ML solutons are ntutve, whch s usually the case. n N 47
48 END 48
Clustering & Unsupervised Learning
Clusterng & Unsupervsed Learnng Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 2012 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
More informationClustering & (Ken Kreutz-Delgado) UCSD
Clusterng & Unsupervsed Learnng Nuno Vasconcelos (Ken Kreutz-Delgado) UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y ), fnd an approxmatng
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
More information2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
More informationLecture 12: Classification
Lecture : Classfcaton g Dscrmnant functons g The optmal Bayes classfer g Quadratc classfers g Eucldean and Mahalanobs metrcs g K Nearest Neghbor Classfers Intellgent Sensor Systems Rcardo Guterrez-Osuna
More informationStatistical pattern recognition
Statstcal pattern recognton Bayes theorem Problem: decdng f a patent has a partcular condton based on a partcular test However, the test s mperfect Someone wth the condton may go undetected (false negatve
More informationClassification as a Regression Problem
Target varable y C C, C,, ; Classfcaton as a Regresson Problem { }, 3 L C K To treat classfcaton as a regresson problem we should transform the target y nto numercal values; The choce of numercal class
More informationCS 2750 Machine Learning. Lecture 5. Density estimation. CS 2750 Machine Learning. Announcements
CS 750 Machne Learnng Lecture 5 Densty estmaton Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square CS 750 Machne Learnng Announcements Homework Due on Wednesday before the class Reports: hand n before
More informationWhy Bayesian? 3. Bayes and Normal Models. State of nature: class. Decision rule. Rev. Thomas Bayes ( ) Bayes Theorem (yes, the famous one)
Why Bayesan? 3. Bayes and Normal Models Alex M. Martnez alex@ece.osu.edu Handouts Handoutsfor forece ECE874 874Sp Sp007 If all our research (n PR was to dsappear and you could only save one theory, whch
More informationENG 8801/ Special Topics in Computer Engineering: Pattern Recognition. Memorial University of Newfoundland Pattern Recognition
EG 880/988 - Specal opcs n Computer Engneerng: Pattern Recognton Memoral Unversty of ewfoundland Pattern Recognton Lecture 7 May 3, 006 http://wwwengrmunca/~charlesr Offce Hours: uesdays hursdays 8:30-9:30
More informationLecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
More informationMIMA Group. Chapter 2 Bayesian Decision Theory. School of Computer Science and Technology, Shandong University. Xin-Shun SDU
Group M D L M Chapter Bayesan Decson heory Xn-Shun Xu @ SDU School of Computer Scence and echnology, Shandong Unversty Bayesan Decson heory Bayesan decson theory s a statstcal approach to data mnng/pattern
More informationLogistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
More informationCIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
More informationP R. Lecture 4. Theory and Applications of Pattern Recognition. Dept. of Electrical and Computer Engineering /
Theory and Applcatons of Pattern Recognton 003, Rob Polkar, Rowan Unversty, Glassboro, NJ Lecture 4 Bayes Classfcaton Rule Dept. of Electrcal and Computer Engneerng 0909.40.0 / 0909.504.04 Theory & Applcatons
More informationComposite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
More information9.913 Pattern Recognition for Vision. Class IV Part I Bayesian Decision Theory Yuri Ivanov
9.93 Class IV Part I Bayesan Decson Theory Yur Ivanov TOC Roadmap to Machne Learnng Bayesan Decson Makng Mnmum Error Rate Decsons Mnmum Rsk Decsons Mnmax Crteron Operatng Characterstcs Notaton x - scalar
More informationLinear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
More informationMLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
More information8/25/17. Data Modeling. Data Modeling. Data Modeling. Patrice Koehl Department of Biological Sciences National University of Singapore
8/5/17 Data Modelng Patrce Koehl Department of Bologcal Scences atonal Unversty of Sngapore http://www.cs.ucdavs.edu/~koehl/teachng/bl59 koehl@cs.ucdavs.edu Data Modelng Ø Data Modelng: least squares Ø
More informationOutline. Multivariate Parametric Methods. Multivariate Data. Basic Multivariate Statistics. Steven J Zeil
Outlne Multvarate Parametrc Methods Steven J Zel Old Domnon Unv. Fall 2010 1 Multvarate Data 2 Multvarate ormal Dstrbuton 3 Multvarate Classfcaton Dscrmnants Tunng Complexty Dscrete Features 4 Multvarate
More informationThe exam is closed book, closed notes except your one-page cheat sheet.
CS 89 Fall 206 Introducton to Machne Learnng Fnal Do not open the exam before you are nstructed to do so The exam s closed book, closed notes except your one-page cheat sheet Usage of electronc devces
More informationThe Geometry of Logit and Probit
The Geometry of Logt and Probt Ths short note s meant as a supplement to Chapters and 3 of Spatal Models of Parlamentary Votng and the notaton and reference to fgures n the text below s to those two chapters.
More information10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
More informationThe Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that e have state of the orld X observatons decson functon L[,y] loss of predctn y th Bayes decson rule s the rule
More informationLecture 10 Support Vector Machines II
Lecture 10 Support Vector Machnes II 22 February 2016 Taylor B. Arnold Yale Statstcs STAT 365/665 1/28 Notes: Problem 3 s posted and due ths upcomng Frday There was an early bug n the fake-test data; fxed
More informationSDMML HT MSc Problem Sheet 4
SDMML HT 06 - MSc Problem Sheet 4. The recever operatng characterstc ROC curve plots the senstvty aganst the specfcty of a bnary classfer as the threshold for dscrmnaton s vared. Let the data space be
More informationGeneralized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
More informationSupport Vector Machines. Vibhav Gogate The University of Texas at dallas
Support Vector Machnes Vbhav Gogate he Unversty of exas at dallas What We have Learned So Far? 1. Decson rees. Naïve Bayes 3. Lnear Regresson 4. Logstc Regresson 5. Perceptron 6. Neural networks 7. K-Nearest
More informationWhich Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
More informationC4B Machine Learning Answers II. = σ(z) (1 σ(z)) 1 1 e z. e z = σ(1 σ) (1 + e z )
C4B Machne Learnng Answers II.(a) Show that for the logstc sgmod functon dσ(z) dz = σ(z) ( σ(z)) A. Zsserman, Hlary Term 20 Start from the defnton of σ(z) Note that Then σ(z) = σ = dσ(z) dz = + e z e z
More information3.1 ML and Empirical Distribution
67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum
More informationBayesian decision theory. Nuno Vasconcelos ECE Department, UCSD
Bayesan decson theory Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world observatons decson functon L[,y] loss of predctn y wth the epected value of the
More informationHomework Assignment 3 Due in class, Thursday October 15
Homework Assgnment 3 Due n class, Thursday October 15 SDS 383C Statstcal Modelng I 1 Rdge regresson and Lasso 1. Get the Prostrate cancer data from http://statweb.stanford.edu/~tbs/elemstatlearn/ datasets/prostate.data.
More informationxp(x µ) = 0 p(x = 0 µ) + 1 p(x = 1 µ) = µ
CSE 455/555 Sprng 2013 Homework 7: Parametrc Technques Jason J. Corso Computer Scence and Engneerng SUY at Buffalo jcorso@buffalo.edu Solutons by Yngbo Zhou Ths assgnment does not need to be submtted and
More informationFor now, let us focus on a specific model of neurons. These are simplified from reality but can achieve remarkable results.
Neural Networks : Dervaton compled by Alvn Wan from Professor Jtendra Malk s lecture Ths type of computaton s called deep learnng and s the most popular method for many problems, such as computer vson
More informationMaximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
More informationConjugacy and the Exponential Family
CS281B/Stat241B: Advanced Topcs n Learnng & Decson Makng Conjugacy and the Exponental Famly Lecturer: Mchael I. Jordan Scrbes: Bran Mlch 1 Conjugacy In the prevous lecture, we saw conjugate prors for the
More informationFinite Mixture Models and Expectation Maximization. Most slides are from: Dr. Mario Figueiredo, Dr. Anil Jain and Dr. Rong Jin
Fnte Mxture Models and Expectaton Maxmzaton Most sldes are from: Dr. Maro Fgueredo, Dr. Anl Jan and Dr. Rong Jn Recall: The Supervsed Learnng Problem Gven a set of n samples X {(x, y )},,,n Chapter 3 of
More informationPredictive Analytics : QM901.1x Prof U Dinesh Kumar, IIMB. All Rights Reserved, Indian Institute of Management Bangalore
Sesson Outlne Introducton to classfcaton problems and dscrete choce models. Introducton to Logstcs Regresson. Logstc functon and Logt functon. Maxmum Lkelhood Estmator (MLE) for estmaton of LR parameters.
More informationAPPROXIMATE PRICES OF BASKET AND ASIAN OPTIONS DUPONT OLIVIER. Premia 14
APPROXIMAE PRICES OF BASKE AND ASIAN OPIONS DUPON OLIVIER Prema 14 Contents Introducton 1 1. Framewor 1 1.1. Baset optons 1.. Asan optons. Computng the prce 3. Lower bound 3.1. Closed formula for the prce
More informationprinceton univ. F 17 cos 521: Advanced Algorithm Design Lecture 7: LP Duality Lecturer: Matt Weinberg
prnceton unv. F 17 cos 521: Advanced Algorthm Desgn Lecture 7: LP Dualty Lecturer: Matt Wenberg Scrbe: LP Dualty s an extremely useful tool for analyzng structural propertes of lnear programs. Whle there
More information3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
More information1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
More informationINF 5860 Machine learning for image classification. Lecture 3 : Image classification and regression part II Anne Solberg January 31, 2018
INF 5860 Machne learnng for mage classfcaton Lecture 3 : Image classfcaton and regresson part II Anne Solberg January 3, 08 Today s topcs Multclass logstc regresson and softma Regularzaton Image classfcaton
More informationLinear discriminants. Nuno Vasconcelos ECE Department, UCSD
Lnear dscrmnants Nuno Vasconcelos ECE Department UCSD Classfcaton a classfcaton problem as to tpes of varables e.g. X - vector of observatons features n te orld Y - state class of te orld X R 2 fever blood
More informationNeural networks. Nuno Vasconcelos ECE Department, UCSD
Neural networs Nuno Vasconcelos ECE Department, UCSD Classfcaton a classfcaton problem has two types of varables e.g. X - vector of observatons (features) n the world Y - state (class) of the world x X
More informationCS 3710: Visual Recognition Classification and Detection. Adriana Kovashka Department of Computer Science January 13, 2015
CS 3710: Vsual Recognton Classfcaton and Detecton Adrana Kovashka Department of Computer Scence January 13, 2015 Plan for Today Vsual recognton bascs part 2: Classfcaton and detecton Adrana s research
More informationKernel Methods and SVMs Extension
Kernel Methods and SVMs Extenson The purpose of ths document s to revew materal covered n Machne Learnng 1 Supervsed Learnng regardng support vector machnes (SVMs). Ths document also provdes a general
More informationLinear Feature Engineering 11
Lnear Feature Engneerng 11 2 Least-Squares 2.1 Smple least-squares Consder the followng dataset. We have a bunch of nputs x and correspondng outputs y. The partcular values n ths dataset are x y 0.23 0.19
More informationDepartment of Computer Science Artificial Intelligence Research Laboratory. Iowa State University MACHINE LEARNING
MACHINE LEANING Vasant Honavar Bonformatcs and Computatonal Bology rogram Center for Computatonal Intellgence, Learnng, & Dscovery Iowa State Unversty honavar@cs.astate.edu www.cs.astate.edu/~honavar/
More informationLectures - Week 4 Matrix norms, Conditioning, Vector Spaces, Linear Independence, Spanning sets and Basis, Null space and Range of a Matrix
Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could
More informationLinear Classification, SVMs and Nearest Neighbors
1 CSE 473 Lecture 25 (Chapter 18) Lnear Classfcaton, SVMs and Nearest Neghbors CSE AI faculty + Chrs Bshop, Dan Klen, Stuart Russell, Andrew Moore Motvaton: Face Detecton How do we buld a classfer to dstngush
More informationMachine learning: Density estimation
CS 70 Foundatons of AI Lecture 3 Machne learnng: ensty estmaton Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square ata: ensty estmaton {.. n} x a vector of attrbute values Objectve: estmate the model of
More informationProbabilistic Classification: Bayes Classifiers. Lecture 6:
Probablstc Classfcaton: Bayes Classfers Lecture : Classfcaton Models Sam Rowes January, Generatve model: p(x, y) = p(y)p(x y). p(y) are called class prors. p(x y) are called class condtonal feature dstrbutons.
More informationPattern Classification
attern Classfcaton All materals n these sldes were taken from attern Classfcaton nd ed by R. O. Duda,. E. Hart and D. G. Stork, John Wley & Sons, 000 wth the ermsson of the authors and the ublsher Chater
More informationGlobal Sensitivity. Tuesday 20 th February, 2018
Global Senstvty Tuesday 2 th February, 28 ) Local Senstvty Most senstvty analyses [] are based on local estmates of senstvty, typcally by expandng the response n a Taylor seres about some specfc values
More informationLecture 12: Discrete Laplacian
Lecture 12: Dscrete Laplacan Scrbe: Tanye Lu Our goal s to come up wth a dscrete verson of Laplacan operator for trangulated surfaces, so that we can use t n practce to solve related problems We are mostly
More informationThe big picture. Outline
The bg pcture Vncent Claveau IRISA - CNRS, sldes from E. Kjak INSA Rennes Notatons classes: C = {ω = 1,.., C} tranng set S of sze m, composed of m ponts (x, ω ) per class ω representaton space: R d (=
More informationUsing T.O.M to Estimate Parameter of distributions that have not Single Exponential Family
IOSR Journal of Mathematcs IOSR-JM) ISSN: 2278-5728. Volume 3, Issue 3 Sep-Oct. 202), PP 44-48 www.osrjournals.org Usng T.O.M to Estmate Parameter of dstrbutons that have not Sngle Exponental Famly Jubran
More informationEM and Structure Learning
EM and Structure Learnng Le Song Machne Learnng II: Advanced Topcs CSE 8803ML, Sprng 2012 Partally observed graphcal models Mxture Models N(μ 1, Σ 1 ) Z X N N(μ 2, Σ 2 ) 2 Gaussan mxture model Consder
More informationMachine Learning. Classification. Theory of Classification and Nonparametric Classifier. Representing data: Hypothesis (classifier) Eric Xing
Machne Learnng 0-70/5 70/5-78, 78, Fall 008 Theory of Classfcaton and Nonarametrc Classfer Erc ng Lecture, Setember 0, 008 Readng: Cha.,5 CB and handouts Classfcaton Reresentng data: M K Hyothess classfer
More informationInstance-Based Learning (a.k.a. memory-based learning) Part I: Nearest Neighbor Classification
Instance-Based earnng (a.k.a. memory-based learnng) Part I: Nearest Neghbor Classfcaton Note to other teachers and users of these sldes. Andrew would be delghted f you found ths source materal useful n
More informationTopic 5: Non-Linear Regression
Topc 5: Non-Lnear Regresson The models we ve worked wth so far have been lnear n the parameters. They ve been of the form: y = Xβ + ε Many models based on economc theory are actually non-lnear n the parameters.
More informationAssortment Optimization under MNL
Assortment Optmzaton under MNL Haotan Song Aprl 30, 2017 1 Introducton The assortment optmzaton problem ams to fnd the revenue-maxmzng assortment of products to offer when the prces of products are fxed.
More informationMATH 829: Introduction to Data Mining and Analysis The EM algorithm (part 2)
1/16 MATH 829: Introducton to Data Mnng and Analyss The EM algorthm (part 2) Domnque Gullot Departments of Mathematcal Scences Unversty of Delaware Aprl 20, 2016 Recall 2/16 We are gven ndependent observatons
More informationGenerative classification models
CS 675 Intro to Machne Learnng Lecture Generatve classfcaton models Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square Data: D { d, d,.., dn} d, Classfcaton represents a dscrete class value Goal: learn
More informationSupport Vector Machines
Separatng boundary, defned by w Support Vector Machnes CISC 5800 Professor Danel Leeds Separatng hyperplane splts class 0 and class 1 Plane s defned by lne w perpendcular to plan Is data pont x n class
More informationANSWERS. Problem 1. and the moment generating function (mgf) by. defined for any real t. Use this to show that E( U) var( U)
Econ 413 Exam 13 H ANSWERS Settet er nndelt 9 deloppgaver, A,B,C, som alle anbefales å telle lkt for å gøre det ltt lettere å stå. Svar er gtt . Unfortunately, there s a prntng error n the hnt of
More informationExpected Value and Variance
MATH 38 Expected Value and Varance Dr. Neal, WKU We now shall dscuss how to fnd the average and standard devaton of a random varable X. Expected Value Defnton. The expected value (or average value, or
More informationVQ widely used in coding speech, image, and video
at Scalar quantzers are specal cases of vector quantzers (VQ): they are constraned to look at one sample at a tme (memoryless) VQ does not have such constrant better RD perfomance expected Source codng
More informationMaximum Likelihood Estimation
Maxmum Lkelhood Estmaton INFO-2301: Quanttatve Reasonng 2 Mchael Paul and Jordan Boyd-Graber MARCH 7, 2017 INFO-2301: Quanttatve Reasonng 2 Paul and Boyd-Graber Maxmum Lkelhood Estmaton 1 of 9 Why MLE?
More informationDr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Analyss of Varance and Desgn of Exerments-I MODULE III LECTURE - 2 EXPERIMENTAL DESIGN MODELS Dr. Shalabh Deartment of Mathematcs and Statstcs Indan Insttute of Technology Kanur 2 We consder the models
More informationBasically, if you have a dummy dependent variable you will be estimating a probability.
ECON 497: Lecture Notes 13 Page 1 of 1 Metropoltan State Unversty ECON 497: Research and Forecastng Lecture Notes 13 Dummy Dependent Varable Technques Studenmund Chapter 13 Bascally, f you have a dummy
More informationCSci 6974 and ECSE 6966 Math. Tech. for Vision, Graphics and Robotics Lecture 21, April 17, 2006 Estimating A Plane Homography
CSc 6974 and ECSE 6966 Math. Tech. for Vson, Graphcs and Robotcs Lecture 21, Aprl 17, 2006 Estmatng A Plane Homography Overvew We contnue wth a dscusson of the major ssues, usng estmaton of plane projectve
More information1 Motivation and Introduction
Instructor: Dr. Volkan Cevher EXPECTATION PROPAGATION September 30, 2008 Rce Unversty STAT 63 / ELEC 633: Graphcal Models Scrbes: Ahmad Beram Andrew Waters Matthew Nokleby Index terms: Approxmate nference,
More information1. Inference on Regression Parameters a. Finding Mean, s.d and covariance amongst estimates. 2. Confidence Intervals and Working Hotelling Bands
Content. Inference on Regresson Parameters a. Fndng Mean, s.d and covarance amongst estmates.. Confdence Intervals and Workng Hotellng Bands 3. Cochran s Theorem 4. General Lnear Testng 5. Measures of
More informationMore metrics on cartesian products
More metrcs on cartesan products If (X, d ) are metrc spaces for 1 n, then n Secton II4 of the lecture notes we defned three metrcs on X whose underlyng topologes are the product topology The purpose of
More informationExplaining the Stein Paradox
Explanng the Sten Paradox Kwong Hu Yung 1999/06/10 Abstract Ths report offers several ratonale for the Sten paradox. Sectons 1 and defnes the multvarate normal mean estmaton problem and ntroduces Sten
More informationLimited Dependent Variables
Lmted Dependent Varables. What f the left-hand sde varable s not a contnuous thng spread from mnus nfnty to plus nfnty? That s, gven a model = f (, β, ε, where a. s bounded below at zero, such as wages
More informationClassification. Representing data: Hypothesis (classifier) Lecture 2, September 14, Reading: Eric CMU,
Machne Learnng 10-701/15-781, 781, Fall 2011 Nonparametrc methods Erc Xng Lecture 2, September 14, 2011 Readng: 1 Classfcaton Representng data: Hypothess (classfer) 2 1 Clusterng 3 Supervsed vs. Unsupervsed
More informationPHYS 705: Classical Mechanics. Calculus of Variations II
1 PHYS 705: Classcal Mechancs Calculus of Varatons II 2 Calculus of Varatons: Generalzaton (no constrant yet) Suppose now that F depends on several dependent varables : We need to fnd such that has a statonary
More informationThe conjugate prior to a Bernoulli is. A) Bernoulli B) Gaussian C) Beta D) none of the above
The conjugate pror to a Bernoull s A) Bernoull B) Gaussan C) Beta D) none of the above The conjugate pror to a Gaussan s A) Bernoull B) Gaussan C) Beta D) none of the above MAP estmates A) argmax θ p(θ
More informationChapter Newton s Method
Chapter 9. Newton s Method After readng ths chapter, you should be able to:. Understand how Newton s method s dfferent from the Golden Secton Search method. Understand how Newton s method works 3. Solve
More informationFeb 14: Spatial analysis of data fields
Feb 4: Spatal analyss of data felds Mappng rregularly sampled data onto a regular grd Many analyss technques for geophyscal data requre the data be located at regular ntervals n space and/or tme. hs s
More informationChapter 11: Simple Linear Regression and Correlation
Chapter 11: Smple Lnear Regresson and Correlaton 11-1 Emprcal Models 11-2 Smple Lnear Regresson 11-3 Propertes of the Least Squares Estmators 11-4 Hypothess Test n Smple Lnear Regresson 11-4.1 Use of t-tests
More informationBezier curves. Michael S. Floater. August 25, These notes provide an introduction to Bezier curves. i=0
Bezer curves Mchael S. Floater August 25, 211 These notes provde an ntroducton to Bezer curves. 1 Bernsten polynomals Recall that a real polynomal of a real varable x R, wth degree n, s a functon of the
More information4.3 Poisson Regression
of teratvely reweghted least squares regressons (the IRLS algorthm). We do wthout gvng further detals, but nstead focus on the practcal applcaton. > glm(survval~log(weght)+age, famly="bnomal", data=baby)
More informationEstimation: Part 2. Chapter GREG estimation
Chapter 9 Estmaton: Part 2 9. GREG estmaton In Chapter 8, we have seen that the regresson estmator s an effcent estmator when there s a lnear relatonshp between y and x. In ths chapter, we generalzed the
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far So far Supervsed machne learnng Lnear models Non-lnear models Unsupervsed machne learnng Generc scaffoldng So far
More informationHydrological statistics. Hydrological statistics and extremes
5--0 Stochastc Hydrology Hydrologcal statstcs and extremes Marc F.P. Berkens Professor of Hydrology Faculty of Geoscences Hydrologcal statstcs Mostly concernes wth the statstcal analyss of hydrologcal
More informationBézier curves. Michael S. Floater. September 10, These notes provide an introduction to Bézier curves. i=0
Bézer curves Mchael S. Floater September 1, 215 These notes provde an ntroducton to Bézer curves. 1 Bernsten polynomals Recall that a real polynomal of a real varable x R, wth degree n, s a functon of
More informationLogistic Classifier CISC 5800 Professor Daniel Leeds
lon 9/7/8 Logstc Classfer CISC 58 Professor Danel Leeds Classfcaton strategy: generatve vs. dscrmnatve Generatve, e.g., Bayes/Naïve Bayes: 5 5 Identfy probablty dstrbuton for each class Determne class
More informationHidden Markov Models & The Multivariate Gaussian (10/26/04)
CS281A/Stat241A: Statstcal Learnng Theory Hdden Markov Models & The Multvarate Gaussan (10/26/04) Lecturer: Mchael I. Jordan Scrbes: Jonathan W. Hu 1 Hdden Markov Models As a bref revew, hdden Markov models
More informationStatistical Inference. 2.3 Summary Statistics Measures of Center and Spread. parameters ( population characteristics )
Ismor Fscher, 8//008 Stat 54 / -8.3 Summary Statstcs Measures of Center and Spread Dstrbuton of dscrete contnuous POPULATION Random Varable, numercal True center =??? True spread =???? parameters ( populaton
More informationSupport Vector Machines
Support Vector Machnes Konstantn Tretyakov (kt@ut.ee) MTAT.03.227 Machne Learnng So far Supervsed machne learnng Lnear models Least squares regresson Fsher s dscrmnant, Perceptron, Logstc model Non-lnear
More informationLaboratory 1c: Method of Least Squares
Lab 1c, Least Squares Laboratory 1c: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly
More informationCommunication with AWGN Interference
Communcaton wth AWG Interference m {m } {p(m } Modulator s {s } r=s+n Recever ˆm AWG n m s a dscrete random varable(rv whch takes m wth probablty p(m. Modulator maps each m nto a waveform sgnal s m=m
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Luca Trevisan September 5, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analyss Handout 4s Luca Trevsan September 5, 07 Summary of Lecture 4 In whch we ntroduce semdefnte programmng and apply t to Max Cut. Semdefnte Programmng Recall that
More information