Machine Learning for Signal Processing Linear Gaussian Models
|
|
|
- Meredith Bernadette French
- 5 years ago
- Views:
Transcription
1 Machne Learnng for Sgnal rocessng Lnear Gaussan Models lass 2. 2 Nov 203 Instructor: Bhsha Raj 2 Nov /8797
2 HW3 s up. Admnstrva rojects please send us an update 2 Nov /8797 2
3 Recap: MA stmators MA Mamum A osteror: Fnd a best guess for statstcall gven non = argma Y Y 2 Nov /8797 3
4 Recap: MA estmaton and are jontl Gaussan z [ z] z Var z zz [ ] z N z zz ep 0.5 z z z z 2 zz z s Gaussan 2 Nov /8797 4
5 MA estmaton: Gaussan DF Y F X 2 Nov /8797 5
6 MA estmaton: he Gaussan at a partcular value of X 0 2 Nov /8797 6
7 ondtonal robablt of N [ ] Var he condtonal probablt of gven s also Gaussan he slce n the fgure s Gaussan he mean of ths Gaussan s a functon of he varance of reduces f s non Uncertant s reduced 2 Nov /8797 7
8 MA estmaton: he Gaussan at a partcular value of X Most lel value F 0 2 Nov /8797 8
9 MA stmaton of a Gaussan RV ˆ arg ma [ ] 0 2 Nov /8797 9
10 Its also a mnmum-mean-squared error estmate Mnmze error: Dfferentatng and equatng to 0: 2 Nov / ] ˆ ˆ [ ] ˆ [ 2 rr ] [ 2ˆ ˆ ˆ ] [ ] 2ˆ ˆ ˆ [ rr 0 ˆ ] [ 2 ˆ 2ˆ. d d rr d ] [ ˆ he MMS estmate s the mean of the dstrbuton
11 For the Gaussan: MA = MMS Most lel value s also he MAN value Would be true of an smmetrc dstrbuton 2 Nov /8797
12 MMS estmates for mture dstrbutons 2 Let be a mture denst he MMS estmate of s gven b Just a eghted combnaton of the MMS estmates from the component dstrbutons d ] [ d ] [
13 MMS estmates from a Gaussan mture 2 Nov / s also a Gaussan mture Let be a Gaussan Mture ; N z z z
14 MMS estmates from a Gaussan mture 2 Nov / Let s a Gaussan Mture N N N
15 MMS estmates from a Gaussan mture 2 Nov / [] s also a mture s a mture Gaussan denst N ] [ ] [ ] [
16 MMS estmates from a Gaussan mture 2 Nov / Weghted combnaton of MMS estmates obtaned from ndvdual Gaussans! Weght s easl computed too.. ] [ N
17 MMS estmates from a Gaussan mture A mture of estmates from ndvdual Gaussans 2 Nov /8797 7
18 Voce Morphng Algn tranng recordngs from both speaers epstral vector sequence Learn a GMM on jont vectors Gven speech from one speaer fnd MMS estmate of the other Snthesze from cepstra 2 Nov /8797 8
19 MMS th GMM: Voce ransformaton - Festvo GMM transformaton sute oda ab bdl jm slt ab bdl jm slt 2 Nov /8797 9
20 MA / ML / MMS General statstcal estmators All used to predct a varable based on other parameters related to t.. Most common assumpton: Data are Gaussan all RVs are Gaussan Other probablt denstes ma also be used.. For Gaussans relatonshps are lnear as e sa.. 2 Nov /
21 Gaussans and more Gaussans.. Lnear Gaussan Models.. But frst a recap 2 Nov /8797 2
22 A Bref Recap D B D B rncpal component analss: Fnd the K bases that best eplan the gven data Fnd B and such that the dfference beteen D and B s mnmum Whle constranng that the columns of B are orthonormal 2 Nov /
23 Remember genfaces Appromate ever face f as f = f V + f2 V 2 + f3 V f V stmate V to mnmze the squared error rror s uneplaned b V.. V rror s orthogonal to genfaces 2 Nov /
24 Karhunen Loeve vs. A genvectors of the orrelaton matr: rncpal drectons of tghtest ellpse centered on orgn Drectons that retan mamum energ 2 Nov /
25 Karhunen Loeve vs. A genvectors of the orrelaton matr: rncpal drectons of tghtest ellpse centered on orgn Drectons that retan mamum energ genvectors of the ovarance matr: rncpal drectons of tghtest ellpse centered on data Drectons that retan mamum varance 2 Nov /
26 Karhunen Loeve vs. A genvectors of the orrelaton matr: rncpal drectons of tghtest ellpse centered on orgn Drectons that retan mamum energ genvectors of the ovarance matr: rncpal drectons of tghtest ellpse centered on data Drectons that retan mamum varance 2 Nov /
27 Karhunen Loeve vs. A genvectors of the orrelaton matr: rncpal drectons of tghtest ellpse centered on orgn Drectons that retan mamum energ genvectors of the ovarance matr: rncpal drectons of tghtest ellpse centered on data Drectons that retan mamum varance 2 Nov /
28 Karhunen Loeve vs. A If the data are naturall centered at orgn KL == A Follong sldes refer to A! Assume data centered at orgn for smplct Not essental as e ll see.. 2 Nov /
29 Remember genfaces Appromate ever face f as f = f V + f2 V 2 + f3 V f V stmate V to mnmze the squared error rror s uneplaned b V.. V rror s orthogonal to genfaces 2 Nov /
30 gen Representaton 0 = + e e Illustraton assumng 3D space K-dmensonal representaton rror s orthogonal to representaton Weght and error are specfc to data nstance 2 Nov /
31 Representaton rror s at 90 o to the egenface = 2 + e 2 90 o 2 e 2 Illustraton assumng 3D space K-dmensonal representaton rror s orthogonal to representaton Weght and error are specfc to data nstance 2 Nov /8797 3
32 Representaton 0 All data th the same representaton V le a plane orthogonal to V K-dmensonal representaton rror s orthogonal to representaton 2 Nov /
33 Wth 2 bases rror s at 90 o to the egenfaces 00 = e e 2 Illustraton assumng 3D space K-dmensonal representaton rror s orthogonal to representaton Weght and error are specfc to data nstance 2 Nov /
34 Wth 2 bases rror s at 90 o to the egenfaces = e e 2 Illustraton assumng 3D space K-dmensonal representaton rror s orthogonal to representaton Weght and error are specfc to data nstance 2 Nov /
35 rror s at 90 o to the egenfaces e 2 In Vector Form K-dmensonal representaton X = V + 2 V 2 + e V 2 2 D 2 V rror s orthogonal to representaton Weght and error are specfc to data nstance X V V e 2 Nov /
36 rror s at 90 o to the egenface e 2 In Vector Form X = V + 2 V 2 + e 2 V e 22 V 2 D 2 V K-dmensonal representaton s a D dmensonal vector V s a D K matr s a K dmensonal vector e s a D dmensonal vector 2 Nov /
37 Learnng A For the gven data: fnd the K-dmensonal subspace such that t captures most of the varance n the data Varance n remanng subspace s mnmal 2 Nov /
38 onstrants rror s at 90 o to the egenface V e 2 22 V 2 D 2 e 2 V V V = I : gen vectors are orthogonal to each other For ever vector error s orthogonal to gen vectors e V = 0 Over the collecton of data Average = Dagonal : gen representatons are uncorrelated Determnant e e = mnmum: rror varance s mnmum Mean of error s 0 2 Nov /
39 A Statstcal Formulaton of A rror s at 90 o to the egenface V e 22 e 2 2 V 2 D 2 V e ~ N0 B ~ N0 s a random varable generated accordng to a lnear relaton s dran from an K-dmensonal Gaussan th dagonal covarance e s dran from a 0-mean D-K-ran D-dmensonal Gaussan stmate V and B gven eamples of 2 Nov /
40 Lnear Gaussan Models!! V e ~ N0 B e ~ N0 s a random varable generated accordng to a lnear relaton s dran from a Gaussan e s dran from a 0-mean Gaussan stmate V gven eamples of In the process also estmate B and 40 2 Nov /8797
41 Lnear Gaussan Models!! V e ~ N0 B e ~ N0 s a random varable generated accordng to a lnear relaton s dran from a Gaussan e s dran from a 0-mean Gaussan stmate V gven eamples of In the process also estmate B and 4 2 Nov /8797
42 Lnear Gaussan Models μ V e ~ N0 B e ~ N0 Observatons are lnear functons of to uncorrelated Gaussan random varables A eght varable An error varable e rror not correlated to eght: [e ] = 0 Learnng LGMs: stmate parameters of the model gven nstances of he problem of learnng the dstrbuton of a Gaussan RV 2 Nov /
43 LGMs: robablt Denst μ V e ~ N0 B e ~ N0 he mean of : [ ] μ V[ ] [ e] μ he ovarance of : [ [ ] [ ] ] VBV 2 Nov /
44 he probablt of μ V e e ~ N0 B ~ N0 ~ N μ VBV ep 0.5 D 2 VBV μ VBV μ s a lnear functon of Gaussans: s also Gaussan Its mean and varance are as gven 2 Nov /
45 stmatng the varables of the μ V model e e ~ N0 B ~ N0 ~ N μ VBV stmatng the varables of the LGM s equvalent to estmatng he varables are V B and 2 Nov /
46 stmatng the model μ V e e ~ N0 B ~ N0 ~ N μ VBV he model s ndetermnate: V = V - = V - We need etra constrants to mae the soluton unque Usual constrant : B = I Varance of s an dentt matr 2 Nov /
47 stmatng the varables of the μ V model e ~ N0 I e ~ N0 ~ N μ VV stmatng the varables of the LGM s equvalent to estmatng he varables are V and 2 Nov /
48 he Mamum Lelhood stmate ~ N μ VV Gven tranng set 2.. N fnd V he ML estmate of does not depend on the covarance of the Gaussan μ N 2 Nov /
49 entered Data We can safel assume centered data = 0 If the data are not centered center t stmate mean of data Whch s the mamum lelhood estmate Subtract t from the data 2 Nov /
50 Smplfed Model V e ~ N0 I e ~ N0 ~ N0 VV stmatng the varables of the LGM s equvalent to estmatng he varables are V and 2 Nov /
51 stmatng the model V e ~ N0 VV Gven a collecton of terms 2.. N stmate V and s unnon for each But f assume e no for each then hat do e get: 2 Nov /8797 5
52 stmatng the arameters V e e N0 N V 2 D ep 0.5 V V We ll use a mamum-lelhood estmate he log-lelhood of.. N nong ther s log.. N.. N 0.5N log 0.5 V V 2 Nov /
53 Mamzng the log-lelhood Dfferentatng.r.t. V and settng to 0 2 Nov / N LL 0.5 log 0.5 V V 0 2 V V Dfferentatng.r.t. - and settng to 0 N V
54 stmatng LGMs: If e no But n realt e don t no the for each So ho to deal th ths? M.. 2 Nov / V N V e V 0 N e
55 Recall M Instance from blue dce Instance from red dce Dce unnon ollecton of blue numbers ollecton of red numbers ollecton of blue numbers ollecton of red numbers ollecton of blue numbers ollecton of red numbers We fgured out ho to compute parameters f e ne the mssng nformaton hen e fragmented the observatons accordng to the posteror probablt z and counted as usual In effect e too the epectaton th respect to the a posteror probablt of the mssng data: z 2 Nov /
56 M for LGMs Replace unseen data terms th epectatons taen.r.t. 2 Nov / V N V e V 0 N e N N V ] [ ] [ ] [ V
57 M for LGMs Replace unseen data terms th epectatons taen.r.t. 2 Nov / V N V e V 0 N e N N V ] [ ] [ ] [ V
58 pected Value of gven V e N0 e N0 I N0 VV and are jontl Gaussan! s Gaussan s Gaussan he are lnearl related z z N z zz 2 Nov /
59 pected Value of gven V [ ] e N0 VV N0 I z V z N z zz zz z zz VV V 0 V I and are jontl Gaussan! 2 Nov /
60 he condtonal epectaton of gven z z s a Gaussan 2 Nov / N I V V VV zz zz 0 z V VV V VV V I N VV V ] [ Var ] [ ] [ ] [ I ] [ ] [ ] [ V VV V
61 LGM: he complete M algorthm Intalze V and step: M step: 2 Nov / VV V ] [ I ] [ ] [ ] [ V VV V ] [ ] [ V N N V ] [
62 So hat have e acheved mploed a complcated M algorthm to learn a Gaussan DF for a varable What have e ganed??? Net class: A Sensble A M algorthms for A Factor Analss FA for feature etracton 2 Nov /
63 LGMs : Applcaton Learnng prncpal components V e ~ N0 I e ~ N0 Fnd drectons that capture most of the varaton n the data rror s orthogonal to these varatons 3 Oct /
64 LGMs : Applcaton 2 Learnng th nsuffcent data FULL OV FIGUR he full covarance matr of a Gaussan has D 2 terms Full captures the relatonshps beteen varables roblem: Needs a lot of data to estmate robustl 3 Oct /
65 o be contnued.. Other applcatons.. Net class 2 Nov /
Machine Learning for Signal Processing Linear Gaussian Models
Machne Learnng for Sgnal Processng Lnear Gaussan Models Class 7. 30 Oct 204 Instructor: Bhksha Raj 755/8797 Recap: MAP stmators MAP (Mamum A Posteror: Fnd a best guess for (statstcall, gven knon = argma
Machine Learning for Signal Processing Applications of Linear Gaussian Models
Machne Learnng for Sgnal Processng Applcatons of Lnear Gaussan Models Class 8. 3 Nov 207 Instructor Najm Dehak In collaboraton th Prof Bhksha Raj Recap: MAP stmators MAP (Mamum A Posteror): Fnd most probable
Expectation Maximization Mixture Models HMMs
-755 Machne Learnng for Sgnal Processng Mture Models HMMs Class 9. 2 Sep 200 Learnng Dstrbutons for Data Problem: Gven a collecton of eamples from some data, estmate ts dstrbuton Basc deas of Mamum Lelhood
Machine Learning for Signal Processing Regression and Prediction
Machne Learnng for Sgnal Processng Regresson and Predcton Class 4. 5 Oct 06 Instructor: Bhsha Raj 755/8797 A Common Problem Can ou spot the gltches? 755/8797 How to f ths problem? Gltches n audo Must be
Machine Learning for Signal Processing Regression and Prediction
Machne Learnng for Sgnal Processng Regresson and Predcton Class 4. 7 Oct 0 Instructor: Bhsha Raj 7 Oct 03 755/8797 f... D df Matr Identtes df d d df d d... df dd dd he dervatve of a scalar functon w.r.t.
Discriminative classifier: Logistic Regression. CS534-Machine Learning
Dscrmnatve classfer: Logstc Regresson CS534-Machne Learnng 2 Logstc Regresson Gven tranng set D stc regresson learns the condtonal dstrbuton We ll assume onl to classes and a parametrc form for here s
MACHINE APPLIED MACHINE LEARNING LEARNING. Gaussian Mixture Regression
11 MACHINE APPLIED MACHINE LEARNING LEARNING MACHINE LEARNING Gaussan Mture Regresson 22 MACHINE APPLIED MACHINE LEARNING LEARNING Bref summary of last week s lecture 33 MACHINE APPLIED MACHINE LEARNING
Discriminative classifier: Logistic Regression. CS534-Machine Learning
Dscrmnatve classfer: Logstc Regresson CS534-Machne Learnng robablstc Classfer Gven an nstance, hat does a probablstc classfer do dfferentl compared to, sa, perceptron? It does not drectl predct Instead,
Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Mamum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models for
Generative classification models
CS 675 Intro to Machne Learnng Lecture Generatve classfcaton models Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square Data: D { d, d,.., dn} d, Classfcaton represents a dscrete class value Goal: learn
Speech and Language Processing
Speech and Language rocessng Lecture 3 ayesan network and ayesan nference Informaton and ommuncatons Engneerng ourse Takahro Shnozak 08//5 Lecture lan (Shnozak s part) I gves the frst 6 lectures about
e i is a random error
Chapter - The Smple Lnear Regresson Model The lnear regresson equaton s: where + β + β e for,..., and are observable varables e s a random error How can an estmaton rule be constructed for the unknown
The Gaussian classifier. Nuno Vasconcelos ECE Department, UCSD
he Gaussan classfer Nuno Vasconcelos ECE Department, UCSD Bayesan decson theory recall that we have state of the world X observatons g decson functon L[g,y] loss of predctng y wth g Bayes decson rule s
Generative and Discriminative Models. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
Generatve and Dscrmnatve Models Je Tang Department o Computer Scence & Technolog Tsnghua Unverst 202 ML as Searchng Hpotheses Space ML Methodologes are ncreasngl statstcal Rule-based epert sstems beng
Composite Hypotheses testing
Composte ypotheses testng In many hypothess testng problems there are many possble dstrbutons that can occur under each of the hypotheses. The output of the source s a set of parameters (ponts n a parameter
Chapter 2 - The Simple Linear Regression Model S =0. e i is a random error. S β2 β. This is a minimization problem. Solution is a calculus exercise.
Chapter - The Smple Lnear Regresson Model The lnear regresson equaton s: where y + = β + β e for =,..., y and are observable varables e s a random error How can an estmaton rule be constructed for the
CIS526: Machine Learning Lecture 3 (Sept 16, 2003) Linear Regression. Preparation help: Xiaoying Huang. x 1 θ 1 output... θ M x M
CIS56: achne Learnng Lecture 3 (Sept 6, 003) Preparaton help: Xaoyng Huang Lnear Regresson Lnear regresson can be represented by a functonal form: f(; θ) = θ 0 0 +θ + + θ = θ = 0 ote: 0 s a dummy attrbute
Mixture of Gaussians Expectation Maximization (EM) Part 2
Mture of Gaussans Eectaton Mamaton EM Part 2 Most of the sldes are due to Chrstoher Bsho BCS Summer School Eeter 2003. The rest of the sldes are based on lecture notes by A. Ng Lmtatons of K-means Hard
Mean Field / Variational Approximations
Mean Feld / Varatonal Appromatons resented by Jose Nuñez 0/24/05 Outlne Introducton Mean Feld Appromaton Structured Mean Feld Weghted Mean Feld Varatonal Methods Introducton roblem: We have dstrbuton but
Classification learning II
Lecture 8 Classfcaton learnng II Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square Logstc regresson model Defnes a lnear decson boundar Dscrmnant functons: g g g g here g z / e z f, g g - s a logstc functon
2E Pattern Recognition Solutions to Introduction to Pattern Recognition, Chapter 2: Bayesian pattern classification
E395 - Pattern Recognton Solutons to Introducton to Pattern Recognton, Chapter : Bayesan pattern classfcaton Preface Ths document s a soluton manual for selected exercses from Introducton to Pattern Recognton
Maximum Likelihood Estimation of Binary Dependent Variables Models: Probit and Logit. 1. General Formulation of Binary Dependent Variables Models
ECO 452 -- OE 4: Probt and Logt Models ECO 452 -- OE 4 Maxmum Lkelhood Estmaton of Bnary Dependent Varables Models: Probt and Logt hs note demonstrates how to formulate bnary dependent varables models
Linear Approximation with Regularization and Moving Least Squares
Lnear Approxmaton wth Regularzaton and Movng Least Squares Igor Grešovn May 007 Revson 4.6 (Revson : March 004). 5 4 3 0.5 3 3.5 4 Contents: Lnear Fttng...4. Weghted Least Squares n Functon Approxmaton...
β0 + β1xi and want to estimate the unknown
SLR Models Estmaton Those OLS Estmates Estmators (e ante) v. estmates (e post) The Smple Lnear Regresson (SLR) Condtons -4 An Asde: The Populaton Regresson Functon B and B are Lnear Estmators (condtonal
MLE and Bayesian Estimation. Jie Tang Department of Computer Science & Technology Tsinghua University 2012
MLE and Bayesan Estmaton Je Tang Department of Computer Scence & Technology Tsnghua Unversty 01 1 Lnear Regresson? As the frst step, we need to decde how we re gong to represent the functon f. One example:
6 Supplementary Materials
6 Supplementar Materals 61 Proof of Theorem 31 Proof Let m Xt z 1:T : l m Xt X,z 1:t Wethenhave mxt z1:t ˆm HX Xt z 1:T mxt z1:t m HX Xt z 1:T + mxt z 1:T HX We consder each of the two terms n equaton
A Tutorial on Data Reduction. Linear Discriminant Analysis (LDA) Shireen Elhabian and Aly A. Farag. University of Louisville, CVIP Lab September 2009
A utoral on Data Reducton Lnear Dscrmnant Analss (LDA) hreen Elhaban and Al A Farag Unverst of Lousvlle, CVIP Lab eptember 009 Outlne LDA objectve Recall PCA No LDA LDA o Classes Counter eample LDA C Classes
10-701/ Machine Learning, Fall 2005 Homework 3
10-701/15-781 Machne Learnng, Fall 2005 Homework 3 Out: 10/20/05 Due: begnnng of the class 11/01/05 Instructons Contact questons-10701@autonlaborg for queston Problem 1 Regresson and Cross-valdaton [40
j) = 1 (note sigma notation) ii. Continuous random variable (e.g. Normal distribution) 1. density function: f ( x) 0 and f ( x) dx = 1
Random varables Measure of central tendences and varablty (means and varances) Jont densty functons and ndependence Measures of assocaton (covarance and correlaton) Interestng result Condtonal dstrbutons
MIMA Group. Chapter 2 Bayesian Decision Theory. School of Computer Science and Technology, Shandong University. Xin-Shun SDU
Group M D L M Chapter Bayesan Decson heory Xn-Shun Xu @ SDU School of Computer Scence and echnology, Shandong Unversty Bayesan Decson heory Bayesan decson theory s a statstcal approach to data mnng/pattern
15-381: Artificial Intelligence. Regression and cross validation
15-381: Artfcal Intellgence Regresson and cross valdaton Where e are Inputs Densty Estmator Probablty Inputs Classfer Predct category Inputs Regressor Predct real no. Today Lnear regresson Gven an nput
1. Inference on Regression Parameters a. Finding Mean, s.d and covariance amongst estimates. 2. Confidence Intervals and Working Hotelling Bands
Content. Inference on Regresson Parameters a. Fndng Mean, s.d and covarance amongst estmates.. Confdence Intervals and Workng Hotellng Bands 3. Cochran s Theorem 4. General Lnear Testng 5. Measures of
Chapter 7 Generalized and Weighted Least Squares Estimation. In this method, the deviation between the observed and expected values of
Chapter 7 Generalzed and Weghted Least Squares Estmaton The usual lnear regresson model assumes that all the random error components are dentcally and ndependently dstrbuted wth constant varance. When
The Multiple Classical Linear Regression Model (CLRM): Specification and Assumptions. 1. Introduction
ECONOMICS 5* -- NOTE (Summary) ECON 5* -- NOTE The Multple Classcal Lnear Regresson Model (CLRM): Specfcaton and Assumptons. Introducton CLRM stands for the Classcal Lnear Regresson Model. The CLRM s also
xp(x µ) = 0 p(x = 0 µ) + 1 p(x = 1 µ) = µ
CSE 455/555 Sprng 2013 Homework 7: Parametrc Technques Jason J. Corso Computer Scence and Engneerng SUY at Buffalo jcorso@buffalo.edu Solutons by Yngbo Zhou Ths assgnment does not need to be submtted and
Lecture 10: Dimensionality reduction
Lecture : Dmensonalt reducton g The curse of dmensonalt g Feature etracton s. feature selecton g Prncpal Components Analss g Lnear Dscrmnant Analss Intellgent Sensor Sstems Rcardo Guterrez-Osuna Wrght
Lecture 3 Stat102, Spring 2007
Lecture 3 Stat0, Sprng 007 Chapter 3. 3.: Introducton to regresson analyss Lnear regresson as a descrptve technque The least-squares equatons Chapter 3.3 Samplng dstrbuton of b 0, b. Contnued n net lecture
EM and Structure Learning
EM and Structure Learnng Le Song Machne Learnng II: Advanced Topcs CSE 8803ML, Sprng 2012 Partally observed graphcal models Mxture Models N(μ 1, Σ 1 ) Z X N N(μ 2, Σ 2 ) 2 Gaussan mxture model Consder
+, where 0 x N - n. k k
CO 745, Mdterm Len Cabrera. A multle choce eam has questons, each of whch has ossble answers. A student nows the correct answer to n of these questons. For the remanng - n questons, he checs the answers
Linear discriminants. Nuno Vasconcelos ECE Department, UCSD
Lnear dscrmnants Nuno Vasconcelos ECE Department UCSD Classfcaton a classfcaton problem as to tpes of varables e.g. X - vector of observatons features n te orld Y - state class of te orld X R 2 fever blood
Retrieval Models: Language models
CS-590I Informaton Retreval Retreval Models: Language models Luo S Department of Computer Scence Purdue Unversty Introducton to language model Ungram language model Document language model estmaton Maxmum
Fall 2012 Analysis of Experimental Measurements B. Eisenstein/rev. S. Errede. . For P such independent random variables (aka degrees of freedom): 1 =
Fall Analss of Epermental Measurements B. Esensten/rev. S. Errede More on : The dstrbuton s the.d.f. for a (normalzed sum of squares of ndependent random varables, each one of whch s dstrbuted as N (,.
Parameter estimation class 5
Parameter estmaton class 5 Multple Ve Geometr Comp 9-89 Marc Pollefes Content Background: Projectve geometr (D, 3D), Parameter estmaton, Algortm evaluaton. Sngle Ve: Camera model, Calbraton, Sngle Ve Geometr.
Independent Component Analysis
Indeendent Comonent Analyss Mture Data Data that are mngled from multle sources May not now how many sources May not now the mng mechansm Good Reresentaton Uncorrelated, nformaton-bearng comonents PCA
CHAPTER 3: BAYESIAN DECISION THEORY
HATER 3: BAYESIAN DEISION THEORY Decson mang under uncertanty 3 Data comes from a process that s completely not nown The lac of nowledge can be compensated by modelng t as a random process May be the underlyng
Classification as a Regression Problem
Target varable y C C, C,, ; Classfcaton as a Regresson Problem { }, 3 L C K To treat classfcaton as a regresson problem we should transform the target y nto numercal values; The choce of numercal class
Feature Selection: Part 1
CSE 546: Machne Learnng Lecture 5 Feature Selecton: Part 1 Instructor: Sham Kakade 1 Regresson n the hgh dmensonal settng How do we learn when the number of features d s greater than the sample sze n?
Finite Mixture Models and Expectation Maximization. Most slides are from: Dr. Mario Figueiredo, Dr. Anil Jain and Dr. Rong Jin
Fnte Mxture Models and Expectaton Maxmzaton Most sldes are from: Dr. Maro Fgueredo, Dr. Anl Jan and Dr. Rong Jn Recall: The Supervsed Learnng Problem Gven a set of n samples X {(x, y )},,,n Chapter 3 of
Generalized Linear Methods
Generalzed Lnear Methods 1 Introducton In the Ensemble Methods the general dea s that usng a combnaton of several weak learner one could make a better learner. More formally, assume that we have a set
x i1 =1 for all i (the constant ).
Chapter 5 The Multple Regresson Model Consder an economc model where the dependent varable s a functon of K explanatory varables. The economc model has the form: y = f ( x,x,..., ) xk Approxmate ths by
Estimation of Non-Gaussian Probability Density by Dynamic Bayesian Networks
June -5, KIEX, Geongg-Do, Korea Estmaton of on-gaussan Probablt Denst b Dnamc Baesan etwors Hun C. Cho *, Sam. Fadal **, and Kwon S. Lee *** * Electrcal Engneerng Dept./60, Unverst of evada-reno, V, 89557,
9 : Learning Partially Observed GM : EM Algorithm
10-708: Probablstc Graphcal Models 10-708, Sprng 2014 9 : Learnng Partally Observed GM : EM Algorthm Lecturer: Erc P. Xng Scrbes: Rohan Ramanath, Rahul Goutam 1 Generalzed Iteratve Scalng In ths secton,
Correlation and Regression. Correlation 9.1. Correlation. Chapter 9
Chapter 9 Correlaton and Regresson 9. Correlaton Correlaton A correlaton s a relatonshp between two varables. The data can be represented b the ordered pars (, ) where s the ndependent (or eplanator) varable,
Logistic Regression. CAP 5610: Machine Learning Instructor: Guo-Jun QI
Logstc Regresson CAP 561: achne Learnng Instructor: Guo-Jun QI Bayes Classfer: A Generatve model odel the posteror dstrbuton P(Y X) Estmate class-condtonal dstrbuton P(X Y) for each Y Estmate pror dstrbuton
Computation of Higher Order Moments from Two Multinomial Overdispersion Likelihood Models
Computaton of Hgher Order Moments from Two Multnomal Overdsperson Lkelhood Models BY J. T. NEWCOMER, N. K. NEERCHAL Department of Mathematcs and Statstcs, Unversty of Maryland, Baltmore County, Baltmore,
Machine Learning for Signal Processing
Machne Learnng for Sgnal Processng Lecture 6: Optmzaton Data-drven Representatons Abelno Jmenez Generalzes to nequalty constrants Optmzaton problem wth constrants mn s. t. g ( x) Lagrange multplers l ³
ENG 8801/ Special Topics in Computer Engineering: Pattern Recognition. Memorial University of Newfoundland Pattern Recognition
EG 880/988 - Specal opcs n Computer Engneerng: Pattern Recognton Memoral Unversty of ewfoundland Pattern Recognton Lecture 7 May 3, 006 http://wwwengrmunca/~charlesr Offce Hours: uesdays hursdays 8:30-9:30
Outline. Bayesian Networks: Maximum Likelihood Estimation and Tree Structure Learning. Our Model and Data. Outline
Outlne Bayesan Networks: Maxmum Lkelhood Estmaton and Tree Structure Learnng Huzhen Yu janey.yu@cs.helsnk.f Dept. Computer Scence, Unv. of Helsnk Probablstc Models, Sprng, 200 Notces: I corrected a number
Lecture Notes on Linear Regression
Lecture Notes on Lnear Regresson Feng L fl@sdueducn Shandong Unversty, Chna Lnear Regresson Problem In regresson problem, we am at predct a contnuous target value gven an nput feature vector We assume
Why Bayesian? 3. Bayes and Normal Models. State of nature: class. Decision rule. Rev. Thomas Bayes ( ) Bayes Theorem (yes, the famous one)
Why Bayesan? 3. Bayes and Normal Models Alex M. Martnez alex@ece.osu.edu Handouts Handoutsfor forece ECE874 874Sp Sp007 If all our research (n PR was to dsappear and you could only save one theory, whch
Chapter 14 Simple Linear Regression
Chapter 4 Smple Lnear Regresson Chapter 4 - Smple Lnear Regresson Manageral decsons often are based on the relatonshp between two or more varables. Regresson analss can be used to develop an equaton showng
CS 3710: Visual Recognition Classification and Detection. Adriana Kovashka Department of Computer Science January 13, 2015
CS 3710: Vsual Recognton Classfcaton and Detecton Adrana Kovashka Department of Computer Scence January 13, 2015 Plan for Today Vsual recognton bascs part 2: Classfcaton and detecton Adrana s research
Support Vector Machines
CS 2750: Machne Learnng Support Vector Machnes Prof. Adrana Kovashka Unversty of Pttsburgh February 17, 2016 Announcement Homework 2 deadlne s now 2/29 We ll have covered everythng you need today or at
LINEAR REGRESSION ANALYSIS. MODULE IX Lecture Multicollinearity
LINEAR REGRESSION ANALYSIS MODULE IX Lecture - 30 Multcollnearty Dr. Shalabh Department of Mathematcs and Statstcs Indan Insttute of Technology Kanpur 2 Remedes for multcollnearty Varous technques have
3.1 ML and Empirical Distribution
67577 Intro. to Machne Learnng Fall semester, 2008/9 Lecture 3: Maxmum Lkelhood/ Maxmum Entropy Dualty Lecturer: Amnon Shashua Scrbe: Amnon Shashua 1 In the prevous lecture we defned the prncple of Maxmum
Statistics for Economics & Business
Statstcs for Economcs & Busness Smple Lnear Regresson Learnng Objectves In ths chapter, you learn: How to use regresson analyss to predct the value of a dependent varable based on an ndependent varable
Expectation Maximization Mixture Models
-755 Machne Learnng for Sgnal Processng Mxture Models Understandng (and Predctng Data Many dfferent data streams around us We process, understand and respond What s the response based on? Class 0. Oct
Fall 2012 Analysis of Experimental Measurements B. Eisenstein/rev. S. Errede
Fall 0 Analyss of Expermental easurements B. Esensten/rev. S. Errede We now reformulate the lnear Least Squares ethod n more general terms, sutable for (eventually extendng to the non-lnear case, and also
INF 5860 Machine learning for image classification. Lecture 3 : Image classification and regression part II Anne Solberg January 31, 2018
INF 5860 Machne learnng for mage classfcaton Lecture 3 : Image classfcaton and regresson part II Anne Solberg January 3, 08 Today s topcs Multclass logstc regresson and softma Regularzaton Image classfcaton
b ), which stands for uniform distribution on the interval a x< b. = 0 elsewhere
Fall Analyss of Epermental Measurements B. Esensten/rev. S. Errede Some mportant probablty dstrbutons: Unform Bnomal Posson Gaussan/ormal The Unform dstrbuton s often called U( a, b ), hch stands for unform
Machine learning: Density estimation
CS 70 Foundatons of AI Lecture 3 Machne learnng: ensty estmaton Mlos Hauskrecht mlos@cs.ptt.edu 539 Sennott Square ata: ensty estmaton {.. n} x a vector of attrbute values Objectve: estmate the model of
THE ROYAL STATISTICAL SOCIETY 2006 EXAMINATIONS SOLUTIONS HIGHER CERTIFICATE
THE ROYAL STATISTICAL SOCIETY 6 EXAMINATIONS SOLUTIONS HIGHER CERTIFICATE PAPER I STATISTICAL THEORY The Socety provdes these solutons to assst canddates preparng for the eamnatons n future years and for
Dr. Shalabh Department of Mathematics and Statistics Indian Institute of Technology Kanpur
Analyss of Varance and Desgn of Exerments-I MODULE III LECTURE - 2 EXPERIMENTAL DESIGN MODELS Dr. Shalabh Deartment of Mathematcs and Statstcs Indan Insttute of Technology Kanur 2 We consder the models
Maximum Likelihood Estimation (MLE)
Maxmum Lkelhood Estmaton (MLE) Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 01 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
1 Convex Optimization
Convex Optmzaton We wll consder convex optmzaton problems. Namely, mnmzaton problems where the objectve s convex (we assume no constrants for now). Such problems often arse n machne learnng. For example,
Limited Dependent Variables and Panel Data. Tibor Hanappi
Lmted Dependent Varables and Panel Data Tbor Hanapp 30.06.2010 Lmted Dependent Varables Dscrete: Varables that can take onl a countable number of values Censored/Truncated: Data ponts n some specfc range
Homework Assignment 3 Due in class, Thursday October 15
Homework Assgnment 3 Due n class, Thursday October 15 SDS 383C Statstcal Modelng I 1 Rdge regresson and Lasso 1. Get the Prostrate cancer data from http://statweb.stanford.edu/~tbs/elemstatlearn/ datasets/prostate.data.
LECTURE 9 CANONICAL CORRELATION ANALYSIS
LECURE 9 CANONICAL CORRELAION ANALYSIS Introducton he concept of canoncal correlaton arses when we want to quantfy the assocatons between two sets of varables. For example, suppose that the frst set of
Laboratory 3: Method of Least Squares
Laboratory 3: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly they are correlated wth
Feature Selection & Dynamic Tracking F&P Textbook New: Ch 11, Old: Ch 17 Guido Gerig CS 6320, Spring 2013
Feature Selecton & Dynamc Trackng F&P Textbook New: Ch 11, Old: Ch 17 Gudo Gerg CS 6320, Sprng 2013 Credts: Materal Greg Welch & Gary Bshop, UNC Chapel Hll, some sldes modfed from J.M. Frahm/ M. Pollefeys,
Chapter 13: Multiple Regression
Chapter 13: Multple Regresson 13.1 Developng the multple-regresson Model The general model can be descrbed as: It smplfes for two ndependent varables: The sample ft parameter b 0, b 1, and b are used to
β0 + β1xi. You are interested in estimating the unknown parameters β
Revsed: v3 Ordnar Least Squares (OLS): Smple Lnear Regresson (SLR) Analtcs The SLR Setup Sample Statstcs Ordnar Least Squares (OLS): FOCs and SOCs Back to OLS and Sample Statstcs Predctons (and Resduals)
Clustering & Unsupervised Learning
Clusterng & Unsupervsed Learnng Ken Kreutz-Delgado (Nuno Vasconcelos) ECE 175A Wnter 2012 UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y
P R. Lecture 4. Theory and Applications of Pattern Recognition. Dept. of Electrical and Computer Engineering /
Theory and Applcatons of Pattern Recognton 003, Rob Polkar, Rowan Unversty, Glassboro, NJ Lecture 4 Bayes Classfcaton Rule Dept. of Electrcal and Computer Engneerng 0909.40.0 / 0909.504.04 Theory & Applcatons
Bayesian decision theory. Nuno Vasconcelos ECE Department, UCSD
Bayesan decson theory Nuno Vasconcelos ECE Department UCSD Notaton the notaton n DHS s qute sloppy e.. show that error error z z dz really not clear what ths means we wll use the follown notaton subscrpts
Basic Business Statistics, 10/e
Chapter 13 13-1 Basc Busness Statstcs 11 th Edton Chapter 13 Smple Lnear Regresson Basc Busness Statstcs, 11e 009 Prentce-Hall, Inc. Chap 13-1 Learnng Objectves In ths chapter, you learn: How to use regresson
Laboratory 1c: Method of Least Squares
Lab 1c, Least Squares Laboratory 1c: Method of Least Squares Introducton Consder the graph of expermental data n Fgure 1. In ths experment x s the ndependent varable and y the dependent varable. Clearly
Supporting Information
Supportng Informaton The neural network f n Eq. 1 s gven by: f x l = ReLU W atom x l + b atom, 2 where ReLU s the element-wse rectfed lnear unt, 21.e., ReLUx = max0, x, W atom R d d s the weght matrx to
Which Separator? Spring 1
Whch Separator? 6.034 - Sprng 1 Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng Whch Separator? Mamze the margn to closest ponts 6.034 - Sprng 3 Margn of a pont " # y (w $ + b) proportonal
Econ Statistical Properties of the OLS estimator. Sanjaya DeSilva
Econ 39 - Statstcal Propertes of the OLS estmator Sanjaya DeSlva September, 008 1 Overvew Recall that the true regresson model s Y = β 0 + β 1 X + u (1) Applyng the OLS method to a sample of data, we estmate
Rockefeller College University at Albany
Rockefeller College Unverst at Alban PAD 705 Handout: Maxmum Lkelhood Estmaton Orgnal b Davd A. Wse John F. Kenned School of Government, Harvard Unverst Modfcatons b R. Karl Rethemeer Up to ths pont n
Chat eld, C. and A.J.Collins, Introduction to multivariate analysis. Chapman & Hall, 1980
MT07: Multvarate Statstcal Methods Mke Tso: emal mke.tso@manchester.ac.uk Webpage for notes: http://www.maths.manchester.ac.uk/~mkt/new_teachng.htm. Introducton to multvarate data. Books Chat eld, C. and
[The following data appear in Wooldridge Q2.3.] The table below contains the ACT score and college GPA for eight college students.
![[The following data appear in Wooldridge Q2.3.] The table below contains the ACT score and college GPA for eight college students.](/thumbs/87/96001374.jpg "[The following data appear in Wooldridge Q2.3.] The table below contains the ACT score and college GPA for eight college students.")
PPOL 59-3 Problem Set Exercses n Smple Regresson Due n class /8/7 In ths problem set, you are asked to compute varous statstcs by hand to gve you a better sense of the mechancs of the Pearson correlaton
Multigradient for Neural Networks for Equalizers 1
Multgradent for Neural Netorks for Equalzers 1 Chulhee ee, Jnook Go and Heeyoung Km Department of Electrcal and Electronc Engneerng Yonse Unversty 134 Shnchon-Dong, Seodaemun-Ku, Seoul 1-749, Korea ABSTRACT
Chapter 3. Two-Variable Regression Model: The Problem of Estimation
Chapter 3. Two-Varable Regresson Model: The Problem of Estmaton Ordnary Least Squares Method (OLS) Recall that, PRF: Y = β 1 + β X + u Thus, snce PRF s not drectly observable, t s estmated by SRF; that
Mixture o f of Gaussian Gaussian clustering Nov
Mture of Gaussan clusterng Nov 11 2009 Soft vs hard lusterng Kmeans performs Hard clusterng: Data pont s determnstcally assgned to one and only one cluster But n realty clusters may overlap Soft-clusterng:
3.1 Expectation of Functions of Several Random Variables. )' be a k-dimensional discrete or continuous random vector, with joint PMF p (, E X E X1 E X
Statstcs 1: Probablty Theory II 37 3 EPECTATION OF SEVERAL RANDOM VARIABLES As n Probablty Theory I, the nterest n most stuatons les not on the actual dstrbuton of a random vector, but rather on a number
Clustering & (Ken Kreutz-Delgado) UCSD
Clusterng & Unsupervsed Learnng Nuno Vasconcelos (Ken Kreutz-Delgado) UCSD Statstcal Learnng Goal: Gven a relatonshp between a feature vector x and a vector y, and d data samples (x,y ), fnd an approxmatng
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS. Version 2 ECE IIT, Kharagpur
Module 3 LOSSY IMAGE COMPRESSION SYSTEMS Verson ECE IIT, Kharagpur Lesson 6 Theory of Quantzaton Verson ECE IIT, Kharagpur Instructonal Objectves At the end of ths lesson, the students should be able to:
Multilayer Perceptrons and Backpropagation. Perceptrons. Recap: Perceptrons. Informatics 1 CG: Lecture 6. Mirella Lapata
Multlayer Perceptrons and Informatcs CG: Lecture 6 Mrella Lapata School of Informatcs Unversty of Ednburgh mlap@nf.ed.ac.uk Readng: Kevn Gurney s Introducton to Neural Networks, Chapters 5 6.5 January,