Lecture 7: Linear Classification Methods
|
|
- Hillary Johns
- 5 years ago
- Views:
Transcription
1 Homeork
2 Homeork
3 Lecture 7: Liear lassificatio Methods Fial rojects? Grous Toics Proosal eek 5 Lecture is oster sessio, Jacobs Hall Lobb, sacks Fial reort 5 Jue.
4 What is liear classificatio? lassificatio is itrisicall oliear It uts oidetical thigs i the same class, so a differece i iut vector sometimes causes ero chage i the aser Liear classificatio meas that the art that adats is liear The adative art is folloed b a fied oliearit. It ma be receded b a fied oliearit e.g. oliear basis fuctios. T +, Decisio f adative liear fuctio fied oliear fuctio.5
5 Reresetig the target values for classificatio For to classes, e use a sigle valued outut that has target values for the ositive class ad or for the other class For robabilistic class labels the target value ca the be Pt ad the model outut ca also rereset P. For N classes e ofte use a vector of N target values cotaiig a sigle for the correct class ad eros elsehere. For robabilistic labels e ca the use a vector of class robabilities as the target vector.
6 Three aroaches to classificatio Use discrimiat fuctios directl ithout robabilities: overt iut vector ito real values. A simle oeratio like thresholdig ca get the class. hoose real values to maimie the useable iformatio about the class label that is i the real value. Ifer coditioal class robabilities: class k omute the coditioal robabilit of each class. The make a decisio that miimies some loss fuctio omare the robabilit of the iut uder searate, classsecific, geerative models. E.g. fit a multivariate Gaussia to the iut vectors of each class ad see hich Gaussia makes a test data vector most robable. Is this the best bet?
7 Discrimiat fuctios The laar decisio surface i datasace for the simle liear discrimiat fuctio: T + ³ X o lae > > Distace from lae
8 Discrimiat fuctios for N> classes Oe ossibilit is to use N toa discrimiat fuctios. Each fuctio discrimiates oe class from the rest. Aother ossibilit is to use NN/ toa discrimiat fuctios Each fuctio discrimiates betee to articular classes. Both these methods have roblems More tha oe good aser Toa refereces eed ot be trasitive!
9 Use N discrimiat fuctios, ad ick the ma., k k A simle solutio 4.. i, j k... This is guarateed to give cosistet ad cove decisio regios if is liear. A imlies a + a > a + a A > j for A ad ositive a B k that j B A > j B B Decisio boudar?
10 Maimum Likelihood ad Least Squares from lecture 3 omutig the gradiet ad settig it to ero ields Solvig for, here The MoorePerose seudoiverse,.
11 LSQ for classificatio Each class k is described b its o liear model so that k T k + k 4.3 here k,...,k. We ca coveietl grou these together usig vector otatio so that WT 4.4 osider a traiig set {" #, $ # }, ' N Defie X ad T { } { } LSQ solutio: W XT X XT T X T 4.6 Ad redictio X WT T T X T. 4.7
12 Usig least squares for classificatio It does ot ork as ell as better methods, but it is eas: It reduces classificatio to least squares regressio. logistic regressio least squares regressio
13 PA do t ork ell
14 icture shoig the advatage of Fisher s liear discrimiat Whe rojected oto the lie joiig the class meas, the classes are ot ell searated. Fisher chooses a directio that makes the rojected classes much tighter, eve though their rojected meas are less far aart.
15 Math of Fisher s liear discrimiats What liear trasformatio is best for discrimiatio? The rojectio oto the vector searatig the class meas seems sesible: T µ m m But e also at small variace ithi each class: s s å e å e m m Fisher s objective fuctio is: J m s m + s betee ithi
16 : m m S m m m m S m m m m S S S µ + + Î Î å å W T T W T B W T B T solutio Otimal s s m m J More math of Fisher s liear discrimiats
17 We have robalistic classificatio!
18 Probabilistic Geerative Models for Discrimiatio Bisho 96 Use a geerative model of the iut vectors for each class, ad see hich model makes a test iut vector most robable. The osterior robabilit of class is: l l here e + + is called the logit ad is give b the log odds
19 A eamle for cotiuous iuts Assume iut vectors for each class are Gaussia, all classes have the same covariace matri. For to classes, ad, the osterior is a logistic: { } e k T k k a µ µ S l T T T µ Σ µ µ Σ µ µ µ Σ s iverse covariace matri ormaliig costat
20 ! #$ % & % % & * % *
21 The role of the iverse covariace matri If the Gaussia is sherical o eed to orr about the covariace matri. So, start b trasformig the data sace to make the Gaussia sherical This is called hiteig the data. It remultilies b the matri square root of the iverse covariace matri. I trasformed sace, the eight vector is the differece betee trasformed meas. Σ gives the for aff ad T aff gives for same value as : Σ µ µ µ Σ T aff Σ aff µ
22 The osterior he the covariace matrices are differet for differet classes Bisho Fig The decisio surface is laar he the covariace matrices are the same ad quadratic he ot.
23 Beroulli distributio Radom variable!, oi fliig: heads, tails Beroulli Distributio ML for Beroulli Give:
24 The logistic fuctio The outut is a smooth fuctio of the iuts ad the eights. d d e i i i i T + + s.5 Its odd to eress it i terms of.
25 ! " # $ & $ Observatios Likelihood & $! $,! 4 $, Loglikelihood Miimie log like Derivative Logistic regressio Bisho 5 EF! 4 $,
26 Logistic regressio age 5 Whe there are ol to classes e ca model the coditioal robabilit of the ositive class as T s + here s + e If e use the right error fuctio, somethig ice haes: The gradiet of the logistic ad the gradiet of the error fuctio cacel each other: E l t, ÑE å t N
27 The atural error fuctio for the logistic Fittig logistic model usig maimum likelihood, requires miimiig the egative log robabilit of the correct aser summed over the traiig set. l l l N N t t t E t t t E + + å å error derivative o traiig case if t if t
28 Usig the chai rule to get the error derivatives T t d d E E d d t E,, +
29 Softma fuctio For the case of K>classes, e have k k k j j j ea k j ea j 4.6 a k l k k l is also ko as the softma fuctio, as it reresets
30 rossetro or softma fuctio for multiclass classificatio i i j i j j i j j j i i i i j i t E E t E e e j i å å å l The outut uits use a olocal oliearit: The atural cost fuctio is the egative log rob of the right aser The steeess of E eactl balaces the flatess of the softma. outut uits 3 3 target value
31 A secial case of softma for to classes So the logistic is just a secial case that avoids usig redudat arameters: Addig the same costat to both ad has o effect. The overarameteriatio of the softma is because the robabilities must add to. e e e e + +
Lecture 7: Linear Classification Methods
Homeork Homeork Lecture 7: Linear lassification Methods Final rojects? Grous oics Proosal eek 5 Lecture is oster session, Jacobs Hall Lobby, snacks Final reort 5 June. What is linear classification? lassification
More informationClassification with linear models
Lecture 8 Classificatio with liear models Milos Hauskrecht milos@cs.pitt.edu 539 Seott Square Geerative approach to classificatio Idea:. Represet ad lear the distributio, ). Use it to defie probabilistic
More informationMachine Learning. Logistic Regression -- generative verses discriminative classifier. Le Song /15-781, Spring 2008
Machie Learig 070/578 Srig 008 Logistic Regressio geerative verses discrimiative classifier Le Sog Lecture 5 Setember 4 0 Based o slides from Eric Xig CMU Readig: Cha. 3..34 CB Geerative vs. Discrimiative
More informationFMA901F: Machine Learning Lecture 4: Linear Models for Classification. Cristian Sminchisescu
FMA90F: Machie Learig Lecture 4: Liear Models for Classificatio Cristia Smichisescu Liear Classificatio Classificatio is itrisically o liear because of the traiig costraits that place o idetical iputs
More informationMachine Learning Regression I Hamid R. Rabiee [Slides are based on Bishop Book] Spring
Machie Learig Regressio I Hamid R. Rabiee [Slides are based o Bishop Book] Sprig 015 http://ce.sharif.edu/courses/93-94//ce717-1 Liear Regressio Liear regressio: ivolves a respose variable ad a sigle predictor
More informationChapter 7. Support Vector Machine
Chapter 7 Support Vector Machie able of Cotet Margi ad support vectors SVM formulatio Slack variables ad hige loss SVM for multiple class SVM ith Kerels Relevace Vector Machie Support Vector Machie (SVM)
More informationINF Introduction to classifiction Anne Solberg Based on Chapter 2 ( ) in Duda and Hart: Pattern Classification
INF 4300 90 Itroductio to classifictio Ae Solberg ae@ifiuioo Based o Chapter -6 i Duda ad Hart: atter Classificatio 90 INF 4300 Madator proect Mai task: classificatio You must implemet a classificatio
More informationLecture 4. Hw 1 and 2 will be reoped after class for every body. New deadline 4/20 Hw 3 and 4 online (Nima is lead)
Lecture 4 Homework Hw 1 ad 2 will be reoped after class for every body. New deadlie 4/20 Hw 3 ad 4 olie (Nima is lead) Pod-cast lecture o-lie Fial projects Nima will register groups ext week. Email/tell
More information( ) = is larger than. the variance of X V
Stat 400, sectio 6. Methods of Poit Estimatio otes by Tim Pilachoski A oit estimate of a arameter is a sigle umber that ca be regarded as a sesible value for The selected statistic is called the oit estimator
More informationTomoki Toda. Augmented Human Communication Laboratory Graduate School of Information Science
Seuetial Data Modelig d class Basics of seuetial data odelig ooki oda Augeted Hua Couicatio Laboratory Graduate School of Iforatio Sciece Basic Aroaches How to efficietly odel joit robability of high diesioal
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5
CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio
More informationHypothesis Testing. H 0 : θ 1 1. H a : θ 1 1 (but > 0... required in distribution) Simple Hypothesis - only checks 1 value
Hyothesis estig ME's are oit estimates of arameters/coefficiets really have a distributio Basic Cocet - develo regio i which we accet the hyothesis ad oe where we reject it H - reresets all ossible values
More informationOutline. Linear regression. Regularization functions. Polynomial curve fitting. Stochastic gradient descent for regression. MLE for regression
REGRESSION 1 Outlie Liear regressio Regularizatio fuctios Polyomial curve fittig Stochastic gradiet descet for regressio MLE for regressio Step-wise forward regressio Regressio methods Statistical techiques
More informationECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015
ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationChapter 12 EM algorithms The Expectation-Maximization (EM) algorithm is a maximum likelihood method for models that have hidden variables eg. Gaussian
Chapter 2 EM algorithms The Expectatio-Maximizatio (EM) algorithm is a maximum likelihood method for models that have hidde variables eg. Gaussia Mixture Models (GMMs), Liear Dyamic Systems (LDSs) ad Hidde
More informationWEIGHTED LEAST SQUARES - used to give more emphasis to selected points in the analysis. Recall, in OLS we minimize Q =! % =!
WEIGHTED LEAST SQUARES - used to give more emphasis to selected poits i the aalysis What are eighted least squares?! " i=1 i=1 Recall, i OLS e miimize Q =! % =!(Y - " - " X ) or Q = (Y_ - X "_) (Y_ - X
More informationSpecial Modeling Techniques
Colorado School of Mies CHEN43 Secial Modelig Techiques Secial Modelig Techiques Summary of Toics Deviatio Variables No-Liear Differetial Equatios 3 Liearizatio of ODEs for Aroximate Solutios 4 Coversio
More informationThree classification models Discriminant Model: learn the decision boundary directly and apply it to determine the class of each data point
Review of Last Wee Three classificatio models Discrimiat Model: lear the decisio boudary directly ad aly it to determie the class of each data oit Discrimiative Model: lear PY directly Geerative Model:
More informationStep 1: Function Set. Otherwise, output C 2. Function set: Including all different w and b
Logistic Regressio Step : Fuctio Set We wat to fid P w,b C x σ z = + exp z If P w,b C x.5, output C Otherwise, output C 2 z P w,b C x = σ z z = w x + b = w i x i + b i z Fuctio set: f w,b x = P w,b C x
More informationQuestions and Answers on Maximum Likelihood
Questios ad Aswers o Maximum Likelihood L. Magee Fall, 2008 1. Give: a observatio-specific log likelihood fuctio l i (θ) = l f(y i x i, θ) the log likelihood fuctio l(θ y, X) = l i(θ) a data set (x i,
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationChapter 7. Transformation
Chapter 7 Trasformatio 7.. Trasformatio Is liear regressio appropriate? 7.. Trasformatio The assumptio of liear relatioship does ot alwas hold We ca trasform The predictor The respose Both to achieve the
More informationPUTNAM TRAINING PROBABILITY
PUTNAM TRAINING PROBABILITY (Last udated: December, 207) Remark. This is a list of exercises o robability. Miguel A. Lerma Exercises. Prove that the umber of subsets of {, 2,..., } with odd cardiality
More informationElementary manipulations of probabilities
Elemetary maipulatios of probabilities Set probability of multi-valued r.v. {=Odd} = +3+5 = /6+/6+/6 = ½ X X,, X i j X i j Multi-variat distributio: Joit probability: X true true X X,, X X i j i j X X
More informationInverse Matrix. A meaning that matrix B is an inverse of matrix A.
Iverse Matrix Two square matrices A ad B of dimesios are called iverses to oe aother if the followig holds, AB BA I (11) The otio is dual but we ofte write 1 B A meaig that matrix B is a iverse of matrix
More informationFirst Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >
More informationLogit regression Logit regression
Logit regressio Logit regressio models the probability of Y= as the cumulative stadard logistic distributio fuctio, evaluated at z = β 0 + β X: Pr(Y = X) = F(β 0 + β X) F is the cumulative logistic distributio
More informationSample Size Estimation in the Proportional Hazards Model for K-sample or Regression Settings Scott S. Emerson, M.D., Ph.D.
ample ie Estimatio i the Proportioal Haards Model for K-sample or Regressio ettigs cott. Emerso, M.D., Ph.D. ample ie Formula for a Normally Distributed tatistic uppose a statistic is kow to be ormally
More informationLecture 22: Review for Exam 2. 1 Basic Model Assumptions (without Gaussian Noise)
Lecture 22: Review for Exam 2 Basic Model Assumptios (without Gaussia Noise) We model oe cotiuous respose variable Y, as a liear fuctio of p umerical predictors, plus oise: Y = β 0 + β X +... β p X p +
More information15-780: Graduate Artificial Intelligence. Density estimation
5-780: Graduate Artificial Itelligece Desity estimatio Coditioal Probability Tables (CPT) But where do we get them? P(B)=.05 B P(E)=. E P(A B,E) )=.95 P(A B, E) =.85 P(A B,E) )=.5 P(A B, E) =.05 A P(J
More informationExponential Families and Bayesian Inference
Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where
More informationA Note on Sums of Independent Random Variables
Cotemorary Mathematics Volume 00 XXXX A Note o Sums of Ideedet Radom Variables Pawe l Hitczeko ad Stehe Motgomery-Smith Abstract I this ote a two sided boud o the tail robability of sums of ideedet ad
More informationChapter 18: Sampling Distribution Models
Chater 18: Samlig Distributio Models This is the last bit of theory before we get back to real-world methods. Samlig Distributios: The Big Idea Take a samle ad summarize it with a statistic. Now take aother
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationOutline. CSCI-567: Machine Learning (Spring 2019) Outline. Prof. Victor Adamchik. Mar. 26, 2019
Outlie CSCI-567: Machie Learig Sprig 209 Gaussia mixture models Prof. Victor Adamchik 2 Desity estimatio U of Souther Califoria Mar. 26, 209 3 Naive Bayes Revisited March 26, 209 / 57 March 26, 209 2 /
More informationLecture 2 October 11
Itroductio to probabilistic graphical models 203/204 Lecture 2 October Lecturer: Guillaume Oboziski Scribes: Aymeric Reshef, Claire Verade Course webpage: http://www.di.es.fr/~fbach/courses/fall203/ 2.
More informationThe Bayesian Learning Framework. Back to Maximum Likelihood. Naïve Bayes. Simple Example: Coin Tosses. Given a generative model
Back to Maximum Likelihood Give a geerative model f (x, y = k) =π k f k (x) Usig a geerative modellig approach, we assume a parametric form for f k (x) =f (x; k ) ad compute the MLE θ of θ =(π k, k ) k=
More informationChapter 6: BINOMIAL PROBABILITIES
Charles Bocelet, Probability, Statistics, ad Radom Sigals," Oxford Uiversity Press, 016. ISBN: 978-0-19-00051-0 Chater 6: BINOMIAL PROBABILITIES Sectios 6.1 Basics of the Biomial Distributio 6. Comutig
More informationBivariate Sample Statistics Geog 210C Introduction to Spatial Data Analysis. Chris Funk. Lecture 7
Bivariate Sample Statistics Geog 210C Itroductio to Spatial Data Aalysis Chris Fuk Lecture 7 Overview Real statistical applicatio: Remote moitorig of east Africa log rais Lead up to Lab 5-6 Review of bivariate/multivariate
More informationElliptic Curves Spring 2017 Problem Set #1
18.783 Ellitic Curves Srig 017 Problem Set #1 These roblems are related to the material covered i Lectures 1-3. Some of them require the use of Sage; you will eed to create a accout at the SageMathCloud.
More informationSeptember 2012 C1 Note. C1 Notes (Edexcel) Copyright - For AS, A2 notes and IGCSE / GCSE worksheets 1
September 0 s (Edecel) Copyright www.pgmaths.co.uk - For AS, A otes ad IGCSE / GCSE worksheets September 0 Copyright www.pgmaths.co.uk - For AS, A otes ad IGCSE / GCSE worksheets September 0 Copyright
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationClassification of DT signals
Comlex exoetial A discrete time sigal may be comlex valued I digital commuicatios comlex sigals arise aturally A comlex sigal may be rereseted i two forms: jarg { z( ) } { } z ( ) = Re { z ( )} + jim {
More information18.01 Calculus Jason Starr Fall 2005
Lecture 18. October 5, 005 Homework. Problem Set 5 Part I: (c). Practice Problems. Course Reader: 3G 1, 3G, 3G 4, 3G 5. 1. Approximatig Riema itegrals. Ofte, there is o simpler expressio for the atiderivative
More informationPattern Classification
Patter Classificatio All materials i these slides were tae from Patter Classificatio (d ed) by R. O. Duda, P. E. Hart ad D. G. Stor, Joh Wiley & Sos, 000 with the permissio of the authors ad the publisher
More informationMachine Learning. Ilya Narsky, Caltech
Machie Learig Ilya Narsky, Caltech Lecture 4 Multi-class problems. Multi-class versios of Neural Networks, Decisio Trees, Support Vector Machies ad AdaBoost. Reductio of a multi-class problem to a set
More information10-701/ Machine Learning Mid-term Exam Solution
0-70/5-78 Machie Learig Mid-term Exam Solutio Your Name: Your Adrew ID: True or False (Give oe setece explaatio) (20%). (F) For a cotiuous radom variable x ad its probability distributio fuctio p(x), it
More informationEstimation Theory Chapter 3
stimatio Theory Chater 3 Likelihood Fuctio Higher deedece of data PDF o ukow arameter results i higher estimatio accuracy amle : If ˆ If large, W, Choose  P  small,  W POOR GOOD i Oly data samle Data
More informationBIOSTATISTICAL METHODS FOR TRANSLATIONAL & CLINICAL RESEARCH
BIOSAISICAL MEHODS FOR RANSLAIONAL & CLINICAL RESEARCH Direct Bioassays: REGRESSION APPLICAIONS COMPONENS OF A BIOASSAY he subject is usually a aimal, a huma tissue, or a bacteria culture, he aget is usually
More informationf(x i ; ) L(x; p) = i=1 To estimate the value of that maximizes L or equivalently ln L we will set =0, for i =1, 2,...,m p x i (1 p) 1 x i i=1
Parameter Estimatio Samples from a probability distributio F () are: [,,..., ] T.Theprobabilitydistributio has a parameter vector [,,..., m ] T. Estimator: Statistic used to estimate ukow. Estimate: Observed
More informationThe Maximum-Likelihood Decoding Performance of Error-Correcting Codes
The Maximum-Lielihood Decodig Performace of Error-Correctig Codes Hery D. Pfister ECE Departmet Texas A&M Uiversity August 27th, 2007 (rev. 0) November 2st, 203 (rev. ) Performace of Codes. Notatio X,
More informationNonequilibrium Excess Carriers in Semiconductors
Lecture 8 Semicoductor Physics VI Noequilibrium Excess Carriers i Semicoductors Noequilibrium coditios. Excess electros i the coductio bad ad excess holes i the valece bad Ambiolar trasort : Excess electros
More informationRegression and generalization
Regressio ad geeralizatio CE-717: Machie Learig Sharif Uiversity of Techology M. Soleymai Fall 2016 Curve fittig: probabilistic perspective Describig ucertaity over value of target variable as a probability
More informationECE534, Spring 2018: Final Exam
ECE534, Srig 2018: Fial Exam Problem 1 Let X N (0, 1) ad Y N (0, 1) be ideedet radom variables. variables V = X + Y ad W = X 2Y. Defie the radom (a) Are V, W joitly Gaussia? Justify your aswer. (b) Comute
More informationPC5215 Numerical Recipes with Applications - Review Problems
PC55 Numerical Recipes with Applicatios - Review Problems Give the IEEE 754 sigle precisio bit patter (biary or he format) of the followig umbers: 0 0 05 00 0 00 Note that it has 8 bits for the epoet,
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More informationBasics of Inference. Lecture 21: Bayesian Inference. Review - Example - Defective Parts, cont. Review - Example - Defective Parts
Basics of Iferece Lecture 21: Sta230 / Mth230 Coli Rudel Aril 16, 2014 U util this oit i the class you have almost exclusively bee reseted with roblems where we are usig a robability model where the model
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationEGN 3353C Fluid Mechanics
Chapter 7: DIMENSIONAL ANALYSIS AND MODELING Lecture 3 dimesio measure of a physical quatity ithout umerical values (e.g., legth) uit assigs a umber to that dimesio (e.g., meter) 7 fudametal dimesios from
More informationSession 5. (1) Principal component analysis and Karhunen-Loève transformation
200 Autum semester Patter Iformatio Processig Topic 2 Image compressio by orthogoal trasformatio Sessio 5 () Pricipal compoet aalysis ad Karhue-Loève trasformatio Topic 2 of this course explais the image
More informationMA Advanced Econometrics: Properties of Least Squares Estimators
MA Advaced Ecoometrics: Properties of Least Squares Estimators Karl Whela School of Ecoomics, UCD February 5, 20 Karl Whela UCD Least Squares Estimators February 5, 20 / 5 Part I Least Squares: Some Fiite-Sample
More informationVector Quantization: a Limiting Case of EM
. Itroductio & defiitios Assume that you are give a data set X = { x j }, j { 2,,, }, of d -dimesioal vectors. The vector quatizatio (VQ) problem requires that we fid a set of prototype vectors Z = { z
More information5.4 The spatial error model Regression model with spatially autocorrelated errors
54 The spatial error model 54 Regressio model with spatiall autocorrelated errors I a multiple regressio model, the depedet variable Y depeds o k regressors X (=), X,, X k ad a disturbace ε: (4) is a x
More informationFactor Analysis. Lecture 10: Factor Analysis and Principal Component Analysis. Sam Roweis
Lecture 10: Factor Aalysis ad Pricipal Compoet Aalysis Sam Roweis February 9, 2004 Whe we assume that the subspace is liear ad that the uderlyig latet variable has a Gaussia distributio we get a model
More informationECO 312 Fall 2013 Chris Sims LIKELIHOOD, POSTERIORS, DIAGNOSING NON-NORMALITY
ECO 312 Fall 2013 Chris Sims LIKELIHOOD, POSTERIORS, DIAGNOSING NON-NORMALITY (1) A distributio that allows asymmetry differet probabilities for egative ad positive outliers is the asymmetric double expoetial,
More informationOutput Analysis (2, Chapters 10 &11 Law)
B. Maddah ENMG 6 Simulatio Output Aalysis (, Chapters 10 &11 Law) Comparig alterative system cofiguratio Sice the output of a simulatio is radom, the comparig differet systems via simulatio should be doe
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationNANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS
NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS STRUCTURE OF EXAMINATION PAPER. There will be oe 2-hour paper cosistig of 4 questios.
More informationBHW #13 1/ Cooper. ENGR 323 Probabilistic Analysis Beautiful Homework # 13
BHW # /5 ENGR Probabilistic Aalysis Beautiful Homework # Three differet roads feed ito a particular freeway etrace. Suppose that durig a fixed time period, the umber of cars comig from each road oto the
More informationNonlinear regression
oliear regressio How to aalyse data? How to aalyse data? Plot! How to aalyse data? Plot! Huma brai is oe the most powerfull computatioall tools Works differetly tha a computer What if data have o liear
More informationMachine Learning: Logistic Regression. Lecture 04
Machie Learig: Logistic Regressio Razva C. Buescu School of Electrical Egieerig ad Computer Sciece buescu@ohio.edu Supervised Learig ask = lear a uko fuctio t : X that maps iput istaces x Î X to output
More informationPutnam Training Exercise Counting, Probability, Pigeonhole Principle (Answers)
Putam Traiig Exercise Coutig, Probability, Pigeohole Pricile (Aswers) November 24th, 2015 1. Fid the umber of iteger o-egative solutios to the followig Diohatie equatio: x 1 + x 2 + x 3 + x 4 + x 5 = 17.
More informationNaïve Bayes. Naïve Bayes
Statistical Data Miig ad Machie Learig Hilary Term 206 Dio Sejdiovic Departmet of Statistics Oxford Slides ad other materials available at: http://www.stats.ox.ac.uk/~sejdiov/sdmml : aother plug-i classifier
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationCHAPTER I: Vector Spaces
CHAPTER I: Vector Spaces Sectio 1: Itroductio ad Examples This first chapter is largely a review of topics you probably saw i your liear algebra course. So why cover it? (1) Not everyoe remembers everythig
More information6.867 Machine learning, lecture 7 (Jaakkola) 1
6.867 Machie learig, lecture 7 (Jaakkola) 1 Lecture topics: Kerel form of liear regressio Kerels, examples, costructio, properties Liear regressio ad kerels Cosider a slightly simpler model where we omit
More informationReview of Probability Axioms and Laws
Review of robabilit ioms ad Laws Berli Che Deartmet of Comuter ciece & Iformatio Egieerig Natioal Taiwa Normal Uiversit Referece:. D.. Bertsekas, J. N. Tsitsiklis, Itroductio to robabilit, thea cietific,
More informationEfficient GMM LECTURE 12 GMM II
DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet
More informationPolynomial Functions and Their Graphs
Polyomial Fuctios ad Their Graphs I this sectio we begi the study of fuctios defied by polyomial expressios. Polyomial ad ratioal fuctios are the most commo fuctios used to model data, ad are used extesively
More informationTopics Machine learning: lecture 3. Linear regression. Linear regression. Linear regression. Linear regression
6.867 Machie learig: lecture 3 Tommi S. Jaakkola MIT CSAIL tommi@csail.mit.edu Topics Beod liear regressio models additive regressio models, eamples geeralizatio ad cross-validatio populatio miimizer Statistical
More informationLeast Squares Methods
Det. of Biomed. Eg. BME80: Iverse Problems i Bioegieerig Kug ee Uiv. Least Squares Methods Overdetermied liear equatios m where R ad m > More equatios tha ukows Caot solve for i most cases. Least squares
More informationTopics Machine learning: lecture 2. Review: the learning problem. Hypotheses and estimation. Estimation criterion cont d. Estimation criterion
.87 Machie learig: lecture Tommi S. Jaakkola MIT CSAIL tommi@csail.mit.edu Topics The learig problem hypothesis class, estimatio algorithm loss ad estimatio criterio samplig, empirical ad epected losses
More informationCov(aX, cy ) Var(X) Var(Y ) It is completely invariant to affine transformations: for any a, b, c, d R, ρ(ax + b, cy + d) = a.s. X i. as n.
CS 189 Itroductio to Machie Learig Sprig 218 Note 11 1 Caoical Correlatio Aalysis The Pearso Correlatio Coefficiet ρ(x, Y ) is a way to measure how liearly related (i other words, how well a liear model
More informationtests 17.1 Simple versus compound
PAS204: Lecture 17. tests UMP ad asymtotic I this lecture, we will idetify UMP tests, wherever they exist, for comarig a simle ull hyothesis with a comoud alterative. We also look at costructig tests based
More informationMini Lecture 10.1 Radical Expressions and Functions. 81x d. x 4x 4
Mii Lecture 0. Radical Expressios ad Fuctios Learig Objectives:. Evaluate square roots.. Evaluate square root fuctios.. Fid the domai of square root fuctios.. Use models that are square root fuctios. 5.
More informationMixtures of Gaussians and the EM Algorithm
Mixtures of Gaussias ad the EM Algorithm CSE 6363 Machie Learig Vassilis Athitsos Computer Sciece ad Egieerig Departmet Uiversity of Texas at Arligto 1 Gaussias A popular way to estimate probability desity
More informationPattern Classification
Patter Classificatio All materials i these slides were tae from Patter Classificatio (d ed) by R. O. Duda, P. E. Hart ad D. G. Stor, Joh Wiley & Sos, 000 with the permissio of the authors ad the publisher
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationIntroduction to Machine Learning DIS10
CS 189 Fall 017 Itroductio to Machie Learig DIS10 1 Fu with Lagrage Multipliers (a) Miimize the fuctio such that f (x,y) = x + y x + y = 3. Solutio: The Lagragia is: L(x,y,λ) = x + y + λ(x + y 3) Takig
More informationL S => logf y i P x i ;S
Three Classical Tests; Wald, LM(core), ad LR tests uose that we hae the desity y; of a model with the ull hyothesis of the form H ; =. Let L be the log-likelihood fuctio of the model ad be the MLE of.
More informationGeneralizing the DTFT. The z Transform. Complex Exponential Excitation. The Transfer Function. Systems Described by Difference Equations
Geeraliig the DTFT The Trasform M. J. Roberts - All Rights Reserved. Edited by Dr. Robert Akl 1 The forward DTFT is defied by X e jω = x e jω i which = Ω is discrete-time radia frequecy, a real variable.
More informationECON 3150/4150, Spring term Lecture 3
Itroductio Fidig the best fit by regressio Residuals ad R-sq Regressio ad causality Summary ad ext step ECON 3150/4150, Sprig term 2014. Lecture 3 Ragar Nymoe Uiversity of Oslo 21 Jauary 2014 1 / 30 Itroductio
More informationExpectation and Variance of a random variable
Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio
More informationClustering. CM226: Machine Learning for Bioinformatics. Fall Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar.
Clusterig CM226: Machie Learig for Bioiformatics. Fall 216 Sriram Sakararama Ackowledgmets: Fei Sha, Ameet Talwalkar Clusterig 1 / 42 Admiistratio HW 1 due o Moday. Email/post o CCLE if you have questios.
More informationCorrelation Regression
Correlatio Regressio While correlatio methods measure the stregth of a liear relatioship betwee two variables, we might wish to go a little further: How much does oe variable chage for a give chage i aother
More informationMth 95 Notes Module 1 Spring Section 4.1- Solving Systems of Linear Equations in Two Variables by Graphing, Substitution, and Elimination
Mth 9 Notes Module Sprig 4 Sectio 4.- Solvig Sstems of Liear Equatios i Two Variales Graphig, Sustitutio, ad Elimiatio A Solutio to a Sstem of Two (or more) Liear Equatios is the commo poit(s) of itersectio
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationLecture 8: Non-parametric Comparison of Location. GENOME 560, Spring 2016 Doug Fowler, GS
Lecture 8: No-parametric Compariso of Locatio GENOME 560, Sprig 2016 Doug Fowler, GS (dfowler@uw.edu) 1 Review What do we mea by oparametric? What is a desirable locatio statistic for ordial data? What
More informationRADICAL EXPRESSION. If a and x are real numbers and n is a positive integer, then x is an. n th root theorems: Example 1 Simplify
Example 1 Simplify 1.2A Radical Operatios a) 4 2 b) 16 1 2 c) 16 d) 2 e) 8 1 f) 8 What is the relatioship betwee a, b, c? What is the relatioship betwee d, e, f? If x = a, the x = = th root theorems: RADICAL
More information