Estimators as Random Variables
|
|
- Sheena Hall
- 5 years ago
- Views:
Transcription
1 Estimation Theory Overview Properties Bias, Variance, and Mean Square Error Cramér-Rao lower bound Maimum likelihood Consistency Confidence intervals Properties of the mean estimator Introduction Up until now we have defined and discussed properties of random variables and processes In each case we started with some known property (e.g. autocorrelation) and derived other related properties (e.g. PSD) In practical problems we rarely know these properties a priori In stead, we must estimate what we wish to know from finite sets of measurements J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 Terminology Suppose we have N independent, identically-distributed (i.i.d.) observations i } N i= Ideally we would like to know the pdf of the data where R p f(; ) In probability theory, we think about the likeliness of i } N i= given the pdf and In inference, we are given i } N i= and are interested in the likeliness of Called the sampling distribution We will use to denote the parameter (or vector of parameters) we wish to estimate Estimators as Random Variables Our estimator is a function of the measurements ˆ [ ] i } N i= It is therefore a random variable It will be different for every different set of observations It is called an estimate or, if is a scalar, a point estimate Of course we want ˆ to be as close to the true as possible This could be, for eample, the process mean μ J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 3 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 4
2 Natural Estimators ˆμ = ˆ [ ] i } N i= = N N n= This is the obvious or natural estimator of the process mean Sometimes called the average or sample mean It will also turn out to be the best estimator I will define best shortly ˆσ = ˆ [ ] i } N i= = N i N ( i ˆμ ) n= This is the obvious or natural estimator of the process variance Not the best Good Estimators Without loss of generality, let us consider a scalar parameter for the time being What is a good estimator Distribution of ˆ should be centered at the true value Want the distribution to be as narrow as possible Lower-order moments enable coarse measurements of good ˆ J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 5 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 6 Bias Bias of an estimator ˆ of a parameter is defined as B(ˆ) E[ˆ] Unbiased: an estimator is said to be unbiased if B(ˆ) =0 This implies the pdf of the estimator is centered at the true value The sample mean is unbiased The estimator of variance on the earlier slide is biased Unbiased estimators are generally good, but they are not always best (more later) Variance Variance of an estimator ˆ of a parameter is defined as [ ] var(ˆ) =σ ˆ E ˆ E [ˆ ] A measure of the spread of ˆ about its mean Would like the variance to be as small as possible J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 7 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 8
3 Bias-Variance Tradeoff The Bias-Variance Tradeoff ˆ ˆ In many cases minimizing variance conflicts with minimizing bias Note that ˆ 0 has zero variance, but is generally biased In these cases we must trade variance for bias (or vice versa) ˆ Understanding of the bias-variance tradeoff is crucial to this course Unbiased models are not always best The methods we will use to estimate the model coefficients are biased But they may be more accurate, because they have less variance This idea applies to nonlinear models as well ˆ J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 9 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 0 Bias, Variance, and Modeling y() =g()+ε ŷ() =ĝ() In the modeling contet, we are usually interested in estimating a function For a given input, this function is a scalar We can define = g() Thus, all of the ideas that apply to estimating parameters also apply to estimating functional relationships Notation and Prediction Error y = g()+ε g = g() ĝ =ĝ() ĝ e =E[ĝ()] Epectation is taken over the distribution of data sets used to construct ĝ() and the distribution of the process noise f(ε) Everything is a function of Recall that ε is i.i.d. with zero mean We are treating as a fied, non-random variable The dependence on is not shown to simplify notation The prediction error for a new, given input is defined as PE() = E[(y ĝ) ] = E[((g ĝ)+ε) ] = E[(g ĝ) ]+E[(g ĝ)ε]+e[ε ] = MSE()+σ ε J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6
4 The Bias-Variance Tradeoff Derivation y = g()+ε g = g() ĝ =ĝ() ĝ e =E[ĝ()] Only ĝ is a random function Nothing else is dependent on the data set MSE() = E[(g ĝ) ] = E[(g ĝ e ) (ĝ ĝ e )} ] = E (g ĝ e ) (g ĝ e )(ĝ ĝ e ) +(ĝ ĝ e ) }}}} Thus Bias-Variance Tradeoff Derivation Continued = E[(g ĝ e ) (g ĝ e )(ĝ ĝ e )] = E[g gĝ e +ĝe g(ĝ ĝ e )] + ĝe ĝe = E[g gĝ e +ĝe gĝ +gĝ e ] = E[g gĝ +ĝe] = g g E[ĝ]+ĝe = g gĝ e +ĝ e = (g ĝ e ) MSE() = + = (g ĝ e ) +E[(ĝ ĝ e ) ] = (g E[ĝ]) +E [ (ĝ E[ĝ]) ] J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 3 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 4 Bias-Variance Tradeoff Comments MSE() =(g E[ĝ]) +E [ (ĝ E[ĝ]) ] = Bias + Variance Large variance: the model is sensitive to small changes in the data set Large bias: if the model was compared to the true function on a large number of data sets, the epected value of the model ĝ() would not be close to the true function g() If the model is sensitive to small changes in the data, a biased model may have smaller error (MSE) than an unbiased model If the data is strongly collinear, biased models can result in more accurate models! Bias-Variance Tradeoff Comments Continued MSE() =(g E[ĝ]) +E [ (ĝ E[ĝ]) ] = Bias + Variance Large variance, small bias If the model is too fleible, it can overfit the data The model will change dramatically from one data set to another In this case it has high variance, but potentially low variance Small variance, large bias If the model is not very fleible, it may not capture the true relationship between the inputs and the output It will not vary as much from one data set to another In this case the model has low variance, but potentially high bias J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 5 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 6
5 Mean Square Error Mean Square Error of an estimator ˆ of a parameter is defined as MSE() E[ ˆ ]=σ ˆ + B(ˆ) We will use often use MSE as a global measure of estimator performance Note that two different estimators may have the same MSE but different bias and variance This criterion is convenient for building estimators Creating a problem we can solve Note the rationale is due to convenience: Picking MSE results in a simple bias/variance decomposition Other error measures generally do not have such a decomposition var(ˆ) E Cramér-Rao Lower Bound [ ln f; (;) ] = E[ ln f ; (;) ] Minimum Variance Unbiased (MVU): Estimators that are both unbiased and have the smallest variance of all possible estimators Note that these do not necessarily achieve the minimum MSE Cramér-Rao Lower Bound (CRLB) shown above is a lower bound on unbiased estimators Log Likelihood Function of is ln f ; (; ) Note that the pdf f ; (; ) describes the distribution of the data (stochastic process), not the parameter is not a random variable, it is a parameter that defines the distribution J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 7 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 8 Cramér-Rao Lower Bound Comments var(ˆ) E [ ln f; (;) ] = [ ] ln f E ; (;) Efficient Estimator: an unbiased estimate that achieves the CRLB with equality If it eists, then the unique solution is given by ln f ; (; ) =0 where the pdf is evaluated at the observed outcome (ζ) Maimum Likelihood (ML) Estimate: an estimator that satisfies the equation above This can be generalized to vectors of parameters Limited use f ; (; ) is rarely known in practice Consistency Consistent Estimator an estimator such that lim MSE(ˆ) =0 N Implies the following as the sample size grows (N ) The estimator becomes unbiased The variance approaches zero The distribution fˆ() becomes an impulse centered at J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 9 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 0
6 Confidence Intervals Confidence Interval: interval, a b, that has a specified probability of covering the unknown true parameter value Pr a < b} = α The interval is estimated from the data, therefore it is also a pair of random variables Confidence Level: coverage probability of a confidence interval, α The confidence interval is not uniquely defined by the confidence level Properties of the Sample Mean ˆμ N E[ˆμ ]=μ var(ˆμ )= σ N N k=0 N l= N (n) The estimator is unbiased Can also be shown that Has minimum variance If Gaussian, is the maimum likelihood estimator If Gaussian, attains the Cramér-Rao Lower Bound J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 fˆμ (ˆμ ) = Sample Mean Confidence Intervals [ π(σ / N) ep Pr μ k σ < ˆμ <μ + k σ } Pr ˆμ k σ <μ < ˆμ + k σ } ( ) ] ˆμ μ σ / N = = α In general, we don t know the pdf If we can assume the process is Gaussian and IID, we know the pdf (sampling distribution) of the estimator If N is large and the distribution doesn t have heavy tails, the distribution of ˆμ is Gaussian by the Central Limit Theorem (CLT) Sample Mean Confidence Intervals Comments Pr ˆμ k σ <μ < ˆμ + k σ } = α In many cases the confidence intervals are accurate, even if they are only approimate We can choose k such that α equals any probability we like In general, the user picks α This controls how often the confidence interval does not cover μ 95% and 99% are common choices J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 3 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 4
7 Sample Mean Variance when Gaussian and IID Pr ˆμ k σ <μ < ˆμ + k σ } = α If σ is unknown (usually), must estimate from the data ˆσ = N [(n) ˆμ ] N n=0 The corresponding z score, has a different distribution If (n) is IID and Gaussian ˆμ μ ˆσ / N has a Students t distribution with v = N degrees of freedom Approaches a Gaussian distribution as v becomes large (> 0) E[ˆμ ]=μ Sample Mean Variance when Gaussian var(ˆμ )= N N l= N ( l ) γ (l) N If (n) is Gaussian but not IID, the sample mean is normal with mean μ The approimate confidence interval is given by a Guassian PDF Pr ˆμ k var(ˆμ ) <μ < ˆμ k } var(ˆμ ) = α J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 5 J. McNames Portland State University ECE 4/557 Estimation Theory Ver..6 6
Terminology Suppose we have N observations {x(n)} N 1. Estimators as Random Variables. {x(n)} N 1
Estimation Theory Overview Properties Bias, Variance, and Mean Square Error Cramér-Rao lower bound Maximum likelihood Consistency Confidence intervals Properties of the mean estimator Properties of the
More information2 Statistical Estimation: Basic Concepts
Technion Israel Institute of Technology, Department of Electrical Engineering Estimation and Identification in Dynamical Systems (048825) Lecture Notes, Fall 2009, Prof. N. Shimkin 2 Statistical Estimation:
More informationPart 4: Multi-parameter and normal models
Part 4: Multi-parameter and normal models 1 The normal model Perhaps the most useful (or utilized) probability model for data analysis is the normal distribution There are several reasons for this, e.g.,
More informationBasic concepts in estimation
Basic concepts in estimation Random and nonrandom parameters Definitions of estimates ML Maimum Lielihood MAP Maimum A Posteriori LS Least Squares MMS Minimum Mean square rror Measures of quality of estimates
More informationProbability Space. J. McNames Portland State University ECE 538/638 Stochastic Signals Ver
Stochastic Signals Overview Definitions Second order statistics Stationarity and ergodicity Random signal variability Power spectral density Linear systems with stationary inputs Random signal memory Correlation
More informationEstimation and Detection
stimation and Detection Lecture 2: Cramér-Rao Lower Bound Dr. ir. Richard C. Hendriks & Dr. Sundeep P. Chepuri 7//207 Remember: Introductory xample Given a process (DC in noise): x[n]=a + w[n], n=0,,n,
More informationECE531 Lecture 10b: Maximum Likelihood Estimation
ECE531 Lecture 10b: Maximum Likelihood Estimation D. Richard Brown III Worcester Polytechnic Institute 05-Apr-2011 Worcester Polytechnic Institute D. Richard Brown III 05-Apr-2011 1 / 23 Introduction So
More informationBias Variance Trade-off
Bias Variance Trade-off The mean squared error of an estimator MSE(ˆθ) = E([ˆθ θ] 2 ) Can be re-expressed MSE(ˆθ) = Var(ˆθ) + (B(ˆθ) 2 ) MSE = VAR + BIAS 2 Proof MSE(ˆθ) = E((ˆθ θ) 2 ) = E(([ˆθ E(ˆθ)]
More informationELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE)
1 ELEG 5633 Detection and Estimation Minimum Variance Unbiased Estimators (MVUE) Jingxian Wu Department of Electrical Engineering University of Arkansas Outline Minimum Variance Unbiased Estimators (MVUE)
More informationStatistical inference
Statistical inference Contents 1. Main definitions 2. Estimation 3. Testing L. Trapani MSc Induction - Statistical inference 1 1 Introduction: definition and preliminary theory In this chapter, we shall
More informationFall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
More informationEstimation theory. Parametric estimation. Properties of estimators. Minimum variance estimator. Cramer-Rao bound. Maximum likelihood estimators
Estimation theory Parametric estimation Properties of estimators Minimum variance estimator Cramer-Rao bound Maximum likelihood estimators Confidence intervals Bayesian estimation 1 Random Variables Let
More informationParametric Techniques
Parametric Techniques Jason J. Corso SUNY at Buffalo J. Corso (SUNY at Buffalo) Parametric Techniques 1 / 39 Introduction When covering Bayesian Decision Theory, we assumed the full probabilistic structure
More informationModule 2. Random Processes. Version 2, ECE IIT, Kharagpur
Module Random Processes Version, ECE IIT, Kharagpur Lesson 9 Introduction to Statistical Signal Processing Version, ECE IIT, Kharagpur After reading this lesson, you will learn about Hypotheses testing
More informationAdvanced Signal Processing Introduction to Estimation Theory
Advanced Signal Processing Introduction to Estimation Theory Danilo Mandic, room 813, ext: 46271 Department of Electrical and Electronic Engineering Imperial College London, UK d.mandic@imperial.ac.uk,
More informationBias. Definition. One of the foremost expectations of an estimator is that it gives accurate estimates.
Bias One of the foremost expectations of an estimator is that it gives accurate estimates. Definition An estimator ˆ is said to be accurate or unbiased if and only if µˆ = E(ˆ ) = 0 (19) In plain language,
More informationParametric Techniques Lecture 3
Parametric Techniques Lecture 3 Jason Corso SUNY at Buffalo 22 January 2009 J. Corso (SUNY at Buffalo) Parametric Techniques Lecture 3 22 January 2009 1 / 39 Introduction In Lecture 2, we learned how to
More informationSimple Linear Regression
Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.
More informationIntroduction to Maximum Likelihood Estimation
Introduction to Maximum Likelihood Estimation Eric Zivot July 26, 2012 The Likelihood Function Let 1 be an iid sample with pdf ( ; ) where is a ( 1) vector of parameters that characterize ( ; ) Example:
More informationIEOR 165 Lecture 7 1 Bias-Variance Tradeoff
IEOR 165 Lecture 7 Bias-Variance Tradeoff 1 Bias-Variance Tradeoff Consider the case of parametric regression with β R, and suppose we would like to analyze the error of the estimate ˆβ in comparison to
More informationReview Quiz. 1. Prove that in a one-dimensional canonical exponential family, the complete and sufficient statistic achieves the
Review Quiz 1. Prove that in a one-dimensional canonical exponential family, the complete and sufficient statistic achieves the Cramér Rao lower bound (CRLB). That is, if where { } and are scalars, then
More informationParameter Estimation
Parameter Estimation Consider a sample of observations on a random variable Y. his generates random variables: (y 1, y 2,, y ). A random sample is a sample (y 1, y 2,, y ) where the random variables y
More informationProblem 1 (20) Log-normal. f(x) Cauchy
ORF 245. Rigollet Date: 11/21/2008 Problem 1 (20) f(x) f(x) 0.0 0.1 0.2 0.3 0.4 0.0 0.2 0.4 0.6 0.8 4 2 0 2 4 Normal (with mean -1) 4 2 0 2 4 Negative-exponential x x f(x) f(x) 0.0 0.1 0.2 0.3 0.4 0.5
More informationwhere r n = dn+1 x(t)
Random Variables Overview Probability Random variables Transforms of pdfs Moments and cumulants Useful distributions Random vectors Linear transformations of random vectors The multivariate normal distribution
More informationParameter estimation! and! forecasting! Cristiano Porciani! AIfA, Uni-Bonn!
Parameter estimation! and! forecasting! Cristiano Porciani! AIfA, Uni-Bonn! Questions?! C. Porciani! Estimation & forecasting! 2! Cosmological parameters! A branch of modern cosmological research focuses
More informationF & B Approaches to a simple model
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 215 http://www.astro.cornell.edu/~cordes/a6523 Lecture 11 Applications: Model comparison Challenges in large-scale surveys
More informationIf we want to analyze experimental or simulated data we might encounter the following tasks:
Chapter 1 Introduction If we want to analyze experimental or simulated data we might encounter the following tasks: Characterization of the source of the signal and diagnosis Studying dependencies Prediction
More informationA Few Notes on Fisher Information (WIP)
A Few Notes on Fisher Information (WIP) David Meyer dmm@{-4-5.net,uoregon.edu} Last update: April 30, 208 Definitions There are so many interesting things about Fisher Information and its theoretical properties
More informationVariations. ECE 6540, Lecture 10 Maximum Likelihood Estimation
Variations ECE 6540, Lecture 10 Last Time BLUE (Best Linear Unbiased Estimator) Formulation Advantages Disadvantages 2 The BLUE A simplification Assume the estimator is a linear system For a single parameter
More informationCOS513: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 10
COS53: FOUNDATIONS OF PROBABILISTIC MODELS LECTURE 0 MELISSA CARROLL, LINJIE LUO. BIAS-VARIANCE TRADE-OFF (CONTINUED FROM LAST LECTURE) If V = (X n, Y n )} are observed data, the linear regression problem
More informationEIE6207: Estimation Theory
EIE6207: Estimation Theory Man-Wai MAK Dept. of Electronic and Information Engineering, The Hong Kong Polytechnic University enmwmak@polyu.edu.hk http://www.eie.polyu.edu.hk/ mwmak References: Steven M.
More informationEstimation Theory Fredrik Rusek. Chapters
Estimation Theory Fredrik Rusek Chapters 3.5-3.10 Recap We deal with unbiased estimators of deterministic parameters Performance of an estimator is measured by the variance of the estimate (due to the
More informationMathematical statistics
October 4 th, 2018 Lecture 12: Information Where are we? Week 1 Week 2 Week 4 Week 7 Week 10 Week 14 Probability reviews Chapter 6: Statistics and Sampling Distributions Chapter 7: Point Estimation Chapter
More informationSGN Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection
SG 21006 Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection Ioan Tabus Department of Signal Processing Tampere University of Technology Finland 1 / 28
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationReview of probability and statistics 1 / 31
Review of probability and statistics 1 / 31 2 / 31 Why? This chapter follows Stock and Watson (all graphs are from Stock and Watson). You may as well refer to the appendix in Wooldridge or any other introduction
More informationProof In the CR proof. and
Question Under what conditions will we be able to attain the Cramér-Rao bound and find a MVUE? Lecture 4 - Consequences of the Cramér-Rao Lower Bound. Searching for a MVUE. Rao-Blackwell Theorem, Lehmann-Scheffé
More informationECON The Simple Regression Model
ECON 351 - The Simple Regression Model Maggie Jones 1 / 41 The Simple Regression Model Our starting point will be the simple regression model where we look at the relationship between two variables In
More informationPrimer on statistics:
Primer on statistics: MLE, Confidence Intervals, and Hypothesis Testing ryan.reece@gmail.com http://rreece.github.io/ Insight Data Science - AI Fellows Workshop Feb 16, 018 Outline 1. Maximum likelihood
More informationSimple linear regression
Simple linear regression Biometry 755 Spring 2008 Simple linear regression p. 1/40 Overview of regression analysis Evaluate relationship between one or more independent variables (X 1,...,X k ) and a single
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationSYSM 6303: Quantitative Introduction to Risk and Uncertainty in Business Lecture 4: Fitting Data to Distributions
SYSM 6303: Quantitative Introduction to Risk and Uncertainty in Business Lecture 4: Fitting Data to Distributions M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu
More information2. A Basic Statistical Toolbox
. A Basic Statistical Toolbo Statistics is a mathematical science pertaining to the collection, analysis, interpretation, and presentation of data. Wikipedia definition Mathematical statistics: concerned
More informationInferring from data. Theory of estimators
Inferring from data Theory of estimators 1 Estimators Estimator is any function of the data e(x) used to provide an estimate ( a measurement ) of an unknown parameter. Because estimators are functions
More informationReminders. Thought questions should be submitted on eclass. Please list the section related to the thought question
Linear regression Reminders Thought questions should be submitted on eclass Please list the section related to the thought question If it is a more general, open-ended question not exactly related to a
More informationExpression arrays, normalization, and error models
1 Epression arrays, normalization, and error models There are a number of different array technologies available for measuring mrna transcript levels in cell populations, from spotted cdna arrays to in
More information14.30 Introduction to Statistical Methods in Economics Spring 2009
MIT OpenCourseWare http://ocw.mit.edu 4.0 Introduction to Statistical Methods in Economics Spring 009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationInference in Normal Regression Model. Dr. Frank Wood
Inference in Normal Regression Model Dr. Frank Wood Remember We know that the point estimator of b 1 is b 1 = (Xi X )(Y i Ȳ ) (Xi X ) 2 Last class we derived the sampling distribution of b 1, it being
More informationPractice Problems Section Problems
Practice Problems Section 4-4-3 4-4 4-5 4-6 4-7 4-8 4-10 Supplemental Problems 4-1 to 4-9 4-13, 14, 15, 17, 19, 0 4-3, 34, 36, 38 4-47, 49, 5, 54, 55 4-59, 60, 63 4-66, 68, 69, 70, 74 4-79, 81, 84 4-85,
More informationRegression Estimation - Least Squares and Maximum Likelihood. Dr. Frank Wood
Regression Estimation - Least Squares and Maximum Likelihood Dr. Frank Wood Least Squares Max(min)imization Function to minimize w.r.t. β 0, β 1 Q = n (Y i (β 0 + β 1 X i )) 2 i=1 Minimize this by maximizing
More informationMS&E 226: Small Data
MS&E 226: Small Data Lecture 6: Model complexity scores (v3) Ramesh Johari ramesh.johari@stanford.edu Fall 2015 1 / 34 Estimating prediction error 2 / 34 Estimating prediction error We saw how we can estimate
More information10. Linear Models and Maximum Likelihood Estimation
10. Linear Models and Maximum Likelihood Estimation ECE 830, Spring 2017 Rebecca Willett 1 / 34 Primary Goal General problem statement: We observe y i iid pθ, θ Θ and the goal is to determine the θ that
More informationEconomics 241B Review of Limit Theorems for Sequences of Random Variables
Economics 241B Review of Limit Theorems for Sequences of Random Variables Convergence in Distribution The previous de nitions of convergence focus on the outcome sequences of a random variable. Convergence
More informationRegression Estimation Least Squares and Maximum Likelihood
Regression Estimation Least Squares and Maximum Likelihood Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 3, Slide 1 Least Squares Max(min)imization Function to minimize
More informationwhere x and ȳ are the sample means of x 1,, x n
y y Animal Studies of Side Effects Simple Linear Regression Basic Ideas In simple linear regression there is an approximately linear relation between two variables say y = pressure in the pancreas x =
More informationElements of statistics (MATH0487-1)
Elements of statistics (MATH0487-1) Prof. Dr. Dr. K. Van Steen University of Liège, Belgium November 12, 2012 Introduction to Statistics Basic Probability Revisited Sampling Exploratory Data Analysis -
More informationRegression with a Single Regressor: Hypothesis Tests and Confidence Intervals
Regression with a Single Regressor: Hypothesis Tests and Confidence Intervals (SW Chapter 5) Outline. The standard error of ˆ. Hypothesis tests concerning β 3. Confidence intervals for β 4. Regression
More informationMachine Learning CSE546 Carlos Guestrin University of Washington. September 30, What about continuous variables?
Linear Regression Machine Learning CSE546 Carlos Guestrin University of Washington September 30, 2014 1 What about continuous variables? n Billionaire says: If I am measuring a continuous variable, what
More informationBusiness Statistics: A Decision-Making Approach 6 th Edition. Chapter Goals
Chapter 6 Student Lecture Notes 6-1 Business Statistics: A Decision-Making Approach 6 th Edition Chapter 6 Introduction to Sampling Distributions Chap 6-1 Chapter Goals To use information from the sample
More informationEcon 6900: Statistical Problems. Instructor: Yogesh Uppal
Econ 6900: Statistical Problems Instructor: Yogesh Uppal Email: yuppal@ysu.edu Chapter 7, Part A Sampling and Sampling Distributions Simple Random Sampling Point Estimation Introduction to Sampling Distributions
More informationIntroduction to Simple Linear Regression
Introduction to Simple Linear Regression Yang Feng http://www.stat.columbia.edu/~yangfeng Yang Feng (Columbia University) Introduction to Simple Linear Regression 1 / 68 About me Faculty in the Department
More information6.867 Machine learning
6.867 Machine learning Mid-term eam October 8, 6 ( points) Your name and MIT ID: .5.5 y.5 y.5 a).5.5 b).5.5.5.5 y.5 y.5 c).5.5 d).5.5 Figure : Plots of linear regression results with different types of
More informationSimple and Multiple Linear Regression
Sta. 113 Chapter 12 and 13 of Devore March 12, 2010 Table of contents 1 Simple Linear Regression 2 Model Simple Linear Regression A simple linear regression model is given by Y = β 0 + β 1 x + ɛ where
More informationSpace Telescope Science Institute statistics mini-course. October Inference I: Estimation, Confidence Intervals, and Tests of Hypotheses
Space Telescope Science Institute statistics mini-course October 2011 Inference I: Estimation, Confidence Intervals, and Tests of Hypotheses James L Rosenberger Acknowledgements: Donald Richards, William
More informationEstimating σ 2. We can do simple prediction of Y and estimation of the mean of Y at any value of X.
Estimating σ 2 We can do simple prediction of Y and estimation of the mean of Y at any value of X. To perform inferences about our regression line, we must estimate σ 2, the variance of the error term.
More information{X i } realize. n i=1 X i. Note that again X is a random variable. If we are to
3 Convergence This topic will overview a variety of extremely powerful analysis results that span statistics, estimation theorem, and big data. It provides a framework to think about how to aggregate more
More informationStat 579: Generalized Linear Models and Extensions
Stat 579: Generalized Linear Models and Extensions Mixed models Yan Lu March, 2018, week 8 1 / 32 Restricted Maximum Likelihood (REML) REML: uses a likelihood function calculated from the transformed set
More informationECNS 561 Multiple Regression Analysis
ECNS 561 Multiple Regression Analysis Model with Two Independent Variables Consider the following model Crime i = β 0 + β 1 Educ i + β 2 [what else would we like to control for?] + ε i Here, we are taking
More information6.867 Machine Learning
6.867 Machine Learning Problem set 1 Solutions Thursday, September 19 What and how to turn in? Turn in short written answers to the questions explicitly stated, and when requested to explain or prove.
More informationCh 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
More informationMachine Learning CSE546 Carlos Guestrin University of Washington. September 30, 2013
Bayesian Methods Machine Learning CSE546 Carlos Guestrin University of Washington September 30, 2013 1 What about prior n Billionaire says: Wait, I know that the thumbtack is close to 50-50. What can you
More informationWooldridge, Introductory Econometrics, 4th ed. Appendix C: Fundamentals of mathematical statistics
Wooldridge, Introductory Econometrics, 4th ed. Appendix C: Fundamentals of mathematical statistics A short review of the principles of mathematical statistics (or, what you should have learned in EC 151).
More informationAll other items including (and especially) CELL PHONES must be left at the front of the room.
TEST #2 / STA 5327 (Inference) / Spring 2017 (April 24, 2017) Name: Directions This exam is closed book and closed notes. You will be supplied with scratch paper, and a copy of the Table of Common Distributions
More informationEstimation, Inference, and Hypothesis Testing
Chapter 2 Estimation, Inference, and Hypothesis Testing Note: The primary reference for these notes is Ch. 7 and 8 of Casella & Berger 2. This text may be challenging if new to this topic and Ch. 7 of
More information6. Vector Random Variables
6. Vector Random Variables In the previous chapter we presented methods for dealing with two random variables. In this chapter we etend these methods to the case of n random variables in the following
More informationData Mining Stat 588
Data Mining Stat 588 Lecture 02: Linear Methods for Regression Department of Statistics & Biostatistics Rutgers University September 13 2011 Regression Problem Quantitative generic output variable Y. Generic
More informationEconomics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
More informationMachine Learning Basics: Estimators, Bias and Variance
Machine Learning Basics: Estiators, Bias and Variance Sargur N. srihari@cedar.buffalo.edu This is part of lecture slides on Deep Learning: http://www.cedar.buffalo.edu/~srihari/cse676 1 Topics in Basics
More informationDetection & Estimation Lecture 1
Detection & Estimation Lecture 1 Intro, MVUE, CRLB Xiliang Luo General Course Information Textbooks & References Fundamentals of Statistical Signal Processing: Estimation Theory/Detection Theory, Steven
More informationMachine Learning for OR & FE
Machine Learning for OR & FE Supervised Learning: Regression I Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com Some of the
More informationStatistic: a that can be from a sample without making use of any unknown. In practice we will use to establish unknown parameters.
Chapter 9: Sampling Distributions 9.1: Sampling Distributions IDEA: How often would a given method of sampling give a correct answer if it was repeated many times? That is, if you took repeated samples
More informationMachine Learning CSE546 Sham Kakade University of Washington. Oct 4, What about continuous variables?
Linear Regression Machine Learning CSE546 Sham Kakade University of Washington Oct 4, 2016 1 What about continuous variables? Billionaire says: If I am measuring a continuous variable, what can you do
More informationMS&E 226: Small Data. Lecture 11: Maximum likelihood (v2) Ramesh Johari
MS&E 226: Small Data Lecture 11: Maximum likelihood (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 18 The likelihood function 2 / 18 Estimating the parameter This lecture develops the methodology behind
More informationThe Simple Linear Regression Model
The Simple Linear Regression Model Lesson 3 Ryan Safner 1 1 Department of Economics Hood College ECON 480 - Econometrics Fall 2017 Ryan Safner (Hood College) ECON 480 - Lesson 3 Fall 2017 1 / 77 Bivariate
More informationReview of Probability
Review of robabilit robabilit Theor: Man techniques in speech processing require the manipulation of probabilities and statistics. The two principal application areas we will encounter are: Statistical
More informationParameter estimation and forecasting. Cristiano Porciani AIfA, Uni-Bonn
Parameter estimation and forecasting Cristiano Porciani AIfA, Uni-Bonn Questions? C. Porciani Estimation & forecasting 2 Cosmological parameters A branch of modern cosmological research focuses on measuring
More informationReview. December 4 th, Review
December 4 th, 2017 Att. Final exam: Course evaluation Friday, 12/14/2018, 10:30am 12:30pm Gore Hall 115 Overview Week 2 Week 4 Week 7 Week 10 Week 12 Chapter 6: Statistics and Sampling Distributions Chapter
More informationCSC321 Lecture 18: Learning Probabilistic Models
CSC321 Lecture 18: Learning Probabilistic Models Roger Grosse Roger Grosse CSC321 Lecture 18: Learning Probabilistic Models 1 / 25 Overview So far in this course: mainly supervised learning Language modeling
More informationProbabilities & Statistics Revision
Probabilities & Statistics Revision Christopher Ting Christopher Ting http://www.mysmu.edu/faculty/christophert/ : christopherting@smu.edu.sg : 6828 0364 : LKCSB 5036 January 6, 2017 Christopher Ting QF
More informationSIO 221B, Rudnick adapted from Davis 1. 1 x lim. N x 2 n = 1 N. { x} 1 N. N x = 1 N. N x = 1 ( N N x ) x = 0 (3) = 1 x N 2
SIO B, Rudnick adapted from Davis VII. Sampling errors We do not have access to the true statistics, so we must compute sample statistics. By this we mean that the number of realizations we average over
More informationEstimation of Parameters
CHAPTER Probability, Statistics, and Reliability for Engineers and Scientists FUNDAMENTALS OF STATISTICAL ANALYSIS Second Edition A. J. Clark School of Engineering Department of Civil and Environmental
More informationSection 8.1: Interval Estimation
Section 8.1: Interval Estimation Discrete-Event Simulation: A First Course c 2006 Pearson Ed., Inc. 0-13-142917-5 Discrete-Event Simulation: A First Course Section 8.1: Interval Estimation 1/ 35 Section
More informationMathematical statistics
October 18 th, 2018 Lecture 16: Midterm review Countdown to mid-term exam: 7 days Week 1 Chapter 1: Probability review Week 2 Week 4 Week 7 Chapter 6: Statistics Chapter 7: Point Estimation Chapter 8:
More informationEstimation Tasks. Short Course on Image Quality. Matthew A. Kupinski. Introduction
Estimation Tasks Short Course on Image Quality Matthew A. Kupinski Introduction Section 13.3 in B&M Keep in mind the similarities between estimation and classification Image-quality is a statistical concept
More informationSystem Identification
System Identification Lecture : Statistical properties of parameter estimators, Instrumental variable methods Roy Smith 8--8. 8--8. Statistical basis for estimation methods Parametrised models: G Gp, zq,
More informationLecture 4: Types of errors. Bayesian regression models. Logistic regression
Lecture 4: Types of errors. Bayesian regression models. Logistic regression A Bayesian interpretation of regularization Bayesian vs maximum likelihood fitting more generally COMP-652 and ECSE-68, Lecture
More informationSTATISTICS 4, S4 (4769) A2
(4769) A2 Objectives To provide students with the opportunity to explore ideas in more advanced statistics to a greater depth. Assessment Examination (72 marks) 1 hour 30 minutes There are four options
More informationDetection & Estimation Lecture 1
Detection & Estimation Lecture 1 Intro, MVUE, CRLB Xiliang Luo General Course Information Textbooks & References Fundamentals of Statistical Signal Processing: Estimation Theory/Detection Theory, Steven
More informationApplied Econometrics (QEM)
Applied Econometrics (QEM) based on Prinicples of Econometrics Jakub Mućk Department of Quantitative Economics Jakub Mućk Applied Econometrics (QEM) Meeting #3 1 / 42 Outline 1 2 3 t-test P-value Linear
More informationBTRY 4090: Spring 2009 Theory of Statistics
BTRY 4090: Spring 2009 Theory of Statistics Guozhang Wang September 25, 2010 1 Review of Probability We begin with a real example of using probability to solve computationally intensive (or infeasible)
More information