Econ 5150: Applied Econometrics Dynamic Demand Model Model Selection. Sung Y. Park CUHK
|
|
- Baldwin Gregory
- 5 years ago
- Views:
Transcription
1 Econ 5150: Applied Econometrics Dynamic Demand Model Model Selection Sung Y. Park CUHK
2 Simple dynamic models A typical simple model: y t = α 0 + α 1 y t 1 + α 2 y t 2 + x tβ 0 x t 1β 1 + u t, where y t is per-capita U.S. gasoline consumption and x t is a vector of exogenous variables, for example, x t =(1, p t, z t ). The lag operator: y t 1 = Ly t y t 2 = Ly t 1 = L 2 y t = Then (1 α 1 L α 2 L 2 )y t = α 0 +(β 0 + β 1 L) x t + u t
3 Simple dynamic models More compactly A(L)y t = α 0 B(L)x t + u t It is tempting to solve the above by writing y t = A(L) 1 α 0 + A(L) 1 B(L) x t + A(L) 1 u t This model is called linear transfer function model. How can we interpret the above model? We may want to explain the notion of equilibrium forms of the above model.
4 Simple dynamic models Stability in linear difference equations: Consider the simplest possible case By repeated substitution X t = ax t 1 X t = ax t 1 = a 2 X t 2 = = a t X 0 where X 0 denotes an initial condition. a < 1: X t 0 a > 1: X t a =1: eitherx t X 0 or X t = ±X 0
5 Simple dynamic models Consider the second order difference equation: X t = a 1 X t 1 + a 2 X t 2. The solutions take the form, X t = A 1 θ t 1 + A 2 θ t 2, where A 1 and A 2 are parameters determined by initial conditions and the θ s are dependent on the a s. By substituting A 1 θ t 1 + A 2θ t 2 = a 1(A 1 θ t A 2 θ t 1 2 )+a 2 (A 1 θ t A 2 θ t 2 2 ) or 0=A 1 θ t 1 (1 a 1θ 1 1 a 2 θ 2 1 )+A 2θ t 2 (1 a 1θ 1 2 a 2 θ 2 2 )
6 Simple dynamic models Suppose we find the roots of the quadratic equation 1 a 1 z a 2 z 2 =0 and call these roots θ1 1 and θ2 1. Done... Stability? Suppose that all the roots are real: both θ 1 and θ 2 must be less than one in absolute value. θ is complex: θ = λ 1 + λ 2 i we can represent θ in polar coordinates θ = r(cos(φ)+i sin(φ)), where r =(λ λ2 2 )1/2,cos(φ) =λ 1 /r, sin(φ) =λ 2 /r.
7 Simple dynamic models Thus, it is necessary that the roots of the equation 1 a 1 z a 2 z 2 =0 should lie outside of the unit circle. Roots outside unit circle are good (stability). Roots inside unit circle: explosive behavior. Roots on the unit circle: Unit root.
8 Impulse response functions Interpreting the expression D(L) =A(L) 1 B(L) Consider or B(L) =A(L)D(L) β 0 + β 1 L + + β s L s =(1 α 1 L α r L r )(δ 0 + δ 1 L + )
9 Impulse response functions For j s β 0 = δ 0 β 1 = δ 0 α 1 + δ 1 β 2 = δ 0 α 2 δ 1 α 1 + δ 2. β j = δ 0 α j δ j 1 α 1 + δ j This means that a system can be solved recursively given the α, β s for the δ s. More generally, δ j = { j r i=1 α iδ j i + β j j s j r i=1 α iδ j i j > s
10 Impulse response functions The function of cumulative sums of the δ s Δ(j) = j i=1 δ i the impulse response function: provide a complete picture of the time pathof the response of y to a once-and-for-all unit shock in x. Case: a single exogenous variable x stays at x 0 for a long time. Thus y is randomly fluctuating around an equilibrium value y 0.Nowx changes tt x 1 and stays there. What happens to y?
11 Impulse response functions EΔy t = A(L) 1 B(L)Δx t = D(L)Δx t D(1)Δx = δ i Δx i=1 a new equilibrium : the accumulation of the short-run impulse response a new equilibrium : can be calculated simply by letting y t = y e and x t = x e. if the roots of the A(z) = 0 lie outside the unit circle... Inferences?
12 Error correction form Consider the following simple dynamic model: y t = α 1 y t 1 + α 0 + β 0 x t + β 1 x t 1 + u t In equilibrium with x t x e y e = α 0 1 α 1 + β 0 + β 1 1 α 1 x e α 1 u t subtract y t from bothsides of the model and add and subtract β 0 x t 1 or Δy t =(α 1 1)y t 1 + α 0 + β 0 Δx t +(β 0 + β 1 )x t 1 + u t Δy t = β 0 Δx t +(α 1 1)[y t 1 α 0 1 α 1 β 0 + β 1 1 α 1 x t 1 ]+u t
13 Model selection Consider a collection of parametric models: {f i (x,θ)}, where θ Θ j for j =1,, J. Some linear structure usually imposed on the parameter space: Θ j = m j θj,wherem j is a linear subspace of R p J and p 1 < p 2 < < p J. Also assume that the models are nested: θ 1 θ 2 θ J.
14 Model selection Akaike information criterion [Akaike (1969)] AIC(j) =l j (ˆθ) p j, where l j (ˆθ) denotes the log-likelihood corresponding to the j t h model. Akaike s selection rule is simply choose the model j which maximizes AIC(j). Schwarz s information criterion [Schwarz (1978)] SIC(j) =l j (ˆθ) 1 2 p j log n where ĵ =argmaxs(j). p(ĵ = j ) 1. (1/2) log n > 1forn > 8 the SIC penalty is larger than the AIC penalty.
15 Model selection Connection with classical hypothesis testing: Under quite general conditions for nested models for p j > p i = p. 2(l j (ˆθ j ) l i (ˆθ i )) χ 2 p j p i SIC would choose j over i iff 2(l j l i ) p j p i > log n log n can be interpreted as an implicit critical value for the model selection decision based on SIC Make sense? AIC: an implicit critical value is 2: positive probability of Type I error.
16 Model selection SIC in the linear regression model: consider the Gaussian linear regression model: l(β,σ) = n 2 log(2π) n 2 log σ2 S 2σ 2 where S =(y X β) (y X β). Evaluating at ˆβ and ˆσ 2 = S/n l( ˆβ, ˆσ) = n 2 log(2π) n 2 n log ˆσ2 2 Thus we maximize SIC l i 1 2 p i log(n) which is the same as minimizing log ˆσ 2 j +(p j /n)logn
17 Model selection Connection with F-test statistic: Note l i l j = n 2 (log ˆσ2 j log ˆσ 2 i ) = n 2 log(ˆσ2 j /ˆσ i 2 ) ( ) = n 2 log 1 ˆσ2 i ˆσ j 2 ˆσ i 2 Usual Taylor-series approximation for log(1 ± a) fora small 2(l i l j ) n(ˆσ2 j ˆσ i 2) ˆσ i 2.
18 Model selection, Shrinkage and the LASSO The information criterion approach: balance the two objectives of simplicity (penalty) and goodness-of-fit (fidelity). Too simple model risks serious bias Too complicate model risks high degree of uncertainty Start with Bayesian method for linear regression model: Shrinkage methods or Stein-rule methods
19 Model selection, Shrinkage and the LASSO Consider the linear model: where u N(0,σ 2 I). y = X β + u, L(y b) =(2π) n/2 σ n exp{ 1 2σ 2 ( ˆβ b) X X ( ˆβ b)} Suppose that we have a prior that β N(β 0, Ω), i.e., π(b) =(2π) p/2 Ω 1/2 exp{ 1 2 (b β 0) Ω 1 (b β 0 )} Using the Bayes rule p(b y) = L(y b) π(b) L(y b)π(b)db.
20 Model selection, Shrinkage and the LASSO Then p(b y) =κ exp{ 1 2 (b β) (σ 2 X X +Ω 1 )(b β)} where κ is a constant and β =(σ 2 (X X )+Ω 1 ) 1 (σ 2 (X X ) ˆβ +Ω 1 β 0 ). the posterior distribution is also Gaussian with mean β. ˆβ and β0 have covariance matrices σ 2 (x x) 1 and Ω, respectively. They are weighted by the inverses of the covariance matrices.
21 Model selection, Shrinkage and the LASSO Tibshirani (1996) considered the l 1 norm in the penalty term Pen(θ) = p θ i i=1 and he proposed the following regression model min (yi x i θ)2 + λpen(θ) θ for some appropriately chosen λ the lasso (least absolute shrinkage and selection operator). Ridge regression: min θ (yi x i θ)2 + λ p i=1 θ 2 i
22 Model selection, Shrinkage and the LASSO One can also use the l 1 fidelity criterion: min yi x i θ + λpen(θ) θ This has been done by Wang, Li and Jiang (JBES, 2007).
23 Model selection, Shrinkage and the LASSO Figure: LASSO and Ridge shrinkage
24 Bias and Variance Consider the following stylized situation in regression (long-model) (short-model) y = X β + Zγ + u y = X β + v What are the price we pay when we misspecify the model...
25 Bias and Variance Assume that the long model is true and we estimate the short model (omitted variables). E ˆβ s = E(X X ) 1 X y = E(X X ) 1 X (X β + Zγ + u) = β +(X X ) 1 X Zγ the bias associated with estimation of β Gγ =(X X ) 1 X Zγ where G is obtained by regressing the columns Z on the columns of X. Bias vanishes if γ =0orifX is orthogonal to Z.
26 Bias and Variance Example: One estimates a static model when a dynamic one is the true model. Suppose the correct specification: y t = α + p β i x t i + u t i=0 where x t is exogenous variable. Instead we estimate the static model y t = α + β 0 x t + v t the relationship between our estimate of β 0 in the static model and the coefficients of the dynamic model...
27 Bias and Variance E ˆβ 0 = β 0 + p g i β i where g i denotes the slope coefficient of the obtained in a regression of x t i on x t, and an intercept. If x t is strongly trended, then these g i will tend to be close to one and E ˆβ 0 will be close to p i=0 β i: long-run effect. i=1
28 Bias and Variance Assume that the short model is true and we estimate the long model. bias?... E ˆβ L = E(X M Z X ) 1 X M Z y = E(X M Z X ) 1 X M Z (X β + u) = β Happy? There is a price to be paid of estimating parameters γ...
29 Bias and Variance Proposition ˆβ s = ˆβ L + G ˆγ L Proposition Assuming V (y) =E(y Ey)(y Ey) = σ 2 I, V ( ˆβ L )=V ( ˆβ s )+GV (ˆγ L )G... the variability of the long estimate always exceeds the variability of the short estimate... but...
30 Fishing concerns the difficulties associated with preliminary testing and model selection... based on Freedman (1983, American Statistician) (seealso Leeb and Pötscher (2005, ET)) He consider a model of the form: y i = x i β 0 + u i where u i iidn (0,σ 2 ). The matrix X =(x i )isn p and X X = I p. And p as n so that p/n ρ for some 0 <ρ<1. He also assumes β 0 =0.
31 Fishing Theorem For the above model, R 2 n ρ and F n 1. Proof: The usual F n statistic for the model is really distributed as F.So EF n =(n p)/(n p 2) which tends to 1. And so F n = n p 1 p R 2 n 1 Rn 2 ( ) n p 1 Rn 2 = F / + F p Thus since F 1wehavethatR 2 n ρ.
32 Fishing Now consider the following case: all p variables are initially tried. Those attaining α-level of significance in a standard t-test are retained, say, q n,α of them. Then the model is reestimated with only these variables. Theorem For the above model, R 2 n,α g(λ α )ρ and F n,α where g(λ) = z >λ and λ is chosen so Φ(λ) =1 α/2. z 2 φ(z)dz ( ) g(λα) α / ( ) 1 g(λ)ρ 1 αρ,
33 Fishing Example: Suppose that n = 100, p = 50, so ρ =1/2. Set α =0.25 so λ =1.15 and g(λ) =0.72. Then E(Z 2 z >λ) 2.9 Rn,α 2 g(λ) ( ) g(λ) F n,α α (1 g(λ)ρ) 4.0 (1 αρ) Eq n,α = αρn = F 12,88,0.05 =1.88 P(F 12,88 > 4.0)
Model comparison and selection
BS2 Statistical Inference, Lectures 9 and 10, Hilary Term 2008 March 2, 2008 Hypothesis testing Consider two alternative models M 1 = {f (x; θ), θ Θ 1 } and M 2 = {f (x; θ), θ Θ 2 } for a sample (X = x)
More informationTesting Restrictions and Comparing Models
Econ. 513, Time Series Econometrics Fall 00 Chris Sims Testing Restrictions and Comparing Models 1. THE PROBLEM We consider here the problem of comparing two parametric models for the data X, defined by
More informationEconomics 536 Lecture 7. Introduction to Specification Testing in Dynamic Econometric Models
University of Illinois Fall 2016 Department of Economics Roger Koenker Economics 536 Lecture 7 Introduction to Specification Testing in Dynamic Econometric Models In this lecture I want to briefly describe
More informationLasso Maximum Likelihood Estimation of Parametric Models with Singular Information Matrices
Article Lasso Maximum Likelihood Estimation of Parametric Models with Singular Information Matrices Fei Jin 1,2 and Lung-fei Lee 3, * 1 School of Economics, Shanghai University of Finance and Economics,
More informationModel Selection and Geometry
Model Selection and Geometry Pascal Massart Université Paris-Sud, Orsay Leipzig, February Purpose of the talk! Concentration of measure plays a fundamental role in the theory of model selection! Model
More informationNonconcave Penalized Likelihood with A Diverging Number of Parameters
Nonconcave Penalized Likelihood with A Diverging Number of Parameters Jianqing Fan and Heng Peng Presenter: Jiale Xu March 12, 2010 Jianqing Fan and Heng Peng Presenter: JialeNonconcave Xu () Penalized
More informationMS-C1620 Statistical inference
MS-C1620 Statistical inference 10 Linear regression III Joni Virta Department of Mathematics and Systems Analysis School of Science Aalto University Academic year 2018 2019 Period III - IV 1 / 32 Contents
More informationSparse Linear Models (10/7/13)
STA56: Probabilistic machine learning Sparse Linear Models (0/7/) Lecturer: Barbara Engelhardt Scribes: Jiaji Huang, Xin Jiang, Albert Oh Sparsity Sparsity has been a hot topic in statistics and machine
More informationBayesian Estimation of Regression Coefficients Under Extended Balanced Loss Function
Communications in Statistics Theory and Methods, 43: 4253 4264, 2014 Copyright Taylor & Francis Group, LLC ISSN: 0361-0926 print / 1532-415X online DOI: 10.1080/03610926.2012.725498 Bayesian Estimation
More informationMISCELLANEOUS TOPICS RELATED TO LIKELIHOOD. Copyright c 2012 (Iowa State University) Statistics / 30
MISCELLANEOUS TOPICS RELATED TO LIKELIHOOD Copyright c 2012 (Iowa State University) Statistics 511 1 / 30 INFORMATION CRITERIA Akaike s Information criterion is given by AIC = 2l(ˆθ) + 2k, where l(ˆθ)
More informationEcon 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines
Econ 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines Maximilian Kasy Department of Economics, Harvard University 1 / 37 Agenda 6 equivalent representations of the
More informationAnalysis of the AIC Statistic for Optimal Detection of Small Changes in Dynamic Systems
Analysis of the AIC Statistic for Optimal Detection of Small Changes in Dynamic Systems Jeremy S. Conner and Dale E. Seborg Department of Chemical Engineering University of California, Santa Barbara, CA
More informationOn the equivalence of confidence interval estimation based on frequentist model averaging and least-squares of the full model in linear regression
Working Paper 2016:1 Department of Statistics On the equivalence of confidence interval estimation based on frequentist model averaging and least-squares of the full model in linear regression Sebastian
More informationRegression I: Mean Squared Error and Measuring Quality of Fit
Regression I: Mean Squared Error and Measuring Quality of Fit -Applied Multivariate Analysis- Lecturer: Darren Homrighausen, PhD 1 The Setup Suppose there is a scientific problem we are interested in solving
More informationExtended Bayesian Information Criteria for Model Selection with Large Model Spaces
Extended Bayesian Information Criteria for Model Selection with Large Model Spaces Jiahua Chen, University of British Columbia Zehua Chen, National University of Singapore (Biometrika, 2008) 1 / 18 Variable
More informationLeast Squares Regression
E0 70 Machine Learning Lecture 4 Jan 7, 03) Least Squares Regression Lecturer: Shivani Agarwal Disclaimer: These notes are a brief summary of the topics covered in the lecture. They are not a substitute
More informationMachine Learning for OR & FE
Machine Learning for OR & FE Regression II: Regularization and Shrinkage Methods Martin Haugh Department of Industrial Engineering and Operations Research Columbia University Email: martin.b.haugh@gmail.com
More informationAnalysis Methods for Supersaturated Design: Some Comparisons
Journal of Data Science 1(2003), 249-260 Analysis Methods for Supersaturated Design: Some Comparisons Runze Li 1 and Dennis K. J. Lin 2 The Pennsylvania State University Abstract: Supersaturated designs
More informationStatistics 203: Introduction to Regression and Analysis of Variance Penalized models
Statistics 203: Introduction to Regression and Analysis of Variance Penalized models Jonathan Taylor - p. 1/15 Today s class Bias-Variance tradeoff. Penalized regression. Cross-validation. - p. 2/15 Bias-variance
More informationRegression, Ridge Regression, Lasso
Regression, Ridge Regression, Lasso Fabio G. Cozman - fgcozman@usp.br October 2, 2018 A general definition Regression studies the relationship between a response variable Y and covariates X 1,..., X n.
More informationBIO5312 Biostatistics Lecture 13: Maximum Likelihood Estimation
BIO5312 Biostatistics Lecture 13: Maximum Likelihood Estimation Yujin Chung November 29th, 2016 Fall 2016 Yujin Chung Lec13: MLE Fall 2016 1/24 Previous Parametric tests Mean comparisons (normality assumption)
More informationIntroductory Econometrics
Based on the textbook by Wooldridge: : A Modern Approach Robert M. Kunst robert.kunst@univie.ac.at University of Vienna and Institute for Advanced Studies Vienna November 23, 2013 Outline Introduction
More informationLeast Squares Regression
CIS 50: Machine Learning Spring 08: Lecture 4 Least Squares Regression Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may not cover all the
More informationDay 4: Shrinkage Estimators
Day 4: Shrinkage Estimators Kenneth Benoit Data Mining and Statistical Learning March 9, 2015 n versus p (aka k) Classical regression framework: n > p. Without this inequality, the OLS coefficients have
More informationSpring 2017 Econ 574 Roger Koenker. Lecture 14 GEE-GMM
University of Illinois Department of Economics Spring 2017 Econ 574 Roger Koenker Lecture 14 GEE-GMM Throughout the course we have emphasized methods of estimation and inference based on the principle
More informationLinear Regression. Junhui Qian. October 27, 2014
Linear Regression Junhui Qian October 27, 2014 Outline The Model Estimation Ordinary Least Square Method of Moments Maximum Likelihood Estimation Properties of OLS Estimator Unbiasedness Consistency Efficiency
More informationEmpirical Market Microstructure Analysis (EMMA)
Empirical Market Microstructure Analysis (EMMA) Lecture 3: Statistical Building Blocks and Econometric Basics Prof. Dr. Michael Stein michael.stein@vwl.uni-freiburg.de Albert-Ludwigs-University of Freiburg
More informationVector Auto-Regressive Models
Vector Auto-Regressive Models Laurent Ferrara 1 1 University of Paris Nanterre M2 Oct. 2018 Overview of the presentation 1. Vector Auto-Regressions Definition Estimation Testing 2. Impulse responses functions
More informationVAR Models and Applications
VAR Models and Applications Laurent Ferrara 1 1 University of Paris West M2 EIPMC Oct. 2016 Overview of the presentation 1. Vector Auto-Regressions Definition Estimation Testing 2. Impulse responses functions
More informationPrelim Examination. Friday August 11, Time limit: 150 minutes
University of Pennsylvania Economics 706, Fall 2017 Prelim Prelim Examination Friday August 11, 2017. Time limit: 150 minutes Instructions: (i) The total number of points is 80, the number of points for
More informationThe regression model with one fixed regressor cont d
The regression model with one fixed regressor cont d 3150/4150 Lecture 4 Ragnar Nymoen 27 January 2012 The model with transformed variables Regression with transformed variables I References HGL Ch 2.8
More informationMachine Learning Linear Classification. Prof. Matteo Matteucci
Machine Learning Linear Classification Prof. Matteo Matteucci Recall from the first lecture 2 X R p Regression Y R Continuous Output X R p Y {Ω 0, Ω 1,, Ω K } Classification Discrete Output X R p Y (X)
More informationECON 4160: Econometrics-Modelling and Systems Estimation Lecture 9: Multiple equation models II
ECON 4160: Econometrics-Modelling and Systems Estimation Lecture 9: Multiple equation models II Ragnar Nymoen Department of Economics University of Oslo 9 October 2018 The reference to this lecture is:
More informationQuick Review on Linear Multiple Regression
Quick Review on Linear Multiple Regression Mei-Yuan Chen Department of Finance National Chung Hsing University March 6, 2007 Introduction for Conditional Mean Modeling Suppose random variables Y, X 1,
More informationISyE 691 Data mining and analytics
ISyE 691 Data mining and analytics Regression Instructor: Prof. Kaibo Liu Department of Industrial and Systems Engineering UW-Madison Email: kliu8@wisc.edu Office: Room 3017 (Mechanical Engineering Building)
More informationBayesian Decision and Bayesian Learning
Bayesian Decision and Bayesian Learning Ying Wu Electrical Engineering and Computer Science Northwestern University Evanston, IL 60208 http://www.eecs.northwestern.edu/~yingwu 1 / 30 Bayes Rule p(x ω i
More informationSTAT 535 Lecture 5 November, 2018 Brief overview of Model Selection and Regularization c Marina Meilă
STAT 535 Lecture 5 November, 2018 Brief overview of Model Selection and Regularization c Marina Meilă mmp@stat.washington.edu Reading: Murphy: BIC, AIC 8.4.2 (pp 255), SRM 6.5 (pp 204) Hastie, Tibshirani
More informationSGN Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection
SG 21006 Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection Ioan Tabus Department of Signal Processing Tampere University of Technology Finland 1 / 28
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationA Significance Test for the Lasso
A Significance Test for the Lasso Lockhart R, Taylor J, Tibshirani R, and Tibshirani R Ashley Petersen May 14, 2013 1 Last time Problem: Many clinical covariates which are important to a certain medical
More informationGraduate Econometrics I: Unbiased Estimation
Graduate Econometrics I: Unbiased Estimation Yves Dominicy Université libre de Bruxelles Solvay Brussels School of Economics and Management ECARES Yves Dominicy Graduate Econometrics I: Unbiased Estimation
More informationBayesian Gaussian / Linear Models. Read Sections and 3.3 in the text by Bishop
Bayesian Gaussian / Linear Models Read Sections 2.3.3 and 3.3 in the text by Bishop Multivariate Gaussian Model with Multivariate Gaussian Prior Suppose we model the observed vector b as having a multivariate
More informationStatistics 910, #5 1. Regression Methods
Statistics 910, #5 1 Overview Regression Methods 1. Idea: effects of dependence 2. Examples of estimation (in R) 3. Review of regression 4. Comparisons and relative efficiencies Idea Decomposition Well-known
More informationProbabilistic machine learning group, Aalto University Bayesian theory and methods, approximative integration, model
Aki Vehtari, Aalto University, Finland Probabilistic machine learning group, Aalto University http://research.cs.aalto.fi/pml/ Bayesian theory and methods, approximative integration, model assessment and
More informationCovariance function estimation in Gaussian process regression
Covariance function estimation in Gaussian process regression François Bachoc Department of Statistics and Operations Research, University of Vienna WU Research Seminar - May 2015 François Bachoc Gaussian
More informationAdvanced Statistics I : Gaussian Linear Model (and beyond)
Advanced Statistics I : Gaussian Linear Model (and beyond) Aurélien Garivier CNRS / Telecom ParisTech Centrale Outline One and Two-Sample Statistics Linear Gaussian Model Model Reduction and model Selection
More informationLecture 2 Machine Learning Review
Lecture 2 Machine Learning Review CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago March 29, 2017 Things we will look at today Formal Setup for Supervised Learning Things
More informationOn the Behavior of Marginal and Conditional Akaike Information Criteria in Linear Mixed Models
On the Behavior of Marginal and Conditional Akaike Information Criteria in Linear Mixed Models Thomas Kneib Department of Mathematics Carl von Ossietzky University Oldenburg Sonja Greven Department of
More informationOn the Behavior of Marginal and Conditional Akaike Information Criteria in Linear Mixed Models
On the Behavior of Marginal and Conditional Akaike Information Criteria in Linear Mixed Models Thomas Kneib Institute of Statistics and Econometrics Georg-August-University Göttingen Department of Statistics
More informationSummer School in Statistics for Astronomers V June 1 - June 6, Regression. Mosuk Chow Statistics Department Penn State University.
Summer School in Statistics for Astronomers V June 1 - June 6, 2009 Regression Mosuk Chow Statistics Department Penn State University. Adapted from notes prepared by RL Karandikar Mean and variance Recall
More informationBayesian Decision Theory
Bayesian Decision Theory Selim Aksoy Department of Computer Engineering Bilkent University saksoy@cs.bilkent.edu.tr CS 551, Fall 2017 CS 551, Fall 2017 c 2017, Selim Aksoy (Bilkent University) 1 / 46 Bayesian
More informationModel Selection Tutorial 2: Problems With Using AIC to Select a Subset of Exposures in a Regression Model
Model Selection Tutorial 2: Problems With Using AIC to Select a Subset of Exposures in a Regression Model Centre for Molecular, Environmental, Genetic & Analytic (MEGA) Epidemiology School of Population
More informationRestricted Maximum Likelihood in Linear Regression and Linear Mixed-Effects Model
Restricted Maximum Likelihood in Linear Regression and Linear Mixed-Effects Model Xiuming Zhang zhangxiuming@u.nus.edu A*STAR-NUS Clinical Imaging Research Center October, 015 Summary This report derives
More informationPaper Review: Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties by Jianqing Fan and Runze Li (2001)
Paper Review: Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties by Jianqing Fan and Runze Li (2001) Presented by Yang Zhao March 5, 2010 1 / 36 Outlines 2 / 36 Motivation
More informationData Mining Stat 588
Data Mining Stat 588 Lecture 02: Linear Methods for Regression Department of Statistics & Biostatistics Rutgers University September 13 2011 Regression Problem Quantitative generic output variable Y. Generic
More informationStatistics 262: Intermediate Biostatistics Model selection
Statistics 262: Intermediate Biostatistics Model selection Jonathan Taylor & Kristin Cobb Statistics 262: Intermediate Biostatistics p.1/?? Today s class Model selection. Strategies for model selection.
More informationSTA414/2104 Statistical Methods for Machine Learning II
STA414/2104 Statistical Methods for Machine Learning II Murat A. Erdogdu & David Duvenaud Department of Computer Science Department of Statistical Sciences Lecture 3 Slide credits: Russ Salakhutdinov Announcements
More informationModelling the Covariance
Modelling the Covariance Jamie Monogan Washington University in St Louis February 9, 2010 Jamie Monogan (WUStL) Modelling the Covariance February 9, 2010 1 / 13 Objectives By the end of this meeting, participants
More informationARIMA Modelling and Forecasting
ARIMA Modelling and Forecasting Economic time series often appear nonstationary, because of trends, seasonal patterns, cycles, etc. However, the differences may appear stationary. Δx t x t x t 1 (first
More informationQuantile Regression for Panel/Longitudinal Data
Quantile Regression for Panel/Longitudinal Data Roger Koenker University of Illinois, Urbana-Champaign University of Minho 12-14 June 2017 y it 0 5 10 15 20 25 i = 1 i = 2 i = 3 0 2 4 6 8 Roger Koenker
More informationTransformations The bias-variance tradeoff Model selection criteria Remarks. Model selection I. Patrick Breheny. February 17
Model selection I February 17 Remedial measures Suppose one of your diagnostic plots indicates a problem with the model s fit or assumptions; what options are available to you? Generally speaking, you
More informationEstimating prediction error in mixed models
Estimating prediction error in mixed models benjamin saefken, thomas kneib georg-august university goettingen sonja greven ludwig-maximilians-university munich 1 / 12 GLMM - Generalized linear mixed models
More informationDecision theory. 1 We may also consider randomized decision rules, where δ maps observed data D to a probability distribution over
Point estimation Suppose we are interested in the value of a parameter θ, for example the unknown bias of a coin. We have already seen how one may use the Bayesian method to reason about θ; namely, we
More informationVector Autoregressive Model. Vector Autoregressions II. Estimation of Vector Autoregressions II. Estimation of Vector Autoregressions I.
Vector Autoregressive Model Vector Autoregressions II Empirical Macroeconomics - Lect 2 Dr. Ana Beatriz Galvao Queen Mary University of London January 2012 A VAR(p) model of the m 1 vector of time series
More informationShort T Panels - Review
Short T Panels - Review We have looked at methods for estimating parameters on time-varying explanatory variables consistently in panels with many cross-section observation units but a small number of
More informationHigh-dimensional Covariance Estimation Based On Gaussian Graphical Models
High-dimensional Covariance Estimation Based On Gaussian Graphical Models Shuheng Zhou, Philipp Rutimann, Min Xu and Peter Buhlmann February 3, 2012 Problem definition Want to estimate the covariance matrix
More informationChapter 3: Maximum Likelihood Theory
Chapter 3: Maximum Likelihood Theory Florian Pelgrin HEC September-December, 2010 Florian Pelgrin (HEC) Maximum Likelihood Theory September-December, 2010 1 / 40 1 Introduction Example 2 Maximum likelihood
More informationLecture 14: Shrinkage
Lecture 14: Shrinkage Reading: Section 6.2 STATS 202: Data mining and analysis October 27, 2017 1 / 19 Shrinkage methods The idea is to perform a linear regression, while regularizing or shrinking the
More informationARMA MODELS Herman J. Bierens Pennsylvania State University February 23, 2009
1. Introduction Given a covariance stationary process µ ' E[ ], the Wold decomposition states that where U t ARMA MODELS Herman J. Bierens Pennsylvania State University February 23, 2009 with vanishing
More informationLinear Model Selection and Regularization
Linear Model Selection and Regularization Recall the linear model Y = β 0 + β 1 X 1 + + β p X p + ɛ. In the lectures that follow, we consider some approaches for extending the linear model framework. In
More informationECON 4160, Autumn term Lecture 1
ECON 4160, Autumn term 2017. Lecture 1 a) Maximum Likelihood based inference. b) The bivariate normal model Ragnar Nymoen University of Oslo 24 August 2017 1 / 54 Principles of inference I Ordinary least
More informationTopic 12 Overview of Estimation
Topic 12 Overview of Estimation Classical Statistics 1 / 9 Outline Introduction Parameter Estimation Classical Statistics Densities and Likelihoods 2 / 9 Introduction In the simplest possible terms, the
More informationFrequentist-Bayesian Model Comparisons: A Simple Example
Frequentist-Bayesian Model Comparisons: A Simple Example Consider data that consist of a signal y with additive noise: Data vector (N elements): D = y + n The additive noise n has zero mean and diagonal
More informationHigh-dimensional regression with unknown variance
High-dimensional regression with unknown variance Christophe Giraud Ecole Polytechnique march 2012 Setting Gaussian regression with unknown variance: Y i = f i + ε i with ε i i.i.d. N (0, σ 2 ) f = (f
More informationTesting methodology. It often the case that we try to determine the form of the model on the basis of data
Testing methodology It often the case that we try to determine the form of the model on the basis of data The simplest case: we try to determine the set of explanatory variables in the model Testing for
More informationAssociation studies and regression
Association studies and regression CM226: Machine Learning for Bioinformatics. Fall 2016 Sriram Sankararaman Acknowledgments: Fei Sha, Ameet Talwalkar Association studies and regression 1 / 104 Administration
More informationStatistical Measures of Uncertainty in Inverse Problems
Statistical Measures of Uncertainty in Inverse Problems Workshop on Uncertainty in Inverse Problems Institute for Mathematics and Its Applications Minneapolis, MN 19-26 April 2002 P.B. Stark Department
More informationMultiple Regression Analysis. Part III. Multiple Regression Analysis
Part III Multiple Regression Analysis As of Sep 26, 2017 1 Multiple Regression Analysis Estimation Matrix form Goodness-of-Fit R-square Adjusted R-square Expected values of the OLS estimators Irrelevant
More informationMachine Learning Linear Regression. Prof. Matteo Matteucci
Machine Learning Linear Regression Prof. Matteo Matteucci Outline 2 o Simple Linear Regression Model Least Squares Fit Measures of Fit Inference in Regression o Multi Variate Regession Model Least Squares
More information5. Erroneous Selection of Exogenous Variables (Violation of Assumption #A1)
5. Erroneous Selection of Exogenous Variables (Violation of Assumption #A1) Assumption #A1: Our regression model does not lack of any further relevant exogenous variables beyond x 1i, x 2i,..., x Ki and
More informationGeneral Linear Model: Statistical Inference
Chapter 6 General Linear Model: Statistical Inference 6.1 Introduction So far we have discussed formulation of linear models (Chapter 1), estimability of parameters in a linear model (Chapter 4), least
More informationStat 5100 Handout #26: Variations on OLS Linear Regression (Ch. 11, 13)
Stat 5100 Handout #26: Variations on OLS Linear Regression (Ch. 11, 13) 1. Weighted Least Squares (textbook 11.1) Recall regression model Y = β 0 + β 1 X 1 +... + β p 1 X p 1 + ε in matrix form: (Ch. 5,
More informationFor more information about how to cite these materials visit
Author(s): Kerby Shedden, Ph.D., 2010 License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution Share Alike 3.0 License: http://creativecommons.org/licenses/by-sa/3.0/
More informationIntroduction to Bayesian Inference
University of Pennsylvania EABCN Training School May 10, 2016 Bayesian Inference Ingredients of Bayesian Analysis: Likelihood function p(y φ) Prior density p(φ) Marginal data density p(y ) = p(y φ)p(φ)dφ
More informationShort Questions (Do two out of three) 15 points each
Econometrics Short Questions Do two out of three) 5 points each ) Let y = Xβ + u and Z be a set of instruments for X When we estimate β with OLS we project y onto the space spanned by X along a path orthogonal
More informationDETECTION theory deals primarily with techniques for
ADVANCED SIGNAL PROCESSING SE Optimum Detection of Deterministic and Random Signals Stefan Tertinek Graz University of Technology turtle@sbox.tugraz.at Abstract This paper introduces various methods for
More informationDavid Giles Bayesian Econometrics
9. Model Selection - Theory David Giles Bayesian Econometrics One nice feature of the Bayesian analysis is that we can apply it to drawing inferences about entire models, not just parameters. Can't do
More informationLinear Models A linear model is defined by the expression
Linear Models A linear model is defined by the expression x = F β + ɛ. where x = (x 1, x 2,..., x n ) is vector of size n usually known as the response vector. β = (β 1, β 2,..., β p ) is the transpose
More information11. Simultaneous-Equation Models
11. Simultaneous-Equation Models Up to now: Estimation and inference in single-equation models Now: Modeling and estimation of a system of equations 328 Example: [I] Analysis of the impact of advertisement
More informationEconometrics of Panel Data
Econometrics of Panel Data Jakub Mućk Meeting # 6 Jakub Mućk Econometrics of Panel Data Meeting # 6 1 / 36 Outline 1 The First-Difference (FD) estimator 2 Dynamic panel data models 3 The Anderson and Hsiao
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More information7. Integrated Processes
7. Integrated Processes Up to now: Analysis of stationary processes (stationary ARMA(p, q) processes) Problem: Many economic time series exhibit non-stationary patterns over time 226 Example: We consider
More informationMLR Model Selection. Author: Nicholas G Reich, Jeff Goldsmith. This material is part of the statsteachr project
MLR Model Selection Author: Nicholas G Reich, Jeff Goldsmith This material is part of the statsteachr project Made available under the Creative Commons Attribution-ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-sa/3.0/deed.en
More informationLinear Regression (9/11/13)
STA561: Probabilistic machine learning Linear Regression (9/11/13) Lecturer: Barbara Engelhardt Scribes: Zachary Abzug, Mike Gloudemans, Zhuosheng Gu, Zhao Song 1 Why use linear regression? Figure 1: Scatter
More informationIEOR 165 Lecture 7 1 Bias-Variance Tradeoff
IEOR 165 Lecture 7 Bias-Variance Tradeoff 1 Bias-Variance Tradeoff Consider the case of parametric regression with β R, and suppose we would like to analyze the error of the estimate ˆβ in comparison to
More informationVector Autoregression
Vector Autoregression Jamie Monogan University of Georgia February 27, 2018 Jamie Monogan (UGA) Vector Autoregression February 27, 2018 1 / 17 Objectives By the end of these meetings, participants should
More information1 Hypothesis Testing and Model Selection
A Short Course on Bayesian Inference (based on An Introduction to Bayesian Analysis: Theory and Methods by Ghosh, Delampady and Samanta) Module 6: From Chapter 6 of GDS 1 Hypothesis Testing and Model Selection
More informationMidterm Suggested Solutions
CUHK Dept. of Economics Spring 2011 ECON 4120 Sung Y. Park Midterm Suggested Solutions Q1 (a) In time series, autocorrelation measures the correlation between y t and its lag y t τ. It is defined as. ρ(τ)
More informationUNIVERSITETET I OSLO
UNIVERSITETET I OSLO Det matematisk-naturvitenskapelige fakultet Examination in: STK4030 Modern data analysis - FASIT Day of examination: Friday 13. Desember 2013. Examination hours: 14.30 18.30. This
More informationECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters
ECE531 Lecture 6: Detection of Discrete-Time Signals with Random Parameters D. Richard Brown III Worcester Polytechnic Institute 26-February-2009 Worcester Polytechnic Institute D. Richard Brown III 26-February-2009
More information