Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory
|
|
|
- Christopher Payne
- 5 years ago
- Views:
Transcription
1 Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory Andreas Buja joint with the PoSI Group: Richard Berk, Lawrence Brown, Linda Zhao, Kai Zhang Ed George, Mikhail Traskin, Emil Pitkin, Dan McCarthy Mostly at the Department of Statistics, The Wharton School University of Pennsylvania WHOA-PSI 2016/10/02
2 Outline A list of mind puzzlers: Why are we treating regressors X as fixed? Why is the justification for conditioning on X wrong? Can we treat regressors as random? Degrees of misspecification are universal. Errors are not the only source of sampling variation. Model-trusting standard errors can be wrong. What is the meaning of regression parameters when the model is wrong? There exists a notion of well-specification without models. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 2 / 20
3 Principle: Models, yes, but do Not Believe in Them! Models are useful... to think about data, to imagine generative mechanisms, to formalize quantities of interest (in the form of parameters), to guide us toward inference (testing, CIs), and to specify ideal conditions for inference. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 3 / 20
4 Principle: Models, yes, but do Not Believe in Them! Models are useful... to think about data, to imagine generative mechanisms, to formalize quantities of interest (in the form of parameters), to guide us toward inference (testing, CIs), and to specify ideal conditions for inference. Traditions of not believing in model correctness: G.E.P. Box:... Cox (1995): The very word model implies simplification and idealization. A modern reason: Models are obtained by model selection. (Model selection makes good compromises, but does not find truth.) To be shown: Lip service to these maxims is not enough; there are serious consequences. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 3 / 20
5 PoSI, just briefly PoSI: Reduction of post-selection inference to simultaneous inference Computational cost: exponential in p, linear in N PoSI covering all submodels feasible for p 20. Sparse PoSI for M 5, e.g., feasible for p 60. Coverage guarantees are marginal, not conditional. Statistical practice is minimally changed: Use a single ˆσ across all submodels M. For PoSI-valid CIs, enlarge the multiplier of SE[ ˆβ j M ] suitably. PoSI covers for formal, informal, and post-hoc selection. PoSI solves the circularity problem of selective inference: Selecting regressors with PoSI tests does not invalidate the tests. PoSI is possible for response and transformation selection. Current PoSI uses fixed-x theory, allows 1st order misspecification, but assumes normality, homoskedasticity and a valid ˆσ. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 4 / 20
6 From Fixed-X to Random-X Regression A different but obvious solution: Data Splitting Split the data into a model selection sample and an estimation & inference sample. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 5 / 20
7 From Fixed-X to Random-X Regression A different but obvious solution: Data Splitting Split the data into Pros: a model selection sample and an estimation & inference sample. Valid inference for the selected model. Works for any model Often less conservative than PoSI. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 5 / 20
8 From Fixed-X to Random-X Regression A different but obvious solution: Data Splitting Split the data into Pros: a model selection sample and an estimation & inference sample. Valid inference for the selected model. Works for any model Often less conservative than PoSI. Cons: Randomness from single split Reduced effective sample size More model selection uncertainty More estimation uncertainty Loss of conditionality on X Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 5 / 20
9 Why Conditioning on X is Wrong With split-sampling and CV we break the fixed-x paradigm. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 6 / 20
10 Why Conditioning on X is Wrong With split-sampling and CV we break the fixed-x paradigm. What is the justification for conditioning on X, i.e., treating X as fixed? Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 6 / 20
11 Why Conditioning on X is Wrong With split-sampling and CV we break the fixed-x paradigm. What is the justification for conditioning on X, i.e., treating X as fixed? Answer: Fisher s ancillarity argument for X. p(y, x; β (1) ) p(y, x; β (0) ) = p(y x; β(1) )p(x) p(y x; β (0) )p(x) = p(y x; β(1) ) p(y x; β (0) ) Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 6 / 20
12 Why Conditioning on X is Wrong With split-sampling and CV we break the fixed-x paradigm. What is the justification for conditioning on X, i.e., treating X as fixed? Answer: Fisher s ancillarity argument for X. p(y, x; β (1) ) p(y, x; β (0) ) = p(y x; β(1) )p(x) p(y x; β (0) )p(x) = p(y x; β(1) ) p(y x; β (0) ) Important! The ancillarity argument assumes correctness of the model. Refutation of Regressor Ancillarity under Misspecification: Assume xi random, y i = f (x i ) nonlinear deterministic, no errors. (There could be error, but this is not relevant.) Fit a linear function ŷi = β 0 + β 1 x i. A situation with misspecification, but no errors. Conditional on x i, estimates ˆβ j have no sampling variability. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 6 / 20
13 Why Conditioning on X is Wrong With split-sampling and CV we break the fixed-x paradigm. What is the justification for conditioning on X, i.e., treating X as fixed? Answer: Fisher s ancillarity argument for X. p(y, x; β (1) ) p(y, x; β (0) ) = p(y x; β(1) )p(x) p(y x; β (0) )p(x) = p(y x; β(1) ) p(y x; β (0) ) Important! The ancillarity argument assumes correctness of the model. Refutation of Regressor Ancillarity under Misspecification: Assume xi random, y i = f (x i ) nonlinear deterministic, no errors. (There could be error, but this is not relevant.) Fit a linear function ŷi = β 0 + β 1 x i. A situation with misspecification, but no errors. Conditional on x i, estimates ˆβ j have no sampling variability. However, watch The Unconditional Movie! source(" buja/src-conspiracy-animation2.r") Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 6 / 20
14 Random X and Misspecification Randomness of X and misspecification conspire to create sampling variability in the estimates. Consequence: Under misspecification, conditioning on X is wrong. Source 1 of model-robust & unconditional inference: Asymptotic inference based on Eicker/Huber/White s Sandwich Estimator of Standard Error. V[ ˆβ] = E[ X X ] 1 E[(Y X β) X X ] E[ X X ] 1. Source 2 of model-robust & unconditional inference: the Pairs or x-y Bootstrap (not the residual bootstrap) Validity of inference is in an assumption-lean/model-robust framework... Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 7 / 20
15 An Assumption-Lean Framework joint distribution, i.i.d. sampling: (y i, x i ) P(dy, d x) = P Y, X Assume properties sufficient to grant CLTs for estimates of interest.!!! No assumptions on µ( x) = E[Y X = x ], σ 2 ( x) = V[Y X = x]!!! Define a population OLS parameter: [ ( ) ] 2 β := argmin β E Y β X = E[ X X ] -1 E[ X Y ] This is the target of inference: β = β(p) β is a statistical functional a Regression Functional. Statistical Functional View of Regression (Random X Theory) Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 8 / 20
16 Implication 1 of Misspecification/Model-Robustness misspecified well-specified Y Y Y = µ(x) Y = µ(x) X Under misspecification parameters depend on the X distribution: β(p Y, X ) = β(p Y X, P X ) β(p Y, X ) = β(p Y X ) Upside down: Ancillarity = parameters do not affect the distribution. Upside up: "Ancillarity = the parameters are not affected by the distribution. X Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 9 / 20
17 Implication 2 of Misspecification/Model-Robustness misspecified well-specified y y x x Misspecification and random X generate sampling variability: V[ E[ ˆβ X ] ] > 0 V[ E[ ˆβ X ] ] = 0 Recall the unconditional movie. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 10 / 20
18 Mis/Well-Specification of Regression Functionals A powerful extension of the idea of mis/well-specification: Write a regression functional as β(p) = β(p Y X, P X ). Definition: A regression functional β(p) is well-specified for P Y X if β(p Y X, P X ) = β(p Y X ) Under well-specification... β(p) does not depend on the X distribution, and E[ ˆβ X] does not have sampling variability due to conspiracy. A new form of regressor ancillarity OLS: β(p) is well-specified for P iff E[Y X = x] = β x. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 11 / 20
19 Sampling Variability of Regression Functionals Plug-in estimates ˆβ = β(ˆp) of regression functionals (such as OLS estimators) have two sources of variability: the conditional distribution of y X, the marginal distribution of X combined with misspecification. This can be expressed with the formula V[ ˆβ ] = E[ V[ ˆβ X] ] + V[ E[ ˆβ X] ]. V[ ˆβ X ]: The only source of sampling variability in linear models theory. E[ ˆβ X ]: The target of estimation in linear models theory, but really a random variable under misspecification, hence a source of sampling variability. V[ E[ ˆβ X ] is the conspiracy part of sampling variability, caused by a synergy of randomness of X and misspecification. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 12 / 20
20 CLTs under Misspecification: Sandwich Form N ( ˆβ β(p)) N ( ˆβ E[ ˆβ X]) N (E[ ˆβ X] β(p)) D N (0, E[ X X ] 1 E[ m 2 ( X) X X ] E[ X X ) ] 1 D N (0, E[ X X ] 1 E[ σ 2 ( X) X X ] E[ X X ) ] 1 D N (0, E[ X X ] 1 E[ η 2 ( X) X X ] E[ X X ) ] 1 µ( X) = E[Y X] response surface η( X) = µ( X) β(p) X nonlinearity σ 2 ( X) = V[Y X] conditional noise variance ɛ = Y µ( X), noise δ = Y β(p) X = η( X) + ɛ, population residual m 2 ( X) = E[ δ 2 X] = η 2 ( X) + σ 2 ( X) conditional MSE Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 13 / 20
21 The x-y Bootstrap for Regression Model-robust framework: The nonparametric x-y bootstrap applies. Estimate SE( ˆβ j ) by Resample ( x i, y i ) pairs ( x i, y i ) ˆβ. ˆ SEboot ( ˆβ j ) = SD (β j ). Let SElin ˆ ( ˆβ j ) = models theory. ˆσ x j be the usual standard error erstimate of linear Question: Is the following always true? ˆ SE boot ( ˆβ j )? ˆ SElin ( ˆβ j ) Compare conventional and bootstrap standard errors empirically... Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 14 / 20
22 Conventional vs Bootstrap Std Errors LA Homeless Data (Richard Berk, UPenn) Response: StreetTotal of homeless in a census tract, N = 505 R , residual dfs = 498 ˆβ j SE lin SE boot SE boot /SE lin t lin MedianInc PropVacant PropMinority PerResidential PerCommercial PerIndustrial Reason for the discrepancy: misspecification. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 15 / 20
23 Reason 1 for SE boot SE lin : Nonlinearity Which has the smallest/largest true SE( ˆβ)? x y RAV ~ x y RAV ~ x y RAV ~ 1 Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 16 / 20
24 Reason 2 for SE boot SE lin : Heteroskedasticity Which has the smallest/largest true SE( ˆβ)? x y RAV ~ x y RAV ~ x y RAV ~ 1 Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 17 / 20
25 Sandwich and x-y Bootstrap Estimators The x-y bootstrap is asymptotically correct in the assumption-lean/model-robust framework, and so is the sandwich estimator of standard error. There exists a connection, based on the M-of-N bootstrap: Resample datasets of size M out of N with replacement and rescale ˆ SE M:N [ ˆβ j ] := (M/N) 1/2 SD [ ˆβ j ] Proposition: SEM:N ˆ [ ˆβ j ] SEsand ˆ [ ˆβ j ] (M ) I.e., the sandwich estimator is the limit of the M-of-N boostrap estimator. This holds for all sandwich estimators, not just those of OLS. Bootstrap estimators for small M may have advantages. Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 18 / 20
26 The Meaning of Slopes under Misspecification Allowing misspecification messes up our regression practice: What is the meaning of slopes under misspecification? y y x x Case-wise slopes: ˆβ = i w i b i, b i := y i x i, w i := x 2 i i x 2 i Pairwise slopes: ˆβ = ik w ik b ik, b ik := y i y k x i x k, w ik := (x i x k ) 2 i k (x i x k )2 Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 19 / 20
27 Conclusions Use models, but don t believe in them. Main use of models: Defining parameters. Extend the parameters beyond the model regression functionals. There is a new notion of well-specification for regression functionals. Random X & model misspecification generate sampling variability. Use inference that does not assume model correctness. Outlook: PoSI under complete misspecification Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 20 / 20
28 Conclusions Use models, but don t believe in them. Main use of models: Defining parameters. Extend the parameters beyond the model regression functionals. There is a new notion of well-specification for regression functionals. Random X & model misspecification generate sampling variability. Use inference that does not assume model correctness. Outlook: PoSI under complete misspecification THANKS! Andreas Buja (Wharton, UPenn) Why Do Statisticians Treat Predictors as Fixed? A Conspiracy Theory 2016/10/02 20 / 20
Higher-Order von Mises Expansions, Bagging and Assumption-Lean Inference
Higher-Order von Mises Expansions, Bagging and Assumption-Lean Inference Andreas Buja joint with: Richard Berk, Lawrence Brown, Linda Zhao, Arun Kuchibhotla, Kai Zhang Werner Stützle, Ed George, Mikhail
Post-Selection Inference for Models that are Approximations
Post-Selection Inference for Models that are Approximations Andreas Buja joint work with the PoSI Group: Richard Berk, Lawrence Brown, Linda Zhao, Kai Zhang Ed George, Mikhail Traskin, Emil Pitkin, Dan
Models as Approximations, Part I: A Conspiracy of Nonlinearity and Random Regressors in Linear Regression
Submitted to Statistical Science Models as Approximations, Part I: A Conspiracy of Nonlinearity and Random Regressors in Linear Regression Andreas Buja,,, Richard Berk, Lawrence Brown,, Edward George,
Generalized Cp (GCp) in a Model Lean Framework
Generalized Cp (GCp) in a Model Lean Framework Linda Zhao University of Pennsylvania Dedicated to Lawrence Brown (1940-2018) September 9th, 2018 WHOA 3 Joint work with Larry Brown, Juhui Cai, Arun Kumar
Models as Approximations A Conspiracy of Random Regressors and Model Misspecification Against Classical Inference in Regression
Submitted to Statistical Science Models as Approximations A Conspiracy of Random Regressors and Model Misspecification Against Classical Inference in Regression Andreas Buja,,, Richard Berk, Lawrence Brown,,
Lawrence D. Brown* and Daniel McCarthy*
Comments on the paper, An adaptive resampling test for detecting the presence of significant predictors by I. W. McKeague and M. Qian Lawrence D. Brown* and Daniel McCarthy* ABSTRACT: This commentary deals
Construction of PoSI Statistics 1
Construction of PoSI Statistics 1 Andreas Buja and Arun Kumar Kuchibhotla Department of Statistics University of Pennsylvania September 8, 2018 WHOA-PSI 2018 1 Joint work with "Larry s Group" at Wharton,
Inference for Approximating Regression Models
University of Pennsylvania ScholarlyCommons Publicly Accessible Penn Dissertations 1-1-2014 Inference for Approximating Regression Models Emil Pitkin University of Pennsylvania, pitkin@wharton.upenn.edu
Models as Approximations Part II: A General Theory of Model-Robust Regression
Submitted to Statistical Science Models as Approximations Part II: A General Theory of Model-Robust Regression Andreas Buja,,, Richard Berk, Lawrence Brown,, Ed George,, Arun Kumar Kuchibhotla,, and Linda
Fall 2017 STAT 532 Homework Peter Hoff. 1. Let P be a probability measure on a collection of sets A.
1. Let P be a probability measure on a collection of sets A. (a) For each n N, let H n be a set in A such that H n H n+1. Show that P (H n ) monotonically converges to P ( k=1 H k) as n. (b) For each n
Department of Criminology
Department of Criminology Working Paper No. 2015-13.01 Calibrated Percentile Double Bootstrap for Robust Linear Regression Inference Daniel McCarthy University of Pennsylvania Kai Zhang University of North
Mallows Cp for Out-of-sample Prediction
Mallows Cp for Out-of-sample Prediction Lawrence D. Brown Statistics Department, Wharton School, University of Pennsylvania lbrown@wharton.upenn.edu WHOA-PSI conference, St. Louis, Oct 1, 2016 Joint work
PoSI and its Geometry
PoSI and its Geometry Andreas Buja joint work with Richard Berk, Lawrence Brown, Kai Zhang, Linda Zhao Department of Statistics, The Wharton School University of Pennsylvania Philadelphia, USA Simon Fraser
PoSI Valid Post-Selection Inference
PoSI Valid Post-Selection Inference Andreas Buja joint work with Richard Berk, Lawrence Brown, Kai Zhang, Linda Zhao Department of Statistics, The Wharton School University of Pennsylvania Philadelphia,
Assumption Lean Regression
Assumption Lean Regression Richard Berk, Andreas Buja, Lawrence Brown, Edward George Arun Kumar Kuchibhotla, Weijie Su, and Linda Zhao University of Pennsylvania November 26, 2018 Abstract It is well known
Models as Approximations II: A Model-Free Theory of Parametric Regression
Submitted to Statistical Science Models as Approximations II: A Model-Free Theory of Parametric Regression Andreas Buja,, Lawrence Brown,, Arun Kumar Kuchibhotla, Richard Berk, Ed George,, and Linda Zhao,,
Discussant: Lawrence D Brown* Statistics Department, Wharton, Univ. of Penn.
Discussion of Estimation and Accuracy After Model Selection By Bradley Efron Discussant: Lawrence D Brown* Statistics Department, Wharton, Univ. of Penn. lbrown@wharton.upenn.edu JSM, Boston, Aug. 4, 2014
Variable Selection Insurance aka Valid Statistical Inference After Model Selection
Variable Selection Insurance aka Valid Statistical Inference After Model Selection L. Brown Wharton School, Univ. of Pennsylvania with Richard Berk, Andreas Buja, Ed George, Michael Traskin, Linda Zhao,
Discussion of Bootstrap prediction intervals for linear, nonlinear, and nonparametric autoregressions, by Li Pan and Dimitris Politis
Discussion of Bootstrap prediction intervals for linear, nonlinear, and nonparametric autoregressions, by Li Pan and Dimitris Politis Sílvia Gonçalves and Benoit Perron Département de sciences économiques,
Regression Models - Introduction
Regression Models - Introduction In regression models there are two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent
Outline of GLMs. Definitions
Outline of GLMs Definitions This is a short outline of GLM details, adapted from the book Nonparametric Regression and Generalized Linear Models, by Green and Silverman. The responses Y i have density
A Bootstrap Lasso + Partial Ridge Method to Construct Confidence Intervals for Parameters in High-dimensional Sparse Linear Models
A Bootstrap Lasso + Partial Ridge Method to Construct Confidence Intervals for Parameters in High-dimensional Sparse Linear Models Jingyi Jessica Li Department of Statistics University of California, Los
Bootstrap, Jackknife and other resampling methods
Bootstrap, Jackknife and other resampling methods Part VI: Cross-validation Rozenn Dahyot Room 128, Department of Statistics Trinity College Dublin, Ireland dahyot@mee.tcd.ie 2005 R. Dahyot (TCD) 453 Modern
Model-free prediction intervals for regression and autoregression. Dimitris N. Politis University of California, San Diego
Model-free prediction intervals for regression and autoregression Dimitris N. Politis University of California, San Diego To explain or to predict? Models are indispensable for exploring/utilizing relationships
Regression with a Single Regressor: Hypothesis Tests and Confidence Intervals
Regression with a Single Regressor: Hypothesis Tests and Confidence Intervals (SW Chapter 5) Outline. The standard error of ˆ. Hypothesis tests concerning β 3. Confidence intervals for β 4. Regression
Formal Statement of Simple Linear Regression Model
Formal Statement of Simple Linear Regression Model Y i = β 0 + β 1 X i + ɛ i Y i value of the response variable in the i th trial β 0 and β 1 are parameters X i is a known constant, the value of the predictor
Multiple Regression Analysis
Multiple Regression Analysis y = 0 + 1 x 1 + x +... k x k + u 6. Heteroskedasticity What is Heteroskedasticity?! Recall the assumption of homoskedasticity implied that conditional on the explanatory variables,
Bootstrap & Confidence/Prediction intervals
Bootstrap & Confidence/Prediction intervals Olivier Roustant Mines Saint-Étienne 2017/11 Olivier Roustant (EMSE) Bootstrap & Confidence/Prediction intervals 2017/11 1 / 9 Framework Consider a model with
Simple and Multiple Linear Regression
Sta. 113 Chapter 12 and 13 of Devore March 12, 2010 Table of contents 1 Simple Linear Regression 2 Model Simple Linear Regression A simple linear regression model is given by Y = β 0 + β 1 x + ɛ where
Central Bank of Chile October 29-31, 2013 Bruce Hansen (University of Wisconsin) Structural Breaks October 29-31, / 91. Bruce E.
Forecasting Lecture 3 Structural Breaks Central Bank of Chile October 29-31, 2013 Bruce Hansen (University of Wisconsin) Structural Breaks October 29-31, 2013 1 / 91 Bruce E. Hansen Organization Detection
Inference For High Dimensional M-estimates. Fixed Design Results
: Fixed Design Results Lihua Lei Advisors: Peter J. Bickel, Michael I. Jordan joint work with Peter J. Bickel and Noureddine El Karoui Dec. 8, 2016 1/57 Table of Contents 1 Background 2 Main Results and
Lectures on Simple Linear Regression Stat 431, Summer 2012
Lectures on Simple Linear Regression Stat 43, Summer 0 Hyunseung Kang July 6-8, 0 Last Updated: July 8, 0 :59PM Introduction Previously, we have been investigating various properties of the population
Econometrics I KS. Module 1: Bivariate Linear Regression. Alexander Ahammer. This version: March 12, 2018
Econometrics I KS Module 1: Bivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: March 12, 2018 Alexander Ahammer (JKU) Module 1: Bivariate
Post-Selection Inference
Classical Inference start end start Post-Selection Inference selected end model data inference data selection model data inference Post-Selection Inference Todd Kuffner Washington University in St. Louis
Econometrics I KS. Module 2: Multivariate Linear Regression. Alexander Ahammer. This version: April 16, 2018
Econometrics I KS Module 2: Multivariate Linear Regression Alexander Ahammer Department of Economics Johannes Kepler University of Linz This version: April 16, 2018 Alexander Ahammer (JKU) Module 2: Multivariate
Review of Econometrics
Review of Econometrics Zheng Tian June 5th, 2017 1 The Essence of the OLS Estimation Multiple regression model involves the models as follows Y i = β 0 + β 1 X 1i + β 2 X 2i + + β k X ki + u i, i = 1,...,
Model Selection, Estimation, and Bootstrap Smoothing. Bradley Efron Stanford University
Model Selection, Estimation, and Bootstrap Smoothing Bradley Efron Stanford University Estimation After Model Selection Usually: (a) look at data (b) choose model (linear, quad, cubic...?) (c) fit estimates
Linear models and their mathematical foundations: Simple linear regression
Linear models and their mathematical foundations: Simple linear regression Steffen Unkel Department of Medical Statistics University Medical Center Göttingen, Germany Winter term 2018/19 1/21 Introduction
Regression Models - Introduction
Regression Models - Introduction In regression models, two types of variables that are studied: A dependent variable, Y, also called response variable. It is modeled as random. An independent variable,
Marginal Screening and Post-Selection Inference
Marginal Screening and Post-Selection Inference Ian McKeague August 13, 2017 Ian McKeague (Columbia University) Marginal Screening August 13, 2017 1 / 29 Outline 1 Background on Marginal Screening 2 2
STAT420 Midterm Exam. University of Illinois Urbana-Champaign October 19 (Friday), :00 4:15p. SOLUTIONS (Yellow)
STAT40 Midterm Exam University of Illinois Urbana-Champaign October 19 (Friday), 018 3:00 4:15p SOLUTIONS (Yellow) Question 1 (15 points) (10 points) 3 (50 points) extra ( points) Total (77 points) Points
Inference For High Dimensional M-estimates: Fixed Design Results
Inference For High Dimensional M-estimates: Fixed Design Results Lihua Lei, Peter Bickel and Noureddine El Karoui Department of Statistics, UC Berkeley Berkeley-Stanford Econometrics Jamboree, 2017 1/49
Recent Advances in the Field of Trade Theory and Policy Analysis Using Micro-Level Data
Recent Advances in the Field of Trade Theory and Policy Analysis Using Micro-Level Data July 2012 Bangkok, Thailand Cosimo Beverelli (World Trade Organization) 1 Content a) Classical regression model b)
Ch 2: Simple Linear Regression
Ch 2: Simple Linear Regression 1. Simple Linear Regression Model A simple regression model with a single regressor x is y = β 0 + β 1 x + ɛ, where we assume that the error ɛ is independent random component
Lecture 10 Multiple Linear Regression
Lecture 10 Multiple Linear Regression STAT 512 Spring 2011 Background Reading KNNL: 6.1-6.5 10-1 Topic Overview Multiple Linear Regression Model 10-2 Data for Multiple Regression Y i is the response variable
Regression Diagnostics for Survey Data
Regression Diagnostics for Survey Data Richard Valliant Joint Program in Survey Methodology, University of Maryland and University of Michigan USA Jianzhu Li (Westat), Dan Liao (JPSM) 1 Introduction Topics
Graduate Econometrics Lecture 4: Heteroskedasticity
Graduate Econometrics Lecture 4: Heteroskedasticity Department of Economics University of Gothenburg November 30, 2014 1/43 and Autocorrelation Consequences for OLS Estimator Begin from the linear model
Confidence Intervals for Low-dimensional Parameters with High-dimensional Data
Confidence Intervals for Low-dimensional Parameters with High-dimensional Data Cun-Hui Zhang and Stephanie S. Zhang Rutgers University and Columbia University September 14, 2012 Outline Introduction Methodology
Cross-Validation with Confidence
Cross-Validation with Confidence Jing Lei Department of Statistics, Carnegie Mellon University WHOA-PSI Workshop, St Louis, 2017 Quotes from Day 1 and Day 2 Good model or pure model? Occam s razor We really
Bootstrapping Heteroskedasticity Consistent Covariance Matrix Estimator
Bootstrapping Heteroskedasticity Consistent Covariance Matrix Estimator by Emmanuel Flachaire Eurequa, University Paris I Panthéon-Sorbonne December 2001 Abstract Recent results of Cribari-Neto and Zarkos
Selective Inference for Effect Modification
Inference for Modification (Joint work with Dylan Small and Ashkan Ertefaie) Department of Statistics, University of Pennsylvania May 24, ACIC 2017 Manuscript and slides are available at http://www-stat.wharton.upenn.edu/~qyzhao/.
Robustness to Parametric Assumptions in Missing Data Models
Robustness to Parametric Assumptions in Missing Data Models Bryan Graham NYU Keisuke Hirano University of Arizona April 2011 Motivation Motivation We consider the classic missing data problem. In practice
ECON3150/4150 Spring 2015
ECON3150/4150 Spring 2015 Lecture 3&4 - The linear regression model Siv-Elisabeth Skjelbred University of Oslo January 29, 2015 1 / 67 Chapter 4 in S&W Section 17.1 in S&W (extended OLS assumptions) 2
1. The OLS Estimator. 1.1 Population model and notation
1. The OLS Estimator OLS stands for Ordinary Least Squares. There are 6 assumptions ordinarily made, and the method of fitting a line through data is by least-squares. OLS is a common estimation methodology
Chapter 8 Heteroskedasticity
Chapter 8 Walter R. Paczkowski Rutgers University Page 1 Chapter Contents 8.1 The Nature of 8. Detecting 8.3 -Consistent Standard Errors 8.4 Generalized Least Squares: Known Form of Variance 8.5 Generalized
Multiple Regression Analysis: Heteroskedasticity
Multiple Regression Analysis: Heteroskedasticity y = β 0 + β 1 x 1 + β x +... β k x k + u Read chapter 8. EE45 -Chaiyuth Punyasavatsut 1 topics 8.1 Heteroskedasticity and OLS 8. Robust estimation 8.3 Testing
University of California, Berkeley
University of California, Berkeley U.C. Berkeley Division of Biostatistics Working Paper Series Year 2009 Paper 251 Nonparametric population average models: deriving the form of approximate population
The Simple Linear Regression Model
The Simple Linear Regression Model Lesson 3 Ryan Safner 1 1 Department of Economics Hood College ECON 480 - Econometrics Fall 2017 Ryan Safner (Hood College) ECON 480 - Lesson 3 Fall 2017 1 / 77 Bivariate
Econometrics - 30C00200
Econometrics - 30C00200 Lecture 11: Heteroskedasticity Antti Saastamoinen VATT Institute for Economic Research Fall 2015 30C00200 Lecture 11: Heteroskedasticity 12.10.2015 Aalto University School of Business
Module 17: Bayesian Statistics for Genetics Lecture 4: Linear regression
1/37 The linear regression model Module 17: Bayesian Statistics for Genetics Lecture 4: Linear regression Ken Rice Department of Biostatistics University of Washington 2/37 The linear regression model
Econometrics Multiple Regression Analysis: Heteroskedasticity
Econometrics Multiple Regression Analysis: João Valle e Azevedo Faculdade de Economia Universidade Nova de Lisboa Spring Semester João Valle e Azevedo (FEUNL) Econometrics Lisbon, April 2011 1 / 19 Properties
Quantile methods. Class Notes Manuel Arellano December 1, Let F (r) =Pr(Y r). Forτ (0, 1), theτth population quantile of Y is defined to be
Quantile methods Class Notes Manuel Arellano December 1, 2009 1 Unconditional quantiles Let F (r) =Pr(Y r). Forτ (0, 1), theτth population quantile of Y is defined to be Q τ (Y ) q τ F 1 (τ) =inf{r : F
Homoskedasticity. Var (u X) = σ 2. (23)
Homoskedasticity How big is the difference between the OLS estimator and the true parameter? To answer this question, we make an additional assumption called homoskedasticity: Var (u X) = σ 2. (23) This
A Course in Applied Econometrics Lecture 14: Control Functions and Related Methods. Jeff Wooldridge IRP Lectures, UW Madison, August 2008
A Course in Applied Econometrics Lecture 14: Control Functions and Related Methods Jeff Wooldridge IRP Lectures, UW Madison, August 2008 1. Linear-in-Parameters Models: IV versus Control Functions 2. Correlated
Introduction to Econometrics
Introduction to Econometrics Lecture 3 : Regression: CEF and Simple OLS Zhaopeng Qu Business School,Nanjing University Oct 9th, 2017 Zhaopeng Qu (Nanjing University) Introduction to Econometrics Oct 9th,
Statistical Inference
Statistical Inference Liu Yang Florida State University October 27, 2016 Liu Yang, Libo Wang (Florida State University) Statistical Inference October 27, 2016 1 / 27 Outline The Bayesian Lasso Trevor Park
A Practitioner s Guide to Cluster-Robust Inference
A Practitioner s Guide to Cluster-Robust Inference A. C. Cameron and D. L. Miller presented by Federico Curci March 4, 2015 Cameron Miller Cluster Clinic II March 4, 2015 1 / 20 In the previous episode
Measurement error effects on bias and variance in two-stage regression, with application to air pollution epidemiology
Measurement error effects on bias and variance in two-stage regression, with application to air pollution epidemiology Chris Paciorek Department of Statistics, University of California, Berkeley and Adam
Nonlinear Regression Functions
Nonlinear Regression Functions (SW Chapter 8) Outline 1. Nonlinear regression functions general comments 2. Nonlinear functions of one variable 3. Nonlinear functions of two variables: interactions 4.
11. Bootstrap Methods
11. Bootstrap Methods c A. Colin Cameron & Pravin K. Trivedi 2006 These transparencies were prepared in 20043. They can be used as an adjunct to Chapter 11 of our subsequent book Microeconometrics: Methods
Lecture 5: LDA and Logistic Regression
Lecture 5: and Logistic Regression Hao Helen Zhang Hao Helen Zhang Lecture 5: and Logistic Regression 1 / 39 Outline Linear Classification Methods Two Popular Linear Models for Classification Linear Discriminant
Simple Linear Regression for the MPG Data
Simple Linear Regression for the MPG Data 2000 2500 3000 3500 15 20 25 30 35 40 45 Wgt MPG What do we do with the data? y i = MPG of i th car x i = Weight of i th car i =1,...,n n = Sample Size Exploratory
The consequences of misspecifying the random effects distribution when fitting generalized linear mixed models
The consequences of misspecifying the random effects distribution when fitting generalized linear mixed models John M. Neuhaus Charles E. McCulloch Division of Biostatistics University of California, San
Inference in Regression Analysis
Inference in Regression Analysis Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 4, Slide 1 Today: Normal Error Regression Model Y i = β 0 + β 1 X i + ǫ i Y i value
LECTURE 2 LINEAR REGRESSION MODEL AND OLS
SEPTEMBER 29, 2014 LECTURE 2 LINEAR REGRESSION MODEL AND OLS Definitions A common question in econometrics is to study the effect of one group of variables X i, usually called the regressors, on another
Stat 579: Generalized Linear Models and Extensions
Stat 579: Generalized Linear Models and Extensions Linear Mixed Models for Longitudinal Data Yan Lu April, 2018, week 15 1 / 38 Data structure t1 t2 tn i 1st subject y 11 y 12 y 1n1 Experimental 2nd subject
Measurement error as missing data: the case of epidemiologic assays. Roderick J. Little
Measurement error as missing data: the case of epidemiologic assays Roderick J. Little Outline Discuss two related calibration topics where classical methods are deficient (A) Limit of quantification methods
ECON The Simple Regression Model
ECON 351 - The Simple Regression Model Maggie Jones 1 / 41 The Simple Regression Model Our starting point will be the simple regression model where we look at the relationship between two variables In
A Primer on Asymptotics
A Primer on Asymptotics Eric Zivot Department of Economics University of Washington September 30, 2003 Revised: October 7, 2009 Introduction The two main concepts in asymptotic theory covered in these
Contents. 1 Review of Residuals. 2 Detecting Outliers. 3 Influential Observations. 4 Multicollinearity and its Effects
Contents 1 Review of Residuals 2 Detecting Outliers 3 Influential Observations 4 Multicollinearity and its Effects W. Zhou (Colorado State University) STAT 540 July 6th, 2015 1 / 32 Model Diagnostics:
i=1 h n (ˆθ n ) = 0. (2)
Stat 8112 Lecture Notes Unbiased Estimating Equations Charles J. Geyer April 29, 2012 1 Introduction In this handout we generalize the notion of maximum likelihood estimation to solution of unbiased estimating
Statistics - Lecture One. Outline. Charlotte Wickham 1. Basic ideas about estimation
Statistics - Lecture One Charlotte Wickham wickham@stat.berkeley.edu http://www.stat.berkeley.edu/~wickham/ Outline 1. Basic ideas about estimation 2. Method of Moments 3. Maximum Likelihood 4. Confidence
Unit 10: Simple Linear Regression and Correlation
Unit 10: Simple Linear Regression and Correlation Statistics 571: Statistical Methods Ramón V. León 6/28/2004 Unit 10 - Stat 571 - Ramón V. León 1 Introductory Remarks Regression analysis is a method for
Mostly Dangerous Econometrics: How to do Model Selection with Inference in Mind
Outline Introduction Analysis in Low Dimensional Settings Analysis in High-Dimensional Settings Bonus Track: Genaralizations Econometrics: How to do Model Selection with Inference in Mind June 25, 2015,
Simple Linear Regression
Simple Linear Regression In simple linear regression we are concerned about the relationship between two variables, X and Y. There are two components to such a relationship. 1. The strength of the relationship.
Introductory Econometrics
Based on the textbook by Wooldridge: : A Modern Approach Robert M. Kunst robert.kunst@univie.ac.at University of Vienna and Institute for Advanced Studies Vienna December 11, 2012 Outline Heteroskedasticity
statistical sense, from the distributions of the xs. The model may now be generalized to the case of k regressors:
Wooldridge, Introductory Econometrics, d ed. Chapter 3: Multiple regression analysis: Estimation In multiple regression analysis, we extend the simple (two-variable) regression model to consider the possibility
Why experimenters should not randomize, and what they should do instead
Why experimenters should not randomize, and what they should do instead Maximilian Kasy Department of Economics, Harvard University Maximilian Kasy (Harvard) Experimental design 1 / 42 project STAR Introduction
Distribution-Free Predictive Inference for Regression
Distribution-Free Predictive Inference for Regression Jing Lei, Max G Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman Department of Statistics, Carnegie Mellon University Abstract We
ECON Introductory Econometrics. Lecture 5: OLS with One Regressor: Hypothesis Tests
ECON4150 - Introductory Econometrics Lecture 5: OLS with One Regressor: Hypothesis Tests Monique de Haan (moniqued@econ.uio.no) Stock and Watson Chapter 5 Lecture outline 2 Testing Hypotheses about one
MS&E 226: Small Data
MS&E 226: Small Data Lecture 12: Frequentist properties of estimators (v4) Ramesh Johari ramesh.johari@stanford.edu 1 / 39 Frequentist inference 2 / 39 Thinking like a frequentist Suppose that for some
Cross-Validation with Confidence
Cross-Validation with Confidence Jing Lei Department of Statistics, Carnegie Mellon University UMN Statistics Seminar, Mar 30, 2017 Overview Parameter est. Model selection Point est. MLE, M-est.,... Cross-validation
Introductory Econometrics
Based on the textbook by Wooldridge: : A Modern Approach Robert M. Kunst robert.kunst@univie.ac.at University of Vienna and Institute for Advanced Studies Vienna December 17, 2012 Outline Heteroskedasticity
MA 575 Linear Models: Cedric E. Ginestet, Boston University Bootstrap for Regression Week 9, Lecture 1
MA 575 Linear Models: Cedric E. Ginestet, Boston University Bootstrap for Regression Week 9, Lecture 1 1 The General Bootstrap This is a computer-intensive resampling algorithm for estimating the empirical
multilevel modeling: concepts, applications and interpretations
multilevel modeling: concepts, applications and interpretations lynne c. messer 27 october 2010 warning social and reproductive / perinatal epidemiologist concepts why context matters multilevel models
Note on Bivariate Regression: Connecting Practice and Theory. Konstantin Kashin
Note on Bivariate Regression: Connecting Practice and Theory Konstantin Kashin Fall 2012 1 This note will explain - in less theoretical terms - the basics of a bivariate linear regression, including testing
1 Introduction to Generalized Least Squares
ECONOMICS 7344, Spring 2017 Bent E. Sørensen April 12, 2017 1 Introduction to Generalized Least Squares Consider the model Y = Xβ + ɛ, where the N K matrix of regressors X is fixed, independent of the
Nonstationary time series models
13 November, 2009 Goals Trends in economic data. Alternative models of time series trends: deterministic trend, and stochastic trend. Comparison of deterministic and stochastic trend models The statistical
Economics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
Bias Variance Trade-off
Bias Variance Trade-off The mean squared error of an estimator MSE(ˆθ) = E([ˆθ θ] 2 ) Can be re-expressed MSE(ˆθ) = Var(ˆθ) + (B(ˆθ) 2 ) MSE = VAR + BIAS 2 Proof MSE(ˆθ) = E((ˆθ θ) 2 ) = E(([ˆθ E(ˆθ)]