Econ 582 Nonparametric Regression
|
|
- Derrick O’Brien’
- 5 years ago
- Views:
Transcription
1 Econ 582 Nonparametric Regression Eric Zivot May 28, 2013
2 Nonparametric Regression Sofarwehaveonlyconsideredlinearregressionmodels = x 0 β + [ x ]=0 [ x = x] =x 0 β = [ x = x] [ x = x] x = β The assume that [ x = x] =x 0 β is a linear function of is often done for convenience. In general, when the components of are continuously distributed then can take on any nonlinear shape. [ x = x] = (x)
3 Two cases to consider If [ x = x] = (x) = (x θ) for θ R then we have a parametric nonlinear regression model = (x θ)+ and the parameters θ and be estimated using nonlinear regression techniques If [ x = x] = (x) cannot be modeled parametrically or the parametric form (x θ) is unknown then we have a non-parametric regression = (x)+ and we can estimate the function (x) at each point x using nonparametric regression techniques.
4 Binned Estimation of (x) Consider a nonparametric regression with a single covariate = ( )+ Fix the point = and consider estimating ( ) using a local average of values associated values near such that ˆ ( ) = P =1 1( ) P =1 1( ) 1( ) = 1 if ;0 otherwise ( ) = 1( ) P =1 1( ) Note, P =1 ( )=1 = X =1 ( )
5 Example: Nonparametric regression (Hansen) Thetruemodelis = ( )+ =1 100 ( ) = 10 log( ) (4 1) (0 16) ( ( ) 16) Forbinnedestimationlet = and =0 5
6 Remarks: Binned estimator is a step-function (discontinuous estimate of ( )) Forcoarsegridof the steps (squares in figure) are large For a fine grid of the steps (solid line in figure) are smaller The bandwidth determines the smoothness of the estimate: Small gives small bins and less smoothness
7 Figure 1: Binned estimator at = with =1 2 and NW estimator with Epanechnikov kernel
8 Kernel Regression Binned estimator is discontinuous because ( ) is constructed from indicator functions which are discontinuous If ( ) is constructed from a continuous function then ˆ ( ) will also be continuous. Kernel estimators of ( ) are continuous estimators based on continuous kernel weight functions
9 Example: Kernel weight function based on uniform distribution Define the weights 1( ) in terms of the uniform density on [ 1 1] 0 ( ) = 1 1( 1) = rectangular kernel 2 Then and µµ 1( ) =1 ˆ ( ) = 1 P ³ =1 0 P ³ =1 0 µ =2 0
10 Definition: A second-order kernel function ( ) satisfies 0 ( ) ( ) = ( ) R ( ) =1 2 = R 2 ( )
11 Kernel Estimator Given a kernel weight function ( ) akernelestimatorof ( ) has the form where ˆ ( ) = = ( )= P =1 ³ P =1 ³ X =1 ( ) ³ P =1 ³ Note: The kernel estimator is also known as the Nadaraya-Watson estimator, the kernel regression estimator or the local constant estimator.
12 Role of Bandwidth Parameter Bandwidth determines smoothness of estimator: large gives smoother ˆ ( ); smaller gives rougher (more erratic) ˆ ( ) 0 ˆ ( ) ˆ ( )
13 Commonly used Kernels 1. Epanechnikov kernel 1 ( ) = 3 4 (1 2 )1 ( 1) 2. Gaussian kernel 4 ( ) = 1 Ã! exp Two important properties of kernels 2 Z = = 2 ( ) Z ( )2
14 Properties of Commonly Used Kernels Kernel Equation 2 Uniform 0 ( ) = 1 21( 1) 1/2 1/3 Epanechnikov 1 ( ) = 3 4 (1 2 )1 ( 1) 3/5 1/5 Biweight 2 ( ) = (1 2 ) 2 1( 1) 5/7 1/7 Triweight 3 ( ) = (1 2 µ ) 3 1( 1) 350/429 1/9 Gaussian 4 ( ) = 1 exp 1/(2 )
15 Local Linear Estimator The Nadaraya-Watson (NW) kernel estimator is often called a local constant estimator as it locally (about x) approximates ( ) as a constant function. In fact, the NW estimator solves the minimization problem ˆ ( ) =argmin X =1 µ ( ) 2 This is a weighted regression of on an intercept only.
16 A local linear approximation solves the minimization problem nˆ ( ) ˆ ( ) o =argmin X =1 µ ( ( )) 2 The local linear estimator of ( ) is the estimated intercept ˆ ( ) =ˆ ( ) The local linear estimator of the regression derivative ( ) is the estimated slope coefficient d ( ) =ˆ ( )
17 Matrix notation Define z = à 1! µ = Then the LL estimator is the weighted LS estimator à ˆ ( ) ˆ ( )! = X =1 ( )z ( )z ( ) 0 = (Z 0 KZ) 1 Z 0 Ky 1 X =1 ( )z ( ) where K ( ) = 1 ( )... ( )
18 Remarks ( ) ˆ + ˆ because ( ) 1 NW does better than LL when ( ) is close to a flat line LL does better than NW when ( ) is meaningfully nonconstant LL does better than NW for values near the boundary of support of
19 Figure 2: Local linear estimator.
20 Nonparametric Residuals and Regression Fit Define the nonparametric residual as ˆ = ˆ ( ) Problem: ˆ is not a good error measure for small because ˆ ( ) as 0 and so ˆ 0 as 0 Need a residual that does not suffer from this overfitting problem
21 Leave-one-out (Jacknife) Residuals (NW Estimator) Idea: For the NW estimator, we can prevent ˆ ( ) as 0 by leaving out and from the non-parametric fit ˆ ( ) = P 6= ³ P 6= ³ The leave-one-out (Jacknife) NW predictor and residual for observation are = ˆ ( ) =
22 Leave-one-out (Jacknife) Residuals (LL Estimator) The Jacknife LL estimator has the form Ã! = X 6= z z 0 1 z = µ = and the LL residual is à 1 X 6=! = z
23 Cross Validation and Bandwidth Selection = ( )+ ( )= 2 for all ˆ ( ) = nonparametric estimate of ( ) Problem: How to choose? large smoother estimator (smaller variance of ˆ ( )) but higher bias at each small noiser estimator (higher variance of ˆ ( )) but lower bias at each (recall, ˆ ( ) as 0) Key point: Desirable to choose to minimize the bias-variance tradeoff
24 MSE, IMSE and MSFE The mean-squared error (MSE) at is defined as ( ) = h (ˆ ( ) ( )) 2i = (ˆ ( ) ( )) 2 + (ˆ ( )) and is a function of both and The integrated MSE a weighted average MSE over all is ( ) = ( ) = pdf of and is only a function of Z ( ) ( ) = [ ( )] Goal: Find to minimize ( )
25 Problem: ( ) depends on ( ) which is unknown Result: ( ) can be estimated using the sample mean-squared forecast error (MSFE) Let ( ) be out-of-sample observations independent of the sample. The prediction of +1 given +1 is The MSFE is defined as ˆ +1 =ˆ ( +1 ) ( ) = h ( +1 ˆ +1 ) 2i = h ( +1 ˆ ( +1 )) 2i
26 Using the trivial identity +1 ˆ ( +1 ) = +1 ( +1 )+ ( +1 ) ˆ ( +1 ) = +1 + ( +1 ) ˆ ( +1 ) It can be shown that ( ) = h ( +1 + ( +1 ) ˆ ( +1 )) 2i = 2 + Z ( ) ( ) = 2 + ( ) Hence, minimizing ( ) is equivalent to minimizing ( )
27 Estimating ( ) Using the Jacknife nonparametric residuals ( ) = ( ) an estimate of ( ) is \ ( ) = 1 X =1 ( ) 2 Treated as a function of \ ( ) is called the cross-validation criterion ( ) = 1 X =1 ( ) 2
28 Optimal Bandwidth Estimation The bandwidth that minimizes an estimate of the IMSE solves ˆ = arg min 0 ( ) Notes: Typically, the univariate minimization is done by evaluating ( ) over a grid [ 1 2 ] and choosing ˆ as the value that gives the smallest ( ) over the grid. Plots of ( ) against provide a visual guide to choosing
29 Asymptotic Distribution Theory Theorem. Let ˆ ( ) denote either the NW or LL estimator of ( ). If is interior to the support of and ( ) 0 then as and 0 such that where (ˆ ( ) ( ) 2 2 ( )) ˆ ( ) Ã Ã ( )+ 2 2 ( ) 2 ( ) ( ) 2 ( ) = [ 2 = ] 2 = Z 2 ( ) = Z 0 2 ( ) ( )! ( )2!
30 Figure 3: Cross-validation criteria, NW and LL estimators.
31 Figure 4: NW and LL estimates using data-dependent CV bandwidths.
32 TheasymptoticbiastermsfortheNWandLLestimatorsare ( ) = ( )+ ( ) 1 0 ( ) 0 ( ) ( ) = 1 2 ( ) 00 ( )
33 Remarks: Asymptotic variances of NW and LL estimators are the same but biases differ ˆ ( ) converges at rate instead of the usual CLT rate of Because 0 as diverges slower than Hence, nonparametric estimators converge more slowly to their asymptotic distributions than parametric estimators ˆ ( ) hasanasymptoticbiasterm 2 2 ( ) which depends on 2 0 ( ) 00 ( ) and ( ) and 0 ( )
34 Asymptotic bias decreases in andasymptoticvarianceincreasesin ( ) depends on both 0 ( ) and 0 ( ) whereas ( ) only depends on 00 ( ) ( ) = ( ) =0if ( ) is constant (i.e., 0 ( ) = 00 ( ) =0) ( ) is typically lower than ( )
35 Estimating Asymptotic Standard Errors The asymptotic distribution theory gives the result (ˆ ( )) = 2 ( ) ( ) The known quantities are and The unknown quantities are 2 ( ) = [ 2 = ] and ( ) An estimate of (ˆ ( )) uses estimates for 2 ( ) and ( ) \ (ˆ ( )) = ˆ 2 ( ) ˆ ( ) [ (ˆ ( )) = Question: How to estimate 2 ( ) and ( )? v u t ˆ 2 ( ) ˆ ( )
36 Nonparametric Estimation of 2 ( ) = [ 2 = ] and ( ) A nonparametric estimate of 2 ( ) has the form where is the Jackknife residual. 2 ( ) =P =1 ( ) 2 P =1 ( ) A nonparametric estimate of ( ) has the form ˆ ( ) = 1 X =1 µ
37 Extension to Multiple Regression = [ x = x]+ [ x = x] = (x)+ x = ( 1 ) 0 For any vector x and observation define the kernel weights and bandwidth vector Ã! Ã! Ã! 1 (x) = h = ( 1 ) 0 2
38 Nonparametric Estimators Multivariate NW estimator: ˆ (x) = P =1 (x) P =1 (x) Multivariate LL estimator: Ã ˆ (x) ˆ (x) z =! = Ã 1 x x X =1! (x)z (x)z (x) 0 = (Z 0 KZ) 1 Z 0 Ky 1 X =1 (x)z (x)
39 Remarks Finding the cross-validation bandwidth vector ĥ =argmin (h) h is a cumbersome numerical problem if is large Asymptotic distribution theory is similar to univariate case with one important difference: convergence rate to asymptotic normal distribution depends on the dimension of x The higher is thesloweristheconvergence rate. This is called the curse of dimensionality and is a major problem in nonparametric regression.
Time Series and Forecasting Lecture 4 NonLinear Time Series
Time Series and Forecasting Lecture 4 NonLinear Time Series Bruce E. Hansen Summer School in Economics and Econometrics University of Crete July 23-27, 2012 Bruce Hansen (University of Wisconsin) Foundations
More informationNonparametric Methods
Nonparametric Methods Michael R. Roberts Department of Finance The Wharton School University of Pennsylvania July 28, 2009 Michael R. Roberts Nonparametric Methods 1/42 Overview Great for data analysis
More informationModelling Non-linear and Non-stationary Time Series
Modelling Non-linear and Non-stationary Time Series Chapter 2: Non-parametric methods Henrik Madsen Advanced Time Series Analysis September 206 Henrik Madsen (02427 Adv. TS Analysis) Lecture Notes September
More informationDensity estimation Nonparametric conditional mean estimation Semiparametric conditional mean estimation. Nonparametrics. Gabriel Montes-Rojas
0 0 5 Motivation: Regression discontinuity (Angrist&Pischke) Outcome.5 1 1.5 A. Linear E[Y 0i X i] 0.2.4.6.8 1 X Outcome.5 1 1.5 B. Nonlinear E[Y 0i X i] i 0.2.4.6.8 1 X utcome.5 1 1.5 C. Nonlinearity
More informationIntroduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones
Introduction to machine learning and pattern recognition Lecture 2 Coryn Bailer-Jones http://www.mpia.de/homes/calj/mlpr_mpia2008.html 1 1 Last week... supervised and unsupervised methods need adaptive
More informationNonparametric Density Estimation
Nonparametric Density Estimation Econ 690 Purdue University Justin L. Tobias (Purdue) Nonparametric Density Estimation 1 / 29 Density Estimation Suppose that you had some data, say on wages, and you wanted
More information41903: Introduction to Nonparametrics
41903: Notes 5 Introduction Nonparametrics fundamentally about fitting flexible models: want model that is flexible enough to accommodate important patterns but not so flexible it overspecializes to specific
More informationSingle Index Quantile Regression for Heteroscedastic Data
Single Index Quantile Regression for Heteroscedastic Data E. Christou M. G. Akritas Department of Statistics The Pennsylvania State University JSM, 2015 E. Christou, M. G. Akritas (PSU) SIQR JSM, 2015
More informationLocal Polynomial Modelling and Its Applications
Local Polynomial Modelling and Its Applications J. Fan Department of Statistics University of North Carolina Chapel Hill, USA and I. Gijbels Institute of Statistics Catholic University oflouvain Louvain-la-Neuve,
More informationIntroduction to Regression
Introduction to Regression p. 1/97 Introduction to Regression Chad Schafer cschafer@stat.cmu.edu Carnegie Mellon University Introduction to Regression p. 1/97 Acknowledgement Larry Wasserman, All of Nonparametric
More informationStatistics for Python
Statistics for Python An extension module for the Python scripting language Michiel de Hoon, Columbia University 2 September 2010 Statistics for Python, an extension module for the Python scripting language.
More informationPreface. 1 Nonparametric Density Estimation and Testing. 1.1 Introduction. 1.2 Univariate Density Estimation
Preface Nonparametric econometrics has become one of the most important sub-fields in modern econometrics. The primary goal of this lecture note is to introduce various nonparametric and semiparametric
More informationIntroduction to Maximum Likelihood Estimation
Introduction to Maximum Likelihood Estimation Eric Zivot July 26, 2012 The Likelihood Function Let 1 be an iid sample with pdf ( ; ) where is a ( 1) vector of parameters that characterize ( ; ) Example:
More informationIntroduction to Regression
Introduction to Regression Chad M. Schafer May 20, 2015 Outline General Concepts of Regression, Bias-Variance Tradeoff Linear Regression Nonparametric Procedures Cross Validation Local Polynomial Regression
More informationNonparametric Regression. Badr Missaoui
Badr Missaoui Outline Kernel and local polynomial regression. Penalized regression. We are given n pairs of observations (X 1, Y 1 ),...,(X n, Y n ) where Y i = r(x i ) + ε i, i = 1,..., n and r(x) = E(Y
More informationLocal Polynomial Regression
VI Local Polynomial Regression (1) Global polynomial regression We observe random pairs (X 1, Y 1 ),, (X n, Y n ) where (X 1, Y 1 ),, (X n, Y n ) iid (X, Y ). We want to estimate m(x) = E(Y X = x) based
More informationIntroduction to Regression
Introduction to Regression Chad M. Schafer cschafer@stat.cmu.edu Carnegie Mellon University Introduction to Regression p. 1/100 Outline General Concepts of Regression, Bias-Variance Tradeoff Linear Regression
More informationDensity Estimation (II)
Density Estimation (II) Yesterday Overview & Issues Histogram Kernel estimators Ideogram Today Further development of optimization Estimating variance and bias Adaptive kernels Multivariate kernel estimation
More informationWeek 5 Quantitative Analysis of Financial Markets Modeling and Forecasting Trend
Week 5 Quantitative Analysis of Financial Markets Modeling and Forecasting Trend Christopher Ting http://www.mysmu.edu/faculty/christophert/ Christopher Ting : christopherting@smu.edu.sg : 6828 0364 :
More informationSingle Index Quantile Regression for Heteroscedastic Data
Single Index Quantile Regression for Heteroscedastic Data E. Christou M. G. Akritas Department of Statistics The Pennsylvania State University SMAC, November 6, 2015 E. Christou, M. G. Akritas (PSU) SIQR
More informationIntroduction. Linear Regression. coefficient estimates for the wage equation: E(Y X) = X 1 β X d β d = X β
Introduction - Introduction -2 Introduction Linear Regression E(Y X) = X β +...+X d β d = X β Example: Wage equation Y = log wages, X = schooling (measured in years), labor market experience (measured
More informationEstimation of cumulative distribution function with spline functions
INTERNATIONAL JOURNAL OF ECONOMICS AND STATISTICS Volume 5, 017 Estimation of cumulative distribution function with functions Akhlitdin Nizamitdinov, Aladdin Shamilov Abstract The estimation of the cumulative
More information12 - Nonparametric Density Estimation
ST 697 Fall 2017 1/49 12 - Nonparametric Density Estimation ST 697 Fall 2017 University of Alabama Density Review ST 697 Fall 2017 2/49 Continuous Random Variables ST 697 Fall 2017 3/49 1.0 0.8 F(x) 0.6
More informationIntroduction to Regression
Introduction to Regression David E Jones (slides mostly by Chad M Schafer) June 1, 2016 1 / 102 Outline General Concepts of Regression, Bias-Variance Tradeoff Linear Regression Nonparametric Procedures
More informationLocal regression I. Patrick Breheny. November 1. Kernel weighted averages Local linear regression
Local regression I Patrick Breheny November 1 Patrick Breheny STA 621: Nonparametric Statistics 1/27 Simple local models Kernel weighted averages The Nadaraya-Watson estimator Expected loss and prediction
More informationNonparametric Econometrics
Applied Microeconometrics with Stata Nonparametric Econometrics Spring Term 2011 1 / 37 Contents Introduction The histogram estimator The kernel density estimator Nonparametric regression estimators Semi-
More informationPenalized Splines, Mixed Models, and Recent Large-Sample Results
Penalized Splines, Mixed Models, and Recent Large-Sample Results David Ruppert Operations Research & Information Engineering, Cornell University Feb 4, 2011 Collaborators Matt Wand, University of Wollongong
More information9. Model Selection. statistical models. overview of model selection. information criteria. goodness-of-fit measures
FE661 - Statistical Methods for Financial Engineering 9. Model Selection Jitkomut Songsiri statistical models overview of model selection information criteria goodness-of-fit measures 9-1 Statistical models
More informationPackage NonpModelCheck
Type Package Package NonpModelCheck April 27, 2017 Title Model Checking and Variable Selection in Nonparametric Regression Version 3.0 Date 2017-04-27 Author Adriano Zanin Zambom Maintainer Adriano Zanin
More informationAlternatives to Basis Expansions. Kernels in Density Estimation. Kernels and Bandwidth. Idea Behind Kernel Methods
Alternatives to Basis Expansions Basis expansions require either choice of a discrete set of basis or choice of smoothing penalty and smoothing parameter Both of which impose prior beliefs on data. Alternatives
More informationECON 721: Lecture Notes on Nonparametric Density and Regression Estimation. Petra E. Todd
ECON 721: Lecture Notes on Nonparametric Density and Regression Estimation Petra E. Todd Fall, 2014 2 Contents 1 Review of Stochastic Order Symbols 1 2 Nonparametric Density Estimation 3 2.1 Histogram
More informationSingle Equation Linear GMM with Serially Correlated Moment Conditions
Single Equation Linear GMM with Serially Correlated Moment Conditions Eric Zivot November 2, 2011 Univariate Time Series Let {y t } be an ergodic-stationary time series with E[y t ]=μ and var(y t )
More informationAdditive Isotonic Regression
Additive Isotonic Regression Enno Mammen and Kyusang Yu 11. July 2006 INTRODUCTION: We have i.i.d. random vectors (Y 1, X 1 ),..., (Y n, X n ) with X i = (X1 i,..., X d i ) and we consider the additive
More informationSingle Equation Linear GMM with Serially Correlated Moment Conditions
Single Equation Linear GMM with Serially Correlated Moment Conditions Eric Zivot October 28, 2009 Univariate Time Series Let {y t } be an ergodic-stationary time series with E[y t ]=μ and var(y t )
More informationMichael Lechner Causal Analysis RDD 2014 page 1. Lecture 7. The Regression Discontinuity Design. RDD fuzzy and sharp
page 1 Lecture 7 The Regression Discontinuity Design fuzzy and sharp page 2 Regression Discontinuity Design () Introduction (1) The design is a quasi-experimental design with the defining characteristic
More informationExtending clustered point process-based rainfall models to a non-stationary climate
Extending clustered point process-based rainfall models to a non-stationary climate Jo Kaczmarska 1, 2 Valerie Isham 2 Paul Northrop 2 1 Risk Management Solutions 2 Department of Statistical Science, University
More informationAUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET. Questions AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPINGS UNIVERSITET
The Problem Identification of Linear and onlinear Dynamical Systems Theme : Curve Fitting Division of Automatic Control Linköping University Sweden Data from Gripen Questions How do the control surface
More informationStat 5100 Handout #26: Variations on OLS Linear Regression (Ch. 11, 13)
Stat 5100 Handout #26: Variations on OLS Linear Regression (Ch. 11, 13) 1. Weighted Least Squares (textbook 11.1) Recall regression model Y = β 0 + β 1 X 1 +... + β p 1 X p 1 + ε in matrix form: (Ch. 5,
More informationMotivational Example
Motivational Example Data: Observational longitudinal study of obesity from birth to adulthood. Overall Goal: Build age-, gender-, height-specific growth charts (under 3 year) to diagnose growth abnomalities.
More informationNonparametric Regression. Changliang Zou
Nonparametric Regression Institute of Statistics, Nankai University Email: nk.chlzou@gmail.com Smoothing parameter selection An overall measure of how well m h (x) performs in estimating m(x) over x (0,
More informationLeast Squares Model Averaging. Bruce E. Hansen University of Wisconsin. January 2006 Revised: August 2006
Least Squares Model Averaging Bruce E. Hansen University of Wisconsin January 2006 Revised: August 2006 Introduction This paper developes a model averaging estimator for linear regression. Model averaging
More informationIntroduction to Statistical modeling: handout for Math 489/583
Introduction to Statistical modeling: handout for Math 489/583 Statistical modeling occurs when we are trying to model some data using statistical tools. From the start, we recognize that no model is perfect
More informationCSE446: non-parametric methods Spring 2017
CSE446: non-parametric methods Spring 2017 Ali Farhadi Slides adapted from Carlos Guestrin and Luke Zettlemoyer Linear Regression: What can go wrong? What do we do if the bias is too strong? Might want
More informationSection 7: Local linear regression (loess) and regression discontinuity designs
Section 7: Local linear regression (loess) and regression discontinuity designs Yotam Shem-Tov Fall 2015 Yotam Shem-Tov STAT 239/ PS 236A October 26, 2015 1 / 57 Motivation We will focus on local linear
More informationNonparametric Regression
Nonparametric Regression Econ 674 Purdue University April 8, 2009 Justin L. Tobias (Purdue) Nonparametric Regression April 8, 2009 1 / 31 Consider the univariate nonparametric regression model: where y
More informationLinear model selection and regularization
Linear model selection and regularization Problems with linear regression with least square 1. Prediction Accuracy: linear regression has low bias but suffer from high variance, especially when n p. It
More informationMaximum Likelihood Estimation. only training data is available to design a classifier
Introduction to Pattern Recognition [ Part 5 ] Mahdi Vasighi Introduction Bayesian Decision Theory shows that we could design an optimal classifier if we knew: P( i ) : priors p(x i ) : class-conditional
More informationNonparametric Regression Härdle, Müller, Sperlich, Werwarz, 1995, Nonparametric and Semiparametric Models, An Introduction
Härdle, Müller, Sperlich, Werwarz, 1995, Nonparametric and Semiparametric Models, An Introduction Tine Buch-Kromann Univariate Kernel Regression The relationship between two variables, X and Y where m(
More informationAGEC 661 Note Eleven Ximing Wu. Exponential regression model: m (x, θ) = exp (xθ) for y 0
AGEC 661 ote Eleven Ximing Wu M-estimator So far we ve focused on linear models, where the estimators have a closed form solution. If the population model is nonlinear, the estimators often do not have
More informationChapter 9. Non-Parametric Density Function Estimation
9-1 Density Estimation Version 1.2 Chapter 9 Non-Parametric Density Function Estimation 9.1. Introduction We have discussed several estimation techniques: method of moments, maximum likelihood, and least
More informationAsymptotic Multivariate Kriging Using Estimated Parameters with Bayesian Prediction Methods for Non-linear Predictands
Asymptotic Multivariate Kriging Using Estimated Parameters with Bayesian Prediction Methods for Non-linear Predictands Elizabeth C. Mannshardt-Shamseldin Advisor: Richard L. Smith Duke University Department
More informationMinimax Rate of Convergence for an Estimator of the Functional Component in a Semiparametric Multivariate Partially Linear Model.
Minimax Rate of Convergence for an Estimator of the Functional Component in a Semiparametric Multivariate Partially Linear Model By Michael Levine Purdue University Technical Report #14-03 Department of
More informationChap 1. Overview of Statistical Learning (HTF, , 2.9) Yongdai Kim Seoul National University
Chap 1. Overview of Statistical Learning (HTF, 2.1-2.6, 2.9) Yongdai Kim Seoul National University 0. Learning vs Statistical learning Learning procedure Construct a claim by observing data or using logics
More informationClassification via kernel regression based on univariate product density estimators
Classification via kernel regression based on univariate product density estimators Bezza Hafidi 1, Abdelkarim Merbouha 2, and Abdallah Mkhadri 1 1 Department of Mathematics, Cadi Ayyad University, BP
More informationLecture 3: Statistical Decision Theory (Part II)
Lecture 3: Statistical Decision Theory (Part II) Hao Helen Zhang Hao Helen Zhang Lecture 3: Statistical Decision Theory (Part II) 1 / 27 Outline of This Note Part I: Statistics Decision Theory (Classical
More informationIntroduction to Nonparametric Regression
Introduction to Nonparametric Regression Nathaniel E. Helwig Assistant Professor of Psychology and Statistics University of Minnesota (Twin Cities) Updated 04-Jan-2017 Nathaniel E. Helwig (U of Minnesota)
More information9/26/17. Ridge regression. What our model needs to do. Ridge Regression: L2 penalty. Ridge coefficients. Ridge coefficients
What our model needs to do regression Usually, we are not just trying to explain observed data We want to uncover meaningful trends And predict future observations Our questions then are Is β" a good estimate
More informationNon-parametric Methods
Non-parametric Methods Machine Learning Torsten Möller Möller/Mori 1 Reading Chapter 2 of Pattern Recognition and Machine Learning by Bishop (with an emphasis on section 2.5) Möller/Mori 2 Outline Last
More information7 Semiparametric Estimation of Additive Models
7 Semiparametric Estimation of Additive Models Additive models are very useful for approximating the high-dimensional regression mean functions. They and their extensions have become one of the most widely
More informationChapter 9. Non-Parametric Density Function Estimation
9-1 Density Estimation Version 1.1 Chapter 9 Non-Parametric Density Function Estimation 9.1. Introduction We have discussed several estimation techniques: method of moments, maximum likelihood, and least
More information1 Empirical Likelihood
Empirical Likelihood February 3, 2016 Debdeep Pati 1 Empirical Likelihood Empirical likelihood a nonparametric method without having to assume the form of the underlying distribution. It retains some of
More informationOFFICE OF NAVAL RESEARCH FINAL REPORT for TASK NO. NR PRINCIPAL INVESTIGATORS: Jeffrey D. Hart Thomas E. Wehrly
AD-A240 830 S ~September 1 10, 1991 OFFICE OF NAVAL RESEARCH FINAL REPORT for 1 OCTOBER 1985 THROUGH 31 AUGUST 1991 CONTRACT N00014-85-K-0723 TASK NO. NR 042-551 Nonparametric Estimation of Functions Based
More informationKernel density estimation
Kernel density estimation Patrick Breheny October 18 Patrick Breheny STA 621: Nonparametric Statistics 1/34 Introduction Kernel Density Estimation We ve looked at one method for estimating density: histograms
More informationNonparametric Estimation of Luminosity Functions
x x Nonparametric Estimation of Luminosity Functions Chad Schafer Department of Statistics, Carnegie Mellon University cschafer@stat.cmu.edu 1 Luminosity Functions The luminosity function gives the number
More informationIntensity Analysis of Spatial Point Patterns Geog 210C Introduction to Spatial Data Analysis
Intensity Analysis of Spatial Point Patterns Geog 210C Introduction to Spatial Data Analysis Chris Funk Lecture 4 Spatial Point Patterns Definition Set of point locations with recorded events" within study
More information6.435, System Identification
System Identification 6.435 SET 3 Nonparametric Identification Munther A. Dahleh 1 Nonparametric Methods for System ID Time domain methods Impulse response Step response Correlation analysis / time Frequency
More information3 Nonparametric Density Estimation
3 Nonparametric Density Estimation Example: Income distribution Source: U.K. Family Expenditure Survey (FES) 1968-1995 Approximately 7000 British Households per year For each household many different variables
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationEconomics 583: Econometric Theory I A Primer on Asymptotics
Economics 583: Econometric Theory I A Primer on Asymptotics Eric Zivot January 14, 2013 The two main concepts in asymptotic theory that we will use are Consistency Asymptotic Normality Intuition consistency:
More informationCOMPARISON OF GMM WITH SECOND-ORDER LEAST SQUARES ESTIMATION IN NONLINEAR MODELS. Abstract
Far East J. Theo. Stat. 0() (006), 179-196 COMPARISON OF GMM WITH SECOND-ORDER LEAST SQUARES ESTIMATION IN NONLINEAR MODELS Department of Statistics University of Manitoba Winnipeg, Manitoba, Canada R3T
More informationA Note on Data-Adaptive Bandwidth Selection for Sequential Kernel Smoothers
6th St.Petersburg Workshop on Simulation (2009) 1-3 A Note on Data-Adaptive Bandwidth Selection for Sequential Kernel Smoothers Ansgar Steland 1 Abstract Sequential kernel smoothers form a class of procedures
More informationModel-free prediction intervals for regression and autoregression. Dimitris N. Politis University of California, San Diego
Model-free prediction intervals for regression and autoregression Dimitris N. Politis University of California, San Diego To explain or to predict? Models are indispensable for exploring/utilizing relationships
More informationLecture 02 Linear classification methods I
Lecture 02 Linear classification methods I 22 January 2016 Taylor B. Arnold Yale Statistics STAT 365/665 1/32 Coursewebsite: A copy of the whole course syllabus, including a more detailed description of
More informationForecasting Lecture 2: Forecast Combination, Multi-Step Forecasts
Forecasting Lecture 2: Forecast Combination, Multi-Step Forecasts Bruce E. Hansen Central Bank of Chile October 29-31, 2013 Bruce Hansen (University of Wisconsin) Forecast Combination and Multi-Step Forecasts
More informationSpatial Regression. 6. Specification Spatial Heterogeneity. Luc Anselin.
Spatial Regression 6. Specification Spatial Heterogeneity Luc Anselin http://spatial.uchicago.edu 1 homogeneity and heterogeneity spatial regimes spatially varying coefficients spatial random effects 2
More informationHigh-dimensional regression with unknown variance
High-dimensional regression with unknown variance Christophe Giraud Ecole Polytechnique march 2012 Setting Gaussian regression with unknown variance: Y i = f i + ε i with ε i i.i.d. N (0, σ 2 ) f = (f
More informationQuantitative Economics for the Evaluation of the European Policy. Dipartimento di Economia e Management
Quantitative Economics for the Evaluation of the European Policy Dipartimento di Economia e Management Irene Brunetti 1 Davide Fiaschi 2 Angela Parenti 3 9 ottobre 2015 1 ireneb@ec.unipi.it. 2 davide.fiaschi@unipi.it.
More informationLWP. Locally Weighted Polynomials toolbox for Matlab/Octave
LWP Locally Weighted Polynomials toolbox for Matlab/Octave ver. 2.2 Gints Jekabsons http://www.cs.rtu.lv/jekabsons/ User's manual September, 2016 Copyright 2009-2016 Gints Jekabsons CONTENTS 1. INTRODUCTION...3
More informationESTIMATING AVERAGE TREATMENT EFFECTS: REGRESSION DISCONTINUITY DESIGNS Jeff Wooldridge Michigan State University BGSE/IZA Course in Microeconometrics
ESTIMATING AVERAGE TREATMENT EFFECTS: REGRESSION DISCONTINUITY DESIGNS Jeff Wooldridge Michigan State University BGSE/IZA Course in Microeconometrics July 2009 1. Introduction 2. The Sharp RD Design 3.
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationResampling techniques for statistical modeling
Resampling techniques for statistical modeling Gianluca Bontempi Département d Informatique Boulevard de Triomphe - CP 212 http://www.ulb.ac.be/di Resampling techniques p.1/33 Beyond the empirical error
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationNonparametric Methods Lecture 5
Nonparametric Methods Lecture 5 Jason Corso SUNY at Buffalo 17 Feb. 29 J. Corso (SUNY at Buffalo) Nonparametric Methods Lecture 5 17 Feb. 29 1 / 49 Nonparametric Methods Lecture 5 Overview Previously,
More informationSTAT 705 Nonlinear regression
STAT 705 Nonlinear regression Adapted from Timothy Hanson Department of Statistics, University of South Carolina Stat 705: Data Analysis II 1 / 1 Chapter 13 Parametric nonlinear regression Throughout most
More informationTitolo Smooth Backfitting with R
Rapporto n. 176 Titolo Smooth Backfitting with R Autori Alberto Arcagni, Luca Bagnato ottobre 2009 Dipartimento di Metodi Quantitativi per le Scienze Economiche ed Aziendali Università degli Studi di Milano
More informationCovariance function estimation in Gaussian process regression
Covariance function estimation in Gaussian process regression François Bachoc Department of Statistics and Operations Research, University of Vienna WU Research Seminar - May 2015 François Bachoc Gaussian
More informationNonparametric Inference via Bootstrapping the Debiased Estimator
Nonparametric Inference via Bootstrapping the Debiased Estimator Yen-Chi Chen Department of Statistics, University of Washington ICSA-Canada Chapter Symposium 2017 1 / 21 Problem Setup Let X 1,, X n be
More information13 Endogeneity and Nonparametric IV
13 Endogeneity and Nonparametric IV 13.1 Nonparametric Endogeneity A nonparametric IV equation is Y i = g (X i ) + e i (1) E (e i j i ) = 0 In this model, some elements of X i are potentially endogenous,
More informationA New Method for Varying Adaptive Bandwidth Selection
IEEE TRASACTIOS O SIGAL PROCESSIG, VOL. 47, O. 9, SEPTEMBER 1999 2567 TABLE I SQUARE ROOT MEA SQUARED ERRORS (SRMSE) OF ESTIMATIO USIG THE LPA AD VARIOUS WAVELET METHODS A ew Method for Varying Adaptive
More informationMachine Learning Lecture 3
Announcements Machine Learning Lecture 3 Eam dates We re in the process of fiing the first eam date Probability Density Estimation II 9.0.207 Eercises The first eercise sheet is available on L2P now First
More informationRegression I: Mean Squared Error and Measuring Quality of Fit
Regression I: Mean Squared Error and Measuring Quality of Fit -Applied Multivariate Analysis- Lecturer: Darren Homrighausen, PhD 1 The Setup Suppose there is a scientific problem we are interested in solving
More informationNonparametric Cointegrating Regression with Endogeneity and Long Memory
Nonparametric Cointegrating Regression with Endogeneity and Long Memory Qiying Wang School of Mathematics and Statistics TheUniversityofSydney Peter C. B. Phillips Yale University, University of Auckland
More informationLecture 24: Weighted and Generalized Least Squares
Lecture 24: Weighted and Generalized Least Squares 1 Weighted Least Squares When we use ordinary least squares to estimate linear regression, we minimize the mean squared error: MSE(b) = 1 n (Y i X i β)
More informationWEIGHTED QUANTILE REGRESSION THEORY AND ITS APPLICATION. Abstract
Journal of Data Science,17(1). P. 145-160,2019 DOI:10.6339/JDS.201901_17(1).0007 WEIGHTED QUANTILE REGRESSION THEORY AND ITS APPLICATION Wei Xiong *, Maozai Tian 2 1 School of Statistics, University of
More informationMachine Learning. Nonparametric Methods. Space of ML Problems. Todo. Histograms. Instance-Based Learning (aka non-parametric methods)
Machine Learning InstanceBased Learning (aka nonparametric methods) Supervised Learning Unsupervised Learning Reinforcement Learning Parametric Non parametric CSE 446 Machine Learning Daniel Weld March
More informationEstimation of Parameters
CHAPTER Probability, Statistics, and Reliability for Engineers and Scientists FUNDAMENTALS OF STATISTICAL ANALYSIS Second Edition A. J. Clark School of Engineering Department of Civil and Environmental
More informationF & B Approaches to a simple model
A6523 Signal Modeling, Statistical Inference and Data Mining in Astrophysics Spring 215 http://www.astro.cornell.edu/~cordes/a6523 Lecture 11 Applications: Model comparison Challenges in large-scale surveys
More informationOptimal Kernel Shapes for Local Linear Regression
Optimal Kernel Shapes for Local Linear Regression Dirk Ormoneit Trevor Hastie Department of Statistics Stanford University Stanford, CA 94305-4065 ormoneit@stat.stanjord.edu Abstract Local linear regression
More informationNonparametric Principal Components Regression
Int. Statistical Inst.: Proc. 58th World Statistical Congress, 2011, Dublin (Session CPS031) p.4574 Nonparametric Principal Components Regression Barrios, Erniel University of the Philippines Diliman,
More informationCurve Fitting Re-visited, Bishop1.2.5
Curve Fitting Re-visited, Bishop1.2.5 Maximum Likelihood Bishop 1.2.5 Model Likelihood differentiation p(t x, w, β) = Maximum Likelihood N N ( t n y(x n, w), β 1). (1.61) n=1 As we did in the case of the
More information