Statistical Inference Based on Extremum Estimators
|
|
- Isabella Sharyl Wiggins
- 5 years ago
- Views:
Transcription
1 T. Rotheberg Fall, 2007 Statistical Iferece Based o Extremum Estimators Itroductio Suppose 0, the true value of a p-dimesioal parameter, is kow to lie i some subset S R p : Ofte we choose to estimate 0 by miimizig (over 2 S) some objective fuctio Q Q(; y), where the radom vector y represets the data from a sample of size. Some examples are:. Liear least squares regressio, where Q = (y X) 0 (y X) ad X 0 is the coditioal expectatio of the radom vector y give the regressor matrix X: 2. Noliear least squares (NLS), where Q = [y g(; X)] 0 [y g(; X)]; g( 0 ; X) is the coditioal expectatio of y give X, ad g has a kow fuctioal form. 3. Least absolute deviatio (LAD) regressio where Q = P i jy i x 0 ij ad the coditioal media of y i give x i is equal to x 0 i 0 : 4. Maximum likelihood (ML), where Q is mius the log of the joit probability desity (or mass) fuctio of the data i a fully speci ed parametric model. 5. Geeralized method of momets (GMM), where Q = m(; y) 0 W m(; y); m(; y) is a q-dimesioal vector such that plim m( 0 ; y) = 0, ad W is a q q positive de ite matrix. 2 Asymptotic Properties of Extremum Estimators Clearly, ot every fuctio Q will lead to good estimates. We shall cosider fuctios that have the followig property: whe is large, Q(; y)= (viewed as a fuctio of ) is with high probability very close to a oradom fuctio Q () that has a proouced miimum at 0. The, i large samples, it seems plausible that ^, the miimizer of Q(; y), should with high probability be close to 0 ; the miimizer of Q (). This ituitio is made precise i the followig cosistecy theorem: Suppose Q(,y) is a cotiuous fuctio of i the compact parameter set SR p. The, if Q(,y) coverges i probability uiformly to a fuctio which has a uique miimum at 0, the value ^ that miimizes Q(,y) coverges i probability to 0. Regularity assumptios o the exogeous variables ad weak depedece across trials ofte imply that these coditios for cosistecy hold. EXAMPLE: I the liear model y = X 0 + u, suppose the u s are i.i.d. with mea zero, variace 2 ad idepedet of X. Assume further that X 0 X coverges i probability to a positive de ite matrix B. For the least-squares criterio fuctio, we have Q = (y X)0 (y X) = u0 u 2( 0 )X 0 u + ( 0 ) 0 X 0 X( 0 ) But u 0 u p! 2 ad X 0 u p! 0. Thus plim Q = ( 0 ) 0 B( 0 ) where the covergece is uiform i ay compact set of parameter values. Sice B is positive de ite, this limitig fuctio has a uique miimum at = 0.
2 Give cosistecy ad employig liearizatio methods, we ca ofte show that extremum estimators are approximately ormal whe the sample size is large. Suppose Q(; y) is twice di eretiable i with rst derivative vector S () ad cotiuous secod derivative matrix H (). (S is ofte called the score ad H the hessia for Q. Of course both will also deped o the sample data y, but this depedecy will be surpressed to simplify the otatio.) If ^ is a regular iterior miimum of Q, the S (^) = 0 ad H (^) is positive de ite. Usig the mea value theorem, we ca write 0 = =2 S (^) = =2 S ( 0 ) + H p (^ 0 ) where H is the hessia H with each elemet evaluated at a value somewhere betwee ^ ad 0. Suppose that =2 S ( 0 ) coverges i distributio to a ormal radom vector havig mea zero ad p p covariace matrix A. The, if H coverges i probability to a pp positive de ite matrix B, it follows that p (^ 0 ) has the same limitig distributio as B =2 S ( 0 ) ad hece p (^ 0 ) d! N(0; B AB ): () I large samples, oe might the act as though ^ were ormal with mea 0 ad variace matrix B b A b B b, where A b ad B b are cosistet estimates of A ad B. To prove that () is valid, we must show that =2 S ( 0 ) teds to a ormal radom variable ad that H teds to a oradom, full-rak matrix. I may problems S ( 0 ) is the sum of idepedet (or weakly depedet) mea-zero radom variables; stadard cetral limit theorems ca the be employed. Demostratig the covergece of H is usually more di cult sice we typically do ot have a closed form expressio for H: However, cotiuity argumets ad the law of large umbers ofte ca be employed to show that H coverges i probability to B = lim E H ( 0 ). Rigorous proofs of the asymptotic ormality of extremal estimators will ot be attempted here. The LAD case where Q is odi eretiable is particularly tricky sice the liearizatio o loger ca be obtaied by the mea value theorem. I cotrast, the OLS case where Q is quadratic is simple sice the H does ot deped o. 3 Computig Extremum Estimates I practice, the computatioal problem of dig the miimum of Q(; y) is otrivial. Computer itesive iterative methods are ofte successful. If Q(; y) is twice di eretiable i ; the quadratic Taylor series approximatio aroud a poit = a 0 is give by Q(; y) Q(a 0 ; y) + (a 0 ) 0 S (a0 ) 0 H 0 (a 0 ) where S 0 is the gradiet of Q ad H 0 is the matrix of secod derivatives, both evaluated at = a 0. The Newto-Raphso algorithm for miimizig Q starts with a trial value a 0 ad, if H 0 is positive de ite, miimizes the quadratic approximatio obtaiig a = a 0 H0 S0. [If H 0 is ot de ite, oe ca replace it with H + ci for some small umber c.] The ext step is to evaluate S ad H at = a ad repeat the calculatio. Usig the recursio a r+ = a r Hr S r, oe cotiues util a r+ a r. The, as log as H r is positive de ite, oe uses a r as the estimate ^ sice S(a r ) 0. Of course, this yields at best oly a local miimum; oe must try various startig values a 0 to be co det that oe has truly miimized Q. The Gauss-Newto algorithm is a variat of Newto-Raphso for the special case where Q ca be writte as e 0 e=2 ad the elemets of the -dimesioal vector e are oliear fuctios of. De ig Z to be the p 0, the gradiet vector S ca be 2
3 writte as Z 0 e. Moreover, the hessia H is equal to Z 0 Z plus a term that teds to be smaller. The G-N algorithm drops this smaller term ad uses the recursio a r+ = a r (Z 0 rz r ) Z 0 re r, where Z ad e are evaluated at the previous estimate a r. Note that the G-N algorithm ca be implemeted by a sequece of least squares regressios. 4 Two-step Estimators We ofte ecouter problems where Q () is di cult to miimize because some elemets of eter i a complicated way. I those cases, a two-step estimatio procedure is ofte employed. Suppose the parameter vector is partitioed ito two parts ( ; 2 ) ad that eters ito Q i a simple way so that, if 2 were kow, Q could easily be miimized over :That is, de ig the gradiet of Q with respect to by S, we assume that the equatio S ( ; 2 ) = 0 ca easily by solved as = h( 2 ; y). If oe could d a simple estimate say ~ 2, oe might estimate by e = h( e 2 ; y). That is, we would estimate by miimizig Q( ; e 2 ). If both the (computatioally di cult) true extremal estimator b ad the simple estimator e 2 are cosistet ad joitly asymptotically ormal, the it ca be show that the estimator e is also cosistet ad asymptotically ormal. Its asymptotic variace will typicaly deped o the asymptotic variace of e 2 ad ca be computed by liearizig the rst-order coditio S ( e ; e 2 ) = 0. If, for example, =2 S ( e ; e 2 ) = =2 S ( 0 ; 0 2) + B p ( e 0 ) + B 2 p ( e 2 0 2) + o p () the the asymptotic variace of e ca be calculated from p ( e 0 ) = B [ =2 S ( 0 ; 0 2) + B 2 p ( e 2 0 2)] + o p () as log as oe kows the joit limitig distributio of =2 S ( 0 ; 0 2) ad p ( e 2 0 2): 5 Asymptotic Tests Based o Extremum Estimators Cosider the ull hypothesis that the true parameter value 0 satis es the equatio g( 0 ) = 0, where g is a vector of q smooth fuctios with cotiuous qp Jacobia matrix G() For otatioal coveiece we de e G G( 0 ). Suppose we have estimated by miimizig some objective fuctio Q(; y) as i sectio 2 ad that the stadardized estimator p (^ 0 ) is asymptotically N(0; V ) where V = B AB. (A ad B are de ed i that sectio.) The, usig the delta method, we d p [g( b ) g( 0 )] G p ( b 0 ) d! N(0; GV G 0 ): Uder the ull hypothesis, g(^) 0 (GV G 0 ) g(^) is asymptotically 2 (q). Replacig G with the cosistet estimate c G G( b ) does ot a ect the asymptotic distributio. Hece, for some cosistet estimate b V ; we might reject the hypothesis that g( 0 ) = 0 if the Wald statistic W g( b ) 0 ( b G b V b G 0 ) g( b ) (2) is larger tha the 95% quatile of a 2 (q) distributio. Sometimes solvig for b is computatioally di cult ad oe seeks a way to test the hypothesis that g( 0 ) = 0 that avoids this computatio. Suppose there exists a easy-tocalculate estimate ~ that satis es g( ~ ) = 0: Suppose further that, whe the ull hypothesis is true, ~ is cosistet, asymptotically ormal ad the usual liear approximatios are valid: p [g(^) g( ~ )] = G p (^ ~ ) + op () 3
4 =2 [S (^) S ( ~ )] = B p (^ ~ ) + op (): Assumig B is a cotiuous fuctio of 0 ; de e ~ G = G( ~ ) ad ~ B = B( ~ ): Sice g( ~ ) ad S (^) are both zero vectors, we see that W is asymptotically equivalet uder the ull hypothesis to ( b ~ ) 0 b G 0 ( b G b V b G 0 ) b G( b ~ ) (3) ad to S ( ~ ) 0 ~ B ~ G 0 [ ~ G ~ V ~ G 0 ] ~ G ~ B S ( ~ ): (4) Note that (4) ca be computed without solvig the miimizatio problem. I the liear regressio model where Q = 2 (y X)0 (y X); we d H = X 0 X ad var[s ()] = 2 X 0 X: Suppose the ull hypothesis is the liear costrait G 0 = 0: Estimatig A by s 2 X 0 X= ad B by X 0 X=, we obtai W = b 0 G 0 [G(X 0 X) G 0 ] G b =s 2 which is q times the usual F-statistic. This feature of LS regressio (that the variace matrix for the score is proportioal to the expectatio of the hessia) holds i may estimatio problems. Whe it occurs, ot oly do (2), (3) ad (4) simplify but also some additioal asymptotically equivalet expressios for the Wald statistic are available. Cosider the problem of miimizig Q(; y) subject to the costrait g() = 0. The solutio ca be used for our ~ : The rst order coditio for a iterior miimum is S ( ~ ) = ~G 0 which implies = ( ~ GB ~ G 0 ) ~ G 0 B S ( ~ ) ad S ( ~ ) = ~ G 0 ( ~ GB ~ G 0 ) ~ G 0 B S ( ~ ): If A = cb ad ~ is the costraied miimizer of Q ; the test statistics (3) ad (4) simplify to (^ ~ ) 0 b B(^ ~ )=^c (3 ) ad S ( ~ ) 0 ~ B S ( ~ )=^c : (4 ) Furthermore, a Taylor s series expasio of the statistic 2[Q (^) Q ( ~ )]=^c (5) shows that it too is asymptotically equivalet to (3 ) ad hece to W. Thus, if A = cb for some ozero scalar c; plim ^c = c, ad ~ is the costraied miimizer of Q ; we have four aymptotically equivalet statistics that might be used for testig g( 0 ) = 0: (a) a quadratic form i g( b ) : g( b ) 0 ( b G b B b G 0 ) g( b )=^c (b) a quadratic form i the estimator di erece ~ b ( b e ) 0 b B( b e )=^c (c) a quadratic form i the score (which is the same as a quadratic form i the Lagrage multiplier ) S( e ) 0 e B S( e )=^c 0 e G e B e G 0 =^c (d) the di erece i costraied ad ucostraied miimized objective fuctio (multiplied by 2/^c): 2[Q( e ; y) Q( b ; y)]=^c: 4
5 Although our discussio has cocered oly the ull distributio of these tests, the asymptotic equivalece holds also uder earby alteratives. Exact equivalece occurs i the liear regressio case because there H ad G are oradom ad do ot deped o the ukow 0. I the geeral case where H may be radom ad both may deped o 0, we d oly asymptotic equivalece. There are may di eret ways to cosistetly estimate B, icludig the hessias H ( b ) ad H ( ~ ) as well as the aalytic expressio EH () evaluated at b or ~. Thus there are really lots of asymptotically equivalet test statistics available. Computatioal coveiece is ofte used as a basis for choice amog asymptotically equivalet tests. However, if p ad q are large, the asymptotic approximatios are sometimes poor. It is usually wise to perform some simulatios to verify that the chose test statistic has approximately the correct small sample rejectio probability uder the ull hypothesis. Whe Q is mius the log likelihood fuctio we d that A = B sice A ad B are the alterative expressios for the limitig iformatio matrix; (d) is the the likelihood ratio statistic ad (c) is the score statistic. Whe the correct weightig matrix is used i the quadratic form de ig GMM, we also have A proportioal to B ad a choice of tests. 6 Score Tests as Diagostics The followig type of problem ofte occurs i ecoometrics. We postulate a fairly simple probability model for the data but etertai the possibility that a more complicated model may be eeded. Let Q ( ; 2 ) be the objective fuctio assumig the complicated model is correct; is the p-dimesioal parameter vector i the simple model ad 2 is the q- dimesioal vector of additioal parameters eed to cope with the complicatio. The simple model beig correct is equivalet to the parametric hypothesis that 2 = 0. Thus, before publishig estimates of based o the simple model, oe might wat to check that 2 is really close to zero. A Wald or likelihood ratio test would require actually estimatig the complicated model. The score test does ot ad is therefore commoly used as a diagostic. Assumig that A = B, the score test statistic for 2 = 0 is S( e ) 0 e B S( e ), where e miimizes Q subject to the costrait. The score vector S ca be partitioed ito two subvectors, say S ad S 2. But, whe evaluated at the costraied estimate e, S must be zero (sice that is the rst-order coditio for miimizig Q subject to 2 = 0.) Thus the test statistic ca be writte as S 2 ( e ) 0 CS 2 ( e ), where C is the q q lower right had block of B. Usig H( e ) as a estimate of B ad partitioig coformably, a atural estimate of C is ( e H 22 e H2 e H e H 2 ). I other words, to test the hypothesis that the simple model is valid: rst, compute e, the MLE for the parameters of the simple model; secod, evaluate the score S 2 at e = ( e ; 0); third, compute the test statistic S 2 ( e ) 0 ( e H 22 e H2 e H e H 2 ) S 2 ( e ). I may examples, the iformatio matrix is block diagoal implyig plim H 2 = 0; the the test statistic ca be simpli ed to S 2 ( e ) 0 e H 22 S 2( e ). I other words, whe the iformatio matrix is block diagoal, oe ca test 2 = 0 by pretedig that were kow ad equal to the estimate e. 5
Economics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More informationNotes On Median and Quantile Regression. James L. Powell Department of Economics University of California, Berkeley
Notes O Media ad Quatile Regressio James L. Powell Departmet of Ecoomics Uiversity of Califoria, Berkeley Coditioal Media Restrictios ad Least Absolute Deviatios It is well-kow that the expected value
More informationAsymptotic Results for the Linear Regression Model
Asymptotic Results for the Liear Regressio Model C. Fli November 29, 2000 1. Asymptotic Results uder Classical Assumptios The followig results apply to the liear regressio model y = Xβ + ε, where X is
More informationLecture 6 Testing Nonlinear Restrictions 1. The previous lectures prepare us for the tests of nonlinear restrictions of the form:
Eco 75 Lecture 6 Testig Noliear Restrictios The previous lectures prepare us for the tests of oliear restrictios of the form: H 0 : h( 0 ) = 0 versus H : h( 0 ) 6= 0: () I this lecture, we cosier Wal,
More informationEfficient GMM LECTURE 12 GMM II
DECEMBER 1 010 LECTURE 1 II Efficiet The estimator depeds o the choice of the weight matrix A. The efficiet estimator is the oe that has the smallest asymptotic variace amog all estimators defied by differet
More information1 General linear Model Continued..
Geeral liear Model Cotiued.. We have We kow y = X + u X o radom u v N(0; I ) b = (X 0 X) X 0 y E( b ) = V ar( b ) = (X 0 X) We saw that b = (X 0 X) X 0 u so b is a liear fuctio of a ormally distributed
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationResampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.
Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More informationLarge Sample Theory. Convergence. Central Limit Theorems Asymptotic Distribution Delta Method. Convergence in Probability Convergence in Distribution
Large Sample Theory Covergece Covergece i Probability Covergece i Distributio Cetral Limit Theorems Asymptotic Distributio Delta Method Covergece i Probability A sequece of radom scalars {z } = (z 1,z,
More informationProperties and Hypothesis Testing
Chapter 3 Properties ad Hypothesis Testig 3.1 Types of data The regressio techiques developed i previous chapters ca be applied to three differet kids of data. 1. Cross-sectioal data. 2. Time series data.
More informationSlide Set 13 Linear Model with Endogenous Regressors and the GMM estimator
Slide Set 13 Liear Model with Edogeous Regressors ad the GMM estimator Pietro Coretto pcoretto@uisa.it Ecoometrics Master i Ecoomics ad Fiace (MEF) Uiversità degli Studi di Napoli Federico II Versio: Friday
More informationSolution to Chapter 2 Analytical Exercises
Nov. 25, 23, Revised Dec. 27, 23 Hayashi Ecoometrics Solutio to Chapter 2 Aalytical Exercises. For ay ε >, So, plim z =. O the other had, which meas that lim E(z =. 2. As show i the hit, Prob( z > ε =
More information1 Covariance Estimation
Eco 75 Lecture 5 Covariace Estimatio ad Optimal Weightig Matrices I this lecture, we cosider estimatio of the asymptotic covariace matrix B B of the extremum estimator b : Covariace Estimatio Lemma 4.
More informationLecture 33: Bootstrap
Lecture 33: ootstrap Motivatio To evaluate ad compare differet estimators, we eed cosistet estimators of variaces or asymptotic variaces of estimators. This is also importat for hypothesis testig ad cofidece
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More information11 THE GMM ESTIMATION
Cotets THE GMM ESTIMATION 2. Cosistecy ad Asymptotic Normality..................... 3.2 Regularity Coditios ad Idetificatio..................... 4.3 The GMM Iterpretatio of the OLS Estimatio.................
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationx iu i E(x u) 0. In order to obtain a consistent estimator of β, we find the instrumental variable z which satisfies E(z u) = 0. z iu i E(z u) = 0.
27 However, β MM is icosistet whe E(x u) 0, i.e., β MM = (X X) X y = β + (X X) X u = β + ( X X ) ( X u ) \ β. Note as follows: X u = x iu i E(x u) 0. I order to obtai a cosistet estimator of β, we fid
More informationLinear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d
Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y
More informationStudy the bias (due to the nite dimensional approximation) and variance of the estimators
2 Series Methods 2. Geeral Approach A model has parameters (; ) where is ite-dimesioal ad is oparametric. (Sometimes, there is o :) We will focus o regressio. The fuctio is approximated by a series a ite
More informationSOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker
SOME THEORY AND PRACTICE OF STATISTICS by Howard G. Tucker CHAPTER 9. POINT ESTIMATION 9. Covergece i Probability. The bases of poit estimatio have already bee laid out i previous chapters. I chapter 5
More informationPOWER COMPARISON OF EMPIRICAL LIKELIHOOD RATIO TESTS: SMALL SAMPLE PROPERTIES THROUGH MONTE CARLO STUDIES*
Kobe Uiversity Ecoomic Review 50(2004) 3 POWER COMPARISON OF EMPIRICAL LIKELIHOOD RATIO TESTS: SMALL SAMPLE PROPERTIES THROUGH MONTE CARLO STUDIES* By HISASHI TANIZAKI There are various kids of oparametric
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationJanuary 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS
Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationAn Introduction to Asymptotic Theory
A Itroductio to Asymptotic Theory Pig Yu School of Ecoomics ad Fiace The Uiversity of Hog Kog Pig Yu (HKU) Asymptotic Theory 1 / 20 Five Weapos i Asymptotic Theory Five Weapos i Asymptotic Theory Pig Yu
More informationLecture 7 Testing Nonlinear Inequality Restrictions 1
Eco 75 Lecture 7 Testig Noliear Iequality Restrictios I Lecture 6, we discussed te testig problems were te ull ypotesis is de ed by oliear equality restrictios: H : ( ) = versus H : ( ) 6= : () We sowed
More informationLecture 2: Monte Carlo Simulation
STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationLogit regression Logit regression
Logit regressio Logit regressio models the probability of Y= as the cumulative stadard logistic distributio fuctio, evaluated at z = β 0 + β X: Pr(Y = X) = F(β 0 + β X) F is the cumulative logistic distributio
More informationIntroduction to Optimization Techniques. How to Solve Equations
Itroductio to Optimizatio Techiques How to Solve Equatios Iterative Methods of Optimizatio Iterative methods of optimizatio Solutio of the oliear equatios resultig form a optimizatio problem is usually
More informationFirst Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationThe standard deviation of the mean
Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider
More informationChapter 6 Sampling Distributions
Chapter 6 Samplig Distributios 1 I most experimets, we have more tha oe measuremet for ay give variable, each measuremet beig associated with oe radomly selected a member of a populatio. Hece we eed to
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationECE 8527: Introduction to Machine Learning and Pattern Recognition Midterm # 1. Vaishali Amin Fall, 2015
ECE 8527: Itroductio to Machie Learig ad Patter Recogitio Midterm # 1 Vaishali Ami Fall, 2015 tue39624@temple.edu Problem No. 1: Cosider a two-class discrete distributio problem: ω 1 :{[0,0], [2,0], [2,2],
More informationStatistical Inference (Chapter 10) Statistical inference = learn about a population based on the information provided by a sample.
Statistical Iferece (Chapter 10) Statistical iferece = lear about a populatio based o the iformatio provided by a sample. Populatio: The set of all values of a radom variable X of iterest. Characterized
More informationRegression with an Evaporating Logarithmic Trend
Regressio with a Evaporatig Logarithmic Tred Peter C. B. Phillips Cowles Foudatio, Yale Uiversity, Uiversity of Aucklad & Uiversity of York ad Yixiao Su Departmet of Ecoomics Yale Uiversity October 5,
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationBull. Korean Math. Soc. 36 (1999), No. 3, pp. 451{457 THE STRONG CONSISTENCY OF NONLINEAR REGRESSION QUANTILES ESTIMATORS Seung Hoe Choi and Hae Kyung
Bull. Korea Math. Soc. 36 (999), No. 3, pp. 45{457 THE STRONG CONSISTENCY OF NONLINEAR REGRESSION QUANTILES ESTIMATORS Abstract. This paper provides suciet coditios which esure the strog cosistecy of regressio
More informationA Note on Box-Cox Quantile Regression Estimation of the Parameters of the Generalized Pareto Distribution
A Note o Box-Cox Quatile Regressio Estimatio of the Parameters of the Geeralized Pareto Distributio JM va Zyl Abstract: Makig use of the quatile equatio, Box-Cox regressio ad Laplace distributed disturbaces,
More information( θ. sup θ Θ f X (x θ) = L. sup Pr (Λ (X) < c) = α. x : Λ (x) = sup θ H 0. sup θ Θ f X (x θ) = ) < c. NH : θ 1 = θ 2 against AH : θ 1 θ 2
82 CHAPTER 4. MAXIMUM IKEIHOOD ESTIMATION Defiitio: et X be a radom sample with joit p.m/d.f. f X x θ. The geeralised likelihood ratio test g.l.r.t. of the NH : θ H 0 agaist the alterative AH : θ H 1,
More informationLECTURE 14 NOTES. A sequence of α-level tests {ϕ n (x)} is consistent if
LECTURE 14 NOTES 1. Asymptotic power of tests. Defiitio 1.1. A sequece of -level tests {ϕ x)} is cosistet if β θ) := E θ [ ϕ x) ] 1 as, for ay θ Θ 1. Just like cosistecy of a sequece of estimators, Defiitio
More informationMachine Learning Brett Bernstein
Machie Learig Brett Berstei Week Lecture: Cocept Check Exercises Starred problems are optioal. Statistical Learig Theory. Suppose A = Y = R ad X is some other set. Furthermore, assume P X Y is a discrete
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationLecture 3: MLE and Regression
STAT/Q SCI 403: Itroductio to Resamplig Methods Sprig 207 Istructor: Ye-Chi Che Lecture 3: MLE ad Regressio 3. Parameters ad Distributios Some distributios are idexed by their uderlyig parameters. Thus,
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5
CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio
More informationFACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures
FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals
More informationStatistical and Mathematical Methods DS-GA 1002 December 8, Sample Final Problems Solutions
Statistical ad Mathematical Methods DS-GA 00 December 8, 05. Short questios Sample Fial Problems Solutios a. Ax b has a solutio if b is i the rage of A. The dimesio of the rage of A is because A has liearly-idepedet
More information10. Comparative Tests among Spatial Regression Models. Here we revisit the example in Section 8.1 of estimating the mean of a normal random
Part III. Areal Data Aalysis 0. Comparative Tests amog Spatial Regressio Models While the otio of relative likelihood values for differet models is somewhat difficult to iterpret directly (as metioed above),
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationOutput Analysis and Run-Length Control
IEOR E4703: Mote Carlo Simulatio Columbia Uiversity c 2017 by Marti Haugh Output Aalysis ad Ru-Legth Cotrol I these otes we describe how the Cetral Limit Theorem ca be used to costruct approximate (1 α%
More informationCS537. Numerical Analysis and Computing
CS57 Numerical Aalysis ad Computig Lecture Locatig Roots o Equatios Proessor Ju Zhag Departmet o Computer Sciece Uiversity o Ketucky Leigto KY 456-6 Jauary 9 9 What is the Root May physical system ca be
More informationIntroductory statistics
CM9S: Machie Learig for Bioiformatics Lecture - 03/3/06 Itroductory statistics Lecturer: Sriram Sakararama Scribe: Sriram Sakararama We will provide a overview of statistical iferece focussig o the key
More information6 Sample Size Calculations
6 Sample Size Calculatios Oe of the major resposibilities of a cliical trial statisticia is to aid the ivestigators i determiig the sample size required to coduct a study The most commo procedure for determiig
More information6. Sufficient, Complete, and Ancillary Statistics
Sufficiet, Complete ad Acillary Statistics http://www.math.uah.edu/stat/poit/sufficiet.xhtml 1 of 7 7/16/2009 6:13 AM Virtual Laboratories > 7. Poit Estimatio > 1 2 3 4 5 6 6. Sufficiet, Complete, ad Acillary
More informationNonlinear regression
oliear regressio How to aalyse data? How to aalyse data? Plot! How to aalyse data? Plot! Huma brai is oe the most powerfull computatioall tools Works differetly tha a computer What if data have o liear
More informationStochastic Simulation
Stochastic Simulatio 1 Itroductio Readig Assigmet: Read Chapter 1 of text. We shall itroduce may of the key issues to be discussed i this course via a couple of model problems. Model Problem 1 (Jackso
More informationSTATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. Comments:
Recall: STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Commets:. So far we have estimates of the parameters! 0 ad!, but have o idea how good these estimates are. Assumptio: E(Y x)! 0 +! x (liear coditioal
More informationExercise 4.3 Use the Continuity Theorem to prove the Cramér-Wold Theorem, Theorem. (1) φ a X(1).
Assigmet 7 Exercise 4.3 Use the Cotiuity Theorem to prove the Cramér-Wold Theorem, Theorem 4.12. Hit: a X d a X implies that φ a X (1) φ a X(1). Sketch of solutio: As we poited out i class, the oly tricky
More informationEcon 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara
Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio
More informationA RANK STATISTIC FOR NON-PARAMETRIC K-SAMPLE AND CHANGE POINT PROBLEMS
J. Japa Statist. Soc. Vol. 41 No. 1 2011 67 73 A RANK STATISTIC FOR NON-PARAMETRIC K-SAMPLE AND CHANGE POINT PROBLEMS Yoichi Nishiyama* We cosider k-sample ad chage poit problems for idepedet data i a
More informationLecture 3. Properties of Summary Statistics: Sampling Distribution
Lecture 3 Properties of Summary Statistics: Samplig Distributio Mai Theme How ca we use math to justify that our umerical summaries from the sample are good summaries of the populatio? Lecture Summary
More informationProblem Set 4 Due Oct, 12
EE226: Radom Processes i Systems Lecturer: Jea C. Walrad Problem Set 4 Due Oct, 12 Fall 06 GSI: Assae Gueye This problem set essetially reviews detectio theory ad hypothesis testig ad some basic otios
More informationKernel density estimator
Jauary, 07 NONPARAMETRIC ERNEL DENSITY ESTIMATION I this lecture, we discuss kerel estimatio of probability desity fuctios PDF Noparametric desity estimatio is oe of the cetral problems i statistics I
More informationMathematical Statistics - MS
Paper Specific Istructios. The examiatio is of hours duratio. There are a total of 60 questios carryig 00 marks. The etire paper is divided ito three sectios, A, B ad C. All sectios are compulsory. Questios
More informationTMA4245 Statistics. Corrected 30 May and 4 June Norwegian University of Science and Technology Department of Mathematical Sciences.
Norwegia Uiversity of Sciece ad Techology Departmet of Mathematical Scieces Corrected 3 May ad 4 Jue Solutios TMA445 Statistics Saturday 6 May 9: 3: Problem Sow desity a The probability is.9.5 6x x dx
More informationMatrix Representation of Data in Experiment
Matrix Represetatio of Data i Experimet Cosider a very simple model for resposes y ij : y ij i ij, i 1,; j 1,,..., (ote that for simplicity we are assumig the two () groups are of equal sample size ) Y
More informationRates of Convergence by Moduli of Continuity
Rates of Covergece by Moduli of Cotiuity Joh Duchi: Notes for Statistics 300b March, 017 1 Itroductio I this ote, we give a presetatio showig the importace, ad relatioship betwee, the modulis of cotiuity
More informationExponential Families and Bayesian Inference
Computer Visio Expoetial Families ad Bayesia Iferece Lecture Expoetial Families A expoetial family of distributios is a d-parameter family f(x; havig the followig form: f(x; = h(xe g(t T (x B(, (. where
More informationLecture 7: Properties of Random Samples
Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ
More informationDifferentiable Convex Functions
Differetiable Covex Fuctios The followig picture motivates Theorem 11. f ( x) f ( x) f '( x)( x x) ˆx x 1 Theorem 11 : Let f : R R be differetiable. The, f is covex o the covex set C R if, ad oly if for
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationThe Method of Least Squares. To understand least squares fitting of data.
The Method of Least Squares KEY WORDS Curve fittig, least square GOAL To uderstad least squares fittig of data To uderstad the least squares solutio of icosistet systems of liear equatios 1 Motivatio Curve
More informationThe central limit theorem for Student s distribution. Problem Karim M. Abadir and Jan R. Magnus. Econometric Theory, 19, 1195 (2003)
The cetral limit theorem for Studet s distributio Problem 03.6.1 Karim M. Abadir ad Ja R. Magus Ecoometric Theory, 19, 1195 (003) Z Ecoometric Theory, 19, 003, 1195 1198+ Prited i the Uited States of America+
More informationIIT JAM Mathematical Statistics (MS) 2006 SECTION A
IIT JAM Mathematical Statistics (MS) 6 SECTION A. If a > for ad lim a / L >, the which of the followig series is ot coverget? (a) (b) (c) (d) (d) = = a = a = a a + / a lim a a / + = lim a / a / + = lim
More informationQuestions and Answers on Maximum Likelihood
Questios ad Aswers o Maximum Likelihood L. Magee Fall, 2008 1. Give: a observatio-specific log likelihood fuctio l i (θ) = l f(y i x i, θ) the log likelihood fuctio l(θ y, X) = l i(θ) a data set (x i,
More informationLecture Stat Maximum Likelihood Estimation
Lecture Stat 461-561 Maximum Likelihood Estimatio A.D. Jauary 2008 A.D. () Jauary 2008 1 / 63 Maximum Likelihood Estimatio Ivariace Cosistecy E ciecy Nuisace Parameters A.D. () Jauary 2008 2 / 63 Parametric
More informationRandom Matrices with Blocks of Intermediate Scale Strongly Correlated Band Matrices
Radom Matrices with Blocks of Itermediate Scale Strogly Correlated Bad Matrices Jiayi Tog Advisor: Dr. Todd Kemp May 30, 07 Departmet of Mathematics Uiversity of Califoria, Sa Diego Cotets Itroductio Notatio
More informationTests of Hypotheses Based on a Single Sample (Devore Chapter Eight)
Tests of Hypotheses Based o a Sigle Sample Devore Chapter Eight MATH-252-01: Probability ad Statistics II Sprig 2018 Cotets 1 Hypothesis Tests illustrated with z-tests 1 1.1 Overview of Hypothesis Testig..........
More informationNYU Center for Data Science: DS-GA 1003 Machine Learning and Computational Statistics (Spring 2018)
NYU Ceter for Data Sciece: DS-GA 003 Machie Learig ad Computatioal Statistics (Sprig 208) Brett Berstei, David Roseberg, Be Jakubowski Jauary 20, 208 Istructios: Followig most lab ad lecture sectios, we
More informationIt should be unbiased, or approximately unbiased. Variance of the variance estimator should be small. That is, the variance estimator is stable.
Chapter 10 Variace Estimatio 10.1 Itroductio Variace estimatio is a importat practical problem i survey samplig. Variace estimates are used i two purposes. Oe is the aalytic purpose such as costructig
More informationt distribution [34] : used to test a mean against an hypothesized value (H 0 : µ = µ 0 ) or the difference
EXST30 Backgroud material Page From the textbook The Statistical Sleuth Mea [0]: I your text the word mea deotes a populatio mea (µ) while the work average deotes a sample average ( ). Variace [0]: The
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationDS 100: Principles and Techniques of Data Science Date: April 13, Discussion #10
DS 00: Priciples ad Techiques of Data Sciece Date: April 3, 208 Name: Hypothesis Testig Discussio #0. Defie these terms below as they relate to hypothesis testig. a) Data Geeratio Model: Solutio: A set
More informationCSE 527, Additional notes on MLE & EM
CSE 57 Lecture Notes: MLE & EM CSE 57, Additioal otes o MLE & EM Based o earlier otes by C. Grat & M. Narasimha Itroductio Last lecture we bega a examiatio of model based clusterig. This lecture will be
More informationLECTURE 8: ASYMPTOTICS I
LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece
More informationSection 14. Simple linear regression.
Sectio 14 Simple liear regressio. Let us look at the cigarette dataset from [1] (available to dowload from joural s website) ad []. The cigarette dataset cotais measuremets of tar, icotie, weight ad carbo
More informationMathematics 170B Selected HW Solutions.
Mathematics 17B Selected HW Solutios. F 4. Suppose X is B(,p). (a)fidthemometgeeratigfuctiom (s)of(x p)/ p(1 p). Write q = 1 p. The MGF of X is (pe s + q), sice X ca be writte as the sum of idepedet Beroulli
More informationNANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS
NANYANG TECHNOLOGICAL UNIVERSITY SYLLABUS FOR ENTRANCE EXAMINATION FOR INTERNATIONAL STUDENTS AO-LEVEL MATHEMATICS STRUCTURE OF EXAMINATION PAPER. There will be oe 2-hour paper cosistig of 4 questios.
More informationTAMS24: Notations and Formulas
TAMS4: Notatios ad Formulas Basic otatios ad defiitios X: radom variable stokastiska variabel Mea Vätevärde: µ = X = by Xiagfeg Yag kpx k, if X is discrete, xf Xxdx, if X is cotiuous Variace Varias: =
More information1 Introduction to reducing variance in Monte Carlo simulations
Copyright c 010 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a ukow mea µ = E(X) of a distributio by
More informationLinear Regression Demystified
Liear Regressio Demystified Liear regressio is a importat subject i statistics. I elemetary statistics courses, formulae related to liear regressio are ofte stated without derivatio. This ote iteds to
More informationRandom assignment with integer costs
Radom assigmet with iteger costs Robert Parviaie Departmet of Mathematics, Uppsala Uiversity P.O. Box 480, SE-7506 Uppsala, Swede robert.parviaie@math.uu.se Jue 4, 200 Abstract The radom assigmet problem
More informationCS321. Numerical Analysis and Computing
CS Numerical Aalysis ad Computig Lecture Locatig Roots o Equatios Proessor Ju Zhag Departmet o Computer Sciece Uiversity o Ketucky Leigto KY 456-6 September 8 5 What is the Root May physical system ca
More information1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More information