Nonparametric regression: minimax upper and lower bounds
|
|
- Amy Richard
- 5 years ago
- Views:
Transcription
1 Capter 4 Noparametric regressio: miimax upper ad lower bouds 4. Itroductio We cosider oe of te two te most classical o-parametric problems i tis example: estimatig a regressio fuctio o a subset of te real lie (te most classical problem beig estimatio of a desity). I o-parametric regressio, we assume tere is a ukow fuctio f : R R, were f belogs to a pre-determied class of fuctios F; usually tis class is parameterized by some type of smootess guaratee. To make our problems cocrete, we will assume tat te ukow fuctio f is L-Lipscitz ad defied o [0,]. Let F deote tis class. (For a fuller tecical itroductio ito oparametric estimatio, see te book by Tsybakov [].) Figure 4.. Observatios i a o-parametric regressio problem, wit fuctio f plotted. (Here f(x) = si(x+cos (3x)).) 4
2 Staford Statistics 3/Electrical Egieerig 377 Jo Duci I te stadard o-parametric regressio problem, we obtai observatios of te form Y i = f(x i )+ε i (4..) were ε i are idepedet, mea zero coditioal o X i, ad E[ε i ] σ. See Figure 4. for a example. We also assume tat we fix te locatios of te X i as X i = i/ [0,], tat is, te X i are evely spaced i [0,]. Give observatios Y i, we ask two questios: () ow ca we estimate f? ad () wat are te optimal rates at wic it is possible to estimate f? 4. Kerel estimates of te fuctio A atural strategy is to place small bumps aroud te observed poits, ad estimate f i a eigborood of a poit x by weigted averages of te Y values for oter poits ear x. We ow formalize a strategy for doig tis. Suppose we ave a kerel fuctio K : R R +, wic is cotiuous, ot idetically zero, as support supp K = [, ], ad satisfies te tecical coditio λ 0 supk(x) if K(x), (4..) x x / were λ 0 > 0 (tis says te kerel as some widt to it). A atural example is te tet fuctio give by K tet (x) = [ x ] +, wic satisfies iequality (4..) wit λ 0 = /. See Fig. 4. for two examples, oe te tet fuctio ad te oter te fuctio ( K(x) = { x < }exp (x ) wic is ifiitely differetiable ad supported o [, ]. ) exp ( (x+) ), Figure 4.: Left: tet kerel. Rigt: ifiitely differetiable compactly supported kerel. Now we cosider a atural estimator of te fuctio f based o observatios (4..) kow as te Nadaraya-Watso estimator. Fix a badwidt, wic we will see later smoots te estimated fuctios f. For all x, defie weigts W i (x) := ( ) Xi x K ( K Xj x 4 )
3 Staford Statistics 3/Electrical Egieerig 377 Jo Duci ad defie te estimated fuctio f (x) := Y i W i (x). Te ituitio ere is tat we ave a locally weigted regressio fuctio, were poits X i i te eigborood of x are give iger weigt ta furter poits. Usig tis fuctio f as our estimator, it is possible to provide a guaratee o te bias ad variace of te estimated fuctio at eac poit x [0,]. Propositio 4.. Let te observatio model (4..) old ad assume coditio (4..). I additio assume te badwidt is suitably large tat / ad tat te X i are evely spaced o [0,]. Te for ay x [0,], we ave E[ f (x)] f(x) L ad Var( f (x)) σ λ 0. Proof To boud te bias, we ote tat (coditioig implicitly o X i ) E[ f (x)] = E[Y i W i (x)] = E[f(X i )W i (x)+ε i W i (x)] = f(x i )W i (x). Tus we ave tat te bias is bouded as E[ f (x)] f(x) f(x i ) f(x) W i (x) i: X i x To boud te variace, we claim tat f(x i ) f(x) W i (x) L W i (x) mi W i (x) = L. { } λ 0,. (4..) Ideed, we ave tat W i (x) = ( Xi x ) K ( K Xj x ) = K ( ) Xi x j: X j x / K ( Xj x ) ( ) K Xi x λ 0 sup x K(x) {j : X j x /}, ad because tere are at least / idices satisfyig X j x, we obtai te claim (4..). Usig te claim, we ave [( ) ] Var( f (x)) = E (Y i f(x i ))W i (x) [( = E = ) ] ε i W i (x) W i (x) E[ε i] σ W i (x). 43
4 Staford Statistics 3/Electrical Egieerig 377 Jo Duci Notig tat W i (x) /λ 0 ad W i(x) =, we ave completig te proof. σ W i (x) σ maxw i (x) W i (x) i }{{} = σ λ 0, Wit te propositio i place, we ca te provide a teorem boudig te worst case poitwise mea squared error for estimatio of a fuctio f F. Teorem 4.. Uder te coditios of Propositio 4., coose = (σ /L λ 0 ) /3 /3. Te tere exists a uiversal (umerical) costat C < suc tat for ay f F, sup E[( f (x) f(x)) ] C x [0,] ( Lσ Proof Usig Propositio 4., we ave for ay x [0,] tat E[( f (x) f(x)) ] = λ 0 ) /3 3. ( ) E[ f (x)] f(x) +E[( f (x) E[ f (x)]) ] σ λ 0 +L. Coosig to balace te above bias/variace tradeoff, we obtai te toerem. By itegratig te result i Teorem 4. over te iterval [0, ], we immediately obtai te followig corollary. Corollary 4.3. Uder te coditios of Teorem 4., if we use te tet kerel K tet, we ave were C is a uiversal costat. supe f [ f f ] C f F ( ) Lσ /3, I Propositio 4., it is possible to sow tat a more clever coice of kerels oes tat are ot always positive ca attai bias E[ f (x)] f(x) = O( β ) if f as Lipscitz (β )t derivative. I tis case, we immediately obtai tat te rate ca be improved to supe[( f (x) f(x)) ] C β+, x ad every additioal degree of smootess gives a correspodig improvemet i covergece rate. We also remark tat rates of tis form, wic are muc larger ta, are caracteristic of oparametric problems; essetially, we must adaptively coose a dimesio tat balaces te sample size, so tat rates of / are difficult or impossible to acieve. 44
5 Staford Statistics 3/Electrical Egieerig 377 Jo Duci 4.3 Miimax lower bouds o estimatio wit Assouad s metod Now we ca ask weter te results we ave give are i fact sarp; do tere exist estimators attaiig a faster rate of covergece ta our kerel-based (locally weigted) estimator? Usig Assouad s metod, we sow tat, i fact, tese results are all tigt. I particular, we prove te followig result o miimax estimatio of a regressio fuctio f F, were F cosists of - Lipscitz fuctios defied o [0,], i te error, tat is, f g = 0 (f(t) g(t)) dt. Teorem 4.4. Let te observatio poits X i be spaced evely o [0,], ad assume te observatio model (4..). Te tere exists a uiversal costat c > 0 suc tat M (F, ) := if sup f f F ] ( σ E f [ f f c Deferrig te proof of te teorem temporarily, we make a few remarks. It is i fact possible to sow usig a completely idetical tecique tat if F β deotes te class of fuctios wit β derivatives, were te (β )t derivative is Lipscitz, te M (F β, ) c ( σ ) β β+. So for ay smootess class, we ca ever acieve te parametric σ / rate, but we ca come arbitrarily close. As aoter remark, wic we do ot prove, i dimesios d, te miimax rate for estimatio of fuctios f wit Lipscitz (β )t derivative scales as M (F β, ) c ( σ ) β β+d. Tis result ca, similarly, be proved usig a variat of Assouad s metod; see, for example, te book of Györfi et al. [, Capter 3], wic is available olie. Tis is a strikig example of te curse of dimesioality: te pealty for icreasig dimesio results i worse rates of covergece. For example, suppose tat β =. I dimesio, we require 90 (.05) 3/ observatios to acieve accuracy.05 i estimatio of f, wile we require 8000 = (.05) (+d)/ eve we te dimesio d = 4, ad observatios eve i 0 dimesios, wic is a relatively small problem. Tat is, te problem is made expoetially more difficult by dimesio icreases. We ow tur to provig Teorem 4.4. To establis te result, we sow ow to costruct a family of problems idexed by biary vectors v {,} k so tat our estimatio problem satisfies te separatio (3..), te we sow tat iformatio based o observig oisy versios of te fuctios we ave defied is small. We te coose k to make our resultig lower boud as ig as possible. Costructio of a separated family of fuctios To costruct our separatio i Hammig metric, asrequiredbyeq.(3..), fixsomek N; wewillcoosek later. Tisapproacissomewat differet from our stadard approac of usig a fixed dimesioality ad scalig te separatio directly; i o-parametric problems, we scale te dimesio itself to adjust te difficulty of te estimatio problem. Defie te fuctio g(x) = [/ x / ] +, so tat g is -Lipscitz ad is 0 outside of te iterval [0,]. Te for ay v {,} k, defie te bump fuctios g j (x) := ( (k k g x j )) k 45 ad f v (x) := ) 3. v j g j (x),
6 Staford Statistics 3/Electrical Egieerig 377 Jo Duci wic we see is -Lipscitz. Now, cosider ay fuctio f : [0,] R, ad let be sortad for te itervals = [(j )/k,j/k] for j =,...,k. We must fid a mappig idetifyig a fuctio f wit poits i te ypercube {,} k. To tat ed, we may defie a vector v(f) {,} k by v j (f) = argmi (f(t) sg j (t)) dt. s {,} We claim tat for ay fuctio f, ( (f(t) f v (t)) dt ) ( { vj (f) v j } f v (t) dt ). (4.3.) Ideed, o te set, we ave v j g j (t) = f v (t), ad tus g j (t) dt = f v (t) dt. Te by te triagle iequality, we ave ( ) ( ) { v j (f) v j } g j (t) dt = (( v j (f) v j )g j (t)) dt ( (f(t) v j g j (t)) dt ( (f(t) f v (t)) dt ) + ( ), (f(t) v j (f)g j (t)) dt by defiitio of te sig v j (f). Wit te defiitio of v ad iequality (4.3.), we see tat for ay vector v {,} k, we ave f f v = I particular, we kow tat f v (t) dt = k (f(t) f v (t)) dt /k 0 g(kt) dt = k 3 { v j (f) v j } f v (t) dt. 0 g(u) du c k 3, were c is a umerical costat. I particular, we ave te desired separatio f f v c k 3 ) { v j (f) v j }. (4.3.) Boudig te biary testig error Let P v deote te distributio of te observatios Y i = f v (X i ) + ε i we f v is te true regressio fuctio. Te iequality (4.3.) implies via Assouad s lemma tat M (F, ) c k 3 [ P +j P j ] TV. (4.3.3) Now, we use covexity ad Pisker s iequality to ote tat P +j P j TV max P v v,+j Pv, j 46 TV max v D ( kl P v,+j Pv, j ).
7 Staford Statistics 3/Electrical Egieerig 377 Jo Duci For ay two fuctios f v ad f v, we ave tat te observatios Y i are idepedet ad ormal wit meas f v (X i ) or f v (X i ), respectively. Tus D kl (Pv Pv ) = ( D kl N(fv (X i ),σ ) N(f v (X i ),σ ) ) = σ (f v(x i ) f v (X i )). (4.3.4) Now we must sow tat te expressio (4.3.4) scales more slowly ta, wic we will see must be te case as weever d am (v,v ). Ituitively, most of te observatios ave te same distributio by our costructio of te f v as bump fuctios; let us make tis rigorous. We may assume witout loss of geerality tat v j = v j for j >. As te X i = i/, we tus ave tat oly X i for i ear ca ave o-zero values i te tesorizatio (4.3.4). I particular, f v (i/) = f v (i/) for all i s.t. i k, i.e. i k. Rewritig expressio (4.3.4), te, ad otig tat f v (x) [ /k,/k] for all x by costructio, we ave /k σ (f v(x i ) f v (X i )) σ (f v(x i ) f v (X i )) σ k Combiig tis wit iequality (4.3.4) ad te miimax boud (4.3.3), we obtai k = k 3 σ. so P +j P j TV M (F, ) c k 3 k 3 σ, [ ] k 3 σ. Coosig k for optimal tradeoffs Now we simply coose k; i particular, settig ( ) /3 k = σ te k 3 σ /4 =, ad we arrive at M (F, ) c k 3 = c k c were c > 0 is a uiversal costat. Teorem 4.4 is proved. ( ) σ /3, 47
8 Bibliograpy [] L. Györfi, M. Koler, A. Krzyżak, ad H. Walk. A Distributio-Free Teory of Noparametric Regressio. Spriger, 00. [] A. B. Tsybakov. Itroductio to Noparametric Estimatio. Spriger,
5.1 A mutual information bound based on metric entropy
Chapter 5 Global Fao Method I this chapter, we exted the techiques of Chapter 2.4 o Fao s method the local Fao method) to a more global costructio. I particular, we show that, rather tha costructig a local
More informationLECTURE 2 LEAST SQUARES CROSS-VALIDATION FOR KERNEL DENSITY ESTIMATION
Jauary 3 07 LECTURE LEAST SQUARES CROSS-VALIDATION FOR ERNEL DENSITY ESTIMATION Noparametric kerel estimatio is extremely sesitive to te coice of badwidt as larger values of result i averagig over more
More informationON LOCAL LINEAR ESTIMATION IN NONPARAMETRIC ERRORS-IN-VARIABLES MODELS 1
Teory of Stocastic Processes Vol2 28, o3-4, 2006, pp*-* SILVELYN ZWANZIG ON LOCAL LINEAR ESTIMATION IN NONPARAMETRIC ERRORS-IN-VARIABLES MODELS Local liear metods are applied to a oparametric regressio
More informationLecture 9: Regression: Regressogram and Kernel Regression
STAT 425: Itroductio to Noparametric Statistics Witer 208 Lecture 9: Regressio: Regressogram ad erel Regressio Istructor: Ye-Ci Ce Referece: Capter 5 of All of oparametric statistics 9 Itroductio Let X,
More information4 Conditional Distribution Estimation
4 Coditioal Distributio Estimatio 4. Estimators Te coditioal distributio (CDF) of y i give X i = x is F (y j x) = P (y i y j X i = x) = E ( (y i y) j X i = x) : Tis is te coditioal mea of te radom variable
More informationCONCENTRATION INEQUALITIES
CONCENTRATION INEQUALITIES MAXIM RAGINSKY I te previous lecture, te followig result was stated witout proof. If X 1,..., X are idepedet Beroulliθ radom variables represetig te outcomes of a sequece of
More informationWeek 6. Intuition: Let p (x, y) be the joint density of (X, Y ) and p (x) be the marginal density of X. Then. h dy =
Week 6 Lecture Kerel regressio ad miimax rates Model: Observe Y i = f (X i ) + ξ i were ξ i, i =,,,, iid wit Eξ i = 0 We ofte assume X i are iid or X i = i/ Nadaraya Watso estimator Let w i (x) = K ( X
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationUnbiased Estimation. February 7-12, 2008
Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationNotes On Nonparametric Density Estimation. James L. Powell Department of Economics University of California, Berkeley
Notes O Noparametric Desity Estimatio James L Powell Departmet of Ecoomics Uiversity of Califoria, Berkeley Uivariate Desity Estimatio via Numerical Derivatives Cosider te problem of estimatig te desity
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationLecture 7 Testing Nonlinear Inequality Restrictions 1
Eco 75 Lecture 7 Testig Noliear Iequality Restrictios I Lecture 6, we discussed te testig problems were te ull ypotesis is de ed by oliear equality restrictios: H : ( ) = versus H : ( ) 6= : () We sowed
More informationA survey on penalized empirical risk minimization Sara A. van de Geer
A survey o pealized empirical risk miimizatio Sara A. va de Geer We address the questio how to choose the pealty i empirical risk miimizatio. Roughly speakig, this pealty should be a good boud for the
More informationFirst Year Quantitative Comp Exam Spring, Part I - 203A. f X (x) = 0 otherwise
First Year Quatitative Comp Exam Sprig, 2012 Istructio: There are three parts. Aswer every questio i every part. Questio I-1 Part I - 203A A radom variable X is distributed with the margial desity: >
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationLecture 7: Density Estimation: k-nearest Neighbor and Basis Approach
STAT 425: Itroductio to Noparametric Statistics Witer 28 Lecture 7: Desity Estimatio: k-nearest Neighbor ad Basis Approach Istructor: Ye-Chi Che Referece: Sectio 8.4 of All of Noparametric Statistics.
More informationResampling Methods. X (1/2), i.e., Pr (X i m) = 1/2. We order the data: X (1) X (2) X (n). Define the sample median: ( n.
Jauary 1, 2019 Resamplig Methods Motivatio We have so may estimators with the property θ θ d N 0, σ 2 We ca also write θ a N θ, σ 2 /, where a meas approximately distributed as Oce we have a cosistet estimator
More informationREAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS
REAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS 18th Feb, 016 Defiitio (Lipschitz fuctio). A fuctio f : R R is said to be Lipschitz if there exists a positive real umber c such that for ay x, y i the domai
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationFall 2013 MTH431/531 Real analysis Section Notes
Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters
More informationChapter 6 Principles of Data Reduction
Chapter 6 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 0 Chapter 6 Priciples of Data Reductio Sectio 6. Itroductio Goal: To summarize or reduce the data X, X,, X to get iformatio about a
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationComputation Of Asymptotic Distribution For Semiparametric GMM Estimators
Computatio Of Asymptotic Distributio For Semiparametric GMM Estimators Hideiko Icimura Departmet of Ecoomics Uiversity College Lodo Cemmap UCL ad IFS April 9, 2004 Abstract A set of su ciet coditios for
More informationUniformly Consistency of the Cauchy-Transformation Kernel Density Estimation Underlying Strong Mixing
Appl. Mat. If. Sci. 7, No. L, 5-9 (203) 5 Applied Matematics & Iformatio Scieces A Iteratioal Joural c 203 NSP Natural Scieces Publisig Cor. Uiformly Cosistecy of te Caucy-Trasformatio Kerel Desity Estimatio
More information32 estimating the cumulative distribution function
32 estimatig the cumulative distributio fuctio 4.6 types of cofidece itervals/bads Let F be a class of distributio fuctios F ad let θ be some quatity of iterest, such as the mea of F or the whole fuctio
More informationEXTREMAL PROPERTIES OF HALF-SPACES FOR LOG-CONCAVE DISTRIBUTIONS. BY S. BOBKOV Syktyvkar University
Te Aals of Probability 1996, Vol. 24, No. 1, 3548 EXTREMAL PROPERTIES O HAL-SPACES OR LOG-CONCAVE DISTRIBUTIONS BY S. BOBKOV Syktyvkar Uiversity Te isoperimetric problem for log-cocave product measures
More informationEconomics 241B Relation to Method of Moments and Maximum Likelihood OLSE as a Maximum Likelihood Estimator
Ecoomics 24B Relatio to Method of Momets ad Maximum Likelihood OLSE as a Maximum Likelihood Estimator Uder Assumptio 5 we have speci ed the distributio of the error, so we ca estimate the model parameters
More informationThis exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.
Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the
More information10/ Statistical Machine Learning Homework #1 Solutions
Caregie Mello Uiversity Departet of Statistics & Data Sciece 0/36-70 Statistical Macie Learig Hoework # Solutios Proble [40 pts.] DUE: February, 08 Let X,..., X P were X i [0, ] ad P as desity p. Let p
More informationEstimation of the essential supremum of a regression function
Estimatio of the essetial supremum of a regressio fuctio Michael ohler, Adam rzyżak 2, ad Harro Walk 3 Fachbereich Mathematik, Techische Uiversität Darmstadt, Schlossgartestr. 7, 64289 Darmstadt, Germay,
More informationMath 61CM - Solutions to homework 3
Math 6CM - Solutios to homework 3 Cédric De Groote October 2 th, 208 Problem : Let F be a field, m 0 a fixed oegative iteger ad let V = {a 0 + a x + + a m x m a 0,, a m F} be the vector space cosistig
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationStat 421-SP2012 Interval Estimation Section
Stat 41-SP01 Iterval Estimatio Sectio 11.1-11. We ow uderstad (Chapter 10) how to fid poit estimators of a ukow parameter. o However, a poit estimate does ot provide ay iformatio about the ucertaity (possible
More informationRegression with quadratic loss
Regressio with quadratic loss Maxim Ragisky October 13, 2015 Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X,Y, where, as before,
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationEssential Microeconomics EXISTENCE OF EQUILIBRIUM Core ideas: continuity of excess demand functions, Fixed point theorems
Essetial Microecoomics -- 5.3 EXISTENCE OF EQUILIBRIUM Core ideas: cotiuity of excess demad fuctios, Fixed oit teorems Two commodity excage ecoomy 2 Excage ecoomy wit may commodities 5 Discotiuous demad
More informationREGRESSION WITH QUADRATIC LOSS
REGRESSION WITH QUADRATIC LOSS MAXIM RAGINSKY Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X, Y ), where, as before, X is a R d
More informationKernel density estimator
Jauary, 07 NONPARAMETRIC ERNEL DENSITY ESTIMATION I this lecture, we discuss kerel estimatio of probability desity fuctios PDF Noparametric desity estimatio is oe of the cetral problems i statistics I
More informationUniversity of Colorado Denver Dept. Math. & Stat. Sciences Applied Analysis Preliminary Exam 13 January 2012, 10:00 am 2:00 pm. Good luck!
Uiversity of Colorado Dever Dept. Math. & Stat. Scieces Applied Aalysis Prelimiary Exam 13 Jauary 01, 10:00 am :00 pm Name: The proctor will let you read the followig coditios before the exam begis, ad
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationLecture 15: Density estimation
Lecture 15: Desity estimatio Why do we estimate a desity? Suppose that X 1,...,X are i.i.d. radom variables from F ad that F is ukow but has a Lebesgue p.d.f. f. Estimatio of F ca be doe by estimatig f.
More informationStability analysis of numerical methods for stochastic systems with additive noise
Stability aalysis of umerical metods for stoctic systems wit additive oise Yosiiro SAITO Abstract Stoctic differetial equatios (SDEs) represet pysical peomea domiated by stoctic processes As for determiistic
More informationA NOTE ON INVARIANT SETS OF ITERATED FUNCTION SYSTEMS
Acta Math. Hugar., 2007 DOI: 10.1007/s10474-007-7013-6 A NOTE ON INVARIANT SETS OF ITERATED FUNCTION SYSTEMS L. L. STACHÓ ad L. I. SZABÓ Bolyai Istitute, Uiversity of Szeged, Aradi vértaúk tere 1, H-6720
More informationMath Solutions to homework 6
Math 175 - Solutios to homework 6 Cédric De Groote November 16, 2017 Problem 1 (8.11 i the book): Let K be a compact Hermitia operator o a Hilbert space H ad let the kerel of K be {0}. Show that there
More informationProbability and Statistics
ICME Refresher Course: robability ad Statistics Staford Uiversity robability ad Statistics Luyag Che September 20, 2016 1 Basic robability Theory 11 robability Spaces A probability space is a triple (Ω,
More informationME 501A Seminar in Engineering Analysis Page 1
Accurac, Stabilit ad Sstems of Equatios November 0, 07 Numerical Solutios of Ordiar Differetial Equatios Accurac, Stabilit ad Sstems of Equatios Larr Caretto Mecaical Egieerig 0AB Semiar i Egieerig Aalsis
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationJournal of Multivariate Analysis. Superefficient estimation of the marginals by exploiting knowledge on the copula
Joural of Multivariate Aalysis 102 (2011) 1315 1319 Cotets lists available at ScieceDirect Joural of Multivariate Aalysis joural homepage: www.elsevier.com/locate/jmva Superefficiet estimatio of the margials
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More informationWeek 10. f2 j=2 2 j k ; j; k 2 Zg is an orthonormal basis for L 2 (R). This function is called mother wavelet, which can be often constructed
Wee 0 A Itroductio to Wavelet regressio. De itio: Wavelet is a fuctio such that f j= j ; j; Zg is a orthoormal basis for L (R). This fuctio is called mother wavelet, which ca be ofte costructed from father
More informationExpectation and Variance of a random variable
Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio
More informationAn Adaptive Empirical Likelihood Test For Time Series Models 1
A Adaptive Empirical Likeliood Test For Time Series Models 1 By Sog Xi Ce ad Jiti Gao We exted te adaptive ad rate-optimal test of Horowitz ad Spokoiy (001 for specificatio of parametric regressio models
More informationSequences. Notation. Convergence of a Sequence
Sequeces A sequece is essetially just a list. Defiitio (Sequece of Real Numbers). A sequece of real umbers is a fuctio Z (, ) R for some real umber. Do t let the descriptio of the domai cofuse you; it
More informationn=1 a n is the sequence (s n ) n 1 n=1 a n converges to s. We write a n = s, n=1 n=1 a n
Series. Defiitios ad first properties A series is a ifiite sum a + a + a +..., deoted i short by a. The sequece of partial sums of the series a is the sequece s ) defied by s = a k = a +... + a,. k= Defiitio
More information1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable
More informationLecture 3. Properties of Summary Statistics: Sampling Distribution
Lecture 3 Properties of Summary Statistics: Samplig Distributio Mai Theme How ca we use math to justify that our umerical summaries from the sample are good summaries of the populatio? Lecture Summary
More informationLecture 10 October Minimaxity and least favorable prior sequences
STATS 300A: Theory of Statistics Fall 205 Lecture 0 October 22 Lecturer: Lester Mackey Scribe: Brya He, Rahul Makhijai Warig: These otes may cotai factual ad/or typographic errors. 0. Miimaxity ad least
More informationStatistics 511 Additional Materials
Cofidece Itervals o mu Statistics 511 Additioal Materials This topic officially moves us from probability to statistics. We begi to discuss makig ifereces about the populatio. Oe way to differetiate probability
More information1 Convergence in Probability and the Weak Law of Large Numbers
36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec
More informationMath 341 Lecture #31 6.5: Power Series
Math 341 Lecture #31 6.5: Power Series We ow tur our attetio to a particular kid of series of fuctios, amely, power series, f(x = a x = a 0 + a 1 x + a 2 x 2 + where a R for all N. I terms of a series
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More information6 Integers Modulo n. integer k can be written as k = qn + r, with q,r, 0 r b. So any integer.
6 Itegers Modulo I Example 2.3(e), we have defied the cogruece of two itegers a,b with respect to a modulus. Let us recall that a b (mod ) meas a b. We have proved that cogruece is a equivalece relatio
More informationSupplemental Material: Proofs
Proof to Theorem Supplemetal Material: Proofs Proof. Let be the miimal umber of traiig items to esure a uique solutio θ. First cosider the case. It happes if ad oly if θ ad Rak(A) d, which is a special
More information10. Comparative Tests among Spatial Regression Models. Here we revisit the example in Section 8.1 of estimating the mean of a normal random
Part III. Areal Data Aalysis 0. Comparative Tests amog Spatial Regressio Models While the otio of relative likelihood values for differet models is somewhat difficult to iterpret directly (as metioed above),
More informationPAPER : IIT-JAM 2010
MATHEMATICS-MA (CODE A) Q.-Q.5: Oly oe optio is correct for each questio. Each questio carries (+6) marks for correct aswer ad ( ) marks for icorrect aswer.. Which of the followig coditios does NOT esure
More informationLecture 27: Optimal Estimators and Functional Delta Method
Stat210B: Theoretical Statistics Lecture Date: April 19, 2007 Lecture 27: Optimal Estimators ad Fuctioal Delta Method Lecturer: Michael I. Jorda Scribe: Guilherme V. Rocha 1 Achievig Optimal Estimators
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More informationGlivenko-Cantelli Classes
CS28B/Stat24B (Sprig 2008 Statistical Learig Theory Lecture: 4 Gliveko-Catelli Classes Lecturer: Peter Bartlett Scribe: Michelle Besi Itroductio This lecture will cover Gliveko-Catelli (GC classes ad itroduce
More informationSieve Estimators: Consistency and Rates of Convergence
EECS 598: Statistical Learig Theory, Witer 2014 Topic 6 Sieve Estimators: Cosistecy ad Rates of Covergece Lecturer: Clayto Scott Scribe: Julia Katz-Samuels, Brado Oselio, Pi-Yu Che Disclaimer: These otes
More informationModel-based Variance Estimation for Systematic Sampling
ASA Sectio o Survey Researc Metods Model-based Variace Estimatio for Systematic Samplig Xiaoxi Li ad J D Opsomer Ceter for Survey Statistics ad Metodology, ad Departmet of Statistics, Iowa State Uiversity,
More informationMAS111 Convergence and Continuity
MAS Covergece ad Cotiuity Key Objectives At the ed of the course, studets should kow the followig topics ad be able to apply the basic priciples ad theorems therei to solvig various problems cocerig covergece
More informationMaximum Likelihood Estimation and Complexity Regularization
ECE90 Sprig 004 Statistical Regularizatio ad Learig Theory Lecture: 4 Maximum Likelihood Estimatio ad Complexity Regularizatio Lecturer: Rob Nowak Scribe: Pam Limpiti Review : Maximum Likelihood Estimatio
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 6 9/23/2013. Brownian motion. Introduction
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 6 9/23/203 Browia motio. Itroductio Cotet.. A heuristic costructio of a Browia motio from a radom walk. 2. Defiitio ad basic properties
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationHypothesis Testing. Evaluation of Performance of Learned h. Issues. Trade-off Between Bias and Variance
Hypothesis Testig Empirically evaluatig accuracy of hypotheses: importat activity i ML. Three questios: Give observed accuracy over a sample set, how well does this estimate apply over additioal samples?
More informationLecture 7: October 18, 2017
Iformatio ad Codig Theory Autum 207 Lecturer: Madhur Tulsiai Lecture 7: October 8, 207 Biary hypothesis testig I this lecture, we apply the tools developed i the past few lectures to uderstad the problem
More informationLearning Theory: Lecture Notes
Learig Theory: Lecture Notes Kamalika Chaudhuri October 4, 0 Cocetratio of Averages Cocetratio of measure is very useful i showig bouds o the errors of machie-learig algorithms. We will begi with a basic
More informationThis section is optional.
4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore
More informationMA131 - Analysis 1. Workbook 2 Sequences I
MA3 - Aalysis Workbook 2 Sequeces I Autum 203 Cotets 2 Sequeces I 2. Itroductio.............................. 2.2 Icreasig ad Decreasig Sequeces................ 2 2.3 Bouded Sequeces..........................
More informationKLMED8004 Medical statistics. Part I, autumn Estimation. We have previously learned: Population and sample. New questions
We have previously leared: KLMED8004 Medical statistics Part I, autum 00 How kow probability distributios (e.g. biomial distributio, ormal distributio) with kow populatio parameters (mea, variace) ca give
More informationMa 4121: Introduction to Lebesgue Integration Solutions to Homework Assignment 5
Ma 42: Itroductio to Lebesgue Itegratio Solutios to Homework Assigmet 5 Prof. Wickerhauser Due Thursday, April th, 23 Please retur your solutios to the istructor by the ed of class o the due date. You
More informationALLOCATING SAMPLE TO STRATA PROPORTIONAL TO AGGREGATE MEASURE OF SIZE WITH BOTH UPPER AND LOWER BOUNDS ON THE NUMBER OF UNITS IN EACH STRATUM
ALLOCATING SAPLE TO STRATA PROPORTIONAL TO AGGREGATE EASURE OF SIZE WIT BOT UPPER AND LOWER BOUNDS ON TE NUBER OF UNITS IN EAC STRATU Lawrece R. Erst ad Cristoper J. Guciardo Erst_L@bls.gov, Guciardo_C@bls.gov
More informationMonte Carlo Integration
Mote Carlo Itegratio I these otes we first review basic umerical itegratio methods (usig Riema approximatio ad the trapezoidal rule) ad their limitatios for evaluatig multidimesioal itegrals. Next we itroduce
More informationMath 2784 (or 2794W) University of Connecticut
ORDERS OF GROWTH PAT SMITH Math 2784 (or 2794W) Uiversity of Coecticut Date: Mar. 2, 22. ORDERS OF GROWTH. Itroductio Gaiig a ituitive feel for the relative growth of fuctios is importat if you really
More informationLecture 7: Properties of Random Samples
Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ
More informationLECTURE 8: ASYMPTOTICS I
LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece
More informationLecture 2: Monte Carlo Simulation
STAT/Q SCI 43: Itroductio to Resamplig ethods Sprig 27 Istructor: Ye-Chi Che Lecture 2: ote Carlo Simulatio 2 ote Carlo Itegratio Assume we wat to evaluate the followig itegratio: e x3 dx What ca we do?
More informationSequences I. Chapter Introduction
Chapter 2 Sequeces I 2. Itroductio A sequece is a list of umbers i a defiite order so that we kow which umber is i the first place, which umber is i the secod place ad, for ay atural umber, we kow which
More information2.1. The Algebraic and Order Properties of R Definition. A binary operation on a set F is a function B : F F! F.
CHAPTER 2 The Real Numbers 2.. The Algebraic ad Order Properties of R Defiitio. A biary operatio o a set F is a fuctio B : F F! F. For the biary operatios of + ad, we replace B(a, b) by a + b ad a b, respectively.
More informationThe Central Limit Theorem
Chapter The Cetral Limit Theorem Deote by Z the stadard ormal radom variable with desity 2π e x2 /2. Lemma.. Ee itz = e t2 /2 Proof. We use the same calculatio as for the momet geeratig fuctio: exp(itx
More informationLecture 19. sup y 1,..., yn B d n
STAT 06A: Polyomials of adom Variables Lecture date: Nov Lecture 19 Grothedieck s Iequality Scribe: Be Hough The scribes are based o a guest lecture by ya O Doell. I this lecture we prove Grothedieck s
More informationECE 901 Lecture 14: Maximum Likelihood Estimation and Complexity Regularization
ECE 90 Lecture 4: Maximum Likelihood Estimatio ad Complexity Regularizatio R Nowak 5/7/009 Review : Maximum Likelihood Estimatio We have iid observatios draw from a ukow distributio Y i iid p θ, i,, where
More information