ECE 901 Lecture 4: Estimation of Lipschitz smooth functions
|
|
- Oswald Holmes
- 6 years ago
- Views:
Transcription
1 ECE 9 Lecture 4: Estiatio of Lipschitz sooth fuctios R. Nowak 5/7/29 Cosider the followig settig. Let Y f (X) + W, where X is a rado variable (r.v.) o X [, ], W is a r.v. o Y R, idepedet of X ad satisfyig Fially let f : [, ] R be a fuctio satisfyig E[W ] ad E[W 2 ] σ 2 <. f (t) f (s) L t s, t, s [, ], () where L > is a costat. A fuctio satisfyig coditio () is said to be Lipschitz o [, ]. Notice that such a fuctio ust be cotiuous, but it is ot ecessarily differetiable. A exaple of such a fuctio is depicted i Figure (a) (a) (b) Figure : Exaple of a Lipschitz fuctio, ad our observatios settig. (a) rado saplig of f, the poits correspod to (X i, Y i ), i,..., ; (b) deteriistic saplig of f, the poits correspod to (i/, Y i ), i,...,. Note that E[Y X x] E[f (X) + W X x] E[f (x) + W X x] f (x) + E[W ] f (x). Cosider our usual setup: Estiate f usig traiig exaples {X i, Y i } i.i.d. i P XY, Y i f (X i ) + W i, i {,..., },
2 where i.i.d. eas idepedetly ad idetically distributed. Figure (a) illustrates this setup. For siplicity we will cosider a slightly differet settig. I ay applicatios we ca saple X [, ] as we like, ad ot ecessarily at rado. For exaple we ca take saples uiforly spaced o [, ] x i i, i,...,, Y i f (x i ) + W i ( ) i f + W i. We will proceed with this setup (as i Figure (b)) i the rest of the lecture. Our goal is to fid f such that E[ f f 2 ], as (here is the usual L 2 -or; i.e., f f 2 f (t) f (t) 2 dt). Let F {f : f is Lipschitz with costat L}. The Risk is defied as R(f) f f 2 f (t) f(t) 2 dt. The Expected Risk (recall that our estiator f is based o {x i, Y i } ad hece is a r.v.) is defied as Fially the Epirical Risk is defied as E[R( f )] E[ f f 2 ]. R (f) i ( f ( ) ) 2 i Y i. For the estiatio task we will use stair fuctios. Let N ad defie the class of piecewise costat fuctios F f : f(t) c j { j t< j }, c j R. F is the space of fuctios that are costat o itervals [ j,, j ), j,...,. Clearly if is rather large we ca approxiate alost ay bouded fuctio arbitrarily well. So it ake soe sese to use these classes to costruct a set of sieves. Let < 2 3 be a sequece of itegers satisfyig as. That is, for each value of there is a associated iteger value. Defie the Sieve F, F 2, F 3,..., F f : f(t) c j {t Ij,}, c j R. Fro here o we will use istead of ad istead of, for otatioal ease. Defie f(t) F to be a approxiatio of f, i particular f(t) c j {t Ij}, where c j ( ) i, N j i: i Ij f Where N j {i {,..., } : i }. Let be the uber of eleets of N j, ad assue is ot too large relative to so that >. I fact N j so as log as grows slightly slower tha we are okay. 2
3 Exercise Upper boud the error of approxiatio of f f 2. f f 2 f (t) f(t) 2 dt f (t) f(t) 2 dt f (t) c j 2 dt 2 f (t) ( i f dt ) i: i Ij ( ( )) 2 i f (t) f dt 2 ( ) i f (t) f dt 2 L dt dt. The above iplies that f f 2 as, sice as. I words, with sufficietly large we ca approxiate f to arbitrary accuracy usig odels i F (eve if the fuctios we are usig to approxiate f are ot Lipschitz!). Of course we caot copute f without kowig f, so let s use the data to fid a good odel i F. For ay f F, f c j {t Ij}, we have 2 R (f) c j {t Ij} Y i i (c j Y i ) 2. i: i Ij Let f arg i f F R (f). The f (t) ĉ j {t Ij}, where ĉ j Y i (2) N j i: i Ij 3
4 Exercise 2 Show (2). Note that E[ĉ j ] c j ad therefore E[ f (t)] f(t). Lets aalyze ow the expected risk of f : E[ f f 2 ] E[ f f + f f 2 ] f f 2 + E[ f f 2 ] + 2E[ f f, f f ] f f 2 + E[ f f 2 ] + 2 f f, E[ f f ] f f 2 + E[ f f 2 ], (3) where the fial step follows fro the fact that E[ f (t)] f(t). A couple of iportat rearks pertaiig the right-had-side of equatio (3): The first ter, f f 2, correspods to the approxiatio error, ad idicates how well ca we approxiate the fuctio f with a fuctio fro F. Clearly, the larger the class F is, the sallest we ca ake this ter. This ter is precisely the squared bias of the estiator f. The secod ter, E[ f f 2 ], is the estiatio error, the variace of our estiator. We will see that the estiatio error is sall if the class of possible estiators F is also sall. The behavior of the first ter i (3) was already studied. Cosider the other ter: [ ] E[ f f 2 ] E f(t) f (t) 2 dt E ( c j ĉ j ) 2 {t Ij}dt E ( c j ĉ j ) 2 dt for ay ɛ > provided / is large eough. Cobiig all the facts derived we have E[ f f 2 ] E [ ( c j ĉ j ) 2] 2 E (f (i/) Y i ) dt 2 E (W i ) dt σ 2 σ 2 / σ 2 / σ2 ( + ɛ)σ2, L2 2 + σ2 O ( { ax 2, }). (4) The otatio x O(y ) (that reads x is big-o y, or x is of the order of y as goes to ifiity ) eas that x Cy, where C is a positive costat ad y is a o-egative sequece. 4
5 What is the best choice of? If is sall the the approxiatio error (i.e., O(/ 2 )) is goig to be large, but the estiatio error (i.e., O(/)) is goig to be sall, ad vice-versa. This two coflictig goals provide a tradeoff that directs our choice of (as a fuctio of ). I Figure 2 we depict this tradeoff. I Figure 2(a) we cosidered a large value, ad we see that the approxiatio of f by a fuctio i the class F ca be very accurate (that is, our estiate will have a sall bias), but whe we use the easured data our estiate looks very bad (high variace). O the other had, as illustrated i Figure 2(b), usig a very sall allows our estiator to get very close to the best approxiatig fuctio i the class F, so we have a low variace estiator, but the bias of our estiator (i.e., the differece betwee f ad f ) is quite cosiderable (a) (b) Figure 2: Approxiatio ad estiatio of f (i blue) for 6. The fuctio f is depicted i gree ad the fuctio f is depicted i red. I (a)we have 6 ad i (b) we have 6. We eed to balace the two ters i the right-had-side of (4) i order to axiize the rate of decay (with ) of the expected risk. This iplies that 2 therefore /3 ad the Mea Squared Error (MSE) is E[ f f 2 ] O( 2/3 ). So the sieve F, F 2, with /3 produces a F-cosistet estiator for f F. It is iterestig to ote that the rate of decay of the MSE we obtai with this strategy caot be further iproved by usig ore sophisticated estiatio techiques (that is, 2/3 is the iiax MSE rate for this proble). Also, rather surprisigly, we are cosiderig classes of odels F that are actually ot Lipschitz, therefore our estiator of f is ot a Lipschitz fuctio, ulike f itself. 5
A Simple Regression Problem
A Siple Regression Proble R. M. Castro March 23, 2 In this brief note a siple regression proble will be introduced, illustrating clearly the bias-variance tradeoff. Let Y i f(x i ) + W i, i,..., n, where
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationA PROBABILITY PROBLEM
A PROBABILITY PROBLEM A big superarket chai has the followig policy: For every Euros you sped per buy, you ear oe poit (suppose, e.g., that = 3; i this case, if you sped 8.45 Euros, you get two poits,
More informationLecture 19. Curve fitting I. 1 Introduction. 2 Fitting a constant to measured data
Lecture 9 Curve fittig I Itroductio Suppose we are preseted with eight poits of easured data (x i, y j ). As show i Fig. o the left, we could represet the uderlyig fuctio of which these data are saples
More informationWe have also learned that, thanks to the Central Limit Theorem and the Law of Large Numbers,
Cofidece Itervals III What we kow so far: We have see how to set cofidece itervals for the ea, or expected value, of a oral probability distributio, both whe the variace is kow (usig the stadard oral,
More informationIntegrals of Functions of Several Variables
Itegrals of Fuctios of Several Variables We ofte resort to itegratios i order to deterie the exact value I of soe quatity which we are uable to evaluate by perforig a fiite uber of additio or ultiplicatio
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More information10/ Statistical Machine Learning Homework #1 Solutions
Caregie Mello Uiversity Departet of Statistics & Data Sciece 0/36-70 Statistical Macie Learig Hoework # Solutios Proble [40 pts.] DUE: February, 08 Let X,..., X P were X i [0, ] ad P as desity p. Let p
More informationJacobi symbols. p 1. Note: The Jacobi symbol does not necessarily distinguish between quadratic residues and nonresidues. That is, we could have ( a
Jacobi sybols efiitio Let be a odd positive iteger If 1, the Jacobi sybol : Z C is the costat fuctio 1 1 If > 1, it has a decopositio ( as ) a product of (ot ecessarily distict) pries p 1 p r The Jacobi
More informationLecture 10: Bounded Linear Operators and Orthogonality in Hilbert Spaces
Lecture : Bouded Liear Operators ad Orthogoality i Hilbert Spaces 34 Bouded Liear Operator Let ( X, ), ( Y, ) i i be ored liear vector spaces ad { } X Y The, T is said to be bouded if a real uber c such
More informationStatistics for Applications Fall Problem Set 7
18.650. Statistics for Applicatios Fall 016. Proble Set 7 Due Friday, Oct. 8 at 1 oo Proble 1 QQ-plots Recall that the Laplace distributio with paraeter λ > 0 is the cotiuous probaλ bility easure with
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 3 9/11/2013. Large deviations Theory. Cramér s Theorem
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 3 9//203 Large deviatios Theory. Cramér s Theorem Cotet.. Cramér s Theorem. 2. Rate fuctio ad properties. 3. Chage of measure techique.
More informationData Analysis and Statistical Methods Statistics 651
Data Aalysis ad Statistical Methods Statistics 651 http://www.stat.tau.edu/~suhasii/teachig.htl Suhasii Subba Rao Exaple The itroge cotet of three differet clover plats is give below. 3DOK1 3DOK5 3DOK7
More informationDefine a Markov chain on {1,..., 6} with transition probability matrix P =
Pla Group Work 0. The title says it all Next Tie: MCMC ad Geeral-state Markov Chais Midter Exa: Tuesday 8 March i class Hoework 4 due Thursday Uless otherwise oted, let X be a irreducible, aperiodic Markov
More informationContents Two Sample t Tests Two Sample t Tests
Cotets 3.5.3 Two Saple t Tests................................... 3.5.3 Two Saple t Tests Setup: Two Saples We ow focus o a sceario where we have two idepedet saples fro possibly differet populatios. Our
More informationMath 4707 Spring 2018 (Darij Grinberg): midterm 2 page 1. Math 4707 Spring 2018 (Darij Grinberg): midterm 2 with solutions [preliminary version]
Math 4707 Sprig 08 Darij Griberg: idter page Math 4707 Sprig 08 Darij Griberg: idter with solutios [preliiary versio] Cotets 0.. Coutig first-eve tuples......................... 3 0.. Coutig legal paths
More informationBertrand s postulate Chapter 2
Bertrad s postulate Chapter We have see that the sequece of prie ubers, 3, 5, 7,... is ifiite. To see that the size of its gaps is ot bouded, let N := 3 5 p deote the product of all prie ubers that are
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationMath 113, Calculus II Winter 2007 Final Exam Solutions
Math, Calculus II Witer 7 Fial Exam Solutios (5 poits) Use the limit defiitio of the defiite itegral ad the sum formulas to compute x x + dx The check your aswer usig the Evaluatio Theorem Solutio: I this
More informationChapter 6 Infinite Series
Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat
More informationDiscrete Mathematics: Lectures 8 and 9 Principle of Inclusion and Exclusion Instructor: Arijit Bishnu Date: August 11 and 13, 2009
Discrete Matheatics: Lectures 8 ad 9 Priciple of Iclusio ad Exclusio Istructor: Arijit Bishu Date: August ad 3, 009 As you ca observe by ow, we ca cout i various ways. Oe such ethod is the age-old priciple
More informationExponential Functions and Taylor Series
MATH 4530: Aalysis Oe Expoetial Fuctios ad Taylor Series James K. Peterso Departmet of Biological Scieces ad Departmet of Mathematical Scieces Clemso Uiversity March 29, 2017 MATH 4530: Aalysis Oe Outlie
More informationStatistics and Data Analysis in MATLAB Kendrick Kay, February 28, Lecture 4: Model fitting
Statistics ad Data Aalysis i MATLAB Kedrick Kay, kedrick.kay@wustl.edu February 28, 2014 Lecture 4: Model fittig 1. The basics - Suppose that we have a set of data ad suppose that we have selected the
More informationThe Hypergeometric Coupon Collection Problem and its Dual
Joural of Idustrial ad Systes Egieerig Vol., o., pp -7 Sprig 7 The Hypergeoetric Coupo Collectio Proble ad its Dual Sheldo M. Ross Epstei Departet of Idustrial ad Systes Egieerig, Uiversity of Souther
More informationIntroduction to Extreme Value Theory Laurens de Haan, ISM Japan, Erasmus University Rotterdam, NL University of Lisbon, PT
Itroductio to Extreme Value Theory Laures de Haa, ISM Japa, 202 Itroductio to Extreme Value Theory Laures de Haa Erasmus Uiversity Rotterdam, NL Uiversity of Lisbo, PT Itroductio to Extreme Value Theory
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationChapter 2. Asymptotic Notation
Asyptotic Notatio 3 Chapter Asyptotic Notatio Goal : To siplify the aalysis of ruig tie by gettig rid of details which ay be affected by specific ipleetatio ad hardware. [1] The Big Oh (O-Notatio) : It
More informationNotes 19 : Martingale CLT
Notes 9 : Martigale CLT Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: [Bil95, Chapter 35], [Roc, Chapter 3]. Sice we have ot ecoutered weak covergece i some time, we first recall
More informationStanford Statistics 311/Electrical Engineering 377
I. Uiversal predictio ad codig a. Gae: sequecex ofdata, adwattopredict(orcode)aswellasifwekew distributio of data b. Two versios: probabilistic ad adversarial. I either case, let p ad q be desities or
More informationLecture 11 and 12: Basic estimation theory
Lecture ad 2: Basic estimatio theory Sprig 202 - EE 94 Networked estimatio ad cotrol Prof. Kha March 2 202 I. MAXIMUM-LIKELIHOOD ESTIMATORS The maximum likelihood priciple is deceptively simple. Louis
More informationWeek 10 Spring Lecture 19. Estimation of Large Covariance Matrices: Upper bound Observe. is contained in the following parameter space,
Week 0 Sprig 009 Lecture 9. stiatio of Large Covariace Matrices: Upper boud Observe ; ; : : : ; i.i.d. fro a p-variate Gaussia distributio, N (; pp ). We assue that the covariace atrix pp = ( ij ) i;jp
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More information5.6 Binomial Multi-section Matching Transformer
4/14/21 5_6 Bioial Multisectio Matchig Trasforers 1/1 5.6 Bioial Multi-sectio Matchig Trasforer Readig Assiget: pp. 246-25 Oe way to axiize badwidth is to costruct a ultisectio Γ f that is axially flat.
More informationProbability Theory. Exercise Sheet 4. ETH Zurich HS 2017
ETH Zurich HS 2017 D-MATH, D-PHYS Prof. A.-S. Szita Coordiator Yili Wag Probability Theory Exercise Sheet 4 Exercise 4.1 Let X ) N be a sequece of i.i.d. rado variables i a probability space Ω, A, P ).
More informationLesson 10: Limits and Continuity
www.scimsacademy.com Lesso 10: Limits ad Cotiuity SCIMS Academy 1 Limit of a fuctio The cocept of limit of a fuctio is cetral to all other cocepts i calculus (like cotiuity, derivative, defiite itegrals
More informationREAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS
REAL ANALYSIS II: PROBLEM SET 1 - SOLUTIONS 18th Feb, 016 Defiitio (Lipschitz fuctio). A fuctio f : R R is said to be Lipschitz if there exists a positive real umber c such that for ay x, y i the domai
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationRegression with quadratic loss
Regressio with quadratic loss Maxim Ragisky October 13, 2015 Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X,Y, where, as before,
More informationCOMP 2804 Solutions Assignment 1
COMP 2804 Solutios Assiget 1 Questio 1: O the first page of your assiget, write your ae ad studet uber Solutio: Nae: Jaes Bod Studet uber: 007 Questio 2: I Tic-Tac-Toe, we are give a 3 3 grid, cosistig
More informationExponential Functions and Taylor Series
Expoetial Fuctios ad Taylor Series James K. Peterso Departmet of Biological Scieces ad Departmet of Mathematical Scieces Clemso Uiversity March 29, 207 Outlie Revistig the Expoetial Fuctio Taylor Series
More informationx !1! + 1!2!
4 Euler-Maclauri Suatio Forula 4. Beroulli Nuber & Beroulli Polyoial 4.. Defiitio of Beroulli Nuber Beroulli ubers B (,,3,) are defied as coefficiets of the followig equatio. x e x - B x! 4.. Expreesio
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationA string of not-so-obvious statements about correlation in the data. (This refers to the mechanical calculation of correlation in the data.
STAT-UB.003 NOTES for Wedesday 0.MAY.0 We will use the file JulieApartet.tw. We ll give the regressio of Price o SqFt, show residual versus fitted plot, save residuals ad fitted. Give plot of (Resid, Price,
More informationSeunghee Ye Ma 8: Week 5 Oct 28
Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationPRACTICE FINAL/STUDY GUIDE SOLUTIONS
Last edited December 9, 03 at 4:33pm) Feel free to sed me ay feedback, icludig commets, typos, ad mathematical errors Problem Give the precise meaig of the followig statemets i) a f) L ii) a + f) L iii)
More informationREGRESSION WITH QUADRATIC LOSS
REGRESSION WITH QUADRATIC LOSS MAXIM RAGINSKY Regressio with quadratic loss is aother basic problem studied i statistical learig theory. We have a radom couple Z = X, Y ), where, as before, X is a R d
More information(s)h(s) = K( s + 8 ) = 5 and one finite zero is located at z 1
ROOT LOCUS TECHNIQUE 93 should be desiged differetly to eet differet specificatios depedig o its area of applicatio. We have observed i Sectio 6.4 of Chapter 6, how the variatio of a sigle paraeter like
More informationLecture 8: Convergence of transformations and law of large numbers
Lecture 8: Covergece of trasformatios ad law of large umbers Trasformatio ad covergece Trasformatio is a importat tool i statistics. If X coverges to X i some sese, we ofte eed to check whether g(x ) coverges
More informationCS434a/541a: Pattern Recognition Prof. Olga Veksler. Lecture 5
CS434a/54a: Patter Recogitio Prof. Olga Veksler Lecture 5 Today Itroductio to parameter estimatio Two methods for parameter estimatio Maimum Likelihood Estimatio Bayesia Estimatio Itroducto Bayesia Decisio
More informationOptimally Sparse SVMs
A. Proof of Lemma 3. We here prove a lower boud o the umber of support vectors to achieve geeralizatio bouds of the form which we cosider. Importatly, this result holds ot oly for liear classifiers, but
More information6 Integers Modulo n. integer k can be written as k = qn + r, with q,r, 0 r b. So any integer.
6 Itegers Modulo I Example 2.3(e), we have defied the cogruece of two itegers a,b with respect to a modulus. Let us recall that a b (mod ) meas a b. We have proved that cogruece is a equivalece relatio
More informationBernoulli Polynomials Talks given at LSBU, October and November 2015 Tony Forbes
Beroulli Polyoials Tals give at LSBU, October ad Noveber 5 Toy Forbes Beroulli Polyoials The Beroulli polyoials B (x) are defied by B (x), Thus B (x) B (x) ad B (x) x, B (x) x x + 6, B (x) dx,. () B 3
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More informationLecture 6: Integration and the Mean Value Theorem. slope =
Math 8 Istructor: Padraic Bartlett Lecture 6: Itegratio ad the Mea Value Theorem Week 6 Caltech 202 The Mea Value Theorem The Mea Value Theorem abbreviated MVT is the followig result: Theorem. Suppose
More informationThe Growth of Functions. Theoretical Supplement
The Growth of Fuctios Theoretical Supplemet The Triagle Iequality The triagle iequality is a algebraic tool that is ofte useful i maipulatig absolute values of fuctios. The triagle iequality says that
More informationThe Binomial Multi-Section Transformer
4/15/2010 The Bioial Multisectio Matchig Trasforer preset.doc 1/24 The Bioial Multi-Sectio Trasforer Recall that a ulti-sectio atchig etwork ca be described usig the theory of sall reflectios as: where:
More information5.6 Binomial Multi-section Matching Transformer
4/14/2010 5_6 Bioial Multisectio Matchig Trasforers 1/1 5.6 Bioial Multi-sectio Matchig Trasforer Readig Assiget: pp. 246-250 Oe way to axiize badwidth is to costruct a ultisectio Γ f that is axially flat.
More informationMA131 - Analysis 1. Workbook 2 Sequences I
MA3 - Aalysis Workbook 2 Sequeces I Autum 203 Cotets 2 Sequeces I 2. Itroductio.............................. 2.2 Icreasig ad Decreasig Sequeces................ 2 2.3 Bouded Sequeces..........................
More informationMath 128A: Homework 1 Solutions
Math 8A: Homework Solutios Due: Jue. Determie the limits of the followig sequeces as. a) a = +. lim a + = lim =. b) a = + ). c) a = si4 +6) +. lim a = lim = lim + ) [ + ) ] = [ e ] = e 6. Observe that
More informationMath 216A Notes, Week 5
Math 6A Notes, Week 5 Scribe: Ayastassia Sebolt Disclaimer: These otes are ot early as polished (ad quite possibly ot early as correct) as a published paper. Please use them at your ow risk.. Thresholds
More informationSequences I. Chapter Introduction
Chapter 2 Sequeces I 2. Itroductio A sequece is a list of umbers i a defiite order so that we kow which umber is i the first place, which umber is i the secod place ad, for ay atural umber, we kow which
More informationMaximum Likelihood Estimation and Complexity Regularization
ECE90 Sprig 004 Statistical Regularizatio ad Learig Theory Lecture: 4 Maximum Likelihood Estimatio ad Complexity Regularizatio Lecturer: Rob Nowak Scribe: Pam Limpiti Review : Maximum Likelihood Estimatio
More informationAnalytic Continuation
Aalytic Cotiuatio The stadard example of this is give by Example Let h (z) = 1 + z + z 2 + z 3 +... kow to coverge oly for z < 1. I fact h (z) = 1/ (1 z) for such z. Yet H (z) = 1/ (1 z) is defied for
More informationIt is often useful to approximate complicated functions using simpler ones. We consider the task of approximating a function by a polynomial.
Taylor Polyomials ad Taylor Series It is ofte useful to approximate complicated fuctios usig simpler oes We cosider the task of approximatig a fuctio by a polyomial If f is at least -times differetiable
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationMATH4822E FOURIER ANALYSIS AND ITS APPLICATIONS
MATH48E FOURIER ANALYSIS AND ITS APPLICATIONS 7.. Cesàro summability. 7. Summability methods Arithmetic meas. The followig idea is due to the Italia geometer Eresto Cesàro (859-96). He shows that eve if
More informationUniversity of Colorado Denver Dept. Math. & Stat. Sciences Applied Analysis Preliminary Exam 13 January 2012, 10:00 am 2:00 pm. Good luck!
Uiversity of Colorado Dever Dept. Math. & Stat. Scieces Applied Aalysis Prelimiary Exam 13 Jauary 01, 10:00 am :00 pm Name: The proctor will let you read the followig coditios before the exam begis, ad
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationST5215: Advanced Statistical Theory
ST525: Advaced Statistical Theory Departmet of Statistics & Applied Probability Tuesday, September 7, 2 ST525: Advaced Statistical Theory Lecture : The law of large umbers The Law of Large Numbers The
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationRandom Variables, Sampling and Estimation
Chapter 1 Radom Variables, Samplig ad Estimatio 1.1 Itroductio This chapter will cover the most importat basic statistical theory you eed i order to uderstad the ecoometric material that will be comig
More informationMATH301 Real Analysis (2008 Fall) Tutorial Note #7. k=1 f k (x) converges pointwise to S(x) on E if and
MATH01 Real Aalysis (2008 Fall) Tutorial Note #7 Sequece ad Series of fuctio 1: Poitwise Covergece ad Uiform Covergece Part I: Poitwise Covergece Defiitio of poitwise covergece: A sequece of fuctios f
More informationECE Spring Prof. David R. Jackson ECE Dept. Notes 20
ECE 6341 Sprig 016 Prof. David R. Jackso ECE Dept. Notes 0 1 Spherical Wave Fuctios Cosider solvig ψ + k ψ = 0 i spherical coordiates z φ θ r y x Spherical Wave Fuctios (cot.) I spherical coordiates we
More informationOn Modeling On Minimum Description Length Modeling. M-closed
O Modelig O Miiu Descriptio Legth Modelig M M-closed M-ope Do you believe that the data geeratig echais really is i your odel class M? 7 73 Miiu Descriptio Legth Priciple o-m-closed predictive iferece
More informationIntroduction to Computational Biology Homework 2 Solution
Itroductio to Computatioal Biology Homework 2 Solutio Problem 1: Cocave gap pealty fuctio Let γ be a gap pealty fuctio defied over o-egative itegers. The fuctio γ is called sub-additive iff it satisfies
More informationIntro to Learning Theory
Lecture 1, October 18, 2016 Itro to Learig Theory Ruth Urer 1 Machie Learig ad Learig Theory Comig soo 2 Formal Framework 21 Basic otios I our formal model for machie learig, the istaces to be classified
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More information1 Review and Overview
CS9T/STATS3: Statistical Learig Theory Lecturer: Tegyu Ma Lecture #6 Scribe: Jay Whag ad Patrick Cho October 0, 08 Review ad Overview Recall i the last lecture that for ay family of scalar fuctios F, we
More informationSECTION 2.6 THE SECOND ALTERNATIVE
54 SECTION 2.6 THE SECOND ALTERNATIVE We ow discuss the probles where the Secod Alterative holds. The suppositio is that there is a otrivial solutio for L(y) =, B (y) = B 2 (y) =. The Fredhol Theores assure
More informationECE 330:541, Stochastic Signals and Systems Lecture Notes on Limit Theorems from Probability Fall 2002
ECE 330:541, Stochastic Sigals ad Systems Lecture Notes o Limit Theorems from robability Fall 00 I practice, there are two ways we ca costruct a ew sequece of radom variables from a old sequece of radom
More informationLecture Chapter 6: Convergence of Random Sequences
ECE5: Aalysis of Radom Sigals Fall 6 Lecture Chapter 6: Covergece of Radom Sequeces Dr Salim El Rouayheb Scribe: Abhay Ashutosh Doel, Qibo Zhag, Peiwe Tia, Pegzhe Wag, Lu Liu Radom sequece Defiitio A ifiite
More informationDirection: This test is worth 250 points. You are required to complete this test within 50 minutes.
Term Test October 3, 003 Name Math 56 Studet Number Directio: This test is worth 50 poits. You are required to complete this test withi 50 miutes. I order to receive full credit, aswer each problem completely
More informationDisjoint set (Union-Find)
CS124 Lecture 7 Fall 2018 Disjoit set (Uio-Fid) For Kruskal s algorithm for the miimum spaig tree problem, we foud that we eeded a data structure for maitaiig a collectio of disjoit sets. That is, we eed
More information19.1 The dictionary problem
CS125 Lecture 19 Fall 2016 19.1 The dictioary proble Cosider the followig data structural proble, usually called the dictioary proble. We have a set of ites. Each ite is a (key, value pair. Keys are i
More informationMath 341 Lecture #31 6.5: Power Series
Math 341 Lecture #31 6.5: Power Series We ow tur our attetio to a particular kid of series of fuctios, amely, power series, f(x = a x = a 0 + a 1 x + a 2 x 2 + where a R for all N. I terms of a series
More informationPower series are analytic
Power series are aalytic Horia Corea 1 1 The expoetial ad the logarithm For every x R we defie the fuctio give by exp(x) := 1 + x + x + + x + = x. If x = 0 we have exp(0) = 1. If x 0, cosider the series
More informationInverse Matrix. A meaning that matrix B is an inverse of matrix A.
Iverse Matrix Two square matrices A ad B of dimesios are called iverses to oe aother if the followig holds, AB BA I (11) The otio is dual but we ofte write 1 B A meaig that matrix B is a iverse of matrix
More informationREVIEW OF CALCULUS Herman J. Bierens Pennsylvania State University (January 28, 2004) x 2., or x 1. x j. ' ' n i'1 x i well.,y 2
REVIEW OF CALCULUS Hera J. Bieres Pesylvaia State Uiversity (Jauary 28, 2004) 1. Suatio Let x 1,x 2,...,x e a sequece of uers. The su of these uers is usually deoted y x 1 % x 2 %...% x ' j x j, or x 1
More informationLecture 15: Density estimation
Lecture 15: Desity estimatio Why do we estimate a desity? Suppose that X 1,...,X are i.i.d. radom variables from F ad that F is ukow but has a Lebesgue p.d.f. f. Estimatio of F ca be doe by estimatig f.
More informationECE534, Spring 2018: Final Exam
ECE534, Srig 2018: Fial Exam Problem 1 Let X N (0, 1) ad Y N (0, 1) be ideedet radom variables. variables V = X + Y ad W = X 2Y. Defie the radom (a) Are V, W joitly Gaussia? Justify your aswer. (b) Comute
More informationf(1), and so, if f is continuous, f(x) = f(1)x.
2.2.35: Let f be a additive fuctio. i Clearly fx = fx ad therefore f x = fx for all Z+ ad x R. Hece, for ay, Z +, f = f, ad so, if f is cotiuous, fx = fx. ii Suppose that f is bouded o soe o-epty ope set.
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationBinomial transform of products
Jauary 02 207 Bioial trasfor of products Khristo N Boyadzhiev Departet of Matheatics ad Statistics Ohio Norther Uiversity Ada OH 4580 USA -boyadzhiev@ouedu Abstract Give the bioial trasfors { b } ad {
More informationLecture 18: Sampling distributions
Lecture 18: Samplig distributios I may applicatios, the populatio is oe or several ormal distributios (or approximately). We ow study properties of some importat statistics based o a radom sample from
More information