MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013. Large Deviations for i.i.d. Random Variables
|
|
- Felix Jefferson
- 5 years ago
- Views:
Transcription
1 MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013 Large Deviatios for i.i.d. Radom Variables Cotet. Cheroff boud usig expoetial momet geeratig fuctios. Properties of a momet geeratig fuctios. Legedre trasforms. 1 Prelimiary otes The Weak Law of Large Numbers tells us that if X 1, X 2,..., is a i.i.d. sequece of radom variables with mea µ E[X 1 ] < the for every E > 0 as. X X P( µ > E) 0, But how quickly does this covergece to zero occur? We ca try to use Chebyshev iequality which says X X Var(X 1 ) P( µ > E). E 2 This suggest a decay rate of order 1 if we treat Var(X 1 ) ad E as a costat. Is this a accurate rate? Far from so... I fact if the higher momet of X 1 was fiite, for example, E[X1 2m ] <, the usig a similar boud, we could show that the decay rate is at least 1 (exercise). The goal of the large deviatio theory is to show that i may iterestig cases the decay rate is i fact expoetial: e c. The expoet c > 0 is called the large deviatios rate, ad i may cases it ca be computed explicitly or umerically. m 1
2 2 Large deviatios upper boud (Cheroff boud) Cosider a i.i.d. sequece with a commo probability distributio fuctio F (x) = P(X x), x R. Fix a value a > µ, where µ is agai a expectatio correspodig to the distributio F. We cosider probability that the average of X 1,..., X exceeds a. The WLLN tells us that this happes with probability covergig to zero as icreases, ad ow we obtai a estimate o this probability. Fix a positive parameter θ > 0. We have 1 i X i P( > a) = P( X i > a) 1 i θ X = P(e 1 i i > e θa ) θ E[e 1 i X i ] e θa E[ i eθx i ] = θa ), (e Markov iequality But recall that X θx i i s are i.i.d. Therefore E[ i e ] = (E[eθX 1 ]). Thus we obtai a upper boud ( ) 1 i X i E[e θx 1 ] P( > a). (1) e θa Of course this boud is meaigful oly if the ratio E[e θx 1 ]/e θa is less tha uity. We recogize E[e θx 1 ] as the momet geeratig fuctio of X 1 ad deote it by M(θ). For the boud to be useful, we eed E[e θx 1 ] to be at least fiite. If we could show that this ratio is less tha uity, we would be doe expoetially fast decay of the probability would be established. Similarly, suppose we wat to estimate 1 i X i P( < a), for some a < µ. Fixig ow a egative θ < 0, we obtai 1 i X i θ X P( < a) = P(e 1 i i > e θa ) ( ) M(θ), e θa 2
3 ad ow we eed to fid a egative θ such that M(θ) < e θa. I particular, we eed to focus o θ for which the momet geeratig fuctio is fiite. For this purpose let D(M) {θ : M(θ) < }. Namely D(M) is the set of values θ for which the momet geeratig fuctio is fiite. Thus we call D the domai of M. 3 Momet geeratig fuctio. Examples ad properties Let us cosider some examples of computig the momet geeratig fuctios. Expoetial distributio. Cosider a expoetially distributed radom variable X with parameter λ. The M(θ) = 0 e θx λe λx dx = λ e (λ θ)x dx. 0 1 λ θ (λ θ)x 0 Whe θ < λ this itegral is equal to e = 1/(λ θ). But whe θ λ, the itegral is ifiite. Thus the exp. momet geeratig fuctio is fiite iff θ < λ ad is M(θ) = λ/(λ θ). I this case the domai of the momet geeratig fuctio is D(M) = (, λ). Stadard Normal distributio. Whe X has stadard Normal distributio, we obtai 1 θx x 2 2π 1 x 2 2θx+θ 2 θ 2 M(θ) = E[e θx ] = e e 2 dx = e 2 dx 2π θ 2 1 (x θ)2 = e 2 e 2 dx 2π Itroducig chage of variables y = x θ we obtai that the itegral 2 is equal to 1 e y 2 dy = 1 (itegral of the desity of the stadard 2π θ Normal distributio). Therefore M(θ) = e 2 2. We see that it is always fiite ad D(M) = R. I a retrospect it is ot surprisig that i this case M(θ) is fiite for all θ. The desity of the stadard Normal distributio decays like e x2 ad 3
4 this is faster tha just expoetial growth e θx. So o matter how large is θ the overall product is fiite. Poisso distributio. Suppose X has a Poisso distributio with parameter λ. The θm λm λ M(θ) = E[e θx ] = e e m! m=0 (e θ λ) m λ = e m! m=0 e θ λ λ = e, t m m 0 m! (where we use the formula = e t ). Thus agai D(M) = R. This agai has to do with the fact that λ m /m! decays at the rate similar to 1/m! which is faster the ay expoetial growth rate e θm. We ow establish several properties of the momet geeratig fuctios. Propositio 1. The momet geeratig fuctio M(θ) of a radom variable X satisfies the followig properties: (a) M(0) = 1. If M(θ) < for some θ > 0 the M(θ ' ) < for all θ ' [0, θ]. Similarly, if M(θ) < for some θ < 0 the M(θ ' ) < for all θ ' [θ, 0]. I particular, the domai D(M) is a iterval cotaiig zero. (b) Suppose (θ 1, θ 2 ) D(M). The M(θ) as a fuctio of θ is differetiable i θ for every θ 0 (θ 1, θ 2 ), ad furthermore, d M(θ) = E[Xe θ 0 X ] <. dθ θ=θ 0 Namely, the order of differetiatio ad expectatio operators ca be chaged. Proof. Part (a) is left as a exercise. We ow establish part (b). Fix ay θ 0 (θ 1, θ 2 ) ad cosider a θ-idexed sequece of radom variables exp(θx) exp(θ 0 X) Y θ. θ θ 0 4
5 d Sice dθ exp(θx) = x exp(θx), the almost surely Y θ X exp(θ 0 X), as θ θ 0. Thus to establish the claim it suffices to show that covergece of expectatios holds as well, amely lim θ θ0 E[Y θ ] = E[X exp(θ 0 X)], ad E[X exp(θ 0 X)] <. For this purpose we will use the Domiated Covergece Theorem. Namely, we will idetify a radom variable Z such that Y θ Z almost surely i some iterval (θ 0 E, θ 0 + E), ad E[Z] <. Fix E > 0 small eough so that (θ 0 E, θ 0 + E) (θ 1, θ 2 ). Let Z = E 1 exp(θ 0 X + E X ). Usig the Taylor expasio of exp( ) fuctio, for every θ (θ 0 E, θ 0 + E), we have ( ) Y θ = exp(θ 0 X) X + (θ θ 0 )X 2 + (θ θ 0 ) 2 X (θ θ 0 ) 1 X +, 2! 3!! which gives ( ) 1 1 Y θ exp(θ 0 X) X + (θ θ 0 ) X (θ θ 0 ) 1 X + 2!! ( ) 1 1 exp(θ 0 X) X + E X E 1 X + 2!! = exp(θ 0 X)E 1 (exp(e X ) 1) exp(θ 0 X)E 1 exp(e X ) = Z. It remais to show that E[Z] <. We have E[Z] = E 1 E[exp(θ 0 X + EX)1{X 0}] + E 1 E[exp(θ 0 X EX)1{X < 0}] E 1 E[exp(θ 0 X + EX)] + E 1 E[exp(θ 0 X EX)] = E 1 M(θ 0 + E) + E 1 M(θ 0 E) <, sice E was chose so that (θ 0 E, θ 0 + E) (θ 1, θ 2 ) D(M). This completes the proof of the propositio. Problem 1. (a) Establish part (a) of Propositio 1. (b) Costruct a example of a radom variable for which the correspodig iterval is trivial {0}. Namely, M(θ) = for every θ > 0. 5
6 (c) Costruct a example of a radom variable X such that D(M) = [θ 1, θ 2 ] for some θ 1 < 0 < θ 2. Namely, the the domai D is a o-zero legth closed iterval cotaiig zero. Now suppose the i.i.d. sequece X i, i 1 is such that 0 (θ 1, θ 2 ) D(M), where M is the momet geeratig fuctio of X 1. Namely, M is fiite i a eighborhood of 0. Let a > µ = E[X 1 ]. Applyig Propositio 1, let us differetiate this ratio with respect to θ at θ = 0: d M(θ) E[X θx 1 ]e θa E[e θx 1 1 e θa ae ] = = µ a < 0. dθ eθa e2θa Note that M(θ)/e θa = 1 whe θ = 0. Therefore, for sufficietly small positive θ, the ratio M(θ)/e θa is smaller tha uity, ad (1) provides a expoetial boud o the tail probability for the average of X 1,..., X. Similarly, if a < µ, the ratio M(θ)/e θa < 1 for sufficietly small egative θ. We ow summarize our fidigs. Theorem 1 (Cheroff boud). Give a i.i.d. sequece X 1,..., X suppose the momet geeratig fuctio M(θ) is fiite i some iterval (θ 1, θ 2 ) : 0. Let a > µ = E[X 1 ]. The there exists θ > 0, such that M(θ)/e θa < 1 ad ( ) X 1 i i M(θ) P( > a). e θa Similarly, if a < µ, the there exists θ < 0, such that M(θ)/e θa < 1 ad 1 i X ( ) i M(θ) P( < a). e θa How small ca we make the ratio M(θ)/ exp(θa)? We have some freedom i choosig θ as log as E[e θx 1 ] is fiite. So we could try to fid θ which miimizes the ratio M(θ)/e θa. This is what we will do i the rest of the lecture. The surprisig coclusio of the large deviatios theory is very ofte that such a miimizig value θ exists ad is tight. Namely it provides the correct decay rate! I this case we will be able to say 1 i X i P( > a) exp( I(a, θ )), a where I(a, θ ) = log M(θ )/e θ. 6
7 4 Legedre trasforms Theorem 1 gave us a large deviatios boud (M(θ)/e θa ) which we rewrite as e (θa log M(θ)). We ow study i more detail the expoet θa log M(θ). Defiitio 1. A Legedre trasform of a radom variable X is the fuctio I(a) sup θ R (θa log M(θ)). Let us go over the examples of some distributios ad compute their correspodig Legedre trasforms. Expoetial distributio with parameter λ. Recall that M(θ) = λ/(λ θ) whe θ < λ ad M(θ) = otherwise. Therefore whe θ < λ λ I(a) = sup (aθ log ) λ θ θ = sup (aθ log λ + log(λ θ)), θ ad I(a) = otherwise. Settig the derivative of g(θ) = aθ log λ + log(λ θ) equal to zero we obtai the equatio a 1/(λ θ) = 0 which has the uique solutio θ = λ 1/a. For the boudary cases, we have aθ log λ + log(λ θ)) whe either θ λ or θ (check). Therefore I(a) = a(λ 1/a) log λ + log(λ λ + 1/a) = aλ 1 log λ + log(1/a) = aλ 1 log λ log a. The large deviatios boud the tells us that whe a > 1/λ 1 i X i (aλ 1 log λ log a) P( > a) e. (.2 log 1.2) Say λ = 1 ad a = 1.2. The the approximatio gives us e. Note that we ca obtai a exact expressio for this tail probability. Ideed, X 1, X 1 +X 2,..., X 1 +X 2 +X,... are the evets of a Poisso process with parameter λ = 1. Therefore we ca compute the probability P( 1 i X i > 1.2) exactly: it is the probability that the Poisso 7
8 process has at most 1 evets before time 1.2. Thus 1 i X i P( > 1.2) = P( X i > 1.2) 1 i (1.2) k 1.2 = e. k! 0 k 1 It is ot at all clear how revealig this expressio is. I hidsight, we kow that it is approximately e (.2 log 1.2), obtaied via large deviatios theory. Stadard Normal distributio. Recall that M(θ) = e θ 2 2 whe X 1 has the stadard Normal distributio. The expected value µ = 0. Thus we fix a > 0 ad obtai θ 2 I(a) = sup (aθ ) θ 2 a 2 =, 2 achieved at θ = a. Thus for a > 0, the large deviatios theory predicts that X i 2 1 i a P( > a) e 2. 1 i X i Agai we could compute this probability directly. We kow that is distributed as a Normal radom variable with mea zero ad variace 1/. Thus 1 i P( X i > a) = e t2 2 dt. 2π a After a little bit of techical work oe could show that this itegral is a+e domiated by its part aroud a, amely,, which is further approxa 2 a 2π imated by the value of the fuctio itself at a, amely e 2. This is cosistet with the value give by the large deviatios theory. Simply the lower order magitude term disappears i the approximatio o the 2π log scale. 8
9 Poisso distributio. Suppose X has a Poisso distributio with parame eter λ. Recall that i this case M(θ) = e θ λ λ. The I(a) = sup (aθ (e θ λ λ)). θ Settig derivative to zero we obtai θ = log(a/λ) ad I(a) = a log(a/λ) (a λ). Thus for a > λ, the large deviatios theory predicts that 1 i X i (a log(a/λ) a+λ) P( > a) e. I this case as well we ca compute the large deviatios probability explicitly. The sum X X of Poisso radom variables is also a Poisso radom variable with parameter λ. Therefore (λ) m λ P( X i > a) = e. m! 1 i m>a But agai it is hard to ifer a more explicit rate of decay usig this expressio 5 Additioal readig materials Chapter 0 of [2]. This is o-techical itroductio to the field which describes motivatio ad various applicatios of the large deviatios theory. Soft readig. Chapter 2.2 of [1]. Refereces [1] A. Dembo ad O. Zeitoui, Large deviatios techiques ad applicatios, Spriger, [2] A. Shwartz ad A. Weiss, Large deviatios for performace aalysis, Chapma ad Hall,
10 MIT OpeCourseWare J / 6.265J Advaced Stochastic Processes Fall 2013 For iformatio about citig these materials or our Terms of Use, visit:
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 3 9/11/2013. Large deviations Theory. Cramér s Theorem
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 3 9//203 Large deviatios Theory. Cramér s Theorem Cotet.. Cramér s Theorem. 2. Rate fuctio ad properties. 3. Chage of measure techique.
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationThis section is optional.
4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 6 9/23/2013. Brownian motion. Introduction
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 6 9/23/203 Browia motio. Itroductio Cotet.. A heuristic costructio of a Browia motio from a radom walk. 2. Defiitio ad basic properties
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece
More informationStochastic Simulation
Stochastic Simulatio 1 Itroductio Readig Assigmet: Read Chapter 1 of text. We shall itroduce may of the key issues to be discussed i this course via a couple of model problems. Model Problem 1 (Jackso
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More informationFall 2013 MTH431/531 Real analysis Section Notes
Fall 013 MTH431/531 Real aalysis Sectio 8.1-8. Notes Yi Su 013.11.1 1. Defiitio of uiform covergece. We look at a sequece of fuctios f (x) ad study the coverget property. Notice we have two parameters
More information1 Convergence in Probability and the Weak Law of Large Numbers
36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall Midterm Solutions
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.65/5.070J Fall 0 Midterm Solutios Problem Suppose a radom variable X is such that P(X > ) = 0 ad P(X > E) > 0 for every E > 0. Recall that the large deviatios rate
More informationEE 4TM4: Digital Communications II Probability Theory
1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair
More informationApplication to Random Graphs
A Applicatio to Radom Graphs Brachig processes have a umber of iterestig ad importat applicatios. We shall cosider oe of the most famous of them, the Erdős-Réyi radom graph theory. 1 Defiitio A.1. Let
More informationMIT Spring 2016
MIT 18.655 Dr. Kempthore Sprig 2016 1 MIT 18.655 Outlie 1 2 MIT 18.655 Beroulli s Weak Law of Large Numbers X 1, X 2,... iid Beroulli(θ). S i=1 = X i Biomial(, θ). S P θ. Proof: Apply Chebychev s Iequality,
More informationMATH 320: Probability and Statistics 9. Estimation and Testing of Parameters. Readings: Pruim, Chapter 4
MATH 30: Probability ad Statistics 9. Estimatio ad Testig of Parameters Estimatio ad Testig of Parameters We have bee dealig situatios i which we have full kowledge of the distributio of a radom variable.
More informationProbability 2 - Notes 10. Lemma. If X is a random variable and g(x) 0 for all x in the support of f X, then P(g(X) 1) E[g(X)].
Probability 2 - Notes 0 Some Useful Iequalities. Lemma. If X is a radom variable ad g(x 0 for all x i the support of f X, the P(g(X E[g(X]. Proof. (cotiuous case P(g(X Corollaries x:g(x f X (xdx x:g(x
More information1.010 Uncertainty in Engineering Fall 2008
MIT OpeCourseWare http://ocw.mit.edu.00 Ucertaity i Egieerig Fall 2008 For iformatio about citig these materials or our Terms of Use, visit: http://ocw.mit.edu.terms. .00 - Brief Notes # 9 Poit ad Iterval
More informationBasics of Probability Theory (for Theory of Computation courses)
Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. oded.goldreich@weizma.ac.il November 24, 2008 Preface.
More informationUnbiased Estimation. February 7-12, 2008
Ubiased Estimatio February 7-2, 2008 We begi with a sample X = (X,..., X ) of radom variables chose accordig to oe of a family of probabilities P θ where θ is elemet from the parameter space Θ. For radom
More informationRates of Convergence by Moduli of Continuity
Rates of Covergece by Moduli of Cotiuity Joh Duchi: Notes for Statistics 300b March, 017 1 Itroductio I this ote, we give a presetatio showig the importace, ad relatioship betwee, the modulis of cotiuity
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationLimit Theorems. Convergence in Probability. Let X be the number of heads observed in n tosses. Then, E[X] = np and Var[X] = np(1-p).
Limit Theorems Covergece i Probability Let X be the umber of heads observed i tosses. The, E[X] = p ad Var[X] = p(-p). L O This P x p NM QP P x p should be close to uity for large if our ituitio is correct.
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More information1 of 7 7/16/2009 6:06 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 6. Order Statistics Defiitios Suppose agai that we have a basic radom experimet, ad that X is a real-valued radom variable
More information(A sequence also can be thought of as the list of function values attained for a function f :ℵ X, where f (n) = x n for n 1.) x 1 x N +k x N +4 x 3
MATH 337 Sequeces Dr. Neal, WKU Let X be a metric space with distace fuctio d. We shall defie the geeral cocept of sequece ad limit i a metric space, the apply the results i particular to some special
More informationLecture 2: Concentration Bounds
CSE 52: Desig ad Aalysis of Algorithms I Sprig 206 Lecture 2: Cocetratio Bouds Lecturer: Shaya Oveis Ghara March 30th Scribe: Syuzaa Sargsya Disclaimer: These otes have ot bee subjected to the usual scrutiy
More informationLearning Theory: Lecture Notes
Learig Theory: Lecture Notes Kamalika Chaudhuri October 4, 0 Cocetratio of Averages Cocetratio of measure is very useful i showig bouds o the errors of machie-learig algorithms. We will begi with a basic
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationLecture 12: September 27
36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.
More informationChapter 7 Isoperimetric problem
Chapter 7 Isoperimetric problem Recall that the isoperimetric problem (see the itroductio its coectio with ido s proble) is oe of the most classical problem of a shape optimizatio. It ca be formulated
More informationLecture 2 February 8, 2016
MIT 6.854/8.45: Advaced Algorithms Sprig 206 Prof. Akur Moitra Lecture 2 February 8, 206 Scribe: Calvi Huag, Lih V. Nguye I this lecture, we aalyze the problem of schedulig equal size tasks arrivig olie
More informationJanuary 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS
Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we
More informationST5215: Advanced Statistical Theory
ST525: Advaced Statistical Theory Departmet of Statistics & Applied Probability Tuesday, September 7, 2 ST525: Advaced Statistical Theory Lecture : The law of large umbers The Law of Large Numbers The
More informationMath 113 Exam 3 Practice
Math Exam Practice Exam will cover.-.9. This sheet has three sectios. The first sectio will remid you about techiques ad formulas that you should kow. The secod gives a umber of practice questios for you
More informationLECTURE 8: ASYMPTOTICS I
LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationThis exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.
Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the
More informationMath 113, Calculus II Winter 2007 Final Exam Solutions
Math, Calculus II Witer 7 Fial Exam Solutios (5 poits) Use the limit defiitio of the defiite itegral ad the sum formulas to compute x x + dx The check your aswer usig the Evaluatio Theorem Solutio: I this
More informationSDS 321: Introduction to Probability and Statistics
SDS 321: Itroductio to Probability ad Statistics Lecture 23: Cotiuous radom variables- Iequalities, CLT Puramrita Sarkar Departmet of Statistics ad Data Sciece The Uiversity of Texas at Austi www.cs.cmu.edu/
More informationIntroduction to Extreme Value Theory Laurens de Haan, ISM Japan, Erasmus University Rotterdam, NL University of Lisbon, PT
Itroductio to Extreme Value Theory Laures de Haa, ISM Japa, 202 Itroductio to Extreme Value Theory Laures de Haa Erasmus Uiversity Rotterdam, NL Uiversity of Lisbo, PT Itroductio to Extreme Value Theory
More informationMathematics 170B Selected HW Solutions.
Mathematics 17B Selected HW Solutios. F 4. Suppose X is B(,p). (a)fidthemometgeeratigfuctiom (s)of(x p)/ p(1 p). Write q = 1 p. The MGF of X is (pe s + q), sice X ca be writte as the sum of idepedet Beroulli
More informationExpectation and Variance of a random variable
Chapter 11 Expectatio ad Variace of a radom variable The aim of this lecture is to defie ad itroduce mathematical Expectatio ad variace of a fuctio of discrete & cotiuous radom variables ad the distributio
More informationMath 525: Lecture 5. January 18, 2018
Math 525: Lecture 5 Jauary 18, 2018 1 Series (review) Defiitio 1.1. A sequece (a ) R coverges to a poit L R (writte a L or lim a = L) if for each ǫ > 0, we ca fid N such that a L < ǫ for all N. If the
More informationAdditional Notes on Power Series
Additioal Notes o Power Series Mauela Girotti MATH 37-0 Advaced Calculus of oe variable Cotets Quick recall 2 Abel s Theorem 2 3 Differetiatio ad Itegratio of Power series 4 Quick recall We recall here
More informationAdvanced Analysis. Min Yan Department of Mathematics Hong Kong University of Science and Technology
Advaced Aalysis Mi Ya Departmet of Mathematics Hog Kog Uiversity of Sciece ad Techology September 3, 009 Cotets Limit ad Cotiuity 7 Limit of Sequece 8 Defiitio 8 Property 3 3 Ifiity ad Ifiitesimal 8 4
More informationMa 530 Introduction to Power Series
Ma 530 Itroductio to Power Series Please ote that there is material o power series at Visual Calculus. Some of this material was used as part of the presetatio of the topics that follow. What is a Power
More informationChapter 10: Power Series
Chapter : Power Series 57 Chapter Overview: Power Series The reaso series are part of a Calculus course is that there are fuctios which caot be itegrated. All power series, though, ca be itegrated because
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More informationStatistical Theory MT 2009 Problems 1: Solution sketches
Statistical Theory MT 009 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. (a) Let 0 < θ < ad put f(x, θ) = ( θ)θ x ; x = 0,,,... (b) (c) where
More informationAn Introduction to Asymptotic Theory
A Itroductio to Asymptotic Theory Pig Yu School of Ecoomics ad Fiace The Uiversity of Hog Kog Pig Yu (HKU) Asymptotic Theory 1 / 20 Five Weapos i Asymptotic Theory Five Weapos i Asymptotic Theory Pig Yu
More informationAda Boost, Risk Bounds, Concentration Inequalities. 1 AdaBoost and Estimates of Conditional Probabilities
CS8B/Stat4B Sprig 008) Statistical Learig Theory Lecture: Ada Boost, Risk Bouds, Cocetratio Iequalities Lecturer: Peter Bartlett Scribe: Subhrasu Maji AdaBoost ad Estimates of Coditioal Probabilities We
More informationNotes 5 : More on the a.s. convergence of sums
Notes 5 : More o the a.s. covergece of sums Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: Dur0, Sectios.5; Wil9, Sectio 4.7, Shi96, Sectio IV.4, Dur0, Sectio.. Radom series. Three-series
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationIIT JAM Mathematical Statistics (MS) 2006 SECTION A
IIT JAM Mathematical Statistics (MS) 6 SECTION A. If a > for ad lim a / L >, the which of the followig series is ot coverget? (a) (b) (c) (d) (d) = = a = a = a a + / a lim a a / + = lim a / a / + = lim
More informationIntroduction to Probability. Ariel Yadin
Itroductio to robability Ariel Yadi Lecture 2 *** Ja. 7 ***. Covergece of Radom Variables As i the case of sequeces of umbers, we would like to talk about covergece of radom variables. There are may ways
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More informationThe standard deviation of the mean
Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationProduct measures, Tonelli s and Fubini s theorems For use in MAT3400/4400, autumn 2014 Nadia S. Larsen. Version of 13 October 2014.
Product measures, Toelli s ad Fubii s theorems For use i MAT3400/4400, autum 2014 Nadia S. Larse Versio of 13 October 2014. 1. Costructio of the product measure The purpose of these otes is to preset the
More informationIntegrable Functions. { f n } is called a determining sequence for f. If f is integrable with respect to, then f d does exist as a finite real number
MATH 532 Itegrable Fuctios Dr. Neal, WKU We ow shall defie what it meas for a measurable fuctio to be itegrable, show that all itegral properties of simple fuctios still hold, ad the give some coditios
More informationStatistical Theory MT 2008 Problems 1: Solution sketches
Statistical Theory MT 008 Problems : Solutio sketches. Which of the followig desities are withi a expoetial family? Explai your reasoig. a) Let 0 < θ < ad put fx, θ) = θ)θ x ; x = 0,,,... b) c) where α
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationIt is often useful to approximate complicated functions using simpler ones. We consider the task of approximating a function by a polynomial.
Taylor Polyomials ad Taylor Series It is ofte useful to approximate complicated fuctios usig simpler oes We cosider the task of approximatig a fuctio by a polyomial If f is at least -times differetiable
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2018 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More information1 Approximating Integrals using Taylor Polynomials
Seughee Ye Ma 8: Week 7 Nov Week 7 Summary This week, we will lear how we ca approximate itegrals usig Taylor series ad umerical methods. Topics Page Approximatig Itegrals usig Taylor Polyomials. Defiitios................................................
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationReview Questions, Chapters 8, 9. f(y) = 0, elsewhere. F (y) = f Y(1) = n ( e y/θ) n 1 1 θ e y/θ = n θ e yn
Stat 366 Lab 2 Solutios (September 2, 2006) page TA: Yury Petracheko, CAB 484, yuryp@ualberta.ca, http://www.ualberta.ca/ yuryp/ Review Questios, Chapters 8, 9 8.5 Suppose that Y, Y 2,..., Y deote a radom
More informationEntropy Rates and Asymptotic Equipartition
Chapter 29 Etropy Rates ad Asymptotic Equipartitio Sectio 29. itroduces the etropy rate the asymptotic etropy per time-step of a stochastic process ad shows that it is well-defied; ad similarly for iformatio,
More informationECE 330:541, Stochastic Signals and Systems Lecture Notes on Limit Theorems from Probability Fall 2002
ECE 330:541, Stochastic Sigals ad Systems Lecture Notes o Limit Theorems from robability Fall 00 I practice, there are two ways we ca costruct a ew sequece of radom variables from a old sequece of radom
More informationLecture 6 Chi Square Distribution (χ 2 ) and Least Squares Fitting
Lecture 6 Chi Square Distributio (χ ) ad Least Squares Fittig Chi Square Distributio (χ ) Suppose: We have a set of measuremets {x 1, x, x }. We kow the true value of each x i (x t1, x t, x t ). We would
More informationENGI Series Page 6-01
ENGI 3425 6 Series Page 6-01 6. Series Cotets: 6.01 Sequeces; geeral term, limits, covergece 6.02 Series; summatio otatio, covergece, divergece test 6.03 Stadard Series; telescopig series, geometric series,
More informationSeunghee Ye Ma 8: Week 5 Oct 28
Week 5 Summary I Sectio, we go over the Mea Value Theorem ad its applicatios. I Sectio 2, we will recap what we have covered so far this term. Topics Page Mea Value Theorem. Applicatios of the Mea Value
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationSection 11.8: Power Series
Sectio 11.8: Power Series 1. Power Series I this sectio, we cosider geeralizig the cocept of a series. Recall that a series is a ifiite sum of umbers a. We ca talk about whether or ot it coverges ad i
More informationMath 312 Lecture Notes One Dimensional Maps
Math 312 Lecture Notes Oe Dimesioal Maps Warre Weckesser Departmet of Mathematics Colgate Uiversity 21-23 February 25 A Example We begi with the simplest model of populatio growth. Suppose, for example,
More informationLecture 4: Unique-SAT, Parity-SAT, and Approximate Counting
Advaced Complexity Theory Sprig 206 Lecture 4: Uique-SAT, Parity-SAT, ad Approximate Coutig Prof. Daa Moshkovitz Scribe: Aoymous Studet Scribe Date: Fall 202 Overview I this lecture we begi talkig about
More informationCentral Limit Theorem using Characteristic functions
Cetral Limit Theorem usig Characteristic fuctios RogXi Guo MAT 477 Jauary 20, 2014 RogXi Guo (2014 Cetral Limit Theorem usig Characteristic fuctios Jauary 20, 2014 1 / 15 Itroductio study a radom variable
More informationMA131 - Analysis 1. Workbook 3 Sequences II
MA3 - Aalysis Workbook 3 Sequeces II Autum 2004 Cotets 2.8 Coverget Sequeces........................ 2.9 Algebra of Limits......................... 2 2.0 Further Useful Results........................
More informationDiscrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22
CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first
More informationMa 530 Infinite Series I
Ma 50 Ifiite Series I Please ote that i additio to the material below this lecture icorporated material from the Visual Calculus web site. The material o sequeces is at Visual Sequeces. (To use this li
More information6a Time change b Quadratic variation c Planar Brownian motion d Conformal local martingales e Hints to exercises...
Tel Aviv Uiversity, 28 Browia motio 59 6 Time chage 6a Time chage..................... 59 6b Quadratic variatio................. 61 6c Plaar Browia motio.............. 64 6d Coformal local martigales............
More informationTopic 9: Sampling Distributions of Estimators
Topic 9: Samplig Distributios of Estimators Course 003, 2016 Page 0 Samplig distributios of estimators Sice our estimators are statistics (particular fuctios of radom variables), their distributio ca be
More informationMath 2784 (or 2794W) University of Connecticut
ORDERS OF GROWTH PAT SMITH Math 2784 (or 2794W) Uiversity of Coecticut Date: Mar. 2, 22. ORDERS OF GROWTH. Itroductio Gaiig a ituitive feel for the relative growth of fuctios is importat if you really
More informationWe are mainly going to be concerned with power series in x, such as. (x)} converges - that is, lims N n
Review of Power Series, Power Series Solutios A power series i x - a is a ifiite series of the form c (x a) =c +c (x a)+(x a) +... We also call this a power series cetered at a. Ex. (x+) is cetered at
More informationSequences. Notation. Convergence of a Sequence
Sequeces A sequece is essetially just a list. Defiitio (Sequece of Real Numbers). A sequece of real umbers is a fuctio Z (, ) R for some real umber. Do t let the descriptio of the domai cofuse you; it
More informationECONOMETRIC THEORY. MODULE XIII Lecture - 34 Asymptotic Theory and Stochastic Regressors
ECONOMETRIC THEORY MODULE XIII Lecture - 34 Asymptotic Theory ad Stochastic Regressors Dr. Shalabh Departmet of Mathematics ad Statistics Idia Istitute of Techology Kapur Asymptotic theory The asymptotic
More information3. Z Transform. Recall that the Fourier transform (FT) of a DT signal xn [ ] is ( ) [ ] = In order for the FT to exist in the finite magnitude sense,
3. Z Trasform Referece: Etire Chapter 3 of text. Recall that the Fourier trasform (FT) of a DT sigal x [ ] is ω ( ) [ ] X e = j jω k = xe I order for the FT to exist i the fiite magitude sese, S = x [
More informationFINAL EXAMINATION IN FOUNDATION OF ANALYSIS (TMA4225)
Norwegia Uiversity of Sciece ad Techology Departmet of Mathematical Scieces Page of 7 Cotact durig exam: Eugeia Maliikova (735) 52 57 FINAL EXAMINATION IN FOUNDATION OF ANALYSIS (TMA4225) Moday, December
More informationAsymptotic distribution of products of sums of independent random variables
Proc. Idia Acad. Sci. Math. Sci. Vol. 3, No., May 03, pp. 83 9. c Idia Academy of Scieces Asymptotic distributio of products of sums of idepedet radom variables YANLING WANG, SUXIA YAO ad HONGXIA DU ollege
More informationLecture Notes 15 Hypothesis Testing (Chapter 10)
1 Itroductio Lecture Notes 15 Hypothesis Testig Chapter 10) Let X 1,..., X p θ x). Suppose we we wat to kow if θ = θ 0 or ot, where θ 0 is a specific value of θ. For example, if we are flippig a coi, we
More informationLecture 4 February 16, 2016
MIT 6.854/18.415: Advaced Algorithms Sprig 16 Prof. Akur Moitra Lecture 4 February 16, 16 Scribe: Be Eysebach, Devi Neal 1 Last Time Cosistet Hashig - hash fuctios that evolve well Radom Trees - routig
More informationInformation Theory and Statistics Lecture 4: Lempel-Ziv code
Iformatio Theory ad Statistics Lecture 4: Lempel-Ziv code Łukasz Dębowski ldebowsk@ipipa.waw.pl Ph. D. Programme 203/204 Etropy rate is the limitig compressio rate Theorem For a statioary process (X i)
More informationLecture 4. We also define the set of possible values for the random walk as the set of all x R d such that P(S n = x) > 0 for some n.
Radom Walks ad Browia Motio Tel Aviv Uiversity Sprig 20 Lecture date: Mar 2, 20 Lecture 4 Istructor: Ro Peled Scribe: Lira Rotem This lecture deals primarily with recurrece for geeral radom walks. We preset
More informationLecture 20: Multivariate convergence and the Central Limit Theorem
Lecture 20: Multivariate covergece ad the Cetral Limit Theorem Covergece i distributio for radom vectors Let Z,Z 1,Z 2,... be radom vectors o R k. If the cdf of Z is cotiuous, the we ca defie covergece
More information