Chapter 5. Inequalities. 5.1 The Markov and Chebyshev inequalities
|
|
- Daniel Augustine Ross
- 5 years ago
- Views:
Transcription
1 Chapter 5 Iequalities 5.1 The Markov ad Chebyshev iequalities As you have probably see o today s frot page: every perso i the upper teth percetile ears at least 1 times more tha the average salary. I other words, if you pick up a radom perso, the probability that his salary is more tha 1 times the average salary is more tha 0.1. Put ito a formal settig, deote by the set of people edowed with a uiform probability ad let X(!) be the perso s salary, the P(X > 1E[X]) 0.1. The followig theorem will show that this is ot possible. Theorem 5.1 (Markov iequality) Let X be a radom variable assumig oegative values, i.e., X(!) 0 for every!. The for every a 0, P(X a) E[X] a. Commet: Adrey Adreyevich Markov ( ) was a Russia mathematicia otably kow for his work o Stochastic processes.
2 10 Chapter 5 Commet: Note first that this is a vacuous statemet for a < E[X]. For a > E[X] this iequality limits the probability that X assumes values larger tha its expected value. This is the first time i this course that we derive a iequality. Iequalities, i geeral, are a importat tool i aalysis, where estimates (rather tha exact idetities) are ofte eeded. The stregth of such a iequality is that it holds for every radom variable. As a result, this estimate may be far from beig tight. Proof : Let A be the evet that X is at least a, that is A = {! X(!) a}. Defie a ew radom variable Y = ai A, where I A is the idicator of A. That is, a X(!) a Y(!) = 0 otherwise. Clearly, X Y, so by the mootoicity of expectatio E[X] E[Y] = a E[I A ] = ap(a) = ap(x a), where we used the fact that I A is a Beroulli variable, hece, its expectatio equals the probability that it is equal to 1. A corollary of Markov s iequality is aother iequality due to Chebyshev: Theorem 5. (Chebyshev s iequality) Let X be a radom variable with expected value µ ad variace. The, for every a > 0 P(X µ a) a. Commet: Pafuty Lvovich Chebyshev ( ) was a Russia mathematicia kow for may cotributios; he was Markov s advisor. Commet: If we write a = k, this theorem states that the probability that a radom variable assumes a value whose absolute distace from its expected value is more tha k times its stadard deviatio is at most 1k, P(X µ k ) 1 k.
3 Iequalities 11 Proof : Defie Y = X µ. Y is a o-egative radom variable, satisfyig, by defiitio, E[Y] =. Applyig Markov s iequality for Y, P(X µ a) = P(Y a ) E[Y] a = a. Example: O average, Joh Doe driks 5 liters of wie every week. What ca be said about the probability that he driks more tha 50 liters of wie o a give week? Here we apply the Markov iequality. If X is the amout of wie Joh Doe driks o a give week, the P(X > 50) E[X] 50 = 1. Note that this may be a very loose estimate; it may well be that Joe Doe driks exactly 5 liters of wie every week, i which case this probability is zero. Example: Let X B (, 1), The, E[X] = ad Var[X] = 4. If you toss a fair coi 10 6 times, the distributio of the umber of Heads is X B (10 6, 1); the probability of gettig betwee 495,000 ad 505,000 Heads ca be bouded by P(X E[X] < 5000) = 1 P(X E[X] 5000) = 0.99 Sice a biomial variable B (, p) ca be represeted as a sum of Beroulli variables, the followig geeralizatio is called for. Let X 1, X,...,X by idepedet ad idetically distributed radom variables, with E[X 1 ] = µ ad Var[X 1 ] =. Defie X = k=1 X k.
4 1 Chapter 5 The, by the liearity of the expectatio ad the idepedece of the radom variables, E[X] = µ ad Var[X] =. Chebyshev s iequality yields P(X µ a) a. This iequality starts gettig useful whe a >. For further referece, we ote that i the particular case where µ = 0, we get a fortiori that for every a > 0, P(X a) a. 5. Hoe dig s iequality We have just see that if we cosider a sum of idepedet radom variables, we ca use Chebyshev s iequality to boud the probability that this sum deviates by much from its expectatio. We ow ask: is this result tight? It turs out there are much better bouds that we ca get i this settig. Theorem 5.3 (Hoeffdig s iequality) Let X 1, X,...,X be idepedet radom variables (ot ecessarily idetically-distributed) satisfyig E[X i ] = 0 ad X i 1 for all 1 i (that is, for every i ad every!, X i (!) 1). The, for every a > 0 P X i a exp a. Recall the defiitio of the momet geeratig fuctio of X M X (t) = E[e tx ]. The proof will use momet geeratig fuctios ad Markov s iequality. We eed the followig lemma. Lemma 5.1 Let X be a radom variable satisfyig E[X] = 0 ad X i 1. The for all t R M X (t) exp t.
5 Iequalities 13 Proof : Fix t R, ad cosider the fuctio f (x) = e tx. This fuctio is covex (the secod derivative of f is everywhere positive), meaig that ay lie segmet betwee two poits o its graph lies above the graph; i.e., for every a, b R ad 0 1, f ( a + (1 )b) f (a) + (1 ) f (b). I particular, takig a = 1 ad b = 1 we get that for every t ad every 0 1, e t( (1 )) e t + (1 )e t. Settig x = 1, i.e., = (1 + x) ad 1 = (1 x), we get that for ay 1 x 1, e tx 1 + x et + 1 x e t = et + e t Sice X is always betwee 1 ad 1, it follows that e tx et + e t + X et e t, + x et e t. which is a iequality betwee radom variables. By mootoicity of expectatio, usig the liearity of the expectatio ad the fact that E[X] = 0, It remais to show that for all t R M X (t) = E[e tx ] et + e t. e t + e t exp t. This is doe by compariso of the Taylor series aroud 0, e t + e t = 1 t k + ( t) k k=0 k! = `=0 t ` (`)! `=0 t ` ``! = (t )` `=0 `! = exp t, where we used the fact that (`)! ``!. Proof :(of Hoe dig s iequality) Let X = X i. First ote that M X (t) = E[exp(t X i )] = E[ e tx i ] = E[e tx i ] = M Xi (t)
6 14 Chapter 5 where we used Propositio 4.3 ad the idepedece of the radom variables. Usig the above lemma we get that M X (t) exp t. For t > 0 the evet X a is the same as the evet e tx e ta. Usig Markov s iequality we get that for ay t > 0 P(X a) = P(e tx e ta ) E[etX ] e ta = M X(t) e ta exp t ta. The right-had side depeds o the parameter t, whereas the left-had side does ot; to get a tighter boud, we may choose t such to miimize the right-had side. The miimum is achieved whe t = a, hece P(X a) exp (a) (a)a = exp a. Commet: Wassily Hoe dig ( ) was a Fiish statisticia. The iequality amed after him was proved i Commet: We have stated Hoe dig s iequality uder the assumptios that the radom variables are bouded by 1 i absolute value ad the expectatio is 0. I geeral, if we have bouded radom variables we ca apply a a e trasformatio to get a radom variables which satisfy the coditio of the theorem. For example, if E[X i ] = µ ad X i b, the Y i = X i µ b + µ satisfies, E[Y i ] = 0 ad Y i X i + µ b + µ 1. Example: Goig back to the last example from the previous sectio, if X i are idepedet Beroulli 1 radom variables, the X = X i B, 1. We defie
7 Iequalities 15 ew radom variables Y i = X i 1 so that Y i 1 ad E[Y i ] = 0. Furthermore, set Y = Y i, so that Y = X. The evet X + 5 equals the evet Y 10. Hoe dig s iequality gives us the boud P Y 10 exp 100 = e 50. A similar boud holds for X 5. Put together, P X 5 e 50. So, if you tossed a fair coi 10 6 times, the probability that you get betwee 495, 000 ad 505, 000 heads is at least 1 e Quite a improvemet over our previous boud of 99 percet! 5.3 Jese s iequality Recall that a fuctio f R R is called covex if its graph betwee every two poits lies uder the secat, i.e., if for every x, y R ad 0 < < 1, f ( x + (1 )y) f (x) + (1 ) f (y). Propositio 5.1 (Jese s iequality) If g is a covex real-valued fuctio ad X is a real-valued radom variable, the provided that the expectatios exist. g(e[x]) E[g(X)], Proof : Let s start with a proof uder the additioal assumptio that g is twice di eretiable. Sice g is covex, it follows that g (x) 0 for all x.
8 16 Chapter 5 Taylor expadig g about µ = E[X], g(x) = g(µ) + g (µ)(x µ) + 1 g ( )(x µ) for some. The last term is o-egative, so we get that for all x g(x) g(µ) + g (µ)(x µ), ad as a iequality betwee radom variables: By the mootoicity of expectatio, g(x) g(µ) + g (µ)(x µ). E[g(X)] E[g(µ) + g (µ)(x µ)] = g(µ) + g (µ)(e[x] µ) = g(e[x]) which is precisely what we eed to show. What about the more geeral case? Ay covex fuctio is cotiuous, ad has oe-sided derivatives with For every m [g (µ), g +(µ)] g g(x) g(y) g(x) g(y) (x) = lim lim = g y x x y y x x y +(x). g(x) g(µ) + m(x µ), so the same proof holds with m replacig g (µ). Commet: Jese s iequality is valid also for covex fuctios of several variables. Example: Sice exp is a covex fuctio, exp(t E[X]) E[e tx ] = M X (t), or, E[X] 1 t log M X(t), for all t > 0.
9 Iequalities 17 Example: Cosider a discrete radom variable X assumig the positive values x 1,...,x with equal probability 1. Jese s iequality for g(x) = log(x) gives, log 1 x i 1 Reversig sigs, ad expoetiatig, we get 1 log(x i ) = log x i 1 x i, x 1 i. which is the classical arithmetic mea-geometric mea iequality. I fact, this iequality ca be geeralized for arbitrary distributios, p X (x i ) = p i, yieldig p i x i x p i i. 5.4 Kolmogorov s iequality (Not taught i 016-7) The Kolmogorov iequality may first seem to be of similar flavor as Chebyshev s iequality, but it is cosiderably stroger. I have decided to iclude it here because its proof ivolves some iterestig subtleties. First, a lemma: Lemma 5. If X, Y, Z are radom variables such that Y is idepedet of X ad Z, the E[XYZ] = E[XZ] E[Y]. Proof : The fact that Y is idepedet of both X, Z implies that (i the case of discrete variables), p X,Y,Z (x, y, z) = p X,Z (x, z)p Y (y).
10 18 Chapter 5 Now, for every z, E[XYZ = z] = xy p X,YZ (x, yz) x,y = xy p X,Y,Z(x, y, z) x,y p Z (z) = xy p X,Z(x, z)p Y (y) x,y p Z (z) = y yp Y (y) x p X,Z(x, z) x p Z (z) = E[Y] E[XZ = z]. Theorem 5.4 (Kolmogorov s iequality) Let X 1,...,X be idepedet radom variables such that E[X k ] = 0 ad Var[X k ] = k <. The, for all a > 0, P max 1 k X 1 + +X k a 1 a i. Commet: For = 1 this is othig but the Chebyshev iequality. For > 1 it would still be Chebyshev s iequality if the maximum over 1 k was replaced by k =, sice by idepedece Var[X 1 + +X ] = i. Proof : We itroduce the otatio S k = X 1 + +X k. This theorem is cocered with the probability that S k > a for some k. We defie the radom variable N(!) to be the smallest iteger k for which S k > a; if there is o such umber we set N(!) =. We observe the equivalece of evets,! max 1 k S k > a =! S N(!) > a,
11 Iequalities 19 ad from the Markov iequality P max 1 k S k > a 1 a E[S N]. We eed to estimate the right had side. If we could replace E[S N] by E[S ] = Var[S ] = i, the we would be doe. The trick is to show that E[S N ] E[S ] by usig coditioal expectatios. If E[S NN = k] E[S N = k] for all 1 k the the iequality holds, sice we have the a iequality betwee radom variables E[S N N] E[S N], ad applyig expectatios o both sides gives the desired result. For k =, the idetity hold trivially. Otherwise, we write E[S NN = ] = E[S N = ], E[S N = k] = E[S kn = k] + E[(X k+1 + +X ) N = k] + E[S k (X k+1 + +X )N = k] The first term o the right had side equals E[S NN = k], whereas the secod terms is o-egative. Remais the third term for which we remark that X k+1 + +X is idepedet of both S k ad N, ad by the previous lemma, E[S k (X k+1 + +X )N = k] = E[S k N = k] E[X k+1 + +X ] = 0. Puttig it all together, E[S N = k] E[S NN = k]. Sice this holds for all k s we have thus show that E[S N] E[S ] = i, which completes the proof.
Learning Theory: Lecture Notes
Learig Theory: Lecture Notes Kamalika Chaudhuri October 4, 0 Cocetratio of Averages Cocetratio of measure is very useful i showig bouds o the errors of machie-learig algorithms. We will begi with a basic
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationConvergence of random variables. (telegram style notes) P.J.C. Spreij
Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space
More information7.1 Convergence of sequences of random variables
Chapter 7 Limit theorems Throughout this sectio we will assume a probability space (Ω, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 19 11/17/2008 LAWS OF LARGE NUMBERS II THE STRONG LAW OF LARGE NUMBERS
MASSACHUSTTS INSTITUT OF TCHNOLOGY 6.436J/5.085J Fall 2008 Lecture 9 /7/2008 LAWS OF LARG NUMBRS II Cotets. The strog law of large umbers 2. The Cheroff boud TH STRONG LAW OF LARG NUMBRS While the weak
More informationDistribution of Random Samples & Limit theorems
STAT/MATH 395 A - PROBABILITY II UW Witer Quarter 2017 Néhémy Lim Distributio of Radom Samples & Limit theorems 1 Distributio of i.i.d. Samples Motivatig example. Assume that the goal of a study is to
More informationEE 4TM4: Digital Communications II Probability Theory
1 EE 4TM4: Digital Commuicatios II Probability Theory I. RANDOM VARIABLES A radom variable is a real-valued fuctio defied o the sample space. Example: Suppose that our experimet cosists of tossig two fair
More informationSTAT Homework 1 - Solutions
STAT-36700 Homework 1 - Solutios Fall 018 September 11, 018 This cotais solutios for Homework 1. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better
More informationLecture 2: Concentration Bounds
CSE 52: Desig ad Aalysis of Algorithms I Sprig 206 Lecture 2: Cocetratio Bouds Lecturer: Shaya Oveis Ghara March 30th Scribe: Syuzaa Sargsya Disclaimer: These otes have ot bee subjected to the usual scrutiy
More informationThis section is optional.
4 Momet Geeratig Fuctios* This sectio is optioal. The momet geeratig fuctio g : R R of a radom variable X is defied as g(t) = E[e tx ]. Propositio 1. We have g () (0) = E[X ] for = 1, 2,... Proof. Therefore
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 3 9/11/2013. Large deviations Theory. Cramér s Theorem
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/5.070J Fall 203 Lecture 3 9//203 Large deviatios Theory. Cramér s Theorem Cotet.. Cramér s Theorem. 2. Rate fuctio ad properties. 3. Chage of measure techique.
More informationAn Introduction to Randomized Algorithms
A Itroductio to Radomized Algorithms The focus of this lecture is to study a radomized algorithm for quick sort, aalyze it usig probabilistic recurrece relatios, ad also provide more geeral tools for aalysis
More informationMath 525: Lecture 5. January 18, 2018
Math 525: Lecture 5 Jauary 18, 2018 1 Series (review) Defiitio 1.1. A sequece (a ) R coverges to a poit L R (writte a L or lim a = L) if for each ǫ > 0, we ca fid N such that a L < ǫ for all N. If the
More informationn outcome is (+1,+1, 1,..., 1). Let the r.v. X denote our position (relative to our starting point 0) after n moves. Thus X = X 1 + X 2 + +X n,
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 9 Variace Questio: At each time step, I flip a fair coi. If it comes up Heads, I walk oe step to the right; if it comes up Tails, I walk oe
More informationLecture 7: Properties of Random Samples
Lecture 7: Properties of Radom Samples 1 Cotiued From Last Class Theorem 1.1. Let X 1, X,...X be a radom sample from a populatio with mea µ ad variace σ
More informationProbability for mathematicians INDEPENDENCE TAU
Probability for mathematicias INDEPENDENCE TAU 2013 28 Cotets 3 Ifiite idepedet sequeces 28 3a Idepedet evets........................ 28 3b Idepedet radom variables.................. 33 3 Ifiite idepedet
More informationDiscrete Mathematics for CS Spring 2007 Luca Trevisan Lecture 22
CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first
More informationSTAT Homework 2 - Solutions
STAT-36700 Homework - Solutios Fall 08 September 4, 08 This cotais solutios for Homework. Please ote that we have icluded several additioal commets ad approaches to the problems to give you better isight.
More informationThis exam contains 19 pages (including this cover page) and 10 questions. A Formulae sheet is provided with the exam.
Probability ad Statistics FS 07 Secod Sessio Exam 09.0.08 Time Limit: 80 Miutes Name: Studet ID: This exam cotais 9 pages (icludig this cover page) ad 0 questios. A Formulae sheet is provided with the
More informationThe standard deviation of the mean
Physics 6C Fall 20 The stadard deviatio of the mea These otes provide some clarificatio o the distictio betwee the stadard deviatio ad the stadard deviatio of the mea.. The sample mea ad variace Cosider
More information1 = δ2 (0, ), Y Y n nδ. , T n = Y Y n n. ( U n,k + X ) ( f U n,k + Y ) n 2n f U n,k + θ Y ) 2 E X1 2 X1
8. The cetral limit theorems 8.1. The cetral limit theorem for i.i.d. sequeces. ecall that C ( is N -separatig. Theorem 8.1. Let X 1, X,... be i.i.d. radom variables with EX 1 = ad EX 1 = σ (,. Suppose
More informationFACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING. Lectures
FACULTY OF MATHEMATICAL STUDIES MATHEMATICS FOR PART I ENGINEERING Lectures MODULE 5 STATISTICS II. Mea ad stadard error of sample data. Biomial distributio. Normal distributio 4. Samplig 5. Cofidece itervals
More informationCS166 Handout 02 Spring 2018 April 3, 2018 Mathematical Terms and Identities
CS166 Hadout 02 Sprig 2018 April 3, 2018 Mathematical Terms ad Idetities Thaks to Ady Nguye ad Julie Tibshirai for their advice o this hadout. This hadout covers mathematical otatio ad idetities that may
More informationRandomized Algorithms I, Spring 2018, Department of Computer Science, University of Helsinki Homework 1: Solutions (Discussed January 25, 2018)
Radomized Algorithms I, Sprig 08, Departmet of Computer Sciece, Uiversity of Helsiki Homework : Solutios Discussed Jauary 5, 08). Exercise.: Cosider the followig balls-ad-bi game. We start with oe black
More informationEcon 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.
Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio
More information4. Partial Sums and the Central Limit Theorem
1 of 10 7/16/2009 6:05 AM Virtual Laboratories > 6. Radom Samples > 1 2 3 4 5 6 7 4. Partial Sums ad the Cetral Limit Theorem The cetral limit theorem ad the law of large umbers are the two fudametal theorems
More informationLecture 01: the Central Limit Theorem. 1 Central Limit Theorem for i.i.d. random variables
CSCI-B609: A Theorist s Toolkit, Fall 06 Aug 3 Lecture 0: the Cetral Limit Theorem Lecturer: Yua Zhou Scribe: Yua Xie & Yua Zhou Cetral Limit Theorem for iid radom variables Let us say that we wat to aalyze
More information0, otherwise. EX = E(X 1 + X n ) = EX j = np and. Var(X j ) = np(1 p). Var(X) = Var(X X n ) =
PROBABILITY MODELS 35 10. Discrete probability distributios I this sectio, we discuss several well-ow discrete probability distributios ad study some of their properties. Some of these distributios, lie
More informationGlivenko-Cantelli Classes
CS28B/Stat24B (Sprig 2008 Statistical Learig Theory Lecture: 4 Gliveko-Catelli Classes Lecturer: Peter Bartlett Scribe: Michelle Besi Itroductio This lecture will cover Gliveko-Catelli (GC classes ad itroduce
More information1 Convergence in Probability and the Weak Law of Large Numbers
36-752 Advaced Probability Overview Sprig 2018 8. Covergece Cocepts: i Probability, i L p ad Almost Surely Istructor: Alessadro Rialdo Associated readig: Sec 2.4, 2.5, ad 4.11 of Ash ad Doléas-Dade; Sec
More informationLecture 19: Convergence
Lecture 19: Covergece Asymptotic approach I statistical aalysis or iferece, a key to the success of fidig a good procedure is beig able to fid some momets ad/or distributios of various statistics. I may
More informationProblems from 9th edition of Probability and Statistical Inference by Hogg, Tanis and Zimmerman:
Math 224 Fall 2017 Homework 4 Drew Armstrog Problems from 9th editio of Probability ad Statistical Iferece by Hogg, Tais ad Zimmerma: Sectio 2.3, Exercises 16(a,d),18. Sectio 2.4, Exercises 13, 14. Sectio
More informationInfinite Sequences and Series
Chapter 6 Ifiite Sequeces ad Series 6.1 Ifiite Sequeces 6.1.1 Elemetary Cocepts Simply speakig, a sequece is a ordered list of umbers writte: {a 1, a 2, a 3,...a, a +1,...} where the elemets a i represet
More informationLecture 3 : Random variables and their distributions
Lecture 3 : Radom variables ad their distributios 3.1 Radom variables Let (Ω, F) ad (S, S) be two measurable spaces. A map X : Ω S is measurable or a radom variable (deoted r.v.) if X 1 (A) {ω : X(ω) A}
More informationSequences and Series of Functions
Chapter 6 Sequeces ad Series of Fuctios 6.1. Covergece of a Sequece of Fuctios Poitwise Covergece. Defiitio 6.1. Let, for each N, fuctio f : A R be defied. If, for each x A, the sequece (f (x)) coverges
More informationNotes 19 : Martingale CLT
Notes 9 : Martigale CLT Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: [Bil95, Chapter 35], [Roc, Chapter 3]. Sice we have ot ecoutered weak covergece i some time, we first recall
More informationLecture 8: Convergence of transformations and law of large numbers
Lecture 8: Covergece of trasformatios ad law of large umbers Trasformatio ad covergece Trasformatio is a importat tool i statistics. If X coverges to X i some sese, we ofte eed to check whether g(x ) coverges
More informationProbability 2 - Notes 10. Lemma. If X is a random variable and g(x) 0 for all x in the support of f X, then P(g(X) 1) E[g(X)].
Probability 2 - Notes 0 Some Useful Iequalities. Lemma. If X is a radom variable ad g(x 0 for all x i the support of f X, the P(g(X E[g(X]. Proof. (cotiuous case P(g(X Corollaries x:g(x f X (xdx x:g(x
More informationECE 330:541, Stochastic Signals and Systems Lecture Notes on Limit Theorems from Probability Fall 2002
ECE 330:541, Stochastic Sigals ad Systems Lecture Notes o Limit Theorems from robability Fall 00 I practice, there are two ways we ca costruct a ew sequece of radom variables from a old sequece of radom
More informationCEE 522 Autumn Uncertainty Concepts for Geotechnical Engineering
CEE 5 Autum 005 Ucertaity Cocepts for Geotechical Egieerig Basic Termiology Set A set is a collectio of (mutually exclusive) objects or evets. The sample space is the (collectively exhaustive) collectio
More informationCS 330 Discussion - Probability
CS 330 Discussio - Probability March 24 2017 1 Fudametals of Probability 11 Radom Variables ad Evets A radom variable X is oe whose value is o-determiistic For example, suppose we flip a coi ad set X =
More informationJanuary 25, 2017 INTRODUCTION TO MATHEMATICAL STATISTICS
Jauary 25, 207 INTRODUCTION TO MATHEMATICAL STATISTICS Abstract. A basic itroductio to statistics assumig kowledge of probability theory.. Probability I a typical udergraduate problem i probability, we
More informationLet us give one more example of MLE. Example 3. The uniform distribution U[0, θ] on the interval [0, θ] has p.d.f.
Lecture 5 Let us give oe more example of MLE. Example 3. The uiform distributio U[0, ] o the iterval [0, ] has p.d.f. { 1 f(x =, 0 x, 0, otherwise The likelihood fuctio ϕ( = f(x i = 1 I(X 1,..., X [0,
More information2.2. Central limit theorem.
36.. Cetral limit theorem. The most ideal case of the CLT is that the radom variables are iid with fiite variace. Although it is a special case of the more geeral Lideberg-Feller CLT, it is most stadard
More informationLecture 12: November 13, 2018
Mathematical Toolkit Autum 2018 Lecturer: Madhur Tulsiai Lecture 12: November 13, 2018 1 Radomized polyomial idetity testig We will use our kowledge of coditioal probability to prove the followig lemma,
More informationProblem Set 2 Solutions
CS271 Radomess & Computatio, Sprig 2018 Problem Set 2 Solutios Poit totals are i the margi; the maximum total umber of poits was 52. 1. Probabilistic method for domiatig sets 6pts Pick a radom subset S
More informationST5215: Advanced Statistical Theory
ST525: Advaced Statistical Theory Departmet of Statistics & Applied Probability Tuesday, September 7, 2 ST525: Advaced Statistical Theory Lecture : The law of large umbers The Law of Large Numbers The
More informationRademacher Complexity
EECS 598: Statistical Learig Theory, Witer 204 Topic 0 Rademacher Complexity Lecturer: Clayto Scott Scribe: Ya Deg, Kevi Moo Disclaimer: These otes have ot bee subjected to the usual scrutiy reserved for
More informationDiscrete Mathematics and Probability Theory Summer 2014 James Cook Note 15
CS 70 Discrete Mathematics ad Probability Theory Summer 2014 James Cook Note 15 Some Importat Distributios I this ote we will itroduce three importat probability distributios that are widely used to model
More informationMath 216A Notes, Week 5
Math 6A Notes, Week 5 Scribe: Ayastassia Sebolt Disclaimer: These otes are ot early as polished (ad quite possibly ot early as correct) as a published paper. Please use them at your ow risk.. Thresholds
More informationLecture 2: April 3, 2013
TTIC/CMSC 350 Mathematical Toolkit Sprig 203 Madhur Tulsiai Lecture 2: April 3, 203 Scribe: Shubhedu Trivedi Coi tosses cotiued We retur to the coi tossig example from the last lecture agai: Example. Give,
More informationBasics of Probability Theory (for Theory of Computation courses)
Basics of Probability Theory (for Theory of Computatio courses) Oded Goldreich Departmet of Computer Sciece Weizma Istitute of Sciece Rehovot, Israel. oded.goldreich@weizma.ac.il November 24, 2008 Preface.
More informationChapter 3. Strong convergence. 3.1 Definition of almost sure convergence
Chapter 3 Strog covergece As poited out i the Chapter 2, there are multiple ways to defie the otio of covergece of a sequece of radom variables. That chapter defied covergece i probability, covergece i
More informationNotes 5 : More on the a.s. convergence of sums
Notes 5 : More o the a.s. covergece of sums Math 733-734: Theory of Probability Lecturer: Sebastie Roch Refereces: Dur0, Sectios.5; Wil9, Sectio 4.7, Shi96, Sectio IV.4, Dur0, Sectio.. Radom series. Three-series
More informationDiscrete Mathematics for CS Spring 2005 Clancy/Wagner Notes 21. Some Important Distributions
CS 70 Discrete Mathematics for CS Sprig 2005 Clacy/Wager Notes 21 Some Importat Distributios Questio: A biased coi with Heads probability p is tossed repeatedly util the first Head appears. What is the
More informationApproximations and more PMFs and PDFs
Approximatios ad more PMFs ad PDFs Saad Meimeh 1 Approximatio of biomial with Poisso Cosider the biomial distributio ( b(k,,p = p k (1 p k, k λ: k Assume that is large, ad p is small, but p λ at the limit.
More informationChapter 6 Infinite Series
Chapter 6 Ifiite Series I the previous chapter we cosidered itegrals which were improper i the sese that the iterval of itegratio was ubouded. I this chapter we are goig to discuss a topic which is somewhat
More informationLECTURE 8: ASYMPTOTICS I
LECTURE 8: ASYMPTOTICS I We are iterested i the properties of estimators as. Cosider a sequece of radom variables {, X 1}. N. M. Kiefer, Corell Uiversity, Ecoomics 60 1 Defiitio: (Weak covergece) A sequece
More informationApplication to Random Graphs
A Applicatio to Radom Graphs Brachig processes have a umber of iterestig ad importat applicatios. We shall cosider oe of the most famous of them, the Erdős-Réyi radom graph theory. 1 Defiitio A.1. Let
More informationOn Random Line Segments in the Unit Square
O Radom Lie Segmets i the Uit Square Thomas A. Courtade Departmet of Electrical Egieerig Uiversity of Califoria Los Ageles, Califoria 90095 Email: tacourta@ee.ucla.edu I. INTRODUCTION Let Q = [0, 1] [0,
More informationAdvanced Stochastic Processes.
Advaced Stochastic Processes. David Gamarik LECTURE 2 Radom variables ad measurable fuctios. Strog Law of Large Numbers (SLLN). Scary stuff cotiued... Outlie of Lecture Radom variables ad measurable fuctios.
More informationIIT JAM Mathematical Statistics (MS) 2006 SECTION A
IIT JAM Mathematical Statistics (MS) 6 SECTION A. If a > for ad lim a / L >, the which of the followig series is ot coverget? (a) (b) (c) (d) (d) = = a = a = a a + / a lim a a / + = lim a / a / + = lim
More informationParameter, Statistic and Random Samples
Parameter, Statistic ad Radom Samples A parameter is a umber that describes the populatio. It is a fixed umber, but i practice we do ot kow its value. A statistic is a fuctio of the sample data, i.e.,
More information4.1 Sigma Notation and Riemann Sums
0 the itegral. Sigma Notatio ad Riema Sums Oe strategy for calculatig the area of a regio is to cut the regio ito simple shapes, calculate the area of each simple shape, ad the add these smaller areas
More informationProbability and Random Processes
Probability ad Radom Processes Lecture 5 Probability ad radom variables The law of large umbers Mikael Skoglud, Probability ad radom processes 1/21 Why Measure Theoretic Probability? Stroger limit theorems
More informationTopic 8: Expected Values
Topic 8: Jue 6, 20 The simplest summary of quatitative data is the sample mea. Give a radom variable, the correspodig cocept is called the distributioal mea, the epectatio or the epected value. We begi
More informationLecture 3 The Lebesgue Integral
Lecture 3: The Lebesgue Itegral 1 of 14 Course: Theory of Probability I Term: Fall 2013 Istructor: Gorda Zitkovic Lecture 3 The Lebesgue Itegral The costructio of the itegral Uless expressly specified
More informationParameter, Statistic and Random Samples
Parameter, Statistic ad Radom Samples A parameter is a umber that describes the populatio. It is a fixed umber, but i practice we do ot kow its value. A statistic is a fuctio of the sample data, i.e.,
More informationEstimation for Complete Data
Estimatio for Complete Data complete data: there is o loss of iformatio durig study. complete idividual complete data= grouped data A complete idividual data is the oe i which the complete iformatio of
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013. Large Deviations for i.i.d. Random Variables
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 2 9/9/2013 Large Deviatios for i.i.d. Radom Variables Cotet. Cheroff boud usig expoetial momet geeratig fuctios. Properties of a momet
More informationEECS564 Estimation, Filtering, and Detection Hwk 2 Solns. Winter p θ (z) = (2θz + 1 θ), 0 z 1
EECS564 Estimatio, Filterig, ad Detectio Hwk 2 Sols. Witer 25 4. Let Z be a sigle observatio havig desity fuctio where. p (z) = (2z + ), z (a) Assumig that is a oradom parameter, fid ad plot the maximum
More informationLecture 12: September 27
36-705: Itermediate Statistics Fall 207 Lecturer: Siva Balakrisha Lecture 2: September 27 Today we will discuss sufficiecy i more detail ad the begi to discuss some geeral strategies for costructig estimators.
More informationEcon 325: Introduction to Empirical Economics
Eco 35: Itroductio to Empirical Ecoomics Lecture 3 Discrete Radom Variables ad Probability Distributios Copyright 010 Pearso Educatio, Ic. Publishig as Pretice Hall Ch. 4-1 4.1 Itroductio to Probability
More informationf X (12) = Pr(X = 12) = Pr({(6, 6)}) = 1/36
Probability Distributios A Example With Dice If X is a radom variable o sample space S, the the probablity that X takes o the value c is Similarly, Pr(X = c) = Pr({s S X(s) = c} Pr(X c) = Pr({s S X(s)
More informationProduct measures, Tonelli s and Fubini s theorems For use in MAT3400/4400, autumn 2014 Nadia S. Larsen. Version of 13 October 2014.
Product measures, Toelli s ad Fubii s theorems For use i MAT3400/4400, autum 2014 Nadia S. Larse Versio of 13 October 2014. 1. Costructio of the product measure The purpose of these otes is to preset the
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.265/15.070J Fall 2013 Lecture 21 11/27/2013 Fuctioal Law of Large Numbers. Costructio of the Wieer Measure Cotet. 1. Additioal techical results o weak covergece
More information6.3 Testing Series With Positive Terms
6.3. TESTING SERIES WITH POSITIVE TERMS 307 6.3 Testig Series With Positive Terms 6.3. Review of what is kow up to ow I theory, testig a series a i for covergece amouts to fidig the i= sequece of partial
More informationLinear regression. Daniel Hsu (COMS 4771) (y i x T i β)2 2πσ. 2 2σ 2. 1 n. (x T i β y i ) 2. 1 ˆβ arg min. β R n d
Liear regressio Daiel Hsu (COMS 477) Maximum likelihood estimatio Oe of the simplest liear regressio models is the followig: (X, Y ),..., (X, Y ), (X, Y ) are iid radom pairs takig values i R d R, ad Y
More informationSelf-normalized deviation inequalities with application to t-statistic
Self-ormalized deviatio iequalities with applicatio to t-statistic Xiequa Fa Ceter for Applied Mathematics, Tiaji Uiversity, 30007 Tiaji, Chia Abstract Let ξ i i 1 be a sequece of idepedet ad symmetric
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 6 9/24/2008 DISCRETE RANDOM VARIABLES AND THEIR EXPECTATIONS Cotets 1. A few useful discrete radom variables 2. Joit, margial, ad
More informationEcon 325 Notes on Point Estimator and Confidence Interval 1 By Hiro Kasahara
Poit Estimator Eco 325 Notes o Poit Estimator ad Cofidece Iterval 1 By Hiro Kasahara Parameter, Estimator, ad Estimate The ormal probability desity fuctio is fully characterized by two costats: populatio
More informationBertrand s Postulate
Bertrad s Postulate Lola Thompso Ross Program July 3, 2009 Lola Thompso (Ross Program Bertrad s Postulate July 3, 2009 1 / 33 Bertrad s Postulate I ve said it oce ad I ll say it agai: There s always a
More informationBertrand s Postulate. Theorem (Bertrand s Postulate): For every positive integer n, there is a prime p satisfying n < p 2n.
Bertrad s Postulate Our goal is to prove the followig Theorem Bertrad s Postulate: For every positive iteger, there is a prime p satisfyig < p We remark that Bertrad s Postulate is true by ispectio for,,
More informationCS161 Handout 05 Summer 2013 July 10, 2013 Mathematical Terms and Identities
CS161 Hadout 05 Summer 2013 July 10, 2013 Mathematical Terms ad Idetities Thaks to Ady Nguye ad Julie Tibshirai for their advice o this hadout. This hadout covers mathematical otatio ad idetities that
More informationBinomial Distribution
0.0 0.5 1.0 1.5 2.0 2.5 3.0 0 1 2 3 4 5 6 7 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Overview Example: coi tossed three times Defiitio Formula Recall that a r.v. is discrete if there are either a fiite umber of possible
More informationMathematics 170B Selected HW Solutions.
Mathematics 17B Selected HW Solutios. F 4. Suppose X is B(,p). (a)fidthemometgeeratigfuctiom (s)of(x p)/ p(1 p). Write q = 1 p. The MGF of X is (pe s + q), sice X ca be writte as the sum of idepedet Beroulli
More informationLecture 3: August 31
36-705: Itermediate Statistics Fall 018 Lecturer: Siva Balakrisha Lecture 3: August 31 This lecture will be mostly a summary of other useful expoetial tail bouds We will ot prove ay of these i lecture,
More informationBHW #13 1/ Cooper. ENGR 323 Probabilistic Analysis Beautiful Homework # 13
BHW # /5 ENGR Probabilistic Aalysis Beautiful Homework # Three differet roads feed ito a particular freeway etrace. Suppose that durig a fixed time period, the umber of cars comig from each road oto the
More informationAgnostic Learning and Concentration Inequalities
ECE901 Sprig 2004 Statistical Regularizatio ad Learig Theory Lecture: 7 Agostic Learig ad Cocetratio Iequalities Lecturer: Rob Nowak Scribe: Aravid Kailas 1 Itroductio 1.1 Motivatio I the last lecture
More informationECE 901 Lecture 12: Complexity Regularization and the Squared Loss
ECE 90 Lecture : Complexity Regularizatio ad the Squared Loss R. Nowak 5/7/009 I the previous lectures we made use of the Cheroff/Hoeffdig bouds for our aalysis of classifier errors. Hoeffdig s iequality
More informationAMS570 Lecture Notes #2
AMS570 Lecture Notes # Review of Probability (cotiued) Probability distributios. () Biomial distributio Biomial Experimet: ) It cosists of trials ) Each trial results i of possible outcomes, S or F 3)
More informationDiscrete Mathematics for CS Spring 2008 David Wagner Note 22
CS 70 Discrete Mathematics for CS Sprig 2008 David Wager Note 22 I.I.D. Radom Variables Estimatig the bias of a coi Questio: We wat to estimate the proportio p of Democrats i the US populatio, by takig
More informations = and t = with C ij = A i B j F. (i) Note that cs = M and so ca i µ(a i ) I E (cs) = = c a i µ(a i ) = ci E (s). (ii) Note that s + t = M and so
3 From the otes we see that the parts of Theorem 4. that cocer us are: Let s ad t be two simple o-egative F-measurable fuctios o X, F, µ ad E, F F. The i I E cs ci E s for all c R, ii I E s + t I E s +
More informationThe variance of a sum of independent variables is the sum of their variances, since covariances are zero. Therefore. V (xi )= n n 2 σ2 = σ2.
SAMPLE STATISTICS A radom sample x 1,x,,x from a distributio f(x) is a set of idepedetly ad idetically variables with x i f(x) for all i Their joit pdf is f(x 1,x,,x )=f(x 1 )f(x ) f(x )= f(x i ) The sample
More informationMassachusetts Institute of Technology
Solutios to Quiz : Sprig 006 Problem : Each of the followig statemets is either True or False. There will be o partial credit give for the True False questios, thus ay explaatios will ot be graded. Please
More informationWe are mainly going to be concerned with power series in x, such as. (x)} converges - that is, lims N n
Review of Power Series, Power Series Solutios A power series i x - a is a ifiite series of the form c (x a) =c +c (x a)+(x a) +... We also call this a power series cetered at a. Ex. (x+) is cetered at
More informationLinear Regression Demystified
Liear Regressio Demystified Liear regressio is a importat subject i statistics. I elemetary statistics courses, formulae related to liear regressio are ofte stated without derivatio. This ote iteds to
More informationLimit Theorems. Convergence in Probability. Let X be the number of heads observed in n tosses. Then, E[X] = np and Var[X] = np(1-p).
Limit Theorems Covergece i Probability Let X be the umber of heads observed i tosses. The, E[X] = p ad Var[X] = p(-p). L O This P x p NM QP P x p should be close to uity for large if our ituitio is correct.
More informationDiscrete Mathematics and Probability Theory Spring 2012 Alistair Sinclair Note 15
CS 70 Discrete Mathematics ad Probability Theory Sprig 2012 Alistair Siclair Note 15 Some Importat Distributios The first importat distributio we leared about i the last Lecture Note is the biomial distributio
More informationMath 299 Supplement: Real Analysis Nov 2013
Math 299 Supplemet: Real Aalysis Nov 203 Algebra Axioms. I Real Aalysis, we work withi the axiomatic system of real umbers: the set R alog with the additio ad multiplicatio operatios +,, ad the iequality
More information